2024-05-15T22:40:51.698 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-15T22:40:51.699 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754 --owner scheduled_yuriw@teuthology --name yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} -- /home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754/orig.config.yaml 2024-05-15T22:40:51.743 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-15T22:40:51.823 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754 branch: wip-yuri5-testing-2024-05-15-0804 description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} email: yweinste@redhat.com first_in_suite: false job_id: '7707754' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri5-testing-2024-05-15-0804 ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd heartbeat use min delay socket: true osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - osd_map_cache_size - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum mon_bind_msgr2: false sha1: 6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri5-testing-2024-05-15-0804 sha1: 6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 6651 sha1: 6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri5-testing-2024-05-15-0804 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_6cd0f8013e2dea00c3a29a0d8b10656b132d7c80/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 targets: smithi084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK9Nf4cHWUuLl3T7vPa3UepgO9fpmPaqZFc3jDJWWFvigOE6SkIM8vIpDYoXQqZHWdFJb7CUlq6DSbADMftJz+M= smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNpKE5Nb7ypAPxSuFMWe0Q/sPYGfgEw475HpNeOGwj8Vrg7jwUodI/qj+KqvcUcPn00OO479Zk1hcgfbtfztS5g= smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKVJiiXiSPIa49WcVsk5RBTt8Yy+lKpFPg7I+1ipH/OgvGENv9W5uCOj4qcxk+NZJSuHPWyqc1PfUMkCqacxr0k= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: nautilus exclude_packages: - cephadm - ceph-mgr-cephadm - ceph-immutable-object-cache - python3-rados - python3-rgw - python3-rbd - python3-cephfs - ceph-volume extra_packages: - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: conf: mon: auth allow insecure global id reclaim: true - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: chance_pgnum_grow: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 timeout: 1800 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-15_21:09:29 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-15T22:40:51.824 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_6cd0f8013e2dea00c3a29a0d8b10656b132d7c80/qa; will attempt to use it 2024-05-15T22:40:51.824 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_6cd0f8013e2dea00c3a29a0d8b10656b132d7c80/qa/tasks 2024-05-15T22:40:51.825 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-15T22:40:51.826 INFO:teuthology.task.internal:Saving configuration 2024-05-15T22:40:51.843 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-15T22:40:51.849 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T22:40:51.876 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 22:35:06.053221', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK9Nf4cHWUuLl3T7vPa3UepgO9fpmPaqZFc3jDJWWFvigOE6SkIM8vIpDYoXQqZHWdFJb7CUlq6DSbADMftJz+M='} 2024-05-15T22:40:51.901 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 22:35:06.055009', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNpKE5Nb7ypAPxSuFMWe0Q/sPYGfgEw475HpNeOGwj8Vrg7jwUodI/qj+KqvcUcPn00OO479Zk1hcgfbtfztS5g='} 2024-05-15T22:40:51.922 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 22:35:06.051182', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKVJiiXiSPIa49WcVsk5RBTt8Yy+lKpFPg7I+1ipH/OgvGENv9W5uCOj4qcxk+NZJSuHPWyqc1PfUMkCqacxr0k='} 2024-05-15T22:40:51.922 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-15T22:40:51.927 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-05-15T22:40:51.928 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-05-15T22:40:51.928 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-05-15T22:40:51.928 INFO:teuthology.run_tasks:Running task console_log... 2024-05-15T22:40:52.037 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1ef970af70>, signals=[15]) 2024-05-15T22:40:52.037 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-15T22:40:52.043 INFO:teuthology.task.internal:Opening connections... 2024-05-15T22:40:52.043 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi084.front.sepia.ceph.com 2024-05-15T22:40:52.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:40:52.120 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2024-05-15T22:40:52.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:40:52.194 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-05-15T22:40:52.194 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:40:52.267 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-15T22:40:52.273 DEBUG:teuthology.orchestra.run.smithi084:> uname -m 2024-05-15T22:40:52.297 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2024-05-15T22:40:52.297 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2024-05-15T22:40:52.358 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Stream" 2024-05-15T22:40:52.358 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="8" 2024-05-15T22:40:52.358 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos" 2024-05-15T22:40:52.358 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora" 2024-05-15T22:40:52.358 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="8" 2024-05-15T22:40:52.358 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el8" 2024-05-15T22:40:52.359 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Stream 8" 2024-05-15T22:40:52.359 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2024-05-15T22:40:52.359 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-05-15T22:40:52.359 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://centos.org/" 2024-05-15T22:40:52.359 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-15T22:40:52.359 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-05-15T22:40:52.359 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-15T22:40:52.359 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2024-05-15T22:40:52.388 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2024-05-15T22:40:52.411 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2024-05-15T22:40:52.411 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2024-05-15T22:40:52.472 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream" 2024-05-15T22:40:52.472 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="8" 2024-05-15T22:40:52.472 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos" 2024-05-15T22:40:52.472 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora" 2024-05-15T22:40:52.473 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="8" 2024-05-15T22:40:52.473 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el8" 2024-05-15T22:40:52.473 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 8" 2024-05-15T22:40:52.473 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2024-05-15T22:40:52.473 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-05-15T22:40:52.473 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/" 2024-05-15T22:40:52.473 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-15T22:40:52.473 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-05-15T22:40:52.473 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-15T22:40:52.473 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2024-05-15T22:40:52.496 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-05-15T22:40:52.519 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-05-15T22:40:52.519 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-05-15T22:40:52.580 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2024-05-15T22:40:52.581 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="8" 2024-05-15T22:40:52.581 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2024-05-15T22:40:52.581 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2024-05-15T22:40:52.581 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="8" 2024-05-15T22:40:52.581 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el8" 2024-05-15T22:40:52.581 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 8" 2024-05-15T22:40:52.581 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2024-05-15T22:40:52.581 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-05-15T22:40:52.581 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2024-05-15T22:40:52.581 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-15T22:40:52.581 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-05-15T22:40:52.581 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-15T22:40:52.582 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-05-15T22:40:52.604 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-15T22:40:52.614 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-15T22:40:52.619 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-15T22:40:52.619 DEBUG:teuthology.orchestra.run.smithi084:> test '!' -e /home/ubuntu/cephtest 2024-05-15T22:40:52.621 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2024-05-15T22:40:52.624 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-05-15T22:40:52.645 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-15T22:40:52.650 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-15T22:40:52.650 DEBUG:teuthology.orchestra.run.smithi084:> test -z $(ls -A /var/lib/ceph) 2024-05-15T22:40:52.683 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2024-05-15T22:40:52.685 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-05-15T22:40:52.714 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-15T22:40:52.785 INFO:teuthology.run_tasks:Running task kernel... 2024-05-15T22:40:52.799 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-15T22:40:52.799 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-15T22:40:52.800 DEBUG:teuthology.orchestra.run.smithi084:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-15T22:40:52.800 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-15T22:40:52.801 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-15T22:40:52.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T22:40:52.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T22:40:52.823 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-05-15T22:40:52.823 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-05-15T22:40:52.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T22:40:52.824 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-05-15T22:40:52.883 INFO:teuthology.orchestra.run.smithi088.stdout:4.18.0-489.el8.x86_64 2024-05-15T22:40:52.883 INFO:teuthology.task.kernel:Running kernel on smithi088: 4.18.0-489.el8.x86_64 2024-05-15T22:40:52.883 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum install -y kernel 2024-05-15T22:40:52.884 INFO:teuthology.orchestra.run.smithi084.stdout:4.18.0-489.el8.x86_64 2024-05-15T22:40:52.884 INFO:teuthology.task.kernel:Running kernel on smithi084: 4.18.0-489.el8.x86_64 2024-05-15T22:40:52.885 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum install -y kernel 2024-05-15T22:40:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:4.18.0-489.el8.x86_64 2024-05-15T22:40:52.887 INFO:teuthology.task.kernel:Running kernel on smithi090: 4.18.0-489.el8.x86_64 2024-05-15T22:40:52.887 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum install -y kernel 2024-05-15T22:40:57.377 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - AppStream 14 MB/s | 28 MB 00:01 2024-05-15T22:40:57.383 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - AppStream 15 MB/s | 28 MB 00:01 2024-05-15T22:40:58.157 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - AppStream 15 MB/s | 28 MB 00:01 2024-05-15T22:41:03.810 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - BaseOS 7.1 MB/s | 10 MB 00:01 2024-05-15T22:41:04.383 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - BaseOS 5.3 MB/s | 10 MB 00:01 2024-05-15T22:41:04.424 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - BaseOS 8.6 MB/s | 10 MB 00:01 2024-05-15T22:41:05.586 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - Extras 120 kB/s | 18 kB 00:00 2024-05-15T22:41:05.861 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - Extras common packages 46 kB/s | 7.7 kB 00:00 2024-05-15T22:41:06.079 INFO:teuthology.orchestra.run.smithi088.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-05-15T22:41:06.292 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - Extras 66 kB/s | 18 kB 00:00 2024-05-15T22:41:06.358 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - Extras 64 kB/s | 18 kB 00:00 2024-05-15T22:41:06.652 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - Extras common packages 41 kB/s | 7.7 kB 00:00 2024-05-15T22:41:06.701 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - Extras common packages 26 kB/s | 7.7 kB 00:00 2024-05-15T22:41:06.904 INFO:teuthology.orchestra.run.smithi090.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-05-15T22:41:06.919 INFO:teuthology.orchestra.run.smithi084.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-05-15T22:41:07.870 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 16 MB/s | 14 MB 00:00 2024-05-15T22:41:11.812 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 2.8 MB/s | 14 MB 00:04 2024-05-15T22:41:12.120 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 433 kB/s | 24 kB 00:00 2024-05-15T22:41:15.216 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-05-15T22:41:15.216 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-05-15T22:41:15.330 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:41:15.330 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:41:15.330 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-05-15T22:41:15.330 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:41:15.330 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-05-15T22:41:15.331 INFO:teuthology.orchestra.run.smithi084.stdout: kernel x86_64 4.18.0-552.1.1.el8 baseos 10 M 2024-05-15T22:41:15.331 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-05-15T22:41:15.331 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core x86_64 4.18.0-552.1.1.el8 baseos 44 M 2024-05-15T22:41:15.331 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules x86_64 4.18.0-552.1.1.el8 baseos 36 M 2024-05-15T22:41:15.331 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:41:15.331 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-05-15T22:41:15.331 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:41:15.331 INFO:teuthology.orchestra.run.smithi084.stdout:Install 3 Packages 2024-05-15T22:41:15.331 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:41:15.332 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 90 M 2024-05-15T22:41:15.332 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 96 M 2024-05-15T22:41:15.332 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-05-15T22:41:16.122 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 290 kB/s | 24 kB 00:00 2024-05-15T22:41:16.162 INFO:teuthology.orchestra.run.smithi084.stdout:(1/3): kernel-4.18.0-552.1.1.el8.x86_64.rpm 13 MB/s | 10 MB 00:00 2024-05-15T22:41:17.287 INFO:teuthology.orchestra.run.smithi084.stdout:(2/3): kernel-modules-4.18.0-552.1.1.el8.x86_64 19 MB/s | 36 MB 00:01 2024-05-15T22:41:17.887 INFO:teuthology.orchestra.run.smithi084.stdout:(3/3): kernel-core-4.18.0-552.1.1.el8.x86_64.rp 17 MB/s | 44 MB 00:02 2024-05-15T22:41:17.888 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:41:17.888 INFO:teuthology.orchestra.run.smithi084.stdout:Total 35 MB/s | 90 MB 00:02 2024-05-15T22:41:18.568 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-05-15T22:41:18.995 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-05-15T22:41:18.995 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-05-15T22:41:19.295 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-05-15T22:41:19.295 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-05-15T22:41:19.414 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:41:19.415 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:41:19.415 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-05-15T22:41:19.415 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:41:19.415 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-05-15T22:41:19.415 INFO:teuthology.orchestra.run.smithi090.stdout: kernel x86_64 4.18.0-552.1.1.el8 baseos 10 M 2024-05-15T22:41:19.415 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-05-15T22:41:19.415 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core x86_64 4.18.0-552.1.1.el8 baseos 44 M 2024-05-15T22:41:19.415 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules x86_64 4.18.0-552.1.1.el8 baseos 36 M 2024-05-15T22:41:19.415 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:41:19.415 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-05-15T22:41:19.415 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:41:19.415 INFO:teuthology.orchestra.run.smithi090.stdout:Install 3 Packages 2024-05-15T22:41:19.415 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:41:19.416 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 90 M 2024-05-15T22:41:19.416 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 96 M 2024-05-15T22:41:19.416 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-05-15T22:41:19.564 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 14 MB 00:13 2024-05-15T22:41:20.166 INFO:teuthology.orchestra.run.smithi090.stdout:(1/3): kernel-4.18.0-552.1.1.el8.x86_64.rpm 15 MB/s | 10 MB 00:00 2024-05-15T22:41:20.238 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-05-15T22:41:20.263 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-05-15T22:41:21.291 INFO:teuthology.orchestra.run.smithi090.stdout:(2/3): kernel-modules-4.18.0-552.1.1.el8.x86_64 20 MB/s | 36 MB 00:01 2024-05-15T22:41:21.809 INFO:teuthology.orchestra.run.smithi090.stdout:(3/3): kernel-core-4.18.0-552.1.1.el8.x86_64.rp 19 MB/s | 44 MB 00:02 2024-05-15T22:41:21.809 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:41:21.809 INFO:teuthology.orchestra.run.smithi090.stdout:Total 38 MB/s | 90 MB 00:02 2024-05-15T22:41:22.487 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-05-15T22:41:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-05-15T22:41:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-05-15T22:41:23.063 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-05-15T22:41:23.595 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T22:41:23.969 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 310 kB/s | 24 kB 00:00 2024-05-15T22:41:24.212 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-05-15T22:41:24.247 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-05-15T22:41:27.068 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-05-15T22:41:27.074 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-05-15T22:41:27.074 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-05-15T22:41:27.170 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T22:41:27.190 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:41:27.191 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:41:27.191 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-05-15T22:41:27.191 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:41:27.192 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T22:41:27.192 INFO:teuthology.orchestra.run.smithi088.stdout: kernel x86_64 4.18.0-552.1.1.el8 baseos 10 M 2024-05-15T22:41:27.192 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T22:41:27.192 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core x86_64 4.18.0-552.1.1.el8 baseos 44 M 2024-05-15T22:41:27.192 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules x86_64 4.18.0-552.1.1.el8 baseos 36 M 2024-05-15T22:41:27.192 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:41:27.192 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T22:41:27.192 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:41:27.193 INFO:teuthology.orchestra.run.smithi088.stdout:Install 3 Packages 2024-05-15T22:41:27.193 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:41:27.194 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 90 M 2024-05-15T22:41:27.194 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 96 M 2024-05-15T22:41:27.194 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T22:41:27.544 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T22:41:27.594 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T22:41:27.979 INFO:teuthology.orchestra.run.smithi088.stdout:(1/3): kernel-4.18.0-552.1.1.el8.x86_64.rpm 14 MB/s | 10 MB 00:00 2024-05-15T22:41:28.717 INFO:teuthology.orchestra.run.smithi088.stdout:(2/3): kernel-modules-4.18.0-552.1.1.el8.x86_64 25 MB/s | 36 MB 00:01 2024-05-15T22:41:29.379 INFO:teuthology.orchestra.run.smithi088.stdout:(3/3): kernel-core-4.18.0-552.1.1.el8.x86_64.rp 21 MB/s | 44 MB 00:02 2024-05-15T22:41:29.379 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:41:29.379 INFO:teuthology.orchestra.run.smithi088.stdout:Total 41 MB/s | 90 MB 00:02 2024-05-15T22:41:30.071 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T22:41:30.496 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T22:41:30.496 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T22:41:30.981 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T22:41:31.149 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T22:41:31.285 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T22:41:31.618 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T22:41:31.738 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T22:41:31.754 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T22:41:34.539 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T22:41:35.017 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T22:41:35.062 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T22:41:35.392 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T22:41:38.558 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T22:41:38.949 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T22:41:42.376 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T22:41:42.672 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T22:41:53.098 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T22:41:53.122 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T22:41:53.486 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T22:41:53.486 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T22:41:53.486 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-core-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T22:41:53.972 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T22:41:53.972 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:41:53.972 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-05-15T22:41:53.972 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:53.972 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:53.973 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:53.973 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:41:53.973 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:41:54.644 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T22:41:54.858 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:54.859 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:54.859 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:54.859 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-15T22:41:54.859 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-15T22:41:54.859 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-15T22:41:54.859 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi084.front.sepia.ceph.com, path=None, version=distro) 2024-05-15T22:41:54.860 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T22:41:55.068 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:55.068 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:55.068 DEBUG:teuthology.orchestra.run.smithi084:> sudo rpm -qi grub2-tools 2024-05-15T22:41:55.102 INFO:teuthology.orchestra.run.smithi084.stdout:Name : grub2-tools 2024-05-15T22:41:55.102 INFO:teuthology.orchestra.run.smithi084.stdout:Epoch : 1 2024-05-15T22:41:55.102 INFO:teuthology.orchestra.run.smithi084.stdout:Version : 2.02 2024-05-15T22:41:55.102 INFO:teuthology.orchestra.run.smithi084.stdout:Release : 129.el8 2024-05-15T22:41:55.102 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: x86_64 2024-05-15T22:41:55.102 INFO:teuthology.orchestra.run.smithi084.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-05-15T22:41:55.102 INFO:teuthology.orchestra.run.smithi084.stdout:Group : System Environment/Base 2024-05-15T22:41:55.102 INFO:teuthology.orchestra.run.smithi084.stdout:Size : 9265586 2024-05-15T22:41:55.102 INFO:teuthology.orchestra.run.smithi084.stdout:License : GPLv3+ 2024-05-15T22:41:55.103 INFO:teuthology.orchestra.run.smithi084.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-05-15T22:41:55.103 INFO:teuthology.orchestra.run.smithi084.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-05-15T22:41:55.103 INFO:teuthology.orchestra.run.smithi084.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-05-15T22:41:55.103 INFO:teuthology.orchestra.run.smithi084.stdout:Build Host : kbuilder.bsys.centos.org 2024-05-15T22:41:55.103 INFO:teuthology.orchestra.run.smithi084.stdout:Relocations : (not relocatable) 2024-05-15T22:41:55.103 INFO:teuthology.orchestra.run.smithi084.stdout:Packager : CentOS BuildSystem 2024-05-15T22:41:55.103 INFO:teuthology.orchestra.run.smithi084.stdout:Vendor : CentOS 2024-05-15T22:41:55.103 INFO:teuthology.orchestra.run.smithi084.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-15T22:41:55.103 INFO:teuthology.orchestra.run.smithi084.stdout:Summary : Support tools for GRUB. 2024-05-15T22:41:55.103 INFO:teuthology.orchestra.run.smithi084.stdout:Description : 2024-05-15T22:41:55.103 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:41:55.104 INFO:teuthology.orchestra.run.smithi084.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-15T22:41:55.104 INFO:teuthology.orchestra.run.smithi084.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-15T22:41:55.104 INFO:teuthology.orchestra.run.smithi084.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-15T22:41:55.104 INFO:teuthology.orchestra.run.smithi084.stdout:hardware devices. 2024-05-15T22:41:55.104 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:41:55.104 INFO:teuthology.orchestra.run.smithi084.stdout:This subpackage provides tools for support of all platforms. 2024-05-15T22:41:55.105 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-15T22:41:55.105 INFO:teuthology.task.kernel:Updating grub on smithi084 to boot 4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:55.105 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-15T22:41:56.495 INFO:teuthology.orchestra.run.smithi084.stderr:Generating grub configuration file ... 2024-05-15T22:41:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T22:41:57.764 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T22:41:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T22:41:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T22:41:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-core-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T22:41:58.689 INFO:teuthology.orchestra.run.smithi084.stderr:done 2024-05-15T22:41:58.690 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-05-15T22:41:58.724 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.3jl6NnujDz 2024-05-15T22:41:58.725 DEBUG:teuthology.orchestra.run.smithi084:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3jl6NnujDz 2024-05-15T22:41:58.798 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.3jl6NnujDz 2024-05-15T22:41:58.942 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.3jl6NnujDz is 6KB 2024-05-15T22:41:58.990 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.3jl6NnujDz 2024-05-15T22:41:59.007 DEBUG:teuthology.orchestra.run.smithi084:> sudo /bin/ls /boot/loader/entries || true 2024-05-15T22:41:59.072 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T22:41:59.073 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:41:59.073 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-05-15T22:41:59.073 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:59.073 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:59.073 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:59.073 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:41:59.073 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:41:59.083 INFO:teuthology.orchestra.run.smithi084.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-05-15T22:41:59.083 INFO:teuthology.orchestra.run.smithi084.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-05-15T22:41:59.083 INFO:teuthology.orchestra.run.smithi084.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-05-15T22:41:59.084 INFO:teuthology.orchestra.run.smithi084.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64.conf 2024-05-15T22:41:59.085 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:59.210 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T22:41:59.227 DEBUG:teuthology.orchestra.run.smithi084:> sudo shutdown -r now 2024-05-15T22:41:59.429 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:59.429 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:59.429 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:59.430 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-15T22:41:59.430 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-15T22:41:59.430 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-05-15T22:41:59.430 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=None, version=distro) 2024-05-15T22:41:59.430 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T22:41:59.641 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:59.641 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:59.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -qi grub2-tools 2024-05-15T22:41:59.674 INFO:teuthology.orchestra.run.smithi090.stdout:Name : grub2-tools 2024-05-15T22:41:59.675 INFO:teuthology.orchestra.run.smithi090.stdout:Epoch : 1 2024-05-15T22:41:59.675 INFO:teuthology.orchestra.run.smithi090.stdout:Version : 2.02 2024-05-15T22:41:59.675 INFO:teuthology.orchestra.run.smithi090.stdout:Release : 129.el8 2024-05-15T22:41:59.675 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: x86_64 2024-05-15T22:41:59.675 INFO:teuthology.orchestra.run.smithi090.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-05-15T22:41:59.675 INFO:teuthology.orchestra.run.smithi090.stdout:Group : System Environment/Base 2024-05-15T22:41:59.675 INFO:teuthology.orchestra.run.smithi090.stdout:Size : 9265586 2024-05-15T22:41:59.675 INFO:teuthology.orchestra.run.smithi090.stdout:License : GPLv3+ 2024-05-15T22:41:59.675 INFO:teuthology.orchestra.run.smithi090.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-05-15T22:41:59.675 INFO:teuthology.orchestra.run.smithi090.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-05-15T22:41:59.675 INFO:teuthology.orchestra.run.smithi090.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-05-15T22:41:59.675 INFO:teuthology.orchestra.run.smithi090.stdout:Build Host : kbuilder.bsys.centos.org 2024-05-15T22:41:59.676 INFO:teuthology.orchestra.run.smithi090.stdout:Relocations : (not relocatable) 2024-05-15T22:41:59.676 INFO:teuthology.orchestra.run.smithi090.stdout:Packager : CentOS BuildSystem 2024-05-15T22:41:59.676 INFO:teuthology.orchestra.run.smithi090.stdout:Vendor : CentOS 2024-05-15T22:41:59.676 INFO:teuthology.orchestra.run.smithi090.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-15T22:41:59.676 INFO:teuthology.orchestra.run.smithi090.stdout:Summary : Support tools for GRUB. 2024-05-15T22:41:59.676 INFO:teuthology.orchestra.run.smithi090.stdout:Description : 2024-05-15T22:41:59.676 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:41:59.676 INFO:teuthology.orchestra.run.smithi090.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-15T22:41:59.676 INFO:teuthology.orchestra.run.smithi090.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-15T22:41:59.676 INFO:teuthology.orchestra.run.smithi090.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-15T22:41:59.676 INFO:teuthology.orchestra.run.smithi090.stdout:hardware devices. 2024-05-15T22:41:59.676 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:41:59.676 INFO:teuthology.orchestra.run.smithi090.stdout:This subpackage provides tools for support of all platforms. 2024-05-15T22:41:59.677 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-15T22:41:59.677 INFO:teuthology.task.kernel:Updating grub on smithi090 to boot 4.18.0-552.1.1.el8.x86_64 2024-05-15T22:41:59.678 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-15T22:42:01.183 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2024-05-15T22:42:04.147 INFO:teuthology.orchestra.run.smithi090.stderr:done 2024-05-15T22:42:04.149 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-05-15T22:42:04.201 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.IxUY54Ihjn 2024-05-15T22:42:04.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.IxUY54Ihjn 2024-05-15T22:42:04.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.IxUY54Ihjn 2024-05-15T22:42:04.343 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.IxUY54Ihjn is 6KB 2024-05-15T22:42:04.395 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.IxUY54Ihjn 2024-05-15T22:42:04.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo /bin/ls /boot/loader/entries || true 2024-05-15T22:42:04.479 INFO:teuthology.orchestra.run.smithi090.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-05-15T22:42:04.479 INFO:teuthology.orchestra.run.smithi090.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-05-15T22:42:04.479 INFO:teuthology.orchestra.run.smithi090.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-05-15T22:42:04.479 INFO:teuthology.orchestra.run.smithi090.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64.conf 2024-05-15T22:42:04.480 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:42:04.552 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T22:42:04.575 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T22:42:04.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2024-05-15T22:42:04.913 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T22:42:04.914 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-4.18.0-552.1.1.el8.x86_64 1/3 2024-05-15T22:42:04.914 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-core-4.18.0-552.1.1.el8.x86_64 2/3 2024-05-15T22:42:05.380 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-4.18.0-552.1.1.el8.x86_64 3/3 2024-05-15T22:42:05.380 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:42:05.380 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T22:42:05.380 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:42:05.380 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:42:05.380 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:42:05.380 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:42:05.380 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:42:05.930 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T22:42:06.149 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:42:06.150 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-15T22:42:06.150 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.1.1.el8.x86_64 2024-05-15T22:42:06.150 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-15T22:42:06.150 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-15T22:42:06.150 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-05-15T22:42:06.150 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2024-05-15T22:42:06.150 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T22:42:06.360 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:42:06.360 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.1.1.el8.x86_64 2024-05-15T22:42:06.361 DEBUG:teuthology.orchestra.run.smithi088:> sudo rpm -qi grub2-tools 2024-05-15T22:42:06.394 INFO:teuthology.orchestra.run.smithi088.stdout:Name : grub2-tools 2024-05-15T22:42:06.394 INFO:teuthology.orchestra.run.smithi088.stdout:Epoch : 1 2024-05-15T22:42:06.394 INFO:teuthology.orchestra.run.smithi088.stdout:Version : 2.02 2024-05-15T22:42:06.394 INFO:teuthology.orchestra.run.smithi088.stdout:Release : 129.el8 2024-05-15T22:42:06.395 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: x86_64 2024-05-15T22:42:06.395 INFO:teuthology.orchestra.run.smithi088.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-05-15T22:42:06.395 INFO:teuthology.orchestra.run.smithi088.stdout:Group : System Environment/Base 2024-05-15T22:42:06.395 INFO:teuthology.orchestra.run.smithi088.stdout:Size : 9265586 2024-05-15T22:42:06.395 INFO:teuthology.orchestra.run.smithi088.stdout:License : GPLv3+ 2024-05-15T22:42:06.395 INFO:teuthology.orchestra.run.smithi088.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-05-15T22:42:06.395 INFO:teuthology.orchestra.run.smithi088.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-05-15T22:42:06.395 INFO:teuthology.orchestra.run.smithi088.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-05-15T22:42:06.395 INFO:teuthology.orchestra.run.smithi088.stdout:Build Host : kbuilder.bsys.centos.org 2024-05-15T22:42:06.396 INFO:teuthology.orchestra.run.smithi088.stdout:Relocations : (not relocatable) 2024-05-15T22:42:06.396 INFO:teuthology.orchestra.run.smithi088.stdout:Packager : CentOS BuildSystem 2024-05-15T22:42:06.396 INFO:teuthology.orchestra.run.smithi088.stdout:Vendor : CentOS 2024-05-15T22:42:06.396 INFO:teuthology.orchestra.run.smithi088.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-15T22:42:06.396 INFO:teuthology.orchestra.run.smithi088.stdout:Summary : Support tools for GRUB. 2024-05-15T22:42:06.396 INFO:teuthology.orchestra.run.smithi088.stdout:Description : 2024-05-15T22:42:06.396 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:42:06.396 INFO:teuthology.orchestra.run.smithi088.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-15T22:42:06.396 INFO:teuthology.orchestra.run.smithi088.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-15T22:42:06.396 INFO:teuthology.orchestra.run.smithi088.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-15T22:42:06.396 INFO:teuthology.orchestra.run.smithi088.stdout:hardware devices. 2024-05-15T22:42:06.396 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:42:06.396 INFO:teuthology.orchestra.run.smithi088.stdout:This subpackage provides tools for support of all platforms. 2024-05-15T22:42:06.397 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-15T22:42:06.398 INFO:teuthology.task.kernel:Updating grub on smithi088 to boot 4.18.0-552.1.1.el8.x86_64 2024-05-15T22:42:06.398 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-15T22:42:07.712 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2024-05-15T22:42:10.021 INFO:teuthology.orchestra.run.smithi088.stderr:done 2024-05-15T22:42:10.023 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-05-15T22:42:10.053 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.KeIcPFoLRi 2024-05-15T22:42:10.054 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KeIcPFoLRi 2024-05-15T22:42:10.127 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.KeIcPFoLRi 2024-05-15T22:42:10.273 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.KeIcPFoLRi is 6KB 2024-05-15T22:42:10.321 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.KeIcPFoLRi 2024-05-15T22:42:10.339 DEBUG:teuthology.orchestra.run.smithi088:> sudo /bin/ls /boot/loader/entries || true 2024-05-15T22:42:10.404 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-05-15T22:42:10.405 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-05-15T22:42:10.405 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-05-15T22:42:10.405 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64.conf 2024-05-15T22:42:10.406 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.1.1.el8.x86_64 2024-05-15T22:42:10.544 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2024-05-15T22:42:29.246 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-15T22:42:29.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-05-15T22:42:29.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:42:34.658 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-05-15T22:42:34.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-05-15T22:42:34.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:42:40.550 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-15T22:42:40.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-15T22:42:40.551 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:43:03.839 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-05-15T22:43:09.218 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-05-15T22:43:12.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-05-15T22:43:12.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:43:15.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-05-15T22:43:18.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-05-15T22:43:18.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:43:21.282 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-05-15T22:43:24.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-15T22:43:24.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:43:27.166 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-05-15T22:43:31.230 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-05-15T22:43:33.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-05-15T22:43:33.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:43:36.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-05-15T22:43:39.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-15T22:43:39.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:43:42.238 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-05-15T22:43:43.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-05-15T22:43:43.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:43:43.580 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-05-15T22:43:43.830 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-05-15T22:43:43.830 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.1.1.el8.x86_64"... 2024-05-15T22:43:43.830 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-05-15T22:43:43.848 INFO:teuthology.orchestra.run.smithi084.stdout:4.18.0-552.1.1.el8.x86_64 2024-05-15T22:43:43.848 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.1.1.el8.x86_64 vs 4.18.0-552.1.1.el8.x86_64 2024-05-15T22:43:43.848 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-15T22:43:43.848 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-15T22:43:44.849 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-15T22:43:44.849 DEBUG:teuthology.orchestra.run.smithi084:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-15T22:43:44.999 INFO:teuthology.orchestra.run.smithi084.stdout:ttyS1 2024-05-15T22:43:45.016 DEBUG:teuthology.parallel:result is None 2024-05-15T22:43:51.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-05-15T22:43:51.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:43:54.434 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-05-15T22:43:57.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-15T22:43:57.241 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:44:00.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-05-15T22:44:12.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-05-15T22:44:12.436 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:44:12.846 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-05-15T22:44:13.184 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-05-15T22:44:13.184 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.1.1.el8.x86_64"... 2024-05-15T22:44:13.185 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-05-15T22:44:13.205 INFO:teuthology.orchestra.run.smithi090.stdout:4.18.0-552.1.1.el8.x86_64 2024-05-15T22:44:13.205 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.1.1.el8.x86_64 vs 4.18.0-552.1.1.el8.x86_64 2024-05-15T22:44:13.205 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-15T22:44:13.205 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-15T22:44:14.206 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-05-15T22:44:14.206 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-15T22:44:14.313 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2024-05-15T22:44:14.327 DEBUG:teuthology.parallel:result is None 2024-05-15T22:44:18.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-15T22:44:18.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:44:18.592 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-05-15T22:44:18.928 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-15T22:44:18.929 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.1.1.el8.x86_64"... 2024-05-15T22:44:18.929 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-05-15T22:44:18.950 INFO:teuthology.orchestra.run.smithi088.stdout:4.18.0-552.1.1.el8.x86_64 2024-05-15T22:44:18.950 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.1.1.el8.x86_64 vs 4.18.0-552.1.1.el8.x86_64 2024-05-15T22:44:18.950 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-15T22:44:18.950 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-15T22:44:19.951 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-15T22:44:19.951 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-15T22:44:20.027 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2024-05-15T22:44:20.040 DEBUG:teuthology.parallel:result is None 2024-05-15T22:44:20.041 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-15T22:44:20.049 INFO:teuthology.task.internal:Creating test directory... 2024-05-15T22:44:20.049 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-15T22:44:20.053 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-15T22:44:20.055 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-15T22:44:20.077 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-15T22:44:20.083 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-15T22:44:20.092 INFO:teuthology.task.internal:Creating archive directory... 2024-05-15T22:44:20.092 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-15T22:44:20.116 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-15T22:44:20.118 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-15T22:44:20.157 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-15T22:44:20.164 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-15T22:44:20.164 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-15T22:44:20.194 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-15T22:44:20.196 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-15T22:44:20.233 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T22:44:20.236 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T22:44:20.237 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T22:44:20.244 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T22:44:20.248 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T22:44:20.249 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T22:44:20.250 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-15T22:44:20.257 INFO:teuthology.task.internal:Configuring sudo... 2024-05-15T22:44:20.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-15T22:44:20.287 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-15T22:44:20.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-15T22:44:20.323 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-15T22:44:20.331 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-15T22:44:20.331 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-15T22:44:20.357 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-15T22:44:20.363 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-15T22:44:20.382 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T22:44:20.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T22:44:20.517 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T22:44:20.602 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T22:44:20.645 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-05-15T22:44:20.645 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-15T22:44:20.712 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T22:44:20.759 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T22:44:20.826 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T22:44:20.874 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T22:44:20.922 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T22:44:20.922 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-15T22:44:20.995 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T22:44:21.042 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T22:44:21.098 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T22:44:21.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T22:44:21.230 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-05-15T22:44:21.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-15T22:44:21.300 DEBUG:teuthology.orchestra.run.smithi084:> sudo service rsyslog restart 2024-05-15T22:44:21.302 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2024-05-15T22:44:21.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-05-15T22:44:21.366 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T22:44:21.368 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T22:44:21.410 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T22:44:21.878 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-15T22:44:21.886 INFO:teuthology.task.internal:Starting timer... 2024-05-15T22:44:21.886 INFO:teuthology.run_tasks:Running task pcp... 2024-05-15T22:44:21.912 INFO:teuthology.run_tasks:Running task selinux... 2024-05-15T22:44:21.920 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-05-15T22:44:21.921 DEBUG:teuthology.orchestra.run.smithi084:> sudo service auditd rotate 2024-05-15T22:44:21.992 INFO:teuthology.orchestra.run.smithi084.stdout:Rotating logs: 2024-05-15T22:44:21.993 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2024-05-15T22:44:22.084 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: 2024-05-15T22:44:22.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2024-05-15T22:44:22.186 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2024-05-15T22:44:22.188 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-15T22:44:22.188 DEBUG:teuthology.orchestra.run.smithi084:> /usr/sbin/getenforce 2024-05-15T22:44:22.229 INFO:teuthology.orchestra.run.smithi084.stdout:Permissive 2024-05-15T22:44:22.229 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2024-05-15T22:44:22.259 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2024-05-15T22:44:22.259 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2024-05-15T22:44:22.287 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2024-05-15T22:44:22.287 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi084.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi088.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi090.front.sepia.ceph.com': 'permissive'} 2024-05-15T22:44:22.287 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-05-15T22:44:22.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T22:44:22.314 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-05-15T22:44:22.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T22:44:22.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-05-15T22:44:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T22:44:22.370 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-15T22:44:22.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo /usr/sbin/setenforce permissive 2024-05-15T22:44:22.397 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2024-05-15T22:44:22.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2024-05-15T22:44:22.453 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-15T22:44:22.463 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-15T22:44:22.484 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-15T22:44:22.486 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi084.front.sepia.ceph.com,smithi088.front.sepia.ceph.com,smithi090.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-15T22:50:21.432 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi084.front.sepia.ceph.com'), Remote(name='ubuntu@smithi088.front.sepia.ceph.com'), Remote(name='ubuntu@smithi090.front.sepia.ceph.com')] 2024-05-15T22:50:21.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-05-15T22:50:21.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:50:21.513 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-05-15T22:50:21.580 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-05-15T22:50:21.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-15T22:50:21.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:50:21.663 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-05-15T22:50:21.735 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-15T22:50:21.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-05-15T22:50:21.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T22:50:21.812 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-05-15T22:50:21.886 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-05-15T22:50:21.886 INFO:teuthology.run_tasks:Running task clock... 2024-05-15T22:50:21.897 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-15T22:50:21.898 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-15T22:50:21.898 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T22:50:21.900 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-15T22:50:21.900 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T22:50:21.903 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-15T22:50:21.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T22:50:21.931 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-15T22:50:21.939 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-15T22:50:21.945 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-15T22:50:21.955 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-15T22:50:21.959 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-15T22:50:21.974 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-15T22:50:22.013 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: ntpd: command not found 2024-05-15T22:50:22.022 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: ntpd: command not found 2024-05-15T22:50:22.024 INFO:teuthology.orchestra.run.smithi084.stdout:506 Cannot talk to daemon 2024-05-15T22:50:22.036 INFO:teuthology.orchestra.run.smithi088.stdout:506 Cannot talk to daemon 2024-05-15T22:50:22.038 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-15T22:50:22.047 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2024-05-15T22:50:22.052 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-15T22:50:22.053 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-15T22:50:22.058 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2024-05-15T22:50:22.069 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-15T22:50:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-15T22:50:22.087 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-15T22:50:22.122 INFO:teuthology.orchestra.run.smithi084.stderr:bash: ntpq: command not found 2024-05-15T22:50:22.125 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T22:50:22.125 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-05-15T22:50:22.126 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T22:50:22.126 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T22:50:22.126 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T22:50:22.126 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T22:50:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:bash: ntpq: command not found 2024-05-15T22:50:22.150 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T22:50:22.150 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-05-15T22:50:22.150 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T22:50:22.150 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T22:50:22.150 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T22:50:22.150 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T22:50:22.154 INFO:teuthology.orchestra.run.smithi090.stderr:bash: ntpq: command not found 2024-05-15T22:50:22.159 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T22:50:22.160 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-05-15T22:50:22.160 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T22:50:22.160 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T22:50:22.160 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T22:50:22.160 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T22:50:22.160 INFO:teuthology.run_tasks:Running task pexec... 2024-05-15T22:50:22.170 INFO:teuthology.task.pexec:Executing custom commands... 2024-05-15T22:50:22.170 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi084.front.sepia.ceph.com 2024-05-15T22:50:22.170 DEBUG:teuthology.orchestra.run.smithi084:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-05-15T22:50:22.171 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi088.front.sepia.ceph.com 2024-05-15T22:50:22.171 DEBUG:teuthology.orchestra.run.smithi088:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-05-15T22:50:22.171 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi090.front.sepia.ceph.com 2024-05-15T22:50:22.171 DEBUG:teuthology.orchestra.run.smithi090:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-05-15T22:50:22.922 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:26 ago on Wed 15 May 2024 10:48:56 PM UTC. 2024-05-15T22:50:22.948 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:25 ago on Wed 15 May 2024 10:48:57 PM UTC. 2024-05-15T22:50:22.961 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:23 ago on Wed 15 May 2024 10:48:59 PM UTC. 2024-05-15T22:50:25.089 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:50:25.090 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:50:25.090 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-05-15T22:50:25.090 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:50:25.090 INFO:teuthology.orchestra.run.smithi084.stdout:Resetting modules: 2024-05-15T22:50:25.090 INFO:teuthology.orchestra.run.smithi084.stdout: container-tools 2024-05-15T22:50:25.091 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:50:25.091 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-05-15T22:50:25.091 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:50:25.091 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:50:25.117 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:50:25.118 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:50:25.118 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-05-15T22:50:25.118 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:50:25.119 INFO:teuthology.orchestra.run.smithi090.stdout:Resetting modules: 2024-05-15T22:50:25.119 INFO:teuthology.orchestra.run.smithi090.stdout: container-tools 2024-05-15T22:50:25.119 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:50:25.119 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-05-15T22:50:25.119 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:50:25.119 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:50:25.131 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:50:25.132 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:50:25.132 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-05-15T22:50:25.132 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:50:25.132 INFO:teuthology.orchestra.run.smithi088.stdout:Resetting modules: 2024-05-15T22:50:25.132 INFO:teuthology.orchestra.run.smithi088.stdout: container-tools 2024-05-15T22:50:25.132 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:50:25.132 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T22:50:25.132 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:50:25.132 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:50:25.346 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:50:25.367 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:50:25.384 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:50:26.121 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:29 ago on Wed 15 May 2024 10:48:56 PM UTC. 2024-05-15T22:50:26.173 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:28 ago on Wed 15 May 2024 10:48:57 PM UTC. 2024-05-15T22:50:26.176 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:26 ago on Wed 15 May 2024 10:48:59 PM UTC. 2024-05-15T22:50:28.289 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:50:28.326 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:50:28.326 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-05-15T22:50:28.326 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:50:28.326 INFO:teuthology.orchestra.run.smithi084.stdout:Installing group/module packages: 2024-05-15T22:50:28.326 INFO:teuthology.orchestra.run.smithi084.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-05-15T22:50:28.327 INFO:teuthology.orchestra.run.smithi084.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-05-15T22:50:28.327 INFO:teuthology.orchestra.run.smithi084.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-05-15T22:50:28.327 INFO:teuthology.orchestra.run.smithi084.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-05-15T22:50:28.327 INFO:teuthology.orchestra.run.smithi084.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-05-15T22:50:28.327 INFO:teuthology.orchestra.run.smithi084.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-05-15T22:50:28.327 INFO:teuthology.orchestra.run.smithi084.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-05-15T22:50:28.327 INFO:teuthology.orchestra.run.smithi084.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-05-15T22:50:28.327 INFO:teuthology.orchestra.run.smithi084.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-05-15T22:50:28.327 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-05-15T22:50:28.327 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-05-15T22:50:28.327 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-05-15T22:50:28.327 INFO:teuthology.orchestra.run.smithi084.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-05-15T22:50:28.327 INFO:teuthology.orchestra.run.smithi084.stdout:Installing weak dependencies: 2024-05-15T22:50:28.327 INFO:teuthology.orchestra.run.smithi084.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-05-15T22:50:28.327 INFO:teuthology.orchestra.run.smithi084.stdout:Installing module profiles: 2024-05-15T22:50:28.328 INFO:teuthology.orchestra.run.smithi084.stdout: container-tools/common 2024-05-15T22:50:28.328 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:50:28.328 INFO:teuthology.orchestra.run.smithi084.stdout:Enabling module streams: 2024-05-15T22:50:28.328 INFO:teuthology.orchestra.run.smithi084.stdout: container-tools rhel8 2024-05-15T22:50:28.328 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:50:28.328 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-05-15T22:50:28.328 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:50:28.328 INFO:teuthology.orchestra.run.smithi084.stdout:Install 13 Packages 2024-05-15T22:50:28.328 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:50:28.329 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 27 M 2024-05-15T22:50:28.329 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 91 M 2024-05-15T22:50:28.329 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-05-15T22:50:28.337 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:50:28.347 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:50:28.374 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:50:28.375 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T22:50:28.375 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:50:28.375 INFO:teuthology.orchestra.run.smithi088.stdout:Installing group/module packages: 2024-05-15T22:50:28.375 INFO:teuthology.orchestra.run.smithi088.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-05-15T22:50:28.375 INFO:teuthology.orchestra.run.smithi088.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-05-15T22:50:28.375 INFO:teuthology.orchestra.run.smithi088.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-05-15T22:50:28.375 INFO:teuthology.orchestra.run.smithi088.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-05-15T22:50:28.375 INFO:teuthology.orchestra.run.smithi088.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-05-15T22:50:28.375 INFO:teuthology.orchestra.run.smithi088.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-05-15T22:50:28.375 INFO:teuthology.orchestra.run.smithi088.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-05-15T22:50:28.375 INFO:teuthology.orchestra.run.smithi088.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-05-15T22:50:28.376 INFO:teuthology.orchestra.run.smithi088.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-05-15T22:50:28.376 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T22:50:28.376 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-05-15T22:50:28.376 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-05-15T22:50:28.376 INFO:teuthology.orchestra.run.smithi088.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-05-15T22:50:28.376 INFO:teuthology.orchestra.run.smithi088.stdout:Installing weak dependencies: 2024-05-15T22:50:28.376 INFO:teuthology.orchestra.run.smithi088.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-05-15T22:50:28.376 INFO:teuthology.orchestra.run.smithi088.stdout:Installing module profiles: 2024-05-15T22:50:28.376 INFO:teuthology.orchestra.run.smithi088.stdout: container-tools/common 2024-05-15T22:50:28.376 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:50:28.376 INFO:teuthology.orchestra.run.smithi088.stdout:Enabling module streams: 2024-05-15T22:50:28.376 INFO:teuthology.orchestra.run.smithi088.stdout: container-tools rhel8 2024-05-15T22:50:28.376 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:50:28.376 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T22:50:28.376 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:50:28.377 INFO:teuthology.orchestra.run.smithi088.stdout:Install 13 Packages 2024-05-15T22:50:28.377 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:50:28.377 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 27 M 2024-05-15T22:50:28.377 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 91 M 2024-05-15T22:50:28.377 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T22:50:28.387 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:50:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-05-15T22:50:28.387 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:50:28.387 INFO:teuthology.orchestra.run.smithi090.stdout:Installing group/module packages: 2024-05-15T22:50:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-05-15T22:50:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-05-15T22:50:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-05-15T22:50:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-05-15T22:50:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-05-15T22:50:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-05-15T22:50:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-05-15T22:50:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-05-15T22:50:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-05-15T22:50:28.388 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-05-15T22:50:28.388 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-05-15T22:50:28.388 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-05-15T22:50:28.388 INFO:teuthology.orchestra.run.smithi090.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-05-15T22:50:28.388 INFO:teuthology.orchestra.run.smithi090.stdout:Installing weak dependencies: 2024-05-15T22:50:28.388 INFO:teuthology.orchestra.run.smithi090.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-05-15T22:50:28.388 INFO:teuthology.orchestra.run.smithi090.stdout:Installing module profiles: 2024-05-15T22:50:28.388 INFO:teuthology.orchestra.run.smithi090.stdout: container-tools/common 2024-05-15T22:50:28.388 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:50:28.388 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling module streams: 2024-05-15T22:50:28.388 INFO:teuthology.orchestra.run.smithi090.stdout: container-tools rhel8 2024-05-15T22:50:28.388 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:50:28.388 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-05-15T22:50:28.388 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:50:28.388 INFO:teuthology.orchestra.run.smithi090.stdout:Install 13 Packages 2024-05-15T22:50:28.389 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:50:28.389 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 27 M 2024-05-15T22:50:28.389 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 91 M 2024-05-15T22:50:28.389 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-05-15T22:50:29.073 INFO:teuthology.orchestra.run.smithi088.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 1.2 MB/s | 689 kB 00:00 2024-05-15T22:50:29.123 INFO:teuthology.orchestra.run.smithi088.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.6 MB/s | 1.0 MB 00:00 2024-05-15T22:50:29.138 INFO:teuthology.orchestra.run.smithi090.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 1.1 MB/s | 689 kB 00:00 2024-05-15T22:50:29.155 INFO:teuthology.orchestra.run.smithi084.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 1.2 MB/s | 689 kB 00:00 2024-05-15T22:50:29.171 INFO:teuthology.orchestra.run.smithi090.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.5 MB/s | 1.0 MB 00:00 2024-05-15T22:50:29.182 INFO:teuthology.orchestra.run.smithi088.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 349 kB/s | 38 kB 00:00 2024-05-15T22:50:29.206 INFO:teuthology.orchestra.run.smithi084.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.6 MB/s | 1.0 MB 00:00 2024-05-15T22:50:29.224 INFO:teuthology.orchestra.run.smithi088.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 2.5 MB/s | 258 kB 00:00 2024-05-15T22:50:29.238 INFO:teuthology.orchestra.run.smithi090.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 378 kB/s | 38 kB 00:00 2024-05-15T22:50:29.248 INFO:teuthology.orchestra.run.smithi084.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 412 kB/s | 38 kB 00:00 2024-05-15T22:50:29.272 INFO:teuthology.orchestra.run.smithi090.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 2.5 MB/s | 258 kB 00:00 2024-05-15T22:50:29.315 INFO:teuthology.orchestra.run.smithi084.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 2.3 MB/s | 258 kB 00:00 2024-05-15T22:50:29.324 INFO:teuthology.orchestra.run.smithi088.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.8 MB/s | 180 kB 00:00 2024-05-15T22:50:29.381 INFO:teuthology.orchestra.run.smithi090.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.6 MB/s | 180 kB 00:00 2024-05-15T22:50:29.415 INFO:teuthology.orchestra.run.smithi084.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.8 MB/s | 180 kB 00:00 2024-05-15T22:50:29.416 INFO:teuthology.orchestra.run.smithi088.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 270 kB/s | 25 kB 00:00 2024-05-15T22:50:29.481 INFO:teuthology.orchestra.run.smithi090.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 247 kB/s | 25 kB 00:00 2024-05-15T22:50:29.507 INFO:teuthology.orchestra.run.smithi084.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 271 kB/s | 25 kB 00:00 2024-05-15T22:50:29.525 INFO:teuthology.orchestra.run.smithi088.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 867 kB/s | 94 kB 00:00 2024-05-15T22:50:29.583 INFO:teuthology.orchestra.run.smithi090.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 939 kB/s | 94 kB 00:00 2024-05-15T22:50:29.608 INFO:teuthology.orchestra.run.smithi088.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 9.7 MB/s | 4.1 MB 00:00 2024-05-15T22:50:29.608 INFO:teuthology.orchestra.run.smithi084.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 940 kB/s | 94 kB 00:00 2024-05-15T22:50:29.666 INFO:teuthology.orchestra.run.smithi090.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 9.6 MB/s | 4.1 MB 00:00 2024-05-15T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 9.5 MB/s | 4.1 MB 00:00 2024-05-15T22:50:29.710 INFO:teuthology.orchestra.run.smithi088.stdout:(9/13): buildah-1.33.6-2.module_el8+976+1136073 8.0 MB/s | 9.6 MB 00:01 2024-05-15T22:50:29.794 INFO:teuthology.orchestra.run.smithi084.stdout:(9/13): buildah-1.33.6-2.module_el8+976+1136073 8.0 MB/s | 9.6 MB 00:01 2024-05-15T22:50:29.802 INFO:teuthology.orchestra.run.smithi090.stdout:(9/13): buildah-1.33.6-2.module_el8+976+1136073 7.5 MB/s | 9.6 MB 00:01 2024-05-15T22:50:29.809 INFO:teuthology.orchestra.run.smithi088.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 527 kB/s | 53 kB 00:00 2024-05-15T22:50:29.864 INFO:teuthology.orchestra.run.smithi088.stdout:(11/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 10 MB/s | 2.5 MB 00:00 2024-05-15T22:50:29.883 INFO:teuthology.orchestra.run.smithi084.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 575 kB/s | 53 kB 00:00 2024-05-15T22:50:29.909 INFO:teuthology.orchestra.run.smithi088.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 410 kB/s | 41 kB 00:00 2024-05-15T22:50:29.916 INFO:teuthology.orchestra.run.smithi090.stdout:(10/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 10 MB/s | 2.5 MB 00:00 2024-05-15T22:50:29.933 INFO:teuthology.orchestra.run.smithi084.stdout:(11/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 10 MB/s | 2.5 MB 00:00 2024-05-15T22:50:29.943 INFO:teuthology.orchestra.run.smithi090.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 371 kB/s | 53 kB 00:00 2024-05-15T22:50:29.975 INFO:teuthology.orchestra.run.smithi084.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 446 kB/s | 41 kB 00:00 2024-05-15T22:50:30.008 INFO:teuthology.orchestra.run.smithi090.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 447 kB/s | 41 kB 00:00 2024-05-15T22:50:30.211 INFO:teuthology.orchestra.run.smithi088.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 13 MB/s | 8.8 MB 00:00 2024-05-15T22:50:30.211 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:50:30.211 INFO:teuthology.orchestra.run.smithi088.stdout:Total 15 MB/s | 27 MB 00:01 2024-05-15T22:50:30.250 INFO:teuthology.orchestra.run.smithi090.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 13 MB/s | 8.8 MB 00:00 2024-05-15T22:50:30.251 INFO:teuthology.orchestra.run.smithi084.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 14 MB/s | 8.8 MB 00:00 2024-05-15T22:50:30.251 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:50:30.251 INFO:teuthology.orchestra.run.smithi090.stdout:Total 15 MB/s | 27 MB 00:01 2024-05-15T22:50:30.252 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:50:30.252 INFO:teuthology.orchestra.run.smithi084.stdout:Total 14 MB/s | 27 MB 00:01 2024-05-15T22:50:30.485 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T22:50:30.505 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T22:50:30.505 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T22:50:30.525 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-05-15T22:50:30.530 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-05-15T22:50:30.546 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-05-15T22:50:30.546 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-05-15T22:50:30.551 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-05-15T22:50:30.551 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-05-15T22:50:30.774 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T22:50:30.777 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T22:50:30.796 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-05-15T22:50:30.799 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-05-15T22:50:30.816 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-05-15T22:50:30.820 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-05-15T22:50:31.422 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T22:50:31.439 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-05-15T22:50:31.441 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-05-15T22:50:32.342 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-05-15T22:50:32.365 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-05-15T22:50:32.394 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-05-15T22:50:32.535 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-05-15T22:50:32.599 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-05-15T22:50:32.608 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-05-15T22:50:32.722 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-05-15T22:50:32.736 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-05-15T22:50:32.777 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-05-15T22:50:32.943 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-05-15T22:50:32.979 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-05-15T22:50:32.985 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-05-15T22:50:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-05-15T22:50:33.198 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-05-15T22:50:33.212 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-05-15T22:50:33.712 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-05-15T22:50:33.731 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-05-15T22:50:33.964 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-05-15T22:50:34.666 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-05-15T22:50:34.712 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-05-15T22:50:34.901 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-05-15T22:50:34.933 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-05-15T22:50:35.001 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-05-15T22:50:35.136 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-05-15T22:50:35.205 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-05-15T22:50:35.211 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-05-15T22:50:35.406 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-05-15T22:50:35.568 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-05-15T22:50:35.629 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-05-15T22:50:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-15T22:50:35.809 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-05-15T22:50:35.842 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-15T22:50:35.955 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-15T22:50:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-15T22:50:36.044 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-15T22:50:36.172 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-15T22:50:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-15T22:50:36.279 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-15T22:50:36.386 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-15T22:50:36.599 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-15T22:50:36.599 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-05-15T22:50:36.599 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-05-15T22:50:36.599 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-05-15T22:50:36.599 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-05-15T22:50:36.599 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-05-15T22:50:36.599 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-05-15T22:50:36.599 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-05-15T22:50:36.600 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-05-15T22:50:36.600 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-05-15T22:50:36.600 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-05-15T22:50:36.600 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-15T22:50:36.600 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-15T22:50:36.706 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-15T22:50:36.707 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-05-15T22:50:36.707 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-05-15T22:50:36.707 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-05-15T22:50:36.707 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-05-15T22:50:36.707 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-05-15T22:50:36.707 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-05-15T22:50:36.707 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-05-15T22:50:36.707 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-05-15T22:50:36.707 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-05-15T22:50:36.707 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-05-15T22:50:36.707 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-15T22:50:36.707 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-15T22:50:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-05-15T22:50:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-05-15T22:50:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-05-15T22:50:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-05-15T22:50:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-05-15T22:50:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-05-15T22:50:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-05-15T22:50:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-05-15T22:50:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-05-15T22:50:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-05-15T22:50:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-05-15T22:50:36.805 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-05-15T22:50:36.805 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-05-15T22:50:37.251 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-05-15T22:50:37.251 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:50:37.252 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T22:50:37.252 INFO:teuthology.orchestra.run.smithi088.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-05-15T22:50:37.252 INFO:teuthology.orchestra.run.smithi088.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-05-15T22:50:37.252 INFO:teuthology.orchestra.run.smithi088.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-05-15T22:50:37.252 INFO:teuthology.orchestra.run.smithi088.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-05-15T22:50:37.252 INFO:teuthology.orchestra.run.smithi088.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-05-15T22:50:37.252 INFO:teuthology.orchestra.run.smithi088.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-05-15T22:50:37.252 INFO:teuthology.orchestra.run.smithi088.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-05-15T22:50:37.252 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-05-15T22:50:37.252 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-05-15T22:50:37.252 INFO:teuthology.orchestra.run.smithi088.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-05-15T22:50:37.252 INFO:teuthology.orchestra.run.smithi088.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-05-15T22:50:37.252 INFO:teuthology.orchestra.run.smithi088.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-05-15T22:50:37.252 INFO:teuthology.orchestra.run.smithi088.stdout: yajl-2.1.0-12.el8.x86_64 2024-05-15T22:50:37.253 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:50:37.253 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:50:37.350 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-05-15T22:50:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:50:37.351 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-05-15T22:50:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-05-15T22:50:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-05-15T22:50:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-05-15T22:50:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-05-15T22:50:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-05-15T22:50:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-05-15T22:50:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-05-15T22:50:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-05-15T22:50:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-05-15T22:50:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-05-15T22:50:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-05-15T22:50:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-05-15T22:50:37.352 INFO:teuthology.orchestra.run.smithi090.stdout: yajl-2.1.0-12.el8.x86_64 2024-05-15T22:50:37.352 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:50:37.352 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:50:37.352 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-05-15T22:50:37.352 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:50:37.352 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-05-15T22:50:37.352 INFO:teuthology.orchestra.run.smithi084.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-05-15T22:50:37.352 INFO:teuthology.orchestra.run.smithi084.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-05-15T22:50:37.352 INFO:teuthology.orchestra.run.smithi084.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-05-15T22:50:37.353 INFO:teuthology.orchestra.run.smithi084.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-05-15T22:50:37.353 INFO:teuthology.orchestra.run.smithi084.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-05-15T22:50:37.353 INFO:teuthology.orchestra.run.smithi084.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-05-15T22:50:37.353 INFO:teuthology.orchestra.run.smithi084.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-05-15T22:50:37.353 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-05-15T22:50:37.353 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-05-15T22:50:37.353 INFO:teuthology.orchestra.run.smithi084.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-05-15T22:50:37.353 INFO:teuthology.orchestra.run.smithi084.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-05-15T22:50:37.353 INFO:teuthology.orchestra.run.smithi084.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-05-15T22:50:37.353 INFO:teuthology.orchestra.run.smithi084.stdout: yajl-2.1.0-12.el8.x86_64 2024-05-15T22:50:37.353 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:50:37.353 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:50:37.509 DEBUG:teuthology.parallel:result is None 2024-05-15T22:50:37.611 DEBUG:teuthology.parallel:result is None 2024-05-15T22:50:37.637 DEBUG:teuthology.parallel:result is None 2024-05-15T22:50:37.638 INFO:teuthology.run_tasks:Running task install... 2024-05-15T22:50:37.647 DEBUG:teuthology.task.install:project ceph 2024-05-15T22:50:37.647 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '6cd0f8013e2dea00c3a29a0d8b10656b132d7c80'}} 2024-05-15T22:50:37.647 DEBUG:teuthology.task.install:config {'branch': 'nautilus', 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'flavor': 'default', 'sha1': '6cd0f8013e2dea00c3a29a0d8b10656b132d7c80'} 2024-05-15T22:50:37.647 INFO:teuthology.task.install:Using flavor: default 2024-05-15T22:50:37.653 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-15T22:50:37.653 INFO:teuthology.task.install:extra packages: ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'] 2024-05-15T22:50:37.654 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'nautilus', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6cd0f8013e2dea00c3a29a0d8b10656b132d7c80', 'tag': None, 'wait_for_package': False} 2024-05-15T22:50:37.654 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T22:50:37.654 INFO:teuthology.packaging:ref: None 2024-05-15T22:50:37.654 INFO:teuthology.packaging:tag: None 2024-05-15T22:50:37.654 INFO:teuthology.packaging:branch: nautilus 2024-05-15T22:50:37.654 INFO:teuthology.packaging:sha1: 6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 2024-05-15T22:50:37.655 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-05-15T22:50:37.656 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'nautilus', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6cd0f8013e2dea00c3a29a0d8b10656b132d7c80', 'tag': None, 'wait_for_package': False} 2024-05-15T22:50:37.657 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T22:50:37.657 INFO:teuthology.packaging:ref: None 2024-05-15T22:50:37.657 INFO:teuthology.packaging:tag: None 2024-05-15T22:50:37.657 INFO:teuthology.packaging:branch: nautilus 2024-05-15T22:50:37.657 INFO:teuthology.packaging:sha1: 6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 2024-05-15T22:50:37.657 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-05-15T22:50:37.658 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'nautilus', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6cd0f8013e2dea00c3a29a0d8b10656b132d7c80', 'tag': None, 'wait_for_package': False} 2024-05-15T22:50:37.658 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T22:50:37.658 INFO:teuthology.packaging:ref: None 2024-05-15T22:50:37.658 INFO:teuthology.packaging:tag: None 2024-05-15T22:50:37.658 INFO:teuthology.packaging:branch: nautilus 2024-05-15T22:50:37.658 INFO:teuthology.packaging:sha1: 6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 2024-05-15T22:50:37.658 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-05-15T22:50:37.855 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/ 2024-05-15T22:50:37.855 INFO:teuthology.task.install.rpm:Package version is 14.2.22-0 2024-05-15T22:50:37.892 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-15T22:50:37.892 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-05-15T22:50:37.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-15T22:50:37.920 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/ 2024-05-15T22:50:37.920 INFO:teuthology.task.install.rpm:Package version is 14.2.22-0 2024-05-15T22:50:37.922 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, rbd-fuse, rbd-mirror, rbd-nbd, python-rados, python-rgw, python-rbd, python-cephfs on remote rpm x86_64 2024-05-15T22:50:37.922 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T22:50:37.922 INFO:teuthology.packaging:ref: None 2024-05-15T22:50:37.922 INFO:teuthology.packaging:tag: None 2024-05-15T22:50:37.922 INFO:teuthology.packaging:branch: nautilus 2024-05-15T22:50:37.922 INFO:teuthology.packaging:sha1: 6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 2024-05-15T22:50:37.922 DEBUG:teuthology.orchestra.run.smithi084:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/nautilus/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-15T22:50:37.933 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/ 2024-05-15T22:50:37.934 INFO:teuthology.task.install.rpm:Package version is 14.2.22-0 2024-05-15T22:50:37.962 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-15T22:50:37.962 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-05-15T22:50:37.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-15T22:50:37.970 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-15T22:50:37.970 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T22:50:37.970 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-15T22:50:37.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-15T22:50:37.991 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, rbd-fuse, rbd-mirror, rbd-nbd, python-rados, python-rgw, python-rbd, python-cephfs on remote rpm x86_64 2024-05-15T22:50:37.991 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T22:50:37.991 INFO:teuthology.packaging:ref: None 2024-05-15T22:50:37.991 INFO:teuthology.packaging:tag: None 2024-05-15T22:50:37.991 INFO:teuthology.packaging:branch: nautilus 2024-05-15T22:50:37.991 INFO:teuthology.packaging:sha1: 6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 2024-05-15T22:50:37.992 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/nautilus/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-15T22:50:37.997 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, rbd-fuse, rbd-mirror, rbd-nbd, python-rados, python-rgw, python-rbd, python-cephfs on remote rpm x86_64 2024-05-15T22:50:37.997 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T22:50:37.997 INFO:teuthology.packaging:ref: None 2024-05-15T22:50:37.997 INFO:teuthology.packaging:tag: None 2024-05-15T22:50:37.997 INFO:teuthology.packaging:branch: nautilus 2024-05-15T22:50:37.997 INFO:teuthology.packaging:sha1: 6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 2024-05-15T22:50:37.997 DEBUG:teuthology.orchestra.run.smithi088:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/nautilus/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-15T22:50:38.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-15T22:50:38.064 DEBUG:teuthology.orchestra.run.smithi088:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-15T22:50:38.067 DEBUG:teuthology.orchestra.run.smithi084:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-15T22:50:38.133 INFO:teuthology.orchestra.run.smithi084.stdout:check_obsoletes = 1 2024-05-15T22:50:38.134 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum clean all 2024-05-15T22:50:38.135 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-15T22:50:38.140 DEBUG:teuthology.orchestra.run.smithi088:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-15T22:50:38.202 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2024-05-15T22:50:38.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2024-05-15T22:50:38.206 INFO:teuthology.orchestra.run.smithi088.stdout:check_obsoletes = 1 2024-05-15T22:50:38.207 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum clean all 2024-05-15T22:50:38.403 INFO:teuthology.orchestra.run.smithi084.stdout:78 files removed 2024-05-15T22:50:38.431 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-radosgw 2024-05-15T22:50:38.472 INFO:teuthology.orchestra.run.smithi090.stdout:78 files removed 2024-05-15T22:50:38.475 INFO:teuthology.orchestra.run.smithi088.stdout:78 files removed 2024-05-15T22:50:38.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw 2024-05-15T22:50:38.505 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw 2024-05-15T22:50:38.907 INFO:teuthology.orchestra.run.smithi084.stdout:ceph packages for x86_64 889 kB/s | 65 kB 00:00 2024-05-15T22:50:38.984 INFO:teuthology.orchestra.run.smithi088.stdout:ceph packages for x86_64 825 kB/s | 65 kB 00:00 2024-05-15T22:50:39.024 INFO:teuthology.orchestra.run.smithi084.stdout:ceph noarch packages 142 kB/s | 11 kB 00:00 2024-05-15T22:50:39.041 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 466 kB/s | 65 kB 00:00 2024-05-15T22:50:39.102 INFO:teuthology.orchestra.run.smithi088.stdout:ceph noarch packages 142 kB/s | 11 kB 00:00 2024-05-15T22:50:39.125 INFO:teuthology.orchestra.run.smithi084.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2024-05-15T22:50:39.202 INFO:teuthology.orchestra.run.smithi088.stdout:ceph source packages 23 kB/s | 1.7 kB 00:00 2024-05-15T22:50:39.260 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 63 kB/s | 11 kB 00:00 2024-05-15T22:50:39.360 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 23 kB/s | 1.7 kB 00:00 2024-05-15T22:50:41.301 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - AppStream 13 MB/s | 28 MB 00:02 2024-05-15T22:50:41.403 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - AppStream 13 MB/s | 28 MB 00:02 2024-05-15T22:50:41.428 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - AppStream 14 MB/s | 28 MB 00:02 2024-05-15T22:50:46.934 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - BaseOS 17 MB/s | 10 MB 00:00 2024-05-15T22:50:47.195 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - BaseOS 13 MB/s | 10 MB 00:00 2024-05-15T22:50:47.779 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - BaseOS 7.5 MB/s | 10 MB 00:01 2024-05-15T22:50:48.859 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - Extras 57 kB/s | 18 kB 00:00 2024-05-15T22:50:49.087 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - Extras 64 kB/s | 18 kB 00:00 2024-05-15T22:50:49.277 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - Extras common packages 25 kB/s | 7.7 kB 00:00 2024-05-15T22:50:49.379 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - Extras common packages 42 kB/s | 7.7 kB 00:00 2024-05-15T22:50:49.730 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - Extras 55 kB/s | 18 kB 00:00 2024-05-15T22:50:50.182 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 7.7 kB 00:00 2024-05-15T22:50:50.236 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - PowerTools 5.7 MB/s | 5.0 MB 00:00 2024-05-15T22:50:50.347 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - PowerTools 5.7 MB/s | 5.0 MB 00:00 2024-05-15T22:50:51.092 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - PowerTools 6.0 MB/s | 5.0 MB 00:00 2024-05-15T22:50:51.554 INFO:teuthology.orchestra.run.smithi084.stdout:Copr repo for python3-asyncssh owned by ceph 9.5 kB/s | 3.5 kB 00:00 2024-05-15T22:50:51.574 INFO:teuthology.orchestra.run.smithi088.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2024-05-15T22:50:52.227 INFO:teuthology.orchestra.run.smithi090.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2024-05-15T22:50:53.257 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 8.5 MB/s | 14 MB 00:01 2024-05-15T22:50:54.854 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 5.4 MB/s | 14 MB 00:02 2024-05-15T22:50:57.615 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 440 kB/s | 24 kB 00:00 2024-05-15T22:50:59.095 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 393 kB/s | 24 kB 00:00 2024-05-15T22:51:01.339 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:51:01.341 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:51:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T22:51:01.341 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:51:01.341 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T22:51:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw x86_64 2:14.2.22-0.el8 ceph 6.0 M 2024-05-15T22:51:01.341 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrading: 2024-05-15T22:51:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: librados2 x86_64 2:14.2.22-0.el8 ceph 3.5 M 2024-05-15T22:51:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1 x86_64 2:14.2.22-0.el8 ceph 1.7 M 2024-05-15T22:51:01.342 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T22:51:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base x86_64 2:14.2.22-0.el8 ceph 5.4 M 2024-05-15T22:51:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common x86_64 2:14.2.22-0.el8 ceph 19 M 2024-05-15T22:51:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux x86_64 2:14.2.22-0.el8 ceph 25 k 2024-05-15T22:51:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-05-15T22:51:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-05-15T22:51:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2 x86_64 2:14.2.22-0.el8 ceph 520 k 2024-05-15T22:51:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-05-15T22:51:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-05-15T22:51:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 x86_64 2:14.2.22-0.el8 ceph 376 k 2024-05-15T22:51:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-05-15T22:51:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2 x86_64 2:14.2.22-0.el8 ceph 5.3 M 2024-05-15T22:51:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-05-15T22:51:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-05-15T22:51:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse x86_64 2:14.2.22-0.el8 ceph 39 k 2024-05-15T22:51:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs x86_64 2:14.2.22-0.el8 ceph 159 k 2024-05-15T22:51:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-05-15T22:51:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados x86_64 2:14.2.22-0.el8 ceph 323 k 2024-05-15T22:51:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd x86_64 2:14.2.22-0.el8 ceph 291 k 2024-05-15T22:51:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw x86_64 2:14.2.22-0.el8 ceph 113 k 2024-05-15T22:51:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:51:01.343 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T22:51:01.343 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:51:01.343 INFO:teuthology.orchestra.run.smithi088.stdout:Install 20 Packages 2024-05-15T22:51:01.343 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrade 2 Packages 2024-05-15T22:51:01.344 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:51:01.344 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 45 M 2024-05-15T22:51:01.344 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T22:51:02.183 INFO:teuthology.orchestra.run.smithi088.stdout:(1/22): ceph-base-14.2.22-0.el8.x86_64.rpm 23 MB/s | 5.4 MB 00:00 2024-05-15T22:51:02.356 INFO:teuthology.orchestra.run.smithi088.stdout:(2/22): ceph-selinux-14.2.22-0.el8.x86_64.rpm 144 kB/s | 25 kB 00:00 2024-05-15T22:51:02.409 INFO:teuthology.orchestra.run.smithi088.stdout:(3/22): libcephfs2-14.2.22-0.el8.x86_64.rpm 9.7 MB/s | 520 kB 00:00 2024-05-15T22:51:02.501 INFO:teuthology.orchestra.run.smithi088.stdout:(4/22): ceph-radosgw-14.2.22-0.el8.x86_64.rpm 11 MB/s | 6.0 MB 00:00 2024-05-15T22:51:02.526 INFO:teuthology.orchestra.run.smithi088.stdout:(5/22): libradosstriper1-14.2.22-0.el8.x86_64.r 3.2 MB/s | 376 kB 00:00 2024-05-15T22:51:02.701 INFO:teuthology.orchestra.run.smithi088.stdout:(6/22): librgw2-14.2.22-0.el8.x86_64.rpm 26 MB/s | 5.3 MB 00:00 2024-05-15T22:51:02.726 INFO:teuthology.orchestra.run.smithi088.stdout:(7/22): python3-ceph-argparse-14.2.22-0.el8.x86 196 kB/s | 39 kB 00:00 2024-05-15T22:51:02.752 INFO:teuthology.orchestra.run.smithi088.stdout:(8/22): python3-cephfs-14.2.22-0.el8.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-05-15T22:51:02.809 INFO:teuthology.orchestra.run.smithi088.stdout:(9/22): python3-rados-14.2.22-0.el8.x86_64.rpm 3.9 MB/s | 323 kB 00:00 2024-05-15T22:51:02.818 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:51:02.820 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:51:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-05-15T22:51:02.821 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:51:02.821 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-05-15T22:51:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:14.2.22-0.el8 ceph 6.0 M 2024-05-15T22:51:02.821 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-05-15T22:51:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:14.2.22-0.el8 ceph 3.5 M 2024-05-15T22:51:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:14.2.22-0.el8 ceph 1.7 M 2024-05-15T22:51:02.821 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-05-15T22:51:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:14.2.22-0.el8 ceph 5.4 M 2024-05-15T22:51:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:14.2.22-0.el8 ceph 19 M 2024-05-15T22:51:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:14.2.22-0.el8 ceph 25 k 2024-05-15T22:51:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-05-15T22:51:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-05-15T22:51:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:14.2.22-0.el8 ceph 520 k 2024-05-15T22:51:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-05-15T22:51:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-05-15T22:51:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:14.2.22-0.el8 ceph 376 k 2024-05-15T22:51:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-05-15T22:51:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:14.2.22-0.el8 ceph 5.3 M 2024-05-15T22:51:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-05-15T22:51:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-05-15T22:51:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse x86_64 2:14.2.22-0.el8 ceph 39 k 2024-05-15T22:51:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:14.2.22-0.el8 ceph 159 k 2024-05-15T22:51:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-05-15T22:51:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:14.2.22-0.el8 ceph 323 k 2024-05-15T22:51:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:14.2.22-0.el8 ceph 291 k 2024-05-15T22:51:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:14.2.22-0.el8 ceph 113 k 2024-05-15T22:51:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:51:02.824 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-05-15T22:51:02.824 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:51:02.824 INFO:teuthology.orchestra.run.smithi090.stdout:Install 20 Packages 2024-05-15T22:51:02.824 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 2 Packages 2024-05-15T22:51:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:51:02.825 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 45 M 2024-05-15T22:51:02.825 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-05-15T22:51:02.844 INFO:teuthology.orchestra.run.smithi088.stdout:(10/22): python3-rbd-14.2.22-0.el8.x86_64.rpm 3.1 MB/s | 291 kB 00:00 2024-05-15T22:51:02.869 INFO:teuthology.orchestra.run.smithi088.stdout:(11/22): python3-rgw-14.2.22-0.el8.x86_64.rpm 1.8 MB/s | 113 kB 00:00 2024-05-15T22:51:03.095 INFO:teuthology.orchestra.run.smithi088.stdout:(12/22): ceph-common-14.2.22-0.el8.x86_64.rpm 17 MB/s | 19 MB 00:01 2024-05-15T22:51:03.280 INFO:teuthology.orchestra.run.smithi090.stdout:(1/22): ceph-base-14.2.22-0.el8.x86_64.rpm 38 MB/s | 5.4 MB 00:00 2024-05-15T22:51:03.306 INFO:teuthology.orchestra.run.smithi090.stdout:(2/22): ceph-selinux-14.2.22-0.el8.x86_64.rpm 998 kB/s | 25 kB 00:00 2024-05-15T22:51:03.341 INFO:teuthology.orchestra.run.smithi090.stdout:(3/22): libcephfs2-14.2.22-0.el8.x86_64.rpm 15 MB/s | 520 kB 00:00 2024-05-15T22:51:03.366 INFO:teuthology.orchestra.run.smithi090.stdout:(4/22): libradosstriper1-14.2.22-0.el8.x86_64.r 15 MB/s | 376 kB 00:00 2024-05-15T22:51:03.420 INFO:teuthology.orchestra.run.smithi088.stdout:(13/22): python3-prettytable-0.7.2-14.el8.noarc 135 kB/s | 44 kB 00:00 2024-05-15T22:51:03.467 INFO:teuthology.orchestra.run.smithi090.stdout:(5/22): ceph-radosgw-14.2.22-0.el8.x86_64.rpm 18 MB/s | 6.0 MB 00:00 2024-05-15T22:51:03.487 INFO:teuthology.orchestra.run.smithi088.stdout:(14/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 421 kB/s | 259 kB 00:00 2024-05-15T22:51:03.492 INFO:teuthology.orchestra.run.smithi090.stdout:(6/22): python3-ceph-argparse-14.2.22-0.el8.x86 1.6 MB/s | 39 kB 00:00 2024-05-15T22:51:03.517 INFO:teuthology.orchestra.run.smithi090.stdout:(7/22): python3-cephfs-14.2.22-0.el8.x86_64.rpm 6.2 MB/s | 159 kB 00:00 2024-05-15T22:51:03.545 INFO:teuthology.orchestra.run.smithi088.stdout:(15/22): librabbitmq-0.9.0-5.el8.x86_64.rpm 376 kB/s | 47 kB 00:00 2024-05-15T22:51:03.551 INFO:teuthology.orchestra.run.smithi090.stdout:(8/22): python3-rados-14.2.22-0.el8.x86_64.rpm 9.5 MB/s | 323 kB 00:00 2024-05-15T22:51:03.579 INFO:teuthology.orchestra.run.smithi088.stdout:(16/22): librdkafka-1.6.1-1.el8.x86_64.rpm 921 kB/s | 677 kB 00:00 2024-05-15T22:51:03.584 INFO:teuthology.orchestra.run.smithi090.stdout:(9/22): python3-rbd-14.2.22-0.el8.x86_64.rpm 8.6 MB/s | 291 kB 00:00 2024-05-15T22:51:03.610 INFO:teuthology.orchestra.run.smithi090.stdout:(10/22): python3-rgw-14.2.22-0.el8.x86_64.rpm 4.4 MB/s | 113 kB 00:00 2024-05-15T22:51:03.681 INFO:teuthology.orchestra.run.smithi088.stdout:(17/22): gperftools-libs-2.7-9.el8.x86_64.rpm 1.5 MB/s | 306 kB 00:00 2024-05-15T22:51:03.713 INFO:teuthology.orchestra.run.smithi088.stdout:(18/22): liboath-2.6.2-3.el8.x86_64.rpm 440 kB/s | 59 kB 00:00 2024-05-15T22:51:03.718 INFO:teuthology.orchestra.run.smithi090.stdout:(11/22): librgw2-14.2.22-0.el8.x86_64.rpm 15 MB/s | 5.3 MB 00:00 2024-05-15T22:51:03.738 INFO:teuthology.orchestra.run.smithi088.stdout:(19/22): leveldb-1.22-1.el8.x86_64.rpm 940 kB/s | 181 kB 00:00 2024-05-15T22:51:03.763 INFO:teuthology.orchestra.run.smithi088.stdout:(20/22): libunwind-1.3.1-3.el8.x86_64.rpm 920 kB/s | 75 kB 00:00 2024-05-15T22:51:03.864 INFO:teuthology.orchestra.run.smithi088.stdout:(21/22): librbd1-14.2.22-0.el8.x86_64.rpm 14 MB/s | 1.7 MB 00:00 2024-05-15T22:51:03.930 INFO:teuthology.orchestra.run.smithi088.stdout:(22/22): librados2-14.2.22-0.el8.x86_64.rpm 16 MB/s | 3.5 MB 00:00 2024-05-15T22:51:03.932 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:51:03.933 INFO:teuthology.orchestra.run.smithi088.stdout:Total 17 MB/s | 45 MB 00:02 2024-05-15T22:51:03.976 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T22:51:04.023 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T22:51:04.023 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T22:51:04.083 INFO:teuthology.orchestra.run.smithi090.stdout:(12/22): ceph-common-14.2.22-0.el8.x86_64.rpm 21 MB/s | 19 MB 00:00 2024-05-15T22:51:04.202 INFO:teuthology.orchestra.run.smithi090.stdout:(13/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 537 kB/s | 259 kB 00:00 2024-05-15T22:51:04.270 INFO:teuthology.orchestra.run.smithi090.stdout:(14/22): librdkafka-1.6.1-1.el8.x86_64.rpm 1.0 MB/s | 677 kB 00:00 2024-05-15T22:51:04.312 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T22:51:04.315 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T22:51:04.337 INFO:teuthology.orchestra.run.smithi090.stdout:(15/22): librabbitmq-0.9.0-5.el8.x86_64.rpm 352 kB/s | 47 kB 00:00 2024-05-15T22:51:04.404 INFO:teuthology.orchestra.run.smithi090.stdout:(16/22): python3-prettytable-0.7.2-14.el8.noarc 137 kB/s | 44 kB 00:00 2024-05-15T22:51:04.438 INFO:teuthology.orchestra.run.smithi090.stdout:(17/22): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2024-05-15T22:51:04.479 INFO:teuthology.orchestra.run.smithi090.stdout:(18/22): leveldb-1.22-1.el8.x86_64.rpm 1.2 MB/s | 181 kB 00:00 2024-05-15T22:51:04.504 INFO:teuthology.orchestra.run.smithi090.stdout:(19/22): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2024-05-15T22:51:04.530 INFO:teuthology.orchestra.run.smithi090.stdout:(20/22): liboath-2.6.2-3.el8.x86_64.rpm 467 kB/s | 59 kB 00:00 2024-05-15T22:51:04.580 INFO:teuthology.orchestra.run.smithi090.stdout:(21/22): librbd1-14.2.22-0.el8.x86_64.rpm 23 MB/s | 1.7 MB 00:00 2024-05-15T22:51:04.812 INFO:teuthology.orchestra.run.smithi090.stdout:(22/22): librados2-14.2.22-0.el8.x86_64.rpm 11 MB/s | 3.5 MB 00:00 2024-05-15T22:51:04.814 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:51:04.814 INFO:teuthology.orchestra.run.smithi090.stdout:Total 22 MB/s | 45 MB 00:01 2024-05-15T22:51:04.859 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-05-15T22:51:04.907 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-05-15T22:51:04.907 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-05-15T22:51:05.242 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T22:51:05.434 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2024-05-15T22:51:05.448 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-05-15T22:51:05.451 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-05-15T22:51:05.623 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-05-15T22:51:05.982 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-05-15T22:51:06.206 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librados2-2:14.2.22-0.el8.x86_64 3/24 2024-05-15T22:51:06.283 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:14.2.22-0.el8.x86_64 3/24 2024-05-15T22:51:06.498 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-2:14.2.22-0.el8.x86_64 4/24 2024-05-15T22:51:07.039 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-05-15T22:51:07.219 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2024-05-15T22:51:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-05-15T22:51:07.615 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-05-15T22:51:07.743 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-05-15T22:51:07.841 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-05-15T22:51:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:14.2.22-0.el8.x86_64 3/24 2024-05-15T22:51:08.056 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-05-15T22:51:08.067 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:14.2.22-0.el8.x86_64 3/24 2024-05-15T22:51:08.130 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-05-15T22:51:08.299 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:14.2.22-0.el8.x86_64 4/24 2024-05-15T22:51:08.476 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-05-15T22:51:08.677 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-05-15T22:51:08.770 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/24 2024-05-15T22:51:08.930 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-05-15T22:51:08.934 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-05-15T22:51:08.996 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-05-15T22:51:09.017 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-05-15T22:51:09.227 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : leveldb-1.22-1.el8.x86_64 9/24 2024-05-15T22:51:09.390 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 10/24 2024-05-15T22:51:09.583 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-2:14.2.22-0.el8.x86_64 11/24 2024-05-15T22:51:09.605 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/24 2024-05-15T22:51:09.743 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-05-15T22:51:09.787 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-05-15T22:51:09.820 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-05-15T22:51:10.012 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-2:14.2.22-0.el8.x86_64 13/24 2024-05-15T22:51:10.014 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : leveldb-1.22-1.el8.x86_64 9/24 2024-05-15T22:51:10.164 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-05-15T22:51:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 10/24 2024-05-15T22:51:10.221 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-05-15T22:51:10.354 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:14.2.22-0.el8.x86_64 11/24 2024-05-15T22:51:10.457 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2024-05-15T22:51:10.599 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-05-15T22:51:10.647 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2024-05-15T22:51:10.808 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:14.2.22-0.el8.x86_64 13/24 2024-05-15T22:51:10.865 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2024-05-15T22:51:10.968 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-05-15T22:51:11.022 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 18/24 2024-05-15T22:51:11.042 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-05-15T22:51:11.271 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2024-05-15T22:51:11.518 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2024-05-15T22:51:11.703 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2024-05-15T22:51:11.861 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 18/24 2024-05-15T22:51:13.122 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-05-15T22:51:13.338 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-05-15T22:51:13.982 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-05-15T22:51:14.020 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-05-15T22:51:14.202 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-05-15T22:51:14.239 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-05-15T22:51:14.361 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-05-15T22:51:14.521 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-05-15T22:51:14.818 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-05-15T22:51:15.055 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-05-15T22:51:15.161 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-05-15T22:51:15.293 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-05-15T22:51:33.174 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-05-15T22:51:33.174 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /sys 2024-05-15T22:51:33.174 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /proc 2024-05-15T22:51:33.174 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /dev 2024-05-15T22:51:33.174 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /run 2024-05-15T22:51:33.174 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /mnt 2024-05-15T22:51:33.174 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /var/tmp 2024-05-15T22:51:33.175 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /home 2024-05-15T22:51:33.175 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /tmp 2024-05-15T22:51:33.175 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /dev 2024-05-15T22:51:33.175 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:51:33.883 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-05-15T22:51:33.903 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-05-15T22:51:33.903 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-05-15T22:51:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-05-15T22:51:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2024-05-15T22:51:34.804 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2024-05-15T22:51:34.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-2:14.2.22-0.el8.x86_64 1/24 2024-05-15T22:51:34.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-2:14.2.22-0.el8.x86_64 2/24 2024-05-15T22:51:34.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-2:14.2.22-0.el8.x86_64 3/24 2024-05-15T22:51:34.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-selinux-2:14.2.22-0.el8.x86_64 4/24 2024-05-15T22:51:34.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-05-15T22:51:34.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libradosstriper1-2:14.2.22-0.el8.x86_64 6/24 2024-05-15T22:51:34.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librgw2-2:14.2.22-0.el8.x86_64 7/24 2024-05-15T22:51:34.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 8/24 2024-05-15T22:51:34.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-2:14.2.22-0.el8.x86_64 9/24 2024-05-15T22:51:34.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-2:14.2.22-0.el8.x86_64 10/24 2024-05-15T22:51:34.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-2:14.2.22-0.el8.x86_64 11/24 2024-05-15T22:51:34.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-05-15T22:51:34.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 13/24 2024-05-15T22:51:34.806 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 14/24 2024-05-15T22:51:34.806 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 15/24 2024-05-15T22:51:34.806 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 16/24 2024-05-15T22:51:34.806 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 17/24 2024-05-15T22:51:34.806 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : leveldb-1.22-1.el8.x86_64 18/24 2024-05-15T22:51:34.806 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2024-05-15T22:51:34.806 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2024-05-15T22:51:34.806 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:14.2.22-0.el8.x86_64 21/24 2024-05-15T22:51:34.806 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2024-05-15T22:51:34.806 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:14.2.22-0.el8.x86_64 23/24 2024-05-15T22:51:34.919 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-05-15T22:51:34.919 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2024-05-15T22:51:34.919 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2024-05-15T22:51:34.919 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /dev 2024-05-15T22:51:34.919 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /run 2024-05-15T22:51:34.919 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2024-05-15T22:51:34.919 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2024-05-15T22:51:34.919 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2024-05-15T22:51:34.919 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2024-05-15T22:51:34.919 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /dev 2024-05-15T22:51:34.920 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:51:35.604 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-05-15T22:51:35.623 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-05-15T22:51:35.624 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-05-15T22:51:35.793 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2024-05-15T22:51:35.793 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:51:35.793 INFO:teuthology.orchestra.run.smithi088.stdout:Upgraded: 2024-05-15T22:51:35.793 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-2:14.2.22-0.el8.x86_64 librbd1-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:35.793 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T22:51:35.793 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:35.793 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:35.794 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:35.794 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:35.794 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-05-15T22:51:35.794 INFO:teuthology.orchestra.run.smithi088.stdout: leveldb-1.22-1.el8.x86_64 2024-05-15T22:51:35.794 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:35.794 INFO:teuthology.orchestra.run.smithi088.stdout: liboath-2.6.2-3.el8.x86_64 2024-05-15T22:51:35.794 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-05-15T22:51:35.794 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:35.794 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-05-15T22:51:35.794 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:35.794 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind-1.3.1-3.el8.x86_64 2024-05-15T22:51:35.794 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-05-15T22:51:35.794 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:35.795 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:35.795 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-05-15T22:51:35.795 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:35.795 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:35.795 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:35.795 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:51:35.795 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:51:35.851 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-05-15T22:51:35.851 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2024-05-15T22:51:36.089 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-test 2024-05-15T22:51:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2024-05-15T22:51:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:14.2.22-0.el8.x86_64 1/24 2024-05-15T22:51:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:14.2.22-0.el8.x86_64 2/24 2024-05-15T22:51:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:14.2.22-0.el8.x86_64 3/24 2024-05-15T22:51:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:14.2.22-0.el8.x86_64 4/24 2024-05-15T22:51:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-05-15T22:51:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:14.2.22-0.el8.x86_64 6/24 2024-05-15T22:51:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:14.2.22-0.el8.x86_64 7/24 2024-05-15T22:51:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 8/24 2024-05-15T22:51:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:14.2.22-0.el8.x86_64 9/24 2024-05-15T22:51:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:14.2.22-0.el8.x86_64 10/24 2024-05-15T22:51:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:14.2.22-0.el8.x86_64 11/24 2024-05-15T22:51:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-05-15T22:51:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 13/24 2024-05-15T22:51:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 14/24 2024-05-15T22:51:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 15/24 2024-05-15T22:51:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 16/24 2024-05-15T22:51:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 17/24 2024-05-15T22:51:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : leveldb-1.22-1.el8.x86_64 18/24 2024-05-15T22:51:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2024-05-15T22:51:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2024-05-15T22:51:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:14.2.22-0.el8.x86_64 21/24 2024-05-15T22:51:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2024-05-15T22:51:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:14.2.22-0.el8.x86_64 23/24 2024-05-15T22:51:36.822 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:39 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:51:37.449 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2024-05-15T22:51:37.449 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:51:37.449 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-05-15T22:51:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:14.2.22-0.el8.x86_64 librbd1-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:37.450 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-05-15T22:51:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-05-15T22:51:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: leveldb-1.22-1.el8.x86_64 2024-05-15T22:51:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.2-3.el8.x86_64 2024-05-15T22:51:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-05-15T22:51:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-05-15T22:51:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.3.1-3.el8.x86_64 2024-05-15T22:51:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-05-15T22:51:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-05-15T22:51:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:14.2.22-0.el8.x86_64 2024-05-15T22:51:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:51:37.451 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:51:37.747 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-test 2024-05-15T22:51:38.482 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:51:38.482 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:51:38.483 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-05-15T22:51:38.483 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:51:38.483 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T22:51:38.483 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test x86_64 2:14.2.22-0.el8 ceph 35 M 2024-05-15T22:51:38.483 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T22:51:38.483 INFO:teuthology.orchestra.run.smithi088.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-05-15T22:51:38.483 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-05-15T22:51:38.483 INFO:teuthology.orchestra.run.smithi088.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-05-15T22:51:38.483 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-05-15T22:51:38.483 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:51:38.483 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T22:51:38.483 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:51:38.483 INFO:teuthology.orchestra.run.smithi088.stdout:Install 5 Packages 2024-05-15T22:51:38.484 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:51:38.484 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:39 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:51:38.484 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 36 M 2024-05-15T22:51:38.485 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 127 M 2024-05-15T22:51:38.485 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T22:51:38.948 INFO:teuthology.orchestra.run.smithi088.stdout:(1/5): jq-1.6-8.el8.x86_64.rpm 502 kB/s | 203 kB 00:00 2024-05-15T22:51:38.973 INFO:teuthology.orchestra.run.smithi088.stdout:(2/5): oniguruma-6.8.2-3.el8.x86_64.rpm 438 kB/s | 188 kB 00:00 2024-05-15T22:51:39.073 INFO:teuthology.orchestra.run.smithi088.stdout:(3/5): xmlstarlet-1.6.1-20.el8.x86_64.rpm 706 kB/s | 71 kB 00:00 2024-05-15T22:51:39.123 INFO:teuthology.orchestra.run.smithi088.stdout:(4/5): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2024-05-15T22:51:39.690 INFO:teuthology.orchestra.run.smithi088.stdout:(5/5): ceph-test-14.2.22-0.el8.x86_64.rpm 30 MB/s | 35 MB 00:01 2024-05-15T22:51:39.691 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:51:39.692 INFO:teuthology.orchestra.run.smithi088.stdout:Total 29 MB/s | 36 MB 00:01 2024-05-15T22:51:39.734 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T22:51:39.755 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T22:51:39.755 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T22:51:39.977 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T22:51:39.980 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T22:51:40.132 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:51:40.133 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:51:40.133 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-05-15T22:51:40.133 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:51:40.133 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-05-15T22:51:40.133 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:14.2.22-0.el8 ceph 35 M 2024-05-15T22:51:40.133 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-05-15T22:51:40.133 INFO:teuthology.orchestra.run.smithi090.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-05-15T22:51:40.133 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-05-15T22:51:40.133 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-05-15T22:51:40.133 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-05-15T22:51:40.133 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:51:40.133 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-05-15T22:51:40.133 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:51:40.134 INFO:teuthology.orchestra.run.smithi090.stdout:Install 5 Packages 2024-05-15T22:51:40.134 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:51:40.134 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 36 M 2024-05-15T22:51:40.134 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 127 M 2024-05-15T22:51:40.134 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-05-15T22:51:40.389 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T22:51:40.621 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/5 2024-05-15T22:51:40.723 INFO:teuthology.orchestra.run.smithi090.stdout:(1/5): ceph-test-14.2.22-0.el8.x86_64.rpm 66 MB/s | 35 MB 00:00 2024-05-15T22:51:40.747 INFO:teuthology.orchestra.run.smithi090.stdout:(2/5): oniguruma-6.8.2-3.el8.x86_64.rpm 339 kB/s | 188 kB 00:00 2024-05-15T22:51:40.773 INFO:teuthology.orchestra.run.smithi090.stdout:(3/5): jq-1.6-8.el8.x86_64.rpm 350 kB/s | 203 kB 00:00 2024-05-15T22:51:40.809 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2024-05-15T22:51:40.848 INFO:teuthology.orchestra.run.smithi090.stdout:(4/5): xmlstarlet-1.6.1-20.el8.x86_64.rpm 704 kB/s | 71 kB 00:00 2024-05-15T22:51:40.990 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-05-15T22:51:41.070 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/5 2024-05-15T22:51:41.190 INFO:teuthology.orchestra.run.smithi090.stdout:(5/5): socat-1.7.4.1-1.el8.x86_64.rpm 692 kB/s | 323 kB 00:00 2024-05-15T22:51:41.191 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:51:41.191 INFO:teuthology.orchestra.run.smithi090.stdout:Total 34 MB/s | 36 MB 00:01 2024-05-15T22:51:41.234 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-05-15T22:51:41.254 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-05-15T22:51:41.254 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-05-15T22:51:41.476 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-05-15T22:51:41.479 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-05-15T22:51:41.889 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-05-15T22:51:42.084 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/5 2024-05-15T22:51:42.269 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2024-05-15T22:51:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-05-15T22:51:42.494 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/5 2024-05-15T22:51:44.258 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : jq-1.6-8.el8.x86_64 4/5 2024-05-15T22:51:44.504 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-05-15T22:51:44.782 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-05-15T22:51:44.782 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-2:14.2.22-0.el8.x86_64 1/5 2024-05-15T22:51:44.782 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : jq-1.6-8.el8.x86_64 2/5 2024-05-15T22:51:44.782 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-05-15T22:51:44.782 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2024-05-15T22:51:45.220 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 5/5 2024-05-15T22:51:45.220 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:51:45.221 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T22:51:45.221 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-2:14.2.22-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-05-15T22:51:45.221 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma-6.8.2-3.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2024-05-15T22:51:45.221 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-05-15T22:51:45.221 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:51:45.221 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:51:45.584 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph 2024-05-15T22:51:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : jq-1.6-8.el8.x86_64 4/5 2024-05-15T22:51:45.990 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-05-15T22:51:46.275 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-05-15T22:51:46.276 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:14.2.22-0.el8.x86_64 1/5 2024-05-15T22:51:46.276 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : jq-1.6-8.el8.x86_64 2/5 2024-05-15T22:51:46.276 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-05-15T22:51:46.276 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2024-05-15T22:51:46.315 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:49 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:51:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 5/5 2024-05-15T22:51:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:51:46.680 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-05-15T22:51:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:14.2.22-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-05-15T22:51:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma-6.8.2-3.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2024-05-15T22:51:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-05-15T22:51:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:51:46.681 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:51:46.868 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph 2024-05-15T22:51:47.599 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:48 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:51:47.989 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:51:47.992 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:51:47.992 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T22:51:47.992 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:51:47.992 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T22:51:47.992 INFO:teuthology.orchestra.run.smithi088.stdout: ceph x86_64 2:14.2.22-0.el8 ceph 6.3 k 2024-05-15T22:51:47.992 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T22:51:47.992 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds x86_64 2:14.2.22-0.el8 ceph 2.0 M 2024-05-15T22:51:47.993 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr x86_64 2:14.2.22-0.el8 ceph 1.6 M 2024-05-15T22:51:47.993 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon x86_64 2:14.2.22-0.el8 ceph 4.5 M 2024-05-15T22:51:47.993 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd x86_64 2:14.2.22-0.el8 ceph 17 M 2024-05-15T22:51:47.993 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-05-15T22:51:47.993 INFO:teuthology.orchestra.run.smithi088.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-05-15T22:51:47.993 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-05-15T22:51:47.993 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-05-15T22:51:47.993 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-05-15T22:51:47.993 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-05-15T22:51:47.993 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-05-15T22:51:47.993 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-05-15T22:51:47.993 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-05-15T22:51:47.993 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-05-15T22:51:47.993 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-05-15T22:51:47.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-05-15T22:51:47.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-05-15T22:51:47.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-05-15T22:51:47.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-05-15T22:51:47.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-05-15T22:51:47.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-05-15T22:51:47.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-05-15T22:51:47.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-05-15T22:51:47.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-05-15T22:51:47.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-05-15T22:51:47.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-05-15T22:51:47.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-05-15T22:51:47.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-05-15T22:51:47.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-05-15T22:51:47.995 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-05-15T22:51:47.995 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:51:47.995 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T22:51:47.995 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:51:47.995 INFO:teuthology.orchestra.run.smithi088.stdout:Install 31 Packages 2024-05-15T22:51:47.995 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:51:47.996 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 28 M 2024-05-15T22:51:47.996 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 93 M 2024-05-15T22:51:47.996 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T22:51:48.515 INFO:teuthology.orchestra.run.smithi088.stdout:(1/31): ceph-14.2.22-0.el8.x86_64.rpm 89 kB/s | 6.3 kB 00:00 2024-05-15T22:51:48.565 INFO:teuthology.orchestra.run.smithi088.stdout:(2/31): ceph-mds-14.2.22-0.el8.x86_64.rpm 16 MB/s | 2.0 MB 00:00 2024-05-15T22:51:49.090 INFO:teuthology.orchestra.run.smithi088.stdout:(3/31): ceph-mgr-14.2.22-0.el8.x86_64.rpm 2.4 MB/s | 1.6 MB 00:00 2024-05-15T22:51:49.158 INFO:teuthology.orchestra.run.smithi088.stdout:(4/31): ceph-mon-14.2.22-0.el8.x86_64.rpm 6.9 MB/s | 4.5 MB 00:00 2024-05-15T22:51:49.256 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:51:49.259 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:51:49.259 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-05-15T22:51:49.259 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:51:49.259 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-05-15T22:51:49.259 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:14.2.22-0.el8 ceph 6.3 k 2024-05-15T22:51:49.259 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-05-15T22:51:49.259 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:14.2.22-0.el8 ceph 2.0 M 2024-05-15T22:51:49.259 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:14.2.22-0.el8 ceph 1.6 M 2024-05-15T22:51:49.260 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:14.2.22-0.el8 ceph 4.5 M 2024-05-15T22:51:49.260 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:14.2.22-0.el8 ceph 17 M 2024-05-15T22:51:49.260 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-05-15T22:51:49.260 INFO:teuthology.orchestra.run.smithi090.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-05-15T22:51:49.260 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-05-15T22:51:49.260 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-05-15T22:51:49.260 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-05-15T22:51:49.260 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-05-15T22:51:49.260 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-05-15T22:51:49.260 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-05-15T22:51:49.260 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-05-15T22:51:49.260 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-05-15T22:51:49.260 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-05-15T22:51:49.260 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-05-15T22:51:49.260 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-05-15T22:51:49.261 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-05-15T22:51:49.261 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-05-15T22:51:49.261 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-05-15T22:51:49.261 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-05-15T22:51:49.261 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-05-15T22:51:49.261 INFO:teuthology.orchestra.run.smithi090.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-05-15T22:51:49.261 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-05-15T22:51:49.261 INFO:teuthology.orchestra.run.smithi090.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-05-15T22:51:49.261 INFO:teuthology.orchestra.run.smithi090.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-05-15T22:51:49.261 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-05-15T22:51:49.261 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-05-15T22:51:49.261 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-05-15T22:51:49.261 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-05-15T22:51:49.261 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:51:49.262 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-05-15T22:51:49.262 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:51:49.262 INFO:teuthology.orchestra.run.smithi090.stdout:Install 31 Packages 2024-05-15T22:51:49.262 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:51:49.262 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 28 M 2024-05-15T22:51:49.262 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 93 M 2024-05-15T22:51:49.262 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-05-15T22:51:49.358 INFO:teuthology.orchestra.run.smithi088.stdout:(5/31): ceph-osd-14.2.22-0.el8.x86_64.rpm 21 MB/s | 17 MB 00:00 2024-05-15T22:51:49.475 INFO:teuthology.orchestra.run.smithi088.stdout:(6/31): python3-mako-1.0.6-14.el8.noarch.rpm 409 kB/s | 157 kB 00:00 2024-05-15T22:51:49.525 INFO:teuthology.orchestra.run.smithi088.stdout:(7/31): python3-markupsafe-0.23-19.el8.x86_64.r 106 kB/s | 39 kB 00:00 2024-05-15T22:51:49.575 INFO:teuthology.orchestra.run.smithi088.stdout:(8/31): python3-pytz-2017.2-11.el8.noarch.rpm 568 kB/s | 57 kB 00:00 2024-05-15T22:51:49.731 INFO:teuthology.orchestra.run.smithi090.stdout:(1/31): ceph-14.2.22-0.el8.x86_64.rpm 103 kB/s | 6.3 kB 00:00 2024-05-15T22:51:49.779 INFO:teuthology.orchestra.run.smithi088.stdout:(9/31): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 246 kB/s | 103 kB 00:00 2024-05-15T22:51:49.789 INFO:teuthology.orchestra.run.smithi090.stdout:(2/31): ceph-mgr-14.2.22-0.el8.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-05-15T22:51:49.826 INFO:teuthology.orchestra.run.smithi088.stdout:(10/31): python3-cffi-1.11.5-6.el8.x86_64.rpm 993 kB/s | 249 kB 00:00 2024-05-15T22:51:49.848 INFO:teuthology.orchestra.run.smithi090.stdout:(3/31): ceph-mds-14.2.22-0.el8.x86_64.rpm 11 MB/s | 2.0 MB 00:00 2024-05-15T22:51:49.851 INFO:teuthology.orchestra.run.smithi088.stdout:(11/31): python3-werkzeug-0.12.2-4.el8.noarch.r 1.4 MB/s | 457 kB 00:00 2024-05-15T22:51:49.877 INFO:teuthology.orchestra.run.smithi088.stdout:(12/31): python3-pycparser-2.14-14.el8.noarch.r 2.1 MB/s | 109 kB 00:00 2024-05-15T22:51:49.940 INFO:teuthology.orchestra.run.smithi090.stdout:(4/31): ceph-mon-14.2.22-0.el8.x86_64.rpm 21 MB/s | 4.5 MB 00:00 2024-05-15T22:51:50.027 INFO:teuthology.orchestra.run.smithi088.stdout:(13/31): python3-cryptography-3.2.1-7.el8.x86_6 2.5 MB/s | 641 kB 00:00 2024-05-15T22:51:50.060 INFO:teuthology.orchestra.run.smithi088.stdout:(14/31): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 209 kB/s | 44 kB 00:00 2024-05-15T22:51:50.111 INFO:teuthology.orchestra.run.smithi088.stdout:(15/31): python3-beautifulsoup4-4.6.3-2.el8.1.n 791 kB/s | 185 kB 00:00 2024-05-15T22:51:50.178 INFO:teuthology.orchestra.run.smithi088.stdout:(16/31): python3-jaraco-6.2-6.el8.noarch.rpm 167 kB/s | 11 kB 00:00 2024-05-15T22:51:50.190 INFO:teuthology.orchestra.run.smithi090.stdout:(5/31): ceph-osd-14.2.22-0.el8.x86_64.rpm 42 MB/s | 17 MB 00:00 2024-05-15T22:51:50.245 INFO:teuthology.orchestra.run.smithi088.stdout:(17/31): python3-jaraco-functools-2.0-4.el8.noa 264 kB/s | 18 kB 00:00 2024-05-15T22:51:50.257 INFO:teuthology.orchestra.run.smithi090.stdout:(6/31): python3-mako-1.0.6-14.el8.noarch.rpm 384 kB/s | 157 kB 00:00 2024-05-15T22:51:50.270 INFO:teuthology.orchestra.run.smithi088.stdout:(18/31): python3-cheroot-8.5.2-1.el8.noarch.rpm 712 kB/s | 173 kB 00:00 2024-05-15T22:51:50.295 INFO:teuthology.orchestra.run.smithi088.stdout:(19/31): python3-cherrypy-18.4.0-1.el8.noarch.r 1.6 MB/s | 384 kB 00:00 2024-05-15T22:51:50.321 INFO:teuthology.orchestra.run.smithi088.stdout:(20/31): python3-logutils-0.3.5-11.el8.noarch.r 645 kB/s | 49 kB 00:00 2024-05-15T22:51:50.358 INFO:teuthology.orchestra.run.smithi090.stdout:(7/31): python3-pytz-2017.2-11.el8.noarch.rpm 569 kB/s | 57 kB 00:00 2024-05-15T22:51:50.362 INFO:teuthology.orchestra.run.smithi088.stdout:(21/31): python3-more-itertools-7.2.0-3.el8.noa 880 kB/s | 59 kB 00:00 2024-05-15T22:51:50.383 INFO:teuthology.orchestra.run.smithi090.stdout:(8/31): python3-markupsafe-0.23-19.el8.x86_64.r 88 kB/s | 39 kB 00:00 2024-05-15T22:51:50.388 INFO:teuthology.orchestra.run.smithi088.stdout:(22/31): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2024-05-15T22:51:50.421 INFO:teuthology.orchestra.run.smithi088.stdout:(23/31): python3-simplegeneric-0.8.1-17.el8.noa 320 kB/s | 19 kB 00:00 2024-05-15T22:51:50.455 INFO:teuthology.orchestra.run.smithi088.stdout:(24/31): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2024-05-15T22:51:50.472 INFO:teuthology.orchestra.run.smithi088.stdout:(25/31): python3-singledispatch-3.4.0.3-18.el8. 285 kB/s | 24 kB 00:00 2024-05-15T22:51:50.489 INFO:teuthology.orchestra.run.smithi088.stdout:(26/31): python3-tempora-1.14.1-5.el8.noarch.rp 434 kB/s | 29 kB 00:00 2024-05-15T22:51:50.522 INFO:teuthology.orchestra.run.smithi088.stdout:(27/31): python3-trustme-0.6.0-4.el8.noarch.rpm 410 kB/s | 27 kB 00:00 2024-05-15T22:51:50.556 INFO:teuthology.orchestra.run.smithi088.stdout:(28/31): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.7 MB/s | 251 kB 00:00 2024-05-15T22:51:50.575 INFO:teuthology.orchestra.run.smithi090.stdout:(9/31): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.3 MB/s | 249 kB 00:00 2024-05-15T22:51:50.589 INFO:teuthology.orchestra.run.smithi088.stdout:(29/31): python3-waitress-1.4.3-1.el8.noarch.rp 2.1 MB/s | 255 kB 00:00 2024-05-15T22:51:50.600 INFO:teuthology.orchestra.run.smithi090.stdout:(10/31): python3-pyOpenSSL-19.0.0-1.el8.noarch. 252 kB/s | 103 kB 00:00 2024-05-15T22:51:50.615 INFO:teuthology.orchestra.run.smithi088.stdout:(30/31): python3-webtest-2.0.33-1.el8.noarch.rp 927 kB/s | 85 kB 00:00 2024-05-15T22:51:50.634 INFO:teuthology.orchestra.run.smithi090.stdout:(11/31): python3-werkzeug-0.12.2-4.el8.noarch.r 1.6 MB/s | 457 kB 00:00 2024-05-15T22:51:50.657 INFO:teuthology.orchestra.run.smithi088.stdout:(31/31): python3-zc-lockfile-2.0-2.el8.noarch.r 228 kB/s | 23 kB 00:00 2024-05-15T22:51:50.658 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:51:50.658 INFO:teuthology.orchestra.run.smithi088.stdout:Total 11 MB/s | 28 MB 00:02 2024-05-15T22:51:50.668 INFO:teuthology.orchestra.run.smithi090.stdout:(12/31): python3-cryptography-3.2.1-7.el8.x86_6 6.8 MB/s | 641 kB 00:00 2024-05-15T22:51:50.740 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T22:51:50.768 INFO:teuthology.orchestra.run.smithi090.stdout:(13/31): python3-pycparser-2.14-14.el8.noarch.r 651 kB/s | 109 kB 00:00 2024-05-15T22:51:50.779 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T22:51:50.779 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T22:51:51.044 INFO:teuthology.orchestra.run.smithi090.stdout:(14/31): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 107 kB/s | 44 kB 00:00 2024-05-15T22:51:51.076 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T22:51:51.079 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T22:51:51.260 INFO:teuthology.orchestra.run.smithi090.stdout:(15/31): python3-beautifulsoup4-4.6.3-2.el8.1.n 313 kB/s | 185 kB 00:00 2024-05-15T22:51:51.352 INFO:teuthology.orchestra.run.smithi090.stdout:(16/31): python3-cheroot-8.5.2-1.el8.noarch.rpm 298 kB/s | 173 kB 00:00 2024-05-15T22:51:51.378 INFO:teuthology.orchestra.run.smithi090.stdout:(17/31): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2024-05-15T22:51:51.429 INFO:teuthology.orchestra.run.smithi090.stdout:(18/31): python3-cherrypy-18.4.0-1.el8.noarch.r 998 kB/s | 384 kB 00:00 2024-05-15T22:51:51.480 INFO:teuthology.orchestra.run.smithi090.stdout:(19/31): python3-jaraco-functools-2.0-4.el8.noa 138 kB/s | 18 kB 00:00 2024-05-15T22:51:51.505 INFO:teuthology.orchestra.run.smithi090.stdout:(20/31): python3-logutils-0.3.5-11.el8.noarch.r 384 kB/s | 49 kB 00:00 2024-05-15T22:51:51.547 INFO:teuthology.orchestra.run.smithi090.stdout:(21/31): python3-more-itertools-7.2.0-3.el8.noa 500 kB/s | 59 kB 00:00 2024-05-15T22:51:51.622 INFO:teuthology.orchestra.run.smithi090.stdout:(22/31): python3-portend-2.6-1.el8.noarch.rpm 133 kB/s | 16 kB 00:00 2024-05-15T22:51:51.664 INFO:teuthology.orchestra.run.smithi090.stdout:(23/31): python3-simplegeneric-0.8.1-17.el8.noa 160 kB/s | 19 kB 00:00 2024-05-15T22:51:51.698 INFO:teuthology.orchestra.run.smithi090.stdout:(24/31): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2024-05-15T22:51:51.732 INFO:teuthology.orchestra.run.smithi090.stdout:(25/31): python3-singledispatch-3.4.0.3-18.el8. 219 kB/s | 24 kB 00:00 2024-05-15T22:51:51.774 INFO:teuthology.orchestra.run.smithi090.stdout:(26/31): python3-tempora-1.14.1-5.el8.noarch.rp 268 kB/s | 29 kB 00:00 2024-05-15T22:51:51.807 INFO:teuthology.orchestra.run.smithi090.stdout:(27/31): python3-trustme-0.6.0-4.el8.noarch.rpm 252 kB/s | 27 kB 00:00 2024-05-15T22:51:51.933 INFO:teuthology.orchestra.run.smithi090.stdout:(28/31): python3-webtest-2.0.33-1.el8.noarch.rp 682 kB/s | 85 kB 00:00 2024-05-15T22:51:51.958 INFO:teuthology.orchestra.run.smithi090.stdout:(29/31): python3-waitress-1.4.3-1.el8.noarch.rp 1.1 MB/s | 255 kB 00:00 2024-05-15T22:51:51.983 INFO:teuthology.orchestra.run.smithi090.stdout:(30/31): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.2 MB/s | 251 kB 00:00 2024-05-15T22:51:52.050 INFO:teuthology.orchestra.run.smithi090.stdout:(31/31): python3-zc-lockfile-2.0-2.el8.noarch.r 196 kB/s | 23 kB 00:00 2024-05-15T22:51:52.055 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:51:52.055 INFO:teuthology.orchestra.run.smithi090.stdout:Total 10 MB/s | 28 MB 00:02 2024-05-15T22:51:52.143 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-05-15T22:51:52.151 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T22:51:52.187 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-05-15T22:51:52.187 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-05-15T22:51:52.359 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/31 2024-05-15T22:51:52.485 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-05-15T22:51:52.489 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-05-15T22:51:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/31 2024-05-15T22:51:52.855 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/31 2024-05-15T22:51:53.083 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/31 2024-05-15T22:51:53.299 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/31 2024-05-15T22:51:53.472 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/31 2024-05-15T22:51:53.547 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-05-15T22:51:53.656 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/31 2024-05-15T22:51:53.779 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/31 2024-05-15T22:51:53.835 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/31 2024-05-15T22:51:54.002 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/31 2024-05-15T22:51:54.044 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/31 2024-05-15T22:51:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/31 2024-05-15T22:51:54.307 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/31 2024-05-15T22:51:54.504 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/31 2024-05-15T22:51:54.520 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/31 2024-05-15T22:51:54.703 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/31 2024-05-15T22:51:54.768 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/31 2024-05-15T22:51:54.859 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/31 2024-05-15T22:51:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/31 2024-05-15T22:51:55.180 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/31 2024-05-15T22:51:55.187 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/31 2024-05-15T22:51:55.425 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/31 2024-05-15T22:51:55.429 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/31 2024-05-15T22:51:55.612 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/31 2024-05-15T22:51:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/31 2024-05-15T22:51:55.876 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/31 2024-05-15T22:51:55.897 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/31 2024-05-15T22:51:56.114 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/31 2024-05-15T22:51:56.119 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/31 2024-05-15T22:51:56.454 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/31 2024-05-15T22:51:56.518 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/31 2024-05-15T22:51:56.677 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/31 2024-05-15T22:51:56.776 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/31 2024-05-15T22:51:56.857 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/31 2024-05-15T22:51:56.948 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/31 2024-05-15T22:51:57.025 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/31 2024-05-15T22:51:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/31 2024-05-15T22:51:57.267 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/31 2024-05-15T22:51:57.425 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/31 2024-05-15T22:51:57.465 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/31 2024-05-15T22:51:57.734 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/31 2024-05-15T22:51:57.767 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/31 2024-05-15T22:51:57.979 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/31 2024-05-15T22:51:58.058 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/31 2024-05-15T22:51:58.143 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/31 2024-05-15T22:51:58.304 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/31 2024-05-15T22:51:58.497 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/31 2024-05-15T22:51:58.548 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/31 2024-05-15T22:51:58.736 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-05-15T22:51:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/31 2024-05-15T22:51:58.975 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/31 2024-05-15T22:51:59.292 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/31 2024-05-15T22:51:59.700 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/31 2024-05-15T22:51:59.911 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-05-15T22:52:00.103 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-05-15T22:52:00.302 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-05-15T22:52:00.697 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-05-15T22:52:00.869 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-05-15T22:52:01.051 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-05-15T22:52:01.269 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-05-15T22:52:01.280 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-05-15T22:52:01.286 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-05-15T22:52:01.410 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-2:14.2.22-0.el8.x86_64 31/31 2024-05-15T22:52:01.461 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-05-15T22:52:01.803 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-2:14.2.22-0.el8.x86_64 31/31 2024-05-15T22:52:01.804 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-2:14.2.22-0.el8.x86_64 1/31 2024-05-15T22:52:01.804 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-2:14.2.22-0.el8.x86_64 2/31 2024-05-15T22:52:01.804 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-2:14.2.22-0.el8.x86_64 3/31 2024-05-15T22:52:01.804 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-2:14.2.22-0.el8.x86_64 4/31 2024-05-15T22:52:01.804 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-2:14.2.22-0.el8.x86_64 5/31 2024-05-15T22:52:01.804 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 6/31 2024-05-15T22:52:01.804 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 7/31 2024-05-15T22:52:01.804 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 8/31 2024-05-15T22:52:01.804 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 9/31 2024-05-15T22:52:01.804 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 10/31 2024-05-15T22:52:01.804 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 11/31 2024-05-15T22:52:01.804 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 12/31 2024-05-15T22:52:01.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 13/31 2024-05-15T22:52:01.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 14/31 2024-05-15T22:52:01.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 15/31 2024-05-15T22:52:01.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 16/31 2024-05-15T22:52:01.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 17/31 2024-05-15T22:52:01.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 18/31 2024-05-15T22:52:01.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 19/31 2024-05-15T22:52:01.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 20/31 2024-05-15T22:52:01.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 21/31 2024-05-15T22:52:01.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 22/31 2024-05-15T22:52:01.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-portend-2.6-1.el8.noarch 23/31 2024-05-15T22:52:01.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 24/31 2024-05-15T22:52:01.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 25/31 2024-05-15T22:52:01.805 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 26/31 2024-05-15T22:52:01.806 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 27/31 2024-05-15T22:52:01.806 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 28/31 2024-05-15T22:52:01.806 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 29/31 2024-05-15T22:52:01.806 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 30/31 2024-05-15T22:52:01.853 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-05-15T22:52:02.016 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-05-15T22:52:02.200 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-05-15T22:52:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-05-15T22:52:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-05-15T22:52:02.582 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:14.2.22-0.el8.x86_64 31/31 2024-05-15T22:52:02.630 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 31/31 2024-05-15T22:52:02.630 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:02.630 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T22:52:02.630 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-2:14.2.22-0.el8.x86_64 2024-05-15T22:52:02.630 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-2:14.2.22-0.el8.x86_64 2024-05-15T22:52:02.630 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-2:14.2.22-0.el8.x86_64 2024-05-15T22:52:02.630 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-2:14.2.22-0.el8.x86_64 2024-05-15T22:52:02.630 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-2:14.2.22-0.el8.x86_64 2024-05-15T22:52:02.630 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-05-15T22:52:02.631 INFO:teuthology.orchestra.run.smithi088.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-05-15T22:52:02.631 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-05-15T22:52:02.631 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-05-15T22:52:02.631 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-05-15T22:52:02.631 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-05-15T22:52:02.631 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-6.2-6.el8.noarch 2024-05-15T22:52:02.631 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-05-15T22:52:02.631 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-05-15T22:52:02.631 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako-1.0.6-14.el8.noarch 2024-05-15T22:52:02.631 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-05-15T22:52:02.631 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-05-15T22:52:02.631 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-05-15T22:52:02.631 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend-2.6-1.el8.noarch 2024-05-15T22:52:02.631 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-05-15T22:52:02.632 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser-2.14-14.el8.noarch 2024-05-15T22:52:02.632 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz-2017.2-11.el8.noarch 2024-05-15T22:52:02.632 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-05-15T22:52:02.632 INFO:teuthology.orchestra.run.smithi088.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-05-15T22:52:02.632 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-05-15T22:52:02.632 INFO:teuthology.orchestra.run.smithi088.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-05-15T22:52:02.632 INFO:teuthology.orchestra.run.smithi088.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-05-15T22:52:02.632 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-05-15T22:52:02.632 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-05-15T22:52:02.632 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-05-15T22:52:02.632 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-05-15T22:52:02.632 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:02.632 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:52:02.891 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-base 2024-05-15T22:52:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-2:14.2.22-0.el8.x86_64 31/31 2024-05-15T22:52:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:14.2.22-0.el8.x86_64 1/31 2024-05-15T22:52:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:14.2.22-0.el8.x86_64 2/31 2024-05-15T22:52:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:14.2.22-0.el8.x86_64 3/31 2024-05-15T22:52:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:14.2.22-0.el8.x86_64 4/31 2024-05-15T22:52:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:14.2.22-0.el8.x86_64 5/31 2024-05-15T22:52:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 6/31 2024-05-15T22:52:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 7/31 2024-05-15T22:52:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 8/31 2024-05-15T22:52:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 9/31 2024-05-15T22:52:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 10/31 2024-05-15T22:52:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 11/31 2024-05-15T22:52:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 12/31 2024-05-15T22:52:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 13/31 2024-05-15T22:52:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 14/31 2024-05-15T22:52:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 15/31 2024-05-15T22:52:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 16/31 2024-05-15T22:52:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 17/31 2024-05-15T22:52:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 18/31 2024-05-15T22:52:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 19/31 2024-05-15T22:52:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 20/31 2024-05-15T22:52:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 21/31 2024-05-15T22:52:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 22/31 2024-05-15T22:52:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-2.6-1.el8.noarch 23/31 2024-05-15T22:52:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 24/31 2024-05-15T22:52:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 25/31 2024-05-15T22:52:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 26/31 2024-05-15T22:52:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 27/31 2024-05-15T22:52:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 28/31 2024-05-15T22:52:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 29/31 2024-05-15T22:52:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 30/31 2024-05-15T22:52:03.623 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:06 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:52:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 31/31 2024-05-15T22:52:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:03.884 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-05-15T22:52:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:14.2.22-0.el8.x86_64 2024-05-15T22:52:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:14.2.22-0.el8.x86_64 2024-05-15T22:52:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:14.2.22-0.el8.x86_64 2024-05-15T22:52:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:14.2.22-0.el8.x86_64 2024-05-15T22:52:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:14.2.22-0.el8.x86_64 2024-05-15T22:52:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-05-15T22:52:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-05-15T22:52:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-05-15T22:52:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-05-15T22:52:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-05-15T22:52:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-05-15T22:52:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-6.2-6.el8.noarch 2024-05-15T22:52:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-05-15T22:52:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-05-15T22:52:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.0.6-14.el8.noarch 2024-05-15T22:52:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-05-15T22:52:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-05-15T22:52:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-05-15T22:52:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-2.6-1.el8.noarch 2024-05-15T22:52:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-05-15T22:52:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.14-14.el8.noarch 2024-05-15T22:52:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2017.2-11.el8.noarch 2024-05-15T22:52:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-05-15T22:52:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-05-15T22:52:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-05-15T22:52:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-05-15T22:52:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-05-15T22:52:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-05-15T22:52:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-05-15T22:52:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-05-15T22:52:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-05-15T22:52:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:03.886 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:52:04.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-base 2024-05-15T22:52:04.913 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:05 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:52:05.233 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-base-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:52:05.282 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:52:05.283 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T22:52:05.283 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:52:05.398 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr 2024-05-15T22:52:06.135 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:09 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:52:06.516 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-base-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:52:06.564 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:52:06.565 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-05-15T22:52:06.566 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:52:06.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr 2024-05-15T22:52:07.410 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:08 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:52:07.736 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-mgr-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:52:07.784 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:52:07.785 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T22:52:07.785 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:52:07.915 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-dashboard 2024-05-15T22:52:08.669 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:11 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:52:09.014 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-mgr-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:52:09.063 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:52:09.064 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-05-15T22:52:09.064 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:52:09.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-dashboard 2024-05-15T22:52:09.933 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:10 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:52:10.321 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:52:10.322 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T22:52:10.322 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:10.322 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T22:52:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard noarch 2:14.2.22-0.el8 ceph-noarch 3.7 M 2024-05-15T22:52:10.322 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T22:52:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards noarch 2:14.2.22-0.el8 ceph-noarch 24 k 2024-05-15T22:52:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-05-15T22:52:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-05-15T22:52:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-05-15T22:52:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:10.323 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T22:52:10.323 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:10.323 INFO:teuthology.orchestra.run.smithi088.stdout:Install 5 Packages 2024-05-15T22:52:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:10.323 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 4.0 M 2024-05-15T22:52:10.323 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 17 M 2024-05-15T22:52:10.323 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T22:52:10.495 INFO:teuthology.orchestra.run.smithi088.stdout:(1/5): ceph-grafana-dashboards-14.2.22-0.el8.no 410 kB/s | 24 kB 00:00 2024-05-15T22:52:10.587 INFO:teuthology.orchestra.run.smithi088.stdout:(2/5): ceph-mgr-dashboard-14.2.22-0.el8.noarch. 25 MB/s | 3.7 MB 00:00 2024-05-15T22:52:10.613 INFO:teuthology.orchestra.run.smithi088.stdout:(3/5): python3-jwt-1.6.1-2.el8.noarch.rpm 242 kB/s | 43 kB 00:00 2024-05-15T22:52:10.696 INFO:teuthology.orchestra.run.smithi088.stdout:(4/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 168 kB/s | 34 kB 00:00 2024-05-15T22:52:10.797 INFO:teuthology.orchestra.run.smithi088.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 941 kB/s | 196 kB 00:00 2024-05-15T22:52:10.797 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:52:10.798 INFO:teuthology.orchestra.run.smithi088.stdout:Total 8.5 MB/s | 4.0 MB 00:00 2024-05-15T22:52:10.807 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T22:52:10.817 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T22:52:10.817 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T22:52:10.985 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T22:52:10.990 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T22:52:11.242 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T22:52:11.480 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-05-15T22:52:11.574 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:52:11.575 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:11.575 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-05-15T22:52:11.575 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:11.575 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-05-15T22:52:11.575 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:14.2.22-0.el8 ceph-noarch 3.7 M 2024-05-15T22:52:11.575 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-05-15T22:52:11.575 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:14.2.22-0.el8 ceph-noarch 24 k 2024-05-15T22:52:11.575 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-05-15T22:52:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-05-15T22:52:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-05-15T22:52:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:11.576 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-05-15T22:52:11.576 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:11.576 INFO:teuthology.orchestra.run.smithi090.stdout:Install 5 Packages 2024-05-15T22:52:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:11.576 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 4.0 M 2024-05-15T22:52:11.577 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 17 M 2024-05-15T22:52:11.577 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-05-15T22:52:11.725 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-05-15T22:52:11.879 INFO:teuthology.orchestra.run.smithi090.stdout:(1/5): ceph-grafana-dashboards-14.2.22-0.el8.no 437 kB/s | 24 kB 00:00 2024-05-15T22:52:11.940 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-05-15T22:52:11.954 INFO:teuthology.orchestra.run.smithi090.stdout:(2/5): ceph-mgr-dashboard-14.2.22-0.el8.noarch. 29 MB/s | 3.7 MB 00:00 2024-05-15T22:52:11.988 INFO:teuthology.orchestra.run.smithi090.stdout:(3/5): python3-jwt-1.6.1-2.el8.noarch.rpm 260 kB/s | 43 kB 00:00 2024-05-15T22:52:12.046 INFO:teuthology.orchestra.run.smithi090.stdout:(4/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 201 kB/s | 34 kB 00:00 2024-05-15T22:52:12.113 INFO:teuthology.orchestra.run.smithi090.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2024-05-15T22:52:12.114 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:52:12.115 INFO:teuthology.orchestra.run.smithi090.stdout:Total 7.5 MB/s | 4.0 MB 00:00 2024-05-15T22:52:12.124 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-05-15T22:52:12.133 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-05-15T22:52:12.133 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-05-15T22:52:12.298 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-05-15T22:52:12.301 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-05-15T22:52:12.540 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-05-15T22:52:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-05-15T22:52:12.844 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 4/5 2024-05-15T22:52:13.041 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-05-15T22:52:13.239 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-05-15T22:52:13.963 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-05-15T22:52:14.132 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 4/5 2024-05-15T22:52:14.401 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-05-15T22:52:14.686 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-05-15T22:52:14.686 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 1/5 2024-05-15T22:52:14.686 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2/5 2024-05-15T22:52:14.686 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-05-15T22:52:14.686 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-05-15T22:52:14.698 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-05-15T22:52:14.698 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 1/5 2024-05-15T22:52:14.698 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2/5 2024-05-15T22:52:14.698 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-05-15T22:52:14.698 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-05-15T22:52:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-05-15T22:52:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:15.066 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-05-15T22:52:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 2024-05-15T22:52:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2024-05-15T22:52:15.067 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-05-15T22:52:15.067 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-05-15T22:52:15.067 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.4.1-12.el8.noarch 2024-05-15T22:52:15.067 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:15.067 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:52:15.083 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-05-15T22:52:15.083 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:15.083 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T22:52:15.083 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 2024-05-15T22:52:15.083 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2024-05-15T22:52:15.083 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-05-15T22:52:15.084 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-05-15T22:52:15.084 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes-2.4.1-12.el8.noarch 2024-05-15T22:52:15.084 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:15.084 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:52:15.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-15T22:52:15.216 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-15T22:52:15.914 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:16 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:52:15.942 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:18 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:52:17.571 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:52:17.572 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2024-05-15T22:52:17.572 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T22:52:17.572 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2024-05-15T22:52:17.573 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T22:52:17.573 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local noarch 2:14.2.22-0.el8 ceph-noarch 1.1 M 2024-05-15T22:52:17.573 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T22:52:17.573 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran x86_64 8.5.0-22.el8 baseos 645 k 2024-05-15T22:52:17.573 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath x86_64 8.5.0-22.el8 baseos 172 k 2024-05-15T22:52:17.573 INFO:teuthology.orchestra.run.smithi088.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-05-15T22:52:17.573 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-05-15T22:52:17.573 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-05-15T22:52:17.573 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-05-15T22:52:17.573 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-05-15T22:52:17.573 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:17.573 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T22:52:17.573 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2024-05-15T22:52:17.574 INFO:teuthology.orchestra.run.smithi088.stdout:Install 8 Packages 2024-05-15T22:52:17.574 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:17.575 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 30 M 2024-05-15T22:52:17.575 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 172 M 2024-05-15T22:52:17.575 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T22:52:17.575 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:52:17.575 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-05-15T22:52:17.575 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-05-15T22:52:17.575 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-05-15T22:52:17.576 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-05-15T22:52:17.576 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:14.2.22-0.el8 ceph-noarch 1.1 M 2024-05-15T22:52:17.576 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-05-15T22:52:17.576 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 8.5.0-22.el8 baseos 645 k 2024-05-15T22:52:17.576 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 8.5.0-22.el8 baseos 172 k 2024-05-15T22:52:17.576 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-05-15T22:52:17.576 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-05-15T22:52:17.576 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-05-15T22:52:17.576 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-05-15T22:52:17.576 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-05-15T22:52:17.576 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:17.576 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-05-15T22:52:17.576 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-05-15T22:52:17.576 INFO:teuthology.orchestra.run.smithi090.stdout:Install 8 Packages 2024-05-15T22:52:17.577 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:17.577 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 30 M 2024-05-15T22:52:17.577 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 172 M 2024-05-15T22:52:17.577 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-05-15T22:52:17.936 INFO:teuthology.orchestra.run.smithi090.stdout:(1/8): ceph-mgr-diskprediction-local-14.2.22-0. 10 MB/s | 1.1 MB 00:00 2024-05-15T22:52:18.008 INFO:teuthology.orchestra.run.smithi088.stdout:(1/8): ceph-mgr-diskprediction-local-14.2.22-0. 20 MB/s | 1.1 MB 00:00 2024-05-15T22:52:18.746 INFO:teuthology.orchestra.run.smithi090.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 5.1 MB/s | 4.7 MB 00:00 2024-05-15T22:52:18.808 INFO:teuthology.orchestra.run.smithi088.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.7 MB/s | 3.7 MB 00:00 2024-05-15T22:52:18.821 INFO:teuthology.orchestra.run.smithi090.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.9 MB/s | 4.9 MB 00:00 2024-05-15T22:52:18.883 INFO:teuthology.orchestra.run.smithi088.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 5.1 MB/s | 4.7 MB 00:00 2024-05-15T22:52:18.896 INFO:teuthology.orchestra.run.smithi090.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.9 MB/s | 3.7 MB 00:00 2024-05-15T22:52:18.922 INFO:teuthology.orchestra.run.smithi090.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.3 MB/s | 225 kB 00:00 2024-05-15T22:52:18.942 INFO:teuthology.orchestra.run.smithi088.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.9 MB/s | 4.9 MB 00:00 2024-05-15T22:52:18.967 INFO:teuthology.orchestra.run.smithi088.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.4 MB/s | 225 kB 00:00 2024-05-15T22:52:19.081 INFO:teuthology.orchestra.run.smithi090.stdout:(6/8): libquadmath-8.5.0-22.el8.x86_64.rpm 1.1 MB/s | 172 kB 00:00 2024-05-15T22:52:19.134 INFO:teuthology.orchestra.run.smithi088.stdout:(6/8): libquadmath-8.5.0-22.el8.x86_64.rpm 1.0 MB/s | 172 kB 00:00 2024-05-15T22:52:19.139 INFO:teuthology.orchestra.run.smithi090.stdout:(7/8): libgfortran-8.5.0-22.el8.x86_64.rpm 2.6 MB/s | 645 kB 00:00 2024-05-15T22:52:19.192 INFO:teuthology.orchestra.run.smithi088.stdout:(7/8): libgfortran-8.5.0-22.el8.x86_64.rpm 2.5 MB/s | 645 kB 00:00 2024-05-15T22:52:19.650 INFO:teuthology.orchestra.run.smithi090.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 17 MB/s | 14 MB 00:00 2024-05-15T22:52:19.650 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:52:19.650 INFO:teuthology.orchestra.run.smithi090.stdout:Total 14 MB/s | 30 MB 00:02 2024-05-15T22:52:19.760 INFO:teuthology.orchestra.run.smithi088.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 16 MB/s | 14 MB 00:00 2024-05-15T22:52:19.760 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:52:19.761 INFO:teuthology.orchestra.run.smithi088.stdout:Total 14 MB/s | 30 MB 00:02 2024-05-15T22:52:19.882 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-05-15T22:52:19.900 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-05-15T22:52:19.900 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-05-15T22:52:19.995 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T22:52:20.014 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T22:52:20.014 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T22:52:20.213 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-05-15T22:52:20.216 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-05-15T22:52:20.331 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T22:52:20.334 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T22:52:20.733 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-05-15T22:52:20.877 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T22:52:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-8.5.0-22.el8.x86_64 1/8 2024-05-15T22:52:21.005 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libquadmath-8.5.0-22.el8.x86_64 1/8 2024-05-15T22:52:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libquadmath-8.5.0-22.el8.x86_64 1/8 2024-05-15T22:52:21.156 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libquadmath-8.5.0-22.el8.x86_64 1/8 2024-05-15T22:52:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-8.5.0-22.el8.x86_64 2/8 2024-05-15T22:52:21.305 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libgfortran-8.5.0-22.el8.x86_64 2/8 2024-05-15T22:52:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libgfortran-8.5.0-22.el8.x86_64 2/8 2024-05-15T22:52:21.882 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libgfortran-8.5.0-22.el8.x86_64 2/8 2024-05-15T22:52:21.989 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-05-15T22:52:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-05-15T22:52:22.611 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-05-15T22:52:22.663 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-05-15T22:52:22.764 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-05-15T22:52:22.830 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-05-15T22:52:23.464 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-05-15T22:52:23.510 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-05-15T22:52:23.784 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-05-15T22:52:23.833 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-05-15T22:52:24.023 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-05-15T22:52:24.055 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-05-15T22:52:25.796 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-05-15T22:52:25.800 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-05-15T22:52:26.375 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-05-15T22:52:26.399 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-05-15T22:52:26.579 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-05-15T22:52:26.597 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-05-15T22:52:26.887 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-05-15T22:52:26.887 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 1/8 2024-05-15T22:52:26.887 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-05-15T22:52:26.887 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-05-15T22:52:26.887 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-05-15T22:52:26.887 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-05-15T22:52:26.887 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-05-15T22:52:26.887 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libgfortran-8.5.0-22.el8.x86_64 7/8 2024-05-15T22:52:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-05-15T22:52:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 1/8 2024-05-15T22:52:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-05-15T22:52:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-05-15T22:52:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-05-15T22:52:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-05-15T22:52:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-05-15T22:52:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-8.5.0-22.el8.x86_64 7/8 2024-05-15T22:52:27.304 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libquadmath-8.5.0-22.el8.x86_64 8/8 2024-05-15T22:52:27.304 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:27.304 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T22:52:27.304 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 2024-05-15T22:52:27.305 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran-8.5.0-22.el8.x86_64 2024-05-15T22:52:27.305 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath-8.5.0-22.el8.x86_64 2024-05-15T22:52:27.305 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-0.3.15-4.el8.x86_64 2024-05-15T22:52:27.305 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-05-15T22:52:27.305 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-05-15T22:52:27.305 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-05-15T22:52:27.305 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-05-15T22:52:27.305 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:27.305 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:52:27.342 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-8.5.0-22.el8.x86_64 8/8 2024-05-15T22:52:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:27.343 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-05-15T22:52:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 2024-05-15T22:52:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-8.5.0-22.el8.x86_64 2024-05-15T22:52:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-8.5.0-22.el8.x86_64 2024-05-15T22:52:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.15-4.el8.x86_64 2024-05-15T22:52:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-05-15T22:52:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-05-15T22:52:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-05-15T22:52:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-05-15T22:52:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:27.343 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:52:27.451 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-rook 2024-05-15T22:52:27.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-rook 2024-05-15T22:52:28.178 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:31 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:52:28.194 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:29 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:52:29.804 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:52:29.806 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:29.806 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T22:52:29.806 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:29.806 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T22:52:29.806 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook noarch 2:14.2.22-0.el8 ceph-noarch 15 k 2024-05-15T22:52:29.806 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T22:52:29.806 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-05-15T22:52:29.806 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-05-15T22:52:29.806 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-05-15T22:52:29.806 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-05-15T22:52:29.806 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-05-15T22:52:29.807 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-05-15T22:52:29.807 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-05-15T22:52:29.807 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-05-15T22:52:29.807 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-05-15T22:52:29.807 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-05-15T22:52:29.807 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:29.807 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T22:52:29.807 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:29.807 INFO:teuthology.orchestra.run.smithi088.stdout:Install 11 Packages 2024-05-15T22:52:29.807 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:29.808 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 1.9 M 2024-05-15T22:52:29.808 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 24 M 2024-05-15T22:52:29.808 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T22:52:29.841 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:52:29.842 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:29.842 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-05-15T22:52:29.842 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:29.843 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-05-15T22:52:29.843 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:14.2.22-0.el8 ceph-noarch 15 k 2024-05-15T22:52:29.843 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-05-15T22:52:29.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-05-15T22:52:29.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-05-15T22:52:29.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-05-15T22:52:29.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-05-15T22:52:29.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-05-15T22:52:29.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-05-15T22:52:29.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-05-15T22:52:29.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-05-15T22:52:29.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-05-15T22:52:29.843 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-05-15T22:52:29.843 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:29.843 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-05-15T22:52:29.844 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:29.844 INFO:teuthology.orchestra.run.smithi090.stdout:Install 11 Packages 2024-05-15T22:52:29.844 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:29.844 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 1.9 M 2024-05-15T22:52:29.844 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 24 M 2024-05-15T22:52:29.845 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-05-15T22:52:30.228 INFO:teuthology.orchestra.run.smithi090.stdout:(1/11): ceph-mgr-rook-14.2.22-0.el8.noarch.rpm 318 kB/s | 15 kB 00:00 2024-05-15T22:52:30.387 INFO:teuthology.orchestra.run.smithi090.stdout:(2/11): python3-oauthlib-2.1.0-1.el8.noarch.rpm 977 kB/s | 155 kB 00:00 2024-05-15T22:52:30.446 INFO:teuthology.orchestra.run.smithi090.stdout:(3/11): python3-requests-oauthlib-1.0.0-1.el8.n 730 kB/s | 43 kB 00:00 2024-05-15T22:52:30.579 INFO:teuthology.orchestra.run.smithi090.stdout:(4/11): python3-pyasn1-0.3.7-6.el8.noarch.rpm 317 kB/s | 126 kB 00:00 2024-05-15T22:52:30.605 INFO:teuthology.orchestra.run.smithi090.stdout:(5/11): python3-pyasn1-modules-0.3.7-6.el8.noar 259 kB/s | 110 kB 00:00 2024-05-15T22:52:30.708 INFO:teuthology.orchestra.run.smithi088.stdout:(1/11): ceph-mgr-rook-14.2.22-0.el8.noarch.rpm 28 kB/s | 15 kB 00:00 2024-05-15T22:52:30.713 INFO:teuthology.orchestra.run.smithi090.stdout:(6/11): python3-cachetools-3.1.1-4.el8.noarch.r 123 kB/s | 33 kB 00:00 2024-05-15T22:52:30.831 INFO:teuthology.orchestra.run.smithi090.stdout:(7/11): python3-certifi-2018.10.15-7.el8.noarch 65 kB/s | 16 kB 00:00 2024-05-15T22:52:30.939 INFO:teuthology.orchestra.run.smithi090.stdout:(8/11): python3-google-auth-1.1.1-10.el8.noarch 243 kB/s | 81 kB 00:00 2024-05-15T22:52:31.017 INFO:teuthology.orchestra.run.smithi090.stdout:(9/11): python3-rsa-4.9-2.el8.noarch.rpm 349 kB/s | 64 kB 00:00 2024-05-15T22:52:31.065 INFO:teuthology.orchestra.run.smithi090.stdout:(10/11): python3-websocket-client-0.56.0-5.el8. 489 kB/s | 61 kB 00:00 2024-05-15T22:52:31.159 INFO:teuthology.orchestra.run.smithi088.stdout:(2/11): python3-oauthlib-2.1.0-1.el8.noarch.rpm 344 kB/s | 155 kB 00:00 2024-05-15T22:52:31.174 INFO:teuthology.orchestra.run.smithi090.stdout:(11/11): python3-kubernetes-11.0.0-6.el8.noarch 2.5 MB/s | 1.2 MB 00:00 2024-05-15T22:52:31.174 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:52:31.175 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2024-05-15T22:52:31.213 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-05-15T22:52:31.217 INFO:teuthology.orchestra.run.smithi088.stdout:(3/11): python3-pyasn1-modules-0.3.7-6.el8.noar 105 kB/s | 110 kB 00:01 2024-05-15T22:52:31.229 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-05-15T22:52:31.229 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-05-15T22:52:31.243 INFO:teuthology.orchestra.run.smithi088.stdout:(4/11): python3-pyasn1-0.3.7-6.el8.noarch.rpm 117 kB/s | 126 kB 00:01 2024-05-15T22:52:31.268 INFO:teuthology.orchestra.run.smithi088.stdout:(5/11): python3-requests-oauthlib-1.0.0-1.el8.n 391 kB/s | 43 kB 00:00 2024-05-15T22:52:31.303 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-05-15T22:52:31.306 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-05-15T22:52:31.360 INFO:teuthology.orchestra.run.smithi088.stdout:(6/11): python3-certifi-2018.10.15-7.el8.noarch 139 kB/s | 16 kB 00:00 2024-05-15T22:52:31.385 INFO:teuthology.orchestra.run.smithi088.stdout:(7/11): python3-cachetools-3.1.1-4.el8.noarch.r 197 kB/s | 33 kB 00:00 2024-05-15T22:52:31.410 INFO:teuthology.orchestra.run.smithi088.stdout:(8/11): python3-google-auth-1.1.1-10.el8.noarch 572 kB/s | 81 kB 00:00 2024-05-15T22:52:31.461 INFO:teuthology.orchestra.run.smithi088.stdout:(9/11): python3-rsa-4.9-2.el8.noarch.rpm 854 kB/s | 64 kB 00:00 2024-05-15T22:52:31.486 INFO:teuthology.orchestra.run.smithi088.stdout:(10/11): python3-websocket-client-0.56.0-5.el8. 814 kB/s | 61 kB 00:00 2024-05-15T22:52:31.644 INFO:teuthology.orchestra.run.smithi088.stdout:(11/11): python3-kubernetes-11.0.0-6.el8.noarch 4.1 MB/s | 1.2 MB 00:00 2024-05-15T22:52:31.646 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:52:31.646 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.0 MB/s | 1.9 MB 00:01 2024-05-15T22:52:31.686 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T22:52:31.701 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T22:52:31.702 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T22:52:31.777 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T22:52:31.780 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T22:52:31.807 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-05-15T22:52:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/11 2024-05-15T22:52:32.297 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T22:52:32.558 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/11 2024-05-15T22:52:32.596 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/11 2024-05-15T22:52:32.739 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/11 2024-05-15T22:52:32.973 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/11 2024-05-15T22:52:33.177 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/11 2024-05-15T22:52:33.418 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/11 2024-05-15T22:52:33.642 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-05-15T22:52:33.651 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/11 2024-05-15T22:52:33.896 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/11 2024-05-15T22:52:33.952 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/11 2024-05-15T22:52:34.137 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/11 2024-05-15T22:52:34.194 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/11 2024-05-15T22:52:34.369 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-05-15T22:52:34.653 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/11 2024-05-15T22:52:34.909 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/11 2024-05-15T22:52:35.317 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/11 2024-05-15T22:52:35.580 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/11 2024-05-15T22:52:35.718 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-05-15T22:52:35.923 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/11 2024-05-15T22:52:36.011 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-05-15T22:52:36.011 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-rook-2:14.2.22-0.el8.noarch 1/11 2024-05-15T22:52:36.012 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 2/11 2024-05-15T22:52:36.012 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/11 2024-05-15T22:52:36.012 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 4/11 2024-05-15T22:52:36.012 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/11 2024-05-15T22:52:36.012 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-05-15T22:52:36.012 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 7/11 2024-05-15T22:52:36.012 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 8/11 2024-05-15T22:52:36.012 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 9/11 2024-05-15T22:52:36.012 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 10/11 2024-05-15T22:52:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/11 2024-05-15T22:52:36.430 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-05-15T22:52:36.451 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 11/11 2024-05-15T22:52:36.451 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:36.451 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T22:52:36.451 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook-2:14.2.22-0.el8.noarch 2024-05-15T22:52:36.451 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-05-15T22:52:36.451 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-05-15T22:52:36.451 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-05-15T22:52:36.451 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-05-15T22:52:36.451 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-05-15T22:52:36.452 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-05-15T22:52:36.452 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-05-15T22:52:36.452 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-05-15T22:52:36.452 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa-4.9-2.el8.noarch 2024-05-15T22:52:36.452 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-05-15T22:52:36.452 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:36.452 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:52:36.580 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-fuse 2024-05-15T22:52:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-05-15T22:52:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:14.2.22-0.el8.noarch 1/11 2024-05-15T22:52:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 2/11 2024-05-15T22:52:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/11 2024-05-15T22:52:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 4/11 2024-05-15T22:52:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/11 2024-05-15T22:52:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-05-15T22:52:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 7/11 2024-05-15T22:52:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 8/11 2024-05-15T22:52:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 9/11 2024-05-15T22:52:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 10/11 2024-05-15T22:52:37.158 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 11/11 2024-05-15T22:52:37.158 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:37.158 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-05-15T22:52:37.158 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:14.2.22-0.el8.noarch 2024-05-15T22:52:37.158 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-05-15T22:52:37.158 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-05-15T22:52:37.159 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-05-15T22:52:37.159 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-05-15T22:52:37.159 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-05-15T22:52:37.159 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-05-15T22:52:37.159 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-05-15T22:52:37.159 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-05-15T22:52:37.159 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.9-2.el8.noarch 2024-05-15T22:52:37.159 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-05-15T22:52:37.159 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:37.159 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:52:37.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-fuse 2024-05-15T22:52:37.333 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:40 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:52:38.032 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:39 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:52:38.965 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:52:38.966 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:38.966 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-05-15T22:52:38.966 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:38.966 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T22:52:38.966 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse x86_64 2:14.2.22-0.el8 ceph 590 k 2024-05-15T22:52:38.966 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T22:52:38.966 INFO:teuthology.orchestra.run.smithi088.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-05-15T22:52:38.966 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:38.966 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T22:52:38.967 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:38.967 INFO:teuthology.orchestra.run.smithi088.stdout:Install 2 Packages 2024-05-15T22:52:38.967 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:38.967 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 673 k 2024-05-15T22:52:38.967 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 1.9 M 2024-05-15T22:52:38.967 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T22:52:39.102 INFO:teuthology.orchestra.run.smithi088.stdout:(1/2): ceph-fuse-14.2.22-0.el8.x86_64.rpm 7.5 MB/s | 590 kB 00:00 2024-05-15T22:52:39.160 INFO:teuthology.orchestra.run.smithi088.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 615 kB/s | 83 kB 00:00 2024-05-15T22:52:39.161 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:52:39.161 INFO:teuthology.orchestra.run.smithi088.stdout:Total 3.4 MB/s | 673 kB 00:00 2024-05-15T22:52:39.171 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T22:52:39.192 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T22:52:39.193 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T22:52:39.277 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T22:52:39.280 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T22:52:39.463 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T22:52:39.684 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:52:39.684 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:39.685 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-05-15T22:52:39.685 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:39.685 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-05-15T22:52:39.685 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:14.2.22-0.el8 ceph 590 k 2024-05-15T22:52:39.685 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-05-15T22:52:39.685 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-05-15T22:52:39.685 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:39.685 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-05-15T22:52:39.685 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:39.685 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2024-05-15T22:52:39.685 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:39.686 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 673 k 2024-05-15T22:52:39.686 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 1.9 M 2024-05-15T22:52:39.686 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-05-15T22:52:39.732 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-05-15T22:52:39.796 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): ceph-fuse-14.2.22-0.el8.x86_64.rpm 11 MB/s | 590 kB 00:00 2024-05-15T22:52:39.896 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 544 kB/s | 83 kB 00:00 2024-05-15T22:52:39.896 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:52:39.897 INFO:teuthology.orchestra.run.smithi090.stdout:Total 3.1 MB/s | 673 kB 00:00 2024-05-15T22:52:39.906 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-05-15T22:52:39.928 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-05-15T22:52:39.928 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-05-15T22:52:39.959 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-05-15T22:52:40.012 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-05-15T22:52:40.015 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-05-15T22:52:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-05-15T22:52:40.350 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-05-15T22:52:40.350 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-2:14.2.22-0.el8.x86_64 1/2 2024-05-15T22:52:40.430 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-05-15T22:52:40.647 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-05-15T22:52:40.751 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-05-15T22:52:40.751 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:40.751 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T22:52:40.752 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-2:14.2.22-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-05-15T22:52:40.752 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:40.752 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:52:40.991 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados-devel 2024-05-15T22:52:41.044 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-05-15T22:52:41.044 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:14.2.22-0.el8.x86_64 1/2 2024-05-15T22:52:41.452 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-05-15T22:52:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:41.453 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-05-15T22:52:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:14.2.22-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-05-15T22:52:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:41.453 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:52:41.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados-devel 2024-05-15T22:52:41.750 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:44 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:52:42.402 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:43 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:52:43.407 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:52:43.408 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:43.408 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-05-15T22:52:43.408 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:43.408 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T22:52:43.408 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel x86_64 2:14.2.22-0.el8 ceph 90 k 2024-05-15T22:52:43.408 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:43.408 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T22:52:43.408 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:43.409 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T22:52:43.409 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:43.409 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 90 k 2024-05-15T22:52:43.409 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 310 k 2024-05-15T22:52:43.409 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T22:52:43.489 INFO:teuthology.orchestra.run.smithi088.stdout:librados-devel-14.2.22-0.el8.x86_64.rpm 1.1 MB/s | 90 kB 00:00 2024-05-15T22:52:43.489 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:52:43.490 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.1 MB/s | 90 kB 00:00 2024-05-15T22:52:43.490 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T22:52:43.500 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T22:52:43.500 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T22:52:43.550 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T22:52:43.553 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T22:52:43.662 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T22:52:43.874 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:52:44.020 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:52:44.020 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:44.020 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-05-15T22:52:44.021 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:44.021 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-05-15T22:52:44.021 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:14.2.22-0.el8 ceph 90 k 2024-05-15T22:52:44.021 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:44.021 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-05-15T22:52:44.021 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:44.021 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-05-15T22:52:44.021 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:44.022 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 90 k 2024-05-15T22:52:44.022 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 310 k 2024-05-15T22:52:44.022 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-05-15T22:52:44.151 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:52:44.362 INFO:teuthology.orchestra.run.smithi090.stdout:librados-devel-14.2.22-0.el8.x86_64.rpm 265 kB/s | 90 kB 00:00 2024-05-15T22:52:44.363 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:52:44.363 INFO:teuthology.orchestra.run.smithi090.stdout:Total 264 kB/s | 90 kB 00:00 2024-05-15T22:52:44.363 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-05-15T22:52:44.374 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-05-15T22:52:44.374 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-05-15T22:52:44.426 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-05-15T22:52:44.429 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-05-15T22:52:44.571 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:52:44.571 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:44.571 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T22:52:44.572 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-2:14.2.22-0.el8.x86_64 2024-05-15T22:52:44.572 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:44.572 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:52:44.797 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-05-15T22:52:44.823 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs2 2024-05-15T22:52:45.003 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:52:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:52:45.591 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:48 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:52:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:52:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:45.698 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-05-15T22:52:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:14.2.22-0.el8.x86_64 2024-05-15T22:52:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:45.698 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:52:45.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs2 2024-05-15T22:52:46.674 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:47 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:52:47.198 INFO:teuthology.orchestra.run.smithi088.stdout:Package libcephfs2-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:52:47.247 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:52:47.248 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T22:52:47.248 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:52:47.350 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs-devel 2024-05-15T22:52:48.097 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:51 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:52:48.266 INFO:teuthology.orchestra.run.smithi090.stdout:Package libcephfs2-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:52:48.314 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:52:48.315 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-05-15T22:52:48.315 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:52:48.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs-devel 2024-05-15T22:52:49.189 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:50 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:52:49.754 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:52:49.755 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:49.755 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-05-15T22:52:49.755 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:49.755 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T22:52:49.755 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel x86_64 2:14.2.22-0.el8 ceph 22 k 2024-05-15T22:52:49.755 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:49.755 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T22:52:49.755 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:49.755 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T22:52:49.755 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:49.755 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 22 k 2024-05-15T22:52:49.756 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 77 k 2024-05-15T22:52:49.756 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T22:52:49.827 INFO:teuthology.orchestra.run.smithi088.stdout:libcephfs-devel-14.2.22-0.el8.x86_64.rpm 294 kB/s | 22 kB 00:00 2024-05-15T22:52:49.828 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:52:49.828 INFO:teuthology.orchestra.run.smithi088.stdout:Total 290 kB/s | 22 kB 00:00 2024-05-15T22:52:49.828 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T22:52:49.831 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T22:52:49.831 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T22:52:49.834 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T22:52:49.836 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T22:52:49.881 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T22:52:50.058 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:52:50.323 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:52:50.654 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:52:50.654 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:50.654 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T22:52:50.654 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel-2:14.2.22-0.el8.x86_64 2024-05-15T22:52:50.654 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:50.654 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:52:50.780 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados2 2024-05-15T22:52:50.862 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:52:50.862 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:50.862 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-05-15T22:52:50.863 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:50.863 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-05-15T22:52:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:14.2.22-0.el8 ceph 22 k 2024-05-15T22:52:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:50.863 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-05-15T22:52:50.863 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:50.863 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-05-15T22:52:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:50.863 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 22 k 2024-05-15T22:52:50.864 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 77 k 2024-05-15T22:52:50.864 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-05-15T22:52:50.912 INFO:teuthology.orchestra.run.smithi090.stdout:libcephfs-devel-14.2.22-0.el8.x86_64.rpm 442 kB/s | 22 kB 00:00 2024-05-15T22:52:50.912 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:52:50.912 INFO:teuthology.orchestra.run.smithi090.stdout:Total 432 kB/s | 22 kB 00:00 2024-05-15T22:52:50.913 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-05-15T22:52:50.916 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-05-15T22:52:50.916 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-05-15T22:52:50.918 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-05-15T22:52:50.921 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-05-15T22:52:50.973 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-05-15T22:52:51.162 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:52:51.413 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:52:51.529 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:54 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:52:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:52:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:51.763 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-05-15T22:52:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:14.2.22-0.el8.x86_64 2024-05-15T22:52:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:51.764 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:52:51.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados2 2024-05-15T22:52:52.625 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:53 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:52:53.134 INFO:teuthology.orchestra.run.smithi088.stdout:Package librados2-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:52:53.182 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:52:53.184 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T22:52:53.184 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:52:53.293 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librbd1 2024-05-15T22:52:54.035 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:57 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:52:54.197 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:52:54.246 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:52:54.247 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-05-15T22:52:54.247 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:52:54.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librbd1 2024-05-15T22:52:55.127 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:56 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:52:55.616 INFO:teuthology.orchestra.run.smithi088.stdout:Package librbd1-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:52:55.664 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:52:55.665 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T22:52:55.665 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:52:55.778 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-fuse 2024-05-15T22:52:56.523 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:59 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:52:56.731 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:52:56.779 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:52:56.780 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-05-15T22:52:56.781 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:52:57.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-fuse 2024-05-15T22:52:57.906 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:58 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:52:58.164 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:52:58.165 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:58.165 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-05-15T22:52:58.165 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:58.165 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T22:52:58.165 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse x86_64 2:14.2.22-0.el8 ceph 73 k 2024-05-15T22:52:58.165 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:58.165 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T22:52:58.165 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:52:58.166 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T22:52:58.166 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:58.166 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 73 k 2024-05-15T22:52:58.166 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 188 k 2024-05-15T22:52:58.166 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T22:52:58.272 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-fuse-14.2.22-0.el8.x86_64.rpm 693 kB/s | 73 kB 00:00 2024-05-15T22:52:58.272 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:52:58.272 INFO:teuthology.orchestra.run.smithi088.stdout:Total 685 kB/s | 73 kB 00:00 2024-05-15T22:52:58.273 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T22:52:58.283 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T22:52:58.283 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T22:52:58.331 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T22:52:58.334 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T22:52:58.438 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T22:52:58.670 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:52:58.976 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:52:59.361 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:52:59.361 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:59.361 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T22:52:59.361 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-2:14.2.22-0.el8.x86_64 2024-05-15T22:52:59.361 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:52:59.362 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:52:59.524 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:52:59.525 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:59.525 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-05-15T22:52:59.525 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:59.525 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-05-15T22:52:59.525 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:14.2.22-0.el8 ceph 73 k 2024-05-15T22:52:59.525 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:59.526 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-05-15T22:52:59.526 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:52:59.526 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-05-15T22:52:59.526 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:52:59.526 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 73 k 2024-05-15T22:52:59.526 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 188 k 2024-05-15T22:52:59.526 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-05-15T22:52:59.570 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-fuse-14.2.22-0.el8.x86_64.rpm 1.6 MB/s | 73 kB 00:00 2024-05-15T22:52:59.571 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:52:59.571 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.6 MB/s | 73 kB 00:00 2024-05-15T22:52:59.571 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-05-15T22:52:59.582 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-05-15T22:52:59.582 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-05-15T22:52:59.597 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-mirror 2024-05-15T22:52:59.630 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-05-15T22:52:59.633 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-05-15T22:52:59.733 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-05-15T22:52:59.953 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:53:00.241 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:53:00.355 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:03 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:53:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:53:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:53:00.628 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-05-15T22:53:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:53:00.628 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:53:00.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-mirror 2024-05-15T22:53:01.619 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:02 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:53:02.006 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:53:02.006 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:53:02.006 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-05-15T22:53:02.007 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:53:02.007 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T22:53:02.007 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror x86_64 2:14.2.22-0.el8 ceph 2.3 M 2024-05-15T22:53:02.007 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:53:02.007 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T22:53:02.007 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:53:02.007 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T22:53:02.007 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:53:02.007 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 2.3 M 2024-05-15T22:53:02.007 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 8.4 M 2024-05-15T22:53:02.007 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T22:53:02.101 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-mirror-14.2.22-0.el8.x86_64.rpm 25 MB/s | 2.3 MB 00:00 2024-05-15T22:53:02.101 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:53:02.102 INFO:teuthology.orchestra.run.smithi088.stdout:Total 24 MB/s | 2.3 MB 00:00 2024-05-15T22:53:02.102 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T22:53:02.114 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T22:53:02.114 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T22:53:02.171 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T22:53:02.174 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T22:53:02.489 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T22:53:02.721 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:53:03.138 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:53:03.268 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:53:03.269 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:53:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-05-15T22:53:03.270 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:53:03.270 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-05-15T22:53:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:14.2.22-0.el8 ceph 2.3 M 2024-05-15T22:53:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:53:03.270 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-05-15T22:53:03.270 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:53:03.270 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-05-15T22:53:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:53:03.271 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.3 M 2024-05-15T22:53:03.271 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 8.4 M 2024-05-15T22:53:03.271 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-05-15T22:53:03.361 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-mirror-14.2.22-0.el8.x86_64.rpm 26 MB/s | 2.3 MB 00:00 2024-05-15T22:53:03.362 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:53:03.362 INFO:teuthology.orchestra.run.smithi090.stdout:Total 25 MB/s | 2.3 MB 00:00 2024-05-15T22:53:03.362 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-05-15T22:53:03.374 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-05-15T22:53:03.374 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-05-15T22:53:03.432 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-05-15T22:53:03.435 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-05-15T22:53:03.735 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-05-15T22:53:03.802 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:53:03.802 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:53:03.802 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T22:53:03.802 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:03.802 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:53:03.802 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:53:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:53:04.000 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-nbd 2024-05-15T22:53:04.211 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 105 kB/s | 14 MB 02:12 2024-05-15T22:53:04.374 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:53:04.767 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:07 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:53:04.805 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:53:04.805 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:53:04.805 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-05-15T22:53:04.805 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:04.805 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:53:04.805 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:53:04.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-nbd 2024-05-15T22:53:05.756 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:06 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:53:06.406 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:53:06.406 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:53:06.406 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-05-15T22:53:06.407 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:53:06.407 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T22:53:06.407 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd x86_64 2:14.2.22-0.el8 ceph 142 k 2024-05-15T22:53:06.407 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:53:06.407 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T22:53:06.407 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T22:53:06.407 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T22:53:06.407 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:53:06.408 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 142 k 2024-05-15T22:53:06.408 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 422 k 2024-05-15T22:53:06.408 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T22:53:06.463 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-nbd-14.2.22-0.el8.x86_64.rpm 2.5 MB/s | 142 kB 00:00 2024-05-15T22:53:06.464 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:53:06.464 INFO:teuthology.orchestra.run.smithi088.stdout:Total 2.4 MB/s | 142 kB 00:00 2024-05-15T22:53:06.464 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T22:53:06.475 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T22:53:06.475 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T22:53:06.535 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T22:53:06.538 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T22:53:06.651 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T22:53:06.844 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:53:07.134 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:53:07.398 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:53:07.399 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:53:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-05-15T22:53:07.399 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:53:07.400 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-05-15T22:53:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:14.2.22-0.el8 ceph 142 k 2024-05-15T22:53:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:53:07.400 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-05-15T22:53:07.400 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-05-15T22:53:07.400 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-05-15T22:53:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:53:07.400 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 142 k 2024-05-15T22:53:07.400 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 422 k 2024-05-15T22:53:07.400 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-05-15T22:53:07.458 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-nbd-14.2.22-0.el8.x86_64.rpm 2.4 MB/s | 142 kB 00:00 2024-05-15T22:53:07.459 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:53:07.459 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.3 MB/s | 142 kB 00:00 2024-05-15T22:53:07.459 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-05-15T22:53:07.470 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-05-15T22:53:07.470 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-05-15T22:53:07.496 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:53:07.497 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:53:07.497 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T22:53:07.497 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:07.497 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T22:53:07.497 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:53:07.530 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-05-15T22:53:07.532 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-05-15T22:53:07.716 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python-rados 2024-05-15T22:53:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-05-15T22:53:08.086 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:53:08.374 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:53:08.485 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 413 kB/s | 24 kB 00:00 2024-05-15T22:53:08.488 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:11 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:53:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:53:08.761 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:53:08.761 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-05-15T22:53:08.762 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:08.762 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:53:08.762 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:53:08.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python-rados 2024-05-15T22:53:09.753 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:10 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:53:10.096 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rados-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:53:10.151 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:53:10.151 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T22:53:10.151 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:53:10.254 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python-rgw 2024-05-15T22:53:11.009 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:14 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:53:11.353 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rados-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:53:11.402 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:53:11.403 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-05-15T22:53:11.403 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:53:11.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python-rgw 2024-05-15T22:53:12.193 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:53:12.195 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:53:12.195 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-05-15T22:53:12.195 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:53:12.196 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-05-15T22:53:12.196 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw x86_64 2:14.2.22-0.el8 ceph 6.0 M 2024-05-15T22:53:12.196 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrading: 2024-05-15T22:53:12.196 INFO:teuthology.orchestra.run.smithi084.stdout: librados2 x86_64 2:14.2.22-0.el8 ceph 3.5 M 2024-05-15T22:53:12.196 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1 x86_64 2:14.2.22-0.el8 ceph 1.7 M 2024-05-15T22:53:12.196 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-05-15T22:53:12.196 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base x86_64 2:14.2.22-0.el8 ceph 5.4 M 2024-05-15T22:53:12.196 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common x86_64 2:14.2.22-0.el8 ceph 19 M 2024-05-15T22:53:12.196 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux x86_64 2:14.2.22-0.el8 ceph 25 k 2024-05-15T22:53:12.196 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-05-15T22:53:12.196 INFO:teuthology.orchestra.run.smithi084.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-05-15T22:53:12.196 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2 x86_64 2:14.2.22-0.el8 ceph 520 k 2024-05-15T22:53:12.196 INFO:teuthology.orchestra.run.smithi084.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-05-15T22:53:12.197 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-05-15T22:53:12.197 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1 x86_64 2:14.2.22-0.el8 ceph 376 k 2024-05-15T22:53:12.197 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-05-15T22:53:12.197 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2 x86_64 2:14.2.22-0.el8 ceph 5.3 M 2024-05-15T22:53:12.197 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-05-15T22:53:12.197 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-05-15T22:53:12.197 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse x86_64 2:14.2.22-0.el8 ceph 39 k 2024-05-15T22:53:12.197 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs x86_64 2:14.2.22-0.el8 ceph 159 k 2024-05-15T22:53:12.197 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-05-15T22:53:12.197 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados x86_64 2:14.2.22-0.el8 ceph 323 k 2024-05-15T22:53:12.197 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd x86_64 2:14.2.22-0.el8 ceph 291 k 2024-05-15T22:53:12.197 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw x86_64 2:14.2.22-0.el8 ceph 113 k 2024-05-15T22:53:12.197 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:53:12.197 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-05-15T22:53:12.197 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:53:12.198 INFO:teuthology.orchestra.run.smithi084.stdout:Install 20 Packages 2024-05-15T22:53:12.198 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrade 2 Packages 2024-05-15T22:53:12.198 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:53:12.198 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 45 M 2024-05-15T22:53:12.198 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-05-15T22:53:12.268 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:13 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:53:12.608 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rgw-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:53:12.645 INFO:teuthology.orchestra.run.smithi084.stdout:(1/22): ceph-base-14.2.22-0.el8.x86_64.rpm 38 MB/s | 5.4 MB 00:00 2024-05-15T22:53:12.658 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:53:12.659 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T22:53:12.659 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:53:12.669 INFO:teuthology.orchestra.run.smithi084.stdout:(2/22): ceph-selinux-14.2.22-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-05-15T22:53:12.703 INFO:teuthology.orchestra.run.smithi084.stdout:(3/22): libcephfs2-14.2.22-0.el8.x86_64.rpm 15 MB/s | 520 kB 00:00 2024-05-15T22:53:12.737 INFO:teuthology.orchestra.run.smithi084.stdout:(4/22): libradosstriper1-14.2.22-0.el8.x86_64.r 11 MB/s | 376 kB 00:00 2024-05-15T22:53:12.766 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python-rbd 2024-05-15T22:53:12.846 INFO:teuthology.orchestra.run.smithi084.stdout:(5/22): ceph-radosgw-14.2.22-0.el8.x86_64.rpm 18 MB/s | 6.0 MB 00:00 2024-05-15T22:53:12.871 INFO:teuthology.orchestra.run.smithi084.stdout:(6/22): python3-ceph-argparse-14.2.22-0.el8.x86 1.5 MB/s | 39 kB 00:00 2024-05-15T22:53:12.929 INFO:teuthology.orchestra.run.smithi084.stdout:(7/22): python3-cephfs-14.2.22-0.el8.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-05-15T22:53:12.955 INFO:teuthology.orchestra.run.smithi084.stdout:(8/22): python3-rados-14.2.22-0.el8.x86_64.rpm 13 MB/s | 323 kB 00:00 2024-05-15T22:53:13.021 INFO:teuthology.orchestra.run.smithi084.stdout:(9/22): librgw2-14.2.22-0.el8.x86_64.rpm 19 MB/s | 5.3 MB 00:00 2024-05-15T22:53:13.047 INFO:teuthology.orchestra.run.smithi084.stdout:(10/22): python3-rgw-14.2.22-0.el8.x86_64.rpm 4.4 MB/s | 113 kB 00:00 2024-05-15T22:53:13.081 INFO:teuthology.orchestra.run.smithi084.stdout:(11/22): python3-rbd-14.2.22-0.el8.x86_64.rpm 2.3 MB/s | 291 kB 00:00 2024-05-15T22:53:13.298 INFO:teuthology.orchestra.run.smithi084.stdout:(12/22): ceph-common-14.2.22-0.el8.x86_64.rpm 25 MB/s | 19 MB 00:00 2024-05-15T22:53:13.517 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:16 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:53:13.623 INFO:teuthology.orchestra.run.smithi084.stdout:(13/22): python3-prettytable-0.7.2-14.el8.noarc 135 kB/s | 44 kB 00:00 2024-05-15T22:53:13.690 INFO:teuthology.orchestra.run.smithi084.stdout:(14/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 426 kB/s | 259 kB 00:00 2024-05-15T22:53:13.790 INFO:teuthology.orchestra.run.smithi084.stdout:(15/22): librdkafka-1.6.1-1.el8.x86_64.rpm 911 kB/s | 677 kB 00:00 2024-05-15T22:53:13.815 INFO:teuthology.orchestra.run.smithi084.stdout:(16/22): librabbitmq-0.9.0-5.el8.x86_64.rpm 245 kB/s | 47 kB 00:00 2024-05-15T22:53:13.863 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rgw-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:53:13.899 INFO:teuthology.orchestra.run.smithi084.stdout:(17/22): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2024-05-15T22:53:13.912 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:53:13.913 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-05-15T22:53:13.913 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:53:13.966 INFO:teuthology.orchestra.run.smithi084.stdout:(18/22): leveldb-1.22-1.el8.x86_64.rpm 1.0 MB/s | 181 kB 00:00 2024-05-15T22:53:13.991 INFO:teuthology.orchestra.run.smithi084.stdout:(19/22): libunwind-1.3.1-3.el8.x86_64.rpm 818 kB/s | 75 kB 00:00 2024-05-15T22:53:14.016 INFO:teuthology.orchestra.run.smithi084.stdout:(20/22): liboath-2.6.2-3.el8.x86_64.rpm 300 kB/s | 59 kB 00:00 2024-05-15T22:53:14.042 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python-rbd 2024-05-15T22:53:14.067 INFO:teuthology.orchestra.run.smithi084.stdout:(21/22): librbd1-14.2.22-0.el8.x86_64.rpm 23 MB/s | 1.7 MB 00:00 2024-05-15T22:53:14.117 INFO:teuthology.orchestra.run.smithi084.stdout:(22/22): librados2-14.2.22-0.el8.x86_64.rpm 23 MB/s | 3.5 MB 00:00 2024-05-15T22:53:14.121 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:53:14.121 INFO:teuthology.orchestra.run.smithi084.stdout:Total 23 MB/s | 45 MB 00:01 2024-05-15T22:53:14.164 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-05-15T22:53:14.212 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-05-15T22:53:14.212 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-05-15T22:53:14.496 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-05-15T22:53:14.499 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-05-15T22:53:14.795 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:15 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:53:15.119 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rbd-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:53:15.169 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:53:15.170 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T22:53:15.170 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:53:15.279 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python-cephfs 2024-05-15T22:53:15.414 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-05-15T22:53:15.622 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2024-05-15T22:53:15.814 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-05-15T22:53:16.031 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:19 ago on Wed 15 May 2024 10:50:57 PM UTC. 2024-05-15T22:53:16.179 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-05-15T22:53:16.372 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librados2-2:14.2.22-0.el8.x86_64 3/24 2024-05-15T22:53:16.401 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rbd-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:53:16.443 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:14.2.22-0.el8.x86_64 3/24 2024-05-15T22:53:16.449 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:53:16.451 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-05-15T22:53:16.451 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:53:16.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python-cephfs 2024-05-15T22:53:16.647 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rados-2:14.2.22-0.el8.x86_64 4/24 2024-05-15T22:53:16.822 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-05-15T22:53:17.029 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-05-15T22:53:17.221 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-05-15T22:53:17.276 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-05-15T22:53:17.321 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:18 ago on Wed 15 May 2024 10:50:59 PM UTC. 2024-05-15T22:53:17.622 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-cephfs-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:53:17.672 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T22:53:17.673 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T22:53:17.673 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T22:53:17.787 DEBUG:teuthology.parallel:result is None 2024-05-15T22:53:17.870 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/24 2024-05-15T22:53:18.046 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-05-15T22:53:18.110 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-05-15T22:53:18.284 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : leveldb-1.22-1.el8.x86_64 9/24 2024-05-15T22:53:18.430 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 10/24 2024-05-15T22:53:18.599 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cephfs-2:14.2.22-0.el8.x86_64 11/24 2024-05-15T22:53:18.813 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-05-15T22:53:18.918 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-cephfs-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:53:18.966 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-05-15T22:53:18.968 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-05-15T22:53:18.968 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-05-15T22:53:19.002 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rbd-2:14.2.22-0.el8.x86_64 13/24 2024-05-15T22:53:19.133 DEBUG:teuthology.parallel:result is None 2024-05-15T22:53:19.171 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-05-15T22:53:19.231 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-05-15T22:53:19.439 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2024-05-15T22:53:19.612 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2024-05-15T22:53:19.821 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2024-05-15T22:53:19.988 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 18/24 2024-05-15T22:53:22.081 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-05-15T22:53:22.321 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-05-15T22:53:22.952 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-05-15T22:53:23.258 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-05-15T22:53:23.364 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-05-15T22:53:23.504 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-05-15T22:53:42.579 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-05-15T22:53:42.579 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /sys 2024-05-15T22:53:42.579 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /proc 2024-05-15T22:53:42.579 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /dev 2024-05-15T22:53:42.579 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /run 2024-05-15T22:53:42.579 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /mnt 2024-05-15T22:53:42.579 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /var/tmp 2024-05-15T22:53:42.579 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /home 2024-05-15T22:53:42.580 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /tmp 2024-05-15T22:53:42.580 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /dev 2024-05-15T22:53:42.580 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:53:43.286 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-05-15T22:53:43.306 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-05-15T22:53:43.306 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-05-15T22:53:43.492 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-05-15T22:53:43.492 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2024-05-15T22:53:44.252 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2024-05-15T22:53:44.252 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-base-2:14.2.22-0.el8.x86_64 1/24 2024-05-15T22:53:44.252 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-common-2:14.2.22-0.el8.x86_64 2/24 2024-05-15T22:53:44.252 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-radosgw-2:14.2.22-0.el8.x86_64 3/24 2024-05-15T22:53:44.253 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-selinux-2:14.2.22-0.el8.x86_64 4/24 2024-05-15T22:53:44.253 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-05-15T22:53:44.253 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libradosstriper1-2:14.2.22-0.el8.x86_64 6/24 2024-05-15T22:53:44.253 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librgw2-2:14.2.22-0.el8.x86_64 7/24 2024-05-15T22:53:44.253 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 8/24 2024-05-15T22:53:44.253 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cephfs-2:14.2.22-0.el8.x86_64 9/24 2024-05-15T22:53:44.253 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rados-2:14.2.22-0.el8.x86_64 10/24 2024-05-15T22:53:44.253 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rbd-2:14.2.22-0.el8.x86_64 11/24 2024-05-15T22:53:44.253 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-05-15T22:53:44.253 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 13/24 2024-05-15T22:53:44.253 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 14/24 2024-05-15T22:53:44.253 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 15/24 2024-05-15T22:53:44.253 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 16/24 2024-05-15T22:53:44.253 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 17/24 2024-05-15T22:53:44.254 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : leveldb-1.22-1.el8.x86_64 18/24 2024-05-15T22:53:44.254 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2024-05-15T22:53:44.254 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2024-05-15T22:53:44.254 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:14.2.22-0.el8.x86_64 21/24 2024-05-15T22:53:44.254 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2024-05-15T22:53:44.254 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:14.2.22-0.el8.x86_64 23/24 2024-05-15T22:53:45.160 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2024-05-15T22:53:45.160 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:53:45.160 INFO:teuthology.orchestra.run.smithi084.stdout:Upgraded: 2024-05-15T22:53:45.160 INFO:teuthology.orchestra.run.smithi084.stdout: librados2-2:14.2.22-0.el8.x86_64 librbd1-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:45.160 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-05-15T22:53:45.160 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:45.160 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:45.160 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:45.160 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:45.160 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-05-15T22:53:45.160 INFO:teuthology.orchestra.run.smithi084.stdout: leveldb-1.22-1.el8.x86_64 2024-05-15T22:53:45.161 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:45.161 INFO:teuthology.orchestra.run.smithi084.stdout: liboath-2.6.2-3.el8.x86_64 2024-05-15T22:53:45.161 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-05-15T22:53:45.161 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:45.161 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-05-15T22:53:45.161 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:45.161 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind-1.3.1-3.el8.x86_64 2024-05-15T22:53:45.161 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-05-15T22:53:45.161 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:45.161 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:45.161 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-05-15T22:53:45.161 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:45.161 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:45.161 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw-2:14.2.22-0.el8.x86_64 2024-05-15T22:53:45.161 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:53:45.162 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:53:45.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-test 2024-05-15T22:53:46.181 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:38 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:53:47.830 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:53:47.831 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:53:47.831 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-05-15T22:53:47.831 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:53:47.831 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-05-15T22:53:47.831 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test x86_64 2:14.2.22-0.el8 ceph 35 M 2024-05-15T22:53:47.831 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-05-15T22:53:47.832 INFO:teuthology.orchestra.run.smithi084.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-05-15T22:53:47.832 INFO:teuthology.orchestra.run.smithi084.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-05-15T22:53:47.832 INFO:teuthology.orchestra.run.smithi084.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-05-15T22:53:47.832 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-05-15T22:53:47.832 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:53:47.832 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-05-15T22:53:47.832 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:53:47.832 INFO:teuthology.orchestra.run.smithi084.stdout:Install 5 Packages 2024-05-15T22:53:47.832 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:53:47.833 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 36 M 2024-05-15T22:53:47.833 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 127 M 2024-05-15T22:53:47.833 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-05-15T22:53:48.363 INFO:teuthology.orchestra.run.smithi084.stdout:(1/5): ceph-test-14.2.22-0.el8.x86_64.rpm 74 MB/s | 35 MB 00:00 2024-05-15T22:53:48.746 INFO:teuthology.orchestra.run.smithi084.stdout:(2/5): jq-1.6-8.el8.x86_64.rpm 238 kB/s | 203 kB 00:00 2024-05-15T22:53:48.771 INFO:teuthology.orchestra.run.smithi084.stdout:(3/5): oniguruma-6.8.2-3.el8.x86_64.rpm 214 kB/s | 188 kB 00:00 2024-05-15T22:53:48.847 INFO:teuthology.orchestra.run.smithi084.stdout:(4/5): socat-1.7.4.1-1.el8.x86_64.rpm 668 kB/s | 323 kB 00:00 2024-05-15T22:53:48.872 INFO:teuthology.orchestra.run.smithi084.stdout:(5/5): xmlstarlet-1.6.1-20.el8.x86_64.rpm 564 kB/s | 71 kB 00:00 2024-05-15T22:53:48.873 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:53:48.873 INFO:teuthology.orchestra.run.smithi084.stdout:Total 34 MB/s | 36 MB 00:01 2024-05-15T22:53:48.916 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-05-15T22:53:48.936 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-05-15T22:53:48.936 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-05-15T22:53:49.156 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-05-15T22:53:49.159 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-05-15T22:53:49.553 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-05-15T22:53:49.760 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/5 2024-05-15T22:53:50.000 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2024-05-15T22:53:50.147 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-05-15T22:53:50.226 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/5 2024-05-15T22:53:53.446 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : jq-1.6-8.el8.x86_64 4/5 2024-05-15T22:53:53.670 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-05-15T22:53:53.956 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-05-15T22:53:53.956 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-test-2:14.2.22-0.el8.x86_64 1/5 2024-05-15T22:53:53.957 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : jq-1.6-8.el8.x86_64 2/5 2024-05-15T22:53:53.957 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-05-15T22:53:53.957 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2024-05-15T22:53:54.367 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 5/5 2024-05-15T22:53:54.368 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:53:54.368 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-05-15T22:53:54.368 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test-2:14.2.22-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-05-15T22:53:54.368 INFO:teuthology.orchestra.run.smithi084.stdout: oniguruma-6.8.2-3.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2024-05-15T22:53:54.368 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-05-15T22:53:54.368 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:53:54.368 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:53:54.616 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph 2024-05-15T22:53:55.340 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:47 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:53:56.973 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:53:56.976 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:53:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-05-15T22:53:56.976 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:53:56.976 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-05-15T22:53:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: ceph x86_64 2:14.2.22-0.el8 ceph 6.3 k 2024-05-15T22:53:56.976 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-05-15T22:53:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds x86_64 2:14.2.22-0.el8 ceph 2.0 M 2024-05-15T22:53:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr x86_64 2:14.2.22-0.el8 ceph 1.6 M 2024-05-15T22:53:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon x86_64 2:14.2.22-0.el8 ceph 4.5 M 2024-05-15T22:53:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd x86_64 2:14.2.22-0.el8 ceph 17 M 2024-05-15T22:53:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-05-15T22:53:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-05-15T22:53:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-05-15T22:53:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-05-15T22:53:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-05-15T22:53:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-05-15T22:53:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-05-15T22:53:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-05-15T22:53:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-05-15T22:53:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-05-15T22:53:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-05-15T22:53:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-05-15T22:53:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-05-15T22:53:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-05-15T22:53:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-05-15T22:53:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-05-15T22:53:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-05-15T22:53:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-05-15T22:53:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-05-15T22:53:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-05-15T22:53:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-05-15T22:53:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-05-15T22:53:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-05-15T22:53:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-05-15T22:53:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-05-15T22:53:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-05-15T22:53:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:53:56.979 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-05-15T22:53:56.979 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:53:56.979 INFO:teuthology.orchestra.run.smithi084.stdout:Install 31 Packages 2024-05-15T22:53:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:53:56.979 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 28 M 2024-05-15T22:53:56.980 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 93 M 2024-05-15T22:53:56.980 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-05-15T22:53:57.171 INFO:teuthology.orchestra.run.smithi084.stdout:(1/31): ceph-14.2.22-0.el8.x86_64.rpm 142 kB/s | 6.3 kB 00:00 2024-05-15T22:53:57.213 INFO:teuthology.orchestra.run.smithi084.stdout:(2/31): ceph-mgr-14.2.22-0.el8.x86_64.rpm 18 MB/s | 1.6 MB 00:00 2024-05-15T22:53:57.255 INFO:teuthology.orchestra.run.smithi084.stdout:(3/31): ceph-mds-14.2.22-0.el8.x86_64.rpm 15 MB/s | 2.0 MB 00:00 2024-05-15T22:53:57.347 INFO:teuthology.orchestra.run.smithi084.stdout:(4/31): ceph-mon-14.2.22-0.el8.x86_64.rpm 25 MB/s | 4.5 MB 00:00 2024-05-15T22:53:57.564 INFO:teuthology.orchestra.run.smithi084.stdout:(5/31): ceph-osd-14.2.22-0.el8.x86_64.rpm 48 MB/s | 17 MB 00:00 2024-05-15T22:53:57.673 INFO:teuthology.orchestra.run.smithi084.stdout:(6/31): python3-mako-1.0.6-14.el8.noarch.rpm 376 kB/s | 157 kB 00:00 2024-05-15T22:53:57.740 INFO:teuthology.orchestra.run.smithi084.stdout:(7/31): python3-markupsafe-0.23-19.el8.x86_64.r 99 kB/s | 39 kB 00:00 2024-05-15T22:53:57.773 INFO:teuthology.orchestra.run.smithi084.stdout:(8/31): python3-pytz-2017.2-11.el8.noarch.rpm 569 kB/s | 57 kB 00:00 2024-05-15T22:53:57.957 INFO:teuthology.orchestra.run.smithi084.stdout:(9/31): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 264 kB/s | 103 kB 00:00 2024-05-15T22:53:57.982 INFO:teuthology.orchestra.run.smithi084.stdout:(10/31): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.2 MB/s | 249 kB 00:00 2024-05-15T22:53:58.041 INFO:teuthology.orchestra.run.smithi084.stdout:(11/31): python3-pycparser-2.14-14.el8.noarch.r 1.8 MB/s | 109 kB 00:00 2024-05-15T22:53:58.083 INFO:teuthology.orchestra.run.smithi084.stdout:(12/31): python3-werkzeug-0.12.2-4.el8.noarch.r 1.3 MB/s | 457 kB 00:00 2024-05-15T22:53:58.508 INFO:teuthology.orchestra.run.smithi084.stdout:(13/31): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 93 kB/s | 44 kB 00:00 2024-05-15T22:53:58.567 INFO:teuthology.orchestra.run.smithi084.stdout:(14/31): python3-cryptography-3.2.1-7.el8.x86_6 1.0 MB/s | 641 kB 00:00 2024-05-15T22:53:58.734 INFO:teuthology.orchestra.run.smithi084.stdout:(15/31): python3-beautifulsoup4-4.6.3-2.el8.1.n 285 kB/s | 185 kB 00:00 2024-05-15T22:53:58.851 INFO:teuthology.orchestra.run.smithi084.stdout:(16/31): python3-cheroot-8.5.2-1.el8.noarch.rpm 504 kB/s | 173 kB 00:00 2024-05-15T22:53:58.876 INFO:teuthology.orchestra.run.smithi084.stdout:(17/31): python3-jaraco-6.2-6.el8.noarch.rpm 78 kB/s | 11 kB 00:00 2024-05-15T22:53:58.960 INFO:teuthology.orchestra.run.smithi084.stdout:(18/31): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2024-05-15T22:53:59.001 INFO:teuthology.orchestra.run.smithi084.stdout:(19/31): python3-logutils-0.3.5-11.el8.noarch.r 389 kB/s | 49 kB 00:00 2024-05-15T22:53:59.085 INFO:teuthology.orchestra.run.smithi084.stdout:(20/31): python3-more-itertools-7.2.0-3.el8.noa 469 kB/s | 59 kB 00:00 2024-05-15T22:53:59.194 INFO:teuthology.orchestra.run.smithi084.stdout:(21/31): python3-cherrypy-18.4.0-1.el8.noarch.r 614 kB/s | 384 kB 00:00 2024-05-15T22:53:59.219 INFO:teuthology.orchestra.run.smithi084.stdout:(22/31): python3-portend-2.6-1.el8.noarch.rpm 117 kB/s | 16 kB 00:00 2024-05-15T22:53:59.244 INFO:teuthology.orchestra.run.smithi084.stdout:(23/31): python3-pecan-1.3.2-9.el8.noarch.rpm 1.1 MB/s | 283 kB 00:00 2024-05-15T22:53:59.303 INFO:teuthology.orchestra.run.smithi084.stdout:(24/31): python3-simplegeneric-0.8.1-17.el8.noa 172 kB/s | 19 kB 00:00 2024-05-15T22:53:59.329 INFO:teuthology.orchestra.run.smithi084.stdout:(25/31): python3-singledispatch-3.4.0.3-18.el8. 220 kB/s | 24 kB 00:00 2024-05-15T22:53:59.353 INFO:teuthology.orchestra.run.smithi084.stdout:(26/31): python3-tempora-1.14.1-5.el8.noarch.rp 269 kB/s | 29 kB 00:00 2024-05-15T22:53:59.412 INFO:teuthology.orchestra.run.smithi084.stdout:(27/31): python3-trustme-0.6.0-4.el8.noarch.rpm 253 kB/s | 27 kB 00:00 2024-05-15T22:53:59.471 INFO:teuthology.orchestra.run.smithi084.stdout:(28/31): python3-waitress-1.4.3-1.el8.noarch.rp 1.8 MB/s | 255 kB 00:00 2024-05-15T22:53:59.496 INFO:teuthology.orchestra.run.smithi084.stdout:(29/31): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.7 MB/s | 251 kB 00:00 2024-05-15T22:53:59.546 INFO:teuthology.orchestra.run.smithi084.stdout:(30/31): python3-webtest-2.0.33-1.el8.noarch.rp 638 kB/s | 85 kB 00:00 2024-05-15T22:53:59.571 INFO:teuthology.orchestra.run.smithi084.stdout:(31/31): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2024-05-15T22:53:59.572 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:53:59.573 INFO:teuthology.orchestra.run.smithi084.stdout:Total 11 MB/s | 28 MB 00:02 2024-05-15T22:53:59.655 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-05-15T22:53:59.695 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-05-15T22:53:59.695 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-05-15T22:53:59.987 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-05-15T22:53:59.990 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-05-15T22:54:01.035 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-05-15T22:54:01.297 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/31 2024-05-15T22:54:01.524 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/31 2024-05-15T22:54:01.760 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/31 2024-05-15T22:54:01.998 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/31 2024-05-15T22:54:02.194 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/31 2024-05-15T22:54:02.389 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/31 2024-05-15T22:54:02.589 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/31 2024-05-15T22:54:02.734 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/31 2024-05-15T22:54:02.938 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/31 2024-05-15T22:54:03.165 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/31 2024-05-15T22:54:03.421 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/31 2024-05-15T22:54:03.678 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/31 2024-05-15T22:54:04.107 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/31 2024-05-15T22:54:04.382 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/31 2024-05-15T22:54:04.569 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/31 2024-05-15T22:54:04.841 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/31 2024-05-15T22:54:05.037 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/31 2024-05-15T22:54:05.351 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/31 2024-05-15T22:54:05.592 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/31 2024-05-15T22:54:05.747 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/31 2024-05-15T22:54:05.924 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/31 2024-05-15T22:54:06.181 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/31 2024-05-15T22:54:06.390 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/31 2024-05-15T22:54:06.592 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/31 2024-05-15T22:54:06.927 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/31 2024-05-15T22:54:07.345 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/31 2024-05-15T22:54:07.614 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-05-15T22:54:08.983 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-05-15T22:54:09.164 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-05-15T22:54:09.553 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-05-15T22:54:09.727 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-05-15T22:54:09.915 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-05-15T22:54:10.102 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-05-15T22:54:10.119 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-05-15T22:54:10.234 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-2:14.2.22-0.el8.x86_64 31/31 2024-05-15T22:54:10.633 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-2:14.2.22-0.el8.x86_64 31/31 2024-05-15T22:54:10.633 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-2:14.2.22-0.el8.x86_64 1/31 2024-05-15T22:54:10.633 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mds-2:14.2.22-0.el8.x86_64 2/31 2024-05-15T22:54:10.633 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-2:14.2.22-0.el8.x86_64 3/31 2024-05-15T22:54:10.633 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mon-2:14.2.22-0.el8.x86_64 4/31 2024-05-15T22:54:10.634 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-osd-2:14.2.22-0.el8.x86_64 5/31 2024-05-15T22:54:10.634 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 6/31 2024-05-15T22:54:10.634 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 7/31 2024-05-15T22:54:10.634 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 8/31 2024-05-15T22:54:10.634 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 9/31 2024-05-15T22:54:10.634 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 10/31 2024-05-15T22:54:10.634 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 11/31 2024-05-15T22:54:10.634 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 12/31 2024-05-15T22:54:10.634 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 13/31 2024-05-15T22:54:10.634 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 14/31 2024-05-15T22:54:10.634 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 15/31 2024-05-15T22:54:10.634 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 16/31 2024-05-15T22:54:10.634 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 17/31 2024-05-15T22:54:10.634 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 18/31 2024-05-15T22:54:10.635 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 19/31 2024-05-15T22:54:10.635 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 20/31 2024-05-15T22:54:10.635 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 21/31 2024-05-15T22:54:10.635 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 22/31 2024-05-15T22:54:10.635 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-portend-2.6-1.el8.noarch 23/31 2024-05-15T22:54:10.635 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 24/31 2024-05-15T22:54:10.635 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 25/31 2024-05-15T22:54:10.635 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 26/31 2024-05-15T22:54:10.635 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 27/31 2024-05-15T22:54:10.635 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 28/31 2024-05-15T22:54:10.636 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 29/31 2024-05-15T22:54:10.636 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 30/31 2024-05-15T22:54:11.465 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 31/31 2024-05-15T22:54:11.465 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:11.465 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-05-15T22:54:11.465 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-2:14.2.22-0.el8.x86_64 2024-05-15T22:54:11.465 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds-2:14.2.22-0.el8.x86_64 2024-05-15T22:54:11.465 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-2:14.2.22-0.el8.x86_64 2024-05-15T22:54:11.466 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon-2:14.2.22-0.el8.x86_64 2024-05-15T22:54:11.466 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd-2:14.2.22-0.el8.x86_64 2024-05-15T22:54:11.466 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-05-15T22:54:11.466 INFO:teuthology.orchestra.run.smithi084.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-05-15T22:54:11.466 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-05-15T22:54:11.466 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-05-15T22:54:11.466 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-05-15T22:54:11.466 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-05-15T22:54:11.466 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-6.2-6.el8.noarch 2024-05-15T22:54:11.466 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-05-15T22:54:11.466 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-05-15T22:54:11.466 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako-1.0.6-14.el8.noarch 2024-05-15T22:54:11.466 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-05-15T22:54:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-05-15T22:54:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-05-15T22:54:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend-2.6-1.el8.noarch 2024-05-15T22:54:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-05-15T22:54:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser-2.14-14.el8.noarch 2024-05-15T22:54:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz-2017.2-11.el8.noarch 2024-05-15T22:54:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-05-15T22:54:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-05-15T22:54:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-05-15T22:54:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-05-15T22:54:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-05-15T22:54:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-05-15T22:54:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-05-15T22:54:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-05-15T22:54:11.468 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-05-15T22:54:11.468 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:11.468 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:54:11.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-base 2024-05-15T22:54:12.480 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:04 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:54:14.082 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-base-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:54:14.130 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:54:14.131 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-05-15T22:54:14.131 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:54:14.249 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr 2024-05-15T22:54:14.977 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:06 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:54:16.583 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-mgr-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:54:16.634 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:54:16.635 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-05-15T22:54:16.635 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:54:16.764 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-dashboard 2024-05-15T22:54:17.489 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:09 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:54:19.143 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:54:19.144 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:54:19.144 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-05-15T22:54:19.145 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:54:19.145 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-05-15T22:54:19.145 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard noarch 2:14.2.22-0.el8 ceph-noarch 3.7 M 2024-05-15T22:54:19.145 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-05-15T22:54:19.145 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards noarch 2:14.2.22-0.el8 ceph-noarch 24 k 2024-05-15T22:54:19.145 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-05-15T22:54:19.145 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-05-15T22:54:19.145 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-05-15T22:54:19.145 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:19.145 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-05-15T22:54:19.145 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:54:19.146 INFO:teuthology.orchestra.run.smithi084.stdout:Install 5 Packages 2024-05-15T22:54:19.146 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:19.147 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 4.0 M 2024-05-15T22:54:19.147 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 17 M 2024-05-15T22:54:19.147 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-05-15T22:54:19.525 INFO:teuthology.orchestra.run.smithi084.stdout:(1/5): ceph-grafana-dashboards-14.2.22-0.el8.no 398 kB/s | 24 kB 00:00 2024-05-15T22:54:19.592 INFO:teuthology.orchestra.run.smithi084.stdout:(2/5): ceph-mgr-dashboard-14.2.22-0.el8.noarch. 29 MB/s | 3.7 MB 00:00 2024-05-15T22:54:19.692 INFO:teuthology.orchestra.run.smithi084.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 201 kB/s | 34 kB 00:00 2024-05-15T22:54:19.726 INFO:teuthology.orchestra.run.smithi084.stdout:(4/5): python3-jwt-1.6.1-2.el8.noarch.rpm 163 kB/s | 43 kB 00:00 2024-05-15T22:54:19.817 INFO:teuthology.orchestra.run.smithi084.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 872 kB/s | 196 kB 00:00 2024-05-15T22:54:19.818 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:54:19.818 INFO:teuthology.orchestra.run.smithi084.stdout:Total 6.0 MB/s | 4.0 MB 00:00 2024-05-15T22:54:19.828 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-05-15T22:54:19.838 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-05-15T22:54:19.838 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-05-15T22:54:20.006 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-05-15T22:54:20.009 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-05-15T22:54:20.265 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-05-15T22:54:20.462 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-05-15T22:54:20.733 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-05-15T22:54:20.939 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-05-15T22:54:21.817 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 4/5 2024-05-15T22:54:22.085 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-05-15T22:54:22.625 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-05-15T22:54:22.625 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 1/5 2024-05-15T22:54:22.625 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2/5 2024-05-15T22:54:22.625 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-05-15T22:54:22.625 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-05-15T22:54:23.771 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-05-15T22:54:23.771 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:23.771 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-05-15T22:54:23.772 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 2024-05-15T22:54:23.772 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2024-05-15T22:54:23.772 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-05-15T22:54:23.772 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-05-15T22:54:23.772 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes-2.4.1-12.el8.noarch 2024-05-15T22:54:23.772 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:23.772 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:54:23.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-15T22:54:24.662 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:16 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:54:26.316 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:54:26.317 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-05-15T22:54:26.317 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-05-15T22:54:26.317 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-05-15T22:54:26.317 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-05-15T22:54:26.317 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local noarch 2:14.2.22-0.el8 ceph-noarch 1.1 M 2024-05-15T22:54:26.317 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-05-15T22:54:26.318 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran x86_64 8.5.0-22.el8 baseos 645 k 2024-05-15T22:54:26.318 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath x86_64 8.5.0-22.el8 baseos 172 k 2024-05-15T22:54:26.318 INFO:teuthology.orchestra.run.smithi084.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-05-15T22:54:26.318 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-05-15T22:54:26.318 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-05-15T22:54:26.318 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-05-15T22:54:26.318 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-05-15T22:54:26.318 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:26.318 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-05-15T22:54:26.318 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-05-15T22:54:26.318 INFO:teuthology.orchestra.run.smithi084.stdout:Install 8 Packages 2024-05-15T22:54:26.318 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:26.319 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 30 M 2024-05-15T22:54:26.319 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 172 M 2024-05-15T22:54:26.319 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-05-15T22:54:26.786 INFO:teuthology.orchestra.run.smithi084.stdout:(1/8): ceph-mgr-diskprediction-local-14.2.22-0. 18 MB/s | 1.1 MB 00:00 2024-05-15T22:54:27.595 INFO:teuthology.orchestra.run.smithi084.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.6 MB/s | 4.9 MB 00:00 2024-05-15T22:54:27.653 INFO:teuthology.orchestra.run.smithi084.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 5.1 MB/s | 4.7 MB 00:00 2024-05-15T22:54:27.720 INFO:teuthology.orchestra.run.smithi084.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.0 MB/s | 3.7 MB 00:00 2024-05-15T22:54:27.745 INFO:teuthology.orchestra.run.smithi084.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.5 MB/s | 225 kB 00:00 2024-05-15T22:54:27.912 INFO:teuthology.orchestra.run.smithi084.stdout:(6/8): libquadmath-8.5.0-22.el8.x86_64.rpm 1.0 MB/s | 172 kB 00:00 2024-05-15T22:54:27.996 INFO:teuthology.orchestra.run.smithi084.stdout:(7/8): libgfortran-8.5.0-22.el8.x86_64.rpm 2.3 MB/s | 645 kB 00:00 2024-05-15T22:54:28.463 INFO:teuthology.orchestra.run.smithi084.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 18 MB/s | 14 MB 00:00 2024-05-15T22:54:28.463 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:54:28.464 INFO:teuthology.orchestra.run.smithi084.stdout:Total 14 MB/s | 30 MB 00:02 2024-05-15T22:54:28.696 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-05-15T22:54:28.714 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-05-15T22:54:28.714 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-05-15T22:54:29.018 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-05-15T22:54:29.021 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-05-15T22:54:29.558 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-05-15T22:54:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libquadmath-8.5.0-22.el8.x86_64 1/8 2024-05-15T22:54:29.820 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libquadmath-8.5.0-22.el8.x86_64 1/8 2024-05-15T22:54:29.963 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libgfortran-8.5.0-22.el8.x86_64 2/8 2024-05-15T22:54:30.579 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libgfortran-8.5.0-22.el8.x86_64 2/8 2024-05-15T22:54:30.730 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-05-15T22:54:31.350 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-05-15T22:54:31.480 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-05-15T22:54:32.175 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-05-15T22:54:32.498 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-05-15T22:54:32.737 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-05-15T22:54:34.472 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-05-15T22:54:35.039 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-05-15T22:54:35.254 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-05-15T22:54:35.574 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-05-15T22:54:35.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 1/8 2024-05-15T22:54:35.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-05-15T22:54:35.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-05-15T22:54:35.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-05-15T22:54:35.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-05-15T22:54:35.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-05-15T22:54:35.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libgfortran-8.5.0-22.el8.x86_64 7/8 2024-05-15T22:54:35.998 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libquadmath-8.5.0-22.el8.x86_64 8/8 2024-05-15T22:54:35.998 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:35.998 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-05-15T22:54:35.998 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 2024-05-15T22:54:35.998 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran-8.5.0-22.el8.x86_64 2024-05-15T22:54:35.998 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath-8.5.0-22.el8.x86_64 2024-05-15T22:54:35.999 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-0.3.15-4.el8.x86_64 2024-05-15T22:54:35.999 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-05-15T22:54:35.999 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-05-15T22:54:35.999 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-05-15T22:54:35.999 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-05-15T22:54:35.999 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:35.999 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:54:36.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-rook 2024-05-15T22:54:36.869 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:28 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:54:38.550 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:54:38.552 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:54:38.552 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-05-15T22:54:38.552 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:54:38.552 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-05-15T22:54:38.553 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook noarch 2:14.2.22-0.el8 ceph-noarch 15 k 2024-05-15T22:54:38.553 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-05-15T22:54:38.553 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-05-15T22:54:38.553 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-05-15T22:54:38.553 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-05-15T22:54:38.553 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-05-15T22:54:38.553 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-05-15T22:54:38.553 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-05-15T22:54:38.553 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-05-15T22:54:38.553 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-05-15T22:54:38.553 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-05-15T22:54:38.553 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-05-15T22:54:38.553 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:38.553 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-05-15T22:54:38.553 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:54:38.554 INFO:teuthology.orchestra.run.smithi084.stdout:Install 11 Packages 2024-05-15T22:54:38.554 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:38.554 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 1.9 M 2024-05-15T22:54:38.554 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 24 M 2024-05-15T22:54:38.554 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-05-15T22:54:38.870 INFO:teuthology.orchestra.run.smithi084.stdout:(1/11): ceph-mgr-rook-14.2.22-0.el8.noarch.rpm 375 kB/s | 15 kB 00:00 2024-05-15T22:54:39.062 INFO:teuthology.orchestra.run.smithi084.stdout:(2/11): python3-oauthlib-2.1.0-1.el8.noarch.rpm 808 kB/s | 155 kB 00:00 2024-05-15T22:54:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:(3/11): python3-requests-oauthlib-1.0.0-1.el8.n 206 kB/s | 43 kB 00:00 2024-05-15T22:54:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:(4/11): python3-pyasn1-modules-0.3.7-6.el8.noar 182 kB/s | 110 kB 00:00 2024-05-15T22:54:39.530 INFO:teuthology.orchestra.run.smithi084.stdout:(5/11): python3-pyasn1-0.3.7-6.el8.noarch.rpm 180 kB/s | 126 kB 00:00 2024-05-15T22:54:39.915 INFO:teuthology.orchestra.run.smithi084.stdout:(6/11): python3-certifi-2018.10.15-7.el8.noarch 34 kB/s | 16 kB 00:00 2024-05-15T22:54:40.286 INFO:teuthology.orchestra.run.smithi084.stdout:(7/11): python3-cachetools-3.1.1-4.el8.noarch.r 32 kB/s | 33 kB 00:01 2024-05-15T22:54:40.415 INFO:teuthology.orchestra.run.smithi084.stdout:(8/11): python3-google-auth-1.1.1-10.el8.noarch 92 kB/s | 81 kB 00:00 2024-05-15T22:54:40.441 INFO:teuthology.orchestra.run.smithi084.stdout:(9/11): python3-rsa-4.9-2.el8.noarch.rpm 415 kB/s | 64 kB 00:00 2024-05-15T22:54:40.491 INFO:teuthology.orchestra.run.smithi084.stdout:(10/11): python3-websocket-client-0.56.0-5.el8. 816 kB/s | 61 kB 00:00 2024-05-15T22:54:40.774 INFO:teuthology.orchestra.run.smithi084.stdout:(11/11): python3-kubernetes-11.0.0-6.el8.noarch 1.4 MB/s | 1.2 MB 00:00 2024-05-15T22:54:40.775 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:54:40.776 INFO:teuthology.orchestra.run.smithi084.stdout:Total 854 kB/s | 1.9 MB 00:02 2024-05-15T22:54:40.815 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-05-15T22:54:40.831 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-05-15T22:54:40.831 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-05-15T22:54:40.906 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-05-15T22:54:40.910 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-05-15T22:54:41.402 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-05-15T22:54:41.629 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/11 2024-05-15T22:54:41.818 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/11 2024-05-15T22:54:42.054 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/11 2024-05-15T22:54:42.267 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/11 2024-05-15T22:54:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/11 2024-05-15T22:54:42.749 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-05-15T22:54:43.032 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/11 2024-05-15T22:54:43.278 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/11 2024-05-15T22:54:44.318 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/11 2024-05-15T22:54:44.587 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/11 2024-05-15T22:54:44.743 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-05-15T22:54:45.027 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-05-15T22:54:45.028 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-rook-2:14.2.22-0.el8.noarch 1/11 2024-05-15T22:54:45.028 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 2/11 2024-05-15T22:54:45.028 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/11 2024-05-15T22:54:45.028 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 4/11 2024-05-15T22:54:45.028 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/11 2024-05-15T22:54:45.028 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-05-15T22:54:45.028 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 7/11 2024-05-15T22:54:45.028 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 8/11 2024-05-15T22:54:45.028 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 9/11 2024-05-15T22:54:45.028 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 10/11 2024-05-15T22:54:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 11/11 2024-05-15T22:54:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:45.461 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-05-15T22:54:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook-2:14.2.22-0.el8.noarch 2024-05-15T22:54:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-05-15T22:54:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-05-15T22:54:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-05-15T22:54:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-05-15T22:54:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-05-15T22:54:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-05-15T22:54:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-05-15T22:54:45.462 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-05-15T22:54:45.462 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa-4.9-2.el8.noarch 2024-05-15T22:54:45.462 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-05-15T22:54:45.462 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:45.462 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:54:45.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-fuse 2024-05-15T22:54:46.334 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:38 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:54:47.972 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:54:47.972 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:54:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-05-15T22:54:47.972 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:54:47.972 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-05-15T22:54:47.972 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse x86_64 2:14.2.22-0.el8 ceph 590 k 2024-05-15T22:54:47.973 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-05-15T22:54:47.973 INFO:teuthology.orchestra.run.smithi084.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-05-15T22:54:47.973 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:47.973 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-05-15T22:54:47.973 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:54:47.973 INFO:teuthology.orchestra.run.smithi084.stdout:Install 2 Packages 2024-05-15T22:54:47.973 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:47.974 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 673 k 2024-05-15T22:54:47.974 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 1.9 M 2024-05-15T22:54:47.974 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-05-15T22:54:48.213 INFO:teuthology.orchestra.run.smithi084.stdout:(1/2): ceph-fuse-14.2.22-0.el8.x86_64.rpm 10 MB/s | 590 kB 00:00 2024-05-15T22:54:48.330 INFO:teuthology.orchestra.run.smithi084.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 482 kB/s | 83 kB 00:00 2024-05-15T22:54:48.330 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:54:48.330 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.8 MB/s | 673 kB 00:00 2024-05-15T22:54:48.340 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-05-15T22:54:48.362 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-05-15T22:54:48.453 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-05-15T22:54:48.454 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-05-15T22:54:48.454 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-05-15T22:54:48.623 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-05-15T22:54:48.890 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-05-15T22:54:49.092 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-05-15T22:54:49.477 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-05-15T22:54:49.478 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-fuse-2:14.2.22-0.el8.x86_64 1/2 2024-05-15T22:54:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-05-15T22:54:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:49.900 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-05-15T22:54:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse-2:14.2.22-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-05-15T22:54:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:49.900 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:54:50.123 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados-devel 2024-05-15T22:54:50.882 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:42 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:54:52.520 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:54:52.521 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:54:52.521 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-05-15T22:54:52.521 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:54:52.521 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-05-15T22:54:52.521 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel x86_64 2:14.2.22-0.el8 ceph 90 k 2024-05-15T22:54:52.521 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:52.521 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-05-15T22:54:52.521 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:54:52.522 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-05-15T22:54:52.522 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:52.522 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 90 k 2024-05-15T22:54:52.522 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 310 k 2024-05-15T22:54:52.522 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-05-15T22:54:52.574 INFO:teuthology.orchestra.run.smithi084.stdout:librados-devel-14.2.22-0.el8.x86_64.rpm 1.7 MB/s | 90 kB 00:00 2024-05-15T22:54:52.575 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:54:52.575 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.6 MB/s | 90 kB 00:00 2024-05-15T22:54:52.575 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-05-15T22:54:52.586 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-05-15T22:54:52.586 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-05-15T22:54:52.634 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-05-15T22:54:52.637 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-05-15T22:54:52.779 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-05-15T22:54:52.973 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:54:53.253 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:54:53.617 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:54:53.617 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:53.617 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-05-15T22:54:53.617 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel-2:14.2.22-0.el8.x86_64 2024-05-15T22:54:53.617 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:53.618 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:54:53.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs2 2024-05-15T22:54:54.636 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:46 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:54:56.228 INFO:teuthology.orchestra.run.smithi084.stdout:Package libcephfs2-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:54:56.278 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:54:56.279 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-05-15T22:54:56.279 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:54:56.387 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs-devel 2024-05-15T22:54:57.140 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:49 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:54:58.795 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:54:58.804 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:54:58.804 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-05-15T22:54:58.805 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:54:58.805 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-05-15T22:54:58.805 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel x86_64 2:14.2.22-0.el8 ceph 22 k 2024-05-15T22:54:58.805 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:58.805 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-05-15T22:54:58.805 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:54:58.805 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-05-15T22:54:58.805 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:58.805 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 22 k 2024-05-15T22:54:58.805 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 77 k 2024-05-15T22:54:58.805 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-05-15T22:54:58.845 INFO:teuthology.orchestra.run.smithi084.stdout:libcephfs-devel-14.2.22-0.el8.x86_64.rpm 464 kB/s | 22 kB 00:00 2024-05-15T22:54:58.845 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:54:58.847 INFO:teuthology.orchestra.run.smithi084.stdout:Total 452 kB/s | 22 kB 00:00 2024-05-15T22:54:58.847 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-05-15T22:54:58.851 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-05-15T22:54:58.852 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-05-15T22:54:58.852 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-05-15T22:54:58.854 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-05-15T22:54:58.907 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-05-15T22:54:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:54:59.354 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:54:59.707 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:54:59.707 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:59.707 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-05-15T22:54:59.707 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel-2:14.2.22-0.el8.x86_64 2024-05-15T22:54:59.707 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:54:59.707 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:54:59.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados2 2024-05-15T22:55:00.579 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:52 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:55:02.158 INFO:teuthology.orchestra.run.smithi084.stdout:Package librados2-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:55:02.206 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:55:02.207 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-05-15T22:55:02.207 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:55:02.321 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librbd1 2024-05-15T22:55:03.082 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:55 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:55:04.689 INFO:teuthology.orchestra.run.smithi084.stdout:Package librbd1-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:55:04.737 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:55:04.738 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-05-15T22:55:04.738 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:55:04.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-fuse 2024-05-15T22:55:05.594 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:57 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:55:07.236 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:55:07.236 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:55:07.236 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-05-15T22:55:07.236 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:55:07.236 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-05-15T22:55:07.236 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse x86_64 2:14.2.22-0.el8 ceph 73 k 2024-05-15T22:55:07.236 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:55:07.237 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-05-15T22:55:07.237 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:55:07.237 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-05-15T22:55:07.237 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:55:07.237 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 73 k 2024-05-15T22:55:07.237 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 188 k 2024-05-15T22:55:07.237 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-05-15T22:55:07.682 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-fuse-14.2.22-0.el8.x86_64.rpm 164 kB/s | 73 kB 00:00 2024-05-15T22:55:07.683 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:55:07.683 INFO:teuthology.orchestra.run.smithi084.stdout:Total 163 kB/s | 73 kB 00:00 2024-05-15T22:55:07.683 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-05-15T22:55:07.694 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-05-15T22:55:07.695 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-05-15T22:55:07.744 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-05-15T22:55:07.747 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-05-15T22:55:07.855 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-05-15T22:55:08.103 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:55:08.391 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:55:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:55:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:55:08.760 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-05-15T22:55:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse-2:14.2.22-0.el8.x86_64 2024-05-15T22:55:08.761 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:55:08.761 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:55:08.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-mirror 2024-05-15T22:55:09.742 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:01 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:55:11.402 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:55:11.402 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:55:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-05-15T22:55:11.403 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:55:11.403 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-05-15T22:55:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror x86_64 2:14.2.22-0.el8 ceph 2.3 M 2024-05-15T22:55:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:55:11.403 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-05-15T22:55:11.403 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:55:11.403 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-05-15T22:55:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:55:11.403 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 2.3 M 2024-05-15T22:55:11.403 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 8.4 M 2024-05-15T22:55:11.403 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-05-15T22:55:11.484 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-mirror-14.2.22-0.el8.x86_64.rpm 29 MB/s | 2.3 MB 00:00 2024-05-15T22:55:11.485 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:55:11.485 INFO:teuthology.orchestra.run.smithi084.stdout:Total 28 MB/s | 2.3 MB 00:00 2024-05-15T22:55:11.485 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-05-15T22:55:11.497 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-05-15T22:55:11.497 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-05-15T22:55:11.554 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-05-15T22:55:11.557 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-05-15T22:55:11.856 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-05-15T22:55:12.029 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:55:12.428 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:55:13.048 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:55:13.048 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:55:13.048 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-05-15T22:55:13.048 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror-2:14.2.22-0.el8.x86_64 2024-05-15T22:55:13.048 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:55:13.048 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:55:13.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-nbd 2024-05-15T22:55:14.029 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:06 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:55:15.671 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:55:15.672 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:55:15.672 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-05-15T22:55:15.672 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:55:15.672 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-05-15T22:55:15.672 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd x86_64 2:14.2.22-0.el8 ceph 142 k 2024-05-15T22:55:15.672 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:55:15.672 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-05-15T22:55:15.672 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-05-15T22:55:15.672 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-05-15T22:55:15.672 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:55:15.673 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 142 k 2024-05-15T22:55:15.673 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 422 k 2024-05-15T22:55:15.673 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-05-15T22:55:15.720 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-nbd-14.2.22-0.el8.x86_64.rpm 3.0 MB/s | 142 kB 00:00 2024-05-15T22:55:15.720 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:55:15.721 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.9 MB/s | 142 kB 00:00 2024-05-15T22:55:15.721 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-05-15T22:55:15.732 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-05-15T22:55:15.732 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-05-15T22:55:15.792 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-05-15T22:55:15.795 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-05-15T22:55:15.911 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-05-15T22:55:16.085 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:55:16.375 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:55:16.758 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-05-15T22:55:16.759 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:55:16.759 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-05-15T22:55:16.759 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd-2:14.2.22-0.el8.x86_64 2024-05-15T22:55:16.759 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:55:16.759 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:55:16.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python-rados 2024-05-15T22:55:17.752 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:09 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:55:19.343 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rados-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:55:19.393 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:55:19.394 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-05-15T22:55:19.394 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:55:19.500 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python-rgw 2024-05-15T22:55:20.252 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:12 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:55:21.840 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rgw-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:55:21.889 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:55:21.890 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-05-15T22:55:21.890 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:55:21.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python-rbd 2024-05-15T22:55:22.749 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:14 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:55:24.347 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rbd-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:55:24.397 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:55:24.397 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-05-15T22:55:24.398 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:55:24.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python-cephfs 2024-05-15T22:55:25.273 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:17 ago on Wed 15 May 2024 10:53:08 PM UTC. 2024-05-15T22:55:26.867 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-cephfs-2:14.2.22-0.el8.x86_64 is already installed. 2024-05-15T22:55:26.917 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-05-15T22:55:26.918 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-05-15T22:55:26.918 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-05-15T22:55:27.033 DEBUG:teuthology.parallel:result is None 2024-05-15T22:55:27.034 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T22:55:27.034 INFO:teuthology.packaging:ref: None 2024-05-15T22:55:27.034 INFO:teuthology.packaging:tag: None 2024-05-15T22:55:27.034 INFO:teuthology.packaging:branch: nautilus 2024-05-15T22:55:27.034 INFO:teuthology.packaging:sha1: 6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 2024-05-15T22:55:27.034 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-05-15T22:55:27.228 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-15T22:55:27.250 INFO:teuthology.orchestra.run.smithi084.stdout:14.2.22-0.el8 2024-05-15T22:55:27.251 INFO:teuthology.packaging:The installed version of ceph is 14.2.22-0.el8 2024-05-15T22:55:27.251 INFO:teuthology.task.install:The correct ceph version 14.2.22-0 is installed. 2024-05-15T22:55:27.253 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T22:55:27.253 INFO:teuthology.packaging:ref: None 2024-05-15T22:55:27.253 INFO:teuthology.packaging:tag: None 2024-05-15T22:55:27.253 INFO:teuthology.packaging:branch: nautilus 2024-05-15T22:55:27.253 INFO:teuthology.packaging:sha1: 6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 2024-05-15T22:55:27.253 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-05-15T22:55:27.443 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-15T22:55:27.470 INFO:teuthology.orchestra.run.smithi088.stdout:14.2.22-0.el8 2024-05-15T22:55:27.470 INFO:teuthology.packaging:The installed version of ceph is 14.2.22-0.el8 2024-05-15T22:55:27.470 INFO:teuthology.task.install:The correct ceph version 14.2.22-0 is installed. 2024-05-15T22:55:27.472 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T22:55:27.472 INFO:teuthology.packaging:ref: None 2024-05-15T22:55:27.473 INFO:teuthology.packaging:tag: None 2024-05-15T22:55:27.473 INFO:teuthology.packaging:branch: nautilus 2024-05-15T22:55:27.473 INFO:teuthology.packaging:sha1: 6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 2024-05-15T22:55:27.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-05-15T22:55:27.733 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-15T22:55:27.759 INFO:teuthology.orchestra.run.smithi090.stdout:14.2.22-0.el8 2024-05-15T22:55:27.759 INFO:teuthology.packaging:The installed version of ceph is 14.2.22-0.el8 2024-05-15T22:55:27.759 INFO:teuthology.task.install:The correct ceph version 14.2.22-0 is installed. 2024-05-15T22:55:27.761 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-15T22:55:27.761 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-05-15T22:55:27.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-15T22:55:27.790 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T22:55:27.791 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-15T22:55:27.823 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-05-15T22:55:27.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-15T22:55:27.855 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-15T22:55:27.855 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-05-15T22:55:27.855 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/daemon-helper 2024-05-15T22:55:27.884 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-15T22:55:27.952 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T22:55:27.952 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2024-05-15T22:55:27.984 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-15T22:55:28.054 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-05-15T22:55:28.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-05-15T22:55:28.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-15T22:55:28.156 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-15T22:55:28.156 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-05-15T22:55:28.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-15T22:55:28.185 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-15T22:55:28.254 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T22:55:28.254 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-15T22:55:28.284 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-15T22:55:28.354 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-05-15T22:55:28.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-15T22:55:28.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-15T22:55:28.454 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-15T22:55:28.455 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-05-15T22:55:28.455 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/stdin-killer 2024-05-15T22:55:28.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-15T22:55:28.552 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T22:55:28.552 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2024-05-15T22:55:28.583 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-15T22:55:28.654 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-05-15T22:55:28.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-05-15T22:55:28.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-15T22:55:28.753 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-15T22:55:28.863 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd heartbeat use min delay socket': True, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'mon_bind_msgr2': False, 'sha1': '6cd0f8013e2dea00c3a29a0d8b10656b132d7c80'} 2024-05-15T22:55:28.863 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 2024-05-15T22:55:28.863 INFO:tasks.cephadm:Cluster fsid is 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:55:28.863 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-15T22:55:28.864 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.84:6789]', 'mon.b': '[v1:172.21.15.88:6789]', 'mon.c': '[v1:172.21.15.90:6789]'} 2024-05-15T22:55:28.864 INFO:tasks.cephadm:First mon is mon.a on smithi084 2024-05-15T22:55:28.864 INFO:tasks.cephadm:First mgr is y 2024-05-15T22:55:28.864 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-15T22:55:28.864 DEBUG:teuthology.orchestra.run.smithi084:> sudo hostname $(hostname -s) 2024-05-15T22:55:28.896 DEBUG:teuthology.orchestra.run.smithi088:> sudo hostname $(hostname -s) 2024-05-15T22:55:28.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo hostname $(hostname -s) 2024-05-15T22:55:28.957 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-15T22:55:28.958 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 2024-05-15T22:55:29.103 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '6cd0f8013e2dea00c3a29a0d8b10656b132d7c80', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/79412/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3728-g6cd0f801', 'node_name': '172.21.4.66+confusa04', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3728.g6cd0f801'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-05-15-0804/6cd0f8013e2dea00c3a29a0d8b10656b132d7c80/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-05-15 21:27:25.705869', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri5-testing-2024-05-15-0804', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-yuri5-testing-2024-05-15-0804/6cd0f8013e2dea00c3a29a0d8b10656b132d7c80/centos/8/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}] 2024-05-15T22:55:29.255 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-yuri5-testing-2024-05-15-0804, sha1 6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 2024-05-15T22:55:29.256 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-yuri5-testing-2024-05-15-0804/6cd0f8013e2dea00c3a29a0d8b10656b132d7c80/centos/8/x86_64/flavors/default/cephadm 2024-05-15T22:55:29.256 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/wip-yuri5-testing-2024-05-15-0804/6cd0f8013e2dea00c3a29a0d8b10656b132d7c80/centos/8/x86_64/flavors/default/cephadm 2024-05-15T22:55:29.256 DEBUG:teuthology.orchestra.run.smithi084:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-yuri5-testing-2024-05-15-0804/6cd0f8013e2dea00c3a29a0d8b10656b132d7c80/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-15T22:55:29.756 INFO:teuthology.orchestra.run.smithi084.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2297301 May 15 22:55 /home/ubuntu/cephtest/cephadm 2024-05-15T22:55:29.757 DEBUG:teuthology.orchestra.run.smithi088:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-yuri5-testing-2024-05-15-0804/6cd0f8013e2dea00c3a29a0d8b10656b132d7c80/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-15T22:55:30.438 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2297301 May 15 22:55 /home/ubuntu/cephtest/cephadm 2024-05-15T22:55:30.439 DEBUG:teuthology.orchestra.run.smithi090:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-yuri5-testing-2024-05-15-0804/6cd0f8013e2dea00c3a29a0d8b10656b132d7c80/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-15T22:55:31.118 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2297301 May 15 22:55 /home/ubuntu/cephtest/cephadm 2024-05-15T22:55:31.119 DEBUG:teuthology.orchestra.run.smithi084:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-15T22:55:31.141 DEBUG:teuthology.orchestra.run.smithi088:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-15T22:55:31.164 DEBUG:teuthology.orchestra.run.smithi090:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-15T22:55:31.198 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 on all hosts... 2024-05-15T22:55:31.198 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 pull 2024-05-15T22:55:31.201 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 pull 2024-05-15T22:55:31.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 pull 2024-05-15T22:55:32.230 INFO:teuthology.orchestra.run.smithi088.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80... 2024-05-15T22:55:32.232 INFO:teuthology.orchestra.run.smithi090.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80... 2024-05-15T22:55:32.362 INFO:teuthology.orchestra.run.smithi084.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80... 2024-05-15T22:56:08.081 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T22:56:08.082 INFO:teuthology.orchestra.run.smithi084.stdout: "ceph_version": "ceph version 19.0.0-3728-g6cd0f801 (6cd0f8013e2dea00c3a29a0d8b10656b132d7c80) squid (dev)", 2024-05-15T22:56:08.082 INFO:teuthology.orchestra.run.smithi084.stdout: "image_id": "41a626c9cc374bd028aa22b4f080d069b610f0be9e0ef97f9a1a54e447e148a3", 2024-05-15T22:56:08.082 INFO:teuthology.orchestra.run.smithi084.stdout: "repo_digests": [ 2024-05-15T22:56:08.082 INFO:teuthology.orchestra.run.smithi084.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5c5f12459ebcbf3753cc9d18fdb5ddd79e42ac036cf0c5c0a49db825e6a86185" 2024-05-15T22:56:08.082 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T22:56:08.082 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T22:56:19.615 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T22:56:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-3728-g6cd0f801 (6cd0f8013e2dea00c3a29a0d8b10656b132d7c80) squid (dev)", 2024-05-15T22:56:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: "image_id": "41a626c9cc374bd028aa22b4f080d069b610f0be9e0ef97f9a1a54e447e148a3", 2024-05-15T22:56:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: "repo_digests": [ 2024-05-15T22:56:19.615 INFO:teuthology.orchestra.run.smithi090.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5c5f12459ebcbf3753cc9d18fdb5ddd79e42ac036cf0c5c0a49db825e6a86185" 2024-05-15T22:56:19.616 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T22:56:19.616 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T22:56:21.722 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T22:56:21.722 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3728-g6cd0f801 (6cd0f8013e2dea00c3a29a0d8b10656b132d7c80) squid (dev)", 2024-05-15T22:56:21.722 INFO:teuthology.orchestra.run.smithi088.stdout: "image_id": "41a626c9cc374bd028aa22b4f080d069b610f0be9e0ef97f9a1a54e447e148a3", 2024-05-15T22:56:21.722 INFO:teuthology.orchestra.run.smithi088.stdout: "repo_digests": [ 2024-05-15T22:56:21.722 INFO:teuthology.orchestra.run.smithi088.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5c5f12459ebcbf3753cc9d18fdb5ddd79e42ac036cf0c5c0a49db825e6a86185" 2024-05-15T22:56:21.722 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T22:56:21.722 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T22:56:21.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /etc/ceph 2024-05-15T22:56:21.788 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph 2024-05-15T22:56:21.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph 2024-05-15T22:56:21.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 777 /etc/ceph 2024-05-15T22:56:21.889 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /etc/ceph 2024-05-15T22:56:21.916 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /etc/ceph 2024-05-15T22:56:21.949 INFO:tasks.cephadm:Writing seed config... 2024-05-15T22:56:21.950 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-05-15T22:56:21.950 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-15T22:56:21.951 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-15T22:56:21.951 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-15T22:56:21.951 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-05-15T22:56:21.951 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-05-15T22:56:21.951 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-05-15T22:56:21.951 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-05-15T22:56:21.951 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-05-15T22:56:21.951 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-05-15T22:56:21.951 INFO:tasks.cephadm: override: [client.2] debug ms = 1 2024-05-15T22:56:21.952 INFO:tasks.cephadm: override: [client.2] debug objecter = 20 2024-05-15T22:56:21.952 INFO:tasks.cephadm: override: [client.2] debug rados = 20 2024-05-15T22:56:21.952 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-05-15T22:56:21.952 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-05-15T22:56:21.952 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-05-15T22:56:21.952 INFO:tasks.cephadm: override: [global] ms inject socket failures = 5000 2024-05-15T22:56:21.952 INFO:tasks.cephadm: override: [global] ms type = async 2024-05-15T22:56:21.952 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-05-15T22:56:21.952 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 2 2024-05-15T22:56:21.952 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-15T22:56:21.952 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-15T22:56:21.952 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] osd backoff on peering = True 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] osd heartbeat use min delay socket = True 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] osd map cache size = 1 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] osd max backfills = 6 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] osd scrub during recovery = False 2024-05-15T22:56:21.953 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-05-15T22:56:21.954 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-05-15T22:56:21.954 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-15T22:56:21.954 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-05-15T22:56:21.954 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-15T22:56:21.976 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3999fb7e-130e-11ef-bc99-c7b262605968 mon client directed command retry = 5 mon election default strategy = 1 ms bind msgr2 = False ms inject socket failures = 5000 ms type = async osd_pool_default_min_size = 2 osd_pool_default_size = 2 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 osd backoff on peering = True osd blocked scrub grace period = 3600 osd heartbeat use min delay socket = True osd map cache size = 1 osd max backfills = 6 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub during recovery = False osd scrub min interval = 60 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon scrub interval = 300 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.2] debug ms = 1 debug objecter = 20 debug rados = 20 2024-05-15T22:56:21.976 DEBUG:teuthology.orchestra.run.smithi084:mon.a> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mon.a.service 2024-05-15T22:56:22.019 DEBUG:teuthology.orchestra.run.smithi084:mgr.y> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mgr.y.service 2024-05-15T22:56:22.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:-- Logs begin at Wed 2024-05-15 22:43:24 UTC. -- 2024-05-15T22:56:22.052 INFO:tasks.cephadm:Bootstrapping... 2024-05-15T22:56:22.052 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 -v bootstrap --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.84:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-15T22:56:22.081 INFO:journalctl@ceph.mgr.y.smithi084.stdout:-- Logs begin at Wed 2024-05-15 22:43:24 UTC. -- 2024-05-15T22:56:22.261 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-05-15T22:56:22.261 INFO:teuthology.orchestra.run.smithi084.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80', '-v', 'bootstrap', '--fsid', '3999fb7e-130e-11ef-bc99-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.84:6789]', '--skip-admin-label'] 2024-05-15T22:56:22.262 INFO:teuthology.orchestra.run.smithi084.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-15T22:56:22.262 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying podman|docker is present... 2024-05-15T22:56:22.294 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-15T22:56:22.294 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying lvm2 is present... 2024-05-15T22:56:22.294 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying time synchronization is in place... 2024-05-15T22:56:22.303 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-15T22:56:22.303 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-15T22:56:22.307 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-15T22:56:22.307 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-05-15T22:56:22.313 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout enabled 2024-05-15T22:56:22.319 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout active 2024-05-15T22:56:22.319 INFO:teuthology.orchestra.run.smithi084.stdout:Unit chronyd.service is enabled and running 2024-05-15T22:56:22.319 INFO:teuthology.orchestra.run.smithi084.stdout:Repeating the final host check... 2024-05-15T22:56:22.350 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-15T22:56:22.350 INFO:teuthology.orchestra.run.smithi084.stdout:podman (/bin/podman) version 4.9.4 is present 2024-05-15T22:56:22.350 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl is present 2024-05-15T22:56:22.351 INFO:teuthology.orchestra.run.smithi084.stdout:lvcreate is present 2024-05-15T22:56:22.356 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-15T22:56:22.356 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-15T22:56:22.362 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-15T22:56:22.362 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-05-15T22:56:22.368 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout enabled 2024-05-15T22:56:22.374 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout active 2024-05-15T22:56:22.374 INFO:teuthology.orchestra.run.smithi084.stdout:Unit chronyd.service is enabled and running 2024-05-15T22:56:22.374 INFO:teuthology.orchestra.run.smithi084.stdout:Host looks OK 2024-05-15T22:56:22.374 INFO:teuthology.orchestra.run.smithi084.stdout:Cluster fsid: 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:56:22.374 INFO:teuthology.orchestra.run.smithi084.stdout:Acquiring lock 139709101110160 on /run/cephadm/3999fb7e-130e-11ef-bc99-c7b262605968.lock 2024-05-15T22:56:22.374 INFO:teuthology.orchestra.run.smithi084.stdout:Lock 139709101110160 acquired on /run/cephadm/3999fb7e-130e-11ef-bc99-c7b262605968.lock 2024-05-15T22:56:22.374 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying IP 172.21.15.84 port 6789 ... 2024-05-15T22:56:22.375 INFO:teuthology.orchestra.run.smithi084.stdout:Base mon IP(s) is [172.21.15.84:6789], mon addrv is [v1:172.21.15.84:6789] 2024-05-15T22:56:22.378 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.84 metric 100 2024-05-15T22:56:22.379 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.84 metric 100 2024-05-15T22:56:22.381 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-15T22:56:22.381 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-05-15T22:56:22.382 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1780sec hoplimit 64 pref medium 2024-05-15T22:56:22.384 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-15T22:56:22.384 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-15T22:56:22.384 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-15T22:56:22.384 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-15T22:56:22.384 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d751/64 scope link 2024-05-15T22:56:22.385 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-15T22:56:22.385 INFO:teuthology.orchestra.run.smithi084.stdout:Mon IP `172.21.15.84` is in CIDR network `172.21.0.0/20` 2024-05-15T22:56:22.385 INFO:teuthology.orchestra.run.smithi084.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-05-15T22:56:22.385 INFO:teuthology.orchestra.run.smithi084.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-15T22:56:22.386 INFO:teuthology.orchestra.run.smithi084.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80... 2024-05-15T22:56:22.841 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stdout 41a626c9cc374bd028aa22b4f080d069b610f0be9e0ef97f9a1a54e447e148a3 2024-05-15T22:56:22.841 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80... 2024-05-15T22:56:22.841 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Getting image source signatures 2024-05-15T22:56:22.841 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Copying blob sha256:322a931fbf62dac647a06ca6d7cc6d0c80fc5fd5d3ff1649477e88a311b8685a 2024-05-15T22:56:22.841 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Copying blob sha256:d0e808cd455827ca3108db1b0125dcee9ffc148daa1d7dde963f550bc831d501 2024-05-15T22:56:22.841 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Copying config sha256:41a626c9cc374bd028aa22b4f080d069b610f0be9e0ef97f9a1a54e447e148a3 2024-05-15T22:56:22.841 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-15T22:56:23.909 INFO:teuthology.orchestra.run.smithi084.stdout:ceph: stdout ceph version 19.0.0-3728-g6cd0f801 (6cd0f8013e2dea00c3a29a0d8b10656b132d7c80) squid (dev) 2024-05-15T22:56:23.910 INFO:teuthology.orchestra.run.smithi084.stdout:Ceph version: ceph version 19.0.0-3728-g6cd0f801 (6cd0f8013e2dea00c3a29a0d8b10656b132d7c80) squid (dev) 2024-05-15T22:56:23.910 INFO:teuthology.orchestra.run.smithi084.stdout:Extracting ceph user uid/gid from container image... 2024-05-15T22:56:25.107 INFO:teuthology.orchestra.run.smithi084.stdout:stat: stdout 167 167 2024-05-15T22:56:25.108 INFO:teuthology.orchestra.run.smithi084.stdout:Creating initial keys... 2024-05-15T22:56:26.161 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph-authtool: stdout AQCZPUVmlFaQHxAAXvSk/FGpB8pRg8dydK8tGA== 2024-05-15T22:56:27.225 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph-authtool: stdout AQCaPUVmjZzbIBAA9C3QlMKiiJQJl8ErfcjzAA== 2024-05-15T22:56:28.174 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph-authtool: stdout AQCbPUVmo3IXJhAAOhrOPSDxGEdIOCBwyEpdzg== 2024-05-15T22:56:28.175 INFO:teuthology.orchestra.run.smithi084.stdout:Creating initial monmap... 2024-05-15T22:56:34.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-15T22:56:34.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-15T22:56:34.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:56:34.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-15T22:56:34.342 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool for a [v1:172.21.15.84:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-15T22:56:34.342 INFO:teuthology.orchestra.run.smithi084.stdout:setting min_mon_release = quincy 2024-05-15T22:56:34.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: set fsid to 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:56:34.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-15T22:56:34.343 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:56:34.343 INFO:teuthology.orchestra.run.smithi084.stdout:Creating mon... 2024-05-15T22:56:40.023 INFO:teuthology.orchestra.run.smithi084.stdout:create mon.a on 2024-05-15T22:56:40.153 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-05-15T22:56:40.243 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-15T22:56:40.366 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3999fb7e-130e-11ef-bc99-c7b262605968.target → /etc/systemd/system/ceph-3999fb7e-130e-11ef-bc99-c7b262605968.target. 2024-05-15T22:56:40.367 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-3999fb7e-130e-11ef-bc99-c7b262605968.target → /etc/systemd/system/ceph-3999fb7e-130e-11ef-bc99-c7b262605968.target. 2024-05-15T22:56:40.551 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mon.a 2024-05-15T22:56:40.552 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mon.a.service: Unit ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mon.a.service not loaded. 2024-05-15T22:56:40.641 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3999fb7e-130e-11ef-bc99-c7b262605968.target.wants/ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mon.a.service → /etc/systemd/system/ceph-3999fb7e-130e-11ef-bc99-c7b262605968@.service. 2024-05-15T22:56:41.247 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-15T22:56:41.247 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-05-15T22:56:41.254 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-15T22:56:41.254 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-05-15T22:56:41.254 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-05-15T22:56:41.254 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to enable service . firewalld.service is not available 2024-05-15T22:56:41.254 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mon to start... 2024-05-15T22:56:41.254 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mon... 2024-05-15T22:56:41.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:41 smithi084 bash[39261]: 22fbf6bed5d1aab9399ef0c34d87d20e40ec4ceb35ef76680cc717653b0e6c8c 2024-05-15T22:56:41.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:41 smithi084 systemd[1]: Started Ceph mon.a for 3999fb7e-130e-11ef-bc99-c7b262605968. 2024-05-15T22:56:42.401 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout cluster: 2024-05-15T22:56:42.401 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout id: 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:56:42.401 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-15T22:56:42.403 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-05-15T22:56:42.403 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout services: 2024-05-15T22:56:42.403 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.642151s) 2024-05-15T22:56:42.403 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-15T22:56:42.403 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-15T22:56:42.403 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-05-15T22:56:42.403 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout data: 2024-05-15T22:56:42.403 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-15T22:56:42.403 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-15T22:56:42.403 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-15T22:56:42.404 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout pgs: 2024-05-15T22:56:42.404 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-05-15T22:56:42.404 INFO:teuthology.orchestra.run.smithi084.stdout:mon is available 2024-05-15T22:56:42.404 INFO:teuthology.orchestra.run.smithi084.stdout:Assimilating anything we can from ceph.conf... 2024-05-15T22:56:42.404 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:42 smithi084 ceph-mon[39299]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-15T22:56:42.404 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:42 smithi084 ceph-mon[39299]: monmap epoch 1 2024-05-15T22:56:42.404 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:42 smithi084 ceph-mon[39299]: fsid 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:56:42.404 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:42 smithi084 ceph-mon[39299]: last_changed 2024-05-15T22:56:28.559279+0000 2024-05-15T22:56:42.404 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:42 smithi084 ceph-mon[39299]: created 2024-05-15T22:56:28.559279+0000 2024-05-15T22:56:42.404 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:42 smithi084 ceph-mon[39299]: min_mon_release 19 (squid) 2024-05-15T22:56:42.404 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:42 smithi084 ceph-mon[39299]: election_strategy: 1 2024-05-15T22:56:42.404 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:42 smithi084 ceph-mon[39299]: 0: v1:172.21.15.84:6789/0 mon.a 2024-05-15T22:56:42.405 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:42 smithi084 ceph-mon[39299]: fsmap 2024-05-15T22:56:42.405 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:42 smithi084 ceph-mon[39299]: osdmap e1: 0 total, 0 up, 0 in 2024-05-15T22:56:42.405 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:42 smithi084 ceph-mon[39299]: mgrmap e1: no daemons active 2024-05-15T22:56:42.405 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:42 smithi084 ceph-mon[39299]: from='client.? v1:172.21.15.84:0/4234696237' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-15T22:56:43.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:43 smithi084 ceph-mon[39299]: from='client.? v1:172.21.15.84:0/3336913313' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-15T22:56:43.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:43 smithi084 ceph-mon[39299]: from='client.? v1:172.21.15.84:0/3336913313' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-15T22:56:43.681 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-05-15T22:56:43.684 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [global] 2024-05-15T22:56:43.684 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout fsid = 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:56:43.684 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-15T22:56:43.684 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.84:6789] 2024-05-15T22:56:43.684 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-15T22:56:43.684 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-15T22:56:43.684 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-15T22:56:43.684 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-15T22:56:43.684 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-05-15T22:56:43.684 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-15T22:56:43.684 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-15T22:56:43.685 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-05-15T22:56:43.685 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [osd] 2024-05-15T22:56:43.685 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-15T22:56:43.685 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-15T22:56:43.685 INFO:teuthology.orchestra.run.smithi084.stdout:Generating new minimal ceph.conf... 2024-05-15T22:56:44.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:44 smithi084 ceph-mon[39299]: from='client.? v1:172.21.15.84:0/2525768137' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:56:44.771 INFO:teuthology.orchestra.run.smithi084.stdout:Restarting the monitor... 2024-05-15T22:56:45.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:44 smithi084 systemd[1]: Stopping Ceph mon.a for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T22:56:45.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:44 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39276]: 2024-05-15T22:56:44.992+0000 7f08f2149700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T22:56:45.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:44 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39276]: 2024-05-15T22:56:44.992+0000 7f08f2149700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-15T22:56:45.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:45 smithi084 bash[39623]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a 2024-05-15T22:56:46.356 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:45 smithi084 systemd[1]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mon.a.service: Succeeded. 2024-05-15T22:56:46.356 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:45 smithi084 systemd[1]: Stopped Ceph mon.a for 3999fb7e-130e-11ef-bc99-c7b262605968. 2024-05-15T22:56:46.356 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:45 smithi084 systemd[1]: Starting Ceph mon.a for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T22:56:46.553 INFO:teuthology.orchestra.run.smithi084.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-15T22:56:46.745 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: set uid:gid to 167:167 (ceph:ceph) 2024-05-15T22:56:46.746 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: ceph version 19.0.0-3728-g6cd0f801 (6cd0f8013e2dea00c3a29a0d8b10656b132d7c80) squid (dev), process ceph-mon, pid 7 2024-05-15T22:56:46.746 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: pidfile_write: ignore empty --pid-file 2024-05-15T22:56:46.746 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: load: jerasure load: lrc 2024-05-15T22:56:46.746 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: RocksDB version: 7.9.2 2024-05-15T22:56:46.746 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Git sha 0 2024-05-15T22:56:46.746 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Compile date 2024-05-15 18:21:30 2024-05-15T22:56:46.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: DB SUMMARY 2024-05-15T22:56:46.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: DB Session ID: J0XXOQFRAKVY0T1F0PQC 2024-05-15T22:56:46.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: CURRENT file: CURRENT 2024-05-15T22:56:46.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: IDENTITY file: IDENTITY 2024-05-15T22:56:46.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-15T22:56:46.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-15T22:56:46.748 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98293 ; 2024-05-15T22:56:46.748 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.error_if_exists: 0 2024-05-15T22:56:46.748 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.create_if_missing: 0 2024-05-15T22:56:46.748 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.paranoid_checks: 1 2024-05-15T22:56:46.748 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-15T22:56:46.748 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-15T22:56:46.749 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-15T22:56:46.749 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.env: 0x55923c435cc0 2024-05-15T22:56:46.749 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.fs: PosixFileSystem 2024-05-15T22:56:46.749 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.info_log: 0x55923e7afdc0 2024-05-15T22:56:46.749 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_file_opening_threads: 16 2024-05-15T22:56:46.749 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.statistics: (nil) 2024-05-15T22:56:46.750 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.use_fsync: 0 2024-05-15T22:56:46.750 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_log_file_size: 0 2024-05-15T22:56:46.750 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-15T22:56:46.750 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-15T22:56:46.750 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.keep_log_file_num: 1000 2024-05-15T22:56:46.751 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.recycle_log_file_num: 0 2024-05-15T22:56:46.751 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.allow_fallocate: 1 2024-05-15T22:56:46.751 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.allow_mmap_reads: 0 2024-05-15T22:56:46.751 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.allow_mmap_writes: 0 2024-05-15T22:56:46.751 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.use_direct_reads: 0 2024-05-15T22:56:46.752 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-15T22:56:46.752 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.create_missing_column_families: 0 2024-05-15T22:56:46.752 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.db_log_dir: 2024-05-15T22:56:46.752 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.wal_dir: 2024-05-15T22:56:46.752 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-15T22:56:46.752 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-15T22:56:46.753 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-15T22:56:46.753 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-15T22:56:46.753 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-15T22:56:46.753 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-15T22:56:46.753 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.advise_random_on_open: 1 2024-05-15T22:56:46.754 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.db_write_buffer_size: 0 2024-05-15T22:56:46.754 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.write_buffer_manager: 0x55923e7810e0 2024-05-15T22:56:46.754 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-15T22:56:46.754 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-15T22:56:46.754 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-15T22:56:46.754 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.rate_limiter: (nil) 2024-05-15T22:56:46.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-15T22:56:46.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.wal_recovery_mode: 2 2024-05-15T22:56:46.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.enable_thread_tracking: 0 2024-05-15T22:56:46.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.enable_pipelined_write: 0 2024-05-15T22:56:46.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.unordered_write: 0 2024-05-15T22:56:46.756 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-15T22:56:46.756 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-15T22:56:46.756 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-15T22:56:46.756 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-15T22:56:46.756 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.row_cache: None 2024-05-15T22:56:46.756 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.wal_filter: None 2024-05-15T22:56:46.757 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-15T22:56:46.757 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.allow_ingest_behind: 0 2024-05-15T22:56:46.757 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.two_write_queues: 0 2024-05-15T22:56:46.757 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.manual_wal_flush: 0 2024-05-15T22:56:46.757 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.wal_compression: 0 2024-05-15T22:56:46.757 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.atomic_flush: 0 2024-05-15T22:56:46.758 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-15T22:56:46.758 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-15T22:56:46.758 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-15T22:56:46.758 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.log_readahead_size: 0 2024-05-15T22:56:46.758 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-15T22:56:46.758 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.best_efforts_recovery: 0 2024-05-15T22:56:46.758 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-15T22:56:46.759 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-15T22:56:46.759 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.allow_data_in_errors: 0 2024-05-15T22:56:46.759 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.db_host_id: __hostname__ 2024-05-15T22:56:46.759 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-15T22:56:46.759 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_background_jobs: 2 2024-05-15T22:56:46.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_background_compactions: -1 2024-05-15T22:56:46.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_subcompactions: 1 2024-05-15T22:56:46.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-15T22:56:46.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-15T22:56:46.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-15T22:56:46.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_total_wal_size: 0 2024-05-15T22:56:46.761 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-15T22:56:46.761 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-15T22:56:46.761 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-15T22:56:46.761 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-15T22:56:46.761 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_open_files: -1 2024-05-15T22:56:46.762 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.bytes_per_sync: 0 2024-05-15T22:56:46.762 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-15T22:56:46.762 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-15T22:56:46.762 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compaction_readahead_size: 0 2024-05-15T22:56:46.762 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_background_flushes: -1 2024-05-15T22:56:46.762 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Compression algorithms supported: 2024-05-15T22:56:46.763 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-15T22:56:46.763 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: kZSTD supported: 0 2024-05-15T22:56:46.763 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: kXpressCompression supported: 0 2024-05-15T22:56:46.763 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: kLZ4HCCompression supported: 1 2024-05-15T22:56:46.763 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: kZlibCompression supported: 1 2024-05-15T22:56:46.763 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: kSnappyCompression supported: 1 2024-05-15T22:56:46.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: kLZ4Compression supported: 1 2024-05-15T22:56:46.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: kBZip2Compression supported: 0 2024-05-15T22:56:46.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-15T22:56:46.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-15T22:56:46.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-15T22:56:46.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-15T22:56:46.765 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-15T22:56:46.765 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.merge_operator: 2024-05-15T22:56:46.765 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compaction_filter: None 2024-05-15T22:56:46.765 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compaction_filter_factory: None 2024-05-15T22:56:46.765 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.sst_partitioner_factory: None 2024-05-15T22:56:46.765 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-15T22:56:46.765 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-15T22:56:46.766 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55923e7aff80) 2024-05-15T22:56:46.766 INFO:journalctl@ceph.mon.a.smithi084.stdout: cache_index_and_filter_blocks: 1 2024-05-15T22:56:46.766 INFO:journalctl@ceph.mon.a.smithi084.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-15T22:56:46.766 INFO:journalctl@ceph.mon.a.smithi084.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-15T22:56:46.766 INFO:journalctl@ceph.mon.a.smithi084.stdout: pin_top_level_index_and_filter: 1 2024-05-15T22:56:46.766 INFO:journalctl@ceph.mon.a.smithi084.stdout: index_type: 0 2024-05-15T22:56:46.766 INFO:journalctl@ceph.mon.a.smithi084.stdout: data_block_index_type: 0 2024-05-15T22:56:46.766 INFO:journalctl@ceph.mon.a.smithi084.stdout: index_shortening: 1 2024-05-15T22:56:46.767 INFO:journalctl@ceph.mon.a.smithi084.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-15T22:56:46.767 INFO:journalctl@ceph.mon.a.smithi084.stdout: checksum: 4 2024-05-15T22:56:46.767 INFO:journalctl@ceph.mon.a.smithi084.stdout: no_block_cache: 0 2024-05-15T22:56:46.767 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache: 0x55923e7b31f0 2024-05-15T22:56:46.767 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache_name: BinnedLRUCache 2024-05-15T22:56:46.767 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache_options: 2024-05-15T22:56:46.767 INFO:journalctl@ceph.mon.a.smithi084.stdout: capacity : 536870912 2024-05-15T22:56:46.768 INFO:journalctl@ceph.mon.a.smithi084.stdout: num_shard_bits : 4 2024-05-15T22:56:46.768 INFO:journalctl@ceph.mon.a.smithi084.stdout: strict_capacity_limit : 0 2024-05-15T22:56:46.768 INFO:journalctl@ceph.mon.a.smithi084.stdout: high_pri_pool_ratio: 0.000 2024-05-15T22:56:46.768 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache_compressed: (nil) 2024-05-15T22:56:46.768 INFO:journalctl@ceph.mon.a.smithi084.stdout: persistent_cache: (nil) 2024-05-15T22:56:46.768 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_size: 4096 2024-05-15T22:56:46.768 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_size_deviation: 10 2024-05-15T22:56:46.768 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_restart_interval: 16 2024-05-15T22:56:46.768 INFO:journalctl@ceph.mon.a.smithi084.stdout: index_block_restart_interval: 1 2024-05-15T22:56:46.769 INFO:journalctl@ceph.mon.a.smithi084.stdout: metadata_block_size: 4096 2024-05-15T22:56:46.769 INFO:journalctl@ceph.mon.a.smithi084.stdout: partition_filters: 0 2024-05-15T22:56:46.769 INFO:journalctl@ceph.mon.a.smithi084.stdout: use_delta_encoding: 1 2024-05-15T22:56:46.769 INFO:journalctl@ceph.mon.a.smithi084.stdout: filter_policy: bloomfilter 2024-05-15T22:56:46.769 INFO:journalctl@ceph.mon.a.smithi084.stdout: whole_key_filtering: 1 2024-05-15T22:56:46.769 INFO:journalctl@ceph.mon.a.smithi084.stdout: verify_compression: 0 2024-05-15T22:56:46.769 INFO:journalctl@ceph.mon.a.smithi084.stdout: read_amp_bytes_per_bit: 0 2024-05-15T22:56:46.769 INFO:journalctl@ceph.mon.a.smithi084.stdout: format_version: 5 2024-05-15T22:56:46.770 INFO:journalctl@ceph.mon.a.smithi084.stdout: enable_index_compression: 1 2024-05-15T22:56:46.770 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_align: 0 2024-05-15T22:56:46.770 INFO:journalctl@ceph.mon.a.smithi084.stdout: max_auto_readahead_size: 262144 2024-05-15T22:56:46.770 INFO:journalctl@ceph.mon.a.smithi084.stdout: prepopulate_block_cache: 0 2024-05-15T22:56:46.770 INFO:journalctl@ceph.mon.a.smithi084.stdout: initial_auto_readahead_size: 8192 2024-05-15T22:56:46.770 INFO:journalctl@ceph.mon.a.smithi084.stdout: num_file_reads_for_auto_readahead: 2 2024-05-15T22:56:46.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.write_buffer_size: 33554432 2024-05-15T22:56:46.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_write_buffer_number: 2 2024-05-15T22:56:46.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compression: NoCompression 2024-05-15T22:56:46.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.bottommost_compression: Disabled 2024-05-15T22:56:46.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.prefix_extractor: nullptr 2024-05-15T22:56:46.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-15T22:56:46.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.num_levels: 7 2024-05-15T22:56:46.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-15T22:56:46.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-15T22:56:46.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-15T22:56:46.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-15T22:56:46.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-15T22:56:46.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-15T22:56:46.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-15T22:56:46.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-15T22:56:46.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-15T22:56:46.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-15T22:56:46.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-15T22:56:46.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-15T22:56:46.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-15T22:56:46.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compression_opts.level: 32767 2024-05-15T22:56:46.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compression_opts.strategy: 0 2024-05-15T22:56:46.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-15T22:56:46.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-15T22:56:46.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-15T22:56:46.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-15T22:56:46.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compression_opts.enabled: false 2024-05-15T22:56:46.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-15T22:56:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-15T22:56:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-15T22:56:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-15T22:56:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.target_file_size_base: 67108864 2024-05-15T22:56:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-15T22:56:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-15T22:56:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-15T22:56:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-15T22:56:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-15T22:56:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-15T22:56:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-15T22:56:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-15T22:56:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-15T22:56:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-15T22:56:46.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-15T22:56:46.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-15T22:56:46.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-15T22:56:46.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-15T22:56:46.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.arena_block_size: 1048576 2024-05-15T22:56:46.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-15T22:56:46.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-15T22:56:46.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.disable_auto_compactions: 0 2024-05-15T22:56:46.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-15T22:56:46.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-15T22:56:46.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-15T22:56:46.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-15T22:56:46.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-15T22:56:46.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-15T22:56:46.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-15T22:56:46.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-15T22:56:46.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-15T22:56:46.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-15T22:56:46.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-15T22:56:46.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.inplace_update_support: 0 2024-05-15T22:56:46.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-15T22:56:46.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-15T22:56:46.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-15T22:56:46.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-15T22:56:46.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.bloom_locality: 0 2024-05-15T22:56:46.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.max_successive_merges: 0 2024-05-15T22:56:46.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-15T22:56:46.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.paranoid_file_checks: 0 2024-05-15T22:56:46.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.force_consistency_checks: 1 2024-05-15T22:56:46.776 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.report_bg_io_stats: 0 2024-05-15T22:56:46.776 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.ttl: 2592000 2024-05-15T22:56:46.776 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-15T22:56:46.776 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-15T22:56:46.776 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-15T22:56:46.776 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.enable_blob_files: false 2024-05-15T22:56:46.776 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.min_blob_size: 0 2024-05-15T22:56:46.776 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.blob_file_size: 268435456 2024-05-15T22:56:46.776 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-15T22:56:46.776 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-15T22:56:46.776 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-15T22:56:46.776 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-15T22:56:46.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-15T22:56:46.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.blob_file_starting_level: 0 2024-05-15T22:56:46.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-15T22:56:46.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-15T22:56:46.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-15T22:56:46.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d88b26be-ab5b-4a28-bf6a-26b836b69fc7 2024-05-15T22:56:46.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715813806493797, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-15T22:56:46.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-15T22:56:46.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715813806494784, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93679, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 306, "table_properties": {"data_size": 91680, "index_size": 255, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13221, "raw_average_key_size": 49, "raw_value_size": 84490, "raw_average_value_size": 318, "num_data_blocks": 11, "num_entries": 265, "num_filter_entries": 265, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715813806, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d88b26be-ab5b-4a28-bf6a-26b836b69fc7", "db_session_id": "J0XXOQFRAKVY0T1F0PQC", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-15T22:56:46.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715813806494861, "job": 1, "event": "recovery_finished"} 2024-05-15T22:56:46.778 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-15T22:56:46.778 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-15T22:56:46.778 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55923e8d0000 2024-05-15T22:56:46.778 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: DB pointer 0x55923e8bc000 2024-05-15T22:56:46.778 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-15T22:56:46.778 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-15T22:56:46.778 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** DB Stats ** 2024-05-15T22:56:46.778 INFO:journalctl@ceph.mon.a.smithi084.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-15T22:56:46.778 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-15T22:56:46.778 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-15T22:56:46.778 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-15T22:56:46.778 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-15T22:56:46.778 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-15T22:56:46.778 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-15T22:56:46.779 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-05-15T22:56:46.779 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** Compaction Stats [default] ** 2024-05-15T22:56:46.779 INFO:journalctl@ceph.mon.a.smithi084.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-15T22:56:46.779 INFO:journalctl@ceph.mon.a.smithi084.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-15T22:56:46.779 INFO:journalctl@ceph.mon.a.smithi084.stdout: L0 2/0 93.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-15T22:56:46.779 INFO:journalctl@ceph.mon.a.smithi084.stdout: Sum 2/0 93.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-15T22:56:46.779 INFO:journalctl@ceph.mon.a.smithi084.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-15T22:56:46.779 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-05-15T22:56:46.779 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** Compaction Stats [default] ** 2024-05-15T22:56:46.779 INFO:journalctl@ceph.mon.a.smithi084.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-15T22:56:46.779 INFO:journalctl@ceph.mon.a.smithi084.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-15T22:56:46.779 INFO:journalctl@ceph.mon.a.smithi084.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-15T22:56:46.779 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-05-15T22:56:46.779 INFO:journalctl@ceph.mon.a.smithi084.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-15T22:56:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-05-15T22:56:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-15T22:56:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-15T22:56:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-15T22:56:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-15T22:56:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-15T22:56:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-15T22:56:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative compaction: 0.00 GB write, 29.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-15T22:56:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval compaction: 0.00 GB write, 29.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-15T22:56:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-15T22:56:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout: Block cache BinnedLRUCache@0x55923e7b31f0#7 capacity: 512.00 MB usage: 1.33 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-05-15T22:56:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-05-15T22:56:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-05-15T22:56:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-15T22:56:46.781 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: starting mon.a rank 0 at public addrs v1:172.21.15.84:6789/0 at bind addrs v1:172.21.15.84:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:56:46.781 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: mon.a@-1(???) e1 preinit fsid 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:56:46.781 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: mon.a@-1(???).mds e1 new map 2024-05-15T22:56:46.781 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: mon.a@-1(???).mds e1 print_map 2024-05-15T22:56:46.781 INFO:journalctl@ceph.mon.a.smithi084.stdout: e1 2024-05-15T22:56:46.781 INFO:journalctl@ceph.mon.a.smithi084.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-15T22:56:46.781 INFO:journalctl@ceph.mon.a.smithi084.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-15T22:56:46.781 INFO:journalctl@ceph.mon.a.smithi084.stdout: legacy client fscid: -1 2024-05-15T22:56:46.781 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-05-15T22:56:46.781 INFO:journalctl@ceph.mon.a.smithi084.stdout: No filesystems configured 2024-05-15T22:56:46.781 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-15T22:56:46.781 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-15T22:56:46.781 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-15T22:56:46.781 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-15T22:56:46.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-15T22:56:46.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 bash[39747]: 1bdec6f208f170faa19283cfc47261e9574437b834e90d764d7b3ae50dbfb91f 2024-05-15T22:56:46.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 systemd[1]: Started Ceph mon.a for 3999fb7e-130e-11ef-bc99-c7b262605968. 2024-05-15T22:56:46.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-15T22:56:46.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: monmap epoch 1 2024-05-15T22:56:46.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: fsid 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:56:46.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: last_changed 2024-05-15T22:56:28.559279+0000 2024-05-15T22:56:46.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: created 2024-05-15T22:56:28.559279+0000 2024-05-15T22:56:46.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: min_mon_release 19 (squid) 2024-05-15T22:56:46.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: election_strategy: 1 2024-05-15T22:56:46.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: 0: v1:172.21.15.84:6789/0 mon.a 2024-05-15T22:56:46.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: fsmap 2024-05-15T22:56:46.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: osdmap e1: 0 total, 0 up, 0 in 2024-05-15T22:56:46.783 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:46 smithi084 ceph-mon[39784]: mgrmap e1: no daemons active 2024-05-15T22:56:47.697 INFO:teuthology.orchestra.run.smithi084.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-15T22:56:47.722 INFO:teuthology.orchestra.run.smithi084.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-15T22:56:47.722 INFO:teuthology.orchestra.run.smithi084.stdout:Creating mgr... 2024-05-15T22:56:47.723 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-15T22:56:47.723 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-15T22:56:47.904 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mgr.y 2024-05-15T22:56:47.904 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mgr.y.service: Unit ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mgr.y.service not loaded. 2024-05-15T22:56:47.994 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3999fb7e-130e-11ef-bc99-c7b262605968.target.wants/ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-3999fb7e-130e-11ef-bc99-c7b262605968@.service. 2024-05-15T22:56:48.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3438667946' entity='client.admin' 2024-05-15T22:56:48.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:48 smithi084 systemd[1]: Starting Ceph mgr.y for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T22:56:48.641 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-15T22:56:48.641 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-05-15T22:56:48.651 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-15T22:56:48.651 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-05-15T22:56:48.651 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-05-15T22:56:48.651 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to enable service . firewalld.service is not available 2024-05-15T22:56:48.660 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-15T22:56:48.661 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-05-15T22:56:48.669 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-15T22:56:48.669 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-05-15T22:56:48.670 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-05-15T22:56:48.670 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-15T22:56:48.670 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr to start... 2024-05-15T22:56:48.670 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr... 2024-05-15T22:56:48.898 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:48 smithi084 bash[40020]: f6de58af7be0d9faf9790d70e2821f0c4223b6d3fb93b57b9582e93444b685ff 2024-05-15T22:56:48.899 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:48 smithi084 systemd[1]: Started Ceph mgr.y for 3999fb7e-130e-11ef-bc99-c7b262605968. 2024-05-15T22:56:48.899 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:48 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:48.672+0000 7f6e5d3a5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T22:56:48.899 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:48 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:48.844+0000 7f6e5d3a5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T22:56:49.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:49 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2904956830' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-15T22:56:49.620 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:49 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:49.190+0000 7f6e5d3a5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T22:56:50.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:49 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:49.715+0000 7f6e5d3a5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T22:56:50.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:49 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:49.961+0000 7f6e5d3a5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T22:56:50.197 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-05-15T22:56:50.197 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-05-15T22:56:50.197 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsid": "3999fb7e-130e-11ef-bc99-c7b262605968", 2024-05-15T22:56:50.197 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "health": { 2024-05-15T22:56:50.197 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-15T22:56:50.197 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-15T22:56:50.197 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-15T22:56:50.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:50.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-15T22:56:50.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-15T22:56:50.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 0 2024-05-15T22:56:50.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-05-15T22:56:50.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-15T22:56:50.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "a" 2024-05-15T22:56:50.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-05-15T22:56:50.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-05-15T22:56:50.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-15T22:56:50.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T22:56:50.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-15T22:56:50.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-15T22:56:50.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:50.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-15T22:56:50.199 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T22:56:50.199 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-15T22:56:50.199 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-15T22:56:50.199 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-15T22:56:50.199 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-15T22:56:50.199 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-15T22:56:50.199 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-15T22:56:50.199 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:50.199 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-15T22:56:50.202 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-15T22:56:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-15T22:56:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-15T22:56:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-15T22:56:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-15T22:56:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-15T22:56:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-15T22:56:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-15T22:56:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-15T22:56:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T22:56:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-15T22:56:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-15T22:56:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-15T22:56:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-15T22:56:50.204 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-15T22:56:50.204 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-15T22:56:50.204 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-15T22:56:50.204 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-15T22:56:50.204 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "restful" 2024-05-15T22:56:50.204 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-05-15T22:56:50.204 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T22:56:50.204 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:50.204 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-15T22:56:50.204 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T22:56:50.204 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modified": "2024-05-15T22:56:41.212925+0000", 2024-05-15T22:56:50.205 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T22:56:50.205 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:50.205 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-15T22:56:50.205 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-05-15T22:56:50.205 INFO:teuthology.orchestra.run.smithi084.stdout:mgr not available, waiting (1/15)... 2024-05-15T22:56:50.620 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:50 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:50.396+0000 7f6e5d3a5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T22:56:51.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:51 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:51.108+0000 7f6e5d3a5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T22:56:51.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:51 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:51.194+0000 7f6e5d3a5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T22:56:52.120 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:51 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:51.787+0000 7f6e5d3a5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T22:56:52.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:51 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:51.870+0000 7f6e5d3a5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T22:56:52.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:51 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:51.957+0000 7f6e5d3a5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T22:56:52.463 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:52 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:52.181+0000 7f6e5d3a5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T22:56:52.463 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:52 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:52.280+0000 7f6e5d3a5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T22:56:52.463 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:52 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:52.376+0000 7f6e5d3a5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T22:56:53.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:52 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2790998361' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-15T22:56:53.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:52 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:52.618+0000 7f6e5d3a5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T22:56:53.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:52 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:52.720+0000 7f6e5d3a5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T22:56:53.530 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-05-15T22:56:53.530 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-05-15T22:56:53.530 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsid": "3999fb7e-130e-11ef-bc99-c7b262605968", 2024-05-15T22:56:53.530 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "health": { 2024-05-15T22:56:53.530 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-15T22:56:53.530 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-15T22:56:53.530 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-15T22:56:53.531 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:53.531 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-15T22:56:53.531 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-15T22:56:53.531 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 0 2024-05-15T22:56:53.531 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-05-15T22:56:53.531 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-15T22:56:53.531 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "a" 2024-05-15T22:56:53.531 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-05-15T22:56:53.531 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-05-15T22:56:53.531 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-15T22:56:53.531 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T22:56:53.531 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-15T22:56:53.532 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-15T22:56:53.532 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:53.532 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-15T22:56:53.532 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T22:56:53.532 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-15T22:56:53.532 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-15T22:56:53.532 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-15T22:56:53.532 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-15T22:56:53.532 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-15T22:56:53.532 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-15T22:56:53.532 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:53.532 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-15T22:56:53.533 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-15T22:56:53.533 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-15T22:56:53.533 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-15T22:56:53.533 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-15T22:56:53.533 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-15T22:56:53.533 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-15T22:56:53.533 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-15T22:56:53.533 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-15T22:56:53.533 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:53.533 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-15T22:56:53.533 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T22:56:53.533 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-15T22:56:53.534 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-15T22:56:53.534 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:53.534 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-15T22:56:53.534 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-15T22:56:53.534 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-15T22:56:53.534 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-15T22:56:53.535 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-15T22:56:53.535 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-15T22:56:53.535 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "restful" 2024-05-15T22:56:53.535 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-05-15T22:56:53.535 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T22:56:53.535 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:53.535 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-15T22:56:53.535 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:53 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:53.207+0000 7f6e5d3a5200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-15T22:56:53.535 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:53 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:53.305+0000 7f6e5d3a5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T22:56:53.536 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:53 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:53.390+0000 7f6e5d3a5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T22:56:53.536 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T22:56:53.536 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modified": "2024-05-15T22:56:41.212925+0000", 2024-05-15T22:56:53.536 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T22:56:53.536 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:53.537 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-15T22:56:53.537 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-05-15T22:56:53.537 INFO:teuthology.orchestra.run.smithi084.stdout:mgr not available, waiting (2/15)... 2024-05-15T22:56:54.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:53 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:53.978+0000 7f6e5d3a5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T22:56:54.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:54 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:54.190+0000 7f6e5d3a5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T22:56:54.870 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:54 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:54.586+0000 7f6e5d3a5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T22:56:55.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:55 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:55.036+0000 7f6e5d3a5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T22:56:55.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:55 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:55.236+0000 7f6e5d3a5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T22:56:55.781 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:55 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:55.419+0000 7f6e5d3a5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T22:56:55.782 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:56:55 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:56:55.536+0000 7f6e5d3a5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T22:56:55.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: Activating manager daemon y 2024-05-15T22:56:55.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: mgrmap e2: y(active, starting, since 0.00397458s) 2024-05-15T22:56:55.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: from='mgr.14100 v1:172.21.15.84:0/1596402300' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-15T22:56:55.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: from='mgr.14100 v1:172.21.15.84:0/1596402300' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-15T22:56:55.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: from='mgr.14100 v1:172.21.15.84:0/1596402300' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-15T22:56:55.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: from='mgr.14100 v1:172.21.15.84:0/1596402300' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-15T22:56:55.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: from='mgr.14100 v1:172.21.15.84:0/1596402300' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-15T22:56:55.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: from='mgr.14100 v1:172.21.15.84:0/1596402300' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-15T22:56:55.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: from='mgr.14100 v1:172.21.15.84:0/1596402300' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:56:55.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: from='mgr.14100 v1:172.21.15.84:0/1596402300' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-15T22:56:55.783 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: Manager daemon y is now available 2024-05-15T22:56:55.783 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: from='mgr.14100 v1:172.21.15.84:0/1596402300' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-15T22:56:55.783 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: from='mgr.14100 v1:172.21.15.84:0/1596402300' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-15T22:56:55.783 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: from='mgr.14100 v1:172.21.15.84:0/1596402300' entity='mgr.y' 2024-05-15T22:56:55.783 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: from='mgr.14100 v1:172.21.15.84:0/1596402300' entity='mgr.y' 2024-05-15T22:56:55.783 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:55 smithi084 ceph-mon[39784]: from='mgr.14100 v1:172.21.15.84:0/1596402300' entity='mgr.y' 2024-05-15T22:56:57.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:57 smithi084 ceph-mon[39784]: mgrmap e3: y(active, since 1.00641s) 2024-05-15T22:56:59.669 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:59 smithi084 ceph-mon[39784]: mgrmap e4: y(active, since 3s) 2024-05-15T22:56:59.669 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:56:59 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3081333713' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-15T22:56:59.955 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-05-15T22:56:59.959 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-05-15T22:56:59.959 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsid": "3999fb7e-130e-11ef-bc99-c7b262605968", 2024-05-15T22:56:59.959 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "health": { 2024-05-15T22:56:59.959 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-15T22:56:59.959 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-15T22:56:59.959 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-15T22:56:59.959 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:59.960 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-15T22:56:59.960 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-15T22:56:59.960 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 0 2024-05-15T22:56:59.960 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-05-15T22:56:59.960 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-15T22:56:59.960 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "a" 2024-05-15T22:56:59.960 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-05-15T22:56:59.960 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-05-15T22:56:59.960 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-15T22:56:59.960 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T22:56:59.960 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-15T22:56:59.960 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-15T22:56:59.960 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:59.960 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-15T22:56:59.960 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T22:56:59.961 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-15T22:56:59.961 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-15T22:56:59.961 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-15T22:56:59.961 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-15T22:56:59.961 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-15T22:56:59.961 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-15T22:56:59.964 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:59.964 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-15T22:56:59.964 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-15T22:56:59.964 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-15T22:56:59.964 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-15T22:56:59.964 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-15T22:56:59.964 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-15T22:56:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "restful" 2024-05-15T22:56:59.966 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-05-15T22:56:59.966 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T22:56:59.966 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:59.966 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-15T22:56:59.966 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T22:56:59.966 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modified": "2024-05-15T22:56:41.212925+0000", 2024-05-15T22:56:59.966 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T22:56:59.966 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-05-15T22:56:59.966 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-15T22:56:59.966 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-05-15T22:56:59.966 INFO:teuthology.orchestra.run.smithi084.stdout:mgr is available 2024-05-15T22:57:01.266 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:00 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3861301042' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-15T22:57:01.512 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-05-15T22:57:01.512 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [global] 2024-05-15T22:57:01.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout fsid = 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:57:01.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-15T22:57:01.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-15T22:57:01.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-15T22:57:01.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-15T22:57:01.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-15T22:57:01.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-05-15T22:57:01.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-15T22:57:01.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-15T22:57:01.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-05-15T22:57:01.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [osd] 2024-05-15T22:57:01.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-15T22:57:01.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-15T22:57:01.513 INFO:teuthology.orchestra.run.smithi084.stdout:Enabling cephadm module... 2024-05-15T22:57:02.666 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4188431254' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-15T22:57:02.666 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:02 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: ignoring --setuser ceph since I am not root 2024-05-15T22:57:02.666 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:02 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: ignoring --setgroup ceph since I am not root 2024-05-15T22:57:02.667 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:02 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:02.534+0000 7ff3c17e9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T22:57:03.048 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:02 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:02.706+0000 7ff3c17e9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T22:57:03.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:03 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:03.044+0000 7ff3c17e9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T22:57:03.710 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4188431254' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-15T22:57:03.711 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:03 smithi084 ceph-mon[39784]: mgrmap e5: y(active, since 6s) 2024-05-15T22:57:03.711 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:03 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:03.559+0000 7ff3c17e9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T22:57:04.037 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:03 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:03.811+0000 7ff3c17e9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T22:57:04.341 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-05-15T22:57:04.341 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-15T22:57:04.341 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-15T22:57:04.341 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-05-15T22:57:04.341 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-15T22:57:04.341 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-05-15T22:57:04.341 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for the mgr to restart... 2024-05-15T22:57:04.341 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr epoch 5... 2024-05-15T22:57:04.343 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:04 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:04.249+0000 7ff3c17e9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T22:57:04.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1798467562' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-15T22:57:05.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:04 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:04.993+0000 7ff3c17e9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T22:57:05.371 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:05 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:05.079+0000 7ff3c17e9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T22:57:06.086 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:05 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:05.688+0000 7ff3c17e9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T22:57:06.086 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:05 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:05.772+0000 7ff3c17e9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T22:57:06.086 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:05 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:05.860+0000 7ff3c17e9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T22:57:06.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:06 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:06.083+0000 7ff3c17e9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T22:57:06.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:06 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:06.174+0000 7ff3c17e9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T22:57:06.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:06 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:06.268+0000 7ff3c17e9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T22:57:06.870 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:06 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:06.498+0000 7ff3c17e9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T22:57:06.870 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:06 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:06.595+0000 7ff3c17e9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T22:57:07.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:07 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:07.067+0000 7ff3c17e9200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-15T22:57:07.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:07 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:07.164+0000 7ff3c17e9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T22:57:07.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:07 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:07.249+0000 7ff3c17e9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T22:57:08.120 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:07 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:07.832+0000 7ff3c17e9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T22:57:08.120 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:08 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:08.042+0000 7ff3c17e9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T22:57:08.870 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:08 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:08.443+0000 7ff3c17e9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T22:57:09.278 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:08 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:08.896+0000 7ff3c17e9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T22:57:09.278 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:09 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:09.093+0000 7ff3c17e9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T22:57:09.278 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:09 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:09.274+0000 7ff3c17e9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T22:57:09.620 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:09 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:09.389+0000 7ff3c17e9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T22:57:09.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:09 smithi084 ceph-mon[39784]: Active manager daemon y restarted 2024-05-15T22:57:09.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:09 smithi084 ceph-mon[39784]: Activating manager daemon y 2024-05-15T22:57:09.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:09 smithi084 ceph-mon[39784]: osdmap e2: 0 total, 0 up, 0 in 2024-05-15T22:57:09.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:09 smithi084 ceph-mon[39784]: mgrmap e6: y(active, starting, since 0.00329266s) 2024-05-15T22:57:09.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:09 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:57:09.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:09 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-15T22:57:09.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:09 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-15T22:57:09.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:09 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-15T22:57:09.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:09 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-15T22:57:12.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:12 smithi084 ceph-mon[39784]: Manager daemon y is now available 2024-05-15T22:57:12.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:12 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' 2024-05-15T22:57:12.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:12 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' 2024-05-15T22:57:12.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:12 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:57:12.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:12 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:57:12.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:12 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-15T22:57:13.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-05-15T22:57:13.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-15T22:57:13.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-15T22:57:13.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-05-15T22:57:13.080 INFO:teuthology.orchestra.run.smithi084.stdout:mgr epoch 5 is available 2024-05-15T22:57:13.080 INFO:teuthology.orchestra.run.smithi084.stdout:Setting orchestrator backend to cephadm... 2024-05-15T22:57:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:13 smithi084 ceph-mon[39784]: Found migration_current of "None". Setting to last migration. 2024-05-15T22:57:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:13 smithi084 ceph-mon[39784]: mgrmap e7: y(active, since 3s) 2024-05-15T22:57:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:13 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-15T22:57:13.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:13 smithi084 ceph-mon[39784]: from='client.14123 v1:172.21.15.84:0/2111914830' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-15T22:57:13.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:13 smithi084 ceph-mon[39784]: from='client.14123 v1:172.21.15.84:0/2111914830' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-15T22:57:13.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:13 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' 2024-05-15T22:57:13.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:13 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' 2024-05-15T22:57:13.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:13 smithi084 ceph-mon[39784]: [15/May/2024:22:57:12] ENGINE Bus STARTING 2024-05-15T22:57:13.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:13 smithi084 ceph-mon[39784]: [15/May/2024:22:57:13] ENGINE Serving on https://172.21.15.84:7150 2024-05-15T22:57:13.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:13 smithi084 ceph-mon[39784]: [15/May/2024:22:57:13] ENGINE Serving on http://172.21.15.84:8765 2024-05-15T22:57:13.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:13 smithi084 ceph-mon[39784]: [15/May/2024:22:57:13] ENGINE Bus STARTED 2024-05-15T22:57:13.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:13 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:57:15.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:14 smithi084 ceph-mon[39784]: mgrmap e8: y(active, since 4s) 2024-05-15T22:57:15.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:14 smithi084 ceph-mon[39784]: from='client.14132 v1:172.21.15.84:0/1102903675' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:57:15.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:14 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' 2024-05-15T22:57:15.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:14 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:57:15.826 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-15T22:57:15.826 INFO:teuthology.orchestra.run.smithi084.stdout:Generating ssh key... 2024-05-15T22:57:15.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:15 smithi084 ceph-mon[39784]: from='client.14134 v1:172.21.15.84:0/1621070735' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:57:17.120 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: Generating public/private rsa key pair. 2024-05-15T22:57:17.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: Your identification has been saved in /tmp/tmpakxg4wgr/key. 2024-05-15T22:57:17.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: Your public key has been saved in /tmp/tmpakxg4wgr/key.pub. 2024-05-15T22:57:17.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: The key fingerprint is: 2024-05-15T22:57:17.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: SHA256:KBXya0RMlqpvuo/wIJG6gwSDn9MqIoRjB23XiPos2MY ceph-3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:57:17.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: The key's randomart image is: 2024-05-15T22:57:17.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: +---[RSA 3072]----+ 2024-05-15T22:57:17.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: | .o=. | 2024-05-15T22:57:17.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: | =o. | 2024-05-15T22:57:17.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: |. . ..* | 2024-05-15T22:57:17.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: |+o +.= + | 2024-05-15T22:57:17.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: |=o++o + S | 2024-05-15T22:57:17.121 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: |+*=..o | 2024-05-15T22:57:17.122 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: |@+++ | 2024-05-15T22:57:17.122 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: |O*E+o | 2024-05-15T22:57:17.122 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: |++*=. | 2024-05-15T22:57:17.122 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: +----[SHA256]-----+ 2024-05-15T22:57:18.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:17 smithi084 ceph-mon[39784]: from='client.14136 v1:172.21.15.84:0/717773546' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:57:18.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:17 smithi084 ceph-mon[39784]: Generating ssh key... 2024-05-15T22:57:18.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:17 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' 2024-05-15T22:57:18.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:17 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' 2024-05-15T22:57:18.697 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8LmvnPoAUVI2y/qqcoixSY8c74tAX+ySJPhmmDxpiu9D9NTnFHNYyB/++AuCKXU3FwDVR4zBmqqq1i3Y/fj6tNphUL5wc4g46++8+iqpcR9ahAE/7Cs1JjR+2OY8E8qUPtghZEsygHX+bcqz6lWyU3vZsWiMmci511ZrH1SYkslXo8c1pJwxDMAUf+s3Ow3P6jTEIHx6as3Lhqrhk9765ZEcwSJVTFGZ+dx3eqCLuwRTDZULErHQMs+jaZkLVRPBhAzzSkusttkETLU64T0DbsLzICCTdvK53YN8r6z2k3NIJ2jiY42AhUZ7AxQ0EBK4PTyXaywNi8NW+CItAVqtybM39Ulq62zeKgwTdcHAM7Nb2NDoJiZgsNm2UygNIYr1lWfynVUiks37kwuyjyMRTA4lc3BRhg9/M5p65KO62BYTPEEmUL2tweSPLnCj6n5sn2k2Vw33WS5H63ykbtLEczP208K4zWxaaWHvPXemUowXZs9L/ONfsYiLjcpHZTJs= ceph-3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:57:18.697 INFO:teuthology.orchestra.run.smithi084.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-15T22:57:18.697 INFO:teuthology.orchestra.run.smithi084.stdout:Adding key to root@localhost authorized_keys... 2024-05-15T22:57:18.698 INFO:teuthology.orchestra.run.smithi084.stdout:Adding host smithi084... 2024-05-15T22:57:19.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:18 smithi084 ceph-mon[39784]: from='client.14138 v1:172.21.15.84:0/577064826' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:57:19.962 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:19 smithi084 ceph-mon[39784]: from='client.14140 v1:172.21.15.84:0/574066159' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi084", "addr": "172.21.15.84", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:57:21.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:20 smithi084 ceph-mon[39784]: Deploying cephadm binary to smithi084 2024-05-15T22:57:23.813 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout Added host 'smithi084' with addr '172.21.15.84' 2024-05-15T22:57:23.813 INFO:teuthology.orchestra.run.smithi084.stdout:Deploying unmanaged mon service... 2024-05-15T22:57:24.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:24 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' 2024-05-15T22:57:24.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:24 smithi084 ceph-mon[39784]: Added host smithi084 2024-05-15T22:57:24.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:24 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:57:25.434 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-15T22:57:25.434 INFO:teuthology.orchestra.run.smithi084.stdout:Deploying unmanaged mgr service... 2024-05-15T22:57:25.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:25 smithi084 ceph-mon[39784]: from='client.14142 v1:172.21.15.84:0/4218129567' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:57:25.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:25 smithi084 ceph-mon[39784]: Saving service mon spec with placement count:5 2024-05-15T22:57:25.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:25 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' 2024-05-15T22:57:27.051 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-15T22:57:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:27 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' 2024-05-15T22:57:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:27 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' 2024-05-15T22:57:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:27 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' 2024-05-15T22:57:28.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:28 smithi084 ceph-mon[39784]: from='client.14144 v1:172.21.15.84:0/2909323980' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:57:28.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:28 smithi084 ceph-mon[39784]: Saving service mgr spec with placement count:2 2024-05-15T22:57:28.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4181311946' entity='client.admin' 2024-05-15T22:57:29.832 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:29 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' 2024-05-15T22:57:29.832 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:29 smithi084 ceph-mon[39784]: from='mgr.14119 v1:172.21.15.84:0/2552384221' entity='mgr.y' 2024-05-15T22:57:30.143 INFO:teuthology.orchestra.run.smithi084.stdout:Enabling the dashboard module... 2024-05-15T22:57:30.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:30 smithi084 ceph-mon[39784]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:57:30.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:30 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/347471479' entity='client.admin' 2024-05-15T22:57:31.600 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2026270235' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-15T22:57:31.601 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:31 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: ignoring --setuser ceph since I am not root 2024-05-15T22:57:31.601 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:31 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: ignoring --setgroup ceph since I am not root 2024-05-15T22:57:31.601 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:31 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:31.597+0000 7fcef1f0a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T22:57:31.870 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:31 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:31.763+0000 7fcef1f0a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T22:57:32.210 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:32 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:32.098+0000 7fcef1f0a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T22:57:32.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:32 smithi084 ceph-mon[39784]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:57:32.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2026270235' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-15T22:57:32.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:32 smithi084 ceph-mon[39784]: mgrmap e9: y(active, since 22s) 2024-05-15T22:57:32.866 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:32 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:32.611+0000 7fcef1f0a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T22:57:33.120 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:32 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:32.863+0000 7fcef1f0a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T22:57:33.607 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-05-15T22:57:33.608 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-15T22:57:33.608 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-15T22:57:33.608 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-05-15T22:57:33.608 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-15T22:57:33.608 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-05-15T22:57:33.608 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for the mgr to restart... 2024-05-15T22:57:33.608 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr epoch 9... 2024-05-15T22:57:33.608 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:33 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1530943732' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-15T22:57:33.609 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:33 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:33.314+0000 7fcef1f0a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T22:57:34.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:34 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:34.056+0000 7fcef1f0a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T22:57:34.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:34 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:34.149+0000 7fcef1f0a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T22:57:35.120 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:34 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:34.748+0000 7fcef1f0a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T22:57:35.120 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:34 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:34.833+0000 7fcef1f0a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T22:57:35.120 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:34 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:34.921+0000 7fcef1f0a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T22:57:35.565 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:35 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:35.145+0000 7fcef1f0a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T22:57:35.566 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:35 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:35.238+0000 7fcef1f0a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T22:57:35.566 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:35 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:35.331+0000 7fcef1f0a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T22:57:35.870 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:35 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:35.563+0000 7fcef1f0a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T22:57:35.870 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:35 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:35.661+0000 7fcef1f0a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T22:57:36.620 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:36 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:36.136+0000 7fcef1f0a200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-15T22:57:36.620 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:36 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:36.234+0000 7fcef1f0a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T22:57:36.620 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:36 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:36.320+0000 7fcef1f0a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T22:57:37.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:36 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:36.908+0000 7fcef1f0a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T22:57:37.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:37 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:37.122+0000 7fcef1f0a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T22:57:37.870 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:37 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:37.528+0000 7fcef1f0a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T22:57:38.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:37 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:37.988+0000 7fcef1f0a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T22:57:38.370 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:38 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:38.189+0000 7fcef1f0a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T22:57:38.628 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:38 smithi084 ceph-mon[39784]: Active manager daemon y restarted 2024-05-15T22:57:38.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:38 smithi084 ceph-mon[39784]: Activating manager daemon y 2024-05-15T22:57:38.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:38 smithi084 ceph-mon[39784]: osdmap e3: 0 total, 0 up, 0 in 2024-05-15T22:57:38.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:38 smithi084 ceph-mon[39784]: mgrmap e10: y(active, starting, since 0.00396159s) 2024-05-15T22:57:38.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:57:38.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-15T22:57:38.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-15T22:57:38.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-15T22:57:38.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-15T22:57:38.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:38 smithi084 ceph-mon[39784]: Manager daemon y is now available 2024-05-15T22:57:38.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:57:38.630 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:38 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:38.375+0000 7fcef1f0a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T22:57:38.630 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:57:38 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:57:38.491+0000 7fcef1f0a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T22:57:39.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-15T22:57:39.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-15T22:57:39.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:39 smithi084 ceph-mon[39784]: [15/May/2024:22:57:39] ENGINE Bus STARTING 2024-05-15T22:57:39.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:39 smithi084 ceph-mon[39784]: [15/May/2024:22:57:39] ENGINE Serving on https://172.21.15.84:7150 2024-05-15T22:57:39.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:39 smithi084 ceph-mon[39784]: [15/May/2024:22:57:39] ENGINE Serving on http://172.21.15.84:8765 2024-05-15T22:57:39.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:39 smithi084 ceph-mon[39784]: [15/May/2024:22:57:39] ENGINE Bus STARTED 2024-05-15T22:57:39.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:39 smithi084 ceph-mon[39784]: mgrmap e11: y(active, since 1.0055s) 2024-05-15T22:57:40.266 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-05-15T22:57:40.266 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-15T22:57:40.266 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-15T22:57:40.267 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-05-15T22:57:40.267 INFO:teuthology.orchestra.run.smithi084.stdout:mgr epoch 9 is available 2024-05-15T22:57:40.267 INFO:teuthology.orchestra.run.smithi084.stdout:Generating a dashboard self-signed certificate... 2024-05-15T22:57:41.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:40 smithi084 ceph-mon[39784]: from='client.14156 v1:172.21.15.84:0/943872061' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-15T22:57:41.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:40 smithi084 ceph-mon[39784]: from='client.14156 v1:172.21.15.84:0/943872061' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-15T22:57:41.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:40 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:57:41.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:40 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:57:41.833 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-15T22:57:41.833 INFO:teuthology.orchestra.run.smithi084.stdout:Creating initial admin user... 2024-05-15T22:57:42.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:41 smithi084 ceph-mon[39784]: mgrmap e12: y(active, since 2s) 2024-05-15T22:57:42.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:41 smithi084 ceph-mon[39784]: from='client.14164 v1:172.21.15.84:0/1976504329' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:57:42.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:57:42.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:57:43.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:43 smithi084 ceph-mon[39784]: from='client.14166 v1:172.21.15.84:0/651893934' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:57:44.269 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$PXquKxJb1xtiqs8YWhNPOOsA0mkoG1EwmkTQAi/0I5rLEs0Lic4rC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1715813863, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-15T22:57:44.269 INFO:teuthology.orchestra.run.smithi084.stdout:Fetching dashboard port number... 2024-05-15T22:57:45.007 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:44 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:57:45.525 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 8443 2024-05-15T22:57:45.534 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-15T22:57:45.534 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-05-15T22:57:45.541 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-15T22:57:45.542 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-05-15T22:57:45.542 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-05-15T22:57:45.542 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-15T22:57:45.544 INFO:teuthology.orchestra.run.smithi084.stdout:Ceph Dashboard is now available at: 2024-05-15T22:57:45.544 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:57:45.544 INFO:teuthology.orchestra.run.smithi084.stdout: URL: https://smithi084.front.sepia.ceph.com:8443/ 2024-05-15T22:57:45.544 INFO:teuthology.orchestra.run.smithi084.stdout: User: admin 2024-05-15T22:57:45.544 INFO:teuthology.orchestra.run.smithi084.stdout: Password: txefhmue78 2024-05-15T22:57:45.544 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:57:45.544 INFO:teuthology.orchestra.run.smithi084.stdout:Saving cluster configuration to /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config directory 2024-05-15T22:57:45.544 INFO:teuthology.orchestra.run.smithi084.stdout:Enabling autotune for osd_memory_target 2024-05-15T22:57:46.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:45 smithi084 ceph-mon[39784]: mgrmap e13: y(active, since 6s) 2024-05-15T22:57:46.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1606658192' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-15T22:57:48.330 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-15T22:57:48.331 INFO:teuthology.orchestra.run.smithi084.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-15T22:57:48.331 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:57:48.331 INFO:teuthology.orchestra.run.smithi084.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-15T22:57:48.331 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:57:48.331 INFO:teuthology.orchestra.run.smithi084.stdout:Or, if you are only running a single cluster on this host: 2024-05-15T22:57:48.331 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:57:48.331 INFO:teuthology.orchestra.run.smithi084.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-15T22:57:48.331 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:57:48.331 INFO:teuthology.orchestra.run.smithi084.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-15T22:57:48.331 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:57:48.331 INFO:teuthology.orchestra.run.smithi084.stdout: ceph telemetry on 2024-05-15T22:57:48.331 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:57:48.331 INFO:teuthology.orchestra.run.smithi084.stdout:For more information see: 2024-05-15T22:57:48.331 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:57:48.331 INFO:teuthology.orchestra.run.smithi084.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-15T22:57:48.332 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:57:48.332 INFO:teuthology.orchestra.run.smithi084.stdout:Bootstrap complete. 2024-05-15T22:57:48.376 INFO:tasks.cephadm:Fetching config... 2024-05-15T22:57:48.377 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-05-15T22:57:48.377 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-15T22:57:48.394 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-15T22:57:48.394 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-05-15T22:57:48.395 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-15T22:57:48.452 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-15T22:57:48.453 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-05-15T22:57:48.453 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-15T22:57:48.520 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-15T22:57:48.520 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-05-15T22:57:48.520 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-15T22:57:48.578 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-15T22:57:48.578 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8LmvnPoAUVI2y/qqcoixSY8c74tAX+ySJPhmmDxpiu9D9NTnFHNYyB/++AuCKXU3FwDVR4zBmqqq1i3Y/fj6tNphUL5wc4g46++8+iqpcR9ahAE/7Cs1JjR+2OY8E8qUPtghZEsygHX+bcqz6lWyU3vZsWiMmci511ZrH1SYkslXo8c1pJwxDMAUf+s3Ow3P6jTEIHx6as3Lhqrhk9765ZEcwSJVTFGZ+dx3eqCLuwRTDZULErHQMs+jaZkLVRPBhAzzSkusttkETLU64T0DbsLzICCTdvK53YN8r6z2k3NIJ2jiY42AhUZ7AxQ0EBK4PTyXaywNi8NW+CItAVqtybM39Ulq62zeKgwTdcHAM7Nb2NDoJiZgsNm2UygNIYr1lWfynVUiks37kwuyjyMRTA4lc3BRhg9/M5p65KO62BYTPEEmUL2tweSPLnCj6n5sn2k2Vw33WS5H63ykbtLEczP208K4zWxaaWHvPXemUowXZs9L/ONfsYiLjcpHZTJs= ceph-3999fb7e-130e-11ef-bc99-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-15T22:57:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8LmvnPoAUVI2y/qqcoixSY8c74tAX+ySJPhmmDxpiu9D9NTnFHNYyB/++AuCKXU3FwDVR4zBmqqq1i3Y/fj6tNphUL5wc4g46++8+iqpcR9ahAE/7Cs1JjR+2OY8E8qUPtghZEsygHX+bcqz6lWyU3vZsWiMmci511ZrH1SYkslXo8c1pJwxDMAUf+s3Ow3P6jTEIHx6as3Lhqrhk9765ZEcwSJVTFGZ+dx3eqCLuwRTDZULErHQMs+jaZkLVRPBhAzzSkusttkETLU64T0DbsLzICCTdvK53YN8r6z2k3NIJ2jiY42AhUZ7AxQ0EBK4PTyXaywNi8NW+CItAVqtybM39Ulq62zeKgwTdcHAM7Nb2NDoJiZgsNm2UygNIYr1lWfynVUiks37kwuyjyMRTA4lc3BRhg9/M5p65KO62BYTPEEmUL2tweSPLnCj6n5sn2k2Vw33WS5H63ykbtLEczP208K4zWxaaWHvPXemUowXZs9L/ONfsYiLjcpHZTJs= ceph-3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:57:48.666 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8LmvnPoAUVI2y/qqcoixSY8c74tAX+ySJPhmmDxpiu9D9NTnFHNYyB/++AuCKXU3FwDVR4zBmqqq1i3Y/fj6tNphUL5wc4g46++8+iqpcR9ahAE/7Cs1JjR+2OY8E8qUPtghZEsygHX+bcqz6lWyU3vZsWiMmci511ZrH1SYkslXo8c1pJwxDMAUf+s3Ow3P6jTEIHx6as3Lhqrhk9765ZEcwSJVTFGZ+dx3eqCLuwRTDZULErHQMs+jaZkLVRPBhAzzSkusttkETLU64T0DbsLzICCTdvK53YN8r6z2k3NIJ2jiY42AhUZ7AxQ0EBK4PTyXaywNi8NW+CItAVqtybM39Ulq62zeKgwTdcHAM7Nb2NDoJiZgsNm2UygNIYr1lWfynVUiks37kwuyjyMRTA4lc3BRhg9/M5p65KO62BYTPEEmUL2tweSPLnCj6n5sn2k2Vw33WS5H63ykbtLEczP208K4zWxaaWHvPXemUowXZs9L/ONfsYiLjcpHZTJs= ceph-3999fb7e-130e-11ef-bc99-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-15T22:57:48.712 INFO:teuthology.orchestra.run.smithi088.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8LmvnPoAUVI2y/qqcoixSY8c74tAX+ySJPhmmDxpiu9D9NTnFHNYyB/++AuCKXU3FwDVR4zBmqqq1i3Y/fj6tNphUL5wc4g46++8+iqpcR9ahAE/7Cs1JjR+2OY8E8qUPtghZEsygHX+bcqz6lWyU3vZsWiMmci511ZrH1SYkslXo8c1pJwxDMAUf+s3Ow3P6jTEIHx6as3Lhqrhk9765ZEcwSJVTFGZ+dx3eqCLuwRTDZULErHQMs+jaZkLVRPBhAzzSkusttkETLU64T0DbsLzICCTdvK53YN8r6z2k3NIJ2jiY42AhUZ7AxQ0EBK4PTyXaywNi8NW+CItAVqtybM39Ulq62zeKgwTdcHAM7Nb2NDoJiZgsNm2UygNIYr1lWfynVUiks37kwuyjyMRTA4lc3BRhg9/M5p65KO62BYTPEEmUL2tweSPLnCj6n5sn2k2Vw33WS5H63ykbtLEczP208K4zWxaaWHvPXemUowXZs9L/ONfsYiLjcpHZTJs= ceph-3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:57:48.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8LmvnPoAUVI2y/qqcoixSY8c74tAX+ySJPhmmDxpiu9D9NTnFHNYyB/++AuCKXU3FwDVR4zBmqqq1i3Y/fj6tNphUL5wc4g46++8+iqpcR9ahAE/7Cs1JjR+2OY8E8qUPtghZEsygHX+bcqz6lWyU3vZsWiMmci511ZrH1SYkslXo8c1pJwxDMAUf+s3Ow3P6jTEIHx6as3Lhqrhk9765ZEcwSJVTFGZ+dx3eqCLuwRTDZULErHQMs+jaZkLVRPBhAzzSkusttkETLU64T0DbsLzICCTdvK53YN8r6z2k3NIJ2jiY42AhUZ7AxQ0EBK4PTyXaywNi8NW+CItAVqtybM39Ulq62zeKgwTdcHAM7Nb2NDoJiZgsNm2UygNIYr1lWfynVUiks37kwuyjyMRTA4lc3BRhg9/M5p65KO62BYTPEEmUL2tweSPLnCj6n5sn2k2Vw33WS5H63ykbtLEczP208K4zWxaaWHvPXemUowXZs9L/ONfsYiLjcpHZTJs= ceph-3999fb7e-130e-11ef-bc99-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-15T22:57:48.768 INFO:teuthology.orchestra.run.smithi090.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8LmvnPoAUVI2y/qqcoixSY8c74tAX+ySJPhmmDxpiu9D9NTnFHNYyB/++AuCKXU3FwDVR4zBmqqq1i3Y/fj6tNphUL5wc4g46++8+iqpcR9ahAE/7Cs1JjR+2OY8E8qUPtghZEsygHX+bcqz6lWyU3vZsWiMmci511ZrH1SYkslXo8c1pJwxDMAUf+s3Ow3P6jTEIHx6as3Lhqrhk9765ZEcwSJVTFGZ+dx3eqCLuwRTDZULErHQMs+jaZkLVRPBhAzzSkusttkETLU64T0DbsLzICCTdvK53YN8r6z2k3NIJ2jiY42AhUZ7AxQ0EBK4PTyXaywNi8NW+CItAVqtybM39Ulq62zeKgwTdcHAM7Nb2NDoJiZgsNm2UygNIYr1lWfynVUiks37kwuyjyMRTA4lc3BRhg9/M5p65KO62BYTPEEmUL2tweSPLnCj6n5sn2k2Vw33WS5H63ykbtLEczP208K4zWxaaWHvPXemUowXZs9L/ONfsYiLjcpHZTJs= ceph-3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:57:48.780 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-15T22:57:49.046 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T22:57:49.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/874987687' entity='client.admin' 2024-05-15T22:57:52.438 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-15T22:57:52.438 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-15T22:57:52.943 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T22:57:53.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:52 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1629682267' entity='client.admin' 2024-05-15T22:57:54.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:57:54.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:57:54.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi084", "name": "osd_memory_target"}]: dispatch 2024-05-15T22:57:54.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:57:54.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:57:54.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:57:55.580 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi088 2024-05-15T22:57:55.580 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T22:57:55.580 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/etc/ceph/ceph.conf 2024-05-15T22:57:55.601 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T22:57:55.602 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-15T22:57:55.662 INFO:tasks.cephadm:Adding host smithi088 to orchestrator... 2024-05-15T22:57:55.662 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch host add smithi088 2024-05-15T22:57:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:55 smithi084 ceph-mon[39784]: from='client.14176 v1:172.21.15.84:0/2892882006' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:57:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:57:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:57:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:57:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:57:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:55 smithi084 ceph-mon[39784]: Updating smithi084:/etc/ceph/ceph.conf 2024-05-15T22:57:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:55 smithi084 ceph-mon[39784]: Updating smithi084:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.conf 2024-05-15T22:57:55.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:55 smithi084 ceph-mon[39784]: Updating smithi084:/etc/ceph/ceph.client.admin.keyring 2024-05-15T22:57:55.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:55 smithi084 ceph-mon[39784]: Updating smithi084:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T22:57:55.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:57:55.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:57:55.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:57:55.925 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T22:57:58.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:58 smithi084 ceph-mon[39784]: from='client.14178 v1:172.21.15.84:0/206460929' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi088", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:57:58.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:58 smithi084 ceph-mon[39784]: Deploying cephadm binary to smithi088 2024-05-15T22:57:59.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:57:59 smithi084 ceph-mon[39784]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:01.614 INFO:teuthology.orchestra.run.smithi084.stdout:Added host 'smithi088' with addr '172.21.15.88' 2024-05-15T22:58:01.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:01 smithi084 ceph-mon[39784]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:02.283 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch host ls --format=json 2024-05-15T22:58:02.552 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T22:58:02.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:02.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:02 smithi084 ceph-mon[39784]: Added host smithi088 2024-05-15T22:58:02.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:58:02.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:02.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:04.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:04 smithi084 ceph-mon[39784]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:04.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:04 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:04.543 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:58:04.543 INFO:teuthology.orchestra.run.smithi084.stdout:[{"addr": "172.21.15.84", "hostname": "smithi084", "labels": [], "status": ""}, {"addr": "172.21.15.88", "hostname": "smithi088", "labels": [], "status": ""}] 2024-05-15T22:58:05.188 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi090 2024-05-15T22:58:05.188 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-05-15T22:58:05.188 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/etc/ceph/ceph.conf 2024-05-15T22:58:05.209 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-05-15T22:58:05.209 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-15T22:58:05.271 INFO:tasks.cephadm:Adding host smithi090 to orchestrator... 2024-05-15T22:58:05.271 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch host add smithi090 2024-05-15T22:58:05.534 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T22:58:06.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:06 smithi084 ceph-mon[39784]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:06.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:06 smithi084 ceph-mon[39784]: from='client.14180 v1:172.21.15.84:0/2176315714' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T22:58:07.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:07.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:07.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:07.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:07.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:07.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi088", "name": "osd_memory_target"}]: dispatch 2024-05-15T22:58:07.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:07.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:58:07.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: Updating smithi088:/etc/ceph/ceph.conf 2024-05-15T22:58:07.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: Updating smithi088:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.conf 2024-05-15T22:58:07.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2024-05-15T22:58:07.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: Updating smithi088:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T22:58:07.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: from='client.14182 v1:172.21.15.84:0/239099113' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi090", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:58:07.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:07.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:07.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:08.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:08 smithi084 ceph-mon[39784]: Deploying cephadm binary to smithi090 2024-05-15T22:58:09.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:09 smithi084 ceph-mon[39784]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:11.196 INFO:teuthology.orchestra.run.smithi084.stdout:Added host 'smithi090' with addr '172.21.15.90' 2024-05-15T22:58:11.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch host ls --format=json 2024-05-15T22:58:12.150 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T22:58:12.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:12 smithi084 ceph-mon[39784]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:12.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:12.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:12 smithi084 ceph-mon[39784]: Added host smithi090 2024-05-15T22:58:12.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:58:12.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:12.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:13 smithi084 ceph-mon[39784]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:13 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:13.917 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T22:58:13.917 INFO:teuthology.orchestra.run.smithi084.stdout:[{"addr": "172.21.15.84", "hostname": "smithi084", "labels": [], "status": ""}, {"addr": "172.21.15.88", "hostname": "smithi088", "labels": [], "status": ""}, {"addr": "172.21.15.90", "hostname": "smithi090", "labels": [], "status": ""}] 2024-05-15T22:58:14.535 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-15T22:58:14.535 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd crush tunables default 2024-05-15T22:58:14.799 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T22:58:14.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:14 smithi084 ceph-mon[39784]: from='client.14184 v1:172.21.15.84:0/384847702' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T22:58:15.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:15 smithi084 ceph-mon[39784]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:16.894 INFO:teuthology.orchestra.run.smithi084.stderr:adjusted tunables profile to default 2024-05-15T22:58:17.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:17.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:17.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:17.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:17.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi090", "name": "osd_memory_target"}]: dispatch 2024-05-15T22:58:17.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:17.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:58:17.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:16 smithi084 ceph-mon[39784]: Updating smithi090:/etc/ceph/ceph.conf 2024-05-15T22:58:17.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:16 smithi084 ceph-mon[39784]: Updating smithi090:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.conf 2024-05-15T22:58:17.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:16 smithi084 ceph-mon[39784]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-05-15T22:58:17.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:16 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3767890894' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-15T22:58:17.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:17.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:17.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:17.490 INFO:tasks.cephadm:Adding mon.a on smithi084 2024-05-15T22:58:17.491 INFO:tasks.cephadm:Adding mon.b on smithi088 2024-05-15T22:58:17.491 INFO:tasks.cephadm:Adding mon.c on smithi090 2024-05-15T22:58:17.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch apply mon '3;smithi084:[v1:172.21.15.84:6789]=a;smithi088:[v1:172.21.15.88:6789]=b;smithi090:[v1:172.21.15.90:6789]=c' 2024-05-15T22:58:17.772 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.conf 2024-05-15T22:58:18.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:17 smithi084 ceph-mon[39784]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:18.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:17 smithi084 ceph-mon[39784]: Updating smithi090:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T22:58:18.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:17 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3767890894' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-15T22:58:18.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:17 smithi084 ceph-mon[39784]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T22:58:19.550 INFO:teuthology.orchestra.run.smithi090.stdout:Scheduled mon update... 2024-05-15T22:58:20.187 DEBUG:teuthology.orchestra.run.smithi088:mon.b> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mon.b.service 2024-05-15T22:58:20.190 DEBUG:teuthology.orchestra.run.smithi090:mon.c> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mon.c.service 2024-05-15T22:58:20.192 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-15T22:58:20.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph mon dump -f json 2024-05-15T22:58:20.221 INFO:journalctl@ceph.mon.c.smithi090.stdout:-- Logs begin at Wed 2024-05-15 22:43:42 UTC. -- 2024-05-15T22:58:20.225 INFO:journalctl@ceph.mon.b.smithi088.stdout:-- Logs begin at Wed 2024-05-15 22:43:48 UTC. -- 2024-05-15T22:58:20.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:19 smithi084 ceph-mon[39784]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:20.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:20.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:58:20.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:20.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:58:20.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:20.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T22:58:20.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:21.275 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.conf 2024-05-15T22:58:21.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:20 smithi084 ceph-mon[39784]: from='client.14188 v1:172.21.15.90:0/2521005691' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi084:[v1:172.21.15.84:6789]=a;smithi088:[v1:172.21.15.88:6789]=b;smithi090:[v1:172.21.15.90:6789]=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:58:21.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:20 smithi084 ceph-mon[39784]: Saving service mon spec with placement smithi084:[v1:172.21.15.84:6789]=a;smithi088:[v1:172.21.15.88:6789]=b;smithi090:[v1:172.21.15.90:6789]=c;count:3 2024-05-15T22:58:21.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:20 smithi084 ceph-mon[39784]: Deploying daemon mon.c on smithi090 2024-05-15T22:58:22.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:21 smithi084 ceph-mon[39784]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:23 smithi084 ceph-mon[39784]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:24 smithi090 bash[44370]: 38a5bbd7cbc16c5b78b0b4ca76ed3f54c3924f016874bb0929e90174d4fab16a 2024-05-15T22:58:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:24 smithi090 systemd[1]: Started Ceph mon.c for 3999fb7e-130e-11ef-bc99-c7b262605968. 2024-05-15T22:58:29.144 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:58:29.144 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":2,"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","modified":"2024-05-15T22:58:24.095688Z","created":"2024-05-15T22:56:28.559279Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-15T22:58:29.147 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 2 2024-05-15T22:58:29.492 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:58:29.492 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: mon.a calling monitor election 2024-05-15T22:58:29.492 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:29.492 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.90:0/3389712351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T22:58:29.492 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:29.492 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:29.492 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:29.492 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: mon.c calling monitor election 2024-05-15T22:58:29.492 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:29.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:29.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:29.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:29.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-15T22:58:29.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: monmap epoch 2 2024-05-15T22:58:29.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: fsid 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:58:29.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: last_changed 2024-05-15T22:58:24.095688+0000 2024-05-15T22:58:29.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: created 2024-05-15T22:56:28.559279+0000 2024-05-15T22:58:29.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: min_mon_release 19 (squid) 2024-05-15T22:58:29.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: election_strategy: 1 2024-05-15T22:58:29.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: 0: v1:172.21.15.84:6789/0 mon.a 2024-05-15T22:58:29.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: 1: v1:172.21.15.90:6789/0 mon.c 2024-05-15T22:58:29.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: fsmap 2024-05-15T22:58:29.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T22:58:29.494 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: mgrmap e13: y(active, since 50s) 2024-05-15T22:58:29.494 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: overall HEALTH_OK 2024-05-15T22:58:29.494 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:29.494 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:29.494 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:29.494 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T22:58:29.494 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:29.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:58:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: mon.a calling monitor election 2024-05-15T22:58:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.90:0/3389712351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T22:58:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: mon.c calling monitor election 2024-05-15T22:58:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-15T22:58:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: monmap epoch 2 2024-05-15T22:58:29.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: fsid 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:58:29.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: last_changed 2024-05-15T22:58:24.095688+0000 2024-05-15T22:58:29.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: created 2024-05-15T22:56:28.559279+0000 2024-05-15T22:58:29.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: min_mon_release 19 (squid) 2024-05-15T22:58:29.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: election_strategy: 1 2024-05-15T22:58:29.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: 0: v1:172.21.15.84:6789/0 mon.a 2024-05-15T22:58:29.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: 1: v1:172.21.15.90:6789/0 mon.c 2024-05-15T22:58:29.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: fsmap 2024-05-15T22:58:29.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T22:58:29.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: mgrmap e13: y(active, since 50s) 2024-05-15T22:58:29.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: overall HEALTH_OK 2024-05-15T22:58:29.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:29.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:29.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:29.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T22:58:29.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:30 smithi090 ceph-mon[44434]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:30 smithi090 ceph-mon[44434]: Deploying daemon mon.b on smithi088 2024-05-15T22:58:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:30 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:30 smithi084 ceph-mon[39784]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:30 smithi084 ceph-mon[39784]: Deploying daemon mon.b on smithi088 2024-05-15T22:58:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:30 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:30.762 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-15T22:58:30.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph mon dump -f json 2024-05-15T22:58:31.034 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T22:58:32.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:32 smithi090 ceph-mon[44434]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:32.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:32 smithi084 ceph-mon[39784]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:58:32.836 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":2,"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","modified":"2024-05-15T22:58:24.095688Z","created":"2024-05-15T22:56:28.559279Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-15T22:58:32.838 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 2 2024-05-15T22:58:33.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:33 smithi088 bash[44661]: 04b863ea652bb1b53467b03a3f9dcb91a37764d34ee5b1e71d33ea288e1ead3b 2024-05-15T22:58:33.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:33 smithi088 systemd[1]: Started Ceph mon.b for 3999fb7e-130e-11ef-bc99-c7b262605968. 2024-05-15T22:58:34.454 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-15T22:58:34.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph mon dump -f json 2024-05-15T22:58:34.724 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T22:58:38.520 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-05-15T22:58:38.520 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":3,"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","modified":"2024-05-15T22:58:33.172648Z","created":"2024-05-15T22:56:28.559279Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-15T22:58:38.522 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 3 2024-05-15T22:58:38.544 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:38.544 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: mon.c calling monitor election 2024-05-15T22:58:38.544 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:58:38.544 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: mon.a calling monitor election 2024-05-15T22:58:38.544 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:38.544 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:38.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:38.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:38.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:38.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:38.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:38.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:38.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:38.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-15T22:58:38.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: monmap epoch 3 2024-05-15T22:58:38.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: fsid 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:58:38.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: last_changed 2024-05-15T22:58:33.172648+0000 2024-05-15T22:58:38.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: created 2024-05-15T22:56:28.559279+0000 2024-05-15T22:58:38.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: min_mon_release 19 (squid) 2024-05-15T22:58:38.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: election_strategy: 1 2024-05-15T22:58:38.545 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: 0: v1:172.21.15.84:6789/0 mon.a 2024-05-15T22:58:38.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: 1: v1:172.21.15.90:6789/0 mon.c 2024-05-15T22:58:38.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: 2: v1:172.21.15.88:6789/0 mon.b 2024-05-15T22:58:38.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: fsmap 2024-05-15T22:58:38.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T22:58:38.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: mgrmap e13: y(active, since 59s) 2024-05-15T22:58:38.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-05-15T22:58:38.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-05-15T22:58:38.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-05-15T22:58:38.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: mon.b (rank 2) addr v1:172.21.15.88:6789/0 is down (out of quorum) 2024-05-15T22:58:38.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:38.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:38.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:38.546 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:38.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:38.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: mon.c calling monitor election 2024-05-15T22:58:38.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:58:38.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: mon.a calling monitor election 2024-05-15T22:58:38.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:38.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:38.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:38.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:38.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:38.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:38.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:38.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:38.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:38.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-15T22:58:38.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: monmap epoch 3 2024-05-15T22:58:38.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: fsid 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:58:38.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: last_changed 2024-05-15T22:58:33.172648+0000 2024-05-15T22:58:38.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: created 2024-05-15T22:56:28.559279+0000 2024-05-15T22:58:38.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: min_mon_release 19 (squid) 2024-05-15T22:58:38.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: election_strategy: 1 2024-05-15T22:58:38.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: 0: v1:172.21.15.84:6789/0 mon.a 2024-05-15T22:58:38.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: 1: v1:172.21.15.90:6789/0 mon.c 2024-05-15T22:58:38.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: 2: v1:172.21.15.88:6789/0 mon.b 2024-05-15T22:58:38.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: fsmap 2024-05-15T22:58:38.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T22:58:38.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: mgrmap e13: y(active, since 59s) 2024-05-15T22:58:38.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-05-15T22:58:38.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-05-15T22:58:38.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-05-15T22:58:38.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: mon.b (rank 2) addr v1:172.21.15.88:6789/0 is down (out of quorum) 2024-05-15T22:58:38.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:38.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:38.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:38.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:39.236 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-15T22:58:39.236 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph config generate-minimal-conf 2024-05-15T22:58:39.512 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T22:58:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:58:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.90:0/2380733988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T22:58:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:39.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:58:39.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.90:0/2380733988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T22:58:39.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:39.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: mon.c calling monitor election 2024-05-15T22:58:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:58:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: mon.a calling monitor election 2024-05-15T22:58:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:58:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:40.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:40.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:40.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:40.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:40.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:40.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-15T22:58:40.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: monmap epoch 3 2024-05-15T22:58:40.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: fsid 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:58:40.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: last_changed 2024-05-15T22:58:33.172648+0000 2024-05-15T22:58:40.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: created 2024-05-15T22:56:28.559279+0000 2024-05-15T22:58:40.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: min_mon_release 19 (squid) 2024-05-15T22:58:40.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: election_strategy: 1 2024-05-15T22:58:40.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: 0: v1:172.21.15.84:6789/0 mon.a 2024-05-15T22:58:40.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: 1: v1:172.21.15.90:6789/0 mon.c 2024-05-15T22:58:40.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: 2: v1:172.21.15.88:6789/0 mon.b 2024-05-15T22:58:40.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: fsmap 2024-05-15T22:58:40.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T22:58:40.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: mgrmap e13: y(active, since 59s) 2024-05-15T22:58:40.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-05-15T22:58:40.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-05-15T22:58:40.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-05-15T22:58:40.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: mon.b (rank 2) addr v1:172.21.15.88:6789/0 is down (out of quorum) 2024-05-15T22:58:40.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:40.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:40.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:40.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:40.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:58:40.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.90:0/2380733988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T22:58:40.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:40.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:41.083 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: mon.b calling monitor election 2024-05-15T22:58:41.083 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: Updating smithi084:/etc/ceph/ceph.conf 2024-05-15T22:58:41.083 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: Updating smithi088:/etc/ceph/ceph.conf 2024-05-15T22:58:41.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: Updating smithi090:/etc/ceph/ceph.conf 2024-05-15T22:58:41.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: mon.b calling monitor election 2024-05-15T22:58:41.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: mon.a calling monitor election 2024-05-15T22:58:41.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: mon.c calling monitor election 2024-05-15T22:58:41.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-15T22:58:41.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: monmap epoch 3 2024-05-15T22:58:41.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: fsid 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:58:41.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: last_changed 2024-05-15T22:58:33.172648+0000 2024-05-15T22:58:41.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: created 2024-05-15T22:56:28.559279+0000 2024-05-15T22:58:41.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: min_mon_release 19 (squid) 2024-05-15T22:58:41.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: election_strategy: 1 2024-05-15T22:58:41.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: 0: v1:172.21.15.84:6789/0 mon.a 2024-05-15T22:58:41.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: 1: v1:172.21.15.90:6789/0 mon.c 2024-05-15T22:58:41.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: 2: v1:172.21.15.88:6789/0 mon.b 2024-05-15T22:58:41.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: fsmap 2024-05-15T22:58:41.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T22:58:41.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: mgrmap e13: y(active, since 61s) 2024-05-15T22:58:41.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-05-15T22:58:41.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T22:58:41.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: overall HEALTH_OK 2024-05-15T22:58:41.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: Updating smithi084:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.conf 2024-05-15T22:58:41.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: Updating smithi088:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.conf 2024-05-15T22:58:41.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: Updating smithi090:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.conf 2024-05-15T22:58:41.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T22:58:41.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T22:58:41.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: mon.b calling monitor election 2024-05-15T22:58:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: Updating smithi084:/etc/ceph/ceph.conf 2024-05-15T22:58:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: Updating smithi088:/etc/ceph/ceph.conf 2024-05-15T22:58:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: Updating smithi090:/etc/ceph/ceph.conf 2024-05-15T22:58:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: mon.b calling monitor election 2024-05-15T22:58:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: mon.a calling monitor election 2024-05-15T22:58:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: mon.c calling monitor election 2024-05-15T22:58:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-15T22:58:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: monmap epoch 3 2024-05-15T22:58:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: fsid 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:58:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: last_changed 2024-05-15T22:58:33.172648+0000 2024-05-15T22:58:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: created 2024-05-15T22:56:28.559279+0000 2024-05-15T22:58:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: min_mon_release 19 (squid) 2024-05-15T22:58:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: election_strategy: 1 2024-05-15T22:58:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: 0: v1:172.21.15.84:6789/0 mon.a 2024-05-15T22:58:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: 1: v1:172.21.15.90:6789/0 mon.c 2024-05-15T22:58:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: 2: v1:172.21.15.88:6789/0 mon.b 2024-05-15T22:58:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: fsmap 2024-05-15T22:58:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T22:58:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: mgrmap e13: y(active, since 61s) 2024-05-15T22:58:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-05-15T22:58:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T22:58:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: overall HEALTH_OK 2024-05-15T22:58:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: Updating smithi084:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.conf 2024-05-15T22:58:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: Updating smithi088:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.conf 2024-05-15T22:58:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: Updating smithi090:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.conf 2024-05-15T22:58:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T22:58:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T22:58:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: mon.b calling monitor election 2024-05-15T22:58:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: Updating smithi084:/etc/ceph/ceph.conf 2024-05-15T22:58:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: Updating smithi088:/etc/ceph/ceph.conf 2024-05-15T22:58:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: Updating smithi090:/etc/ceph/ceph.conf 2024-05-15T22:58:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: mon.b calling monitor election 2024-05-15T22:58:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: mon.a calling monitor election 2024-05-15T22:58:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: mon.c calling monitor election 2024-05-15T22:58:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-15T22:58:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: monmap epoch 3 2024-05-15T22:58:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: fsid 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:58:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: last_changed 2024-05-15T22:58:33.172648+0000 2024-05-15T22:58:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: created 2024-05-15T22:56:28.559279+0000 2024-05-15T22:58:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: min_mon_release 19 (squid) 2024-05-15T22:58:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: election_strategy: 1 2024-05-15T22:58:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: 0: v1:172.21.15.84:6789/0 mon.a 2024-05-15T22:58:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: 1: v1:172.21.15.90:6789/0 mon.c 2024-05-15T22:58:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: 2: v1:172.21.15.88:6789/0 mon.b 2024-05-15T22:58:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: fsmap 2024-05-15T22:58:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T22:58:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: mgrmap e13: y(active, since 61s) 2024-05-15T22:58:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-05-15T22:58:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T22:58:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: overall HEALTH_OK 2024-05-15T22:58:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: Updating smithi084:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.conf 2024-05-15T22:58:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: Updating smithi088:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.conf 2024-05-15T22:58:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: Updating smithi090:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/config/ceph.conf 2024-05-15T22:58:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:41.326 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T22:58:41.326 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T22:58:41.326 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:41.544 INFO:teuthology.orchestra.run.smithi084.stdout:# minimal ceph.conf for 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:58:41.544 INFO:teuthology.orchestra.run.smithi084.stdout:[global] 2024-05-15T22:58:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: fsid = 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T22:58:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: mon_host = 172.21.15.84:6789/0 172.21.15.88:6789/0 172.21.15.90:6789/0 2024-05-15T22:58:42.227 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-15T22:58:42.227 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-05-15T22:58:42.227 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.conf 2024-05-15T22:58:42.257 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-05-15T22:58:42.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-15T22:58:42.324 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T22:58:42.324 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.conf 2024-05-15T22:58:42.360 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T22:58:42.360 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-15T22:58:42.431 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-05-15T22:58:42.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.conf 2024-05-15T22:58:42.468 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-05-15T22:58:42.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-15T22:58:42.540 INFO:tasks.cephadm:Adding mgr.y on smithi084 2024-05-15T22:58:42.540 INFO:tasks.cephadm:Adding mgr.x on smithi088 2024-05-15T22:58:42.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch apply mgr '2;smithi084=y;smithi088=x' 2024-05-15T22:58:42.607 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:42 smithi090 ceph-mon[44434]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:42.607 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:42 smithi090 ceph-mon[44434]: Reconfiguring mon.a (unknown last config time)... 2024-05-15T22:58:42.607 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:42 smithi090 ceph-mon[44434]: Reconfiguring daemon mon.a on smithi084 2024-05-15T22:58:42.607 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:42 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:42.607 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:42 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2988079547' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:42.620 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 22:58:42 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T22:58:42.373+0000 7fcebd0fb700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-15T22:58:42.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:42 smithi084 ceph-mon[39784]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:42.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:42 smithi084 ceph-mon[39784]: Reconfiguring mon.a (unknown last config time)... 2024-05-15T22:58:42.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:42 smithi084 ceph-mon[39784]: Reconfiguring daemon mon.a on smithi084 2024-05-15T22:58:42.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:42 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:42.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:42 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2988079547' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:42.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:42 smithi088 ceph-mon[44699]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:42.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:42 smithi088 ceph-mon[44699]: Reconfiguring mon.a (unknown last config time)... 2024-05-15T22:58:42.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:42 smithi088 ceph-mon[44699]: Reconfiguring daemon mon.a on smithi084 2024-05-15T22:58:42.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:42 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:58:42.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:42 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2988079547' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:42.839 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T22:58:44.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:44 smithi090 ceph-mon[44434]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:44.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:44 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:44.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:44 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:44.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:44 smithi090 ceph-mon[44434]: Reconfiguring mon.b (monmap changed)... 2024-05-15T22:58:44.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:44 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T22:58:44.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:44 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T22:58:44.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:44 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:44.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:44 smithi090 ceph-mon[44434]: Reconfiguring daemon mon.b on smithi088 2024-05-15T22:58:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:44 smithi088 ceph-mon[44699]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:44 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:44 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:44 smithi088 ceph-mon[44699]: Reconfiguring mon.b (monmap changed)... 2024-05-15T22:58:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:44 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T22:58:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:44 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T22:58:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:44 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:44 smithi088 ceph-mon[44699]: Reconfiguring daemon mon.b on smithi088 2024-05-15T22:58:44.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:44 smithi084 ceph-mon[39784]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:44.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:44 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:44.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:44 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:44.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:44 smithi084 ceph-mon[39784]: Reconfiguring mon.b (monmap changed)... 2024-05-15T22:58:44.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:44 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T22:58:44.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:44 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T22:58:44.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:44 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:44.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:44 smithi084 ceph-mon[39784]: Reconfiguring daemon mon.b on smithi088 2024-05-15T22:58:44.623 INFO:teuthology.orchestra.run.smithi090.stdout:Scheduled mgr update... 2024-05-15T22:58:45.183 DEBUG:teuthology.orchestra.run.smithi088:mgr.x> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mgr.x.service 2024-05-15T22:58:45.185 INFO:tasks.cephadm:Deploying OSDs... 2024-05-15T22:58:45.185 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-05-15T22:58:45.185 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T22:58:45.208 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-15T22:58:45.208 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_1 2024-05-15T22:58:45.215 INFO:journalctl@ceph.mgr.x.smithi088.stdout:-- Logs begin at Wed 2024-05-15 22:43:48 UTC. -- 2024-05-15T22:58:45.269 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-15T22:58:45.269 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T22:58:45.269 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 171404 Links: 1 2024-05-15T22:58:45.269 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T22:58:45.269 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T22:58:45.269 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-05-15 22:57:52.057016340 +0000 2024-05-15T22:58:45.269 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-05-15 22:49:46.152242124 +0000 2024-05-15T22:58:45.269 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-05-15 22:49:46.152242124 +0000 2024-05-15T22:58:45.269 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-05-15T22:58:45.270 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-15T22:58:45.338 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-05-15T22:58:45.339 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-05-15T22:58:45.339 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000191951 s, 2.7 MB/s 2024-05-15T22:58:45.340 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-15T22:58:45.399 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_2 2024-05-15T22:58:45.458 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-15T22:58:45.459 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T22:58:45.459 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 168188 Links: 1 2024-05-15T22:58:45.459 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T22:58:45.459 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T22:58:45.459 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-05-15 22:57:52.095015583 +0000 2024-05-15T22:58:45.459 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-05-15 22:49:46.494235420 +0000 2024-05-15T22:58:45.459 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-05-15 22:49:46.494235420 +0000 2024-05-15T22:58:45.459 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-05-15T22:58:45.459 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-15T22:58:45.526 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-05-15T22:58:45.526 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-05-15T22:58:45.526 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000216177 s, 2.4 MB/s 2024-05-15T22:58:45.527 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-15T22:58:45.586 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_3 2024-05-15T22:58:45.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:45 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:45.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:45 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:45.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:45 smithi084 ceph-mon[39784]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:45.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:45 smithi084 ceph-mon[39784]: Reconfiguring mon.c (monmap changed)... 2024-05-15T22:58:45.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:45 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T22:58:45.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:45 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T22:58:45.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:45 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:45.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:45 smithi084 ceph-mon[39784]: Reconfiguring daemon mon.c on smithi090 2024-05-15T22:58:45.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:45 smithi084 ceph-mon[39784]: from='client.24103 v1:172.21.15.90:0/3925942214' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi084=y;smithi088=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:58:45.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:45 smithi084 ceph-mon[39784]: Saving service mgr spec with placement smithi084=y;smithi088=x;count:2 2024-05-15T22:58:45.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:45 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:45.638 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-15T22:58:45.638 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T22:58:45.638 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 171550 Links: 1 2024-05-15T22:58:45.638 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T22:58:45.638 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T22:58:45.639 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-05-15 22:57:52.135014785 +0000 2024-05-15T22:58:45.639 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-05-15 22:49:46.857228305 +0000 2024-05-15T22:58:45.639 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-05-15 22:49:46.857228305 +0000 2024-05-15T22:58:45.639 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-05-15T22:58:45.639 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-15T22:58:45.704 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-05-15T22:58:45.704 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-05-15T22:58:45.705 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000213321 s, 2.4 MB/s 2024-05-15T22:58:45.705 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-15T22:58:45.764 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_4 2024-05-15T22:58:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:45 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:45 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:45 smithi090 ceph-mon[44434]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:45 smithi090 ceph-mon[44434]: Reconfiguring mon.c (monmap changed)... 2024-05-15T22:58:45.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:45 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T22:58:45.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:45 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T22:58:45.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:45 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:45.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:45 smithi090 ceph-mon[44434]: Reconfiguring daemon mon.c on smithi090 2024-05-15T22:58:45.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:45 smithi090 ceph-mon[44434]: from='client.24103 v1:172.21.15.90:0/3925942214' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi084=y;smithi088=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:58:45.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:45 smithi090 ceph-mon[44434]: Saving service mgr spec with placement smithi084=y;smithi088=x;count:2 2024-05-15T22:58:45.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:45 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:45.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:45 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:45.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:45 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:45.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:45 smithi088 ceph-mon[44699]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:45.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:45 smithi088 ceph-mon[44699]: Reconfiguring mon.c (monmap changed)... 2024-05-15T22:58:45.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:45 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T22:58:45.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:45 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T22:58:45.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:45 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:45.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:45 smithi088 ceph-mon[44699]: Reconfiguring daemon mon.c on smithi090 2024-05-15T22:58:45.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:45 smithi088 ceph-mon[44699]: from='client.24103 v1:172.21.15.90:0/3925942214' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi084=y;smithi088=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:58:45.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:45 smithi088 ceph-mon[44699]: Saving service mgr spec with placement smithi084=y;smithi088=x;count:2 2024-05-15T22:58:45.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:45 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:45.824 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-15T22:58:45.824 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T22:58:45.824 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 163800 Links: 1 2024-05-15T22:58:45.824 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T22:58:45.824 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T22:58:45.824 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-05-15 22:57:52.177013948 +0000 2024-05-15T22:58:45.824 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-05-15 22:49:47.209221405 +0000 2024-05-15T22:58:45.824 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-05-15 22:49:47.209221405 +0000 2024-05-15T22:58:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-05-15T22:58:45.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-15T22:58:45.890 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-05-15T22:58:45.890 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-05-15T22:58:45.890 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000226155 s, 2.3 MB/s 2024-05-15T22:58:45.891 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-15T22:58:45.950 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T22:58:45.951 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T22:58:45.973 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-15T22:58:45.973 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2024-05-15T22:58:46.035 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-15T22:58:46.036 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T22:58:46.036 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 171249 Links: 1 2024-05-15T22:58:46.036 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T22:58:46.036 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T22:58:46.036 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-05-15 22:58:32.577241178 +0000 2024-05-15T22:58:46.036 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-05-15 22:49:49.197267037 +0000 2024-05-15T22:58:46.036 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-05-15 22:49:49.197267037 +0000 2024-05-15T22:58:46.036 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2024-05-15T22:58:46.036 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-15T22:58:46.106 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-05-15T22:58:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-05-15T22:58:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000192197 s, 2.7 MB/s 2024-05-15T22:58:46.108 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-15T22:58:46.169 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2024-05-15T22:58:46.229 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-15T22:58:46.229 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T22:58:46.229 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 167385 Links: 1 2024-05-15T22:58:46.229 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T22:58:46.229 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T22:58:46.229 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-05-15 22:58:32.577241178 +0000 2024-05-15T22:58:46.229 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-05-15 22:49:49.565259658 +0000 2024-05-15T22:58:46.229 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-05-15 22:49:49.565259658 +0000 2024-05-15T22:58:46.230 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2024-05-15T22:58:46.230 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-15T22:58:46.298 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-05-15T22:58:46.298 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-05-15T22:58:46.298 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000249879 s, 2.0 MB/s 2024-05-15T22:58:46.299 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-15T22:58:46.359 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2024-05-15T22:58:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-15T22:58:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T22:58:46.418 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 167464 Links: 1 2024-05-15T22:58:46.418 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T22:58:46.418 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T22:58:46.418 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-05-15 22:58:32.577241178 +0000 2024-05-15T22:58:46.418 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-05-15 22:49:49.951251918 +0000 2024-05-15T22:58:46.418 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-05-15 22:49:49.951251918 +0000 2024-05-15T22:58:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2024-05-15T22:58:46.419 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-15T22:58:46.485 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-05-15T22:58:46.485 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-05-15T22:58:46.485 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.00020736 s, 2.5 MB/s 2024-05-15T22:58:46.486 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-15T22:58:46.545 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2024-05-15T22:58:46.603 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-15T22:58:46.604 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T22:58:46.604 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 172261 Links: 1 2024-05-15T22:58:46.604 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T22:58:46.604 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T22:58:46.604 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-05-15 22:58:32.577241178 +0000 2024-05-15T22:58:46.604 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-05-15 22:49:50.293245059 +0000 2024-05-15T22:58:46.604 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-05-15 22:49:50.293245059 +0000 2024-05-15T22:58:46.604 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2024-05-15T22:58:46.604 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-15T22:58:46.669 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-05-15T22:58:46.669 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-05-15T22:58:46.669 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000229062 s, 2.2 MB/s 2024-05-15T22:58:46.670 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-15T22:58:46.728 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-05-15T22:58:46.728 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T22:58:46.749 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-15T22:58:46.749 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2024-05-15T22:58:46.809 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-15T22:58:46.809 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T22:58:46.809 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 171290 Links: 1 2024-05-15T22:58:46.809 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T22:58:46.809 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T22:58:46.809 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-05-15 22:58:23.045413049 +0000 2024-05-15T22:58:46.809 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-05-15 22:49:47.691363885 +0000 2024-05-15T22:58:46.809 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-05-15 22:49:47.691363885 +0000 2024-05-15T22:58:46.809 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2024-05-15T22:58:46.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-15T22:58:46.880 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-05-15T22:58:46.880 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-05-15T22:58:46.880 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000213591 s, 2.4 MB/s 2024-05-15T22:58:46.882 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-15T22:58:46.942 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2024-05-15T22:58:47.002 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-15T22:58:47.060 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T22:58:47.060 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 172152 Links: 1 2024-05-15T22:58:47.061 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T22:58:47.061 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T22:58:47.061 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-05-15 22:58:23.045413049 +0000 2024-05-15T22:58:47.061 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-05-15 22:49:48.027358860 +0000 2024-05-15T22:58:47.061 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-05-15 22:49:48.027358860 +0000 2024-05-15T22:58:47.061 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2024-05-15T22:58:47.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-15T22:58:47.088 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-05-15T22:58:47.088 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-05-15T22:58:47.088 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000196188 s, 2.6 MB/s 2024-05-15T22:58:47.089 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-15T22:58:47.149 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2024-05-15T22:58:47.207 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-15T22:58:47.208 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T22:58:47.208 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 169190 Links: 1 2024-05-15T22:58:47.208 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T22:58:47.208 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T22:58:47.208 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-05-15 22:58:23.045413049 +0000 2024-05-15T22:58:47.208 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-05-15 22:49:48.381353566 +0000 2024-05-15T22:58:47.209 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-05-15 22:49:48.381353566 +0000 2024-05-15T22:58:47.209 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2024-05-15T22:58:47.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-15T22:58:47.273 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-05-15T22:58:47.273 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-05-15T22:58:47.273 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000186656 s, 2.7 MB/s 2024-05-15T22:58:47.275 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-15T22:58:47.334 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2024-05-15T22:58:47.392 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-15T22:58:47.392 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T22:58:47.392 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 169244 Links: 1 2024-05-15T22:58:47.393 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T22:58:47.393 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T22:58:47.393 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-05-15 22:58:23.045413049 +0000 2024-05-15T22:58:47.393 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-05-15 22:49:48.713348601 +0000 2024-05-15T22:58:47.393 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-05-15 22:49:48.713348601 +0000 2024-05-15T22:58:47.393 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2024-05-15T22:58:47.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-15T22:58:47.458 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-05-15T22:58:47.458 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-05-15T22:58:47.458 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.00023273 s, 2.2 MB/s 2024-05-15T22:58:47.459 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-15T22:58:47.511 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:47 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:47.511 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:47 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:47.511 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:47 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:58:47.512 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:47 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:47.512 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:47 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:58:47.512 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:47 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:47.512 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:47 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T22:58:47.512 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:47 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-15T22:58:47.512 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:47 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T22:58:47.512 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:47 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:47.518 INFO:tasks.cephadm:Deploying osd.0 on smithi084 with /dev/vg_nvme/lv_4... 2024-05-15T22:58:47.518 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-15T22:58:47.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:47 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:47.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:47 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:47.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:47 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:58:47.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:47 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:47.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:47 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:58:47.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:47 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:47.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:47 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T22:58:47.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:47 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-15T22:58:47.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:47 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T22:58:47.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:47 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:47 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:47 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:47 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:58:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:47 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:47 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:58:47.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:47 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:47.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:47 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T22:58:47.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:47 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-15T22:58:47.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:47 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T22:58:47.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:47 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:47.786 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T22:58:48.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:48 smithi090 ceph-mon[44434]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:48.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:48 smithi090 ceph-mon[44434]: Deploying daemon mgr.x on smithi088 2024-05-15T22:58:48.557 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:48 smithi084 ceph-mon[39784]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:48.557 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:48 smithi084 ceph-mon[39784]: Deploying daemon mgr.x on smithi088 2024-05-15T22:58:48.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:48 smithi088 ceph-mon[44699]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:48.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:48 smithi088 ceph-mon[44699]: Deploying daemon mgr.x on smithi088 2024-05-15T22:58:48.572 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:48 smithi088 systemd[1]: Starting Ceph mgr.x for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T22:58:49.322 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:49 smithi088 bash[46303]: 7c95469af8e578163f9ea0b76a26d7580d30ce3f2d8784e2d7c44a9436068cba 2024-05-15T22:58:49.322 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:49 smithi088 systemd[1]: Started Ceph mgr.x for 3999fb7e-130e-11ef-bc99-c7b262605968. 2024-05-15T22:58:49.322 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:49 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:49.095+0000 7ff2b6c01200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T22:58:49.322 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:49 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:49.265+0000 7ff2b6c01200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T22:58:49.754 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-15T22:58:49.754 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-15T22:58:49.754 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10+0 records in 2024-05-15T22:58:49.755 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records out 2024-05-15T22:58:49.755 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111645 s, 939 MB/s 2024-05-15T22:58:49.755 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping successful for: 2024-05-15T22:58:50.050 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:49 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:49.610+0000 7ff2b6c01200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T22:58:50.322 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:50 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:50.149+0000 7ff2b6c01200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T22:58:50.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:50 smithi088 ceph-mon[44699]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:50.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:50 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:50.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:50 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:50.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:50 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:50.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:50 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:50.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:50 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:58:50.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:50 smithi084 ceph-mon[39784]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:50.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:50 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:50.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:50 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:50.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:50 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:50.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:50 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:50.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:50 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:58:50.411 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch daemon add osd smithi084:vg_nvme/lv_4 2024-05-15T22:58:50.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:50 smithi090 ceph-mon[44434]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:50.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:50 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:50.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:50 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:50.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:50 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:50.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:50 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:50.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:50 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:58:50.720 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T22:58:50.822 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:50 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:50.402+0000 7ff2b6c01200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T22:58:51.321 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:50 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:50.842+0000 7ff2b6c01200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T22:58:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:58:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T22:58:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T22:58:51.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:58:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:51.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T22:58:51.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T22:58:51.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:51.822 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:51 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:51.559+0000 7ff2b6c01200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T22:58:51.822 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:51 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:51.645+0000 7ff2b6c01200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T22:58:51.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:51.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:51.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:51.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:58:51.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:51.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T22:58:51.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T22:58:51.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:52.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:52 smithi088 ceph-mon[44699]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:52.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:52 smithi088 ceph-mon[44699]: Reconfiguring mgr.y (unknown last config time)... 2024-05-15T22:58:52.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:52 smithi088 ceph-mon[44699]: Reconfiguring daemon mgr.y on smithi084 2024-05-15T22:58:52.573 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:52 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:52.241+0000 7ff2b6c01200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T22:58:52.573 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:52 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:52.326+0000 7ff2b6c01200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T22:58:52.573 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:52 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:52.414+0000 7ff2b6c01200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T22:58:52.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:52 smithi084 ceph-mon[39784]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:52.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:52 smithi084 ceph-mon[39784]: Reconfiguring mgr.y (unknown last config time)... 2024-05-15T22:58:52.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:52 smithi084 ceph-mon[39784]: Reconfiguring daemon mgr.y on smithi084 2024-05-15T22:58:52.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:52 smithi090 ceph-mon[44434]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:52.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:52 smithi090 ceph-mon[44434]: Reconfiguring mgr.y (unknown last config time)... 2024-05-15T22:58:52.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:52 smithi090 ceph-mon[44434]: Reconfiguring daemon mgr.y on smithi084 2024-05-15T22:58:53.061 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:52 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:52.640+0000 7ff2b6c01200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T22:58:53.061 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:52 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:52.733+0000 7ff2b6c01200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T22:58:53.061 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:52 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:52.827+0000 7ff2b6c01200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T22:58:53.321 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:53 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:53.058+0000 7ff2b6c01200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T22:58:53.321 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:53 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:53.154+0000 7ff2b6c01200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T22:58:53.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T22:58:53.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T22:58:53.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:53.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:53.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:53.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:58:53.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:53.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:58:53.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:53.728 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:53 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:53.629+0000 7ff2b6c01200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-15T22:58:53.729 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T22:58:53.729 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T22:58:53.729 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:53.729 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:53.729 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:53.729 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:58:53.729 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:53.729 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:58:53.729 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:53.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T22:58:53.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T22:58:53.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:53.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:53.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:53.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:58:53.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:58:53.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:58:53.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:54.071 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:53 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:53.725+0000 7ff2b6c01200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T22:58:54.072 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:53 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:53.811+0000 7ff2b6c01200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T22:58:54.721 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:54 smithi084 ceph-mon[39784]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:54.721 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:54 smithi084 ceph-mon[39784]: from='client.14223 v1:172.21.15.84:0/4193134319' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:58:54.721 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:54 smithi090 ceph-mon[44434]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:54 smithi090 ceph-mon[44434]: from='client.14223 v1:172.21.15.84:0/4193134319' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:58:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:54.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:54 smithi088 ceph-mon[44699]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:54 smithi088 ceph-mon[44699]: from='client.14223 v1:172.21.15.84:0/4193134319' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:58:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:58:54.822 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:54 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:54.397+0000 7ff2b6c01200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T22:58:54.822 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:54 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:54.607+0000 7ff2b6c01200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T22:58:55.322 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:55 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:55.008+0000 7ff2b6c01200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T22:58:55.821 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:55 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:55.463+0000 7ff2b6c01200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T22:58:55.822 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:55 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:55.662+0000 7ff2b6c01200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T22:58:56.322 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:55 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:55.845+0000 7ff2b6c01200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T22:58:56.322 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 22:58:55 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x[46318]: 2024-05-15T22:58:55.959+0000 7ff2b6c01200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T22:58:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:56 smithi090 ceph-mon[44434]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/539982314' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80159cc8-1b6c-4aa6-a114-966786d7be1e"}]: dispatch 2024-05-15T22:58:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/539982314' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80159cc8-1b6c-4aa6-a114-966786d7be1e"}]': finished 2024-05-15T22:58:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:56 smithi090 ceph-mon[44434]: osdmap e5: 1 total, 0 up, 1 in 2024-05-15T22:58:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:56 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:58:56.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:56 smithi090 ceph-mon[44434]: Standby manager daemon x started 2024-05-15T22:58:56.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:56 smithi090 ceph-mon[44434]: from='mgr.? v1:172.21.15.88:0/2645130310' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-15T22:58:56.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:56 smithi090 ceph-mon[44434]: from='mgr.? v1:172.21.15.88:0/2645130310' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-15T22:58:56.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:56 smithi090 ceph-mon[44434]: from='mgr.? v1:172.21.15.88:0/2645130310' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-15T22:58:56.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:56 smithi090 ceph-mon[44434]: from='mgr.? v1:172.21.15.88:0/2645130310' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-15T22:58:56.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1067708091' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T22:58:56.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:56 smithi088 ceph-mon[44699]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:56.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/539982314' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80159cc8-1b6c-4aa6-a114-966786d7be1e"}]: dispatch 2024-05-15T22:58:56.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/539982314' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80159cc8-1b6c-4aa6-a114-966786d7be1e"}]': finished 2024-05-15T22:58:56.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:56 smithi088 ceph-mon[44699]: osdmap e5: 1 total, 0 up, 1 in 2024-05-15T22:58:56.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:56 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:58:56.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:56 smithi088 ceph-mon[44699]: Standby manager daemon x started 2024-05-15T22:58:56.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:56 smithi088 ceph-mon[44699]: from='mgr.? v1:172.21.15.88:0/2645130310' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-15T22:58:56.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:56 smithi088 ceph-mon[44699]: from='mgr.? v1:172.21.15.88:0/2645130310' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-15T22:58:56.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:56 smithi088 ceph-mon[44699]: from='mgr.? v1:172.21.15.88:0/2645130310' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-15T22:58:56.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:56 smithi088 ceph-mon[44699]: from='mgr.? v1:172.21.15.88:0/2645130310' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-15T22:58:56.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1067708091' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T22:58:56.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:56 smithi084 ceph-mon[39784]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:56.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/539982314' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80159cc8-1b6c-4aa6-a114-966786d7be1e"}]: dispatch 2024-05-15T22:58:56.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/539982314' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80159cc8-1b6c-4aa6-a114-966786d7be1e"}]': finished 2024-05-15T22:58:56.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:56 smithi084 ceph-mon[39784]: osdmap e5: 1 total, 0 up, 1 in 2024-05-15T22:58:56.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:56 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:58:56.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:56 smithi084 ceph-mon[39784]: Standby manager daemon x started 2024-05-15T22:58:56.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:56 smithi084 ceph-mon[39784]: from='mgr.? v1:172.21.15.88:0/2645130310' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-15T22:58:56.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:56 smithi084 ceph-mon[39784]: from='mgr.? v1:172.21.15.88:0/2645130310' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-15T22:58:56.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:56 smithi084 ceph-mon[39784]: from='mgr.? v1:172.21.15.88:0/2645130310' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-15T22:58:56.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:56 smithi084 ceph-mon[39784]: from='mgr.? v1:172.21.15.88:0/2645130310' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-15T22:58:56.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1067708091' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T22:58:57.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:57 smithi084 ceph-mon[39784]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:57.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:57 smithi084 ceph-mon[39784]: mgrmap e14: y(active, since 78s), standbys: x 2024-05-15T22:58:57.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-15T22:58:58.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:57 smithi090 ceph-mon[44434]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:58.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:57 smithi090 ceph-mon[44434]: mgrmap e14: y(active, since 78s), standbys: x 2024-05-15T22:58:58.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-15T22:58:58.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:57 smithi088 ceph-mon[44699]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:58:58.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:57 smithi088 ceph-mon[44699]: mgrmap e14: y(active, since 78s), standbys: x 2024-05-15T22:58:58.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-15T22:58:59.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:58:59 smithi084 ceph-mon[39784]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:00.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:58:59 smithi090 ceph-mon[44434]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:00.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:58:59 smithi088 ceph-mon[44699]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:01.709 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:01 smithi084 ceph-mon[39784]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:02.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:01 smithi090 ceph-mon[44434]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:02.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:01 smithi088 ceph-mon[44699]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:02.768 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-15T22:59:02.769 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:02.769 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:02 smithi084 ceph-mon[39784]: Deploying daemon osd.0 on smithi084 2024-05-15T22:59:03.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:02 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-15T22:59:03.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:02 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:03.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:02 smithi090 ceph-mon[44434]: Deploying daemon osd.0 on smithi084 2024-05-15T22:59:03.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:02 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-15T22:59:03.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:02 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:03.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:02 smithi088 ceph-mon[44699]: Deploying daemon osd.0 on smithi084 2024-05-15T22:59:03.785 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:03 smithi084 ceph-mon[39784]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:04.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:03 smithi090 ceph-mon[44434]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:04.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:03 smithi088 ceph-mon[44699]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:05.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:05 smithi084 ceph-mon[39784]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:06.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:05 smithi090 ceph-mon[44434]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:06.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:05 smithi088 ceph-mon[44699]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:08.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:08 smithi090 ceph-mon[44434]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:08.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:08 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:08.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:08 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:08.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:08 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:59:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:08 smithi088 ceph-mon[44699]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:08 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:08 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:08.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:08 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:59:08.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:08 smithi084 ceph-mon[39784]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:08.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:08 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:08.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:08 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:08.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:08 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:59:10.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:10 smithi090 ceph-mon[44434]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:10.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:10 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:10.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:10 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:10.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:10 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:10.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:10 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:59:10.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:10 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:10.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:10 smithi088 ceph-mon[44699]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:10.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:10 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:10.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:10 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:10.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:10 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:10.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:10 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:59:10.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:10 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:10.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:10 smithi084 ceph-mon[39784]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:10.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:10 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:10.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:10 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:10.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:10 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:10.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:10 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:59:10.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:10 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:10.912 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 0 on host 'smithi084' 2024-05-15T22:59:11.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:11 smithi090 ceph-mon[44434]: from='osd.0 v1:172.21.15.84:6801/1084680531' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-15T22:59:11.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:11 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:11.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:11 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:59:11.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:11 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:11 smithi088 ceph-mon[44699]: from='osd.0 v1:172.21.15.84:6801/1084680531' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-15T22:59:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:11 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:11.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:11 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:59:11.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:11 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:11.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:11 smithi084 ceph-mon[39784]: from='osd.0 v1:172.21.15.84:6801/1084680531' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-15T22:59:11.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:11 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:11.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:11 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:59:11.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:11 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:11.653 DEBUG:teuthology.orchestra.run.smithi084:osd.0> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.0.service 2024-05-15T22:59:11.655 INFO:tasks.cephadm:Deploying osd.1 on smithi084 with /dev/vg_nvme/lv_3... 2024-05-15T22:59:11.656 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-15T22:59:11.686 INFO:journalctl@ceph.osd.0.smithi084.stdout:-- Logs begin at Wed 2024-05-15 22:43:24 UTC. -- 2024-05-15T22:59:12.433 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T22:59:12.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:12 smithi084 ceph-mon[39784]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:12.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:12 smithi084 ceph-mon[39784]: from='osd.0 v1:172.21.15.84:6801/1084680531' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-15T22:59:12.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:12 smithi084 ceph-mon[39784]: osdmap e6: 1 total, 0 up, 1 in 2024-05-15T22:59:12.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:12 smithi084 ceph-mon[39784]: from='osd.0 v1:172.21.15.84:6801/1084680531' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T22:59:12.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:59:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:12 smithi090 ceph-mon[44434]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:12 smithi090 ceph-mon[44434]: from='osd.0 v1:172.21.15.84:6801/1084680531' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-15T22:59:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:12 smithi090 ceph-mon[44434]: osdmap e6: 1 total, 0 up, 1 in 2024-05-15T22:59:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:12 smithi090 ceph-mon[44434]: from='osd.0 v1:172.21.15.84:6801/1084680531' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T22:59:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:59:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:12 smithi088 ceph-mon[44699]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:12 smithi088 ceph-mon[44699]: from='osd.0 v1:172.21.15.84:6801/1084680531' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-15T22:59:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:12 smithi088 ceph-mon[44699]: osdmap e6: 1 total, 0 up, 1 in 2024-05-15T22:59:12.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:12 smithi088 ceph-mon[44699]: from='osd.0 v1:172.21.15.84:6801/1084680531' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T22:59:12.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:59:12.870 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 22:59:12 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-0[57754]: 2024-05-15T22:59:12.748+0000 7f19c7236700 -1 osd.0 0 waiting for initial osdmap 2024-05-15T22:59:12.870 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 22:59:12 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-0[57754]: 2024-05-15T22:59:12.750+0000 7f19be820700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T22:59:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:13 smithi084 ceph-mon[39784]: from='osd.0 v1:172.21.15.84:6801/1084680531' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-05-15T22:59:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:13 smithi084 ceph-mon[39784]: osdmap e7: 1 total, 0 up, 1 in 2024-05-15T22:59:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:13 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:59:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:13 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:59:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:13 smithi084 ceph-mon[39784]: from='osd.0 v1:172.21.15.84:6801/1084680531' entity='osd.0' 2024-05-15T22:59:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:13 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:59:13.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:13 smithi090 ceph-mon[44434]: from='osd.0 v1:172.21.15.84:6801/1084680531' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-05-15T22:59:13.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:13 smithi090 ceph-mon[44434]: osdmap e7: 1 total, 0 up, 1 in 2024-05-15T22:59:13.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:13 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:59:13.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:13 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:59:13.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:13 smithi090 ceph-mon[44434]: from='osd.0 v1:172.21.15.84:6801/1084680531' entity='osd.0' 2024-05-15T22:59:13.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:13 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:59:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:13 smithi088 ceph-mon[44699]: from='osd.0 v1:172.21.15.84:6801/1084680531' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-05-15T22:59:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:13 smithi088 ceph-mon[44699]: osdmap e7: 1 total, 0 up, 1 in 2024-05-15T22:59:13.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:13 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:59:13.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:13 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:59:13.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:13 smithi088 ceph-mon[44699]: from='osd.0 v1:172.21.15.84:6801/1084680531' entity='osd.0' 2024-05-15T22:59:13.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:13 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:59:14.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:14 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T22:59:14.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:14 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T22:59:14.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T22:59:14.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:14 smithi084 ceph-mon[39784]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:14.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T22:59:14.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:14 smithi084 ceph-mon[39784]: osd.0 v1:172.21.15.84:6801/1084680531 boot 2024-05-15T22:59:14.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:14 smithi084 ceph-mon[39784]: osdmap e8: 1 total, 1 up, 1 in 2024-05-15T22:59:14.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:14 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:59:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:14 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T22:59:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:14 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T22:59:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T22:59:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:14 smithi090 ceph-mon[44434]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T22:59:14.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:14 smithi090 ceph-mon[44434]: osd.0 v1:172.21.15.84:6801/1084680531 boot 2024-05-15T22:59:14.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:14 smithi090 ceph-mon[44434]: osdmap e8: 1 total, 1 up, 1 in 2024-05-15T22:59:14.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:14 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:59:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:14 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T22:59:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:14 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T22:59:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T22:59:14.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:14 smithi088 ceph-mon[44699]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T22:59:14.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T22:59:14.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:14 smithi088 ceph-mon[44699]: osd.0 v1:172.21.15.84:6801/1084680531 boot 2024-05-15T22:59:14.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:14 smithi088 ceph-mon[44699]: osdmap e8: 1 total, 1 up, 1 in 2024-05-15T22:59:14.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:14 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T22:59:15.575 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-15T22:59:15.575 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-15T22:59:15.575 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10+0 records in 2024-05-15T22:59:15.576 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records out 2024-05-15T22:59:15.576 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110897 s, 946 MB/s 2024-05-15T22:59:15.576 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping successful for: 2024-05-15T22:59:16.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:16 smithi084 ceph-mon[39784]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:16.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:16 smithi084 ceph-mon[39784]: osdmap e9: 1 total, 1 up, 1 in 2024-05-15T22:59:16.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:16.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:16.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T22:59:16.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:16.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:16.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:59:16.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:16.513 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch daemon add osd smithi084:vg_nvme/lv_3 2024-05-15T22:59:16.779 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T22:59:16.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:16 smithi090 ceph-mon[44434]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:16.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:16 smithi090 ceph-mon[44434]: osdmap e9: 1 total, 1 up, 1 in 2024-05-15T22:59:16.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:16.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:16.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T22:59:16.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:16.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:16.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:59:16.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:16.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:16 smithi088 ceph-mon[44699]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:16.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:16 smithi088 ceph-mon[44699]: osdmap e9: 1 total, 1 up, 1 in 2024-05-15T22:59:16.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:16.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:16.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T22:59:16.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:16.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:16.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:59:16.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:17.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:17 smithi084 ceph-mon[39784]: Detected new or changed devices on smithi084 2024-05-15T22:59:17.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:17 smithi084 ceph-mon[39784]: Adjusting osd_memory_target on smithi084 to 17122M 2024-05-15T22:59:17.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:17 smithi090 ceph-mon[44434]: Detected new or changed devices on smithi084 2024-05-15T22:59:17.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:17 smithi090 ceph-mon[44434]: Adjusting osd_memory_target on smithi084 to 17122M 2024-05-15T22:59:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:17 smithi088 ceph-mon[44699]: Detected new or changed devices on smithi084 2024-05-15T22:59:17.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:17 smithi088 ceph-mon[44699]: Adjusting osd_memory_target on smithi084 to 17122M 2024-05-15T22:59:18.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:18 smithi084 ceph-mon[39784]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:18.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:18 smithi090 ceph-mon[44434]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:18.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:18 smithi088 ceph-mon[44699]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:19.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T22:59:19.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T22:59:19.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:19 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T22:59:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:19 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T22:59:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:19 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:19.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:19 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T22:59:19.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:19 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T22:59:19.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:19 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:20.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:20 smithi084 ceph-mon[39784]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:20.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:20 smithi084 ceph-mon[39784]: from='client.14250 v1:172.21.15.84:0/525609117' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:59:20.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:20 smithi090 ceph-mon[44434]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:20.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:20 smithi090 ceph-mon[44434]: from='client.14250 v1:172.21.15.84:0/525609117' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:59:20.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:20 smithi088 ceph-mon[44699]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:20.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:20 smithi088 ceph-mon[44699]: from='client.14250 v1:172.21.15.84:0/525609117' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:59:21.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:21 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1186177782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c2c76c9-ce0f-4c2d-9d96-c87a10187607"}]: dispatch 2024-05-15T22:59:21.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:21 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1186177782' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c2c76c9-ce0f-4c2d-9d96-c87a10187607"}]': finished 2024-05-15T22:59:21.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:21 smithi084 ceph-mon[39784]: osdmap e10: 2 total, 1 up, 2 in 2024-05-15T22:59:21.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:21 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:21.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:21 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1186177782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c2c76c9-ce0f-4c2d-9d96-c87a10187607"}]: dispatch 2024-05-15T22:59:21.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:21 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1186177782' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c2c76c9-ce0f-4c2d-9d96-c87a10187607"}]': finished 2024-05-15T22:59:21.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:21 smithi090 ceph-mon[44434]: osdmap e10: 2 total, 1 up, 2 in 2024-05-15T22:59:21.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:21 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:21 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1186177782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c2c76c9-ce0f-4c2d-9d96-c87a10187607"}]: dispatch 2024-05-15T22:59:21.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:21 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1186177782' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c2c76c9-ce0f-4c2d-9d96-c87a10187607"}]': finished 2024-05-15T22:59:21.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:21 smithi088 ceph-mon[44699]: osdmap e10: 2 total, 1 up, 2 in 2024-05-15T22:59:21.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:21 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:22.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:22 smithi084 ceph-mon[39784]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:22.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1899339830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T22:59:22.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:22 smithi090 ceph-mon[44434]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:22.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1899339830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T22:59:22.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:22 smithi088 ceph-mon[44699]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:22.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1899339830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T22:59:24.600 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:24 smithi084 ceph-mon[39784]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:24.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:24 smithi090 ceph-mon[44434]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:24.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:24 smithi088 ceph-mon[44699]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:26.410 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:26 smithi084 ceph-mon[39784]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:26.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:26 smithi090 ceph-mon[44434]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:26.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:26 smithi088 ceph-mon[44699]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:28.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:28 smithi084 ceph-mon[39784]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:28.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-15T22:59:28.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:28.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:28 smithi084 ceph-mon[39784]: Deploying daemon osd.1 on smithi084 2024-05-15T22:59:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:28 smithi090 ceph-mon[44434]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-15T22:59:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:28 smithi090 ceph-mon[44434]: Deploying daemon osd.1 on smithi084 2024-05-15T22:59:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:28 smithi088 ceph-mon[44699]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:28.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-15T22:59:28.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:28.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:28 smithi088 ceph-mon[44699]: Deploying daemon osd.1 on smithi084 2024-05-15T22:59:30.502 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:30 smithi084 ceph-mon[39784]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:30.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:30 smithi090 ceph-mon[44434]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:30.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:30 smithi088 ceph-mon[44699]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:32 smithi090 ceph-mon[44434]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:32 smithi088 ceph-mon[44699]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:32.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:32 smithi084 ceph-mon[39784]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:34.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:33 smithi090 ceph-mon[44434]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:34.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:34.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:34.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:59:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:33 smithi088 ceph-mon[44699]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:34.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:34.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:59:34.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:33 smithi084 ceph-mon[39784]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:34.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:34.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:34.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:59:36.385 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 1 on host 'smithi084' 2024-05-15T22:59:36.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:36 smithi084 ceph-mon[39784]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:59:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:36 smithi084 ceph-mon[39784]: from='osd.1 v1:172.21.15.84:6805/2140830229' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-15T22:59:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:59:36.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:36 smithi090 ceph-mon[44434]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:36.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:36.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:36.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:36.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:59:36.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:36.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:36 smithi090 ceph-mon[44434]: from='osd.1 v1:172.21.15.84:6805/2140830229' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-15T22:59:36.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:36.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:36.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:59:36.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:36 smithi088 ceph-mon[44699]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:36.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:36.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:36.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:36.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:59:36.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:36.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:36 smithi088 ceph-mon[44699]: from='osd.1 v1:172.21.15.84:6805/2140830229' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-15T22:59:36.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:36.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:36.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:59:37.127 DEBUG:teuthology.orchestra.run.smithi084:osd.1> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.1.service 2024-05-15T22:59:37.131 INFO:tasks.cephadm:Deploying osd.2 on smithi084 with /dev/vg_nvme/lv_2... 2024-05-15T22:59:37.131 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-15T22:59:37.160 INFO:journalctl@ceph.osd.1.smithi084.stdout:-- Logs begin at Wed 2024-05-15 22:43:24 UTC. -- 2024-05-15T22:59:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:37 smithi090 ceph-mon[44434]: from='osd.1 v1:172.21.15.84:6805/2140830229' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-15T22:59:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:37 smithi090 ceph-mon[44434]: osdmap e11: 2 total, 1 up, 2 in 2024-05-15T22:59:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:37 smithi090 ceph-mon[44434]: from='osd.1 v1:172.21.15.84:6805/2140830229' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T22:59:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:37 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:37 smithi088 ceph-mon[44699]: from='osd.1 v1:172.21.15.84:6805/2140830229' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-15T22:59:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:37 smithi088 ceph-mon[44699]: osdmap e11: 2 total, 1 up, 2 in 2024-05-15T22:59:37.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:37 smithi088 ceph-mon[44699]: from='osd.1 v1:172.21.15.84:6805/2140830229' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T22:59:37.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:37 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:37.822 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T22:59:37.870 INFO:journalctl@ceph.osd.1.smithi084.stdout:May 15 22:59:37 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-1[65448]: 2024-05-15T22:59:37.484+0000 7fb381f66700 -1 osd.1 0 waiting for initial osdmap 2024-05-15T22:59:37.870 INFO:journalctl@ceph.osd.1.smithi084.stdout:May 15 22:59:37 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-1[65448]: 2024-05-15T22:59:37.493+0000 7fb37dd59700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T22:59:37.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:37 smithi084 ceph-mon[39784]: from='osd.1 v1:172.21.15.84:6805/2140830229' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-15T22:59:37.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:37 smithi084 ceph-mon[39784]: osdmap e11: 2 total, 1 up, 2 in 2024-05-15T22:59:37.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:37 smithi084 ceph-mon[39784]: from='osd.1 v1:172.21.15.84:6805/2140830229' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T22:59:37.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:37 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:38.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:38 smithi084 ceph-mon[39784]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:38.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:38 smithi084 ceph-mon[39784]: from='osd.1 v1:172.21.15.84:6805/2140830229' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-05-15T22:59:38.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:38 smithi084 ceph-mon[39784]: osdmap e12: 2 total, 1 up, 2 in 2024-05-15T22:59:38.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:38.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:38.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:38.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:38 smithi090 ceph-mon[44434]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:38.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:38 smithi090 ceph-mon[44434]: from='osd.1 v1:172.21.15.84:6805/2140830229' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-05-15T22:59:38.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:38 smithi090 ceph-mon[44434]: osdmap e12: 2 total, 1 up, 2 in 2024-05-15T22:59:38.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:38.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:38.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:38.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:38 smithi088 ceph-mon[44699]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:38.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:38 smithi088 ceph-mon[44699]: from='osd.1 v1:172.21.15.84:6805/2140830229' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-05-15T22:59:38.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:38 smithi088 ceph-mon[44699]: osdmap e12: 2 total, 1 up, 2 in 2024-05-15T22:59:38.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:38 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:38.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:38 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:38.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:38 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:39.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:39 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T22:59:39.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:39 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T22:59:39.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T22:59:39.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:39 smithi090 ceph-mon[44434]: osd.1 v1:172.21.15.84:6805/2140830229 boot 2024-05-15T22:59:39.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:39 smithi090 ceph-mon[44434]: osdmap e13: 2 total, 2 up, 2 in 2024-05-15T22:59:39.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:39.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-15T22:59:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:39 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T22:59:39.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:39 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T22:59:39.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T22:59:39.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:39 smithi088 ceph-mon[44699]: osd.1 v1:172.21.15.84:6805/2140830229 boot 2024-05-15T22:59:39.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:39 smithi088 ceph-mon[44699]: osdmap e13: 2 total, 2 up, 2 in 2024-05-15T22:59:39.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:39 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:39.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:39 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-15T22:59:39.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:39 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T22:59:39.846 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:39 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T22:59:39.846 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T22:59:39.846 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:39 smithi084 ceph-mon[39784]: osd.1 v1:172.21.15.84:6805/2140830229 boot 2024-05-15T22:59:39.846 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:39 smithi084 ceph-mon[39784]: osdmap e13: 2 total, 2 up, 2 in 2024-05-15T22:59:39.846 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T22:59:39.846 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-15T22:59:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:40 smithi090 ceph-mon[44434]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:40 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-15T22:59:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:40 smithi090 ceph-mon[44434]: osdmap e14: 2 total, 2 up, 2 in 2024-05-15T22:59:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:40 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T22:59:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:40 smithi090 ceph-mon[44434]: Detected new or changed devices on smithi084 2024-05-15T22:59:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:40 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:40 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:40 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T22:59:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:40 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T22:59:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:40 smithi090 ceph-mon[44434]: Adjusting osd_memory_target on smithi084 to 8561M 2024-05-15T22:59:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:40 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:40 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:40 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:59:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:40 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:40 smithi088 ceph-mon[44699]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-15T22:59:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:40 smithi088 ceph-mon[44699]: osdmap e14: 2 total, 2 up, 2 in 2024-05-15T22:59:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T22:59:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:40 smithi088 ceph-mon[44699]: Detected new or changed devices on smithi084 2024-05-15T22:59:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T22:59:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T22:59:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:40 smithi088 ceph-mon[44699]: Adjusting osd_memory_target on smithi084 to 8561M 2024-05-15T22:59:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:59:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:40 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:40 smithi084 ceph-mon[39784]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T22:59:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:40 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-15T22:59:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:40 smithi084 ceph-mon[39784]: osdmap e14: 2 total, 2 up, 2 in 2024-05-15T22:59:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:40 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T22:59:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:40 smithi084 ceph-mon[39784]: Detected new or changed devices on smithi084 2024-05-15T22:59:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:40 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:40 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:40 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T22:59:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:40 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T22:59:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:40 smithi084 ceph-mon[39784]: Adjusting osd_memory_target on smithi084 to 8561M 2024-05-15T22:59:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:40 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:40 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:40 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T22:59:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:40 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:41.281 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-15T22:59:41.281 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-15T22:59:41.281 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10+0 records in 2024-05-15T22:59:41.281 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records out 2024-05-15T22:59:41.281 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111827 s, 938 MB/s 2024-05-15T22:59:41.281 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping successful for: 2024-05-15T22:59:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70848]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-15T22:59:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70848]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T22:59:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70848]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T22:59:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70848]: pam_unix(sudo:session): session closed for user root 2024-05-15T22:59:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70853]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-15T22:59:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70853]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T22:59:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70853]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T22:59:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70853]: pam_unix(sudo:session): session closed for user root 2024-05-15T22:59:41.371 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70828]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-15T22:59:41.371 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70828]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T22:59:41.371 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70828]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T22:59:41.371 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70828]: pam_unix(sudo:session): session closed for user root 2024-05-15T22:59:41.371 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70833]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-15T22:59:41.371 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70833]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T22:59:41.372 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70833]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T22:59:41.372 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70833]: pam_unix(sudo:session): session closed for user root 2024-05-15T22:59:41.372 INFO:journalctl@ceph.osd.1.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70840]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-15T22:59:41.372 INFO:journalctl@ceph.osd.1.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70840]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T22:59:41.372 INFO:journalctl@ceph.osd.1.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70840]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T22:59:41.372 INFO:journalctl@ceph.osd.1.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70840]: pam_unix(sudo:session): session closed for user root 2024-05-15T22:59:41.372 INFO:journalctl@ceph.osd.1.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70845]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-15T22:59:41.372 INFO:journalctl@ceph.osd.1.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70845]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T22:59:41.372 INFO:journalctl@ceph.osd.1.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70845]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T22:59:41.372 INFO:journalctl@ceph.osd.1.smithi084.stdout:May 15 22:59:41 smithi084 sudo[70845]: pam_unix(sudo:session): session closed for user root 2024-05-15T22:59:41.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 sudo[46809]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-15T22:59:41.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 sudo[46809]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T22:59:41.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 sudo[46809]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T22:59:41.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 sudo[46809]: pam_unix(sudo:session): session closed for user root 2024-05-15T22:59:41.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 sudo[46812]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-15T22:59:41.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 sudo[46812]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T22:59:41.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 sudo[46812]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T22:59:41.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 sudo[46812]: pam_unix(sudo:session): session closed for user root 2024-05-15T22:59:41.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 sudo[47031]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-15T22:59:41.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 sudo[47031]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T22:59:41.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 sudo[47031]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T22:59:41.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 sudo[47031]: pam_unix(sudo:session): session closed for user root 2024-05-15T22:59:41.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 sudo[47034]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-15T22:59:41.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 sudo[47034]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T22:59:41.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 sudo[47034]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T22:59:41.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 sudo[47034]: pam_unix(sudo:session): session closed for user root 2024-05-15T22:59:41.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-15T22:59:41.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: osdmap e15: 2 total, 2 up, 2 in 2024-05-15T22:59:41.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: pgmap v66: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:41.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:41.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:59:41.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:59:41.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:59:41.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T22:59:41.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T22:59:41.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:59:41.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:59:41.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T22:59:41.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:59:41.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T22:59:41.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:41.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:59:41.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:59:41.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T22:59:41.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:59:41.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T22:59:41.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:41.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-15T22:59:41.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: osdmap e15: 2 total, 2 up, 2 in 2024-05-15T22:59:41.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: pgmap v66: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:41.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:41.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:59:41.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:59:41.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:59:41.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T22:59:41.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T22:59:41.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:59:41.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:59:41.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T22:59:41.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:59:41.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T22:59:41.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:41.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:59:41.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:59:41.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T22:59:41.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:59:41.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T22:59:41.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-15T22:59:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: osdmap e15: 2 total, 2 up, 2 in 2024-05-15T22:59:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: pgmap v66: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:59:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:59:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:59:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T22:59:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T22:59:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:59:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:59:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T22:59:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:59:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T22:59:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T22:59:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T22:59:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T22:59:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T22:59:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T22:59:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:41.919 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch daemon add osd smithi084:vg_nvme/lv_2 2024-05-15T22:59:42.184 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T22:59:42.638 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:42 smithi084 ceph-mon[39784]: osdmap e16: 2 total, 2 up, 2 in 2024-05-15T22:59:42.638 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:42 smithi084 ceph-mon[39784]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-15T22:59:42.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:42 smithi090 ceph-mon[44434]: osdmap e16: 2 total, 2 up, 2 in 2024-05-15T22:59:42.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:42 smithi090 ceph-mon[44434]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-15T22:59:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:42 smithi088 ceph-mon[44699]: osdmap e16: 2 total, 2 up, 2 in 2024-05-15T22:59:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:42 smithi088 ceph-mon[44699]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-15T22:59:43.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:43 smithi090 ceph-mon[44434]: pgmap v68: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:43.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:43 smithi088 ceph-mon[44699]: pgmap v68: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:43 smithi084 ceph-mon[39784]: pgmap v68: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:44.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:44 smithi090 ceph-mon[44434]: from='client.14286 v1:172.21.15.84:0/1042924674' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:59:44.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:44 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T22:59:44.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:44 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T22:59:44.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:44 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:44.818 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:44 smithi084 ceph-mon[39784]: from='client.14286 v1:172.21.15.84:0/1042924674' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:59:44.818 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:44 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T22:59:44.818 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:44 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T22:59:44.818 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:44 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:44 smithi088 ceph-mon[44699]: from='client.14286 v1:172.21.15.84:0/1042924674' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T22:59:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:44 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T22:59:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:44 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T22:59:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:44 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:45.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:45 smithi084 ceph-mon[39784]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:45 smithi090 ceph-mon[44434]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:45.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:45 smithi088 ceph-mon[44699]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:47.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:47 smithi084 ceph-mon[39784]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:47.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:47 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3737999323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8514daa-b5f9-4f2f-a432-53f4a4eb86bc"}]: dispatch 2024-05-15T22:59:47.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:47 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3737999323' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8514daa-b5f9-4f2f-a432-53f4a4eb86bc"}]': finished 2024-05-15T22:59:47.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:47 smithi084 ceph-mon[39784]: osdmap e17: 3 total, 2 up, 3 in 2024-05-15T22:59:47.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:47 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T22:59:47.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:47 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2240652368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T22:59:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:47 smithi090 ceph-mon[44434]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:47 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3737999323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8514daa-b5f9-4f2f-a432-53f4a4eb86bc"}]: dispatch 2024-05-15T22:59:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:47 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3737999323' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8514daa-b5f9-4f2f-a432-53f4a4eb86bc"}]': finished 2024-05-15T22:59:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:47 smithi090 ceph-mon[44434]: osdmap e17: 3 total, 2 up, 3 in 2024-05-15T22:59:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:47 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T22:59:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:47 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2240652368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T22:59:48.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:47 smithi088 ceph-mon[44699]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:47 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3737999323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8514daa-b5f9-4f2f-a432-53f4a4eb86bc"}]: dispatch 2024-05-15T22:59:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:47 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3737999323' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8514daa-b5f9-4f2f-a432-53f4a4eb86bc"}]': finished 2024-05-15T22:59:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:47 smithi088 ceph-mon[44699]: osdmap e17: 3 total, 2 up, 3 in 2024-05-15T22:59:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:47 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T22:59:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:47 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2240652368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T22:59:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:49 smithi084 ceph-mon[39784]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:50.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:49 smithi090 ceph-mon[44434]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:50.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:49 smithi088 ceph-mon[44699]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:51.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:51 smithi084 ceph-mon[39784]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:52.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:51 smithi090 ceph-mon[44434]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:52.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:51 smithi088 ceph-mon[44699]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:53.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:53 smithi084 ceph-mon[39784]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:53.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-15T22:59:53.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:53.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:53 smithi084 ceph-mon[39784]: Deploying daemon osd.2 on smithi084 2024-05-15T22:59:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:53 smithi090 ceph-mon[44434]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-15T22:59:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:53 smithi090 ceph-mon[44434]: Deploying daemon osd.2 on smithi084 2024-05-15T22:59:54.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:53 smithi088 ceph-mon[44699]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:54.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-15T22:59:54.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T22:59:54.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:53 smithi088 ceph-mon[44699]: Deploying daemon osd.2 on smithi084 2024-05-15T22:59:55.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:55 smithi084 ceph-mon[39784]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:56.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:55 smithi090 ceph-mon[44434]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:56.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:55 smithi088 ceph-mon[44699]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:57.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:57 smithi090 ceph-mon[44434]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:57 smithi088 ceph-mon[44699]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:57.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:57 smithi084 ceph-mon[39784]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T22:59:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:59 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:59 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:59:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 22:59:59 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:59 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:59 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:59:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 22:59:59 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:59 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T22:59:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:59 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T22:59:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 22:59:59 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:00.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:00 smithi084 ceph-mon[39784]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T23:00:00.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:00 smithi084 ceph-mon[39784]: overall HEALTH_OK 2024-05-15T23:00:00.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:00 smithi090 ceph-mon[44434]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T23:00:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:00 smithi090 ceph-mon[44434]: overall HEALTH_OK 2024-05-15T23:00:00.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:00 smithi088 ceph-mon[44699]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T23:00:00.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:00 smithi088 ceph-mon[44699]: overall HEALTH_OK 2024-05-15T23:00:01.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:01 smithi084 ceph-mon[39784]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T23:00:01.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:01.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:01.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:01.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:01.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:01.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:01 smithi084 ceph-mon[39784]: from='osd.2 v1:172.21.15.84:6809/2699996290' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-15T23:00:01.822 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 2 on host 'smithi084' 2024-05-15T23:00:02.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:01 smithi090 ceph-mon[44434]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T23:00:02.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:02.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:02.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:02.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:02.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:02.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:01 smithi090 ceph-mon[44434]: from='osd.2 v1:172.21.15.84:6809/2699996290' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-15T23:00:02.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:01 smithi088 ceph-mon[44699]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T23:00:02.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:02.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:02.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:02.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:02.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:02.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:01 smithi088 ceph-mon[44699]: from='osd.2 v1:172.21.15.84:6809/2699996290' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-15T23:00:02.536 DEBUG:teuthology.orchestra.run.smithi084:osd.2> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.2.service 2024-05-15T23:00:02.539 INFO:tasks.cephadm:Deploying osd.3 on smithi084 with /dev/vg_nvme/lv_1... 2024-05-15T23:00:02.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-15T23:00:02.570 INFO:journalctl@ceph.osd.2.smithi084.stdout:-- Logs begin at Wed 2024-05-15 22:43:24 UTC. -- 2024-05-15T23:00:03.025 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:03.025 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:03.025 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:00:03.025 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:02 smithi084 ceph-mon[39784]: from='osd.2 v1:172.21.15.84:6809/2699996290' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-15T23:00:03.025 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:02 smithi084 ceph-mon[39784]: osdmap e18: 3 total, 2 up, 3 in 2024-05-15T23:00:03.025 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:02 smithi084 ceph-mon[39784]: from='osd.2 v1:172.21.15.84:6809/2699996290' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T23:00:03.025 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:00:03.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:02 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:03.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:02 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:03.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:02 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:00:03.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:02 smithi090 ceph-mon[44434]: from='osd.2 v1:172.21.15.84:6809/2699996290' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-15T23:00:03.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:02 smithi090 ceph-mon[44434]: osdmap e18: 3 total, 2 up, 3 in 2024-05-15T23:00:03.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:02 smithi090 ceph-mon[44434]: from='osd.2 v1:172.21.15.84:6809/2699996290' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T23:00:03.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:02 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:00:03.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:02 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:03.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:02 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:03.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:02 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:00:03.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:02 smithi088 ceph-mon[44699]: from='osd.2 v1:172.21.15.84:6809/2699996290' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-15T23:00:03.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:02 smithi088 ceph-mon[44699]: osdmap e18: 3 total, 2 up, 3 in 2024-05-15T23:00:03.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:02 smithi088 ceph-mon[44699]: from='osd.2 v1:172.21.15.84:6809/2699996290' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T23:00:03.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:02 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:00:03.619 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:00:03 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-2[73385]: 2024-05-15T23:00:03.287+0000 7fdeb0dd3700 -1 osd.2 0 waiting for initial osdmap 2024-05-15T23:00:03.620 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:00:03 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-2[73385]: 2024-05-15T23:00:03.291+0000 7fdeacbc6700 -1 osd.2 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T23:00:03.744 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:00:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:04 smithi090 ceph-mon[44434]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T23:00:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:04 smithi090 ceph-mon[44434]: from='osd.2 v1:172.21.15.84:6809/2699996290' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-05-15T23:00:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:04 smithi090 ceph-mon[44434]: osdmap e19: 3 total, 2 up, 3 in 2024-05-15T23:00:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:04 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:00:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:04 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:00:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:04 smithi088 ceph-mon[44699]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T23:00:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:04 smithi088 ceph-mon[44699]: from='osd.2 v1:172.21.15.84:6809/2699996290' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-05-15T23:00:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:04 smithi088 ceph-mon[44699]: osdmap e19: 3 total, 2 up, 3 in 2024-05-15T23:00:04.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:04 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:00:04.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:04 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:00:04.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:04 smithi084 ceph-mon[39784]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T23:00:04.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:04 smithi084 ceph-mon[39784]: from='osd.2 v1:172.21.15.84:6809/2699996290' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-05-15T23:00:04.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:04 smithi084 ceph-mon[39784]: osdmap e19: 3 total, 2 up, 3 in 2024-05-15T23:00:04.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:04 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:00:04.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:04 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:00:05.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:05 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:00:05.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:05 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:00:05.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:00:05.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:05 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:00:05.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:05 smithi090 ceph-mon[44434]: osd.2 v1:172.21.15.84:6809/2699996290 boot 2024-05-15T23:00:05.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:05 smithi090 ceph-mon[44434]: osdmap e20: 3 total, 3 up, 3 in 2024-05-15T23:00:05.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:05 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:00:05.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:05 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:00:05.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:05 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:00:05.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:00:05.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:05 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:00:05.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:05 smithi088 ceph-mon[44699]: osd.2 v1:172.21.15.84:6809/2699996290 boot 2024-05-15T23:00:05.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:05 smithi088 ceph-mon[44699]: osdmap e20: 3 total, 3 up, 3 in 2024-05-15T23:00:05.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:05 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:00:05.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:05 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:00:05.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:05 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:00:05.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:00:05.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:05 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:00:05.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:05 smithi084 ceph-mon[39784]: osd.2 v1:172.21.15.84:6809/2699996290 boot 2024-05-15T23:00:05.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:05 smithi084 ceph-mon[39784]: osdmap e20: 3 total, 3 up, 3 in 2024-05-15T23:00:05.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:05 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:00:06.534 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-15T23:00:06.534 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-15T23:00:06.534 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10+0 records in 2024-05-15T23:00:06.534 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records out 2024-05-15T23:00:06.534 INFO:teuthology.orchestra.run.smithi084.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112514 s, 932 MB/s 2024-05-15T23:00:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping successful for: 2024-05-15T23:00:06.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:06 smithi090 ceph-mon[44434]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T23:00:06.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:06 smithi090 ceph-mon[44434]: osdmap e21: 3 total, 3 up, 3 in 2024-05-15T23:00:06.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:06 smithi084 ceph-mon[39784]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T23:00:06.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:06 smithi084 ceph-mon[39784]: osdmap e21: 3 total, 3 up, 3 in 2024-05-15T23:00:06.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:06 smithi088 ceph-mon[44699]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-05-15T23:00:06.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:06 smithi088 ceph-mon[44699]: osdmap e21: 3 total, 3 up, 3 in 2024-05-15T23:00:07.117 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch daemon add osd smithi084:vg_nvme/lv_1 2024-05-15T23:00:07.382 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:00:07.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:07 smithi088 ceph-mon[44699]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:07.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:07 smithi088 ceph-mon[44699]: Detected new or changed devices on smithi084 2024-05-15T23:00:07.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:07.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:07.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:07.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:07.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:07.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:07 smithi088 ceph-mon[44699]: Adjusting osd_memory_target on smithi084 to 5707M 2024-05-15T23:00:07.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:07.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:07.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:07.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:07.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:07 smithi084 ceph-mon[39784]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:07.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:07 smithi084 ceph-mon[39784]: Detected new or changed devices on smithi084 2024-05-15T23:00:07.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:07.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:07.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:07.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:07.852 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:07.852 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:07 smithi084 ceph-mon[39784]: Adjusting osd_memory_target on smithi084 to 5707M 2024-05-15T23:00:07.852 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:07.852 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:07.852 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:07.852 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:08.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:07 smithi090 ceph-mon[44434]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:08.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:07 smithi090 ceph-mon[44434]: Detected new or changed devices on smithi084 2024-05-15T23:00:08.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:08.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:08.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:08.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:08.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:08.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:07 smithi090 ceph-mon[44434]: Adjusting osd_memory_target on smithi084 to 5707M 2024-05-15T23:00:08.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:08.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:08.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:08.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:09.822 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:09 smithi084 ceph-mon[39784]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:09.822 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:09 smithi084 ceph-mon[39784]: from='client.14313 v1:172.21.15.84:0/3233935217' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:00:09.822 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:09 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:00:09.822 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:09 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:00:09.822 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:09 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:09.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:09 smithi088 ceph-mon[44699]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:09.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:09 smithi088 ceph-mon[44699]: from='client.14313 v1:172.21.15.84:0/3233935217' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:00:09.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:09 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:00:09.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:09 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:00:09.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:09 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:10.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:09 smithi090 ceph-mon[44434]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:10.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:09 smithi090 ceph-mon[44434]: from='client.14313 v1:172.21.15.84:0/3233935217' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:00:10.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:09 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:00:10.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:09 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:00:10.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:09 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:11 smithi088 ceph-mon[44699]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:11.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:11 smithi084 ceph-mon[39784]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:11 smithi090 ceph-mon[44434]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:12 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3470647912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d9965f9-c0b2-48ce-869c-582990985623"}]: dispatch 2024-05-15T23:00:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:12 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3470647912' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d9965f9-c0b2-48ce-869c-582990985623"}]': finished 2024-05-15T23:00:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:12 smithi088 ceph-mon[44699]: osdmap e22: 4 total, 3 up, 4 in 2024-05-15T23:00:12.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:12.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:12 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3766535121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:00:12.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:12 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3470647912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d9965f9-c0b2-48ce-869c-582990985623"}]: dispatch 2024-05-15T23:00:12.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:12 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3470647912' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d9965f9-c0b2-48ce-869c-582990985623"}]': finished 2024-05-15T23:00:12.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:12 smithi084 ceph-mon[39784]: osdmap e22: 4 total, 3 up, 4 in 2024-05-15T23:00:12.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:12.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:12 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3766535121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:00:13.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:12 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3470647912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d9965f9-c0b2-48ce-869c-582990985623"}]: dispatch 2024-05-15T23:00:13.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:12 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3470647912' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d9965f9-c0b2-48ce-869c-582990985623"}]': finished 2024-05-15T23:00:13.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:12 smithi090 ceph-mon[44434]: osdmap e22: 4 total, 3 up, 4 in 2024-05-15T23:00:13.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:13.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:12 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3766535121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:00:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:13 smithi090 ceph-mon[44434]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:13 smithi088 ceph-mon[44699]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:14.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:13 smithi084 ceph-mon[39784]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:16.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:15 smithi084 ceph-mon[39784]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:16.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:15 smithi090 ceph-mon[44434]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:16.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:15 smithi088 ceph-mon[44699]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:17.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:17 smithi090 ceph-mon[44434]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:17.818 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:17 smithi084 ceph-mon[39784]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:17 smithi088 ceph-mon[44699]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:18.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-15T23:00:18.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:18.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:18 smithi084 ceph-mon[39784]: Deploying daemon osd.3 on smithi084 2024-05-15T23:00:18.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-15T23:00:18.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:18.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:18 smithi090 ceph-mon[44434]: Deploying daemon osd.3 on smithi084 2024-05-15T23:00:18.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-15T23:00:18.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:18.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:18 smithi088 ceph-mon[44699]: Deploying daemon osd.3 on smithi084 2024-05-15T23:00:19.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:19 smithi084 ceph-mon[39784]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:19.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:19 smithi090 ceph-mon[44434]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:19.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:19 smithi088 ceph-mon[44699]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:21.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:21 smithi084 ceph-mon[39784]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:21.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:21 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:00:21.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:21 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:00:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:21 smithi088 ceph-mon[44699]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:21 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:00:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:21 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:00:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:21 smithi090 ceph-mon[44434]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:21 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:00:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:21 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:00:22.808 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:22 smithi084 ceph-mon[39784]: osdmap e23: 4 total, 3 up, 4 in 2024-05-15T23:00:22.808 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:22 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:22 smithi088 ceph-mon[44699]: osdmap e23: 4 total, 3 up, 4 in 2024-05-15T23:00:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:22 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:23.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:22 smithi090 ceph-mon[44434]: osdmap e23: 4 total, 3 up, 4 in 2024-05-15T23:00:23.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:22 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:23 smithi088 ceph-mon[44699]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:23 smithi084 ceph-mon[39784]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:24.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:23 smithi090 ceph-mon[44434]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:25.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:25.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:25.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:00:25.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:25.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:25.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:00:25.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:25.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:25.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:00:26.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:25 smithi084 ceph-mon[39784]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:26.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:25 smithi090 ceph-mon[44434]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:26.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:25 smithi088 ceph-mon[44699]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:27.229 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 3 on host 'smithi084' 2024-05-15T23:00:27.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:27.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:27.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:27.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:27.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:27.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:27 smithi084 ceph-mon[39784]: from='osd.3 v1:172.21.15.84:6813/2204039828' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-15T23:00:27.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:27.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:00:27.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:27 smithi090 ceph-mon[44434]: from='osd.3 v1:172.21.15.84:6813/2204039828' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-15T23:00:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:00:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:27.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:27.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:27.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:27.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:27.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:27.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:27 smithi088 ceph-mon[44699]: from='osd.3 v1:172.21.15.84:6813/2204039828' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-15T23:00:27.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:27.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:00:27.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:27.896 DEBUG:teuthology.orchestra.run.smithi084:osd.3> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.3.service 2024-05-15T23:00:27.898 INFO:tasks.cephadm:Deploying osd.4 on smithi088 with /dev/vg_nvme/lv_4... 2024-05-15T23:00:27.898 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-15T23:00:27.930 INFO:journalctl@ceph.osd.3.smithi084.stdout:-- Logs begin at Wed 2024-05-15 22:43:24 UTC. -- 2024-05-15T23:00:28.166 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:00:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:28 smithi084 ceph-mon[39784]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:28 smithi084 ceph-mon[39784]: from='osd.3 v1:172.21.15.84:6813/2204039828' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-15T23:00:28.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:28 smithi084 ceph-mon[39784]: osdmap e24: 4 total, 3 up, 4 in 2024-05-15T23:00:28.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:28 smithi084 ceph-mon[39784]: from='osd.3 v1:172.21.15.84:6813/2204039828' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T23:00:28.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:28.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:28 smithi088 ceph-mon[44699]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:28.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:28 smithi088 ceph-mon[44699]: from='osd.3 v1:172.21.15.84:6813/2204039828' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-15T23:00:28.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:28 smithi088 ceph-mon[44699]: osdmap e24: 4 total, 3 up, 4 in 2024-05-15T23:00:28.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:28 smithi088 ceph-mon[44699]: from='osd.3 v1:172.21.15.84:6813/2204039828' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T23:00:28.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:28 smithi090 ceph-mon[44434]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:28 smithi090 ceph-mon[44434]: from='osd.3 v1:172.21.15.84:6813/2204039828' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-15T23:00:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:28 smithi090 ceph-mon[44434]: osdmap e24: 4 total, 3 up, 4 in 2024-05-15T23:00:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:28 smithi090 ceph-mon[44434]: from='osd.3 v1:172.21.15.84:6813/2204039828' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T23:00:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:29.035 INFO:journalctl@ceph.osd.3.smithi084.stdout:May 15 23:00:28 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-3[80671]: 2024-05-15T23:00:28.988+0000 7f8f12ab1700 -1 osd.3 0 waiting for initial osdmap 2024-05-15T23:00:29.036 INFO:journalctl@ceph.osd.3.smithi084.stdout:May 15 23:00:28 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-3[80671]: 2024-05-15T23:00:28.993+0000 7f8f0a89c700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T23:00:29.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:29 smithi084 ceph-mon[39784]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:29.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:29 smithi084 ceph-mon[39784]: from='osd.3 v1:172.21.15.84:6813/2204039828' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-05-15T23:00:29.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:29 smithi084 ceph-mon[39784]: osdmap e25: 4 total, 3 up, 4 in 2024-05-15T23:00:29.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:29.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:29.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:29 smithi084 ceph-mon[39784]: from='osd.3 v1:172.21.15.84:6813/2204039828' entity='osd.3' 2024-05-15T23:00:29.677 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:29 smithi088 ceph-mon[44699]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:29.677 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:29 smithi088 ceph-mon[44699]: from='osd.3 v1:172.21.15.84:6813/2204039828' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-05-15T23:00:29.677 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:29 smithi088 ceph-mon[44699]: osdmap e25: 4 total, 3 up, 4 in 2024-05-15T23:00:29.677 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:29 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:29.678 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:29 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:29.678 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:29 smithi088 ceph-mon[44699]: from='osd.3 v1:172.21.15.84:6813/2204039828' entity='osd.3' 2024-05-15T23:00:29.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:29 smithi090 ceph-mon[44434]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:29.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:29 smithi090 ceph-mon[44434]: from='osd.3 v1:172.21.15.84:6813/2204039828' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-05-15T23:00:29.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:29 smithi090 ceph-mon[44434]: osdmap e25: 4 total, 3 up, 4 in 2024-05-15T23:00:29.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:29.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:29.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:29 smithi090 ceph-mon[44434]: from='osd.3 v1:172.21.15.84:6813/2204039828' entity='osd.3' 2024-05-15T23:00:30.292 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-15T23:00:30.292 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-15T23:00:30.292 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2024-05-15T23:00:30.293 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2024-05-15T23:00:30.293 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108175 s, 969 MB/s 2024-05-15T23:00:30.293 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2024-05-15T23:00:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:30 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:00:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:30 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:00:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:00:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:00:30.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:30 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:30.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:30 smithi084 ceph-mon[39784]: osd.3 v1:172.21.15.84:6813/2204039828 boot 2024-05-15T23:00:30.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:30 smithi084 ceph-mon[39784]: osdmap e26: 4 total, 4 up, 4 in 2024-05-15T23:00:30.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:30 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:30.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:30 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:00:30.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:30 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:00:30.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:00:30.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:00:30.687 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:30 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:30.688 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:30 smithi088 ceph-mon[44699]: osd.3 v1:172.21.15.84:6813/2204039828 boot 2024-05-15T23:00:30.688 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:30 smithi088 ceph-mon[44699]: osdmap e26: 4 total, 4 up, 4 in 2024-05-15T23:00:30.688 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:30 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:30.976 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch daemon add osd smithi088:vg_nvme/lv_4 2024-05-15T23:00:31.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:30 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:00:31.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:30 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:00:31.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:00:31.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:00:31.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:30 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:31.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:30 smithi090 ceph-mon[44434]: osd.3 v1:172.21.15.84:6813/2204039828 boot 2024-05-15T23:00:31.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:30 smithi090 ceph-mon[44434]: osdmap e26: 4 total, 4 up, 4 in 2024-05-15T23:00:31.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:30 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T23:00:31.239 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:00:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:31 smithi090 ceph-mon[44434]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:31 smithi090 ceph-mon[44434]: Detected new or changed devices on smithi084 2024-05-15T23:00:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:31 smithi090 ceph-mon[44434]: Adjusting osd_memory_target on smithi084 to 4280M 2024-05-15T23:00:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:32.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:32.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:32.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:32.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:31 smithi090 ceph-mon[44434]: osdmap e27: 4 total, 4 up, 4 in 2024-05-15T23:00:32.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:31 smithi088 ceph-mon[44699]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:32.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:31 smithi088 ceph-mon[44699]: Detected new or changed devices on smithi084 2024-05-15T23:00:32.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:32.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:32.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:32.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:32.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:32.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:32.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:31 smithi088 ceph-mon[44699]: Adjusting osd_memory_target on smithi084 to 4280M 2024-05-15T23:00:32.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:32.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:32.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:32.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:32.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:31 smithi088 ceph-mon[44699]: osdmap e27: 4 total, 4 up, 4 in 2024-05-15T23:00:32.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:31 smithi084 ceph-mon[39784]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-15T23:00:32.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:31 smithi084 ceph-mon[39784]: Detected new or changed devices on smithi084 2024-05-15T23:00:32.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:32.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:32.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:32.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:32.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:32.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:32.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:31 smithi084 ceph-mon[39784]: Adjusting osd_memory_target on smithi084 to 4280M 2024-05-15T23:00:32.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:32.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:32.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:32.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:32.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:31 smithi084 ceph-mon[39784]: osdmap e27: 4 total, 4 up, 4 in 2024-05-15T23:00:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:33 smithi090 ceph-mon[44434]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:33 smithi090 ceph-mon[44434]: from='client.34229 v1:172.21.15.88:0/902834889' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:00:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:00:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:00:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:33 smithi088 ceph-mon[44699]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:33 smithi088 ceph-mon[44699]: from='client.34229 v1:172.21.15.88:0/902834889' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:00:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:00:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:00:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:34.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:33 smithi084 ceph-mon[39784]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:34.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:33 smithi084 ceph-mon[39784]: from='client.34229 v1:172.21.15.88:0/902834889' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:00:34.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:00:34.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:00:34.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:36.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:35 smithi090 ceph-mon[44434]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:36.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:35 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.88:0/1161545375' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bc50523-8480-44d2-8d76-31479e839c85"}]: dispatch 2024-05-15T23:00:36.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:35 smithi090 ceph-mon[44434]: from='client.34235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bc50523-8480-44d2-8d76-31479e839c85"}]: dispatch 2024-05-15T23:00:36.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:35 smithi090 ceph-mon[44434]: from='client.34235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4bc50523-8480-44d2-8d76-31479e839c85"}]': finished 2024-05-15T23:00:36.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:35 smithi090 ceph-mon[44434]: osdmap e28: 5 total, 4 up, 5 in 2024-05-15T23:00:36.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:35 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:36.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:35 smithi088 ceph-mon[44699]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:36.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:35 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.88:0/1161545375' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bc50523-8480-44d2-8d76-31479e839c85"}]: dispatch 2024-05-15T23:00:36.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:35 smithi088 ceph-mon[44699]: from='client.34235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bc50523-8480-44d2-8d76-31479e839c85"}]: dispatch 2024-05-15T23:00:36.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:35 smithi088 ceph-mon[44699]: from='client.34235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4bc50523-8480-44d2-8d76-31479e839c85"}]': finished 2024-05-15T23:00:36.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:35 smithi088 ceph-mon[44699]: osdmap e28: 5 total, 4 up, 5 in 2024-05-15T23:00:36.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:35 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:36.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:35 smithi084 ceph-mon[39784]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:36.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:35 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.88:0/1161545375' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bc50523-8480-44d2-8d76-31479e839c85"}]: dispatch 2024-05-15T23:00:36.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:35 smithi084 ceph-mon[39784]: from='client.34235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bc50523-8480-44d2-8d76-31479e839c85"}]: dispatch 2024-05-15T23:00:36.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:35 smithi084 ceph-mon[39784]: from='client.34235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4bc50523-8480-44d2-8d76-31479e839c85"}]': finished 2024-05-15T23:00:36.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:35 smithi084 ceph-mon[39784]: osdmap e28: 5 total, 4 up, 5 in 2024-05-15T23:00:36.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:35 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:37.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.88:0/3050571628' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:00:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.88:0/3050571628' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:00:37.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.88:0/3050571628' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:00:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:38 smithi090 ceph-mon[44434]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:38 smithi088 ceph-mon[44699]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:38 smithi084 ceph-mon[39784]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:40.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:40 smithi090 ceph-mon[44434]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:40.317 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:40 smithi088 ceph-mon[44699]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:40 smithi084 ceph-mon[39784]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:42.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:42 smithi090 ceph-mon[44434]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:42.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:42 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-15T23:00:42.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:42 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:42.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:42 smithi088 ceph-mon[44699]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:42.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:42 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-15T23:00:42.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:42 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:42.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:42 smithi084 ceph-mon[39784]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:42.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:42 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-15T23:00:42.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:42 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:43.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:43 smithi090 ceph-mon[44434]: Deploying daemon osd.4 on smithi088 2024-05-15T23:00:43.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:43 smithi088 ceph-mon[44699]: Deploying daemon osd.4 on smithi088 2024-05-15T23:00:43.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:43 smithi084 ceph-mon[39784]: Deploying daemon osd.4 on smithi088 2024-05-15T23:00:44.220 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:44 smithi088 ceph-mon[44699]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:44.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:44 smithi090 ceph-mon[44434]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:44.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:44 smithi084 ceph-mon[39784]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:46.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:46 smithi090 ceph-mon[44434]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:46.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:46 smithi088 ceph-mon[44699]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:46.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:46 smithi084 ceph-mon[39784]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:48.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:48 smithi090 ceph-mon[44434]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:48.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:48 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:48.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:48 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:48.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:48 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:00:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:48 smithi088 ceph-mon[44699]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:48 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:48 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:48 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:00:48.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:48 smithi084 ceph-mon[39784]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:48.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:48 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:48.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:48 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:48.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:48 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:00:50.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:50 smithi088 ceph-mon[44699]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:50.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:50 smithi090 ceph-mon[44434]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:50.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:50 smithi084 ceph-mon[39784]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:51.439 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:51.439 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:51.439 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:51.439 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:51.439 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:51.439 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:51 smithi088 ceph-mon[44699]: from='osd.4 v1:172.21.15.88:6800/2267955292' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T23:00:51.439 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:51 smithi088 ceph-mon[44699]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T23:00:51.482 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 4 on host 'smithi088' 2024-05-15T23:00:51.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:51 smithi084 ceph-mon[39784]: from='osd.4 v1:172.21.15.88:6800/2267955292' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T23:00:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:51 smithi084 ceph-mon[39784]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T23:00:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:51 smithi090 ceph-mon[44434]: from='osd.4 v1:172.21.15.88:6800/2267955292' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T23:00:51.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:51 smithi090 ceph-mon[44434]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T23:00:52.103 DEBUG:teuthology.orchestra.run.smithi088:osd.4> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.4.service 2024-05-15T23:00:52.105 INFO:tasks.cephadm:Deploying osd.5 on smithi088 with /dev/vg_nvme/lv_3... 2024-05-15T23:00:52.106 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-15T23:00:52.138 INFO:journalctl@ceph.osd.4.smithi088.stdout:-- Logs begin at Wed 2024-05-15 22:43:48 UTC. -- 2024-05-15T23:00:52.619 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:52 smithi088 ceph-mon[44699]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:52.619 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:52 smithi088 ceph-mon[44699]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-15T23:00:52.619 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:52 smithi088 ceph-mon[44699]: from='osd.4 v1:172.21.15.88:6800/2267955292' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:00:52.619 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:52 smithi088 ceph-mon[44699]: osdmap e29: 5 total, 4 up, 5 in 2024-05-15T23:00:52.619 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:52 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:52.619 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:52 smithi088 ceph-mon[44699]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:00:52.619 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:52 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:52.619 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:52 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:52.619 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:52 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:00:52.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:52 smithi084 ceph-mon[39784]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:52.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:52 smithi084 ceph-mon[39784]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-15T23:00:52.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:52 smithi084 ceph-mon[39784]: from='osd.4 v1:172.21.15.88:6800/2267955292' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:00:52.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:52 smithi084 ceph-mon[39784]: osdmap e29: 5 total, 4 up, 5 in 2024-05-15T23:00:52.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:52 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:52.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:52 smithi084 ceph-mon[39784]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:00:52.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:52 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:52.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:52 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:52.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:52 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:00:52.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:52 smithi090 ceph-mon[44434]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:52.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:52 smithi090 ceph-mon[44434]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-15T23:00:52.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:52 smithi090 ceph-mon[44434]: from='osd.4 v1:172.21.15.88:6800/2267955292' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:00:52.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:52 smithi090 ceph-mon[44434]: osdmap e29: 5 total, 4 up, 5 in 2024-05-15T23:00:52.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:52 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:52.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:52 smithi090 ceph-mon[44434]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:00:52.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:52 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:52.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:52 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:52.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:52 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:00:52.930 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:00:53.071 INFO:journalctl@ceph.osd.4.smithi088.stdout:May 15 23:00:52 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-4[50000]: 2024-05-15T23:00:52.779+0000 7f424b13b700 -1 osd.4 0 waiting for initial osdmap 2024-05-15T23:00:53.071 INFO:journalctl@ceph.osd.4.smithi088.stdout:May 15 23:00:52 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-4[50000]: 2024-05-15T23:00:52.783+0000 7f4242f26700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T23:00:53.420 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:53 smithi088 ceph-mon[44699]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-05-15T23:00:53.420 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:53 smithi088 ceph-mon[44699]: osdmap e30: 5 total, 4 up, 5 in 2024-05-15T23:00:53.421 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:53.421 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:53.421 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:53 smithi088 ceph-mon[44699]: from='osd.4 ' entity='osd.4' 2024-05-15T23:00:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:53 smithi084 ceph-mon[39784]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-05-15T23:00:53.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:53 smithi084 ceph-mon[39784]: osdmap e30: 5 total, 4 up, 5 in 2024-05-15T23:00:53.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:53.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:53.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:53 smithi084 ceph-mon[39784]: from='osd.4 ' entity='osd.4' 2024-05-15T23:00:53.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:53 smithi090 ceph-mon[44434]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-05-15T23:00:53.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:53 smithi090 ceph-mon[44434]: osdmap e30: 5 total, 4 up, 5 in 2024-05-15T23:00:53.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:53.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:53.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:53 smithi090 ceph-mon[44434]: from='osd.4 ' entity='osd.4' 2024-05-15T23:00:54.430 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:54 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:00:54.430 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:54 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:00:54.430 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:00:54.431 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:54 smithi088 ceph-mon[44699]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:54.431 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:00:54.431 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:54.431 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:54 smithi088 ceph-mon[44699]: osd.4 v1:172.21.15.88:6800/2267955292 boot 2024-05-15T23:00:54.431 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:54 smithi088 ceph-mon[44699]: osdmap e31: 5 total, 5 up, 5 in 2024-05-15T23:00:54.431 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:54.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:54 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:00:54.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:54 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:00:54.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:00:54.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:54 smithi084 ceph-mon[39784]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:54.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:00:54.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:54.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:54 smithi084 ceph-mon[39784]: osd.4 v1:172.21.15.88:6800/2267955292 boot 2024-05-15T23:00:54.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:54 smithi084 ceph-mon[39784]: osdmap e31: 5 total, 5 up, 5 in 2024-05-15T23:00:54.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:54 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:00:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:54 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:00:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:00:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:54 smithi090 ceph-mon[44434]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-15T23:00:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:00:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:54 smithi090 ceph-mon[44434]: osd.4 v1:172.21.15.88:6800/2267955292 boot 2024-05-15T23:00:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:54 smithi090 ceph-mon[44434]: osdmap e31: 5 total, 5 up, 5 in 2024-05-15T23:00:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T23:00:55.511 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-15T23:00:55.512 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-15T23:00:55.512 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2024-05-15T23:00:55.512 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2024-05-15T23:00:55.512 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109786 s, 955 MB/s 2024-05-15T23:00:55.512 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2024-05-15T23:00:55.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:55 smithi088 ceph-mon[44699]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:00:55.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:55 smithi088 ceph-mon[44699]: osdmap e32: 5 total, 5 up, 5 in 2024-05-15T23:00:56.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:55 smithi090 ceph-mon[44434]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:00:56.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:55 smithi090 ceph-mon[44434]: osdmap e32: 5 total, 5 up, 5 in 2024-05-15T23:00:56.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:55 smithi084 ceph-mon[39784]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:00:56.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:55 smithi084 ceph-mon[39784]: osdmap e32: 5 total, 5 up, 5 in 2024-05-15T23:00:56.248 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch daemon add osd smithi088:vg_nvme/lv_3 2024-05-15T23:00:56.925 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:00:58.242 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:57 smithi088 ceph-mon[44699]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:00:58.242 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:57 smithi088 ceph-mon[44699]: Detected new or changed devices on smithi088 2024-05-15T23:00:58.242 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:58.243 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:58.243 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:58.243 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:57 smithi088 ceph-mon[44699]: Adjusting osd_memory_target on smithi088 to 17122M 2024-05-15T23:00:58.243 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:58.243 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:58.243 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:58.243 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:57 smithi090 ceph-mon[44434]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:00:58.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:57 smithi090 ceph-mon[44434]: Detected new or changed devices on smithi088 2024-05-15T23:00:58.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:58.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:58.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:58.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:57 smithi090 ceph-mon[44434]: Adjusting osd_memory_target on smithi088 to 17122M 2024-05-15T23:00:58.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:58.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:58.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:58.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:58.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:57 smithi084 ceph-mon[39784]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:00:58.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:57 smithi084 ceph-mon[39784]: Detected new or changed devices on smithi088 2024-05-15T23:00:58.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:58.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:58.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:00:58.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:57 smithi084 ceph-mon[39784]: Adjusting osd_memory_target on smithi088 to 17122M 2024-05-15T23:00:58.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:00:58.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:00:58.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:00:58.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:01:00.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:59 smithi090 ceph-mon[44434]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:00.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:59 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:01:00.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:59 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:01:00.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:00:59 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:59 smithi088 ceph-mon[44699]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:59 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:01:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:59 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:01:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:00:59 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:00.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:59 smithi084 ceph-mon[39784]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:00.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:59 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:01:00.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:59 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:01:00.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:00:59 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:00 smithi090 ceph-mon[44434]: from='client.34256 v1:172.21.15.88:0/2488602949' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:01:01.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:00 smithi088 ceph-mon[44699]: from='client.34256 v1:172.21.15.88:0/2488602949' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:01:01.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:00 smithi084 ceph-mon[39784]: from='client.34256 v1:172.21.15.88:0/2488602949' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:01:02.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:01 smithi088 ceph-mon[44699]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:01 smithi090 ceph-mon[44434]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:02.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:01 smithi084 ceph-mon[39784]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.88:0/3913191225' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20"}]: dispatch 2024-05-15T23:01:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:02 smithi090 ceph-mon[44434]: from='client.34262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20"}]: dispatch 2024-05-15T23:01:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:02 smithi090 ceph-mon[44434]: from='client.34262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20"}]': finished 2024-05-15T23:01:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:02 smithi090 ceph-mon[44434]: osdmap e33: 6 total, 5 up, 6 in 2024-05-15T23:01:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:02 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.88:0/3913191225' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20"}]: dispatch 2024-05-15T23:01:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:02 smithi088 ceph-mon[44699]: from='client.34262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20"}]: dispatch 2024-05-15T23:01:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:02 smithi088 ceph-mon[44699]: from='client.34262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20"}]': finished 2024-05-15T23:01:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:02 smithi088 ceph-mon[44699]: osdmap e33: 6 total, 5 up, 6 in 2024-05-15T23:01:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:02 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:03.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.88:0/3913191225' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20"}]: dispatch 2024-05-15T23:01:03.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:02 smithi084 ceph-mon[39784]: from='client.34262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20"}]: dispatch 2024-05-15T23:01:03.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:02 smithi084 ceph-mon[39784]: from='client.34262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20"}]': finished 2024-05-15T23:01:03.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:02 smithi084 ceph-mon[39784]: osdmap e33: 6 total, 5 up, 6 in 2024-05-15T23:01:03.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:03 smithi090 ceph-mon[44434]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.88:0/3908617213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:01:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:03 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:01:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:03 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:01:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:03 smithi088 ceph-mon[44699]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.88:0/3908617213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:01:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:03 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:01:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:03 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:01:04.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:03 smithi084 ceph-mon[39784]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:04.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.88:0/3908617213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:01:04.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:03 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:01:04.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:03 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:01:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:04 smithi090 ceph-mon[44434]: osdmap e34: 6 total, 5 up, 6 in 2024-05-15T23:01:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:04 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:04 smithi088 ceph-mon[44699]: osdmap e34: 6 total, 5 up, 6 in 2024-05-15T23:01:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:04 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:05.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:04 smithi084 ceph-mon[39784]: osdmap e34: 6 total, 5 up, 6 in 2024-05-15T23:01:05.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:04 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:06.118 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:05 smithi088 ceph-mon[44699]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:05 smithi090 ceph-mon[44434]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:06.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:05 smithi084 ceph-mon[39784]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:07 smithi090 ceph-mon[44434]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:07 smithi088 ceph-mon[44699]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:07 smithi084 ceph-mon[39784]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:08 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:01:09.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:08 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:01:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:08 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:01:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:08 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:01:09.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:08 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:01:09.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:08 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:01:10.240 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:09 smithi088 ceph-mon[44699]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:10.240 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:09 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-15T23:01:10.240 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:09 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:10.240 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:09 smithi088 ceph-mon[44699]: Deploying daemon osd.5 on smithi088 2024-05-15T23:01:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:09 smithi090 ceph-mon[44434]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:09 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-15T23:01:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:09 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:09 smithi090 ceph-mon[44434]: Deploying daemon osd.5 on smithi088 2024-05-15T23:01:10.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:09 smithi084 ceph-mon[39784]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:10.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:09 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-15T23:01:10.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:09 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:10.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:09 smithi084 ceph-mon[39784]: Deploying daemon osd.5 on smithi088 2024-05-15T23:01:12.246 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:11 smithi088 ceph-mon[44699]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:12.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:11 smithi090 ceph-mon[44434]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:11 smithi084 ceph-mon[39784]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:14.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:13 smithi088 ceph-mon[44699]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:13 smithi090 ceph-mon[44434]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:14.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:13 smithi084 ceph-mon[39784]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:15.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:14 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:15.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:14 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:01:15.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:14 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:01:15.319 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:14 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:15.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:14 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:01:15.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:14 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:01:15.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:14 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:15.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:14 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:01:15.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:14 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:01:16.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:15 smithi090 ceph-mon[44434]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:16.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:15 smithi088 ceph-mon[44699]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:16.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:15 smithi084 ceph-mon[39784]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:17.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:01:17.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:01:17.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:01:17.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:01:17.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:01:17.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:01:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:01:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:01:17.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:01:18.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:18 smithi088 ceph-mon[44699]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:18 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:18 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:18 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:18 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:18.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:18 smithi088 ceph-mon[44699]: from='osd.5 v1:172.21.15.88:6804/661041538' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T23:01:18.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:18 smithi088 ceph-mon[44699]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T23:01:18.426 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 5 on host 'smithi088' 2024-05-15T23:01:18.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:18 smithi090 ceph-mon[44434]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:18.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:18 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:18.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:18 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:18.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:18.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:18.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:18 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:18.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:18.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:18.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:18.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:18 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:18.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:18 smithi090 ceph-mon[44434]: from='osd.5 v1:172.21.15.88:6804/661041538' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T23:01:18.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:18 smithi090 ceph-mon[44434]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T23:01:18.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:18 smithi084 ceph-mon[39784]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:18.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:18 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:18.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:18 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:18.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:18.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:18.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:18 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:18.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:18.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:18.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:18.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:18 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:18.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:18 smithi084 ceph-mon[39784]: from='osd.5 v1:172.21.15.88:6804/661041538' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T23:01:18.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:18 smithi084 ceph-mon[39784]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T23:01:19.158 DEBUG:teuthology.orchestra.run.smithi088:osd.5> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.5.service 2024-05-15T23:01:19.160 INFO:tasks.cephadm:Deploying osd.6 on smithi088 with /dev/vg_nvme/lv_2... 2024-05-15T23:01:19.160 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-15T23:01:19.192 INFO:journalctl@ceph.osd.5.smithi088.stdout:-- Logs begin at Wed 2024-05-15 22:43:48 UTC. -- 2024-05-15T23:01:19.396 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:19 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:19.396 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:19 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:19.396 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:19 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:19.396 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:19 smithi088 ceph-mon[44699]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-15T23:01:19.397 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:19 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:19.397 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:19 smithi088 ceph-mon[44699]: from='osd.5 v1:172.21.15.88:6804/661041538' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:01:19.397 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:19 smithi088 ceph-mon[44699]: osdmap e35: 6 total, 5 up, 6 in 2024-05-15T23:01:19.397 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:19 smithi088 ceph-mon[44699]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:01:19.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:19 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:19 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:19 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:19 smithi090 ceph-mon[44434]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-15T23:01:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:19 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:19 smithi090 ceph-mon[44434]: from='osd.5 v1:172.21.15.88:6804/661041538' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:01:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:19 smithi090 ceph-mon[44434]: osdmap e35: 6 total, 5 up, 6 in 2024-05-15T23:01:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:19 smithi090 ceph-mon[44434]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:01:19.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:19 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:19.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:19 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:19.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:19.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:19 smithi084 ceph-mon[39784]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-15T23:01:19.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:19.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:19 smithi084 ceph-mon[39784]: from='osd.5 v1:172.21.15.88:6804/661041538' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:01:19.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:19 smithi084 ceph-mon[39784]: osdmap e35: 6 total, 5 up, 6 in 2024-05-15T23:01:19.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:19 smithi084 ceph-mon[39784]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:01:19.993 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:01:20.321 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:01:19 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[57847]: 2024-05-15T23:01:19.983+0000 7f0a03e14700 -1 osd.5 0 waiting for initial osdmap 2024-05-15T23:01:20.321 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:01:19 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[57847]: 2024-05-15T23:01:19.988+0000 7f09fbbff700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T23:01:20.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:20 smithi088 ceph-mon[44699]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:20.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:20 smithi088 ceph-mon[44699]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-05-15T23:01:20.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:20 smithi088 ceph-mon[44699]: osdmap e36: 6 total, 5 up, 6 in 2024-05-15T23:01:20.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:20 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:20.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:20 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:20.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:20 smithi088 ceph-mon[44699]: from='osd.5 ' entity='osd.5' 2024-05-15T23:01:20.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:20 smithi090 ceph-mon[44434]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:20.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:20 smithi090 ceph-mon[44434]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-05-15T23:01:20.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:20 smithi090 ceph-mon[44434]: osdmap e36: 6 total, 5 up, 6 in 2024-05-15T23:01:20.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:20 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:20.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:20 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:20.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:20 smithi090 ceph-mon[44434]: from='osd.5 ' entity='osd.5' 2024-05-15T23:01:20.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:20 smithi084 ceph-mon[39784]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:20.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:20 smithi084 ceph-mon[39784]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-05-15T23:01:20.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:20 smithi084 ceph-mon[39784]: osdmap e36: 6 total, 5 up, 6 in 2024-05-15T23:01:20.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:20 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:20.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:20 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:20.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:20 smithi084 ceph-mon[39784]: from='osd.5 ' entity='osd.5' 2024-05-15T23:01:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:21 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:01:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:21 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:01:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:01:21.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:01:21.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:21 smithi088 ceph-mon[44699]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:21.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:21 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:21.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:21 smithi088 ceph-mon[44699]: osd.5 v1:172.21.15.88:6804/661041538 boot 2024-05-15T23:01:21.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:21 smithi088 ceph-mon[44699]: osdmap e37: 6 total, 6 up, 6 in 2024-05-15T23:01:21.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:21 smithi088 ceph-mon[44699]: osdmap e38: 6 total, 6 up, 6 in 2024-05-15T23:01:21.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:21 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:01:21.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:21 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:01:21.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:01:21.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:01:21.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:21 smithi084 ceph-mon[39784]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:21.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:21 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:21.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:21 smithi084 ceph-mon[39784]: osd.5 v1:172.21.15.88:6804/661041538 boot 2024-05-15T23:01:21.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:21 smithi084 ceph-mon[39784]: osdmap e37: 6 total, 6 up, 6 in 2024-05-15T23:01:21.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:21 smithi084 ceph-mon[39784]: osdmap e38: 6 total, 6 up, 6 in 2024-05-15T23:01:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:21 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:01:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:21 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:01:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:01:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:01:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:21 smithi090 ceph-mon[44434]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-15T23:01:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:21 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:01:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:21 smithi090 ceph-mon[44434]: osd.5 v1:172.21.15.88:6804/661041538 boot 2024-05-15T23:01:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:21 smithi090 ceph-mon[44434]: osdmap e37: 6 total, 6 up, 6 in 2024-05-15T23:01:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:21 smithi090 ceph-mon[44434]: osdmap e38: 6 total, 6 up, 6 in 2024-05-15T23:01:23.033 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-15T23:01:23.033 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-15T23:01:23.033 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2024-05-15T23:01:23.033 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2024-05-15T23:01:23.033 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109701 s, 956 MB/s 2024-05-15T23:01:23.033 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2024-05-15T23:01:23.815 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:23 smithi088 ceph-mon[44699]: pgmap v140: 1 pgs: 1 peering; 0 B data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:23.815 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:23 smithi088 ceph-mon[44699]: osdmap e39: 6 total, 6 up, 6 in 2024-05-15T23:01:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:23 smithi084 ceph-mon[39784]: pgmap v140: 1 pgs: 1 peering; 0 B data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:23 smithi084 ceph-mon[39784]: osdmap e39: 6 total, 6 up, 6 in 2024-05-15T23:01:24.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:23 smithi090 ceph-mon[44434]: pgmap v140: 1 pgs: 1 peering; 0 B data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:24.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:23 smithi090 ceph-mon[44434]: osdmap e39: 6 total, 6 up, 6 in 2024-05-15T23:01:24.335 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch daemon add osd smithi088:vg_nvme/lv_2 2024-05-15T23:01:24.596 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:01:25.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:25.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:25.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:25.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:25.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:25.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:25.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:01:25.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:01:25.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:01:25.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: Adjusting osd_memory_target on smithi088 to 8561M 2024-05-15T23:01:25.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:25.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:25.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:25.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:25.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:25.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:25.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:25.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:24 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:25.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:25.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:25.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:25.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:25.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:25.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:25.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:01:25.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:01:25.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:01:25.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: Adjusting osd_memory_target on smithi088 to 8561M 2024-05-15T23:01:25.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:25.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:25.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:25.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:25.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:25.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:25.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:25.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:24 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:01:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:01:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:01:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: Adjusting osd_memory_target on smithi088 to 8561M 2024-05-15T23:01:25.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:25.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:25.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:25.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:25.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:25.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:25.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:25.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:24 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:26 smithi088 ceph-mon[44699]: pgmap v142: 1 pgs: 1 peering; 0 B data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:26.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:26 smithi090 ceph-mon[44434]: pgmap v142: 1 pgs: 1 peering; 0 B data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:26.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:26 smithi084 ceph-mon[39784]: pgmap v142: 1 pgs: 1 peering; 0 B data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:27.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:27 smithi088 ceph-mon[44699]: from='client.34283 v1:172.21.15.88:0/3200317367' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:01:27.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:01:27.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:01:27.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:27 smithi090 ceph-mon[44434]: from='client.34283 v1:172.21.15.88:0/3200317367' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:01:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:01:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:01:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:27.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:27 smithi084 ceph-mon[39784]: from='client.34283 v1:172.21.15.88:0/3200317367' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:01:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:01:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:01:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:28.441 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:28 smithi088 ceph-mon[44699]: pgmap v143: 1 pgs: 1 peering; 0 B data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:28.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:28 smithi090 ceph-mon[44434]: pgmap v143: 1 pgs: 1 peering; 0 B data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:28.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:28 smithi084 ceph-mon[39784]: pgmap v143: 1 pgs: 1 peering; 0 B data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:30 smithi090 ceph-mon[44434]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:30 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.88:0/3678574307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f"}]: dispatch 2024-05-15T23:01:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:30 smithi090 ceph-mon[44434]: from='client.34289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f"}]: dispatch 2024-05-15T23:01:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:30 smithi090 ceph-mon[44434]: from='client.34289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f"}]': finished 2024-05-15T23:01:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:30 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:30 smithi090 ceph-mon[44434]: osdmap e40: 7 total, 6 up, 7 in 2024-05-15T23:01:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:30 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.88:0/1312664089' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:01:30.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:30 smithi088 ceph-mon[44699]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:30.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:30 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.88:0/3678574307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f"}]: dispatch 2024-05-15T23:01:30.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:30 smithi088 ceph-mon[44699]: from='client.34289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f"}]: dispatch 2024-05-15T23:01:30.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:30 smithi088 ceph-mon[44699]: from='client.34289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f"}]': finished 2024-05-15T23:01:30.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:30 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:30.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:30 smithi088 ceph-mon[44699]: osdmap e40: 7 total, 6 up, 7 in 2024-05-15T23:01:30.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:30 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.88:0/1312664089' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:01:30.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:30 smithi084 ceph-mon[39784]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:30 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.88:0/3678574307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f"}]: dispatch 2024-05-15T23:01:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:30 smithi084 ceph-mon[39784]: from='client.34289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f"}]: dispatch 2024-05-15T23:01:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:30 smithi084 ceph-mon[39784]: from='client.34289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f"}]': finished 2024-05-15T23:01:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:30 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:30 smithi084 ceph-mon[39784]: osdmap e40: 7 total, 6 up, 7 in 2024-05-15T23:01:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:30 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.88:0/1312664089' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:01:32.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:32 smithi090 ceph-mon[44434]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:32.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:32 smithi088 ceph-mon[44699]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:32.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:32 smithi084 ceph-mon[39784]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:33.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:33 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:01:33.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:33 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:01:33.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:33 smithi088 ceph-mon[44699]: osdmap e41: 7 total, 6 up, 7 in 2024-05-15T23:01:33.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:33.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:33 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:01:33.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:33 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:01:33.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:33 smithi090 ceph-mon[44434]: osdmap e41: 7 total, 6 up, 7 in 2024-05-15T23:01:33.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:33 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:01:33.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:33 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:01:33.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:33 smithi084 ceph-mon[39784]: osdmap e41: 7 total, 6 up, 7 in 2024-05-15T23:01:33.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:34.414 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:34 smithi088 ceph-mon[44699]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:34.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:34 smithi090 ceph-mon[44434]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:34.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:34 smithi084 ceph-mon[39784]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:36.408 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:36 smithi088 ceph-mon[44699]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:36.408 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:36 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:01:36.408 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:36 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:01:36.409 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-15T23:01:36.409 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:36 smithi090 ceph-mon[44434]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:36 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:01:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:36 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:01:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-15T23:01:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:36.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:36 smithi084 ceph-mon[39784]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:36.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:36 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:01:36.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:36 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:01:36.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-15T23:01:36.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:37 smithi088 ceph-mon[44699]: Deploying daemon osd.6 on smithi088 2024-05-15T23:01:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:37 smithi088 ceph-mon[44699]: osdmap e42: 7 total, 6 up, 7 in 2024-05-15T23:01:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:37 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:37 smithi090 ceph-mon[44434]: Deploying daemon osd.6 on smithi088 2024-05-15T23:01:37.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:37 smithi090 ceph-mon[44434]: osdmap e42: 7 total, 6 up, 7 in 2024-05-15T23:01:37.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:37 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:37.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:37 smithi084 ceph-mon[39784]: Deploying daemon osd.6 on smithi088 2024-05-15T23:01:37.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:37 smithi084 ceph-mon[39784]: osdmap e42: 7 total, 6 up, 7 in 2024-05-15T23:01:37.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:37 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:38.521 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:38 smithi088 ceph-mon[44699]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:38 smithi090 ceph-mon[44434]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:38.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:38 smithi084 ceph-mon[39784]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:40.478 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:40 smithi088 ceph-mon[44699]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:40 smithi090 ceph-mon[44434]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:40.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:40 smithi084 ceph-mon[39784]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:42.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:42 smithi090 ceph-mon[44434]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:42.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:42 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:42.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:42 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:42.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:42 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:42.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:42 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:01:42.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:42 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:01:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:42 smithi088 ceph-mon[44699]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:42 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:42 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:42 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:42 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:01:42.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:42 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:01:42.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:42 smithi084 ceph-mon[39784]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:42 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:42 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:42 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:42 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:01:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:42 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:01:43.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:43 smithi090 ceph-mon[44434]: osdmap e43: 7 total, 6 up, 7 in 2024-05-15T23:01:43.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:43 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:43.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:43 smithi088 ceph-mon[44699]: osdmap e43: 7 total, 6 up, 7 in 2024-05-15T23:01:43.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:43 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:43 smithi084 ceph-mon[39784]: osdmap e43: 7 total, 6 up, 7 in 2024-05-15T23:01:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:43 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:44.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:44 smithi090 ceph-mon[44434]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:44.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:44 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:44.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:44 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:44 smithi088 ceph-mon[44699]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:44 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:44 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:44.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:44 smithi084 ceph-mon[39784]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:44.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:44 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:44.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:44 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:45.324 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 6 on host 'smithi088' 2024-05-15T23:01:45.686 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:45 smithi088 ceph-mon[44699]: from='osd.6 v1:172.21.15.88:6808/380856869' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T23:01:45.686 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:45 smithi088 ceph-mon[44699]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T23:01:45.686 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:45 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:45.686 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:45 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:45.686 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:45 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:45.686 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:45 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:45.686 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:45 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:45.686 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:45 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:45 smithi090 ceph-mon[44434]: from='osd.6 v1:172.21.15.88:6808/380856869' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T23:01:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:45 smithi090 ceph-mon[44434]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T23:01:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:45 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:45 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:45 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:45 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:45 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:45 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:45.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:45 smithi084 ceph-mon[39784]: from='osd.6 v1:172.21.15.88:6808/380856869' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T23:01:45.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:45 smithi084 ceph-mon[39784]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T23:01:45.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:45 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:45.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:45 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:45.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:45 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:45.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:45 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:45.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:45 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:45.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:45 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:01:46.049 DEBUG:teuthology.orchestra.run.smithi088:osd.6> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.6.service 2024-05-15T23:01:46.052 INFO:tasks.cephadm:Deploying osd.7 on smithi088 with /dev/vg_nvme/lv_1... 2024-05-15T23:01:46.052 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-15T23:01:46.081 INFO:journalctl@ceph.osd.6.smithi088.stdout:-- Logs begin at Wed 2024-05-15 22:43:48 UTC. -- 2024-05-15T23:01:46.571 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:01:46 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:01:46.463+0000 7ff5c8985700 -1 osd.6 0 waiting for initial osdmap 2024-05-15T23:01:46.571 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:01:46 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:01:46.470+0000 7ff5c4778700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T23:01:46.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:46 smithi088 ceph-mon[44699]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:46.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:46 smithi088 ceph-mon[44699]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-15T23:01:46.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:46 smithi088 ceph-mon[44699]: osdmap e44: 7 total, 6 up, 7 in 2024-05-15T23:01:46.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:46 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:46.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:46 smithi088 ceph-mon[44699]: from='osd.6 v1:172.21.15.88:6808/380856869' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:01:46.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:46 smithi088 ceph-mon[44699]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:01:46.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:46 smithi090 ceph-mon[44434]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:46 smithi090 ceph-mon[44434]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-15T23:01:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:46 smithi090 ceph-mon[44434]: osdmap e44: 7 total, 6 up, 7 in 2024-05-15T23:01:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:46 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:46 smithi090 ceph-mon[44434]: from='osd.6 v1:172.21.15.88:6808/380856869' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:01:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:46 smithi090 ceph-mon[44434]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:01:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:01:46.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:46 smithi084 ceph-mon[39784]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:46.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:46 smithi084 ceph-mon[39784]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-15T23:01:46.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:46 smithi084 ceph-mon[39784]: osdmap e44: 7 total, 6 up, 7 in 2024-05-15T23:01:46.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:46 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:46.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:46 smithi084 ceph-mon[39784]: from='osd.6 v1:172.21.15.88:6808/380856869' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:01:46.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:46 smithi084 ceph-mon[39784]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:01:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:47 smithi088 ceph-mon[44699]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-05-15T23:01:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:47 smithi088 ceph-mon[44699]: osdmap e45: 7 total, 6 up, 7 in 2024-05-15T23:01:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:47 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:47 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:47 smithi088 ceph-mon[44699]: osd.6 v1:172.21.15.88:6808/380856869 boot 2024-05-15T23:01:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:47 smithi088 ceph-mon[44699]: osdmap e46: 7 total, 7 up, 7 in 2024-05-15T23:01:47.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:47 smithi090 ceph-mon[44434]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-05-15T23:01:47.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:47 smithi090 ceph-mon[44434]: osdmap e45: 7 total, 6 up, 7 in 2024-05-15T23:01:47.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:47 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:47.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:47 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:47.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:47 smithi090 ceph-mon[44434]: osd.6 v1:172.21.15.88:6808/380856869 boot 2024-05-15T23:01:47.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:47 smithi090 ceph-mon[44434]: osdmap e46: 7 total, 7 up, 7 in 2024-05-15T23:01:47.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:47 smithi084 ceph-mon[39784]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-05-15T23:01:47.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:47 smithi084 ceph-mon[39784]: osdmap e45: 7 total, 6 up, 7 in 2024-05-15T23:01:47.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:47 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:47.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:47 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T23:01:47.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:47 smithi084 ceph-mon[39784]: osd.6 v1:172.21.15.88:6808/380856869 boot 2024-05-15T23:01:47.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:47 smithi084 ceph-mon[39784]: osdmap e46: 7 total, 7 up, 7 in 2024-05-15T23:01:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:48 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:01:48.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:48 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:01:48.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:01:48.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:48 smithi090 ceph-mon[44434]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:48.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:48 smithi090 ceph-mon[44434]: osdmap e47: 7 total, 7 up, 7 in 2024-05-15T23:01:48.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:48 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:01:48.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:48 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:01:48.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:01:48.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:48 smithi088 ceph-mon[44699]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:48.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:48 smithi088 ceph-mon[44699]: osdmap e47: 7 total, 7 up, 7 in 2024-05-15T23:01:48.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:48 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:01:48.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:48 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:01:48.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:01:48.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:48 smithi084 ceph-mon[39784]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-05-15T23:01:48.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:48 smithi084 ceph-mon[39784]: osdmap e47: 7 total, 7 up, 7 in 2024-05-15T23:01:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:49 smithi088 ceph-mon[44699]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:49 smithi088 ceph-mon[44699]: osdmap e48: 7 total, 7 up, 7 in 2024-05-15T23:01:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:49 smithi084 ceph-mon[39784]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:49 smithi084 ceph-mon[39784]: osdmap e48: 7 total, 7 up, 7 in 2024-05-15T23:01:50.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:49 smithi090 ceph-mon[44434]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:50.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:49 smithi090 ceph-mon[44434]: osdmap e48: 7 total, 7 up, 7 in 2024-05-15T23:01:50.547 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-15T23:01:50.547 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-15T23:01:50.547 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2024-05-15T23:01:50.547 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2024-05-15T23:01:50.547 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111505 s, 940 MB/s 2024-05-15T23:01:50.548 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2024-05-15T23:01:51.261 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:51 smithi088 ceph-mon[44699]: Detected new or changed devices on smithi088 2024-05-15T23:01:51.261 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:51 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:51.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:51 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:51.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:51 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:51 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:51 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:51 smithi088 ceph-mon[44699]: Adjusting osd_memory_target on smithi088 to 5707M 2024-05-15T23:01:51.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:51 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:51.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:51.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:51 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:51.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:51 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:51.289 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch daemon add osd smithi088:vg_nvme/lv_1 2024-05-15T23:01:51.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:51 smithi090 ceph-mon[44434]: Detected new or changed devices on smithi088 2024-05-15T23:01:51.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:51 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:51.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:51 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:51.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:51 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:51 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:51 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:51 smithi090 ceph-mon[44434]: Adjusting osd_memory_target on smithi088 to 5707M 2024-05-15T23:01:51.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:51 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:51.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:51.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:51 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:51.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:51 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:51 smithi084 ceph-mon[39784]: Detected new or changed devices on smithi088 2024-05-15T23:01:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:51 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:51 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:51 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:51 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:51 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:01:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:51 smithi084 ceph-mon[39784]: Adjusting osd_memory_target on smithi088 to 5707M 2024-05-15T23:01:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:51 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:51.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:51 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:01:51.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:51 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:01:51.553 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:01:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:52 smithi088 ceph-mon[44699]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:52.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:52 smithi084 ceph-mon[39784]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:52 smithi090 ceph-mon[44434]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:54 smithi088 ceph-mon[44699]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:01:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:01:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:54.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:54 smithi084 ceph-mon[39784]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:54.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:01:54.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:01:54.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:54 smithi090 ceph-mon[44434]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:01:54.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:01:54.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:01:55.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:55 smithi088 ceph-mon[44699]: from='client.34310 v1:172.21.15.88:0/118734903' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:01:55.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:55 smithi084 ceph-mon[39784]: from='client.34310 v1:172.21.15.88:0/118734903' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:01:55.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:55 smithi090 ceph-mon[44434]: from='client.34310 v1:172.21.15.88:0/118734903' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:01:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:56 smithi088 ceph-mon[44699]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:56.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:56 smithi084 ceph-mon[39784]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:56.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:56 smithi090 ceph-mon[44434]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:57.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.88:0/2815131512' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34"}]: dispatch 2024-05-15T23:01:57.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:57 smithi088 ceph-mon[44699]: from='client.34316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34"}]: dispatch 2024-05-15T23:01:57.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:57 smithi088 ceph-mon[44699]: from='client.34316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34"}]': finished 2024-05-15T23:01:57.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:01:57.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:57 smithi088 ceph-mon[44699]: osdmap e49: 8 total, 7 up, 8 in 2024-05-15T23:01:57.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.88:0/954832904' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:01:57.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.88:0/2815131512' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34"}]: dispatch 2024-05-15T23:01:57.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:57 smithi084 ceph-mon[39784]: from='client.34316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34"}]: dispatch 2024-05-15T23:01:57.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:57 smithi084 ceph-mon[39784]: from='client.34316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34"}]': finished 2024-05-15T23:01:57.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:01:57.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:57 smithi084 ceph-mon[39784]: osdmap e49: 8 total, 7 up, 8 in 2024-05-15T23:01:57.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.88:0/954832904' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:01:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.88:0/2815131512' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34"}]: dispatch 2024-05-15T23:01:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:57 smithi090 ceph-mon[44434]: from='client.34316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34"}]: dispatch 2024-05-15T23:01:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:57 smithi090 ceph-mon[44434]: from='client.34316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34"}]': finished 2024-05-15T23:01:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:01:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:57 smithi090 ceph-mon[44434]: osdmap e49: 8 total, 7 up, 8 in 2024-05-15T23:01:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.88:0/954832904' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:01:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:58 smithi088 ceph-mon[44699]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:58.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:58 smithi084 ceph-mon[39784]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:58.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:58 smithi090 ceph-mon[44434]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:59.793 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:59 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:01:59.794 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:59 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:01:59.794 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:59 smithi088 ceph-mon[44699]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:59.794 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:59 smithi088 ceph-mon[44699]: osdmap e50: 8 total, 7 up, 8 in 2024-05-15T23:01:59.794 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:01:59 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:01:59.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:59 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:01:59.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:59 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:01:59.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:59 smithi084 ceph-mon[39784]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:01:59.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:59 smithi084 ceph-mon[39784]: osdmap e50: 8 total, 7 up, 8 in 2024-05-15T23:01:59.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:01:59 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:02:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:59 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:02:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:59 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:02:00.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:59 smithi090 ceph-mon[44434]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:00.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:59 smithi090 ceph-mon[44434]: osdmap e50: 8 total, 7 up, 8 in 2024-05-15T23:02:00.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:01:59 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:02:01.706 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:01 smithi088 ceph-mon[44699]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:01.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:01 smithi084 ceph-mon[39784]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:02.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:01 smithi090 ceph-mon[44434]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:03.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:03 smithi088 ceph-mon[44699]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:03.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-15T23:02:03.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:03.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:03 smithi088 ceph-mon[44699]: Deploying daemon osd.7 on smithi088 2024-05-15T23:02:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:03 smithi084 ceph-mon[39784]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-15T23:02:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:03 smithi084 ceph-mon[39784]: Deploying daemon osd.7 on smithi088 2024-05-15T23:02:04.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:03 smithi090 ceph-mon[44434]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:04.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-15T23:02:04.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:04.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:03 smithi090 ceph-mon[44434]: Deploying daemon osd.7 on smithi088 2024-05-15T23:02:05.673 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:05 smithi088 ceph-mon[44699]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:05 smithi084 ceph-mon[39784]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:06.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:05 smithi090 ceph-mon[44434]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:07.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:07 smithi088 ceph-mon[44699]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:07.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:07 smithi084 ceph-mon[39784]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:08.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:07 smithi090 ceph-mon[44434]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:10.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:09 smithi090 ceph-mon[44434]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:10.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:09 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:10.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:09 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:10.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:09 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:10.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:09 smithi088 ceph-mon[44699]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:10.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:09 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:10.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:09 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:10.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:09 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:09 smithi084 ceph-mon[39784]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:09 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:09 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:09 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:12.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:11 smithi090 ceph-mon[44434]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:11 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:11 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:11 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:11 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:02:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:11 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:11 smithi088 ceph-mon[44699]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:11 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:11 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:11 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:11 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:02:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:11 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:12.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:11 smithi084 ceph-mon[39784]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:12.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:11 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:12.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:11 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:12.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:11 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:12.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:11 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:02:12.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:11 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:12.546 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 7 on host 'smithi088' 2024-05-15T23:02:13.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:12 smithi090 ceph-mon[44434]: from='osd.7 v1:172.21.15.88:6812/1044709063' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T23:02:13.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:12 smithi090 ceph-mon[44434]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T23:02:13.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:12 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:13.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:12 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:13.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:13.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:12 smithi088 ceph-mon[44699]: from='osd.7 v1:172.21.15.88:6812/1044709063' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T23:02:13.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:12 smithi088 ceph-mon[44699]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T23:02:13.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:12 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:13.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:12 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:13.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:13.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:12 smithi084 ceph-mon[39784]: from='osd.7 v1:172.21.15.88:6812/1044709063' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T23:02:13.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:12 smithi084 ceph-mon[39784]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T23:02:13.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:12 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:13.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:12 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:13.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:13.307 DEBUG:teuthology.orchestra.run.smithi088:osd.7> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.7.service 2024-05-15T23:02:13.309 INFO:tasks.cephadm:Deploying osd.8 on smithi090 with /dev/vg_nvme/lv_4... 2024-05-15T23:02:13.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-15T23:02:13.340 INFO:journalctl@ceph.osd.7.smithi088.stdout:-- Logs begin at Wed 2024-05-15 22:43:48 UTC. -- 2024-05-15T23:02:13.577 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:02:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:13 smithi088 ceph-mon[44699]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:13 smithi088 ceph-mon[44699]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-15T23:02:13.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:13 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:02:13.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:13 smithi088 ceph-mon[44699]: osdmap e51: 8 total, 7 up, 8 in 2024-05-15T23:02:13.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:13 smithi088 ceph-mon[44699]: from='osd.7 v1:172.21.15.88:6812/1044709063' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:02:13.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:13 smithi088 ceph-mon[44699]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:02:13.822 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:02:13 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:02:13.778+0000 7f7db21a8700 -1 osd.7 0 waiting for initial osdmap 2024-05-15T23:02:13.822 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:02:13 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:02:13.787+0000 7f7dadf9b700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T23:02:14.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:13 smithi090 ceph-mon[44434]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:14.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:13 smithi090 ceph-mon[44434]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-15T23:02:14.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:13 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:02:14.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:13 smithi090 ceph-mon[44434]: osdmap e51: 8 total, 7 up, 8 in 2024-05-15T23:02:14.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:13 smithi090 ceph-mon[44434]: from='osd.7 v1:172.21.15.88:6812/1044709063' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:02:14.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:13 smithi090 ceph-mon[44434]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:02:14.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:13 smithi084 ceph-mon[39784]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:14.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:13 smithi084 ceph-mon[39784]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-15T23:02:14.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:13 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:02:14.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:13 smithi084 ceph-mon[39784]: osdmap e51: 8 total, 7 up, 8 in 2024-05-15T23:02:14.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:13 smithi084 ceph-mon[39784]: from='osd.7 v1:172.21.15.88:6812/1044709063' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:02:14.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:13 smithi084 ceph-mon[39784]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:02:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:14 smithi088 ceph-mon[44699]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-05-15T23:02:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:14 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:02:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:14 smithi088 ceph-mon[44699]: osdmap e52: 8 total, 7 up, 8 in 2024-05-15T23:02:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:14 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:02:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:14 smithi088 ceph-mon[44699]: osd.7 v1:172.21.15.88:6812/1044709063 boot 2024-05-15T23:02:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:14 smithi088 ceph-mon[44699]: osdmap e53: 8 total, 8 up, 8 in 2024-05-15T23:02:14.937 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:14 smithi090 ceph-mon[44434]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-05-15T23:02:14.937 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:14 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:02:14.937 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:14 smithi090 ceph-mon[44434]: osdmap e52: 8 total, 7 up, 8 in 2024-05-15T23:02:14.937 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:14 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:02:14.937 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:14 smithi090 ceph-mon[44434]: osd.7 v1:172.21.15.88:6812/1044709063 boot 2024-05-15T23:02:14.938 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:14 smithi090 ceph-mon[44434]: osdmap e53: 8 total, 8 up, 8 in 2024-05-15T23:02:15.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:14 smithi084 ceph-mon[39784]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-05-15T23:02:15.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:14 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:02:15.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:14 smithi084 ceph-mon[39784]: osdmap e52: 8 total, 7 up, 8 in 2024-05-15T23:02:15.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:14 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:02:15.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:14 smithi084 ceph-mon[39784]: osd.7 v1:172.21.15.88:6812/1044709063 boot 2024-05-15T23:02:15.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:14 smithi084 ceph-mon[39784]: osdmap e53: 8 total, 8 up, 8 in 2024-05-15T23:02:15.489 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-15T23:02:15.490 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-15T23:02:15.490 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-05-15T23:02:15.490 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-05-15T23:02:15.490 INFO:teuthology.orchestra.run.smithi090.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112113 s, 935 MB/s 2024-05-15T23:02:15.490 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 2024-05-15T23:02:15.490 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-05-15T23:02:16.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:15 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:02:16.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:15 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:02:16.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:02:16.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:15 smithi090 ceph-mon[44434]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:16.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:15 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:02:16.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:15 smithi090 ceph-mon[44434]: osdmap e54: 8 total, 8 up, 8 in 2024-05-15T23:02:16.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:15 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:02:16.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:15 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:02:16.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:02:16.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:15 smithi088 ceph-mon[44699]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:16.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:15 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:02:16.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:15 smithi088 ceph-mon[44699]: osdmap e54: 8 total, 8 up, 8 in 2024-05-15T23:02:16.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:15 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:02:16.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:15 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:02:16.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:02:16.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:15 smithi084 ceph-mon[39784]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-05-15T23:02:16.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:15 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T23:02:16.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:15 smithi084 ceph-mon[39784]: osdmap e54: 8 total, 8 up, 8 in 2024-05-15T23:02:16.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_4 2024-05-15T23:02:16.441 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:02:17.787 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: Detected new or changed devices on smithi088 2024-05-15T23:02:17.787 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:17.787 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:17.787 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.787 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.787 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.787 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.787 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.787 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.788 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.788 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.788 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: Adjusting osd_memory_target on smithi088 to 4280M 2024-05-15T23:02:17.788 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:17.788 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:17.788 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:17.788 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:02:17.788 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:17.788 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:17 smithi090 ceph-mon[44434]: osdmap e55: 8 total, 8 up, 8 in 2024-05-15T23:02:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: Detected new or changed devices on smithi088 2024-05-15T23:02:17.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:17.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:17.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: Adjusting osd_memory_target on smithi088 to 4280M 2024-05-15T23:02:17.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:17.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:17.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:17.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:02:17.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:17.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:17 smithi088 ceph-mon[44699]: osdmap e55: 8 total, 8 up, 8 in 2024-05-15T23:02:17.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: Detected new or changed devices on smithi088 2024-05-15T23:02:17.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:17.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:17.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:17.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: Adjusting osd_memory_target on smithi088 to 4280M 2024-05-15T23:02:17.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:17.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:17.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:17.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:02:17.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:17.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:17 smithi084 ceph-mon[39784]: osdmap e55: 8 total, 8 up, 8 in 2024-05-15T23:02:18.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:18 smithi088 ceph-mon[44699]: from='client.24331 v1:172.21.15.90:0/1711967382' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:02:18.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:02:18.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:02:18.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:18.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:18 smithi084 ceph-mon[39784]: from='client.24331 v1:172.21.15.90:0/1711967382' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:02:18.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:02:18.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:02:18.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:18.955 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:18 smithi090 ceph-mon[44434]: from='client.24331 v1:172.21.15.90:0/1711967382' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:02:18.956 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:02:18.956 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:02:18.956 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:19.716 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:19 smithi090 ceph-mon[44434]: pgmap v185: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:19.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:19 smithi084 ceph-mon[39784]: pgmap v185: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:19 smithi088 ceph-mon[44699]: pgmap v185: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:21 smithi090 ceph-mon[44434]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T23:02:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:21 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.90:0/564028574' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02cab286-d487-4a66-982c-3cf28f55e851"}]: dispatch 2024-05-15T23:02:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:21 smithi090 ceph-mon[44434]: from='client.24337 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02cab286-d487-4a66-982c-3cf28f55e851"}]: dispatch 2024-05-15T23:02:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:21 smithi090 ceph-mon[44434]: from='client.24337 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02cab286-d487-4a66-982c-3cf28f55e851"}]': finished 2024-05-15T23:02:22.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:21 smithi090 ceph-mon[44434]: osdmap e56: 9 total, 8 up, 9 in 2024-05-15T23:02:22.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:21 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:22.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:21 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.90:0/1919102022' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:02:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:21 smithi088 ceph-mon[44699]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T23:02:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:21 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.90:0/564028574' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02cab286-d487-4a66-982c-3cf28f55e851"}]: dispatch 2024-05-15T23:02:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:21 smithi088 ceph-mon[44699]: from='client.24337 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02cab286-d487-4a66-982c-3cf28f55e851"}]: dispatch 2024-05-15T23:02:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:21 smithi088 ceph-mon[44699]: from='client.24337 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02cab286-d487-4a66-982c-3cf28f55e851"}]': finished 2024-05-15T23:02:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:21 smithi088 ceph-mon[44699]: osdmap e56: 9 total, 8 up, 9 in 2024-05-15T23:02:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:21 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:21 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.90:0/1919102022' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:02:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:21 smithi084 ceph-mon[39784]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T23:02:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:21 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.90:0/564028574' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02cab286-d487-4a66-982c-3cf28f55e851"}]: dispatch 2024-05-15T23:02:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:21 smithi084 ceph-mon[39784]: from='client.24337 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02cab286-d487-4a66-982c-3cf28f55e851"}]: dispatch 2024-05-15T23:02:22.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:21 smithi084 ceph-mon[39784]: from='client.24337 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02cab286-d487-4a66-982c-3cf28f55e851"}]': finished 2024-05-15T23:02:22.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:21 smithi084 ceph-mon[39784]: osdmap e56: 9 total, 8 up, 9 in 2024-05-15T23:02:22.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:21 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:22.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:21 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.90:0/1919102022' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:02:24.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:02:24.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:02:24.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:23 smithi090 ceph-mon[44434]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-15T23:02:24.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:23 smithi090 ceph-mon[44434]: osdmap e57: 9 total, 8 up, 9 in 2024-05-15T23:02:24.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:23 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:24.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:02:24.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:02:24.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:23 smithi088 ceph-mon[44699]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-15T23:02:24.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:23 smithi088 ceph-mon[44699]: osdmap e57: 9 total, 8 up, 9 in 2024-05-15T23:02:24.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:23 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:02:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:02:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:23 smithi084 ceph-mon[39784]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-15T23:02:24.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:23 smithi084 ceph-mon[39784]: osdmap e57: 9 total, 8 up, 9 in 2024-05-15T23:02:24.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:23 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:25.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:25 smithi090 ceph-mon[44434]: 1.0 deep-scrub starts 2024-05-15T23:02:25.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:25 smithi090 ceph-mon[44434]: 1.0 deep-scrub ok 2024-05-15T23:02:25.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:25 smithi090 ceph-mon[44434]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-15T23:02:26.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:25 smithi088 ceph-mon[44699]: 1.0 deep-scrub starts 2024-05-15T23:02:26.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:25 smithi088 ceph-mon[44699]: 1.0 deep-scrub ok 2024-05-15T23:02:26.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:25 smithi088 ceph-mon[44699]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-15T23:02:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:25 smithi084 ceph-mon[39784]: 1.0 deep-scrub starts 2024-05-15T23:02:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:25 smithi084 ceph-mon[39784]: 1.0 deep-scrub ok 2024-05-15T23:02:26.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:25 smithi084 ceph-mon[39784]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-15T23:02:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:27 smithi090 ceph-mon[44434]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T23:02:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-15T23:02:28.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:28.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:27 smithi090 ceph-mon[44434]: Deploying daemon osd.8 on smithi090 2024-05-15T23:02:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:27 smithi088 ceph-mon[44699]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T23:02:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-15T23:02:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:27 smithi088 ceph-mon[44699]: Deploying daemon osd.8 on smithi090 2024-05-15T23:02:28.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:27 smithi084 ceph-mon[39784]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T23:02:28.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-15T23:02:28.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:28.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:27 smithi084 ceph-mon[39784]: Deploying daemon osd.8 on smithi090 2024-05-15T23:02:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:28 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:02:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:28 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:02:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:28 smithi090 ceph-mon[44434]: osdmap e58: 9 total, 8 up, 9 in 2024-05-15T23:02:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:28 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:02:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:28 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:02:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:28 smithi088 ceph-mon[44699]: osdmap e58: 9 total, 8 up, 9 in 2024-05-15T23:02:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:28 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:02:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:28 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:02:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:28 smithi084 ceph-mon[39784]: osdmap e58: 9 total, 8 up, 9 in 2024-05-15T23:02:29.799 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:29 smithi090 ceph-mon[44434]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:30.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:29 smithi088 ceph-mon[44699]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:29 smithi084 ceph-mon[39784]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:31.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:31 smithi090 ceph-mon[44434]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:31 smithi088 ceph-mon[44699]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:32.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:31 smithi084 ceph-mon[39784]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:33.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:33 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:33.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:33 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:33.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:33 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:33 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:33.762 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:33 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:33.762 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:33 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:33.762 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:34.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:34 smithi088 ceph-mon[44699]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:34.591 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:34 smithi090 ceph-mon[44434]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:34.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:34 smithi084 ceph-mon[39784]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:35 smithi090 ceph-mon[44434]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:35 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:35 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:35 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:35 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:02:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:35 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:35 smithi090 ceph-mon[44434]: from='osd.8 v1:172.21.15.90:6800/522459126' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T23:02:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:35 smithi090 ceph-mon[44434]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T23:02:35.867 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 8 on host 'smithi090' 2024-05-15T23:02:36.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:35 smithi088 ceph-mon[44699]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:36.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:35 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:36.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:35 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:36.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:35 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:36.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:35 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:02:36.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:35 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:36.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:35 smithi088 ceph-mon[44699]: from='osd.8 v1:172.21.15.90:6800/522459126' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T23:02:36.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:35 smithi088 ceph-mon[44699]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T23:02:36.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:35 smithi084 ceph-mon[39784]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:36.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:35 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:36.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:35 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:36.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:35 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:36.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:35 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:02:36.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:35 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:36.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:35 smithi084 ceph-mon[39784]: from='osd.8 v1:172.21.15.90:6800/522459126' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T23:02:36.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:35 smithi084 ceph-mon[39784]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T23:02:36.607 DEBUG:teuthology.orchestra.run.smithi090:osd.8> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.8.service 2024-05-15T23:02:36.609 INFO:tasks.cephadm:Deploying osd.9 on smithi090 with /dev/vg_nvme/lv_3... 2024-05-15T23:02:36.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-15T23:02:36.638 INFO:journalctl@ceph.osd.8.smithi090.stdout:-- Logs begin at Wed 2024-05-15 22:43:42 UTC. -- 2024-05-15T23:02:37.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:36 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:02:37.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:36 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:02:37.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:36 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:37.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:37.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:36 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:37.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:36 smithi084 ceph-mon[39784]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-15T23:02:37.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:36 smithi084 ceph-mon[39784]: from='osd.8 v1:172.21.15.90:6800/522459126' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:02:37.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:36 smithi084 ceph-mon[39784]: osdmap e59: 9 total, 8 up, 9 in 2024-05-15T23:02:37.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:36 smithi084 ceph-mon[39784]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:02:37.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:37.302 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:02:37.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:36 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:02:37.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:36 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:02:37.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:36 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:37.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:37.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:36 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:37.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:36 smithi090 ceph-mon[44434]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-15T23:02:37.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:36 smithi090 ceph-mon[44434]: from='osd.8 v1:172.21.15.90:6800/522459126' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:02:37.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:36 smithi090 ceph-mon[44434]: osdmap e59: 9 total, 8 up, 9 in 2024-05-15T23:02:37.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:36 smithi090 ceph-mon[44434]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:02:37.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:36 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:02:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:36 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:02:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:36 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:36 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:36 smithi088 ceph-mon[44699]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-15T23:02:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:36 smithi088 ceph-mon[44699]: from='osd.8 v1:172.21.15.90:6800/522459126' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:02:37.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:36 smithi088 ceph-mon[44699]: osdmap e59: 9 total, 8 up, 9 in 2024-05-15T23:02:37.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:36 smithi088 ceph-mon[44699]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:02:37.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:37.756 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:02:37 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[50278]: 2024-05-15T23:02:37.329+0000 7f0ac3e49700 -1 osd.8 0 waiting for initial osdmap 2024-05-15T23:02:37.756 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:02:37 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[50278]: 2024-05-15T23:02:37.337+0000 7f0abfc3c700 -1 osd.8 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T23:02:38.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:38 smithi090 ceph-mon[44434]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:38.543 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:38 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:02:38.543 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:38 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:02:38.543 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:38 smithi090 ceph-mon[44434]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-05-15T23:02:38.543 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:38 smithi090 ceph-mon[44434]: osdmap e60: 9 total, 8 up, 9 in 2024-05-15T23:02:38.543 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:38.543 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:38.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:38 smithi084 ceph-mon[39784]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:38.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:38 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:02:38.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:38 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:02:38.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:38 smithi084 ceph-mon[39784]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-05-15T23:02:38.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:38 smithi084 ceph-mon[39784]: osdmap e60: 9 total, 8 up, 9 in 2024-05-15T23:02:38.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:38.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:38.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:38 smithi088 ceph-mon[44699]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:38.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:38 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:02:38.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:38 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:02:38.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:38 smithi088 ceph-mon[44699]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-05-15T23:02:38.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:38 smithi088 ceph-mon[44699]: osdmap e60: 9 total, 8 up, 9 in 2024-05-15T23:02:38.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:38 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:38.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:38 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:39 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:02:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:39 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:02:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:02:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:39 smithi090 ceph-mon[44434]: osd.8 v1:172.21.15.90:6800/522459126 boot 2024-05-15T23:02:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:39 smithi090 ceph-mon[44434]: osdmap e61: 9 total, 9 up, 9 in 2024-05-15T23:02:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:39.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:39 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:02:39.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:39 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:02:39.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:02:39.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:39 smithi084 ceph-mon[39784]: osd.8 v1:172.21.15.90:6800/522459126 boot 2024-05-15T23:02:39.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:39 smithi084 ceph-mon[39784]: osdmap e61: 9 total, 9 up, 9 in 2024-05-15T23:02:39.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:39 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:02:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:39 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:02:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:02:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:39 smithi088 ceph-mon[44699]: osd.8 v1:172.21.15.90:6800/522459126 boot 2024-05-15T23:02:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:39 smithi088 ceph-mon[44699]: osdmap e61: 9 total, 9 up, 9 in 2024-05-15T23:02:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:39 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:02:39.889 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-15T23:02:39.890 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-15T23:02:39.890 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-05-15T23:02:39.890 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-05-15T23:02:39.890 INFO:teuthology.orchestra.run.smithi090.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114739 s, 914 MB/s 2024-05-15T23:02:39.890 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 2024-05-15T23:02:39.890 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-05-15T23:02:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:40 smithi090 ceph-mon[44434]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:40 smithi090 ceph-mon[44434]: osdmap e62: 9 total, 9 up, 9 in 2024-05-15T23:02:40.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:40 smithi084 ceph-mon[39784]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:40.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:40 smithi084 ceph-mon[39784]: osdmap e62: 9 total, 9 up, 9 in 2024-05-15T23:02:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:40 smithi088 ceph-mon[44699]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-05-15T23:02:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:40 smithi088 ceph-mon[44699]: osdmap e62: 9 total, 9 up, 9 in 2024-05-15T23:02:41.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_3 2024-05-15T23:02:41.472 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:02:41.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:41 smithi084 ceph-mon[39784]: osdmap e63: 9 total, 9 up, 9 in 2024-05-15T23:02:41.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:41 smithi090 ceph-mon[44434]: osdmap e63: 9 total, 9 up, 9 in 2024-05-15T23:02:41.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:41 smithi088 ceph-mon[44699]: osdmap e63: 9 total, 9 up, 9 in 2024-05-15T23:02:42.647 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:02:42.648 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: Detected new or changed devices on smithi090 2024-05-15T23:02:42.648 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:42.648 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:42.648 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:42.648 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:42.648 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: Adjusting osd_memory_target on smithi090 to 21218M 2024-05-15T23:02:42.649 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:42.649 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:42.649 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:02:42.649 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:42.649 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:02:42.649 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:02:42.649 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:02:42.649 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:42.649 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:42.650 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:02:42.650 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:42 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:02:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:02:42.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: Detected new or changed devices on smithi090 2024-05-15T23:02:42.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:42.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:42.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:42.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:42.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: Adjusting osd_memory_target on smithi090 to 21218M 2024-05-15T23:02:42.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:42.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:42.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:02:42.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:42.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:02:42.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:02:42.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:02:42.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:42.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:42.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:02:42.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:42 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:02:42.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:02:42.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: Detected new or changed devices on smithi090 2024-05-15T23:02:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:02:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: Adjusting osd_memory_target on smithi090 to 21218M 2024-05-15T23:02:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:02:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:02:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:02:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:02:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:02:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:42.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:42.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:02:42.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:42 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:02:43.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:43 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:02:43.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:43 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:02:43.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:43 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:43.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:43 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:02:43.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:43 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:02:43.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:43 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:43 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:02:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:43 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:02:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:43 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:44.751 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:44 smithi090 ceph-mon[44434]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:02:44.751 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:44 smithi090 ceph-mon[44434]: from='client.24358 v1:172.21.15.90:0/3670570642' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:02:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:44 smithi088 ceph-mon[44699]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:02:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:44 smithi088 ceph-mon[44699]: from='client.24358 v1:172.21.15.90:0/3670570642' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:02:44.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:44 smithi084 ceph-mon[39784]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:02:44.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:44 smithi084 ceph-mon[39784]: from='client.24358 v1:172.21.15.90:0/3670570642' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:02:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:46 smithi090 ceph-mon[44434]: pgmap v206: 1 pgs: 1 peering; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:02:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:46 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.90:0/2716078113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613"}]: dispatch 2024-05-15T23:02:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:46 smithi090 ceph-mon[44434]: from='client.24364 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613"}]: dispatch 2024-05-15T23:02:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:46 smithi090 ceph-mon[44434]: from='client.24364 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613"}]': finished 2024-05-15T23:02:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:46 smithi090 ceph-mon[44434]: osdmap e64: 10 total, 9 up, 10 in 2024-05-15T23:02:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:46 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:02:46.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:46 smithi088 ceph-mon[44699]: pgmap v206: 1 pgs: 1 peering; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:02:46.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:46 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.90:0/2716078113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613"}]: dispatch 2024-05-15T23:02:46.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:46 smithi088 ceph-mon[44699]: from='client.24364 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613"}]: dispatch 2024-05-15T23:02:46.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:46 smithi088 ceph-mon[44699]: from='client.24364 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613"}]': finished 2024-05-15T23:02:46.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:46 smithi088 ceph-mon[44699]: osdmap e64: 10 total, 9 up, 10 in 2024-05-15T23:02:46.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:46 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:02:46.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:46 smithi084 ceph-mon[39784]: pgmap v206: 1 pgs: 1 peering; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:02:46.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:46 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.90:0/2716078113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613"}]: dispatch 2024-05-15T23:02:46.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:46 smithi084 ceph-mon[39784]: from='client.24364 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613"}]: dispatch 2024-05-15T23:02:46.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:46 smithi084 ceph-mon[39784]: from='client.24364 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613"}]': finished 2024-05-15T23:02:46.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:46 smithi084 ceph-mon[39784]: osdmap e64: 10 total, 9 up, 10 in 2024-05-15T23:02:46.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:46 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:02:47.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:47 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.90:0/989099410' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:02:47.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:47 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.90:0/989099410' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:02:47.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:47 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.90:0/989099410' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:02:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:48 smithi090 ceph-mon[44434]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-05-15T23:02:48.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:48 smithi088 ceph-mon[44699]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-05-15T23:02:48.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:48 smithi084 ceph-mon[39784]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-05-15T23:02:50.772 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:50 smithi090 ceph-mon[44434]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-05-15T23:02:50.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:50 smithi088 ceph-mon[44699]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-05-15T23:02:50.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:50 smithi084 ceph-mon[39784]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-05-15T23:02:52.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:52 smithi090 ceph-mon[44434]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-15T23:02:52.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:52 smithi088 ceph-mon[44699]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-15T23:02:52.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:52 smithi084 ceph-mon[39784]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-15T23:02:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-15T23:02:53.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:53.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:53 smithi090 ceph-mon[44434]: Deploying daemon osd.9 on smithi090 2024-05-15T23:02:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-15T23:02:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:53 smithi088 ceph-mon[44699]: Deploying daemon osd.9 on smithi090 2024-05-15T23:02:53.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-15T23:02:53.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:02:53.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:53 smithi084 ceph-mon[39784]: Deploying daemon osd.9 on smithi090 2024-05-15T23:02:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:54 smithi090 ceph-mon[44434]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-15T23:02:54.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:54 smithi088 ceph-mon[44699]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-15T23:02:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:54 smithi084 ceph-mon[39784]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-15T23:02:55.666 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:55 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:02:55.666 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:55 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:02:55.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:55 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:02:55.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:55 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:02:55.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:55 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:02:55.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:55 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:02:56.717 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:56 smithi090 ceph-mon[44434]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-15T23:02:56.717 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:56 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:02:56.717 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:56 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:02:56.717 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:56 smithi090 ceph-mon[44434]: osdmap e65: 10 total, 9 up, 10 in 2024-05-15T23:02:56.717 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:56 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:02:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:56 smithi088 ceph-mon[44699]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-15T23:02:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:56 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:02:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:56 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:02:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:56 smithi088 ceph-mon[44699]: osdmap e65: 10 total, 9 up, 10 in 2024-05-15T23:02:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:56 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:02:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:56 smithi084 ceph-mon[39784]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-15T23:02:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:56 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:02:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:56 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:02:56.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:56 smithi084 ceph-mon[39784]: osdmap e65: 10 total, 9 up, 10 in 2024-05-15T23:02:56.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:56 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:02:57.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:57 smithi084 ceph-mon[39784]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:02:57.915 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:57 smithi090 ceph-mon[44434]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:02:58.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:57 smithi088 ceph-mon[44699]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:02:59.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:58 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:02:59.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:58 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:02:59.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:58 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:59.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:58 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:02:59.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:58 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:02:59.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:58 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:02:59.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:58 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:02:59.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:58 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:02:59.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:58 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:00.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:02:59 smithi090 ceph-mon[44434]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:03:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:02:59 smithi088 ceph-mon[44699]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:03:00.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:02:59 smithi084 ceph-mon[39784]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:03:01.270 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 9 on host 'smithi090' 2024-05-15T23:03:01.887 DEBUG:teuthology.orchestra.run.smithi090:osd.9> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.9.service 2024-05-15T23:03:01.890 INFO:tasks.cephadm:Deploying osd.10 on smithi090 with /dev/vg_nvme/lv_2... 2024-05-15T23:03:01.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-15T23:03:01.923 INFO:journalctl@ceph.osd.9.smithi090.stdout:-- Logs begin at Wed 2024-05-15 22:43:42 UTC. -- 2024-05-15T23:03:02.201 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:01 smithi090 ceph-mon[44434]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:03:02.201 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:01 smithi090 ceph-mon[44434]: from='osd.9 v1:172.21.15.90:6804/2794571133' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-15T23:03:02.202 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:01 smithi090 ceph-mon[44434]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-15T23:03:02.202 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:02.202 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:02.202 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:02.202 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:01 smithi088 ceph-mon[44699]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:03:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:01 smithi088 ceph-mon[44699]: from='osd.9 v1:172.21.15.90:6804/2794571133' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-15T23:03:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:01 smithi088 ceph-mon[44699]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-15T23:03:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:02.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:02.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:01 smithi084 ceph-mon[39784]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:03:02.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:01 smithi084 ceph-mon[39784]: from='osd.9 v1:172.21.15.90:6804/2794571133' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-15T23:03:02.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:01 smithi084 ceph-mon[39784]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-15T23:03:02.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:02.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:02.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:02.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:02.709 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:03:03.267 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:03:02 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:03:02.998+0000 7fd16b027700 -1 osd.9 0 waiting for initial osdmap 2024-05-15T23:03:03.268 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:03:02 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:03:02.998+0000 7fd1627a5700 -1 osd.9 0 failed to load OSD map for epoch 12, got 0 bytes 2024-05-15T23:03:03.268 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:03:03 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:03:03.005+0000 7fd166e1a700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T23:03:03.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:02 smithi090 ceph-mon[44434]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-15T23:03:03.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:02 smithi090 ceph-mon[44434]: osdmap e66: 10 total, 9 up, 10 in 2024-05-15T23:03:03.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:02 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:03:03.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:02 smithi090 ceph-mon[44434]: from='osd.9 v1:172.21.15.90:6804/2794571133' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:03:03.268 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:02 smithi090 ceph-mon[44434]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:03:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:02 smithi088 ceph-mon[44699]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-15T23:03:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:02 smithi088 ceph-mon[44699]: osdmap e66: 10 total, 9 up, 10 in 2024-05-15T23:03:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:02 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:03:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:02 smithi088 ceph-mon[44699]: from='osd.9 v1:172.21.15.90:6804/2794571133' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:03:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:02 smithi088 ceph-mon[44699]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:03:03.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:02 smithi084 ceph-mon[39784]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-15T23:03:03.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:02 smithi084 ceph-mon[39784]: osdmap e66: 10 total, 9 up, 10 in 2024-05-15T23:03:03.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:03:03.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:02 smithi084 ceph-mon[39784]: from='osd.9 v1:172.21.15.90:6804/2794571133' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:03:03.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:02 smithi084 ceph-mon[39784]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:03:04.288 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:03 smithi090 ceph-mon[44434]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:03:04.288 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:03 smithi090 ceph-mon[44434]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-05-15T23:03:04.288 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:03 smithi090 ceph-mon[44434]: osdmap e67: 10 total, 9 up, 10 in 2024-05-15T23:03:04.288 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:03:04.288 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:03:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:03 smithi088 ceph-mon[44699]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:03:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:03 smithi088 ceph-mon[44699]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-05-15T23:03:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:03 smithi088 ceph-mon[44699]: osdmap e67: 10 total, 9 up, 10 in 2024-05-15T23:03:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:03:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:03:04.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:03 smithi084 ceph-mon[39784]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:03:04.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:03 smithi084 ceph-mon[39784]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-05-15T23:03:04.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:03 smithi084 ceph-mon[39784]: osdmap e67: 10 total, 9 up, 10 in 2024-05-15T23:03:04.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:03:04.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:03:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:04 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:03:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:04 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:03:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:03:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:04 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:03:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:04 smithi090 ceph-mon[44434]: osd.9 v1:172.21.15.90:6804/2794571133 boot 2024-05-15T23:03:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:04 smithi090 ceph-mon[44434]: osdmap e68: 10 total, 10 up, 10 in 2024-05-15T23:03:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:04 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:03:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:04 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:03:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:04 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:03:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:03:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:04 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:03:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:04 smithi088 ceph-mon[44699]: osd.9 v1:172.21.15.90:6804/2794571133 boot 2024-05-15T23:03:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:04 smithi088 ceph-mon[44699]: osdmap e68: 10 total, 10 up, 10 in 2024-05-15T23:03:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:04 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:03:05.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:04 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:03:05.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:04 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:03:05.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:03:05.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:04 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:03:05.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:04 smithi084 ceph-mon[39784]: osd.9 v1:172.21.15.90:6804/2794571133 boot 2024-05-15T23:03:05.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:04 smithi084 ceph-mon[39784]: osdmap e68: 10 total, 10 up, 10 in 2024-05-15T23:03:05.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:04 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-15T23:03:05.757 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-15T23:03:05.758 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-15T23:03:05.758 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-05-15T23:03:05.758 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-05-15T23:03:05.758 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128589 s, 815 MB/s 2024-05-15T23:03:05.758 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-05-15T23:03:06.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:06 smithi090 ceph-mon[44434]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:03:06.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:06 smithi090 ceph-mon[44434]: osdmap e69: 10 total, 10 up, 10 in 2024-05-15T23:03:06.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:06.191 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:03:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:03:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:03:06.192 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:06 smithi088 ceph-mon[44699]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:03:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:06 smithi088 ceph-mon[44699]: osdmap e69: 10 total, 10 up, 10 in 2024-05-15T23:03:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:06.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:03:06.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:03:06.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:06.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:06.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:03:06.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:06.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:06 smithi084 ceph-mon[39784]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-05-15T23:03:06.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:06 smithi084 ceph-mon[39784]: osdmap e69: 10 total, 10 up, 10 in 2024-05-15T23:03:06.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:06.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:06.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:03:06.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:03:06.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:06.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:06.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:03:06.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:06.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_2 2024-05-15T23:03:06.740 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:03:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:07 smithi090 ceph-mon[44434]: Detected new or changed devices on smithi090 2024-05-15T23:03:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:07 smithi090 ceph-mon[44434]: Adjusting osd_memory_target on smithi090 to 10609M 2024-05-15T23:03:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:03:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:07 smithi088 ceph-mon[44699]: Detected new or changed devices on smithi090 2024-05-15T23:03:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:07 smithi088 ceph-mon[44699]: Adjusting osd_memory_target on smithi090 to 10609M 2024-05-15T23:03:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:03:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:07.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:07 smithi084 ceph-mon[39784]: Detected new or changed devices on smithi090 2024-05-15T23:03:07.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:07 smithi084 ceph-mon[39784]: Adjusting osd_memory_target on smithi090 to 10609M 2024-05-15T23:03:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:03:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:08.271 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:08 smithi090 ceph-mon[44434]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:08 smithi088 ceph-mon[44699]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:08.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:08 smithi084 ceph-mon[39784]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:09.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:09 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:03:09.281 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:09 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:03:09.281 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:09 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:09 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:03:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:09 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:03:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:09 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:09.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:09 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:03:09.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:09 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:03:09.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:09 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:10.218 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:10 smithi090 ceph-mon[44434]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:10.218 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:10 smithi090 ceph-mon[44434]: from='client.24385 v1:172.21.15.90:0/1750235256' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:03:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:10 smithi088 ceph-mon[44699]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:10 smithi088 ceph-mon[44699]: from='client.24385 v1:172.21.15.90:0/1750235256' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:03:10.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:10 smithi084 ceph-mon[39784]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:10.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:10 smithi084 ceph-mon[39784]: from='client.24385 v1:172.21.15.90:0/1750235256' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:03:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:12 smithi090 ceph-mon[44434]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:12 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.90:0/4190740049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473"}]: dispatch 2024-05-15T23:03:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:12 smithi090 ceph-mon[44434]: from='client.24391 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473"}]: dispatch 2024-05-15T23:03:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:12 smithi090 ceph-mon[44434]: from='client.24391 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473"}]': finished 2024-05-15T23:03:12.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:12 smithi090 ceph-mon[44434]: osdmap e70: 11 total, 10 up, 11 in 2024-05-15T23:03:12.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:12 smithi088 ceph-mon[44699]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:12 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.90:0/4190740049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473"}]: dispatch 2024-05-15T23:03:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:12 smithi088 ceph-mon[44699]: from='client.24391 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473"}]: dispatch 2024-05-15T23:03:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:12 smithi088 ceph-mon[44699]: from='client.24391 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473"}]': finished 2024-05-15T23:03:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:12 smithi088 ceph-mon[44699]: osdmap e70: 11 total, 10 up, 11 in 2024-05-15T23:03:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:12 smithi084 ceph-mon[39784]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:12.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:12 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.90:0/4190740049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473"}]: dispatch 2024-05-15T23:03:12.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:12 smithi084 ceph-mon[39784]: from='client.24391 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473"}]: dispatch 2024-05-15T23:03:12.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:12 smithi084 ceph-mon[39784]: from='client.24391 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473"}]': finished 2024-05-15T23:03:12.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:12 smithi084 ceph-mon[39784]: osdmap e70: 11 total, 10 up, 11 in 2024-05-15T23:03:12.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:13.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:13 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:03:13.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:13 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:03:13.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.90:0/2544705274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:03:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:13 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:03:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:13 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:03:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.90:0/2544705274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:03:13.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:13 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:03:13.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:13 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:03:13.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.90:0/2544705274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:03:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:14 smithi090 ceph-mon[44434]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:14 smithi090 ceph-mon[44434]: osdmap e71: 11 total, 10 up, 11 in 2024-05-15T23:03:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:14 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:14 smithi088 ceph-mon[44699]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:14 smithi088 ceph-mon[44699]: osdmap e71: 11 total, 10 up, 11 in 2024-05-15T23:03:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:14 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:14.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:14 smithi084 ceph-mon[39784]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:14.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:14 smithi084 ceph-mon[39784]: osdmap e71: 11 total, 10 up, 11 in 2024-05-15T23:03:14.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:14 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:16.076 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:16 smithi090 ceph-mon[44434]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:16.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:16 smithi088 ceph-mon[44699]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:16.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:16 smithi084 ceph-mon[39784]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:17.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:17 smithi084 ceph-mon[39784]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:18.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:17 smithi090 ceph-mon[44434]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:17 smithi088 ceph-mon[44699]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:18.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-15T23:03:18.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:18.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:18 smithi090 ceph-mon[44434]: Deploying daemon osd.10 on smithi090 2024-05-15T23:03:18.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-15T23:03:18.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:18.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:18 smithi084 ceph-mon[39784]: Deploying daemon osd.10 on smithi090 2024-05-15T23:03:19.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-15T23:03:19.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:19.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:18 smithi088 ceph-mon[44699]: Deploying daemon osd.10 on smithi090 2024-05-15T23:03:19.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:19 smithi084 ceph-mon[39784]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:20.046 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:19 smithi090 ceph-mon[44434]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:20.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:19 smithi088 ceph-mon[44699]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:21.656 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:21 smithi090 ceph-mon[44434]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:21.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:21 smithi084 ceph-mon[39784]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:21 smithi088 ceph-mon[44699]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:23.847 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:23 smithi090 ceph-mon[44434]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:23 smithi084 ceph-mon[39784]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:24.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:23 smithi088 ceph-mon[44699]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:25.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:25 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:25.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:25 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:25.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:25 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:25 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:25 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:25 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:25.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:25 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:25.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:25 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:25.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:25 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:26.216 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:26 smithi090 ceph-mon[44434]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:26.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:26 smithi088 ceph-mon[44699]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:26.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:26 smithi084 ceph-mon[39784]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:27.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:27 smithi090 ceph-mon[44434]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:27.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:27.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:27.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:27 smithi090 ceph-mon[44434]: from='osd.10 v1:172.21.15.90:6808/3735380744' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-15T23:03:27.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:27.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:03:27.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:27.850 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 10 on host 'smithi090' 2024-05-15T23:03:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:27 smithi088 ceph-mon[44699]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:27 smithi088 ceph-mon[44699]: from='osd.10 v1:172.21.15.90:6808/3735380744' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-15T23:03:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:03:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:28.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:27 smithi084 ceph-mon[39784]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:28.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:28.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:28.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:27 smithi084 ceph-mon[39784]: from='osd.10 v1:172.21.15.90:6808/3735380744' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-15T23:03:28.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:28.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:03:28.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:28.540 DEBUG:teuthology.orchestra.run.smithi090:osd.10> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.10.service 2024-05-15T23:03:28.545 INFO:tasks.cephadm:Deploying osd.11 on smithi090 with /dev/vg_nvme/lv_1... 2024-05-15T23:03:28.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-15T23:03:28.573 INFO:journalctl@ceph.osd.10.smithi090.stdout:-- Logs begin at Wed 2024-05-15 22:43:42 UTC. -- 2024-05-15T23:03:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:28 smithi084 ceph-mon[39784]: from='osd.10 v1:172.21.15.90:6808/3735380744' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-15T23:03:29.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:28 smithi084 ceph-mon[39784]: osdmap e72: 11 total, 10 up, 11 in 2024-05-15T23:03:29.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:28 smithi084 ceph-mon[39784]: from='osd.10 v1:172.21.15.90:6808/3735380744' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:03:29.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:29.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:28 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:03:29.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:28 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:03:29.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:29.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:29.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:29.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:28 smithi090 ceph-mon[44434]: from='osd.10 v1:172.21.15.90:6808/3735380744' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-15T23:03:29.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:28 smithi090 ceph-mon[44434]: osdmap e72: 11 total, 10 up, 11 in 2024-05-15T23:03:29.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:28 smithi090 ceph-mon[44434]: from='osd.10 v1:172.21.15.90:6808/3735380744' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:03:29.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:29.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:28 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:03:29.177 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:28 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:03:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:28 smithi088 ceph-mon[44699]: from='osd.10 v1:172.21.15.90:6808/3735380744' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-15T23:03:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:28 smithi088 ceph-mon[44699]: osdmap e72: 11 total, 10 up, 11 in 2024-05-15T23:03:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:28 smithi088 ceph-mon[44699]: from='osd.10 v1:172.21.15.90:6808/3735380744' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:03:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:29.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:28 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:03:29.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:28 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:03:29.552 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:03:29 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:03:29.174+0000 7efe81b1a700 -1 osd.10 0 waiting for initial osdmap 2024-05-15T23:03:29.552 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:03:29 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:03:29.174+0000 7efe79298700 -1 osd.10 0 failed to load OSD map for epoch 12, got 0 bytes 2024-05-15T23:03:29.553 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:03:29 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:03:29.186+0000 7efe7d90d700 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T23:03:29.756 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:03:30.012 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:29 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:03:30.012 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:29 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:03:30.012 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:29 smithi090 ceph-mon[44434]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:30.012 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:29 smithi090 ceph-mon[44434]: from='osd.10 v1:172.21.15.90:6808/3735380744' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-05-15T23:03:30.012 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:29 smithi090 ceph-mon[44434]: osdmap e73: 11 total, 10 up, 11 in 2024-05-15T23:03:30.012 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:30.012 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:29 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:03:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:29 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:03:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:29 smithi084 ceph-mon[39784]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:30.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:29 smithi084 ceph-mon[39784]: from='osd.10 v1:172.21.15.90:6808/3735380744' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-05-15T23:03:30.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:29 smithi084 ceph-mon[39784]: osdmap e73: 11 total, 10 up, 11 in 2024-05-15T23:03:30.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:30.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:30.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:29 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:03:30.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:29 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:03:30.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:29 smithi088 ceph-mon[44699]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 894 GiB / 894 GiB avail 2024-05-15T23:03:30.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:29 smithi088 ceph-mon[44699]: from='osd.10 v1:172.21.15.90:6808/3735380744' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-05-15T23:03:30.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:29 smithi088 ceph-mon[44699]: osdmap e73: 11 total, 10 up, 11 in 2024-05-15T23:03:30.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:29 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:30.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:29 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:31.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:31 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:03:31.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:31 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:03:31.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:03:31.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:31 smithi090 ceph-mon[44434]: osd.10 v1:172.21.15.90:6808/3735380744 boot 2024-05-15T23:03:31.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:31 smithi090 ceph-mon[44434]: osdmap e74: 11 total, 11 up, 11 in 2024-05-15T23:03:31.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:31.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:31 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:03:31.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:31 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:03:31.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:03:31.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:31 smithi088 ceph-mon[44699]: osd.10 v1:172.21.15.90:6808/3735380744 boot 2024-05-15T23:03:31.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:31 smithi088 ceph-mon[44699]: osdmap e74: 11 total, 11 up, 11 in 2024-05-15T23:03:31.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:31 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:03:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:31 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:03:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:03:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:31 smithi084 ceph-mon[39784]: osd.10 v1:172.21.15.90:6808/3735380744 boot 2024-05-15T23:03:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:31 smithi084 ceph-mon[39784]: osdmap e74: 11 total, 11 up, 11 in 2024-05-15T23:03:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-15T23:03:32.519 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-15T23:03:32.520 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-15T23:03:32.520 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-05-15T23:03:32.520 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-05-15T23:03:32.520 INFO:teuthology.orchestra.run.smithi090.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110851 s, 946 MB/s 2024-05-15T23:03:32.520 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-05-15T23:03:32.551 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:32 smithi090 ceph-mon[44434]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:32.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:32 smithi088 ceph-mon[44699]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:32.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:32 smithi084 ceph-mon[39784]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:33.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:33 smithi090 ceph-mon[44434]: osdmap e75: 11 total, 11 up, 11 in 2024-05-15T23:03:33.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:33.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:33.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:03:33.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:03:33.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:03:33.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:33.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:33.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:03:33.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:33.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_1 2024-05-15T23:03:33.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:33 smithi088 ceph-mon[44699]: osdmap e75: 11 total, 11 up, 11 in 2024-05-15T23:03:33.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:33.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:33.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:03:33.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:03:33.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:03:33.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:33.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:33.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:03:33.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:33 smithi084 ceph-mon[39784]: osdmap e75: 11 total, 11 up, 11 in 2024-05-15T23:03:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:03:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:03:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:03:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:33.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:33.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:03:33.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:33.674 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:03:34.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:34 smithi090 ceph-mon[44434]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:34.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:34 smithi090 ceph-mon[44434]: Detected new or changed devices on smithi090 2024-05-15T23:03:34.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:34 smithi090 ceph-mon[44434]: Adjusting osd_memory_target on smithi090 to 7072M 2024-05-15T23:03:34.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:34 smithi088 ceph-mon[44699]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:34.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:34 smithi088 ceph-mon[44699]: Detected new or changed devices on smithi090 2024-05-15T23:03:34.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:34 smithi088 ceph-mon[44699]: Adjusting osd_memory_target on smithi090 to 7072M 2024-05-15T23:03:34.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:34 smithi084 ceph-mon[39784]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:34.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:34 smithi084 ceph-mon[39784]: Detected new or changed devices on smithi090 2024-05-15T23:03:34.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:34 smithi084 ceph-mon[39784]: Adjusting osd_memory_target on smithi090 to 7072M 2024-05-15T23:03:36.456 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:36 smithi090 ceph-mon[44434]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:36.457 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:03:36.457 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:03:36.457 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:36 smithi088 ceph-mon[44699]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:03:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:03:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:36.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:36 smithi084 ceph-mon[39784]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:36.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T23:03:36.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T23:03:36.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:37.298 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:37 smithi090 ceph-mon[44434]: from='client.24412 v1:172.21.15.90:0/1308173218' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:03:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:37 smithi088 ceph-mon[44699]: from='client.24412 v1:172.21.15.90:0/1308173218' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:03:37.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:37 smithi084 ceph-mon[39784]: from='client.24412 v1:172.21.15.90:0/1308173218' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T23:03:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:38 smithi090 ceph-mon[44434]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:38.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:38 smithi088 ceph-mon[44699]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:38.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:38 smithi084 ceph-mon[39784]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.90:0/2205038149' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5"}]: dispatch 2024-05-15T23:03:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:39 smithi090 ceph-mon[44434]: from='client.24418 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5"}]: dispatch 2024-05-15T23:03:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:39 smithi090 ceph-mon[44434]: from='client.24418 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5"}]': finished 2024-05-15T23:03:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:39 smithi090 ceph-mon[44434]: osdmap e76: 12 total, 11 up, 12 in 2024-05-15T23:03:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.90:0/915082685' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:03:39.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.90:0/2205038149' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5"}]: dispatch 2024-05-15T23:03:39.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:39 smithi088 ceph-mon[44699]: from='client.24418 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5"}]: dispatch 2024-05-15T23:03:39.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:39 smithi088 ceph-mon[44699]: from='client.24418 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5"}]': finished 2024-05-15T23:03:39.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:39 smithi088 ceph-mon[44699]: osdmap e76: 12 total, 11 up, 12 in 2024-05-15T23:03:39.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:39 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:39.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.90:0/915082685' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:03:39.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.90:0/2205038149' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5"}]: dispatch 2024-05-15T23:03:39.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:39 smithi084 ceph-mon[39784]: from='client.24418 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5"}]: dispatch 2024-05-15T23:03:39.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:39 smithi084 ceph-mon[39784]: from='client.24418 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5"}]': finished 2024-05-15T23:03:39.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:39 smithi084 ceph-mon[39784]: osdmap e76: 12 total, 11 up, 12 in 2024-05-15T23:03:39.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:39.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.90:0/915082685' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T23:03:40.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {auth=23,config=2,health=12,kv=53,logm=10} crc {auth=616185976,config=1548763834,health=3897446751,kv=204887964,logm=1698375103}) 2024-05-15T23:03:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=375935965}) 2024-05-15T23:03:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1300210543}) 2024-05-15T23:03:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3015866984}) 2024-05-15T23:03:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=133084949}) 2024-05-15T23:03:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2681826027}) 2024-05-15T23:03:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4178308808}) 2024-05-15T23:03:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2875189440}) 2024-05-15T23:03:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2527971578}) 2024-05-15T23:03:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=974660429}) 2024-05-15T23:03:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=624282795}) 2024-05-15T23:03:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942822724}) 2024-05-15T23:03:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4136317045}) 2024-05-15T23:03:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3819310107}) 2024-05-15T23:03:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2780883826}) 2024-05-15T23:03:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1788601906}) 2024-05-15T23:03:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=43,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=23} crc {logm=1666387205,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=3306116768,mon_config_key=2180306020}) 2024-05-15T23:03:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=919274403}) 2024-05-15T23:03:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3951336826}) 2024-05-15T23:03:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=49,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=34} crc {mon_config_key=220575596,monmap=1478146745,osd_metadata=2185583256,osd_pg_creating=454211687,osdmap=669751588}) 2024-05-15T23:03:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=75} crc {osdmap=1272218232}) 2024-05-15T23:03:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {auth=23,config=2,health=12,kv=53,logm=10} crc {auth=616185976,config=1548763834,health=3897446751,kv=204887964,logm=1698375103}) 2024-05-15T23:03:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=375935965}) 2024-05-15T23:03:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1300210543}) 2024-05-15T23:03:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3015866984}) 2024-05-15T23:03:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=133084949}) 2024-05-15T23:03:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2681826027}) 2024-05-15T23:03:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4178308808}) 2024-05-15T23:03:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2875189440}) 2024-05-15T23:03:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2527971578}) 2024-05-15T23:03:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=974660429}) 2024-05-15T23:03:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=624282795}) 2024-05-15T23:03:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942822724}) 2024-05-15T23:03:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4136317045}) 2024-05-15T23:03:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3819310107}) 2024-05-15T23:03:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2780883826}) 2024-05-15T23:03:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1788601906}) 2024-05-15T23:03:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=43,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=23} crc {logm=1666387205,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=3306116768,mon_config_key=2180306020}) 2024-05-15T23:03:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=919274403}) 2024-05-15T23:03:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3951336826}) 2024-05-15T23:03:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=49,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=34} crc {mon_config_key=220575596,monmap=1478146745,osd_metadata=2185583256,osd_pg_creating=454211687,osdmap=669751588}) 2024-05-15T23:03:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=75} crc {osdmap=1272218232}) 2024-05-15T23:03:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {auth=23,config=2,health=12,kv=53,logm=10} crc {auth=616185976,config=1548763834,health=3897446751,kv=204887964,logm=1698375103}) 2024-05-15T23:03:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=375935965}) 2024-05-15T23:03:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1300210543}) 2024-05-15T23:03:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3015866984}) 2024-05-15T23:03:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=133084949}) 2024-05-15T23:03:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2681826027}) 2024-05-15T23:03:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4178308808}) 2024-05-15T23:03:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2875189440}) 2024-05-15T23:03:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2527971578}) 2024-05-15T23:03:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=974660429}) 2024-05-15T23:03:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=624282795}) 2024-05-15T23:03:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942822724}) 2024-05-15T23:03:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4136317045}) 2024-05-15T23:03:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3819310107}) 2024-05-15T23:03:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2780883826}) 2024-05-15T23:03:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1788601906}) 2024-05-15T23:03:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=43,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=23} crc {logm=1666387205,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=3306116768,mon_config_key=2180306020}) 2024-05-15T23:03:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=919274403}) 2024-05-15T23:03:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3951336826}) 2024-05-15T23:03:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=49,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=34} crc {mon_config_key=220575596,monmap=1478146745,osd_metadata=2185583256,osd_pg_creating=454211687,osdmap=669751588}) 2024-05-15T23:03:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=75} crc {osdmap=1272218232}) 2024-05-15T23:03:42.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:42 smithi090 ceph-mon[44434]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:42.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:42 smithi088 ceph-mon[44699]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:42.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:42 smithi084 ceph-mon[39784]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:44.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:44 smithi090 ceph-mon[44434]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:44.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:44 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:03:44.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:44 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:03:44.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:44 smithi090 ceph-mon[44434]: osdmap e77: 12 total, 11 up, 12 in 2024-05-15T23:03:44.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:44 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:44 smithi088 ceph-mon[44699]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:44 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:03:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:44 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:03:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:44 smithi088 ceph-mon[44699]: osdmap e77: 12 total, 11 up, 12 in 2024-05-15T23:03:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:44 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:44.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:44 smithi084 ceph-mon[39784]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:44.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:44 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:03:44.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:44 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:03:44.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:44 smithi084 ceph-mon[39784]: osdmap e77: 12 total, 11 up, 12 in 2024-05-15T23:03:44.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:44 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:45 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:03:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:45 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:03:45.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:45 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:03:45.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:45 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:03:45.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:45 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:03:45.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:45 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:03:46.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:46 smithi090 ceph-mon[44434]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:46.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:46 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-15T23:03:46.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:46 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:46.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:46 smithi088 ceph-mon[44699]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:46.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:46 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-15T23:03:46.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:46 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:46.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:46 smithi084 ceph-mon[39784]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:46.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:46 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-15T23:03:46.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:46 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:47 smithi090 ceph-mon[44434]: Deploying daemon osd.11 on smithi090 2024-05-15T23:03:47.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:47 smithi088 ceph-mon[44699]: Deploying daemon osd.11 on smithi090 2024-05-15T23:03:47.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:47 smithi084 ceph-mon[39784]: Deploying daemon osd.11 on smithi090 2024-05-15T23:03:48.655 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:48 smithi090 ceph-mon[44434]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:48.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:48 smithi088 ceph-mon[44699]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:48.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:48 smithi084 ceph-mon[39784]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:50.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:50 smithi090 ceph-mon[44434]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:50.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:50 smithi088 ceph-mon[44699]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:50.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:50 smithi084 ceph-mon[39784]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:52.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:52 smithi084 ceph-mon[39784]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:52.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:52 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:52.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:52 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:52.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:52 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:52.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:52 smithi090 ceph-mon[44434]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:52.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:52 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:52.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:52 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:52.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:52 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:52.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:52 smithi088 ceph-mon[44699]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:52.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:52 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:52.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:52 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:52.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:52 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:53.790 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:53 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:03:53.790 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:53 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:03:53.790 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:53 smithi090 ceph-mon[44434]: osdmap e78: 12 total, 11 up, 12 in 2024-05-15T23:03:53.791 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:53 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:03:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:53 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:03:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:53 smithi088 ceph-mon[44699]: osdmap e78: 12 total, 11 up, 12 in 2024-05-15T23:03:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:53.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:53 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:03:53.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:53 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:03:53.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:53 smithi084 ceph-mon[39784]: osdmap e78: 12 total, 11 up, 12 in 2024-05-15T23:03:53.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:54.678 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:54 smithi090 ceph-mon[44434]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:54.678 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:54.678 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:54.678 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:54.678 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:03:54.679 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:54.679 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:54 smithi090 ceph-mon[44434]: from='osd.11 v1:172.21.15.90:6812/2265615270' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-15T23:03:54.679 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:54 smithi090 ceph-mon[44434]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-15T23:03:54.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:54 smithi088 ceph-mon[44699]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:54.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:54.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:54.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:54.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:03:54.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:54.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:54 smithi088 ceph-mon[44699]: from='osd.11 v1:172.21.15.90:6812/2265615270' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-15T23:03:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:54 smithi088 ceph-mon[44699]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-15T23:03:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:54 smithi084 ceph-mon[39784]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:03:54.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:03:54.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:54.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:54 smithi084 ceph-mon[39784]: from='osd.11 v1:172.21.15.90:6812/2265615270' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-15T23:03:54.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:54 smithi084 ceph-mon[39784]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-15T23:03:55.002 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 11 on host 'smithi090' 2024-05-15T23:03:55.731 DEBUG:teuthology.orchestra.run.smithi090:osd.11> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.11.service 2024-05-15T23:03:55.733 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-15T23:03:55.734 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd stat -f json 2024-05-15T23:03:55.766 INFO:journalctl@ceph.osd.11.smithi090.stdout:-- Logs begin at Wed 2024-05-15 22:43:42 UTC. -- 2024-05-15T23:03:55.994 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:55 smithi090 ceph-mon[44434]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:55.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:55 smithi090 ceph-mon[44434]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-15T23:03:55.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:55 smithi090 ceph-mon[44434]: osdmap e79: 12 total, 11 up, 12 in 2024-05-15T23:03:55.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:55 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:55.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:55 smithi090 ceph-mon[44434]: from='osd.11 v1:172.21.15.90:6812/2265615270' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:03:55.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:55 smithi090 ceph-mon[44434]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:03:55.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:55 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:55.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:55 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:55.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:55 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:56.011 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:03:56.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:55 smithi084 ceph-mon[39784]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:56.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:55 smithi084 ceph-mon[39784]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-15T23:03:56.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:55 smithi084 ceph-mon[39784]: osdmap e79: 12 total, 11 up, 12 in 2024-05-15T23:03:56.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:56.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:55 smithi084 ceph-mon[39784]: from='osd.11 v1:172.21.15.90:6812/2265615270' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:03:56.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:55 smithi084 ceph-mon[39784]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:03:56.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:56.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:56.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:55 smithi088 ceph-mon[44699]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:55 smithi088 ceph-mon[44699]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-15T23:03:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:55 smithi088 ceph-mon[44699]: osdmap e79: 12 total, 11 up, 12 in 2024-05-15T23:03:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:55 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:55 smithi088 ceph-mon[44699]: from='osd.11 v1:172.21.15.90:6812/2265615270' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:03:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:55 smithi088 ceph-mon[44699]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:03:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:55 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:55 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:03:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:55 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:03:56.707 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:03:56 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:03:56.345+0000 7fe565bdd700 -1 osd.11 0 waiting for initial osdmap 2024-05-15T23:03:56.708 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:03:56 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:03:56.346+0000 7fe559353700 -1 osd.11 0 failed to load OSD map for epoch 23, got 0 bytes 2024-05-15T23:03:56.708 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:03:56 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:03:56.355+0000 7fe55d9c8700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T23:03:57.008 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:56 smithi090 ceph-mon[44434]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-05-15T23:03:57.008 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:56 smithi090 ceph-mon[44434]: osdmap e80: 12 total, 11 up, 12 in 2024-05-15T23:03:57.008 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:56 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:57.008 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:56 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:57.008 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:56 smithi090 ceph-mon[44434]: from='osd.11 ' entity='osd.11' 2024-05-15T23:03:57.008 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:56 smithi090 ceph-mon[44434]: osd.11 v1:172.21.15.90:6812/2265615270 boot 2024-05-15T23:03:57.008 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:56 smithi090 ceph-mon[44434]: osdmap e81: 12 total, 12 up, 12 in 2024-05-15T23:03:57.008 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:56 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:56 smithi088 ceph-mon[44699]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-05-15T23:03:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:56 smithi088 ceph-mon[44699]: osdmap e80: 12 total, 11 up, 12 in 2024-05-15T23:03:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:56 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:56 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:56 smithi088 ceph-mon[44699]: from='osd.11 ' entity='osd.11' 2024-05-15T23:03:57.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:56 smithi088 ceph-mon[44699]: osd.11 v1:172.21.15.90:6812/2265615270 boot 2024-05-15T23:03:57.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:56 smithi088 ceph-mon[44699]: osdmap e81: 12 total, 12 up, 12 in 2024-05-15T23:03:57.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:56 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:57.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:56 smithi084 ceph-mon[39784]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-05-15T23:03:57.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:56 smithi084 ceph-mon[39784]: osdmap e80: 12 total, 11 up, 12 in 2024-05-15T23:03:57.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:56 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:57.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:56 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:57.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:56 smithi084 ceph-mon[39784]: from='osd.11 ' entity='osd.11' 2024-05-15T23:03:57.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:56 smithi084 ceph-mon[39784]: osd.11 v1:172.21.15.90:6812/2265615270 boot 2024-05-15T23:03:57.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:56 smithi084 ceph-mon[39784]: osdmap e81: 12 total, 12 up, 12 in 2024-05-15T23:03:57.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:56 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-15T23:03:58.065 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:03:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:57 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:03:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:57 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:03:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:03:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:03:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:57 smithi090 ceph-mon[44434]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:58.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:57 smithi090 ceph-mon[44434]: osdmap e82: 12 total, 12 up, 12 in 2024-05-15T23:03:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:57 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:03:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:57 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:03:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:03:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:03:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:57 smithi088 ceph-mon[44699]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:57 smithi088 ceph-mon[44699]: osdmap e82: 12 total, 12 up, 12 in 2024-05-15T23:03:58.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:57 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:03:58.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:57 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:03:58.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:03:58.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:03:58.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:57 smithi084 ceph-mon[39784]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-05-15T23:03:58.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:57 smithi084 ceph-mon[39784]: osdmap e82: 12 total, 12 up, 12 in 2024-05-15T23:03:58.768 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1715814236,"num_in_osds":12,"osd_in_since":1715814218,"num_remapped_pgs":0} 2024-05-15T23:03:58.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump --format=json 2024-05-15T23:03:59.032 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:03:59.032 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:58 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1624967273' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-15T23:03:59.032 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:03:58 smithi090 ceph-mon[44434]: osdmap e83: 12 total, 12 up, 12 in 2024-05-15T23:03:59.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:58 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1624967273' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-15T23:03:59.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:03:58 smithi084 ceph-mon[39784]: osdmap e83: 12 total, 12 up, 12 in 2024-05-15T23:03:59.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:58 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1624967273' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-15T23:03:59.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:03:58 smithi088 ceph-mon[44699]: osdmap e83: 12 total, 12 up, 12 in 2024-05-15T23:04:00.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:00 smithi084 ceph-mon[39784]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:00.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:00 smithi084 ceph-mon[39784]: Detected new or changed devices on smithi090 2024-05-15T23:04:00.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:04:00.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:04:00.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:04:00.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:04:00.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:04:00.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:04:00.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:00 smithi084 ceph-mon[39784]: Adjusting osd_memory_target on smithi090 to 5304M 2024-05-15T23:04:00.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:04:00.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:04:00.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:04:00.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:04:00.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:00 smithi090 ceph-mon[44434]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:00.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:00 smithi090 ceph-mon[44434]: Detected new or changed devices on smithi090 2024-05-15T23:04:00.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:04:00.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:04:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:04:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:04:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:04:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:04:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:00 smithi090 ceph-mon[44434]: Adjusting osd_memory_target on smithi090 to 5304M 2024-05-15T23:04:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:04:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:04:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:04:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:04:00.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:00 smithi088 ceph-mon[44699]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:00.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:00 smithi088 ceph-mon[44699]: Detected new or changed devices on smithi090 2024-05-15T23:04:00.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:04:00.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:04:00.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:04:00.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:04:00.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:04:00.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-15T23:04:00.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:00 smithi088 ceph-mon[44699]: Adjusting osd_memory_target on smithi090 to 5304M 2024-05-15T23:04:00.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:04:00.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:04:00.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:04:00.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:04:01.010 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:04:01.011 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":83,"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","created":"2024-05-15T22:56:41.212493+0000","modified":"2024-05-15T23:03:58.586087+0000","last_up_change":"2024-05-15T23:03:56.576073+0000","last_in_change":"2024-05-15T23:03:38.460671+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T22:59:38.603575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80159cc8-1b6c-4aa6-a114-966786d7be1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6801","nonce":1084680531}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6802","nonce":1084680531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6804","nonce":1084680531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6803","nonce":1084680531}]},"public_addr":"172.21.15.84:6801/1084680531","cluster_addr":"172.21.15.84:6802/1084680531","heartbeat_back_addr":"172.21.15.84:6804/1084680531","heartbeat_front_addr":"172.21.15.84:6803/1084680531","state":["exists","up"]},{"osd":1,"uuid":"9c2c76c9-ce0f-4c2d-9d96-c87a10187607","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6805","nonce":2140830229}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6806","nonce":2140830229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6808","nonce":2140830229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6807","nonce":2140830229}]},"public_addr":"172.21.15.84:6805/2140830229","cluster_addr":"172.21.15.84:6806/2140830229","heartbeat_back_addr":"172.21.15.84:6808/2140830229","heartbeat_front_addr":"172.21.15.84:6807/2140830229","state":["exists","up"]},{"osd":2,"uuid":"e8514daa-b5f9-4f2f-a432-53f4a4eb86bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6809","nonce":2699996290}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6810","nonce":2699996290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6812","nonce":2699996290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6811","nonce":2699996290}]},"public_addr":"172.21.15.84:6809/2699996290","cluster_addr":"172.21.15.84:6810/2699996290","heartbeat_back_addr":"172.21.15.84:6812/2699996290","heartbeat_front_addr":"172.21.15.84:6811/2699996290","state":["exists","up"]},{"osd":3,"uuid":"5d9965f9-c0b2-48ce-869c-582990985623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6813","nonce":2204039828}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6814","nonce":2204039828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6816","nonce":2204039828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6815","nonce":2204039828}]},"public_addr":"172.21.15.84:6813/2204039828","cluster_addr":"172.21.15.84:6814/2204039828","heartbeat_back_addr":"172.21.15.84:6816/2204039828","heartbeat_front_addr":"172.21.15.84:6815/2204039828","state":["exists","up"]},{"osd":4,"uuid":"4bc50523-8480-44d2-8d76-31479e839c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6800","nonce":2267955292}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":2267955292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":2267955292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":2267955292}]},"public_addr":"172.21.15.88:6800/2267955292","cluster_addr":"172.21.15.88:6801/2267955292","heartbeat_back_addr":"172.21.15.88:6803/2267955292","heartbeat_front_addr":"172.21.15.88:6802/2267955292","state":["exists","up"]},{"osd":5,"uuid":"d4db8d7f-afc7-43dc-81fe-f70c290d8b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":661041538}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":661041538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":661041538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":661041538}]},"public_addr":"172.21.15.88:6804/661041538","cluster_addr":"172.21.15.88:6805/661041538","heartbeat_back_addr":"172.21.15.88:6807/661041538","heartbeat_front_addr":"172.21.15.88:6806/661041538","state":["exists","up"]},{"osd":6,"uuid":"a2773454-2318-4a5b-8f31-0952b1b07f2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":380856869}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":380856869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":380856869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":380856869}]},"public_addr":"172.21.15.88:6808/380856869","cluster_addr":"172.21.15.88:6809/380856869","heartbeat_back_addr":"172.21.15.88:6811/380856869","heartbeat_front_addr":"172.21.15.88:6810/380856869","state":["exists","up"]},{"osd":7,"uuid":"028ca7b6-3e00-4466-a33f-b726cd60cb34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":1044709063}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":1044709063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":1044709063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":1044709063}]},"public_addr":"172.21.15.88:6812/1044709063","cluster_addr":"172.21.15.88:6813/1044709063","heartbeat_back_addr":"172.21.15.88:6815/1044709063","heartbeat_front_addr":"172.21.15.88:6814/1044709063","state":["exists","up"]},{"osd":8,"uuid":"02cab286-d487-4a66-982c-3cf28f55e851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6800","nonce":522459126}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6801","nonce":522459126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6803","nonce":522459126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6802","nonce":522459126}]},"public_addr":"172.21.15.90:6800/522459126","cluster_addr":"172.21.15.90:6801/522459126","heartbeat_back_addr":"172.21.15.90:6803/522459126","heartbeat_front_addr":"172.21.15.90:6802/522459126","state":["exists","up"]},{"osd":9,"uuid":"47d38bc7-39ba-4b35-af24-e282354dd613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6804","nonce":2794571133}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6805","nonce":2794571133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6807","nonce":2794571133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6806","nonce":2794571133}]},"public_addr":"172.21.15.90:6804/2794571133","cluster_addr":"172.21.15.90:6805/2794571133","heartbeat_back_addr":"172.21.15.90:6807/2794571133","heartbeat_front_addr":"172.21.15.90:6806/2794571133","state":["exists","up"]},{"osd":10,"uuid":"d7b2aeae-d012-41eb-9f91-e4c05c649473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6808","nonce":3735380744}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6809","nonce":3735380744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6811","nonce":3735380744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6810","nonce":3735380744}]},"public_addr":"172.21.15.90:6808/3735380744","cluster_addr":"172.21.15.90:6809/3735380744","heartbeat_back_addr":"172.21.15.90:6811/3735380744","heartbeat_front_addr":"172.21.15.90:6810/3735380744","state":["exists","up"]},{"osd":11,"uuid":"2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6812","nonce":2265615270}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6813","nonce":2265615270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6815","nonce":2265615270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6814","nonce":2265615270}]},"public_addr":"172.21.15.90:6812/2265615270","cluster_addr":"172.21.15.90:6813/2265615270","heartbeat_back_addr":"172.21.15.90:6815/2265615270","heartbeat_front_addr":"172.21.15.90:6814/2265615270","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:02:54.736546+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:28.444415+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:02:35.556159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:52.123662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:12.220515+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:43.452092+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:02:55.160040+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:28.064342+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:02:36.442958+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:02.040127+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:28.234215+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:55.433179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/2581408231":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/1136984393":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/2443201864":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1252671888":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1451001371":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/4084814866":"2024-05-16T22:57:38.505263+0000","172.21.15.84:0/4076409285":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/2988977694":"2024-05-16T22:57:38.505263+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T23:04:01.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:01 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:04:01.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:01 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:04:01.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:01 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3382777094' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:04:01.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:01 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:04:01.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:01 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:04:01.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:01 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3382777094' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:04:01.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:01 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:04:01.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:01 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:04:01.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:01 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3382777094' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:04:01.767 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T22:59:38.603575+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-15T23:04:01.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-15T23:04:02.032 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:02.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:02 smithi084 ceph-mon[39784]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s, 0 objects/s recovering 2024-05-15T23:04:02.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:02 smithi084 ceph-mon[39784]: osdmap e84: 12 total, 12 up, 12 in 2024-05-15T23:04:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:02 smithi090 ceph-mon[44434]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s, 0 objects/s recovering 2024-05-15T23:04:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:02 smithi090 ceph-mon[44434]: osdmap e84: 12 total, 12 up, 12 in 2024-05-15T23:04:02.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:02 smithi088 ceph-mon[44699]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s, 0 objects/s recovering 2024-05-15T23:04:02.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:02 smithi088 ceph-mon[44699]: osdmap e84: 12 total, 12 up, 12 in 2024-05-15T23:04:04.030 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 1 2024-05-15T23:04:04.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:04 smithi084 ceph-mon[39784]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T23:04:04.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:04 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:04:04.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:04 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:04:04.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:04 smithi084 ceph-mon[39784]: osdmap e85: 12 total, 12 up, 12 in 2024-05-15T23:04:04.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3984012200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-15T23:04:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:04 smithi090 ceph-mon[44434]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T23:04:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:04 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:04:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:04 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:04:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:04 smithi090 ceph-mon[44434]: osdmap e85: 12 total, 12 up, 12 in 2024-05-15T23:04:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3984012200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-15T23:04:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:04 smithi088 ceph-mon[44699]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T23:04:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:04 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:04:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:04 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:04:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:04 smithi088 ceph-mon[44699]: osdmap e85: 12 total, 12 up, 12 in 2024-05-15T23:04:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3984012200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-15T23:04:04.808 INFO:tasks.cephadm:Setting up client nodes... 2024-05-15T23:04:04.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-15T23:04:05.071 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:06.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:06 smithi084 ceph-mon[39784]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T23:04:06.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:06 smithi090 ceph-mon[44434]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T23:04:06.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:06 smithi088 ceph-mon[44699]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-15T23:04:07.267 INFO:teuthology.orchestra.run.smithi084.stdout:[client.0] 2024-05-15T23:04:07.268 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQBnP0VmE3DPDxAAAq+V2Jq8Imcuf1daSu01BA== 2024-05-15T23:04:07.886 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-05-15T23:04:07.886 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-15T23:04:07.886 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-15T23:04:07.927 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-15T23:04:08.203 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:04:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:08 smithi088 ceph-mon[44699]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T23:04:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/501739294' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T23:04:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/501739294' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T23:04:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:08 smithi084 ceph-mon[39784]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T23:04:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/501739294' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T23:04:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/501739294' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T23:04:08.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:08 smithi090 ceph-mon[44434]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T23:04:08.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/501739294' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T23:04:08.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/501739294' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T23:04:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:10 smithi088 ceph-mon[44699]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:10.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:10 smithi084 ceph-mon[39784]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:10.525 INFO:teuthology.orchestra.run.smithi088.stdout:[client.1] 2024-05-15T23:04:10.525 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQBqP0Vm/jYIHxAAvJLnjc6G7/ZWwjOTE/XLuw== 2024-05-15T23:04:10.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:10 smithi090 ceph-mon[44434]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:11.189 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:11 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.88:0/1969939829' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T23:04:11.189 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:11 smithi088 ceph-mon[44699]: from='client.34463 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T23:04:11.189 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:11 smithi088 ceph-mon[44699]: from='client.34463 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T23:04:11.218 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T23:04:11.218 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-05-15T23:04:11.218 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-15T23:04:11.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-15T23:04:11.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:11 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.88:0/1969939829' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T23:04:11.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:11 smithi084 ceph-mon[39784]: from='client.34463 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T23:04:11.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:11 smithi084 ceph-mon[39784]: from='client.34463 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T23:04:11.538 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:04:11.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:11 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.88:0/1969939829' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T23:04:11.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:11 smithi090 ceph-mon[44434]: from='client.34463 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T23:04:11.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:11 smithi090 ceph-mon[44434]: from='client.34463 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T23:04:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:12 smithi090 ceph-mon[44434]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:12 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:04:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:12 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:04:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:12 smithi090 ceph-mon[44434]: osdmap e86: 12 total, 12 up, 12 in 2024-05-15T23:04:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:12 smithi088 ceph-mon[44699]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:12 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:04:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:12 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:04:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:12 smithi088 ceph-mon[44699]: osdmap e86: 12 total, 12 up, 12 in 2024-05-15T23:04:12.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:12 smithi084 ceph-mon[39784]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:12.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:12 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:04:12.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:12 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:04:12.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:12 smithi084 ceph-mon[39784]: osdmap e86: 12 total, 12 up, 12 in 2024-05-15T23:04:13.913 INFO:teuthology.orchestra.run.smithi090.stdout:[client.2] 2024-05-15T23:04:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBtP0Vmw78gNhAAKuRCqGsSdCW2iOs+rWpuPA== 2024-05-15T23:04:14.591 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:14 smithi090 ceph-mon[44434]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:14.591 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:14 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.90:0/985255825' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T23:04:14.591 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:14 smithi090 ceph-mon[44434]: from='client.24469 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T23:04:14.591 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:14 smithi090 ceph-mon[44434]: from='client.24469 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T23:04:14.616 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-05-15T23:04:14.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-05-15T23:04:14.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-05-15T23:04:14.656 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-15T23:04:14.656 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-15T23:04:14.656 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph mgr dump --format=json 2024-05-15T23:04:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:14 smithi088 ceph-mon[44699]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:14 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.90:0/985255825' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T23:04:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:14 smithi088 ceph-mon[44699]: from='client.24469 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T23:04:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:14 smithi088 ceph-mon[44699]: from='client.24469 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T23:04:14.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:14 smithi084 ceph-mon[39784]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:14.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:14 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.90:0/985255825' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T23:04:14.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:14 smithi084 ceph-mon[39784]: from='client.24469 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T23:04:14.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:14 smithi084 ceph-mon[39784]: from='client.24469 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T23:04:14.933 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:16.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:15 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:04:16.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:15 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:04:16.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:15 smithi084 ceph-mon[39784]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:16.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:15 smithi084 ceph-mon[39784]: osdmap e87: 12 total, 12 up, 12 in 2024-05-15T23:04:16.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:15 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:04:16.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:15 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:04:16.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:15 smithi090 ceph-mon[44434]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:16.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:15 smithi090 ceph-mon[44434]: osdmap e87: 12 total, 12 up, 12 in 2024-05-15T23:04:16.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:15 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:04:16.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:15 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:04:16.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:15 smithi088 ceph-mon[44699]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:16.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:15 smithi088 ceph-mon[44699]: osdmap e87: 12 total, 12 up, 12 in 2024-05-15T23:04:17.025 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:04:17.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:17 smithi084 ceph-mon[39784]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:17.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:17 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/93915985' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-15T23:04:17.816 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":15,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6800","nonce":3061622445}]},"active_addr":"172.21.15.84:6800/3061622445","active_change":"2024-05-15T22:57:38.505369+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":34106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.84:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":4215405527}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":3897905141}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":389990635}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":4135158899}]}]} 2024-05-15T23:04:17.820 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-15T23:04:17.820 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-15T23:04:17.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump --format=json 2024-05-15T23:04:18.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:17 smithi090 ceph-mon[44434]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:18.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:17 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/93915985' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-15T23:04:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:17 smithi088 ceph-mon[44699]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:17 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/93915985' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-15T23:04:18.083 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:19.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:19 smithi084 ceph-mon[39784]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:19.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:19 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:04:19.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:19 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:04:20.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:19 smithi090 ceph-mon[44434]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:20.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:19 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:04:20.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:19 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:04:20.064 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:04:20.064 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":88,"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","created":"2024-05-15T22:56:41.212493+0000","modified":"2024-05-15T23:04:19.588572+0000","last_up_change":"2024-05-15T23:03:56.576073+0000","last_in_change":"2024-05-15T23:03:38.460671+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T22:59:38.603575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80159cc8-1b6c-4aa6-a114-966786d7be1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6801","nonce":1084680531}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6802","nonce":1084680531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6804","nonce":1084680531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6803","nonce":1084680531}]},"public_addr":"172.21.15.84:6801/1084680531","cluster_addr":"172.21.15.84:6802/1084680531","heartbeat_back_addr":"172.21.15.84:6804/1084680531","heartbeat_front_addr":"172.21.15.84:6803/1084680531","state":["exists","up"]},{"osd":1,"uuid":"9c2c76c9-ce0f-4c2d-9d96-c87a10187607","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6805","nonce":2140830229}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6806","nonce":2140830229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6808","nonce":2140830229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6807","nonce":2140830229}]},"public_addr":"172.21.15.84:6805/2140830229","cluster_addr":"172.21.15.84:6806/2140830229","heartbeat_back_addr":"172.21.15.84:6808/2140830229","heartbeat_front_addr":"172.21.15.84:6807/2140830229","state":["exists","up"]},{"osd":2,"uuid":"e8514daa-b5f9-4f2f-a432-53f4a4eb86bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6809","nonce":2699996290}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6810","nonce":2699996290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6812","nonce":2699996290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6811","nonce":2699996290}]},"public_addr":"172.21.15.84:6809/2699996290","cluster_addr":"172.21.15.84:6810/2699996290","heartbeat_back_addr":"172.21.15.84:6812/2699996290","heartbeat_front_addr":"172.21.15.84:6811/2699996290","state":["exists","up"]},{"osd":3,"uuid":"5d9965f9-c0b2-48ce-869c-582990985623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6813","nonce":2204039828}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6814","nonce":2204039828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6816","nonce":2204039828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6815","nonce":2204039828}]},"public_addr":"172.21.15.84:6813/2204039828","cluster_addr":"172.21.15.84:6814/2204039828","heartbeat_back_addr":"172.21.15.84:6816/2204039828","heartbeat_front_addr":"172.21.15.84:6815/2204039828","state":["exists","up"]},{"osd":4,"uuid":"4bc50523-8480-44d2-8d76-31479e839c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6800","nonce":2267955292}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":2267955292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":2267955292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":2267955292}]},"public_addr":"172.21.15.88:6800/2267955292","cluster_addr":"172.21.15.88:6801/2267955292","heartbeat_back_addr":"172.21.15.88:6803/2267955292","heartbeat_front_addr":"172.21.15.88:6802/2267955292","state":["exists","up"]},{"osd":5,"uuid":"d4db8d7f-afc7-43dc-81fe-f70c290d8b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":661041538}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":661041538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":661041538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":661041538}]},"public_addr":"172.21.15.88:6804/661041538","cluster_addr":"172.21.15.88:6805/661041538","heartbeat_back_addr":"172.21.15.88:6807/661041538","heartbeat_front_addr":"172.21.15.88:6806/661041538","state":["exists","up"]},{"osd":6,"uuid":"a2773454-2318-4a5b-8f31-0952b1b07f2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":380856869}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":380856869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":380856869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":380856869}]},"public_addr":"172.21.15.88:6808/380856869","cluster_addr":"172.21.15.88:6809/380856869","heartbeat_back_addr":"172.21.15.88:6811/380856869","heartbeat_front_addr":"172.21.15.88:6810/380856869","state":["exists","up"]},{"osd":7,"uuid":"028ca7b6-3e00-4466-a33f-b726cd60cb34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":1044709063}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":1044709063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":1044709063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":1044709063}]},"public_addr":"172.21.15.88:6812/1044709063","cluster_addr":"172.21.15.88:6813/1044709063","heartbeat_back_addr":"172.21.15.88:6815/1044709063","heartbeat_front_addr":"172.21.15.88:6814/1044709063","state":["exists","up"]},{"osd":8,"uuid":"02cab286-d487-4a66-982c-3cf28f55e851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6800","nonce":522459126}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6801","nonce":522459126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6803","nonce":522459126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6802","nonce":522459126}]},"public_addr":"172.21.15.90:6800/522459126","cluster_addr":"172.21.15.90:6801/522459126","heartbeat_back_addr":"172.21.15.90:6803/522459126","heartbeat_front_addr":"172.21.15.90:6802/522459126","state":["exists","up"]},{"osd":9,"uuid":"47d38bc7-39ba-4b35-af24-e282354dd613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6804","nonce":2794571133}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6805","nonce":2794571133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6807","nonce":2794571133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6806","nonce":2794571133}]},"public_addr":"172.21.15.90:6804/2794571133","cluster_addr":"172.21.15.90:6805/2794571133","heartbeat_back_addr":"172.21.15.90:6807/2794571133","heartbeat_front_addr":"172.21.15.90:6806/2794571133","state":["exists","up"]},{"osd":10,"uuid":"d7b2aeae-d012-41eb-9f91-e4c05c649473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6808","nonce":3735380744}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6809","nonce":3735380744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6811","nonce":3735380744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6810","nonce":3735380744}]},"public_addr":"172.21.15.90:6808/3735380744","cluster_addr":"172.21.15.90:6809/3735380744","heartbeat_back_addr":"172.21.15.90:6811/3735380744","heartbeat_front_addr":"172.21.15.90:6810/3735380744","state":["exists","up"]},{"osd":11,"uuid":"2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6812","nonce":2265615270}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6813","nonce":2265615270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6815","nonce":2265615270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6814","nonce":2265615270}]},"public_addr":"172.21.15.90:6812/2265615270","cluster_addr":"172.21.15.90:6813/2265615270","heartbeat_back_addr":"172.21.15.90:6815/2265615270","heartbeat_front_addr":"172.21.15.90:6814/2265615270","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:02:54.736546+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:28.444415+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:04:00.294225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:52.123662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:04:14.117098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:43.452092+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:04:10.694461+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:28.064342+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:04:03.061558+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:04:18.910306+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:28.234215+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:55.433179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/2581408231":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/1136984393":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/2443201864":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1252671888":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1451001371":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/4084814866":"2024-05-16T22:57:38.505263+0000","172.21.15.84:0/4076409285":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/2988977694":"2024-05-16T22:57:38.505263+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T23:04:20.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:19 smithi088 ceph-mon[44699]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:19 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:04:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:19 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:04:20.732 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:20 smithi084 ceph-mon[39784]: osdmap e88: 12 total, 12 up, 12 in 2024-05-15T23:04:20.732 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:20 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/547597640' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:04:20.757 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-15T23:04:20.757 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump --format=json 2024-05-15T23:04:21.021 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:21.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:20 smithi090 ceph-mon[44434]: osdmap e88: 12 total, 12 up, 12 in 2024-05-15T23:04:21.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:20 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/547597640' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:04:21.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:20 smithi088 ceph-mon[44699]: osdmap e88: 12 total, 12 up, 12 in 2024-05-15T23:04:21.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:20 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/547597640' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:04:21.932 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:21 smithi084 ceph-mon[39784]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:22.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:21 smithi090 ceph-mon[44434]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:21 smithi088 ceph-mon[44699]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:22.923 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:04:22.923 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":88,"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","created":"2024-05-15T22:56:41.212493+0000","modified":"2024-05-15T23:04:19.588572+0000","last_up_change":"2024-05-15T23:03:56.576073+0000","last_in_change":"2024-05-15T23:03:38.460671+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T22:59:38.603575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80159cc8-1b6c-4aa6-a114-966786d7be1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6801","nonce":1084680531}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6802","nonce":1084680531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6804","nonce":1084680531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6803","nonce":1084680531}]},"public_addr":"172.21.15.84:6801/1084680531","cluster_addr":"172.21.15.84:6802/1084680531","heartbeat_back_addr":"172.21.15.84:6804/1084680531","heartbeat_front_addr":"172.21.15.84:6803/1084680531","state":["exists","up"]},{"osd":1,"uuid":"9c2c76c9-ce0f-4c2d-9d96-c87a10187607","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6805","nonce":2140830229}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6806","nonce":2140830229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6808","nonce":2140830229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6807","nonce":2140830229}]},"public_addr":"172.21.15.84:6805/2140830229","cluster_addr":"172.21.15.84:6806/2140830229","heartbeat_back_addr":"172.21.15.84:6808/2140830229","heartbeat_front_addr":"172.21.15.84:6807/2140830229","state":["exists","up"]},{"osd":2,"uuid":"e8514daa-b5f9-4f2f-a432-53f4a4eb86bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6809","nonce":2699996290}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6810","nonce":2699996290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6812","nonce":2699996290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6811","nonce":2699996290}]},"public_addr":"172.21.15.84:6809/2699996290","cluster_addr":"172.21.15.84:6810/2699996290","heartbeat_back_addr":"172.21.15.84:6812/2699996290","heartbeat_front_addr":"172.21.15.84:6811/2699996290","state":["exists","up"]},{"osd":3,"uuid":"5d9965f9-c0b2-48ce-869c-582990985623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6813","nonce":2204039828}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6814","nonce":2204039828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6816","nonce":2204039828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6815","nonce":2204039828}]},"public_addr":"172.21.15.84:6813/2204039828","cluster_addr":"172.21.15.84:6814/2204039828","heartbeat_back_addr":"172.21.15.84:6816/2204039828","heartbeat_front_addr":"172.21.15.84:6815/2204039828","state":["exists","up"]},{"osd":4,"uuid":"4bc50523-8480-44d2-8d76-31479e839c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6800","nonce":2267955292}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":2267955292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":2267955292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":2267955292}]},"public_addr":"172.21.15.88:6800/2267955292","cluster_addr":"172.21.15.88:6801/2267955292","heartbeat_back_addr":"172.21.15.88:6803/2267955292","heartbeat_front_addr":"172.21.15.88:6802/2267955292","state":["exists","up"]},{"osd":5,"uuid":"d4db8d7f-afc7-43dc-81fe-f70c290d8b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":661041538}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":661041538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":661041538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":661041538}]},"public_addr":"172.21.15.88:6804/661041538","cluster_addr":"172.21.15.88:6805/661041538","heartbeat_back_addr":"172.21.15.88:6807/661041538","heartbeat_front_addr":"172.21.15.88:6806/661041538","state":["exists","up"]},{"osd":6,"uuid":"a2773454-2318-4a5b-8f31-0952b1b07f2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":380856869}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":380856869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":380856869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":380856869}]},"public_addr":"172.21.15.88:6808/380856869","cluster_addr":"172.21.15.88:6809/380856869","heartbeat_back_addr":"172.21.15.88:6811/380856869","heartbeat_front_addr":"172.21.15.88:6810/380856869","state":["exists","up"]},{"osd":7,"uuid":"028ca7b6-3e00-4466-a33f-b726cd60cb34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":1044709063}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":1044709063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":1044709063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":1044709063}]},"public_addr":"172.21.15.88:6812/1044709063","cluster_addr":"172.21.15.88:6813/1044709063","heartbeat_back_addr":"172.21.15.88:6815/1044709063","heartbeat_front_addr":"172.21.15.88:6814/1044709063","state":["exists","up"]},{"osd":8,"uuid":"02cab286-d487-4a66-982c-3cf28f55e851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6800","nonce":522459126}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6801","nonce":522459126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6803","nonce":522459126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6802","nonce":522459126}]},"public_addr":"172.21.15.90:6800/522459126","cluster_addr":"172.21.15.90:6801/522459126","heartbeat_back_addr":"172.21.15.90:6803/522459126","heartbeat_front_addr":"172.21.15.90:6802/522459126","state":["exists","up"]},{"osd":9,"uuid":"47d38bc7-39ba-4b35-af24-e282354dd613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6804","nonce":2794571133}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6805","nonce":2794571133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6807","nonce":2794571133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6806","nonce":2794571133}]},"public_addr":"172.21.15.90:6804/2794571133","cluster_addr":"172.21.15.90:6805/2794571133","heartbeat_back_addr":"172.21.15.90:6807/2794571133","heartbeat_front_addr":"172.21.15.90:6806/2794571133","state":["exists","up"]},{"osd":10,"uuid":"d7b2aeae-d012-41eb-9f91-e4c05c649473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6808","nonce":3735380744}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6809","nonce":3735380744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6811","nonce":3735380744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6810","nonce":3735380744}]},"public_addr":"172.21.15.90:6808/3735380744","cluster_addr":"172.21.15.90:6809/3735380744","heartbeat_back_addr":"172.21.15.90:6811/3735380744","heartbeat_front_addr":"172.21.15.90:6810/3735380744","state":["exists","up"]},{"osd":11,"uuid":"2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6812","nonce":2265615270}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6813","nonce":2265615270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6815","nonce":2265615270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6814","nonce":2265615270}]},"public_addr":"172.21.15.90:6812/2265615270","cluster_addr":"172.21.15.90:6813/2265615270","heartbeat_back_addr":"172.21.15.90:6815/2265615270","heartbeat_front_addr":"172.21.15.90:6814/2265615270","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:02:54.736546+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:28.444415+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:04:00.294225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:52.123662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:04:14.117098+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:43.452092+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:04:10.694461+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:28.064342+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:04:03.061558+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:04:18.910306+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:28.234215+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:03:55.433179+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/2581408231":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/1136984393":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/2443201864":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1252671888":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1451001371":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/4084814866":"2024-05-16T22:57:38.505263+0000","172.21.15.84:0/4076409285":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/2988977694":"2024-05-16T22:57:38.505263+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T23:04:23.736 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-15T23:04:23.736 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-15T23:04:23.737 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-15T23:04:23.737 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-15T23:04:23.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-15T23:04:23.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-15T23:04:23.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-15T23:04:23.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-15T23:04:23.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-15T23:04:23.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-15T23:04:23.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-15T23:04:23.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-15T23:04:23.973 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:23 smithi084 ceph-mon[39784]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:23.973 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:23 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3854128760' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:04:24.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:23 smithi090 ceph-mon[44434]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:24.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:23 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3854128760' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:04:24.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:23 smithi088 ceph-mon[44699]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:24.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:23 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3854128760' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:04:24.424 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:24.752 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:24.752 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:24.762 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:24.762 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:24.766 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:24.770 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:24.782 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:24.783 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:25.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:24 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:04:25.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:24 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:04:25.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:24 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:04:25.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:24 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:04:25.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:24 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:04:25.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:24 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:04:25.318 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:25.318 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:25.820 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:25 smithi084 ceph-mon[39784]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:25.820 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:25 smithi084 ceph-mon[39784]: osdmap e89: 12 total, 12 up, 12 in 2024-05-15T23:04:26.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:25 smithi090 ceph-mon[44434]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:26.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:25 smithi090 ceph-mon[44434]: osdmap e89: 12 total, 12 up, 12 in 2024-05-15T23:04:26.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:25 smithi088 ceph-mon[44699]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:26.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:25 smithi088 ceph-mon[44699]: osdmap e89: 12 total, 12 up, 12 in 2024-05-15T23:04:27.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:27 smithi084 ceph-mon[39784]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:27 smithi090 ceph-mon[44434]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:27 smithi088 ceph-mon[44699]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:29 smithi090 ceph-mon[44434]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:30.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:29 smithi088 ceph-mon[44699]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:29 smithi084 ceph-mon[39784]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:31.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:31 smithi084 ceph-mon[39784]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:32.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:31 smithi090 ceph-mon[44434]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:32.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:31 smithi088 ceph-mon[44699]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:33.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:33 smithi084 ceph-mon[39784]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:34.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:33 smithi090 ceph-mon[44434]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:34.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:33 smithi088 ceph-mon[44699]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:35.907 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:35 smithi084 ceph-mon[39784]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:36.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:35 smithi090 ceph-mon[44434]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:36.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:35 smithi088 ceph-mon[44699]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:37.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:37 smithi084 ceph-mon[39784]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:37.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:37 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:04:37.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:37 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:04:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:37 smithi090 ceph-mon[44434]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:38.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:37 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:04:38.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:37 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:04:38.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:37 smithi088 ceph-mon[44699]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:38.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:37 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:04:38.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:37 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:04:38.408 INFO:teuthology.orchestra.run.smithi084.stdout:158913789992 2024-05-15T23:04:38.409 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-15T23:04:38.706 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:38 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:04:38.706 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:38 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:04:38.706 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:38 smithi084 ceph-mon[39784]: osdmap e90: 12 total, 12 up, 12 in 2024-05-15T23:04:39.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:38 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:04:39.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:38 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:04:39.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:38 smithi090 ceph-mon[44434]: osdmap e90: 12 total, 12 up, 12 in 2024-05-15T23:04:39.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:38 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:04:39.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:38 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:04:39.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:38 smithi088 ceph-mon[44699]: osdmap e90: 12 total, 12 up, 12 in 2024-05-15T23:04:39.607 INFO:teuthology.orchestra.run.smithi084.stdout:55834574909 2024-05-15T23:04:39.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-15T23:04:39.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:39 smithi084 ceph-mon[39784]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:39.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:39 smithi084 ceph-mon[39784]: osdmap e91: 12 total, 12 up, 12 in 2024-05-15T23:04:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:39 smithi090 ceph-mon[44434]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:39 smithi090 ceph-mon[44434]: osdmap e91: 12 total, 12 up, 12 in 2024-05-15T23:04:40.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:39 smithi088 ceph-mon[44699]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:40.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:39 smithi088 ceph-mon[44699]: osdmap e91: 12 total, 12 up, 12 in 2024-05-15T23:04:41.624 INFO:teuthology.orchestra.run.smithi084.stdout:85899345977 2024-05-15T23:04:41.624 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-15T23:04:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:41 smithi084 ceph-mon[39784]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:42.044 INFO:teuthology.orchestra.run.smithi084.stdout:292057776150 2024-05-15T23:04:42.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-15T23:04:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:41 smithi090 ceph-mon[44434]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:41 smithi088 ceph-mon[44699]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:42.841 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:43 smithi084 ceph-mon[39784]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:44.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:43 smithi090 ceph-mon[44434]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:44.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:43 smithi088 ceph-mon[44699]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:44.805 INFO:teuthology.orchestra.run.smithi084.stdout:317827579919 2024-05-15T23:04:44.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-15T23:04:45.744 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:45 smithi084 ceph-mon[39784]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:45 smithi090 ceph-mon[44434]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:46.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:45 smithi088 ceph-mon[44699]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:46.918 INFO:teuthology.orchestra.run.smithi084.stdout:227633266720 2024-05-15T23:04:46.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-15T23:04:47.554 INFO:teuthology.orchestra.run.smithi084.stdout:347892350987 2024-05-15T23:04:47.554 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-15T23:04:47.839 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:47 smithi084 ceph-mon[39784]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:47.871 INFO:teuthology.orchestra.run.smithi084.stdout:111669149749 2024-05-15T23:04:47.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-15T23:04:48.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:47 smithi090 ceph-mon[44434]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:48.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:47 smithi088 ceph-mon[44699]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:49.744 INFO:teuthology.orchestra.run.smithi084.stdout:133143986224 2024-05-15T23:04:49.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-15T23:04:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:49 smithi084 ceph-mon[39784]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:50.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:49 smithi090 ceph-mon[44434]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:50.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:49 smithi088 ceph-mon[44699]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:50.168 INFO:teuthology.orchestra.run.smithi084.stdout:197568495655 2024-05-15T23:04:50.168 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-15T23:04:50.596 INFO:teuthology.orchestra.run.smithi084.stdout:261993005085 2024-05-15T23:04:50.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-15T23:04:50.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:50 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:04:50.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:50 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:04:50.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:50 smithi084 ceph-mon[39784]: osdmap e92: 12 total, 12 up, 12 in 2024-05-15T23:04:50.949 INFO:teuthology.orchestra.run.smithi084.stdout:34359738438 2024-05-15T23:04:50.949 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-15T23:04:50.990 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:51.017 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:51.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:50 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:04:51.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:50 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:04:51.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:50 smithi090 ceph-mon[44434]: osdmap e92: 12 total, 12 up, 12 in 2024-05-15T23:04:51.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:50 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:04:51.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:50 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:04:51.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:50 smithi088 ceph-mon[44699]: osdmap e92: 12 total, 12 up, 12 in 2024-05-15T23:04:52.031 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:51 smithi084 ceph-mon[39784]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:52.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:51 smithi090 ceph-mon[44434]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:52.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:51 smithi088 ceph-mon[44699]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:52.359 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:52.363 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:52.363 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:52.366 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:52.366 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:52.372 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:52.372 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:52.373 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:52.373 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:04:53.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:53 smithi084 ceph-mon[39784]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:54.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:53 smithi090 ceph-mon[44434]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:54.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:53 smithi088 ceph-mon[44699]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:55.635 INFO:teuthology.orchestra.run.smithi084.stdout:317827579922 2024-05-15T23:04:55.812 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:55 smithi084 ceph-mon[39784]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:56.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:55 smithi090 ceph-mon[44434]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:56.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:55 smithi088 ceph-mon[44699]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:56.509 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579919 got 317827579922 for osd.10 2024-05-15T23:04:56.510 DEBUG:teuthology.parallel:result is None 2024-05-15T23:04:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/141243582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-15T23:04:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/141243582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-15T23:04:57.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/141243582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-15T23:04:57.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:57 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:04:57.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:57 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:04:57.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:57 smithi084 ceph-mon[39784]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:57.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:57 smithi084 ceph-mon[39784]: osdmap e93: 12 total, 12 up, 12 in 2024-05-15T23:04:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:57 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:04:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:57 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:04:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:57 smithi090 ceph-mon[44434]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:57 smithi090 ceph-mon[44434]: osdmap e93: 12 total, 12 up, 12 in 2024-05-15T23:04:58.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:57 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:04:58.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:57 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:04:58.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:57 smithi088 ceph-mon[44699]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:04:58.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:57 smithi088 ceph-mon[44699]: osdmap e93: 12 total, 12 up, 12 in 2024-05-15T23:05:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:59 smithi090 ceph-mon[44434]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:04:59 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:05:00.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:59 smithi088 ceph-mon[44699]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:00.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:04:59 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:05:00.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:59 smithi084 ceph-mon[39784]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:00.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:04:59 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:05:00.319 INFO:teuthology.orchestra.run.smithi084.stdout:85899345981 2024-05-15T23:05:00.982 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:05:00.983 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:05:00.983 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:05:00.983 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:00 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/998795711' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-15T23:05:01.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:05:01.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:05:01.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:05:01.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:00 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/998795711' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-15T23:05:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:05:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:05:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:05:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:00 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/998795711' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-15T23:05:01.917 INFO:teuthology.orchestra.run.smithi084.stdout:158913789997 2024-05-15T23:05:02.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:01 smithi084 ceph-mon[39784]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:01 smithi090 ceph-mon[44434]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:01 smithi088 ceph-mon[44699]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3953381444' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-15T23:05:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3953381444' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-15T23:05:03.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3953381444' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-15T23:05:03.944 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:03 smithi084 ceph-mon[39784]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:04.084 INFO:teuthology.orchestra.run.smithi084.stdout:292057776155 2024-05-15T23:05:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:03 smithi090 ceph-mon[44434]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:03 smithi088 ceph-mon[44699]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:04.448 INFO:teuthology.orchestra.run.smithi084.stdout:133143986227 2024-05-15T23:05:04.752 INFO:tasks.cephadm.ceph_manager.ceph:need seq 85899345977 got 85899345981 for osd.2 2024-05-15T23:05:04.753 DEBUG:teuthology.parallel:result is None 2024-05-15T23:05:05.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2427872385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-15T23:05:05.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2330489768' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-15T23:05:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2427872385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-15T23:05:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2330489768' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-15T23:05:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2427872385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-15T23:05:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2330489768' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-15T23:05:05.886 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776150 got 292057776155 for osd.9 2024-05-15T23:05:05.886 DEBUG:teuthology.parallel:result is None 2024-05-15T23:05:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:05 smithi084 ceph-mon[39784]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:05 smithi090 ceph-mon[44434]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:06.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:05 smithi088 ceph-mon[44699]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:06.860 INFO:teuthology.orchestra.run.smithi084.stdout:261993005089 2024-05-15T23:05:07.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:06 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:05:07.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:06 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:05:07.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:06 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3084568374' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-15T23:05:07.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:06 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:05:07.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:06 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:05:07.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:06 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3084568374' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-15T23:05:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:06 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:05:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:06 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:05:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:06 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3084568374' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-15T23:05:07.705 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789992 got 158913789997 for osd.5 2024-05-15T23:05:07.705 DEBUG:teuthology.parallel:result is None 2024-05-15T23:05:07.932 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:07 smithi084 ceph-mon[39784]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:07.932 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:07 smithi084 ceph-mon[39784]: osdmap e94: 12 total, 12 up, 12 in 2024-05-15T23:05:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:07 smithi090 ceph-mon[44434]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:07 smithi090 ceph-mon[44434]: osdmap e94: 12 total, 12 up, 12 in 2024-05-15T23:05:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:07 smithi088 ceph-mon[44699]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:07 smithi088 ceph-mon[44699]: osdmap e94: 12 total, 12 up, 12 in 2024-05-15T23:05:09.773 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005085 got 261993005089 for osd.8 2024-05-15T23:05:09.773 DEBUG:teuthology.parallel:result is None 2024-05-15T23:05:09.872 INFO:teuthology.orchestra.run.smithi084.stdout:55834574916 2024-05-15T23:05:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:09 smithi084 ceph-mon[39784]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:09 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1112554237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-15T23:05:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:09 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/416468502' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-15T23:05:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:09 smithi090 ceph-mon[44434]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:09 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1112554237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-15T23:05:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:09 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/416468502' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-15T23:05:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:09 smithi088 ceph-mon[44699]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:09 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1112554237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-15T23:05:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:09 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/416468502' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-15T23:05:10.617 INFO:teuthology.orchestra.run.smithi084.stdout:227633266725 2024-05-15T23:05:10.633 INFO:teuthology.orchestra.run.smithi084.stdout:111669149755 2024-05-15T23:05:11.243 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:10 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1552320640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-15T23:05:11.269 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266720 got 227633266725 for osd.7 2024-05-15T23:05:11.270 DEBUG:teuthology.parallel:result is None 2024-05-15T23:05:11.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:10 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1552320640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-15T23:05:11.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:10 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1552320640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-15T23:05:11.978 INFO:teuthology.orchestra.run.smithi084.stdout:347892350992 2024-05-15T23:05:12.116 INFO:teuthology.orchestra.run.smithi084.stdout:197568495659 2024-05-15T23:05:12.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:11 smithi090 ceph-mon[44434]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:12.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:11 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:05:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:11 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:05:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:11 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:05:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:11 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:05:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:11 smithi088 ceph-mon[44699]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:11 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:05:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:11 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:05:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:11 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:05:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:11 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:05:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:11 smithi084 ceph-mon[39784]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:11 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:05:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:11 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:05:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:11 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:05:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:11 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:05:12.853 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495655 got 197568495659 for osd.6 2024-05-15T23:05:12.854 DEBUG:teuthology.parallel:result is None 2024-05-15T23:05:13.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:12 smithi084 ceph-mon[39784]: osdmap e95: 12 total, 12 up, 12 in 2024-05-15T23:05:13.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:12 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2149975464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-15T23:05:13.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:12 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/758009653' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-15T23:05:13.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:12 smithi090 ceph-mon[44434]: osdmap e95: 12 total, 12 up, 12 in 2024-05-15T23:05:13.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:12 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2149975464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-15T23:05:13.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:12 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/758009653' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-15T23:05:13.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:12 smithi088 ceph-mon[44699]: osdmap e95: 12 total, 12 up, 12 in 2024-05-15T23:05:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:12 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2149975464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-15T23:05:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:12 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/758009653' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-15T23:05:13.418 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149749 got 111669149755 for osd.3 2024-05-15T23:05:13.418 DEBUG:teuthology.parallel:result is None 2024-05-15T23:05:14.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:13 smithi084 ceph-mon[39784]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:14.115 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:14.152 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350987 got 347892350992 for osd.11 2024-05-15T23:05:14.152 DEBUG:teuthology.parallel:result is None 2024-05-15T23:05:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:13 smithi090 ceph-mon[44434]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:14.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:13 smithi088 ceph-mon[44699]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:14.647 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574909 got 55834574916 for osd.1 2024-05-15T23:05:14.647 DEBUG:teuthology.parallel:result is None 2024-05-15T23:05:14.789 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986224 got 133143986227 for osd.4 2024-05-15T23:05:14.789 DEBUG:teuthology.parallel:result is None 2024-05-15T23:05:16.261 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:15 smithi084 ceph-mon[39784]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:16.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:15 smithi090 ceph-mon[44434]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:16.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:15 smithi088 ceph-mon[44699]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:16.734 INFO:teuthology.orchestra.run.smithi084.stdout:34359738444 2024-05-15T23:05:17.072 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:16 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:05:17.072 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:16 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:05:17.072 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:16 smithi084 ceph-mon[39784]: osdmap e96: 12 total, 12 up, 12 in 2024-05-15T23:05:17.073 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:16 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/778761911' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-15T23:05:17.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:16 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:05:17.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:16 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:05:17.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:16 smithi090 ceph-mon[44434]: osdmap e96: 12 total, 12 up, 12 in 2024-05-15T23:05:17.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:16 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/778761911' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-15T23:05:17.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:16 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:05:17.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:16 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:05:17.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:16 smithi088 ceph-mon[44699]: osdmap e96: 12 total, 12 up, 12 in 2024-05-15T23:05:17.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:16 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/778761911' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-15T23:05:17.487 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738438 got 34359738444 for osd.0 2024-05-15T23:05:17.488 DEBUG:teuthology.parallel:result is None 2024-05-15T23:05:17.488 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-15T23:05:17.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:05:17.754 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:18.283 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:17 smithi084 ceph-mon[39784]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:18.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:17 smithi090 ceph-mon[44434]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:18.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:17 smithi088 ceph-mon[44699]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:19.696 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:05:19.699 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:05:20.043 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:19 smithi084 ceph-mon[39784]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:20.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:19 smithi090 ceph-mon[44434]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:20.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:19 smithi088 ceph-mon[44699]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:20.399 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-05-15T23:05:18.597561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":337340,"kb_used_data":8060,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124604484,"statfs":{"total":1151940427776,"available":1151594991616,"internally_reserved":0,"allocated":8253440,"data_stored":4335019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":336115002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001462"},"pg_stats":[{"pgid":"1.0","version":"57'94","reported_seq":141,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-05-15T23:05:11.924634+0000","last_change":"2024-05-15T23:05:11.075182+0000","last_active":"2024-05-15T23:05:11.924634+0000","last_peered":"2024-05-15T23:05:11.924634+0000","last_clean":"2024-05-15T23:05:11.924634+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:05:11.924634+0000","last_undegraded":"2024-05-15T23:05:11.924634+0000","last_fullsized":"2024-05-15T23:05:11.924634+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:05:11.075069+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:05:11.075069+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:06:21.551937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350994,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28204,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716948,"statfs":{"total":95995035648,"available":95966154752,"internally_reserved":0,"allocated":1130496,"data_stored":801539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":598016,"data_stored":272631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776158,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":598016,"data_stored":272631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005091,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":598016,"data_stored":272631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28204,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716948,"statfs":{"total":95995035648,"available":95966154752,"internally_reserved":0,"allocated":1130496,"data_stored":801559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717536,"statfs":{"total":95995035648,"available":95966756864,"internally_reserved":0,"allocated":610304,"data_stored":278242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":598016,"data_stored":272631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31844,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713308,"statfs":{"total":95995035648,"available":95962427392,"internally_reserved":0,"allocated":598016,"data_stored":272631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345985,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":598016,"data_stored":272631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":598016,"data_stored":272631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986230,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":598016,"data_stored":272631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":598016,"data_stored":272631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:05:20.400 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:05:20.662 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:21.160 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:20 smithi084 ceph-mon[39784]: from='client.14745 v1:172.21.15.84:0/667685223' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:05:21.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:20 smithi090 ceph-mon[44434]: from='client.14745 v1:172.21.15.84:0/667685223' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:05:21.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:20 smithi088 ceph-mon[44699]: from='client.14745 v1:172.21.15.84:0/667685223' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:05:22.197 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:21 smithi084 ceph-mon[39784]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:22.197 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:21 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:05:22.197 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:21 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:05:22.197 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:21 smithi084 ceph-mon[39784]: osdmap e97: 12 total, 12 up, 12 in 2024-05-15T23:05:22.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:21 smithi090 ceph-mon[44434]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:22.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:21 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:05:22.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:21 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:05:22.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:21 smithi090 ceph-mon[44434]: osdmap e97: 12 total, 12 up, 12 in 2024-05-15T23:05:22.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:21 smithi088 ceph-mon[44699]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:22.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:21 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:05:22.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:21 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:05:22.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:21 smithi088 ceph-mon[44699]: osdmap e97: 12 total, 12 up, 12 in 2024-05-15T23:05:22.662 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:05:22.664 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:05:23.434 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-05-15T23:05:22.598295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":336604,"kb_used_data":8204,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124605220,"statfs":{"total":1151940427776,"available":1151595745280,"internally_reserved":0,"allocated":8400896,"data_stored":4402351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":336115002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001349"},"pg_stats":[{"pgid":"1.0","version":"57'94","reported_seq":143,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-05-15T23:05:15.936465+0000","last_change":"2024-05-15T23:05:11.075182+0000","last_active":"2024-05-15T23:05:15.936465+0000","last_peered":"2024-05-15T23:05:15.936465+0000","last_clean":"2024-05-15T23:05:15.936465+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:05:15.936465+0000","last_undegraded":"2024-05-15T23:05:15.936465+0000","last_fullsized":"2024-05-15T23:05:15.936465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:05:11.075069+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:05:11.075069+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:06:21.551937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350995,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1142784,"data_stored":807150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579928,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717536,"statfs":{"total":95995035648,"available":95966756864,"internally_reserved":0,"allocated":610304,"data_stored":278242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776159,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717536,"statfs":{"total":95995035648,"available":95966756864,"internally_reserved":0,"allocated":610304,"data_stored":278242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005092,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717536,"statfs":{"total":95995035648,"available":95966756864,"internally_reserved":0,"allocated":610304,"data_stored":278242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1142784,"data_stored":807170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27628,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717524,"statfs":{"total":95995035648,"available":95966744576,"internally_reserved":0,"allocated":622592,"data_stored":283853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717536,"statfs":{"total":95995035648,"available":95966756864,"internally_reserved":0,"allocated":610304,"data_stored":278242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31776,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713376,"statfs":{"total":95995035648,"available":95962497024,"internally_reserved":0,"allocated":610304,"data_stored":278242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345986,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717536,"statfs":{"total":95995035648,"available":95966756864,"internally_reserved":0,"allocated":610304,"data_stored":278242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717536,"statfs":{"total":95995035648,"available":95966756864,"internally_reserved":0,"allocated":610304,"data_stored":278242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986231,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717536,"statfs":{"total":95995035648,"available":95966756864,"internally_reserved":0,"allocated":610304,"data_stored":278242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717536,"statfs":{"total":95995035648,"available":95966756864,"internally_reserved":0,"allocated":610304,"data_stored":278242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:05:23.435 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-15T23:05:23.435 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-15T23:05:23.435 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-15T23:05:23.436 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph health --format=json 2024-05-15T23:05:23.702 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:24.220 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:23 smithi084 ceph-mon[39784]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:24.220 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:23 smithi084 ceph-mon[39784]: from='client.14751 v1:172.21.15.84:0/982593851' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:05:24.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:23 smithi090 ceph-mon[44434]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:24.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:23 smithi090 ceph-mon[44434]: from='client.14751 v1:172.21.15.84:0/982593851' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:05:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:23 smithi088 ceph-mon[44699]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:23 smithi088 ceph-mon[44699]: from='client.14751 v1:172.21.15.84:0/982593851' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:05:25.292 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:24 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:05:25.292 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:24 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:05:25.292 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:24 smithi084 ceph-mon[39784]: osdmap e98: 12 total, 12 up, 12 in 2024-05-15T23:05:25.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:24 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:05:25.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:24 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:05:25.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:24 smithi090 ceph-mon[44434]: osdmap e98: 12 total, 12 up, 12 in 2024-05-15T23:05:25.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:24 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:05:25.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:24 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:05:25.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:24 smithi088 ceph-mon[44699]: osdmap e98: 12 total, 12 up, 12 in 2024-05-15T23:05:25.796 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:05:25.796 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-15T23:05:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:25 smithi084 ceph-mon[39784]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:25 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3405650967' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-15T23:05:26.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:25 smithi090 ceph-mon[44434]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:26.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:25 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3405650967' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-15T23:05:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:25 smithi088 ceph-mon[44699]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:25 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3405650967' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-15T23:05:26.502 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-15T23:05:26.502 INFO:tasks.cephadm:Setup complete, yielding 2024-05-15T23:05:26.503 INFO:teuthology.run_tasks:Running task exec... 2024-05-15T23:05:26.513 INFO:teuthology.task.exec:Executing custom commands... 2024-05-15T23:05:26.513 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi084.front.sepia.ceph.com 2024-05-15T23:05:26.513 DEBUG:teuthology.orchestra.run.smithi084:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! ceph balancer status ; do sleep 1 ; done' 2024-05-15T23:05:27.043 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:05:27.044 INFO:teuthology.orchestra.run.smithi084.stdout: "active": true, 2024-05-15T23:05:27.044 INFO:teuthology.orchestra.run.smithi084.stdout: "last_optimize_duration": "0:00:00.000267", 2024-05-15T23:05:27.044 INFO:teuthology.orchestra.run.smithi084.stdout: "last_optimize_started": "Wed May 15 23:04:38 2024", 2024-05-15T23:05:27.044 INFO:teuthology.orchestra.run.smithi084.stdout: "mode": "upmap", 2024-05-15T23:05:27.044 INFO:teuthology.orchestra.run.smithi084.stdout: "no_optimization_needed": true, 2024-05-15T23:05:27.044 INFO:teuthology.orchestra.run.smithi084.stdout: "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", 2024-05-15T23:05:27.044 INFO:teuthology.orchestra.run.smithi084.stdout: "plans": [] 2024-05-15T23:05:27.044 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:05:27.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode crush-compat' 2024-05-15T23:05:28.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2024-05-15T23:05:28.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:27 smithi090 ceph-mon[44434]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:28.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:27 smithi090 ceph-mon[44434]: from='client.14763 v1:172.21.15.84:0/1368186194' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mgr", ""]}]: dispatch 2024-05-15T23:05:28.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:27 smithi090 ceph-mon[44434]: from='client.14769 v1:172.21.15.84:0/2306217477' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mgr", ""]}]: dispatch 2024-05-15T23:05:28.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-05-15T23:05:28.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:27 smithi088 ceph-mon[44699]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:28.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:27 smithi088 ceph-mon[44699]: from='client.14763 v1:172.21.15.84:0/1368186194' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mgr", ""]}]: dispatch 2024-05-15T23:05:28.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:27 smithi088 ceph-mon[44699]: from='client.14769 v1:172.21.15.84:0/2306217477' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mgr", ""]}]: dispatch 2024-05-15T23:05:28.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-05-15T23:05:28.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:27 smithi084 ceph-mon[39784]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:28.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:27 smithi084 ceph-mon[39784]: from='client.14763 v1:172.21.15.84:0/1368186194' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mgr", ""]}]: dispatch 2024-05-15T23:05:28.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:27 smithi084 ceph-mon[39784]: from='client.14769 v1:172.21.15.84:0/2306217477' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mgr", ""]}]: dispatch 2024-05-15T23:05:28.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-05-15T23:05:28.378 INFO:teuthology.run_tasks:Running task thrashosds... 2024-05-15T23:05:28.388 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-05-15T23:05:28.388 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuri5-testing-2024-05-15-0804'}, 'ceph': {'conf': {'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd heartbeat use min delay socket': True, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'mon_bind_msgr2': False, 'sha1': '6cd0f8013e2dea00c3a29a0d8b10656b132d7c80'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '6cd0f8013e2dea00c3a29a0d8b10656b132d7c80'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri5-testing-2024-05-15-0804', 'sha1': '6cd0f8013e2dea00c3a29a0d8b10656b132d7c80'}} 2024-05-15T23:05:28.388 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-05-15T23:05:28.389 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-05-15T23:05:28.649 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-05-15T23:05:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:28 smithi084 ceph-mon[39784]: osdmap e99: 12 total, 12 up, 12 in 2024-05-15T23:05:29.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-05-15T23:05:29.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:05:29.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:05:29.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:05:29.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:05:29.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:05:29.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:28 smithi084 ceph-mon[39784]: from='client.14775 v1:172.21.15.84:0/3060132271' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mgr", ""]}]: dispatch 2024-05-15T23:05:29.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-05-15T23:05:29.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:28 smithi090 ceph-mon[44434]: osdmap e99: 12 total, 12 up, 12 in 2024-05-15T23:05:29.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-05-15T23:05:29.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:05:29.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:05:29.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:05:29.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:05:29.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:05:29.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:28 smithi090 ceph-mon[44434]: from='client.14775 v1:172.21.15.84:0/3060132271' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mgr", ""]}]: dispatch 2024-05-15T23:05:29.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-05-15T23:05:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:28 smithi088 ceph-mon[44699]: osdmap e99: 12 total, 12 up, 12 in 2024-05-15T23:05:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-05-15T23:05:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:05:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:05:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:05:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:05:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:05:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:28 smithi088 ceph-mon[44699]: from='client.14775 v1:172.21.15.84:0/3060132271' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mgr", ""]}]: dispatch 2024-05-15T23:05:30.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:30 smithi090 ceph-mon[44434]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:30.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:30 smithi088 ceph-mon[44699]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:30.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:30 smithi084 ceph-mon[39784]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:31.557 INFO:tasks.thrashosds:Beginning thrashosds... 2024-05-15T23:05:31.557 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-15T23:05:31.557 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:05:31.820 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:32.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:31 smithi084 ceph-mon[39784]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:32.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/386120454' entity='client.admin' 2024-05-15T23:05:32.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:05:32.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:05:32.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:05:32.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:05:32.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:31 smithi090 ceph-mon[44434]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:32.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/386120454' entity='client.admin' 2024-05-15T23:05:32.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:05:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:05:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:05:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:05:32.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:31 smithi088 ceph-mon[44699]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:32.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/386120454' entity='client.admin' 2024-05-15T23:05:32.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:05:32.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:05:32.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:05:32.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:05:33.799 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:05:33.801 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:05:34.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:33 smithi084 ceph-mon[39784]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:33 smithi090 ceph-mon[44434]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:33 smithi088 ceph-mon[44699]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:34.524 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-05-15T23:05:32.600191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":336988,"kb_used_data":8588,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124604836,"statfs":{"total":1151940427776,"available":1151595352064,"internally_reserved":0,"allocated":8794112,"data_stored":4596987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":336115002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.646955"},"pg_stats":[{"pgid":"1.0","version":"57'94","reported_seq":149,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-05-15T23:05:29.172244+0000","last_change":"2024-05-15T23:05:11.075182+0000","last_active":"2024-05-15T23:05:29.172244+0000","last_peered":"2024-05-15T23:05:29.172244+0000","last_clean":"2024-05-15T23:05:29.172244+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:05:29.172244+0000","last_undegraded":"2024-05-15T23:05:29.172244+0000","last_fullsized":"2024-05-15T23:05:29.172244+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:05:11.075069+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:05:11.075069+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:06:21.551937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350997,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1179648,"data_stored":825659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579930,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776161,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005094,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1167360,"data_stored":818392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495664,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717512,"statfs":{"total":95995035648,"available":95966732288,"internally_reserved":0,"allocated":634880,"data_stored":289464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31800,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713352,"statfs":{"total":95995035648,"available":95962472448,"internally_reserved":0,"allocated":634880,"data_stored":289464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345988,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986233,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:05:34.525 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:05:34.790 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:35.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:34 smithi090 ceph-mon[44434]: from='client.14787 v1:172.21.15.84:0/1131426593' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:05:35.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:34 smithi088 ceph-mon[44699]: from='client.14787 v1:172.21.15.84:0/1131426593' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:05:35.358 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:34 smithi084 ceph-mon[39784]: from='client.14787 v1:172.21.15.84:0/1131426593' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:05:35.933 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:35 smithi084 ceph-mon[39784]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:36.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:35 smithi090 ceph-mon[44434]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:36.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:35 smithi088 ceph-mon[44699]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:36.705 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:05:36.708 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:05:37.076 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:36 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:05:37.076 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:36 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:05:37.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:36 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:05:37.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:36 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:05:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:36 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:05:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:36 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:05:37.424 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-05-15T23:05:36.600888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":337024,"kb_used_data":8624,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124604800,"statfs":{"total":1151940427776,"available":1151595315200,"internally_reserved":0,"allocated":8830976,"data_stored":4618848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":336115002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001722"},"pg_stats":[{"pgid":"1.0","version":"57'94","reported_seq":149,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-05-15T23:05:29.172244+0000","last_change":"2024-05-15T23:05:11.075182+0000","last_active":"2024-05-15T23:05:29.172244+0000","last_peered":"2024-05-15T23:05:29.172244+0000","last_clean":"2024-05-15T23:05:29.172244+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:05:29.172244+0000","last_undegraded":"2024-05-15T23:05:29.172244+0000","last_fullsized":"2024-05-15T23:05:29.172244+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:05:11.075069+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:05:11.075069+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:06:21.551937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350997,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1179648,"data_stored":825659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579931,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776162,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005095,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1179648,"data_stored":825679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495664,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31812,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713340,"statfs":{"total":95995035648,"available":95962460160,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986234,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":647168,"data_stored":296751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:05:37.425 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-15T23:05:37.425 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump 2024-05-15T23:05:37.689 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:38.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:37 smithi084 ceph-mon[39784]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:38.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:37 smithi084 ceph-mon[39784]: from='client.14793 v1:172.21.15.84:0/3665989604' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:05:38.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:37 smithi084 ceph-mon[39784]: osdmap e100: 12 total, 12 up, 12 in 2024-05-15T23:05:38.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:37 smithi090 ceph-mon[44434]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:38.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:37 smithi090 ceph-mon[44434]: from='client.14793 v1:172.21.15.84:0/3665989604' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:05:38.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:37 smithi090 ceph-mon[44434]: osdmap e100: 12 total, 12 up, 12 in 2024-05-15T23:05:38.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:37 smithi088 ceph-mon[44699]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:37 smithi088 ceph-mon[44699]: from='client.14793 v1:172.21.15.84:0/3665989604' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:05:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:37 smithi088 ceph-mon[44699]: osdmap e100: 12 total, 12 up, 12 in 2024-05-15T23:05:38.946 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:38 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:05:38.946 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:38 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:05:38.946 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:38 smithi084 ceph-mon[39784]: osdmap e101: 12 total, 12 up, 12 in 2024-05-15T23:05:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:38 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:05:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:38 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:05:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:38 smithi090 ceph-mon[44434]: osdmap e101: 12 total, 12 up, 12 in 2024-05-15T23:05:39.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:38 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:05:39.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:38 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:05:39.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:38 smithi088 ceph-mon[44699]: osdmap e101: 12 total, 12 up, 12 in 2024-05-15T23:05:39.763 INFO:teuthology.orchestra.run.smithi084.stdout:epoch 101 2024-05-15T23:05:39.763 INFO:teuthology.orchestra.run.smithi084.stdout:fsid 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T23:05:39.763 INFO:teuthology.orchestra.run.smithi084.stdout:created 2024-05-15T22:56:41.212493+0000 2024-05-15T23:05:39.763 INFO:teuthology.orchestra.run.smithi084.stdout:modified 2024-05-15T23:05:38.601429+0000 2024-05-15T23:05:39.763 INFO:teuthology.orchestra.run.smithi084.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-15T23:05:39.763 INFO:teuthology.orchestra.run.smithi084.stdout:crush_version 27 2024-05-15T23:05:39.763 INFO:teuthology.orchestra.run.smithi084.stdout:full_ratio 0.95 2024-05-15T23:05:39.763 INFO:teuthology.orchestra.run.smithi084.stdout:backfillfull_ratio 0.9 2024-05-15T23:05:39.763 INFO:teuthology.orchestra.run.smithi084.stdout:nearfull_ratio 0.85 2024-05-15T23:05:39.763 INFO:teuthology.orchestra.run.smithi084.stdout:require_min_compat_client luminous 2024-05-15T23:05:39.763 INFO:teuthology.orchestra.run.smithi084.stdout:min_compat_client jewel 2024-05-15T23:05:39.764 INFO:teuthology.orchestra.run.smithi084.stdout:require_osd_release squid 2024-05-15T23:05:39.764 INFO:teuthology.orchestra.run.smithi084.stdout:stretch_mode_enabled false 2024-05-15T23:05:39.764 INFO:teuthology.orchestra.run.smithi084.stdout:pool 1 '.mgr' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 16 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 11.76 2024-05-15T23:05:39.764 INFO:teuthology.orchestra.run.smithi084.stdout:max_osd 12 2024-05-15T23:05:39.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0 up in weight 1 up_from 8 up_thru 53 down_at 0 last_clean_interval [0,0) v1:172.21.15.84:6801/1084680531 v1:172.21.15.84:6802/1084680531 exists,up 80159cc8-1b6c-4aa6-a114-966786d7be1e 2024-05-15T23:05:39.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) v1:172.21.15.84:6805/2140830229 v1:172.21.15.84:6806/2140830229 exists,up 9c2c76c9-ce0f-4c2d-9d96-c87a10187607 2024-05-15T23:05:39.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.84:6809/2699996290 v1:172.21.15.84:6810/2699996290 exists,up e8514daa-b5f9-4f2f-a432-53f4a4eb86bc 2024-05-15T23:05:39.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.84:6813/2204039828 v1:172.21.15.84:6814/2204039828 exists,up 5d9965f9-c0b2-48ce-869c-582990985623 2024-05-15T23:05:39.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.88:6800/2267955292 v1:172.21.15.88:6801/2267955292 exists,up 4bc50523-8480-44d2-8d76-31479e839c85 2024-05-15T23:05:39.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5 up in weight 1 up_from 37 up_thru 38 down_at 0 last_clean_interval [0,0) v1:172.21.15.88:6804/661041538 v1:172.21.15.88:6805/661041538 exists,up d4db8d7f-afc7-43dc-81fe-f70c290d8b20 2024-05-15T23:05:39.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6 up in weight 1 up_from 46 up_thru 47 down_at 0 last_clean_interval [0,0) v1:172.21.15.88:6808/380856869 v1:172.21.15.88:6809/380856869 exists,up a2773454-2318-4a5b-8f31-0952b1b07f2f 2024-05-15T23:05:39.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7 up in weight 1 up_from 53 up_thru 82 down_at 0 last_clean_interval [0,0) v1:172.21.15.88:6812/1044709063 v1:172.21.15.88:6813/1044709063 exists,up 028ca7b6-3e00-4466-a33f-b726cd60cb34 2024-05-15T23:05:39.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6800/522459126 v1:172.21.15.90:6801/522459126 exists,up 02cab286-d487-4a66-982c-3cf28f55e851 2024-05-15T23:05:39.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9 up in weight 1 up_from 68 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6804/2794571133 v1:172.21.15.90:6805/2794571133 exists,up 47d38bc7-39ba-4b35-af24-e282354dd613 2024-05-15T23:05:39.765 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10 up in weight 1 up_from 74 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6808/3735380744 v1:172.21.15.90:6809/3735380744 exists,up d7b2aeae-d012-41eb-9f91-e4c05c649473 2024-05-15T23:05:39.765 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11 up in weight 1 up_from 81 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6812/2265615270 v1:172.21.15.90:6813/2265615270 exists,up 2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5 2024-05-15T23:05:39.765 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/2581408231 expires 2024-05-16T22:57:38.505263+0000 2024-05-15T23:05:39.765 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:6800/1136984393 expires 2024-05-16T22:57:38.505263+0000 2024-05-15T23:05:39.765 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:6800/2443201864 expires 2024-05-16T22:57:09.403385+0000 2024-05-15T23:05:39.765 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/1252671888 expires 2024-05-16T22:57:09.403385+0000 2024-05-15T23:05:39.765 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/1451001371 expires 2024-05-16T22:57:09.403385+0000 2024-05-15T23:05:39.765 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/4084814866 expires 2024-05-16T22:57:38.505263+0000 2024-05-15T23:05:39.765 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/4076409285 expires 2024-05-16T22:57:09.403385+0000 2024-05-15T23:05:39.765 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/2988977694 expires 2024-05-16T22:57:38.505263+0000 2024-05-15T23:05:40.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:39 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:05:40.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:39 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:05:40.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:39 smithi084 ceph-mon[39784]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:40.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2732102856' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-15T23:05:40.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:39 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:05:40.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:39 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:05:40.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:39 smithi090 ceph-mon[44434]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:40.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2732102856' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-15T23:05:40.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:39 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:05:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:39 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:05:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:39 smithi088 ceph-mon[44699]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2732102856' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-15T23:05:40.488 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 53 down_at 0 last_clean_interval [0,0) v1:172.21.15.84:6801/1084680531 v1:172.21.15.84:6802/1084680531 exists,up 80159cc8-1b6c-4aa6-a114-966786d7be1e', 'osd.1 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) v1:172.21.15.84:6805/2140830229 v1:172.21.15.84:6806/2140830229 exists,up 9c2c76c9-ce0f-4c2d-9d96-c87a10187607', 'osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.84:6809/2699996290 v1:172.21.15.84:6810/2699996290 exists,up e8514daa-b5f9-4f2f-a432-53f4a4eb86bc', 'osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.84:6813/2204039828 v1:172.21.15.84:6814/2204039828 exists,up 5d9965f9-c0b2-48ce-869c-582990985623', 'osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.88:6800/2267955292 v1:172.21.15.88:6801/2267955292 exists,up 4bc50523-8480-44d2-8d76-31479e839c85', 'osd.5 up in weight 1 up_from 37 up_thru 38 down_at 0 last_clean_interval [0,0) v1:172.21.15.88:6804/661041538 v1:172.21.15.88:6805/661041538 exists,up d4db8d7f-afc7-43dc-81fe-f70c290d8b20', 'osd.6 up in weight 1 up_from 46 up_thru 47 down_at 0 last_clean_interval [0,0) v1:172.21.15.88:6808/380856869 v1:172.21.15.88:6809/380856869 exists,up a2773454-2318-4a5b-8f31-0952b1b07f2f', 'osd.7 up in weight 1 up_from 53 up_thru 82 down_at 0 last_clean_interval [0,0) v1:172.21.15.88:6812/1044709063 v1:172.21.15.88:6813/1044709063 exists,up 028ca7b6-3e00-4466-a33f-b726cd60cb34', 'osd.8 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6800/522459126 v1:172.21.15.90:6801/522459126 exists,up 02cab286-d487-4a66-982c-3cf28f55e851', 'osd.9 up in weight 1 up_from 68 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6804/2794571133 v1:172.21.15.90:6805/2794571133 exists,up 47d38bc7-39ba-4b35-af24-e282354dd613', 'osd.10 up in weight 1 up_from 74 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6808/3735380744 v1:172.21.15.90:6809/3735380744 exists,up d7b2aeae-d012-41eb-9f91-e4c05c649473', 'osd.11 up in weight 1 up_from 81 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.90:6812/2265615270 v1:172.21.15.90:6813/2265615270 exists,up 2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5'] 2024-05-15T23:05:40.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon mon.a config show 2024-05-15T23:05:40.752 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:42.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:41 smithi084 ceph-mon[39784]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:42.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:41 smithi090 ceph-mon[44434]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:42.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:41 smithi088 ceph-mon[44699]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:42.417 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:05:42.418 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "mon.a", 2024-05-15T23:05:42.418 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster": "ceph", 2024-05-15T23:05:42.418 INFO:teuthology.orchestra.run.smithi084.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-05-15T23:05:42.418 INFO:teuthology.orchestra.run.smithi084.stdout: "admin_socket_mode": "", 2024-05-15T23:05:42.418 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-05-15T23:05:42.418 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_client_required": "cephx, none", 2024-05-15T23:05:42.418 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_cluster_required": "cephx", 2024-05-15T23:05:42.418 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_debug": "true", 2024-05-15T23:05:42.418 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-05-15T23:05:42.418 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-05-15T23:05:42.418 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_service_required": "cephx", 2024-05-15T23:05:42.419 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_service_ticket_ttl": "240.000000", 2024-05-15T23:05:42.419 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_supported": "", 2024-05-15T23:05:42.419 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio": "true", 2024-05-15T23:05:42.419 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio_max_queue_depth": "1024", 2024-05-15T23:05:42.419 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio_poll_ms": "250", 2024-05-15T23:05:42.419 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio_reap_max": "16", 2024-05-15T23:05:42.419 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_async_discard_threads": "0", 2024-05-15T23:05:42.419 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_block_size": "4096", 2024-05-15T23:05:42.419 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_aio": "false", 2024-05-15T23:05:42.419 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-05-15T23:05:42.419 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-05-15T23:05:42.419 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_inflight_ios": "false", 2024-05-15T23:05:42.419 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_enable_discard": "false", 2024-05-15T23:05:42.419 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_flock_retry": "3", 2024-05-15T23:05:42.419 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_flock_retry_interval": "0.100000", 2024-05-15T23:05:42.420 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_inject_crash": "0", 2024-05-15T23:05:42.420 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_inject_crash_flush_delay": "2", 2024-05-15T23:05:42.420 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_ioring": "false", 2024-05-15T23:05:42.420 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_ioring_hipri": "false", 2024-05-15T23:05:42.420 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_ioring_sqthread_poll": "false", 2024-05-15T23:05:42.420 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-05-15T23:05:42.420 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_read_buffer_alignment": "4096", 2024-05-15T23:05:42.420 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-05-15T23:05:42.420 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_type": "", 2024-05-15T23:05:42.420 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_alloc_size": "1048576", 2024-05-15T23:05:42.420 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_allocator": "hybrid", 2024-05-15T23:05:42.420 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_buffered_io": "true", 2024-05-15T23:05:42.420 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_check_for_zeros": "false", 2024-05-15T23:05:42.420 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_check_volume_selector_often": "false", 2024-05-15T23:05:42.421 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-05-15T23:05:42.421 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_compact_log_sync": "false", 2024-05-15T23:05:42.421 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-05-15T23:05:42.421 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-05-15T23:05:42.421 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_log_compact_min_size": "16777216", 2024-05-15T23:05:42.421 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_log_replay_check_allocations": "true", 2024-05-15T23:05:42.421 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_max_log_runway": "4194304", 2024-05-15T23:05:42.421 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_max_prefetch": "1048576", 2024-05-15T23:05:42.421 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_min_flush_size": "524288", 2024-05-15T23:05:42.421 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_min_log_runway": "1048576", 2024-05-15T23:05:42.421 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_replay_recovery": "false", 2024-05-15T23:05:42.421 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-05-15T23:05:42.421 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_shared_alloc_size": "65536", 2024-05-15T23:05:42.421 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_sync_write": "false", 2024-05-15T23:05:42.421 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-05-15T23:05:42.422 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-05-15T23:05:42.422 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-05-15T23:05:42.422 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_allocation_from_file": "true", 2024-05-15T23:05:42.422 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_allocator": "hybrid", 2024-05-15T23:05:42.422 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-05-15T23:05:42.422 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-05-15T23:05:42.422 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-05-15T23:05:42.422 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-05-15T23:05:42.422 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-05-15T23:05:42.422 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-05-15T23:05:42.422 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_blobid_prealloc": "10240", 2024-05-15T23:05:42.422 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_create": "true", 2024-05-15T23:05:42.423 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_db_create": "false", 2024-05-15T23:05:42.423 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_db_path": "", 2024-05-15T23:05:42.423 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_db_size": "0", 2024-05-15T23:05:42.423 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_path": "", 2024-05-15T23:05:42.423 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_preallocate_file": "false", 2024-05-15T23:05:42.423 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_size": "107374182400", 2024-05-15T23:05:42.423 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_wal_create": "false", 2024-05-15T23:05:42.423 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_wal_path": "", 2024-05-15T23:05:42.423 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_wal_size": "100663296", 2024-05-15T23:05:42.423 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs": "true", 2024-05-15T23:05:42.423 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-05-15T23:05:42.423 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs_env_mirror": "false", 2024-05-15T23:05:42.423 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-05-15T23:05:42.423 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-05-15T23:05:42.424 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-05-15T23:05:42.424 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-05-15T23:05:42.424 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-05-15T23:05:42.424 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-05-15T23:05:42.424 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_autotune": "true", 2024-05-15T23:05:42.424 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-05-15T23:05:42.424 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-05-15T23:05:42.424 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-05-15T23:05:42.424 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-05-15T23:05:42.424 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_size": "0", 2024-05-15T23:05:42.424 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-05-15T23:05:42.424 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-05-15T23:05:42.424 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-05-15T23:05:42.424 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-05-15T23:05:42.424 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_type": "2q", 2024-05-15T23:05:42.425 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-05-15T23:05:42.425 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_clone_cow": "true", 2024-05-15T23:05:42.425 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_algorithm": "snappy", 2024-05-15T23:05:42.425 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_max_blob_size": "0", 2024-05-15T23:05:42.425 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-05-15T23:05:42.425 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-05-15T23:05:42.425 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_min_blob_size": "0", 2024-05-15T23:05:42.425 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-05-15T23:05:42.425 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-05-15T23:05:42.425 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_mode": "none", 2024-05-15T23:05:42.425 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-05-15T23:05:42.425 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_csum_type": "crc32c", 2024-05-15T23:05:42.425 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-05-15T23:05:42.425 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_enforce_settings": "default", 2024-05-15T23:05:42.425 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_freelist": "false", 2024-05-15T23:05:42.426 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_fsck_abort": "false", 2024-05-15T23:05:42.426 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-05-15T23:05:42.426 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-05-15T23:05:42.426 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_inject_read_err": "false", 2024-05-15T23:05:42.426 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_legacy_omap": "false", 2024-05-15T23:05:42.426 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-05-15T23:05:42.426 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-05-15T23:05:42.426 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-05-15T23:05:42.426 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-05-15T23:05:42.426 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_prefill": "0.000000", 2024-05-15T23:05:42.426 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-05-15T23:05:42.426 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-05-15T23:05:42.426 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-05-15T23:05:42.427 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_small_allocations": "0", 2024-05-15T23:05:42.427 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-05-15T23:05:42.427 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_default_buffered_read": "true", 2024-05-15T23:05:42.427 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_default_buffered_write": "false", 2024-05-15T23:05:42.427 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_deferred_batch_ops": "0", 2024-05-15T23:05:42.427 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-05-15T23:05:42.427 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-05-15T23:05:42.427 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_elastic_shared_blobs": "true", 2024-05-15T23:05:42.427 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-05-15T23:05:42.427 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-05-15T23:05:42.427 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-05-15T23:05:42.427 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-05-15T23:05:42.427 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-05-15T23:05:42.427 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fail_eio": "false", 2024-05-15T23:05:42.427 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-05-15T23:05:42.428 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-05-15T23:05:42.428 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-05-15T23:05:42.428 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-05-15T23:05:42.428 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mkfs": "true", 2024-05-15T23:05:42.428 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-05-15T23:05:42.428 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mount": "false", 2024-05-15T23:05:42.428 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-05-15T23:05:42.428 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_umount": "false", 2024-05-15T23:05:42.428 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-05-15T23:05:42.428 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-05-15T23:05:42.428 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-05-15T23:05:42.428 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-05-15T23:05:42.428 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-05-15T23:05:42.428 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-05-15T23:05:42.428 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-05-15T23:05:42.429 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-05-15T23:05:42.429 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_ignore_data_csum": "false", 2024-05-15T23:05:42.429 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-05-15T23:05:42.429 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_kvbackend": "rocksdb", 2024-05-15T23:05:42.429 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-05-15T23:05:42.429 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-05-15T23:05:42.429 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_log_op_age": "5.000000", 2024-05-15T23:05:42.429 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_alloc_size": "0", 2024-05-15T23:05:42.429 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_blob_size": "0", 2024-05-15T23:05:42.429 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-05-15T23:05:42.429 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-05-15T23:05:42.429 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_defer_interval": "3.000000", 2024-05-15T23:05:42.429 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_deferred_txc": "32", 2024-05-15T23:05:42.429 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_min_alloc_size": "0", 2024-05-15T23:05:42.429 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-05-15T23:05:42.430 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-05-15T23:05:42.430 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_nid_prealloc": "1024", 2024-05-15T23:05:42.430 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_prefer_deferred_size": "0", 2024-05-15T23:05:42.430 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-05-15T23:05:42.430 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-05-15T23:05:42.430 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_qfsck_on_mount": "true", 2024-05-15T23:05:42.430 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_retry_disk_reads": "3", 2024-05-15T23:05:42.430 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_cf": "true", 2024-05-15T23:05:42.430 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-05-15T23:05:42.430 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-05-15T23:05:42.430 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_options_annex": "", 2024-05-15T23:05:42.430 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_coremask": "0x1", 2024-05-15T23:05:42.430 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_io_sleep": "5", 2024-05-15T23:05:42.430 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_max_io_completion": "0", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_mem": "512", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_sync_submit_transaction": "false", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_bytes": "67108864", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_cost_per_io": "0", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_tracing": "false", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_volume_selection_reserved": "0", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-05-15T23:05:42.431 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-05-15T23:05:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-05-15T23:05:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-05-15T23:05:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_zero_block_detection": "false", 2024-05-15T23:05:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-05-15T23:05:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_action_update_interval": "2", 2024-05-15T23:05:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-05-15T23:05:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-05-15T23:05:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-05-15T23:05:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-05-15T23:05:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_mount_timeout": "10", 2024-05-15T23:05:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-05-15T23:05:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-05-15T23:05:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-05-15T23:05:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-05-15T23:05:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-05-15T23:05:42.433 INFO:teuthology.orchestra.run.smithi084.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-05-15T23:05:42.433 INFO:teuthology.orchestra.run.smithi084.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-05-15T23:05:42.433 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_cluster_require_signatures": "false", 2024-05-15T23:05:42.433 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_cluster_require_version": "2", 2024-05-15T23:05:42.433 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_require_signatures": "false", 2024-05-15T23:05:42.433 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_require_version": "2", 2024-05-15T23:05:42.433 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_service_require_signatures": "false", 2024-05-15T23:05:42.433 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_service_require_version": "2", 2024-05-15T23:05:42.433 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_sign_messages": "true", 2024-05-15T23:05:42.433 INFO:teuthology.orchestra.run.smithi084.stdout: "chdir": "", 2024-05-15T23:05:42.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client_acl_type": "", 2024-05-15T23:05:42.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client_asio_thread_count": "2", 2024-05-15T23:05:42.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client_cache_mid": "0.750000", 2024-05-15T23:05:42.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client_cache_size": "16384", 2024-05-15T23:05:42.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client_caps_release_delay": "5", 2024-05-15T23:05:42.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_check_pool_perm": "true", 2024-05-15T23:05:42.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_collect_and_send_global_metrics": "false", 2024-05-15T23:05:42.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_debug_force_sync_read": "false", 2024-05-15T23:05:42.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_debug_getattr_caps": "false", 2024-05-15T23:05:42.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_debug_inject_features": "", 2024-05-15T23:05:42.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_debug_inject_tick_delay": "0", 2024-05-15T23:05:42.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-05-15T23:05:42.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_die_on_failed_remount": "false", 2024-05-15T23:05:42.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_dirsize_rbytes": "true", 2024-05-15T23:05:42.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_force_lazyio": "false", 2024-05-15T23:05:42.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_fs": "", 2024-05-15T23:05:42.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_inject_fixed_oldest_tid": "false", 2024-05-15T23:05:42.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_inject_release_failure": "false", 2024-05-15T23:05:42.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_max_inline_size": "4096", 2024-05-15T23:05:42.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_max_retries_on_remount_failure": "5", 2024-05-15T23:05:42.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mds_namespace": "", 2024-05-15T23:05:42.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_metadata": "", 2024-05-15T23:05:42.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mount_gid": "-1", 2024-05-15T23:05:42.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mount_timeout": "300", 2024-05-15T23:05:42.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mount_uid": "-1", 2024-05-15T23:05:42.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mountpoint": "/", 2024-05-15T23:05:42.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_notify_timeout": "10", 2024-05-15T23:05:42.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc": "true", 2024-05-15T23:05:42.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_max_dirty": "104857600", 2024-05-15T23:05:42.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_max_dirty_age": "5.000000", 2024-05-15T23:05:42.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_max_objects": "1000", 2024-05-15T23:05:42.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_size": "209715200", 2024-05-15T23:05:42.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_target_dirty": "8388608", 2024-05-15T23:05:42.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_permissions": "true", 2024-05-15T23:05:42.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_quota": "true", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "client_quota_df": "true", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "client_readahead_max_bytes": "0", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "client_readahead_max_periods": "4", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "client_readahead_min": "131072", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "client_reconnect_stale": "false", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "client_shutdown_timeout": "30", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "client_snapdir": ".snap", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "client_tick_interval": "1", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "client_trace": "", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "client_try_dentry_invalidate": "false", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "client_use_faked_inos": "false", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "client_use_random_mds": "false", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_graylog": "false", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_graylog_port": "12201", 2024-05-15T23:05:42.436 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_monitors": "default=true", 2024-05-15T23:05:42.437 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_syslog": "false", 2024-05-15T23:05:42.437 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-05-15T23:05:42.437 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_syslog_level": "info", 2024-05-15T23:05:42.437 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "-", 2024-05-15T23:05:42.437 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_network": "", 2024-05-15T23:05:42.437 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_network_interface": "", 2024-05-15T23:05:42.437 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zlib_isal": "false", 2024-05-15T23:05:42.437 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zlib_level": "5", 2024-05-15T23:05:42.437 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zlib_winsize": "-15", 2024-05-15T23:05:42.437 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zstd_level": "1", 2024-05-15T23:05:42.437 INFO:teuthology.orchestra.run.smithi084.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5c5f12459ebcbf3753cc9d18fdb5ddd79e42ac036cf0c5c0a49db825e6a86185", 2024-05-15T23:05:42.437 INFO:teuthology.orchestra.run.smithi084.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-05-15T23:05:42.437 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_alien_op_num_threads": "6", 2024-05-15T23:05:42.437 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_alien_thread_cpu_cores": "", 2024-05-15T23:05:42.437 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_osd_obc_lru_size": "512", 2024-05-15T23:05:42.438 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-05-15T23:05:42.438 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_osd_stat_interval": "0", 2024-05-15T23:05:42.438 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_seastar_cpu_cores": "", 2024-05-15T23:05:42.438 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location": "", 2024-05-15T23:05:42.438 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location_hook": "", 2024-05-15T23:05:42.438 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location_hook_timeout": "10", 2024-05-15T23:05:42.438 INFO:teuthology.orchestra.run.smithi084.stdout: "daemonize": "false", 2024-05-15T23:05:42.438 INFO:teuthology.orchestra.run.smithi084.stdout: "daos_pool": "tank", 2024-05-15T23:05:42.438 INFO:teuthology.orchestra.run.smithi084.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-05-15T23:05:42.438 INFO:teuthology.orchestra.run.smithi084.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-05-15T23:05:42.438 INFO:teuthology.orchestra.run.smithi084.stdout: "dbstore_db_name_prefix": "dbstore", 2024-05-15T23:05:42.442 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_alienstore": "0/5", 2024-05-15T23:05:42.442 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_allow_any_pool_priority": "false", 2024-05-15T23:05:42.442 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_asok": "1/5", 2024-05-15T23:05:42.442 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_asok_assert_abort": "false", 2024-05-15T23:05:42.442 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_asserts_on_shutdown": "true", 2024-05-15T23:05:42.442 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_auth": "1/5", 2024-05-15T23:05:42.442 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_bdev": "1/3", 2024-05-15T23:05:42.442 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_bluefs": "1/5", 2024-05-15T23:05:42.442 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_bluestore": "1/5", 2024-05-15T23:05:42.442 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_buffer": "0/1", 2024-05-15T23:05:42.442 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_ceph_dedup": "0/5", 2024-05-15T23:05:42.442 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_ceph_exporter": "1/5", 2024-05-15T23:05:42.443 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_cephfs_mirror": "0/5", 2024-05-15T23:05:42.443 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_cephsqlite": "0/5", 2024-05-15T23:05:42.443 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_client": "0/5", 2024-05-15T23:05:42.443 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_compressor": "1/5", 2024-05-15T23:05:42.443 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_context": "0/1", 2024-05-15T23:05:42.443 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_crimson_interrupt": "0/5", 2024-05-15T23:05:42.443 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_crush": "1/1", 2024-05-15T23:05:42.443 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_crypto": "1/5", 2024-05-15T23:05:42.443 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_cyanstore": "0/5", 2024-05-15T23:05:42.443 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_deliberately_leak_memory": "false", 2024-05-15T23:05:42.443 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_disable_randomized_ping": "false", 2024-05-15T23:05:42.443 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_dpdk": "1/5", 2024-05-15T23:05:42.443 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_eventtrace": "1/5", 2024-05-15T23:05:42.444 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_filer": "0/1", 2024-05-15T23:05:42.444 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_filestore": "1/3", 2024-05-15T23:05:42.444 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_finisher": "1/1", 2024-05-15T23:05:42.444 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_fuse": "1/5", 2024-05-15T23:05:42.444 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_heartbeat_testing_span": "0", 2024-05-15T23:05:42.444 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_heartbeatmap": "1/5", 2024-05-15T23:05:42.444 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_immutable_obj_cache": "0/5", 2024-05-15T23:05:42.444 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_javaclient": "1/5", 2024-05-15T23:05:42.444 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_journal": "1/3", 2024-05-15T23:05:42.444 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_journaler": "0/5", 2024-05-15T23:05:42.444 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_kstore": "1/5", 2024-05-15T23:05:42.444 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_lockdep": "0/1", 2024-05-15T23:05:42.444 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mclock": "1/5", 2024-05-15T23:05:42.444 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds": "1/5", 2024-05-15T23:05:42.444 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_balancer": "1/5", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_locker": "1/5", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_log": "1/5", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_log_expire": "1/5", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_migrator": "1/5", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_quiesce": "3/5", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_memstore": "1/5", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mgr": "2/5", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mgrc": "1/5", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mon": "20/20", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_monc": "0/10", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_ms": "1/1", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_none": "0/5", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_objclass": "0/5", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_objectcacher": "0/5", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_objecter": "0/1", 2024-05-15T23:05:42.445 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_optracker": "0/5", 2024-05-15T23:05:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_osd": "1/5", 2024-05-15T23:05:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_paxos": "20/20", 2024-05-15T23:05:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_perfcounter": "1/5", 2024-05-15T23:05:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_prioritycache": "1/5", 2024-05-15T23:05:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rados": "0/5", 2024-05-15T23:05:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd": "0/5", 2024-05-15T23:05:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd_mirror": "0/5", 2024-05-15T23:05:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd_pwl": "0/5", 2024-05-15T23:05:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd_replay": "0/5", 2024-05-15T23:05:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_refs": "0/0", 2024-05-15T23:05:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_reserver": "1/1", 2024-05-15T23:05:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw": "1/5", 2024-05-15T23:05:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_access": "1/5", 2024-05-15T23:05:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_datacache": "1/5", 2024-05-15T23:05:42.446 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_dbstore": "1/5", 2024-05-15T23:05:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_flight": "1/5", 2024-05-15T23:05:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_lifecycle": "1/5", 2024-05-15T23:05:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_notification": "1/5", 2024-05-15T23:05:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_sync": "1/5", 2024-05-15T23:05:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rocksdb": "4/5", 2024-05-15T23:05:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore": "0/5", 2024-05-15T23:05:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_backref": "0/5", 2024-05-15T23:05:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_cache": "0/5", 2024-05-15T23:05:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_cleaner": "0/5", 2024-05-15T23:05:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_device": "0/5", 2024-05-15T23:05:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_epm": "0/5", 2024-05-15T23:05:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-05-15T23:05:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_journal": "0/5", 2024-05-15T23:05:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_lba": "0/5", 2024-05-15T23:05:42.447 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_odata": "0/5", 2024-05-15T23:05:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_omap": "0/5", 2024-05-15T23:05:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_onode": "0/5", 2024-05-15T23:05:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_t": "0/5", 2024-05-15T23:05:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_tm": "0/5", 2024-05-15T23:05:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_striper": "0/1", 2024-05-15T23:05:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_test": "0/5", 2024-05-15T23:05:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_throttle": "1/1", 2024-05-15T23:05:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_timer": "0/1", 2024-05-15T23:05:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_tp": "0/5", 2024-05-15T23:05:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_trace": "1/5", 2024-05-15T23:05:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-05-15T23:05:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "device_failure_prediction_mode": "none", 2024-05-15T23:05:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-05-15T23:05:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-05-15T23:05:42.448 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_graylog": "false", 2024-05-15T23:05:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_journald": "false", 2024-05-15T23:05:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_stderr": "true", 2024-05-15T23:05:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_syslog": "false", 2024-05-15T23:05:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event_tracing": "false", 2024-05-15T23:05:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_addr": "0.0.0.0", 2024-05-15T23:05:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_http_port": "9926", 2024-05-15T23:05:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_prio_limit": "5", 2024-05-15T23:05:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-05-15T23:05:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_sort_metrics": "true", 2024-05-15T23:05:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_stats_period": "5", 2024-05-15T23:05:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "fake_statfs_for_testing": "0", 2024-05-15T23:05:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "fatal_signal_handlers": "true", 2024-05-15T23:05:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "filer_max_purge_ops": "10", 2024-05-15T23:05:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "filer_max_truncate_ops": "128", 2024-05-15T23:05:42.449 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_apply_finisher_threads": "1", 2024-05-15T23:05:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_blackhole": "false", 2024-05-15T23:05:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_btrfs_clone_range": "true", 2024-05-15T23:05:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_btrfs_snap": "true", 2024-05-15T23:05:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_caller_concurrency": "10", 2024-05-15T23:05:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_collect_device_partition_information": "true", 2024-05-15T23:05:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_commit_timeout": "600.000000", 2024-05-15T23:05:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_debug_inject_read_err": "false", 2024-05-15T23:05:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_debug_omap_check": "false", 2024-05-15T23:05:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_debug_verify_split": "false", 2024-05-15T23:05:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_dump_file": "", 2024-05-15T23:05:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-05-15T23:05:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-05-15T23:05:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fadvise": "true", 2024-05-15T23:05:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fail_eio": "true", 2024-05-15T23:05:42.450 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fd_cache_shards": "16", 2024-05-15T23:05:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fd_cache_size": "128", 2024-05-15T23:05:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fiemap": "false", 2024-05-15T23:05:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fiemap_threshold": "4096", 2024-05-15T23:05:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-05-15T23:05:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_index_retry_probability": "0.000000", 2024-05-15T23:05:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_inject_stall": "0", 2024-05-15T23:05:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_journal_parallel": "false", 2024-05-15T23:05:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_journal_trailing": "false", 2024-05-15T23:05:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_journal_writeahead": "false", 2024-05-15T23:05:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_kill_at": "0", 2024-05-15T23:05:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-05-15T23:05:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size": "0", 2024-05-15T23:05:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-05-15T23:05:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-05-15T23:05:42.451 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs": "0", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs_other": "2", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_sync_interval": "5.000000", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size": "0", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_merge_threshold": "-10", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_min_sync_interval": "0.010000", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_odsync_write": "false", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_omap_backend": "rocksdb", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_omap_backend_path": "", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_omap_header_cache_size": "1024", 2024-05-15T23:05:42.452 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_ondisk_finisher_threads": "1", 2024-05-15T23:05:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-05-15T23:05:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_op_thread_timeout": "60", 2024-05-15T23:05:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_op_threads": "2", 2024-05-15T23:05:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_punch_hole": "false", 2024-05-15T23:05:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-05-15T23:05:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-05-15T23:05:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-05-15T23:05:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-05-15T23:05:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-05-15T23:05:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_bytes": "104857600", 2024-05-15T23:05:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-05-15T23:05:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-05-15T23:05:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-05-15T23:05:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_ops": "50", 2024-05-15T23:05:42.453 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-05-15T23:05:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_seek_data_hole": "false", 2024-05-15T23:05:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_sloppy_crc": "false", 2024-05-15T23:05:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-05-15T23:05:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_splice": "false", 2024-05-15T23:05:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_split_multiple": "2", 2024-05-15T23:05:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_split_rand_factor": "20", 2024-05-15T23:05:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_update_to": "1000", 2024-05-15T23:05:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-05-15T23:05:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-05-15T23:05:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-05-15T23:05:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-05-15T23:05:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-05-15T23:05:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-05-15T23:05:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_enable": "true", 2024-05-15T23:05:42.454 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_xfs_extsize": "false", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_zfs_snap": "false", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "fio_dir": "/tmp/fio", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "3999fb7e-130e-11ef-bc99-c7b262605968", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_allow_other": "true", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_atomic_o_trunc": "true", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_big_writes": "true", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_debug": "false", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_default_permissions": "false", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_disable_pagecache": "false", 2024-05-15T23:05:42.455 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_max_write": "0", 2024-05-15T23:05:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_multithreaded": "true", 2024-05-15T23:05:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_require_active_mds": "true", 2024-05-15T23:05:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_set_user_groups": "true", 2024-05-15T23:05:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_splice_move": "true", 2024-05-15T23:05:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_splice_read": "true", 2024-05-15T23:05:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_splice_write": "true", 2024-05-15T23:05:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_syncfs_on_mksnap": "true", 2024-05-15T23:05:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_use_invalidate_cb": "true", 2024-05-15T23:05:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-05-15T23:05:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "gss_target_name": "ceph", 2024-05-15T23:05:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_file": "", 2024-05-15T23:05:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_inject_failure": "0", 2024-05-15T23:05:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_interval": "5", 2024-05-15T23:05:42.456 INFO:teuthology.orchestra.run.smithi084.stdout: "host": "smithi084", 2024-05-15T23:05:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-05-15T23:05:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-05-15T23:05:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_max_size": "1073741824", 2024-05-15T23:05:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-05-15T23:05:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-05-15T23:05:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-05-15T23:05:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-05-15T23:05:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-05-15T23:05:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-05-15T23:05:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-05-15T23:05:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-05-15T23:05:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-05-15T23:05:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_watermark": "0.900000", 2024-05-15T23:05:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "inject_early_sigterm": "false", 2024-05-15T23:05:42.457 INFO:teuthology.orchestra.run.smithi084.stdout: "jaeger_agent_port": "6799", 2024-05-15T23:05:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "jaeger_tracing_enable": "false", 2024-05-15T23:05:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_aio": "true", 2024-05-15T23:05:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_align_min_size": "65536", 2024-05-15T23:05:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_block_align": "true", 2024-05-15T23:05:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_block_size": "4096", 2024-05-15T23:05:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_dio": "true", 2024-05-15T23:05:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_discard": "false", 2024-05-15T23:05:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_force_aio": "false", 2024-05-15T23:05:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_ignore_corruption": "false", 2024-05-15T23:05:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_max_write_bytes": "10485760", 2024-05-15T23:05:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_max_write_entries": "100", 2024-05-15T23:05:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_replay_from": "0", 2024-05-15T23:05:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_high_multiple": "0.000000", 2024-05-15T23:05:42.458 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_max_multiple": "0.000000", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_write_header_frequency": "0", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_zero_on_create": "false", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "journaler_prefetch_periods": "10", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "journaler_prezero_periods": "5", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "journaler_write_head_interval": "15", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "key": "", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "keyfile": "", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_backend": "rocksdb", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_default_stripe_size": "65536", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_fsck_on_mount": "false", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_fsck_on_mount_deep": "true", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_max_bytes": "67108864", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_max_ops": "512", 2024-05-15T23:05:42.459 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_nid_prealloc": "1024", 2024-05-15T23:05:42.460 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_onode_map_size": "1024", 2024-05-15T23:05:42.460 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-05-15T23:05:42.460 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_sync_submit_transaction": "false", 2024-05-15T23:05:42.460 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_sync_transaction": "false", 2024-05-15T23:05:42.460 INFO:teuthology.orchestra.run.smithi084.stdout: "librados_thread_count": "2", 2024-05-15T23:05:42.460 INFO:teuthology.orchestra.run.smithi084.stdout: "lockdep": "false", 2024-05-15T23:05:42.460 INFO:teuthology.orchestra.run.smithi084.stdout: "lockdep_force_backtrace": "false", 2024-05-15T23:05:42.460 INFO:teuthology.orchestra.run.smithi084.stdout: "log_coarse_timestamps": "true", 2024-05-15T23:05:42.460 INFO:teuthology.orchestra.run.smithi084.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-05-15T23:05:42.460 INFO:teuthology.orchestra.run.smithi084.stdout: "log_flush_on_exit": "false", 2024-05-15T23:05:42.460 INFO:teuthology.orchestra.run.smithi084.stdout: "log_graylog_host": "127.0.0.1", 2024-05-15T23:05:42.460 INFO:teuthology.orchestra.run.smithi084.stdout: "log_graylog_port": "12201", 2024-05-15T23:05:42.460 INFO:teuthology.orchestra.run.smithi084.stdout: "log_max_new": "1000", 2024-05-15T23:05:42.460 INFO:teuthology.orchestra.run.smithi084.stdout: "log_max_recent": "10000", 2024-05-15T23:05:42.460 INFO:teuthology.orchestra.run.smithi084.stdout: "log_stderr_prefix": "", 2024-05-15T23:05:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "log_stop_at_utilization": "0.970000", 2024-05-15T23:05:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_file": "true", 2024-05-15T23:05:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_graylog": "false", 2024-05-15T23:05:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_journald": "false", 2024-05-15T23:05:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_stderr": "false", 2024-05-15T23:05:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_syslog": "false", 2024-05-15T23:05:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "max_rotating_auth_attempts": "10", 2024-05-15T23:05:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-05-15T23:05:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_action_on_write_error": "1", 2024-05-15T23:05:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_alternate_name_max": "8192", 2024-05-15T23:05:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_asio_thread_count": "2", 2024-05-15T23:05:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_export_pin": "true", 2024-05-15T23:05:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_dirs": "true", 2024-05-15T23:05:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-05-15T23:05:42.461 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_interval": "5", 2024-05-15T23:05:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_size_max": "100000", 2024-05-15T23:05:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_idle_threshold": "0.000000", 2024-05-15T23:05:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_interval": "10", 2024-05-15T23:05:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_max": "-1", 2024-05-15T23:05:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_max_until": "-1", 2024-05-15T23:05:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_merge_size": "50", 2024-05-15T23:05:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_midchunk": "0.300000", 2024-05-15T23:05:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_min_rebalance": "0.100000", 2024-05-15T23:05:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_min_start": "0.200000", 2024-05-15T23:05:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_minchunk": "0.001000", 2024-05-15T23:05:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_mode": "0", 2024-05-15T23:05:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_need_max": "1.200000", 2024-05-15T23:05:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_need_min": "0.800000", 2024-05-15T23:05:42.462 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_overload_epochs": "2", 2024-05-15T23:05:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-05-15T23:05:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_sample_interval": "3.000000", 2024-05-15T23:05:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_bits": "3", 2024-05-15T23:05:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_rd": "25000.000000", 2024-05-15T23:05:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_size": "10000", 2024-05-15T23:05:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_wr": "10000.000000", 2024-05-15T23:05:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_target_decay": "10.000000", 2024-05-15T23:05:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-05-15T23:05:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_beacon_grace": "15.000000", 2024-05-15T23:05:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_beacon_interval": "4.000000", 2024-05-15T23:05:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_beacon_mon_down_grace": "60", 2024-05-15T23:05:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_memory_limit": "4294967296", 2024-05-15T23:05:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_mid": "0.700000", 2024-05-15T23:05:42.463 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-05-15T23:05:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_quiesce_delay": "0", 2024-05-15T23:05:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_quiesce_sleep": "200", 2024-05-15T23:05:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_quiesce_splitauth": "true", 2024-05-15T23:05:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_quiesce_threshold": "524288", 2024-05-15T23:05:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_release_free_interval": "10", 2024-05-15T23:05:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_reservation": "0.050000", 2024-05-15T23:05:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-05-15T23:05:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_trim_interval": "1", 2024-05-15T23:05:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_trim_threshold": "262144", 2024-05-15T23:05:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-05-15T23:05:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-05-15T23:05:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_client_delegate_inos_pct": "50", 2024-05-15T23:05:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_client_prealloc_inos": "1000", 2024-05-15T23:05:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-05-15T23:05:42.464 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_connect_bootstrapping": "false", 2024-05-15T23:05:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_damage_table_max_entries": "10000", 2024-05-15T23:05:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-05-15T23:05:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_auth_pins": "false", 2024-05-15T23:05:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_frag": "false", 2024-05-15T23:05:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_scatterstat": "false", 2024-05-15T23:05:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_subtrees": "false", 2024-05-15T23:05:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_decay_halflife": "5.000000", 2024-05-15T23:05:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_default_dir_hash": "2", 2024-05-15T23:05:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_defer_session_stale": "true", 2024-05-15T23:05:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_deny_all_reconnect": "false", 2024-05-15T23:05:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_keys_per_op": "16384", 2024-05-15T23:05:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_max_commit_size": "10", 2024-05-15T23:05:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_max_entries": "0", 2024-05-15T23:05:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_prefetch": "true", 2024-05-15T23:05:42.465 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dirstat_min_interval": "1.000000", 2024-05-15T23:05:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_after_rejoin": "false", 2024-05-15T23:05:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_on_map": "false", 2024-05-15T23:05:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_threshold_file": "0", 2024-05-15T23:05:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-05-15T23:05:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_early_reply": "true", 2024-05-15T23:05:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_enable_op_tracker": "true", 2024-05-15T23:05:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_enforce_unique_name": "true", 2024-05-15T23:05:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_distributed": "true", 2024-05-15T23:05:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-05-15T23:05:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_random": "true", 2024-05-15T23:05:42.466 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-05-15T23:05:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-05-15T23:05:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_forward_all_requests_to_auth": "false", 2024-05-15T23:05:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-05-15T23:05:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-05-15T23:05:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-05-15T23:05:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-05-15T23:05:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_health_cache_threshold": "1.500000", 2024-05-15T23:05:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_health_summarize_threshold": "10", 2024-05-15T23:05:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_heartbeat_grace": "15.000000", 2024-05-15T23:05:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_heartbeat_reset_grace": "1000", 2024-05-15T23:05:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_health_dummy": "false", 2024-05-15T23:05:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-05-15T23:05:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_migrator_session_race": "false", 2024-05-15T23:05:42.467 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-05-15T23:05:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-05-15T23:05:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-05-15T23:05:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_join_fs": "", 2024-05-15T23:05:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_journal_format": "1", 2024-05-15T23:05:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-05-15T23:05:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_create_at": "0", 2024-05-15T23:05:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_export_at": "0", 2024-05-15T23:05:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_import_at": "0", 2024-05-15T23:05:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_journal_at": "0", 2024-05-15T23:05:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_journal_expire_at": "0", 2024-05-15T23:05:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_journal_replay_at": "0", 2024-05-15T23:05:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_link_at": "0", 2024-05-15T23:05:42.468 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_mdstable_at": "0", 2024-05-15T23:05:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_openc_at": "0", 2024-05-15T23:05:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_rename_at": "0", 2024-05-15T23:05:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_shutdown_at": "0", 2024-05-15T23:05:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_event_large_threshold": "524288", 2024-05-15T23:05:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_events_per_segment": "1024", 2024-05-15T23:05:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_major_segment_event_ratio": "12", 2024-05-15T23:05:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_max_events": "-1", 2024-05-15T23:05:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_max_segments": "128", 2024-05-15T23:05:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_pause": "false", 2024-05-15T23:05:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_segment_size": "0", 2024-05-15T23:05:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_skip_corrupt_events": "false", 2024-05-15T23:05:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_skip_unbounded_events": "false", 2024-05-15T23:05:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-05-15T23:05:42.469 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_trim_threshold": "128", 2024-05-15T23:05:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-05-15T23:05:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_warn_factor": "2.000000", 2024-05-15T23:05:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_caps_per_client": "1048576", 2024-05-15T23:05:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_completed_flushes": "100000", 2024-05-15T23:05:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_completed_requests": "100000", 2024-05-15T23:05:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_export_size": "20971520", 2024-05-15T23:05:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_file_recover": "32", 2024-05-15T23:05:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_purge_files": "64", 2024-05-15T23:05:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_purge_ops": "8192", 2024-05-15T23:05:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-05-15T23:05:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-05-15T23:05:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_snaps_per_dir": "100", 2024-05-15T23:05:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_metrics_update_interval": "2", 2024-05-15T23:05:42.470 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_min_caps_per_client": "100", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_min_caps_working_set": "10000", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_numa_node": "-1", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_complaint_time": "30.000000", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_duration": "600", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_size": "20", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_slow_op_size": "20", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_slow_op_threshold": "10", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_log_threshold": "5", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_ping_grace": "15", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_ping_interval": "5", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_max_caps": "30000", 2024-05-15T23:05:42.471 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-05-15T23:05:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_max_decay_threshold": "131072", 2024-05-15T23:05:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-05-15T23:05:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_warning_threshold": "262144", 2024-05-15T23:05:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_reconnect_timeout": "45.000000", 2024-05-15T23:05:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_replay_interval": "1.000000", 2024-05-15T23:05:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-05-15T23:05:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-05-15T23:05:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_root_ino_gid": "0", 2024-05-15T23:05:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_root_ino_uid": "0", 2024-05-15T23:05:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-05-15T23:05:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-05-15T23:05:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-05-15T23:05:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_blocklist_on_evict": "true", 2024-05-15T23:05:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_blocklist_on_timeout": "true", 2024-05-15T23:05:42.472 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-05-15T23:05:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-05-15T23:05:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-05-15T23:05:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-05-15T23:05:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-05-15T23:05:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_metadata_threshold": "16777216", 2024-05-15T23:05:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-05-15T23:05:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_shutdown_check": "0", 2024-05-15T23:05:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_skip_ino": "0", 2024-05-15T23:05:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_sleep_rank_change": "0.000000", 2024-05-15T23:05:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_snap_max_uid": "4294967294", 2024-05-15T23:05:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_snap_min_uid": "0", 2024-05-15T23:05:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_snap_rstat": "false", 2024-05-15T23:05:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_standby_replay_damaged": "false", 2024-05-15T23:05:42.473 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_symlink_recovery": "true", 2024-05-15T23:05:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_task_status_update_interval": "2.000000", 2024-05-15T23:05:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_thrash_exports": "0", 2024-05-15T23:05:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_thrash_fragments": "0", 2024-05-15T23:05:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_tick_interval": "5.000000", 2024-05-15T23:05:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_valgrind_exit": "false", 2024-05-15T23:05:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_verify_backtrace": "1", 2024-05-15T23:05:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_verify_scatter": "false", 2024-05-15T23:05:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_wipe_ino_prealloc": "false", 2024-05-15T23:05:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_wipe_sessions": "false", 2024-05-15T23:05:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "mempool_debug": "false", 2024-05-15T23:05:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_debug_omit_block_device_write": "false", 2024-05-15T23:05:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_device_bytes": "1073741824", 2024-05-15T23:05:42.474 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_page_set": "false", 2024-05-15T23:05:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_page_size": "65536", 2024-05-15T23:05:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_client_bytes": "134217728", 2024-05-15T23:05:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_client_messages": "512", 2024-05-15T23:05:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-05-15T23:05:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_connect_retry_interval": "1.000000", 2024-05-15T23:05:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-05-15T23:05:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-05-15T23:05:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_disabled_modules": "", 2024-05-15T23:05:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_enable_op_tracker": "true", 2024-05-15T23:05:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-05-15T23:05:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_max_pg_creating": "1024", 2024-05-15T23:05:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_max_pg_num_change": "128", 2024-05-15T23:05:42.475 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mds_bytes": "134217728", 2024-05-15T23:05:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mds_messages": "128", 2024-05-15T23:05:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-05-15T23:05:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mon_bytes": "134217728", 2024-05-15T23:05:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mon_messages": "128", 2024-05-15T23:05:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_num_op_tracker_shard": "32", 2024-05-15T23:05:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_complaint_time": "30.000000", 2024-05-15T23:05:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_history_duration": "600", 2024-05-15T23:05:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_history_size": "20", 2024-05-15T23:05:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_history_slow_op_size": "20", 2024-05-15T23:05:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-05-15T23:05:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_log_threshold": "5", 2024-05-15T23:05:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_osd_bytes": "536870912", 2024-05-15T23:05:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_osd_messages": "8192", 2024-05-15T23:05:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_pool": "true", 2024-05-15T23:05:42.476 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_service_beacon_grace": "60.000000", 2024-05-15T23:05:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_standby_modules": "true", 2024-05-15T23:05:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_stats_period": "5", 2024-05-15T23:05:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_stats_threshold": "5", 2024-05-15T23:05:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_tick_period": "2", 2024-05-15T23:05:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-05-15T23:05:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_accept_timeout_factor": "2.000000", 2024-05-15T23:05:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_allow_pool_delete": "true", 2024-05-15T23:05:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_allow_pool_size_one": "false", 2024-05-15T23:05:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_auth_validate_all_caps": "true", 2024-05-15T23:05:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-05-15T23:05:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-05-15T23:05:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_bytes": "104857600", 2024-05-15T23:05:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_directed_command_retry": "5", 2024-05-15T23:05:42.477 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval": "3.000000", 2024-05-15T23:05:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-05-15T23:05:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-05-15T23:05:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-05-15T23:05:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_parallel": "3", 2024-05-15T23:05:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_log_interval": "1.000000", 2024-05-15T23:05:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-05-15T23:05:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_ping_interval": "10.000000", 2024-05-15T23:05:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_ping_timeout": "30.000000", 2024-05-15T23:05:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_target_rank": "-1", 2024-05-15T23:05:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_clock_drift_allowed": "1.000000", 2024-05-15T23:05:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-05-15T23:05:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-05-15T23:05:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_level": "debug", 2024-05-15T23:05:42.478 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_file": "true", 2024-05-15T23:05:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_graylog": "false", 2024-05-15T23:05:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-05-15T23:05:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-05-15T23:05:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_journald": "true", 2024-05-15T23:05:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_stderr": "false", 2024-05-15T23:05:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-05-15T23:05:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-05-15T23:05:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_compact_on_bootstrap": "false", 2024-05-15T23:05:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_compact_on_start": "false", 2024-05-15T23:05:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_compact_on_trim": "true", 2024-05-15T23:05:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_con_tracker_persist_interval": "10", 2024-05-15T23:05:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_con_tracker_score_halflife": "43200", 2024-05-15T23:05:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_config_key_max_entry_size": "65536", 2024-05-15T23:05:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cpu_threads": "4", 2024-05-15T23:05:42.479 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_crush_min_required_version": "hammer", 2024-05-15T23:05:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_daemon_bytes": "419430400", 2024-05-15T23:05:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-05-15T23:05:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data_avail_crit": "5", 2024-05-15T23:05:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data_avail_warn": "5", 2024-05-15T23:05:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data_size_warn": "16106127360", 2024-05-15T23:05:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_block_osdmap_trim": "false", 2024-05-15T23:05:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-05-15T23:05:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_dump_json": "false", 2024-05-15T23:05:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-05-15T23:05:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_dump_transactions": "false", 2024-05-15T23:05:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_extra_checks": "false", 2024-05-15T23:05:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-05-15T23:05:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-05-15T23:05:42.480 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_require_reef": "false", 2024-05-15T23:05:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_require_squid": "false", 2024-05-15T23:05:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-05-15T23:05:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_delta_reset_interval": "10.000000", 2024-05-15T23:05:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_dns_srv_name": "ceph-mon", 2024-05-15T23:05:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_down_mkfs_grace": "60", 2024-05-15T23:05:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_down_uptime_grace": "60", 2024-05-15T23:05:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_election_default_strategy": "1", 2024-05-15T23:05:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_election_timeout": "5.000000", 2024-05-15T23:05:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-05-15T23:05:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_elector_ping_divisor": "2", 2024-05-15T23:05:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_elector_ping_timeout": "2.000000", 2024-05-15T23:05:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_enable_op_tracker": "true", 2024-05-15T23:05:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_fake_pool_delete": "false", 2024-05-15T23:05:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_force_quorum_join": "false", 2024-05-15T23:05:42.481 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_globalid_prealloc": "10000", 2024-05-15T23:05:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_detail_to_clog": "true", 2024-05-15T23:05:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_log_update_period": "5", 2024-05-15T23:05:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_max_detail": "50", 2024-05-15T23:05:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_to_clog": "true", 2024-05-15T23:05:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_to_clog_interval": "600", 2024-05-15T23:05:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-05-15T23:05:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_host": "172.21.15.84:6789/0", 2024-05-15T23:05:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_host_override": "", 2024-05-15T23:05:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_initial_members": "", 2024-05-15T23:05:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-05-15T23:05:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-05-15T23:05:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-05-15T23:05:42.482 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-05-15T23:05:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_keyvaluedb": "rocksdb", 2024-05-15T23:05:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_lease": "5.000000", 2024-05-15T23:05:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-05-15T23:05:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-05-15T23:05:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_log_full_interval": "50", 2024-05-15T23:05:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_log_max": "10000", 2024-05-15T23:05:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_log_max_summary": "50", 2024-05-15T23:05:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_log_entries_per_event": "4096", 2024-05-15T23:05:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_log_epochs": "500", 2024-05-15T23:05:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_mdsmap_epochs": "500", 2024-05-15T23:05:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_mgrmap_epochs": "500", 2024-05-15T23:05:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_osd": "10000", 2024-05-15T23:05:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_pg_per_osd": "10000", 2024-05-15T23:05:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_pool_pg_num": "65536", 2024-05-15T23:05:42.483 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-05-15T23:05:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-05-15T23:05:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mds_force_trim_to": "0", 2024-05-15T23:05:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mds_skip_sanity": "false", 2024-05-15T23:05:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_memory_autotune": "true", 2024-05-15T23:05:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_memory_target": "2147483648", 2024-05-15T23:05:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_beacon_grace": "30", 2024-05-15T23:05:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-05-15T23:05:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_digest_period": "5", 2024-05-15T23:05:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_inactive_grace": "60", 2024-05-15T23:05:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_mkfs_grace": "240", 2024-05-15T23:05:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-05-15T23:05:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_min_osdmap_epochs": "50", 2024-05-15T23:05:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_complaint_time": "30", 2024-05-15T23:05:42.484 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_duration": "600", 2024-05-15T23:05:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_size": "20", 2024-05-15T23:05:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_slow_op_size": "20", 2024-05-15T23:05:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_slow_op_threshold": "10", 2024-05-15T23:05:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_log_threshold": "5", 2024-05-15T23:05:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-05-15T23:05:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-05-15T23:05:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-05-15T23:05:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_auto_mark_in": "false", 2024-05-15T23:05:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_auto_mark_new_in": "true", 2024-05-15T23:05:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-05-15T23:05:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-05-15T23:05:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_cache_size": "500", 2024-05-15T23:05:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_cache_size_min": "134217728", 2024-05-15T23:05:42.485 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_crush_smoke_test": "true", 2024-05-15T23:05:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_destroyed_out_interval": "600", 2024-05-15T23:05:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_down_out_interval": "600", 2024-05-15T23:05:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-05-15T23:05:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-05-15T23:05:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_force_trim_to": "0", 2024-05-15T23:05:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_full_ratio": "0.950000", 2024-05-15T23:05:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-05-15T23:05:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_laggy_halflife": "3600", 2024-05-15T23:05:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_laggy_max_interval": "300", 2024-05-15T23:05:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_laggy_weight": "0.300000", 2024-05-15T23:05:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-05-15T23:05:42.486 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_max_initial_pgs": "1024", 2024-05-15T23:05:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_min_down_reporters": "2", 2024-05-15T23:05:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-05-15T23:05:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-05-15T23:05:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-05-15T23:05:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_prime_pg_temp": "true", 2024-05-15T23:05:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-05-15T23:05:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-05-15T23:05:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_report_timeout": "900", 2024-05-15T23:05:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-05-15T23:05:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-05-15T23:05:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_warn_num_repaired": "10", 2024-05-15T23:05:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_warn_op_age": "32.000000", 2024-05-15T23:05:42.487 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-05-15T23:05:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_interval": "2", 2024-05-15T23:05:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_min": "15", 2024-05-15T23:05:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-05-15T23:05:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-05-15T23:05:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_stuck_threshold": "60", 2024-05-15T23:05:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-05-15T23:05:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_min_objects": "10000", 2024-05-15T23:05:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_min_per_osd": "0", 2024-05-15T23:05:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-05-15T23:05:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pool_quota_crit_threshold": "0", 2024-05-15T23:05:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pool_quota_warn_threshold": "0", 2024-05-15T23:05:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_probe_timeout": "2.000000", 2024-05-15T23:05:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_max_change": "0.050000", 2024-05-15T23:05:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_max_osds": "4", 2024-05-15T23:05:42.488 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-05-15T23:05:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-05-15T23:05:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-05-15T23:05:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-05-15T23:05:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-05-15T23:05:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_interval": "300", 2024-05-15T23:05:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_max_keys": "100", 2024-05-15T23:05:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_timeout": "300", 2024-05-15T23:05:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_session_timeout": "300", 2024-05-15T23:05:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_smart_report_timeout": "5", 2024-05-15T23:05:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stat_smooth_intervals": "6", 2024-05-15T23:05:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-05-15T23:05:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_pool_min_size": "2", 2024-05-15T23:05:42.489 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_pool_size": "4", 2024-05-15T23:05:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-05-15T23:05:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_subscribe_interval": "86400.000000", 2024-05-15T23:05:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_debug": "false", 2024-05-15T23:05:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_max_payload_keys": "2000", 2024-05-15T23:05:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_max_payload_size": "1048576", 2024-05-15T23:05:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_provider_kill_at": "0", 2024-05-15T23:05:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_requester_kill_at": "0", 2024-05-15T23:05:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_timeout": "60.000000", 2024-05-15T23:05:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_target_pg_per_osd": "100", 2024-05-15T23:05:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_tick_interval": "5", 2024-05-15T23:05:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_timecheck_interval": "300.000000", 2024-05-15T23:05:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-05-15T23:05:42.490 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_use_min_delay_socket": "false", 2024-05-15T23:05:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_older_version_delay": "604800", 2024-05-15T23:05:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-05-15T23:05:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-05-15T23:05:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-05-15T23:05:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_filestore_osds": "true", 2024-05-15T23:05:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-05-15T23:05:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-05-15T23:05:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-05-15T23:05:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_misplaced": "false", 2024-05-15T23:05:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-05-15T23:05:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_older_version": "true", 2024-05-15T23:05:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-05-15T23:05:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_pool_no_app": "true", 2024-05-15T23:05:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-05-15T23:05:42.491 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-05-15T23:05:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-05-15T23:05:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-05-15T23:05:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_too_few_osds": "false", 2024-05-15T23:05:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-05-15T23:05:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-05-15T23:05:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "monmap": "", 2024-05-15T23:05:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-05-15T23:05:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-05-15T23:05:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-05-15T23:05:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-05-15T23:05:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-05-15T23:05:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-05-15T23:05:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_tracing_enabled": "false", 2024-05-15T23:05:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_op_threads": "3", 2024-05-15T23:05:42.492 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_buffer_size": "131072", 2024-05-15T23:05:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_cm": "false", 2024-05-15T23:05:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_device_name": "", 2024-05-15T23:05:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_dscp": "96", 2024-05-15T23:05:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-05-15T23:05:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_gid_idx": "0", 2024-05-15T23:05:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_local_gid": "", 2024-05-15T23:05:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_polling_us": "1000", 2024-05-15T23:05:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_port_num": "1", 2024-05-15T23:05:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-05-15T23:05:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-05-15T23:05:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_roce_ver": "1", 2024-05-15T23:05:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_send_buffers": "1024", 2024-05-15T23:05:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_sl": "3", 2024-05-15T23:05:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_support_srq": "true", 2024-05-15T23:05:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_type": "ib", 2024-05-15T23:05:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_reap_threshold": "5", 2024-05-15T23:05:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_before_connect": "false", 2024-05-15T23:05:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_ipv4": "true", 2024-05-15T23:05:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_ipv6": "false", 2024-05-15T23:05:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_msgr1": "true", 2024-05-15T23:05:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_msgr2": "false", 2024-05-15T23:05:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_port_max": "7568", 2024-05-15T23:05:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_port_min": "6800", 2024-05-15T23:05:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_prefer_ipv4": "false", 2024-05-15T23:05:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_retry_count": "3", 2024-05-15T23:05:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_retry_delay": "5", 2024-05-15T23:05:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_client": "false", 2024-05-15T23:05:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_mds": "false", 2024-05-15T23:05:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_mgr": "false", 2024-05-15T23:05:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_mon": "false", 2024-05-15T23:05:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_osd": "false", 2024-05-15T23:05:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_client_mode": "crc secure", 2024-05-15T23:05:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_cluster_mode": "crc secure", 2024-05-15T23:05:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_cluster_type": "", 2024-05-15T23:05:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_compress_secure": "false", 2024-05-15T23:05:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_connection_idle_timeout": "900", 2024-05-15T23:05:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_connection_ready_timeout": "10", 2024-05-15T23:05:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_crc_data": "true", 2024-05-15T23:05:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_crc_header": "true", 2024-05-15T23:05:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_bad_msg": "false", 2024-05-15T23:05:42.495 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_bug": "true", 2024-05-15T23:05:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_old_message": "true", 2024-05-15T23:05:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_skipped_message": "false", 2024-05-15T23:05:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_unhandled_msg": "false", 2024-05-15T23:05:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-05-15T23:05:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_coremask": "0xF", 2024-05-15T23:05:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-05-15T23:05:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_devs_allowlist": "", 2024-05-15T23:05:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_enable_tso": "true", 2024-05-15T23:05:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-05-15T23:05:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-05-15T23:05:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_hugepages": "", 2024-05-15T23:05:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_hw_flow_control": "true", 2024-05-15T23:05:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-05-15T23:05:42.496 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_lro": "true", 2024-05-15T23:05:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_memory_channel": "4", 2024-05-15T23:05:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-05-15T23:05:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_pmd": "", 2024-05-15T23:05:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_port_id": "0", 2024-05-15T23:05:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-05-15T23:05:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dump_corrupt_message_level": "1", 2024-05-15T23:05:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dump_on_send": "false", 2024-05-15T23:05:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_initial_backoff": "0.200000", 2024-05-15T23:05:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_delay_max": "1.000000", 2024-05-15T23:05:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_delay_probability": "0.000000", 2024-05-15T23:05:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_delay_type": "", 2024-05-15T23:05:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_internal_delays": "0.000000", 2024-05-15T23:05:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_network_congestion": "0", 2024-05-15T23:05:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_socket_failures": "5000", 2024-05-15T23:05:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_learn_addr_from_peer": "true", 2024-05-15T23:05:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_max_accept_failures": "4", 2024-05-15T23:05:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_max_backoff": "15.000000", 2024-05-15T23:05:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_mon_client_mode": "secure crc", 2024-05-15T23:05:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_mon_cluster_mode": "secure crc", 2024-05-15T23:05:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_mon_service_mode": "secure crc", 2024-05-15T23:05:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_osd_compress_min_size": "1024", 2024-05-15T23:05:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_osd_compress_mode": "none", 2024-05-15T23:05:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_osd_compression_algorithm": "snappy", 2024-05-15T23:05:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-05-15T23:05:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_pq_min_cost": "65536", 2024-05-15T23:05:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_public_type": "", 2024-05-15T23:05:42.498 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_service_mode": "crc secure", 2024-05-15T23:05:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_listen_backlog": "512", 2024-05-15T23:05:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_nodelay": "true", 2024-05-15T23:05:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-05-15T23:05:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_rcvbuf": "0", 2024-05-15T23:05:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_type": "async+posix", 2024-05-15T23:05:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "no_config_file": "false", 2024-05-15T23:05:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_completion_locks_per_session": "32", 2024-05-15T23:05:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_debug_inject_relock_delay": "false", 2024-05-15T23:05:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_inflight_op_bytes": "104857600", 2024-05-15T23:05:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_inflight_ops": "1024", 2024-05-15T23:05:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_inject_no_watch_ping": "false", 2024-05-15T23:05:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-05-15T23:05:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_tick_interval": "5.000000", 2024-05-15T23:05:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_timeout": "10.000000", 2024-05-15T23:05:42.499 INFO:teuthology.orchestra.run.smithi084.stdout: "objectstore_blackhole": "false", 2024-05-15T23:05:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-05-15T23:05:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "openssl_engine_opts": "", 2024-05-15T23:05:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_delay_time": "5.000000", 2024-05-15T23:05:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_hist_halflife": "1000", 2024-05-15T23:05:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_max_low_ops": "2", 2024-05-15T23:05:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_max_ops": "4", 2024-05-15T23:05:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-05-15T23:05:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_quantize_effort": "0.100000", 2024-05-15T23:05:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_slop": "0.020000", 2024-05-15T23:05:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-05-15T23:05:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_allow_recovery_below_min_size": "true", 2024-05-15T23:05:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_asio_thread_count": "2", 2024-05-15T23:05:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_async_recovery_min_cost": "100", 2024-05-15T23:05:42.500 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_auto_mark_unfound_lost": "false", 2024-05-15T23:05:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backfill_retry_interval": "30.000000", 2024-05-15T23:05:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backfill_scan_max": "512", 2024-05-15T23:05:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backfill_scan_min": "64", 2024-05-15T23:05:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backoff_on_degraded": "false", 2024-05-15T23:05:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backoff_on_peering": "false", 2024-05-15T23:05:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backoff_on_unfound": "true", 2024-05-15T23:05:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_beacon_report_interval": "300", 2024-05-15T23:05:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_duration": "30", 2024-05-15T23:05:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-05-15T23:05:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_max_block_size": "67108864", 2024-05-15T23:05:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_small_size_max_iops": "100", 2024-05-15T23:05:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_blkin_trace_all": "false", 2024-05-15T23:05:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_blocked_scrub_grace_period": "120", 2024-05-15T23:05:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-05-15T23:05:42.501 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-05-15T23:05:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-05-15T23:05:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_check_for_log_corruption": "false", 2024-05-15T23:05:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-05-15T23:05:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-05-15T23:05:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-05-15T23:05:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-05-15T23:05:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_update_on_start": "true", 2024-05-15T23:05:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_message_cap": "256", 2024-05-15T23:05:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_message_size_cap": "524288000", 2024-05-15T23:05:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_op_priority": "63", 2024-05-15T23:05:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_watch_timeout": "30", 2024-05-15T23:05:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_command_max_records": "256", 2024-05-15T23:05:42.502 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_compact_on_start": "false", 2024-05-15T23:05:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-05-15T23:05:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_chooseleaf_type": "1", 2024-05-15T23:05:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_initial_weight": "-1.000000", 2024-05-15T23:05:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_update_on_start": "true", 2024-05-15T23:05:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_update_weight_set": "true", 2024-05-15T23:05:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-05-15T23:05:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-05-15T23:05:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-05-15T23:05:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_drop_ping_duration": "0", 2024-05-15T23:05:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-05-15T23:05:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_feed_pullee": "-1", 2024-05-15T23:05:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-05-15T23:05:42.503 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-05-15T23:05:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-05-15T23:05:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_misdirected_ops": "false", 2024-05-15T23:05:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_no_acting_change": "false", 2024-05-15T23:05:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_no_purge_strays": "false", 2024-05-15T23:05:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_op_order": "false", 2024-05-15T23:05:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_pg_log_writeout": "false", 2024-05-15T23:05:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_pretend_recovery_active": "false", 2024-05-15T23:05:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-05-15T23:05:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-05-15T23:05:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_shutdown": "false", 2024-05-15T23:05:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-05-15T23:05:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-05-15T23:05:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_trim_objects": "false", 2024-05-15T23:05:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_cached_snaps": "false", 2024-05-15T23:05:42.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_missing_on_start": "false", 2024-05-15T23:05:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_snaps": "false", 2024-05-15T23:05:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-05-15T23:05:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-05-15T23:05:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_keys": "1024", 2024-05-15T23:05:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-05-15T23:05:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-05-15T23:05:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-05-15T23:05:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_stride": "524288", 2024-05-15T23:05:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-05-15T23:05:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_default_data_pool_replay_window": "45", 2024-05-15T23:05:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_default_notify_timeout": "30", 2024-05-15T23:05:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep": "0.000000", 2024-05-15T23:05:42.505 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-05-15T23:05:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-05-15T23:05:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-05-15T23:05:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_discard_disconnected_ops": "true", 2024-05-15T23:05:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_enable_op_tracker": "true", 2024-05-15T23:05:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-05-15T23:05:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_extblkdev_plugins": "vdo", 2024-05-15T23:05:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-05-15T23:05:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-05-15T23:05:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_info": "true", 2024-05-15T23:05:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_shutdown": "true", 2024-05-15T23:05:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-05-15T23:05:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_shutdown_timeout": "15", 2024-05-15T23:05:42.506 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-05-15T23:05:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-05-15T23:05:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-05-15T23:05:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_function_tracing": "false", 2024-05-15T23:05:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_grace": "20", 2024-05-15T23:05:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_interval": "6", 2024-05-15T23:05:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-05-15T23:05:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_min_peers": "10", 2024-05-15T23:05:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_min_size": "2000", 2024-05-15T23:05:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_stale": "600", 2024-05-15T23:05:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-05-15T23:05:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_hit_set_max_size": "100000", 2024-05-15T23:05:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_hit_set_min_size": "1000", 2024-05-15T23:05:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-05-15T23:05:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-05-15T23:05:42.507 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-05-15T23:05:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-05-15T23:05:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-05-15T23:05:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_journal_flush_on_shutdown": "true", 2024-05-15T23:05:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_journal_size": "5120", 2024-05-15T23:05:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_kill_backfill_at": "0", 2024-05-15T23:05:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_loop_before_reset_tphandle": "64", 2024-05-15T23:05:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_cache_size": "50", 2024-05-15T23:05:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_dedup": "true", 2024-05-15T23:05:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_message_max": "40", 2024-05-15T23:05:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_message_max_bytes": "10485760", 2024-05-15T23:05:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_share_max_epochs": "40", 2024-05-15T23:05:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_attr_name_len": "100", 2024-05-15T23:05:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_attr_size": "0", 2024-05-15T23:05:42.508 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_backfills": "1", 2024-05-15T23:05:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_markdown_count": "5", 2024-05-15T23:05:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_markdown_period": "600", 2024-05-15T23:05:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_object_name_len": "2048", 2024-05-15T23:05:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_object_namespace_len": "256", 2024-05-15T23:05:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_object_size": "134217728", 2024-05-15T23:05:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-05-15T23:05:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_omap_entries_per_request": "1024", 2024-05-15T23:05:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pg_blocked_by": "16", 2024-05-15T23:05:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pg_log_entries": "10000", 2024-05-15T23:05:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-05-15T23:05:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pgls": "1024", 2024-05-15T23:05:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_push_cost": "8388608", 2024-05-15T23:05:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_push_objects": "10", 2024-05-15T23:05:42.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_scrubs": "3", 2024-05-15T23:05:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-05-15T23:05:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_trimming_pgs": "2", 2024-05-15T23:05:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_write_op_reply_len": "64", 2024-05-15T23:05:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_write_size": "90", 2024-05-15T23:05:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-05-15T23:05:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-05-15T23:05:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-05-15T23:05:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-05-15T23:05:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-05-15T23:05:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-05-15T23:05:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-05-15T23:05:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_override_recovery_settings": "false", 2024-05-15T23:05:42.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_profile": "balanced", 2024-05-15T23:05:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-05-15T23:05:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-05-15T23:05:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-05-15T23:05:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-05-15T23:05:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-05-15T23:05:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-05-15T23:05:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-05-15T23:05:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-05-15T23:05:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-05-15T23:05:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-05-15T23:05:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_skip_benchmark": "false", 2024-05-15T23:05:42.511 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_base": "805306368", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_cache_min": "134217728", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_target": "4294967296", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_target_autotune": "false", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_min_pg_log_entries": "250", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_min_recovery_priority": "0", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_heartbeat_interval": "30", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_report_interval": "5", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_report_max_in_flight": "2", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_num_cache_shards": "32", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_num_op_tracker_shard": "32", 2024-05-15T23:05:42.512 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_numa_auto_affinity": "true", 2024-05-15T23:05:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_numa_node": "-1", 2024-05-15T23:05:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_numa_prefer_iface": "true", 2024-05-15T23:05:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-05-15T23:05:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objecter_finishers": "1", 2024-05-15T23:05:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objectstore": "bluestore", 2024-05-15T23:05:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objectstore_fuse": "false", 2024-05-15T23:05:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objectstore_tracing": "false", 2024-05-15T23:05:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_complaint_time": "30.000000", 2024-05-15T23:05:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_duration": "600", 2024-05-15T23:05:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_size": "20", 2024-05-15T23:05:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_slow_op_size": "20", 2024-05-15T23:05:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-05-15T23:05:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_log_threshold": "5", 2024-05-15T23:05:42.513 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_shards": "0", 2024-05-15T23:05:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_shards_hdd": "5", 2024-05-15T23:05:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_shards_ssd": "8", 2024-05-15T23:05:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_threads_per_shard": "0", 2024-05-15T23:05:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-05-15T23:05:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-05-15T23:05:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-05-15T23:05:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_pq_min_cost": "65536", 2024-05-15T23:05:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_queue": "mclock_scheduler", 2024-05-15T23:05:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_queue_cut_off": "high", 2024-05-15T23:05:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_thread_suicide_timeout": "150", 2024-05-15T23:05:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_thread_timeout": "15", 2024-05-15T23:05:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_open_classes_on_start": "true", 2024-05-15T23:05:42.514 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_os_flags": "0", 2024-05-15T23:05:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_peering_op_priority": "255", 2024-05-15T23:05:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_delete_cost": "1048576", 2024-05-15T23:05:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_delete_priority": "5", 2024-05-15T23:05:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-05-15T23:05:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-05-15T23:05:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_log_dups_tracked": "3000", 2024-05-15T23:05:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_log_trim_max": "10000", 2024-05-15T23:05:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_log_trim_min": "100", 2024-05-15T23:05:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-05-15T23:05:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_object_context_cache_count": "64", 2024-05-15T23:05:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-05-15T23:05:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-05-15T23:05:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-05-15T23:05:42.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-05-15T23:05:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-05-15T23:05:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-05-15T23:05:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-05-15T23:05:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-05-15T23:05:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_crimson": "false", 2024-05-15T23:05:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_crush_rule": "-1", 2024-05-15T23:05:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_ec_fast_read": "false", 2024-05-15T23:05:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-05-15T23:05:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_bulk": "false", 2024-05-15T23:05:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-05-15T23:05:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_nodelete": "false", 2024-05-15T23:05:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-05-15T23:05:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-05-15T23:05:42.516 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flags": "0", 2024-05-15T23:05:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-05-15T23:05:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_min_size": "2", 2024-05-15T23:05:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-05-15T23:05:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_pg_num": "32", 2024-05-15T23:05:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_pgp_num": "0", 2024-05-15T23:05:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-05-15T23:05:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_read_ratio": "70", 2024-05-15T23:05:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_size": "2", 2024-05-15T23:05:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_type": "replicated", 2024-05-15T23:05:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-05-15T23:05:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_use_gmt_hitset": "true", 2024-05-15T23:05:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_push_per_object_cost": "1000", 2024-05-15T23:05:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_read_ec_check_for_errors": "false", 2024-05-15T23:05:42.517 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recover_clone_overlap": "true", 2024-05-15T23:05:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recover_clone_overlap_limit": "10", 2024-05-15T23:05:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_cost": "20971520", 2024-05-15T23:05:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_delay_start": "0.000000", 2024-05-15T23:05:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_active": "0", 2024-05-15T23:05:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_active_hdd": "3", 2024-05-15T23:05:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_active_ssd": "10", 2024-05-15T23:05:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_chunk": "8388608", 2024-05-15T23:05:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-05-15T23:05:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_single_start": "1", 2024-05-15T23:05:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_op_priority": "3", 2024-05-15T23:05:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_op_warn_multiple": "16", 2024-05-15T23:05:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_priority": "5", 2024-05-15T23:05:42.518 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_retry_interval": "30.000000", 2024-05-15T23:05:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep": "0.000000", 2024-05-15T23:05:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-05-15T23:05:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-05-15T23:05:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-05-15T23:05:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_repair_during_recovery": "false", 2024-05-15T23:05:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_requested_scrub_priority": "120", 2024-05-15T23:05:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-05-15T23:05:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_rollback_to_cluster_snap": "", 2024-05-15T23:05:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_auto_repair": "false", 2024-05-15T23:05:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-05-15T23:05:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-05-15T23:05:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_begin_hour": "0", 2024-05-15T23:05:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_begin_week_day": "0", 2024-05-15T23:05:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_chunk_max": "25", 2024-05-15T23:05:42.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_chunk_min": "5", 2024-05-15T23:05:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_cost": "52428800", 2024-05-15T23:05:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-05-15T23:05:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_during_recovery": "false", 2024-05-15T23:05:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_end_hour": "0", 2024-05-15T23:05:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_end_week_day": "0", 2024-05-15T23:05:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_event_cost": "4096", 2024-05-15T23:05:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-05-15T23:05:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-05-15T23:05:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_invalid_stats": "true", 2024-05-15T23:05:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_load_threshold": "0.500000", 2024-05-15T23:05:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_max_interval": "604800.000000", 2024-05-15T23:05:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_max_preemptions": "5", 2024-05-15T23:05:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_min_interval": "86400.000000", 2024-05-15T23:05:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_priority": "5", 2024-05-15T23:05:42.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_sleep": "0.000000", 2024-05-15T23:05:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-05-15T23:05:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-05-15T23:05:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_shutdown_pgref_assert": "false", 2024-05-15T23:05:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-05-15T23:05:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_skip_data_digest": "false", 2024-05-15T23:05:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_smart_report_timeout": "5", 2024-05-15T23:05:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_cost": "1048576", 2024-05-15T23:05:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_priority": "5", 2024-05-15T23:05:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep": "0.000000", 2024-05-15T23:05:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-05-15T23:05:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-05-15T23:05:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-05-15T23:05:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-05-15T23:05:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_stats_update_period_scrubbing": "15", 2024-05-15T23:05:42.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-05-15T23:05:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_target_transaction_size": "30", 2024-05-15T23:05:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-05-15T23:05:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-05-15T23:05:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-05-15T23:05:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-05-15T23:05:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-05-15T23:05:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-05-15T23:05:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-05-15T23:05:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_mode": "writeback", 2024-05-15T23:05:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-05-15T23:05:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-05-15T23:05:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tracing": "false", 2024-05-15T23:05:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_use_stale_snap": "false", 2024-05-15T23:05:42.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-05-15T23:05:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "osdc_blkin_trace_all": "false", 2024-05-15T23:05:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_kill_at": "0", 2024-05-15T23:05:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_max_join_drift": "10", 2024-05-15T23:05:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_min": "500", 2024-05-15T23:05:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_min_wait": "0.050000", 2024-05-15T23:05:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_propose_interval": "1.000000", 2024-05-15T23:05:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_service_trim_max": "500", 2024-05-15T23:05:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_service_trim_max_multiplier": "20", 2024-05-15T23:05:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_service_trim_min": "10", 2024-05-15T23:05:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_stash_full_interval": "25", 2024-05-15T23:05:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_trim_max": "500", 2024-05-15T23:05:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_trim_min": "250", 2024-05-15T23:05:42.523 INFO:teuthology.orchestra.run.smithi084.stdout: "perf": "true", 2024-05-15T23:05:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "pid_file": "", 2024-05-15T23:05:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-05-15T23:05:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-05-15T23:05:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "-", 2024-05-15T23:05:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrv": "", 2024-05-15T23:05:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "public_bind_addr": "-", 2024-05-15T23:05:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "public_network": "", 2024-05-15T23:05:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "public_network_interface": "", 2024-05-15T23:05:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "qat_compressor_enabled": "false", 2024-05-15T23:05:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "qat_compressor_session_max_number": "256", 2024-05-15T23:05:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "rados_mon_op_timeout": "0", 2024-05-15T23:05:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "rados_osd_op_timeout": "0", 2024-05-15T23:05:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "rados_tracing": "false", 2024-05-15T23:05:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_atime_update_interval": "60", 2024-05-15T23:05:42.524 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-05-15T23:05:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_balance_parent_reads": "false", 2024-05-15T23:05:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_balance_snap_reads": "false", 2024-05-15T23:05:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_blkin_trace_all": "false", 2024-05-15T23:05:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_blocklist_expire_seconds": "0", 2024-05-15T23:05:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_blocklist_on_break_lock": "true", 2024-05-15T23:05:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache": "true", 2024-05-15T23:05:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_block_writes_upfront": "false", 2024-05-15T23:05:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_max_dirty": "25165824", 2024-05-15T23:05:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-05-15T23:05:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_max_dirty_object": "0", 2024-05-15T23:05:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_policy": "writearound", 2024-05-15T23:05:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_size": "33554432", 2024-05-15T23:05:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_target_dirty": "16777216", 2024-05-15T23:05:42.525 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-05-15T23:05:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_clone_copy_on_read": "false", 2024-05-15T23:05:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_compression_hint": "none", 2024-05-15T23:05:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_concurrent_management_ops": "10", 2024-05-15T23:05:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-05-15T23:05:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_clone_format": "auto", 2024-05-15T23:05:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_data_pool": "", 2024-05-15T23:05:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_features": "61", 2024-05-15T23:05:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_format": "2", 2024-05-15T23:05:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_map_options": "", 2024-05-15T23:05:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_order": "22", 2024-05-15T23:05:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_pool": "rbd", 2024-05-15T23:05:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-05-15T23:05:42.526 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_stripe_count": "0", 2024-05-15T23:05:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_stripe_unit": "0", 2024-05-15T23:05:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_disable_zero_copy_writes": "true", 2024-05-15T23:05:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_discard_granularity_bytes": "65536", 2024-05-15T23:05:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-05-15T23:05:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_enable_alloc_hint": "true", 2024-05-15T23:05:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-05-15T23:05:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_io_scheduler": "simple", 2024-05-15T23:05:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-05-15T23:05:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_commit_age": "5.000000", 2024-05-15T23:05:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-05-15T23:05:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-05-15T23:05:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-05-15T23:05:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-05-15T23:05:42.527 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_flush_interval": "0", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_order": "24", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_pool": "", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_splay_width": "4", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_localize_parent_reads": "false", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_localize_snap_reads": "false", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_type": "simple", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-05-15T23:05:42.528 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-05-15T23:05:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-05-15T23:05:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-05-15T23:05:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-05-15T23:05:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-05-15T23:05:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-05-15T23:05:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_autotune": "true", 2024-05-15T23:05:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_base": "805306368", 2024-05-15T23:05:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-05-15T23:05:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-05-15T23:05:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-05-15T23:05:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-05-15T23:05:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_target": "4294967296", 2024-05-15T23:05:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-05-15T23:05:42.529 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-05-15T23:05:42.530 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-05-15T23:05:42.530 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_delete_delay": "0", 2024-05-15T23:05:42.530 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-05-15T23:05:42.530 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_replay_delay": "0", 2024-05-15T23:05:42.530 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-05-15T23:05:42.530 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-05-15T23:05:42.530 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_move_to_trash_on_remove": "false", 2024-05-15T23:05:42.530 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-05-15T23:05:42.530 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mtime_update_interval": "60", 2024-05-15T23:05:42.530 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_non_blocking_aio": "true", 2024-05-15T23:05:42.530 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_op_thread_timeout": "60", 2024-05-15T23:05:42.530 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_op_threads": "1", 2024-05-15T23:05:42.530 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_parent_cache_enabled": "false", 2024-05-15T23:05:42.530 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_persistent_cache_mode": "disabled", 2024-05-15T23:05:42.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_persistent_cache_path": "/tmp", 2024-05-15T23:05:42.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_persistent_cache_size": "1073741824", 2024-05-15T23:05:42.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_plugins": "", 2024-05-15T23:05:42.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_bps_burst": "0", 2024-05-15T23:05:42.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-05-15T23:05:42.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_bps_limit": "0", 2024-05-15T23:05:42.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_exclude_ops": "0", 2024-05-15T23:05:42.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_iops_burst": "0", 2024-05-15T23:05:42.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-05-15T23:05:42.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_iops_limit": "0", 2024-05-15T23:05:42.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_bps_burst": "0", 2024-05-15T23:05:42.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-05-15T23:05:42.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_bps_limit": "0", 2024-05-15T23:05:42.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_iops_burst": "0", 2024-05-15T23:05:42.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-05-15T23:05:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_iops_limit": "0", 2024-05-15T23:05:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_schedule_tick_min": "50", 2024-05-15T23:05:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_bps_burst": "0", 2024-05-15T23:05:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-05-15T23:05:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_bps_limit": "0", 2024-05-15T23:05:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_iops_burst": "0", 2024-05-15T23:05:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-05-15T23:05:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_iops_limit": "0", 2024-05-15T23:05:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_quiesce_notification_attempts": "10", 2024-05-15T23:05:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_read_from_replica_policy": "default", 2024-05-15T23:05:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-05-15T23:05:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_readahead_max_bytes": "524288", 2024-05-15T23:05:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_readahead_trigger_requests": "10", 2024-05-15T23:05:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_request_timed_out_seconds": "30", 2024-05-15T23:05:42.532 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_skip_partial_discard": "true", 2024-05-15T23:05:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-05-15T23:05:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_tracing": "false", 2024-05-15T23:05:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_validate_names": "true", 2024-05-15T23:05:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_validate_pool": "true", 2024-05-15T23:05:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "restapi_base_url": "", 2024-05-15T23:05:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "restapi_log_level": "", 2024-05-15T23:05:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-05-15T23:05:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_account_default_quota_max_size": "-1", 2024-05-15T23:05:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_acl_grants_max_num": "100", 2024-05-15T23:05:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_admin_entry": "admin", 2024-05-15T23:05:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-05-15T23:05:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_backend_store": "rados", 2024-05-15T23:05:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_barbican_url": "", 2024-05-15T23:05:42.533 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_beast_enable_async": "true", 2024-05-15T23:05:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_counters_cache": "false", 2024-05-15T23:05:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-05-15T23:05:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-05-15T23:05:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-05-15T23:05:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_index_max_aio": "128", 2024-05-15T23:05:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-05-15T23:05:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-05-15T23:05:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_quota_ttl": "600", 2024-05-15T23:05:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-05-15T23:05:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cache_enabled": "true", 2024-05-15T23:05:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cache_expiry_interval": "900", 2024-05-15T23:05:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cache_lru_size": "25000", 2024-05-15T23:05:42.534 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_config_store": "rados", 2024-05-15T23:05:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_content_length_compat": "false", 2024-05-15T23:05:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_copy_obj_progress": "true", 2024-05-15T23:05:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-05-15T23:05:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cors_rules_max_num": "100", 2024-05-15T23:05:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cross_domain_policy": "", 2024-05-15T23:05:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_default_encryption_key": "", 2024-05-15T23:05:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_addr": "", 2024-05-15T23:05:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_ca_path": "", 2024-05-15T23:05:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_client_cert": "", 2024-05-15T23:05:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_client_key": "", 2024-05-15T23:05:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-05-15T23:05:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_password": "", 2024-05-15T23:05:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-05-15T23:05:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_username": "", 2024-05-15T23:05:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_require_ssl": "true", 2024-05-15T23:05:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-05-15T23:05:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-05-15T23:05:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-05-15T23:05:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-05-15T23:05:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-05-15T23:05:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-05-15T23:05:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-05-15T23:05:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-05-15T23:05:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-05-15T23:05:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-05-15T23:05:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-05-15T23:05:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-05-15T23:05:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-05-15T23:05:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-05-15T23:05:42.536 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_suppress_logs": "true", 2024-05-15T23:05:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_addr": "", 2024-05-15T23:05:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_auth": "token", 2024-05-15T23:05:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_namespace": "", 2024-05-15T23:05:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_prefix": "", 2024-05-15T23:05:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-05-15T23:05:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-05-15T23:05:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-05-15T23:05:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-05-15T23:05:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_token_file": "", 2024-05-15T23:05:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-05-15T23:05:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_buffersize": "524288", 2024-05-15T23:05:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_low_speed_limit": "1024", 2024-05-15T23:05:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_low_speed_time": "300", 2024-05-15T23:05:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_tcp_keepalive": "0", 2024-05-15T23:05:42.537 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-05-15T23:05:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-05-15T23:05:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-05-15T23:05:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-05-15T23:05:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-05-15T23:05:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_fadvise": "4", 2024-05-15T23:05:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-05-15T23:05:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-05-15T23:05:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-05-15T23:05:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-05-15T23:05:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-05-15T23:05:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-05-15T23:05:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-05-15T23:05:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-05-15T23:05:42.538 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_l1_fadvise": "4", 2024-05-15T23:05:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-05-15T23:05:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-05-15T23:05:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-05-15T23:05:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_changes_size": "1000", 2024-05-15T23:05:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_num_shards": "128", 2024-05-15T23:05:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-05-15T23:05:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_window": "30", 2024-05-15T23:05:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_notify_interval_msec": "0", 2024-05-15T23:05:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_sync_poll_interval": "20", 2024-05-15T23:05:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_sync_spawn_window": "20", 2024-05-15T23:05:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-05-15T23:05:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-05-15T23:05:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-05-15T23:05:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_data_log_backing": "fifo", 2024-05-15T23:05:42.539 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_region_info_oid": "default.region", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_defer_to_bucket_acls": "", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_data_res": "500.000000", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-05-15T23:05:42.540 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-05-15T23:05:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-05-15T23:05:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dns_name": "", 2024-05-15T23:05:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dns_s3website_name": "", 2024-05-15T23:05:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dynamic_resharding": "true", 2024-05-15T23:05:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-05-15T23:05:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_gc_threads": "true", 2024-05-15T23:05:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_lc_threads": "true", 2024-05-15T23:05:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_mdsearch": "true", 2024-05-15T23:05:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_ops_log": "false", 2024-05-15T23:05:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_quota_threads": "true", 2024-05-15T23:05:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_static_website": "false", 2024-05-15T23:05:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_usage_log": "false", 2024-05-15T23:05:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enforce_swift_acls": "true", 2024-05-15T23:05:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_exit_timeout_secs": "120", 2024-05-15T23:05:42.541 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_expose_bucket": "false", 2024-05-15T23:05:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_extended_http_attrs": "", 2024-05-15T23:05:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_filter": "none", 2024-05-15T23:05:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-05-15T23:05:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_frontends": "beast port=7480", 2024-05-15T23:05:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_concurrent_io": "10", 2024-05-15T23:05:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_deferred": "50", 2024-05-15T23:05:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-05-15T23:05:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_objs": "32", 2024-05-15T23:05:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_queue_size": "134213632", 2024-05-15T23:05:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_trim_chunk": "16", 2024-05-15T23:05:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_obj_min_wait": "7200", 2024-05-15T23:05:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_processor_max_time": "3600", 2024-05-15T23:05:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_processor_period": "3600", 2024-05-15T23:05:42.542 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-05-15T23:05:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_get_obj_window_size": "16777216", 2024-05-15T23:05:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_healthcheck_disabling_path": "", 2024-05-15T23:05:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ignore_get_invalid_range": "false", 2024-05-15T23:05:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_init_timeout": "300", 2024-05-15T23:05:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-05-15T23:05:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-05-15T23:05:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_kafka_connection_idle": "300", 2024-05-15T23:05:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_kafka_message_timeout": "5000", 2024-05-15T23:05:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_kafka_sleep_timeout": "10", 2024-05-15T23:05:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-05-15T23:05:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-05-15T23:05:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-05-15T23:05:42.543 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_domain": "", 2024-05-15T23:05:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_password": "", 2024-05-15T23:05:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_password_path": "", 2024-05-15T23:05:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_project": "", 2024-05-15T23:05:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_tenant": "", 2024-05-15T23:05:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_token": "", 2024-05-15T23:05:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_token_path": "", 2024-05-15T23:05:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_user": "", 2024-05-15T23:05:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_api_version": "2", 2024-05-15T23:05:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_domain": "", 2024-05-15T23:05:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_password": "", 2024-05-15T23:05:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_project": "", 2024-05-15T23:05:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_tenant": "", 2024-05-15T23:05:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_user": "", 2024-05-15T23:05:42.544 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-05-15T23:05:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_implicit_tenants": "false", 2024-05-15T23:05:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-05-15T23:05:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_service_token_enabled": "false", 2024-05-15T23:05:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_token_cache_size": "10000", 2024-05-15T23:05:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_url": "", 2024-05-15T23:05:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_verify_ssl": "true", 2024-05-15T23:05:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_debug_interval": "-1", 2024-05-15T23:05:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_lock_max_time": "90", 2024-05-15T23:05:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_objs": "32", 2024-05-15T23:05:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_rules": "1000", 2024-05-15T23:05:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_worker": "3", 2024-05-15T23:05:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_wp_worker": "3", 2024-05-15T23:05:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_thread_delay": "0", 2024-05-15T23:05:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-05-15T23:05:42.545 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_dnattr": "uid", 2024-05-15T23:05:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-05-15T23:05:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_searchfilter": "", 2024-05-15T23:05:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-05-15T23:05:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_uri": "", 2024-05-15T23:05:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lfuda_sync_frequency": "60", 2024-05-15T23:05:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-05-15T23:05:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-05-15T23:05:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-05-15T23:05:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_http_headers": "", 2024-05-15T23:05:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_nonexistent_bucket": "false", 2024-05-15T23:05:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-05-15T23:05:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_object_name_utc": "false", 2024-05-15T23:05:42.546 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-05-15T23:05:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-05-15T23:05:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_attr_name_len": "0", 2024-05-15T23:05:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_attr_size": "0", 2024-05-15T23:05:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_attrs_num_in_req": "0", 2024-05-15T23:05:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_chunk_size": "4194304", 2024-05-15T23:05:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_concurrent_requests": "1024", 2024-05-15T23:05:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-05-15T23:05:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_dynamic_shards": "1999", 2024-05-15T23:05:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_listing_results": "1000", 2024-05-15T23:05:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_notify_retries": "10", 2024-05-15T23:05:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_objs_per_shard": "100000", 2024-05-15T23:05:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_put_param_size": "1048576", 2024-05-15T23:05:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_put_size": "5368709120", 2024-05-15T23:05:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_slo_entries": "1000", 2024-05-15T23:05:42.547 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_md_log_max_shards": "64", 2024-05-15T23:05:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_md_notify_interval_msec": "200", 2024-05-15T23:05:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_meta_sync_poll_interval": "20", 2024-05-15T23:05:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_meta_sync_spawn_window": "20", 2024-05-15T23:05:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-05-15T23:05:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_mp_lock_max_time": "600", 2024-05-15T23:05:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-05-15T23:05:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_multipart_min_part_size": "5242880", 2024-05-15T23:05:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-05-15T23:05:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-05-15T23:05:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_fhcache_size": "2017", 2024-05-15T23:05:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-05-15T23:05:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-05-15T23:05:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_lru_lanes": "5", 2024-05-15T23:05:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_max_gc": "300", 2024-05-15T23:05:42.548 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-05-15T23:05:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_gc_threads": "false", 2024-05-15T23:05:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_lc_threads": "false", 2024-05-15T23:05:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_quota_threads": "false", 2024-05-15T23:05:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_sync_thread": "false", 2024-05-15T23:05:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-05-15T23:05:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-05-15T23:05:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_num_async_rados_threads": "32", 2024-05-15T23:05:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_num_control_oids": "8", 2024-05-15T23:05:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_numa_node": "-1", 2024-05-15T23:05:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_obj_stripe_size": "4194304", 2024-05-15T23:05:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-05-15T23:05:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_objexp_chunk_size": "100", 2024-05-15T23:05:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_objexp_gc_interval": "600", 2024-05-15T23:05:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_objexp_hints_num_shards": "127", 2024-05-15T23:05:42.549 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-05-15T23:05:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-05-15T23:05:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_op_thread_timeout": "600", 2024-05-15T23:05:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_op_tracing": "false", 2024-05-15T23:05:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_opa_token": "", 2024-05-15T23:05:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_opa_url": "", 2024-05-15T23:05:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_opa_verify_ssl": "true", 2024-05-15T23:05:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-05-15T23:05:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-05-15T23:05:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_rados": "false", 2024-05-15T23:05:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_socket_path": "", 2024-05-15T23:05:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-05-15T23:05:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-05-15T23:05:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-05-15T23:05:42.550 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_push_interval": "2.000000", 2024-05-15T23:05:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_push_interval_max": "30.000000", 2024-05-15T23:05:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_root_pool": ".rgw.root", 2024-05-15T23:05:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-05-15T23:05:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-05-15T23:05:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_lanes": "3", 2024-05-15T23:05:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-05-15T23:05:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_max_buckets": "100", 2024-05-15T23:05:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_partitions": "3", 2024-05-15T23:05:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-05-15T23:05:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_print_continue": "true", 2024-05-15T23:05:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_print_prohibited_content_length": "false", 2024-05-15T23:05:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-05-15T23:05:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-05-15T23:05:42.551 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rados_tracing": "false", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_realm": "", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_realm_id": "", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_region": "", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_region_root_pool": ".rgw.root", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_relaxed_region_enforcement": "false", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_request_uri": "", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_batch_size": "64", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_max_aio": "128", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_num_logs": "16", 2024-05-15T23:05:42.552 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_thread_interval": "600", 2024-05-15T23:05:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_resolve_cname": "false", 2024-05-15T23:05:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rest_getusage_op_compat": "false", 2024-05-15T23:05:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_run_sync_thread": "true", 2024-05-15T23:05:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-05-15T23:05:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-05-15T23:05:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_keystone": "false", 2024-05-15T23:05:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_ldap": "false", 2024-05-15T23:05:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_rados": "true", 2024-05-15T23:05:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_sts": "false", 2024-05-15T23:05:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-05-15T23:05:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_success_create_obj_status": "0", 2024-05-15T23:05:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-05-15T23:05:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_scheduler_type": "throttler", 2024-05-15T23:05:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_script_uri": "", 2024-05-15T23:05:42.553 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_service_provider_name": "", 2024-05-15T23:05:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-05-15T23:05:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_client_id": "", 2024-05-15T23:05:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_client_secret": "", 2024-05-15T23:05:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_entry": "sts", 2024-05-15T23:05:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_key": "sts", 2024-05-15T23:05:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_max_session_duration": "43200", 2024-05-15T23:05:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_min_session_duration": "900", 2024-05-15T23:05:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_token_introspection_url": "", 2024-05-15T23:05:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_account_in_url": "false", 2024-05-15T23:05:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_auth_entry": "auth", 2024-05-15T23:05:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_auth_url": "", 2024-05-15T23:05:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_custom_header": "", 2024-05-15T23:05:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_enforce_content_length": "false", 2024-05-15T23:05:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_need_stats": "true", 2024-05-15T23:05:42.554 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_tenant_name": "", 2024-05-15T23:05:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_token_expiration": "86400", 2024-05-15T23:05:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_url": "", 2024-05-15T23:05:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_url_prefix": "swift", 2024-05-15T23:05:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_versioning_enabled": "false", 2024-05-15T23:05:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-05-15T23:05:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-05-15T23:05:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_lease_period": "120", 2024-05-15T23:05:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-05-15T23:05:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_interval": "1200", 2024-05-15T23:05:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-05-15T23:05:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-05-15T23:05:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-05-15T23:05:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_obj_etag_verify": "false", 2024-05-15T23:05:42.555 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_trace_history_size": "4096", 2024-05-15T23:05:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-05-15T23:05:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-05-15T23:05:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_thread_pool_size": "512", 2024-05-15T23:05:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_topic_persistency_max_retries": "0", 2024-05-15T23:05:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-05-15T23:05:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-05-15T23:05:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_topic_require_publish_policy": "false", 2024-05-15T23:05:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_comment": "", 2024-05-15T23:05:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_createby": "", 2024-05-15T23:05:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_encoding": "", 2024-05-15T23:05:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_flag": "false", 2024-05-15T23:05:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_max_size": "5368709120", 2024-05-15T23:05:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_origin": "", 2024-05-15T23:05:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_sha_unit": "524288", 2024-05-15T23:05:42.556 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_tracker": "", 2024-05-15T23:05:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_trust_forwarded_https": "false", 2024-05-15T23:05:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-05-15T23:05:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_log_tick_interval": "30", 2024-05-15T23:05:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_max_shards": "32", 2024-05-15T23:05:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_max_user_shards": "1", 2024-05-15T23:05:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_use_opa_authz": "false", 2024-05-15T23:05:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_counters_cache": "false", 2024-05-15T23:05:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_counters_cache_size": "10000", 2024-05-15T23:05:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-05-15T23:05:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_default_quota_max_size": "-1", 2024-05-15T23:05:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_max_buckets": "1000", 2024-05-15T23:05:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_policies_max_num": "100", 2024-05-15T23:05:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-05-15T23:05:42.557 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-05-15T23:05:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_sync_interval": "86400", 2024-05-15T23:05:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-05-15T23:05:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_unique_email": "true", 2024-05-15T23:05:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_verify_ssl": "true", 2024-05-15T23:05:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_website_routing_rules_max_num": "50", 2024-05-15T23:05:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zone": "", 2024-05-15T23:05:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zone_id": "", 2024-05-15T23:05:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-05-15T23:05:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zonegroup": "", 2024-05-15T23:05:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zonegroup_id": "", 2024-05-15T23:05:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-05-15T23:05:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "rgwlc_auto_session_clear": "true", 2024-05-15T23:05:42.558 INFO:teuthology.orchestra.run.smithi084.stdout: "rgwlc_skip_bucket_step": "false", 2024-05-15T23:05:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_block_size": "4096", 2024-05-15T23:05:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-05-15T23:05:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-05-15T23:05:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-05-15T23:05:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-05-15T23:05:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_shard_bits": "4", 2024-05-15T23:05:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_size": "536870912", 2024-05-15T23:05:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_type": "binned_lru", 2024-05-15T23:05:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-05-15T23:05:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-05-15T23:05:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-05-15T23:05:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_collect_compaction_stats": "false", 2024-05-15T23:05:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_collect_extended_stats": "false", 2024-05-15T23:05:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_collect_memory_stats": "false", 2024-05-15T23:05:42.559 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-05-15T23:05:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_index_type": "binary_search", 2024-05-15T23:05:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_log_to_ceph_log": "true", 2024-05-15T23:05:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_metadata_block_size": "4096", 2024-05-15T23:05:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_partition_filters": "false", 2024-05-15T23:05:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_perf": "false", 2024-05-15T23:05:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-05-15T23:05:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-05-15T23:05:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "rotating_keys_renewal_timeout": "10", 2024-05-15T23:05:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "run_dir": "/var/run/ceph", 2024-05-15T23:05:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_block_create": "true", 2024-05-15T23:05:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_cache_lru_size": "67108864", 2024-05-15T23:05:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_cbjournal_size": "5368709120", 2024-05-15T23:05:42.560 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_data_delta_based_overwrite": "0", 2024-05-15T23:05:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_default_max_object_size": "16777216", 2024-05-15T23:05:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-05-15T23:05:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_device_size": "53687091200", 2024-05-15T23:05:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_full_integrity_check": "false", 2024-05-15T23:05:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_batch_capacity": "16", 2024-05-15T23:05:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-05-15T23:05:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-05-15T23:05:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_iodepth_limit": "5", 2024-05-15T23:05:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_main_device_type": "SSD", 2024-05-15T23:05:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_max_concurrent_transactions": "8", 2024-05-15T23:05:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_max_data_allocation_size": "32768", 2024-05-15T23:05:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-05-15T23:05:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-05-15T23:05:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-05-15T23:05:42.561 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-05-15T23:05:42.562 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_segment_size": "67108864", 2024-05-15T23:05:42.562 INFO:teuthology.orchestra.run.smithi084.stdout: "set_keepcaps": "false", 2024-05-15T23:05:42.562 INFO:teuthology.orchestra.run.smithi084.stdout: "setgroup": "ceph", 2024-05-15T23:05:42.562 INFO:teuthology.orchestra.run.smithi084.stdout: "setuser": "ceph", 2024-05-15T23:05:42.562 INFO:teuthology.orchestra.run.smithi084.stdout: "setuser_match_path": "", 2024-05-15T23:05:42.562 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_misplaced_ratio": "0.050000", 2024-05-15T23:05:42.562 INFO:teuthology.orchestra.run.smithi084.stdout: "thp": "false", 2024-05-15T23:05:42.562 INFO:teuthology.orchestra.run.smithi084.stdout: "threadpool_default_timeout": "60", 2024-05-15T23:05:42.562 INFO:teuthology.orchestra.run.smithi084.stdout: "threadpool_empty_queue_max_wait": "2", 2024-05-15T23:05:42.562 INFO:teuthology.orchestra.run.smithi084.stdout: "throttler_perf_counter": "true" 2024-05-15T23:05:42.562 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:05:43.131 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-05-15T23:05:43.394 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:43.955 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:43 smithi084 ceph-mon[39784]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:44.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:43 smithi090 ceph-mon[44434]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:44.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:43 smithi088 ceph-mon[44699]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:45.186 INFO:teuthology.orchestra.run.smithi084.stdout:mon.a: {} 2024-05-15T23:05:45.197 INFO:teuthology.orchestra.run.smithi084.stdout:mon.c: {} 2024-05-15T23:05:45.208 INFO:teuthology.orchestra.run.smithi084.stdout:mon.b: {} 2024-05-15T23:05:45.209 INFO:teuthology.orchestra.run.smithi084.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-15T23:05:45.209 INFO:teuthology.orchestra.run.smithi084.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-15T23:05:45.210 INFO:teuthology.orchestra.run.smithi084.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-15T23:05:45.878 INFO:teuthology.run_tasks:Running task full_sequential... 2024-05-15T23:05:45.888 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T23:05:45.890 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T23:05:45.890 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-05-15T23:05:45.890 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-05-15T23:05:45.890 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-05-15T23:05:45.891 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:05:45.891 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:05:45.891 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-05-15T23:05:45.891 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump -f json-pretty 2024-05-15T23:05:45.891 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-05-15T23:05:45.892 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:05:45.892 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-05-15T23:05:45.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:05:45.892 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-05-15T23:05:45.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T23:05:45.893 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-05-15T23:05:45.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:05:45.927 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:05:45.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:45.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:05:46.054 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:05:46.085 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:46.105 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:05:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:46.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:05:46.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:45 smithi084 ceph-mon[39784]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:46.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/930255484' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T23:05:46.208 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:46.208 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:05:46.234 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:45 smithi088 ceph-mon[44699]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:46.234 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/930255484' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T23:05:46.236 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:46.255 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:46.256 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:05:46.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:46.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:05:46.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:45 smithi090 ceph-mon[44434]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:46.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/930255484' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T23:05:46.360 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:05:46.387 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:46.485 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:05:46.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:46.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:05:46.489 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:46.589 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:05:46.620 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:46.640 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:05:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:46.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:05:46.744 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:05:46.772 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:46.791 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:05:46.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:46.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:05:46.895 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:05:46.923 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:46.942 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:05:46.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:46.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:05:47.046 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:05:47.074 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:47.523 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:05:47.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:47.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:05:47.626 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:05:47.659 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:47.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:47.684 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:05:47.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:05:47.786 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:05:47.820 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:47.841 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:05:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:47.844 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:05:47.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:05:47.976 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:47.996 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:05:47.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:47.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:05:48.100 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:05:48.132 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:48.154 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:05:48.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:48.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:05:48.223 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:47 smithi084 ceph-mon[39784]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:48.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:05:48.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:47 smithi090 ceph-mon[44434]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:48.307 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:05:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:48.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:05:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:47 smithi088 ceph-mon[44699]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:48.412 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:05:48.444 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:48.465 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:05:48.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:48.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:05:48.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:05:48.600 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:48.621 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:05:48.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:48.623 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:05:48.724 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:05:48.756 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:48.777 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:05:48.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:48.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:05:48.811 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:48.881 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:05:48.909 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:49.458 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:05:49.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:49.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:05:49.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:05:49.596 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:49.617 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:05:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:49.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:05:49.722 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:05:49.752 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:49.995 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:50.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:49 smithi084 ceph-mon[39784]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:50.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:49 smithi090 ceph-mon[44434]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:50.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:49 smithi088 ceph-mon[44699]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:50.334 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:05:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:50.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:05:50.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:05:50.470 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:50.491 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:05:50.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:50.494 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:05:50.595 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:05:50.626 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:50.648 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:05:50.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:50.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:05:50.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:05:50.782 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:50.802 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:05:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:50.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:05:50.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:05:50.929 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_0' created 2024-05-15T23:05:50.930 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:05:50.939 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:51.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/641436786' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-05-15T23:05:51.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/646201712' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:05:51.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/641436786' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-05-15T23:05:51.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/646201712' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:05:51.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/641436786' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-05-15T23:05:51.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/646201712' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:05:51.639 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:05:51.640 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:05:51.640 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 102, 2024-05-15T23:05:51.640 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "3999fb7e-130e-11ef-bc99-c7b262605968", 2024-05-15T23:05:51.640 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-05-15T22:56:41.212493+0000", 2024-05-15T23:05:51.640 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-05-15T23:05:50.918160+0000", 2024-05-15T23:05:51.640 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-05-15T23:03:56.576073+0000", 2024-05-15T23:05:51.640 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-05-15T23:03:38.460671+0000", 2024-05-15T23:05:51.640 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-15T23:05:51.640 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5801984, 2024-05-15T23:05:51.640 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-05-15T23:05:51.640 INFO:teuthology.orchestra.run.smithi084.stdout: "noscrub", 2024-05-15T23:05:51.640 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-05-15T23:05:51.640 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-05-15T23:05:51.640 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-05-15T23:05:51.641 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-05-15T23:05:51.641 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:05:51.641 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 27, 2024-05-15T23:05:51.641 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-05-15T23:05:51.641 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-15T23:05:51.641 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-15T23:05:51.641 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-05-15T23:05:51.641 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 2, 2024-05-15T23:05:51.641 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 12, 2024-05-15T23:05:51.641 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-05-15T23:05:51.641 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "jewel", 2024-05-15T23:05:51.641 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-05-15T23:05:51.641 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-05-15T23:05:51.641 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-05-15T22:59:38.603575+0000", 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-05-15T23:05:51.642 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-05-15T23:05:51.643 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 1, 2024-05-15T23:05:51.643 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 1, 2024-05-15T23:05:51.643 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 1, 2024-05-15T23:05:51.643 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 1, 2024-05-15T23:05:51.643 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 1, 2024-05-15T23:05:51.643 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-05-15T23:05:51.643 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-05-15T23:05:51.643 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-05-15T23:05:51.643 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-05-15T23:05:51.643 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-05-15T23:05:51.643 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-05-15T23:05:51.643 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-05-15T23:05:51.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.643 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "16", 2024-05-15T23:05:51.643 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-05-15T23:05:51.644 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-05-15T23:05:51.645 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-05-15T23:05:51.645 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T23:05:51.645 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T23:05:51.645 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T23:05:51.645 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-05-15T23:05:51.645 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-05-15T23:05:51.645 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-05-15T23:05:51.645 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-05-15T23:05:51.645 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-05-15T23:05:51.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.645 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-05-15T23:05:51.645 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-05-15T23:05:51.645 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-05-15T23:05:51.645 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-05-15T23:05:51.645 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-05-15T23:05:51.646 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T23:05:51.646 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-05-15T23:05:51.646 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-05-15T23:05:51.646 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-05-15T23:05:51.646 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-05-15T23:05:51.646 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-05-15T23:05:51.646 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-05-15T23:05:51.646 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-05-15T23:05:51.646 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-05-15T23:05:51.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.646 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-05-15T23:05:51.646 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-05-15T23:05:51.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.646 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-05-15T23:05:51.647 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-05-15T23:05:51.647 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 11.760000228881836, 2024-05-15T23:05:51.647 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 11.760000228881836, 2024-05-15T23:05:51.647 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.17000000178813934, 2024-05-15T23:05:51.647 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-05-15T23:05:51.647 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-05-15T23:05:51.647 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-05-15T23:05:51.647 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-05-15T23:05:51.647 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-05-15T23:05:51.647 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.647 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-05-15T23:05:51.647 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-05-15T23:05:51.647 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-05-15T23:05:50.137063+0000", 2024-05-15T23:05:51.648 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 32769, 2024-05-15T23:05:51.648 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,creating", 2024-05-15T23:05:51.648 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-05-15T23:05:51.648 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2024-05-15T23:05:51.648 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-05-15T23:05:51.648 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-05-15T23:05:51.648 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-05-15T23:05:51.648 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-05-15T23:05:51.648 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T23:05:51.648 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T23:05:51.648 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-05-15T23:05:51.648 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-05-15T23:05:51.648 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 16, 2024-05-15T23:05:51.648 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 16, 2024-05-15T23:05:51.648 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 16, 2024-05-15T23:05:51.649 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 16, 2024-05-15T23:05:51.649 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 16, 2024-05-15T23:05:51.649 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-05-15T23:05:51.649 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-05-15T23:05:51.649 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-05-15T23:05:51.649 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-05-15T23:05:51.649 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-05-15T23:05:51.649 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-05-15T23:05:51.649 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-05-15T23:05:51.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.649 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "102", 2024-05-15T23:05:51.649 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-05-15T23:05:51.649 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T23:05:51.649 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T23:05:51.650 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-05-15T23:05:51.650 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-05-15T23:05:51.650 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-05-15T23:05:51.650 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-05-15T23:05:51.650 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-05-15T23:05:51.650 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-05-15T23:05:51.650 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-05-15T23:05:51.650 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-05-15T23:05:51.650 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-05-15T23:05:51.650 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-05-15T23:05:51.651 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-05-15T23:05:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-05-15T23:05:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-05-15T23:05:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T23:05:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-05-15T23:05:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-05-15T23:05:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-05-15T23:05:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-05-15T23:05:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-05-15T23:05:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-05-15T23:05:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": {}, 2024-05-15T23:05:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-05-15T23:05:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-05-15T23:05:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.25, 2024-05-15T23:05:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.25, 2024-05-15T23:05:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.25, 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.25, 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "80159cc8-1b6c-4aa6-a114-966786d7be1e", 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:05:51.653 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 53, 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1084680531 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.654 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:05:51.655 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.655 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.655 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-05-15T23:05:51.655 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1084680531 2024-05-15T23:05:51.655 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.655 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.655 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:05:51.655 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.655 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.655 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-05-15T23:05:51.656 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1084680531 2024-05-15T23:05:51.656 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.656 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.656 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:05:51.656 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.656 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.656 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-05-15T23:05:51.656 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1084680531 2024-05-15T23:05:51.656 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.656 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.657 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/1084680531", 2024-05-15T23:05:51.657 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/1084680531", 2024-05-15T23:05:51.657 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/1084680531", 2024-05-15T23:05:51.657 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/1084680531", 2024-05-15T23:05:51.657 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:05:51.657 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:05:51.657 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:05:51.657 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.657 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-05-15T23:05:51.657 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "9c2c76c9-ce0f-4c2d-9d96-c87a10187607", 2024-05-15T23:05:51.658 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:05:51.658 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:05:51.658 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:05:51.658 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:05:51.658 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:05:51.658 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:05:51.658 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-05-15T23:05:51.658 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 14, 2024-05-15T23:05:51.658 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:05:51.658 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:05:51.658 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:05:51.658 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.660 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:05:51.660 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.660 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.660 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-05-15T23:05:51.660 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:05:51.660 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.660 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.660 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:05:51.660 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.660 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.661 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-05-15T23:05:51.661 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:05:51.661 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.661 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.661 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/2140830229", 2024-05-15T23:05:51.661 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/2140830229", 2024-05-15T23:05:51.661 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/2140830229", 2024-05-15T23:05:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/2140830229", 2024-05-15T23:05:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:05:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:05:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:05:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-05-15T23:05:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "e8514daa-b5f9-4f2f-a432-53f4a4eb86bc", 2024-05-15T23:05:51.663 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:05:51.663 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:05:51.663 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:05:51.663 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:05:51.663 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:05:51.663 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:05:51.663 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 20, 2024-05-15T23:05:51.663 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2024-05-15T23:05:51.663 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:05:51.663 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:05:51.664 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:05:51.664 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.664 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.664 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-05-15T23:05:51.664 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2699996290 2024-05-15T23:05:51.664 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.664 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.665 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:05:51.665 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.665 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.665 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-05-15T23:05:51.665 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2699996290 2024-05-15T23:05:51.665 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.665 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.666 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:05:51.666 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.666 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.666 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-05-15T23:05:51.666 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2699996290 2024-05-15T23:05:51.666 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.666 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.667 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:05:51.667 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.667 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.667 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-05-15T23:05:51.667 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2699996290 2024-05-15T23:05:51.667 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.667 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.668 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/2699996290", 2024-05-15T23:05:51.668 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/2699996290", 2024-05-15T23:05:51.668 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/2699996290", 2024-05-15T23:05:51.668 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/2699996290", 2024-05-15T23:05:51.668 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:05:51.668 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:05:51.668 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:05:51.668 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.669 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-05-15T23:05:51.669 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "5d9965f9-c0b2-48ce-869c-582990985623", 2024-05-15T23:05:51.669 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:05:51.669 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:05:51.669 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:05:51.669 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:05:51.669 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:05:51.669 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:05:51.669 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 26, 2024-05-15T23:05:51.669 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2024-05-15T23:05:51.669 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:05:51.669 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:05:51.670 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:05:51.670 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.670 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.670 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-05-15T23:05:51.670 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:05:51.670 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.670 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.670 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:05:51.670 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.670 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.671 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-05-15T23:05:51.671 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:05:51.671 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.671 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.671 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:05:51.671 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.671 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.671 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-05-15T23:05:51.671 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:05:51.671 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.671 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.671 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:05:51.672 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.672 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.672 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-05-15T23:05:51.672 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:05:51.672 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.672 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.672 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2204039828", 2024-05-15T23:05:51.672 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2204039828", 2024-05-15T23:05:51.672 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2204039828", 2024-05-15T23:05:51.672 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2204039828", 2024-05-15T23:05:51.672 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:05:51.672 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:05:51.672 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:05:51.673 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.673 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-05-15T23:05:51.673 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4bc50523-8480-44d2-8d76-31479e839c85", 2024-05-15T23:05:51.673 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:05:51.673 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:05:51.673 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:05:51.673 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:05:51.673 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:05:51.673 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:05:51.673 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-05-15T23:05:51.673 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2024-05-15T23:05:51.673 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:05:51.673 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:05:51.674 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:05:51.674 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.674 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.674 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6800", 2024-05-15T23:05:51.674 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:05:51.674 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.674 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.674 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:05:51.674 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.674 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6801", 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6803", 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:05:51.675 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6802", 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6800/2267955292", 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6801/2267955292", 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6803/2267955292", 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6802/2267955292", 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.677 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-05-15T23:05:51.677 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20", 2024-05-15T23:05:51.677 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:05:51.677 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:05:51.677 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:05:51.677 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:05:51.677 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:05:51.677 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:05:51.677 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 37, 2024-05-15T23:05:51.677 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 38, 2024-05-15T23:05:51.678 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:05:51.678 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:05:51.678 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:05:51.678 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.678 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.678 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6804", 2024-05-15T23:05:51.678 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:05:51.678 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.678 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.678 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:05:51.678 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.679 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.679 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6805", 2024-05-15T23:05:51.679 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:05:51.679 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.679 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.679 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:05:51.679 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.679 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.679 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6807", 2024-05-15T23:05:51.680 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:05:51.680 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.680 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.680 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:05:51.680 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.680 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.680 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6806", 2024-05-15T23:05:51.680 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:05:51.680 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.680 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.681 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6804/661041538", 2024-05-15T23:05:51.681 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6805/661041538", 2024-05-15T23:05:51.681 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6807/661041538", 2024-05-15T23:05:51.681 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6806/661041538", 2024-05-15T23:05:51.681 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:05:51.681 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:05:51.681 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:05:51.681 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.681 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-05-15T23:05:51.681 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f", 2024-05-15T23:05:51.681 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:05:51.681 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:05:51.682 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:05:51.682 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:05:51.682 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:05:51.682 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:05:51.682 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 46, 2024-05-15T23:05:51.682 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 47, 2024-05-15T23:05:51.682 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:05:51.682 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:05:51.682 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:05:51.682 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.682 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.682 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6808", 2024-05-15T23:05:51.683 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:05:51.683 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.683 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.683 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:05:51.683 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.683 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.683 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6809", 2024-05-15T23:05:51.683 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:05:51.683 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.683 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.683 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:05:51.684 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.684 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.684 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6811", 2024-05-15T23:05:51.684 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:05:51.684 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.684 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.684 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:05:51.684 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.684 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.684 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6810", 2024-05-15T23:05:51.684 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:05:51.684 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6808/380856869", 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6809/380856869", 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6811/380856869", 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6810/380856869", 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34", 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:05:51.685 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:05:51.686 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:05:51.686 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:05:51.686 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:05:51.686 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:05:51.686 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 53, 2024-05-15T23:05:51.686 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 82, 2024-05-15T23:05:51.686 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:05:51.686 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:05:51.686 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:05:51.686 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.686 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.686 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6812", 2024-05-15T23:05:51.687 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:05:51.687 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.687 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.687 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:05:51.687 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.687 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.687 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6813", 2024-05-15T23:05:51.687 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:05:51.687 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.687 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.687 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:05:51.687 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6815", 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6814", 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.689 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6812/1044709063", 2024-05-15T23:05:51.689 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6813/1044709063", 2024-05-15T23:05:51.689 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6815/1044709063", 2024-05-15T23:05:51.689 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6814/1044709063", 2024-05-15T23:05:51.689 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:05:51.689 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:05:51.689 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:05:51.689 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.689 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-05-15T23:05:51.689 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "02cab286-d487-4a66-982c-3cf28f55e851", 2024-05-15T23:05:51.689 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:05:51.689 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:05:51.690 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:05:51.690 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:05:51.690 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:05:51.690 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:05:51.690 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 61, 2024-05-15T23:05:51.690 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2024-05-15T23:05:51.690 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:05:51.690 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:05:51.690 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:05:51.690 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.690 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.690 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6800", 2024-05-15T23:05:51.690 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 522459126 2024-05-15T23:05:51.691 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.691 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.691 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:05:51.691 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.691 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.691 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6801", 2024-05-15T23:05:51.691 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 522459126 2024-05-15T23:05:51.691 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.691 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.691 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:05:51.691 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.692 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.692 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6803", 2024-05-15T23:05:51.692 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 522459126 2024-05-15T23:05:51.692 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.692 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.692 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:05:51.692 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.692 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.692 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6802", 2024-05-15T23:05:51.692 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 522459126 2024-05-15T23:05:51.692 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.692 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.693 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6800/522459126", 2024-05-15T23:05:51.693 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6801/522459126", 2024-05-15T23:05:51.693 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6803/522459126", 2024-05-15T23:05:51.693 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6802/522459126", 2024-05-15T23:05:51.693 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:05:51.693 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:05:51.693 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:05:51.693 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.693 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-05-15T23:05:51.693 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613", 2024-05-15T23:05:51.693 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:05:51.693 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:05:51.694 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:05:51.694 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:05:51.694 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:05:51.694 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:05:51.694 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 68, 2024-05-15T23:05:51.694 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2024-05-15T23:05:51.694 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:05:51.694 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:05:51.694 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:05:51.694 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.694 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.694 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6804", 2024-05-15T23:05:51.694 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6805", 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.696 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6807", 2024-05-15T23:05:51.696 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:05:51.696 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.696 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.696 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:05:51.696 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.696 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.696 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6806", 2024-05-15T23:05:51.696 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:05:51.696 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.696 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.696 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6804/2794571133", 2024-05-15T23:05:51.697 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6805/2794571133", 2024-05-15T23:05:51.697 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6807/2794571133", 2024-05-15T23:05:51.697 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6806/2794571133", 2024-05-15T23:05:51.697 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:05:51.697 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:05:51.697 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:05:51.697 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.697 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-05-15T23:05:51.697 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473", 2024-05-15T23:05:51.697 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:05:51.697 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:05:51.697 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:05:51.697 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:05:51.698 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:05:51.698 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:05:51.698 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 74, 2024-05-15T23:05:51.698 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2024-05-15T23:05:51.698 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:05:51.698 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:05:51.698 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:05:51.698 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.698 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.698 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6808", 2024-05-15T23:05:51.698 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:05:51.698 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.699 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.699 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:05:51.699 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.699 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.699 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6809", 2024-05-15T23:05:51.699 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:05:51.699 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.699 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.699 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:05:51.699 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.700 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.700 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6811", 2024-05-15T23:05:51.700 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:05:51.700 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.700 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.700 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:05:51.700 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.700 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.700 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6810", 2024-05-15T23:05:51.700 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:05:51.700 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.700 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.701 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6808/3735380744", 2024-05-15T23:05:51.701 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6809/3735380744", 2024-05-15T23:05:51.701 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6811/3735380744", 2024-05-15T23:05:51.701 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6810/3735380744", 2024-05-15T23:05:51.701 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:05:51.701 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:05:51.701 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:05:51.701 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.701 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-05-15T23:05:51.701 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5", 2024-05-15T23:05:51.701 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:05:51.701 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:05:51.701 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:05:51.702 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:05:51.702 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:05:51.702 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:05:51.702 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 81, 2024-05-15T23:05:51.702 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2024-05-15T23:05:51.702 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:05:51.702 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:05:51.702 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:05:51.702 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.703 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.703 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6812", 2024-05-15T23:05:51.703 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:05:51.703 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.703 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.703 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:05:51.703 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.703 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.703 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6813", 2024-05-15T23:05:51.703 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:05:51.703 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.703 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6815", 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6814", 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:05:51.704 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6812/2265615270", 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6813/2265615270", 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6815/2265615270", 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6814/2265615270", 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-05-15T23:05:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:05:51.706 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:05:51.706 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:05:51.706 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:05:51.706 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:05:51.706 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:05:36.073777+0000", 2024-05-15T23:05:51.706 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:05:51.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.706 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-05-15T23:05:51.706 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:05:51.706 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:05:51.706 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:05:51.706 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:05:51.707 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:05:51.707 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:04:37.591135+0000", 2024-05-15T23:05:51.707 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:05:51.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.707 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-05-15T23:05:51.707 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:05:51.707 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:05:51.707 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:05:51.707 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:05:51.707 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:05:51.707 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:05:20.624193+0000", 2024-05-15T23:05:51.707 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:05:51.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.708 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-05-15T23:05:51.708 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:05:51.708 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:05:51.708 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:05:51.708 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:05:51.708 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:05:51.708 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:05:11.351028+0000", 2024-05-15T23:05:51.708 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:05:51.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.708 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-05-15T23:05:51.708 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:05:51.709 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:05:51.709 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:05:51.709 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:05:51.709 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:05:51.709 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:05:38.362254+0000", 2024-05-15T23:05:51.709 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:05:51.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.709 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-05-15T23:05:51.709 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:05:51.709 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:05:51.709 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:05:51.710 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:05:51.710 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:05:51.710 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:04:49.088415+0000", 2024-05-15T23:05:51.710 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:05:51.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.710 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-05-15T23:05:51.710 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:05:51.710 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:05:51.710 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:05:51.710 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:05:51.710 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:05:51.710 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:05:37.820222+0000", 2024-05-15T23:05:51.711 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:05:51.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.711 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-05-15T23:05:51.711 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:05:51.711 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:05:51.711 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:05:51.711 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:05:51.711 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:05:51.711 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:04:55.986981+0000", 2024-05-15T23:05:51.711 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:05:51.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.711 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-05-15T23:05:51.712 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:05:51.712 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:05:51.712 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:05:51.712 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:05:51.712 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:05:51.712 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:05:06.187913+0000", 2024-05-15T23:05:51.712 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:05:51.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.712 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-05-15T23:05:51.712 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:05:51.712 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:05:51.712 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:05:51.712 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:05:51.712 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:05:23.099371+0000", 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:04:37.100129+0000", 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-05-15T23:05:51.713 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:05:51.714 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:05:51.714 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:05:51.714 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:05:51.714 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:05:51.714 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:05:15.622583+0000", 2024-05-15T23:05:51.714 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:05:51.714 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.714 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:05:51.714 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [], 2024-05-15T23:05:51.714 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [], 2024-05-15T23:05:51.714 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-05-15T23:05:51.714 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-05-15T23:05:51.714 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2581408231": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/1136984393": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2443201864": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1252671888": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1451001371": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4084814866": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4076409285": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2988977694": "2024-05-16T22:57:38.505263+0000" 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-05-15T23:05:51.715 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [], 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-05-15T23:05:51.716 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:05:51.717 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:05:51.880 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-05-15T23:05:52.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:51 smithi084 ceph-mon[39784]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:52.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:51 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:05:52.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/641436786' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-05-15T23:05:52.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/646201712' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:05:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:51 smithi084 ceph-mon[39784]: osdmap e102: 12 total, 12 up, 12 in 2024-05-15T23:05:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3353548163' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:05:52.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:51 smithi090 ceph-mon[44434]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:52.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:51 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:05:52.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/641436786' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-05-15T23:05:52.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/646201712' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:05:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:51 smithi090 ceph-mon[44434]: osdmap e102: 12 total, 12 up, 12 in 2024-05-15T23:05:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3353548163' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:05:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:51 smithi088 ceph-mon[44699]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:51 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:05:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/641436786' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-05-15T23:05:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/646201712' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:05:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:51 smithi088 ceph-mon[44699]: osdmap e102: 12 total, 12 up, 12 in 2024-05-15T23:05:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3353548163' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:05:53.191 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:52 smithi084 ceph-mon[39784]: osdmap e103: 12 total, 12 up, 12 in 2024-05-15T23:05:53.191 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:05:53.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:53.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:05:53.217 INFO:tasks.thrashosds.thrasher:j is {'epoch': 102, 'fsid': '3999fb7e-130e-11ef-bc99-c7b262605968', 'created': '2024-05-15T22:56:41.212493+0000', 'modified': '2024-05-15T23:05:50.918160+0000', 'last_up_change': '2024-05-15T23:03:56.576073+0000', 'last_in_change': '2024-05-15T23:03:38.460671+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T22:59:38.603575+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-05-15T23:05:50.137063+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '80159cc8-1b6c-4aa6-a114-966786d7be1e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6801', 'nonce': 1084680531}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6802', 'nonce': 1084680531}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6804', 'nonce': 1084680531}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6803', 'nonce': 1084680531}]}, 'public_addr': '172.21.15.84:6801/1084680531', 'cluster_addr': '172.21.15.84:6802/1084680531', 'heartbeat_back_addr': '172.21.15.84:6804/1084680531', 'heartbeat_front_addr': '172.21.15.84:6803/1084680531', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9c2c76c9-ce0f-4c2d-9d96-c87a10187607', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 14, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6805', 'nonce': 2140830229}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6806', 'nonce': 2140830229}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6808', 'nonce': 2140830229}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6807', 'nonce': 2140830229}]}, 'public_addr': '172.21.15.84:6805/2140830229', 'cluster_addr': '172.21.15.84:6806/2140830229', 'heartbeat_back_addr': '172.21.15.84:6808/2140830229', 'heartbeat_front_addr': '172.21.15.84:6807/2140830229', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e8514daa-b5f9-4f2f-a432-53f4a4eb86bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 20, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6809', 'nonce': 2699996290}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6810', 'nonce': 2699996290}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6812', 'nonce': 2699996290}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6811', 'nonce': 2699996290}]}, 'public_addr': '172.21.15.84:6809/2699996290', 'cluster_addr': '172.21.15.84:6810/2699996290', 'heartbeat_back_addr': '172.21.15.84:6812/2699996290', 'heartbeat_front_addr': '172.21.15.84:6811/2699996290', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5d9965f9-c0b2-48ce-869c-582990985623', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6813', 'nonce': 2204039828}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6814', 'nonce': 2204039828}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6816', 'nonce': 2204039828}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6815', 'nonce': 2204039828}]}, 'public_addr': '172.21.15.84:6813/2204039828', 'cluster_addr': '172.21.15.84:6814/2204039828', 'heartbeat_back_addr': '172.21.15.84:6816/2204039828', 'heartbeat_front_addr': '172.21.15.84:6815/2204039828', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4bc50523-8480-44d2-8d76-31479e839c85', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6800', 'nonce': 2267955292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6801', 'nonce': 2267955292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6803', 'nonce': 2267955292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6802', 'nonce': 2267955292}]}, 'public_addr': '172.21.15.88:6800/2267955292', 'cluster_addr': '172.21.15.88:6801/2267955292', 'heartbeat_back_addr': '172.21.15.88:6803/2267955292', 'heartbeat_front_addr': '172.21.15.88:6802/2267955292', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'd4db8d7f-afc7-43dc-81fe-f70c290d8b20', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 38, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6804', 'nonce': 661041538}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6805', 'nonce': 661041538}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6807', 'nonce': 661041538}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6806', 'nonce': 661041538}]}, 'public_addr': '172.21.15.88:6804/661041538', 'cluster_addr': '172.21.15.88:6805/661041538', 'heartbeat_back_addr': '172.21.15.88:6807/661041538', 'heartbeat_front_addr': '172.21.15.88:6806/661041538', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'a2773454-2318-4a5b-8f31-0952b1b07f2f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6808', 'nonce': 380856869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6809', 'nonce': 380856869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6811', 'nonce': 380856869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6810', 'nonce': 380856869}]}, 'public_addr': '172.21.15.88:6808/380856869', 'cluster_addr': '172.21.15.88:6809/380856869', 'heartbeat_back_addr': '172.21.15.88:6811/380856869', 'heartbeat_front_addr': '172.21.15.88:6810/380856869', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '028ca7b6-3e00-4466-a33f-b726cd60cb34', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 82, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6812', 'nonce': 1044709063}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6813', 'nonce': 1044709063}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6815', 'nonce': 1044709063}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6814', 'nonce': 1044709063}]}, 'public_addr': '172.21.15.88:6812/1044709063', 'cluster_addr': '172.21.15.88:6813/1044709063', 'heartbeat_back_addr': '172.21.15.88:6815/1044709063', 'heartbeat_front_addr': '172.21.15.88:6814/1044709063', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '02cab286-d487-4a66-982c-3cf28f55e851', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6800', 'nonce': 522459126}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 522459126}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 522459126}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6802', 'nonce': 522459126}]}, 'public_addr': '172.21.15.90:6800/522459126', 'cluster_addr': '172.21.15.90:6801/522459126', 'heartbeat_back_addr': '172.21.15.90:6803/522459126', 'heartbeat_front_addr': '172.21.15.90:6802/522459126', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '47d38bc7-39ba-4b35-af24-e282354dd613', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 68, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6804', 'nonce': 2794571133}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 2794571133}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 2794571133}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6806', 'nonce': 2794571133}]}, 'public_addr': '172.21.15.90:6804/2794571133', 'cluster_addr': '172.21.15.90:6805/2794571133', 'heartbeat_back_addr': '172.21.15.90:6807/2794571133', 'heartbeat_front_addr': '172.21.15.90:6806/2794571133', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'd7b2aeae-d012-41eb-9f91-e4c05c649473', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6808', 'nonce': 3735380744}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 3735380744}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 3735380744}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 3735380744}]}, 'public_addr': '172.21.15.90:6808/3735380744', 'cluster_addr': '172.21.15.90:6809/3735380744', 'heartbeat_back_addr': '172.21.15.90:6811/3735380744', 'heartbeat_front_addr': '172.21.15.90:6810/3735380744', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6812', 'nonce': 2265615270}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 2265615270}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 2265615270}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 2265615270}]}, 'public_addr': '172.21.15.90:6812/2265615270', 'cluster_addr': '172.21.15.90:6813/2265615270', 'heartbeat_back_addr': '172.21.15.90:6815/2265615270', 'heartbeat_front_addr': '172.21.15.90:6814/2265615270', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:05:36.073777+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:04:37.591135+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:05:20.624193+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:05:11.351028+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:05:38.362254+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:04:49.088415+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:05:37.820222+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:04:55.986981+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:05:06.187913+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:05:23.099371+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:04:37.100129+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:05:15.622583+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/2581408231': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:6800/1136984393': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:6800/2443201864': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/1252671888': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/1451001371': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/4084814866': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:0/4076409285': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/2988977694': '2024-05-16T22:57:38.505263+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-15T23:05:53.217 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:05:53.295 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:05:53.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:52 smithi090 ceph-mon[44434]: osdmap e103: 12 total, 12 up, 12 in 2024-05-15T23:05:53.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:52 smithi088 ceph-mon[44699]: osdmap e103: 12 total, 12 up, 12 in 2024-05-15T23:05:53.328 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:53.349 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:05:53.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:53.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:05:53.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:05:53.484 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:53.506 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:05:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:53.509 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:05:53.610 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:05:53.642 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:53.662 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:05:53.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:53.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:05:53.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:05:53.794 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:54.043 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:05:54.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:54.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:05:54.148 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:05:54.176 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:54.229 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:05:54.230 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:53 smithi084 ceph-mon[39784]: pgmap v340: 17 pgs: 4 creating+peering, 2 active+clean, 11 unknown; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:54.230 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:53 smithi084 ceph-mon[39784]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:05:54.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:53 smithi084 ceph-mon[39784]: osdmap e104: 12 total, 12 up, 12 in 2024-05-15T23:05:54.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:54.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:05:54.265 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:05:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:05:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-05-15T23:05:54.265 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:05:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:53 smithi090 ceph-mon[44434]: pgmap v340: 17 pgs: 4 creating+peering, 2 active+clean, 11 unknown; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:53 smithi090 ceph-mon[44434]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:05:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:53 smithi090 ceph-mon[44434]: osdmap e104: 12 total, 12 up, 12 in 2024-05-15T23:05:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:53 smithi088 ceph-mon[44699]: pgmap v340: 17 pgs: 4 creating+peering, 2 active+clean, 11 unknown; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:53 smithi088 ceph-mon[44699]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:05:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:53 smithi088 ceph-mon[44699]: osdmap e104: 12 total, 12 up, 12 in 2024-05-15T23:05:54.332 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:05:54.364 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:54.385 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:05:54.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:54.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:05:54.489 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:05:54.518 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:54.540 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:05:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:54.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:05:54.635 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:05:54.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:05:54.677 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:54.698 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:05:54.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:54.701 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:05:54.802 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:05:54.832 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:54.851 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:05:54.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:54.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:05:54.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:05:54.982 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:55.019 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:05:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:55.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:05:55.125 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:05:55.157 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:55.178 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:05:55.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:55.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:05:55.282 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:05:55.309 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:55.346 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:05:55.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:55.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:05:55.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:05:55.483 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:55.504 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:05:55.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:55.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:05:55.603 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:55.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:05:55.636 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T23:05:55.637 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:55.658 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:05:55.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:55.661 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:05:55.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:05:55.789 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:55.866 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:05:55.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:55.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:05:55.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:05:56.001 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:56.021 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:05:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:56.024 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:05:56.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:55 smithi084 ceph-mon[39784]: pgmap v342: 17 pgs: 4 creating+peering, 8 active+clean, 5 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:56.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:05:56.152 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:56.172 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:05:56.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:56.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:05:56.183 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:05:56.202 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:05:56.221 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:05:56.239 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:05:56.260 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:05:56.275 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:05:56.279 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:05:56.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:55 smithi090 ceph-mon[44434]: pgmap v342: 17 pgs: 4 creating+peering, 8 active+clean, 5 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:56.302 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:05:56.304 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:55 smithi088 ceph-mon[44699]: pgmap v342: 17 pgs: 4 creating+peering, 8 active+clean, 5 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:56.307 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:56.323 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:05:56.329 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:05:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:56.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:05:56.343 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:05:56.363 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:05:56.384 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:05:56.405 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:05:56.406 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:56.407 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:05:56.407 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:05:56.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:05:56.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:05:56.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:05:56.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:05:56.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:05:56.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:05:56.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:05:56.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:05:56.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:05:56.408 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:56.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:05:56.462 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:56.481 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:05:56.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:56.484 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:05:56.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:05:56.613 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:56.631 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:05:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:56.634 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:05:56.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:05:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:56.779 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:05:56.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:56.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:05:56.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:05:56.909 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:56.927 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:05:56.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:56.930 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:05:57.030 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:05:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1916545044' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:05:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.057 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1916545044' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:05:57.058 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.058 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.058 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.058 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.058 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.058 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.058 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.058 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.058 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.060 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:57.112 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:05:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:57.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:05:57.216 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:05:57.243 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:57.261 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:05:57.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:57.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:05:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1916545044' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:05:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:57.350 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:05:57.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:05:57.392 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:57.410 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:05:57.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:57.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:05:57.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:05:57.539 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:57.557 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:05:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:57.560 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:05:57.661 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:05:57.693 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:57.714 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:05:57.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:57.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:05:57.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:05:57.844 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:57.862 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:05:57.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:57.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:05:57.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:05:57.992 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:58.011 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:05:58.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:58.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:05:58.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:58.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:58.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:58.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:57 smithi088 ceph-mon[44699]: pgmap v343: 17 pgs: 2 creating+peering, 15 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:58.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:05:58.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:58.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:58.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:58.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:57 smithi090 ceph-mon[44434]: pgmap v343: 17 pgs: 2 creating+peering, 15 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:05:58.159 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:05:58.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:05:58.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:05:58.193 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:58.193 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:58.193 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:05:58.193 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:57 smithi084 ceph-mon[39784]: pgmap v343: 17 pgs: 2 creating+peering, 15 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:05:58.263 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:05:58.291 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:00.156 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:05:59 smithi084 ceph-mon[39784]: pgmap v344: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:06:00.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:05:59 smithi090 ceph-mon[44434]: pgmap v344: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:06:00.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:05:59 smithi088 ceph-mon[44699]: pgmap v344: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:06:00.641 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:06:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:00.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:06:00.747 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:06:00.778 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:00.799 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:06:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:00.802 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:06:00.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:06:00.933 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:00.954 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:06:00.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:00.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:06:00.975 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:06:00.977 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:06:01.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:06:01.088 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:01.186 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:06:01.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:01.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:06:01.289 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:06:01.315 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:01.412 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:06:01.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:01.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:06:01.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:06:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:01.568 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:06:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:01.571 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:06:01.664 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:06:01.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:01.739 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:01.953 INFO:teuthology.orchestra.run.smithi084.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-05-15T23:06:01.954 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:06:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:01 smithi090 ceph-mon[44434]: pgmap v345: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:06:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:01 smithi090 ceph-mon[44434]: from='client.14841 v1:172.21.15.84:0/3364606729' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:06:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:01 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2398411992' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:06:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:01 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/333673598' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:06:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:01 smithi088 ceph-mon[44699]: pgmap v345: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:06:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:01 smithi088 ceph-mon[44699]: from='client.14841 v1:172.21.15.84:0/3364606729' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:06:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:01 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2398411992' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:06:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:01 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/333673598' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:06:02.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:01 smithi084 ceph-mon[39784]: pgmap v345: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:06:02.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:01 smithi084 ceph-mon[39784]: from='client.14841 v1:172.21.15.84:0/3364606729' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:06:02.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:01 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2398411992' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:06:02.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:01 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/333673598' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:06:03.006 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-05-15T23:06:00.605802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337784,"kb_used_data":9344,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124604040,"statfs":{"total":1151940427776,"available":1151594536960,"internally_reserved":0,"allocated":9568256,"data_stored":4991736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":336115002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.678769"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:53.776402+0000","last_change":"2024-05-15T23:05:51.938939+0000","last_active":"2024-05-15T23:05:53.776402+0000","last_peered":"2024-05-15T23:05:53.776402+0000","last_clean":"2024-05-15T23:05:53.776402+0000","last_became_active":"2024-05-15T23:05:51.938654+0000","last_became_peered":"2024-05-15T23:05:51.938654+0000","last_unstale":"2024-05-15T23:05:53.776402+0000","last_undegraded":"2024-05-15T23:05:53.776402+0000","last_fullsized":"2024-05-15T23:05:53.776402+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:17.311580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:53.523574+0000","last_change":"2024-05-15T23:05:51.958003+0000","last_active":"2024-05-15T23:05:53.523574+0000","last_peered":"2024-05-15T23:05:53.523574+0000","last_clean":"2024-05-15T23:05:53.523574+0000","last_became_active":"2024-05-15T23:05:51.957730+0000","last_became_peered":"2024-05-15T23:05:51.957730+0000","last_unstale":"2024-05-15T23:05:53.523574+0000","last_undegraded":"2024-05-15T23:05:53.523574+0000","last_fullsized":"2024-05-15T23:05:53.523574+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:06.105883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:53.775862+0000","last_change":"2024-05-15T23:05:51.932448+0000","last_active":"2024-05-15T23:05:53.775862+0000","last_peered":"2024-05-15T23:05:53.775862+0000","last_clean":"2024-05-15T23:05:53.775862+0000","last_became_active":"2024-05-15T23:05:51.932108+0000","last_became_peered":"2024-05-15T23:05:51.932108+0000","last_unstale":"2024-05-15T23:05:53.775862+0000","last_undegraded":"2024-05-15T23:05:53.775862+0000","last_fullsized":"2024-05-15T23:05:53.775862+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:00.465357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:53.775585+0000","last_change":"2024-05-15T23:05:51.957461+0000","last_active":"2024-05-15T23:05:53.775585+0000","last_peered":"2024-05-15T23:05:53.775585+0000","last_clean":"2024-05-15T23:05:53.775585+0000","last_became_active":"2024-05-15T23:05:51.957162+0000","last_became_peered":"2024-05-15T23:05:51.957162+0000","last_unstale":"2024-05-15T23:05:53.775585+0000","last_undegraded":"2024-05-15T23:05:53.775585+0000","last_fullsized":"2024-05-15T23:05:53.775585+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:15.155943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:53.775877+0000","last_change":"2024-05-15T23:05:51.931869+0000","last_active":"2024-05-15T23:05:53.775877+0000","last_peered":"2024-05-15T23:05:53.775877+0000","last_clean":"2024-05-15T23:05:53.775877+0000","last_became_active":"2024-05-15T23:05:51.931638+0000","last_became_peered":"2024-05-15T23:05:51.931638+0000","last_unstale":"2024-05-15T23:05:53.775877+0000","last_undegraded":"2024-05-15T23:05:53.775877+0000","last_fullsized":"2024-05-15T23:05:53.775877+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:08.928445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":159,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:52.937681+0000","last_change":"2024-05-15T23:05:11.075182+0000","last_active":"2024-05-15T23:05:52.937681+0000","last_peered":"2024-05-15T23:05:52.937681+0000","last_clean":"2024-05-15T23:05:52.937681+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:05:52.937681+0000","last_undegraded":"2024-05-15T23:05:52.937681+0000","last_fullsized":"2024-05-15T23:05:52.937681+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:05:11.075069+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:05:11.075069+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:06:21.551937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:52.937859+0000","last_change":"2024-05-15T23:05:51.958943+0000","last_active":"2024-05-15T23:05:52.937859+0000","last_peered":"2024-05-15T23:05:52.937859+0000","last_clean":"2024-05-15T23:05:52.937859+0000","last_became_active":"2024-05-15T23:05:51.958648+0000","last_became_peered":"2024-05-15T23:05:51.958648+0000","last_unstale":"2024-05-15T23:05:52.937859+0000","last_undegraded":"2024-05-15T23:05:52.937859+0000","last_fullsized":"2024-05-15T23:05:52.937859+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:10.797542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:53.776467+0000","last_change":"2024-05-15T23:05:51.957597+0000","last_active":"2024-05-15T23:05:53.776467+0000","last_peered":"2024-05-15T23:05:53.776467+0000","last_clean":"2024-05-15T23:05:53.776467+0000","last_became_active":"2024-05-15T23:05:51.957327+0000","last_became_peered":"2024-05-15T23:05:51.957327+0000","last_unstale":"2024-05-15T23:05:53.776467+0000","last_undegraded":"2024-05-15T23:05:53.776467+0000","last_fullsized":"2024-05-15T23:05:53.776467+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:09.674536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:53.523542+0000","last_change":"2024-05-15T23:05:51.937160+0000","last_active":"2024-05-15T23:05:53.523542+0000","last_peered":"2024-05-15T23:05:53.523542+0000","last_clean":"2024-05-15T23:05:53.523542+0000","last_became_active":"2024-05-15T23:05:51.936955+0000","last_became_peered":"2024-05-15T23:05:51.936955+0000","last_unstale":"2024-05-15T23:05:53.523542+0000","last_undegraded":"2024-05-15T23:05:53.523542+0000","last_fullsized":"2024-05-15T23:05:53.523542+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:15.588743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:53.775868+0000","last_change":"2024-05-15T23:05:51.958903+0000","last_active":"2024-05-15T23:05:53.775868+0000","last_peered":"2024-05-15T23:05:53.775868+0000","last_clean":"2024-05-15T23:05:53.775868+0000","last_became_active":"2024-05-15T23:05:51.958598+0000","last_became_peered":"2024-05-15T23:05:51.958598+0000","last_unstale":"2024-05-15T23:05:53.775868+0000","last_undegraded":"2024-05-15T23:05:53.775868+0000","last_fullsized":"2024-05-15T23:05:53.775868+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:04.534824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:52.937815+0000","last_change":"2024-05-15T23:05:51.937557+0000","last_active":"2024-05-15T23:05:52.937815+0000","last_peered":"2024-05-15T23:05:52.937815+0000","last_clean":"2024-05-15T23:05:52.937815+0000","last_became_active":"2024-05-15T23:05:51.937345+0000","last_became_peered":"2024-05-15T23:05:51.937345+0000","last_unstale":"2024-05-15T23:05:52.937815+0000","last_undegraded":"2024-05-15T23:05:52.937815+0000","last_fullsized":"2024-05-15T23:05:52.937815+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:05.429963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:53.523539+0000","last_change":"2024-05-15T23:05:51.934544+0000","last_active":"2024-05-15T23:05:53.523539+0000","last_peered":"2024-05-15T23:05:53.523539+0000","last_clean":"2024-05-15T23:05:53.523539+0000","last_became_active":"2024-05-15T23:05:51.934294+0000","last_became_peered":"2024-05-15T23:05:51.934294+0000","last_unstale":"2024-05-15T23:05:53.523539+0000","last_undegraded":"2024-05-15T23:05:53.523539+0000","last_fullsized":"2024-05-15T23:05:53.523539+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:03.844921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:53.776510+0000","last_change":"2024-05-15T23:05:51.957579+0000","last_active":"2024-05-15T23:05:53.776510+0000","last_peered":"2024-05-15T23:05:53.776510+0000","last_clean":"2024-05-15T23:05:53.776510+0000","last_became_active":"2024-05-15T23:05:51.957105+0000","last_became_peered":"2024-05-15T23:05:51.957105+0000","last_unstale":"2024-05-15T23:05:53.776510+0000","last_undegraded":"2024-05-15T23:05:53.776510+0000","last_fullsized":"2024-05-15T23:05:53.776510+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:06:56.584508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:53.776410+0000","last_change":"2024-05-15T23:05:51.937574+0000","last_active":"2024-05-15T23:05:53.776410+0000","last_peered":"2024-05-15T23:05:53.776410+0000","last_clean":"2024-05-15T23:05:53.776410+0000","last_became_active":"2024-05-15T23:05:51.937101+0000","last_became_peered":"2024-05-15T23:05:51.937101+0000","last_unstale":"2024-05-15T23:05:53.776410+0000","last_undegraded":"2024-05-15T23:05:53.776410+0000","last_fullsized":"2024-05-15T23:05:53.776410+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:19.599101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:53.776562+0000","last_change":"2024-05-15T23:05:51.957721+0000","last_active":"2024-05-15T23:05:53.776562+0000","last_peered":"2024-05-15T23:05:53.776562+0000","last_clean":"2024-05-15T23:05:53.776562+0000","last_became_active":"2024-05-15T23:05:51.957426+0000","last_became_peered":"2024-05-15T23:05:51.957426+0000","last_unstale":"2024-05-15T23:05:53.776562+0000","last_undegraded":"2024-05-15T23:05:53.776562+0000","last_fullsized":"2024-05-15T23:05:53.776562+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:05.379245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:53.776368+0000","last_change":"2024-05-15T23:05:51.937507+0000","last_active":"2024-05-15T23:05:53.776368+0000","last_peered":"2024-05-15T23:05:53.776368+0000","last_clean":"2024-05-15T23:05:53.776368+0000","last_became_active":"2024-05-15T23:05:51.937178+0000","last_became_peered":"2024-05-15T23:05:51.937178+0000","last_unstale":"2024-05-15T23:05:53.776368+0000","last_undegraded":"2024-05-15T23:05:53.776368+0000","last_fullsized":"2024-05-15T23:05:53.776368+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:06:52.590880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-05-15T23:05:53.775583+0000","last_change":"2024-05-15T23:05:51.957426+0000","last_active":"2024-05-15T23:05:53.775583+0000","last_peered":"2024-05-15T23:05:53.775583+0000","last_clean":"2024-05-15T23:05:53.775583+0000","last_became_active":"2024-05-15T23:05:51.957133+0000","last_became_peered":"2024-05-15T23:05:51.957133+0000","last_unstale":"2024-05-15T23:05:53.775583+0000","last_undegraded":"2024-05-15T23:05:53.775583+0000","last_fullsized":"2024-05-15T23:05:53.775583+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:06:58.279767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351002,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28232,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716920,"statfs":{"total":95995035648,"available":95966126080,"internally_reserved":0,"allocated":1241088,"data_stored":856733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579936,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":708608,"data_stored":327825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776167,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717440,"statfs":{"total":95995035648,"available":95966658560,"internally_reserved":0,"allocated":708608,"data_stored":327825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005100,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":327825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266736,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28236,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716916,"statfs":{"total":95995035648,"available":95966121984,"internally_reserved":0,"allocated":1241088,"data_stored":856753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":327825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":708608,"data_stored":327825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31876,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713276,"statfs":{"total":95995035648,"available":95962394624,"internally_reserved":0,"allocated":708608,"data_stored":327825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345994,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717440,"statfs":{"total":95995035648,"available":95966658560,"internally_reserved":0,"allocated":708608,"data_stored":327825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717440,"statfs":{"total":95995035648,"available":95966658560,"internally_reserved":0,"allocated":708608,"data_stored":327825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986239,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":327825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790009,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717440,"statfs":{"total":95995035648,"available":95966658560,"internally_reserved":0,"allocated":708608,"data_stored":327825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:06:03.007 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [0, 9] 2024-05-15T23:06:03.007 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.7', '0', '9'] 2024-05-15T23:06:03.008 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pg-upmap 2.7 0 9 2024-05-15T23:06:03.208 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:02 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:06:03.208 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:02 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:06:03.208 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:02 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:06:03.208 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2398411992' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:06:03.209 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/333673598' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:06:03.209 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:02 smithi084 ceph-mon[39784]: osdmap e105: 12 total, 12 up, 12 in 2024-05-15T23:06:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:02 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:06:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:02 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:06:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:02 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:06:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2398411992' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:06:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/333673598' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:06:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:02 smithi090 ceph-mon[44434]: osdmap e105: 12 total, 12 up, 12 in 2024-05-15T23:06:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:02 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:06:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:02 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:06:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:02 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:06:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2398411992' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:06:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/333673598' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:06:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:02 smithi088 ceph-mon[44699]: osdmap e105: 12 total, 12 up, 12 in 2024-05-15T23:06:03.496 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:03.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:03.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:03.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:06:03.625 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:03.873 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:06:03.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:03.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:06:03.977 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:06:04.009 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:04.030 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:06:04.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:04.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:06:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:03 smithi084 ceph-mon[39784]: pgmap v347: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:06:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:03 smithi084 ceph-mon[39784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:06:04.134 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:04.161 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:03 smithi090 ceph-mon[44434]: pgmap v347: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:06:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:03 smithi090 ceph-mon[44434]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:06:04.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:03 smithi088 ceph-mon[44699]: pgmap v347: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:06:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:03 smithi088 ceph-mon[44699]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:06:04.517 DEBUG:teuthology.orchestra.run.smithi090:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-05-15T23:06:04.520 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T23:06:04.520 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T23:06:04.748 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:04.749 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:04.791 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:06:04.792 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:06:04.834 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:hints = 1 2024-05-15T23:06:04.834 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T23:06:04.834 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object prefix: benchmark_data_smithi090_78160 2024-05-15T23:06:04.835 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:06:04.835 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 0 0 0 0 0 0 - 0 2024-05-15T23:06:05.010 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:05.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:05.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:05.116 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:05.150 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:05.175 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:05.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:05.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:06:05.324 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:05.835 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 1 16 6040 6024 376.415 376.5 0.00224097 0.00265103 2024-05-15T23:06:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:05 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:06:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:05 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:06:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:05 smithi084 ceph-mon[39784]: pgmap v348: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:06:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:05 smithi084 ceph-mon[39784]: osdmap e106: 12 total, 12 up, 12 in 2024-05-15T23:06:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:05 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:06:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:05 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:06:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:05 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:06:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:05 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:06:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:05 smithi090 ceph-mon[44434]: pgmap v348: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:06:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:05 smithi090 ceph-mon[44434]: osdmap e106: 12 total, 12 up, 12 in 2024-05-15T23:06:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:05 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:06:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:05 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:06:06.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:05 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:06:06.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:05 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:06:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:05 smithi088 ceph-mon[44699]: pgmap v348: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:06:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:05 smithi088 ceph-mon[44699]: osdmap e106: 12 total, 12 up, 12 in 2024-05-15T23:06:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:05 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:06:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:05 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:06:06.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:06:06.835 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 2 16 11398 11382 355.633 334.875 0.00230611 0.00280836 2024-05-15T23:06:07.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:06 smithi090 ceph-mon[44434]: osdmap e107: 12 total, 12 up, 12 in 2024-05-15T23:06:07.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:06 smithi088 ceph-mon[44699]: osdmap e107: 12 total, 12 up, 12 in 2024-05-15T23:06:07.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:06 smithi084 ceph-mon[39784]: osdmap e107: 12 total, 12 up, 12 in 2024-05-15T23:06:07.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T23:06:07.835 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 3 16 16145 16129 335.974 296.688 0.00379391 0.00297349 2024-05-15T23:06:07.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:07 smithi084 ceph-mon[39784]: pgmap v351: 17 pgs: 17 active+clean; 106 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 281 op/s 2024-05-15T23:06:07.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:07 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1812541189' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.7", "id": [0, 9]}]: dispatch 2024-05-15T23:06:07.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:07 smithi084 ceph-mon[39784]: from='client.34748 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.7", "id": [0, 9]}]: dispatch 2024-05-15T23:06:07.997 INFO:teuthology.orchestra.run.smithi084.stderr:set 2.7 pg_upmap mapping to [0,9] 2024-05-15T23:06:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:07 smithi090 ceph-mon[44434]: pgmap v351: 17 pgs: 17 active+clean; 106 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 281 op/s 2024-05-15T23:06:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:07 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1812541189' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.7", "id": [0, 9]}]: dispatch 2024-05-15T23:06:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:07 smithi090 ceph-mon[44434]: from='client.34748 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.7", "id": [0, 9]}]: dispatch 2024-05-15T23:06:08.316 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:07 smithi088 ceph-mon[44699]: pgmap v351: 17 pgs: 17 active+clean; 106 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 281 op/s 2024-05-15T23:06:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:07 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1812541189' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.7", "id": [0, 9]}]: dispatch 2024-05-15T23:06:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:07 smithi088 ceph-mon[44699]: from='client.34748 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.7", "id": [0, 9]}]: dispatch 2024-05-15T23:06:08.835 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 4 16 17021 17005 265.667 54.75 0.00166065 0.00299278 2024-05-15T23:06:09.021 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:06:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:09.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:06:09.041 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:08 smithi084 ceph-mon[39784]: from='client.34748 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.7", "id": [0, 9]}]': finished 2024-05-15T23:06:09.041 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:08 smithi084 ceph-mon[39784]: osdmap e108: 12 total, 12 up, 12 in 2024-05-15T23:06:09.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:06:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:09.177 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:06:09.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:09.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:06:09.281 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:06:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:08 smithi090 ceph-mon[44434]: from='client.34748 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.7", "id": [0, 9]}]': finished 2024-05-15T23:06:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:08 smithi090 ceph-mon[44434]: osdmap e108: 12 total, 12 up, 12 in 2024-05-15T23:06:09.309 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:09.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:08 smithi088 ceph-mon[44699]: from='client.34748 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.7", "id": [0, 9]}]': finished 2024-05-15T23:06:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:08 smithi088 ceph-mon[44699]: osdmap e108: 12 total, 12 up, 12 in 2024-05-15T23:06:09.507 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:06:09.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:09.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:06:09.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:06:09.643 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:09.663 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:06:09.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:09.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:06:09.767 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:06:09.795 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:09.815 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:06:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:09.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:06:09.835 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 5 16 17021 17005 212.535 0 - 0.00299278 2024-05-15T23:06:09.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:06:09.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:06:09.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:06:09.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:06:09.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:06:09.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:06:09.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:06:09.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:06:09.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:06:09.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:06:09.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:06:09.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:06:09.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:06:09.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:06:09.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:06:09.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:06:09.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:06:09.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:06:09.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:06:09.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:06:09.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:06:09.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:06:09.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:06:09.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:06:09.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:06:09.945 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:10.247 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:10 smithi084 ceph-mon[39784]: pgmap v353: 17 pgs: 17 active+clean; 253 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 672 op/s 2024-05-15T23:06:10.247 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:10 smithi084 ceph-mon[39784]: osdmap e109: 12 total, 12 up, 12 in 2024-05-15T23:06:10.247 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:10 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3621786342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:06:10.247 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.247 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.247 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.247 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.247 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.247 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.247 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.247 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.247 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.248 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:09 smithi090 ceph-mon[44434]: pgmap v353: 17 pgs: 17 active+clean; 253 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 672 op/s 2024-05-15T23:06:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:09 smithi090 ceph-mon[44434]: osdmap e109: 12 total, 12 up, 12 in 2024-05-15T23:06:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:09 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3621786342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:06:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:09 smithi088 ceph-mon[44699]: pgmap v353: 17 pgs: 17 active+clean; 253 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 672 op/s 2024-05-15T23:06:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:09 smithi088 ceph-mon[44699]: osdmap e109: 12 total, 12 up, 12 in 2024-05-15T23:06:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:09 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3621786342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:06:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:10.594 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:06:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:10.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:06:10.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:06:10.746 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:10.777 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:06:10.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:10.781 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:06:10.835 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 6 16 19904 19888 207.142 90.0938 0.00347817 0.00482554 2024-05-15T23:06:10.882 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:06:10.915 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:10.946 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:06:10.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:10.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:06:11.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:06:11.076 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:11.088 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:11.113 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:06:11.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:11.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:06:11.218 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:06:11.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:11.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:11.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:11 smithi084 ceph-mon[39784]: osdmap e110: 12 total, 12 up, 12 in 2024-05-15T23:06:11.265 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:11.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:11.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:11.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:11 smithi090 ceph-mon[44434]: osdmap e110: 12 total, 12 up, 12 in 2024-05-15T23:06:11.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:11.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:11.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:11 smithi088 ceph-mon[44699]: osdmap e110: 12 total, 12 up, 12 in 2024-05-15T23:06:11.367 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:06:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:11.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:06:11.540 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:06:11.586 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:11.697 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:06:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:11.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:06:11.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:11.835 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 7 15 23722 23707 211.645 238.688 0.00797228 0.00472234 2024-05-15T23:06:11.851 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:12.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:12 smithi084 ceph-mon[39784]: pgmap v356: 17 pgs: 17 active+clean; 966 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 201 MiB/s wr, 3.22k op/s 2024-05-15T23:06:12.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:12 smithi084 ceph-mon[39784]: osdmap e111: 12 total, 12 up, 12 in 2024-05-15T23:06:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:12 smithi090 ceph-mon[44434]: pgmap v356: 17 pgs: 17 active+clean; 966 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 201 MiB/s wr, 3.22k op/s 2024-05-15T23:06:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:12 smithi090 ceph-mon[44434]: osdmap e111: 12 total, 12 up, 12 in 2024-05-15T23:06:12.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:12 smithi088 ceph-mon[44699]: pgmap v356: 17 pgs: 17 active+clean; 966 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 201 MiB/s wr, 3.22k op/s 2024-05-15T23:06:12.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:12 smithi088 ceph-mon[44699]: osdmap e111: 12 total, 12 up, 12 in 2024-05-15T23:06:12.607 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:06:12.835 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 8 16 27331 27315 213.375 225.5 0.00308489 0.00468389 2024-05-15T23:06:12.911 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:12.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:13.021 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:06:13.035 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:06:13.054 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:13.082 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:06:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:13.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:06:13.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1947652777' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:06:13.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:13 smithi084 ceph-mon[39784]: from='client.34760 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:06:13.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:13.227 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1947652777' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:06:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:13 smithi090 ceph-mon[44434]: from='client.34760 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:06:13.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1947652777' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:06:13.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:13 smithi088 ceph-mon[44699]: from='client.34760 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:06:13.387 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:13.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:13.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:13.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:06:13.540 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:06:13.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:13.575 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:06:13.676 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:06:13.719 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:13.826 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:06:13.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:13.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:06:13.836 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 9 16 30642 30626 212.658 206.938 0.00247284 0.0047004 2024-05-15T23:06:13.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:13.981 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:14.045 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:14.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:14.092 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:14.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:14 smithi084 ceph-mon[39784]: pgmap v358: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 1.2 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 206 MiB/s wr, 3.29k op/s; 1376/38546 objects misplaced (3.570%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:06:14.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:14 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:06:14.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:14 smithi084 ceph-mon[39784]: from='client.34760 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:06:14.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:14 smithi084 ceph-mon[39784]: osdmap e112: 12 total, 12 up, 12 in 2024-05-15T23:06:14.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:06:14.161 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:06:14.161 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-05-15T23:06:14.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-05-15T23:06:14.161 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:06:14.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.161 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:17745 2.7 e110/109 2:ee4f9a09:::benchmark_data_smithi090_78160_object17744:head v 110'1066, mlcod=105'21)", 2024-05-15T23:06:14.161 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:10.309448+0000", 2024-05-15T23:06:14.161 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 3.8478168909999999, 2024-05-15T23:06:14.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0090073889999999993, 2024-05-15T23:06:14.161 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.162 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.162 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.162 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:10.309448+0000", 2024-05-15T23:06:14.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:10.309448+0000", 2024-05-15T23:06:14.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:10.309449+0000", 2024-05-15T23:06:14.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3099999999999996e-07 2024-05-15T23:06:14.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:10.309559+0000", 2024-05-15T23:06:14.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010966700000000001 2024-05-15T23:06:14.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:10.309565+0000", 2024-05-15T23:06:14.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9889999999999997e-06 2024-05-15T23:06:14.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:10.309569+0000", 2024-05-15T23:06:14.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0300000000000004e-06 2024-05-15T23:06:14.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:10.309911+0000", 2024-05-15T23:06:14.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00034236299999999999 2024-05-15T23:06:14.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:10.309942+0000", 2024-05-15T23:06:14.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0743e-05 2024-05-15T23:06:14.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:10.318413+0000", 2024-05-15T23:06:14.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084716229999999993 2024-05-15T23:06:14.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:10.318455+0000", 2024-05-15T23:06:14.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2042999999999997e-05 2024-05-15T23:06:14.165 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.165 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.165 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.166 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:23327 2.5 e111/102 2:a71dd5a5:::benchmark_data_smithi090_78160_object23326:head v 111'1450, mlcod=111'1447)", 2024-05-15T23:06:14.166 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:11.728018+0000", 2024-05-15T23:06:14.166 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.429247352, 2024-05-15T23:06:14.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0090466350000000008, 2024-05-15T23:06:14.166 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.166 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.166 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.166 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.728018+0000", 2024-05-15T23:06:14.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.728018+0000", 2024-05-15T23:06:14.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.728019+0000", 2024-05-15T23:06:14.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.378e-06 2024-05-15T23:06:14.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.728085+0000", 2024-05-15T23:06:14.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6441999999999998e-05 2024-05-15T23:06:14.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.728093+0000", 2024-05-15T23:06:14.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0469999999999994e-06 2024-05-15T23:06:14.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.728100+0000", 2024-05-15T23:06:14.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5089999999999999e-06 2024-05-15T23:06:14.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.728153+0000", 2024-05-15T23:06:14.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2664000000000003e-05 2024-05-15T23:06:14.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.728170+0000", 2024-05-15T23:06:14.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7278e-05 2024-05-15T23:06:14.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.737024+0000", 2024-05-15T23:06:14.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0088541479999999992 2024-05-15T23:06:14.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.737064+0000", 2024-05-15T23:06:14.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0169000000000001e-05 2024-05-15T23:06:14.170 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.170 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.170 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.170 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:24040 2.c e111/102 2:3d1e6085:::benchmark_data_smithi090_78160_object24039:head v 111'1499, mlcod=111'1497)", 2024-05-15T23:06:14.170 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:11.930427+0000", 2024-05-15T23:06:14.170 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.2268377529999999, 2024-05-15T23:06:14.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084451449999999994, 2024-05-15T23:06:14.170 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.170 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.170 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.171 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.930427+0000", 2024-05-15T23:06:14.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.930427+0000", 2024-05-15T23:06:14.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.930429+0000", 2024-05-15T23:06:14.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.66e-06 2024-05-15T23:06:14.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.930477+0000", 2024-05-15T23:06:14.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8365999999999997e-05 2024-05-15T23:06:14.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.930488+0000", 2024-05-15T23:06:14.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0311e-05 2024-05-15T23:06:14.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.930496+0000", 2024-05-15T23:06:14.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8680000000000005e-06 2024-05-15T23:06:14.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.930533+0000", 2024-05-15T23:06:14.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7042999999999998e-05 2024-05-15T23:06:14.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.930546+0000", 2024-05-15T23:06:14.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2914999999999999e-05 2024-05-15T23:06:14.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.938816+0000", 2024-05-15T23:06:14.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082700340000000008 2024-05-15T23:06:14.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.938872+0000", 2024-05-15T23:06:14.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5948e-05 2024-05-15T23:06:14.174 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.174 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.174 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.174 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:24064 2.c e111/102 2:3e414657:::benchmark_data_smithi090_78160_object24063:head v 111'1501, mlcod=111'1497)", 2024-05-15T23:06:14.174 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:11.941874+0000", 2024-05-15T23:06:14.175 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.2153907340000001, 2024-05-15T23:06:14.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0088426600000000005, 2024-05-15T23:06:14.175 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.175 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.175 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.175 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.941874+0000", 2024-05-15T23:06:14.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.941874+0000", 2024-05-15T23:06:14.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.941877+0000", 2024-05-15T23:06:14.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.328e-06 2024-05-15T23:06:14.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.941918+0000", 2024-05-15T23:06:14.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1891000000000003e-05 2024-05-15T23:06:14.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.941934+0000", 2024-05-15T23:06:14.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5130000000000001e-05 2024-05-15T23:06:14.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.941943+0000", 2024-05-15T23:06:14.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1290000000000002e-06 2024-05-15T23:06:14.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942147+0000", 2024-05-15T23:06:14.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00020403399999999999 2024-05-15T23:06:14.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942165+0000", 2024-05-15T23:06:14.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8703999999999999e-05 2024-05-15T23:06:14.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.950676+0000", 2024-05-15T23:06:14.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085106969999999994 2024-05-15T23:06:14.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.950717+0000", 2024-05-15T23:06:14.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0747000000000001e-05 2024-05-15T23:06:14.178 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.178 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.179 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.179 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:24069 2.7 e111/109 2:e612aaae:::benchmark_data_smithi090_78160_object24068:head v 111'1468, mlcod=106'239)", 2024-05-15T23:06:14.179 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:11.942003+0000", 2024-05-15T23:06:14.179 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.2152618070000001, 2024-05-15T23:06:14.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087234630000000007, 2024-05-15T23:06:14.179 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.179 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.179 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.179 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942003+0000", 2024-05-15T23:06:14.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942003+0000", 2024-05-15T23:06:14.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942004+0000", 2024-05-15T23:06:14.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.47e-07 2024-05-15T23:06:14.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942045+0000", 2024-05-15T23:06:14.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0630000000000002e-05 2024-05-15T23:06:14.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942054+0000", 2024-05-15T23:06:14.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3940000000000002e-06 2024-05-15T23:06:14.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942062+0000", 2024-05-15T23:06:14.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8949999999999998e-06 2024-05-15T23:06:14.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942142+0000", 2024-05-15T23:06:14.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9754999999999998e-05 2024-05-15T23:06:14.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942171+0000", 2024-05-15T23:06:14.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9189000000000001e-05 2024-05-15T23:06:14.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.950680+0000", 2024-05-15T23:06:14.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085089829999999995 2024-05-15T23:06:14.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.950727+0000", 2024-05-15T23:06:14.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6669999999999999e-05 2024-05-15T23:06:14.183 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.183 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.183 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.183 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:24073 2.7 e111/109 2:eaa1dcc7:::benchmark_data_smithi090_78160_object24072:head v 111'1469, mlcod=106'239)", 2024-05-15T23:06:14.183 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:11.942077+0000", 2024-05-15T23:06:14.183 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.2151882500000002, 2024-05-15T23:06:14.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086946650000000007, 2024-05-15T23:06:14.183 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.183 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.183 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.183 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942077+0000", 2024-05-15T23:06:14.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942077+0000", 2024-05-15T23:06:14.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942077+0000", 2024-05-15T23:06:14.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2099999999999996e-07 2024-05-15T23:06:14.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942109+0000", 2024-05-15T23:06:14.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1404e-05 2024-05-15T23:06:14.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942134+0000", 2024-05-15T23:06:14.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5579e-05 2024-05-15T23:06:14.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942144+0000", 2024-05-15T23:06:14.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.285e-06 2024-05-15T23:06:14.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.943374+0000", 2024-05-15T23:06:14.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001230655 2024-05-15T23:06:14.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.943424+0000", 2024-05-15T23:06:14.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9270000000000001e-05 2024-05-15T23:06:14.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.950731+0000", 2024-05-15T23:06:14.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073072930000000003 2024-05-15T23:06:14.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.950771+0000", 2024-05-15T23:06:14.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0457999999999998e-05 2024-05-15T23:06:14.187 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.187 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.187 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.187 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:24065 2.4 e111/102 2:2b0647ed:::benchmark_data_smithi090_78160_object24064:head v 111'1436, mlcod=111'1433)", 2024-05-15T23:06:14.187 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:11.942180+0000", 2024-05-15T23:06:14.187 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.2150848480000001, 2024-05-15T23:06:14.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085719360000000005, 2024-05-15T23:06:14.188 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.188 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.188 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.188 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942180+0000", 2024-05-15T23:06:14.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942180+0000", 2024-05-15T23:06:14.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942182+0000", 2024-05-15T23:06:14.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5069999999999999e-06 2024-05-15T23:06:14.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942215+0000", 2024-05-15T23:06:14.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3330000000000001e-05 2024-05-15T23:06:14.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942225+0000", 2024-05-15T23:06:14.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0334e-05 2024-05-15T23:06:14.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942231+0000", 2024-05-15T23:06:14.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8660000000000002e-06 2024-05-15T23:06:14.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.943061+0000", 2024-05-15T23:06:14.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00083009000000000002 2024-05-15T23:06:14.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.943083+0000", 2024-05-15T23:06:14.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1506e-05 2024-05-15T23:06:14.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.950721+0000", 2024-05-15T23:06:14.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0076378699999999997 2024-05-15T23:06:14.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.950752+0000", 2024-05-15T23:06:14.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1433000000000002e-05 2024-05-15T23:06:14.191 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.191 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.191 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.192 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:24081 2.5 e111/102 2:a12af455:::benchmark_data_smithi090_78160_object24080:head v 111'1496, mlcod=111'1494)", 2024-05-15T23:06:14.192 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:11.942833+0000", 2024-05-15T23:06:14.192 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.214432022, 2024-05-15T23:06:14.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086035250000000008, 2024-05-15T23:06:14.192 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.192 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.192 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.192 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942833+0000", 2024-05-15T23:06:14.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942833+0000", 2024-05-15T23:06:14.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942835+0000", 2024-05-15T23:06:14.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.823e-06 2024-05-15T23:06:14.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942874+0000", 2024-05-15T23:06:14.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9638999999999998e-05 2024-05-15T23:06:14.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942886+0000", 2024-05-15T23:06:14.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1919000000000001e-05 2024-05-15T23:06:14.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942894+0000", 2024-05-15T23:06:14.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3980000000000002e-06 2024-05-15T23:06:14.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942955+0000", 2024-05-15T23:06:14.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1470000000000006e-05 2024-05-15T23:06:14.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.942969+0000", 2024-05-15T23:06:14.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3805000000000001e-05 2024-05-15T23:06:14.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.951404+0000", 2024-05-15T23:06:14.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084352899999999998 2024-05-15T23:06:14.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:11.951436+0000", 2024-05-15T23:06:14.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2181000000000002e-05 2024-05-15T23:06:14.196 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.196 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.196 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.196 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:25652 2.5 e111/102 2:a8b759fb:::benchmark_data_smithi090_78160_object25651:head v 111'1594, mlcod=111'1590)", 2024-05-15T23:06:14.196 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:12.376682+0000", 2024-05-15T23:06:14.196 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.780583179, 2024-05-15T23:06:14.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011362809999999999, 2024-05-15T23:06:14.197 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.197 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.197 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.197 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.376682+0000", 2024-05-15T23:06:14.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.376682+0000", 2024-05-15T23:06:14.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.376683+0000", 2024-05-15T23:06:14.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1900000000000001e-07 2024-05-15T23:06:14.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.376718+0000", 2024-05-15T23:06:14.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5701000000000003e-05 2024-05-15T23:06:14.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.376725+0000", 2024-05-15T23:06:14.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8050000000000001e-06 2024-05-15T23:06:14.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.376730+0000", 2024-05-15T23:06:14.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2399999999999998e-06 2024-05-15T23:06:14.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.377769+0000", 2024-05-15T23:06:14.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001038574 2024-05-15T23:06:14.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.377789+0000", 2024-05-15T23:06:14.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9715999999999999e-05 2024-05-15T23:06:14.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.388002+0000", 2024-05-15T23:06:14.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010212987999999999 2024-05-15T23:06:14.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.388045+0000", 2024-05-15T23:06:14.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2867000000000001e-05 2024-05-15T23:06:14.200 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.200 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.200 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.201 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:25665 2.5 e111/102 2:a76b644c:::benchmark_data_smithi090_78160_object25664:head v 111'1595, mlcod=111'1592)", 2024-05-15T23:06:14.201 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:12.379852+0000", 2024-05-15T23:06:14.201 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.777412623, 2024-05-15T23:06:14.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082206369999999994, 2024-05-15T23:06:14.201 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.201 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.201 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.201 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.379852+0000", 2024-05-15T23:06:14.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.379852+0000", 2024-05-15T23:06:14.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.379853+0000", 2024-05-15T23:06:14.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.19e-06 2024-05-15T23:06:14.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.379898+0000", 2024-05-15T23:06:14.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4462000000000002e-05 2024-05-15T23:06:14.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.379913+0000", 2024-05-15T23:06:14.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5268e-05 2024-05-15T23:06:14.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.379920+0000", 2024-05-15T23:06:14.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1019999999999999e-06 2024-05-15T23:06:14.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.381186+0000", 2024-05-15T23:06:14.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001265354 2024-05-15T23:06:14.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.381199+0000", 2024-05-15T23:06:14.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3723000000000001e-05 2024-05-15T23:06:14.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.388048+0000", 2024-05-15T23:06:14.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068481369999999998 2024-05-15T23:06:14.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.388073+0000", 2024-05-15T23:06:14.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5401e-05 2024-05-15T23:06:14.204 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.205 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.205 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.205 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:26080 2.7 e111/109 2:e2c9cfa0:::benchmark_data_smithi090_78160_object26079:head v 111'1593, mlcod=106'317)", 2024-05-15T23:06:14.205 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:12.494768+0000", 2024-05-15T23:06:14.205 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.66249687, 2024-05-15T23:06:14.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081520559999999995, 2024-05-15T23:06:14.205 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.205 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.205 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.205 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.494768+0000", 2024-05-15T23:06:14.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.494768+0000", 2024-05-15T23:06:14.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.494769+0000", 2024-05-15T23:06:14.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2699999999999999e-06 2024-05-15T23:06:14.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.494844+0000", 2024-05-15T23:06:14.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4732000000000002e-05 2024-05-15T23:06:14.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.494851+0000", 2024-05-15T23:06:14.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8820000000000003e-06 2024-05-15T23:06:14.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.494856+0000", 2024-05-15T23:06:14.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7369999999999997e-06 2024-05-15T23:06:14.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.495829+0000", 2024-05-15T23:06:14.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00097303400000000001 2024-05-15T23:06:14.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.495851+0000", 2024-05-15T23:06:14.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2450999999999999e-05 2024-05-15T23:06:14.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.502877+0000", 2024-05-15T23:06:14.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070261389999999998 2024-05-15T23:06:14.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.502920+0000", 2024-05-15T23:06:14.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2811000000000001e-05 2024-05-15T23:06:14.209 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.209 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.209 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.209 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:26801 2.7 e111/109 2:e54bb403:::benchmark_data_smithi090_78160_object26800:head v 111'1644, mlcod=106'346)", 2024-05-15T23:06:14.209 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:12.688899+0000", 2024-05-15T23:06:14.209 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.468366286, 2024-05-15T23:06:14.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083637150000000007, 2024-05-15T23:06:14.209 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.209 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.209 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.210 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.688899+0000", 2024-05-15T23:06:14.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.688899+0000", 2024-05-15T23:06:14.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.688900+0000", 2024-05-15T23:06:14.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.029e-06 2024-05-15T23:06:14.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.688929+0000", 2024-05-15T23:06:14.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9247999999999998e-05 2024-05-15T23:06:14.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.688941+0000", 2024-05-15T23:06:14.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2094e-05 2024-05-15T23:06:14.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.688947+0000", 2024-05-15T23:06:14.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1940000000000003e-06 2024-05-15T23:06:14.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.690456+0000", 2024-05-15T23:06:14.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001508498 2024-05-15T23:06:14.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.690478+0000", 2024-05-15T23:06:14.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2243e-05 2024-05-15T23:06:14.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.697224+0000", 2024-05-15T23:06:14.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067464179999999997 2024-05-15T23:06:14.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.697262+0000", 2024-05-15T23:06:14.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7991000000000003e-05 2024-05-15T23:06:14.213 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.213 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.213 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.213 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:27468 2.c e111/102 2:348b7103:::benchmark_data_smithi090_78160_object27467:head v 111'1709, mlcod=111'1705)", 2024-05-15T23:06:14.213 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:12.881420+0000", 2024-05-15T23:06:14.213 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.2758453240000001, 2024-05-15T23:06:14.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0097014360000000008, 2024-05-15T23:06:14.214 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.214 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.214 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.214 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.881420+0000", 2024-05-15T23:06:14.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.881420+0000", 2024-05-15T23:06:14.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.881421+0000", 2024-05-15T23:06:14.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7129999999999999e-06 2024-05-15T23:06:14.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.881459+0000", 2024-05-15T23:06:14.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7552999999999998e-05 2024-05-15T23:06:14.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.881475+0000", 2024-05-15T23:06:14.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6152000000000001e-05 2024-05-15T23:06:14.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.881483+0000", 2024-05-15T23:06:14.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7880000000000007e-06 2024-05-15T23:06:14.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.881608+0000", 2024-05-15T23:06:14.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012490099999999999 2024-05-15T23:06:14.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.881625+0000", 2024-05-15T23:06:14.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7146000000000002e-05 2024-05-15T23:06:14.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.891068+0000", 2024-05-15T23:06:14.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094428670000000006 2024-05-15T23:06:14.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.891121+0000", 2024-05-15T23:06:14.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3316000000000002e-05 2024-05-15T23:06:14.217 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.217 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.217 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.218 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:27643 2.c e111/102 2:3a208c38:::benchmark_data_smithi090_78160_object27642:head v 111'1711, mlcod=111'1709)", 2024-05-15T23:06:14.218 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:12.947774+0000", 2024-05-15T23:06:14.218 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.209491216, 2024-05-15T23:06:14.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081102180000000006, 2024-05-15T23:06:14.218 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.218 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.218 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.218 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.947774+0000", 2024-05-15T23:06:14.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.947774+0000", 2024-05-15T23:06:14.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.947776+0000", 2024-05-15T23:06:14.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4719999999999998e-06 2024-05-15T23:06:14.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.947860+0000", 2024-05-15T23:06:14.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3961999999999993e-05 2024-05-15T23:06:14.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.947872+0000", 2024-05-15T23:06:14.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1639e-05 2024-05-15T23:06:14.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.947880+0000", 2024-05-15T23:06:14.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3669999999999997e-06 2024-05-15T23:06:14.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.947939+0000", 2024-05-15T23:06:14.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8867999999999999e-05 2024-05-15T23:06:14.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.947959+0000", 2024-05-15T23:06:14.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9553999999999999e-05 2024-05-15T23:06:14.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.955841+0000", 2024-05-15T23:06:14.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0078822909999999996 2024-05-15T23:06:14.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:12.955884+0000", 2024-05-15T23:06:14.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3065000000000002e-05 2024-05-15T23:06:14.222 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.222 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.222 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.222 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:28499 2.5 e112/102 2:a06fa7b2:::benchmark_data_smithi090_78160_object28498:head v 112'1765, mlcod=112'1760)", 2024-05-15T23:06:14.222 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:13.211384+0000", 2024-05-15T23:06:14.222 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.94588113100000004, 2024-05-15T23:06:14.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0091564850000000007, 2024-05-15T23:06:14.222 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.222 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.222 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.222 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.211384+0000", 2024-05-15T23:06:14.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.211384+0000", 2024-05-15T23:06:14.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.211385+0000", 2024-05-15T23:06:14.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.33e-07 2024-05-15T23:06:14.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.211439+0000", 2024-05-15T23:06:14.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3838999999999997e-05 2024-05-15T23:06:14.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.211450+0000", 2024-05-15T23:06:14.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.113e-05 2024-05-15T23:06:14.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.211457+0000", 2024-05-15T23:06:14.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9369999999999998e-06 2024-05-15T23:06:14.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.212910+0000", 2024-05-15T23:06:14.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00145309 2024-05-15T23:06:14.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.212931+0000", 2024-05-15T23:06:14.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0939999999999999e-05 2024-05-15T23:06:14.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.220507+0000", 2024-05-15T23:06:14.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0075762570000000003 2024-05-15T23:06:14.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.220540+0000", 2024-05-15T23:06:14.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3358999999999997e-05 2024-05-15T23:06:14.226 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.226 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.226 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.226 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:28828 2.4 e112/102 2:2ea3ca7f:::benchmark_data_smithi090_78160_object28827:head v 112'1711, mlcod=112'1707)", 2024-05-15T23:06:14.226 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:13.308713+0000", 2024-05-15T23:06:14.226 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.84855179000000003, 2024-05-15T23:06:14.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.008485388, 2024-05-15T23:06:14.226 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.226 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.226 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.227 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.308713+0000", 2024-05-15T23:06:14.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.308713+0000", 2024-05-15T23:06:14.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.308714+0000", 2024-05-15T23:06:14.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1209999999999999e-06 2024-05-15T23:06:14.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.308733+0000", 2024-05-15T23:06:14.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8953999999999998e-05 2024-05-15T23:06:14.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.308745+0000", 2024-05-15T23:06:14.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1792e-05 2024-05-15T23:06:14.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.308752+0000", 2024-05-15T23:06:14.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3409999999999997e-06 2024-05-15T23:06:14.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.308808+0000", 2024-05-15T23:06:14.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5266999999999998e-05 2024-05-15T23:06:14.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.308821+0000", 2024-05-15T23:06:14.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3494e-05 2024-05-15T23:06:14.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.317122+0000", 2024-05-15T23:06:14.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083009339999999994 2024-05-15T23:06:14.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.317199+0000", 2024-05-15T23:06:14.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6484999999999997e-05 2024-05-15T23:06:14.230 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.230 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.230 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.230 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:28836 2.7 e112/109 2:e68d511b:::benchmark_data_smithi090_78160_object28835:head v 112'1761, mlcod=107'443)", 2024-05-15T23:06:14.230 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:13.309540+0000", 2024-05-15T23:06:14.230 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.84772484100000001, 2024-05-15T23:06:14.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0093476830000000007, 2024-05-15T23:06:14.231 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.231 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.231 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.231 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.309540+0000", 2024-05-15T23:06:14.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.309540+0000", 2024-05-15T23:06:14.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.309541+0000", 2024-05-15T23:06:14.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.26e-06 2024-05-15T23:06:14.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.309655+0000", 2024-05-15T23:06:14.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113396 2024-05-15T23:06:14.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.309669+0000", 2024-05-15T23:06:14.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4756999999999999e-05 2024-05-15T23:06:14.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.309676+0000", 2024-05-15T23:06:14.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5339999999999996e-06 2024-05-15T23:06:14.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.309721+0000", 2024-05-15T23:06:14.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5136000000000003e-05 2024-05-15T23:06:14.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.309741+0000", 2024-05-15T23:06:14.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0114000000000001e-05 2024-05-15T23:06:14.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.318845+0000", 2024-05-15T23:06:14.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0091039610000000007 2024-05-15T23:06:14.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.318888+0000", 2024-05-15T23:06:14.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2524999999999997e-05 2024-05-15T23:06:14.234 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.234 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.234 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.235 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:30600 2.4 e112/102 2:200760f7:::benchmark_data_smithi090_78160_object30599:head v 112'1828, mlcod=112'1822)", 2024-05-15T23:06:14.235 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:13.821512+0000", 2024-05-15T23:06:14.235 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.33575250499999998, 2024-05-15T23:06:14.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081043049999999991, 2024-05-15T23:06:14.235 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.235 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.235 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.235 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.821512+0000", 2024-05-15T23:06:14.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.821512+0000", 2024-05-15T23:06:14.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.821514+0000", 2024-05-15T23:06:14.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4720000000000001e-06 2024-05-15T23:06:14.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.821540+0000", 2024-05-15T23:06:14.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5689000000000001e-05 2024-05-15T23:06:14.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.821549+0000", 2024-05-15T23:06:14.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.679e-06 2024-05-15T23:06:14.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.821557+0000", 2024-05-15T23:06:14.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6890000000000004e-06 2024-05-15T23:06:14.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.821605+0000", 2024-05-15T23:06:14.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8155000000000002e-05 2024-05-15T23:06:14.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.821623+0000", 2024-05-15T23:06:14.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8354000000000001e-05 2024-05-15T23:06:14.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.829585+0000", 2024-05-15T23:06:14.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0079617290000000007 2024-05-15T23:06:14.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.829617+0000", 2024-05-15T23:06:14.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1538000000000002e-05 2024-05-15T23:06:14.238 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.239 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.239 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.239 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:31039 2.c e112/102 2:3fdf4c7f:::benchmark_data_smithi090_78160_object31038:head v 112'1936, mlcod=112'1933)", 2024-05-15T23:06:14.239 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:13.949871+0000", 2024-05-15T23:06:14.239 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.20739390899999999, 2024-05-15T23:06:14.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084008039999999996, 2024-05-15T23:06:14.239 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.239 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.239 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.239 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.949871+0000", 2024-05-15T23:06:14.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.949871+0000", 2024-05-15T23:06:14.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.949872+0000", 2024-05-15T23:06:14.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.376e-06 2024-05-15T23:06:14.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.949940+0000", 2024-05-15T23:06:14.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7249000000000006e-05 2024-05-15T23:06:14.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.949950+0000", 2024-05-15T23:06:14.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0659000000000001e-05 2024-05-15T23:06:14.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.949958+0000", 2024-05-15T23:06:14.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9799999999999998e-06 2024-05-15T23:06:14.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.950020+0000", 2024-05-15T23:06:14.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1704000000000004e-05 2024-05-15T23:06:14.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.950042+0000", 2024-05-15T23:06:14.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1897000000000001e-05 2024-05-15T23:06:14.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.958237+0000", 2024-05-15T23:06:14.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081952480000000005 2024-05-15T23:06:14.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:13.958272+0000", 2024-05-15T23:06:14.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4690999999999997e-05 2024-05-15T23:06:14.243 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.243 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.243 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.243 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:31264 2.7 e112/109 2:e16b7384:::benchmark_data_smithi090_78160_object31263:head v 112'1903, mlcod=107'562)", 2024-05-15T23:06:14.243 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:14.033469+0000", 2024-05-15T23:06:14.243 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.12379562199999999, 2024-05-15T23:06:14.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082114639999999999, 2024-05-15T23:06:14.243 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:14.243 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:14.244 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:14.244 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:14.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:14.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:14.033469+0000", 2024-05-15T23:06:14.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:14.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:14.033469+0000", 2024-05-15T23:06:14.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:14.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:14.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:14.033471+0000", 2024-05-15T23:06:14.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5540000000000001e-06 2024-05-15T23:06:14.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:14.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:14.033636+0000", 2024-05-15T23:06:14.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016537299999999999 2024-05-15T23:06:14.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:14.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:14.033644+0000", 2024-05-15T23:06:14.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6240000000000002e-06 2024-05-15T23:06:14.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:14.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:14.033650+0000", 2024-05-15T23:06:14.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5050000000000004e-06 2024-05-15T23:06:14.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:14.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:14.033731+0000", 2024-05-15T23:06:14.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0228999999999994e-05 2024-05-15T23:06:14.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:14.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:14.033757+0000", 2024-05-15T23:06:14.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6707e-05 2024-05-15T23:06:14.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:14.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:14.041627+0000", 2024-05-15T23:06:14.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0078701740000000006 2024-05-15T23:06:14.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:14.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:14.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:14.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:14.041681+0000", 2024-05-15T23:06:14.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3297999999999997e-05 2024-05-15T23:06:14.247 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.247 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.247 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.247 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:14.247 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:14.247 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:06:14.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:14.249 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:14.249 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:06:14.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:06:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:14 smithi090 ceph-mon[44434]: pgmap v358: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 1.2 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 206 MiB/s wr, 3.29k op/s; 1376/38546 objects misplaced (3.570%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:06:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:14 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:06:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:14 smithi090 ceph-mon[44434]: from='client.34760 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:06:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:14 smithi090 ceph-mon[44434]: osdmap e112: 12 total, 12 up, 12 in 2024-05-15T23:06:14.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:14 smithi088 ceph-mon[44699]: pgmap v358: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 1.2 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 206 MiB/s wr, 3.29k op/s; 1376/38546 objects misplaced (3.570%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:06:14.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:14 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:06:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:14 smithi088 ceph-mon[44699]: from='client.34760 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:06:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:14 smithi088 ceph-mon[44699]: osdmap e112: 12 total, 12 up, 12 in 2024-05-15T23:06:14.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:06:14.390 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:14.414 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:06:14.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:14.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:06:14.522 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:14.574 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:14.654 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:14.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:14.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:14.759 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:06:14.805 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:14.835 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 10 16 33775 33759 210.972 195.812 0.00880421 0.0047376 2024-05-15T23:06:15.548 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T23:06:15.836 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 11 15 37006 36991 210.155 202 0.00673435 0.00475661 2024-05-15T23:06:15.862 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:06:15.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:15.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:06:15.874 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:06:15.967 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:06:16.020 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:16.098 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:06:16.098 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:06:16.098 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-05-15T23:06:16.098 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T23:06:16.098 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T23:06:16.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 config set filestore_inject_stall 3 2024-05-15T23:06:16.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:16 smithi084 ceph-mon[39784]: pgmap v360: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 234 MiB/s wr, 3.75k op/s; 1376/50196 objects misplaced (2.741%); 3.5 MiB/s, 56 objects/s recovering 2024-05-15T23:06:16.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:16 smithi090 ceph-mon[44434]: pgmap v360: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 234 MiB/s wr, 3.75k op/s; 1376/50196 objects misplaced (2.741%); 3.5 MiB/s, 56 objects/s recovering 2024-05-15T23:06:16.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:16 smithi088 ceph-mon[44699]: pgmap v360: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 234 MiB/s wr, 3.75k op/s; 1376/50196 objects misplaced (2.741%); 3.5 MiB/s, 56 objects/s recovering 2024-05-15T23:06:16.384 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:06:16.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:16.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:06:16.397 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:16.458 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:16.482 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:06:16.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:06:16.538 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:06:16.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:16.710 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:06:16.812 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:16.836 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 12 16 39359 39343 204.89 147 0.00219831 0.00481995 2024-05-15T23:06:16.839 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:17.186 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:06:17.205 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:06:17.223 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:06:17.242 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:06:17.262 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:06:17.281 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:06:17.300 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:06:17.320 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:06:17.339 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:06:17.359 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:06:17.378 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:06:17.398 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:06:17.400 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:06:17.400 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:06:17.400 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:06:17.400 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:06:17.400 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:06:17.487 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:17.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:17.592 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:06:17.621 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:17.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:17 smithi084 ceph-mon[39784]: pgmap v361: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 189 MiB/s wr, 3.02k op/s; 1376/65080 objects misplaced (2.114%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T23:06:17.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:17 smithi084 ceph-mon[39784]: osdmap e113: 12 total, 12 up, 12 in 2024-05-15T23:06:17.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:17 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2605689514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:06:17.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:17.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:17.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:17.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:17.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:17.836 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 13 16 39949 39933 191.966 36.875 0.00701373 0.00520739 2024-05-15T23:06:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:17 smithi090 ceph-mon[44434]: pgmap v361: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 189 MiB/s wr, 3.02k op/s; 1376/65080 objects misplaced (2.114%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T23:06:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:17 smithi090 ceph-mon[44434]: osdmap e113: 12 total, 12 up, 12 in 2024-05-15T23:06:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:17 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2605689514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:06:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:18.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:17 smithi088 ceph-mon[44699]: pgmap v361: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 189 MiB/s wr, 3.02k op/s; 1376/65080 objects misplaced (2.114%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T23:06:18.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:17 smithi088 ceph-mon[44699]: osdmap e113: 12 total, 12 up, 12 in 2024-05-15T23:06:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:17 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2605689514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:06:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:18.678 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:06:18.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:18.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:06:18.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:06:18.829 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:18.836 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 14 16 42577 42561 189.978 164.25 0.00404681 0.00526178 2024-05-15T23:06:19.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:18 smithi090 ceph-mon[44434]: osdmap e114: 12 total, 12 up, 12 in 2024-05-15T23:06:19.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:18 smithi088 ceph-mon[44699]: osdmap e114: 12 total, 12 up, 12 in 2024-05-15T23:06:19.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:19.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:18 smithi084 ceph-mon[39784]: osdmap e114: 12 total, 12 up, 12 in 2024-05-15T23:06:19.829 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:19 smithi084 ceph-mon[39784]: pgmap v364: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 2.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 158 MiB/s wr, 2.52k op/s; 1376/68796 objects misplaced (2.000%) 2024-05-15T23:06:19.837 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 15 15 45030 45015 187.537 153.375 0.00627353 0.00533061 2024-05-15T23:06:20.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:19 smithi090 ceph-mon[44434]: pgmap v364: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 2.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 158 MiB/s wr, 2.52k op/s; 1376/68796 objects misplaced (2.000%) 2024-05-15T23:06:20.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:19 smithi088 ceph-mon[44699]: pgmap v364: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 2.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 158 MiB/s wr, 2.52k op/s; 1376/68796 objects misplaced (2.000%) 2024-05-15T23:06:20.190 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:06:20.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:20.194 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:06:20.295 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:20.312 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:06:20.312 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:06:20.323 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:20.346 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:20.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:20.450 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:20.479 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:20.502 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:20.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:20.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:20.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:06:20.651 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:20.713 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:06:20.837 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 16 16 47462 47446 185.312 151.938 0.00227299 0.00539353 2024-05-15T23:06:21.648 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:06:21.837 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 17 16 50025 50009 183.833 160.188 0.0111282 0.00543792 2024-05-15T23:06:21.942 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:06:21.942 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-15T23:06:21.943 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:06:21.969 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:21 smithi090 ceph-mon[44434]: pgmap v365: 17 pgs: 17 active+clean; 2.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 199 MiB/s wr, 3.18k op/s; 5.7 MiB/s, 90 objects/s recovering 2024-05-15T23:06:21.969 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:21 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1543315905' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:06:21.969 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:21 smithi090 ceph-mon[44434]: from='client.24775 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:06:21.969 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:21 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:06:21.970 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:21 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:06:21.970 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:21 smithi090 ceph-mon[44434]: from='client.24775 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:06:21.970 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:21 smithi090 ceph-mon[44434]: osdmap e115: 12 total, 12 up, 12 in 2024-05-15T23:06:22.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:21 smithi088 ceph-mon[44699]: pgmap v365: 17 pgs: 17 active+clean; 2.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 199 MiB/s wr, 3.18k op/s; 5.7 MiB/s, 90 objects/s recovering 2024-05-15T23:06:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:21 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1543315905' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:06:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:21 smithi088 ceph-mon[44699]: from='client.24775 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:06:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:21 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:06:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:21 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:06:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:21 smithi088 ceph-mon[44699]: from='client.24775 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:06:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:21 smithi088 ceph-mon[44699]: osdmap e115: 12 total, 12 up, 12 in 2024-05-15T23:06:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:21 smithi084 ceph-mon[39784]: pgmap v365: 17 pgs: 17 active+clean; 2.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 199 MiB/s wr, 3.18k op/s; 5.7 MiB/s, 90 objects/s recovering 2024-05-15T23:06:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:21 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1543315905' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:06:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:21 smithi084 ceph-mon[39784]: from='client.24775 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:06:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:21 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:06:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:21 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:06:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:21 smithi084 ceph-mon[39784]: from='client.24775 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:06:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:21 smithi084 ceph-mon[39784]: osdmap e115: 12 total, 12 up, 12 in 2024-05-15T23:06:22.837 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 18 16 52451 52435 182.043 151.625 0.00307742 0.00549139 2024-05-15T23:06:23.837 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 19 16 54958 54942 180.708 156.688 0.00369915 0.00553184 2024-05-15T23:06:24.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:23 smithi090 ceph-mon[44434]: pgmap v367: 17 pgs: 17 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s wr, 2.25k op/s; 7.2 MiB/s, 114 objects/s recovering 2024-05-15T23:06:24.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:23 smithi088 ceph-mon[44699]: pgmap v367: 17 pgs: 17 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s wr, 2.25k op/s; 7.2 MiB/s, 114 objects/s recovering 2024-05-15T23:06:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:23 smithi084 ceph-mon[39784]: pgmap v367: 17 pgs: 17 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s wr, 2.25k op/s; 7.2 MiB/s, 114 objects/s recovering 2024-05-15T23:06:24.837 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:06:24.837137 min lat: 0.000985234 max lat: 2.1501 avg lat: 0.00559802 2024-05-15T23:06:24.837 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:06:24.837 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 20 16 57172 57156 178.591 138.375 0.00597816 0.00559802 2024-05-15T23:06:25.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:24 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:06:25.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:24 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:06:25.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:24 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:06:25.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:24 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:06:25.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:24 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:06:25.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:24 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:06:25.837 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 21 16 59654 59638 177.473 155.125 0.00521105 0.00563303 2024-05-15T23:06:26.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:25 smithi090 ceph-mon[44434]: pgmap v368: 17 pgs: 17 active+clean; 3.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 147 MiB/s wr, 2.35k op/s; 5.4 MiB/s, 86 objects/s recovering 2024-05-15T23:06:26.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:25 smithi088 ceph-mon[44699]: pgmap v368: 17 pgs: 17 active+clean; 3.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 147 MiB/s wr, 2.35k op/s; 5.4 MiB/s, 86 objects/s recovering 2024-05-15T23:06:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:25 smithi084 ceph-mon[39784]: pgmap v368: 17 pgs: 17 active+clean; 3.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 147 MiB/s wr, 2.35k op/s; 5.4 MiB/s, 86 objects/s recovering 2024-05-15T23:06:26.754 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:06:26.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:26.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:06:26.837 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 22 16 62014 61998 176.108 147.5 0.00385225 0.00567689 2024-05-15T23:06:26.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:06:26.904 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:27.265 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:06:27.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:27.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:06:27.287 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T23:06:27.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:27.427 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:27.634 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:27.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:27.659 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:27.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:06:27.739 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:06:27.739 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:06:27.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:06:27.800 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:27.837 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 23 16 64161 64145 174.285 134.188 0.0131913 0.00573565 2024-05-15T23:06:27.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:06:27.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:06:27.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:27 smithi084 ceph-mon[39784]: pgmap v369: 17 pgs: 17 active+clean; 3.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 155 MiB/s wr, 2.48k op/s; 4.8 MiB/s, 76 objects/s recovering 2024-05-15T23:06:27.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:27 smithi084 ceph-mon[39784]: osdmap e116: 12 total, 12 up, 12 in 2024-05-15T23:06:27.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:06:27.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:06:27.938 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:06:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:27.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:06:28.052 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:06:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:06:28.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:06:28.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:27 smithi090 ceph-mon[44434]: pgmap v369: 17 pgs: 17 active+clean; 3.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 155 MiB/s wr, 2.48k op/s; 4.8 MiB/s, 76 objects/s recovering 2024-05-15T23:06:28.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:27 smithi090 ceph-mon[44434]: osdmap e116: 12 total, 12 up, 12 in 2024-05-15T23:06:28.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:06:28.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:06:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:06:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:06:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:27 smithi088 ceph-mon[44699]: pgmap v369: 17 pgs: 17 active+clean; 3.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 155 MiB/s wr, 2.48k op/s; 4.8 MiB/s, 76 objects/s recovering 2024-05-15T23:06:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:27 smithi088 ceph-mon[44699]: osdmap e116: 12 total, 12 up, 12 in 2024-05-15T23:06:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:06:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:06:28.095 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:28.248 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:06:28.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:28.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:06:28.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:06:28.397 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:28.435 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:06:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:28.439 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:06:28.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:06:28.577 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:28.582 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:28.607 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:06:28.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:28.610 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:06:28.711 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:06:28.738 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:28.761 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:06:28.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:28.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:06:28.837 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 24 16 66349 66333 172.721 136.75 0.00363226 0.00578844 2024-05-15T23:06:28.865 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:28.893 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:28.915 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:28.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:28.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:28.978 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:28 smithi084 ceph-mon[39784]: osdmap e117: 12 total, 12 up, 12 in 2024-05-15T23:06:28.978 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:28 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:06:28.978 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:28 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:06:29.019 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:29.044 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:28 smithi088 ceph-mon[44699]: osdmap e117: 12 total, 12 up, 12 in 2024-05-15T23:06:29.044 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:28 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:06:29.044 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:28 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:06:29.047 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:28 smithi090 ceph-mon[44434]: osdmap e117: 12 total, 12 up, 12 in 2024-05-15T23:06:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:28 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:06:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:28 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:06:29.068 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:29.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:29.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:29.172 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:06:29.199 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:29.220 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:06:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:29.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:06:29.325 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:06:29.362 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:29.647 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:06:29.838 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 25 15 68579 68564 171.383 139.438 0.0170949 0.005833 2024-05-15T23:06:29.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:29 smithi084 ceph-mon[39784]: pgmap v372: 17 pgs: 17 active+clean; 3.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.91k op/s 2024-05-15T23:06:29.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:29 smithi084 ceph-mon[39784]: osdmap e118: 12 total, 12 up, 12 in 2024-05-15T23:06:30.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:29 smithi090 ceph-mon[44434]: pgmap v372: 17 pgs: 17 active+clean; 3.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.91k op/s 2024-05-15T23:06:30.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:29 smithi090 ceph-mon[44434]: osdmap e118: 12 total, 12 up, 12 in 2024-05-15T23:06:30.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:29 smithi088 ceph-mon[44699]: pgmap v372: 17 pgs: 17 active+clean; 3.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.91k op/s 2024-05-15T23:06:30.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:29 smithi088 ceph-mon[44699]: osdmap e118: 12 total, 12 up, 12 in 2024-05-15T23:06:30.098 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:30.327 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:06:30.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:30.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:06:30.396 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:06:30.434 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:30.451 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:06:30.485 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:06:30.496 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:30.526 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:06:30.569 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:06:30.608 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:06:30.615 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:30.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:30.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:30.647 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:06:30.687 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:06:30.725 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:06:30.733 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:06:30.752 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:30.765 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:06:30.773 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:06:30.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:30.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:06:30.803 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:06:30.835 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:06:30.839 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 26 16 70585 70569 169.611 125.312 0.00636371 0.0058945 2024-05-15T23:06:30.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:06:30.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:06:30.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:06:30.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:06:30.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:06:30.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:06:30.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:06:30.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:06:30.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:06:30.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:06:30.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:06:30.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:06:30.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:06:30.926 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:30 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2624418881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:06:30.926 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:30.926 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:30.930 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:30.958 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:06:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:30.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:06:31.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:30 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2624418881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:06:31.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.064 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:06:31.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:30 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2624418881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:06:31.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.093 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:31.113 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:06:31.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:31.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:06:31.218 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:31.246 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:31.266 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:31.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:31.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:31.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:06:31.417 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:31.455 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:06:31.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:31.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:06:31.562 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:06:31.621 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:31.730 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.730 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.730 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.730 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.731 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:31 smithi090 ceph-mon[44434]: pgmap v374: 17 pgs: 17 active+clean; 4.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 159 MiB/s wr, 2.54k op/s 2024-05-15T23:06:31.731 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.731 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.731 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.731 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.731 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.731 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.731 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:06:31.731 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:31 smithi090 ceph-mon[44434]: osdmap e119: 12 total, 12 up, 12 in 2024-05-15T23:06:31.839 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 27 15 72469 72454 167.692 117.812 0.00693601 0.00596196 2024-05-15T23:06:31.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:31 smithi084 ceph-mon[39784]: pgmap v374: 17 pgs: 17 active+clean; 4.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 159 MiB/s wr, 2.54k op/s 2024-05-15T23:06:31.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:31.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:06:31.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:31 smithi084 ceph-mon[39784]: osdmap e119: 12 total, 12 up, 12 in 2024-05-15T23:06:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:31 smithi088 ceph-mon[44699]: pgmap v374: 17 pgs: 17 active+clean; 4.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 159 MiB/s wr, 2.54k op/s 2024-05-15T23:06:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:06:32.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:31 smithi088 ceph-mon[44699]: osdmap e119: 12 total, 12 up, 12 in 2024-05-15T23:06:32.404 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:06:32.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:32.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:06:32.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:06:32.516 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:06:32.516 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:06:32.516 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:06:32.517 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:06:32.517 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:06:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:32.581 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:06:32.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:32.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:06:32.692 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:06:32.718 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:32.724 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:06:32.725 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:06:32.740 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:06:32.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:32.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:06:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:06:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:06:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:06:32.839 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 28 16 74503 74487 166.241 127.062 0.00552235 0.00601379 2024-05-15T23:06:32.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:32.911 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:33.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:06:33.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:06:33.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:06:33.071 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:33.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:33.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:33.076 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:06:33.175 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:06:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:33.223 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:06:33.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:33.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:06:33.328 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:33.386 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:33.451 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:33.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:33.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:33.558 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:06:33.610 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:33.668 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:06:33.672 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:06:33.822 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T23:06:33.839 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 29 16 76397 76381 164.59 118.375 0.00703491 0.0060741 2024-05-15T23:06:33.975 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:06:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:33.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:06:33.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:33 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:06:33.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:33 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:06:33.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:33 smithi084 ceph-mon[39784]: pgmap v376: 17 pgs: 17 active+clean; 4.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.12k op/s 2024-05-15T23:06:33.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:33 smithi084 ceph-mon[39784]: osdmap e120: 12 total, 12 up, 12 in 2024-05-15T23:06:34.087 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:34.115 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:34.137 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:34.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:34.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:34.140 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:33 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:06:34.140 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:33 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:06:34.140 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:33 smithi088 ceph-mon[44699]: pgmap v376: 17 pgs: 17 active+clean; 4.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.12k op/s 2024-05-15T23:06:34.141 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:33 smithi088 ceph-mon[44699]: osdmap e120: 12 total, 12 up, 12 in 2024-05-15T23:06:34.241 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:06:34.295 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:34.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:33 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:06:34.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:33 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:06:34.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:33 smithi090 ceph-mon[44434]: pgmap v376: 17 pgs: 17 active+clean; 4.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.12k op/s 2024-05-15T23:06:34.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:33 smithi090 ceph-mon[44434]: osdmap e120: 12 total, 12 up, 12 in 2024-05-15T23:06:34.429 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:06:34.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:34.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:06:34.481 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-05-15T23:06:32.878102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4538040864,"num_objects":69238,"num_object_clones":0,"num_object_copies":138476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69238,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":69359,"num_write_kb":4433254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1010,"num_bytes_recovered":67384928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69330,"ondisk_log_size":69330,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9343288,"kb_used_data":8895584,"kb_used_omap":18,"kb_used_meta":447533,"kb_avail":1115598536,"statfs":{"total":1151940427776,"available":1142372900864,"internally_reserved":0,"allocated":9109078016,"data_stored":9103490542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":458274106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":732626944,"num_objects":11179,"num_object_clones":0,"num_object_copies":22358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11179,"num_write_kb":715456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.181385"},"pg_stats":[{"pgid":"2.f","version":"118'4277","reported_seq":4318,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-05-15T23:06:29.305351+0000","last_change":"2024-05-15T23:05:51.938939+0000","last_active":"2024-05-15T23:06:29.305351+0000","last_peered":"2024-05-15T23:06:29.305351+0000","last_clean":"2024-05-15T23:06:29.305351+0000","last_became_active":"2024-05-15T23:05:51.938654+0000","last_became_peered":"2024-05-15T23:05:51.938654+0000","last_unstale":"2024-05-15T23:06:29.305351+0000","last_undegraded":"2024-05-15T23:06:29.305351+0000","last_fullsized":"2024-05-15T23:06:29.305351+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4277,"log_dups_size":0,"ondisk_log_size":4277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:06.520036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280297472,"num_objects":4277,"num_object_clones":0,"num_object_copies":8554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4277,"num_write_kb":273728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"118'4127","reported_seq":4167,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-05-15T23:06:29.551985+0000","last_change":"2024-05-15T23:05:51.958003+0000","last_active":"2024-05-15T23:06:29.551985+0000","last_peered":"2024-05-15T23:06:29.551985+0000","last_clean":"2024-05-15T23:06:29.551985+0000","last_became_active":"2024-05-15T23:05:51.957730+0000","last_became_peered":"2024-05-15T23:05:51.957730+0000","last_unstale":"2024-05-15T23:06:29.551985+0000","last_undegraded":"2024-05-15T23:06:29.551985+0000","last_fullsized":"2024-05-15T23:06:29.551985+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:06:52.193077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270467072,"num_objects":4127,"num_object_clones":0,"num_object_copies":8254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":264128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"118'4481","reported_seq":4522,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-05-15T23:06:31.550139+0000","last_change":"2024-05-15T23:05:51.932448+0000","last_active":"2024-05-15T23:06:31.550139+0000","last_peered":"2024-05-15T23:06:31.550139+0000","last_clean":"2024-05-15T23:06:31.550139+0000","last_became_active":"2024-05-15T23:05:51.932108+0000","last_became_peered":"2024-05-15T23:05:51.932108+0000","last_unstale":"2024-05-15T23:06:31.550139+0000","last_undegraded":"2024-05-15T23:06:31.550139+0000","last_fullsized":"2024-05-15T23:06:31.550139+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:06:53.199806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293666816,"num_objects":4481,"num_object_clones":0,"num_object_copies":8962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":286784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"118'4261","reported_seq":4302,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-05-15T23:06:30.316989+0000","last_change":"2024-05-15T23:05:51.957461+0000","last_active":"2024-05-15T23:06:30.316989+0000","last_peered":"2024-05-15T23:06:30.316989+0000","last_clean":"2024-05-15T23:06:30.316989+0000","last_became_active":"2024-05-15T23:05:51.957162+0000","last_became_peered":"2024-05-15T23:05:51.957162+0000","last_unstale":"2024-05-15T23:06:30.316989+0000","last_undegraded":"2024-05-15T23:06:30.316989+0000","last_fullsized":"2024-05-15T23:06:30.316989+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4261,"log_dups_size":0,"ondisk_log_size":4261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:06:53.035079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279248896,"num_objects":4261,"num_object_clones":0,"num_object_copies":8522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4261,"num_write_kb":272704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"118'4482","reported_seq":4523,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-05-15T23:06:31.573278+0000","last_change":"2024-05-15T23:05:51.931869+0000","last_active":"2024-05-15T23:06:31.573278+0000","last_peered":"2024-05-15T23:06:31.573278+0000","last_clean":"2024-05-15T23:06:31.573278+0000","last_became_active":"2024-05-15T23:05:51.931638+0000","last_became_peered":"2024-05-15T23:05:51.931638+0000","last_unstale":"2024-05-15T23:06:31.573278+0000","last_undegraded":"2024-05-15T23:06:31.573278+0000","last_fullsized":"2024-05-15T23:06:31.573278+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:15.361777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293732352,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4482,"num_write_kb":286848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":196,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-05-15T23:06:28.709640+0000","last_change":"2024-05-15T23:06:23.753780+0000","last_active":"2024-05-15T23:06:28.709640+0000","last_peered":"2024-05-15T23:06:28.709640+0000","last_clean":"2024-05-15T23:06:28.709640+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:06:28.709640+0000","last_undegraded":"2024-05-15T23:06:28.709640+0000","last_fullsized":"2024-05-15T23:06:28.709640+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:06:23.753652+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:06:23.753652+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:44.178066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"118'4333","reported_seq":4374,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-05-15T23:06:29.810146+0000","last_change":"2024-05-15T23:05:51.958943+0000","last_active":"2024-05-15T23:06:29.810146+0000","last_peered":"2024-05-15T23:06:29.810146+0000","last_clean":"2024-05-15T23:06:29.810146+0000","last_became_active":"2024-05-15T23:05:51.958648+0000","last_became_peered":"2024-05-15T23:05:51.958648+0000","last_unstale":"2024-05-15T23:06:29.810146+0000","last_undegraded":"2024-05-15T23:06:29.810146+0000","last_fullsized":"2024-05-15T23:06:29.810146+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4333,"log_dups_size":0,"ondisk_log_size":4333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:06:53.949011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283967488,"num_objects":4333,"num_object_clones":0,"num_object_copies":8666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4333,"num_write_kb":277312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"118'4223","reported_seq":4264,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-05-15T23:06:29.338169+0000","last_change":"2024-05-15T23:05:51.957597+0000","last_active":"2024-05-15T23:06:29.338169+0000","last_peered":"2024-05-15T23:06:29.338169+0000","last_clean":"2024-05-15T23:06:29.338169+0000","last_became_active":"2024-05-15T23:05:51.957327+0000","last_became_peered":"2024-05-15T23:05:51.957327+0000","last_unstale":"2024-05-15T23:06:29.338169+0000","last_undegraded":"2024-05-15T23:06:29.338169+0000","last_fullsized":"2024-05-15T23:06:29.338169+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4223,"log_dups_size":0,"ondisk_log_size":4223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:06:54.393186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276758528,"num_objects":4223,"num_object_clones":0,"num_object_copies":8446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4223,"num_write_kb":270272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"118'4281","reported_seq":4321,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-05-15T23:06:29.547918+0000","last_change":"2024-05-15T23:05:51.937160+0000","last_active":"2024-05-15T23:06:29.547918+0000","last_peered":"2024-05-15T23:06:29.547918+0000","last_clean":"2024-05-15T23:06:29.547918+0000","last_became_active":"2024-05-15T23:05:51.936955+0000","last_became_peered":"2024-05-15T23:05:51.936955+0000","last_unstale":"2024-05-15T23:06:29.547918+0000","last_undegraded":"2024-05-15T23:06:29.547918+0000","last_fullsized":"2024-05-15T23:06:29.547918+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4281,"log_dups_size":0,"ondisk_log_size":4281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:19.849407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280559616,"num_objects":4281,"num_object_clones":0,"num_object_copies":8562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4281,"num_write_kb":273984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"118'4176","reported_seq":7238,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-05-15T23:06:29.358527+0000","last_change":"2024-05-15T23:06:17.645265+0000","last_active":"2024-05-15T23:06:29.358527+0000","last_peered":"2024-05-15T23:06:29.358527+0000","last_clean":"2024-05-15T23:06:29.358527+0000","last_became_active":"2024-05-15T23:06:17.643653+0000","last_became_peered":"2024-05-15T23:06:17.643653+0000","last_unstale":"2024-05-15T23:06:29.358527+0000","last_undegraded":"2024-05-15T23:06:29.358527+0000","last_fullsized":"2024-05-15T23:06:29.358527+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4176,"log_dups_size":0,"ondisk_log_size":4176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:03.447690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273678336,"num_objects":4176,"num_object_clones":0,"num_object_copies":8352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4176,"num_write_kb":267264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1004,"num_bytes_recovered":65798144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"118'4308","reported_seq":4349,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-05-15T23:06:29.805824+0000","last_change":"2024-05-15T23:05:51.937557+0000","last_active":"2024-05-15T23:06:29.805824+0000","last_peered":"2024-05-15T23:06:29.805824+0000","last_clean":"2024-05-15T23:06:29.805824+0000","last_became_active":"2024-05-15T23:05:51.937345+0000","last_became_peered":"2024-05-15T23:05:51.937345+0000","last_unstale":"2024-05-15T23:06:29.805824+0000","last_undegraded":"2024-05-15T23:06:29.805824+0000","last_fullsized":"2024-05-15T23:06:29.805824+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4308,"log_dups_size":0,"ondisk_log_size":4308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:16.303823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282329088,"num_objects":4308,"num_object_clones":0,"num_object_copies":8616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4308,"num_write_kb":275712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"118'4273","reported_seq":4314,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-05-15T23:06:29.543687+0000","last_change":"2024-05-15T23:05:51.934544+0000","last_active":"2024-05-15T23:06:29.543687+0000","last_peered":"2024-05-15T23:06:29.543687+0000","last_clean":"2024-05-15T23:06:29.543687+0000","last_became_active":"2024-05-15T23:05:51.934294+0000","last_became_peered":"2024-05-15T23:05:51.934294+0000","last_unstale":"2024-05-15T23:06:29.543687+0000","last_undegraded":"2024-05-15T23:06:29.543687+0000","last_fullsized":"2024-05-15T23:06:29.543687+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:06:54.159951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280035328,"num_objects":4273,"num_object_clones":0,"num_object_copies":8546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4273,"num_write_kb":273472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"119'4480","reported_seq":4523,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-05-15T23:06:31.911206+0000","last_change":"2024-05-15T23:05:51.957579+0000","last_active":"2024-05-15T23:06:31.911206+0000","last_peered":"2024-05-15T23:06:31.911206+0000","last_clean":"2024-05-15T23:06:31.911206+0000","last_became_active":"2024-05-15T23:05:51.957105+0000","last_became_peered":"2024-05-15T23:05:51.957105+0000","last_unstale":"2024-05-15T23:06:31.911206+0000","last_undegraded":"2024-05-15T23:06:31.911206+0000","last_fullsized":"2024-05-15T23:06:31.911206+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:05.706392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":4480,"num_object_clones":0,"num_object_copies":8960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4480,"num_write_kb":286720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"118'4328","reported_seq":4369,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-05-15T23:06:29.321067+0000","last_change":"2024-05-15T23:05:51.937574+0000","last_active":"2024-05-15T23:06:29.321067+0000","last_peered":"2024-05-15T23:06:29.321067+0000","last_clean":"2024-05-15T23:06:29.321067+0000","last_became_active":"2024-05-15T23:05:51.937101+0000","last_became_peered":"2024-05-15T23:05:51.937101+0000","last_unstale":"2024-05-15T23:06:29.321067+0000","last_undegraded":"2024-05-15T23:06:29.321067+0000","last_fullsized":"2024-05-15T23:06:29.321067+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:10.287929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283639808,"num_objects":4328,"num_object_clones":0,"num_object_copies":8656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4328,"num_write_kb":276992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"119'4442","reported_seq":4485,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-05-15T23:06:31.910743+0000","last_change":"2024-05-15T23:05:51.957721+0000","last_active":"2024-05-15T23:06:31.910743+0000","last_peered":"2024-05-15T23:06:31.910743+0000","last_clean":"2024-05-15T23:06:31.910743+0000","last_became_active":"2024-05-15T23:05:51.957426+0000","last_became_peered":"2024-05-15T23:05:51.957426+0000","last_unstale":"2024-05-15T23:06:31.910743+0000","last_undegraded":"2024-05-15T23:06:31.910743+0000","last_fullsized":"2024-05-15T23:06:31.910743+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:19.378282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291110912,"num_objects":4442,"num_object_clones":0,"num_object_copies":8884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4442,"num_write_kb":284288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"118'4237","reported_seq":4278,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-05-15T23:06:30.005311+0000","last_change":"2024-05-15T23:05:51.937507+0000","last_active":"2024-05-15T23:06:30.005311+0000","last_peered":"2024-05-15T23:06:30.005311+0000","last_clean":"2024-05-15T23:06:30.005311+0000","last_became_active":"2024-05-15T23:05:51.937178+0000","last_became_peered":"2024-05-15T23:05:51.937178+0000","last_unstale":"2024-05-15T23:06:30.005311+0000","last_undegraded":"2024-05-15T23:06:30.005311+0000","last_fullsized":"2024-05-15T23:06:30.005311+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4237,"log_dups_size":0,"ondisk_log_size":4237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:07:14.419016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277676032,"num_objects":4237,"num_object_clones":0,"num_object_copies":8474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4237,"num_write_kb":271168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"118'4527","reported_seq":4568,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-05-15T23:06:30.298792+0000","last_change":"2024-05-15T23:05:51.957426+0000","last_active":"2024-05-15T23:06:30.298792+0000","last_peered":"2024-05-15T23:06:30.298792+0000","last_clean":"2024-05-15T23:06:30.298792+0000","last_became_active":"2024-05-15T23:05:51.957133+0000","last_became_peered":"2024-05-15T23:05:51.957133+0000","last_unstale":"2024-05-15T23:06:30.298792+0000","last_undegraded":"2024-05-15T23:06:30.298792+0000","last_fullsized":"2024-05-15T23:06:30.298792+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:05:50.918160+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:06:59.946391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296681472,"num_objects":4527,"num_object_clones":0,"num_object_copies":9054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4527,"num_write_kb":289728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4537450496,"num_objects":69236,"num_object_clones":0,"num_object_copies":138472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69236,"num_write_kb":4431104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1004,"num_bytes_recovered":65798144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9220456448,"data_stored":9220456448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69236,"ondisk_log_size":69236,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351009,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895804,"kb_used_data":864624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92849348,"statfs":{"total":95995035648,"available":95077732352,"internally_reserved":0,"allocated":885374976,"data_stored":884900306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579942,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":348588,"kb_used_data":317404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93396564,"statfs":{"total":95995035648,"available":95638081536,"internally_reserved":0,"allocated":325021696,"data_stored":324556671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776173,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1159400,"kb_used_data":1101852,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92585752,"statfs":{"total":95995035648,"available":94807810048,"internally_reserved":0,"allocated":1128296448,"data_stored":1127831423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005106,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1114732,"kb_used_data":1079388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":92630420,"statfs":{"total":95995035648,"available":94853550080,"internally_reserved":0,"allocated":1105293312,"data_stored":1104828287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266742,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":831220,"kb_used_data":804196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92913932,"statfs":{"total":95995035648,"available":95143866368,"internally_reserved":0,"allocated":823496704,"data_stored":823028127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":161068,"kb_used_data":125724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93584084,"statfs":{"total":95995035648,"available":95830102016,"internally_reserved":0,"allocated":128741376,"data_stored":128276351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1116072,"kb_used_data":1080732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":92629080,"statfs":{"total":95995035648,"available":94852177920,"internally_reserved":0,"allocated":1106669568,"data_stored":1106204543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1099628,"kb_used_data":1046236,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92645524,"statfs":{"total":95995035648,"available":94869016576,"internally_reserved":0,"allocated":1071345664,"data_stored":1070880639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346000,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":864420,"kb_used_data":824924,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":92880732,"statfs":{"total":95995035648,"available":95109869568,"internally_reserved":0,"allocated":844722176,"data_stored":844257151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":873572,"kb_used_data":829916,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":92871580,"statfs":{"total":95995035648,"available":95100497920,"internally_reserved":0,"allocated":849833984,"data_stored":849368959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":44693959},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986245,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":286044,"kb_used_data":259024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93459108,"statfs":{"total":95995035648,"available":95702126592,"internally_reserved":0,"allocated":265240576,"data_stored":264781574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790016,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":592740,"kb_used_data":561564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93152412,"statfs":{"total":95995035648,"available":95388069888,"internally_reserved":0,"allocated":575041536,"data_stored":574576511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1089404928,"data_stored":1089404928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1107361792,"data_stored":1107361792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":860225536,"data_stored":860225536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":863240192,"data_stored":863240192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":272236544,"data_stored":272236544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":587530240,"data_stored":587530240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":125894656,"data_stored":125894656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":848035840,"data_stored":848035840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1119420416,"data_stored":1119420416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1128726528,"data_stored":1128726528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":328269824,"data_stored":328269824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":890109952,"data_stored":890109952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:06:34.533 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:34.560 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:34.579 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:34.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:34.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:34.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:06:34.729 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:06:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:34.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:06:34.837 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:34.839 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 30 15 78387 78372 163.251 124.438 0.0103006 0.00612319 2024-05-15T23:06:34.859 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:06:34.886 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:34.906 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:06:34.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:34.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:06:35.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:06:35.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:34 smithi090 ceph-mon[44434]: from='client.34781 v1:172.21.15.84:0/1350956438' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:06:35.063 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:35.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:34 smithi084 ceph-mon[39784]: from='client.34781 v1:172.21.15.84:0/1350956438' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:06:35.145 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:06:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:35.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:06:35.151 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:35.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:06:35.288 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:35.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:34 smithi088 ceph-mon[44699]: from='client.34781 v1:172.21.15.84:0/1350956438' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:06:35.321 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:06:35.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:35.324 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:06:35.425 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:06:35.478 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:35.839 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 31 16 80244 80228 161.727 116 0.00285187 0.00618167 2024-05-15T23:06:36.045 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:06:36.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:36.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:06:36.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:35 smithi084 ceph-mon[39784]: pgmap v378: 17 pgs: 17 active+clean; 4.5 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 161 MiB/s wr, 2.58k op/s 2024-05-15T23:06:36.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:35 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:06:36.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:35 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:06:36.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:35 smithi084 ceph-mon[39784]: osdmap e121: 12 total, 12 up, 12 in 2024-05-15T23:06:36.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:06:36.181 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:35 smithi090 ceph-mon[44434]: pgmap v378: 17 pgs: 17 active+clean; 4.5 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 161 MiB/s wr, 2.58k op/s 2024-05-15T23:06:36.181 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:35 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:06:36.181 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:35 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:06:36.181 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:35 smithi090 ceph-mon[44434]: osdmap e121: 12 total, 12 up, 12 in 2024-05-15T23:06:36.186 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:06:36.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:36.218 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:06:36.318 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:06:36.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:35 smithi088 ceph-mon[44699]: pgmap v378: 17 pgs: 17 active+clean; 4.5 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 161 MiB/s wr, 2.58k op/s 2024-05-15T23:06:36.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:35 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:06:36.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:35 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:06:36.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:35 smithi088 ceph-mon[44699]: osdmap e121: 12 total, 12 up, 12 in 2024-05-15T23:06:36.351 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:36.372 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:06:36.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:36.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:06:36.476 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:06:36.503 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:36.525 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:06:36.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:36.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:06:36.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:36.671 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:36.839 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 32 15 82244 82229 160.581 125.062 0.00587443 0.00622589 2024-05-15T23:06:36.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:06:37.070 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:37.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:37.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:06:37.216 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:37.240 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:06:37.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:37.245 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:06:37.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:06:37.393 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:37.483 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:06:37.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:37.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:06:37.588 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:06:37.618 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:37.641 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:06:37.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:37.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:06:37.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:06:37.801 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:37.841 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 33 16 84037 84021 159.108 112 0.0103148 0.00628333 2024-05-15T23:06:38.075 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:06:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:38.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:06:38.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:37 smithi084 ceph-mon[39784]: pgmap v380: 17 pgs: 17 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 117 MiB/s wr, 1.87k op/s 2024-05-15T23:06:38.150 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:06:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-05-15T23:06:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-05-15T23:06:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:06:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-05-15T23:06:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T22:59:40.510986+0000", 2024-05-15T23:06:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 417.63507846499999, 2024-05-15T23:06:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.54870574400000005, 2024-05-15T23:06:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14158", 2024-05-15T23:06:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.84:0/3897905141", 2024-05-15T23:06:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2024-05-15T23:06:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T22:59:40.510986+0000", 2024-05-15T23:06:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T22:59:40.510986+0000", 2024-05-15T23:06:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T22:59:40.510988+0000", 2024-05-15T23:06:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3960000000000001e-06 2024-05-15T23:06:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T22:59:40.510995+0000", 2024-05-15T23:06:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.793e-06 2024-05-15T23:06:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T22:59:40.511003+0000", 2024-05-15T23:06:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9729999999999997e-06 2024-05-15T23:06:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T22:59:40.511011+0000", 2024-05-15T23:06:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3180000000000007e-06 2024-05-15T23:06:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T22:59:40.511061+0000", 2024-05-15T23:06:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0229000000000003e-05 2024-05-15T23:06:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:06:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T22:59:40.511073+0000", 2024-05-15T23:06:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1446e-05 2024-05-15T23:06:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T22:59:41.058747+0000", 2024-05-15T23:06:38.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.54767401999999998 2024-05-15T23:06:38.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T22:59:41.058892+0000", 2024-05-15T23:06:38.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014515 2024-05-15T23:06:38.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T22:59:41.059014+0000", 2024-05-15T23:06:38.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121822 2024-05-15T23:06:38.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T22:59:41.059449+0000", 2024-05-15T23:06:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00043490400000000001 2024-05-15T23:06:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T22:59:41.059611+0000", 2024-05-15T23:06:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000162662 2024-05-15T23:06:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T22:59:41.059650+0000", 2024-05-15T23:06:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8924000000000002e-05 2024-05-15T23:06:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T22:59:41.059692+0000", 2024-05-15T23:06:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2107000000000002e-05 2024-05-15T23:06:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:69975 2.4 2:2b66d61e:::benchmark_data_smithi090_78160_object69974:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e118)", 2024-05-15T23:06:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:30.517988+0000", 2024-05-15T23:06:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 7.6280765490000002, 2024-05-15T23:06:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.040077412999999999, 2024-05-15T23:06:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 69975 2024-05-15T23:06:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.517988+0000", 2024-05-15T23:06:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.517988+0000", 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.517989+0000", 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0900000000000002e-07 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.518008+0000", 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8819000000000001e-05 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.518012+0000", 2024-05-15T23:06:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1849999999999997e-06 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.518017+0000", 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7190000000000001e-06 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.524437+0000", 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0064209419999999998 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.524543+0000", 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105308 2024-05-15T23:06:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.524663+0000", 2024-05-15T23:06:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120395 2024-05-15T23:06:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.528827+0000", 2024-05-15T23:06:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041637439999999996 2024-05-15T23:06:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.557963+0000", 2024-05-15T23:06:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.029135768999999999 2024-05-15T23:06:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.558022+0000", 2024-05-15T23:06:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9713000000000002e-05 2024-05-15T23:06:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.558066+0000", 2024-05-15T23:06:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3409999999999999e-05 2024-05-15T23:06:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:70305 2.4 2:20b83bfa:::benchmark_data_smithi090_78160_object70304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e118)", 2024-05-15T23:06:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:30.690553+0000", 2024-05-15T23:06:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 7.4555118399999998, 2024-05-15T23:06:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037120350000000003, 2024-05-15T23:06:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 70305 2024-05-15T23:06:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.690553+0000", 2024-05-15T23:06:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.690553+0000", 2024-05-15T23:06:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.690556+0000", 2024-05-15T23:06:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.757e-06 2024-05-15T23:06:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.690604+0000", 2024-05-15T23:06:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8180000000000003e-05 2024-05-15T23:06:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.690619+0000", 2024-05-15T23:06:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5051e-05 2024-05-15T23:06:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.690627+0000", 2024-05-15T23:06:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1990000000000006e-06 2024-05-15T23:06:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.693113+0000", 2024-05-15T23:06:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024856129999999998 2024-05-15T23:06:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.693238+0000", 2024-05-15T23:06:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000124609 2024-05-15T23:06:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.693385+0000", 2024-05-15T23:06:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014792900000000001 2024-05-15T23:06:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.696139+0000", 2024-05-15T23:06:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027539919999999998 2024-05-15T23:06:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.727562+0000", 2024-05-15T23:06:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.031422659999999998 2024-05-15T23:06:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.727621+0000", 2024-05-15T23:06:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8647999999999998e-05 2024-05-15T23:06:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.727673+0000", 2024-05-15T23:06:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2711999999999999e-05 2024-05-15T23:06:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:70841 2.4 2:2b5d3496:::benchmark_data_smithi090_78160_object70840:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e118)", 2024-05-15T23:06:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:30.970438+0000", 2024-05-15T23:06:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 7.1756265969999999, 2024-05-15T23:06:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.02425598, 2024-05-15T23:06:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 70841 2024-05-15T23:06:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.970438+0000", 2024-05-15T23:06:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.970438+0000", 2024-05-15T23:06:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.970439+0000", 2024-05-15T23:06:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.569999999999999e-07 2024-05-15T23:06:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.971976+0000", 2024-05-15T23:06:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001537093 2024-05-15T23:06:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.971989+0000", 2024-05-15T23:06:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.252e-05 2024-05-15T23:06:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.971999+0000", 2024-05-15T23:06:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9000000000000001e-06 2024-05-15T23:06:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.973028+0000", 2024-05-15T23:06:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010296050000000001 2024-05-15T23:06:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.973150+0000", 2024-05-15T23:06:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012133000000000001 2024-05-15T23:06:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.973272+0000", 2024-05-15T23:06:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012267499999999999 2024-05-15T23:06:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.983155+0000", 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009882778 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.994594+0000", 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01143925 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.994650+0000", 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5755999999999999e-05 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:30.994694+0000", 2024-05-15T23:06:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4116000000000002e-05 2024-05-15T23:06:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:72107 2.4 2:2bccb0cf:::benchmark_data_smithi090_78160_object72106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e119)", 2024-05-15T23:06:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:31.640959+0000", 2024-05-15T23:06:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 6.5051057989999999, 2024-05-15T23:06:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032608629, 2024-05-15T23:06:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 72107 2024-05-15T23:06:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:31.640959+0000", 2024-05-15T23:06:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:31.640959+0000", 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:31.640960+0000", 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6199999999999997e-07 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:31.641036+0000", 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6625999999999994e-05 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:31.641041+0000", 2024-05-15T23:06:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5689999999999998e-06 2024-05-15T23:06:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:31.641045+0000", 2024-05-15T23:06:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.58e-06 2024-05-15T23:06:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:31.643291+0000", 2024-05-15T23:06:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002246718 2024-05-15T23:06:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:31.643421+0000", 2024-05-15T23:06:38.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012932300000000001 2024-05-15T23:06:38.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:31.643566+0000", 2024-05-15T23:06:38.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145737 2024-05-15T23:06:38.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:31.649379+0000", 2024-05-15T23:06:38.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058126849999999997 2024-05-15T23:06:38.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:31.673460+0000", 2024-05-15T23:06:38.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024081169999999999 2024-05-15T23:06:38.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:31.673518+0000", 2024-05-15T23:06:38.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8046e-05 2024-05-15T23:06:38.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:31.673568+0000", 2024-05-15T23:06:38.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9512999999999998e-05 2024-05-15T23:06:38.179 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.179 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.180 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.180 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:72834 2.4 2:2a88d7c0:::benchmark_data_smithi090_78160_object72833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e119)", 2024-05-15T23:06:38.180 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:32.022589+0000", 2024-05-15T23:06:38.180 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 6.1234763860000001, 2024-05-15T23:06:38.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025935359000000002, 2024-05-15T23:06:38.180 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.180 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.180 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.180 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.180 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.180 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.180 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 72834 2024-05-15T23:06:38.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.181 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.022589+0000", 2024-05-15T23:06:38.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.022589+0000", 2024-05-15T23:06:38.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.022591+0000", 2024-05-15T23:06:38.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1299999999999999e-06 2024-05-15T23:06:38.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.022630+0000", 2024-05-15T23:06:38.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9697999999999998e-05 2024-05-15T23:06:38.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.022641+0000", 2024-05-15T23:06:38.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0635999999999999e-05 2024-05-15T23:06:38.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.022649+0000", 2024-05-15T23:06:38.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1100000000000003e-06 2024-05-15T23:06:38.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.022777+0000", 2024-05-15T23:06:38.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000127906 2024-05-15T23:06:38.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.022914+0000", 2024-05-15T23:06:38.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013683 2024-05-15T23:06:38.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.023058+0000", 2024-05-15T23:06:38.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014376999999999999 2024-05-15T23:06:38.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.025565+0000", 2024-05-15T23:06:38.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002507446 2024-05-15T23:06:38.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.048423+0000", 2024-05-15T23:06:38.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022858074999999999 2024-05-15T23:06:38.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.048477+0000", 2024-05-15T23:06:38.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3646000000000001e-05 2024-05-15T23:06:38.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.048524+0000", 2024-05-15T23:06:38.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7111999999999999e-05 2024-05-15T23:06:38.185 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.185 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.185 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.186 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:72836 2.4 2:22d7ac42:::benchmark_data_smithi090_78160_object72835:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e119)", 2024-05-15T23:06:38.186 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:32.025427+0000", 2024-05-15T23:06:38.186 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 6.1206379829999999, 2024-05-15T23:06:38.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026942478999999998, 2024-05-15T23:06:38.186 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.186 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.186 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.186 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.186 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.186 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.186 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 72836 2024-05-15T23:06:38.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.186 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.025427+0000", 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.025427+0000", 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.025429+0000", 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9290000000000001e-06 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.025549+0000", 2024-05-15T23:06:38.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120098 2024-05-15T23:06:38.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.025559+0000", 2024-05-15T23:06:38.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8379999999999997e-06 2024-05-15T23:06:38.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.025566+0000", 2024-05-15T23:06:38.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.661e-06 2024-05-15T23:06:38.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.025607+0000", 2024-05-15T23:06:38.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0735999999999998e-05 2024-05-15T23:06:38.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.025691+0000", 2024-05-15T23:06:38.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3862000000000004e-05 2024-05-15T23:06:38.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.025786+0000", 2024-05-15T23:06:38.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5061000000000003e-05 2024-05-15T23:06:38.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.029159+0000", 2024-05-15T23:06:38.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003372805 2024-05-15T23:06:38.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.052299+0000", 2024-05-15T23:06:38.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023140528 2024-05-15T23:06:38.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.052333+0000", 2024-05-15T23:06:38.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3775000000000001e-05 2024-05-15T23:06:38.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:32.052369+0000", 2024-05-15T23:06:38.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6186000000000002e-05 2024-05-15T23:06:38.191 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.191 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.191 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.191 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:75477 2.4 2:2732dccc:::benchmark_data_smithi090_78160_object75476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-05-15T23:06:38.191 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:33.326706+0000", 2024-05-15T23:06:38.191 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 4.8193588710000004, 2024-05-15T23:06:38.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027970464, 2024-05-15T23:06:38.191 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.192 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.192 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.192 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.192 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.192 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.192 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 75477 2024-05-15T23:06:38.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.192 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.326706+0000", 2024-05-15T23:06:38.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.326706+0000", 2024-05-15T23:06:38.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.326709+0000", 2024-05-15T23:06:38.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4770000000000002e-06 2024-05-15T23:06:38.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.326764+0000", 2024-05-15T23:06:38.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4951000000000003e-05 2024-05-15T23:06:38.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.326775+0000", 2024-05-15T23:06:38.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1600000000000001e-05 2024-05-15T23:06:38.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.326784+0000", 2024-05-15T23:06:38.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2790000000000004e-06 2024-05-15T23:06:38.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.327322+0000", 2024-05-15T23:06:38.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00053792399999999995 2024-05-15T23:06:38.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.327490+0000", 2024-05-15T23:06:38.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016810699999999999 2024-05-15T23:06:38.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.327646+0000", 2024-05-15T23:06:38.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000155706 2024-05-15T23:06:38.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.331887+0000", 2024-05-15T23:06:38.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042404180000000001 2024-05-15T23:06:38.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.354620+0000", 2024-05-15T23:06:38.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022733550000000002 2024-05-15T23:06:38.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.354652+0000", 2024-05-15T23:06:38.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2079000000000002e-05 2024-05-15T23:06:38.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.354677+0000", 2024-05-15T23:06:38.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4372999999999999e-05 2024-05-15T23:06:38.197 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.197 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.197 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.197 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:75977 2.6 2:6a3637f6:::benchmark_data_smithi090_78160_object75976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-05-15T23:06:38.197 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:33.610070+0000", 2024-05-15T23:06:38.197 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 4.5359952220000004, 2024-05-15T23:06:38.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026672267999999999, 2024-05-15T23:06:38.197 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.197 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.197 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.197 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.197 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.198 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.198 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 75977 2024-05-15T23:06:38.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.198 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.610070+0000", 2024-05-15T23:06:38.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.610070+0000", 2024-05-15T23:06:38.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.610070+0000", 2024-05-15T23:06:38.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5299999999999999e-07 2024-05-15T23:06:38.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.610110+0000", 2024-05-15T23:06:38.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9746000000000002e-05 2024-05-15T23:06:38.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.610114+0000", 2024-05-15T23:06:38.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4059999999999998e-06 2024-05-15T23:06:38.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.610118+0000", 2024-05-15T23:06:38.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5889999999999999e-06 2024-05-15T23:06:38.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.620211+0000", 2024-05-15T23:06:38.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010092800000000001 2024-05-15T23:06:38.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.620321+0000", 2024-05-15T23:06:38.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011014399999999999 2024-05-15T23:06:38.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 8", 2024-05-15T23:06:38.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.620465+0000", 2024-05-15T23:06:38.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014439699999999999 2024-05-15T23:06:38.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.630524+0000", 2024-05-15T23:06:38.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010058971999999999 2024-05-15T23:06:38.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.636627+0000", 2024-05-15T23:06:38.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061025419999999999 2024-05-15T23:06:38.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.636690+0000", 2024-05-15T23:06:38.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3507e-05 2024-05-15T23:06:38.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:33.636742+0000", 2024-05-15T23:06:38.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1712000000000002e-05 2024-05-15T23:06:38.202 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.202 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.202 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.203 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:78448 2.4 2:2f013c11:::benchmark_data_smithi090_78160_object78447:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e120)", 2024-05-15T23:06:38.203 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:34.884496+0000", 2024-05-15T23:06:38.203 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 3.2615688889999999, 2024-05-15T23:06:38.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028042718000000001, 2024-05-15T23:06:38.203 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.203 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.203 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.203 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.203 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.203 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.203 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 78448 2024-05-15T23:06:38.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.203 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:34.884496+0000", 2024-05-15T23:06:38.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:34.884496+0000", 2024-05-15T23:06:38.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:34.884497+0000", 2024-05-15T23:06:38.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8800000000000002e-07 2024-05-15T23:06:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:34.884526+0000", 2024-05-15T23:06:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8969999999999999e-05 2024-05-15T23:06:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:34.884533+0000", 2024-05-15T23:06:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8480000000000003e-06 2024-05-15T23:06:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:34.884538+0000", 2024-05-15T23:06:38.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5110000000000003e-06 2024-05-15T23:06:38.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:34.887767+0000", 2024-05-15T23:06:38.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003229151 2024-05-15T23:06:38.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:34.887884+0000", 2024-05-15T23:06:38.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011654699999999999 2024-05-15T23:06:38.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:34.888021+0000", 2024-05-15T23:06:38.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000136947 2024-05-15T23:06:38.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:34.897791+0000", 2024-05-15T23:06:38.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0097698920000000005 2024-05-15T23:06:38.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:34.912455+0000", 2024-05-15T23:06:38.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014663996 2024-05-15T23:06:38.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:34.912491+0000", 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6022000000000002e-05 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:34.912539+0000", 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8046e-05 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:80611 2.4 2:2b29d57d:::benchmark_data_smithi090_78160_object80610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e121)", 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:36.016263+0000", 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.129802449, 2024-05-15T23:06:38.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024650980999999999, 2024-05-15T23:06:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 80611 2024-05-15T23:06:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.016263+0000", 2024-05-15T23:06:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.016263+0000", 2024-05-15T23:06:38.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.016264+0000", 2024-05-15T23:06:38.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9649999999999998e-06 2024-05-15T23:06:38.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.016300+0000", 2024-05-15T23:06:38.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5512999999999997e-05 2024-05-15T23:06:38.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.016315+0000", 2024-05-15T23:06:38.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5160000000000001e-05 2024-05-15T23:06:38.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.016323+0000", 2024-05-15T23:06:38.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7710000000000006e-06 2024-05-15T23:06:38.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.016441+0000", 2024-05-15T23:06:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000117663 2024-05-15T23:06:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.016607+0000", 2024-05-15T23:06:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016663100000000001 2024-05-15T23:06:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.016702+0000", 2024-05-15T23:06:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4394999999999993e-05 2024-05-15T23:06:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.021337+0000", 2024-05-15T23:06:38.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046350569999999997 2024-05-15T23:06:38.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.040839+0000", 2024-05-15T23:06:38.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019502578 2024-05-15T23:06:38.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.040879+0000", 2024-05-15T23:06:38.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9987000000000002e-05 2024-05-15T23:06:38.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.040913+0000", 2024-05-15T23:06:38.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4261000000000003e-05 2024-05-15T23:06:38.214 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.214 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.214 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.214 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:80612 2.4 2:28aa77ca:::benchmark_data_smithi090_78160_object80611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e121)", 2024-05-15T23:06:38.214 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:36.016344+0000", 2024-05-15T23:06:38.214 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.129721215, 2024-05-15T23:06:38.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024638067999999999, 2024-05-15T23:06:38.214 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.214 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.214 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.215 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.215 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.215 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.215 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 80612 2024-05-15T23:06:38.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.215 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.016344+0000", 2024-05-15T23:06:38.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.016344+0000", 2024-05-15T23:06:38.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.016345+0000", 2024-05-15T23:06:38.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2299999999999999e-07 2024-05-15T23:06:38.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.016371+0000", 2024-05-15T23:06:38.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6018999999999999e-05 2024-05-15T23:06:38.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.016377+0000", 2024-05-15T23:06:38.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1569999999999997e-06 2024-05-15T23:06:38.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.016382+0000", 2024-05-15T23:06:38.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6420000000000004e-06 2024-05-15T23:06:38.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.018785+0000", 2024-05-15T23:06:38.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002402104 2024-05-15T23:06:38.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.018891+0000", 2024-05-15T23:06:38.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105968 2024-05-15T23:06:38.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.019011+0000", 2024-05-15T23:06:38.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120661 2024-05-15T23:06:38.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.021448+0000", 2024-05-15T23:06:38.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002437177 2024-05-15T23:06:38.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.040939+0000", 2024-05-15T23:06:38.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019490849000000001 2024-05-15T23:06:38.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.040964+0000", 2024-05-15T23:06:38.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4752999999999998e-05 2024-05-15T23:06:38.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.040982+0000", 2024-05-15T23:06:38.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7815000000000001e-05 2024-05-15T23:06:38.220 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.220 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.220 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.220 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:82341 2.4 2:2db6be7b:::benchmark_data_smithi090_78160_object82340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e121)", 2024-05-15T23:06:38.220 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:36.896433+0000", 2024-05-15T23:06:38.220 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.249632259, 2024-05-15T23:06:38.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023778707999999999, 2024-05-15T23:06:38.220 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.220 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.220 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.220 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.220 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.221 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.221 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82341 2024-05-15T23:06:38.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.221 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.896433+0000", 2024-05-15T23:06:38.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.896433+0000", 2024-05-15T23:06:38.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.896436+0000", 2024-05-15T23:06:38.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.275e-06 2024-05-15T23:06:38.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.896478+0000", 2024-05-15T23:06:38.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2524000000000001e-05 2024-05-15T23:06:38.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.896490+0000", 2024-05-15T23:06:38.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1542000000000001e-05 2024-05-15T23:06:38.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.896499+0000", 2024-05-15T23:06:38.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1479999999999995e-06 2024-05-15T23:06:38.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.898321+0000", 2024-05-15T23:06:38.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001821881 2024-05-15T23:06:38.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.898439+0000", 2024-05-15T23:06:38.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000118158 2024-05-15T23:06:38.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.898561+0000", 2024-05-15T23:06:38.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000122032 2024-05-15T23:06:38.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.902804+0000", 2024-05-15T23:06:38.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042422900000000001 2024-05-15T23:06:38.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.920102+0000", 2024-05-15T23:06:38.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017298796000000002 2024-05-15T23:06:38.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.920168+0000", 2024-05-15T23:06:38.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5551999999999999e-05 2024-05-15T23:06:38.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.920211+0000", 2024-05-15T23:06:38.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3510000000000002e-05 2024-05-15T23:06:38.225 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.225 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.226 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.226 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:82345 2.4 2:222ceda8:::benchmark_data_smithi090_78160_object82344:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e121)", 2024-05-15T23:06:38.226 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:36.897316+0000", 2024-05-15T23:06:38.226 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.2487493490000001, 2024-05-15T23:06:38.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025153726000000001, 2024-05-15T23:06:38.226 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.226 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.226 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.226 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.226 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.226 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.227 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82345 2024-05-15T23:06:38.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.227 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.897316+0000", 2024-05-15T23:06:38.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.897316+0000", 2024-05-15T23:06:38.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.897317+0000", 2024-05-15T23:06:38.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.637e-06 2024-05-15T23:06:38.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.897387+0000", 2024-05-15T23:06:38.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0222000000000006e-05 2024-05-15T23:06:38.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.897429+0000", 2024-05-15T23:06:38.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1884000000000001e-05 2024-05-15T23:06:38.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.897437+0000", 2024-05-15T23:06:38.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6599999999999995e-06 2024-05-15T23:06:38.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.900826+0000", 2024-05-15T23:06:38.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033890959999999999 2024-05-15T23:06:38.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.900911+0000", 2024-05-15T23:06:38.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5270000000000002e-05 2024-05-15T23:06:38.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.901003+0000", 2024-05-15T23:06:38.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2120999999999994e-05 2024-05-15T23:06:38.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.902922+0000", 2024-05-15T23:06:38.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001918431 2024-05-15T23:06:38.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.922338+0000", 2024-05-15T23:06:38.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019416473 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.922415+0000", 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7057000000000004e-05 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.922469+0000", 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3875000000000002e-05 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.231 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:82439 2.4 2:2406d8cd:::benchmark_data_smithi090_78160_object82438:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e121)", 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:36.955503+0000", 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.1905622899999999, 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023832111999999999, 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 82439 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.955503+0000", 2024-05-15T23:06:38.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.955503+0000", 2024-05-15T23:06:38.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.955503+0000", 2024-05-15T23:06:38.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8299999999999996e-07 2024-05-15T23:06:38.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.955529+0000", 2024-05-15T23:06:38.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5850000000000002e-05 2024-05-15T23:06:38.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.955535+0000", 2024-05-15T23:06:38.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8989999999999998e-06 2024-05-15T23:06:38.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.955540+0000", 2024-05-15T23:06:38.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2680000000000002e-06 2024-05-15T23:06:38.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.958408+0000", 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002867535 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.958513+0000", 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010488300000000001 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.958640+0000", 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000127038 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.968867+0000", 2024-05-15T23:06:38.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010227465 2024-05-15T23:06:38.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.979241+0000", 2024-05-15T23:06:38.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010373937 2024-05-15T23:06:38.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.979296+0000", 2024-05-15T23:06:38.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.52e-05 2024-05-15T23:06:38.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:36.979335+0000", 2024-05-15T23:06:38.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8353999999999999e-05 2024-05-15T23:06:38.237 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.237 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.237 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.237 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:83434 2.4 2:2a62e88e:::benchmark_data_smithi090_78160_object83433:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e121)", 2024-05-15T23:06:38.237 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:37.502476+0000", 2024-05-15T23:06:38.237 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.64358930599999997, 2024-05-15T23:06:38.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.031422416000000002, 2024-05-15T23:06:38.237 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.238 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.238 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.238 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.238 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.238 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.238 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83434 2024-05-15T23:06:38.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.238 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.502476+0000", 2024-05-15T23:06:38.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.502476+0000", 2024-05-15T23:06:38.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.502476+0000", 2024-05-15T23:06:38.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6899999999999997e-07 2024-05-15T23:06:38.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.502504+0000", 2024-05-15T23:06:38.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7432000000000001e-05 2024-05-15T23:06:38.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.502510+0000", 2024-05-15T23:06:38.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6309999999999997e-06 2024-05-15T23:06:38.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.502515+0000", 2024-05-15T23:06:38.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8960000000000002e-06 2024-05-15T23:06:38.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.505966+0000", 2024-05-15T23:06:38.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003450748 2024-05-15T23:06:38.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.506104+0000", 2024-05-15T23:06:38.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013798900000000001 2024-05-15T23:06:38.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.506253+0000", 2024-05-15T23:06:38.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014861 2024-05-15T23:06:38.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.523608+0000", 2024-05-15T23:06:38.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017354960999999999 2024-05-15T23:06:38.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.533668+0000", 2024-05-15T23:06:38.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010060316999999999 2024-05-15T23:06:38.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.533701+0000", 2024-05-15T23:06:38.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3519000000000002e-05 2024-05-15T23:06:38.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.533898+0000", 2024-05-15T23:06:38.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196644 2024-05-15T23:06:38.242 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.243 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.243 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.243 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:83457 2.4 2:28b6e57f:::benchmark_data_smithi090_78160_object83456:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e121)", 2024-05-15T23:06:38.243 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:37.513542+0000", 2024-05-15T23:06:38.243 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.63252310700000003, 2024-05-15T23:06:38.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.029450124000000001, 2024-05-15T23:06:38.243 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.243 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.243 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.243 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83457 2024-05-15T23:06:38.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.244 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.513542+0000", 2024-05-15T23:06:38.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.513542+0000", 2024-05-15T23:06:38.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.513543+0000", 2024-05-15T23:06:38.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0020000000000001e-06 2024-05-15T23:06:38.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.513617+0000", 2024-05-15T23:06:38.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4188000000000002e-05 2024-05-15T23:06:38.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.513625+0000", 2024-05-15T23:06:38.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7139999999999993e-06 2024-05-15T23:06:38.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.513632+0000", 2024-05-15T23:06:38.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9120000000000001e-06 2024-05-15T23:06:38.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.521008+0000", 2024-05-15T23:06:38.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073759960000000001 2024-05-15T23:06:38.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.521114+0000", 2024-05-15T23:06:38.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000106029 2024-05-15T23:06:38.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.521197+0000", 2024-05-15T23:06:38.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2934000000000003e-05 2024-05-15T23:06:38.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.531080+0000", 2024-05-15T23:06:38.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0098829599999999997 2024-05-15T23:06:38.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.542898+0000", 2024-05-15T23:06:38.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011818162 2024-05-15T23:06:38.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.542949+0000", 2024-05-15T23:06:38.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1039999999999999e-05 2024-05-15T23:06:38.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.542992+0000", 2024-05-15T23:06:38.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3186999999999998e-05 2024-05-15T23:06:38.248 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.248 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.248 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.249 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:83535 2.4 2:2c0388f1:::benchmark_data_smithi090_78160_object83534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e121)", 2024-05-15T23:06:38.249 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:37.566347+0000", 2024-05-15T23:06:38.249 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.57971748999999995, 2024-05-15T23:06:38.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026897388000000001, 2024-05-15T23:06:38.249 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.249 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.249 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.249 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.249 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.249 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.249 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83535 2024-05-15T23:06:38.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.249 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.566347+0000", 2024-05-15T23:06:38.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.566347+0000", 2024-05-15T23:06:38.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.566350+0000", 2024-05-15T23:06:38.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7209999999999999e-06 2024-05-15T23:06:38.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.566395+0000", 2024-05-15T23:06:38.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4773e-05 2024-05-15T23:06:38.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.566406+0000", 2024-05-15T23:06:38.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0764000000000001e-05 2024-05-15T23:06:38.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.566413+0000", 2024-05-15T23:06:38.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6709999999999999e-06 2024-05-15T23:06:38.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.568514+0000", 2024-05-15T23:06:38.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021007080000000002 2024-05-15T23:06:38.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.568600+0000", 2024-05-15T23:06:38.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5674999999999995e-05 2024-05-15T23:06:38.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.568754+0000", 2024-05-15T23:06:38.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000154652 2024-05-15T23:06:38.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.573557+0000", 2024-05-15T23:06:38.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048030010000000003 2024-05-15T23:06:38.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.593130+0000", 2024-05-15T23:06:38.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019572609000000001 2024-05-15T23:06:38.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.593191+0000", 2024-05-15T23:06:38.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1451999999999994e-05 2024-05-15T23:06:38.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.254 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.254 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.593245+0000", 2024-05-15T23:06:38.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3362000000000002e-05 2024-05-15T23:06:38.254 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.254 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.254 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.254 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:83539 2.4 2:2a12e5be:::benchmark_data_smithi090_78160_object83538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e121)", 2024-05-15T23:06:38.254 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:37.568575+0000", 2024-05-15T23:06:38.254 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.577490316, 2024-05-15T23:06:38.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024837512999999999, 2024-05-15T23:06:38.254 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.255 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.255 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.255 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.255 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.255 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.255 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 83539 2024-05-15T23:06:38.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.255 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.255 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:06:38.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.568575+0000", 2024-05-15T23:06:38.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.568575+0000", 2024-05-15T23:06:38.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.568576+0000", 2024-05-15T23:06:38.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0400000000000005e-07 2024-05-15T23:06:38.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.568604+0000", 2024-05-15T23:06:38.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8564999999999998e-05 2024-05-15T23:06:38.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.568611+0000", 2024-05-15T23:06:38.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8940000000000004e-06 2024-05-15T23:06:38.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.568617+0000", 2024-05-15T23:06:38.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2090000000000002e-06 2024-05-15T23:06:38.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.570739+0000", 2024-05-15T23:06:38.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021214960000000001 2024-05-15T23:06:38.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.570864+0000", 2024-05-15T23:06:38.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012578700000000001 2024-05-15T23:06:38.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.571035+0000", 2024-05-15T23:06:38.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000170187 2024-05-15T23:06:38.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.576116+0000", 2024-05-15T23:06:38.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050816910000000002 2024-05-15T23:06:38.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.593300+0000", 2024-05-15T23:06:38.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017183700999999999 2024-05-15T23:06:38.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.593355+0000", 2024-05-15T23:06:38.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4537999999999999e-05 2024-05-15T23:06:38.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:37.593412+0000", 2024-05-15T23:06:38.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7540999999999997e-05 2024-05-15T23:06:38.261 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.261 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.261 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.261 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:84476 2.4 2:24cdb411:::benchmark_data_smithi090_78160_object84475:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e121)", 2024-05-15T23:06:38.261 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:38.077542+0000", 2024-05-15T23:06:38.261 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.068522696999999994, 2024-05-15T23:06:38.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027139748000000002, 2024-05-15T23:06:38.261 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:06:38.262 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:06:38.262 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:06:38.262 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:06:38.262 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:06:38.262 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:06:38.262 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 84476 2024-05-15T23:06:38.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.262 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:06:38.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:06:38.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:38.077542+0000", 2024-05-15T23:06:38.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:06:38.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:38.077542+0000", 2024-05-15T23:06:38.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:06:38.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:06:38.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:38.077544+0000", 2024-05-15T23:06:38.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.513e-06 2024-05-15T23:06:38.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:06:38.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:38.077579+0000", 2024-05-15T23:06:38.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5098000000000002e-05 2024-05-15T23:06:38.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:06:38.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:38.077590+0000", 2024-05-15T23:06:38.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1489e-05 2024-05-15T23:06:38.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:06:38.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:38.077598+0000", 2024-05-15T23:06:38.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3130000000000001e-06 2024-05-15T23:06:38.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:06:38.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:38.080457+0000", 2024-05-15T23:06:38.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028594409999999999 2024-05-15T23:06:38.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:06:38.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:38.080564+0000", 2024-05-15T23:06:38.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010657700000000001 2024-05-15T23:06:38.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:06:38.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:38.080688+0000", 2024-05-15T23:06:38.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000124795 2024-05-15T23:06:38.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:06:38.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:38.090903+0000", 2024-05-15T23:06:38.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010214299 2024-05-15T23:06:38.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:06:38.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:38.104581+0000", 2024-05-15T23:06:38.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013677925 2024-05-15T23:06:38.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:06:38.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:38.104636+0000", 2024-05-15T23:06:38.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5293000000000001e-05 2024-05-15T23:06:38.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:38.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:38.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:06:38.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:38.104682+0000", 2024-05-15T23:06:38.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6004999999999997e-05 2024-05-15T23:06:38.267 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.267 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.267 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.267 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:38.267 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:38.267 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:06:38.282 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:37 smithi088 ceph-mon[44699]: pgmap v380: 17 pgs: 17 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 117 MiB/s wr, 1.87k op/s 2024-05-15T23:06:38.284 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:38.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:37 smithi090 ceph-mon[44434]: pgmap v380: 17 pgs: 17 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 117 MiB/s wr, 1.87k op/s 2024-05-15T23:06:38.305 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:06:38.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:38.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:06:38.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:06:38.456 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:38.478 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:38.482 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:06:38.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:38.487 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:06:38.587 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:38.618 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:38.641 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:38.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:38.746 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:38.776 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:38.795 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:38.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:38.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:38.839 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 34 16 85862 85846 157.783 114.062 0.00581744 0.00633632 2024-05-15T23:06:38.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:06:38.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T23:06:38.954 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:06:38.976 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:39.200 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:06:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:39.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:06:39.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:38 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3141614993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:06:39.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:38 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3141614993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:06:39.312 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:06:39.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:38 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3141614993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:06:39.344 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:39.364 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:06:39.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:39.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:06:39.469 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:06:39.483 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:06:39.483 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:06:39.483 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-05-15T23:06:39.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump -f json-pretty 2024-05-15T23:06:39.511 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:39.539 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:06:39.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:39.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:06:39.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:06:39.684 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:39.710 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:06:39.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:39.715 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:06:39.816 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:06:39.839 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 35 16 87547 87531 156.284 105.312 0.00538869 0.00639683 2024-05-15T23:06:39.846 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:39.865 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:06:39.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:39.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:06:39.969 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:39.996 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:40.018 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:40.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:40.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:40.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:06:40.161 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:40.191 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:06:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:40.195 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:06:40.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:39 smithi090 ceph-mon[44434]: pgmap v381: 17 pgs: 17 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s 2024-05-15T23:06:40.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:39 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:06:40.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3141614993' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:06:40.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:39 smithi090 ceph-mon[44434]: osdmap e122: 12 total, 12 up, 12 in 2024-05-15T23:06:40.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:06:40.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:39 smithi088 ceph-mon[44699]: pgmap v381: 17 pgs: 17 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s 2024-05-15T23:06:40.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:39 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:06:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3141614993' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:06:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:39 smithi088 ceph-mon[44699]: osdmap e122: 12 total, 12 up, 12 in 2024-05-15T23:06:40.336 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:40.365 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:06:40.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:40.369 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:06:40.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:39 smithi084 ceph-mon[39784]: pgmap v381: 17 pgs: 17 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s 2024-05-15T23:06:40.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:39 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:06:40.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3141614993' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:06:40.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:39 smithi084 ceph-mon[39784]: osdmap e122: 12 total, 12 up, 12 in 2024-05-15T23:06:40.470 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:06:40.497 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:40.517 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:06:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:40.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:06:40.621 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:06:40.650 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:40.675 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:06:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:40.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:06:40.779 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:06:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:40.839 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 36 16 89428 89412 155.208 117.562 0.00721001 0.00644145 2024-05-15T23:06:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:06:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:40.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:06:40.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:06:40.996 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:41.025 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:06:41.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:41.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:06:41.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:06:41.185 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:41.213 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:06:41.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:41.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:06:41.320 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:06:41.347 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:41.367 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:06:41.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:41.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:06:41.469 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:41.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:06:41.473 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:41.500 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:41.529 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:06:41.530 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:06:41.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:41.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:06:41.564 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:06:41.605 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:06:41.633 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:06:41.644 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:06:41.666 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:41.674 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:06:41.688 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:06:41.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:41.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:06:41.703 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:06:41.734 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:06:41.769 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:06:41.793 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:06:41.815 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:06:41.839 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 37 16 91059 91043 153.768 101.938 0.00674795 0.00650194 2024-05-15T23:06:41.857 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:41.858 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:06:41.892 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:06:41.930 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:06:41.938 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:06:41.938 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:06:41.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:06:41.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:06:41.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:06:41.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:06:41.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:06:41.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:06:41.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:06:41.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:06:41.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:06:41.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:06:42.018 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:06:42.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:41 smithi084 ceph-mon[39784]: pgmap v383: 17 pgs: 17 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.17k op/s 2024-05-15T23:06:42.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:41 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4174361607' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:06:42.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:42.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:41 smithi084 ceph-mon[39784]: osdmap e123: 12 total, 12 up, 12 in 2024-05-15T23:06:42.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:42.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:42.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:42.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:06:42.038 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:06:42.126 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:42.218 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:42.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:41 smithi090 ceph-mon[44434]: pgmap v383: 17 pgs: 17 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.17k op/s 2024-05-15T23:06:42.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:41 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4174361607' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:06:42.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:42.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:41 smithi090 ceph-mon[44434]: osdmap e123: 12 total, 12 up, 12 in 2024-05-15T23:06:42.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:42.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:42.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:42.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:41 smithi088 ceph-mon[44699]: pgmap v383: 17 pgs: 17 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.17k op/s 2024-05-15T23:06:42.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:41 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4174361607' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:06:42.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:42.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:41 smithi088 ceph-mon[44699]: osdmap e123: 12 total, 12 up, 12 in 2024-05-15T23:06:42.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:42.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:42.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:42.839 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 38 16 92790 92774 152.569 108.188 0.00660022 0.00655309 2024-05-15T23:06:42.882 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:42.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:42.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:06:43.030 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:43.055 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:06:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:43.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:06:43.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:06:43.198 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:43.227 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:06:43.229 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.229 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.229 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.229 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.229 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.229 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.229 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.229 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:43.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:06:43.307 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.307 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.307 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.307 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.307 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:43.331 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:06:43.372 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:43.397 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:06:43.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:43.400 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:06:43.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:06:43.533 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:06:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:43.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:06:43.665 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:06:43.691 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:43.711 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:06:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:43.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:06:43.814 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:06:43.839 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 39 16 94534 94518 151.451 109 0.00513434 0.00660149 2024-05-15T23:06:43.844 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:43.866 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:06:43.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:43.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:06:43.970 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:43.998 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:44.020 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:44.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:44.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:44.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:06:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:44.197 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:06:44.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:44.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:06:44.301 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:06:44.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:43 smithi090 ceph-mon[44434]: pgmap v385: 17 pgs: 17 active+clean; 5.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-05-15T23:06:44.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:43 smithi088 ceph-mon[44699]: pgmap v385: 17 pgs: 17 active+clean; 5.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-05-15T23:06:44.328 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:44.351 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:06:44.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:44.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:06:44.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:43 smithi084 ceph-mon[39784]: pgmap v385: 17 pgs: 17 active+clean; 5.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-05-15T23:06:44.455 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:06:44.486 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:44.515 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:06:44.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:44.520 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:06:44.582 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:44.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:06:44.656 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:44.686 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:06:44.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:44.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:06:44.794 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:44.821 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:44.840 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:06:44.839837 min lat: 0.000985234 max lat: 2.1501 avg lat: 0.00664993 2024-05-15T23:06:44.840 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:06:44.840 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 40 16 96250 96234 150.346 107.25 0.00348249 0.00664993 2024-05-15T23:06:44.841 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:44.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:44.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:44.945 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:44.996 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:45.131 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:45.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:45.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:45.301 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:45.353 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:45.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:45.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:45.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:06:45.503 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:45.530 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:06:45.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:45.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:06:45.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:06:45.646 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:06:45.646 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:06:45.646 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 123, 2024-05-15T23:06:45.646 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "3999fb7e-130e-11ef-bc99-c7b262605968", 2024-05-15T23:06:45.646 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-05-15T22:56:41.212493+0000", 2024-05-15T23:06:45.646 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-05-15T23:06:41.618081+0000", 2024-05-15T23:06:45.646 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-05-15T23:03:56.576073+0000", 2024-05-15T23:06:45.646 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-05-15T23:03:38.460671+0000", 2024-05-15T23:06:45.646 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-15T23:06:45.647 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5801984, 2024-05-15T23:06:45.647 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-05-15T23:06:45.647 INFO:teuthology.orchestra.run.smithi084.stdout: "noscrub", 2024-05-15T23:06:45.647 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-05-15T23:06:45.647 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-05-15T23:06:45.647 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-05-15T23:06:45.647 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-05-15T23:06:45.647 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:06:45.647 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 27, 2024-05-15T23:06:45.647 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-05-15T23:06:45.647 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-15T23:06:45.647 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-15T23:06:45.647 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-05-15T23:06:45.647 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 2, 2024-05-15T23:06:45.647 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 12, 2024-05-15T23:06:45.648 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-05-15T23:06:45.648 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-05-15T23:06:45.648 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-05-15T23:06:45.648 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-05-15T23:06:45.648 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-05-15T23:06:45.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.648 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-05-15T23:06:45.648 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-05-15T23:06:45.648 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-05-15T22:59:38.603575+0000", 2024-05-15T23:06:45.648 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-05-15T23:06:45.648 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-05-15T23:06:45.648 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-05-15T23:06:45.648 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2024-05-15T23:06:45.648 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-05-15T23:06:45.649 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-05-15T23:06:45.649 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-05-15T23:06:45.649 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-05-15T23:06:45.649 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T23:06:45.649 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T23:06:45.649 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-05-15T23:06:45.649 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-05-15T23:06:45.649 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 1, 2024-05-15T23:06:45.649 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 1, 2024-05-15T23:06:45.649 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 1, 2024-05-15T23:06:45.649 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 1, 2024-05-15T23:06:45.649 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 1, 2024-05-15T23:06:45.649 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-05-15T23:06:45.649 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-05-15T23:06:45.650 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-05-15T23:06:45.650 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-05-15T23:06:45.650 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-05-15T23:06:45.650 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-05-15T23:06:45.650 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-05-15T23:06:45.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.650 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "16", 2024-05-15T23:06:45.650 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-05-15T23:06:45.650 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T23:06:45.650 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T23:06:45.650 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-05-15T23:06:45.650 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-05-15T23:06:45.650 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-05-15T23:06:45.651 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-05-15T23:06:45.651 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-05-15T23:06:45.651 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-05-15T23:06:45.651 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-05-15T23:06:45.651 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-05-15T23:06:45.651 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-05-15T23:06:45.651 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-05-15T23:06:45.651 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-05-15T23:06:45.651 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-05-15T23:06:45.651 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-05-15T23:06:45.651 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-05-15T23:06:45.651 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-05-15T23:06:45.651 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T23:06:45.652 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T23:06:45.652 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T23:06:45.652 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-05-15T23:06:45.652 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-05-15T23:06:45.652 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-05-15T23:06:45.652 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-05-15T23:06:45.652 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-05-15T23:06:45.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.652 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-05-15T23:06:45.652 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-05-15T23:06:45.652 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-05-15T23:06:45.652 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-05-15T23:06:45.653 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-05-15T23:06:45.653 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T23:06:45.653 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-05-15T23:06:45.653 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-05-15T23:06:45.653 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-05-15T23:06:45.653 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-05-15T23:06:45.653 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-05-15T23:06:45.653 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-05-15T23:06:45.653 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-05-15T23:06:45.653 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-05-15T23:06:45.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.653 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-05-15T23:06:45.653 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-05-15T23:06:45.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.653 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-05-15T23:06:45.654 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-05-15T23:06:45.654 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 11.760000228881836, 2024-05-15T23:06:45.654 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 11.760000228881836, 2024-05-15T23:06:45.654 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.17000000178813934, 2024-05-15T23:06:45.654 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-05-15T23:06:45.654 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-05-15T23:06:45.654 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-05-15T23:06:45.654 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-05-15T23:06:45.654 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-05-15T23:06:45.654 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.654 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-05-15T23:06:45.655 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-05-15T23:06:45.655 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-05-15T23:05:50.137063+0000", 2024-05-15T23:06:45.655 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-05-15T23:06:45.655 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-05-15T23:06:45.655 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-05-15T23:06:45.655 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2024-05-15T23:06:45.655 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-05-15T23:06:45.655 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-05-15T23:06:45.655 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-05-15T23:06:45.655 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-05-15T23:06:45.655 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T23:06:45.655 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T23:06:45.655 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-05-15T23:06:45.655 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-05-15T23:06:45.655 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 16, 2024-05-15T23:06:45.656 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 16, 2024-05-15T23:06:45.656 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 16, 2024-05-15T23:06:45.656 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 16, 2024-05-15T23:06:45.656 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 16, 2024-05-15T23:06:45.656 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-05-15T23:06:45.656 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-05-15T23:06:45.656 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-05-15T23:06:45.656 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-05-15T23:06:45.656 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-05-15T23:06:45.656 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-05-15T23:06:45.656 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-05-15T23:06:45.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.656 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "105", 2024-05-15T23:06:45.656 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-05-15T23:06:45.657 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T23:06:45.657 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T23:06:45.657 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-05-15T23:06:45.657 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-05-15T23:06:45.657 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-05-15T23:06:45.657 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-05-15T23:06:45.657 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-05-15T23:06:45.657 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-05-15T23:06:45.657 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-05-15T23:06:45.657 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-05-15T23:06:45.657 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-05-15T23:06:45.657 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-05-15T23:06:45.657 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-05-15T23:06:45.657 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-05-15T23:06:45.657 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-05-15T23:06:45.658 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-05-15T23:06:45.658 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-05-15T23:06:45.658 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T23:06:45.658 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T23:06:45.658 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T23:06:45.658 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-05-15T23:06:45.658 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-05-15T23:06:45.658 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-05-15T23:06:45.658 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-05-15T23:06:45.658 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-05-15T23:06:45.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.658 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-05-15T23:06:45.658 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-05-15T23:06:45.658 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-05-15T23:06:45.659 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-05-15T23:06:45.659 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-05-15T23:06:45.659 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T23:06:45.659 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-05-15T23:06:45.659 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-05-15T23:06:45.659 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-05-15T23:06:45.659 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-05-15T23:06:45.659 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-05-15T23:06:45.659 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-05-15T23:06:45.659 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-05-15T23:06:45.659 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-05-15T23:06:45.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.659 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-05-15T23:06:45.659 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.2400000095367432, 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.2400000095367432, 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.92000001668930054, 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.059999942779541, 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.059999942779541, 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "80159cc8-1b6c-4aa6-a114-966786d7be1e", 2024-05-15T23:06:45.660 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:06:45.661 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:06:45.661 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:06:45.661 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:06:45.661 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:06:45.661 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:06:45.661 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-05-15T23:06:45.661 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 113, 2024-05-15T23:06:45.661 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:06:45.661 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:06:45.661 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:06:45.661 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.661 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.662 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-05-15T23:06:45.662 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1084680531 2024-05-15T23:06:45.662 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.662 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.662 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:06:45.662 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.662 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.662 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-05-15T23:06:45.662 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1084680531 2024-05-15T23:06:45.662 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.662 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.663 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:06:45.663 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.663 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.663 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-05-15T23:06:45.663 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1084680531 2024-05-15T23:06:45.663 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.663 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.663 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:06:45.663 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.663 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.663 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-05-15T23:06:45.664 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1084680531 2024-05-15T23:06:45.664 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.664 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.664 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/1084680531", 2024-05-15T23:06:45.664 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/1084680531", 2024-05-15T23:06:45.664 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/1084680531", 2024-05-15T23:06:45.664 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/1084680531", 2024-05-15T23:06:45.664 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:06:45.664 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:06:45.664 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:06:45.664 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.664 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-05-15T23:06:45.665 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "9c2c76c9-ce0f-4c2d-9d96-c87a10187607", 2024-05-15T23:06:45.665 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:06:45.665 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:06:45.665 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:06:45.665 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:06:45.665 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:06:45.665 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:06:45.665 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-05-15T23:06:45.665 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 102, 2024-05-15T23:06:45.665 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:06:45.665 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:06:45.665 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:06:45.665 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.665 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.666 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-05-15T23:06:45.666 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:06:45.666 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.666 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.666 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:06:45.666 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.666 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.666 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-05-15T23:06:45.666 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:06:45.666 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.666 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.667 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:06:45.667 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.667 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.667 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-05-15T23:06:45.667 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:06:45.667 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.667 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.667 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:06:45.667 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.667 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.668 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-05-15T23:06:45.668 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:06:45.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.668 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.668 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/2140830229", 2024-05-15T23:06:45.668 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/2140830229", 2024-05-15T23:06:45.668 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/2140830229", 2024-05-15T23:06:45.668 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/2140830229", 2024-05-15T23:06:45.668 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:06:45.668 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:06:45.668 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:06:45.668 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.669 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-05-15T23:06:45.669 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "e8514daa-b5f9-4f2f-a432-53f4a4eb86bc", 2024-05-15T23:06:45.669 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:06:45.669 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:06:45.669 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:06:45.669 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:06:45.669 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:06:45.669 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:06:45.669 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 20, 2024-05-15T23:06:45.669 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 102, 2024-05-15T23:06:45.669 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:06:45.669 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:06:45.669 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:06:45.670 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.670 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.670 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-05-15T23:06:45.670 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2699996290 2024-05-15T23:06:45.670 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.670 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.670 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:06:45.670 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.670 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.670 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-05-15T23:06:45.670 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2699996290 2024-05-15T23:06:45.671 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.671 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.671 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:06:45.671 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.671 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.671 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-05-15T23:06:45.671 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2699996290 2024-05-15T23:06:45.671 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.671 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.671 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:06:45.671 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.672 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.672 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-05-15T23:06:45.672 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2699996290 2024-05-15T23:06:45.672 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.672 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.672 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/2699996290", 2024-05-15T23:06:45.672 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/2699996290", 2024-05-15T23:06:45.672 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/2699996290", 2024-05-15T23:06:45.672 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/2699996290", 2024-05-15T23:06:45.672 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:06:45.672 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:06:45.672 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:06:45.672 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.673 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-05-15T23:06:45.673 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "5d9965f9-c0b2-48ce-869c-582990985623", 2024-05-15T23:06:45.673 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:06:45.673 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:06:45.673 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:06:45.673 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:06:45.673 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:06:45.673 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:06:45.673 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 26, 2024-05-15T23:06:45.673 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2024-05-15T23:06:45.673 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:06:45.673 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:06:45.674 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:06:45.674 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.674 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.674 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-05-15T23:06:45.674 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:06:45.674 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.674 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.674 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:06:45.674 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.674 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.674 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-05-15T23:06:45.674 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:06:45.675 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.675 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.675 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:06:45.675 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.675 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.675 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-05-15T23:06:45.675 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:06:45.675 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.675 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.675 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:06:45.675 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.676 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.676 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-05-15T23:06:45.676 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:06:45.676 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.676 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.676 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2204039828", 2024-05-15T23:06:45.676 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2204039828", 2024-05-15T23:06:45.676 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2204039828", 2024-05-15T23:06:45.676 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2204039828", 2024-05-15T23:06:45.676 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:06:45.676 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:06:45.676 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:06:45.677 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.677 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-05-15T23:06:45.677 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4bc50523-8480-44d2-8d76-31479e839c85", 2024-05-15T23:06:45.677 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:06:45.677 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:06:45.677 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:06:45.677 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:06:45.677 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:06:45.677 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:06:45.677 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-05-15T23:06:45.677 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2024-05-15T23:06:45.677 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:06:45.677 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:06:45.678 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:06:45.678 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.678 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.678 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6800", 2024-05-15T23:06:45.678 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:06:45.678 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.678 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.678 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:06:45.678 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.678 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.678 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6801", 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6803", 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.680 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.680 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6802", 2024-05-15T23:06:45.680 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:06:45.680 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.680 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.680 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6800/2267955292", 2024-05-15T23:06:45.680 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6801/2267955292", 2024-05-15T23:06:45.680 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6803/2267955292", 2024-05-15T23:06:45.680 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6802/2267955292", 2024-05-15T23:06:45.680 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:06:45.680 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:06:45.680 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:06:45.680 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.681 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-05-15T23:06:45.681 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20", 2024-05-15T23:06:45.681 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:06:45.681 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:06:45.681 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:06:45.681 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:06:45.681 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:06:45.681 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:06:45.681 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 37, 2024-05-15T23:06:45.681 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 102, 2024-05-15T23:06:45.681 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:06:45.681 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:06:45.682 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:06:45.682 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.682 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.682 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6804", 2024-05-15T23:06:45.682 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:06:45.682 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.682 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.682 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:06:45.682 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.682 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.682 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6805", 2024-05-15T23:06:45.683 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:06:45.683 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.683 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.683 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:06:45.683 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.683 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.683 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6807", 2024-05-15T23:06:45.683 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:06:45.683 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.683 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.683 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:06:45.684 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.684 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.684 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6806", 2024-05-15T23:06:45.684 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:06:45.684 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.684 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.684 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6804/661041538", 2024-05-15T23:06:45.684 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6805/661041538", 2024-05-15T23:06:45.684 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6807/661041538", 2024-05-15T23:06:45.684 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6806/661041538", 2024-05-15T23:06:45.684 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:06:45.684 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:06:45.685 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:06:45.685 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.685 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-05-15T23:06:45.685 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f", 2024-05-15T23:06:45.685 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:06:45.685 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:06:45.685 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:06:45.685 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:06:45.685 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:06:45.685 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:06:45.685 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 46, 2024-05-15T23:06:45.685 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 112, 2024-05-15T23:06:45.685 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:06:45.686 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:06:45.686 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:06:45.686 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.686 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.686 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6808", 2024-05-15T23:06:45.686 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:06:45.686 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.686 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.686 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:06:45.686 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6809", 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6811", 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.687 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:06:45.688 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.688 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.688 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6810", 2024-05-15T23:06:45.688 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:06:45.688 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.688 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.688 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6808/380856869", 2024-05-15T23:06:45.688 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6809/380856869", 2024-05-15T23:06:45.688 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6811/380856869", 2024-05-15T23:06:45.688 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6810/380856869", 2024-05-15T23:06:45.688 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:06:45.688 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:06:45.689 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:06:45.689 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.689 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-05-15T23:06:45.689 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34", 2024-05-15T23:06:45.689 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:06:45.689 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:06:45.689 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:06:45.689 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:06:45.689 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:06:45.689 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:06:45.689 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 53, 2024-05-15T23:06:45.689 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 102, 2024-05-15T23:06:45.689 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:06:45.690 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:06:45.690 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:06:45.690 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.690 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.690 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6812", 2024-05-15T23:06:45.690 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:06:45.690 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.690 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.690 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:06:45.690 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.690 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.691 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6813", 2024-05-15T23:06:45.691 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:06:45.691 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.691 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.691 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:06:45.691 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.691 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.691 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6815", 2024-05-15T23:06:45.691 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:06:45.691 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.691 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.692 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:06:45.692 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.692 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.692 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6814", 2024-05-15T23:06:45.692 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:06:45.692 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.692 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.692 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6812/1044709063", 2024-05-15T23:06:45.692 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6813/1044709063", 2024-05-15T23:06:45.692 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6815/1044709063", 2024-05-15T23:06:45.692 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6814/1044709063", 2024-05-15T23:06:45.692 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:06:45.692 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:06:45.693 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:06:45.693 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.693 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-05-15T23:06:45.693 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "02cab286-d487-4a66-982c-3cf28f55e851", 2024-05-15T23:06:45.693 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:06:45.693 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:06:45.693 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:06:45.693 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:06:45.693 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:06:45.693 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:06:45.693 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 61, 2024-05-15T23:06:45.693 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 102, 2024-05-15T23:06:45.693 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:06:45.694 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:06:45.694 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:06:45.694 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.694 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.694 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6800", 2024-05-15T23:06:45.694 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 522459126 2024-05-15T23:06:45.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.694 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.694 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:06:45.694 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.695 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.695 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6801", 2024-05-15T23:06:45.695 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 522459126 2024-05-15T23:06:45.695 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.695 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.695 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:06:45.695 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.695 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.695 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6803", 2024-05-15T23:06:45.695 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 522459126 2024-05-15T23:06:45.695 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.696 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.696 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:06:45.696 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.696 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.696 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6802", 2024-05-15T23:06:45.696 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 522459126 2024-05-15T23:06:45.696 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.696 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.696 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6800/522459126", 2024-05-15T23:06:45.696 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6801/522459126", 2024-05-15T23:06:45.696 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6803/522459126", 2024-05-15T23:06:45.696 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6802/522459126", 2024-05-15T23:06:45.697 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:06:45.697 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:06:45.697 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:06:45.697 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.697 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-05-15T23:06:45.697 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613", 2024-05-15T23:06:45.697 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:06:45.697 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:06:45.697 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:06:45.697 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:06:45.697 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:06:45.697 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:06:45.697 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 68, 2024-05-15T23:06:45.698 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 102, 2024-05-15T23:06:45.698 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:06:45.698 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:06:45.698 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:06:45.698 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.698 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.698 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6804", 2024-05-15T23:06:45.698 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:06:45.698 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.698 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.698 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:06:45.698 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.699 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.699 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6805", 2024-05-15T23:06:45.699 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:06:45.699 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.699 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.699 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:06:45.699 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.699 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.699 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6807", 2024-05-15T23:06:45.699 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:06:45.699 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.700 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.700 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:06:45.700 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.700 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.700 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6806", 2024-05-15T23:06:45.700 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:06:45.700 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.700 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.700 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6804/2794571133", 2024-05-15T23:06:45.700 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6805/2794571133", 2024-05-15T23:06:45.700 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6807/2794571133", 2024-05-15T23:06:45.700 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6806/2794571133", 2024-05-15T23:06:45.701 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:06:45.701 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:06:45.701 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:06:45.701 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.701 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-05-15T23:06:45.701 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473", 2024-05-15T23:06:45.701 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:06:45.701 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:06:45.701 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:06:45.701 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:06:45.701 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:06:45.702 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:06:45.702 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 74, 2024-05-15T23:06:45.702 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 0, 2024-05-15T23:06:45.702 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:06:45.702 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:06:45.702 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:06:45.702 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.702 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.702 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6808", 2024-05-15T23:06:45.702 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:06:45.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.703 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.703 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:06:45.703 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.703 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.703 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6809", 2024-05-15T23:06:45.703 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:06:45.703 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.703 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.703 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:06:45.703 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.704 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.704 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6811", 2024-05-15T23:06:45.704 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:06:45.704 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.704 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.704 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:06:45.704 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.704 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.704 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6810", 2024-05-15T23:06:45.704 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:06:45.704 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.704 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.705 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6808/3735380744", 2024-05-15T23:06:45.705 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6809/3735380744", 2024-05-15T23:06:45.705 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6811/3735380744", 2024-05-15T23:06:45.705 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6810/3735380744", 2024-05-15T23:06:45.705 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:06:45.705 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:06:45.705 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:06:45.705 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.705 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-05-15T23:06:45.705 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5", 2024-05-15T23:06:45.706 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:06:45.706 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:06:45.706 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:06:45.706 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:06:45.706 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:06:45.706 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:06:45.706 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 81, 2024-05-15T23:06:45.706 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 102, 2024-05-15T23:06:45.706 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:06:45.706 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:06:45.706 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:06:45.706 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.707 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.707 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6812", 2024-05-15T23:06:45.707 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:06:45.707 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.707 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.707 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:06:45.707 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.707 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.707 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6813", 2024-05-15T23:06:45.707 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:06:45.707 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.707 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.708 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:06:45.708 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.708 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.708 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6815", 2024-05-15T23:06:45.708 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:06:45.708 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.708 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.708 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:06:45.708 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:06:45.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.708 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:06:45.709 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6814", 2024-05-15T23:06:45.709 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:06:45.709 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.709 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.709 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6812/2265615270", 2024-05-15T23:06:45.709 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6813/2265615270", 2024-05-15T23:06:45.709 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6815/2265615270", 2024-05-15T23:06:45.709 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6814/2265615270", 2024-05-15T23:06:45.709 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:06:45.709 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:06:45.709 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:06:45.709 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.710 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.710 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:06:45.710 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-05-15T23:06:45.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.710 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-05-15T23:06:45.710 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:06:45.710 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:06:45.710 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:06:45.710 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:06:45.710 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:06:45.710 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:05:36.073777+0000", 2024-05-15T23:06:45.710 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:06:45.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:06:05.455859+0000", 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:06:45.711 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:06:45.712 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:06:26.964593+0000", 2024-05-15T23:06:45.712 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:06:45.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.712 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-05-15T23:06:45.712 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:06:45.712 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:06:45.712 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:06:45.712 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:06:45.712 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:06:45.712 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:06:32.154761+0000", 2024-05-15T23:06:45.712 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:06:45.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.713 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-05-15T23:06:45.713 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:06:45.713 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:06:45.713 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:06:45.713 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:06:45.713 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:06:45.713 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:05:38.362254+0000", 2024-05-15T23:06:45.713 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:06:45.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.713 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-05-15T23:06:45.713 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:06:45.714 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:06:45.714 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:06:45.714 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:06:45.714 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:06:45.714 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:06:01.865452+0000", 2024-05-15T23:06:45.714 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:06:45.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.714 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-05-15T23:06:45.714 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:06:45.714 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:06:45.714 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:06:45.714 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:06:45.714 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:06:45.715 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:05:37.820222+0000", 2024-05-15T23:06:45.715 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:06:45.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.715 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-05-15T23:06:45.715 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:06:45.715 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:06:45.715 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:06:45.715 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:06:45.715 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:06:45.715 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:06:26.030494+0000", 2024-05-15T23:06:45.715 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:06:45.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.716 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-05-15T23:06:45.716 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:06:45.716 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:06:45.716 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:06:45.716 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:06:45.716 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:06:45.716 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:06:27.761807+0000", 2024-05-15T23:06:45.716 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:06:45.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.716 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-05-15T23:06:45.716 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:06:45.716 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:06:45.716 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:06:45.717 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:06:45.717 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:06:45.717 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:05:23.099371+0000", 2024-05-15T23:06:45.717 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:06:45.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.717 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-05-15T23:06:45.717 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:06:45.717 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:06:45.717 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:06:45.717 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:06:45.717 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:06:45.717 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:06:04.243811+0000", 2024-05-15T23:06:45.717 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:06:45.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.718 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-05-15T23:06:45.718 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:06:45.718 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:06:45.718 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:06:45.718 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:06:45.718 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:06:45.718 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:06:34.807756+0000", 2024-05-15T23:06:45.718 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:06:45.718 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.718 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:06:45.718 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2024-05-15T23:06:45.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:06:45.719 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.7", 2024-05-15T23:06:45.719 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-05-15T23:06:45.719 INFO:teuthology.orchestra.run.smithi084.stdout: 0, 2024-05-15T23:06:45.719 INFO:teuthology.orchestra.run.smithi084.stdout: 9 2024-05-15T23:06:45.719 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:06:45.719 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.719 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:06:45.719 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [], 2024-05-15T23:06:45.719 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-05-15T23:06:45.719 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-05-15T23:06:45.719 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-05-15T23:06:45.719 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-05-15T23:06:45.720 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2581408231": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:06:45.720 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/1136984393": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:06:45.720 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2443201864": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:06:45.720 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1252671888": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:06:45.720 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1451001371": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:06:45.720 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4084814866": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:06:45.720 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4076409285": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:06:45.720 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2988977694": "2024-05-16T22:57:38.505263+0000" 2024-05-15T23:06:45.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.720 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-05-15T23:06:45.720 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-05-15T23:06:45.720 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-05-15T23:06:45.720 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-05-15T23:06:45.720 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-05-15T23:06:45.720 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-05-15T23:06:45.721 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-05-15T23:06:45.721 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-05-15T23:06:45.721 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:06:45.721 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [], 2024-05-15T23:06:45.721 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2024-05-15T23:06:45.721 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-05-15T23:06:45.721 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-05-15T23:06:45.721 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-05-15T23:06:45.721 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-05-15T23:06:45.721 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-05-15T23:06:45.721 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-05-15T23:06:45.721 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-05-15T23:06:45.722 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-05-15T23:06:45.722 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-05-15T23:06:45.722 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:06:45.722 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:06:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:06:45.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:06:45.824 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:06:45.840 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 41 16 97849 97833 149.117 99.9375 0.00500145 0.00670462 2024-05-15T23:06:45.853 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:45.875 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:06:45.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:45.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:06:45.979 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:46.007 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:46.026 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:46.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:46.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:46.126 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:06:46.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:06:46.167 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:46.192 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:06:46.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:46.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:06:46.298 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:46.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:45 smithi090 ceph-mon[44434]: pgmap v386: 17 pgs: 17 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s 2024-05-15T23:06:46.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1803262715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:06:46.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:45 smithi088 ceph-mon[44699]: pgmap v386: 17 pgs: 17 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s 2024-05-15T23:06:46.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1803262715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:06:46.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:45 smithi084 ceph-mon[39784]: pgmap v386: 17 pgs: 17 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s 2024-05-15T23:06:46.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1803262715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:06:46.350 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:46.422 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:46.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:46.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:46.583 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:46.758 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:46.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:46.840 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 42 16 99493 99477 148.012 102.75 0.0144675 0.00675447 2024-05-15T23:06:46.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:06:46.901 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:46.924 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:06:46.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:46.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:06:47.035 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:06:47.080 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:47.080 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:46 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:06:47.081 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:46 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:06:47.081 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:46 smithi084 ceph-mon[39784]: osdmap e124: 12 total, 12 up, 12 in 2024-05-15T23:06:47.081 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:46 smithi084 ceph-mon[39784]: osdmap e125: 12 total, 12 up, 12 in 2024-05-15T23:06:47.115 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:06:47.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:47.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:06:47.224 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:06:47.256 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:47.256 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:46 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:06:47.256 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:46 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:06:47.256 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:46 smithi090 ceph-mon[44434]: osdmap e124: 12 total, 12 up, 12 in 2024-05-15T23:06:47.256 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:46 smithi090 ceph-mon[44434]: osdmap e125: 12 total, 12 up, 12 in 2024-05-15T23:06:47.291 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:06:47.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:47.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:06:47.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:46 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:06:47.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:46 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:06:47.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:46 smithi088 ceph-mon[44699]: osdmap e124: 12 total, 12 up, 12 in 2024-05-15T23:06:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:46 smithi088 ceph-mon[44699]: osdmap e125: 12 total, 12 up, 12 in 2024-05-15T23:06:47.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:06:47.437 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:47.469 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:06:47.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:47.475 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:06:47.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:06:47.625 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:47.656 INFO:tasks.thrashosds.thrasher:j is {'epoch': 123, 'fsid': '3999fb7e-130e-11ef-bc99-c7b262605968', 'created': '2024-05-15T22:56:41.212493+0000', 'modified': '2024-05-15T23:06:41.618081+0000', 'last_up_change': '2024-05-15T23:03:56.576073+0000', 'last_in_change': '2024-05-15T23:03:38.460671+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T22:59:38.603575+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-05-15T23:05:50.137063+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '80159cc8-1b6c-4aa6-a114-966786d7be1e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 113, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6801', 'nonce': 1084680531}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6802', 'nonce': 1084680531}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6804', 'nonce': 1084680531}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6803', 'nonce': 1084680531}]}, 'public_addr': '172.21.15.84:6801/1084680531', 'cluster_addr': '172.21.15.84:6802/1084680531', 'heartbeat_back_addr': '172.21.15.84:6804/1084680531', 'heartbeat_front_addr': '172.21.15.84:6803/1084680531', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9c2c76c9-ce0f-4c2d-9d96-c87a10187607', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 102, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6805', 'nonce': 2140830229}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6806', 'nonce': 2140830229}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6808', 'nonce': 2140830229}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6807', 'nonce': 2140830229}]}, 'public_addr': '172.21.15.84:6805/2140830229', 'cluster_addr': '172.21.15.84:6806/2140830229', 'heartbeat_back_addr': '172.21.15.84:6808/2140830229', 'heartbeat_front_addr': '172.21.15.84:6807/2140830229', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e8514daa-b5f9-4f2f-a432-53f4a4eb86bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 20, 'up_thru': 102, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6809', 'nonce': 2699996290}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6810', 'nonce': 2699996290}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6812', 'nonce': 2699996290}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6811', 'nonce': 2699996290}]}, 'public_addr': '172.21.15.84:6809/2699996290', 'cluster_addr': '172.21.15.84:6810/2699996290', 'heartbeat_back_addr': '172.21.15.84:6812/2699996290', 'heartbeat_front_addr': '172.21.15.84:6811/2699996290', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5d9965f9-c0b2-48ce-869c-582990985623', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6813', 'nonce': 2204039828}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6814', 'nonce': 2204039828}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6816', 'nonce': 2204039828}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6815', 'nonce': 2204039828}]}, 'public_addr': '172.21.15.84:6813/2204039828', 'cluster_addr': '172.21.15.84:6814/2204039828', 'heartbeat_back_addr': '172.21.15.84:6816/2204039828', 'heartbeat_front_addr': '172.21.15.84:6815/2204039828', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4bc50523-8480-44d2-8d76-31479e839c85', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6800', 'nonce': 2267955292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6801', 'nonce': 2267955292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6803', 'nonce': 2267955292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6802', 'nonce': 2267955292}]}, 'public_addr': '172.21.15.88:6800/2267955292', 'cluster_addr': '172.21.15.88:6801/2267955292', 'heartbeat_back_addr': '172.21.15.88:6803/2267955292', 'heartbeat_front_addr': '172.21.15.88:6802/2267955292', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'd4db8d7f-afc7-43dc-81fe-f70c290d8b20', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 102, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6804', 'nonce': 661041538}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6805', 'nonce': 661041538}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6807', 'nonce': 661041538}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6806', 'nonce': 661041538}]}, 'public_addr': '172.21.15.88:6804/661041538', 'cluster_addr': '172.21.15.88:6805/661041538', 'heartbeat_back_addr': '172.21.15.88:6807/661041538', 'heartbeat_front_addr': '172.21.15.88:6806/661041538', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'a2773454-2318-4a5b-8f31-0952b1b07f2f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 112, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6808', 'nonce': 380856869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6809', 'nonce': 380856869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6811', 'nonce': 380856869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6810', 'nonce': 380856869}]}, 'public_addr': '172.21.15.88:6808/380856869', 'cluster_addr': '172.21.15.88:6809/380856869', 'heartbeat_back_addr': '172.21.15.88:6811/380856869', 'heartbeat_front_addr': '172.21.15.88:6810/380856869', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '028ca7b6-3e00-4466-a33f-b726cd60cb34', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 102, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6812', 'nonce': 1044709063}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6813', 'nonce': 1044709063}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6815', 'nonce': 1044709063}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6814', 'nonce': 1044709063}]}, 'public_addr': '172.21.15.88:6812/1044709063', 'cluster_addr': '172.21.15.88:6813/1044709063', 'heartbeat_back_addr': '172.21.15.88:6815/1044709063', 'heartbeat_front_addr': '172.21.15.88:6814/1044709063', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '02cab286-d487-4a66-982c-3cf28f55e851', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 102, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6800', 'nonce': 522459126}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 522459126}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 522459126}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6802', 'nonce': 522459126}]}, 'public_addr': '172.21.15.90:6800/522459126', 'cluster_addr': '172.21.15.90:6801/522459126', 'heartbeat_back_addr': '172.21.15.90:6803/522459126', 'heartbeat_front_addr': '172.21.15.90:6802/522459126', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '47d38bc7-39ba-4b35-af24-e282354dd613', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 68, 'up_thru': 102, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6804', 'nonce': 2794571133}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 2794571133}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 2794571133}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6806', 'nonce': 2794571133}]}, 'public_addr': '172.21.15.90:6804/2794571133', 'cluster_addr': '172.21.15.90:6805/2794571133', 'heartbeat_back_addr': '172.21.15.90:6807/2794571133', 'heartbeat_front_addr': '172.21.15.90:6806/2794571133', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'd7b2aeae-d012-41eb-9f91-e4c05c649473', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6808', 'nonce': 3735380744}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 3735380744}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 3735380744}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 3735380744}]}, 'public_addr': '172.21.15.90:6808/3735380744', 'cluster_addr': '172.21.15.90:6809/3735380744', 'heartbeat_back_addr': '172.21.15.90:6811/3735380744', 'heartbeat_front_addr': '172.21.15.90:6810/3735380744', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 102, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6812', 'nonce': 2265615270}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 2265615270}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 2265615270}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 2265615270}]}, 'public_addr': '172.21.15.90:6812/2265615270', 'cluster_addr': '172.21.15.90:6813/2265615270', 'heartbeat_back_addr': '172.21.15.90:6815/2265615270', 'heartbeat_front_addr': '172.21.15.90:6814/2265615270', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:05:36.073777+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:06:05.455859+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:06:26.964593+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:06:32.154761+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:05:38.362254+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:06:01.865452+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:05:37.820222+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:06:26.030494+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:06:27.761807+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:05:23.099371+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:06:04.243811+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:06:34.807756+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.7', 'osds': [0, 9]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/2581408231': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:6800/1136984393': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:6800/2443201864': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/1252671888': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/1451001371': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/4084814866': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:0/4076409285': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/2988977694': '2024-05-16T22:57:38.505263+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-15T23:06:47.656 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-05-15T23:06:47.722 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:06:47.723 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:06:47.840 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 43 15 101129 101114 146.949 102.312 0.01468 0.0068039 2024-05-15T23:06:48.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:47 smithi090 ceph-mon[44434]: pgmap v388: 17 pgs: 17 active+clean; 5.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.17k op/s 2024-05-15T23:06:48.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:47 smithi088 ceph-mon[44699]: pgmap v388: 17 pgs: 17 active+clean; 5.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.17k op/s 2024-05-15T23:06:48.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:47 smithi084 ceph-mon[39784]: pgmap v388: 17 pgs: 17 active+clean; 5.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.17k op/s 2024-05-15T23:06:48.391 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T23:06:48.403 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:06:48.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:48.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:06:48.410 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:48.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:06:48.572 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:48.639 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:06:48.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:48.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:06:48.752 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:48.802 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:48.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:48.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:48.840 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 44 16 102740 102724 145.897 100.625 0.00903658 0.00685234 2024-05-15T23:06:48.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:06:48.943 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:48.976 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:06:48.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:48.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:06:49.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:06:49.123 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:06:49.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:49.159 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:06:49.260 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:06:49.286 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:49.307 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:06:49.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:49.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:06:49.410 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:06:49.438 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:49.459 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:06:49.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:49.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:06:49.565 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:06:49.591 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:49.611 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:06:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:49.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:06:49.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:06:49.762 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:49.789 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:06:49.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:49.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:06:49.840 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 45 16 104275 104259 144.786 95.9375 0.00906859 0.00690543 2024-05-15T23:06:49.897 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:49.944 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:50.014 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:06:50.032 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:50.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:50.048 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:50.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:50 smithi090 ceph-mon[44434]: pgmap v390: 17 pgs: 17 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-05-15T23:06:50.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4275783142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:06:50.145 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:06:50.172 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:50.191 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:06:50.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:50.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:06:50.295 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:06:50.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:50 smithi088 ceph-mon[44699]: pgmap v390: 17 pgs: 17 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-05-15T23:06:50.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4275783142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:06:50.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:49 smithi084 ceph-mon[39784]: pgmap v390: 17 pgs: 17 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-05-15T23:06:50.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:49 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4275783142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:06:50.347 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:50.501 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:06:50.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:50.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:06:50.607 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:06:50.633 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:50.656 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:06:50.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:50.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:06:50.760 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:06:50.832 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:50.840 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 46 16 105797 105781 143.706 95.125 0.0211721 0.0069573 2024-05-15T23:06:51.008 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:51 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:06:51.008 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4275783142' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:06:51.008 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:51 smithi084 ceph-mon[39784]: osdmap e126: 12 total, 12 up, 12 in 2024-05-15T23:06:51.008 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2936525800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:06:51.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:06:51.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:06:51.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:06:51.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:06:51.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:06:51.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:06:51.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:06:51.172 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:06:51.189 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:06:51.216 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:06:51.226 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:06:51.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:51.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:06:51.264 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:06:51.298 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:06:51.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:51 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:06:51.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4275783142' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:06:51.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:51 smithi090 ceph-mon[44434]: osdmap e126: 12 total, 12 up, 12 in 2024-05-15T23:06:51.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2936525800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:06:51.315 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:06:51.315 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:06:51.315 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:06:51.315 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:06:51.315 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:06:51.316 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:06:51.316 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:06:51.316 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:06:51.316 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:06:51.316 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:06:51.316 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:06:51.316 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:06:51.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:51 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:06:51.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4275783142' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:06:51.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:51 smithi088 ceph-mon[44699]: osdmap e126: 12 total, 12 up, 12 in 2024-05-15T23:06:51.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2936525800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:06:51.329 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:06:51.386 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:51.473 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:06:51.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:51.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:06:51.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:06:51.625 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:51.649 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:06:51.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:51.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:06:51.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:06:51.797 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:51.825 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:06:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:51.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:06:51.840 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 47 16 107374 107358 142.746 98.5625 0.00763785 0.00700431 2024-05-15T23:06:51.931 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:06:51.958 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:51.978 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:06:51.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:51.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:06:52.021 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:52 smithi090 ceph-mon[44434]: pgmap v392: 17 pgs: 17 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 113 MiB/s wr, 1.81k op/s 2024-05-15T23:06:52.021 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.021 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:52 smithi090 ceph-mon[44434]: osdmap e127: 12 total, 12 up, 12 in 2024-05-15T23:06:52.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:06:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:52 smithi084 ceph-mon[39784]: pgmap v392: 17 pgs: 17 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 113 MiB/s wr, 1.81k op/s 2024-05-15T23:06:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:52 smithi084 ceph-mon[39784]: osdmap e127: 12 total, 12 up, 12 in 2024-05-15T23:06:52.133 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:52 smithi088 ceph-mon[44699]: pgmap v392: 17 pgs: 17 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 113 MiB/s wr, 1.81k op/s 2024-05-15T23:06:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:52.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:52 smithi088 ceph-mon[44699]: osdmap e127: 12 total, 12 up, 12 in 2024-05-15T23:06:52.657 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:06:52.658 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:06:52.658 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-05-15T23:06:52.658 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T23:06:52.658 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T23:06:52.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 config set filestore_inject_stall 3 2024-05-15T23:06:52.803 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:06:52.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:52.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:06:52.840 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 48 16 108955 108939 141.83 98.8125 0.00628437 0.00704936 2024-05-15T23:06:52.908 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:06:52.950 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:52.978 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:06:52.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:52.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:06:53.074 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:06:53.083 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:06:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:53.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:53.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:53.132 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:06:53.136 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:06:53.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:53.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:06:53.240 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:06:53.268 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:53.290 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:06:53.292 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:53.292 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:53.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:06:53.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:53.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:06:53.394 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:06:53.420 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:53.438 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:06:53.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:53.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:06:53.541 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:53.609 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:53.647 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:53.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:53.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:53.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:06:53.798 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:53.840 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 49 16 110217 110201 140.545 78.875 0.0110557 0.00711377 2024-05-15T23:06:53.999 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:06:53.999 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:06:53.999 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:06:53.999 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:06:53.999 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:06:54.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:54 smithi090 ceph-mon[44434]: pgmap v394: 17 pgs: 17 active+clean; 6.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-05-15T23:06:54.276 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:54.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:54 smithi088 ceph-mon[44699]: pgmap v394: 17 pgs: 17 active+clean; 6.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-05-15T23:06:54.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:54 smithi084 ceph-mon[39784]: pgmap v394: 17 pgs: 17 active+clean; 6.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-05-15T23:06:54.663 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:06:54.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:54.668 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:06:54.737 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T23:06:54.769 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:06:54.797 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:54.819 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:06:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:54.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:06:54.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:06:54.840 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 50 16 111704 111688 139.593 92.9375 0.0125757 0.00716189 2024-05-15T23:06:54.923 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:54.949 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:54.968 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:54.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:54.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:55.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:06:55.114 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:55.193 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:06:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:55.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:06:55.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:06:55.337 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:55.703 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:06:55.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:55.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:06:55.810 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:06:55.840 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 51 16 113132 113116 138.606 89.25 0.00610842 0.00721286 2024-05-15T23:06:55.864 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:56.116 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:56.117 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:06:56.117 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:06:56.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:56 smithi084 ceph-mon[39784]: pgmap v395: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-05-15T23:06:56.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:56 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:06:56.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:56 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:06:56.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:56 smithi084 ceph-mon[39784]: osdmap e128: 12 total, 12 up, 12 in 2024-05-15T23:06:56.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:56.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:06:56.229 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:06:56.256 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:56.278 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:06:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:56.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:06:56.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:56 smithi090 ceph-mon[44434]: pgmap v395: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-05-15T23:06:56.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:56 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:06:56.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:56 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:06:56.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:56 smithi090 ceph-mon[44434]: osdmap e128: 12 total, 12 up, 12 in 2024-05-15T23:06:56.316 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:06:56.317 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-15T23:06:56.317 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:06:56.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:56 smithi088 ceph-mon[44699]: pgmap v395: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-05-15T23:06:56.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:56 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:06:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:56 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:06:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:56 smithi088 ceph-mon[44699]: osdmap e128: 12 total, 12 up, 12 in 2024-05-15T23:06:56.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:06:56.442 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:56.574 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:06:56.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:56.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:06:56.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:06:56.726 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:56.841 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 52 16 114532 114516 137.623 87.5 0.00705292 0.00726451 2024-05-15T23:06:57.083 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:06:57.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:57.087 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:06:57.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:06:57.266 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:57.388 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:06:57.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:57.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:06:57.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:06:57.548 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:57.572 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:06:57.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:57.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:06:57.679 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:06:57.706 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:57.726 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:06:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:57.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:06:57.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:57 smithi090 ceph-mon[44434]: pgmap v397: 17 pgs: 17 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-05-15T23:06:57.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:57 smithi090 ceph-mon[44434]: osdmap e129: 12 total, 12 up, 12 in 2024-05-15T23:06:57.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/530035801' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:06:57.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:57 smithi090 ceph-mon[44434]: from='client.34817 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:06:57.830 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:06:57.841 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 53 16 115996 115980 136.753 91.5 0.00709872 0.0073111 2024-05-15T23:06:57.857 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:57.881 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:06:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:06:57.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:06:57.984 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:06:58.026 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:57 smithi084 ceph-mon[39784]: pgmap v397: 17 pgs: 17 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-05-15T23:06:58.026 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:57 smithi084 ceph-mon[39784]: osdmap e129: 12 total, 12 up, 12 in 2024-05-15T23:06:58.026 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/530035801' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:06:58.026 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:57 smithi084 ceph-mon[39784]: from='client.34817 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:06:58.028 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:06:58.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:57 smithi088 ceph-mon[44699]: pgmap v397: 17 pgs: 17 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-05-15T23:06:58.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:57 smithi088 ceph-mon[44699]: osdmap e129: 12 total, 12 up, 12 in 2024-05-15T23:06:58.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/530035801' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:06:58.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:57 smithi088 ceph-mon[44699]: from='client.34817 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:06:58.441 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:06:58.841 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 54 16 117461 117445 135.916 91.5625 0.00880498 0.00735592 2024-05-15T23:06:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:58 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:06:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:58 smithi090 ceph-mon[44434]: from='client.34817 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:06:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:06:58 smithi090 ceph-mon[44434]: osdmap e130: 12 total, 12 up, 12 in 2024-05-15T23:06:59.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:58 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:06:59.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:58 smithi088 ceph-mon[44699]: from='client.34817 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:06:59.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:06:58 smithi088 ceph-mon[44699]: osdmap e130: 12 total, 12 up, 12 in 2024-05-15T23:06:59.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:58 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:06:59.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:58 smithi084 ceph-mon[39784]: from='client.34817 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:06:59.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:06:58 smithi084 ceph-mon[39784]: osdmap e130: 12 total, 12 up, 12 in 2024-05-15T23:06:59.841 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 55 16 118940 118924 135.125 92.4375 0.00764147 0.00739934 2024-05-15T23:07:00.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:00 smithi084 ceph-mon[39784]: pgmap v400: 17 pgs: 17 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-05-15T23:07:00.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:00 smithi090 ceph-mon[44434]: pgmap v400: 17 pgs: 17 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-05-15T23:07:00.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:00 smithi088 ceph-mon[44699]: pgmap v400: 17 pgs: 17 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-05-15T23:07:00.841 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 56 16 120188 120172 134.105 78 0.00367234 0.00745509 2024-05-15T23:07:01.131 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:07:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:01.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:07:01.236 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:01.264 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:01.288 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:01.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:01.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:01.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:01 smithi084 ceph-mon[39784]: 2.1 scrub starts 2024-05-15T23:07:01.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:01 smithi084 ceph-mon[39784]: 2.4 scrub starts 2024-05-15T23:07:01.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:01 smithi084 ceph-mon[39784]: 2.2 scrub starts 2024-05-15T23:07:01.393 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:01.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:01 smithi088 ceph-mon[44699]: 2.1 scrub starts 2024-05-15T23:07:01.418 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:01 smithi088 ceph-mon[44699]: 2.4 scrub starts 2024-05-15T23:07:01.418 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:01 smithi088 ceph-mon[44699]: 2.2 scrub starts 2024-05-15T23:07:01.420 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:01.441 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:01.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:01.545 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:07:01.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:01 smithi090 ceph-mon[44434]: 2.1 scrub starts 2024-05-15T23:07:01.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:01 smithi090 ceph-mon[44434]: 2.4 scrub starts 2024-05-15T23:07:01.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:01 smithi090 ceph-mon[44434]: 2.2 scrub starts 2024-05-15T23:07:01.579 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:01.601 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:07:01.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:01.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:07:01.705 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:01.734 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:01.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:01.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:01.841 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 57 16 121324 121308 132.998 71 0.00914612 0.00751709 2024-05-15T23:07:01.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:07:01.892 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:01.928 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:07:01.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:01.931 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:07:02.032 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:02.060 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:02.087 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:02.091 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:07:02.091 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:07:02.091 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-15T23:07:02.091 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 1.000000 2024-05-15T23:07:02.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 4 1 2024-05-15T23:07:02.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:02.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:02.193 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:02.220 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:02.241 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:02.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:02.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:02 smithi090 ceph-mon[44434]: pgmap v401: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.59k op/s 2024-05-15T23:07:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:02 smithi090 ceph-mon[44434]: 2.5 scrub starts 2024-05-15T23:07:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:02 smithi090 ceph-mon[44434]: 2.9 scrub starts 2024-05-15T23:07:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:02 smithi090 ceph-mon[44434]: osdmap e131: 12 total, 12 up, 12 in 2024-05-15T23:07:02.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:02 smithi088 ceph-mon[44699]: pgmap v401: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.59k op/s 2024-05-15T23:07:02.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:02 smithi088 ceph-mon[44699]: 2.5 scrub starts 2024-05-15T23:07:02.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:02 smithi088 ceph-mon[44699]: 2.9 scrub starts 2024-05-15T23:07:02.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:02 smithi088 ceph-mon[44699]: osdmap e131: 12 total, 12 up, 12 in 2024-05-15T23:07:02.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:07:02.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:02 smithi084 ceph-mon[39784]: pgmap v401: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.59k op/s 2024-05-15T23:07:02.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:02 smithi084 ceph-mon[39784]: 2.5 scrub starts 2024-05-15T23:07:02.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:02 smithi084 ceph-mon[39784]: 2.9 scrub starts 2024-05-15T23:07:02.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:02 smithi084 ceph-mon[39784]: osdmap e131: 12 total, 12 up, 12 in 2024-05-15T23:07:02.409 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:02.460 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:07:02.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:02.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:07:02.579 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:07:02.606 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:02.628 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:07:02.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:02.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:07:02.732 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:02.761 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:02.784 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:02.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:02.841 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 58 16 122483 122467 131.954 72.4375 0.00553095 0.00757688 2024-05-15T23:07:02.888 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:07:02.924 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:02.924 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:07:02.924 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:07:02.924 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:07:02.924 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:07:02.924 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:07:02.924 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:07:02.925 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:07:02.925 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:07:02.925 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:07:02.928 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:07:02.946 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:07:02.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:02.949 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:07:02.970 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:07:03.006 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:07:03.018 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:07:03.018 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:07:03.018 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:07:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:07:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:07:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:07:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:07:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:07:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:07:03.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:07:03.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:07:03.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:07:03.050 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:03.078 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:03.100 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:03.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:03.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:03.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:03.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:03 smithi090 ceph-mon[44434]: 2.1 scrub ok 2024-05-15T23:07:03.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:03 smithi090 ceph-mon[44434]: 2.2 scrub ok 2024-05-15T23:07:03.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:03 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:07:03.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:03 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:07:03.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:03 smithi090 ceph-mon[44434]: 2.5 scrub ok 2024-05-15T23:07:03.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:03 smithi090 ceph-mon[44434]: 2.4 scrub ok 2024-05-15T23:07:03.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/177148268' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:03.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:03.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:03.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:03 smithi090 ceph-mon[44434]: 2.0 scrub starts 2024-05-15T23:07:03.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:03.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:03.263 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:03.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:03.270 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:03.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:07:03.410 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:03.447 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:07:03.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:03.457 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:07:03.558 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:07:03.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:03 smithi088 ceph-mon[44699]: 2.1 scrub ok 2024-05-15T23:07:03.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:03 smithi088 ceph-mon[44699]: 2.2 scrub ok 2024-05-15T23:07:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:03 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:07:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:03 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:07:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:03 smithi088 ceph-mon[44699]: 2.5 scrub ok 2024-05-15T23:07:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:03 smithi088 ceph-mon[44699]: 2.4 scrub ok 2024-05-15T23:07:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/177148268' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:03 smithi088 ceph-mon[44699]: 2.0 scrub starts 2024-05-15T23:07:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:03.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:03 smithi084 ceph-mon[39784]: 2.1 scrub ok 2024-05-15T23:07:03.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:03 smithi084 ceph-mon[39784]: 2.2 scrub ok 2024-05-15T23:07:03.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:03 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:07:03.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:03 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:07:03.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:03 smithi084 ceph-mon[39784]: 2.5 scrub ok 2024-05-15T23:07:03.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:03 smithi084 ceph-mon[39784]: 2.4 scrub ok 2024-05-15T23:07:03.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/177148268' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:03.606 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:03.606 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:03.606 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:03 smithi084 ceph-mon[39784]: 2.0 scrub starts 2024-05-15T23:07:03.606 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:03.606 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:03.609 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:03.752 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:07:03.752 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:07:03.841 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 59 16 123746 123730 131.055 78.9375 0.00368695 0.00762813 2024-05-15T23:07:04.495 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:07:04.498 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.498 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:04 smithi084 ceph-mon[39784]: pgmap v403: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-05-15T23:07:04.498 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.498 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.498 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.498 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.498 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:04 smithi084 ceph-mon[39784]: osdmap e132: 12 total, 12 up, 12 in 2024-05-15T23:07:04.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:04 smithi084 ceph-mon[39784]: 2.9 scrub ok 2024-05-15T23:07:04.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:04 smithi084 ceph-mon[39784]: 2.0 scrub ok 2024-05-15T23:07:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:04.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:07:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:04 smithi090 ceph-mon[44434]: pgmap v403: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-05-15T23:07:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:04 smithi090 ceph-mon[44434]: osdmap e132: 12 total, 12 up, 12 in 2024-05-15T23:07:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:04 smithi090 ceph-mon[44434]: 2.9 scrub ok 2024-05-15T23:07:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:04 smithi090 ceph-mon[44434]: 2.0 scrub ok 2024-05-15T23:07:04.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:04 smithi088 ceph-mon[44699]: pgmap v403: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-05-15T23:07:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:04 smithi088 ceph-mon[44699]: osdmap e132: 12 total, 12 up, 12 in 2024-05-15T23:07:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:04 smithi088 ceph-mon[44699]: 2.9 scrub ok 2024-05-15T23:07:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:04 smithi088 ceph-mon[44699]: 2.0 scrub ok 2024-05-15T23:07:04.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:07:04.634 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:04.671 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:07:04.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:04.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:07:04.774 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:07:04.802 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:04.824 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:07:04.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:04.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:07:04.841 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:07:04.841486 min lat: 0.000985234 max lat: 2.1501 avg lat: 0.00767442 2024-05-15T23:07:04.841 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:07:04.841 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 60 16 125100 125084 130.281 84.625 0.00478322 0.00767442 2024-05-15T23:07:04.928 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:04.954 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:04.974 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:04.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:04.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:04.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T23:07:05.078 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:07:05.148 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:05.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:05 smithi084 ceph-mon[39784]: 2.e scrub starts 2024-05-15T23:07:05.335 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:07:05.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:05.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:07:05.418 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:05.428 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:05.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:07:05.470 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:05 smithi090 ceph-mon[44434]: 2.e scrub starts 2024-05-15T23:07:05.472 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:05.498 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:07:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:05.505 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:07:05.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:05 smithi088 ceph-mon[44699]: 2.e scrub starts 2024-05-15T23:07:05.606 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:05.639 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:05.662 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:05.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:05.765 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:05.796 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:05.815 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:05.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:05.841 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 61 16 126366 126350 129.442 79.125 0.00595295 0.00772388 2024-05-15T23:07:05.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:07:05.959 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:05.999 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:07:06.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:06.005 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:07:06.106 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:07:06.133 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:06.158 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:07:06.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:06.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:07:06.263 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:06.290 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:06 smithi090 ceph-mon[44434]: pgmap v405: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-05-15T23:07:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:06 smithi090 ceph-mon[44434]: 2.e scrub ok 2024-05-15T23:07:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:06.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:06.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:06 smithi084 ceph-mon[39784]: pgmap v405: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-05-15T23:07:06.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:06 smithi084 ceph-mon[39784]: 2.e scrub ok 2024-05-15T23:07:06.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:07:06.471 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:06.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:07:06.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:06 smithi088 ceph-mon[44699]: pgmap v405: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-05-15T23:07:06.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:06 smithi088 ceph-mon[44699]: 2.e scrub ok 2024-05-15T23:07:06.841 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 62 15 127722 127707 128.722 84.8125 0.0158187 0.00776729 2024-05-15T23:07:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:07:06.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:06.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:07:06.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:07:06.994 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:07.376 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:07:07.767 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:07:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:07.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:07:07.841 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 63 16 128955 128939 127.901 77 0.00665999 0.00781736 2024-05-15T23:07:07.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:07:07.929 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:07 smithi090 ceph-mon[44434]: 2.a scrub starts 2024-05-15T23:07:07.929 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:07 smithi090 ceph-mon[44434]: pgmap v406: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.56k op/s 2024-05-15T23:07:07.929 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:07 smithi090 ceph-mon[44434]: osdmap e133: 12 total, 12 up, 12 in 2024-05-15T23:07:07.937 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:07.963 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:07:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:07.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:07:08.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:07 smithi084 ceph-mon[39784]: 2.a scrub starts 2024-05-15T23:07:08.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:07 smithi084 ceph-mon[39784]: pgmap v406: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.56k op/s 2024-05-15T23:07:08.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:07 smithi084 ceph-mon[39784]: osdmap e133: 12 total, 12 up, 12 in 2024-05-15T23:07:08.070 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:07:08.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:07 smithi088 ceph-mon[44699]: 2.a scrub starts 2024-05-15T23:07:08.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:07 smithi088 ceph-mon[44699]: pgmap v406: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.56k op/s 2024-05-15T23:07:08.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:07 smithi088 ceph-mon[44699]: osdmap e133: 12 total, 12 up, 12 in 2024-05-15T23:07:08.096 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:08.116 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:07:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:08.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:07:08.220 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:07:08.273 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:08.531 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:07:08.531 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:07:08.542 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:08.542 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:08.654 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:07:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:08.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:07:08.763 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:07:08.789 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:07:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:08.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:07:08.842 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 64 16 130076 130060 126.998 70.0625 0.00467221 0.00787255 2024-05-15T23:07:08.913 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:07:08.968 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:08 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:07:08.968 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:08 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:07:08.969 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:08 smithi084 ceph-mon[39784]: 2.f scrub starts 2024-05-15T23:07:08.969 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:08 smithi084 ceph-mon[39784]: 2.7 scrub starts 2024-05-15T23:07:08.969 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:08 smithi084 ceph-mon[39784]: 2.a scrub ok 2024-05-15T23:07:08.969 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:08 smithi084 ceph-mon[39784]: osdmap e134: 12 total, 12 up, 12 in 2024-05-15T23:07:08.969 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:09.053 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:07:09.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:08 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:07:09.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:08 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:07:09.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:08 smithi090 ceph-mon[44434]: 2.f scrub starts 2024-05-15T23:07:09.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:08 smithi090 ceph-mon[44434]: 2.7 scrub starts 2024-05-15T23:07:09.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:08 smithi090 ceph-mon[44434]: 2.a scrub ok 2024-05-15T23:07:09.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:08 smithi090 ceph-mon[44434]: osdmap e134: 12 total, 12 up, 12 in 2024-05-15T23:07:09.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:09.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:07:09.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:08 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:07:09.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:08 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:07:09.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:08 smithi088 ceph-mon[44699]: 2.f scrub starts 2024-05-15T23:07:09.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:08 smithi088 ceph-mon[44699]: 2.7 scrub starts 2024-05-15T23:07:09.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:08 smithi088 ceph-mon[44699]: 2.a scrub ok 2024-05-15T23:07:09.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:08 smithi088 ceph-mon[44699]: osdmap e134: 12 total, 12 up, 12 in 2024-05-15T23:07:09.157 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:07:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:09.236 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:07:09.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:09.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:07:09.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:07:09.407 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:09.546 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:07:09.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:09.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:07:09.567 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T23:07:09.648 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:09.676 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:09.695 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:09.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:09.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:09.799 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:07:09.835 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:09.842 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 65 16 131257 131241 126.179 73.8125 0.00887223 0.0079235 2024-05-15T23:07:09.873 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:07:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:09.879 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:07:09.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:07:10.037 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:10.094 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:07:10.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:10.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:07:10.193 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:10 smithi084 ceph-mon[39784]: pgmap v409: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-05-15T23:07:10.194 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:10 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2166434998' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:07:10.194 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:10 smithi084 ceph-mon[39784]: from='client.34826 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:07:10.194 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:10 smithi084 ceph-mon[39784]: 2.f scrub ok 2024-05-15T23:07:10.196 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:10 smithi090 ceph-mon[44434]: pgmap v409: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-05-15T23:07:10.196 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:10 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2166434998' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:07:10.196 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:10 smithi090 ceph-mon[44434]: from='client.34826 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:07:10.196 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:10 smithi090 ceph-mon[44434]: 2.f scrub ok 2024-05-15T23:07:10.197 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:07:10.211 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-05-15T23:07:10.225 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:10.247 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:07:10.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:10.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:07:10.351 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:07:10.382 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:10.411 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:07:10.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:10.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:07:10.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:07:10.560 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:10.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:10 smithi088 ceph-mon[44699]: pgmap v409: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-05-15T23:07:10.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:10 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2166434998' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:07:10.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:10 smithi088 ceph-mon[44699]: from='client.34826 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:07:10.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:10 smithi088 ceph-mon[44699]: 2.f scrub ok 2024-05-15T23:07:10.584 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:07:10.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:10.588 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:07:10.689 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:10.718 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:10.738 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:10.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:10.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:10.842 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 66 16 132597 132581 125.536 83.75 0.0151067 0.00796476 2024-05-15T23:07:10.843 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:07:10.891 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:10.924 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:07:10.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:10.931 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:07:11.032 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:07:11.060 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:11.097 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:07:11.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:11.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:07:11.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:07:11.255 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:11.288 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:07:11.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:11.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:07:11.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:11 smithi084 ceph-mon[39784]: 2.7 scrub ok 2024-05-15T23:07:11.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:11 smithi084 ceph-mon[39784]: from='client.34826 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:07:11.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:11 smithi084 ceph-mon[39784]: osdmap e135: 12 total, 12 up, 12 in 2024-05-15T23:07:11.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:07:11.450 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:11.486 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:07:11.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:11.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:07:11.491 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:11 smithi090 ceph-mon[44434]: 2.7 scrub ok 2024-05-15T23:07:11.492 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:11 smithi090 ceph-mon[44434]: from='client.34826 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:07:11.492 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:11 smithi090 ceph-mon[44434]: osdmap e135: 12 total, 12 up, 12 in 2024-05-15T23:07:11.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:11 smithi088 ceph-mon[44699]: 2.7 scrub ok 2024-05-15T23:07:11.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:11 smithi088 ceph-mon[44699]: from='client.34826 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:07:11.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:11 smithi088 ceph-mon[44699]: osdmap e135: 12 total, 12 up, 12 in 2024-05-15T23:07:11.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:07:11.645 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:11.678 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:07:11.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:11.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:07:11.787 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:11.815 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:11.837 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:11.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:11.842 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 67 16 133753 133737 124.741 72.25 0.0044761 0.00801499 2024-05-15T23:07:11.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:07:11.985 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:12.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:12 smithi084 ceph-mon[39784]: pgmap v411: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-05-15T23:07:12.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:12 smithi084 ceph-mon[39784]: 2.b scrub starts 2024-05-15T23:07:12.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:12 smithi090 ceph-mon[44434]: pgmap v411: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-05-15T23:07:12.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:12 smithi090 ceph-mon[44434]: 2.b scrub starts 2024-05-15T23:07:12.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:12 smithi088 ceph-mon[44699]: pgmap v411: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-05-15T23:07:12.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:12 smithi088 ceph-mon[44699]: 2.b scrub starts 2024-05-15T23:07:12.842 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 68 16 135030 135014 124.08 79.8125 0.00485643 0.00805777 2024-05-15T23:07:12.873 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:07:12.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:12.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:07:12.979 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:07:13.008 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:13.030 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:07:13.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:13.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:07:13.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:07:13.171 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:13.198 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:07:13.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:13.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:07:13.261 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:07:13.301 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:07:13.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:13 smithi088 ceph-mon[44699]: 2.b scrub ok 2024-05-15T23:07:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/457061912' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:07:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:13 smithi088 ceph-mon[44699]: from='client.34832 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:07:13.367 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:13.416 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:07:13.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:13.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:07:13.521 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:13.548 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:13 smithi090 ceph-mon[44434]: 2.b scrub ok 2024-05-15T23:07:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/457061912' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:07:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:13 smithi090 ceph-mon[44434]: from='client.34832 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:07:13.568 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:13.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:13 smithi084 ceph-mon[39784]: 2.b scrub ok 2024-05-15T23:07:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/457061912' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:07:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:13 smithi084 ceph-mon[39784]: from='client.34832 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:07:13.671 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:07:13.738 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:13.844 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 69 16 136289 136273 123.417 78.6875 0.0142687 0.00810104 2024-05-15T23:07:13.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:07:13.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:07:13.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:07:13.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:07:13.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:07:13.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:07:13.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:07:13.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:07:13.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:07:13.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:07:13.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:07:13.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:07:13.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:07:13.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:07:13.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:07:13.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:07:13.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:07:13.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:07:13.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:07:13.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:07:13.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:07:13.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:07:13.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:07:13.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:07:14.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: pgmap v412: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-05-15T23:07:14.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:07:14.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:07:14.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: from='client.34832 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:07:14.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: osdmap e136: 12 total, 12 up, 12 in 2024-05-15T23:07:14.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1186789589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:14.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:07:14.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:14 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:07:14.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: pgmap v412: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-05-15T23:07:14.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:07:14.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:07:14.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: from='client.34832 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:07:14.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: osdmap e136: 12 total, 12 up, 12 in 2024-05-15T23:07:14.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1186789589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:14.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:07:14.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:14 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:07:14.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: pgmap v412: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: from='client.34832 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: osdmap e136: 12 total, 12 up, 12 in 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1186789589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:07:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:14 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:07:14.845 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 70 16 137642 137626 122.862 84.5625 0.0109948 0.00813747 2024-05-15T23:07:15.018 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:07:15.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:15.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:07:15.056 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:15.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:07:15.178 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:15.209 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:07:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:15.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:07:15.315 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:07:15.348 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:15.348 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:15.348 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:15.348 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:15 smithi084 ceph-mon[39784]: osdmap e137: 12 total, 12 up, 12 in 2024-05-15T23:07:15.360 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:15.403 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:07:15.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:15.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:07:15.513 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:07:15.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:15.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:15.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:15.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:15 smithi090 ceph-mon[44434]: osdmap e137: 12 total, 12 up, 12 in 2024-05-15T23:07:15.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:15.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:15.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:15.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:15 smithi088 ceph-mon[44699]: osdmap e137: 12 total, 12 up, 12 in 2024-05-15T23:07:15.572 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:15.660 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:07:15.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:15.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:07:15.765 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:15.794 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:15.816 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:15.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:15.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:15.845 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 71 16 138926 138910 122.262 80.25 0.00964223 0.00817805 2024-05-15T23:07:15.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:15.964 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:15.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:15.998 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:16.099 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:07:16.126 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:16.151 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:07:16.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:16.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:07:16.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:07:16.264 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:16 smithi090 ceph-mon[44434]: pgmap v415: 17 pgs: 17 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-05-15T23:07:16.264 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:16 smithi090 ceph-mon[44434]: 2.d scrub starts 2024-05-15T23:07:16.308 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:16.336 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:07:16.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:16.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:07:16.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:16.478 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:16.510 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:16.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:16.547 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:16 smithi084 ceph-mon[39784]: pgmap v415: 17 pgs: 17 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-05-15T23:07:16.547 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:16 smithi084 ceph-mon[39784]: 2.d scrub starts 2024-05-15T23:07:16.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:07:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:16 smithi088 ceph-mon[44699]: pgmap v415: 17 pgs: 17 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-05-15T23:07:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:16 smithi088 ceph-mon[44699]: 2.d scrub starts 2024-05-15T23:07:16.616 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:07:16.701 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:16.845 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 72 16 140105 140089 121.587 73.6875 0.00985158 0.0082229 2024-05-15T23:07:17.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:07:17.283 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:07:17.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:17.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:07:17.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:07:17.425 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:17.469 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:07:17.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:17.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:07:17.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:17.628 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:17.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:17.648 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:17.662 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:17.669 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:07:17.669 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:07:17.669 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:07:17.669 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:07:17.669 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:07:17.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:07:17.770 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:17.799 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:07:17.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:17.804 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:07:17.845 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 73 16 141233 141217 120.888 70.5 0.0115667 0.00827077 2024-05-15T23:07:17.905 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:07:17.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:17 smithi084 ceph-mon[39784]: pgmap v416: 17 pgs: 17 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T23:07:17.977 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:17 smithi084 ceph-mon[39784]: 2.3 scrub starts 2024-05-15T23:07:17.977 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:17 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:07:17.977 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:17 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:07:17.986 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:17 smithi090 ceph-mon[44434]: pgmap v416: 17 pgs: 17 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T23:07:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:17 smithi090 ceph-mon[44434]: 2.3 scrub starts 2024-05-15T23:07:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:17 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:07:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:17 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:07:18.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:17 smithi088 ceph-mon[44699]: pgmap v416: 17 pgs: 17 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T23:07:18.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:17 smithi088 ceph-mon[44699]: 2.3 scrub starts 2024-05-15T23:07:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:17 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:07:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:17 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:07:18.463 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:07:18.463 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:07:18.463 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-05-15T23:07:18.463 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-05-15T23:07:18.849 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 74 16 142482 142466 120.302 78.0625 0.0121883 0.00831095 2024-05-15T23:07:19.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:18 smithi090 ceph-mon[44434]: 2.d scrub ok 2024-05-15T23:07:19.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:18 smithi090 ceph-mon[44434]: osdmap e138: 12 total, 12 up, 12 in 2024-05-15T23:07:19.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:18 smithi088 ceph-mon[44699]: 2.d scrub ok 2024-05-15T23:07:19.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:18 smithi088 ceph-mon[44699]: osdmap e138: 12 total, 12 up, 12 in 2024-05-15T23:07:19.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:18 smithi084 ceph-mon[39784]: 2.d scrub ok 2024-05-15T23:07:19.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:18 smithi084 ceph-mon[39784]: osdmap e138: 12 total, 12 up, 12 in 2024-05-15T23:07:19.849 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 75 16 143744 143728 119.75 78.875 0.0135226 0.00834931 2024-05-15T23:07:20.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:19 smithi090 ceph-mon[44434]: 2.3 scrub ok 2024-05-15T23:07:20.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:19 smithi090 ceph-mon[44434]: pgmap v418: 17 pgs: 17 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-05-15T23:07:20.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:19 smithi090 ceph-mon[44434]: 2.c scrub starts 2024-05-15T23:07:20.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:19 smithi088 ceph-mon[44699]: 2.3 scrub ok 2024-05-15T23:07:20.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:19 smithi088 ceph-mon[44699]: pgmap v418: 17 pgs: 17 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-05-15T23:07:20.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:19 smithi088 ceph-mon[44699]: 2.c scrub starts 2024-05-15T23:07:20.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:19 smithi084 ceph-mon[39784]: 2.3 scrub ok 2024-05-15T23:07:20.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:19 smithi084 ceph-mon[39784]: pgmap v418: 17 pgs: 17 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-05-15T23:07:20.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:19 smithi084 ceph-mon[39784]: 2.c scrub starts 2024-05-15T23:07:20.200 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T23:07:20.450 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:07:20.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:20.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:07:20.555 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:20.582 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:20.604 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:20.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:20.708 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:20.734 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:20.754 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:20.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:20.849 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 76 16 144875 144859 119.104 70.6875 0.00982114 0.00839429 2024-05-15T23:07:20.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:07:20.909 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:20.951 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:07:20.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:20.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:07:21.060 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:07:21.099 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:21.127 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:07:21.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:21.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:07:21.180 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:21.237 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:07:21.263 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:21.270 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:07:21.282 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:07:21.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:21.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:07:21.315 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:07:21.353 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:07:21.380 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:07:21.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:07:21.412 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:07:21.426 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:21.449 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:07:21.462 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:07:21.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:21.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:07:21.481 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:07:21.519 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:07:21.550 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:07:21.567 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:07:21.588 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:07:21.620 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:21.634 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:07:21.665 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:07:21.684 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:07:21.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:07:21.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:07:21.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:07:21.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:07:21.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:07:21.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:07:21.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:07:21.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:07:21.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:07:21.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:07:21.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:07:21.849 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 77 16 145838 145822 118.339 60.1875 0.0333189 0.00844842 2024-05-15T23:07:22.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:21 smithi090 ceph-mon[44434]: 2.6 scrub starts 2024-05-15T23:07:22.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:21 smithi090 ceph-mon[44434]: pgmap v419: 17 pgs: 17 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-05-15T23:07:22.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:21 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2577388208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:22.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:22.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:21 smithi090 ceph-mon[44434]: 2.c scrub ok 2024-05-15T23:07:22.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:22.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:22.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:21 smithi088 ceph-mon[44699]: 2.6 scrub starts 2024-05-15T23:07:22.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:21 smithi088 ceph-mon[44699]: pgmap v419: 17 pgs: 17 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-05-15T23:07:22.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:21 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2577388208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:21 smithi088 ceph-mon[44699]: 2.c scrub ok 2024-05-15T23:07:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:21 smithi084 ceph-mon[39784]: 2.6 scrub starts 2024-05-15T23:07:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:21 smithi084 ceph-mon[39784]: pgmap v419: 17 pgs: 17 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-05-15T23:07:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:21 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2577388208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:21 smithi084 ceph-mon[39784]: 2.c scrub ok 2024-05-15T23:07:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:22.849 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 78 16 146988 146972 117.744 71.875 0.0176085 0.00849163 2024-05-15T23:07:23.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4224385786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:07:23.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:22 smithi090 ceph-mon[44434]: from='client.34850 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:07:23.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4224385786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:07:23.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:22 smithi088 ceph-mon[44699]: from='client.34850 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:07:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4224385786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:07:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:22 smithi084 ceph-mon[39784]: from='client.34850 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:07:23.849 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 79 16 148190 148174 117.204 75.125 0.0119031 0.00853026 2024-05-15T23:07:24.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:23 smithi090 ceph-mon[44434]: 2.6 scrub ok 2024-05-15T23:07:24.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:23 smithi090 ceph-mon[44434]: 2.8 scrub starts 2024-05-15T23:07:24.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:23 smithi090 ceph-mon[44434]: pgmap v420: 17 pgs: 17 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-05-15T23:07:24.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:23 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:07:24.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:23 smithi090 ceph-mon[44434]: from='client.34850 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:07:24.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:23 smithi090 ceph-mon[44434]: osdmap e139: 12 total, 12 up, 12 in 2024-05-15T23:07:24.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:23 smithi088 ceph-mon[44699]: 2.6 scrub ok 2024-05-15T23:07:24.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:23 smithi088 ceph-mon[44699]: 2.8 scrub starts 2024-05-15T23:07:24.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:23 smithi088 ceph-mon[44699]: pgmap v420: 17 pgs: 17 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-05-15T23:07:24.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:23 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:07:24.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:23 smithi088 ceph-mon[44699]: from='client.34850 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:07:24.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:23 smithi088 ceph-mon[44699]: osdmap e139: 12 total, 12 up, 12 in 2024-05-15T23:07:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:23 smithi084 ceph-mon[39784]: 2.6 scrub ok 2024-05-15T23:07:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:23 smithi084 ceph-mon[39784]: 2.8 scrub starts 2024-05-15T23:07:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:23 smithi084 ceph-mon[39784]: pgmap v420: 17 pgs: 17 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-05-15T23:07:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:23 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:07:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:23 smithi084 ceph-mon[39784]: from='client.34850 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:07:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:23 smithi084 ceph-mon[39784]: osdmap e139: 12 total, 12 up, 12 in 2024-05-15T23:07:24.313 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:07:24.850 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:07:24.849709 min lat: 0.000985234 max lat: 2.1501 avg lat: 0.00856405 2024-05-15T23:07:24.850 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:07:24.850 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 80 16 149479 149463 116.746 80.5625 0.0179077 0.00856405 2024-05-15T23:07:25.850 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 81 16 150705 150689 116.251 76.625 0.00987937 0.0086008 2024-05-15T23:07:26.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:25 smithi090 ceph-mon[44434]: pgmap v422: 17 pgs: 17 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-05-15T23:07:26.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:25 smithi088 ceph-mon[44699]: pgmap v422: 17 pgs: 17 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-05-15T23:07:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:25 smithi084 ceph-mon[39784]: pgmap v422: 17 pgs: 17 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-05-15T23:07:26.613 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:07:26.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:26.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:07:26.721 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:07:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:07:26.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:26.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:07:26.850 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 82 16 151860 151844 115.713 72.1875 0.00482915 0.00863974 2024-05-15T23:07:26.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:07:26.946 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:27.850 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 83 16 153079 153063 115.237 76.1875 0.00467329 0.0086756 2024-05-15T23:07:28.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:27 smithi090 ceph-mon[44434]: pgmap v423: 17 pgs: 17 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-05-15T23:07:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:27 smithi088 ceph-mon[44699]: pgmap v423: 17 pgs: 17 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-05-15T23:07:28.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:27 smithi084 ceph-mon[39784]: pgmap v423: 17 pgs: 17 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-05-15T23:07:28.850 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 84 15 154340 154325 114.804 78.875 0.0198276 0.00870936 2024-05-15T23:07:29.850 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 85 16 155572 155556 114.359 76.9375 0.0124124 0.00874243 2024-05-15T23:07:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:29 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:07:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:29 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:07:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:29 smithi090 ceph-mon[44434]: pgmap v424: 17 pgs: 17 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T23:07:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:29 smithi090 ceph-mon[44434]: osdmap e140: 12 total, 12 up, 12 in 2024-05-15T23:07:30.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:29 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:07:30.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:29 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:07:30.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:29 smithi088 ceph-mon[44699]: pgmap v424: 17 pgs: 17 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T23:07:30.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:29 smithi088 ceph-mon[44699]: osdmap e140: 12 total, 12 up, 12 in 2024-05-15T23:07:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:29 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:07:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:29 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:07:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:29 smithi084 ceph-mon[39784]: pgmap v424: 17 pgs: 17 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T23:07:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:29 smithi084 ceph-mon[39784]: osdmap e140: 12 total, 12 up, 12 in 2024-05-15T23:07:30.850 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 86 16 156768 156752 113.898 74.75 0.0133709 0.00877845 2024-05-15T23:07:31.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:30 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:07:31.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:30 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:07:31.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:30 smithi090 ceph-mon[44434]: osdmap e141: 12 total, 12 up, 12 in 2024-05-15T23:07:31.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:30 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:07:31.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:30 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:07:31.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:30 smithi084 ceph-mon[39784]: osdmap e141: 12 total, 12 up, 12 in 2024-05-15T23:07:31.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:30 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:07:31.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:30 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:07:31.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:30 smithi088 ceph-mon[44699]: osdmap e141: 12 total, 12 up, 12 in 2024-05-15T23:07:31.850 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 87 16 158004 157988 113.477 77.25 0.010852 0.00881124 2024-05-15T23:07:32.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:31 smithi090 ceph-mon[44434]: pgmap v427: 17 pgs: 17 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-05-15T23:07:32.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:31 smithi088 ceph-mon[44699]: pgmap v427: 17 pgs: 17 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-05-15T23:07:32.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:31 smithi084 ceph-mon[39784]: pgmap v427: 17 pgs: 17 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-05-15T23:07:32.850 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 88 16 159142 159126 112.995 71.125 0.00484142 0.00884864 2024-05-15T23:07:32.918 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:07:32.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:32.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:07:33.027 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:33.054 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:33.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:07:33.074 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:33.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:33.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:33.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:07:33.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:07:33.225 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:33.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:07:33.553 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:33.556 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:07:33.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:33.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:07:33.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:07:33.756 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:33.802 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:07:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:33.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:07:33.850 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 89 16 160251 160235 112.505 69.3125 0.00958974 0.00888699 2024-05-15T23:07:33.912 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:33.939 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:33.960 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:33.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:33.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:34.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:07:34.144 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:34.191 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:07:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:34.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:07:34.296 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:07:34.350 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:34.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:34 smithi084 ceph-mon[39784]: pgmap v428: 17 pgs: 17 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-05-15T23:07:34.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:34 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:07:34.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:34 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:07:34.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:34 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:07:34.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:34 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:07:34.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:34 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:07:34.377 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:07:34.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:34.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:07:34.483 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:07:34.509 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:34 smithi088 ceph-mon[44699]: pgmap v428: 17 pgs: 17 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-05-15T23:07:34.509 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:34 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:07:34.509 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:34 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:07:34.509 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:34 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:07:34.509 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:34 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:07:34.509 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:34 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:07:34.510 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:34.533 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:07:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:34.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:07:34.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:34 smithi090 ceph-mon[44434]: pgmap v428: 17 pgs: 17 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-05-15T23:07:34.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:34 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:07:34.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:34 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:07:34.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:34 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:07:34.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:34 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:07:34.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:34 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:07:34.638 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:07:34.689 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:34.850 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total time run: 90.011 2024-05-15T23:07:34.851 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total writes made: 161355 2024-05-15T23:07:34.851 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Write size: 65536 2024-05-15T23:07:34.851 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object size: 65536 2024-05-15T23:07:34.851 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Bandwidth (MB/sec): 112.038 2024-05-15T23:07:34.851 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Bandwidth: 58.2776 2024-05-15T23:07:34.851 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max bandwidth (MB/sec): 376.5 2024-05-15T23:07:34.851 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min bandwidth (MB/sec): 0 2024-05-15T23:07:34.851 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average IOPS: 1792 2024-05-15T23:07:34.851 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev IOPS: 932.444 2024-05-15T23:07:34.851 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max IOPS: 6024 2024-05-15T23:07:34.851 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min IOPS: 0 2024-05-15T23:07:34.851 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average Latency(s): 0.00892489 2024-05-15T23:07:34.851 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Latency(s): 0.024132 2024-05-15T23:07:34.852 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max latency(s): 2.1501 2024-05-15T23:07:34.852 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min latency(s): 0.000985234 2024-05-15T23:07:34.856 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T23:07:34.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:07:35.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-05-15T23:07:35.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:07:35.855 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:07:35.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:35.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:07:35.964 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:07:36.031 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:36.032 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:36.042 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:36.170 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:07:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:36.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:07:36.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:07:36.319 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:36.344 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:07:36.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:36.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:07:36.448 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:07:36.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:36 smithi084 ceph-mon[39784]: pgmap v429: 17 pgs: 17 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T23:07:36.512 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:36 smithi090 ceph-mon[44434]: pgmap v429: 17 pgs: 17 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T23:07:36.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:36 smithi088 ceph-mon[44699]: pgmap v429: 17 pgs: 17 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T23:07:36.653 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:07:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:36.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:07:36.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:07:36.811 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:36.856 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:07:36.856 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:07:36.864 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:07:36.868 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:36.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:36.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:07:36.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:07:37.028 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:37.058 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:07:37.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:37.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:07:37.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:07:37.198 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:37.228 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:07:37.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:37.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:07:37.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:37.378 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:37.402 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:37.405 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:37.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:37.555 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:37.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:37.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:07:37.732 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:37.765 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:07:37.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:37.768 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:07:37.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:37.911 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:37.943 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:37.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:37.946 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:38.047 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:38.079 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:38.101 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:38.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:38.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:38.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:07:38.240 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:38.531 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:38 smithi084 ceph-mon[39784]: pgmap v430: 17 pgs: 17 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 85 MiB/s wr, 1.39k op/s 2024-05-15T23:07:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:38 smithi090 ceph-mon[44434]: pgmap v430: 17 pgs: 17 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 85 MiB/s wr, 1.39k op/s 2024-05-15T23:07:38.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:38 smithi088 ceph-mon[44699]: pgmap v430: 17 pgs: 17 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 85 MiB/s wr, 1.39k op/s 2024-05-15T23:07:39.736 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T23:07:40.002 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:07:40.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:40.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:07:40.043 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:07:40.106 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:40.158 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:40.200 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:40.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:40.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:40.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:07:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:40 smithi084 ceph-mon[39784]: pgmap v431: 17 pgs: 17 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s rd, 71 MiB/s wr, 1.20k op/s 2024-05-15T23:07:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:40 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:07:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:40 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:07:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:40 smithi084 ceph-mon[39784]: osdmap e142: 12 total, 12 up, 12 in 2024-05-15T23:07:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:40 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1564612143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:40.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:40.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:40.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:40.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:40.370 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:40.460 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:07:40.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:40.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:07:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:40 smithi090 ceph-mon[44434]: pgmap v431: 17 pgs: 17 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s rd, 71 MiB/s wr, 1.20k op/s 2024-05-15T23:07:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:40 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:07:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:40 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:07:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:40 smithi090 ceph-mon[44434]: osdmap e142: 12 total, 12 up, 12 in 2024-05-15T23:07:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:40 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1564612143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:40.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:07:40.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:07:40.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:07:40.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:07:40.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:07:40.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:07:40.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:07:40.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:07:40.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:07:40.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:07:40.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:07:40.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:07:40.560 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:07:40.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:07:40.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:07:40.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:07:40.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:07:40.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:07:40.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:07:40.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:07:40.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:07:40.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:07:40.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:07:40.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:07:40.568 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:40 smithi088 ceph-mon[44699]: pgmap v431: 17 pgs: 17 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s rd, 71 MiB/s wr, 1.20k op/s 2024-05-15T23:07:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:40 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:07:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:40 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:07:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:40 smithi088 ceph-mon[44699]: osdmap e142: 12 total, 12 up, 12 in 2024-05-15T23:07:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:40 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1564612143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:40.600 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:41.051 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:41.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:41.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:41.155 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:07:41.198 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:41.249 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:07:41.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:41.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:07:41.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:41.393 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:41.418 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:41.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:41.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:41.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:07:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:41.571 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:41.617 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:07:41.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:41.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:07:41.724 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:07:41.753 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:41.792 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:07:41.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:41.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:07:41.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:07:41.952 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:42.117 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:07:42.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:42.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:07:42.123 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:07:42.123 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:07:42.124 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2024-05-15T23:07:42.124 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:07:42.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:42.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:42 smithi084 ceph-mon[39784]: pgmap v433: 17 pgs: 17 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 42 MiB/s wr, 1.16k op/s 2024-05-15T23:07:42.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:42 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:07:42.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:42 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:07:42.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:42 smithi084 ceph-mon[39784]: osdmap e143: 12 total, 12 up, 12 in 2024-05-15T23:07:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:42.289 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:42.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:42.299 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:42.400 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:42.431 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:42 smithi088 ceph-mon[44699]: pgmap v433: 17 pgs: 17 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 42 MiB/s wr, 1.16k op/s 2024-05-15T23:07:42.431 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:42 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:07:42.432 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:42 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:07:42.432 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:42 smithi088 ceph-mon[44699]: osdmap e143: 12 total, 12 up, 12 in 2024-05-15T23:07:42.435 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:42.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:42 smithi090 ceph-mon[44434]: pgmap v433: 17 pgs: 17 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 42 MiB/s wr, 1.16k op/s 2024-05-15T23:07:42.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:42 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:07:42.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:42 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:07:42.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:42 smithi090 ceph-mon[44434]: osdmap e143: 12 total, 12 up, 12 in 2024-05-15T23:07:42.560 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:42.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:42.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:07:42.733 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:42.759 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:07:42.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:42.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:07:42.864 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:07:42.947 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:43.173 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T23:07:43.266 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:07:43.281 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:07:43.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:43 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/856158109' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:07:43.437 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-05-15T23:07:43.476 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:07:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:43 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/856158109' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:07:43.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:43 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/856158109' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:07:44.032 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:07:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:44.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:07:44.118 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:44.140 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:44.147 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:44.178 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:44.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:44 smithi084 ceph-mon[39784]: pgmap v435: 17 pgs: 17 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 22 MiB/s wr, 1.13k op/s 2024-05-15T23:07:44.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:44 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:07:44.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:44 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:07:44.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:44 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:07:44.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/856158109' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:07:44.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:44 smithi084 ceph-mon[39784]: osdmap e144: 12 total, 12 up, 12 in 2024-05-15T23:07:44.329 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:44.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:44.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:44.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:44.471 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:44.494 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:44.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:44.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:44.535 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:44 smithi088 ceph-mon[44699]: pgmap v435: 17 pgs: 17 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 22 MiB/s wr, 1.13k op/s 2024-05-15T23:07:44.536 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:44 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:07:44.536 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:44 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:07:44.536 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:44 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:07:44.536 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/856158109' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:07:44.536 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:44 smithi088 ceph-mon[44699]: osdmap e144: 12 total, 12 up, 12 in 2024-05-15T23:07:44.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:44 smithi090 ceph-mon[44434]: pgmap v435: 17 pgs: 17 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 22 MiB/s wr, 1.13k op/s 2024-05-15T23:07:44.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:44 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:07:44.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:44 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:07:44.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:44 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:07:44.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/856158109' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:07:44.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:44 smithi090 ceph-mon[44434]: osdmap e144: 12 total, 12 up, 12 in 2024-05-15T23:07:44.599 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:44.626 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:44.877 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:44.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:44.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:45.029 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:45.057 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:45.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:45.059 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:45.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:45.201 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:45.237 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:45.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:45.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:07:45.411 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:45.474 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:07:45.474 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:07:45.474 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:07:45.474 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:07:45.474 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:07:45.497 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:07:45.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:45.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:07:45.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:07:45.643 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:45.908 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:07:45.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:45.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:07:46.017 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:46.047 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:46.066 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:07:46.241 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T23:07:46.246 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:46.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:46.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:46.350 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:07:46.394 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:46.426 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:07:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:46.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:07:46.502 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:46 smithi084 ceph-mon[39784]: pgmap v437: 17 pgs: 17 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s wr, 1.21k op/s 2024-05-15T23:07:46.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:07:46.545 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:07:46.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:46 smithi090 ceph-mon[44434]: pgmap v437: 17 pgs: 17 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s wr, 1.21k op/s 2024-05-15T23:07:46.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:46 smithi088 ceph-mon[44699]: pgmap v437: 17 pgs: 17 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s wr, 1.21k op/s 2024-05-15T23:07:46.605 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:46.841 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:07:46.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:46.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:07:46.951 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:07:46.992 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:47.085 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:07:47.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:47.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:07:47.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:07:47.250 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:47.273 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:07:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:47.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:07:47.377 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:47.409 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:47.429 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:47.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:47.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:47.533 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:07:47.564 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:47.797 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:07:47.798 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:07:47.798 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:07:47.798 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:07:47.815 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:47.834 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:07:47.863 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:07:47.895 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:07:47.922 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:07:47.954 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:07:47.989 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:07:48.015 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:07:48.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:48.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:07:48.023 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:07:48.058 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:07:48.062 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:07:48.062 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:07:48.062 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:07:48.062 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:07:48.062 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:07:48.062 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:07:48.062 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:07:48.063 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:07:48.063 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:07:48.063 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:07:48.063 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:07:48.063 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:07:48.119 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:07:48.154 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:48.256 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:07:48.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:48.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:07:48.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:48 smithi084 ceph-mon[39784]: pgmap v438: 17 pgs: 17 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s wr, 1.46k op/s 2024-05-15T23:07:48.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4121885131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:48.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:48.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:48.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:48.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:48.295 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:48 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:07:48.295 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:48 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:07:48.295 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:48.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:07:48.432 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:48 smithi090 ceph-mon[44434]: pgmap v438: 17 pgs: 17 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s wr, 1.46k op/s 2024-05-15T23:07:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4121885131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:48.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:48 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:07:48.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:48 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:07:48.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:48.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:48 smithi088 ceph-mon[44699]: pgmap v438: 17 pgs: 17 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s wr, 1.46k op/s 2024-05-15T23:07:48.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4121885131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:48.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:48.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:48.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:48.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:48.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:48 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:07:48.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:48 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:07:48.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:48.671 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:07:48.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:48.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:07:48.778 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:48.809 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:49.429 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-05-15T23:07:49.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:49 smithi088 ceph-mon[44699]: osdmap e145: 12 total, 12 up, 12 in 2024-05-15T23:07:49.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:49 smithi084 ceph-mon[39784]: osdmap e145: 12 total, 12 up, 12 in 2024-05-15T23:07:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:49.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:49 smithi090 ceph-mon[44434]: osdmap e145: 12 total, 12 up, 12 in 2024-05-15T23:07:49.889 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:07:49.889 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:07:50.337 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:50.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:50 smithi084 ceph-mon[39784]: pgmap v440: 17 pgs: 17 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 971 op/s 2024-05-15T23:07:50.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:50 smithi090 ceph-mon[44434]: pgmap v440: 17 pgs: 17 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 971 op/s 2024-05-15T23:07:50.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:50 smithi088 ceph-mon[44699]: pgmap v440: 17 pgs: 17 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 971 op/s 2024-05-15T23:07:51.660 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:51.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:51.765 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:07:51.810 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:51.881 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:07:52.080 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:07:52.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:52.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:07:52.181 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T23:07:52.190 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:07:52.253 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:52.276 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:07:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:52.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:07:52.369 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:07:52.380 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:52.405 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:52 smithi088 ceph-mon[44699]: pgmap v441: 17 pgs: 17 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-05-15T23:07:52.406 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:52 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/236824233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:07:52.406 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:52 smithi088 ceph-mon[44699]: from='client.34889 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:07:52.408 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:52.432 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:52.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:52.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:52.502 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:07:52.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:07:52.569 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:52.606 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:07:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:52.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:07:52.612 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:52 smithi090 ceph-mon[44434]: pgmap v441: 17 pgs: 17 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-05-15T23:07:52.612 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:52 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/236824233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:07:52.612 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:52 smithi090 ceph-mon[44434]: from='client.34889 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:07:52.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:52 smithi084 ceph-mon[39784]: pgmap v441: 17 pgs: 17 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-05-15T23:07:52.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:52 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/236824233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:07:52.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:52 smithi084 ceph-mon[39784]: from='client.34889 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:07:52.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:07:52.746 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:52.776 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:07:52.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:52.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:07:52.881 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:52.909 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:52.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:52.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:53.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:53.121 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:53.149 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:53.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:53.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:53.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:53.295 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:53.326 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:53.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:53.330 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:53.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:53 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:07:53.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:53 smithi084 ceph-mon[39784]: from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:07:53.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:53 smithi084 ceph-mon[39784]: osdmap e146: 12 total, 12 up, 12 in 2024-05-15T23:07:53.430 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:07:53.464 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:53.495 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:07:53.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:53.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:07:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:53 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:07:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:53 smithi090 ceph-mon[44434]: from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:07:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:53 smithi090 ceph-mon[44434]: osdmap e146: 12 total, 12 up, 12 in 2024-05-15T23:07:53.607 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:07:53.667 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:53 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:07:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:53 smithi088 ceph-mon[44699]: from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:07:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:53 smithi088 ceph-mon[44699]: osdmap e146: 12 total, 12 up, 12 in 2024-05-15T23:07:53.891 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:07:53.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:53.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:07:53.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-05-15T23:07:53.999 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:07:54.058 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:54.247 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:07:54.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:54.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:07:54.321 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:54.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:07:54.385 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:07:54.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:54.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:07:54.515 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:07:54.516 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:07:54.519 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:07:54.540 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:54 smithi088 ceph-mon[44699]: pgmap v443: 17 pgs: 17 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-05-15T23:07:54.540 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:54 smithi088 ceph-mon[44699]: 2.8 scrub starts 2024-05-15T23:07:54.554 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:07:54.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:54.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:07:54.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:54 smithi084 ceph-mon[39784]: pgmap v443: 17 pgs: 17 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-05-15T23:07:54.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:54 smithi084 ceph-mon[39784]: 2.8 scrub starts 2024-05-15T23:07:54.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:07:54.716 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:54 smithi090 ceph-mon[44434]: pgmap v443: 17 pgs: 17 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-05-15T23:07:54.717 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:54 smithi090 ceph-mon[44434]: 2.8 scrub starts 2024-05-15T23:07:54.721 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:54.746 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:07:54.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:54.751 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:07:54.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:54.884 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:54.913 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:54.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:54.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:55.022 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:55.055 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:55.201 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:07:55.257 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:55.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:55.284 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T23:07:55.363 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:07:55.416 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:55.621 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:07:55.677 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:07:55.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:55.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:07:55.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:07:55.845 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:55.875 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:07:55.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:55.880 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:07:55.981 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:07:56.030 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:56.183 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:07:56.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:56.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:07:56.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:07:56.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:07:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:56.364 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:07:56.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:56.370 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:07:56.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:07:56.505 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:56.528 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:07:56.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:56.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:07:56.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:56 smithi088 ceph-mon[44699]: pgmap v444: 17 pgs: 17 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T23:07:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:56 smithi088 ceph-mon[44699]: 2.8 scrub ok 2024-05-15T23:07:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:56 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:07:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:56 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:07:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:56 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:07:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:56 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:07:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:56 smithi084 ceph-mon[39784]: pgmap v444: 17 pgs: 17 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T23:07:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:56 smithi084 ceph-mon[39784]: 2.8 scrub ok 2024-05-15T23:07:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:56 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:07:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:56 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:07:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:56 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:07:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:56 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:07:56.632 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:07:56.653 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:07:56.660 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:56.689 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:07:56.735 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:07:56.768 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:07:56.801 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:07:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:56 smithi090 ceph-mon[44434]: pgmap v444: 17 pgs: 17 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T23:07:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:56 smithi090 ceph-mon[44434]: 2.8 scrub ok 2024-05-15T23:07:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:56 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:07:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:56 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:07:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:56 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:07:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:56 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:07:56.840 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:07:56.889 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:07:56.928 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:07:56.962 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:07:56.989 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:07:56.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:56.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:07:57.003 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:07:57.057 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:07:57.092 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:07:57.093 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:07:57.115 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:07:57.116 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:07:57.116 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:07:57.116 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:07:57.116 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:07:57.116 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:07:57.116 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:07:57.116 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:07:57.116 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:07:57.116 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:07:57.116 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:07:57.124 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:57.154 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:07:57.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:57.161 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:07:57.262 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:07:57.318 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:57.343 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:07:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:57.347 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:07:57.449 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:07:57.480 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:57.631 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:07:57.634 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:57 smithi088 ceph-mon[44699]: osdmap e147: 12 total, 12 up, 12 in 2024-05-15T23:07:57.634 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1059342153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:57.634 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.634 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.634 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.634 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.634 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.634 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:57.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:07:57.636 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:07:57.636 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:07:57.636 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:07:57.636 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:07:57.636 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:07:57.736 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:07:57.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:57 smithi084 ceph-mon[39784]: osdmap e147: 12 total, 12 up, 12 in 2024-05-15T23:07:57.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1059342153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:57.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.797 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:57 smithi090 ceph-mon[44434]: osdmap e147: 12 total, 12 up, 12 in 2024-05-15T23:07:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1059342153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:07:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:57.906 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:07:57.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:57.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:07:58.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:07:58.053 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:58.208 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:07:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:58.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:07:58.312 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:07:58.380 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:58.453 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T23:07:58.596 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:07:58.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:58.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:07:58.643 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.643 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:58 smithi084 ceph-mon[39784]: pgmap v446: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.36k op/s 2024-05-15T23:07:58.643 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:07:58.736 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.736 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:58 smithi090 ceph-mon[44434]: pgmap v446: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.36k op/s 2024-05-15T23:07:58.736 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.736 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.736 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.736 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.736 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.737 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:58.758 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:07:58.759 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:07:58.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:58.763 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:07:58.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:58 smithi088 ceph-mon[44699]: pgmap v446: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.36k op/s 2024-05-15T23:07:58.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:07:58.864 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:07:58.916 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:07:59.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:59 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:07:59.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:59 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:07:59.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:07:59 smithi090 ceph-mon[44434]: osdmap e148: 12 total, 12 up, 12 in 2024-05-15T23:07:59.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:59 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:07:59.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:59 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:07:59.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:07:59 smithi088 ceph-mon[44699]: osdmap e148: 12 total, 12 up, 12 in 2024-05-15T23:07:59.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:59 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:07:59.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:59 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:07:59.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:07:59 smithi084 ceph-mon[39784]: osdmap e148: 12 total, 12 up, 12 in 2024-05-15T23:07:59.893 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:07:59.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:07:59.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:07:59.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-05-15T23:08:00.003 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:08:00.053 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:00.057 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:00.092 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:08:00.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:00.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:08:00.199 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:00.230 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:00.253 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:00.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:00.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:08:00.388 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:00.418 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:08:00.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:00.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:08:00.529 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:08:00.530 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:00 smithi088 ceph-mon[44699]: pgmap v448: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s 2024-05-15T23:08:00.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:00 smithi084 ceph-mon[39784]: pgmap v448: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s 2024-05-15T23:08:00.593 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:00.631 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:08:00.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:00.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:08:00.643 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:08:00.742 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:08:00.776 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:00 smithi090 ceph-mon[44434]: pgmap v448: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s 2024-05-15T23:08:00.835 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:08:00.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:00.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:08:00.871 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:08:00.871 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:08:00.871 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:08:00.872 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:08:00.872 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.872 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119417 2.3 2:cf4ffc0c:::benchmark_data_smithi090_78160_object119416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:00.872 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.178575+0000", 2024-05-15T23:08:00.872 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.689203657, 2024-05-15T23:08:00.872 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.079319406999999995, 2024-05-15T23:08:00.872 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.872 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.872 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.872 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.872 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.872 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.872 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119417 2024-05-15T23:08:00.873 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.873 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.873 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.873 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.873 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178575+0000", 2024-05-15T23:08:00.873 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.873 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.873 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.873 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.873 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178575+0000", 2024-05-15T23:08:00.873 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.873 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.873 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.873 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.874 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178577+0000", 2024-05-15T23:08:00.874 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9700000000000002e-06 2024-05-15T23:08:00.874 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.874 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.874 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.874 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178632+0000", 2024-05-15T23:08:00.874 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5040000000000002e-05 2024-05-15T23:08:00.874 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.874 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.874 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.874 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178642+0000", 2024-05-15T23:08:00.874 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0397999999999999e-05 2024-05-15T23:08:00.874 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.874 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.875 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178649+0000", 2024-05-15T23:08:00.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6019999999999996e-06 2024-05-15T23:08:00.875 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.875 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.875 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178700+0000", 2024-05-15T23:08:00.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1292000000000003e-05 2024-05-15T23:08:00.875 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.875 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.875 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.875 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178792+0000", 2024-05-15T23:08:00.875 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2165999999999998e-05 2024-05-15T23:08:00.875 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178886+0000", 2024-05-15T23:08:00.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3111999999999999e-05 2024-05-15T23:08:00.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.181534+0000", 2024-05-15T23:08:00.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026488869999999999 2024-05-15T23:08:00.876 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.876 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.876 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.876 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.257736+0000", 2024-05-15T23:08:00.876 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.076201181000000007 2024-05-15T23:08:00.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.257843+0000", 2024-05-15T23:08:00.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00010730099999999999 2024-05-15T23:08:00.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.877 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.877 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.877 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.257894+0000", 2024-05-15T23:08:00.877 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1458000000000001e-05 2024-05-15T23:08:00.877 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.877 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.877 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.878 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.878 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119447 2.3 2:ce53d87e:::benchmark_data_smithi090_78160_object119446:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:00.878 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.194462+0000", 2024-05-15T23:08:00.878 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.673316921000001, 2024-05-15T23:08:00.878 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070550634000000001, 2024-05-15T23:08:00.878 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.878 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.878 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.878 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.878 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.878 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.878 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119447 2024-05-15T23:08:00.878 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.879 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.879 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.879 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.879 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194462+0000", 2024-05-15T23:08:00.879 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.879 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.879 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.879 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.879 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194462+0000", 2024-05-15T23:08:00.879 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.879 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.879 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.879 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.879 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194463+0000", 2024-05-15T23:08:00.880 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8190000000000001e-06 2024-05-15T23:08:00.880 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.880 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.880 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.880 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194503+0000", 2024-05-15T23:08:00.880 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9174999999999997e-05 2024-05-15T23:08:00.880 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.880 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.880 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.880 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194518+0000", 2024-05-15T23:08:00.880 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5460000000000001e-05 2024-05-15T23:08:00.880 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.880 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.880 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.881 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194527+0000", 2024-05-15T23:08:00.881 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.4330000000000004e-06 2024-05-15T23:08:00.881 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.881 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.881 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.881 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194587+0000", 2024-05-15T23:08:00.881 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0207999999999997e-05 2024-05-15T23:08:00.881 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.881 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.881 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.881 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194712+0000", 2024-05-15T23:08:00.881 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000125318 2024-05-15T23:08:00.881 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.881 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.882 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.882 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194863+0000", 2024-05-15T23:08:00.882 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015054399999999999 2024-05-15T23:08:00.882 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.882 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.882 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.882 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.200877+0000", 2024-05-15T23:08:00.882 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0060144400000000002 2024-05-15T23:08:00.882 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.882 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.882 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.882 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.264955+0000", 2024-05-15T23:08:00.882 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.064077814999999996 2024-05-15T23:08:00.882 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.883 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.883 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.883 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.264988+0000", 2024-05-15T23:08:00.883 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2755000000000001e-05 2024-05-15T23:08:00.883 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.884 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.884 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.884 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.265012+0000", 2024-05-15T23:08:00.884 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4666999999999999e-05 2024-05-15T23:08:00.884 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.884 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.884 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.884 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.885 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.885 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119448 2.3 2:c91dd160:::benchmark_data_smithi090_78160_object119447:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:00.885 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.194884+0000", 2024-05-15T23:08:00.885 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.672894243999998, 2024-05-15T23:08:00.885 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.071121264000000003, 2024-05-15T23:08:00.885 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.885 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.885 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.885 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.886 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.886 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.886 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119448 2024-05-15T23:08:00.886 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.886 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.886 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.886 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194884+0000", 2024-05-15T23:08:00.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.887 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.887 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.887 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194884+0000", 2024-05-15T23:08:00.887 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.887 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.887 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.888 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.888 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194885+0000", 2024-05-15T23:08:00.888 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.04e-06 2024-05-15T23:08:00.888 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.888 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.888 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.888 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194946+0000", 2024-05-15T23:08:00.888 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0918000000000002e-05 2024-05-15T23:08:00.888 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.889 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.889 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.889 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194955+0000", 2024-05-15T23:08:00.889 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6929999999999996e-06 2024-05-15T23:08:00.889 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.889 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.889 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.890 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194962+0000", 2024-05-15T23:08:00.890 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8689999999999998e-06 2024-05-15T23:08:00.890 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.890 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.890 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.890 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198056+0000", 2024-05-15T23:08:00.890 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030938900000000002 2024-05-15T23:08:00.890 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.891 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.891 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.891 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198180+0000", 2024-05-15T23:08:00.891 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012426000000000001 2024-05-15T23:08:00.891 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.891 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.891 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.892 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198279+0000", 2024-05-15T23:08:00.892 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8665999999999994e-05 2024-05-15T23:08:00.892 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.892 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.892 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.892 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.200997+0000", 2024-05-15T23:08:00.892 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027182339999999999 2024-05-15T23:08:00.892 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.892 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.893 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.893 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.265881+0000", 2024-05-15T23:08:00.893 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.064884366999999998 2024-05-15T23:08:00.893 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.893 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.893 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.893 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.265938+0000", 2024-05-15T23:08:00.894 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6895999999999999e-05 2024-05-15T23:08:00.894 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.894 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.894 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.894 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.266006+0000", 2024-05-15T23:08:00.894 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7430999999999999e-05 2024-05-15T23:08:00.894 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.894 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.895 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.895 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.895 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.895 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119454 2.2 2:4078f10b:::benchmark_data_smithi090_78160_object119453:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:00.895 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.198357+0000", 2024-05-15T23:08:00.895 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.669421108000002, 2024-05-15T23:08:00.895 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.064184890999999994, 2024-05-15T23:08:00.895 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.895 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.896 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.896 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.896 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.896 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.896 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119454 2024-05-15T23:08:00.896 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.896 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.896 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.897 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.897 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198357+0000", 2024-05-15T23:08:00.897 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.897 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.897 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.897 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.897 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198357+0000", 2024-05-15T23:08:00.897 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.898 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.898 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.898 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.898 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198359+0000", 2024-05-15T23:08:00.898 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8449999999999999e-06 2024-05-15T23:08:00.898 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.898 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.898 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.899 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198406+0000", 2024-05-15T23:08:00.899 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6476e-05 2024-05-15T23:08:00.899 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.899 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.899 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.899 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198417+0000", 2024-05-15T23:08:00.899 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0956e-05 2024-05-15T23:08:00.899 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.900 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.900 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.900 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198425+0000", 2024-05-15T23:08:00.900 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.4509999999999992e-06 2024-05-15T23:08:00.900 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.900 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.900 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.900 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198473+0000", 2024-05-15T23:08:00.901 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.812e-05 2024-05-15T23:08:00.901 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.901 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.901 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.901 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198559+0000", 2024-05-15T23:08:00.901 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5396999999999996e-05 2024-05-15T23:08:00.902 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.902 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.902 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:08:00.902 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198657+0000", 2024-05-15T23:08:00.902 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7847e-05 2024-05-15T23:08:00.902 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.902 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.902 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.902 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.203278+0000", 2024-05-15T23:08:00.902 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0046219039999999996 2024-05-15T23:08:00.902 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.902 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.903 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.903 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.262458+0000", 2024-05-15T23:08:00.903 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.059179555000000002 2024-05-15T23:08:00.903 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.903 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.903 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.903 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.262511+0000", 2024-05-15T23:08:00.903 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3168000000000003e-05 2024-05-15T23:08:00.903 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.903 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.903 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.903 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.262542+0000", 2024-05-15T23:08:00.903 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1171999999999999e-05 2024-05-15T23:08:00.903 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.904 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.904 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.904 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.904 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.904 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119455 2.2 2:43cb5301:::benchmark_data_smithi090_78160_object119454:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:00.904 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.198541+0000", 2024-05-15T23:08:00.904 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.669237973000001, 2024-05-15T23:08:00.904 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.067531026999999993, 2024-05-15T23:08:00.904 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.904 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.904 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.904 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.904 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.905 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.905 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119455 2024-05-15T23:08:00.905 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.905 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.905 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.905 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.905 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198541+0000", 2024-05-15T23:08:00.905 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.905 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.905 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.905 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.905 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198541+0000", 2024-05-15T23:08:00.905 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.905 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.905 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.906 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.906 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198542+0000", 2024-05-15T23:08:00.906 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0449999999999999e-06 2024-05-15T23:08:00.906 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.906 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.906 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.906 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198603+0000", 2024-05-15T23:08:00.906 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1055000000000004e-05 2024-05-15T23:08:00.906 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.907 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.907 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.907 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198613+0000", 2024-05-15T23:08:00.907 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8290000000000003e-06 2024-05-15T23:08:00.907 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.907 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.907 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.907 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198620+0000", 2024-05-15T23:08:00.907 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9400000000000002e-06 2024-05-15T23:08:00.907 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.907 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.907 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.907 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.201083+0000", 2024-05-15T23:08:00.907 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024620750000000002 2024-05-15T23:08:00.908 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.908 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.908 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.908 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.201177+0000", 2024-05-15T23:08:00.908 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4271000000000006e-05 2024-05-15T23:08:00.908 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.908 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.908 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:08:00.908 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.201270+0000", 2024-05-15T23:08:00.908 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3246999999999997e-05 2024-05-15T23:08:00.908 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.908 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.908 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.908 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.204897+0000", 2024-05-15T23:08:00.908 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003627389 2024-05-15T23:08:00.909 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.909 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.909 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.909 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.265985+0000", 2024-05-15T23:08:00.909 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061087352999999997 2024-05-15T23:08:00.909 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.909 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.909 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.909 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.266022+0000", 2024-05-15T23:08:00.909 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7598000000000002e-05 2024-05-15T23:08:00.909 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.909 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.909 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.909 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.266072+0000", 2024-05-15T23:08:00.909 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9224999999999997e-05 2024-05-15T23:08:00.910 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.910 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.910 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.910 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.910 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.910 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119456 2.2 2:4e6d95d4:::benchmark_data_smithi090_78160_object119455:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:00.910 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.201870+0000", 2024-05-15T23:08:00.910 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.665908135000002, 2024-05-15T23:08:00.910 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.064294057000000002, 2024-05-15T23:08:00.910 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.910 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.910 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.910 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.911 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.911 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.911 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119456 2024-05-15T23:08:00.911 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.911 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.911 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.911 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.911 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.201870+0000", 2024-05-15T23:08:00.911 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.911 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.911 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.911 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.911 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.201870+0000", 2024-05-15T23:08:00.911 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.912 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.912 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.912 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.912 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.201872+0000", 2024-05-15T23:08:00.912 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5120000000000001e-06 2024-05-15T23:08:00.912 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.912 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.912 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.913 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.201929+0000", 2024-05-15T23:08:00.913 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6944999999999998e-05 2024-05-15T23:08:00.913 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.913 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.913 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.913 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.201938+0000", 2024-05-15T23:08:00.913 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.132e-06 2024-05-15T23:08:00.913 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.913 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.913 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.914 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.201945+0000", 2024-05-15T23:08:00.914 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3200000000000002e-06 2024-05-15T23:08:00.914 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.914 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.914 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.914 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.203324+0000", 2024-05-15T23:08:00.914 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013786390000000001 2024-05-15T23:08:00.914 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.914 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.914 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.914 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.203405+0000", 2024-05-15T23:08:00.915 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0921999999999995e-05 2024-05-15T23:08:00.915 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.915 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.915 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:08:00.915 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.203500+0000", 2024-05-15T23:08:00.915 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4850000000000002e-05 2024-05-15T23:08:00.915 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.915 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.915 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.915 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.205049+0000", 2024-05-15T23:08:00.915 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0015488839999999999 2024-05-15T23:08:00.915 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.915 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.915 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.916 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.266110+0000", 2024-05-15T23:08:00.916 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061060958999999998 2024-05-15T23:08:00.916 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.916 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.916 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.916 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.266141+0000", 2024-05-15T23:08:00.916 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1585999999999999e-05 2024-05-15T23:08:00.916 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.916 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.916 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.916 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.266165+0000", 2024-05-15T23:08:00.916 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3308000000000001e-05 2024-05-15T23:08:00.916 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.916 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.917 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.917 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.917 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.917 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119714 2.3 2:cc865b7c:::benchmark_data_smithi090_78160_object119713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:00.917 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.447597+0000", 2024-05-15T23:08:00.917 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.420181968000001, 2024-05-15T23:08:00.917 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.050610349999999998, 2024-05-15T23:08:00.917 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.917 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.917 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.917 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.917 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.917 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.918 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119714 2024-05-15T23:08:00.918 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.918 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.918 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.918 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.918 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447597+0000", 2024-05-15T23:08:00.918 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.918 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.918 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.918 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.918 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447597+0000", 2024-05-15T23:08:00.919 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.919 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.919 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.919 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.919 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447599+0000", 2024-05-15T23:08:00.919 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7250000000000002e-06 2024-05-15T23:08:00.919 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.919 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.919 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.919 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447682+0000", 2024-05-15T23:08:00.919 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2856999999999996e-05 2024-05-15T23:08:00.919 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.920 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.920 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.920 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447697+0000", 2024-05-15T23:08:00.920 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4877e-05 2024-05-15T23:08:00.920 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.920 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.920 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.920 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447723+0000", 2024-05-15T23:08:00.920 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5803999999999999e-05 2024-05-15T23:08:00.920 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.920 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.920 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.920 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447790+0000", 2024-05-15T23:08:00.920 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7442999999999998e-05 2024-05-15T23:08:00.921 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.921 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.921 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.921 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447918+0000", 2024-05-15T23:08:00.921 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012783099999999999 2024-05-15T23:08:00.921 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.921 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.921 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.921 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.448024+0000", 2024-05-15T23:08:00.921 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0001061 2024-05-15T23:08:00.921 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.921 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.921 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.922 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.459030+0000", 2024-05-15T23:08:00.922 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011005351 2024-05-15T23:08:00.922 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.922 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.922 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.922 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.498102+0000", 2024-05-15T23:08:00.922 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.039072510999999997 2024-05-15T23:08:00.922 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.922 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.923 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.923 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.498161+0000", 2024-05-15T23:08:00.923 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9191000000000003e-05 2024-05-15T23:08:00.923 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.923 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.923 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.923 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.498207+0000", 2024-05-15T23:08:00.923 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.566e-05 2024-05-15T23:08:00.923 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.923 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.923 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.924 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.924 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.924 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119717 2.3 2:c030e997:::benchmark_data_smithi090_78160_object119716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:00.924 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.447748+0000", 2024-05-15T23:08:00.924 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.420030928999999, 2024-05-15T23:08:00.924 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065365148999999997, 2024-05-15T23:08:00.924 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.924 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.924 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.924 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.924 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.924 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.924 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119717 2024-05-15T23:08:00.924 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.925 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.925 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.925 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.925 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447748+0000", 2024-05-15T23:08:00.925 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.925 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.925 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.925 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.925 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447748+0000", 2024-05-15T23:08:00.925 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.925 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.925 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.925 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.925 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447748+0000", 2024-05-15T23:08:00.925 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2799999999999995e-07 2024-05-15T23:08:00.926 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.926 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.926 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.926 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447778+0000", 2024-05-15T23:08:00.926 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9292999999999999e-05 2024-05-15T23:08:00.926 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.926 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.926 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.926 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447784+0000", 2024-05-15T23:08:00.926 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4779999999999996e-06 2024-05-15T23:08:00.926 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.926 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447790+0000", 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6400000000000002e-06 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.451399+0000", 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036088610000000001 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.451532+0000", 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013333400000000001 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.927 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.928 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.451678+0000", 2024-05-15T23:08:00.928 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014612699999999999 2024-05-15T23:08:00.928 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.928 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.928 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.928 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.459038+0000", 2024-05-15T23:08:00.928 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0073598999999999999 2024-05-15T23:08:00.928 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.928 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.928 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.928 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.513039+0000", 2024-05-15T23:08:00.928 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054001400999999997 2024-05-15T23:08:00.928 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.928 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.928 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.929 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.513073+0000", 2024-05-15T23:08:00.929 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3148000000000002e-05 2024-05-15T23:08:00.929 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.929 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.929 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.929 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.513113+0000", 2024-05-15T23:08:00.929 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0238999999999999e-05 2024-05-15T23:08:00.929 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.929 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.929 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.929 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.929 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.929 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119718 2.3 2:cd032e9d:::benchmark_data_smithi090_78160_object119717:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:00.929 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.447799+0000", 2024-05-15T23:08:00.929 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.419979206999997, 2024-05-15T23:08:00.930 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065419465999999996, 2024-05-15T23:08:00.930 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.930 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.930 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.930 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.930 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.930 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.930 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119718 2024-05-15T23:08:00.930 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.930 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.930 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.930 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.930 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447799+0000", 2024-05-15T23:08:00.931 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.931 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.931 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.931 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.931 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447799+0000", 2024-05-15T23:08:00.931 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.931 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.931 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.931 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.931 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447800+0000", 2024-05-15T23:08:00.931 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1999999999999999e-07 2024-05-15T23:08:00.931 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.931 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.932 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.932 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447849+0000", 2024-05-15T23:08:00.932 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8791000000000001e-05 2024-05-15T23:08:00.932 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.932 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.932 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.932 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447856+0000", 2024-05-15T23:08:00.932 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6389999999999992e-06 2024-05-15T23:08:00.932 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.932 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.932 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.932 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.447862+0000", 2024-05-15T23:08:00.932 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0190000000000003e-06 2024-05-15T23:08:00.933 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.933 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.933 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.933 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.454157+0000", 2024-05-15T23:08:00.933 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0062940629999999999 2024-05-15T23:08:00.933 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.933 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.933 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.934 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.454283+0000", 2024-05-15T23:08:00.934 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000126253 2024-05-15T23:08:00.934 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.934 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.934 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.934 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.454424+0000", 2024-05-15T23:08:00.934 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014081200000000001 2024-05-15T23:08:00.934 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.934 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.935 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.935 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.459044+0000", 2024-05-15T23:08:00.935 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0046209270000000004 2024-05-15T23:08:00.935 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.935 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.935 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.935 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.513151+0000", 2024-05-15T23:08:00.935 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054106620000000001 2024-05-15T23:08:00.935 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.935 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.935 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.935 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.513185+0000", 2024-05-15T23:08:00.935 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3974999999999999e-05 2024-05-15T23:08:00.935 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.936 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.936 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.936 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.513219+0000", 2024-05-15T23:08:00.936 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3747000000000001e-05 2024-05-15T23:08:00.936 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.936 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.936 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.936 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.936 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.936 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119727 2.3 2:ca0d49b0:::benchmark_data_smithi090_78160_object119726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:00.936 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.453742+0000", 2024-05-15T23:08:00.936 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.414036682999999, 2024-05-15T23:08:00.936 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070952512999999995, 2024-05-15T23:08:00.936 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.936 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.937 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.937 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.937 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.937 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.937 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119727 2024-05-15T23:08:00.937 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.937 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.937 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.937 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.937 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.453742+0000", 2024-05-15T23:08:00.937 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.937 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.937 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.937 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.937 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.453742+0000", 2024-05-15T23:08:00.938 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.938 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.938 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.938 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.938 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.453744+0000", 2024-05-15T23:08:00.938 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.781e-06 2024-05-15T23:08:00.938 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.938 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.938 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.938 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.453812+0000", 2024-05-15T23:08:00.938 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8233999999999998e-05 2024-05-15T23:08:00.938 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.938 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.938 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.453823+0000", 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1158999999999999e-05 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.453829+0000", 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.173e-06 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.456675+0000", 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002846094 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.939 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.456802+0000", 2024-05-15T23:08:00.940 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012648100000000001 2024-05-15T23:08:00.940 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.940 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.940 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.940 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.456946+0000", 2024-05-15T23:08:00.940 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000144449 2024-05-15T23:08:00.940 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.940 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.940 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.940 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.461103+0000", 2024-05-15T23:08:00.940 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0041565229999999996 2024-05-15T23:08:00.940 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.940 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.940 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.940 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.524622+0000", 2024-05-15T23:08:00.941 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063519640000000002 2024-05-15T23:08:00.941 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.941 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.941 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.941 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.524668+0000", 2024-05-15T23:08:00.941 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5753000000000001e-05 2024-05-15T23:08:00.941 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.941 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.941 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.941 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.524694+0000", 2024-05-15T23:08:00.941 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6225999999999999e-05 2024-05-15T23:08:00.941 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.941 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.941 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.941 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.942 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.942 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119731 2.3 2:cf461399:::benchmark_data_smithi090_78160_object119730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:00.942 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.454119+0000", 2024-05-15T23:08:00.942 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.413659481000003, 2024-05-15T23:08:00.942 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070677609000000002, 2024-05-15T23:08:00.942 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.942 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.942 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.942 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.942 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.942 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.942 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119731 2024-05-15T23:08:00.942 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.942 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.943 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.943 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.943 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.454119+0000", 2024-05-15T23:08:00.943 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.943 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.943 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.943 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.943 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.454119+0000", 2024-05-15T23:08:00.943 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.943 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.943 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.943 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.943 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.454120+0000", 2024-05-15T23:08:00.943 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0330000000000001e-06 2024-05-15T23:08:00.944 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.944 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.944 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.944 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.454225+0000", 2024-05-15T23:08:00.944 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000105028 2024-05-15T23:08:00.944 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.944 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.944 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.944 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.454230+0000", 2024-05-15T23:08:00.944 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2440000000000001e-06 2024-05-15T23:08:00.944 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.454235+0000", 2024-05-15T23:08:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2069999999999997e-06 2024-05-15T23:08:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.459063+0000", 2024-05-15T23:08:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0048286099999999997 2024-05-15T23:08:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.459136+0000", 2024-05-15T23:08:00.945 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3155e-05 2024-05-15T23:08:00.946 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.946 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.946 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.946 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.459231+0000", 2024-05-15T23:08:00.946 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4890999999999997e-05 2024-05-15T23:08:00.946 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.946 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.946 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.946 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.463369+0000", 2024-05-15T23:08:00.946 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0041376779999999997 2024-05-15T23:08:00.946 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.946 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.947 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.947 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.524735+0000", 2024-05-15T23:08:00.947 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061365885000000002 2024-05-15T23:08:00.947 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.947 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.947 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.947 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.524771+0000", 2024-05-15T23:08:00.947 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6109999999999998e-05 2024-05-15T23:08:00.947 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.947 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.947 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.947 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.524797+0000", 2024-05-15T23:08:00.947 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5768000000000001e-05 2024-05-15T23:08:00.947 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.947 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.948 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.948 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.948 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.948 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119736 2.3 2:c8b46c6a:::benchmark_data_smithi090_78160_object119735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:00.948 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.459643+0000", 2024-05-15T23:08:00.948 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.408135317999999, 2024-05-15T23:08:00.948 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065240878000000002, 2024-05-15T23:08:00.948 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.948 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.948 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.948 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.948 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.948 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.948 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119736 2024-05-15T23:08:00.948 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.949 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.949 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.949 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.949 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.459643+0000", 2024-05-15T23:08:00.949 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.949 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.949 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.949 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.949 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.459643+0000", 2024-05-15T23:08:00.949 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.949 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.949 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.949 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.949 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.459645+0000", 2024-05-15T23:08:00.950 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3060000000000001e-06 2024-05-15T23:08:00.950 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.950 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.950 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.950 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.459702+0000", 2024-05-15T23:08:00.950 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7472000000000001e-05 2024-05-15T23:08:00.950 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.950 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.950 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.950 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.459712+0000", 2024-05-15T23:08:00.950 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6139999999999998e-06 2024-05-15T23:08:00.950 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.950 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.950 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.950 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.459717+0000", 2024-05-15T23:08:00.951 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8109999999999999e-06 2024-05-15T23:08:00.951 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.951 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.951 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.951 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.461120+0000", 2024-05-15T23:08:00.951 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001402903 2024-05-15T23:08:00.951 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.951 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.951 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.951 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.461227+0000", 2024-05-15T23:08:00.951 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00010689899999999999 2024-05-15T23:08:00.951 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.951 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.951 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.951 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.461326+0000", 2024-05-15T23:08:00.952 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8317000000000001e-05 2024-05-15T23:08:00.952 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.952 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.952 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.952 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.463496+0000", 2024-05-15T23:08:00.952 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0021700410000000002 2024-05-15T23:08:00.952 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.952 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.952 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.952 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.524825+0000", 2024-05-15T23:08:00.952 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061328903999999997 2024-05-15T23:08:00.952 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.952 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.952 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.952 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.524853+0000", 2024-05-15T23:08:00.953 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8779999999999999e-05 2024-05-15T23:08:00.953 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.953 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.953 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.953 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.524884+0000", 2024-05-15T23:08:00.953 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0830999999999997e-05 2024-05-15T23:08:00.953 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.953 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.953 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.953 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.953 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.954 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119743 2.3 2:cacd1098:::benchmark_data_smithi090_78160_object119742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:00.954 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.466610+0000", 2024-05-15T23:08:00.954 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 60.401168714000001, 2024-05-15T23:08:00.954 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061546702000000002, 2024-05-15T23:08:00.954 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.954 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.954 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.954 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.954 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.955 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.955 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119743 2024-05-15T23:08:00.955 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.955 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.955 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.955 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.955 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.466610+0000", 2024-05-15T23:08:00.955 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.955 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.955 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.955 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.955 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.466610+0000", 2024-05-15T23:08:00.955 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.955 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.955 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.956 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.956 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.466612+0000", 2024-05-15T23:08:00.956 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.654e-06 2024-05-15T23:08:00.956 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.956 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.956 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.956 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.466634+0000", 2024-05-15T23:08:00.956 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2527999999999998e-05 2024-05-15T23:08:00.956 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.956 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.956 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.956 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.466644+0000", 2024-05-15T23:08:00.956 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0006999999999999e-05 2024-05-15T23:08:00.956 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.956 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.957 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.957 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.466652+0000", 2024-05-15T23:08:00.957 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7219999999999999e-06 2024-05-15T23:08:00.957 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.957 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.957 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.957 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.466698+0000", 2024-05-15T23:08:00.957 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5945000000000001e-05 2024-05-15T23:08:00.957 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.957 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.957 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.957 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.466780+0000", 2024-05-15T23:08:00.957 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2472999999999995e-05 2024-05-15T23:08:00.957 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.957 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.958 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.958 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.466877+0000", 2024-05-15T23:08:00.958 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6558999999999995e-05 2024-05-15T23:08:00.958 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.958 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.958 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.958 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.469394+0000", 2024-05-15T23:08:00.958 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025172990000000002 2024-05-15T23:08:00.958 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.958 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.958 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.958 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.528100+0000", 2024-05-15T23:08:00.958 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058705935000000001 2024-05-15T23:08:00.959 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.959 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.959 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.959 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.528132+0000", 2024-05-15T23:08:00.959 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2400999999999997e-05 2024-05-15T23:08:00.959 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.959 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.959 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.959 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.528157+0000", 2024-05-15T23:08:00.959 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4179e-05 2024-05-15T23:08:00.959 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.959 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.959 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.960 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.960 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.960 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:120265 2.3 2:ce664b33:::benchmark_data_smithi090_78160_object120264:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:00.960 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.905160+0000", 2024-05-15T23:08:00.960 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 59.962618401, 2024-05-15T23:08:00.960 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047628977000000003, 2024-05-15T23:08:00.960 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.960 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.960 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.960 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.960 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.960 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.960 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 120265 2024-05-15T23:08:00.960 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.960 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.961 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.961 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.961 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.905160+0000", 2024-05-15T23:08:00.961 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.961 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.961 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.961 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.961 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.905160+0000", 2024-05-15T23:08:00.961 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.961 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.961 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.961 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.961 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.905162+0000", 2024-05-15T23:08:00.961 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6309999999999999e-06 2024-05-15T23:08:00.961 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.962 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.962 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.962 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.905230+0000", 2024-05-15T23:08:00.962 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7873000000000006e-05 2024-05-15T23:08:00.962 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.962 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.962 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.962 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.905240+0000", 2024-05-15T23:08:00.962 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0142999999999999e-05 2024-05-15T23:08:00.962 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.962 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.962 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.962 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.905248+0000", 2024-05-15T23:08:00.962 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9840000000000001e-06 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.905295+0000", 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6859e-05 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.905375+0000", 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0735999999999994e-05 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.905464+0000", 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8627000000000004e-05 2024-05-15T23:08:00.963 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.964 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.964 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.964 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.911801+0000", 2024-05-15T23:08:00.964 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0063373420000000001 2024-05-15T23:08:00.964 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.964 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.964 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.964 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.952625+0000", 2024-05-15T23:08:00.964 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.040823358999999997 2024-05-15T23:08:00.964 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.964 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.964 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.964 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.952722+0000", 2024-05-15T23:08:00.964 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7250999999999994e-05 2024-05-15T23:08:00.965 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.965 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.965 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.965 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.952789+0000", 2024-05-15T23:08:00.965 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7172e-05 2024-05-15T23:08:00.965 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.965 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.965 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.965 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.966 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.966 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:120269 2.3 2:c430bda8:::benchmark_data_smithi090_78160_object120268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:00.966 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.908886+0000", 2024-05-15T23:08:00.966 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 59.958892261000003, 2024-05-15T23:08:00.966 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044035018000000002, 2024-05-15T23:08:00.966 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.966 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.966 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.966 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.966 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.966 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.966 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 120269 2024-05-15T23:08:00.966 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.966 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.967 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.967 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.967 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.908886+0000", 2024-05-15T23:08:00.967 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.967 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.967 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.967 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.967 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.908886+0000", 2024-05-15T23:08:00.967 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.967 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.967 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.967 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.967 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.908888+0000", 2024-05-15T23:08:00.967 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4839999999999999e-06 2024-05-15T23:08:00.968 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.968 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.968 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.968 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.908945+0000", 2024-05-15T23:08:00.968 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6851999999999997e-05 2024-05-15T23:08:00.968 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.968 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.968 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.968 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.908955+0000", 2024-05-15T23:08:00.968 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0242e-05 2024-05-15T23:08:00.968 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.968 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.968 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.968 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.908964+0000", 2024-05-15T23:08:00.968 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6279999999999994e-06 2024-05-15T23:08:00.969 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.969 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.969 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.969 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.909123+0000", 2024-05-15T23:08:00.969 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000159433 2024-05-15T23:08:00.969 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.969 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.969 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.969 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.909242+0000", 2024-05-15T23:08:00.969 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000118676 2024-05-15T23:08:00.969 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.969 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.969 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.969 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.909331+0000", 2024-05-15T23:08:00.970 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.9237e-05 2024-05-15T23:08:00.970 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.970 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.970 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.970 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.911934+0000", 2024-05-15T23:08:00.970 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002602685 2024-05-15T23:08:00.970 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.970 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.970 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.970 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.952865+0000", 2024-05-15T23:08:00.970 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.040931286999999997 2024-05-15T23:08:00.970 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.970 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.970 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.971 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.952897+0000", 2024-05-15T23:08:00.971 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1797000000000001e-05 2024-05-15T23:08:00.971 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.971 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.971 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.971 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.952921+0000", 2024-05-15T23:08:00.971 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4697000000000001e-05 2024-05-15T23:08:00.971 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.971 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.971 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.971 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.971 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.971 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:140997 2.3 2:cf88a709:::benchmark_data_smithi090_78160_object140996:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e137)", 2024-05-15T23:08:00.972 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:17.618889+0000", 2024-05-15T23:08:00.972 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 43.248889841, 2024-05-15T23:08:00.972 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041484289000000001, 2024-05-15T23:08:00.972 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.972 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.972 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.972 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.972 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.972 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.972 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 140997 2024-05-15T23:08:00.972 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.972 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.972 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.972 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.972 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:17.618889+0000", 2024-05-15T23:08:00.973 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.973 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.973 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.973 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.973 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:17.618889+0000", 2024-05-15T23:08:00.973 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.973 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.973 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.973 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.973 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:17.618892+0000", 2024-05-15T23:08:00.973 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5360000000000001e-06 2024-05-15T23:08:00.973 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.973 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.973 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.974 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:17.618975+0000", 2024-05-15T23:08:00.974 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2403000000000004e-05 2024-05-15T23:08:00.974 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.974 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.974 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.974 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:17.618988+0000", 2024-05-15T23:08:00.974 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2908000000000001e-05 2024-05-15T23:08:00.974 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.974 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.974 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.974 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:17.619005+0000", 2024-05-15T23:08:00.974 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6949e-05 2024-05-15T23:08:00.974 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.975 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.975 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.975 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:17.623975+0000", 2024-05-15T23:08:00.975 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049700279999999996 2024-05-15T23:08:00.975 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.975 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.975 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.975 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:17.624117+0000", 2024-05-15T23:08:00.975 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000142249 2024-05-15T23:08:00.975 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.975 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.975 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.976 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:17.624223+0000", 2024-05-15T23:08:00.976 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000106625 2024-05-15T23:08:00.976 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.976 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.976 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.976 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:17.627806+0000", 2024-05-15T23:08:00.976 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0035827939999999998 2024-05-15T23:08:00.976 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.976 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.976 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.976 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:17.660255+0000", 2024-05-15T23:08:00.976 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.032449137000000003 2024-05-15T23:08:00.976 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.977 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.977 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.977 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:17.660310+0000", 2024-05-15T23:08:00.977 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4388000000000002e-05 2024-05-15T23:08:00.977 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.977 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.977 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.977 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:17.660373+0000", 2024-05-15T23:08:00.977 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3272e-05 2024-05-15T23:08:00.977 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.977 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.977 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.977 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.977 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.977 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:189299 2.3 2:c7e0a181:::benchmark_data_smithi090_78160_object27941:head [delete] snapc 0=[] ondisk+write+known_if_redirected e148)", 2024-05-15T23:08:00.978 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:59.060703+0000", 2024-05-15T23:08:00.978 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 1.8070751110000001, 2024-05-15T23:08:00.978 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.057059081999999997, 2024-05-15T23:08:00.978 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.978 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.978 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.978 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.978 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.978 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.978 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 189299 2024-05-15T23:08:00.979 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.979 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.979 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.979 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.979 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.060703+0000", 2024-05-15T23:08:00.979 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.979 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.979 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.979 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.979 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.060703+0000", 2024-05-15T23:08:00.979 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.980 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.980 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.980 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.980 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.060705+0000", 2024-05-15T23:08:00.980 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6810000000000001e-06 2024-05-15T23:08:00.980 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.980 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.980 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.980 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.060710+0000", 2024-05-15T23:08:00.981 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9080000000000003e-06 2024-05-15T23:08:00.981 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.981 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.981 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.981 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.060720+0000", 2024-05-15T23:08:00.981 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7200000000000001e-06 2024-05-15T23:08:00.981 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.981 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.981 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.981 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.060726+0000", 2024-05-15T23:08:00.981 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3300000000000004e-06 2024-05-15T23:08:00.981 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.981 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.981 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.982 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.060775+0000", 2024-05-15T23:08:00.982 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8869999999999998e-05 2024-05-15T23:08:00.982 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.982 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.982 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.982 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.060877+0000", 2024-05-15T23:08:00.982 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000102168 2024-05-15T23:08:00.982 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.982 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.982 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.982 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.060976+0000", 2024-05-15T23:08:00.982 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8919000000000006e-05 2024-05-15T23:08:00.982 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.982 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.983 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.983 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.066837+0000", 2024-05-15T23:08:00.983 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0058604809999999999 2024-05-15T23:08:00.983 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.983 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.983 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.983 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.117691+0000", 2024-05-15T23:08:00.983 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.050853957999999998 2024-05-15T23:08:00.983 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.983 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.983 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.983 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.117736+0000", 2024-05-15T23:08:00.983 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5878999999999999e-05 2024-05-15T23:08:00.983 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.984 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.984 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.984 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.117763+0000", 2024-05-15T23:08:00.984 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6168000000000001e-05 2024-05-15T23:08:00.984 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.984 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.984 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.984 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.984 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.984 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:189304 2.3 2:cb856e78:::benchmark_data_smithi090_78160_object27946:head [delete] snapc 0=[] ondisk+write+known_if_redirected e148)", 2024-05-15T23:08:00.984 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:59.060865+0000", 2024-05-15T23:08:00.984 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 1.806913693, 2024-05-15T23:08:00.984 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.057029234999999998, 2024-05-15T23:08:00.984 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.984 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.985 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.985 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.985 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.985 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.985 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 189304 2024-05-15T23:08:00.985 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.985 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.985 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.985 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.985 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.060865+0000", 2024-05-15T23:08:00.985 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.985 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.985 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.985 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.986 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.060865+0000", 2024-05-15T23:08:00.986 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.986 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.986 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.986 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.986 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.060866+0000", 2024-05-15T23:08:00.986 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.418e-06 2024-05-15T23:08:00.986 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.986 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.986 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.986 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.060871+0000", 2024-05-15T23:08:00.986 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8439999999999997e-06 2024-05-15T23:08:00.987 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.987 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.987 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.987 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.060879+0000", 2024-05-15T23:08:00.987 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2179999999999999e-06 2024-05-15T23:08:00.987 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.987 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.987 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.987 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.060885+0000", 2024-05-15T23:08:00.987 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4319999999999998e-06 2024-05-15T23:08:00.987 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.987 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.987 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.987 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.064364+0000", 2024-05-15T23:08:00.988 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003479223 2024-05-15T23:08:00.988 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.988 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.988 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.988 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.064460+0000", 2024-05-15T23:08:00.988 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6057000000000005e-05 2024-05-15T23:08:00.988 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.988 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.988 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.988 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.064563+0000", 2024-05-15T23:08:00.988 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00010246099999999999 2024-05-15T23:08:00.988 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.988 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.989 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.989 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.068990+0000", 2024-05-15T23:08:00.989 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0044275 2024-05-15T23:08:00.989 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.989 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.989 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.989 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.117801+0000", 2024-05-15T23:08:00.989 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.048811175999999998 2024-05-15T23:08:00.989 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.989 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.989 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.989 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.117857+0000", 2024-05-15T23:08:00.990 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5519000000000002e-05 2024-05-15T23:08:00.990 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.990 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.990 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.990 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.117894+0000", 2024-05-15T23:08:00.990 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7387e-05 2024-05-15T23:08:00.990 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.990 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.990 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.990 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.990 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.990 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:189309 2.3 2:c5cf9187:::benchmark_data_smithi090_78160_object27951:head [delete] snapc 0=[] ondisk+write+known_if_redirected e148)", 2024-05-15T23:08:00.990 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:59.066178+0000", 2024-05-15T23:08:00.990 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 1.801600702, 2024-05-15T23:08:00.991 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051966130999999999, 2024-05-15T23:08:00.991 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.991 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.991 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.991 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.991 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.991 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.991 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 189309 2024-05-15T23:08:00.991 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.991 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.991 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.991 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.991 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.066178+0000", 2024-05-15T23:08:00.991 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.992 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.992 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.992 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.992 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.066178+0000", 2024-05-15T23:08:00.992 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.992 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.992 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.992 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.992 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.066180+0000", 2024-05-15T23:08:00.992 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9400000000000001e-06 2024-05-15T23:08:00.992 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.992 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.992 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.992 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.066185+0000", 2024-05-15T23:08:00.993 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4360000000000001e-06 2024-05-15T23:08:00.993 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.993 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.993 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.993 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.066197+0000", 2024-05-15T23:08:00.993 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1344e-05 2024-05-15T23:08:00.993 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.993 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.993 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.993 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.066233+0000", 2024-05-15T23:08:00.993 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6389e-05 2024-05-15T23:08:00.993 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.994 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.994 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:00.994 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.066883+0000", 2024-05-15T23:08:00.994 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000649926 2024-05-15T23:08:00.994 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.994 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.994 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:00.994 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.066959+0000", 2024-05-15T23:08:00.994 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5804999999999997e-05 2024-05-15T23:08:00.994 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.994 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.994 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:08:00.994 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.067024+0000", 2024-05-15T23:08:00.994 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5035000000000003e-05 2024-05-15T23:08:00.995 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.995 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.995 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:00.995 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.069111+0000", 2024-05-15T23:08:00.995 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020869410000000001 2024-05-15T23:08:00.995 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.995 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.995 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:00.995 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.118058+0000", 2024-05-15T23:08:00.995 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.048947176000000002 2024-05-15T23:08:00.995 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.995 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.995 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:00.995 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.118105+0000", 2024-05-15T23:08:00.996 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7305999999999997e-05 2024-05-15T23:08:00.996 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.996 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.996 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:00.996 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.118144+0000", 2024-05-15T23:08:00.996 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8832999999999999e-05 2024-05-15T23:08:00.996 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.996 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:00.996 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:00.996 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.996 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.996 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:189312 2.2 2:45519a26:::benchmark_data_smithi090_78160_object27954:head [delete] snapc 0=[] ondisk+write+known_if_redirected e148)", 2024-05-15T23:08:00.996 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:59.071113+0000", 2024-05-15T23:08:00.996 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 1.7966654040000001, 2024-05-15T23:08:00.997 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046335904999999997, 2024-05-15T23:08:00.997 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:00.997 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:00.997 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:00.997 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:00.997 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:00.997 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:00.997 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 189312 2024-05-15T23:08:00.997 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.997 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:00.997 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.997 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:00.997 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.071113+0000", 2024-05-15T23:08:00.997 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.998 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.998 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.998 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:00.998 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.071113+0000", 2024-05-15T23:08:00.998 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:00.998 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.998 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.998 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:00.998 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.071115+0000", 2024-05-15T23:08:00.998 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6470000000000001e-06 2024-05-15T23:08:00.998 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.998 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.998 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:00.999 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.071121+0000", 2024-05-15T23:08:00.999 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.908e-06 2024-05-15T23:08:00.999 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.999 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.999 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:00.999 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.071134+0000", 2024-05-15T23:08:00.999 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3002e-05 2024-05-15T23:08:00.999 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.999 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:00.999 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:00.999 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.071142+0000", 2024-05-15T23:08:00.999 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6400000000000003e-06 2024-05-15T23:08:00.999 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:00.999 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:01.000 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:01.000 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.071203+0000", 2024-05-15T23:08:01.000 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0571e-05 2024-05-15T23:08:01.000 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:01.000 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:01.000 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:01.000 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.071330+0000", 2024-05-15T23:08:01.000 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012753499999999999 2024-05-15T23:08:01.000 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:01.000 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:01.001 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:08:01.001 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.071415+0000", 2024-05-15T23:08:01.001 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.4126000000000001e-05 2024-05-15T23:08:01.001 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:01.001 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:01.001 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:01.001 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.075075+0000", 2024-05-15T23:08:01.001 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036601139999999999 2024-05-15T23:08:01.001 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:01.001 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:01.001 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:01.001 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.117377+0000", 2024-05-15T23:08:01.001 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.042301876000000002 2024-05-15T23:08:01.001 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:01.002 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:01.002 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:01.002 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.117416+0000", 2024-05-15T23:08:01.002 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9057000000000002e-05 2024-05-15T23:08:01.002 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:01.002 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:01.002 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:01.002 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:59.117449+0000", 2024-05-15T23:08:01.002 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3429000000000001e-05 2024-05-15T23:08:01.002 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:01.002 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:01.002 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:01.002 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:01.003 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:01.003 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:08:01.003 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:08:01.044 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:01.075 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:08:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:01.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:08:01.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:08:01.217 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:01.245 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:08:01.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:01.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:08:01.349 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:01.378 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:01.495 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:01.523 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:01.675 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T23:08:01.728 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:01.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:01.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:08:01.873 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:08:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:01.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:08:02.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:08:02.019 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:08:02.085 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:02.119 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:08:02.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:02.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:08:02.228 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:08:02.256 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:02.280 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:08:02.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:02.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:08:02.386 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:08:02.463 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:02.500 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:08:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:02.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:08:02.559 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:02 smithi088 ceph-mon[44699]: pgmap v449: 17 pgs: 17 active+clean; 8.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:08:02.614 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:08:02.646 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:02.799 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:02 smithi084 ceph-mon[39784]: pgmap v449: 17 pgs: 17 active+clean; 8.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:08:02.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:02 smithi090 ceph-mon[44434]: pgmap v449: 17 pgs: 17 active+clean; 8.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:08:02.899 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:08:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:02.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:08:03.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:08:03.061 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:03.273 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:08:03.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:03.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:08:03.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:08:03.423 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:03.447 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:08:03.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:03.450 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:08:03.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:08:03.583 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:03.619 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:08:03.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:03.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:08:03.727 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:08:03.763 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:03.783 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:08:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:03.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:08:03.887 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:03.917 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:03.938 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:03.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:03.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:04.043 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:04.075 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:04.081 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:08:04.081 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:08:04.081 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2024-05-15T23:08:04.081 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:08:04.097 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:04.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:04.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:04.201 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:04.241 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:04.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:04 smithi084 ceph-mon[39784]: pgmap v450: 17 pgs: 17 active+clean; 8.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-05-15T23:08:04.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:04 smithi090 ceph-mon[44434]: pgmap v450: 17 pgs: 17 active+clean; 8.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-05-15T23:08:04.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:04 smithi088 ceph-mon[44699]: pgmap v450: 17 pgs: 17 active+clean; 8.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-05-15T23:08:04.895 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T23:08:05.201 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:08:05.734 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:05 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3733631430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:08:05.734 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:05 smithi088 ceph-mon[44699]: from='client.34919 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:08:05.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:05 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3733631430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:08:05.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:05 smithi090 ceph-mon[44434]: from='client.34919 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:08:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:05 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3733631430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:08:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:05 smithi084 ceph-mon[39784]: from='client.34919 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:08:06.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:06 smithi088 ceph-mon[44699]: pgmap v451: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-05-15T23:08:06.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:06 smithi088 ceph-mon[44699]: 2.1 scrub starts 2024-05-15T23:08:06.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:06 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:08:06.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:06 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:08:06.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:06 smithi088 ceph-mon[44699]: from='client.34919 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:08:06.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:06 smithi088 ceph-mon[44699]: osdmap e149: 12 total, 12 up, 12 in 2024-05-15T23:08:06.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:06 smithi090 ceph-mon[44434]: pgmap v451: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-05-15T23:08:06.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:06 smithi090 ceph-mon[44434]: 2.1 scrub starts 2024-05-15T23:08:06.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:06 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:08:06.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:06 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:08:06.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:06 smithi090 ceph-mon[44434]: from='client.34919 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:08:06.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:06 smithi090 ceph-mon[44434]: osdmap e149: 12 total, 12 up, 12 in 2024-05-15T23:08:06.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:06 smithi084 ceph-mon[39784]: pgmap v451: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-05-15T23:08:06.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:06 smithi084 ceph-mon[39784]: 2.1 scrub starts 2024-05-15T23:08:06.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:06 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:08:06.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:06 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:08:06.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:06 smithi084 ceph-mon[39784]: from='client.34919 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:08:06.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:06 smithi084 ceph-mon[39784]: osdmap e149: 12 total, 12 up, 12 in 2024-05-15T23:08:07.237 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:08:07.237 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:08:07.237 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:08:07.237 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:08:07.237 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:08:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:07 smithi088 ceph-mon[44699]: 2.1 scrub ok 2024-05-15T23:08:07.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:07 smithi090 ceph-mon[44434]: 2.1 scrub ok 2024-05-15T23:08:07.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:07 smithi084 ceph-mon[39784]: 2.1 scrub ok 2024-05-15T23:08:08.006 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T23:08:08.318 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:08:08.660 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:08:08.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:08 smithi090 ceph-mon[44434]: pgmap v453: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s 2024-05-15T23:08:08.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:08 smithi088 ceph-mon[44699]: pgmap v453: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s 2024-05-15T23:08:08.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:08 smithi084 ceph-mon[39784]: pgmap v453: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s 2024-05-15T23:08:10.245 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-05-15T23:08:10.293 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:08:10.293 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:08:10.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:08:10.293 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:08:10.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.293 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16826 2.7 2:e191f622:::benchmark_data_smithi090_78160_object16825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.293 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.017535+0000", 2024-05-15T23:08:10.293 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.27163822200001, 2024-05-15T23:08:10.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.112188986, 2024-05-15T23:08:10.293 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.293 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.293 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.294 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.294 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.294 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.294 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16826 2024-05-15T23:08:10.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.294 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017535+0000", 2024-05-15T23:08:10.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017535+0000", 2024-05-15T23:08:10.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017538+0000", 2024-05-15T23:08:10.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0340000000000001e-06 2024-05-15T23:08:10.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017667+0000", 2024-05-15T23:08:10.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012813000000000001 2024-05-15T23:08:10.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017682+0000", 2024-05-15T23:08:10.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5699e-05 2024-05-15T23:08:10.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017694+0000", 2024-05-15T23:08:10.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2156e-05 2024-05-15T23:08:10.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017787+0000", 2024-05-15T23:08:10.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2972000000000004e-05 2024-05-15T23:08:10.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017809+0000", 2024-05-15T23:08:10.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2014e-05 2024-05-15T23:08:10.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.126600+0000", 2024-05-15T23:08:10.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.108790918 2024-05-15T23:08:10.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.126718+0000", 2024-05-15T23:08:10.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011804800000000001 2024-05-15T23:08:10.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.126855+0000", 2024-05-15T23:08:10.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000136546 2024-05-15T23:08:10.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129204+0000", 2024-05-15T23:08:10.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023486689999999998 2024-05-15T23:08:10.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129254+0000", 2024-05-15T23:08:10.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0640999999999998e-05 2024-05-15T23:08:10.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129380+0000", 2024-05-15T23:08:10.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012590400000000001 2024-05-15T23:08:10.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129430+0000", 2024-05-15T23:08:10.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0371999999999998e-05 2024-05-15T23:08:10.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129724+0000", 2024-05-15T23:08:10.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00029388299999999999 2024-05-15T23:08:10.300 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.300 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.300 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.300 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16861 2.7 2:e22f6f28:::benchmark_data_smithi090_78160_object16860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.300 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.017713+0000", 2024-05-15T23:08:10.300 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.27146012199999, 2024-05-15T23:08:10.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.113245968, 2024-05-15T23:08:10.300 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.300 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.300 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.300 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.301 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.301 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.301 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16861 2024-05-15T23:08:10.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.301 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017713+0000", 2024-05-15T23:08:10.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.301 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017713+0000", 2024-05-15T23:08:10.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017714+0000", 2024-05-15T23:08:10.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1299999999999999e-07 2024-05-15T23:08:10.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017745+0000", 2024-05-15T23:08:10.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0784999999999997e-05 2024-05-15T23:08:10.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017751+0000", 2024-05-15T23:08:10.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6319999999999996e-06 2024-05-15T23:08:10.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017755+0000", 2024-05-15T23:08:10.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1880000000000004e-06 2024-05-15T23:08:10.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017844+0000", 2024-05-15T23:08:10.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.9328999999999998e-05 2024-05-15T23:08:10.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017864+0000", 2024-05-15T23:08:10.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0159000000000001e-05 2024-05-15T23:08:10.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.127841+0000", 2024-05-15T23:08:10.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10997657399999999 2024-05-15T23:08:10.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.127934+0000", 2024-05-15T23:08:10.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3522000000000003e-05 2024-05-15T23:08:10.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.128048+0000", 2024-05-15T23:08:10.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000113461 2024-05-15T23:08:10.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129335+0000", 2024-05-15T23:08:10.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0012868059999999999 2024-05-15T23:08:10.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130512+0000", 2024-05-15T23:08:10.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001176781 2024-05-15T23:08:10.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130637+0000", 2024-05-15T23:08:10.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012557399999999999 2024-05-15T23:08:10.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130673+0000", 2024-05-15T23:08:10.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5803000000000003e-05 2024-05-15T23:08:10.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130959+0000", 2024-05-15T23:08:10.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00028654100000000002 2024-05-15T23:08:10.307 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.307 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.307 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.307 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16878 2.7 2:eadea6ce:::benchmark_data_smithi090_78160_object16877:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.307 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.017762+0000", 2024-05-15T23:08:10.307 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.271411743, 2024-05-15T23:08:10.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.113672233, 2024-05-15T23:08:10.308 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.308 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.308 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.308 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.308 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.308 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.308 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16878 2024-05-15T23:08:10.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.308 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017762+0000", 2024-05-15T23:08:10.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017762+0000", 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017762+0000", 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7000000000000005e-07 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017796+0000", 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3522999999999997e-05 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017806+0000", 2024-05-15T23:08:10.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5580000000000006e-06 2024-05-15T23:08:10.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017810+0000", 2024-05-15T23:08:10.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3059999999999999e-06 2024-05-15T23:08:10.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017909+0000", 2024-05-15T23:08:10.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9451999999999996e-05 2024-05-15T23:08:10.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017921+0000", 2024-05-15T23:08:10.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1572e-05 2024-05-15T23:08:10.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.128568+0000", 2024-05-15T23:08:10.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.110646964 2024-05-15T23:08:10.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.128663+0000", 2024-05-15T23:08:10.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5342999999999998e-05 2024-05-15T23:08:10.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.128781+0000", 2024-05-15T23:08:10.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000117476 2024-05-15T23:08:10.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130466+0000", 2024-05-15T23:08:10.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00168541 2024-05-15T23:08:10.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130599+0000", 2024-05-15T23:08:10.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000132605 2024-05-15T23:08:10.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.131005+0000", 2024-05-15T23:08:10.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00040684899999999998 2024-05-15T23:08:10.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.131047+0000", 2024-05-15T23:08:10.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1885999999999998e-05 2024-05-15T23:08:10.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.131434+0000", 2024-05-15T23:08:10.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000386719 2024-05-15T23:08:10.314 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.314 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.314 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.314 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16883 2.7 2:e347a603:::benchmark_data_smithi090_78160_object16882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.314 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.017816+0000", 2024-05-15T23:08:10.314 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.27135790200001, 2024-05-15T23:08:10.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.114700729, 2024-05-15T23:08:10.314 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.314 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.314 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.314 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.315 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.315 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.315 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16883 2024-05-15T23:08:10.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.315 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017816+0000", 2024-05-15T23:08:10.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017816+0000", 2024-05-15T23:08:10.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017816+0000", 2024-05-15T23:08:10.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4899999999999995e-07 2024-05-15T23:08:10.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017890+0000", 2024-05-15T23:08:10.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3910000000000002e-05 2024-05-15T23:08:10.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017899+0000", 2024-05-15T23:08:10.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3389999999999993e-06 2024-05-15T23:08:10.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017906+0000", 2024-05-15T23:08:10.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1389999999999998e-06 2024-05-15T23:08:10.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017950+0000", 2024-05-15T23:08:10.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3900999999999999e-05 2024-05-15T23:08:10.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017954+0000", 2024-05-15T23:08:10.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5859999999999998e-06 2024-05-15T23:08:10.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129772+0000", 2024-05-15T23:08:10.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.111818023 2024-05-15T23:08:10.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129896+0000", 2024-05-15T23:08:10.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012374700000000001 2024-05-15T23:08:10.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130011+0000", 2024-05-15T23:08:10.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011510600000000001 2024-05-15T23:08:10.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.131450+0000", 2024-05-15T23:08:10.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001439271 2024-05-15T23:08:10.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.131966+0000", 2024-05-15T23:08:10.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00051586000000000004 2024-05-15T23:08:10.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132045+0000", 2024-05-15T23:08:10.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9083000000000002e-05 2024-05-15T23:08:10.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132107+0000", 2024-05-15T23:08:10.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1853999999999994e-05 2024-05-15T23:08:10.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132516+0000", 2024-05-15T23:08:10.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00040926100000000001 2024-05-15T23:08:10.320 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.320 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.320 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.321 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16908 2.7 2:eabdad50:::benchmark_data_smithi090_78160_object16907:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.321 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.017916+0000", 2024-05-15T23:08:10.321 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.271257248, 2024-05-15T23:08:10.321 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11791069899999999, 2024-05-15T23:08:10.321 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.321 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.321 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.321 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.321 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.321 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.321 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16908 2024-05-15T23:08:10.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.321 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017916+0000", 2024-05-15T23:08:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017916+0000", 2024-05-15T23:08:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.322 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017917+0000", 2024-05-15T23:08:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0100000000000004e-07 2024-05-15T23:08:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017986+0000", 2024-05-15T23:08:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8436999999999995e-05 2024-05-15T23:08:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017993+0000", 2024-05-15T23:08:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4719999999999998e-06 2024-05-15T23:08:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018000+0000", 2024-05-15T23:08:10.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5030000000000002e-06 2024-05-15T23:08:10.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018030+0000", 2024-05-15T23:08:10.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0035999999999998e-05 2024-05-15T23:08:10.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018036+0000", 2024-05-15T23:08:10.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5899999999999996e-06 2024-05-15T23:08:10.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132574+0000", 2024-05-15T23:08:10.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11453821 2024-05-15T23:08:10.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132718+0000", 2024-05-15T23:08:10.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014348599999999999 2024-05-15T23:08:10.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132919+0000", 2024-05-15T23:08:10.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000200776 2024-05-15T23:08:10.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135329+0000", 2024-05-15T23:08:10.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024106230000000002 2024-05-15T23:08:10.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135392+0000", 2024-05-15T23:08:10.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3010999999999997e-05 2024-05-15T23:08:10.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.326 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.326 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135410+0000", 2024-05-15T23:08:10.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8062000000000001e-05 2024-05-15T23:08:10.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.327 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.327 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.327 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135465+0000", 2024-05-15T23:08:10.327 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4889000000000003e-05 2024-05-15T23:08:10.327 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.327 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.327 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.327 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135827+0000", 2024-05-15T23:08:10.327 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00036180300000000001 2024-05-15T23:08:10.327 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.327 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.327 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.327 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.327 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.328 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16916 2.7 2:e8125f82:::benchmark_data_smithi090_78160_object16915:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.328 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018012+0000", 2024-05-15T23:08:10.328 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.271162037, 2024-05-15T23:08:10.328 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.120212466, 2024-05-15T23:08:10.328 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.328 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.328 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.328 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.328 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.328 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.328 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16916 2024-05-15T23:08:10.328 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.328 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.328 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.328 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018012+0000", 2024-05-15T23:08:10.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018012+0000", 2024-05-15T23:08:10.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018012+0000", 2024-05-15T23:08:10.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3200000000000004e-07 2024-05-15T23:08:10.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018073+0000", 2024-05-15T23:08:10.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0924999999999997e-05 2024-05-15T23:08:10.330 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.330 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018081+0000", 2024-05-15T23:08:10.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8120000000000008e-06 2024-05-15T23:08:10.330 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.330 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018086+0000", 2024-05-15T23:08:10.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8860000000000003e-06 2024-05-15T23:08:10.330 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.330 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018117+0000", 2024-05-15T23:08:10.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0602000000000003e-05 2024-05-15T23:08:10.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018121+0000", 2024-05-15T23:08:10.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3139999999999997e-06 2024-05-15T23:08:10.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.133639+0000", 2024-05-15T23:08:10.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.115518282 2024-05-15T23:08:10.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.332 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.133762+0000", 2024-05-15T23:08:10.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012241699999999999 2024-05-15T23:08:10.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.332 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.133892+0000", 2024-05-15T23:08:10.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000130004 2024-05-15T23:08:10.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.332 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.332 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135838+0000", 2024-05-15T23:08:10.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019465039999999999 2024-05-15T23:08:10.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.333 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.333 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135910+0000", 2024-05-15T23:08:10.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1577999999999998e-05 2024-05-15T23:08:10.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.333 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.333 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.137826+0000", 2024-05-15T23:08:10.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019163439999999999 2024-05-15T23:08:10.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.333 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.333 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.137885+0000", 2024-05-15T23:08:10.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8805999999999999e-05 2024-05-15T23:08:10.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.334 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.334 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.334 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138224+0000", 2024-05-15T23:08:10.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00033916000000000003 2024-05-15T23:08:10.334 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.334 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.334 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.334 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.334 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16920 2.7 2:ecf12bd4:::benchmark_data_smithi090_78160_object16919:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.334 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018095+0000", 2024-05-15T23:08:10.334 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.27107828699999, 2024-05-15T23:08:10.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.120514865, 2024-05-15T23:08:10.335 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.335 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.335 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.335 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.335 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.335 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.335 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16920 2024-05-15T23:08:10.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.335 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018095+0000", 2024-05-15T23:08:10.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018095+0000", 2024-05-15T23:08:10.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018096+0000", 2024-05-15T23:08:10.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1699999999999997e-07 2024-05-15T23:08:10.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018222+0000", 2024-05-15T23:08:10.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000125718 2024-05-15T23:08:10.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018230+0000", 2024-05-15T23:08:10.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9300000000000003e-06 2024-05-15T23:08:10.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018235+0000", 2024-05-15T23:08:10.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1619999999999999e-06 2024-05-15T23:08:10.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.338 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018269+0000", 2024-05-15T23:08:10.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4529000000000001e-05 2024-05-15T23:08:10.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.338 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.338 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018274+0000", 2024-05-15T23:08:10.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5970000000000002e-06 2024-05-15T23:08:10.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.338 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.338 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.134501+0000", 2024-05-15T23:08:10.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.116226648 2024-05-15T23:08:10.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.134644+0000", 2024-05-15T23:08:10.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014388500000000001 2024-05-15T23:08:10.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.134829+0000", 2024-05-15T23:08:10.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000184279 2024-05-15T23:08:10.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135983+0000", 2024-05-15T23:08:10.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0011546989999999999 2024-05-15T23:08:10.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.136000+0000", 2024-05-15T23:08:10.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6526999999999999e-05 2024-05-15T23:08:10.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138278+0000", 2024-05-15T23:08:10.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0022781310000000001 2024-05-15T23:08:10.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138316+0000", 2024-05-15T23:08:10.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7691999999999998e-05 2024-05-15T23:08:10.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138610+0000", 2024-05-15T23:08:10.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00029435099999999999 2024-05-15T23:08:10.341 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.341 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.341 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.341 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16933 2.7 2:e0794838:::benchmark_data_smithi090_78160_object16932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.342 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018244+0000", 2024-05-15T23:08:10.342 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.27092953899999, 2024-05-15T23:08:10.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12079975699999999, 2024-05-15T23:08:10.342 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.342 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.342 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.342 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.342 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.342 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.342 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16933 2024-05-15T23:08:10.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.342 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018244+0000", 2024-05-15T23:08:10.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018244+0000", 2024-05-15T23:08:10.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018245+0000", 2024-05-15T23:08:10.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9899999999999999e-07 2024-05-15T23:08:10.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.344 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018310+0000", 2024-05-15T23:08:10.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5549999999999994e-05 2024-05-15T23:08:10.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.344 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018317+0000", 2024-05-15T23:08:10.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.063e-06 2024-05-15T23:08:10.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.344 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018323+0000", 2024-05-15T23:08:10.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.04e-06 2024-05-15T23:08:10.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018350+0000", 2024-05-15T23:08:10.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7500000000000001e-05 2024-05-15T23:08:10.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018354+0000", 2024-05-15T23:08:10.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1500000000000001e-06 2024-05-15T23:08:10.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.136048+0000", 2024-05-15T23:08:10.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11769423900000001 2024-05-15T23:08:10.346 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.346 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.346 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.346 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.136216+0000", 2024-05-15T23:08:10.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167236 2024-05-15T23:08:10.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.136400+0000", 2024-05-15T23:08:10.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018410599999999999 2024-05-15T23:08:10.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138622+0000", 2024-05-15T23:08:10.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0022219190000000002 2024-05-15T23:08:10.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138682+0000", 2024-05-15T23:08:10.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0417000000000001e-05 2024-05-15T23:08:10.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138733+0000", 2024-05-15T23:08:10.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0949999999999998e-05 2024-05-15T23:08:10.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138768+0000", 2024-05-15T23:08:10.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4715000000000002e-05 2024-05-15T23:08:10.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.349 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.349 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139044+0000", 2024-05-15T23:08:10.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000276073 2024-05-15T23:08:10.349 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.349 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.349 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.349 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16935 2.7 2:e64d4890:::benchmark_data_smithi090_78160_object16934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.349 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018331+0000", 2024-05-15T23:08:10.349 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.270842403, 2024-05-15T23:08:10.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12112640099999999, 2024-05-15T23:08:10.349 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.350 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.350 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.350 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.350 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.350 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.350 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16935 2024-05-15T23:08:10.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.350 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018331+0000", 2024-05-15T23:08:10.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018331+0000", 2024-05-15T23:08:10.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018332+0000", 2024-05-15T23:08:10.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4900000000000005e-07 2024-05-15T23:08:10.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018394+0000", 2024-05-15T23:08:10.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1666000000000002e-05 2024-05-15T23:08:10.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018400+0000", 2024-05-15T23:08:10.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4999999999999996e-06 2024-05-15T23:08:10.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018404+0000", 2024-05-15T23:08:10.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2640000000000002e-06 2024-05-15T23:08:10.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018437+0000", 2024-05-15T23:08:10.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3002e-05 2024-05-15T23:08:10.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018442+0000", 2024-05-15T23:08:10.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3390000000000003e-06 2024-05-15T23:08:10.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.353 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.353 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.137007+0000", 2024-05-15T23:08:10.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.118565066 2024-05-15T23:08:10.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.137133+0000", 2024-05-15T23:08:10.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012647 2024-05-15T23:08:10.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.137305+0000", 2024-05-15T23:08:10.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017200399999999999 2024-05-15T23:08:10.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139055+0000", 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001750049 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139096+0000", 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0997000000000001e-05 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139142+0000", 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5354e-05 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139175+0000", 2024-05-15T23:08:10.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3386000000000002e-05 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139458+0000", 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00028255499999999998 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16947 2.7 2:edcf61d7:::benchmark_data_smithi090_78160_object16946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018413+0000", 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.270760767, 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124689859, 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.356 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.357 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.357 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.357 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.357 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.357 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16947 2024-05-15T23:08:10.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.357 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.357 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018413+0000", 2024-05-15T23:08:10.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.358 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.358 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018413+0000", 2024-05-15T23:08:10.358 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.358 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.358 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.358 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018414+0000", 2024-05-15T23:08:10.358 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0999999999999998e-07 2024-05-15T23:08:10.358 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.358 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.358 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018469+0000", 2024-05-15T23:08:10.358 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5869999999999999e-05 2024-05-15T23:08:10.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018476+0000", 2024-05-15T23:08:10.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.161e-06 2024-05-15T23:08:10.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018480+0000", 2024-05-15T23:08:10.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2069999999999997e-06 2024-05-15T23:08:10.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018510+0000", 2024-05-15T23:08:10.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0154999999999998e-05 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018515+0000", 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.074e-06 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139510+0000", 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.120994801 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139649+0000", 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013955200000000001 2024-05-15T23:08:10.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139826+0000", 2024-05-15T23:08:10.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017705999999999999 2024-05-15T23:08:10.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142625+0000", 2024-05-15T23:08:10.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027985340000000001 2024-05-15T23:08:10.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142693+0000", 2024-05-15T23:08:10.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8279000000000001e-05 2024-05-15T23:08:10.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142747+0000", 2024-05-15T23:08:10.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3838999999999997e-05 2024-05-15T23:08:10.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142788+0000", 2024-05-15T23:08:10.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0448999999999998e-05 2024-05-15T23:08:10.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143103+0000", 2024-05-15T23:08:10.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000315168 2024-05-15T23:08:10.363 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.363 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.363 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.363 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16960 2.7 2:e8496105:::benchmark_data_smithi090_78160_object16959:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.363 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018488+0000", 2024-05-15T23:08:10.363 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.270685125, 2024-05-15T23:08:10.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.126728282, 2024-05-15T23:08:10.363 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.363 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.363 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.363 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.363 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.363 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.364 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16960 2024-05-15T23:08:10.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.364 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.364 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.364 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018488+0000", 2024-05-15T23:08:10.364 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.364 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.364 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018488+0000", 2024-05-15T23:08:10.364 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.364 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018489+0000", 2024-05-15T23:08:10.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5199999999999996e-07 2024-05-15T23:08:10.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018549+0000", 2024-05-15T23:08:10.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9521999999999997e-05 2024-05-15T23:08:10.365 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.365 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.365 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.365 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018555+0000", 2024-05-15T23:08:10.365 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3620000000000002e-06 2024-05-15T23:08:10.366 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018559+0000", 2024-05-15T23:08:10.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1479999999999999e-06 2024-05-15T23:08:10.366 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018588+0000", 2024-05-15T23:08:10.366 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8538999999999999e-05 2024-05-15T23:08:10.366 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.366 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.366 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.366 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018592+0000", 2024-05-15T23:08:10.367 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1309999999999999e-06 2024-05-15T23:08:10.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.367 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.367 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.140349+0000", 2024-05-15T23:08:10.367 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.121757411 2024-05-15T23:08:10.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.367 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.367 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.140492+0000", 2024-05-15T23:08:10.367 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014264200000000001 2024-05-15T23:08:10.367 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.367 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.367 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.367 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.140606+0000", 2024-05-15T23:08:10.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000114437 2024-05-15T23:08:10.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.368 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143118+0000", 2024-05-15T23:08:10.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025118330000000002 2024-05-15T23:08:10.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.368 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143168+0000", 2024-05-15T23:08:10.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9667999999999999e-05 2024-05-15T23:08:10.368 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.368 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.369 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.144796+0000", 2024-05-15T23:08:10.369 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016276979999999999 2024-05-15T23:08:10.369 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.369 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.369 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.369 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.144849+0000", 2024-05-15T23:08:10.369 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3050000000000002e-05 2024-05-15T23:08:10.369 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.369 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.369 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.369 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145217+0000", 2024-05-15T23:08:10.369 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00036808899999999999 2024-05-15T23:08:10.369 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.369 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.370 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.370 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.370 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.370 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16971 2.7 2:ead74483:::benchmark_data_smithi090_78160_object16970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.370 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018569+0000", 2024-05-15T23:08:10.370 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.270604565, 2024-05-15T23:08:10.370 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12710465400000001, 2024-05-15T23:08:10.370 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.370 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.370 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.370 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.370 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.370 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16971 2024-05-15T23:08:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018569+0000", 2024-05-15T23:08:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018569+0000", 2024-05-15T23:08:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.372 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018570+0000", 2024-05-15T23:08:10.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1399999999999996e-07 2024-05-15T23:08:10.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.372 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.372 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.372 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018621+0000", 2024-05-15T23:08:10.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1483000000000002e-05 2024-05-15T23:08:10.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.372 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.372 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.372 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018627+0000", 2024-05-15T23:08:10.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2020000000000001e-06 2024-05-15T23:08:10.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.372 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.373 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018631+0000", 2024-05-15T23:08:10.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9149999999999998e-06 2024-05-15T23:08:10.373 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.373 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018662+0000", 2024-05-15T23:08:10.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0599000000000003e-05 2024-05-15T23:08:10.373 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.373 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.373 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018668+0000", 2024-05-15T23:08:10.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0990000000000004e-06 2024-05-15T23:08:10.373 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.374 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.374 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.374 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.141123+0000", 2024-05-15T23:08:10.374 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12245492700000001 2024-05-15T23:08:10.374 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.374 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.374 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.374 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.141282+0000", 2024-05-15T23:08:10.374 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015872099999999999 2024-05-15T23:08:10.374 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.374 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.374 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.374 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.141465+0000", 2024-05-15T23:08:10.375 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018367399999999999 2024-05-15T23:08:10.375 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.375 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.375 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.375 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143123+0000", 2024-05-15T23:08:10.375 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016577300000000001 2024-05-15T23:08:10.375 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.375 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.375 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.375 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143216+0000", 2024-05-15T23:08:10.375 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3317000000000001e-05 2024-05-15T23:08:10.375 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.375 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.375 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145289+0000", 2024-05-15T23:08:10.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020730890000000002 2024-05-15T23:08:10.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.376 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.376 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145338+0000", 2024-05-15T23:08:10.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8353000000000003e-05 2024-05-15T23:08:10.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.376 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.376 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145674+0000", 2024-05-15T23:08:10.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00033583099999999998 2024-05-15T23:08:10.376 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.377 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.377 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.377 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16975 2.7 2:e5d20b63:::benchmark_data_smithi090_78160_object16974:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.377 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018639+0000", 2024-05-15T23:08:10.377 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.270535018, 2024-05-15T23:08:10.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12747009100000001, 2024-05-15T23:08:10.377 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.377 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.377 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.377 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.377 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.377 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.377 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16975 2024-05-15T23:08:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018639+0000", 2024-05-15T23:08:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018639+0000", 2024-05-15T23:08:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.379 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018639+0000", 2024-05-15T23:08:10.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1600000000000001e-07 2024-05-15T23:08:10.379 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.379 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.379 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018690+0000", 2024-05-15T23:08:10.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1094000000000002e-05 2024-05-15T23:08:10.379 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.379 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.379 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018696+0000", 2024-05-15T23:08:10.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7540000000000003e-06 2024-05-15T23:08:10.379 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.380 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.380 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018700+0000", 2024-05-15T23:08:10.380 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9330000000000003e-06 2024-05-15T23:08:10.380 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.380 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.380 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018728+0000", 2024-05-15T23:08:10.380 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7447e-05 2024-05-15T23:08:10.380 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.380 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.380 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018732+0000", 2024-05-15T23:08:10.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.126e-06 2024-05-15T23:08:10.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.141950+0000", 2024-05-15T23:08:10.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12321836799999999 2024-05-15T23:08:10.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142073+0000", 2024-05-15T23:08:10.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012294000000000001 2024-05-15T23:08:10.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142192+0000", 2024-05-15T23:08:10.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000118751 2024-05-15T23:08:10.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143227+0000", 2024-05-15T23:08:10.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010353199999999999 2024-05-15T23:08:10.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143266+0000", 2024-05-15T23:08:10.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9242999999999997e-05 2024-05-15T23:08:10.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145771+0000", 2024-05-15T23:08:10.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025051489999999999 2024-05-15T23:08:10.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145808+0000", 2024-05-15T23:08:10.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6304999999999999e-05 2024-05-15T23:08:10.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146109+0000", 2024-05-15T23:08:10.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000300945 2024-05-15T23:08:10.384 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.384 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.384 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.384 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16985 2.7 2:e193a409:::benchmark_data_smithi090_78160_object16984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.384 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018708+0000", 2024-05-15T23:08:10.384 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.270466087, 2024-05-15T23:08:10.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.127922702, 2024-05-15T23:08:10.384 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.384 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.384 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.384 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.385 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.385 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.385 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16985 2024-05-15T23:08:10.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.385 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018708+0000", 2024-05-15T23:08:10.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018708+0000", 2024-05-15T23:08:10.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.386 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.386 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.386 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018708+0000", 2024-05-15T23:08:10.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6400000000000002e-07 2024-05-15T23:08:10.386 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.386 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.386 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.386 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018762+0000", 2024-05-15T23:08:10.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3594000000000002e-05 2024-05-15T23:08:10.386 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.386 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.386 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.386 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018768+0000", 2024-05-15T23:08:10.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9869999999999996e-06 2024-05-15T23:08:10.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018772+0000", 2024-05-15T23:08:10.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9600000000000002e-06 2024-05-15T23:08:10.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018797+0000", 2024-05-15T23:08:10.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5701e-05 2024-05-15T23:08:10.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.388 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018801+0000", 2024-05-15T23:08:10.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0280000000000002e-06 2024-05-15T23:08:10.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.388 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.388 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143297+0000", 2024-05-15T23:08:10.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124495859 2024-05-15T23:08:10.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.388 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.388 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143387+0000", 2024-05-15T23:08:10.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0144999999999998e-05 2024-05-15T23:08:10.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143541+0000", 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015373200000000001 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145726+0000", 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0021849360000000002 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145737+0000", 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0769e-05 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146249+0000", 2024-05-15T23:08:10.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00051189599999999997 2024-05-15T23:08:10.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146303+0000", 2024-05-15T23:08:10.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3795000000000002e-05 2024-05-15T23:08:10.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146630+0000", 2024-05-15T23:08:10.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00032763600000000002 2024-05-15T23:08:10.390 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.390 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.391 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.391 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16996 2.7 2:edffe2d2:::benchmark_data_smithi090_78160_object16995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.391 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018779+0000", 2024-05-15T23:08:10.391 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.270394095, 2024-05-15T23:08:10.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128263827, 2024-05-15T23:08:10.391 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.391 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.391 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.391 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.391 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.391 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.391 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16996 2024-05-15T23:08:10.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.392 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018779+0000", 2024-05-15T23:08:10.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018779+0000", 2024-05-15T23:08:10.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018780+0000", 2024-05-15T23:08:10.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1600000000000001e-07 2024-05-15T23:08:10.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018831+0000", 2024-05-15T23:08:10.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1023000000000003e-05 2024-05-15T23:08:10.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018837+0000", 2024-05-15T23:08:10.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0059999999999998e-06 2024-05-15T23:08:10.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018841+0000", 2024-05-15T23:08:10.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9480000000000001e-06 2024-05-15T23:08:10.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018868+0000", 2024-05-15T23:08:10.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6917999999999999e-05 2024-05-15T23:08:10.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018872+0000", 2024-05-15T23:08:10.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1239999999999998e-06 2024-05-15T23:08:10.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.144067+0000", 2024-05-15T23:08:10.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.125194783 2024-05-15T23:08:10.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.144155+0000", 2024-05-15T23:08:10.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8103999999999995e-05 2024-05-15T23:08:10.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.144283+0000", 2024-05-15T23:08:10.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012799099999999999 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145742+0000", 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0014585710000000001 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146189+0000", 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00044775400000000002 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146681+0000", 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00049144900000000003 2024-05-15T23:08:10.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146718+0000", 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7146e-05 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.147043+0000", 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00032529400000000001 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:17021 2.7 2:e966b9dc:::benchmark_data_smithi090_78160_object17020:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018848+0000", 2024-05-15T23:08:10.397 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.27032531, 2024-05-15T23:08:10.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.132449714, 2024-05-15T23:08:10.398 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.398 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.398 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.398 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.398 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.398 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.398 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 17021 2024-05-15T23:08:10.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.398 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018848+0000", 2024-05-15T23:08:10.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018848+0000", 2024-05-15T23:08:10.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018849+0000", 2024-05-15T23:08:10.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0500000000000003e-07 2024-05-15T23:08:10.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018901+0000", 2024-05-15T23:08:10.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1912e-05 2024-05-15T23:08:10.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018907+0000", 2024-05-15T23:08:10.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9490000000000001e-06 2024-05-15T23:08:10.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018911+0000", 2024-05-15T23:08:10.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8600000000000003e-06 2024-05-15T23:08:10.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018939+0000", 2024-05-15T23:08:10.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8167e-05 2024-05-15T23:08:10.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:08:10.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018943+0000", 2024-05-15T23:08:10.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2540000000000003e-06 2024-05-15T23:08:10.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.147139+0000", 2024-05-15T23:08:10.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12819576399999999 2024-05-15T23:08:10.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.147298+0000", 2024-05-15T23:08:10.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015885199999999999 2024-05-15T23:08:10.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.147472+0000", 2024-05-15T23:08:10.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017376300000000001 2024-05-15T23:08:10.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.150655+0000", 2024-05-15T23:08:10.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031836740000000001 2024-05-15T23:08:10.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.150707+0000", 2024-05-15T23:08:10.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1873000000000003e-05 2024-05-15T23:08:10.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.150771+0000", 2024-05-15T23:08:10.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4231000000000002e-05 2024-05-15T23:08:10.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.150833+0000", 2024-05-15T23:08:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2126999999999997e-05 2024-05-15T23:08:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.151298+0000", 2024-05-15T23:08:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00046458300000000002 2024-05-15T23:08:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:17027 2.7 2:ed2336b5:::benchmark_data_smithi090_78160_object17026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e110)", 2024-05-15T23:08:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.132972+0000", 2024-05-15T23:08:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 120.15620206200001, 2024-05-15T23:08:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020623263999999999, 2024-05-15T23:08:10.405 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.405 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.405 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.405 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.405 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.405 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.405 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 17027 2024-05-15T23:08:10.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.405 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.405 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132972+0000", 2024-05-15T23:08:10.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132972+0000", 2024-05-15T23:08:10.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132974+0000", 2024-05-15T23:08:10.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5100000000000001e-06 2024-05-15T23:08:10.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.133071+0000", 2024-05-15T23:08:10.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6658999999999997e-05 2024-05-15T23:08:10.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.133086+0000", 2024-05-15T23:08:10.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5622999999999999e-05 2024-05-15T23:08:10.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.133096+0000", 2024-05-15T23:08:10.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6129999999999993e-06 2024-05-15T23:08:10.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.148146+0000", 2024-05-15T23:08:10.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015050071 2024-05-15T23:08:10.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.148292+0000", 2024-05-15T23:08:10.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145878 2024-05-15T23:08:10.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.148451+0000", 2024-05-15T23:08:10.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015942899999999999 2024-05-15T23:08:10.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.151316+0000", 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002864391 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.153088+0000", 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0017720209999999999 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.153160+0000", 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1786999999999996e-05 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.153218+0000", 2024-05-15T23:08:10.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8862e-05 2024-05-15T23:08:10.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.153595+0000", 2024-05-15T23:08:10.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00037641999999999998 2024-05-15T23:08:10.410 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.410 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.410 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.410 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:20323 2.7 2:eb28a62d:::benchmark_data_smithi090_78160_object20322:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e110)", 2024-05-15T23:08:10.411 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.934751+0000", 2024-05-15T23:08:10.411 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 119.354422525, 2024-05-15T23:08:10.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.017203597000000001, 2024-05-15T23:08:10.411 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.411 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.411 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.411 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.411 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.411 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.411 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 20323 2024-05-15T23:08:10.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.411 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.934751+0000", 2024-05-15T23:08:10.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.934751+0000", 2024-05-15T23:08:10.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.934752+0000", 2024-05-15T23:08:10.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5300000000000003e-07 2024-05-15T23:08:10.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.934810+0000", 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7992999999999999e-05 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.934818+0000", 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3890000000000005e-06 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.934825+0000", 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8770000000000004e-06 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.944466+0000", 2024-05-15T23:08:10.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096409000000000009 2024-05-15T23:08:10.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.944552+0000", 2024-05-15T23:08:10.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5923999999999999e-05 2024-05-15T23:08:10.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.944644+0000", 2024-05-15T23:08:10.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.1718999999999994e-05 2024-05-15T23:08:10.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.946562+0000", 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019179749999999999 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.951415+0000", 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0048529250000000001 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.951490+0000", 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5202000000000003e-05 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.951553+0000", 2024-05-15T23:08:10.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2769999999999997e-05 2024-05-15T23:08:10.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.951955+0000", 2024-05-15T23:08:10.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00040206999999999998 2024-05-15T23:08:10.416 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.416 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.416 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.416 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:21793 2.7 2:e0c140ee:::benchmark_data_smithi090_78160_object21792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e110)", 2024-05-15T23:08:10.417 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:11.322003+0000", 2024-05-15T23:08:10.417 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 118.967170858, 2024-05-15T23:08:10.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.014509948, 2024-05-15T23:08:10.417 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.417 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.417 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.417 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.417 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.417 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.417 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 21793 2024-05-15T23:08:10.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.417 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:11.322003+0000", 2024-05-15T23:08:10.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:11.322003+0000", 2024-05-15T23:08:10.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:11.322005+0000", 2024-05-15T23:08:10.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0150000000000002e-06 2024-05-15T23:08:10.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:11.322104+0000", 2024-05-15T23:08:10.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9714000000000001e-05 2024-05-15T23:08:10.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:11.322115+0000", 2024-05-15T23:08:10.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0959e-05 2024-05-15T23:08:10.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:11.322122+0000", 2024-05-15T23:08:10.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0419999999999997e-06 2024-05-15T23:08:10.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:11.324934+0000", 2024-05-15T23:08:10.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028113330000000001 2024-05-15T23:08:10.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:11.325035+0000", 2024-05-15T23:08:10.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000101112 2024-05-15T23:08:10.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:11.325136+0000", 2024-05-15T23:08:10.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000100976 2024-05-15T23:08:10.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:11.326252+0000", 2024-05-15T23:08:10.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0011163589999999999 2024-05-15T23:08:10.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:11.327193+0000", 2024-05-15T23:08:10.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00094052199999999995 2024-05-15T23:08:10.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:11.335941+0000", 2024-05-15T23:08:10.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0087482389999999997 2024-05-15T23:08:10.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:11.336026+0000", 2024-05-15T23:08:10.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.4645e-05 2024-05-15T23:08:10.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:11.336513+0000", 2024-05-15T23:08:10.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00048703199999999998 2024-05-15T23:08:10.422 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.423 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.423 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.423 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:33746 2.7 2:e5f96629:::benchmark_data_smithi090_78160_object33745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-05-15T23:08:10.423 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:14.821951+0000", 2024-05-15T23:08:10.423 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 115.46722251200001, 2024-05-15T23:08:10.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.014276580000000001, 2024-05-15T23:08:10.423 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:08:10.423 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:08:10.423 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:10.423 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:08:10.423 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 33746 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:14.821951+0000", 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:14.821951+0000", 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:08:10.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:14.821952+0000", 2024-05-15T23:08:10.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0130000000000001e-06 2024-05-15T23:08:10.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:08:10.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:14.822002+0000", 2024-05-15T23:08:10.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0380000000000002e-05 2024-05-15T23:08:10.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:08:10.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:14.822011+0000", 2024-05-15T23:08:10.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2470000000000008e-06 2024-05-15T23:08:10.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:08:10.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:14.822015+0000", 2024-05-15T23:08:10.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7199999999999997e-06 2024-05-15T23:08:10.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:08:10.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:14.822089+0000", 2024-05-15T23:08:10.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3852999999999999e-05 2024-05-15T23:08:10.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:08:10.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:14.822223+0000", 2024-05-15T23:08:10.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013379099999999999 2024-05-15T23:08:10.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:08:10.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:14.822352+0000", 2024-05-15T23:08:10.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012882200000000001 2024-05-15T23:08:10.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:08:10.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:14.823161+0000", 2024-05-15T23:08:10.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00080948299999999995 2024-05-15T23:08:10.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:14.823673+0000", 2024-05-15T23:08:10.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00051172800000000003 2024-05-15T23:08:10.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:10.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:14.835950+0000", 2024-05-15T23:08:10.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.012277308000000001 2024-05-15T23:08:10.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:08:10.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:14.836021+0000", 2024-05-15T23:08:10.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0349e-05 2024-05-15T23:08:10.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:08:10.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:08:10.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:08:10.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:14.836228+0000", 2024-05-15T23:08:10.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020688600000000001 2024-05-15T23:08:10.429 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.429 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.429 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.429 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:08:10.429 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:08:10.429 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:08:10.560 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:10.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:10.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:10.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:10 smithi088 ceph-mon[44699]: pgmap v454: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:08:10.670 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:10.701 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:10.739 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:10.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:10.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:10.761 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:10 smithi084 ceph-mon[39784]: pgmap v454: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:08:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:10 smithi090 ceph-mon[44434]: pgmap v454: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:08:10.830 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:10.846 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:10.874 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:11.118 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T23:08:11.123 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:11.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:11.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:11.227 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:08:11.274 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:11.432 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:08:11.485 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:08:11.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:11.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:08:11.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:08:11.649 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:11.724 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:08:11.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:11.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:08:11.779 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:08:11.810 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:08:11.832 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:11.850 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:08:11.863 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:11.882 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:08:11.905 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:08:11.929 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:08:11.955 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:08:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:11.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:11.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:11.986 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:08:12.021 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:08:12.059 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:08:12.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:08:12.092 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:08:12.121 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:12.125 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:08:12.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:08:12.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:08:12.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:08:12.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:08:12.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:08:12.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:08:12.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:08:12.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:08:12.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:08:12.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:08:12.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:08:12.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:08:12.150 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:08:12.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:12.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:08:12.257 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:12.437 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:12.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:12.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:12.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:08:12.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:12 smithi088 ceph-mon[44699]: pgmap v455: 17 pgs: 17 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:08:12.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:12 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1878084886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:08:12.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:12.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:12.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:12.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:12.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:12.575 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:12.597 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:08:12.602 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:08:12.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:12.605 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:08:12.706 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:12.736 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:12.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:12 smithi090 ceph-mon[44434]: pgmap v455: 17 pgs: 17 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:08:12.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:12 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1878084886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:08:12.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:12.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:12.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:12.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:12.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:12.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:12 smithi084 ceph-mon[39784]: pgmap v455: 17 pgs: 17 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:08:12.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:12 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1878084886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:08:12.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:12.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:12.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:12.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:12.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:12.937 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:12.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:13.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:08:13.082 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:13.114 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:08:13.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:13.120 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:08:13.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:08:13.258 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:13.285 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:08:13.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:13.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:08:13.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:08:13.427 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:13.450 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:08:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:13.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:08:13.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:08:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.602 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:08:13.602 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:08:13.603 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:13.650 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:08:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:13.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:08:13.754 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:13.785 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:13.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.862 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:13.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:13.968 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:13.996 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:14.351 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T23:08:14.379 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:14.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:14.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:14.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:08:14.484 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:14.498 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-05-15T23:08:14.563 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:14.677 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:08:14.785 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:08:14.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:14 smithi084 ceph-mon[39784]: pgmap v456: 17 pgs: 17 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:08:14.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:14.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:08:14.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:14 smithi090 ceph-mon[44434]: pgmap v456: 17 pgs: 17 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:08:14.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:14 smithi088 ceph-mon[44699]: pgmap v456: 17 pgs: 17 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:08:14.897 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:08:14.944 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:14.974 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:08:14.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:14.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:08:15.078 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:15.128 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:15.507 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:15.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:15.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:15.538 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:15.614 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:08:15.652 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:08:15.700 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:16.005 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:08:16.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:16.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:08:16.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:08:16.156 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:16.176 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:08:16.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:16.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:08:16.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:08:16.317 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:08:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:16.356 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:08:16.457 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:16.488 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:16.540 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:16 smithi090 ceph-mon[44434]: pgmap v457: 17 pgs: 17 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:08:16.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:16 smithi084 ceph-mon[39784]: pgmap v457: 17 pgs: 17 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:08:16.765 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:08:16.765 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:08:16.790 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:16 smithi088 ceph-mon[44699]: pgmap v457: 17 pgs: 17 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:08:17.041 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:17.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:17.149 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:08:17.185 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:17.210 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:08:17.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:17.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:08:17.314 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:17.385 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:17.670 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:17.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:17.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:17.720 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:17.769 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T23:08:17.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:08:17.823 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:17.851 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:08:17.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:17.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:08:17.957 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:17.988 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:18.014 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:18.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:18.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:18.077 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:08:18.121 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:18.168 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:18.242 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:18.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:18.352 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:08:18.409 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:18.442 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:08:18.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:18.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:08:18.549 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:18 smithi088 ceph-mon[44699]: pgmap v458: 17 pgs: 17 active+clean; 7.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:08:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:18 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2738086795' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:08:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:18 smithi088 ceph-mon[44699]: from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:08:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:18 smithi088 ceph-mon[44699]: 2.5 scrub starts 2024-05-15T23:08:18.592 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:08:18.593 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:18.743 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:18.749 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:18 smithi084 ceph-mon[39784]: pgmap v458: 17 pgs: 17 active+clean; 7.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:08:18.749 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:18 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2738086795' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:08:18.749 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:18 smithi084 ceph-mon[39784]: from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:08:18.749 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:18 smithi084 ceph-mon[39784]: 2.5 scrub starts 2024-05-15T23:08:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:18.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:18.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:18 smithi090 ceph-mon[44434]: pgmap v458: 17 pgs: 17 active+clean; 7.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:08:18.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:18 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2738086795' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:08:18.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:18 smithi090 ceph-mon[44434]: from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:08:18.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:18 smithi090 ceph-mon[44434]: 2.5 scrub starts 2024-05-15T23:08:18.851 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:08:18.882 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:18.903 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:08:18.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:18.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:08:19.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:08:19.050 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:19.081 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:08:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:19.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:08:19.187 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:19.215 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:19.415 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:19.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:19.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:19.519 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:08:19.549 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:19.752 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:08:19.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:19.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:08:19.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:19 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:08:19.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:19 smithi088 ceph-mon[44699]: from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:08:19.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:19 smithi088 ceph-mon[44699]: osdmap e150: 12 total, 12 up, 12 in 2024-05-15T23:08:19.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:08:19.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:19 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:08:19.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:19 smithi084 ceph-mon[39784]: from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:08:19.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:19 smithi084 ceph-mon[39784]: osdmap e150: 12 total, 12 up, 12 in 2024-05-15T23:08:19.928 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:20.019 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:08:20.019 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:08:20.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:19 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:08:20.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:19 smithi090 ceph-mon[44434]: from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:08:20.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:19 smithi090 ceph-mon[44434]: osdmap e150: 12 total, 12 up, 12 in 2024-05-15T23:08:20.385 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-05-15T23:08:20.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:20 smithi084 ceph-mon[39784]: pgmap v460: 17 pgs: 17 active+clean; 7.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s 2024-05-15T23:08:20.783 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:20 smithi088 ceph-mon[44699]: pgmap v460: 17 pgs: 17 active+clean; 7.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s 2024-05-15T23:08:20.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T23:08:20.859 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:20 smithi090 ceph-mon[44434]: pgmap v460: 17 pgs: 17 active+clean; 7.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s 2024-05-15T23:08:20.891 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:08:20.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:20.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:08:20.962 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:21.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:08:21.074 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:21.174 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:08:21.233 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:08:21.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:21.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:08:21.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:08:21.394 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:21.664 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:08:21.862 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:08:21.906 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:08:21.945 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:08:21.976 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:08:22.003 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:08:22.041 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:08:22.070 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:08:22.116 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:08:22.148 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:08:22.171 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:08:22.186 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:08:22.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:22.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:08:22.236 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:08:22.267 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:08:22.287 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:08:22.287 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:08:22.288 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:08:22.288 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:08:22.288 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:08:22.288 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:08:22.288 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:08:22.288 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:08:22.288 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:08:22.288 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:08:22.288 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:08:22.288 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:08:22.290 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:22.319 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:22.341 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:22.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:22.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:22.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:08:22.484 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:22.697 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:08:22.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:22.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:08:22.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:08:22.846 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:22 smithi090 ceph-mon[44434]: pgmap v461: 17 pgs: 17 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:08:22.846 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/619863408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:08:22.846 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:22.846 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:22.846 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:22.846 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:22.846 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:22.850 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:22.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:22 smithi084 ceph-mon[39784]: pgmap v461: 17 pgs: 17 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:08:22.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/619863408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:08:22.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:22.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:22.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:22.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:22.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.061 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:08:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:23.066 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:08:23.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:22 smithi088 ceph-mon[44699]: pgmap v461: 17 pgs: 17 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:08:23.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/619863408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:08:23.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.166 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:23.194 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:23.217 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:23.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:23.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:23.353 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:23.374 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:23.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:23.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:23.408 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:08:23.408 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:08:23.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.408 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:217290 2.5 2:ad07ac80:::benchmark_data_smithi090_78160_object55932:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:08:23.408 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:23.391248+0000", 2024-05-15T23:08:23.409 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.015427629999999999, 2024-05-15T23:08:23.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015456826999999999, 2024-05-15T23:08:23.409 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:23.409 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:23.409 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:08:23.409 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:23.409 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:23.409 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:23.409 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 217290 2024-05-15T23:08:23.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.409 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:23.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:23.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.391248+0000", 2024-05-15T23:08:23.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:23.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:23.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.391248+0000", 2024-05-15T23:08:23.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:23.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:23.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.391249+0000", 2024-05-15T23:08:23.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.06e-06 2024-05-15T23:08:23.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:23.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.391253+0000", 2024-05-15T23:08:23.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1230000000000002e-06 2024-05-15T23:08:23.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:23.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.391262+0000", 2024-05-15T23:08:23.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6430000000000001e-06 2024-05-15T23:08:23.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:23.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.391269+0000", 2024-05-15T23:08:23.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.306e-06 2024-05-15T23:08:23.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:23.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.396135+0000", 2024-05-15T23:08:23.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048658290000000003 2024-05-15T23:08:23.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:23.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.396277+0000", 2024-05-15T23:08:23.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000142373 2024-05-15T23:08:23.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:08:23.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.396407+0000", 2024-05-15T23:08:23.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012996399999999999 2024-05-15T23:08:23.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:23.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.406159+0000", 2024-05-15T23:08:23.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0097512129999999999 2024-05-15T23:08:23.413 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:23.413 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:23.413 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:23.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.414 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:217291 2.5 2:ab66bff7:::benchmark_data_smithi090_78160_object55933:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:08:23.414 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:23.391290+0000", 2024-05-15T23:08:23.414 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.015385950000000001, 2024-05-15T23:08:23.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015503467999999999, 2024-05-15T23:08:23.414 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:23.414 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:23.414 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:08:23.414 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:23.414 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:23.414 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:23.414 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 217291 2024-05-15T23:08:23.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.415 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:23.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:23.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.391290+0000", 2024-05-15T23:08:23.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:23.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:23.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.391290+0000", 2024-05-15T23:08:23.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:23.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:23.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.391291+0000", 2024-05-15T23:08:23.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1999999999999998e-07 2024-05-15T23:08:23.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:23.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.391294+0000", 2024-05-15T23:08:23.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.168e-06 2024-05-15T23:08:23.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:23.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.391300+0000", 2024-05-15T23:08:23.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5919999999999997e-06 2024-05-15T23:08:23.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:23.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.391306+0000", 2024-05-15T23:08:23.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6420000000000004e-06 2024-05-15T23:08:23.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:23.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.401144+0000", 2024-05-15T23:08:23.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098379450000000007 2024-05-15T23:08:23.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:23.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.401276+0000", 2024-05-15T23:08:23.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013160599999999999 2024-05-15T23:08:23.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:08:23.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.401402+0000", 2024-05-15T23:08:23.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000126377 2024-05-15T23:08:23.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:23.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.406321+0000", 2024-05-15T23:08:23.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049193939999999997 2024-05-15T23:08:23.418 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:23.418 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:23.418 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:23.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.419 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:217302 2.b 2:d58bc91d:::benchmark_data_smithi090_78160_object55944:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:08:23.419 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:23.399355+0000", 2024-05-15T23:08:23.419 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0073203510000000001, 2024-05-15T23:08:23.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075193389999999999, 2024-05-15T23:08:23.419 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:23.419 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:23.419 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:08:23.419 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:23.419 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:23.419 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:23.419 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 217302 2024-05-15T23:08:23.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.420 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:23.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:23.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.399355+0000", 2024-05-15T23:08:23.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:23.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:23.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.399355+0000", 2024-05-15T23:08:23.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:23.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.399357+0000", 2024-05-15T23:08:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8369999999999999e-06 2024-05-15T23:08:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.399363+0000", 2024-05-15T23:08:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8769999999999998e-06 2024-05-15T23:08:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.399373+0000", 2024-05-15T23:08:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0356999999999999e-05 2024-05-15T23:08:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.399379+0000", 2024-05-15T23:08:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9220000000000002e-06 2024-05-15T23:08:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.399433+0000", 2024-05-15T23:08:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4119999999999997e-05 2024-05-15T23:08:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.399548+0000", 2024-05-15T23:08:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011440800000000001 2024-05-15T23:08:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 11", 2024-05-15T23:08:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.399659+0000", 2024-05-15T23:08:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110937 2024-05-15T23:08:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.405263+0000", 2024-05-15T23:08:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0056041260000000001 2024-05-15T23:08:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:23.424 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:23.424 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:23.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.424 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:217287 2.5 2:a26ee679:::benchmark_data_smithi090_78160_object55929:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:08:23.424 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:23.390839+0000", 2024-05-15T23:08:23.424 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.015836406000000001, 2024-05-15T23:08:23.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016117979000000001, 2024-05-15T23:08:23.424 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:23.424 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:23.424 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:08:23.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:23.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:23.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:23.424 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 217287 2024-05-15T23:08:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.390839+0000", 2024-05-15T23:08:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.390839+0000", 2024-05-15T23:08:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.390842+0000", 2024-05-15T23:08:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.553e-06 2024-05-15T23:08:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.390850+0000", 2024-05-15T23:08:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7600000000000002e-06 2024-05-15T23:08:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.390859+0000", 2024-05-15T23:08:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8099999999999992e-06 2024-05-15T23:08:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.390869+0000", 2024-05-15T23:08:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.346e-06 2024-05-15T23:08:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.390937+0000", 2024-05-15T23:08:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8628999999999996e-05 2024-05-15T23:08:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.391095+0000", 2024-05-15T23:08:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015734899999999999 2024-05-15T23:08:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:08:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.391248+0000", 2024-05-15T23:08:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015345100000000001 2024-05-15T23:08:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.406152+0000", 2024-05-15T23:08:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014903733000000001 2024-05-15T23:08:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-05-15T23:08:23.429 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 4 2024-05-15T23:08:23.429 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:08:23.477 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:23.504 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:23.523 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:23.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:23.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:08:23.669 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:23.779 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.779 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.779 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.779 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.779 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.779 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.780 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:24.141 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:24.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T23:08:24.620 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:08:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:24.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:08:24.660 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-05-15T23:08:24.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:08:24.774 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:24.792 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:08:24.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:24.799 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:08:24.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:24 smithi090 ceph-mon[44434]: pgmap v462: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:08:24.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:24 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:08:24.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:24 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:08:24.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:24 smithi084 ceph-mon[39784]: pgmap v462: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:08:24.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:24 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:08:24.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:24 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:08:24.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:08:24.937 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:24.982 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:08:24.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:24.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:08:25.006 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:08:25.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:24 smithi088 ceph-mon[44699]: pgmap v462: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:08:25.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:24 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:08:25.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:24 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:08:25.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:08:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:08:25.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:25.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:08:25.376 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:08:25.420 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:25.485 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:08:25.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:25.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:08:25.537 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:25.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:25.621 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:25.672 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:25.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:25.785 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:25.812 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:25.833 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:25.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:25.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:25.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:25 smithi084 ceph-mon[39784]: osdmap e151: 12 total, 12 up, 12 in 2024-05-15T23:08:25.939 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:25.993 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:26.006 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:25 smithi090 ceph-mon[44434]: osdmap e151: 12 total, 12 up, 12 in 2024-05-15T23:08:26.032 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:26.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:26.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:26.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:25 smithi088 ceph-mon[44699]: osdmap e151: 12 total, 12 up, 12 in 2024-05-15T23:08:26.143 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:08:26.149 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:08:26.170 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:26.193 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:08:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:26.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:08:26.297 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:08:26.325 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:26.347 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:08:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:26.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:08:26.451 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:08:26.478 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:26.497 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:08:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:26.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:08:26.601 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:26.627 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:26.648 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:26.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:26.755 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:08:26.787 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:26 smithi090 ceph-mon[44434]: pgmap v464: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 971 op/s 2024-05-15T23:08:26.798 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:26.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:26 smithi088 ceph-mon[44699]: pgmap v464: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 971 op/s 2024-05-15T23:08:26.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:26 smithi084 ceph-mon[39784]: pgmap v464: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 971 op/s 2024-05-15T23:08:27.014 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:08:27.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:27.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:08:27.118 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:27.134 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:08:27.134 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:08:27.134 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:08:27.134 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:08:27.135 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:08:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:27.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:27.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:27.270 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:08:27.297 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:27.318 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:08:27.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:27.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:08:27.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:08:27.455 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:28.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T23:08:28.013 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:08:28.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:28.020 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:08:28.121 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:28.147 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:28.169 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:28.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:28.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:28.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:08:28.288 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:28.320 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:28.361 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:08:28.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:28.367 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:08:28.430 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:08:28.468 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:08:28.495 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:08:28.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:28.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:08:28.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:08:28.647 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:28.660 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:08:28.686 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:28 smithi084 ceph-mon[39784]: pgmap v465: 17 pgs: 17 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-05-15T23:08:28.687 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:28 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:08:28.687 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:28 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:08:28.687 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/865339499' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:08:28.687 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:28 smithi084 ceph-mon[39784]: from='client.24940 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:08:28.712 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:08:28.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:28.715 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:08:28.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:08:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:28 smithi088 ceph-mon[44699]: pgmap v465: 17 pgs: 17 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-05-15T23:08:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:28 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:08:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:28 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:08:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/865339499' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:08:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:28 smithi088 ceph-mon[44699]: from='client.24940 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:08:28.876 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:28.957 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:08:28.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:28.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:08:28.960 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:28 smithi090 ceph-mon[44434]: pgmap v465: 17 pgs: 17 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-05-15T23:08:28.961 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:28 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:08:28.961 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:28 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:08:28.961 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/865339499' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:08:28.961 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:28 smithi090 ceph-mon[44434]: from='client.24940 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:08:29.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:08:29.116 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:29.323 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:08:29.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:29.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:08:29.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:08:29.477 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-05-15T23:08:29.478 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:29.513 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:08:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:29.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:08:29.630 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:08:29.656 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:29 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:08:29.657 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:29 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:08:29.657 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:29 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:08:29.657 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:29 smithi090 ceph-mon[44434]: from='client.24940 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:08:29.657 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:29 smithi090 ceph-mon[44434]: osdmap e152: 12 total, 12 up, 12 in 2024-05-15T23:08:29.657 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:29 smithi090 ceph-mon[44434]: pgmap v467: 17 pgs: 17 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 877 op/s 2024-05-15T23:08:29.682 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:29.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:29 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:08:29.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:29 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:08:29.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:29 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:08:29.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:29 smithi084 ceph-mon[39784]: from='client.24940 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:08:29.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:29 smithi084 ceph-mon[39784]: osdmap e152: 12 total, 12 up, 12 in 2024-05-15T23:08:29.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:29 smithi084 ceph-mon[39784]: pgmap v467: 17 pgs: 17 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 877 op/s 2024-05-15T23:08:29.951 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:08:29.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:29.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:08:30.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:08:30.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:29 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:08:30.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:29 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:08:30.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:29 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:08:30.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:29 smithi088 ceph-mon[44699]: from='client.24940 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:08:30.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:29 smithi088 ceph-mon[44699]: osdmap e152: 12 total, 12 up, 12 in 2024-05-15T23:08:30.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:29 smithi088 ceph-mon[44699]: pgmap v467: 17 pgs: 17 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 877 op/s 2024-05-15T23:08:30.094 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:30.557 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:08:30.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:30.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:08:30.660 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:08:30.664 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:08:30.664 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:08:30.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:08:30.664 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:08:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109617 2.5 2:a3ae1bf3:::benchmark_data_smithi090_78160_object109616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:08:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.320545+0000", 2024-05-15T23:08:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.340111519000004, 2024-05-15T23:08:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14569204499999999, 2024-05-15T23:08:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109617 2024-05-15T23:08:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.320545+0000", 2024-05-15T23:08:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.320545+0000", 2024-05-15T23:08:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.320545+0000", 2024-05-15T23:08:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3099999999999996e-07 2024-05-15T23:08:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.320574+0000", 2024-05-15T23:08:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8470999999999999e-05 2024-05-15T23:08:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.320581+0000", 2024-05-15T23:08:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6200000000000001e-06 2024-05-15T23:08:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.320586+0000", 2024-05-15T23:08:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3240000000000002e-06 2024-05-15T23:08:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324708+0000", 2024-05-15T23:08:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004121988 2024-05-15T23:08:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324832+0000", 2024-05-15T23:08:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012456899999999999 2024-05-15T23:08:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:08:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324970+0000", 2024-05-15T23:08:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013749000000000001 2024-05-15T23:08:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329072+0000", 2024-05-15T23:08:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041016309999999997 2024-05-15T23:08:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466125+0000", 2024-05-15T23:08:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13705376399999999 2024-05-15T23:08:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466188+0000", 2024-05-15T23:08:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2574000000000001e-05 2024-05-15T23:08:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466237+0000", 2024-05-15T23:08:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8782999999999997e-05 2024-05-15T23:08:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109625 2.b 2:d2ace5aa:::benchmark_data_smithi090_78160_object109624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:08:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.324751+0000", 2024-05-15T23:08:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.335904812999999, 2024-05-15T23:08:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.141344727, 2024-05-15T23:08:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109625 2024-05-15T23:08:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324751+0000", 2024-05-15T23:08:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324751+0000", 2024-05-15T23:08:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324753+0000", 2024-05-15T23:08:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7460000000000001e-06 2024-05-15T23:08:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324773+0000", 2024-05-15T23:08:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9851e-05 2024-05-15T23:08:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324780+0000", 2024-05-15T23:08:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.165e-06 2024-05-15T23:08:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324786+0000", 2024-05-15T23:08:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3900000000000001e-06 2024-05-15T23:08:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324834+0000", 2024-05-15T23:08:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8701e-05 2024-05-15T23:08:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324916+0000", 2024-05-15T23:08:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1674000000000004e-05 2024-05-15T23:08:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 11", 2024-05-15T23:08:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324995+0000", 2024-05-15T23:08:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8987999999999997e-05 2024-05-15T23:08:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329078+0000", 2024-05-15T23:08:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040834210000000003 2024-05-15T23:08:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465993+0000", 2024-05-15T23:08:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13691505700000001 2024-05-15T23:08:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466052+0000", 2024-05-15T23:08:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8922000000000003e-05 2024-05-15T23:08:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466096+0000", 2024-05-15T23:08:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3812e-05 2024-05-15T23:08:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109626 2.f 2:f5db7710:::benchmark_data_smithi090_78160_object109625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:08:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.325464+0000", 2024-05-15T23:08:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.335191723999998, 2024-05-15T23:08:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14049392499999999, 2024-05-15T23:08:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.677 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.677 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.677 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.677 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.677 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.677 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109626 2024-05-15T23:08:30.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.677 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325464+0000", 2024-05-15T23:08:30.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325464+0000", 2024-05-15T23:08:30.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325466+0000", 2024-05-15T23:08:30.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3289999999999999e-06 2024-05-15T23:08:30.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325488+0000", 2024-05-15T23:08:30.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.196e-05 2024-05-15T23:08:30.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325497+0000", 2024-05-15T23:08:30.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6949999999999995e-06 2024-05-15T23:08:30.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325504+0000", 2024-05-15T23:08:30.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0199999999999997e-06 2024-05-15T23:08:30.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325556+0000", 2024-05-15T23:08:30.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1062e-05 2024-05-15T23:08:30.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325662+0000", 2024-05-15T23:08:30.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106178 2024-05-15T23:08:30.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 9", 2024-05-15T23:08:30.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325779+0000", 2024-05-15T23:08:30.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117321 2024-05-15T23:08:30.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329086+0000", 2024-05-15T23:08:30.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033067629999999999 2024-05-15T23:08:30.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329613+0000", 2024-05-15T23:08:30.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052711200000000004 2024-05-15T23:08:30.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465911+0000", 2024-05-15T23:08:30.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.136297687 2024-05-15T23:08:30.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465958+0000", 2024-05-15T23:08:30.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7797999999999999e-05 2024-05-15T23:08:30.682 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.682 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.682 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.682 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.24745.0:109626 2.f e127/102 ondisk, result = 0)", 2024-05-15T23:08:30.682 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.329536+0000", 2024-05-15T23:08:30.682 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.331120620999997, 2024-05-15T23:08:30.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13643081000000001, 2024-05-15T23:08:30.682 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.682 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.682 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:08:30.683 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329536+0000", 2024-05-15T23:08:30.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329536+0000", 2024-05-15T23:08:30.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329537+0000", 2024-05-15T23:08:30.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.59e-06 2024-05-15T23:08:30.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329543+0000", 2024-05-15T23:08:30.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.891e-06 2024-05-15T23:08:30.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329553+0000", 2024-05-15T23:08:30.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0336999999999999e-05 2024-05-15T23:08:30.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329560+0000", 2024-05-15T23:08:30.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0690000000000004e-06 2024-05-15T23:08:30.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329602+0000", 2024-05-15T23:08:30.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1211999999999998e-05 2024-05-15T23:08:30.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329609+0000", 2024-05-15T23:08:30.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6809999999999998e-06 2024-05-15T23:08:30.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465966+0000", 2024-05-15T23:08:30.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13635702999999999 2024-05-15T23:08:30.686 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.686 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.686 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.686 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.24745.0:109617 2.5 e127/102 ondisk, result = 0)", 2024-05-15T23:08:30.686 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.329717+0000", 2024-05-15T23:08:30.686 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.330938704000005, 2024-05-15T23:08:30.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.136528596, 2024-05-15T23:08:30.686 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.686 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.686 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:08:30.687 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329717+0000", 2024-05-15T23:08:30.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329717+0000", 2024-05-15T23:08:30.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329719+0000", 2024-05-15T23:08:30.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.057e-06 2024-05-15T23:08:30.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329722+0000", 2024-05-15T23:08:30.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8290000000000001e-06 2024-05-15T23:08:30.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465828+0000", 2024-05-15T23:08:30.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13610577099999999 2024-05-15T23:08:30.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465840+0000", 2024-05-15T23:08:30.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2140999999999999e-05 2024-05-15T23:08:30.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466110+0000", 2024-05-15T23:08:30.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027013500000000001 2024-05-15T23:08:30.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466119+0000", 2024-05-15T23:08:30.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4740000000000005e-06 2024-05-15T23:08:30.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466246+0000", 2024-05-15T23:08:30.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012718899999999999 2024-05-15T23:08:30.690 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.690 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.690 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.690 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109634 2.5 2:aefd519a:::benchmark_data_smithi090_78160_object109633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:08:30.690 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.331893+0000", 2024-05-15T23:08:30.690 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.328762924000003, 2024-05-15T23:08:30.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14172101100000001, 2024-05-15T23:08:30.691 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.691 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.691 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.691 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.691 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.691 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.691 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109634 2024-05-15T23:08:30.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.691 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.331893+0000", 2024-05-15T23:08:30.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.331893+0000", 2024-05-15T23:08:30.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.331895+0000", 2024-05-15T23:08:30.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0169999999999999e-06 2024-05-15T23:08:30.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.331918+0000", 2024-05-15T23:08:30.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3051999999999999e-05 2024-05-15T23:08:30.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465774+0000", 2024-05-15T23:08:30.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.133855329 2024-05-15T23:08:30.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465792+0000", 2024-05-15T23:08:30.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8848999999999999e-05 2024-05-15T23:08:30.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.467040+0000", 2024-05-15T23:08:30.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001247822 2024-05-15T23:08:30.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.467148+0000", 2024-05-15T23:08:30.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010783199999999999 2024-05-15T23:08:30.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:08:30.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.467278+0000", 2024-05-15T23:08:30.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013016500000000001 2024-05-15T23:08:30.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.471575+0000", 2024-05-15T23:08:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004296345 2024-05-15T23:08:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.473509+0000", 2024-05-15T23:08:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019343940000000001 2024-05-15T23:08:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.473566+0000", 2024-05-15T23:08:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7225000000000002e-05 2024-05-15T23:08:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.473614+0000", 2024-05-15T23:08:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7981e-05 2024-05-15T23:08:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:109635 2.6 e127/102 2:69c20386:::benchmark_data_smithi090_78160_object109634:head v 127'6850, mlcod=127'6848)", 2024-05-15T23:08:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.333210+0000", 2024-05-15T23:08:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.327445960000006, 2024-05-15T23:08:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13963065999999999, 2024-05-15T23:08:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.333210+0000", 2024-05-15T23:08:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.333210+0000", 2024-05-15T23:08:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.333212+0000", 2024-05-15T23:08:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5770000000000001e-06 2024-05-15T23:08:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.333249+0000", 2024-05-15T23:08:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7134000000000001e-05 2024-05-15T23:08:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466146+0000", 2024-05-15T23:08:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.132896659 2024-05-15T23:08:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466153+0000", 2024-05-15T23:08:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8399999999999995e-06 2024-05-15T23:08:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466367+0000", 2024-05-15T23:08:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021343600000000001 2024-05-15T23:08:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466387+0000", 2024-05-15T23:08:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9922e-05 2024-05-15T23:08:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.472790+0000", 2024-05-15T23:08:30.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064031050000000001 2024-05-15T23:08:30.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.472841+0000", 2024-05-15T23:08:30.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0986999999999998e-05 2024-05-15T23:08:30.700 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.700 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.700 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.700 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:111051 2.b 2:dae0a3ed:::benchmark_data_smithi090_78160_object111050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:08:30.701 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:54.386337+0000", 2024-05-15T23:08:30.701 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 96.274319582000004, 2024-05-15T23:08:30.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063655595999999995, 2024-05-15T23:08:30.701 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.701 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.701 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.701 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.701 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.701 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.701 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 111051 2024-05-15T23:08:30.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.701 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386337+0000", 2024-05-15T23:08:30.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386337+0000", 2024-05-15T23:08:30.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386339+0000", 2024-05-15T23:08:30.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.74e-06 2024-05-15T23:08:30.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386365+0000", 2024-05-15T23:08:30.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5542999999999999e-05 2024-05-15T23:08:30.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386381+0000", 2024-05-15T23:08:30.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6158999999999999e-05 2024-05-15T23:08:30.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386392+0000", 2024-05-15T23:08:30.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0584999999999999e-05 2024-05-15T23:08:30.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386453+0000", 2024-05-15T23:08:30.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1821000000000003e-05 2024-05-15T23:08:30.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386576+0000", 2024-05-15T23:08:30.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122977 2024-05-15T23:08:30.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 11", 2024-05-15T23:08:30.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.399570+0000", 2024-05-15T23:08:30.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012993963000000001 2024-05-15T23:08:30.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.402445+0000", 2024-05-15T23:08:30.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028750550000000001 2024-05-15T23:08:30.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.449874+0000", 2024-05-15T23:08:30.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.047428234 2024-05-15T23:08:30.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.449941+0000", 2024-05-15T23:08:30.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7588000000000004e-05 2024-05-15T23:08:30.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.449992+0000", 2024-05-15T23:08:30.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0930999999999997e-05 2024-05-15T23:08:30.706 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.706 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.706 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.706 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:116822 2.5 2:a4a74574:::benchmark_data_smithi090_78160_object116821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:30.706 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:58.388393+0000", 2024-05-15T23:08:30.706 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 92.272263312000007, 2024-05-15T23:08:30.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049270092000000001, 2024-05-15T23:08:30.707 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.707 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.707 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.707 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.707 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.707 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.707 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 116822 2024-05-15T23:08:30.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.707 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388393+0000", 2024-05-15T23:08:30.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388393+0000", 2024-05-15T23:08:30.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388394+0000", 2024-05-15T23:08:30.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.607e-06 2024-05-15T23:08:30.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388415+0000", 2024-05-15T23:08:30.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0684000000000001e-05 2024-05-15T23:08:30.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388424+0000", 2024-05-15T23:08:30.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2480000000000002e-06 2024-05-15T23:08:30.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388433+0000", 2024-05-15T23:08:30.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-05-15T23:08:30.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388477+0000", 2024-05-15T23:08:30.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3893000000000001e-05 2024-05-15T23:08:30.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388580+0000", 2024-05-15T23:08:30.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102902 2024-05-15T23:08:30.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:08:30.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388686+0000", 2024-05-15T23:08:30.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106543 2024-05-15T23:08:30.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.391560+0000", 2024-05-15T23:08:30.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028739429999999999 2024-05-15T23:08:30.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.437575+0000", 2024-05-15T23:08:30.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046014688999999998 2024-05-15T23:08:30.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.437635+0000", 2024-05-15T23:08:30.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9623000000000002e-05 2024-05-15T23:08:30.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.437663+0000", 2024-05-15T23:08:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8350000000000001e-05 2024-05-15T23:08:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:116838 2.5 2:a8d1e181:::benchmark_data_smithi090_78160_object116837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:08:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:58.403340+0000", 2024-05-15T23:08:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 92.257316305000003, 2024-05-15T23:08:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046982428, 2024-05-15T23:08:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.712 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.713 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.713 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 116838 2024-05-15T23:08:30.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.713 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403340+0000", 2024-05-15T23:08:30.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403340+0000", 2024-05-15T23:08:30.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403342+0000", 2024-05-15T23:08:30.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9709999999999998e-06 2024-05-15T23:08:30.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403366+0000", 2024-05-15T23:08:30.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4416000000000001e-05 2024-05-15T23:08:30.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403382+0000", 2024-05-15T23:08:30.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5381999999999999e-05 2024-05-15T23:08:30.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403390+0000", 2024-05-15T23:08:30.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5979999999999997e-06 2024-05-15T23:08:30.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403499+0000", 2024-05-15T23:08:30.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108922 2024-05-15T23:08:30.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403604+0000", 2024-05-15T23:08:30.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000105085 2024-05-15T23:08:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:08:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403738+0000", 2024-05-15T23:08:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013375400000000001 2024-05-15T23:08:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.406702+0000", 2024-05-15T23:08:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029637930000000002 2024-05-15T23:08:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.450211+0000", 2024-05-15T23:08:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.043509054999999998 2024-05-15T23:08:30.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.450268+0000", 2024-05-15T23:08:30.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6793999999999999e-05 2024-05-15T23:08:30.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.450322+0000", 2024-05-15T23:08:30.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4657999999999997e-05 2024-05-15T23:08:30.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.717 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.718 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:190851 2.b 2:d6bfb2a4:::benchmark_data_smithi090_78160_object29493:head [delete] snapc 0=[] ondisk+write+known_if_redirected e148)", 2024-05-15T23:08:30.718 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:00.419416+0000", 2024-05-15T23:08:30.718 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.241240216000001, 2024-05-15T23:08:30.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045686048, 2024-05-15T23:08:30.718 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.718 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.718 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.718 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.718 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.718 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.718 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 190851 2024-05-15T23:08:30.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.718 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.419416+0000", 2024-05-15T23:08:30.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.419416+0000", 2024-05-15T23:08:30.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.419416+0000", 2024-05-15T23:08:30.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2600000000000002e-07 2024-05-15T23:08:30.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.419419+0000", 2024-05-15T23:08:30.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.565e-06 2024-05-15T23:08:30.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.419424+0000", 2024-05-15T23:08:30.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1399999999999999e-06 2024-05-15T23:08:30.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.419429+0000", 2024-05-15T23:08:30.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3149999999999999e-06 2024-05-15T23:08:30.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.433265+0000", 2024-05-15T23:08:30.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013835953999999999 2024-05-15T23:08:30.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.433400+0000", 2024-05-15T23:08:30.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013417999999999999 2024-05-15T23:08:30.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 11", 2024-05-15T23:08:30.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.433520+0000", 2024-05-15T23:08:30.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120078 2024-05-15T23:08:30.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.452068+0000", 2024-05-15T23:08:30.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018548179000000001 2024-05-15T23:08:30.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.465004+0000", 2024-05-15T23:08:30.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012936509000000001 2024-05-15T23:08:30.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.465057+0000", 2024-05-15T23:08:30.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2321999999999998e-05 2024-05-15T23:08:30.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.465102+0000", 2024-05-15T23:08:30.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.528e-05 2024-05-15T23:08:30.723 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.723 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.723 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.723 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:204121 2.5 2:ab8c208a:::benchmark_data_smithi090_78160_object42763:head [delete] snapc 0=[] ondisk+write+known_if_redirected e149)", 2024-05-15T23:08:30.724 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:11.826244+0000", 2024-05-15T23:08:30.724 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.834412362999998, 2024-05-15T23:08:30.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042297257999999997, 2024-05-15T23:08:30.724 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.724 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.724 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.724 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.724 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.724 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.724 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 204121 2024-05-15T23:08:30.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.724 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.826244+0000", 2024-05-15T23:08:30.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.826244+0000", 2024-05-15T23:08:30.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.826245+0000", 2024-05-15T23:08:30.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.266e-06 2024-05-15T23:08:30.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.826250+0000", 2024-05-15T23:08:30.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.775e-06 2024-05-15T23:08:30.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.826258+0000", 2024-05-15T23:08:30.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4460000000000001e-06 2024-05-15T23:08:30.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.826264+0000", 2024-05-15T23:08:30.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.502e-06 2024-05-15T23:08:30.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.832691+0000", 2024-05-15T23:08:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064274019999999996 2024-05-15T23:08:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.832800+0000", 2024-05-15T23:08:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010875299999999999 2024-05-15T23:08:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:08:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.832866+0000", 2024-05-15T23:08:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6276e-05 2024-05-15T23:08:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.839967+0000", 2024-05-15T23:08:30.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071012150000000001 2024-05-15T23:08:30.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.868431+0000", 2024-05-15T23:08:30.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028463321999999999 2024-05-15T23:08:30.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.868490+0000", 2024-05-15T23:08:30.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9385999999999997e-05 2024-05-15T23:08:30.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.868541+0000", 2024-05-15T23:08:30.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0915000000000003e-05 2024-05-15T23:08:30.729 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.729 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.729 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.729 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:204123 2.5 2:ac7bf2f8:::benchmark_data_smithi090_78160_object42765:head [delete] snapc 0=[] ondisk+write+known_if_redirected e149)", 2024-05-15T23:08:30.729 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:11.826313+0000", 2024-05-15T23:08:30.729 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.834343060999998, 2024-05-15T23:08:30.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042350974, 2024-05-15T23:08:30.729 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.730 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.730 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.730 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.730 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.730 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.730 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 204123 2024-05-15T23:08:30.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.730 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.826313+0000", 2024-05-15T23:08:30.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.826313+0000", 2024-05-15T23:08:30.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.826314+0000", 2024-05-15T23:08:30.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1689999999999999e-06 2024-05-15T23:08:30.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.826318+0000", 2024-05-15T23:08:30.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1980000000000003e-06 2024-05-15T23:08:30.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.826327+0000", 2024-05-15T23:08:30.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9809999999999992e-06 2024-05-15T23:08:30.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.826334+0000", 2024-05-15T23:08:30.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7870000000000004e-06 2024-05-15T23:08:30.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.836459+0000", 2024-05-15T23:08:30.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010125093 2024-05-15T23:08:30.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.836581+0000", 2024-05-15T23:08:30.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012165800000000001 2024-05-15T23:08:30.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:08:30.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.836697+0000", 2024-05-15T23:08:30.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115796 2024-05-15T23:08:30.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.844814+0000", 2024-05-15T23:08:30.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008116992 2024-05-15T23:08:30.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.868580+0000", 2024-05-15T23:08:30.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023766551 2024-05-15T23:08:30.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.868626+0000", 2024-05-15T23:08:30.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5247000000000002e-05 2024-05-15T23:08:30.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:11.868664+0000", 2024-05-15T23:08:30.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8501999999999998e-05 2024-05-15T23:08:30.735 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.735 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.735 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.735 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:218350 2.b 2:d272971d:::benchmark_data_smithi090_78160_object56992:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:08:30.735 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:24.318097+0000", 2024-05-15T23:08:30.735 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.3425590280000002, 2024-05-15T23:08:30.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055657801999999999, 2024-05-15T23:08:30.735 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.735 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.735 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.735 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.735 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.736 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.736 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218350 2024-05-15T23:08:30.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.736 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.318097+0000", 2024-05-15T23:08:30.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.318097+0000", 2024-05-15T23:08:30.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.318099+0000", 2024-05-15T23:08:30.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8789999999999999e-06 2024-05-15T23:08:30.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.318104+0000", 2024-05-15T23:08:30.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3680000000000001e-06 2024-05-15T23:08:30.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.318114+0000", 2024-05-15T23:08:30.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3819999999999993e-06 2024-05-15T23:08:30.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.318120+0000", 2024-05-15T23:08:30.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7100000000000001e-06 2024-05-15T23:08:30.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.319370+0000", 2024-05-15T23:08:30.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012499570000000001 2024-05-15T23:08:30.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.319479+0000", 2024-05-15T23:08:30.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108941 2024-05-15T23:08:30.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 11", 2024-05-15T23:08:30.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.319587+0000", 2024-05-15T23:08:30.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010780700000000001 2024-05-15T23:08:30.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.324031+0000", 2024-05-15T23:08:30.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044442750000000001 2024-05-15T23:08:30.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.337615+0000", 2024-05-15T23:08:30.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01358352 2024-05-15T23:08:30.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.337665+0000", 2024-05-15T23:08:30.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0319999999999999e-05 2024-05-15T23:08:30.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.373755+0000", 2024-05-15T23:08:30.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036089642999999998 2024-05-15T23:08:30.741 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.741 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.741 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.741 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:218364 2.5 2:ac4383fa:::benchmark_data_smithi090_78160_object57006:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:08:30.741 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:24.325312+0000", 2024-05-15T23:08:30.741 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.335344418, 2024-05-15T23:08:30.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.048743983999999997, 2024-05-15T23:08:30.741 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.741 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.741 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.741 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.741 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.742 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.742 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218364 2024-05-15T23:08:30.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.742 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325312+0000", 2024-05-15T23:08:30.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325312+0000", 2024-05-15T23:08:30.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325313+0000", 2024-05-15T23:08:30.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.536e-06 2024-05-15T23:08:30.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325319+0000", 2024-05-15T23:08:30.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4319999999999998e-06 2024-05-15T23:08:30.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325329+0000", 2024-05-15T23:08:30.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0564000000000001e-05 2024-05-15T23:08:30.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325337+0000", 2024-05-15T23:08:30.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5480000000000004e-06 2024-05-15T23:08:30.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325392+0000", 2024-05-15T23:08:30.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5476000000000002e-05 2024-05-15T23:08:30.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325493+0000", 2024-05-15T23:08:30.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100572 2024-05-15T23:08:30.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:08:30.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325591+0000", 2024-05-15T23:08:30.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8259999999999998e-05 2024-05-15T23:08:30.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.333725+0000", 2024-05-15T23:08:30.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0081340900000000001 2024-05-15T23:08:30.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.373961+0000", 2024-05-15T23:08:30.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040235663999999997 2024-05-15T23:08:30.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.374013+0000", 2024-05-15T23:08:30.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1705e-05 2024-05-15T23:08:30.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.374056+0000", 2024-05-15T23:08:30.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3136999999999997e-05 2024-05-15T23:08:30.746 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.746 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.747 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.747 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:218365 2.5 2:a283dc57:::benchmark_data_smithi090_78160_object57007:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:08:30.747 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:24.327336+0000", 2024-05-15T23:08:30.747 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.3333199950000001, 2024-05-15T23:08:30.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.051207694999999998, 2024-05-15T23:08:30.747 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.747 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.747 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.747 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.747 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.747 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.747 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218365 2024-05-15T23:08:30.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.748 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.327336+0000", 2024-05-15T23:08:30.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.327336+0000", 2024-05-15T23:08:30.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.327338+0000", 2024-05-15T23:08:30.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.86e-06 2024-05-15T23:08:30.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.327344+0000", 2024-05-15T23:08:30.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2169999999999999e-06 2024-05-15T23:08:30.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.327356+0000", 2024-05-15T23:08:30.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1292e-05 2024-05-15T23:08:30.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.327363+0000", 2024-05-15T23:08:30.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7200000000000006e-06 2024-05-15T23:08:30.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330074+0000", 2024-05-15T23:08:30.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002710559 2024-05-15T23:08:30.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330177+0000", 2024-05-15T23:08:30.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103081 2024-05-15T23:08:30.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:08:30.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330293+0000", 2024-05-15T23:08:30.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011625000000000001 2024-05-15T23:08:30.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.374099+0000", 2024-05-15T23:08:30.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.043805372000000002 2024-05-15T23:08:30.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.378434+0000", 2024-05-15T23:08:30.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043352629999999998 2024-05-15T23:08:30.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.378489+0000", 2024-05-15T23:08:30.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5680000000000002e-05 2024-05-15T23:08:30.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.378544+0000", 2024-05-15T23:08:30.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4401000000000003e-05 2024-05-15T23:08:30.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.752 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.753 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:218366 2.b 2:d2ab96ff:::benchmark_data_smithi090_78160_object57008:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:08:30.753 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:24.330167+0000", 2024-05-15T23:08:30.753 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.3304896580000003, 2024-05-15T23:08:30.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.043737920999999999, 2024-05-15T23:08:30.753 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.753 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.753 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.753 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218366 2024-05-15T23:08:30.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.753 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330167+0000", 2024-05-15T23:08:30.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330167+0000", 2024-05-15T23:08:30.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330168+0000", 2024-05-15T23:08:30.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6750000000000001e-06 2024-05-15T23:08:30.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330173+0000", 2024-05-15T23:08:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.101e-06 2024-05-15T23:08:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330182+0000", 2024-05-15T23:08:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-05-15T23:08:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330189+0000", 2024-05-15T23:08:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4660000000000002e-06 2024-05-15T23:08:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330243+0000", 2024-05-15T23:08:30.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.384e-05 2024-05-15T23:08:30.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330343+0000", 2024-05-15T23:08:30.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9658e-05 2024-05-15T23:08:30.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 11", 2024-05-15T23:08:30.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330419+0000", 2024-05-15T23:08:30.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6346000000000004e-05 2024-05-15T23:08:30.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.334087+0000", 2024-05-15T23:08:30.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003668283 2024-05-15T23:08:30.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.373806+0000", 2024-05-15T23:08:30.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039718282000000001 2024-05-15T23:08:30.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.373860+0000", 2024-05-15T23:08:30.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4425000000000002e-05 2024-05-15T23:08:30.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.373904+0000", 2024-05-15T23:08:30.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4234999999999999e-05 2024-05-15T23:08:30.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.758 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.758 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:218368 2.5 2:a696423a:::benchmark_data_smithi090_78160_object57010:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:08:30.758 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:24.330243+0000", 2024-05-15T23:08:30.758 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.3304128860000004, 2024-05-15T23:08:30.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.048395308999999997, 2024-05-15T23:08:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218368 2024-05-15T23:08:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330243+0000", 2024-05-15T23:08:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330243+0000", 2024-05-15T23:08:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330244+0000", 2024-05-15T23:08:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6700000000000002e-07 2024-05-15T23:08:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330248+0000", 2024-05-15T23:08:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0160000000000002e-06 2024-05-15T23:08:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330255+0000", 2024-05-15T23:08:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.866e-06 2024-05-15T23:08:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330261+0000", 2024-05-15T23:08:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9170000000000003e-06 2024-05-15T23:08:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.333760+0000", 2024-05-15T23:08:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034988649999999999 2024-05-15T23:08:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.333879+0000", 2024-05-15T23:08:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118651 2024-05-15T23:08:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:08:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.333991+0000", 2024-05-15T23:08:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011229499999999999 2024-05-15T23:08:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.374136+0000", 2024-05-15T23:08:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040145104000000001 2024-05-15T23:08:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.378560+0000", 2024-05-15T23:08:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044240140000000004 2024-05-15T23:08:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.378602+0000", 2024-05-15T23:08:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2450000000000002e-05 2024-05-15T23:08:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.378639+0000", 2024-05-15T23:08:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6164000000000001e-05 2024-05-15T23:08:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:218378 2.b 2:de94d2bb:::benchmark_data_smithi090_78160_object57020:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:08:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:24.339333+0000", 2024-05-15T23:08:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.3213231409999997, 2024-05-15T23:08:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045024103000000003, 2024-05-15T23:08:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218378 2024-05-15T23:08:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.339333+0000", 2024-05-15T23:08:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.339333+0000", 2024-05-15T23:08:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.339335+0000", 2024-05-15T23:08:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.86e-06 2024-05-15T23:08:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.339340+0000", 2024-05-15T23:08:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3380000000000004e-06 2024-05-15T23:08:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.373846+0000", 2024-05-15T23:08:30.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034505894000000002 2024-05-15T23:08:30.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.373858+0000", 2024-05-15T23:08:30.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1941e-05 2024-05-15T23:08:30.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.373932+0000", 2024-05-15T23:08:30.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3648999999999999e-05 2024-05-15T23:08:30.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.374065+0000", 2024-05-15T23:08:30.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133029 2024-05-15T23:08:30.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 11", 2024-05-15T23:08:30.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.374205+0000", 2024-05-15T23:08:30.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000140476 2024-05-15T23:08:30.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.379115+0000", 2024-05-15T23:08:30.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049099929999999996 2024-05-15T23:08:30.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.384262+0000", 2024-05-15T23:08:30.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051469000000000003 2024-05-15T23:08:30.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.384319+0000", 2024-05-15T23:08:30.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6468000000000002e-05 2024-05-15T23:08:30.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.384357+0000", 2024-05-15T23:08:30.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8554999999999999e-05 2024-05-15T23:08:30.770 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.770 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.770 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.770 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:223731 2.f 2:f294b35e:::benchmark_data_smithi090_78160_object62373:head [delete] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-05-15T23:08:30.770 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:29.198071+0000", 2024-05-15T23:08:30.770 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.4625847320000001, 2024-05-15T23:08:30.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.04143993, 2024-05-15T23:08:30.770 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:08:30.770 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:08:30.770 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:08:30.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:08:30.771 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:08:30.771 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:08:30.771 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 223731 2024-05-15T23:08:30.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.771 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:08:30.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:08:30.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:29.198071+0000", 2024-05-15T23:08:30.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:08:30.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:29.198071+0000", 2024-05-15T23:08:30.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:08:30.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:08:30.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:29.198072+0000", 2024-05-15T23:08:30.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4799999999999997e-07 2024-05-15T23:08:30.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:08:30.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:29.198076+0000", 2024-05-15T23:08:30.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4529999999999998e-06 2024-05-15T23:08:30.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:08:30.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:29.198084+0000", 2024-05-15T23:08:30.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1710000000000002e-06 2024-05-15T23:08:30.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:08:30.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:29.198090+0000", 2024-05-15T23:08:30.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0440000000000001e-06 2024-05-15T23:08:30.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:08:30.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:29.203417+0000", 2024-05-15T23:08:30.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005327203 2024-05-15T23:08:30.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:08:30.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:29.203554+0000", 2024-05-15T23:08:30.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013667700000000001 2024-05-15T23:08:30.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 9", 2024-05-15T23:08:30.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:29.203685+0000", 2024-05-15T23:08:30.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013110500000000001 2024-05-15T23:08:30.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:08:30.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:29.217352+0000", 2024-05-15T23:08:30.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013667413999999999 2024-05-15T23:08:30.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:08:30.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:29.235989+0000", 2024-05-15T23:08:30.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018636479000000001 2024-05-15T23:08:30.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:08:30.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:29.239462+0000", 2024-05-15T23:08:30.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034733870000000001 2024-05-15T23:08:30.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:08:30.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:08:30.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:08:30.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:29.239511+0000", 2024-05-15T23:08:30.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9048999999999997e-05 2024-05-15T23:08:30.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.775 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.776 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:08:30.776 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:08:30.776 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:08:30.825 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:30.892 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:08:30.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:30.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:08:30.999 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:08:31.032 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:31.395 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:08:31.395 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:31.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:31.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:08:31.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:31.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T23:08:31.561 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:31.674 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:31.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:31.745 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:31 smithi090 ceph-mon[44434]: pgmap v468: 17 pgs: 17 active+clean; 6.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:08:31.777 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:31.805 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:31.825 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:31.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:31.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:31.918 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:08:31.929 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:31.957 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:31.968 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:31 smithi084 ceph-mon[39784]: pgmap v468: 17 pgs: 17 active+clean; 6.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:08:31.978 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:31.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:31.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:31.991 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:08:32.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:31 smithi088 ceph-mon[44699]: pgmap v468: 17 pgs: 17 active+clean; 6.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:08:32.085 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:32.154 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:32.242 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:32.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:32.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:32.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:08:32.417 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:33.095 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:08:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:33.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:08:33.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:08:33.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:08:33.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:08:33.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:08:33.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:08:33.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:08:33.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:08:33.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:08:33.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:08:33.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:08:33.211 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:08:33.250 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:08:33.260 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:33.295 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:08:33.306 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:08:33.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:08:33.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:08:33.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:08:33.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:08:33.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:08:33.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:08:33.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:08:33.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:08:33.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:08:33.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:08:33.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:08:33.903 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:34.005 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:33 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:08:34.005 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:33 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:08:34.005 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:33 smithi084 ceph-mon[39784]: pgmap v469: 17 pgs: 17 active+clean; 6.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:08:34.005 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:33 smithi084 ceph-mon[39784]: osdmap e153: 12 total, 12 up, 12 in 2024-05-15T23:08:34.006 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:33 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3500192718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:08:34.006 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.006 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.006 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.006 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.006 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.006 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.006 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.006 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.023 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:08:34.023 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:08:34.050 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:33 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:08:34.050 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:33 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:08:34.050 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:33 smithi090 ceph-mon[44434]: pgmap v469: 17 pgs: 17 active+clean; 6.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:08:34.050 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:33 smithi090 ceph-mon[44434]: osdmap e153: 12 total, 12 up, 12 in 2024-05-15T23:08:34.050 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:33 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3500192718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:08:34.050 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.050 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.050 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.050 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.050 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.050 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.050 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:33 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:08:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:33 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:08:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:33 smithi088 ceph-mon[44699]: pgmap v469: 17 pgs: 17 active+clean; 6.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:08:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:33 smithi088 ceph-mon[44699]: osdmap e153: 12 total, 12 up, 12 in 2024-05-15T23:08:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:33 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3500192718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:08:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:34.833 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:08:34.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:34.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:08:34.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:08:34.980 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:35.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:35.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:35.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:35.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:35.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:34 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:08:35.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:34 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:08:35.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:34 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:08:35.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:35.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:35.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:35.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:35.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:34 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:08:35.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:34 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:08:35.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:34 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:08:35.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:35.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:35.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:35.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:35.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:34 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:08:35.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:34 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:08:35.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:34 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:08:35.295 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-05-15T23:08:36.395 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:08:36.486 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:36.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:36 smithi084 ceph-mon[39784]: pgmap v471: 17 pgs: 17 active+clean; 5.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 953 op/s 2024-05-15T23:08:36.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:08:36.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:36 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:08:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:36 smithi088 ceph-mon[44699]: pgmap v471: 17 pgs: 17 active+clean; 5.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 953 op/s 2024-05-15T23:08:36.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:08:36.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:36 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:08:37.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:36 smithi090 ceph-mon[44434]: pgmap v471: 17 pgs: 17 active+clean; 5.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 953 op/s 2024-05-15T23:08:37.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:08:37.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:36 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:08:37.447 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:08:37.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:37.452 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:08:37.477 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T23:08:37.553 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:08:37.645 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:08:37.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:37.703 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:08:37.738 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:08:37.804 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:37.843 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:08:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:37 smithi090 ceph-mon[44434]: pgmap v472: 17 pgs: 17 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s 2024-05-15T23:08:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:37 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3406419428' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:08:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:37 smithi090 ceph-mon[44434]: from='client.34982 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:08:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:37 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:08:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:37 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:08:38.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:37 smithi084 ceph-mon[39784]: pgmap v472: 17 pgs: 17 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s 2024-05-15T23:08:38.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:37 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3406419428' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:08:38.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:37 smithi084 ceph-mon[39784]: from='client.34982 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:08:38.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:37 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:08:38.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:37 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:08:38.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:37 smithi088 ceph-mon[44699]: pgmap v472: 17 pgs: 17 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s 2024-05-15T23:08:38.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:37 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3406419428' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:08:38.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:37 smithi088 ceph-mon[44699]: from='client.34982 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:08:38.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:37 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:08:38.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:37 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:08:38.072 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:38.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:38.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:38.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:08:38.229 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:38.325 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:38.325 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:08:38.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:38.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:08:38.438 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:38.466 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:38.489 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:38.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:38.595 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:08:38.621 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:38.643 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:08:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:38.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:08:38.748 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:08:38.846 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:39.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:08:39.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:38 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:08:39.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:38 smithi090 ceph-mon[44434]: from='client.34982 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:08:39.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:38 smithi090 ceph-mon[44434]: osdmap e154: 12 total, 12 up, 12 in 2024-05-15T23:08:39.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:38 smithi090 ceph-mon[44434]: 2.e scrub starts 2024-05-15T23:08:39.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:38 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:08:39.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:38 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:08:39.058 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:08:39.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:08:39.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:38 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:08:39.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:38 smithi084 ceph-mon[39784]: from='client.34982 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:08:39.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:38 smithi084 ceph-mon[39784]: osdmap e154: 12 total, 12 up, 12 in 2024-05-15T23:08:39.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:38 smithi084 ceph-mon[39784]: 2.e scrub starts 2024-05-15T23:08:39.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:38 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:08:39.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:38 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:08:39.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:39.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:08:39.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:38 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:08:39.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:38 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:08:39.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:38 smithi088 ceph-mon[44699]: from='client.34982 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:08:39.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:38 smithi088 ceph-mon[44699]: osdmap e154: 12 total, 12 up, 12 in 2024-05-15T23:08:39.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:38 smithi088 ceph-mon[44699]: 2.e scrub starts 2024-05-15T23:08:39.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:38 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:08:39.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:38 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:08:39.172 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:39.214 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:40.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:39 smithi090 ceph-mon[44434]: pgmap v474: 17 pgs: 17 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s 2024-05-15T23:08:40.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:39 smithi090 ceph-mon[44434]: osdmap e155: 12 total, 12 up, 12 in 2024-05-15T23:08:40.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:39 smithi090 ceph-mon[44434]: 2.3 scrub starts 2024-05-15T23:08:40.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:39 smithi090 ceph-mon[44434]: 2.e scrub ok 2024-05-15T23:08:40.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:39 smithi088 ceph-mon[44699]: pgmap v474: 17 pgs: 17 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s 2024-05-15T23:08:40.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:39 smithi088 ceph-mon[44699]: osdmap e155: 12 total, 12 up, 12 in 2024-05-15T23:08:40.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:39 smithi088 ceph-mon[44699]: 2.3 scrub starts 2024-05-15T23:08:40.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:39 smithi088 ceph-mon[44699]: 2.e scrub ok 2024-05-15T23:08:40.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:39 smithi084 ceph-mon[39784]: pgmap v474: 17 pgs: 17 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s 2024-05-15T23:08:40.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:39 smithi084 ceph-mon[39784]: osdmap e155: 12 total, 12 up, 12 in 2024-05-15T23:08:40.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:39 smithi084 ceph-mon[39784]: 2.3 scrub starts 2024-05-15T23:08:40.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:39 smithi084 ceph-mon[39784]: 2.e scrub ok 2024-05-15T23:08:40.590 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:40.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:40.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:40.698 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:40.724 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:40.746 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:40.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:40.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:40.785 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:08:40.850 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:08:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: 2.d scrub starts 2024-05-15T23:08:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: 2.0 scrub starts 2024-05-15T23:08:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: 2.3 scrub ok 2024-05-15T23:08:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=54,logm=4} crc {auth=450933107,config=3313370793,health=1997216375,kv=3300458951,logm=2977399903}) 2024-05-15T23:08:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3621621085}) 2024-05-15T23:08:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2650414573}) 2024-05-15T23:08:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518723471}) 2024-05-15T23:08:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3322705727}) 2024-05-15T23:08:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3425607191}) 2024-05-15T23:08:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1750031316}) 2024-05-15T23:08:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1273409153}) 2024-05-15T23:08:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2933815105}) 2024-05-15T23:08:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1432883674}) 2024-05-15T23:08:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2721550962}) 2024-05-15T23:08:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2199476607}) 2024-05-15T23:08:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2516775559}) 2024-05-15T23:08:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1121547100}) 2024-05-15T23:08:40.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=537628518}) 2024-05-15T23:08:40.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=667289590}) 2024-05-15T23:08:40.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477413509}) 2024-05-15T23:08:40.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3527964219}) 2024-05-15T23:08:40.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1575499763}) 2024-05-15T23:08:40.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2118497697}) 2024-05-15T23:08:40.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1360358774}) 2024-05-15T23:08:40.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2338062457}) 2024-05-15T23:08:40.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2920108469}) 2024-05-15T23:08:40.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4023787172}) 2024-05-15T23:08:40.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1442702680}) 2024-05-15T23:08:40.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1184566881}) 2024-05-15T23:08:40.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=50,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=16} crc {logm=704868518,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=3719884697,mon_config_key=3049386775}) 2024-05-15T23:08:40.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=656496947}) 2024-05-15T23:08:40.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2807616862}) 2024-05-15T23:08:40.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=68,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=14} crc {mon_config_key=1118700187,monmap=1478146745,osd_metadata=1317173833,osd_pg_creating=2209559030,osdmap=4173192563}) 2024-05-15T23:08:40.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1384316713}) 2024-05-15T23:08:40.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=11} crc {osdmap=721386556}) 2024-05-15T23:08:40.877 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:40.898 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:08:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:40.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:08:41.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:08:41.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: 2.d scrub starts 2024-05-15T23:08:41.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: 2.0 scrub starts 2024-05-15T23:08:41.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: 2.3 scrub ok 2024-05-15T23:08:41.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=54,logm=4} crc {auth=450933107,config=3313370793,health=1997216375,kv=3300458951,logm=2977399903}) 2024-05-15T23:08:41.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3621621085}) 2024-05-15T23:08:41.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2650414573}) 2024-05-15T23:08:41.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518723471}) 2024-05-15T23:08:41.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3322705727}) 2024-05-15T23:08:41.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3425607191}) 2024-05-15T23:08:41.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1750031316}) 2024-05-15T23:08:41.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1273409153}) 2024-05-15T23:08:41.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2933815105}) 2024-05-15T23:08:41.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1432883674}) 2024-05-15T23:08:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2721550962}) 2024-05-15T23:08:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2199476607}) 2024-05-15T23:08:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2516775559}) 2024-05-15T23:08:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1121547100}) 2024-05-15T23:08:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=537628518}) 2024-05-15T23:08:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=667289590}) 2024-05-15T23:08:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477413509}) 2024-05-15T23:08:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3527964219}) 2024-05-15T23:08:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1575499763}) 2024-05-15T23:08:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2118497697}) 2024-05-15T23:08:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1360358774}) 2024-05-15T23:08:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2338062457}) 2024-05-15T23:08:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2920108469}) 2024-05-15T23:08:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4023787172}) 2024-05-15T23:08:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1442702680}) 2024-05-15T23:08:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1184566881}) 2024-05-15T23:08:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=50,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=16} crc {logm=704868518,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=3719884697,mon_config_key=3049386775}) 2024-05-15T23:08:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=656496947}) 2024-05-15T23:08:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2807616862}) 2024-05-15T23:08:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=68,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=14} crc {mon_config_key=1118700187,monmap=1478146745,osd_metadata=1317173833,osd_pg_creating=2209559030,osdmap=4173192563}) 2024-05-15T23:08:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1384316713}) 2024-05-15T23:08:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=11} crc {osdmap=721386556}) 2024-05-15T23:08:41.069 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: 2.d scrub starts 2024-05-15T23:08:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: 2.0 scrub starts 2024-05-15T23:08:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: 2.3 scrub ok 2024-05-15T23:08:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=54,logm=4} crc {auth=450933107,config=3313370793,health=1997216375,kv=3300458951,logm=2977399903}) 2024-05-15T23:08:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3621621085}) 2024-05-15T23:08:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2650414573}) 2024-05-15T23:08:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518723471}) 2024-05-15T23:08:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3322705727}) 2024-05-15T23:08:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3425607191}) 2024-05-15T23:08:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1750031316}) 2024-05-15T23:08:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1273409153}) 2024-05-15T23:08:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2933815105}) 2024-05-15T23:08:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1432883674}) 2024-05-15T23:08:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2721550962}) 2024-05-15T23:08:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2199476607}) 2024-05-15T23:08:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2516775559}) 2024-05-15T23:08:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1121547100}) 2024-05-15T23:08:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=537628518}) 2024-05-15T23:08:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=667289590}) 2024-05-15T23:08:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477413509}) 2024-05-15T23:08:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3527964219}) 2024-05-15T23:08:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1575499763}) 2024-05-15T23:08:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2118497697}) 2024-05-15T23:08:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1360358774}) 2024-05-15T23:08:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2338062457}) 2024-05-15T23:08:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2920108469}) 2024-05-15T23:08:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4023787172}) 2024-05-15T23:08:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1442702680}) 2024-05-15T23:08:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1184566881}) 2024-05-15T23:08:41.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=50,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=16} crc {logm=704868518,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=3719884697,mon_config_key=3049386775}) 2024-05-15T23:08:41.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=656496947}) 2024-05-15T23:08:41.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2807616862}) 2024-05-15T23:08:41.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=68,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=14} crc {mon_config_key=1118700187,monmap=1478146745,osd_metadata=1317173833,osd_pg_creating=2209559030,osdmap=4173192563}) 2024-05-15T23:08:41.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1384316713}) 2024-05-15T23:08:41.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=11} crc {osdmap=721386556}) 2024-05-15T23:08:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:41 smithi090 ceph-mon[44434]: 2.d scrub ok 2024-05-15T23:08:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:41 smithi090 ceph-mon[44434]: 2.0 scrub ok 2024-05-15T23:08:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:41 smithi090 ceph-mon[44434]: pgmap v476: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:08:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:41 smithi090 ceph-mon[44434]: 2.9 scrub starts 2024-05-15T23:08:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:41 smithi090 ceph-mon[44434]: 2.5 scrub starts 2024-05-15T23:08:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:41 smithi090 ceph-mon[44434]: 2.2 scrub starts 2024-05-15T23:08:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:41 smithi088 ceph-mon[44699]: 2.d scrub ok 2024-05-15T23:08:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:41 smithi088 ceph-mon[44699]: 2.0 scrub ok 2024-05-15T23:08:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:41 smithi088 ceph-mon[44699]: pgmap v476: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:08:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:41 smithi088 ceph-mon[44699]: 2.9 scrub starts 2024-05-15T23:08:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:41 smithi088 ceph-mon[44699]: 2.5 scrub starts 2024-05-15T23:08:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:41 smithi088 ceph-mon[44699]: 2.2 scrub starts 2024-05-15T23:08:42.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:41 smithi084 ceph-mon[39784]: 2.d scrub ok 2024-05-15T23:08:42.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:41 smithi084 ceph-mon[39784]: 2.0 scrub ok 2024-05-15T23:08:42.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:41 smithi084 ceph-mon[39784]: pgmap v476: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:08:42.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:41 smithi084 ceph-mon[39784]: 2.9 scrub starts 2024-05-15T23:08:42.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:41 smithi084 ceph-mon[39784]: 2.5 scrub starts 2024-05-15T23:08:42.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:41 smithi084 ceph-mon[39784]: 2.2 scrub starts 2024-05-15T23:08:42.194 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:08:42.195 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:08:42.963 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:42 smithi090 ceph-mon[44434]: 2.c scrub starts 2024-05-15T23:08:42.963 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:42 smithi090 ceph-mon[44434]: 2.2 scrub ok 2024-05-15T23:08:42.963 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:42 smithi090 ceph-mon[44434]: 2.5 scrub ok 2024-05-15T23:08:43.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T23:08:43.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:42 smithi088 ceph-mon[44699]: 2.c scrub starts 2024-05-15T23:08:43.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:42 smithi088 ceph-mon[44699]: 2.2 scrub ok 2024-05-15T23:08:43.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:42 smithi088 ceph-mon[44699]: 2.5 scrub ok 2024-05-15T23:08:43.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:42 smithi084 ceph-mon[39784]: 2.c scrub starts 2024-05-15T23:08:43.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:42 smithi084 ceph-mon[39784]: 2.2 scrub ok 2024-05-15T23:08:43.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:42 smithi084 ceph-mon[39784]: 2.5 scrub ok 2024-05-15T23:08:43.434 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:08:44.003 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:43 smithi090 ceph-mon[44434]: 2.9 scrub ok 2024-05-15T23:08:44.004 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:43 smithi090 ceph-mon[44434]: pgmap v477: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:08:44.004 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:43 smithi090 ceph-mon[44434]: 2.b scrub starts 2024-05-15T23:08:44.004 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:43 smithi090 ceph-mon[44434]: 2.c scrub ok 2024-05-15T23:08:44.053 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:43 smithi084 ceph-mon[39784]: 2.9 scrub ok 2024-05-15T23:08:44.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:43 smithi084 ceph-mon[39784]: pgmap v477: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:08:44.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:43 smithi084 ceph-mon[39784]: 2.b scrub starts 2024-05-15T23:08:44.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:43 smithi084 ceph-mon[39784]: 2.c scrub ok 2024-05-15T23:08:44.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:43 smithi088 ceph-mon[44699]: 2.9 scrub ok 2024-05-15T23:08:44.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:43 smithi088 ceph-mon[44699]: pgmap v477: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:08:44.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:43 smithi088 ceph-mon[44699]: 2.b scrub starts 2024-05-15T23:08:44.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:43 smithi088 ceph-mon[44699]: 2.c scrub ok 2024-05-15T23:08:45.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:44 smithi084 ceph-mon[39784]: 2.b scrub ok 2024-05-15T23:08:45.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:44 smithi084 ceph-mon[39784]: 2.7 scrub starts 2024-05-15T23:08:45.203 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:44 smithi090 ceph-mon[44434]: 2.b scrub ok 2024-05-15T23:08:45.203 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:44 smithi090 ceph-mon[44434]: 2.7 scrub starts 2024-05-15T23:08:45.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:44 smithi088 ceph-mon[44699]: 2.b scrub ok 2024-05-15T23:08:45.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:44 smithi088 ceph-mon[44699]: 2.7 scrub starts 2024-05-15T23:08:45.542 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:08:45.543 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:08:45.826 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:45 smithi090 ceph-mon[44434]: pgmap v478: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s 2024-05-15T23:08:45.826 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:45 smithi090 ceph-mon[44434]: 2.4 scrub starts 2024-05-15T23:08:45.826 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:45 smithi090 ceph-mon[44434]: 2.7 scrub ok 2024-05-15T23:08:46.081 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:08:46.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:46.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:08:46.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:45 smithi084 ceph-mon[39784]: pgmap v478: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s 2024-05-15T23:08:46.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:45 smithi084 ceph-mon[39784]: 2.4 scrub starts 2024-05-15T23:08:46.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:45 smithi084 ceph-mon[39784]: 2.7 scrub ok 2024-05-15T23:08:46.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:08:46.229 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:46.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:45 smithi088 ceph-mon[44699]: pgmap v478: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s 2024-05-15T23:08:46.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:45 smithi088 ceph-mon[44699]: 2.4 scrub starts 2024-05-15T23:08:46.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:45 smithi088 ceph-mon[44699]: 2.7 scrub ok 2024-05-15T23:08:46.366 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:08:46.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:46.373 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:08:46.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T23:08:46.446 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:46.474 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:08:46.490 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:08:46.500 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:46.520 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:08:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:46.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:08:46.524 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:08:46.551 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:08:46.582 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:08:46.622 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:08:46.624 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:08:46.650 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:46.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:08:46.669 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:08:46.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:46.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:08:46.687 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:08:46.718 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:08:46.767 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:08:46.773 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:08:46.804 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:08:46.831 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:08:46.838 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:08:46.847 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:46.879 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:08:46.881 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:08:46.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:46.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:08:46.902 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:08:46.902 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:08:46.902 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:08:46.902 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:08:46.902 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:08:46.902 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:08:46.902 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:08:46.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:08:46.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:08:46.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:08:46.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:08:46.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:08:46.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:08:47.026 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:47.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:46 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:08:47.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:46 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:08:47.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:46 smithi084 ceph-mon[39784]: 2.a scrub starts 2024-05-15T23:08:47.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:46 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:08:47.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:46 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:08:47.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:46 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3230223999' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:08:47.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:47.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:47.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:47.097 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:08:47.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:47.100 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:08:47.100 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:46 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:08:47.100 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:46 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:08:47.101 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:46 smithi090 ceph-mon[44434]: 2.a scrub starts 2024-05-15T23:08:47.101 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:46 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:08:47.101 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:46 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:08:47.101 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:46 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3230223999' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:08:47.101 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:47.101 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:47.101 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:47.201 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:08:47.247 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:46 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:08:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:46 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:08:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:46 smithi088 ceph-mon[44699]: 2.a scrub starts 2024-05-15T23:08:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:46 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:08:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:46 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:08:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:46 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3230223999' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:08:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:47.388 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:08:47.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:47.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:08:47.500 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:08:47.540 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:47.589 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:08:47.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:47.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:08:47.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:08:47.744 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:47.801 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:08:47.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:47.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:08:47.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:08:47.964 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:48.048 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:47 smithi084 ceph-mon[39784]: 2.4 scrub ok 2024-05-15T23:08:48.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:47 smithi084 ceph-mon[39784]: pgmap v479: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-05-15T23:08:48.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:47 smithi084 ceph-mon[39784]: osdmap e156: 12 total, 12 up, 12 in 2024-05-15T23:08:48.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:47 smithi084 ceph-mon[39784]: 2.a scrub ok 2024-05-15T23:08:48.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:47 smithi090 ceph-mon[44434]: 2.4 scrub ok 2024-05-15T23:08:48.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:47 smithi090 ceph-mon[44434]: pgmap v479: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-05-15T23:08:48.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:47 smithi090 ceph-mon[44434]: osdmap e156: 12 total, 12 up, 12 in 2024-05-15T23:08:48.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:47 smithi090 ceph-mon[44434]: 2.a scrub ok 2024-05-15T23:08:48.202 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:08:48.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:48.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:08:48.306 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:08:48.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:47 smithi088 ceph-mon[44699]: 2.4 scrub ok 2024-05-15T23:08:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:47 smithi088 ceph-mon[44699]: pgmap v479: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-05-15T23:08:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:47 smithi088 ceph-mon[44699]: osdmap e156: 12 total, 12 up, 12 in 2024-05-15T23:08:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:47 smithi088 ceph-mon[44699]: 2.a scrub ok 2024-05-15T23:08:48.334 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:48.358 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:08:48.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:48.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:08:48.462 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:08:48.520 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:48.550 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:08:48.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:48.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:08:48.660 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:08:48.697 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:48.739 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:08:48.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:48.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:08:48.848 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:08:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:48.897 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:08:48.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:48.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:08:48.911 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:08:48.911 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:08:48.911 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-05-15T23:08:48.911 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:08:49.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:08:49.050 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:49.091 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:08:49.091 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:08:49.091 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-15T23:08:49.091 INFO:tasks.thrashosds.thrasher:Setting osd 8 primary_affinity to 0.544863 2024-05-15T23:08:49.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 8 0.5448625852879737 2024-05-15T23:08:49.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:48 smithi084 ceph-mon[39784]: 2.f scrub starts 2024-05-15T23:08:49.145 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:08:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:49.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:08:49.154 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:48 smithi090 ceph-mon[44434]: 2.f scrub starts 2024-05-15T23:08:49.253 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:08:49.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:48 smithi088 ceph-mon[44699]: 2.f scrub starts 2024-05-15T23:08:49.279 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:49.299 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:08:49.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:49.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:08:49.404 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:08:49.437 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:49.566 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:49.747 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T23:08:49.793 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:08:49.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:49.796 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:08:49.895 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:08:49.897 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:08:49.927 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:49.946 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:08:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:49.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:08:50.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:08:50.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:49 smithi090 ceph-mon[44434]: pgmap v481: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-05-15T23:08:50.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:49 smithi090 ceph-mon[44434]: 2.f scrub ok 2024-05-15T23:08:50.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:49 smithi090 ceph-mon[44434]: from='client.15039 v1:172.21.15.84:0/2155240191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:08:50.085 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:08:50.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:49 smithi084 ceph-mon[39784]: pgmap v481: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-05-15T23:08:50.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:49 smithi084 ceph-mon[39784]: 2.f scrub ok 2024-05-15T23:08:50.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:49 smithi084 ceph-mon[39784]: from='client.15039 v1:172.21.15.84:0/2155240191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:08:50.118 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:50.152 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:08:50.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:50.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:08:50.211 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:08:50.257 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:08:50.281 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:49 smithi088 ceph-mon[44699]: pgmap v481: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-05-15T23:08:50.282 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:49 smithi088 ceph-mon[44699]: 2.f scrub ok 2024-05-15T23:08:50.282 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:49 smithi088 ceph-mon[44699]: from='client.15039 v1:172.21.15.84:0/2155240191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:08:50.284 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:50.305 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:08:50.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:50.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:08:50.409 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:50.458 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:50.545 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:50.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:50.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:50.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:08:50.680 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:51.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:50 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:08:51.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:50 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:08:51.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:50 smithi090 ceph-mon[44434]: from='client.15039 v1:172.21.15.84:0/2155240191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:08:51.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:50 smithi090 ceph-mon[44434]: osdmap e157: 12 total, 12 up, 12 in 2024-05-15T23:08:51.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:50 smithi090 ceph-mon[44434]: 2.6 scrub starts 2024-05-15T23:08:51.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:50 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:08:51.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:50 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:08:51.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:50 smithi084 ceph-mon[39784]: from='client.15039 v1:172.21.15.84:0/2155240191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:08:51.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:50 smithi084 ceph-mon[39784]: osdmap e157: 12 total, 12 up, 12 in 2024-05-15T23:08:51.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:50 smithi084 ceph-mon[39784]: 2.6 scrub starts 2024-05-15T23:08:51.155 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:08:51.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:51.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:08:51.260 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:51.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:50 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:08:51.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:50 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:08:51.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:50 smithi088 ceph-mon[44699]: from='client.15039 v1:172.21.15.84:0/2155240191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:08:51.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:50 smithi088 ceph-mon[44699]: osdmap e157: 12 total, 12 up, 12 in 2024-05-15T23:08:51.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:50 smithi088 ceph-mon[44699]: 2.6 scrub starts 2024-05-15T23:08:51.323 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:51.580 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:51.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:51.636 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:51.684 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:08:51.729 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:51.757 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:08:51.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:51.762 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:08:51.862 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:08:51.901 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:52.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:51 smithi090 ceph-mon[44434]: pgmap v483: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-05-15T23:08:52.058 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:08:52.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:52.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:08:52.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:08:52.199 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:52.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:51 smithi088 ceph-mon[44699]: pgmap v483: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-05-15T23:08:52.367 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:08:52.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:51 smithi084 ceph-mon[39784]: pgmap v483: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-05-15T23:08:52.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:52.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:08:52.406 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:08:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:08:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:08:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:08:52.406 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:08:52.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:08:52.536 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:52.581 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:08:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:52.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:08:52.687 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:52.714 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:52.734 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:52.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:52.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:08:52.874 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:52.905 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.8 primary-affinity to 0.544863 (8357082) 2024-05-15T23:08:53.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:52 smithi088 ceph-mon[44699]: 2.6 scrub ok 2024-05-15T23:08:53.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:52 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3147795383' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 0.5448625852879737}]: dispatch 2024-05-15T23:08:53.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:52 smithi088 ceph-mon[44699]: from='client.24970 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 0.5448625852879737}]: dispatch 2024-05-15T23:08:53.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:52 smithi084 ceph-mon[39784]: 2.6 scrub ok 2024-05-15T23:08:53.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:52 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3147795383' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 0.5448625852879737}]: dispatch 2024-05-15T23:08:53.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:52 smithi084 ceph-mon[39784]: from='client.24970 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 0.5448625852879737}]: dispatch 2024-05-15T23:08:53.228 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:52 smithi090 ceph-mon[44434]: 2.6 scrub ok 2024-05-15T23:08:53.228 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:52 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3147795383' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 0.5448625852879737}]: dispatch 2024-05-15T23:08:53.229 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:52 smithi090 ceph-mon[44434]: from='client.24970 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 0.5448625852879737}]: dispatch 2024-05-15T23:08:53.233 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:08:53.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:53.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:08:53.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T23:08:53.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:08:53.364 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:53.383 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:08:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:53.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:08:53.487 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:53.518 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:53.539 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:53.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:53.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:53.557 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:08:53.623 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:08:53.643 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:08:53.669 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:53.818 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:08:53.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:53.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:08:53.922 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:08:53.953 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:53 smithi090 ceph-mon[44434]: pgmap v484: 17 pgs: 17 active+clean; 4.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-05-15T23:08:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:53 smithi090 ceph-mon[44434]: from='client.24970 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 0.5448625852879737}]': finished 2024-05-15T23:08:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:53 smithi090 ceph-mon[44434]: osdmap e158: 12 total, 12 up, 12 in 2024-05-15T23:08:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:53 smithi090 ceph-mon[44434]: osdmap e159: 12 total, 12 up, 12 in 2024-05-15T23:08:54.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:53 smithi084 ceph-mon[39784]: pgmap v484: 17 pgs: 17 active+clean; 4.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-05-15T23:08:54.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:53 smithi084 ceph-mon[39784]: from='client.24970 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 0.5448625852879737}]': finished 2024-05-15T23:08:54.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:53 smithi084 ceph-mon[39784]: osdmap e158: 12 total, 12 up, 12 in 2024-05-15T23:08:54.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:53 smithi084 ceph-mon[39784]: osdmap e159: 12 total, 12 up, 12 in 2024-05-15T23:08:54.219 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:08:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:54.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:08:54.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:53 smithi088 ceph-mon[44699]: pgmap v484: 17 pgs: 17 active+clean; 4.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-05-15T23:08:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:53 smithi088 ceph-mon[44699]: from='client.24970 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 0.5448625852879737}]': finished 2024-05-15T23:08:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:53 smithi088 ceph-mon[44699]: osdmap e158: 12 total, 12 up, 12 in 2024-05-15T23:08:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:53 smithi088 ceph-mon[44699]: osdmap e159: 12 total, 12 up, 12 in 2024-05-15T23:08:54.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:54.381 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:54.601 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:54.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:54.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:54.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:54.745 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:54.810 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:54.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:54.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:54.918 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:54.945 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:54.965 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:54.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:54.968 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:55.040 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:08:55.068 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:08:55.088 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:08:55.094 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:55.116 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:08:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:55.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:08:55.126 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:08:55.179 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:08:55.205 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:08:55.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:08:55.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:08:55.263 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:55.274 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:08:55.302 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:08:55.332 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:08:55.364 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:08:55.397 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:08:55.439 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:08:55.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:08:55.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:08:55.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:08:55.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:08:55.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:08:55.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:08:55.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:08:55.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:08:55.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:08:55.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:08:55.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:08:55.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:08:55.466 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:08:55.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:55.474 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:08:55.522 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:08:55.522 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:08:55.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:08:55.602 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:55.621 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:08:55.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:55.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:08:55.725 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:08:55.754 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:55.781 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:08:55.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:55.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:08:55.888 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:08:55.927 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:56.291 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:08:56.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:55 smithi090 ceph-mon[44434]: pgmap v487: 17 pgs: 17 active+clean; 4.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:08:56.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:55 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1710615212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:08:56.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:56.298 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:08:56.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:55 smithi084 ceph-mon[39784]: pgmap v487: 17 pgs: 17 active+clean; 4.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:08:56.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:55 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1710615212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:08:56.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:55 smithi088 ceph-mon[44699]: pgmap v487: 17 pgs: 17 active+clean; 4.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:08:56.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:55 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1710615212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:08:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:56.398 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:56.427 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:56.446 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:56.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:56.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:56.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:08:56.591 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:56.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T23:08:56.946 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:08:56.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:56.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:08:57.050 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:57.076 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:57.098 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:57.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:57.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:57.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:08:57.253 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:57.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.315 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:08:57.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:08:57.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:57.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:08:57.343 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:08:57.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:08:57.452 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:57.503 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:08:57.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:57.506 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:08:57.607 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:57.634 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:57.661 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:57.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:57.764 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:08:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:57.857 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:08:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:57.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:08:57.965 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:08:58.003 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:58.055 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:08:58.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:58.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:08:58.126 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:57 smithi084 ceph-mon[39784]: pgmap v488: 17 pgs: 17 active+clean; 4.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 924 op/s 2024-05-15T23:08:58.159 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:58.188 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:57 smithi088 ceph-mon[44699]: pgmap v488: 17 pgs: 17 active+clean; 4.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 924 op/s 2024-05-15T23:08:58.190 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:58.211 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:58.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:58.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:58.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:57 smithi090 ceph-mon[44434]: pgmap v488: 17 pgs: 17 active+clean; 4.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 924 op/s 2024-05-15T23:08:58.315 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:08:58.342 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:58.367 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:08:58.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:58.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:08:58.470 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:58.516 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:58.542 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:58.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:58.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:58.646 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:08:58.658 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:08:58.658 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:08:58.658 INFO:tasks.thrashosds.thrasher:Removing osd 9, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:08:58.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd out 9 2024-05-15T23:08:58.673 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:58.692 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:08:58.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:58.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:08:58.798 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:08:58.850 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:58.898 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:08:58.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:58.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:08:58.965 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:08:59.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:08:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:58 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:08:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:58 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:08:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:58 smithi090 ceph-mon[44434]: osdmap e160: 12 total, 12 up, 12 in 2024-05-15T23:08:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:58 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2432576170' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:08:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:58 smithi090 ceph-mon[44434]: from='client.35015 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:08:59.063 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:59.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:58 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:08:59.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:58 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:08:59.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:58 smithi084 ceph-mon[39784]: osdmap e160: 12 total, 12 up, 12 in 2024-05-15T23:08:59.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:58 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2432576170' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:08:59.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:58 smithi084 ceph-mon[39784]: from='client.35015 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:08:59.232 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:08:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:59.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:08:59.255 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:08:59.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:58 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:08:59.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:58 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:08:59.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:58 smithi088 ceph-mon[44699]: osdmap e160: 12 total, 12 up, 12 in 2024-05-15T23:08:59.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:58 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2432576170' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:08:59.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:58 smithi088 ceph-mon[44699]: from='client.35015 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:08:59.337 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:08:59.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:08:59.378 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:59.394 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:08:59.394 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:08:59.418 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:08:59.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:59.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:08:59.532 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:08:59.559 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:59.578 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:08:59.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:59.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:08:59.687 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:08:59.714 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:08:59.735 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:08:59.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:08:59.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:08:59.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:08:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:59 smithi090 ceph-mon[44434]: pgmap v490: 17 pgs: 17 active+clean; 4.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 751 op/s 2024-05-15T23:09:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:59 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:09:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:59 smithi090 ceph-mon[44434]: from='client.35015 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:09:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:08:59 smithi090 ceph-mon[44434]: osdmap e161: 12 total, 12 up, 12 in 2024-05-15T23:09:00.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:59 smithi084 ceph-mon[39784]: pgmap v490: 17 pgs: 17 active+clean; 4.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 751 op/s 2024-05-15T23:09:00.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:59 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:09:00.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:59 smithi084 ceph-mon[39784]: from='client.35015 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:09:00.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:08:59 smithi084 ceph-mon[39784]: osdmap e161: 12 total, 12 up, 12 in 2024-05-15T23:09:00.138 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:09:00.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:00.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:09:00.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T23:09:00.245 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:09:00.292 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:59 smithi088 ceph-mon[44699]: pgmap v490: 17 pgs: 17 active+clean; 4.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 751 op/s 2024-05-15T23:09:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:59 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:09:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:59 smithi088 ceph-mon[44699]: from='client.35015 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:09:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:08:59 smithi088 ceph-mon[44699]: osdmap e161: 12 total, 12 up, 12 in 2024-05-15T23:09:00.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:00.322 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:09:00.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:09:00.423 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:09:00.451 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:09:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:00.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:09:00.513 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:09:00.574 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:09:00.621 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:00.794 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:09:00.800 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:00.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:09:00.903 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:09:00.930 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:00.952 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:09:00.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:00.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:09:01.057 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:09:01.083 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:01.103 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:09:01.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:01.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:09:01.207 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:09:01.243 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:01.807 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:09:01.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:01.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:09:01.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:09:01.917 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:09:01.945 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:01.965 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:09:01.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:01.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:09:02.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:01 smithi084 ceph-mon[39784]: pgmap v492: 17 pgs: 17 active+clean; 4.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-05-15T23:09:02.069 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:09:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:09:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:02.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:09:02.221 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:09:02.284 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:01 smithi090 ceph-mon[44434]: pgmap v492: 17 pgs: 17 active+clean; 4.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-05-15T23:09:02.319 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:09:02.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:01 smithi088 ceph-mon[44699]: pgmap v492: 17 pgs: 17 active+clean; 4.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-05-15T23:09:02.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:02.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:09:02.425 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:09:02.477 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:09:02.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:02.532 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:09:02.633 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:09:02.708 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:02.823 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:09:02.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:02.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:09:02.928 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:09:02.979 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:03.079 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:09:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:03.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:09:03.102 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:09:03.102 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:09:03.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:03.231 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:03.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:03 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:09:03.099+0000 7fa0938b7700 -1 mon.a@0(leader).osd e162 definitely_dead 0 2024-05-15T23:09:03.371 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:03.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:03.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:03.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:09:03.525 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:03.633 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:09:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:03.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:09:03.683 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:03.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:09:03.779 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:03.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T23:09:03.969 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.9. 2024-05-15T23:09:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:03 smithi084 ceph-mon[39784]: pgmap v493: 17 pgs: 17 active+clean; 4.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:09:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:03 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:09:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:03 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:09:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:03 smithi084 ceph-mon[39784]: osdmap e162: 12 total, 12 up, 12 in 2024-05-15T23:09:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3615143630' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-05-15T23:09:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:03 smithi084 ceph-mon[39784]: Client client.admin marked osd.9 out, while it was still marked up 2024-05-15T23:09:04.143 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:09:04.233 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:09:04.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:04.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:09:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:03 smithi090 ceph-mon[44434]: pgmap v493: 17 pgs: 17 active+clean; 4.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:09:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:03 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:09:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:03 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:09:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:03 smithi090 ceph-mon[44434]: osdmap e162: 12 total, 12 up, 12 in 2024-05-15T23:09:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3615143630' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-05-15T23:09:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:03 smithi090 ceph-mon[44434]: Client client.admin marked osd.9 out, while it was still marked up 2024-05-15T23:09:04.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:03 smithi088 ceph-mon[44699]: pgmap v493: 17 pgs: 17 active+clean; 4.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:09:04.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:03 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:09:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:03 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:09:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:03 smithi088 ceph-mon[44699]: osdmap e162: 12 total, 12 up, 12 in 2024-05-15T23:09:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3615143630' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-05-15T23:09:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:03 smithi088 ceph-mon[44699]: Client client.admin marked osd.9 out, while it was still marked up 2024-05-15T23:09:04.338 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:09:04.364 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:04.382 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:09:04.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:04.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:09:04.486 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:09:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:04.531 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:09:04.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:04.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:09:04.634 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:09:04.660 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:04.679 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:09:04.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:04.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:09:04.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:09:04.810 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:04.908 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:09:04.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:04.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:09:05.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:09:05.039 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3615143630' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-05-15T23:09:05.039 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:04 smithi084 ceph-mon[39784]: osdmap e163: 12 total, 12 up, 11 in 2024-05-15T23:09:05.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/470884332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:09:05.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:05.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:05.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:05.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:05.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:05.042 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:05.190 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:09:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:05.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:09:05.294 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:09:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3615143630' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-05-15T23:09:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:04 smithi090 ceph-mon[44434]: osdmap e163: 12 total, 12 up, 11 in 2024-05-15T23:09:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/470884332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:09:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:05.320 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3615143630' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-05-15T23:09:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:04 smithi088 ceph-mon[44699]: osdmap e163: 12 total, 12 up, 11 in 2024-05-15T23:09:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/470884332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:09:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:05.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:09:05.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:09:05.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:09:05.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:09:05.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:09:05.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:09:05.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:09:05.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:09:05.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:09:05.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:09:05.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:09:05.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:09:05.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:09:05.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:09:05.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:09:05.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:09:05.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:09:05.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:09:05.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:09:05.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:09:05.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:09:05.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:09:05.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:09:05.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:09:05.956 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:09:05.956 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:09:05.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:09:05.957 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:09:05.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.957 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109643 2.c 2:35bed8ca:::benchmark_data_smithi090_78160_object109642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:09:05.957 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.334876+0000", 2024-05-15T23:09:05.957 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 132.61848691500001, 2024-05-15T23:09:05.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.131093448, 2024-05-15T23:09:05.957 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:05.957 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:05.957 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:05.957 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:05.957 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:05.957 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:05.957 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109643 2024-05-15T23:09:05.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.958 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:05.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:05.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334876+0000", 2024-05-15T23:09:05.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:05.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334876+0000", 2024-05-15T23:09:05.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334878+0000", 2024-05-15T23:09:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5290000000000001e-06 2024-05-15T23:09:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334899+0000", 2024-05-15T23:09:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0951999999999999e-05 2024-05-15T23:09:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334909+0000", 2024-05-15T23:09:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308e-05 2024-05-15T23:09:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:05.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334917+0000", 2024-05-15T23:09:05.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3119999999999996e-06 2024-05-15T23:09:05.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:05.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.339148+0000", 2024-05-15T23:09:05.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004231004 2024-05-15T23:09:05.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:05.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.339226+0000", 2024-05-15T23:09:05.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8885000000000001e-05 2024-05-15T23:09:05.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:09:05.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.339304+0000", 2024-05-15T23:09:05.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7218000000000005e-05 2024-05-15T23:09:05.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:05.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.343262+0000", 2024-05-15T23:09:05.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003958746 2024-05-15T23:09:05.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:05.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351963+0000", 2024-05-15T23:09:05.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087008099999999998 2024-05-15T23:09:05.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:05.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.352004+0000", 2024-05-15T23:09:05.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0442999999999999e-05 2024-05-15T23:09:05.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:05.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465970+0000", 2024-05-15T23:09:05.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.113966241 2024-05-15T23:09:05.962 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.962 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:05.962 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.963 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109646 2.c 2:300500bb:::benchmark_data_smithi090_78160_object109645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:09:05.963 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.337707+0000", 2024-05-15T23:09:05.963 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 132.615656465, 2024-05-15T23:09:05.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128404672, 2024-05-15T23:09:05.963 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:05.963 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:05.963 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:05.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:05.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:05.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:05.963 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109646 2024-05-15T23:09:05.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.963 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:05.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:05.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337707+0000", 2024-05-15T23:09:05.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:05.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337707+0000", 2024-05-15T23:09:05.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:05.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337708+0000", 2024-05-15T23:09:05.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.511e-06 2024-05-15T23:09:05.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:05.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337728+0000", 2024-05-15T23:09:05.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0055999999999999e-05 2024-05-15T23:09:05.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:05.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337739+0000", 2024-05-15T23:09:05.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0471e-05 2024-05-15T23:09:05.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:05.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337746+0000", 2024-05-15T23:09:05.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4059999999999999e-06 2024-05-15T23:09:05.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:05.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.340943+0000", 2024-05-15T23:09:05.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031963859999999998 2024-05-15T23:09:05.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:05.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.341055+0000", 2024-05-15T23:09:05.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011188600000000001 2024-05-15T23:09:05.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:09:05.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.341175+0000", 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120839 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.343368+0000", 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021920669999999998 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466023+0000", 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.122655237 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466075+0000", 2024-05-15T23:09:05.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1910000000000003e-05 2024-05-15T23:09:05.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:05.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466112+0000", 2024-05-15T23:09:05.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6903000000000002e-05 2024-05-15T23:09:05.968 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.968 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:05.968 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.968 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.24745.0:109643 2.c e127/102 ondisk, result = 0)", 2024-05-15T23:09:05.968 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.351843+0000", 2024-05-15T23:09:05.968 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 132.601520255, 2024-05-15T23:09:05.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.114139029, 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351843+0000", 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351843+0000", 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:05.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351843+0000", 2024-05-15T23:09:05.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9000000000000002e-07 2024-05-15T23:09:05.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:05.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351846+0000", 2024-05-15T23:09:05.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1179999999999999e-06 2024-05-15T23:09:05.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:05.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351849+0000", 2024-05-15T23:09:05.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8510000000000001e-06 2024-05-15T23:09:05.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:05.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351853+0000", 2024-05-15T23:09:05.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3890000000000001e-06 2024-05-15T23:09:05.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:05.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351950+0000", 2024-05-15T23:09:05.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7557000000000001e-05 2024-05-15T23:09:05.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:05.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351958+0000", 2024-05-15T23:09:05.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5310000000000004e-06 2024-05-15T23:09:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465982+0000", 2024-05-15T23:09:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.114024193 2024-05-15T23:09:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.24745.0:109646 2.c e127/102 ondisk, result = 0)", 2024-05-15T23:09:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.351916+0000", 2024-05-15T23:09:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 132.60144786000001, 2024-05-15T23:09:05.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.114202603, 2024-05-15T23:09:05.973 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:05.973 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:05.973 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:09:05.973 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:05.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:05.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351916+0000", 2024-05-15T23:09:05.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:05.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351916+0000", 2024-05-15T23:09:05.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:05.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351917+0000", 2024-05-15T23:09:05.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2759999999999999e-06 2024-05-15T23:09:05.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:05.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351921+0000", 2024-05-15T23:09:05.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1559999999999997e-06 2024-05-15T23:09:05.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:05.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351929+0000", 2024-05-15T23:09:05.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7500000000000003e-06 2024-05-15T23:09:05.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:05.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351934+0000", 2024-05-15T23:09:05.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.079e-06 2024-05-15T23:09:05.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:05.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466009+0000", 2024-05-15T23:09:05.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11407569300000001 2024-05-15T23:09:05.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:05.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466017+0000", 2024-05-15T23:09:05.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6350000000000006e-06 2024-05-15T23:09:05.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:05.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466118+0000", 2024-05-15T23:09:05.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010101400000000001 2024-05-15T23:09:05.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.976 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:05.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.976 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109656 2.a 2:509d81bb:::benchmark_data_smithi090_78160_object109655:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:09:05.976 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.354844+0000", 2024-05-15T23:09:05.976 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 132.59851918999999, 2024-05-15T23:09:05.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.119458562, 2024-05-15T23:09:05.976 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:05.976 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109656 2024-05-15T23:09:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.354844+0000", 2024-05-15T23:09:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.354844+0000", 2024-05-15T23:09:05.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:05.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.354848+0000", 2024-05-15T23:09:05.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3960000000000002e-06 2024-05-15T23:09:05.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:05.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.354888+0000", 2024-05-15T23:09:05.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0673999999999997e-05 2024-05-15T23:09:05.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:05.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466170+0000", 2024-05-15T23:09:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11128214 2024-05-15T23:09:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466180+0000", 2024-05-15T23:09:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6779999999999995e-06 2024-05-15T23:09:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466282+0000", 2024-05-15T23:09:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101661 2024-05-15T23:09:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:05.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466411+0000", 2024-05-15T23:09:05.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012923899999999999 2024-05-15T23:09:05.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:09:05.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466561+0000", 2024-05-15T23:09:05.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014973300000000001 2024-05-15T23:09:05.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:05.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.470610+0000", 2024-05-15T23:09:05.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040489970000000004 2024-05-15T23:09:05.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:05.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.474147+0000", 2024-05-15T23:09:05.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035371539999999998 2024-05-15T23:09:05.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:05.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.474210+0000", 2024-05-15T23:09:05.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3291000000000001e-05 2024-05-15T23:09:05.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:05.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.474303+0000", 2024-05-15T23:09:05.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2598999999999999e-05 2024-05-15T23:09:05.981 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.981 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:05.981 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.982 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:111038 2.a 2:5beaf1f7:::benchmark_data_smithi090_78160_object111037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:09:05.982 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:54.379828+0000", 2024-05-15T23:09:05.982 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 131.573535803, 2024-05-15T23:09:05.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067266323000000003, 2024-05-15T23:09:05.982 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:05.982 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:05.982 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:05.982 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:05.982 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:05.982 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:05.982 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 111038 2024-05-15T23:09:05.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.982 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:05.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:05.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.379828+0000", 2024-05-15T23:09:05.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:05.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.379828+0000", 2024-05-15T23:09:05.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:05.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.379829+0000", 2024-05-15T23:09:05.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1179999999999999e-06 2024-05-15T23:09:05.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:05.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.379849+0000", 2024-05-15T23:09:05.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.039e-05 2024-05-15T23:09:05.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:05.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.379858+0000", 2024-05-15T23:09:05.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.738e-06 2024-05-15T23:09:05.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:05.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.379864+0000", 2024-05-15T23:09:05.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.241e-06 2024-05-15T23:09:05.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:05.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.379910+0000", 2024-05-15T23:09:05.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5918999999999999e-05 2024-05-15T23:09:05.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:05.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.380001+0000", 2024-05-15T23:09:05.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1392999999999997e-05 2024-05-15T23:09:05.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:09:05.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.380079+0000", 2024-05-15T23:09:05.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7715999999999999e-05 2024-05-15T23:09:05.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:05.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.382711+0000", 2024-05-15T23:09:05.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026322070000000001 2024-05-15T23:09:05.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:05.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.446979+0000", 2024-05-15T23:09:05.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.064268061000000001 2024-05-15T23:09:05.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:05.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.447044+0000", 2024-05-15T23:09:05.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5073999999999994e-05 2024-05-15T23:09:05.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:05.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.447094+0000", 2024-05-15T23:09:05.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9465999999999997e-05 2024-05-15T23:09:05.987 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.987 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:05.987 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.987 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:111042 2.a 2:56e8af0b:::benchmark_data_smithi090_78160_object111041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:09:05.987 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:54.382913+0000", 2024-05-15T23:09:05.987 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 131.57045028499999, 2024-05-15T23:09:05.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.070487672000000001, 2024-05-15T23:09:05.988 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:05.988 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:05.988 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:05.988 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:05.988 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:05.988 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:05.988 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 111042 2024-05-15T23:09:05.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.988 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:05.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:05.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.382913+0000", 2024-05-15T23:09:05.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:05.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.382913+0000", 2024-05-15T23:09:05.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:05.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.382915+0000", 2024-05-15T23:09:05.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0109999999999999e-06 2024-05-15T23:09:05.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:05.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.382935+0000", 2024-05-15T23:09:05.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0367999999999999e-05 2024-05-15T23:09:05.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:05.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.382944+0000", 2024-05-15T23:09:05.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9800000000000004e-06 2024-05-15T23:09:05.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:05.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.382951+0000", 2024-05-15T23:09:05.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5989999999999998e-06 2024-05-15T23:09:05.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:05.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.382995+0000", 2024-05-15T23:09:05.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4229e-05 2024-05-15T23:09:05.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:05.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.383111+0000", 2024-05-15T23:09:05.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011558200000000001 2024-05-15T23:09:05.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:09:05.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.383234+0000", 2024-05-15T23:09:05.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122831 2024-05-15T23:09:05.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:05.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.447155+0000", 2024-05-15T23:09:05.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063920925000000003 2024-05-15T23:09:05.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:05.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.453301+0000", 2024-05-15T23:09:05.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061459150000000001 2024-05-15T23:09:05.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:05.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.453355+0000", 2024-05-15T23:09:05.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4589000000000002e-05 2024-05-15T23:09:05.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:05.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.453401+0000", 2024-05-15T23:09:05.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5643000000000003e-05 2024-05-15T23:09:05.993 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.993 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:05.993 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.993 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.24745.0:111038 2.a e127/102 ondisk, result = 0)", 2024-05-15T23:09:05.993 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:54.386501+0000", 2024-05-15T23:09:05.993 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 131.56686186799999, 2024-05-15T23:09:05.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.060600904999999997, 2024-05-15T23:09:05.993 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:05.993 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:05.993 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:09:05.993 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:05.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:05.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386501+0000", 2024-05-15T23:09:05.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:05.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386501+0000", 2024-05-15T23:09:05.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:05.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386503+0000", 2024-05-15T23:09:05.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.632e-06 2024-05-15T23:09:05.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:05.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386510+0000", 2024-05-15T23:09:05.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1740000000000003e-06 2024-05-15T23:09:05.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:05.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.444286+0000", 2024-05-15T23:09:05.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057775323000000003 2024-05-15T23:09:05.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:05.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.444294+0000", 2024-05-15T23:09:05.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.32e-06 2024-05-15T23:09:05.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:05.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.446962+0000", 2024-05-15T23:09:05.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026679130000000001 2024-05-15T23:09:05.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:05.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.446972+0000", 2024-05-15T23:09:05.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0414999999999999e-05 2024-05-15T23:09:05.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:05.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.447102+0000", 2024-05-15T23:09:05.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130128 2024-05-15T23:09:05.996 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.996 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:05.997 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:05.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.997 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:111053 2.a 2:5dc5d97a:::benchmark_data_smithi090_78160_object111052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:09:05.997 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:54.387348+0000", 2024-05-15T23:09:05.997 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 131.56601545999999, 2024-05-15T23:09:05.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.069716921000000001, 2024-05-15T23:09:05.997 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:05.997 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:05.997 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:05.997 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:05.997 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:05.997 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:05.997 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 111053 2024-05-15T23:09:05.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.998 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:05.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:05.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.387348+0000", 2024-05-15T23:09:05.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:05.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.387348+0000", 2024-05-15T23:09:05.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:05.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:05.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.387350+0000", 2024-05-15T23:09:05.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1239999999999999e-06 2024-05-15T23:09:05.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:05.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.387378+0000", 2024-05-15T23:09:05.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7487999999999998e-05 2024-05-15T23:09:05.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:05.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:05.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.444279+0000", 2024-05-15T23:09:05.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.056901936 2024-05-15T23:09:05.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:05.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:06.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.444292+0000", 2024-05-15T23:09:06.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2697999999999999e-05 2024-05-15T23:09:06.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:06.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.447192+0000", 2024-05-15T23:09:06.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028999099999999999 2024-05-15T23:09:06.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:06.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.447397+0000", 2024-05-15T23:09:06.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020483900000000001 2024-05-15T23:09:06.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:09:06.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.447543+0000", 2024-05-15T23:09:06.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000146236 2024-05-15T23:09:06.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:06.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.453287+0000", 2024-05-15T23:09:06.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0057439530000000004 2024-05-15T23:09:06.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:06.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.456968+0000", 2024-05-15T23:09:06.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00368133 2024-05-15T23:09:06.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:06.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.457019+0000", 2024-05-15T23:09:06.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0528000000000001e-05 2024-05-15T23:09:06.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:06.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.457065+0000", 2024-05-15T23:09:06.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5878999999999999e-05 2024-05-15T23:09:06.002 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.002 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:06.002 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.003 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:111051 2.b e127/102 2:dae0a3ed:::benchmark_data_smithi090_78160_object111050:head v 127'7031, mlcod=127'7029)", 2024-05-15T23:09:06.003 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:54.399676+0000", 2024-05-15T23:09:06.003 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 131.55368687500001, 2024-05-15T23:09:06.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050025429000000003, 2024-05-15T23:09:06.003 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:06.003 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:06.003 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:06.003 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:06.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:06.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.399676+0000", 2024-05-15T23:09:06.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:06.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.399676+0000", 2024-05-15T23:09:06.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:06.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.399679+0000", 2024-05-15T23:09:06.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2970000000000002e-06 2024-05-15T23:09:06.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:06.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.399718+0000", 2024-05-15T23:09:06.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9202999999999997e-05 2024-05-15T23:09:06.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:06.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.444283+0000", 2024-05-15T23:09:06.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.044564650999999997 2024-05-15T23:09:06.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:06.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.444294+0000", 2024-05-15T23:09:06.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1313e-05 2024-05-15T23:09:06.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:06.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.444358+0000", 2024-05-15T23:09:06.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4362000000000005e-05 2024-05-15T23:09:06.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.444382+0000", 2024-05-15T23:09:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3538000000000001e-05 2024-05-15T23:09:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.449640+0000", 2024-05-15T23:09:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052580049999999996 2024-05-15T23:09:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.449702+0000", 2024-05-15T23:09:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2059999999999999e-05 2024-05-15T23:09:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:116830 2.c 2:3b68eb5e:::benchmark_data_smithi090_78160_object116829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:09:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:58.395067+0000", 2024-05-15T23:09:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 127.558296544, 2024-05-15T23:09:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.047594221999999999, 2024-05-15T23:09:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 116830 2024-05-15T23:09:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.395067+0000", 2024-05-15T23:09:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.395067+0000", 2024-05-15T23:09:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.395069+0000", 2024-05-15T23:09:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.891e-06 2024-05-15T23:09:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.395089+0000", 2024-05-15T23:09:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0020000000000001e-05 2024-05-15T23:09:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.395099+0000", 2024-05-15T23:09:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0169e-05 2024-05-15T23:09:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.395105+0000", 2024-05-15T23:09:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.049e-06 2024-05-15T23:09:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.395164+0000", 2024-05-15T23:09:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8681999999999998e-05 2024-05-15T23:09:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.395295+0000", 2024-05-15T23:09:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013109499999999999 2024-05-15T23:09:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:09:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.395427+0000", 2024-05-15T23:09:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013236899999999999 2024-05-15T23:09:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.398937+0000", 2024-05-15T23:09:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035100019999999999 2024-05-15T23:09:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.442574+0000", 2024-05-15T23:09:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.043636988000000002 2024-05-15T23:09:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.442611+0000", 2024-05-15T23:09:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7007e-05 2024-05-15T23:09:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.442661+0000", 2024-05-15T23:09:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9950000000000001e-05 2024-05-15T23:09:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:145486 2.c 2:369a4c97:::benchmark_data_smithi090_78160_object145485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e138)", 2024-05-15T23:09:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:07:21.479220+0000", 2024-05-15T23:09:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 104.47414371000001, 2024-05-15T23:09:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.047832826000000002, 2024-05-15T23:09:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 145486 2024-05-15T23:09:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:21.479220+0000", 2024-05-15T23:09:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:21.479220+0000", 2024-05-15T23:09:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:21.479222+0000", 2024-05-15T23:09:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.137e-06 2024-05-15T23:09:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:21.479244+0000", 2024-05-15T23:09:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1963999999999998e-05 2024-05-15T23:09:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:21.479255+0000", 2024-05-15T23:09:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1452e-05 2024-05-15T23:09:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:21.479263+0000", 2024-05-15T23:09:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4660000000000002e-06 2024-05-15T23:09:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:21.479322+0000", 2024-05-15T23:09:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.906e-05 2024-05-15T23:09:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:21.479464+0000", 2024-05-15T23:09:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014254900000000001 2024-05-15T23:09:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:09:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:21.479616+0000", 2024-05-15T23:09:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015129999999999999 2024-05-15T23:09:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:21.496052+0000", 2024-05-15T23:09:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016436315999999999 2024-05-15T23:09:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:21.526988+0000", 2024-05-15T23:09:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030936466999999999 2024-05-15T23:09:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:21.527024+0000", 2024-05-15T23:09:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5565000000000003e-05 2024-05-15T23:09:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:21.527052+0000", 2024-05-15T23:09:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8549999999999999e-05 2024-05-15T23:09:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:172841 2.c 2:389e08a6:::benchmark_data_smithi090_78160_object11483:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e144)", 2024-05-15T23:09:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:07:44.791957+0000", 2024-05-15T23:09:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.161406091000003, 2024-05-15T23:09:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.048787983, 2024-05-15T23:09:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172841 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.791957+0000", 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.791957+0000", 2024-05-15T23:09:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.791958+0000", 2024-05-15T23:09:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7999999999999995e-07 2024-05-15T23:09:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.791961+0000", 2024-05-15T23:09:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6860000000000002e-06 2024-05-15T23:09:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.791966+0000", 2024-05-15T23:09:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9240000000000004e-06 2024-05-15T23:09:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.791972+0000", 2024-05-15T23:09:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2950000000000001e-06 2024-05-15T23:09:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.797020+0000", 2024-05-15T23:09:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050484229999999998 2024-05-15T23:09:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.797146+0000", 2024-05-15T23:09:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012613699999999999 2024-05-15T23:09:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:09:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.797243+0000", 2024-05-15T23:09:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6244999999999998e-05 2024-05-15T23:09:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.805623+0000", 2024-05-15T23:09:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0083808509999999999 2024-05-15T23:09:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.840670+0000", 2024-05-15T23:09:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035046257999999997 2024-05-15T23:09:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.840710+0000", 2024-05-15T23:09:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0442999999999999e-05 2024-05-15T23:09:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.840745+0000", 2024-05-15T23:09:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5141000000000001e-05 2024-05-15T23:09:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:175336 2.c 2:3015f4d3:::benchmark_data_smithi090_78160_object13978:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-05-15T23:09:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:07:46.944326+0000", 2024-05-15T23:09:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.009037844000005, 2024-05-15T23:09:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.047371288999999997, 2024-05-15T23:09:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 175336 2024-05-15T23:09:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.944326+0000", 2024-05-15T23:09:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.944326+0000", 2024-05-15T23:09:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.944328+0000", 2024-05-15T23:09:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9810000000000002e-06 2024-05-15T23:09:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.944334+0000", 2024-05-15T23:09:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8800000000000002e-06 2024-05-15T23:09:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.944347+0000", 2024-05-15T23:09:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2663000000000001e-05 2024-05-15T23:09:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.944356+0000", 2024-05-15T23:09:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1419999999999999e-06 2024-05-15T23:09:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.944405+0000", 2024-05-15T23:09:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9023999999999997e-05 2024-05-15T23:09:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.944491+0000", 2024-05-15T23:09:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5623000000000003e-05 2024-05-15T23:09:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:09:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.944598+0000", 2024-05-15T23:09:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106777 2024-05-15T23:09:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.951374+0000", 2024-05-15T23:09:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0067759639999999998 2024-05-15T23:09:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.991595+0000", 2024-05-15T23:09:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040221804 2024-05-15T23:09:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.991649+0000", 2024-05-15T23:09:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3599e-05 2024-05-15T23:09:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.991697+0000", 2024-05-15T23:09:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7831999999999999e-05 2024-05-15T23:09:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:175337 2.c 2:348f1fb9:::benchmark_data_smithi090_78160_object13979:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-05-15T23:09:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:07:46.945241+0000", 2024-05-15T23:09:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.008121927000005, 2024-05-15T23:09:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046619573999999997, 2024-05-15T23:09:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 175337 2024-05-15T23:09:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.945241+0000", 2024-05-15T23:09:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.945241+0000", 2024-05-15T23:09:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.945243+0000", 2024-05-15T23:09:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2720000000000001e-06 2024-05-15T23:09:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.945247+0000", 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4020000000000004e-06 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.945257+0000", 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7769999999999997e-06 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.945264+0000", 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8210000000000004e-06 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.948319+0000", 2024-05-15T23:09:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030550989999999999 2024-05-15T23:09:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.948401+0000", 2024-05-15T23:09:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2322000000000002e-05 2024-05-15T23:09:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:09:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.948484+0000", 2024-05-15T23:09:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2484000000000005e-05 2024-05-15T23:09:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.951502+0000", 2024-05-15T23:09:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030179199999999999 2024-05-15T23:09:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.991758+0000", 2024-05-15T23:09:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040256187999999998 2024-05-15T23:09:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.991810+0000", 2024-05-15T23:09:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2772999999999998e-05 2024-05-15T23:09:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:46.991861+0000", 2024-05-15T23:09:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0516000000000002e-05 2024-05-15T23:09:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:190528 2.c 2:3b9214c8:::benchmark_data_smithi090_78160_object29170:head [delete] snapc 0=[] ondisk+write+known_if_redirected e148)", 2024-05-15T23:09:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:00.113475+0000", 2024-05-15T23:09:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.839887989000005, 2024-05-15T23:09:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.047261873000000003, 2024-05-15T23:09:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 190528 2024-05-15T23:09:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.113475+0000", 2024-05-15T23:09:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.113475+0000", 2024-05-15T23:09:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:06.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.113478+0000", 2024-05-15T23:09:06.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6249999999999999e-06 2024-05-15T23:09:06.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:06.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.113485+0000", 2024-05-15T23:09:06.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4800000000000004e-06 2024-05-15T23:09:06.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:06.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.113504+0000", 2024-05-15T23:09:06.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8584e-05 2024-05-15T23:09:06.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:06.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.113513+0000", 2024-05-15T23:09:06.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9460000000000004e-06 2024-05-15T23:09:06.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:06.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.113574+0000", 2024-05-15T23:09:06.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1303999999999994e-05 2024-05-15T23:09:06.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:06.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.113697+0000", 2024-05-15T23:09:06.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012255600000000001 2024-05-15T23:09:06.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:09:06.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.113815+0000", 2024-05-15T23:09:06.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011766500000000001 2024-05-15T23:09:06.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:06.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.122906+0000", 2024-05-15T23:09:06.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090917389999999997 2024-05-15T23:09:06.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:06.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.160644+0000", 2024-05-15T23:09:06.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037737791 2024-05-15T23:09:06.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:06.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.160694+0000", 2024-05-15T23:09:06.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0077000000000002e-05 2024-05-15T23:09:06.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:06.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.160737+0000", 2024-05-15T23:09:06.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3105999999999997e-05 2024-05-15T23:09:06.042 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.042 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:06.042 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.042 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:216090 2.c 2:3c166bc5:::benchmark_data_smithi090_78160_object54732:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:09:06.042 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:22.345479+0000", 2024-05-15T23:09:06.042 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.607884538, 2024-05-15T23:09:06.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.053230103000000001, 2024-05-15T23:09:06.042 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:06.042 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:06.043 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:06.043 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:06.043 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:06.043 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:06.043 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 216090 2024-05-15T23:09:06.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.043 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:06.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:06.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.345479+0000", 2024-05-15T23:09:06.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:06.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.345479+0000", 2024-05-15T23:09:06.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:06.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.345481+0000", 2024-05-15T23:09:06.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1840000000000002e-06 2024-05-15T23:09:06.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:06.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.345489+0000", 2024-05-15T23:09:06.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8110000000000003e-06 2024-05-15T23:09:06.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:06.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.345501+0000", 2024-05-15T23:09:06.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.226e-05 2024-05-15T23:09:06.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:06.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.345510+0000", 2024-05-15T23:09:06.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6149999999999997e-06 2024-05-15T23:09:06.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:06.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.345562+0000", 2024-05-15T23:09:06.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2485999999999999e-05 2024-05-15T23:09:06.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:06.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.345694+0000", 2024-05-15T23:09:06.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206099999999999 2024-05-15T23:09:06.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:09:06.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.345787+0000", 2024-05-15T23:09:06.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3084000000000005e-05 2024-05-15T23:09:06.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:06.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.354573+0000", 2024-05-15T23:09:06.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087860230000000004 2024-05-15T23:09:06.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:06.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.398651+0000", 2024-05-15T23:09:06.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.044077301999999999 2024-05-15T23:09:06.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:06.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.398682+0000", 2024-05-15T23:09:06.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1077000000000001e-05 2024-05-15T23:09:06.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:06.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.398709+0000", 2024-05-15T23:09:06.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.72e-05 2024-05-15T23:09:06.048 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.048 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:06.048 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.048 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:216092 2.c 2:39cd22f5:::benchmark_data_smithi090_78160_object54734:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:09:06.049 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:22.345529+0000", 2024-05-15T23:09:06.049 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.607834271999998, 2024-05-15T23:09:06.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.053306727999999998, 2024-05-15T23:09:06.049 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:06.049 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:06.049 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:06.049 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:06.049 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:06.049 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:06.049 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 216092 2024-05-15T23:09:06.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.049 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:06.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:06.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.345529+0000", 2024-05-15T23:09:06.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:06.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.345529+0000", 2024-05-15T23:09:06.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:06.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.345530+0000", 2024-05-15T23:09:06.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4700000000000001e-07 2024-05-15T23:09:06.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:06.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.345533+0000", 2024-05-15T23:09:06.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.631e-06 2024-05-15T23:09:06.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:06.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.345539+0000", 2024-05-15T23:09:06.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5010000000000004e-06 2024-05-15T23:09:06.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:06.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.345543+0000", 2024-05-15T23:09:06.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9509999999999999e-06 2024-05-15T23:09:06.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:06.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.349530+0000", 2024-05-15T23:09:06.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039872029999999999 2024-05-15T23:09:06.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:06.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.349651+0000", 2024-05-15T23:09:06.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012118899999999999 2024-05-15T23:09:06.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:09:06.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.349767+0000", 2024-05-15T23:09:06.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115313 2024-05-15T23:09:06.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:06.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.354740+0000", 2024-05-15T23:09:06.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049738669999999999 2024-05-15T23:09:06.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:06.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.398750+0000", 2024-05-15T23:09:06.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.044009333999999997 2024-05-15T23:09:06.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:06.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.398798+0000", 2024-05-15T23:09:06.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8641999999999999e-05 2024-05-15T23:09:06.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:06.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:22.398836+0000", 2024-05-15T23:09:06.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.735e-05 2024-05-15T23:09:06.055 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.055 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:06.055 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.055 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:246081 2.c 2:3923cb78:::benchmark_data_smithi090_78160_object84723:head [delete] snapc 0=[] ondisk+write+known_if_redirected e157)", 2024-05-15T23:09:06.055 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:50.362683+0000", 2024-05-15T23:09:06.055 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.590680624999999, 2024-05-15T23:09:06.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062152818999999998, 2024-05-15T23:09:06.056 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:06.056 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:06.056 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:06.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:06.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:06.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:06.056 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 246081 2024-05-15T23:09:06.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.056 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:06.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:06.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.362683+0000", 2024-05-15T23:09:06.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:06.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.362683+0000", 2024-05-15T23:09:06.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:06.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.362684+0000", 2024-05-15T23:09:06.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.559e-06 2024-05-15T23:09:06.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:06.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.362690+0000", 2024-05-15T23:09:06.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4110000000000004e-06 2024-05-15T23:09:06.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:06.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.362700+0000", 2024-05-15T23:09:06.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2024-05-15T23:09:06.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:06.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.362707+0000", 2024-05-15T23:09:06.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3130000000000001e-06 2024-05-15T23:09:06.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:06.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.367390+0000", 2024-05-15T23:09:06.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046826790000000004 2024-05-15T23:09:06.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:06.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.367465+0000", 2024-05-15T23:09:06.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5594999999999997e-05 2024-05-15T23:09:06.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:09:06.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.367533+0000", 2024-05-15T23:09:06.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8109999999999997e-05 2024-05-15T23:09:06.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:06.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.375050+0000", 2024-05-15T23:09:06.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075167970000000004 2024-05-15T23:09:06.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:06.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.424729+0000", 2024-05-15T23:09:06.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049678792999999999 2024-05-15T23:09:06.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:06.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.424788+0000", 2024-05-15T23:09:06.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9012999999999999e-05 2024-05-15T23:09:06.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:06.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.424836+0000", 2024-05-15T23:09:06.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7559000000000003e-05 2024-05-15T23:09:06.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.061 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:06.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.062 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:246085 2.c 2:3d2325c3:::benchmark_data_smithi090_78160_object84727:head [delete] snapc 0=[] ondisk+write+known_if_redirected e157)", 2024-05-15T23:09:06.062 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:50.371080+0000", 2024-05-15T23:09:06.062 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.582283168, 2024-05-15T23:09:06.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.053875961, 2024-05-15T23:09:06.062 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:09:06.062 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:09:06.062 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:06.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:09:06.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:09:06.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:06.062 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 246085 2024-05-15T23:09:06.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.063 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:09:06.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:09:06.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.371080+0000", 2024-05-15T23:09:06.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:09:06.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.371080+0000", 2024-05-15T23:09:06.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:09:06.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:09:06.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.371082+0000", 2024-05-15T23:09:06.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0980000000000001e-06 2024-05-15T23:09:06.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:09:06.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.371088+0000", 2024-05-15T23:09:06.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5679999999999999e-06 2024-05-15T23:09:06.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:09:06.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.371100+0000", 2024-05-15T23:09:06.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2366e-05 2024-05-15T23:09:06.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:09:06.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.371108+0000", 2024-05-15T23:09:06.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2260000000000005e-06 2024-05-15T23:09:06.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:09:06.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.371313+0000", 2024-05-15T23:09:06.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020469499999999999 2024-05-15T23:09:06.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:09:06.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.371424+0000", 2024-05-15T23:09:06.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011066200000000001 2024-05-15T23:09:06.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:09:06.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.371526+0000", 2024-05-15T23:09:06.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102376 2024-05-15T23:09:06.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:09:06.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.375218+0000", 2024-05-15T23:09:06.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003692112 2024-05-15T23:09:06.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:06.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.424880+0000", 2024-05-15T23:09:06.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049661504000000002 2024-05-15T23:09:06.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:09:06.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.424922+0000", 2024-05-15T23:09:06.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2256999999999999e-05 2024-05-15T23:09:06.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:09:06.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:09:06.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:09:06.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:50.424956+0000", 2024-05-15T23:09:06.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4097000000000002e-05 2024-05-15T23:09:06.068 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.068 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:06.068 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.068 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:09:06.068 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:09:06.068 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:09:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:05 smithi084 ceph-mon[39784]: pgmap v496: 17 pgs: 1 peering, 1 unknown, 15 active+clean; 4.0 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-05-15T23:09:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:05 smithi084 ceph-mon[39784]: osdmap e164: 12 total, 12 up, 11 in 2024-05-15T23:09:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:05 smithi090 ceph-mon[44434]: pgmap v496: 17 pgs: 1 peering, 1 unknown, 15 active+clean; 4.0 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-05-15T23:09:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:05 smithi090 ceph-mon[44434]: osdmap e164: 12 total, 12 up, 11 in 2024-05-15T23:09:06.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:05 smithi088 ceph-mon[44699]: pgmap v496: 17 pgs: 1 peering, 1 unknown, 15 active+clean; 4.0 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-05-15T23:09:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:05 smithi088 ceph-mon[44699]: osdmap e164: 12 total, 12 up, 11 in 2024-05-15T23:09:06.709 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T23:09:06.784 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:09:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:06.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:09:06.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:09:06.931 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:06.963 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:09:06.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:06.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:09:07.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:07.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:06 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:09:07.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:06 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:09:07.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:06 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:09:07.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:06 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:09:07.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:06 smithi084 ceph-mon[39784]: 2.d starting backfill to osd.3 from (0'0,0'0] MAX to 162'16097 2024-05-15T23:09:07.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:06 smithi084 ceph-mon[39784]: 2.8 starting backfill to osd.0 from (0'0,0'0] MAX to 162'16358 2024-05-15T23:09:07.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:06 smithi084 ceph-mon[39784]: osdmap e165: 12 total, 12 up, 11 in 2024-05-15T23:09:07.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:06 smithi084 ceph-mon[39784]: 2.f starting backfill to osd.4 from (0'0,0'0] MAX to 162'16420 2024-05-15T23:09:07.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:06 smithi084 ceph-mon[39784]: 2.7 starting backfill to osd.6 from (0'0,0'0] MAX to 162'16279 2024-05-15T23:09:07.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:06 smithi084 ceph-mon[39784]: 2.7 starting backfill to osd.10 from (0'0,0'0] MAX to 162'16279 2024-05-15T23:09:07.113 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:07.148 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:07.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:07.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:07.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:09:07.302 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:06 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:09:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:06 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:09:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:06 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:09:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:06 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:09:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:06 smithi090 ceph-mon[44434]: 2.d starting backfill to osd.3 from (0'0,0'0] MAX to 162'16097 2024-05-15T23:09:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:06 smithi090 ceph-mon[44434]: 2.8 starting backfill to osd.0 from (0'0,0'0] MAX to 162'16358 2024-05-15T23:09:07.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:06 smithi090 ceph-mon[44434]: osdmap e165: 12 total, 12 up, 11 in 2024-05-15T23:09:07.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:06 smithi090 ceph-mon[44434]: 2.f starting backfill to osd.4 from (0'0,0'0] MAX to 162'16420 2024-05-15T23:09:07.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:06 smithi090 ceph-mon[44434]: 2.7 starting backfill to osd.6 from (0'0,0'0] MAX to 162'16279 2024-05-15T23:09:07.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:06 smithi090 ceph-mon[44434]: 2.7 starting backfill to osd.10 from (0'0,0'0] MAX to 162'16279 2024-05-15T23:09:07.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:06 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:09:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:06 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:09:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:06 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:09:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:06 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:09:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:06 smithi088 ceph-mon[44699]: 2.d starting backfill to osd.3 from (0'0,0'0] MAX to 162'16097 2024-05-15T23:09:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:06 smithi088 ceph-mon[44699]: 2.8 starting backfill to osd.0 from (0'0,0'0] MAX to 162'16358 2024-05-15T23:09:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:06 smithi088 ceph-mon[44699]: osdmap e165: 12 total, 12 up, 11 in 2024-05-15T23:09:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:06 smithi088 ceph-mon[44699]: 2.f starting backfill to osd.4 from (0'0,0'0] MAX to 162'16420 2024-05-15T23:09:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:06 smithi088 ceph-mon[44699]: 2.7 starting backfill to osd.6 from (0'0,0'0] MAX to 162'16279 2024-05-15T23:09:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:06 smithi088 ceph-mon[44699]: 2.7 starting backfill to osd.10 from (0'0,0'0] MAX to 162'16279 2024-05-15T23:09:07.412 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:09:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:07.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:09:07.491 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:07.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:09:07.556 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:07.579 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:09:07.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:07.582 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:09:07.683 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:09:07.714 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:07.738 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:09:07.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:07.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:09:07.843 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:09:07.873 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:07.897 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:09:07.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:07.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:09:08.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:08.005 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:08 smithi088 ceph-mon[44699]: pgmap v499: 17 pgs: 2 peering, 1 unknown, 14 active+clean; 3.9 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 842 op/s 2024-05-15T23:09:08.005 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:08 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:09:08.013 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:08 smithi090 ceph-mon[44434]: pgmap v499: 17 pgs: 2 peering, 1 unknown, 14 active+clean; 3.9 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 842 op/s 2024-05-15T23:09:08.013 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:08 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:09:08.013 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:08 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:09:08.013 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/349377341' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:09:08.013 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:08 smithi090 ceph-mon[44434]: from='client.35033 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:09:08.026 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:09:08.053 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:08.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:08 smithi084 ceph-mon[39784]: pgmap v499: 17 pgs: 2 peering, 1 unknown, 14 active+clean; 3.9 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 842 op/s 2024-05-15T23:09:08.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:08 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:09:08.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:08 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:09:08.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/349377341' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:09:08.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:08 smithi084 ceph-mon[39784]: from='client.35033 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:09:08.077 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:08.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:08.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:08.182 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:09:08.234 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:08.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:08 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:09:08.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/349377341' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:09:08.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:08 smithi088 ceph-mon[44699]: from='client.35033 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:09:08.373 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:09:08.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:08.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:09:08.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:08.536 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:08.568 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:08.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:08.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:08.681 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:09:08.722 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:08.822 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:09:09.202 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:09 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:09:09.202 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:09 smithi084 ceph-mon[39784]: from='client.35033 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:09:09.203 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:09 smithi084 ceph-mon[39784]: osdmap e166: 12 total, 12 up, 11 in 2024-05-15T23:09:09.204 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:09:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:09.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:09:09.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:09 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:09:09.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:09 smithi090 ceph-mon[44434]: from='client.35033 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:09:09.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:09 smithi090 ceph-mon[44434]: osdmap e166: 12 total, 12 up, 11 in 2024-05-15T23:09:09.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:09:09.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:09 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:09:09.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:09 smithi088 ceph-mon[44699]: from='client.35033 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:09:09.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:09 smithi088 ceph-mon[44699]: osdmap e166: 12 total, 12 up, 11 in 2024-05-15T23:09:09.352 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:09.385 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:09:09.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:09.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:09:09.490 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:09:09.534 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:09.671 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:09:09.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:09.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:09:09.727 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:09.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:09.818 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:09.858 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:09.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:09.964 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:09:09.995 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:10.023 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:09:10.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:10.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:09:10.030 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:10 smithi084 ceph-mon[39784]: pgmap v501: 17 pgs: 2 peering, 1 unknown, 14 active+clean; 3.8 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 797 op/s 2024-05-15T23:09:10.126 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:09:10.156 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:10.179 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:09:10.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:10.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:09:10.219 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11] out_osds: [9] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:09:10.219 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:09:10.219 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-05-15T23:09:10.220 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T23:09:10.220 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T23:09:10.220 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-05-15T23:09:10.259 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:10 smithi088 ceph-mon[44699]: pgmap v501: 17 pgs: 2 peering, 1 unknown, 14 active+clean; 3.8 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 797 op/s 2024-05-15T23:09:10.282 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:09:10.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:10 smithi090 ceph-mon[44434]: pgmap v501: 17 pgs: 2 peering, 1 unknown, 14 active+clean; 3.8 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 797 op/s 2024-05-15T23:09:10.347 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:10.379 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:09:10.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:10.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:09:10.489 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:09:10.552 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:10.583 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:09:10.618 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:09:10.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:10.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:09:10.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:09:10.763 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:10.801 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:09:10.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:10.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:09:10.905 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:09:10.936 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:10.962 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:09:10.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:10.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:09:11.066 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:09:11.124 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:11.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:09:11.565 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:09:11.566 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:09:11.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.566 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:264571 2.5 2:af102ea3:::benchmark_data_smithi090_78160_object103213:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-05-15T23:09:11.566 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:09:10.780381+0000", 2024-05-15T23:09:11.566 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.010071304999999999, 2024-05-15T23:09:11.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010104999, 2024-05-15T23:09:11.566 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:11.566 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:11.566 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:09:11.566 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:11.566 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:11.566 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:11.567 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 264571 2024-05-15T23:09:11.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.567 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:11.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:11.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.780381+0000", 2024-05-15T23:09:11.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:11.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:11.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.780381+0000", 2024-05-15T23:09:11.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:11.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:11.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.780383+0000", 2024-05-15T23:09:11.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1749999999999999e-06 2024-05-15T23:09:11.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:11.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.780401+0000", 2024-05-15T23:09:11.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8000999999999999e-05 2024-05-15T23:09:11.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:11.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.780413+0000", 2024-05-15T23:09:11.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1775e-05 2024-05-15T23:09:11.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:11.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.780421+0000", 2024-05-15T23:09:11.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1650000000000006e-06 2024-05-15T23:09:11.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:11.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.780477+0000", 2024-05-15T23:09:11.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6428000000000003e-05 2024-05-15T23:09:11.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:11.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.780610+0000", 2024-05-15T23:09:11.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013293100000000001 2024-05-15T23:09:11.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 8", 2024-05-15T23:09:11.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.780732+0000", 2024-05-15T23:09:11.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121907 2024-05-15T23:09:11.570 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:11.570 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:11.570 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:11.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.570 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:264580 2.5 2:a40d9ca0:::benchmark_data_smithi090_78160_object103222:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-05-15T23:09:11.571 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:09:10.783337+0000", 2024-05-15T23:09:11.571 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.0071147119999999996, 2024-05-15T23:09:11.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072357020000000001, 2024-05-15T23:09:11.571 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:11.571 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:11.571 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:09:11.571 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:11.571 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:11.571 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:11.571 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 264580 2024-05-15T23:09:11.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.571 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:11.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:11.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.783337+0000", 2024-05-15T23:09:11.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:11.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:11.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.783337+0000", 2024-05-15T23:09:11.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:11.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:11.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.783339+0000", 2024-05-15T23:09:11.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8959999999999999e-06 2024-05-15T23:09:11.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:11.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.783345+0000", 2024-05-15T23:09:11.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5010000000000004e-06 2024-05-15T23:09:11.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:11.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.783356+0000", 2024-05-15T23:09:11.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1124000000000001e-05 2024-05-15T23:09:11.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:11.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.783365+0000", 2024-05-15T23:09:11.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5990000000000002e-06 2024-05-15T23:09:11.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:11.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.785312+0000", 2024-05-15T23:09:11.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019478849999999999 2024-05-15T23:09:11.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:11.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.785421+0000", 2024-05-15T23:09:11.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000108813 2024-05-15T23:09:11.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 8", 2024-05-15T23:09:11.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.785524+0000", 2024-05-15T23:09:11.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102484 2024-05-15T23:09:11.574 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:11.574 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:264572 2.c e166/102 2:36210dd2:::benchmark_data_smithi090_78160_object103214:head v 166'16342, mlcod=166'16339)", 2024-05-15T23:09:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:09:10.785823+0000", 2024-05-15T23:09:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.0046295859999999998, 2024-05-15T23:09:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048300890000000001, 2024-05-15T23:09:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-05-15T23:09:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.785823+0000", 2024-05-15T23:09:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.785823+0000", 2024-05-15T23:09:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.785823+0000", 2024-05-15T23:09:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9899999999999999e-07 2024-05-15T23:09:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.785829+0000", 2024-05-15T23:09:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7529999999999998e-06 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.785837+0000", 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9189999999999999e-06 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.785842+0000", 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8779999999999997e-06 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.786135+0000", 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000293319 2024-05-15T23:09:11.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:11.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.786151+0000", 2024-05-15T23:09:11.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6277e-05 2024-05-15T23:09:11.578 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:11.578 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:11.578 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:11.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.578 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24745.0:264574 2.c e166/102)", 2024-05-15T23:09:11.578 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:09:10.786196+0000", 2024-05-15T23:09:11.578 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.0042557469999999998, 2024-05-15T23:09:11.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045176829999999998, 2024-05-15T23:09:11.578 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:11.579 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:11.579 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:09:11.579 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:11.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:11.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.786196+0000", 2024-05-15T23:09:11.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:11.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:11.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.786196+0000", 2024-05-15T23:09:11.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:11.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:11.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.786197+0000", 2024-05-15T23:09:11.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5000000000000001e-07 2024-05-15T23:09:11.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:11.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.786202+0000", 2024-05-15T23:09:11.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.994e-06 2024-05-15T23:09:11.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:11.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.786210+0000", 2024-05-15T23:09:11.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.576e-06 2024-05-15T23:09:11.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:11.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:11.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:11.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:10.786215+0000", 2024-05-15T23:09:11.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4310000000000002e-06 2024-05-15T23:09:11.581 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:11.581 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:11.581 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:11.581 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:11.581 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:09:11.581 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 4 2024-05-15T23:09:11.581 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:09:12.048 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:12 smithi088 ceph-mon[44699]: pgmap v502: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 3.6 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 931 op/s; 16254/119586 objects misplaced (13.592%); 24 MiB/s, 380 objects/s recovering 2024-05-15T23:09:12.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:12 smithi084 ceph-mon[39784]: pgmap v502: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 3.6 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 931 op/s; 16254/119586 objects misplaced (13.592%); 24 MiB/s, 380 objects/s recovering 2024-05-15T23:09:12.205 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:09:12.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:12.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:09:12.239 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T23:09:12.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:12 smithi090 ceph-mon[44434]: pgmap v502: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 3.6 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 931 op/s; 16254/119586 objects misplaced (13.592%); 24 MiB/s, 380 objects/s recovering 2024-05-15T23:09:12.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:09:12.349 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:12.383 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:09:12.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:12.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:09:12.490 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:09:12.580 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:12.745 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:09:12.745 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-15T23:09:12.745 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:09:12.851 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:09:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:12.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:09:12.926 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:12.937 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:12.965 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:09:12.966 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:09:13.020 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:09:13.023 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:13.066 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:09:13.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1817977385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:09:13.128 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:09:13.176 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:09:13.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:09:13.276 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:09:13.315 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:09:13.354 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:09:13.402 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:09:13.423 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1817977385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:09:13.448 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:09:13.484 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:09:13.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:09:13.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:09:13.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:09:13.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:09:13.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:09:13.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:09:13.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:09:13.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:09:13.489 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:09:13.489 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:09:13.489 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:09:13.489 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:09:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1817977385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:09:13.923 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:09:13.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:13.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:09:14.032 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:09:14.070 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:14.095 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:09:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:14.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:09:14.199 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:09:14.234 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:14.260 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:09:14.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:14.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:09:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:14 smithi088 ceph-mon[44699]: pgmap v503: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 3.6 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 694 op/s; 16254/117248 objects misplaced (13.863%); 18 MiB/s, 295 objects/s recovering 2024-05-15T23:09:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.359 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:14 smithi084 ceph-mon[39784]: pgmap v503: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 3.6 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 694 op/s; 16254/117248 objects misplaced (13.863%); 18 MiB/s, 295 objects/s recovering 2024-05-15T23:09:14.359 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.359 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.359 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.359 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.359 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.359 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.359 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.360 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.360 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:09:14.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:14 smithi090 ceph-mon[44434]: pgmap v503: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 3.6 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 694 op/s; 16254/117248 objects misplaced (13.863%); 18 MiB/s, 295 objects/s recovering 2024-05-15T23:09:14.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:14.399 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:14.426 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:09:14.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:14.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:09:14.535 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:09:14.564 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:14.590 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:09:14.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:14.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:09:14.696 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:09:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:14.750 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:09:14.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:14.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:09:14.857 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:09:14.891 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:14.917 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:09:14.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:14.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:09:15.021 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:09:15.077 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:15.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:15.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:15.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:15.207 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:09:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:15.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:09:15.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:09:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:15.365 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:15.404 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:09:15.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:15.409 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:09:15.409 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:15.409 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:15.409 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:15.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:09:15.555 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:15.615 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:09:15.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:15.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:09:15.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:09:15.831 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:15.870 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:09:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:15.876 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:09:15.976 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:09:16.017 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:16.045 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:09:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:16.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:09:16.120 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:16 smithi090 ceph-mon[44434]: pgmap v504: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 3.5 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 12059/114134 objects misplaced (10.566%); 36 MiB/s, 575 objects/s recovering 2024-05-15T23:09:16.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:09:16.220 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:16.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:16 smithi084 ceph-mon[39784]: pgmap v504: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 3.5 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 12059/114134 objects misplaced (10.566%); 36 MiB/s, 575 objects/s recovering 2024-05-15T23:09:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:16 smithi088 ceph-mon[44699]: pgmap v504: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 3.5 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 12059/114134 objects misplaced (10.566%); 36 MiB/s, 575 objects/s recovering 2024-05-15T23:09:17.143 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:09:17.231 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:09:17.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:17 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3852132668' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:09:17.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:17 smithi084 ceph-mon[39784]: from='client.35045 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:09:17.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:17 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3852132668' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:09:17.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:17 smithi090 ceph-mon[44434]: from='client.35045 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:09:17.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:17 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3852132668' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:09:17.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:17 smithi088 ceph-mon[44699]: from='client.35045 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:09:18.451 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:09:18.451 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:09:18.458 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11] out_osds: [9] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:09:18.459 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:09:18.459 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:09:18.490 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:18 smithi084 ceph-mon[39784]: pgmap v505: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 3.3 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 877 op/s; 11194/108800 objects misplaced (10.289%); 35 MiB/s, 564 objects/s recovering 2024-05-15T23:09:18.490 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:18 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:09:18.490 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:18 smithi084 ceph-mon[39784]: from='client.35045 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:09:18.490 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:18 smithi084 ceph-mon[39784]: osdmap e167: 12 total, 12 up, 11 in 2024-05-15T23:09:18.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:18 smithi090 ceph-mon[44434]: pgmap v505: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 3.3 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 877 op/s; 11194/108800 objects misplaced (10.289%); 35 MiB/s, 564 objects/s recovering 2024-05-15T23:09:18.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:18 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:09:18.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:18 smithi090 ceph-mon[44434]: from='client.35045 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:09:18.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:18 smithi090 ceph-mon[44434]: osdmap e167: 12 total, 12 up, 11 in 2024-05-15T23:09:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:18 smithi088 ceph-mon[44699]: pgmap v505: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 3.3 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 877 op/s; 11194/108800 objects misplaced (10.289%); 35 MiB/s, 564 objects/s recovering 2024-05-15T23:09:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:18 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:09:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:18 smithi088 ceph-mon[44699]: from='client.35045 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:09:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:18 smithi088 ceph-mon[44699]: osdmap e167: 12 total, 12 up, 11 in 2024-05-15T23:09:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:19 smithi090 ceph-mon[44434]: osdmap e168: 12 total, 12 up, 11 in 2024-05-15T23:09:19.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:19 smithi088 ceph-mon[44699]: osdmap e168: 12 total, 12 up, 11 in 2024-05-15T23:09:19.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:19 smithi084 ceph-mon[39784]: osdmap e168: 12 total, 12 up, 11 in 2024-05-15T23:09:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:20 smithi090 ceph-mon[44434]: pgmap v508: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 3.3 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 11194/107620 objects misplaced (10.401%); 26 MiB/s, 419 objects/s recovering 2024-05-15T23:09:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:20 smithi090 ceph-mon[44434]: osdmap e169: 12 total, 12 up, 11 in 2024-05-15T23:09:20.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:20 smithi088 ceph-mon[44699]: pgmap v508: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 3.3 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 11194/107620 objects misplaced (10.401%); 26 MiB/s, 419 objects/s recovering 2024-05-15T23:09:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:20 smithi088 ceph-mon[44699]: osdmap e169: 12 total, 12 up, 11 in 2024-05-15T23:09:20.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:20 smithi084 ceph-mon[39784]: pgmap v508: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 3.3 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 11194/107620 objects misplaced (10.401%); 26 MiB/s, 419 objects/s recovering 2024-05-15T23:09:20.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:20 smithi084 ceph-mon[39784]: osdmap e169: 12 total, 12 up, 11 in 2024-05-15T23:09:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:22 smithi090 ceph-mon[44434]: pgmap v510: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 3.1 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s; 5777/101356 objects misplaced (5.700%); 47 MiB/s, 746 objects/s recovering 2024-05-15T23:09:22.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:22 smithi088 ceph-mon[44699]: pgmap v510: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 3.1 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s; 5777/101356 objects misplaced (5.700%); 47 MiB/s, 746 objects/s recovering 2024-05-15T23:09:22.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:22 smithi084 ceph-mon[39784]: pgmap v510: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 3.1 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s; 5777/101356 objects misplaced (5.700%); 47 MiB/s, 746 objects/s recovering 2024-05-15T23:09:22.925 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:09:22.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:22.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:09:23.030 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:09:23.101 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:24.058 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:09:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:24.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:09:24.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:24.215 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:24.239 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:24.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:24.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:24.349 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:09:24.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:24 smithi084 ceph-mon[39784]: pgmap v511: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 3.0 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 797 op/s; 5777/99218 objects misplaced (5.823%); 40 MiB/s, 641 objects/s recovering 2024-05-15T23:09:24.382 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:24.406 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:09:24.408 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:24 smithi088 ceph-mon[44699]: pgmap v511: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 3.0 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 797 op/s; 5777/99218 objects misplaced (5.823%); 40 MiB/s, 641 objects/s recovering 2024-05-15T23:09:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:24.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:09:24.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T23:09:24.482 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:24.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:24.541 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:24 smithi090 ceph-mon[44434]: pgmap v511: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 3.0 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 797 op/s; 5777/99218 objects misplaced (5.823%); 40 MiB/s, 641 objects/s recovering 2024-05-15T23:09:24.548 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:24.556 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:24.568 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:24.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:24.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:24.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:09:24.738 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:24.778 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:09:24.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:24.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:09:24.885 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:09:24.913 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:24.935 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:09:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:24.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:09:25.039 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:09:25.077 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:25.099 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:09:25.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:25.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:09:25.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:09:25.251 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:25.280 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:09:25.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:25.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:09:25.387 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:09:25.416 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:25.446 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:09:25.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:25.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:09:25.550 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:09:25.580 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:25.601 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:09:25.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:25.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:09:25.709 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:09:25.755 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:26.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:09:26.341 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:09:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:26.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:09:26.355 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:26 smithi084 ceph-mon[39784]: pgmap v512: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 2.9 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 767 op/s; 3580/95980 objects misplaced (3.730%); 42 MiB/s, 678 objects/s recovering 2024-05-15T23:09:26.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:26.492 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:26 smithi090 ceph-mon[44434]: pgmap v512: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 2.9 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 767 op/s; 3580/95980 objects misplaced (3.730%); 42 MiB/s, 678 objects/s recovering 2024-05-15T23:09:26.498 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:26.525 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:26.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:26.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:26.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:26 smithi088 ceph-mon[44699]: pgmap v512: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 2.9 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 767 op/s; 3580/95980 objects misplaced (3.730%); 42 MiB/s, 678 objects/s recovering 2024-05-15T23:09:26.629 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:09:26.693 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:27.372 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:09:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:27.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:09:27.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:09:27.518 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:27.647 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:09:27.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:27.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:09:27.752 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:09:27.784 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:28.149 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:09:28.149 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:09:28.149 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:09:28.150 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:09:28.150 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:09:28.150 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:09:28.150 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:09:28.150 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:09:28.150 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:09:28.150 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:09:28.156 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:28.168 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:09:28.189 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:09:28.190 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:09:28.190 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:09:28.190 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:09:28.190 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:09:28.191 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:09:28.191 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:09:28.191 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:09:28.191 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:09:28.191 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:09:28.191 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:09:28.191 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:09:28.191 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:09:28.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:28 smithi084 ceph-mon[39784]: pgmap v513: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 2.8 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s; 2864/90298 objects misplaced (3.172%); 42 MiB/s, 670 objects/s recovering 2024-05-15T23:09:28.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:28 smithi084 ceph-mon[39784]: osdmap e170: 12 total, 12 up, 11 in 2024-05-15T23:09:28.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3703209992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:09:28.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:28.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:28.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:28.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:28 smithi090 ceph-mon[44434]: pgmap v513: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 2.8 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s; 2864/90298 objects misplaced (3.172%); 42 MiB/s, 670 objects/s recovering 2024-05-15T23:09:28.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:28 smithi090 ceph-mon[44434]: osdmap e170: 12 total, 12 up, 11 in 2024-05-15T23:09:28.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3703209992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:09:28.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:28.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:28.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:28.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:28 smithi088 ceph-mon[44699]: pgmap v513: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 2.8 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s; 2864/90298 objects misplaced (3.172%); 42 MiB/s, 670 objects/s recovering 2024-05-15T23:09:28.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:28 smithi088 ceph-mon[44699]: osdmap e170: 12 total, 12 up, 11 in 2024-05-15T23:09:28.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3703209992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:09:28.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:28.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:28.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:28.604 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:28.727 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:09:28.728 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:09:28.744 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:09:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:28.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:09:28.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:09:28.879 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:29.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:29 smithi090 ceph-mon[44434]: osdmap e171: 12 total, 12 up, 11 in 2024-05-15T23:09:29.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:29 smithi088 ceph-mon[44699]: osdmap e171: 12 total, 12 up, 11 in 2024-05-15T23:09:29.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:29.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:29 smithi084 ceph-mon[39784]: osdmap e171: 12 total, 12 up, 11 in 2024-05-15T23:09:30.007 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":516,"stamp":"2024-05-15T23:09:28.716316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2918253112,"num_objects":44523,"num_object_clones":0,"num_object_copies":89046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2864,"num_objects_unfound":0,"num_objects_dirty":44523,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":278314,"num_write_kb":10328871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12406,"num_bytes_recovered":814102136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160985,"ondisk_log_size":160985,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":6683364,"kb_used_data":5809676,"kb_used_omap":17,"kb_used_meta":873134,"kb_avail":1024513308,"statfs":{"total":1055945392128,"available":1049101627392,"internally_reserved":0,"allocated":5949108224,"data_stored":5945382744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17542,"internal_metadata":894090106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-403374080,"num_objects":-6155,"num_object_clones":0,"num_object_copies":-12310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2913,"num_objects_unfound":0,"num_objects_dirty":-6155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6155,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1892,"num_bytes_recovered":123994112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.011489"},"pg_stats":[{"pgid":"2.f","version":"169'17544","reported_seq":25036,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-05-15T23:09:24.479981+0000","last_change":"2024-05-15T23:09:19.218095+0000","last_active":"2024-05-15T23:09:24.479981+0000","last_peered":"2024-05-15T23:09:24.479981+0000","last_clean":"2024-05-15T23:09:24.479981+0000","last_became_active":"2024-05-15T23:09:19.155054+0000","last_became_peered":"2024-05-15T23:09:19.155054+0000","last_unstale":"2024-05-15T23:09:24.479981+0000","last_undegraded":"2024-05-15T23:09:24.479981+0000","last_fullsized":"2024-05-15T23:09:24.479981+0000","mapping_epoch":168,"log_start":"130'7500","ondisk_log_start":"130'7500","created":102,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'15444","last_scrub_stamp":"2024-05-15T23:08:48.803738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:08:48.803738+0000","objects_scrubbed":4997,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:09:50.754750+0000","scrub_duration":968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187432960,"num_objects":2860,"num_object_clones":0,"num_object_copies":5720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17544,"num_write_kb":652928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3593,"num_bytes_recovered":235405312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"169'17085","reported_seq":17371,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-05-15T23:09:24.790538+0000","last_change":"2024-05-15T23:08:46.395501+0000","last_active":"2024-05-15T23:09:24.790538+0000","last_peered":"2024-05-15T23:09:24.790538+0000","last_clean":"2024-05-15T23:09:24.790538+0000","last_became_active":"2024-05-15T23:05:51.957730+0000","last_became_peered":"2024-05-15T23:05:51.957730+0000","last_unstale":"2024-05-15T23:09:24.790538+0000","last_undegraded":"2024-05-15T23:09:24.790538+0000","last_fullsized":"2024-05-15T23:09:24.790538+0000","mapping_epoch":102,"log_start":"129'7000","ondisk_log_start":"129'7000","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'14877","last_scrub_stamp":"2024-05-15T23:08:46.395388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:08:46.395388+0000","objects_scrubbed":5092,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:10:01.092886+0000","scrub_duration":1499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185794560,"num_objects":2835,"num_object_clones":0,"num_object_copies":5670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17085,"num_write_kb":637440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"169'17267","reported_seq":17554,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-05-15T23:09:26.640240+0000","last_change":"2024-05-15T23:08:42.107403+0000","last_active":"2024-05-15T23:09:26.640240+0000","last_peered":"2024-05-15T23:09:26.640240+0000","last_clean":"2024-05-15T23:09:26.640240+0000","last_became_active":"2024-05-15T23:05:51.932108+0000","last_became_peered":"2024-05-15T23:05:51.932108+0000","last_unstale":"2024-05-15T23:09:26.640240+0000","last_undegraded":"2024-05-15T23:09:26.640240+0000","last_fullsized":"2024-05-15T23:09:26.640240+0000","mapping_epoch":102,"log_start":"129'7200","ondisk_log_start":"129'7200","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'14659","last_scrub_stamp":"2024-05-15T23:08:42.107302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:08:42.107302+0000","objects_scrubbed":5219,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:09:46.244372+0000","scrub_duration":962,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169934848,"num_objects":2593,"num_object_clones":0,"num_object_copies":5186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17267,"num_write_kb":635520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"169'17248","reported_seq":17563,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-05-15T23:09:25.507825+0000","last_change":"2024-05-15T23:08:06.525424+0000","last_active":"2024-05-15T23:09:25.507825+0000","last_peered":"2024-05-15T23:09:25.507825+0000","last_clean":"2024-05-15T23:09:25.507825+0000","last_became_active":"2024-05-15T23:05:51.957162+0000","last_became_peered":"2024-05-15T23:05:51.957162+0000","last_unstale":"2024-05-15T23:09:25.507825+0000","last_undegraded":"2024-05-15T23:09:25.507825+0000","last_fullsized":"2024-05-15T23:09:25.507825+0000","mapping_epoch":102,"log_start":"130'7200","ondisk_log_start":"130'7200","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'12260","last_scrub_stamp":"2024-05-15T23:08:06.525327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:08:06.525327+0000","objects_scrubbed":7857,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:09:34.660271+0000","scrub_duration":1788,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184025088,"num_objects":2808,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17248,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"169'17456","reported_seq":17759,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-05-15T23:09:26.595063+0000","last_change":"2024-05-15T23:08:39.921815+0000","last_active":"2024-05-15T23:09:26.595063+0000","last_peered":"2024-05-15T23:09:26.595063+0000","last_clean":"2024-05-15T23:09:26.595063+0000","last_became_active":"2024-05-15T23:05:51.931638+0000","last_became_peered":"2024-05-15T23:05:51.931638+0000","last_unstale":"2024-05-15T23:09:26.595063+0000","last_undegraded":"2024-05-15T23:09:26.595063+0000","last_fullsized":"2024-05-15T23:09:26.595063+0000","mapping_epoch":102,"log_start":"130'7400","ondisk_log_start":"130'7400","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'14689","last_scrub_stamp":"2024-05-15T23:08:39.921629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:08:39.921629+0000","objects_scrubbed":5477,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:09:45.771373+0000","scrub_duration":852,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":2688,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17456,"num_write_kb":644608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":307,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-05-15T23:09:19.145427+0000","last_change":"2024-05-15T23:07:55.785299+0000","last_active":"2024-05-15T23:09:19.145427+0000","last_peered":"2024-05-15T23:09:19.145427+0000","last_clean":"2024-05-15T23:09:19.145427+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:09:19.145427+0000","last_undegraded":"2024-05-15T23:09:19.145427+0000","last_fullsized":"2024-05-15T23:09:19.145427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:07:55.785205+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:07:55.785205+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"169'17535","reported_seq":17828,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-05-15T23:09:24.874453+0000","last_change":"2024-05-15T23:08:40.654687+0000","last_active":"2024-05-15T23:09:24.874453+0000","last_peered":"2024-05-15T23:09:24.874453+0000","last_clean":"2024-05-15T23:09:24.874453+0000","last_became_active":"2024-05-15T23:05:51.958648+0000","last_became_peered":"2024-05-15T23:05:51.958648+0000","last_unstale":"2024-05-15T23:09:24.874453+0000","last_undegraded":"2024-05-15T23:09:24.874453+0000","last_fullsized":"2024-05-15T23:09:24.874453+0000","mapping_epoch":102,"log_start":"130'7500","ondisk_log_start":"130'7500","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'14918","last_scrub_stamp":"2024-05-15T23:08:40.654513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:08:40.654513+0000","objects_scrubbed":5539,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:10:02.108390+0000","scrub_duration":745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189988864,"num_objects":2899,"num_object_clones":0,"num_object_copies":5798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17535,"num_write_kb":653888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"169'17361","reported_seq":17697,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-05-15T23:09:24.533568+0000","last_change":"2024-05-15T23:08:53.927728+0000","last_active":"2024-05-15T23:09:24.533568+0000","last_peered":"2024-05-15T23:09:24.533568+0000","last_clean":"2024-05-15T23:09:24.533568+0000","last_became_active":"2024-05-15T23:08:53.895071+0000","last_became_peered":"2024-05-15T23:08:53.895071+0000","last_unstale":"2024-05-15T23:09:24.533568+0000","last_undegraded":"2024-05-15T23:09:24.533568+0000","last_fullsized":"2024-05-15T23:09:24.533568+0000","mapping_epoch":158,"log_start":"130'7300","ondisk_log_start":"130'7300","created":102,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'14881","last_scrub_stamp":"2024-05-15T23:08:42.130095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:08:42.130095+0000","objects_scrubbed":5373,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:10:04.241968+0000","scrub_duration":1238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187236352,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17361,"num_write_kb":646976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"169'17422","reported_seq":17719,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-05-15T23:09:24.834753+0000","last_change":"2024-05-15T23:08:51.018413+0000","last_active":"2024-05-15T23:09:24.834753+0000","last_peered":"2024-05-15T23:09:24.834753+0000","last_clean":"2024-05-15T23:09:24.834753+0000","last_became_active":"2024-05-15T23:05:51.936955+0000","last_became_peered":"2024-05-15T23:05:51.936955+0000","last_unstale":"2024-05-15T23:09:24.834753+0000","last_undegraded":"2024-05-15T23:09:24.834753+0000","last_fullsized":"2024-05-15T23:09:24.834753+0000","mapping_epoch":102,"log_start":"130'7400","ondisk_log_start":"130'7400","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'15500","last_scrub_stamp":"2024-05-15T23:08:51.018274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:08:51.018274+0000","objects_scrubbed":4831,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:10:09.505018+0000","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187957248,"num_objects":2868,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17422,"num_write_kb":649280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"169'17424","reported_seq":27558,"reported_epoch":169,"state":"active+remapped+backfilling","last_fresh":"2024-05-15T23:09:24.549010+0000","last_change":"2024-05-15T23:09:06.142621+0000","last_active":"2024-05-15T23:09:24.549010+0000","last_peered":"2024-05-15T23:09:24.549010+0000","last_clean":"2024-05-15T23:09:03.962098+0000","last_became_active":"2024-05-15T23:09:06.089235+0000","last_became_peered":"2024-05-15T23:09:06.089235+0000","last_unstale":"2024-05-15T23:09:24.549010+0000","last_undegraded":"2024-05-15T23:09:24.549010+0000","last_fullsized":"2024-05-15T23:09:24.549010+0000","mapping_epoch":164,"log_start":"130'7400","ondisk_log_start":"130'7400","created":102,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'15087","last_scrub_stamp":"2024-05-15T23:08:45.275422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:08:45.275422+0000","objects_scrubbed":5250,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189136896,"num_objects":2886,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1942,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17424,"num_write_kb":649920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3274,"num_bytes_recovered":214564864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[0,9],"avail_no_missing":["0","9"],"object_location_counts":[{"shards":"0,9","objects":2886}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"169'17523","reported_seq":23049,"reported_epoch":169,"state":"active+remapped+backfilling","last_fresh":"2024-05-15T23:09:24.872379+0000","last_change":"2024-05-15T23:09:06.160790+0000","last_active":"2024-05-15T23:09:24.872379+0000","last_peered":"2024-05-15T23:09:24.872379+0000","last_clean":"2024-05-15T23:09:03.960701+0000","last_became_active":"2024-05-15T23:09:06.099726+0000","last_became_peered":"2024-05-15T23:09:06.099726+0000","last_unstale":"2024-05-15T23:09:24.872379+0000","last_undegraded":"2024-05-15T23:09:24.872379+0000","last_fullsized":"2024-05-15T23:09:24.872379+0000","mapping_epoch":164,"log_start":"130'7500","ondisk_log_start":"130'7500","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'11647","last_scrub_stamp":"2024-05-15T23:07:55.130334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:07:55.130334+0000","objects_scrubbed":8760,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185794560,"num_objects":2835,"num_object_clones":0,"num_object_copies":5670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":627,"num_objects_unfound":0,"num_objects_dirty":2835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17523,"num_write_kb":651456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2636,"num_bytes_recovered":172752896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,9],"avail_no_missing":["7","9"],"object_location_counts":[{"shards":"7,9","objects":2835}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"169'17087","reported_seq":17373,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-05-15T23:09:24.753466+0000","last_change":"2024-05-15T23:08:42.016394+0000","last_active":"2024-05-15T23:09:24.753466+0000","last_peered":"2024-05-15T23:09:24.753466+0000","last_clean":"2024-05-15T23:09:24.753466+0000","last_became_active":"2024-05-15T23:05:51.934294+0000","last_became_peered":"2024-05-15T23:05:51.934294+0000","last_unstale":"2024-05-15T23:09:24.753466+0000","last_undegraded":"2024-05-15T23:09:24.753466+0000","last_fullsized":"2024-05-15T23:09:24.753466+0000","mapping_epoch":102,"log_start":"128'7000","ondisk_log_start":"128'7000","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'14682","last_scrub_stamp":"2024-05-15T23:08:42.016167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:08:42.016167+0000","objects_scrubbed":5199,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:10:02.103194+0000","scrub_duration":1138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181731328,"num_objects":2773,"num_object_clones":0,"num_object_copies":5546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17087,"num_write_kb":635520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"169'17481","reported_seq":17774,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-05-15T23:09:27.032957+0000","last_change":"2024-05-15T23:08:47.069522+0000","last_active":"2024-05-15T23:09:27.032957+0000","last_peered":"2024-05-15T23:09:27.032957+0000","last_clean":"2024-05-15T23:09:27.032957+0000","last_became_active":"2024-05-15T23:05:51.957105+0000","last_became_peered":"2024-05-15T23:05:51.957105+0000","last_unstale":"2024-05-15T23:09:27.032957+0000","last_undegraded":"2024-05-15T23:09:27.032957+0000","last_fullsized":"2024-05-15T23:09:27.032957+0000","mapping_epoch":102,"log_start":"130'7400","ondisk_log_start":"130'7400","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'15114","last_scrub_stamp":"2024-05-15T23:08:47.069362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:08:47.069362+0000","objects_scrubbed":5061,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:10:02.316933+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174784512,"num_objects":2667,"num_object_clones":0,"num_object_copies":5334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17481,"num_write_kb":644736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"169'17514","reported_seq":17808,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-05-15T23:09:24.436574+0000","last_change":"2024-05-15T23:08:43.985597+0000","last_active":"2024-05-15T23:09:24.436574+0000","last_peered":"2024-05-15T23:09:24.436574+0000","last_clean":"2024-05-15T23:09:24.436574+0000","last_became_active":"2024-05-15T23:05:51.937101+0000","last_became_peered":"2024-05-15T23:05:51.937101+0000","last_unstale":"2024-05-15T23:09:24.436574+0000","last_undegraded":"2024-05-15T23:09:24.436574+0000","last_fullsized":"2024-05-15T23:09:24.436574+0000","mapping_epoch":102,"log_start":"130'7500","ondisk_log_start":"130'7500","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'15121","last_scrub_stamp":"2024-05-15T23:08:43.985306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:08:43.985306+0000","objects_scrubbed":5198,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:10:11.862343+0000","scrub_duration":1117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180879360,"num_objects":2760,"num_object_clones":0,"num_object_copies":5520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17514,"num_write_kb":648768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"169'17263","reported_seq":17563,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-05-15T23:09:26.997876+0000","last_change":"2024-05-15T23:08:43.333297+0000","last_active":"2024-05-15T23:09:26.997876+0000","last_peered":"2024-05-15T23:09:26.997876+0000","last_clean":"2024-05-15T23:09:26.997876+0000","last_became_active":"2024-05-15T23:05:51.957426+0000","last_became_peered":"2024-05-15T23:05:51.957426+0000","last_unstale":"2024-05-15T23:09:26.997876+0000","last_undegraded":"2024-05-15T23:09:26.997876+0000","last_fullsized":"2024-05-15T23:09:26.997876+0000","mapping_epoch":102,"log_start":"130'7200","ondisk_log_start":"130'7200","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'14736","last_scrub_stamp":"2024-05-15T23:08:43.333080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:08:43.333080+0000","objects_scrubbed":5268,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:09:56.490768+0000","scrub_duration":1233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176750592,"num_objects":2697,"num_object_clones":0,"num_object_copies":5394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17263,"num_write_kb":638720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"169'17189","reported_seq":23290,"reported_epoch":169,"state":"active+remapped+backfilling","last_fresh":"2024-05-15T23:09:24.391952+0000","last_change":"2024-05-15T23:09:06.141569+0000","last_active":"2024-05-15T23:09:24.391952+0000","last_peered":"2024-05-15T23:09:24.391952+0000","last_clean":"2024-05-15T23:09:03.933656+0000","last_became_active":"2024-05-15T23:09:06.094329+0000","last_became_peered":"2024-05-15T23:09:06.094329+0000","last_unstale":"2024-05-15T23:09:24.391952+0000","last_undegraded":"2024-05-15T23:09:24.391952+0000","last_fullsized":"2024-05-15T23:09:24.391952+0000","mapping_epoch":164,"log_start":"128'7100","ondisk_log_start":"128'7100","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'14572","last_scrub_stamp":"2024-05-15T23:08:40.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:08:40.481506+0000","objects_scrubbed":5425,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1044,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182059032,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":295,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17189,"num_write_kb":638913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2897,"num_bytes_recovered":189792280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,9],"avail_no_missing":["4","9"],"object_location_counts":[{"shards":"4,9","objects":2779}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"169'17792","reported_seq":18087,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-05-15T23:09:25.539527+0000","last_change":"2024-05-15T23:08:39.092294+0000","last_active":"2024-05-15T23:09:25.539527+0000","last_peered":"2024-05-15T23:09:25.539527+0000","last_clean":"2024-05-15T23:09:25.539527+0000","last_became_active":"2024-05-15T23:05:51.957133+0000","last_became_peered":"2024-05-15T23:05:51.957133+0000","last_unstale":"2024-05-15T23:09:25.539527+0000","last_undegraded":"2024-05-15T23:09:25.539527+0000","last_fullsized":"2024-05-15T23:09:25.539527+0000","mapping_epoch":102,"log_start":"130'7700","ondisk_log_start":"130'7700","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'15003","last_scrub_stamp":"2024-05-15T23:08:39.092182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:05:50.918160+0000","last_clean_scrub_stamp":"2024-05-15T23:08:39.092182+0000","objects_scrubbed":5547,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:09:54.715796+0000","scrub_duration":1253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177995776,"num_objects":2716,"num_object_clones":0,"num_object_copies":5432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17792,"num_write_kb":656256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":2917662744,"num_objects":44521,"num_object_clones":0,"num_object_copies":89042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2864,"num_objects_unfound":0,"num_objects_dirty":44521,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":278191,"num_write_kb":10326721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12400,"num_bytes_recovered":812515352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6620852224,"data_stored":6620840008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160891,"ondisk_log_size":160891,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351044,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":580652,"kb_used_data":513380,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93164500,"statfs":{"total":95995035648,"available":95400448000,"internally_reserved":0,"allocated":525701120,"data_stored":525360256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":68876740},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579977,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":385828,"kb_used_data":310236,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93359324,"statfs":{"total":95995035648,"available":95599947776,"internally_reserved":0,"allocated":317681664,"data_stored":317344372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":77396416},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":292057776209,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005141,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":848804,"kb_used_data":741276,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92896348,"statfs":{"total":95995035648,"available":95125860352,"internally_reserved":0,"allocated":759066624,"data_stored":758729332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266777,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":627948,"kb_used_data":556516,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93117204,"statfs":{"total":95995035648,"available":95352016896,"internally_reserved":0,"allocated":569872384,"data_stored":569531540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495712,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":164004,"kb_used_data":132828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93581148,"statfs":{"total":95995035648,"available":95827095552,"internally_reserved":0,"allocated":136015872,"data_stored":135678580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574969,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":830948,"kb_used_data":731740,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92914204,"statfs":{"total":95995035648,"available":95144144896,"internally_reserved":0,"allocated":749301760,"data_stored":748964468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738494,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1002852,"kb_used_data":877276,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92742300,"statfs":{"total":95995035648,"available":94968115200,"internally_reserved":0,"allocated":898330624,"data_stored":897993332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346035,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":601444,"kb_used_data":537948,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93143708,"statfs":{"total":95995035648,"available":95379156992,"internally_reserved":0,"allocated":550858752,"data_stored":550521460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":793960,"kb_used_data":700256,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":92951192,"statfs":{"total":95995035648,"available":95182020608,"internally_reserved":0,"allocated":717062144,"data_stored":716720780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":95877575},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986280,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":434344,"kb_used_data":367072,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93310808,"statfs":{"total":95995035648,"available":95550267392,"internally_reserved":0,"allocated":375881728,"data_stored":375540364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790051,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":412580,"kb_used_data":341148,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93332572,"statfs":{"total":95995035648,"available":95572553728,"internally_reserved":0,"allocated":349335552,"data_stored":348998260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":892665856,"data_stored":892665856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":733020160,"data_stored":733020160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":542572544,"data_stored":542572544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":708644864,"data_stored":708640792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":369758208,"data_stored":369754136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":346030080,"data_stored":346030080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":138739712,"data_stored":138739712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":557318144,"data_stored":557318144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":744947712,"data_stored":744947712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":747573248,"data_stored":747569176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":316735488,"data_stored":316735488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":522846208,"data_stored":522846208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:09:30.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:30 smithi084 ceph-mon[39784]: pgmap v516: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 2.7 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 768 op/s; 2864/89046 objects misplaced (3.216%); 15 MiB/s, 236 objects/s recovering 2024-05-15T23:09:30.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:30 smithi084 ceph-mon[39784]: from='client.15081 v1:172.21.15.84:0/838903792' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:09:30.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:30 smithi084 ceph-mon[39784]: osdmap e172: 12 total, 12 up, 11 in 2024-05-15T23:09:30.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:30 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:09:30.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:30 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:09:30.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:30 smithi090 ceph-mon[44434]: pgmap v516: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 2.7 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 768 op/s; 2864/89046 objects misplaced (3.216%); 15 MiB/s, 236 objects/s recovering 2024-05-15T23:09:30.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:30 smithi090 ceph-mon[44434]: from='client.15081 v1:172.21.15.84:0/838903792' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:09:30.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:30 smithi090 ceph-mon[44434]: osdmap e172: 12 total, 12 up, 11 in 2024-05-15T23:09:30.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:30 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:09:30.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:30 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:09:30.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:30 smithi088 ceph-mon[44699]: pgmap v516: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 2.7 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 768 op/s; 2864/89046 objects misplaced (3.216%); 15 MiB/s, 236 objects/s recovering 2024-05-15T23:09:30.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:30 smithi088 ceph-mon[44699]: from='client.15081 v1:172.21.15.84:0/838903792' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:09:30.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:30 smithi088 ceph-mon[44699]: osdmap e172: 12 total, 12 up, 11 in 2024-05-15T23:09:30.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:30 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:09:30.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:30 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:09:30.733 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:09:30.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:30.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:09:30.844 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:09:30.873 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:30.901 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:09:30.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:30.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:09:31.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:09:31.035 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:31.063 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:09:31.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:31.068 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:09:31.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:09:31.252 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:31.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:31 smithi090 ceph-mon[44434]: osdmap e173: 12 total, 12 up, 11 in 2024-05-15T23:09:31.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:31 smithi088 ceph-mon[44699]: osdmap e173: 12 total, 12 up, 11 in 2024-05-15T23:09:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:31 smithi084 ceph-mon[39784]: osdmap e173: 12 total, 12 up, 11 in 2024-05-15T23:09:32.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:32 smithi084 ceph-mon[39784]: pgmap v519: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.6 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 1942/84488 objects misplaced (2.299%); 14 MiB/s, 223 objects/s recovering 2024-05-15T23:09:32.484 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:09:32.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:32 smithi090 ceph-mon[44434]: pgmap v519: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.6 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 1942/84488 objects misplaced (2.299%); 14 MiB/s, 223 objects/s recovering 2024-05-15T23:09:32.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:32 smithi088 ceph-mon[44699]: pgmap v519: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.6 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 1942/84488 objects misplaced (2.299%); 14 MiB/s, 223 objects/s recovering 2024-05-15T23:09:33.452 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:09:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-05-15T23:09:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-05-15T23:09:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:09:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.452 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:109644 2.7 2:ea3584ac:::benchmark_data_smithi090_78160_object109643:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:09:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:53.339492+0000", 2024-05-15T23:09:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 160.00330732699999, 2024-05-15T23:09:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.139075318, 2024-05-15T23:09:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109644 2024-05-15T23:09:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.339492+0000", 2024-05-15T23:09:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.339492+0000", 2024-05-15T23:09:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.339494+0000", 2024-05-15T23:09:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6300000000000001e-06 2024-05-15T23:09:33.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.339528+0000", 2024-05-15T23:09:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4041000000000002e-05 2024-05-15T23:09:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.339537+0000", 2024-05-15T23:09:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1970000000000002e-06 2024-05-15T23:09:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.339543+0000", 2024-05-15T23:09:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0589999999999999e-06 2024-05-15T23:09:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.339663+0000", 2024-05-15T23:09:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011989000000000001 2024-05-15T23:09:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.339806+0000", 2024-05-15T23:09:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014327099999999999 2024-05-15T23:09:33.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:09:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.339952+0000", 2024-05-15T23:09:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014586199999999999 2024-05-15T23:09:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.347040+0000", 2024-05-15T23:09:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.007087851 2024-05-15T23:09:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.478454+0000", 2024-05-15T23:09:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13141413599999999 2024-05-15T23:09:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.478514+0000", 2024-05-15T23:09:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9996e-05 2024-05-15T23:09:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.478568+0000", 2024-05-15T23:09:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3384999999999998e-05 2024-05-15T23:09:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.458 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:109647 2.7 2:ef9645db:::benchmark_data_smithi090_78160_object109646:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:09:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:53.342336+0000", 2024-05-15T23:09:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 160.000463336, 2024-05-15T23:09:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13636024999999999, 2024-05-15T23:09:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.459 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 109647 2024-05-15T23:09:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.342336+0000", 2024-05-15T23:09:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.342336+0000", 2024-05-15T23:09:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.342339+0000", 2024-05-15T23:09:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5569999999999998e-06 2024-05-15T23:09:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.342377+0000", 2024-05-15T23:09:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7577000000000003e-05 2024-05-15T23:09:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.342395+0000", 2024-05-15T23:09:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8210999999999999e-05 2024-05-15T23:09:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.342405+0000", 2024-05-15T23:09:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0124e-05 2024-05-15T23:09:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.343110+0000", 2024-05-15T23:09:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070469900000000004 2024-05-15T23:09:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.343220+0000", 2024-05-15T23:09:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011090399999999999 2024-05-15T23:09:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:09:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.343350+0000", 2024-05-15T23:09:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000129355 2024-05-15T23:09:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.347045+0000", 2024-05-15T23:09:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0036955019999999998 2024-05-15T23:09:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.478608+0000", 2024-05-15T23:09:33.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.131562808 2024-05-15T23:09:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.478660+0000", 2024-05-15T23:09:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1712000000000002e-05 2024-05-15T23:09:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:53.478697+0000", 2024-05-15T23:09:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6801000000000002e-05 2024-05-15T23:09:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.465 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:116813 2.7 2:e359cec5:::benchmark_data_smithi090_78160_object116812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:09:33.465 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:58.383013+0000", 2024-05-15T23:09:33.465 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 154.959787151, 2024-05-15T23:09:33.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.065674309, 2024-05-15T23:09:33.465 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.465 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.465 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.465 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 116813 2024-05-15T23:09:33.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.465 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.383013+0000", 2024-05-15T23:09:33.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.383013+0000", 2024-05-15T23:09:33.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.383014+0000", 2024-05-15T23:09:33.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.299e-06 2024-05-15T23:09:33.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.383065+0000", 2024-05-15T23:09:33.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1375000000000002e-05 2024-05-15T23:09:33.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.383073+0000", 2024-05-15T23:09:33.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0129999999999993e-06 2024-05-15T23:09:33.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.383079+0000", 2024-05-15T23:09:33.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3199999999999999e-06 2024-05-15T23:09:33.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.385829+0000", 2024-05-15T23:09:33.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002750429 2024-05-15T23:09:33.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.385966+0000", 2024-05-15T23:09:33.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013653600000000001 2024-05-15T23:09:33.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:09:33.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.386111+0000", 2024-05-15T23:09:33.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014507499999999999 2024-05-15T23:09:33.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.437273+0000", 2024-05-15T23:09:33.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051161975999999998 2024-05-15T23:09:33.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.448626+0000", 2024-05-15T23:09:33.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01135388 2024-05-15T23:09:33.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.448659+0000", 2024-05-15T23:09:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2262999999999999e-05 2024-05-15T23:09:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.448687+0000", 2024-05-15T23:09:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8143000000000001e-05 2024-05-15T23:09:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:116820 2.7 2:e0e0f65f:::benchmark_data_smithi090_78160_object116819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:09:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:06:58.388500+0000", 2024-05-15T23:09:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 154.954299943, 2024-05-15T23:09:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.060236588000000001, 2024-05-15T23:09:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 116820 2024-05-15T23:09:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.388500+0000", 2024-05-15T23:09:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.388500+0000", 2024-05-15T23:09:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.388501+0000", 2024-05-15T23:09:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4440000000000001e-06 2024-05-15T23:09:33.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.388555+0000", 2024-05-15T23:09:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3905e-05 2024-05-15T23:09:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.388563+0000", 2024-05-15T23:09:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9300000000000003e-06 2024-05-15T23:09:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.388569+0000", 2024-05-15T23:09:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9290000000000003e-06 2024-05-15T23:09:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.389341+0000", 2024-05-15T23:09:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00077180200000000003 2024-05-15T23:09:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.389457+0000", 2024-05-15T23:09:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011606499999999999 2024-05-15T23:09:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:09:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.389538+0000", 2024-05-15T23:09:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1508000000000006e-05 2024-05-15T23:09:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.437324+0000", 2024-05-15T23:09:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.047785514000000001 2024-05-15T23:09:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.448694+0000", 2024-05-15T23:09:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011370145 2024-05-15T23:09:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.448718+0000", 2024-05-15T23:09:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4099000000000001e-05 2024-05-15T23:09:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:06:58.448736+0000", 2024-05-15T23:09:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8247e-05 2024-05-15T23:09:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.476 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:248868 2.5 2:a9c3f41c:::benchmark_data_smithi090_78160_object87510:head [delete] snapc 0=[] ondisk+write+known_if_redirected e158)", 2024-05-15T23:09:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:08:53.893984+0000", 2024-05-15T23:09:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.448815455999998, 2024-05-15T23:09:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.063734375999999995, 2024-05-15T23:09:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 248868 2024-05-15T23:09:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.893984+0000", 2024-05-15T23:09:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.893984+0000", 2024-05-15T23:09:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.893985+0000", 2024-05-15T23:09:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5999999999999999e-07 2024-05-15T23:09:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.893986+0000", 2024-05-15T23:09:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.823e-06 2024-05-15T23:09:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.893990+0000", 2024-05-15T23:09:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3469999999999999e-06 2024-05-15T23:09:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.893993+0000", 2024-05-15T23:09:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7350000000000001e-06 2024-05-15T23:09:33.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894137+0000", 2024-05-15T23:09:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144196 2024-05-15T23:09:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894140+0000", 2024-05-15T23:09:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0000000000000001e-06 2024-05-15T23:09:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.948114+0000", 2024-05-15T23:09:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.053974126999999997 2024-05-15T23:09:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.948227+0000", 2024-05-15T23:09:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011345699999999999 2024-05-15T23:09:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 8", 2024-05-15T23:09:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.948306+0000", 2024-05-15T23:09:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8428000000000002e-05 2024-05-15T23:09:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.952729+0000", 2024-05-15T23:09:33.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044228089999999998 2024-05-15T23:09:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.957616+0000", 2024-05-15T23:09:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048873379999999998 2024-05-15T23:09:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.957671+0000", 2024-05-15T23:09:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5600000000000003e-05 2024-05-15T23:09:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.957719+0000", 2024-05-15T23:09:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7156e-05 2024-05-15T23:09:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:248883 2.5 2:a0b6385b:::benchmark_data_smithi090_78160_object87525:head [delete] snapc 0=[] ondisk+write+known_if_redirected e158)", 2024-05-15T23:09:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:08:53.894001+0000", 2024-05-15T23:09:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.448799018000003, 2024-05-15T23:09:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073839127000000004, 2024-05-15T23:09:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.483 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 248883 2024-05-15T23:09:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894001+0000", 2024-05-15T23:09:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894001+0000", 2024-05-15T23:09:33.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894001+0000", 2024-05-15T23:09:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1399999999999997e-07 2024-05-15T23:09:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894003+0000", 2024-05-15T23:09:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5770000000000001e-06 2024-05-15T23:09:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894006+0000", 2024-05-15T23:09:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1e-06 2024-05-15T23:09:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894009+0000", 2024-05-15T23:09:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7760000000000002e-06 2024-05-15T23:09:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894158+0000", 2024-05-15T23:09:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000149662 2024-05-15T23:09:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894162+0000", 2024-05-15T23:09:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2720000000000002e-06 2024-05-15T23:09:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.952758+0000", 2024-05-15T23:09:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.058596844000000002 2024-05-15T23:09:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.952836+0000", 2024-05-15T23:09:33.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7360000000000005e-05 2024-05-15T23:09:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 8", 2024-05-15T23:09:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.952919+0000", 2024-05-15T23:09:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2879000000000004e-05 2024-05-15T23:09:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.962746+0000", 2024-05-15T23:09:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0098274269999999997 2024-05-15T23:09:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.967741+0000", 2024-05-15T23:09:33.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049946620000000004 2024-05-15T23:09:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.967788+0000", 2024-05-15T23:09:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7790000000000002e-05 2024-05-15T23:09:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.967840+0000", 2024-05-15T23:09:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1363999999999998e-05 2024-05-15T23:09:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.489 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:248884 2.5 2:a9aa123f:::benchmark_data_smithi090_78160_object87526:head [delete] snapc 0=[] ondisk+write+known_if_redirected e158)", 2024-05-15T23:09:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:08:53.894013+0000", 2024-05-15T23:09:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.448786372999997, 2024-05-15T23:09:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.083327131999999998, 2024-05-15T23:09:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 248884 2024-05-15T23:09:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894013+0000", 2024-05-15T23:09:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894013+0000", 2024-05-15T23:09:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894014+0000", 2024-05-15T23:09:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9799999999999999e-07 2024-05-15T23:09:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894015+0000", 2024-05-15T23:09:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.331e-06 2024-05-15T23:09:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894018+0000", 2024-05-15T23:09:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8590000000000001e-06 2024-05-15T23:09:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894021+0000", 2024-05-15T23:09:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8609999999999998e-06 2024-05-15T23:09:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894180+0000", 2024-05-15T23:09:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000159085 2024-05-15T23:09:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894183+0000", 2024-05-15T23:09:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0489999999999999e-06 2024-05-15T23:09:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.957751+0000", 2024-05-15T23:09:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.063568524000000001 2024-05-15T23:09:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.957870+0000", 2024-05-15T23:09:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000118656 2024-05-15T23:09:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 8", 2024-05-15T23:09:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.957991+0000", 2024-05-15T23:09:33.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120506 2024-05-15T23:09:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.972811+0000", 2024-05-15T23:09:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01482054 2024-05-15T23:09:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.977247+0000", 2024-05-15T23:09:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044363579999999996 2024-05-15T23:09:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.977299+0000", 2024-05-15T23:09:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.134e-05 2024-05-15T23:09:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.977340+0000", 2024-05-15T23:09:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1724999999999998e-05 2024-05-15T23:09:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:248908 2.5 2:aa5927fc:::benchmark_data_smithi090_78160_object87550:head [delete] snapc 0=[] ondisk+write+known_if_redirected e158)", 2024-05-15T23:09:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:08:53.894024+0000", 2024-05-15T23:09:33.496 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.448775390999998, 2024-05-15T23:09:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.083569885999999996, 2024-05-15T23:09:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 248908 2024-05-15T23:09:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894024+0000", 2024-05-15T23:09:33.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894024+0000", 2024-05-15T23:09:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894025+0000", 2024-05-15T23:09:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4499999999999998e-07 2024-05-15T23:09:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894026+0000", 2024-05-15T23:09:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.252e-06 2024-05-15T23:09:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894029+0000", 2024-05-15T23:09:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.473e-06 2024-05-15T23:09:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894032+0000", 2024-05-15T23:09:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6699999999999998e-06 2024-05-15T23:09:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894202+0000", 2024-05-15T23:09:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000169951 2024-05-15T23:09:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894205+0000", 2024-05-15T23:09:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0860000000000002e-06 2024-05-15T23:09:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.962790+0000", 2024-05-15T23:09:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.068585008000000003 2024-05-15T23:09:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.962885+0000", 2024-05-15T23:09:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4712999999999999e-05 2024-05-15T23:09:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 8", 2024-05-15T23:09:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.962987+0000", 2024-05-15T23:09:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010211 2024-05-15T23:09:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.977404+0000", 2024-05-15T23:09:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014416597 2024-05-15T23:09:33.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.977512+0000", 2024-05-15T23:09:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010838800000000001 2024-05-15T23:09:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.977554+0000", 2024-05-15T23:09:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2088000000000001e-05 2024-05-15T23:09:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.977594+0000", 2024-05-15T23:09:33.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0204999999999999e-05 2024-05-15T23:09:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:248949 2.5 2:a1a2aeee:::benchmark_data_smithi090_78160_object87591:head [delete] snapc 0=[] ondisk+write+known_if_redirected e158)", 2024-05-15T23:09:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:08:53.894042+0000", 2024-05-15T23:09:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.448757673000003, 2024-05-15T23:09:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.083631450999999996, 2024-05-15T23:09:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.503 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 248949 2024-05-15T23:09:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894042+0000", 2024-05-15T23:09:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894042+0000", 2024-05-15T23:09:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894042+0000", 2024-05-15T23:09:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.46e-07 2024-05-15T23:09:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894044+0000", 2024-05-15T23:09:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.61e-06 2024-05-15T23:09:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894047+0000", 2024-05-15T23:09:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.05e-06 2024-05-15T23:09:33.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894050+0000", 2024-05-15T23:09:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1140000000000002e-06 2024-05-15T23:09:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894224+0000", 2024-05-15T23:09:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000173475 2024-05-15T23:09:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894227+0000", 2024-05-15T23:09:33.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0110000000000001e-06 2024-05-15T23:09:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.967877+0000", 2024-05-15T23:09:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.073650289999999993 2024-05-15T23:09:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.967979+0000", 2024-05-15T23:09:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102315 2024-05-15T23:09:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 8", 2024-05-15T23:09:33.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.968088+0000", 2024-05-15T23:09:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000108902 2024-05-15T23:09:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.977443+0000", 2024-05-15T23:09:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0093551780000000005 2024-05-15T23:09:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.977602+0000", 2024-05-15T23:09:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015873600000000001 2024-05-15T23:09:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.977640+0000", 2024-05-15T23:09:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8012000000000001e-05 2024-05-15T23:09:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.977673+0000", 2024-05-15T23:09:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3411999999999998e-05 2024-05-15T23:09:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:248957 2.5 2:a9ec1c6e:::benchmark_data_smithi090_78160_object87599:head [delete] snapc 0=[] ondisk+write+known_if_redirected e158)", 2024-05-15T23:09:33.509 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:08:53.894054+0000", 2024-05-15T23:09:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.448746098000001, 2024-05-15T23:09:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093212122999999994, 2024-05-15T23:09:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 248957 2024-05-15T23:09:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894054+0000", 2024-05-15T23:09:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894054+0000", 2024-05-15T23:09:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894054+0000", 2024-05-15T23:09:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.46e-07 2024-05-15T23:09:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894055+0000", 2024-05-15T23:09:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2359999999999999e-06 2024-05-15T23:09:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894058+0000", 2024-05-15T23:09:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5840000000000002e-06 2024-05-15T23:09:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894061+0000", 2024-05-15T23:09:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7190000000000001e-06 2024-05-15T23:09:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894246+0000", 2024-05-15T23:09:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000185052 2024-05-15T23:09:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894249+0000", 2024-05-15T23:09:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1609999999999999e-06 2024-05-15T23:09:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.972859+0000", 2024-05-15T23:09:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.078610228000000004 2024-05-15T23:09:33.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.972990+0000", 2024-05-15T23:09:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013056 2024-05-15T23:09:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 8", 2024-05-15T23:09:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.973065+0000", 2024-05-15T23:09:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5321000000000006e-05 2024-05-15T23:09:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.977497+0000", 2024-05-15T23:09:33.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044324489999999998 2024-05-15T23:09:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.987160+0000", 2024-05-15T23:09:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096627759999999997 2024-05-15T23:09:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.987209+0000", 2024-05-15T23:09:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9205000000000001e-05 2024-05-15T23:09:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.987266+0000", 2024-05-15T23:09:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6486000000000001e-05 2024-05-15T23:09:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:248966 2.5 2:a0df9060:::benchmark_data_smithi090_78160_object87608:head [delete] snapc 0=[] ondisk+write+known_if_redirected e158)", 2024-05-15T23:09:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:08:53.894063+0000", 2024-05-15T23:09:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.448736492999998, 2024-05-15T23:09:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093274418999999997, 2024-05-15T23:09:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.516 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 248966 2024-05-15T23:09:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894063+0000", 2024-05-15T23:09:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894063+0000", 2024-05-15T23:09:33.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894064+0000", 2024-05-15T23:09:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9999999999999999e-07 2024-05-15T23:09:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894065+0000", 2024-05-15T23:09:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1999999999999999e-06 2024-05-15T23:09:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894067+0000", 2024-05-15T23:09:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5349999999999999e-06 2024-05-15T23:09:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894070+0000", 2024-05-15T23:09:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6419999999999999e-06 2024-05-15T23:09:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894269+0000", 2024-05-15T23:09:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019891 2024-05-15T23:09:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894272+0000", 2024-05-15T23:09:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0860000000000002e-06 2024-05-15T23:09:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.977708+0000", 2024-05-15T23:09:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.083435916999999998 2024-05-15T23:09:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.977777+0000", 2024-05-15T23:09:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8870999999999998e-05 2024-05-15T23:09:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 8", 2024-05-15T23:09:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.977869+0000", 2024-05-15T23:09:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2066999999999998e-05 2024-05-15T23:09:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.987148+0000", 2024-05-15T23:09:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0092792110000000008 2024-05-15T23:09:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.987273+0000", 2024-05-15T23:09:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000125148 2024-05-15T23:09:33.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.987307+0000", 2024-05-15T23:09:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4189000000000001e-05 2024-05-15T23:09:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.987338+0000", 2024-05-15T23:09:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0343000000000001e-05 2024-05-15T23:09:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:248967 2.5 2:a893f840:::benchmark_data_smithi090_78160_object87609:head [delete] snapc 0=[] ondisk+write+known_if_redirected e158)", 2024-05-15T23:09:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:08:53.894073+0000", 2024-05-15T23:09:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.448726833999999, 2024-05-15T23:09:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.10228074400000001, 2024-05-15T23:09:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 248967 2024-05-15T23:09:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894073+0000", 2024-05-15T23:09:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894073+0000", 2024-05-15T23:09:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894073+0000", 2024-05-15T23:09:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9999999999999999e-07 2024-05-15T23:09:33.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894074+0000", 2024-05-15T23:09:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9699999999999994e-07 2024-05-15T23:09:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894077+0000", 2024-05-15T23:09:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5210000000000001e-06 2024-05-15T23:09:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894080+0000", 2024-05-15T23:09:33.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1420000000000002e-06 2024-05-15T23:09:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894291+0000", 2024-05-15T23:09:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021126200000000001 2024-05-15T23:09:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894294+0000", 2024-05-15T23:09:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.157e-06 2024-05-15T23:09:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.982367+0000", 2024-05-15T23:09:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.088072658999999998 2024-05-15T23:09:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.982446+0000", 2024-05-15T23:09:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9573999999999994e-05 2024-05-15T23:09:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 8", 2024-05-15T23:09:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.982531+0000", 2024-05-15T23:09:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4067999999999995e-05 2024-05-15T23:09:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.991380+0000", 2024-05-15T23:09:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0088496259999999993 2024-05-15T23:09:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.996270+0000", 2024-05-15T23:09:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048902019999999997 2024-05-15T23:09:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.996307+0000", 2024-05-15T23:09:33.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7024000000000003e-05 2024-05-15T23:09:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.996354+0000", 2024-05-15T23:09:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6211999999999997e-05 2024-05-15T23:09:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:248985 2.5 2:a23ebad0:::benchmark_data_smithi090_78160_object87627:head [delete] snapc 0=[] ondisk+write+known_if_redirected e158)", 2024-05-15T23:09:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:08:53.894083+0000", 2024-05-15T23:09:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.448716546, 2024-05-15T23:09:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.107152632, 2024-05-15T23:09:33.529 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 248985 2024-05-15T23:09:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894083+0000", 2024-05-15T23:09:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894083+0000", 2024-05-15T23:09:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894083+0000", 2024-05-15T23:09:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1100000000000002e-07 2024-05-15T23:09:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894085+0000", 2024-05-15T23:09:33.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2100000000000001e-06 2024-05-15T23:09:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894087+0000", 2024-05-15T23:09:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6709999999999999e-06 2024-05-15T23:09:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894090+0000", 2024-05-15T23:09:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5909999999999998e-06 2024-05-15T23:09:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894313+0000", 2024-05-15T23:09:33.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00022276300000000001 2024-05-15T23:09:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894316+0000", 2024-05-15T23:09:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.101e-06 2024-05-15T23:09:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.987364+0000", 2024-05-15T23:09:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093047719000000001 2024-05-15T23:09:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.987462+0000", 2024-05-15T23:09:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8138000000000001e-05 2024-05-15T23:09:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 8", 2024-05-15T23:09:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.987582+0000", 2024-05-15T23:09:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120315 2024-05-15T23:09:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:54.001101+0000", 2024-05-15T23:09:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013518842 2024-05-15T23:09:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:54.001145+0000", 2024-05-15T23:09:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4239000000000001e-05 2024-05-15T23:09:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:54.001198+0000", 2024-05-15T23:09:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3131999999999999e-05 2024-05-15T23:09:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:54.001236+0000", 2024-05-15T23:09:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7599999999999999e-05 2024-05-15T23:09:33.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:248990 2.5 2:a53ba3e1:::benchmark_data_smithi090_78160_object87632:head [delete] snapc 0=[] ondisk+write+known_if_redirected e158)", 2024-05-15T23:09:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:08:53.894093+0000", 2024-05-15T23:09:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.448706428999998, 2024-05-15T23:09:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.10731796, 2024-05-15T23:09:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 248990 2024-05-15T23:09:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894093+0000", 2024-05-15T23:09:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894093+0000", 2024-05-15T23:09:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894094+0000", 2024-05-15T23:09:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8599999999999999e-07 2024-05-15T23:09:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894095+0000", 2024-05-15T23:09:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.339e-06 2024-05-15T23:09:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894098+0000", 2024-05-15T23:09:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0419999999999998e-06 2024-05-15T23:09:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894101+0000", 2024-05-15T23:09:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5749999999999999e-06 2024-05-15T23:09:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894328+0000", 2024-05-15T23:09:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00022708799999999999 2024-05-15T23:09:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.894331+0000", 2024-05-15T23:09:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0580000000000002e-06 2024-05-15T23:09:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.991504+0000", 2024-05-15T23:09:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097173679999999998 2024-05-15T23:09:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.991609+0000", 2024-05-15T23:09:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105025 2024-05-15T23:09:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 8", 2024-05-15T23:09:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:53.991721+0000", 2024-05-15T23:09:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011150300000000001 2024-05-15T23:09:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:54.001285+0000", 2024-05-15T23:09:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0095642360000000003 2024-05-15T23:09:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:54.001300+0000", 2024-05-15T23:09:33.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4892000000000001e-05 2024-05-15T23:09:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:54.001347+0000", 2024-05-15T23:09:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7246999999999997e-05 2024-05-15T23:09:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:08:54.001411+0000", 2024-05-15T23:09:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3989e-05 2024-05-15T23:09:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.542 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:260048 2.7 2:e6d2c805:::benchmark_data_smithi090_78160_object98690:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e165)", 2024-05-15T23:09:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:09:05.999999+0000", 2024-05-15T23:09:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 27.342800464, 2024-05-15T23:09:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.124037191, 2024-05-15T23:09:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:09:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260048 2024-05-15T23:09:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.543 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:05.999999+0000", 2024-05-15T23:09:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:05.999999+0000", 2024-05-15T23:09:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000003+0000", 2024-05-15T23:09:33.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5999999999999998e-06 2024-05-15T23:09:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000012+0000", 2024-05-15T23:09:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3100000000000006e-06 2024-05-15T23:09:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000027+0000", 2024-05-15T23:09:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5102999999999999e-05 2024-05-15T23:09:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000041+0000", 2024-05-15T23:09:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3835999999999999e-05 2024-05-15T23:09:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000092+0000", 2024-05-15T23:09:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1214999999999997e-05 2024-05-15T23:09:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000105+0000", 2024-05-15T23:09:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2371000000000001e-05 2024-05-15T23:09:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.123967+0000", 2024-05-15T23:09:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.123862544 2024-05-15T23:09:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.124036+0000", 2024-05-15T23:09:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9212000000000001e-05 2024-05-15T23:09:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.547 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:260058 2.7 2:e4bf2cc9:::benchmark_data_smithi090_78160_object98700:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e165)", 2024-05-15T23:09:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:09:06.000065+0000", 2024-05-15T23:09:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 27.342735007000002, 2024-05-15T23:09:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.124204759, 2024-05-15T23:09:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:09:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260058 2024-05-15T23:09:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000065+0000", 2024-05-15T23:09:33.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000065+0000", 2024-05-15T23:09:33.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000066+0000", 2024-05-15T23:09:33.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.046e-06 2024-05-15T23:09:33.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000070+0000", 2024-05-15T23:09:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.25e-06 2024-05-15T23:09:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000077+0000", 2024-05-15T23:09:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2110000000000001e-06 2024-05-15T23:09:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000084+0000", 2024-05-15T23:09:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3289999999999999e-06 2024-05-15T23:09:33.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000125+0000", 2024-05-15T23:09:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1031000000000001e-05 2024-05-15T23:09:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000132+0000", 2024-05-15T23:09:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4290000000000004e-06 2024-05-15T23:09:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.124220+0000", 2024-05-15T23:09:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.124087975 2024-05-15T23:09:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.124269+0000", 2024-05-15T23:09:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9487999999999998e-05 2024-05-15T23:09:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:260070 2.7 2:eab11625:::benchmark_data_smithi090_78160_object98712:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e165)", 2024-05-15T23:09:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:09:06.000094+0000", 2024-05-15T23:09:33.552 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 27.342706089, 2024-05-15T23:09:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14221492899999999, 2024-05-15T23:09:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260070 2024-05-15T23:09:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000094+0000", 2024-05-15T23:09:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000094+0000", 2024-05-15T23:09:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000094+0000", 2024-05-15T23:09:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0399999999999995e-07 2024-05-15T23:09:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000097+0000", 2024-05-15T23:09:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8449999999999999e-06 2024-05-15T23:09:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000104+0000", 2024-05-15T23:09:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7479999999999996e-06 2024-05-15T23:09:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000110+0000", 2024-05-15T23:09:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3169999999999998e-06 2024-05-15T23:09:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000152+0000", 2024-05-15T23:09:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2172000000000002e-05 2024-05-15T23:09:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000159+0000", 2024-05-15T23:09:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2840000000000003e-06 2024-05-15T23:09:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.124296+0000", 2024-05-15T23:09:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12413687 2024-05-15T23:09:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.124383+0000", 2024-05-15T23:09:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7601000000000004e-05 2024-05-15T23:09:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 6,9,10", 2024-05-15T23:09:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.124495+0000", 2024-05-15T23:09:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111377 2024-05-15T23:09:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.133373+0000", 2024-05-15T23:09:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0088782129999999994 2024-05-15T23:09:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.133398+0000", 2024-05-15T23:09:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5695e-05 2024-05-15T23:09:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.133440+0000", 2024-05-15T23:09:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1177999999999998e-05 2024-05-15T23:09:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.142114+0000", 2024-05-15T23:09:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086740159999999997 2024-05-15T23:09:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.142176+0000", 2024-05-15T23:09:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2152999999999999e-05 2024-05-15T23:09:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.142309+0000", 2024-05-15T23:09:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013275599999999999 2024-05-15T23:09:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:260081 2.7 2:ec20205a:::benchmark_data_smithi090_78160_object98723:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165)", 2024-05-15T23:09:33.559 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:09:06.000119+0000", 2024-05-15T23:09:33.560 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 27.342680541, 2024-05-15T23:09:33.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15050044500000001, 2024-05-15T23:09:33.560 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.560 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.560 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.560 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.560 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.560 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.560 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260081 2024-05-15T23:09:33.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.560 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000119+0000", 2024-05-15T23:09:33.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000119+0000", 2024-05-15T23:09:33.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000120+0000", 2024-05-15T23:09:33.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7400000000000003e-07 2024-05-15T23:09:33.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000123+0000", 2024-05-15T23:09:33.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8490000000000002e-06 2024-05-15T23:09:33.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000130+0000", 2024-05-15T23:09:33.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.097e-06 2024-05-15T23:09:33.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000135+0000", 2024-05-15T23:09:33.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3090000000000003e-06 2024-05-15T23:09:33.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000176+0000", 2024-05-15T23:09:33.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1091999999999999e-05 2024-05-15T23:09:33.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000180+0000", 2024-05-15T23:09:33.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8070000000000001e-06 2024-05-15T23:09:33.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.128145+0000", 2024-05-15T23:09:33.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12796486400000001 2024-05-15T23:09:33.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.128268+0000", 2024-05-15T23:09:33.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012366699999999999 2024-05-15T23:09:33.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 6,9,10", 2024-05-15T23:09:33.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.128416+0000", 2024-05-15T23:09:33.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014787200000000001 2024-05-15T23:09:33.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.133490+0000", 2024-05-15T23:09:33.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050740079999999996 2024-05-15T23:09:33.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.133506+0000", 2024-05-15T23:09:33.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6025999999999999e-05 2024-05-15T23:09:33.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.133542+0000", 2024-05-15T23:09:33.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6010000000000003e-05 2024-05-15T23:09:33.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.150408+0000", 2024-05-15T23:09:33.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016865905 2024-05-15T23:09:33.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.150479+0000", 2024-05-15T23:09:33.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0487999999999993e-05 2024-05-15T23:09:33.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.150620+0000", 2024-05-15T23:09:33.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000140877 2024-05-15T23:09:33.566 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.566 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.566 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.567 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:260087 2.7 2:ec1938d6:::benchmark_data_smithi090_78160_object98729:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165)", 2024-05-15T23:09:33.567 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:09:06.000155+0000", 2024-05-15T23:09:33.567 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 27.342644375999999, 2024-05-15T23:09:33.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16143232199999999, 2024-05-15T23:09:33.567 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.567 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.567 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.567 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.567 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.567 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.567 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260087 2024-05-15T23:09:33.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.567 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000155+0000", 2024-05-15T23:09:33.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000155+0000", 2024-05-15T23:09:33.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000156+0000", 2024-05-15T23:09:33.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5300000000000002e-07 2024-05-15T23:09:33.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000160+0000", 2024-05-15T23:09:33.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2739999999999999e-06 2024-05-15T23:09:33.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000166+0000", 2024-05-15T23:09:33.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7159999999999997e-06 2024-05-15T23:09:33.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000172+0000", 2024-05-15T23:09:33.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7629999999999997e-06 2024-05-15T23:09:33.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000240+0000", 2024-05-15T23:09:33.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7807999999999999e-05 2024-05-15T23:09:33.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000244+0000", 2024-05-15T23:09:33.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.985e-06 2024-05-15T23:09:33.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.133578+0000", 2024-05-15T23:09:33.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13333440199999999 2024-05-15T23:09:33.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.133711+0000", 2024-05-15T23:09:33.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000132576 2024-05-15T23:09:33.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 6,9,10", 2024-05-15T23:09:33.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.133854+0000", 2024-05-15T23:09:33.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014310899999999999 2024-05-15T23:09:33.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.142372+0000", 2024-05-15T23:09:33.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085178240000000002 2024-05-15T23:09:33.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.142402+0000", 2024-05-15T23:09:33.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0334000000000001e-05 2024-05-15T23:09:33.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.142440+0000", 2024-05-15T23:09:33.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7684e-05 2024-05-15T23:09:33.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.161381+0000", 2024-05-15T23:09:33.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018941066999999999 2024-05-15T23:09:33.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.161454+0000", 2024-05-15T23:09:33.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3629999999999998e-05 2024-05-15T23:09:33.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.161588+0000", 2024-05-15T23:09:33.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013319699999999999 2024-05-15T23:09:33.573 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.573 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.574 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.574 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24745.0:260091 2.7 2:ed8eb145:::benchmark_data_smithi090_78160_object98733:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165)", 2024-05-15T23:09:33.574 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:09:06.000181+0000", 2024-05-15T23:09:33.574 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 27.342618662, 2024-05-15T23:09:33.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.161637696, 2024-05-15T23:09:33.574 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:09:33.574 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:09:33.574 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:09:33.574 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:09:33.574 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24745", 2024-05-15T23:09:33.574 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:09:33.575 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260091 2024-05-15T23:09:33.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.575 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:09:33.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:09:33.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000181+0000", 2024-05-15T23:09:33.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:09:33.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000181+0000", 2024-05-15T23:09:33.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:09:33.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:09:33.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000182+0000", 2024-05-15T23:09:33.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4600000000000004e-07 2024-05-15T23:09:33.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:09:33.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000185+0000", 2024-05-15T23:09:33.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9639999999999999e-06 2024-05-15T23:09:33.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:09:33.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000227+0000", 2024-05-15T23:09:33.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1881000000000001e-05 2024-05-15T23:09:33.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:09:33.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000235+0000", 2024-05-15T23:09:33.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2199999999999992e-06 2024-05-15T23:09:33.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000266+0000", 2024-05-15T23:09:33.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1337000000000002e-05 2024-05-15T23:09:33.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:09:33.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.000270+0000", 2024-05-15T23:09:33.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7790000000000002e-06 2024-05-15T23:09:33.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:09:33.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.137716+0000", 2024-05-15T23:09:33.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.13744594600000001 2024-05-15T23:09:33.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:09:33.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.137833+0000", 2024-05-15T23:09:33.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011697900000000001 2024-05-15T23:09:33.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 6,9,10", 2024-05-15T23:09:33.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.137916+0000", 2024-05-15T23:09:33.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3047000000000006e-05 2024-05-15T23:09:33.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.142494+0000", 2024-05-15T23:09:33.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045779380000000001 2024-05-15T23:09:33.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:09:33.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.142508+0000", 2024-05-15T23:09:33.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4253000000000001e-05 2024-05-15T23:09:33.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.142551+0000", 2024-05-15T23:09:33.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2967999999999999e-05 2024-05-15T23:09:33.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:09:33.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.161645+0000", 2024-05-15T23:09:33.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019094406000000001 2024-05-15T23:09:33.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:09:33.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.161707+0000", 2024-05-15T23:09:33.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1253000000000005e-05 2024-05-15T23:09:33.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:09:33.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:09:33.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:09:33.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:09:06.161819+0000", 2024-05-15T23:09:33.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112079 2024-05-15T23:09:33.581 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.581 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.581 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.581 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:09:33.581 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:09:33.581 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:09:34.325 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:09:34.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:34 smithi088 ceph-mon[44699]: pgmap v520: 17 pgs: 17 active+clean; 2.5 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 631 op/s; 21 MiB/s, 329 objects/s recovering 2024-05-15T23:09:34.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:34 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1056075972' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:09:34.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:34 smithi084 ceph-mon[39784]: pgmap v520: 17 pgs: 17 active+clean; 2.5 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 631 op/s; 21 MiB/s, 329 objects/s recovering 2024-05-15T23:09:34.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:34 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1056075972' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:09:34.686 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:09:34.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:34.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:09:34.795 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:09:34.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:34 smithi090 ceph-mon[44434]: pgmap v520: 17 pgs: 17 active+clean; 2.5 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 631 op/s; 21 MiB/s, 329 objects/s recovering 2024-05-15T23:09:34.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:34 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1056075972' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:09:34.825 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:34.851 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:09:34.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:34.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:09:34.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:34.983 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:35.009 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11] out_osds: [9] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:09:35.010 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:09:35.010 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11] 2024-05-15T23:09:35.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd out 10 2024-05-15T23:09:35.015 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:35.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:35.019 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:35.119 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:09:35.150 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:35.169 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:09:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:35.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:09:35.247 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T23:09:35.272 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:35.302 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:35.322 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:35.324 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:35.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:35 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:09:35.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:35 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:09:35.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:35 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1056075972' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:09:35.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:35 smithi084 ceph-mon[39784]: osdmap e174: 12 total, 12 up, 11 in 2024-05-15T23:09:35.425 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:09:35.451 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:35 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:09:35.451 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:35 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:09:35.451 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:35 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1056075972' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:09:35.452 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:35 smithi088 ceph-mon[44699]: osdmap e174: 12 total, 12 up, 11 in 2024-05-15T23:09:35.453 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:35.473 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:09:35.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:35.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:09:35.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:35 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:09:35.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:35 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:09:35.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:35 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1056075972' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:09:35.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:35 smithi090 ceph-mon[44434]: osdmap e174: 12 total, 12 up, 11 in 2024-05-15T23:09:35.577 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:09:35.609 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:35.630 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:09:35.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:35.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:09:35.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:09:35.801 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:35.813 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:35.855 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:09:35.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:35.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:09:35.892 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:35.961 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:09:35.989 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:36.012 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:09:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:36.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:09:36.117 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:09:36.165 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:36.391 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:09:36.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:36.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:09:36.434 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:36 smithi084 ceph-mon[39784]: pgmap v522: 17 pgs: 17 active+clean; 2.4 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 790 op/s; 19 MiB/s, 302 objects/s recovering 2024-05-15T23:09:36.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:09:36.527 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:09:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:36.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:09:36.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:09:36.683 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:36 smithi090 ceph-mon[44434]: pgmap v522: 17 pgs: 17 active+clean; 2.4 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 790 op/s; 19 MiB/s, 302 objects/s recovering 2024-05-15T23:09:36.685 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:36.722 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:09:36.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:36.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:09:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:36 smithi088 ceph-mon[44699]: pgmap v522: 17 pgs: 17 active+clean; 2.4 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 790 op/s; 19 MiB/s, 302 objects/s recovering 2024-05-15T23:09:36.827 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:09:36.885 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:36.990 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:09:37.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:37.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:09:37.101 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:09:37.129 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:37.151 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:09:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:37.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:09:37.258 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:09:37.317 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:37.330 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:37.804 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:09:37.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:37.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:09:37.853 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:09:37.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:09:37.947 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:37.971 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:09:37.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:37.976 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:09:38.077 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:09:38.163 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:38.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:38 smithi084 ceph-mon[39784]: pgmap v523: 17 pgs: 17 active+clean; 2.2 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s; 15 MiB/s, 241 objects/s recovering 2024-05-15T23:09:38.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:38 smithi084 ceph-mon[39784]: 2.1 scrub starts 2024-05-15T23:09:38.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:38 smithi084 ceph-mon[39784]: 2.1 scrub ok 2024-05-15T23:09:38.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:09:38.788 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:09:38.788 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:09:38.788 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:09:38.788 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:09:38.788 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:09:38.788 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:09:38.788 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:09:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:38 smithi090 ceph-mon[44434]: pgmap v523: 17 pgs: 17 active+clean; 2.2 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s; 15 MiB/s, 241 objects/s recovering 2024-05-15T23:09:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:38 smithi090 ceph-mon[44434]: 2.1 scrub starts 2024-05-15T23:09:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:38 smithi090 ceph-mon[44434]: 2.1 scrub ok 2024-05-15T23:09:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:09:38.810 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:09:38.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:38 smithi088 ceph-mon[44699]: pgmap v523: 17 pgs: 17 active+clean; 2.2 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s; 15 MiB/s, 241 objects/s recovering 2024-05-15T23:09:38.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:38 smithi088 ceph-mon[44699]: 2.1 scrub starts 2024-05-15T23:09:38.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:38 smithi088 ceph-mon[44699]: 2.1 scrub ok 2024-05-15T23:09:38.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:38 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:09:38.838 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:09:38.872 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:09:38.903 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:09:38.928 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:09:38.957 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:09:38.958 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:09:38.958 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:09:38.958 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:09:38.958 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:09:38.958 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:09:38.958 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:09:38.958 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:09:38.958 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:09:38.958 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:09:38.958 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:09:38.958 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:09:39.183 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:09:39.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:39.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:09:39.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:09:39.328 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:39.350 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:09:39.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:39.352 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:09:39.453 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:09:39.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1823096729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:09:39.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:09:39.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:09:39.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:09:39.511 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:39.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1823096729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:09:39.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:09:39.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:09:39.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:09:39.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1823096729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:09:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:39 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:09:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:39 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:09:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:39 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:09:39.893 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:40.580 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:09:40.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:40.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:09:40.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:40.722 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:40.722 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:40 smithi090 ceph-mon[44434]: pgmap v524: 17 pgs: 17 active+clean; 2.2 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 824 op/s; 6.8 MiB/s, 108 objects/s recovering 2024-05-15T23:09:40.722 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:40.722 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:40.722 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:40 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:09:40.723 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:40 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:09:40.723 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:40.723 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:40 smithi090 ceph-mon[44434]: osdmap e175: 12 total, 12 up, 11 in 2024-05-15T23:09:40.724 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:40.750 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:40.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:40.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:40 smithi088 ceph-mon[44699]: pgmap v524: 17 pgs: 17 active+clean; 2.2 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 824 op/s; 6.8 MiB/s, 108 objects/s recovering 2024-05-15T23:09:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:40 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:09:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:40 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:09:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:40 smithi088 ceph-mon[44699]: osdmap e175: 12 total, 12 up, 11 in 2024-05-15T23:09:40.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:09:40.881 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:40.881 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:40 smithi084 ceph-mon[39784]: pgmap v524: 17 pgs: 17 active+clean; 2.2 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 824 op/s; 6.8 MiB/s, 108 objects/s recovering 2024-05-15T23:09:40.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:40.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:40.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:40 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:09:40.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:40 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:09:40.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:40.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:40 smithi084 ceph-mon[39784]: osdmap e175: 12 total, 12 up, 11 in 2024-05-15T23:09:40.895 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:40.930 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:09:40.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:40.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:09:41.033 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:09:41.063 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:41.085 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:09:41.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:41.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:09:41.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:09:41.218 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:41.248 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:09:41.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:41.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:09:41.351 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:09:41.455 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:41.477 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:09:41.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:41.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:09:41.584 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:09:41.611 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:41.630 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:09:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:41.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:09:41.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:09:41.822 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:41.912 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:09:41.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:41.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:09:42.023 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:09:42.080 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:41 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:09:41.851+0000 7fa0938b7700 -1 mon.a@0(leader).osd e175 definitely_dead 0 2024-05-15T23:09:42.086 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:42.759 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.10. 2024-05-15T23:09:42.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:42 smithi090 ceph-mon[44434]: pgmap v526: 17 pgs: 17 active+clean; 1.9 GiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:09:42.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:42 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3016904017' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-05-15T23:09:42.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:42 smithi090 ceph-mon[44434]: from='client.25018 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-05-15T23:09:42.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:42 smithi090 ceph-mon[44434]: Client client.admin marked osd.10 out, while it was still marked up 2024-05-15T23:09:42.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:42 smithi090 ceph-mon[44434]: 2.8 scrub starts 2024-05-15T23:09:42.803 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:42 smithi084 ceph-mon[39784]: pgmap v526: 17 pgs: 17 active+clean; 1.9 GiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:09:42.803 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:42 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3016904017' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-05-15T23:09:42.803 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:42 smithi084 ceph-mon[39784]: from='client.25018 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-05-15T23:09:42.803 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:42 smithi084 ceph-mon[39784]: Client client.admin marked osd.10 out, while it was still marked up 2024-05-15T23:09:42.803 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:42 smithi084 ceph-mon[39784]: 2.8 scrub starts 2024-05-15T23:09:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:42 smithi088 ceph-mon[44699]: pgmap v526: 17 pgs: 17 active+clean; 1.9 GiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:09:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:42 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3016904017' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-05-15T23:09:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:42 smithi088 ceph-mon[44699]: from='client.25018 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-05-15T23:09:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:42 smithi088 ceph-mon[44699]: Client client.admin marked osd.10 out, while it was still marked up 2024-05-15T23:09:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:42 smithi088 ceph-mon[44699]: 2.8 scrub starts 2024-05-15T23:09:42.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:09:43.567 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:09:43.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:43.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:09:43.672 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:09:43.702 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:43.722 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:09:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:43.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:09:43.826 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:09:43.854 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:43 smithi084 ceph-mon[39784]: 2.8 scrub ok 2024-05-15T23:09:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:43 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:09:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:43 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:09:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:43 smithi084 ceph-mon[39784]: pgmap v527: 17 pgs: 17 active+clean; 1.8 GiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.29k op/s 2024-05-15T23:09:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:43 smithi084 ceph-mon[39784]: from='client.25018 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-05-15T23:09:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:43 smithi084 ceph-mon[39784]: osdmap e176: 12 total, 12 up, 10 in 2024-05-15T23:09:43.873 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:09:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:43.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:09:43.971 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:09:43.971 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:09:43.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:09:44.007 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:43 smithi090 ceph-mon[44434]: 2.8 scrub ok 2024-05-15T23:09:44.007 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:43 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:09:44.007 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:43 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:09:44.007 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:43 smithi090 ceph-mon[44434]: pgmap v527: 17 pgs: 17 active+clean; 1.8 GiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.29k op/s 2024-05-15T23:09:44.007 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:43 smithi090 ceph-mon[44434]: from='client.25018 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-05-15T23:09:44.007 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:43 smithi090 ceph-mon[44434]: osdmap e176: 12 total, 12 up, 10 in 2024-05-15T23:09:44.009 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:44.031 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:09:44.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:44.034 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:09:44.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:43 smithi088 ceph-mon[44699]: 2.8 scrub ok 2024-05-15T23:09:44.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:43 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:09:44.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:43 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:09:44.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:43 smithi088 ceph-mon[44699]: pgmap v527: 17 pgs: 17 active+clean; 1.8 GiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.29k op/s 2024-05-15T23:09:44.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:43 smithi088 ceph-mon[44699]: from='client.25018 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-05-15T23:09:44.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:43 smithi088 ceph-mon[44699]: osdmap e176: 12 total, 12 up, 10 in 2024-05-15T23:09:44.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:09:44.165 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:44.185 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:09:44.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:44.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:09:44.289 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:09:44.318 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:44.735 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:44.761 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:09:44.766 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:09:44.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T23:09:44.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:44.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:09:44.871 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:09:44.897 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:44.970 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:09:44.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:44.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:09:44.995 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:44 smithi084 ceph-mon[39784]: osdmap e177: 12 total, 12 up, 10 in 2024-05-15T23:09:44.996 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3840533571' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:09:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:44 smithi090 ceph-mon[44434]: osdmap e177: 12 total, 12 up, 10 in 2024-05-15T23:09:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3840533571' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:09:45.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:44 smithi088 ceph-mon[44699]: osdmap e177: 12 total, 12 up, 10 in 2024-05-15T23:09:45.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3840533571' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:09:45.073 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:09:45.099 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:45.199 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:09:45.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:45.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:09:45.311 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:09:45.341 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:45.364 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:09:45.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:45.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:09:45.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:09:45.506 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:45.528 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:09:45.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:45.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:09:45.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:45.683 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:45.711 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:45.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:45.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:45.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:45 smithi090 ceph-mon[44434]: pgmap v530: 17 pgs: 17 active+clean; 1.8 GiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-05-15T23:09:45.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:45 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:09:45.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3840533571' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:09:45.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:45 smithi090 ceph-mon[44434]: 2.7 starting backfill to osd.2 from (0'0,0'0] MAX to 175'18565 2024-05-15T23:09:45.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:45 smithi090 ceph-mon[44434]: osdmap e178: 12 total, 12 up, 10 in 2024-05-15T23:09:45.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:45 smithi090 ceph-mon[44434]: 2.0 starting backfill to osd.11 from (0'0,0'0] MAX to 175'18671 2024-05-15T23:09:45.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:09:45.888 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:46.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:45 smithi088 ceph-mon[44699]: pgmap v530: 17 pgs: 17 active+clean; 1.8 GiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-05-15T23:09:46.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:45 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:09:46.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3840533571' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:09:46.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:45 smithi088 ceph-mon[44699]: 2.7 starting backfill to osd.2 from (0'0,0'0] MAX to 175'18565 2024-05-15T23:09:46.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:45 smithi088 ceph-mon[44699]: osdmap e178: 12 total, 12 up, 10 in 2024-05-15T23:09:46.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:45 smithi088 ceph-mon[44699]: 2.0 starting backfill to osd.11 from (0'0,0'0] MAX to 175'18671 2024-05-15T23:09:46.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:45 smithi084 ceph-mon[39784]: pgmap v530: 17 pgs: 17 active+clean; 1.8 GiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-05-15T23:09:46.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:45 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:09:46.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3840533571' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:09:46.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:45 smithi084 ceph-mon[39784]: 2.7 starting backfill to osd.2 from (0'0,0'0] MAX to 175'18565 2024-05-15T23:09:46.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:45 smithi084 ceph-mon[39784]: osdmap e178: 12 total, 12 up, 10 in 2024-05-15T23:09:46.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:45 smithi084 ceph-mon[39784]: 2.0 starting backfill to osd.11 from (0'0,0'0] MAX to 175'18671 2024-05-15T23:09:46.255 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:09:46.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:46.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:09:46.365 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:09:46.415 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:47.374 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:09:47.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:47.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:09:47.461 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:47.487 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:09:47.520 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:47.540 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:09:47.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:47.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:09:47.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:47.674 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:47.716 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:47.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:47.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:47.779 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:47 smithi088 ceph-mon[44699]: pgmap v532: 17 pgs: 1 activating+remapped, 16 active+clean; 1.7 GiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 760 op/s; 1763/54610 objects misplaced (3.228%) 2024-05-15T23:09:47.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:09:47.850 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:09:47.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:47.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:09:47.976 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:09:48.030 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:48.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:47 smithi090 ceph-mon[44434]: pgmap v532: 17 pgs: 1 activating+remapped, 16 active+clean; 1.7 GiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 760 op/s; 1763/54610 objects misplaced (3.228%) 2024-05-15T23:09:48.064 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:09:48.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:48.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:09:48.077 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:47 smithi084 ceph-mon[39784]: pgmap v532: 17 pgs: 1 activating+remapped, 16 active+clean; 1.7 GiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 760 op/s; 1763/54610 objects misplaced (3.228%) 2024-05-15T23:09:48.144 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:09:48.172 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:09:48.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:09:48.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:09:48.215 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:48.242 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:09:48.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:48.247 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:09:48.254 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:09:48.290 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:09:48.332 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:09:48.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:09:48.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:09:48.398 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:09:48.405 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:48.432 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:09:48.451 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:09:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:48.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:09:48.461 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:09:48.499 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:09:48.537 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:09:48.547 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:09:48.547 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:09:48.547 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:09:48.547 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:09:48.547 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:09:48.547 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:09:48.547 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:09:48.547 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:09:48.548 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:09:48.548 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:09:48.548 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:09:48.548 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:09:48.559 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:09:48.626 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:48.725 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:09:48.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:48.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:09:48.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:09:48.885 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:48.914 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:09:48.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:48.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:09:49.022 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:09:49.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2873786498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:09:49.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.062 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2873786498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:09:49.062 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.062 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.062 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.062 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.062 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.062 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2873786498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:09:49.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:49.071 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:49.432 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:09:49.552 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:09:49.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:49.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:09:49.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:09:49.685 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:49.712 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:09:49.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:49.720 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:09:49.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:09:49.852 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:49.880 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:09:49.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:49.882 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:09:49.983 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:09:50.011 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:50.032 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:09:50.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:50.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:09:50.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:49 smithi090 ceph-mon[44434]: pgmap v533: 17 pgs: 1 active+remapped+backfilling, 1 activating+remapped, 15 active+clean; 1.6 GiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 576 op/s; 3070/53460 objects misplaced (5.743%); 3.0 MiB/s, 48 objects/s recovering 2024-05-15T23:09:50.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:49 smithi088 ceph-mon[44699]: pgmap v533: 17 pgs: 1 active+remapped+backfilling, 1 activating+remapped, 15 active+clean; 1.6 GiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 576 op/s; 3070/53460 objects misplaced (5.743%); 3.0 MiB/s, 48 objects/s recovering 2024-05-15T23:09:50.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:09:50.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:49 smithi084 ceph-mon[39784]: pgmap v533: 17 pgs: 1 active+remapped+backfilling, 1 activating+remapped, 15 active+clean; 1.6 GiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 576 op/s; 3070/53460 objects misplaced (5.743%); 3.0 MiB/s, 48 objects/s recovering 2024-05-15T23:09:50.136 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:09:50.201 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:50.374 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:09:50.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:50.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:09:50.408 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:50.482 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:09:50.510 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:50.531 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:09:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:50.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:09:50.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:50.670 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:50.702 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:50.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:50.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:09:50.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:50 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:09:50.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:50 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:09:50.875 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:50.940 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:09:50.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:50.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:09:50.960 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:50 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:09:50.960 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:50 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:09:51.049 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:51.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:50 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:09:51.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:50 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:09:51.075 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:51.095 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:51.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:51.099 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:51.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:09:51.221 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:09:51.221 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:09:51.221 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:09:51.221 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:09:51.222 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:09:51.225 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:51.243 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:09:51.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:51.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:09:51.273 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 11] out_osds: [9, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:09:51.274 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:09:51.274 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-05-15T23:09:51.274 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T23:09:51.274 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T23:09:51.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 config set filestore_inject_stall 3 2024-05-15T23:09:51.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:09:51.372 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:51.390 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:09:51.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:51.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:09:51.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:09:51.521 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:51.544 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:09:51.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:51.547 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:09:51.555 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:09:51.648 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:09:51.675 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:51.695 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:09:51.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:51.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:09:51.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:09:51.830 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:51.850 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:09:51.852 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:51 smithi084 ceph-mon[39784]: pgmap v534: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 1.4 GiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 870 op/s; 1328/46484 objects misplaced (2.857%); 14 MiB/s, 228 objects/s recovering 2024-05-15T23:09:51.852 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:51 smithi084 ceph-mon[39784]: osdmap e179: 12 total, 12 up, 10 in 2024-05-15T23:09:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:51.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:09:51.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:09:51.988 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:52.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:51 smithi088 ceph-mon[44699]: pgmap v534: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 1.4 GiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 870 op/s; 1328/46484 objects misplaced (2.857%); 14 MiB/s, 228 objects/s recovering 2024-05-15T23:09:52.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:51 smithi088 ceph-mon[44699]: osdmap e179: 12 total, 12 up, 10 in 2024-05-15T23:09:52.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:51 smithi090 ceph-mon[44434]: pgmap v534: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 1.4 GiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 870 op/s; 1328/46484 objects misplaced (2.857%); 14 MiB/s, 228 objects/s recovering 2024-05-15T23:09:52.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:51 smithi090 ceph-mon[44434]: osdmap e179: 12 total, 12 up, 10 in 2024-05-15T23:09:52.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:09:52.416 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T23:09:53.049 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:53.079 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:09:53.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:52 smithi084 ceph-mon[39784]: osdmap e180: 12 total, 12 up, 10 in 2024-05-15T23:09:53.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:52 smithi090 ceph-mon[44434]: osdmap e180: 12 total, 12 up, 10 in 2024-05-15T23:09:53.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:52 smithi088 ceph-mon[44699]: osdmap e180: 12 total, 12 up, 10 in 2024-05-15T23:09:53.843 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:09:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:53.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:09:53.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:09:53.981 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:54.035 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:09:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:54.039 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:09:54.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:53 smithi090 ceph-mon[44434]: pgmap v537: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 1.3 GiB data, 3.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 919 op/s; 83/43692 objects misplaced (0.190%); 23 MiB/s, 361 objects/s recovering 2024-05-15T23:09:54.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:53 smithi084 ceph-mon[39784]: pgmap v537: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 1.3 GiB data, 3.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 919 op/s; 83/43692 objects misplaced (0.190%); 23 MiB/s, 361 objects/s recovering 2024-05-15T23:09:54.140 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:09:54.172 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:09:54.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:53 smithi088 ceph-mon[44699]: pgmap v537: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 1.3 GiB data, 3.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 919 op/s; 83/43692 objects misplaced (0.190%); 23 MiB/s, 361 objects/s recovering 2024-05-15T23:09:54.874 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:09:55.109 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:54 smithi090 ceph-mon[44434]: osdmap e181: 12 total, 12 up, 10 in 2024-05-15T23:09:55.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/74912295' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:09:55.110 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:54 smithi090 ceph-mon[44434]: from='client.35090 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:09:55.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:54 smithi084 ceph-mon[39784]: osdmap e181: 12 total, 12 up, 10 in 2024-05-15T23:09:55.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/74912295' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:09:55.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:54 smithi084 ceph-mon[39784]: from='client.35090 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:09:55.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:54 smithi088 ceph-mon[44699]: osdmap e181: 12 total, 12 up, 10 in 2024-05-15T23:09:55.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/74912295' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:09:55.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:54 smithi088 ceph-mon[44699]: from='client.35090 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:09:55.465 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:09:55.465 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "" 2024-05-15T23:09:55.465 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:09:56.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:55 smithi084 ceph-mon[39784]: pgmap v539: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 1.3 GiB data, 3.4 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 971 op/s; 83/41800 objects misplaced (0.199%); 27 MiB/s, 432 objects/s recovering 2024-05-15T23:09:56.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:55 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:09:56.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:55 smithi084 ceph-mon[39784]: from='client.35090 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:09:56.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:55 smithi084 ceph-mon[39784]: osdmap e182: 12 total, 12 up, 10 in 2024-05-15T23:09:56.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:55 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:09:56.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:55 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:09:56.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:55 smithi090 ceph-mon[44434]: pgmap v539: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 1.3 GiB data, 3.4 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 971 op/s; 83/41800 objects misplaced (0.199%); 27 MiB/s, 432 objects/s recovering 2024-05-15T23:09:56.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:55 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:09:56.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:55 smithi090 ceph-mon[44434]: from='client.35090 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:09:56.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:55 smithi090 ceph-mon[44434]: osdmap e182: 12 total, 12 up, 10 in 2024-05-15T23:09:56.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:55 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:09:56.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:55 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:09:56.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:55 smithi088 ceph-mon[44699]: pgmap v539: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 1.3 GiB data, 3.4 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 971 op/s; 83/41800 objects misplaced (0.199%); 27 MiB/s, 432 objects/s recovering 2024-05-15T23:09:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:55 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:09:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:55 smithi088 ceph-mon[44699]: from='client.35090 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:09:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:55 smithi088 ceph-mon[44699]: osdmap e182: 12 total, 12 up, 10 in 2024-05-15T23:09:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:55 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:09:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:55 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:09:57.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:56 smithi084 ceph-mon[39784]: osdmap e183: 12 total, 12 up, 10 in 2024-05-15T23:09:57.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:56 smithi090 ceph-mon[44434]: osdmap e183: 12 total, 12 up, 10 in 2024-05-15T23:09:57.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:56 smithi088 ceph-mon[44699]: osdmap e183: 12 total, 12 up, 10 in 2024-05-15T23:09:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:57 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:09:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:57 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:09:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:57 smithi090 ceph-mon[44434]: pgmap v542: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.1 GiB data, 3.2 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 802 op/s; 62/37450 objects misplaced (0.166%); 14 MiB/s, 219 objects/s recovering 2024-05-15T23:09:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:57 smithi090 ceph-mon[44434]: osdmap e184: 12 total, 12 up, 10 in 2024-05-15T23:09:58.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:57 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:09:58.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:57 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:09:58.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:57 smithi088 ceph-mon[44699]: pgmap v542: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.1 GiB data, 3.2 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 802 op/s; 62/37450 objects misplaced (0.166%); 14 MiB/s, 219 objects/s recovering 2024-05-15T23:09:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:57 smithi088 ceph-mon[44699]: osdmap e184: 12 total, 12 up, 10 in 2024-05-15T23:09:58.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:57 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:09:58.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:57 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:09:58.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:57 smithi084 ceph-mon[39784]: pgmap v542: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.1 GiB data, 3.2 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 802 op/s; 62/37450 objects misplaced (0.166%); 14 MiB/s, 219 objects/s recovering 2024-05-15T23:09:58.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:57 smithi084 ceph-mon[39784]: osdmap e184: 12 total, 12 up, 10 in 2024-05-15T23:09:59.113 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:09:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:09:59.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:09:59.218 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:09:59.262 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:00.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:09:59 smithi084 ceph-mon[39784]: pgmap v544: 17 pgs: 17 active+clean; 1.1 GiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 819 op/s; 1.0 MiB/s, 16 objects/s recovering 2024-05-15T23:10:00.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:09:59 smithi090 ceph-mon[44434]: pgmap v544: 17 pgs: 17 active+clean; 1.1 GiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 819 op/s; 1.0 MiB/s, 16 objects/s recovering 2024-05-15T23:10:00.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:09:59 smithi088 ceph-mon[44699]: pgmap v544: 17 pgs: 17 active+clean; 1.1 GiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 819 op/s; 1.0 MiB/s, 16 objects/s recovering 2024-05-15T23:10:00.752 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:00.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:00.859 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:10:00.914 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:01.046 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:10:01.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:01.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:10:01.057 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:00 smithi084 ceph-mon[39784]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-05-15T23:10:01.057 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:00 smithi084 ceph-mon[39784]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-05-15T23:10:01.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:01.211 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:01.212 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 11] out_osds: [9, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:10:01.212 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:10:01.213 INFO:tasks.thrashosds.thrasher:Adding osd 9 2024-05-15T23:10:01.213 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd in 9 2024-05-15T23:10:01.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:00 smithi090 ceph-mon[44434]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-05-15T23:10:01.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:00 smithi090 ceph-mon[44434]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-05-15T23:10:01.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:00 smithi088 ceph-mon[44699]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-05-15T23:10:01.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:00 smithi088 ceph-mon[44699]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-05-15T23:10:01.324 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:01.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:01.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:01.462 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:01.485 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:01.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:01.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:01.590 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:10:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:01.642 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:10:01.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:01.650 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:10:01.751 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:10:01.781 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:01.806 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:10:01.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:01.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:10:01.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:01.942 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:01.969 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:01.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:01.973 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:02.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:01 smithi088 ceph-mon[44699]: pgmap v545: 17 pgs: 17 active+clean; 873 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s; 841 KiB/s, 13 objects/s recovering 2024-05-15T23:10:02.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:01 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3252630243' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:10:02.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:02.101 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:02.128 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:02.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:02.131 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:02.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:10:02.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:10:02.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:10:02.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:10:02.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:10:02.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:10:02.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:10:02.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:10:02.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:10:02.230 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:10:02.232 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:10:02.259 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:02.270 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:10:02.288 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:10:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:02.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:10:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:01 smithi090 ceph-mon[44434]: pgmap v545: 17 pgs: 17 active+clean; 873 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s; 841 KiB/s, 13 objects/s recovering 2024-05-15T23:10:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:01 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3252630243' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:10:02.306 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:10:02.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:10:02.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:10:02.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:10:02.323 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:10:02.323 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:10:02.323 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:10:02.323 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:10:02.323 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:10:02.323 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:10:02.323 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:10:02.323 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:10:02.323 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:10:02.359 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:01 smithi084 ceph-mon[39784]: pgmap v545: 17 pgs: 17 active+clean; 873 MiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s; 841 KiB/s, 13 objects/s recovering 2024-05-15T23:10:02.359 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:01 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3252630243' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:10:02.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:02.418 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:02.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:02.447 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:02.548 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:10:02.607 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:02.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:10:02.876 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:10:02.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:02.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:10:02.987 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:10:03.015 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:03.034 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:10:03.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:03.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:10:03.089 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:10:03.089 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:10:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:02 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:10:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:02 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:10:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:02 smithi084 ceph-mon[39784]: osdmap e185: 12 total, 12 up, 10 in 2024-05-15T23:10:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.139 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:10:03.170 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:03.190 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:10:03.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:03.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:10:03.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:02 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:10:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:02 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:10:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:02 smithi090 ceph-mon[44434]: osdmap e185: 12 total, 12 up, 10 in 2024-05-15T23:10:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:02 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:10:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:02 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:10:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:02 smithi088 ceph-mon[44699]: osdmap e185: 12 total, 12 up, 10 in 2024-05-15T23:10:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:03.329 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:03.357 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:03.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:03.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:03.497 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:03.518 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:03.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:03.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:03.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:03.675 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:03.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:03.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:03.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:03.813 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:03.835 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:03.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:03.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:03.858 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:03.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:03.969 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:03.992 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:03.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:03.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:04.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:03 smithi084 ceph-mon[39784]: pgmap v547: 17 pgs: 17 active+clean; 757 MiB data, 2.5 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s; 550 KiB/s, 8 objects/s recovering 2024-05-15T23:10:04.129 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:04.154 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:04.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:04.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:04.288 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:04.289 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:04.289 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:04.289 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:03 smithi090 ceph-mon[44434]: pgmap v547: 17 pgs: 17 active+clean; 757 MiB data, 2.5 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s; 550 KiB/s, 8 objects/s recovering 2024-05-15T23:10:04.290 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:04.313 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:04.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:04.318 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:04.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:04.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:03 smithi088 ceph-mon[44699]: pgmap v547: 17 pgs: 17 active+clean; 757 MiB data, 2.5 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s; 550 KiB/s, 8 objects/s recovering 2024-05-15T23:10:04.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:04.448 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:04.466 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:04.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:04.469 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:04.494 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T23:10:04.571 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:10:04.636 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:04.746 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:04.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:04.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:04.859 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:10:04.885 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:04.906 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:10:04.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:04.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:10:05.011 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:05.043 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:05.067 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:05.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:05.070 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:05.171 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:05.238 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:05.352 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:05.365 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:05.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:05.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:05.407 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:05.469 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:10:05.508 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:05.563 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:05.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:05.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:05.666 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:05.741 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:05.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:10:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:06 smithi090 ceph-mon[44434]: pgmap v548: 17 pgs: 17 active+clean; 669 MiB data, 2.3 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s; 471 KiB/s, 7 objects/s recovering 2024-05-15T23:10:06.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:06 smithi088 ceph-mon[44699]: pgmap v548: 17 pgs: 17 active+clean; 669 MiB data, 2.3 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s; 471 KiB/s, 7 objects/s recovering 2024-05-15T23:10:06.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:06 smithi084 ceph-mon[39784]: pgmap v548: 17 pgs: 17 active+clean; 669 MiB data, 2.3 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s; 471 KiB/s, 7 objects/s recovering 2024-05-15T23:10:07.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:07 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:10:07.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:07 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:10:07.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:07 smithi090 ceph-mon[44434]: osdmap e186: 12 total, 12 up, 10 in 2024-05-15T23:10:07.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:07 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:10:07.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:07 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:10:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:07 smithi088 ceph-mon[44699]: osdmap e186: 12 total, 12 up, 10 in 2024-05-15T23:10:07.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:07 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:10:07.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:07 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:10:07.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:07 smithi084 ceph-mon[39784]: osdmap e186: 12 total, 12 up, 10 in 2024-05-15T23:10:08.144 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:08.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:08.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:08.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:08 smithi084 ceph-mon[39784]: pgmap v550: 17 pgs: 17 active+clean; 482 MiB data, 1.9 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:10:08.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:10:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:08 smithi090 ceph-mon[44434]: pgmap v550: 17 pgs: 17 active+clean; 482 MiB data, 1.9 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:10:08.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:08 smithi088 ceph-mon[44699]: pgmap v550: 17 pgs: 17 active+clean; 482 MiB data, 1.9 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:10:08.324 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:08.789 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:10:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:08.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:10:08.897 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:10:08.925 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:08.947 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:10:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:08.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:10:09.051 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:10:09.080 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:10:09.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:09.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:10:09.211 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:10:09.241 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:09.265 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:10:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:09.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:10:09.370 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:10:09.398 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:09.419 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:10:09.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:09.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:10:09.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:10:09.553 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:09.576 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:10:09.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:09.579 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:10:09.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:09.733 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:09.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:09.738 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:09.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:09.875 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:09.898 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:09.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:10.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:10:10.029 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:10.032 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:10.051 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:10:10.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:10.055 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:10:10.071 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:10:10.075 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.9. 2024-05-15T23:10:10.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:10.179 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:10 smithi090 ceph-mon[44434]: pgmap v551: 17 pgs: 17 active+clean; 417 MiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 909 op/s 2024-05-15T23:10:10.179 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:10 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2957569675' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:10:10.180 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:10 smithi090 ceph-mon[44434]: from='client.35102 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:10:10.180 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:10 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1304393543' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-05-15T23:10:10.180 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:10 smithi090 ceph-mon[44434]: from='client.35108 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-05-15T23:10:10.181 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:10.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:10.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:10.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:10.329 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:10.347 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:10.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:10.349 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:10.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:09 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:10:09.867+0000 7fa0938b7700 -1 mon.a@0(leader).osd e186 definitely_dead 0 2024-05-15T23:10:10.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:10 smithi084 ceph-mon[39784]: pgmap v551: 17 pgs: 17 active+clean; 417 MiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 909 op/s 2024-05-15T23:10:10.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:10 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2957569675' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:10:10.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:10 smithi084 ceph-mon[39784]: from='client.35102 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:10:10.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:10 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1304393543' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-05-15T23:10:10.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:10 smithi084 ceph-mon[39784]: from='client.35108 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-05-15T23:10:10.450 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:10.476 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:10.495 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:10.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:10.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:10.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:10 smithi088 ceph-mon[44699]: pgmap v551: 17 pgs: 17 active+clean; 417 MiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 909 op/s 2024-05-15T23:10:10.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:10 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2957569675' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:10:10.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:10 smithi088 ceph-mon[44699]: from='client.35102 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:10:10.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:10 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1304393543' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-05-15T23:10:10.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:10 smithi088 ceph-mon[44699]: from='client.35108 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-05-15T23:10:10.598 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:10:10.626 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:10.645 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:10:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:10.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:10:10.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:10.775 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:10.794 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:10.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:10.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:10.898 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:10:10.925 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:10.944 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:10:10.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:10.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:10:11.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:10:11.075 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:11.094 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:10:11.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:11.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:10:11.198 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:10:11.222 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:11 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:10:11.222 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:11 smithi088 ceph-mon[44699]: from='client.35102 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:10:11.223 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:11 smithi088 ceph-mon[44699]: from='client.35108 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-05-15T23:10:11.223 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:11 smithi088 ceph-mon[44699]: osdmap e187: 12 total, 12 up, 11 in 2024-05-15T23:10:11.223 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:11 smithi088 ceph-mon[44699]: 2.6 scrub starts 2024-05-15T23:10:11.223 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:11 smithi088 ceph-mon[44699]: 2.6 scrub ok 2024-05-15T23:10:11.223 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:11 smithi088 ceph-mon[44699]: osdmap e188: 12 total, 12 up, 11 in 2024-05-15T23:10:11.224 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:11.242 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:10:11.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:11.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:10:11.259 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:10:11.259 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:10:11.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:11 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:10:11.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:11 smithi084 ceph-mon[39784]: from='client.35102 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:10:11.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:11 smithi084 ceph-mon[39784]: from='client.35108 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-05-15T23:10:11.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:11 smithi084 ceph-mon[39784]: osdmap e187: 12 total, 12 up, 11 in 2024-05-15T23:10:11.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:11 smithi084 ceph-mon[39784]: 2.6 scrub starts 2024-05-15T23:10:11.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:11 smithi084 ceph-mon[39784]: 2.6 scrub ok 2024-05-15T23:10:11.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:11 smithi084 ceph-mon[39784]: osdmap e188: 12 total, 12 up, 11 in 2024-05-15T23:10:11.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:11.380 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:11.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:11 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:10:11.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:11 smithi090 ceph-mon[44434]: from='client.35102 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:10:11.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:11 smithi090 ceph-mon[44434]: from='client.35108 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-05-15T23:10:11.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:11 smithi090 ceph-mon[44434]: osdmap e187: 12 total, 12 up, 11 in 2024-05-15T23:10:11.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:11 smithi090 ceph-mon[44434]: 2.6 scrub starts 2024-05-15T23:10:11.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:11 smithi090 ceph-mon[44434]: 2.6 scrub ok 2024-05-15T23:10:11.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:11 smithi090 ceph-mon[44434]: osdmap e188: 12 total, 12 up, 11 in 2024-05-15T23:10:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:12 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:10:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:12 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:10:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:12 smithi084 ceph-mon[39784]: pgmap v553: 17 pgs: 1 peering, 16 active+clean; 338 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 835 op/s 2024-05-15T23:10:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:12 smithi084 ceph-mon[39784]: osdmap e189: 12 total, 12 up, 11 in 2024-05-15T23:10:12.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:12 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:10:12.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:12 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:10:12.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:12 smithi090 ceph-mon[44434]: pgmap v553: 17 pgs: 1 peering, 16 active+clean; 338 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 835 op/s 2024-05-15T23:10:12.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:12 smithi090 ceph-mon[44434]: osdmap e189: 12 total, 12 up, 11 in 2024-05-15T23:10:12.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:12 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:10:12.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:12 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:10:12.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:12 smithi088 ceph-mon[44699]: pgmap v553: 17 pgs: 1 peering, 16 active+clean; 338 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 835 op/s 2024-05-15T23:10:12.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:12 smithi088 ceph-mon[44699]: osdmap e189: 12 total, 12 up, 11 in 2024-05-15T23:10:12.987 INFO:tasks.thrashosds.thrasher:Added osd 9 2024-05-15T23:10:13.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:13 smithi084 ceph-mon[39784]: 2.c scrub starts 2024-05-15T23:10:13.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:13 smithi084 ceph-mon[39784]: 2.0 starting backfill to osd.9 from (0'0,0'0] MAX to 186'20309 2024-05-15T23:10:13.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:13 smithi084 ceph-mon[39784]: 2.8 starting backfill to osd.9 from (130'7600,170'17687] MIN to 186'20223 2024-05-15T23:10:13.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:13 smithi084 ceph-mon[39784]: 2.7 starting backfill to osd.0 from (0'0,0'0] MAX to 186'20214 2024-05-15T23:10:13.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:13 smithi084 ceph-mon[39784]: 2.d starting backfill to osd.9 from (130'7300,169'17355] MIN to 186'19867 2024-05-15T23:10:13.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:13 smithi084 ceph-mon[39784]: 2.f starting backfill to osd.9 from (128'7200,167'17217] MIN to 186'20306 2024-05-15T23:10:13.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:13 smithi084 ceph-mon[39784]: 2.7 starting backfill to osd.9 from (131'7600,171'17643] MIN to 186'20214 2024-05-15T23:10:13.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:13 smithi084 ceph-mon[39784]: 2.c scrub ok 2024-05-15T23:10:13.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:13 smithi084 ceph-mon[39784]: 2.e scrub starts 2024-05-15T23:10:13.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:13 smithi084 ceph-mon[39784]: 2.e scrub ok 2024-05-15T23:10:13.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:13 smithi084 ceph-mon[39784]: 2.4 scrub starts 2024-05-15T23:10:13.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:13 smithi084 ceph-mon[39784]: 2.4 scrub ok 2024-05-15T23:10:13.430 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:13.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:13.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:13.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:13 smithi090 ceph-mon[44434]: 2.c scrub starts 2024-05-15T23:10:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:13 smithi090 ceph-mon[44434]: 2.0 starting backfill to osd.9 from (0'0,0'0] MAX to 186'20309 2024-05-15T23:10:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:13 smithi090 ceph-mon[44434]: 2.8 starting backfill to osd.9 from (130'7600,170'17687] MIN to 186'20223 2024-05-15T23:10:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:13 smithi090 ceph-mon[44434]: 2.7 starting backfill to osd.0 from (0'0,0'0] MAX to 186'20214 2024-05-15T23:10:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:13 smithi090 ceph-mon[44434]: 2.d starting backfill to osd.9 from (130'7300,169'17355] MIN to 186'19867 2024-05-15T23:10:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:13 smithi090 ceph-mon[44434]: 2.f starting backfill to osd.9 from (128'7200,167'17217] MIN to 186'20306 2024-05-15T23:10:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:13 smithi090 ceph-mon[44434]: 2.7 starting backfill to osd.9 from (131'7600,171'17643] MIN to 186'20214 2024-05-15T23:10:13.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:13 smithi090 ceph-mon[44434]: 2.c scrub ok 2024-05-15T23:10:13.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:13 smithi090 ceph-mon[44434]: 2.e scrub starts 2024-05-15T23:10:13.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:13 smithi090 ceph-mon[44434]: 2.e scrub ok 2024-05-15T23:10:13.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:13 smithi090 ceph-mon[44434]: 2.4 scrub starts 2024-05-15T23:10:13.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:13 smithi090 ceph-mon[44434]: 2.4 scrub ok 2024-05-15T23:10:13.561 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:13.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:13 smithi088 ceph-mon[44699]: 2.c scrub starts 2024-05-15T23:10:13.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:13 smithi088 ceph-mon[44699]: 2.0 starting backfill to osd.9 from (0'0,0'0] MAX to 186'20309 2024-05-15T23:10:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:13 smithi088 ceph-mon[44699]: 2.8 starting backfill to osd.9 from (130'7600,170'17687] MIN to 186'20223 2024-05-15T23:10:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:13 smithi088 ceph-mon[44699]: 2.7 starting backfill to osd.0 from (0'0,0'0] MAX to 186'20214 2024-05-15T23:10:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:13 smithi088 ceph-mon[44699]: 2.d starting backfill to osd.9 from (130'7300,169'17355] MIN to 186'19867 2024-05-15T23:10:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:13 smithi088 ceph-mon[44699]: 2.f starting backfill to osd.9 from (128'7200,167'17217] MIN to 186'20306 2024-05-15T23:10:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:13 smithi088 ceph-mon[44699]: 2.7 starting backfill to osd.9 from (131'7600,171'17643] MIN to 186'20214 2024-05-15T23:10:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:13 smithi088 ceph-mon[44699]: 2.c scrub ok 2024-05-15T23:10:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:13 smithi088 ceph-mon[44699]: 2.e scrub starts 2024-05-15T23:10:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:13 smithi088 ceph-mon[44699]: 2.e scrub ok 2024-05-15T23:10:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:13 smithi088 ceph-mon[44699]: 2.4 scrub starts 2024-05-15T23:10:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:13 smithi088 ceph-mon[44699]: 2.4 scrub ok 2024-05-15T23:10:13.579 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:13.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:13.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:13.656 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:10:13.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:13.709 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:13.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:13.731 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:13.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:13.832 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:10:13.861 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:13.913 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:10:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:13.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:10:14.016 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:10:14.033 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:10:14.047 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:14.053 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:10:14.067 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:10:14.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:14.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:10:14.073 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:10:14.095 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:10:14.116 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:10:14.138 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:10:14.161 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:10:14.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:14.185 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:10:14.197 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:14.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:10:14.225 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:14.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:14.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:14.245 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:10:14.275 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:10:14.306 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:10:14.311 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:10:14.312 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:10:14.312 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:10:14.312 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:10:14.312 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:10:14.312 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:10:14.312 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:10:14.312 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:10:14.312 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:10:14.312 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:10:14.312 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:10:14.312 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:10:14.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:14 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:10:14.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:14 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:10:14.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:14 smithi088 ceph-mon[44699]: pgmap v556: 17 pgs: 1 active+clean+scrubbing, 1 peering, 15 active+clean; 283 MiB data, 1.6 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 532 op/s 2024-05-15T23:10:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:14 smithi088 ceph-mon[44699]: osdmap e190: 12 total, 12 up, 11 in 2024-05-15T23:10:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:14 smithi088 ceph-mon[44699]: 2.9 scrub starts 2024-05-15T23:10:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:14 smithi088 ceph-mon[44699]: 2.9 scrub ok 2024-05-15T23:10:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:14 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1642626896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:10:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:14.331 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:14.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:14 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:10:14.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:14 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:10:14.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:14 smithi084 ceph-mon[39784]: pgmap v556: 17 pgs: 1 active+clean+scrubbing, 1 peering, 15 active+clean; 283 MiB data, 1.6 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 532 op/s 2024-05-15T23:10:14.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:14 smithi084 ceph-mon[39784]: osdmap e190: 12 total, 12 up, 11 in 2024-05-15T23:10:14.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:14 smithi084 ceph-mon[39784]: 2.9 scrub starts 2024-05-15T23:10:14.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:14 smithi084 ceph-mon[39784]: 2.9 scrub ok 2024-05-15T23:10:14.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:14 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1642626896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:10:14.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:14.366 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:14.401 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T23:10:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:14 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:10:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:14 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:10:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:14 smithi090 ceph-mon[44434]: pgmap v556: 17 pgs: 1 active+clean+scrubbing, 1 peering, 15 active+clean; 283 MiB data, 1.6 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 532 op/s 2024-05-15T23:10:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:14 smithi090 ceph-mon[44434]: osdmap e190: 12 total, 12 up, 11 in 2024-05-15T23:10:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:14 smithi090 ceph-mon[44434]: 2.9 scrub starts 2024-05-15T23:10:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:14 smithi090 ceph-mon[44434]: 2.9 scrub ok 2024-05-15T23:10:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:14 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1642626896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:10:14.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:14.573 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:14.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:14.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:14.677 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:10:14.706 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:15.022 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:15.054 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:10:15.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:15.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:10:15.159 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:10:15.187 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:15.206 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:10:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:15.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:10:15.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:10:15.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:10:15.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: 2.b scrub starts 2024-05-15T23:10:15.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: 2.b scrub ok 2024-05-15T23:10:15.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: 2.a scrub starts 2024-05-15T23:10:15.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: osdmap e191: 12 total, 12 up, 11 in 2024-05-15T23:10:15.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: 2.a scrub ok 2024-05-15T23:10:15.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.280 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.280 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:10:15.370 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:15.396 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:15.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:15.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:15.506 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:10:15.530 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:10:15.530 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:10:15.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: 2.b scrub starts 2024-05-15T23:10:15.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: 2.b scrub ok 2024-05-15T23:10:15.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: 2.a scrub starts 2024-05-15T23:10:15.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: osdmap e191: 12 total, 12 up, 11 in 2024-05-15T23:10:15.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: 2.a scrub ok 2024-05-15T23:10:15.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.532 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:15.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:10:15.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:10:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: 2.b scrub starts 2024-05-15T23:10:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: 2.b scrub ok 2024-05-15T23:10:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: 2.a scrub starts 2024-05-15T23:10:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: osdmap e191: 12 total, 12 up, 11 in 2024-05-15T23:10:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: 2.a scrub ok 2024-05-15T23:10:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:15.555 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:10:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:15.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:10:15.660 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:10:15.715 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:15.934 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Removed 161355 objects 2024-05-15T23:10:15.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Clean up completed and total clean up time :161.078 2024-05-15T23:10:15.973 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:10:15.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:15.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:10:15.980 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:16.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:16.107 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:16.126 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:16.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:16.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:16.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:16.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:16.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:16.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:16 smithi084 ceph-mon[39784]: pgmap v559: 17 pgs: 1 active+clean+scrubbing, 2 peering, 14 active+clean; 255 MiB data, 1.6 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 334 op/s; 1.8 MiB/s, 28 objects/s recovering 2024-05-15T23:10:16.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:16 smithi084 ceph-mon[39784]: 2.5 scrub starts 2024-05-15T23:10:16.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:16 smithi084 ceph-mon[39784]: 2.5 scrub ok 2024-05-15T23:10:16.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:16 smithi084 ceph-mon[39784]: osdmap e192: 12 total, 12 up, 11 in 2024-05-15T23:10:16.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:16 smithi084 ceph-mon[39784]: 2.0 scrub starts 2024-05-15T23:10:16.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:16 smithi084 ceph-mon[39784]: 2.0 scrub ok 2024-05-15T23:10:16.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:16 smithi084 ceph-mon[39784]: 2.3 scrub starts 2024-05-15T23:10:16.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:16 smithi084 ceph-mon[39784]: 2.3 scrub ok 2024-05-15T23:10:16.255 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:16.273 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:16.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:16.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:16.377 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:10:16.401 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:16.402 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:16.402 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:16 smithi088 ceph-mon[44699]: pgmap v559: 17 pgs: 1 active+clean+scrubbing, 2 peering, 14 active+clean; 255 MiB data, 1.6 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 334 op/s; 1.8 MiB/s, 28 objects/s recovering 2024-05-15T23:10:16.402 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:16 smithi088 ceph-mon[44699]: 2.5 scrub starts 2024-05-15T23:10:16.402 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:16 smithi088 ceph-mon[44699]: 2.5 scrub ok 2024-05-15T23:10:16.402 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:16 smithi088 ceph-mon[44699]: osdmap e192: 12 total, 12 up, 11 in 2024-05-15T23:10:16.402 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:16 smithi088 ceph-mon[44699]: 2.0 scrub starts 2024-05-15T23:10:16.402 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:16 smithi088 ceph-mon[44699]: 2.0 scrub ok 2024-05-15T23:10:16.402 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:16 smithi088 ceph-mon[44699]: 2.3 scrub starts 2024-05-15T23:10:16.402 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:16 smithi088 ceph-mon[44699]: 2.3 scrub ok 2024-05-15T23:10:16.403 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:16.421 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:10:16.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:16.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:10:16.525 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:10:16.550 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:16 smithi090 ceph-mon[44434]: pgmap v559: 17 pgs: 1 active+clean+scrubbing, 2 peering, 14 active+clean; 255 MiB data, 1.6 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 334 op/s; 1.8 MiB/s, 28 objects/s recovering 2024-05-15T23:10:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:16 smithi090 ceph-mon[44434]: 2.5 scrub starts 2024-05-15T23:10:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:16 smithi090 ceph-mon[44434]: 2.5 scrub ok 2024-05-15T23:10:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:16 smithi090 ceph-mon[44434]: osdmap e192: 12 total, 12 up, 11 in 2024-05-15T23:10:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:16 smithi090 ceph-mon[44434]: 2.0 scrub starts 2024-05-15T23:10:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:16 smithi090 ceph-mon[44434]: 2.0 scrub ok 2024-05-15T23:10:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:16 smithi090 ceph-mon[44434]: 2.3 scrub starts 2024-05-15T23:10:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:16 smithi090 ceph-mon[44434]: 2.3 scrub ok 2024-05-15T23:10:16.569 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:10:16.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:16.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:10:16.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:16.701 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:17.024 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:10:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:17 smithi084 ceph-mon[39784]: 2.f scrub starts 2024-05-15T23:10:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:17 smithi084 ceph-mon[39784]: 2.f scrub ok 2024-05-15T23:10:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:17 smithi084 ceph-mon[39784]: 2.7 scrub starts 2024-05-15T23:10:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:17 smithi084 ceph-mon[39784]: 2.7 scrub ok 2024-05-15T23:10:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:17 smithi084 ceph-mon[39784]: 2.2 scrub starts 2024-05-15T23:10:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:17 smithi084 ceph-mon[39784]: 2.2 scrub ok 2024-05-15T23:10:17.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:17 smithi090 ceph-mon[44434]: 2.f scrub starts 2024-05-15T23:10:17.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:17 smithi090 ceph-mon[44434]: 2.f scrub ok 2024-05-15T23:10:17.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:17 smithi090 ceph-mon[44434]: 2.7 scrub starts 2024-05-15T23:10:17.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:17 smithi090 ceph-mon[44434]: 2.7 scrub ok 2024-05-15T23:10:17.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:17 smithi090 ceph-mon[44434]: 2.2 scrub starts 2024-05-15T23:10:17.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:17 smithi090 ceph-mon[44434]: 2.2 scrub ok 2024-05-15T23:10:17.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:17 smithi088 ceph-mon[44699]: 2.f scrub starts 2024-05-15T23:10:17.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:17 smithi088 ceph-mon[44699]: 2.f scrub ok 2024-05-15T23:10:17.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:17 smithi088 ceph-mon[44699]: 2.7 scrub starts 2024-05-15T23:10:17.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:17 smithi088 ceph-mon[44699]: 2.7 scrub ok 2024-05-15T23:10:17.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:17 smithi088 ceph-mon[44699]: 2.2 scrub starts 2024-05-15T23:10:17.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:17 smithi088 ceph-mon[44699]: 2.2 scrub ok 2024-05-15T23:10:17.750 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:17.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:17.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:17.854 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:10:17.887 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:17.908 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:10:17.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:17.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:10:17.988 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 11, 9] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:10:17.988 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:10:17.988 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-05-15T23:10:17.988 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd in 10 2024-05-15T23:10:18.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:18.045 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:18.067 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:18.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:18.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:18.172 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:10:18.202 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:18 smithi088 ceph-mon[44699]: 2.d scrub starts 2024-05-15T23:10:18.203 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:18 smithi088 ceph-mon[44699]: 2.d scrub ok 2024-05-15T23:10:18.203 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:18 smithi088 ceph-mon[44699]: pgmap v561: 17 pgs: 1 active+clean+scrubbing, 1 peering, 15 active+clean; 51 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 873 op/s; 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:10:18.205 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:18.226 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:10:18.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:18.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:10:18.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:18 smithi084 ceph-mon[39784]: 2.d scrub starts 2024-05-15T23:10:18.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:18 smithi084 ceph-mon[39784]: 2.d scrub ok 2024-05-15T23:10:18.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:18 smithi084 ceph-mon[39784]: pgmap v561: 17 pgs: 1 active+clean+scrubbing, 1 peering, 15 active+clean; 51 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 873 op/s; 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:10:18.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:18 smithi090 ceph-mon[44434]: 2.d scrub starts 2024-05-15T23:10:18.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:18 smithi090 ceph-mon[44434]: 2.d scrub ok 2024-05-15T23:10:18.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:18 smithi090 ceph-mon[44434]: pgmap v561: 17 pgs: 1 active+clean+scrubbing, 1 peering, 15 active+clean; 51 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 873 op/s; 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:10:18.330 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:10:18.357 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:18.375 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:10:18.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:18.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:10:18.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:18.510 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:18.531 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:18.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:18.534 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:18.635 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:10:18.668 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:18.689 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:10:18.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:18.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:10:18.695 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:18.705 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:18.712 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:10:18.712 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:10:18.712 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:10:18.712 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:10:18.712 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:10:18.793 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:18.820 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:18.967 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:18.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:18.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:19.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:19.103 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:10:19.104 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:19.126 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:19.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:19.129 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:19.230 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:10:19.259 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3055942237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:10:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3055942237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:10:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3055942237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:10:20.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:20 smithi084 ceph-mon[39784]: pgmap v562: 17 pgs: 1 peering, 16 active+clean; 38 MiB data, 1.1 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 651 op/s; 3.4 MiB/s, 54 objects/s recovering 2024-05-15T23:10:20.215 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:20 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:10:20.215 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:20 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:10:20.215 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:20 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3055942237' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:10:20.215 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:20 smithi084 ceph-mon[39784]: osdmap e193: 12 total, 12 up, 11 in 2024-05-15T23:10:20.509 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:20.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:20.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:20 smithi090 ceph-mon[44434]: pgmap v562: 17 pgs: 1 peering, 16 active+clean; 38 MiB data, 1.1 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 651 op/s; 3.4 MiB/s, 54 objects/s recovering 2024-05-15T23:10:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:20 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:10:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:20 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:10:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:20 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3055942237' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:10:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:20 smithi090 ceph-mon[44434]: osdmap e193: 12 total, 12 up, 11 in 2024-05-15T23:10:20.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:20 smithi088 ceph-mon[44699]: pgmap v562: 17 pgs: 1 peering, 16 active+clean; 38 MiB data, 1.1 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 651 op/s; 3.4 MiB/s, 54 objects/s recovering 2024-05-15T23:10:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:20 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:10:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:20 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:10:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:20 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3055942237' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:10:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:20 smithi088 ceph-mon[44699]: osdmap e193: 12 total, 12 up, 11 in 2024-05-15T23:10:20.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:10:20.645 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:20.666 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:10:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:20.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:10:20.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:20.778 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-05-15T23:10:20.778 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-05-15T23:10:20.800 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:20.820 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:20.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:20.823 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:20.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:10:20.950 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:20.970 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:10:20.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:20.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:10:20.999 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T23:10:21.074 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:10:21.108 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:21.206 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:10:21.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:21.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:10:21.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:10:21.342 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:21.364 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:10:21.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:21.367 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:10:21.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:10:21.495 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:10:21.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:21.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:10:21.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:21.651 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:21.671 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:21.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:21.775 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:21.803 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:21.901 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:21.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:21.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:21.909 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:10:22.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:10:22.032 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:22.067 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:10:22.067 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:10:22.071 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:22.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:22.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:22.075 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:10:22.100 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:10:22.126 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:10:22.129 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:22.148 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:10:22.170 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:10:22.174 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:10:22.194 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:10:22.201 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:22.215 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:10:22.236 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:10:22.256 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:10:22.277 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:10:22.279 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:10:22.279 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:10:22.279 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:10:22.279 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:10:22.279 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:10:22.279 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:10:22.279 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:10:22.279 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:10:22.279 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:10:22.279 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:10:22.280 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:10:22.280 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:10:22.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:22 smithi084 ceph-mon[39784]: pgmap v564: 17 pgs: 17 active+clean; 577 KiB data, 1.0 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 675 op/s; 3.0 MiB/s, 48 objects/s recovering 2024-05-15T23:10:22.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:22 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:10:22.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:22 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:10:22.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:22 smithi084 ceph-mon[39784]: osdmap e194: 12 total, 12 up, 11 in 2024-05-15T23:10:22.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3860089471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:10:22.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:22 smithi090 ceph-mon[44434]: pgmap v564: 17 pgs: 17 active+clean; 577 KiB data, 1.0 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 675 op/s; 3.0 MiB/s, 48 objects/s recovering 2024-05-15T23:10:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:22 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:10:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:22 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:10:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:22 smithi090 ceph-mon[44434]: osdmap e194: 12 total, 12 up, 11 in 2024-05-15T23:10:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3860089471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:10:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:22.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:22 smithi088 ceph-mon[44699]: pgmap v564: 17 pgs: 17 active+clean; 577 KiB data, 1.0 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 675 op/s; 3.0 MiB/s, 48 objects/s recovering 2024-05-15T23:10:22.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:22 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:10:22.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:22 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:10:22.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:22 smithi088 ceph-mon[44699]: osdmap e194: 12 total, 12 up, 11 in 2024-05-15T23:10:22.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3860089471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:10:22.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:22.715 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:22.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:22.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:22.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:22.851 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:22.873 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:22.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:22.877 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:22.977 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:10:23.010 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:23.032 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:10:23.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:23.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:10:23.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:10:23.168 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:23.190 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:10:23.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:23.193 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:10:23.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.212 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.212 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.212 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.212 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:10:23.212 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:10:23.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:10:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:10:23.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:10:23.322 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:23.420 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:10:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:23.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:10:23.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:10:23.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:10:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:10:23.558 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:23.578 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:10:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:23.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:10:23.682 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:10:23.715 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:23.736 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:10:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:23.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:10:23.840 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:10:23.869 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:23.890 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:10:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:23.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:10:23.993 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:10:24.022 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:24.041 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:10:24.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:24.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:10:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:23 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:10:23.752+0000 7fa0938b7700 -1 mon.a@0(leader).osd e195 definitely_dead 0 2024-05-15T23:10:24.117 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.10. 2024-05-15T23:10:24.145 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:10:24.173 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:24.191 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:10:24.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:24.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:10:24.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:10:24.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:24 smithi088 ceph-mon[44699]: pgmap v566: 17 pgs: 17 active+clean; 577 KiB data, 1.0 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 485 op/s; 375 KiB/s, 5 objects/s recovering 2024-05-15T23:10:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:24 smithi088 ceph-mon[44699]: osdmap e195: 12 total, 12 up, 11 in 2024-05-15T23:10:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:24 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2323248302' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-05-15T23:10:24.328 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:24.349 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:10:24.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:24.352 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:10:24.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:24.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:24 smithi084 ceph-mon[39784]: pgmap v566: 17 pgs: 17 active+clean; 577 KiB data, 1.0 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 485 op/s; 375 KiB/s, 5 objects/s recovering 2024-05-15T23:10:24.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:24 smithi084 ceph-mon[39784]: osdmap e195: 12 total, 12 up, 11 in 2024-05-15T23:10:24.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:24 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2323248302' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-05-15T23:10:24.453 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:10:24.483 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:24.522 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:10:24.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:24.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:10:24.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:24.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:24 smithi090 ceph-mon[44434]: pgmap v566: 17 pgs: 17 active+clean; 577 KiB data, 1.0 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 485 op/s; 375 KiB/s, 5 objects/s recovering 2024-05-15T23:10:24.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:24 smithi090 ceph-mon[44434]: osdmap e195: 12 total, 12 up, 11 in 2024-05-15T23:10:24.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:24 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2323248302' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-05-15T23:10:24.627 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:10:24.653 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:24.734 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:25.180 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:25.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:25.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:25.316 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:25.338 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:25.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:25.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:25.442 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:10:25.472 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:25 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2323248302' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-05-15T23:10:25.473 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:25 smithi088 ceph-mon[44699]: osdmap e196: 12 total, 12 up, 12 in 2024-05-15T23:10:25.473 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:25 smithi088 ceph-mon[44699]: osdmap e197: 12 total, 12 up, 12 in 2024-05-15T23:10:25.475 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:25.497 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:10:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:25.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:10:25.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:25 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2323248302' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-05-15T23:10:25.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:25 smithi090 ceph-mon[44434]: osdmap e196: 12 total, 12 up, 12 in 2024-05-15T23:10:25.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:25 smithi090 ceph-mon[44434]: osdmap e197: 12 total, 12 up, 12 in 2024-05-15T23:10:25.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:25 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2323248302' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-05-15T23:10:25.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:25 smithi084 ceph-mon[39784]: osdmap e196: 12 total, 12 up, 12 in 2024-05-15T23:10:25.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:25 smithi084 ceph-mon[39784]: osdmap e197: 12 total, 12 up, 12 in 2024-05-15T23:10:25.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:25.610 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-05-15T23:10:25.634 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:25.656 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:25.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:25.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:25.760 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:10:25.792 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:25.813 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:10:25.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:25.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:10:25.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:25.950 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:25.972 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:25.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:25.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:26.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:10:26.106 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:10:26.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:26.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:10:26.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:10:26.256 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:26 smithi090 ceph-mon[44434]: pgmap v569: 17 pgs: 17 active+clean; 577 KiB data, 1.0 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 106 op/s 2024-05-15T23:10:26.256 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:26 smithi090 ceph-mon[44434]: osdmap e198: 12 total, 12 up, 12 in 2024-05-15T23:10:26.258 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:26.277 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:10:26.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:26.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:10:26.312 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:26.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:26 smithi084 ceph-mon[39784]: pgmap v569: 17 pgs: 17 active+clean; 577 KiB data, 1.0 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 106 op/s 2024-05-15T23:10:26.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:26 smithi084 ceph-mon[39784]: osdmap e198: 12 total, 12 up, 12 in 2024-05-15T23:10:26.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:26.407 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:26.427 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:26.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:26.430 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:26.531 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:10:26.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:26 smithi088 ceph-mon[44699]: pgmap v569: 17 pgs: 17 active+clean; 577 KiB data, 1.0 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 106 op/s 2024-05-15T23:10:26.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:26 smithi088 ceph-mon[44699]: osdmap e198: 12 total, 12 up, 12 in 2024-05-15T23:10:26.564 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:26.585 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:10:26.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:26.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:10:26.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:26.718 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:26.736 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:26.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:26.762 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_0' removed 2024-05-15T23:10:26.840 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:10:26.868 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:26.888 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:10:26.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:26.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:10:26.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:10:27.022 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:27.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:10:27.219 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:27.221 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:27 smithi084 ceph-mon[39784]: 2.0 starting backfill to osd.10 from (138'9000,178'19083] MIN to 192'20434 2024-05-15T23:10:27.221 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:27 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1442715699' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:10:27.221 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:27 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1442715699' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:10:27.221 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:27 smithi084 ceph-mon[39784]: osdmap e199: 12 total, 12 up, 12 in 2024-05-15T23:10:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:27.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:27.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:10:27.352 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:27.371 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:10:27.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:27.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:10:27.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:27.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:27 smithi090 ceph-mon[44434]: 2.0 starting backfill to osd.10 from (138'9000,178'19083] MIN to 192'20434 2024-05-15T23:10:27.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:27 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1442715699' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:10:27.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:27 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1442715699' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:10:27.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:27 smithi090 ceph-mon[44434]: osdmap e199: 12 total, 12 up, 12 in 2024-05-15T23:10:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:27.529 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:27.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:27.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:27.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:27 smithi088 ceph-mon[44699]: 2.0 starting backfill to osd.10 from (138'9000,178'19083] MIN to 192'20434 2024-05-15T23:10:27.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:27 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1442715699' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:10:27.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:27 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1442715699' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:10:27.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:27 smithi088 ceph-mon[44699]: osdmap e199: 12 total, 12 up, 12 in 2024-05-15T23:10:27.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:27.660 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:27.678 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:27.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:27.783 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:10:27.810 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:28.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:28 smithi084 ceph-mon[39784]: pgmap v573: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:28.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:28 smithi090 ceph-mon[44434]: pgmap v573: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:28.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:28 smithi088 ceph-mon[44699]: pgmap v573: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:28.805 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:28.839 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T23:10:28.840 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T23:10:28.840 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-05-15T23:10:28.840 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-05-15T23:10:29.037 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:10:29.037 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:10:29.260 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:10:29.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:29.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:10:29.364 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:10:29.397 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:29.418 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:10:29.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:29.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:10:29.523 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:10:29.553 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:29.573 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:10:29.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:29.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:10:29.677 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:10:29.705 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:29.724 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:10:29.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:29.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:10:29.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:10:29.857 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:29.877 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:29.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:29.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:29.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:30.018 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:30.040 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:30.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:30.043 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:30.144 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:10:30.171 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:30.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:30 smithi090 ceph-mon[44434]: pgmap v574: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:30.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T23:10:30.400 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:30 smithi084 ceph-mon[39784]: pgmap v574: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:30.523 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:10:30.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:30.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:10:30.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:30 smithi088 ceph-mon[44699]: pgmap v574: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:30.611 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 11, 9, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:10:30.611 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:10:30.611 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-15T23:10:30.612 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2024-05-15T23:10:30.612 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 5 1 2024-05-15T23:10:30.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:30.682 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:31.030 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:31.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:31.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:31.136 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:10:31.168 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:31.190 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:10:31.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:31.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:10:31.281 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:10:31.294 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:10:31.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4197802202' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:10:31.322 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:31.521 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:10:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:31.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:10:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4197802202' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:10:31.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4197802202' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:10:31.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:10:31.654 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:31.728 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:10:31.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:31.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:10:31.832 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:31.860 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:31.934 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:31.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:31.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:32.053 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:10:32.086 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:32.107 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:10:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:32.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:10:32.211 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:10:32.241 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:32.261 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:10:32.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:32.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:10:32.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:32.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:32 smithi084 ceph-mon[39784]: pgmap v575: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:32.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:32 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:10:32.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4197802202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:10:32.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:32 smithi084 ceph-mon[39784]: osdmap e200: 12 total, 12 up, 12 in 2024-05-15T23:10:32.395 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:32 smithi090 ceph-mon[44434]: pgmap v575: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:32.396 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:32 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:10:32.396 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4197802202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:10:32.396 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:32 smithi090 ceph-mon[44434]: osdmap e200: 12 total, 12 up, 12 in 2024-05-15T23:10:32.398 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:32.420 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:32.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:32.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:32.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:32.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:32 smithi088 ceph-mon[44699]: pgmap v575: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:32.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:32 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:10:32.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4197802202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:10:32.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:32 smithi088 ceph-mon[44699]: osdmap e200: 12 total, 12 up, 12 in 2024-05-15T23:10:32.572 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:32.575 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:32.676 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:10:32.709 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:34.312 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:34.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:34 smithi090 ceph-mon[44434]: pgmap v577: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:34.569 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:34.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:34 smithi088 ceph-mon[44699]: pgmap v577: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:34.613 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:34.614 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:10:34.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:34 smithi084 ceph-mon[39784]: pgmap v577: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:34.634 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:10:34.653 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:10:34.672 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:10:34.692 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:10:34.712 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:10:34.732 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:10:34.754 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:10:34.774 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:10:34.794 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:10:34.813 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:10:34.832 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:10:34.834 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:10:34.834 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:10:34.834 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:10:34.834 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:10:34.835 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:10:34.835 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:10:34.835 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:10:34.835 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:10:34.835 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:10:34.835 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:10:34.836 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:10:34.836 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:10:34.863 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:34.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:34.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:34.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:35.001 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:35.022 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:35.026 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:35.127 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:10:35.156 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:35.279 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:10:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:35.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:10:35.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:35 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2081202458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:10:35.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:35.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:35.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:35.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:35.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:35.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:35.417 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:35.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:35 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2081202458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:10:35.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:35.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:35.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:35.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:35.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:35.437 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:35.441 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:35.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:35.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:35 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2081202458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:10:35.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:35.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:35.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:35.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:35.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:35.571 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:35.590 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:35.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:35.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:35.723 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:35.743 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:35.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:35.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:35.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:35.876 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:35.895 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:35.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:35.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:35.998 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:10:36.025 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:36.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:36 smithi084 ceph-mon[39784]: pgmap v578: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:36.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:10:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:36 smithi090 ceph-mon[44434]: pgmap v578: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:36 smithi088 ceph-mon[44699]: pgmap v578: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:36.973 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:10:36.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:36.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:10:37.077 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:10:37.114 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:37.136 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:10:37.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:37.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:10:37.241 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:37.272 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:37.293 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:37.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:37.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:37.397 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:37.425 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:37.983 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:37 smithi084 ceph-mon[39784]: pgmap v579: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:37 smithi090 ceph-mon[44434]: pgmap v579: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:38.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:37 smithi088 ceph-mon[44699]: pgmap v579: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:38.803 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:38.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:38.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:38.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:10:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:38.962 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:10:38.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:38.965 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:10:39.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:10:39.095 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:39.115 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:10:39.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:39.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:10:39.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:39.247 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:39.266 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:39.269 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:39.370 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:10:39.403 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:39.424 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:10:39.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:39.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:10:39.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:39.558 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:39.579 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:39.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:39.684 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:39.705 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:10:39.710 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:39.728 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:39.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:39.731 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:39.783 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-05-15T23:10:39.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:10:39.859 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:39 smithi090 ceph-mon[44434]: pgmap v580: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/833970617' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:10:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:10:40.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:39 smithi088 ceph-mon[44699]: pgmap v580: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:40.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/833970617' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:10:40.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:39 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:10:40.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:39 smithi084 ceph-mon[39784]: pgmap v580: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:40.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/833970617' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:10:40.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:10:40.917 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:10:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:10:40.917 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-05-15T23:10:40.917 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:10:40.934 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:40 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/833970617' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:10:40.934 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:40 smithi084 ceph-mon[39784]: osdmap e201: 12 total, 12 up, 12 in 2024-05-15T23:10:40.934 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:40 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3922280524' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-05-15T23:10:41.029 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_1' created 2024-05-15T23:10:41.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:40 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/833970617' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:10:41.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:40 smithi090 ceph-mon[44434]: osdmap e201: 12 total, 12 up, 12 in 2024-05-15T23:10:41.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:40 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3922280524' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-05-15T23:10:41.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:40 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/833970617' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:10:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:40 smithi088 ceph-mon[44699]: osdmap e201: 12 total, 12 up, 12 in 2024-05-15T23:10:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:40 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3922280524' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-05-15T23:10:41.538 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:10:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:41.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:10:41.643 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:10:41.672 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:41.948 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:41 smithi084 ceph-mon[39784]: pgmap v582: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:41.948 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:41 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3922280524' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-05-15T23:10:41.948 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:41 smithi084 ceph-mon[39784]: osdmap e202: 12 total, 12 up, 12 in 2024-05-15T23:10:41.981 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-05-15T23:10:41.986 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:41 smithi090 ceph-mon[44434]: pgmap v582: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:41 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3922280524' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-05-15T23:10:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:41 smithi090 ceph-mon[44434]: osdmap e202: 12 total, 12 up, 12 in 2024-05-15T23:10:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:41 smithi088 ceph-mon[44699]: pgmap v582: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:41 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3922280524' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-05-15T23:10:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:41 smithi088 ceph-mon[44699]: osdmap e202: 12 total, 12 up, 12 in 2024-05-15T23:10:42.422 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:42.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:42.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:42.439 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:42.441 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T23:10:42.527 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:42.560 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:42.581 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:42.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:42.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:42.685 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:10:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:42.740 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:10:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:42.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:10:42.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:10:42.872 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:42.944 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:10:42.945 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:42 smithi084 ceph-mon[39784]: osdmap e203: 12 total, 12 up, 12 in 2024-05-15T23:10:42.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:42.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:10:43.050 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:10:43.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:42 smithi090 ceph-mon[44434]: osdmap e203: 12 total, 12 up, 12 in 2024-05-15T23:10:43.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:42 smithi088 ceph-mon[44699]: osdmap e203: 12 total, 12 up, 12 in 2024-05-15T23:10:43.077 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:43 smithi084 ceph-mon[39784]: pgmap v585: 17 pgs: 2 creating+peering, 10 unknown, 5 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:43 smithi084 ceph-mon[39784]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:10:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:43 smithi084 ceph-mon[39784]: osdmap e204: 12 total, 12 up, 12 in 2024-05-15T23:10:43.926 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:10:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:43.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:10:44.031 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:10:44.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:43 smithi090 ceph-mon[44434]: pgmap v585: 17 pgs: 2 creating+peering, 10 unknown, 5 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:44.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:43 smithi090 ceph-mon[44434]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:10:44.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:43 smithi090 ceph-mon[44434]: osdmap e204: 12 total, 12 up, 12 in 2024-05-15T23:10:44.058 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:44.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:43 smithi088 ceph-mon[44699]: pgmap v585: 17 pgs: 2 creating+peering, 10 unknown, 5 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:44.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:43 smithi088 ceph-mon[44699]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:10:44.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:43 smithi088 ceph-mon[44699]: osdmap e204: 12 total, 12 up, 12 in 2024-05-15T23:10:44.206 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:44.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:44.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:44.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:44.365 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:44.368 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:44.469 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:44.499 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:44.518 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:44.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:44.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:44.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:10:44.651 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:44.723 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:44.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:44.827 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:44.855 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:45.597 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:45.603 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:45.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:45.700 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 11, 9, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:10:45.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 0 1 2024-05-15T23:10:45.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:45.743 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:45.765 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:45.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:45.768 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:45.813 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:10:45.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:10:45.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:45 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:10:45.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:45 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:10:45.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:45 smithi084 ceph-mon[39784]: pgmap v587: 17 pgs: 2 creating+peering, 6 unknown, 9 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:45.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:45 smithi084 ceph-mon[39784]: osdmap e205: 12 total, 12 up, 12 in 2024-05-15T23:10:45.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/833802492' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:10:45.896 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:45 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:10:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:45 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:10:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:45 smithi090 ceph-mon[44434]: pgmap v587: 17 pgs: 2 creating+peering, 6 unknown, 9 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:45 smithi090 ceph-mon[44434]: osdmap e205: 12 total, 12 up, 12 in 2024-05-15T23:10:46.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/833802492' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:10:46.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:45 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:10:46.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:45 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:10:46.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:45 smithi088 ceph-mon[44699]: pgmap v587: 17 pgs: 2 creating+peering, 6 unknown, 9 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:46.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:45 smithi088 ceph-mon[44699]: osdmap e205: 12 total, 12 up, 12 in 2024-05-15T23:10:46.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/833802492' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:10:46.301 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:47.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:46 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:10:47.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:46 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/833802492' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:10:47.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:46 smithi088 ceph-mon[44699]: osdmap e206: 12 total, 12 up, 12 in 2024-05-15T23:10:47.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:46 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:10:47.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:46 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/833802492' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:10:47.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:46 smithi084 ceph-mon[39784]: osdmap e206: 12 total, 12 up, 12 in 2024-05-15T23:10:47.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:46 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:10:47.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:46 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/833802492' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:10:47.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:46 smithi090 ceph-mon[44434]: osdmap e206: 12 total, 12 up, 12 in 2024-05-15T23:10:48.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:47 smithi088 ceph-mon[44699]: pgmap v590: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:48.082 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:10:48.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:47 smithi084 ceph-mon[39784]: pgmap v590: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:48.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:10:48.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:48.217 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:47 smithi090 ceph-mon[44434]: pgmap v590: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:48.241 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:48.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:48.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:48.378 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:49.205 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:49.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:49.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:10:49.309 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:10:49.342 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:49.362 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:10:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:49.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:10:49.467 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:49.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:49 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:10:49.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:49 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:10:49.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:49 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:10:49.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:49 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:10:49.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:49 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:10:49.494 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:49.522 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:49.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:49.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:49.526 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:49.552 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:10:49.571 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:10:49.590 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:10:49.610 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:10:49.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:10:49.630 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:10:49.650 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:10:49.658 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:49.672 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:10:49.679 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:10:49.681 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:49 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:10:49.681 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:49 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:10:49.681 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:49 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:10:49.681 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:49 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:10:49.681 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:49 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:10:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:49.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:10:49.692 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:10:49.712 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:10:49.732 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:10:49.753 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:10:49.772 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:10:49.773 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:10:49.773 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:10:49.773 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:10:49.773 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:10:49.774 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:10:49.774 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:10:49.774 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:10:49.774 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:10:49.774 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:10:49.774 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:10:49.774 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:10:49.774 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:10:49.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:10:49.811 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:49 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:10:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:49 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:10:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:49 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:10:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:49 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:10:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:49 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:10:49.830 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:10:49.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:49.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:10:49.935 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:10:49.968 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:49.989 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:10:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:49.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:10:50.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:50.127 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:50.328 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:50.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:50.430 INFO:teuthology.orchestra.run.smithi084.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-05-15T23:10:50.432 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:10:50.463 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:50.485 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:10:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:50.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:10:50.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:50.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:50 smithi084 ceph-mon[39784]: pgmap v591: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:50.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3707794921' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:10:50.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:50 smithi084 ceph-mon[39784]: from='client.35171 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:10:50.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4228859246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:10:50.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.623 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:50.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:50.647 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:50.748 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:10:50.779 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:50 smithi088 ceph-mon[44699]: pgmap v591: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:50.779 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3707794921' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:10:50.779 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:50 smithi088 ceph-mon[44699]: from='client.35171 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:10:50.780 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4228859246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:10:50.780 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.780 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.780 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.780 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.780 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.781 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.781 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.781 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.781 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.782 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:50.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:50 smithi090 ceph-mon[44434]: pgmap v591: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:50.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3707794921' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:10:50.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:50 smithi090 ceph-mon[44434]: from='client.35171 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:10:50.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4228859246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:10:50.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:50.803 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:10:50.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:50.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:10:50.907 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:10:50.937 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:50.966 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:10:50.966 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:10:51.136 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:51.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:51.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:51.272 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:51.293 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:51.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:51.296 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:51.397 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:10:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:51.452 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:10:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:51.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:10:51.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:51.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:51.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:51.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:51 smithi090 ceph-mon[44434]: from='client.35171 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:10:51.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:51 smithi090 ceph-mon[44434]: osdmap e207: 12 total, 12 up, 12 in 2024-05-15T23:10:51.556 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:10:51.582 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:51.680 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:10:51.682 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:51.682 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:51.682 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:51.682 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:51 smithi084 ceph-mon[39784]: from='client.35171 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:10:51.682 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:51 smithi084 ceph-mon[39784]: osdmap e207: 12 total, 12 up, 12 in 2024-05-15T23:10:51.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:51.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:10:51.784 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:10:51.814 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:51.814 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:51.814 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:10:51.815 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:51 smithi088 ceph-mon[44699]: from='client.35171 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:10:51.815 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:51 smithi088 ceph-mon[44699]: osdmap e207: 12 total, 12 up, 12 in 2024-05-15T23:10:51.817 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:51.838 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:10:51.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:51.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:10:51.943 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:51.976 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:51.997 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:51.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:52.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:52.101 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:10:52.134 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:52.156 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:10:52.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:52.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:10:52.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:52.293 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:52.314 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:52.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:52.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:52.448 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:52.468 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:52.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:52.471 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:52 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:10:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:52 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:10:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:52 smithi090 ceph-mon[44434]: pgmap v593: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:52 smithi090 ceph-mon[44434]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:10:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:52 smithi090 ceph-mon[44434]: osdmap e208: 12 total, 12 up, 12 in 2024-05-15T23:10:52.572 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:52.600 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:52.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:52 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:10:52.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:52 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:10:52.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:52 smithi084 ceph-mon[39784]: pgmap v593: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:52.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:52 smithi084 ceph-mon[39784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:10:52.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:52 smithi084 ceph-mon[39784]: osdmap e208: 12 total, 12 up, 12 in 2024-05-15T23:10:52.620 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:52.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:52.623 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:52.724 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:10:52.731 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T23:10:52.755 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:52 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:10:52.755 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:52 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:10:52.755 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:52 smithi088 ceph-mon[44699]: pgmap v593: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:52.755 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:52 smithi088 ceph-mon[44699]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:10:52.755 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:52 smithi088 ceph-mon[44699]: osdmap e208: 12 total, 12 up, 12 in 2024-05-15T23:10:52.757 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:52.779 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:10:52.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:52.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:10:52.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:52.933 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:52.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:52.936 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:53.038 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:53.064 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:53.138 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:53.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:53.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:53.242 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:10:53.273 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:53.295 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:10:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:53.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:10:53.399 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:10:53.427 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:53.447 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:10:53.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:53.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:10:53.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:53.596 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:53.618 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:53.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:53.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:53.722 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:10:53.754 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:53.774 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:10:53.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:53.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:10:53.878 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:10:53.909 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:10:53.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:53.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:10:53.982 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:54.017 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:54.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:54.067 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:54.089 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:54.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:54.092 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:54.193 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:10:54.224 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:54.244 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:10:54.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:54.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:10:54.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:54.381 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:54.402 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:54.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:54.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:54.436 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2024-05-15T23:10:54.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:54.536 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:54.556 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:54.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:54.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:54 smithi088 ceph-mon[44699]: pgmap v595: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:54.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1982472465' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:10:54.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:54 smithi084 ceph-mon[39784]: pgmap v595: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:54.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1982472465' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:10:54.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:54.686 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:54 smithi090 ceph-mon[44434]: pgmap v595: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:54.686 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1982472465' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:10:54.688 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:54.708 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:54.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:54.811 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:10:54.823 DEBUG:teuthology.orchestra.run.smithi090:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-05-15T23:10:54.825 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T23:10:54.825 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T23:10:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:54.864 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:10:54.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:54.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:10:54.917 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:hints = 1 2024-05-15T23:10:54.918 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T23:10:54.918 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object prefix: benchmark_data_smithi090_90974 2024-05-15T23:10:54.918 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:10:54.918 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 0 0 0 0 0 0 - 0 2024-05-15T23:10:54.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:10:55.005 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:55.030 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:10:55.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:55.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:10:55.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:55.164 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:10:55.172 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:55.198 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:55.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:55.201 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:55.302 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:10:55.345 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:55.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:55 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1982472465' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T23:10:55.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:55 smithi084 ceph-mon[39784]: osdmap e209: 12 total, 12 up, 12 in 2024-05-15T23:10:55.677 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 1 1 2024-05-15T23:10:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:55 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1982472465' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T23:10:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:55 smithi090 ceph-mon[44434]: osdmap e209: 12 total, 12 up, 12 in 2024-05-15T23:10:55.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:55 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1982472465' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T23:10:55.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:55 smithi088 ceph-mon[44699]: osdmap e209: 12 total, 12 up, 12 in 2024-05-15T23:10:55.918 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 1 16 6159 6143 383.917 383.938 0.00155436 0.00260024 2024-05-15T23:10:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:56 smithi084 ceph-mon[39784]: pgmap v597: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:56 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:10:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:56 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:10:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:56 smithi084 ceph-mon[39784]: osdmap e210: 12 total, 12 up, 12 in 2024-05-15T23:10:56.713 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:10:56.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:56.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:10:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:56 smithi090 ceph-mon[44434]: pgmap v597: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:56 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:10:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:56 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:10:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:56 smithi090 ceph-mon[44434]: osdmap e210: 12 total, 12 up, 12 in 2024-05-15T23:10:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:56 smithi088 ceph-mon[44699]: pgmap v597: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:10:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:56 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:10:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:56 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:10:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:56 smithi088 ceph-mon[44699]: osdmap e210: 12 total, 12 up, 12 in 2024-05-15T23:10:56.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:56.865 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:56.890 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:56.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:56.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:56.918 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 2 15 11913 11898 371.787 359.688 0.0026856 0.00268649 2024-05-15T23:10:56.994 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:10:57.036 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:57.141 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:10:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:57.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:10:57.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:10:57.290 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:57.318 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:10:57.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:57.321 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:10:57.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:57.458 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:57.485 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:57.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:57.488 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:10:57.589 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:10:57.636 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:57.659 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:10:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:57.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:10:57.763 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:10:57.807 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:57.892 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:10:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:57.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:10:57.918 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 3 15 16725 16710 348.1 300.75 0.00458695 0.00286989 2024-05-15T23:10:57.997 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:10:57.997 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:10:58.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:10:58.008 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:58.009 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:57 smithi084 ceph-mon[39784]: pgmap v599: 17 pgs: 17 active+clean; 145 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 385 op/s 2024-05-15T23:10:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:57 smithi090 ceph-mon[44434]: pgmap v599: 17 pgs: 17 active+clean; 145 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 385 op/s 2024-05-15T23:10:58.061 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:58.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:57 smithi088 ceph-mon[44699]: pgmap v599: 17 pgs: 17 active+clean; 145 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 385 op/s 2024-05-15T23:10:58.792 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:10:58.920 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 4 16 21199 21183 330.796 279.562 0.00202987 0.00302066 2024-05-15T23:10:58.927 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T23:10:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:58 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1633249349' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:10:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:58 smithi090 ceph-mon[44434]: from='client.35195 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:10:59.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:58 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1633249349' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:10:59.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:58 smithi088 ceph-mon[44699]: from='client.35195 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:10:59.087 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:10:59.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:58 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1633249349' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:10:59.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:58 smithi084 ceph-mon[39784]: from='client.35195 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:10:59.279 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:10:59.819 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:10:59.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:59.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:10:59.858 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:59 smithi084 ceph-mon[39784]: pgmap v600: 17 pgs: 17 active+clean; 348 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 757 op/s 2024-05-15T23:10:59.859 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:59 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:10:59.859 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:59 smithi084 ceph-mon[39784]: from='client.35195 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:10:59.859 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:10:59 smithi084 ceph-mon[39784]: osdmap e211: 12 total, 12 up, 12 in 2024-05-15T23:10:59.920 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 5 16 25306 25290 315.976 256.688 0.00359766 0.00316315 2024-05-15T23:10:59.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:10:59.960 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:10:59.990 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:10:59.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:10:59.994 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:59 smithi090 ceph-mon[44434]: pgmap v600: 17 pgs: 17 active+clean; 348 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 757 op/s 2024-05-15T23:11:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:59 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:11:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:59 smithi090 ceph-mon[44434]: from='client.35195 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:11:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:10:59 smithi090 ceph-mon[44434]: osdmap e211: 12 total, 12 up, 12 in 2024-05-15T23:11:00.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:59 smithi088 ceph-mon[44699]: pgmap v600: 17 pgs: 17 active+clean; 348 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 757 op/s 2024-05-15T23:11:00.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:59 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:11:00.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:59 smithi088 ceph-mon[44699]: from='client.35195 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:11:00.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:10:59 smithi088 ceph-mon[44699]: osdmap e211: 12 total, 12 up, 12 in 2024-05-15T23:11:00.095 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:11:00.133 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:11:00.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:00.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:11:00.265 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:11:00.306 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:00.861 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:11:00.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:00.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:11:00.920 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 6 16 29116 29100 303 238.125 0.0019602 0.00325685 2024-05-15T23:11:00.964 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:11:00.997 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:01.208 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:11:01.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:01.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:11:01.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:01.336 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:11:01.336 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:11:01.356 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:01.382 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:01.385 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:01.487 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:11:01.530 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:01.564 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:11:01.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:01.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:11:01.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:11:01.669 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:11:01.709 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:01.743 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:11:01.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:01.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:11:01.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:11:01.884 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:01.913 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:11:01.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:01.917 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:11:01.920 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 7 16 32572 32556 290.573 216 0.00662363 0.00343929 2024-05-15T23:11:02.018 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:11:02.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:01 smithi090 ceph-mon[44434]: pgmap v602: 17 pgs: 17 active+clean; 1.3 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 205 MiB/s wr, 3.28k op/s 2024-05-15T23:11:02.057 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:01 smithi084 ceph-mon[39784]: pgmap v602: 17 pgs: 17 active+clean; 1.3 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 205 MiB/s wr, 3.28k op/s 2024-05-15T23:11:02.061 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:01 smithi088 ceph-mon[44699]: pgmap v602: 17 pgs: 17 active+clean; 1.3 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 205 MiB/s wr, 3.28k op/s 2024-05-15T23:11:02.062 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:02.098 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T23:11:02.481 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:11:02.523 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:11:02.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:02.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:11:02.632 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:11:02.685 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:02.764 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:11:02.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:02.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:11:02.870 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:11:02.920 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 8 15 35947 35932 280.626 211 0.00248141 0.00356123 2024-05-15T23:11:02.927 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:03.187 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:11:03.224 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:11:03.258 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:11:03.290 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:03.292 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:11:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:03.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:11:03.306 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:11:03.354 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:11:03.392 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:11:03.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:03.422 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:11:03.434 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:03.452 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:11:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:03.466 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:03.467 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:03.485 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:11:03.516 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:11:03.549 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:11:03.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:11:03.585 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:11:03.601 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:03.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:11:03.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:11:03.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:11:03.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:11:03.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:11:03.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:11:03.604 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:11:03.604 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:11:03.604 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:11:03.604 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:11:03.604 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:11:03.604 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:11:03.628 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:11:03.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:03.631 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:11:03.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:11:03.783 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:03.806 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2024-05-15T23:11:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:03 smithi084 ceph-mon[39784]: pgmap v603: 17 pgs: 17 active+clean; 1.7 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 222 MiB/s wr, 3.55k op/s 2024-05-15T23:11:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2367610870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:11:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/749330544' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:11:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:03.920 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 9 15 38807 38792 269.308 178.75 0.0027888 0.00371164 2024-05-15T23:11:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:03 smithi088 ceph-mon[44699]: pgmap v603: 17 pgs: 17 active+clean; 1.7 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 222 MiB/s wr, 3.55k op/s 2024-05-15T23:11:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2367610870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:11:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/749330544' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:11:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:03 smithi090 ceph-mon[44434]: pgmap v603: 17 pgs: 17 active+clean; 1.7 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 222 MiB/s wr, 3.55k op/s 2024-05-15T23:11:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2367610870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:11:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/749330544' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:11:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.540 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:11:04.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:04.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:11:04.645 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:11:04.686 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:04.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:04 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:11:04.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:04 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:11:04.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.820 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.820 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:04.820 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/749330544' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:11:04.820 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:04 smithi084 ceph-mon[39784]: osdmap e212: 12 total, 12 up, 12 in 2024-05-15T23:11:04.920 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 10 16 41902 41886 261.714 193.375 0.00301091 0.00381935 2024-05-15T23:11:05.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:05.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:05.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:05.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:04 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:11:05.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:04 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:11:05.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:05.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:05.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:05.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/749330544' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:11:05.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:04 smithi088 ceph-mon[44699]: osdmap e212: 12 total, 12 up, 12 in 2024-05-15T23:11:05.075 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:11:05.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:05.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:11:05.106 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 2 1 2024-05-15T23:11:05.180 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:11:05.216 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:04 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:11:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:04 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:11:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/749330544' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:11:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:04 smithi090 ceph-mon[44434]: osdmap e212: 12 total, 12 up, 12 in 2024-05-15T23:11:05.474 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:11:05.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:05.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:11:05.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:11:05.618 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:11:05.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:05.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:11:05.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:11:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:05.811 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:11:05.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:05.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:11:05.916 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:11:05.922 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 11 15 44818 44803 254.496 182.312 0.00590618 0.00392777 2024-05-15T23:11:05.951 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:06.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:05 smithi090 ceph-mon[44434]: pgmap v605: 17 pgs: 17 active+clean; 2.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 237 MiB/s wr, 3.79k op/s 2024-05-15T23:11:06.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:05 smithi088 ceph-mon[44699]: pgmap v605: 17 pgs: 17 active+clean; 2.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 237 MiB/s wr, 3.79k op/s 2024-05-15T23:11:06.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:05 smithi084 ceph-mon[39784]: pgmap v605: 17 pgs: 17 active+clean; 2.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 237 MiB/s wr, 3.79k op/s 2024-05-15T23:11:06.158 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:11:06.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:06.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:11:06.228 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:06.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:11:06.319 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:06.921 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 12 15 47897 47882 249.324 192.438 0.00430839 0.00400906 2024-05-15T23:11:07.301 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:11:07.301 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:11:07.301 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:11:07.301 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:11:07.301 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:11:07.921 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 13 16 50765 50749 243.928 179.188 0.00448742 0.00409764 2024-05-15T23:11:08.040 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:07 smithi088 ceph-mon[44699]: pgmap v606: 17 pgs: 17 active+clean; 2.5 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 274 MiB/s wr, 4.38k op/s 2024-05-15T23:11:08.079 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T23:11:08.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:07 smithi084 ceph-mon[39784]: pgmap v606: 17 pgs: 17 active+clean; 2.5 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 274 MiB/s wr, 4.38k op/s 2024-05-15T23:11:08.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:11:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:07 smithi090 ceph-mon[44434]: pgmap v606: 17 pgs: 17 active+clean; 2.5 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 274 MiB/s wr, 4.38k op/s 2024-05-15T23:11:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:11:08.588 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:11:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:08.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:11:08.692 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:11:08.738 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:08.921 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 14 16 53511 53495 238.764 171.625 0.00225592 0.00418662 2024-05-15T23:11:08.945 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:11:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:08.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:11:09.052 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:11:09.089 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:09.205 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:11:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:09.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:11:09.311 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:11:09.349 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:09.377 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:11:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:09.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:11:09.480 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:11:09.516 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:09.618 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:09.840 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:11:09.871 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:11:09.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:09.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:11:09.921 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 15 16 56064 56048 233.483 159.562 0.00309878 0.00428105 2024-05-15T23:11:09.974 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:11:10.024 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:09 smithi084 ceph-mon[39784]: pgmap v607: 17 pgs: 17 active+clean; 2.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 243 MiB/s wr, 3.89k op/s 2024-05-15T23:11:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:09 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:11:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:09 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:11:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:09 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2350516576' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:09 smithi084 ceph-mon[39784]: from='client.35213 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:10.190 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:09 smithi088 ceph-mon[44699]: pgmap v607: 17 pgs: 17 active+clean; 2.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 243 MiB/s wr, 3.89k op/s 2024-05-15T23:11:10.190 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:09 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:11:10.191 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:09 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:11:10.191 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:09 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2350516576' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:10.191 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:09 smithi088 ceph-mon[44699]: from='client.35213 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:09 smithi090 ceph-mon[44434]: pgmap v607: 17 pgs: 17 active+clean; 2.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 243 MiB/s wr, 3.89k op/s 2024-05-15T23:11:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:09 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:11:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:09 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:11:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:09 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2350516576' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:09 smithi090 ceph-mon[44434]: from='client.35213 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:10.533 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:11:10.533 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:11:10.533 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:11:10.533 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:11:10.533 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.534 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:260076 2.d 2:b949063d:::benchmark_data_smithi090_78160_object98718:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e164)", 2024-05-15T23:11:10.534 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:05.983807+0000", 2024-05-15T23:11:10.534 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 124.541677366, 2024-05-15T23:11:10.534 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15722057, 2024-05-15T23:11:10.534 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.534 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.534 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.535 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.535 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:10.535 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:10.535 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 260076 2024-05-15T23:11:10.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.535 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.536 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983807+0000", 2024-05-15T23:11:10.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.536 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.537 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983807+0000", 2024-05-15T23:11:10.537 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.537 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.537 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.537 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.537 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983810+0000", 2024-05-15T23:11:10.537 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.819e-06 2024-05-15T23:11:10.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983819+0000", 2024-05-15T23:11:10.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8270000000000004e-06 2024-05-15T23:11:10.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983834+0000", 2024-05-15T23:11:10.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5194000000000001e-05 2024-05-15T23:11:10.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983847+0000", 2024-05-15T23:11:10.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2447e-05 2024-05-15T23:11:10.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983937+0000", 2024-05-15T23:11:10.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0345000000000003e-05 2024-05-15T23:11:10.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.540 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:10.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983951+0000", 2024-05-15T23:11:10.540 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4389999999999999e-05 2024-05-15T23:11:10.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.540 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.125575+0000", 2024-05-15T23:11:10.540 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14162365199999999 2024-05-15T23:11:10.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.540 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.128147+0000", 2024-05-15T23:11:10.540 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025723819999999998 2024-05-15T23:11:10.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3,9", 2024-05-15T23:11:10.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.128256+0000", 2024-05-15T23:11:10.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00010836999999999999 2024-05-15T23:11:10.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.140453+0000", 2024-05-15T23:11:10.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.012197428999999999 2024-05-15T23:11:10.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.140473+0000", 2024-05-15T23:11:10.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9904000000000001e-05 2024-05-15T23:11:10.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.140829+0000", 2024-05-15T23:11:10.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00035547900000000002 2024-05-15T23:11:10.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.140896+0000", 2024-05-15T23:11:10.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7565999999999997e-05 2024-05-15T23:11:10.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141028+0000", 2024-05-15T23:11:10.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013176600000000001 2024-05-15T23:11:10.543 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.543 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.543 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.543 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.543 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:260078 2.d 2:b751e7be:::benchmark_data_smithi090_78160_object98720:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e164)", 2024-05-15T23:11:10.543 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:05.983869+0000", 2024-05-15T23:11:10.543 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 124.541615959, 2024-05-15T23:11:10.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15740128, 2024-05-15T23:11:10.543 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.544 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.544 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.544 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.544 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:10.544 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:10.544 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 260078 2024-05-15T23:11:10.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.544 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.544 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983869+0000", 2024-05-15T23:11:10.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983869+0000", 2024-05-15T23:11:10.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983870+0000", 2024-05-15T23:11:10.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1100000000000005e-07 2024-05-15T23:11:10.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983873+0000", 2024-05-15T23:11:10.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5149999999999998e-06 2024-05-15T23:11:10.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983880+0000", 2024-05-15T23:11:10.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8140000000000003e-06 2024-05-15T23:11:10.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983887+0000", 2024-05-15T23:11:10.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2989999999999999e-06 2024-05-15T23:11:10.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983990+0000", 2024-05-15T23:11:10.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000102845 2024-05-15T23:11:10.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.547 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:10.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983996+0000", 2024-05-15T23:11:10.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3300000000000004e-06 2024-05-15T23:11:10.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.547 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.131851+0000", 2024-05-15T23:11:10.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14785506700000001 2024-05-15T23:11:10.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.131925+0000", 2024-05-15T23:11:10.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3078999999999996e-05 2024-05-15T23:11:10.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3,9", 2024-05-15T23:11:10.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.131997+0000", 2024-05-15T23:11:10.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2636e-05 2024-05-15T23:11:10.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.140704+0000", 2024-05-15T23:11:10.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0087072350000000007 2024-05-15T23:11:10.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.140721+0000", 2024-05-15T23:11:10.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6801999999999999e-05 2024-05-15T23:11:10.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.549 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141092+0000", 2024-05-15T23:11:10.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00037110700000000002 2024-05-15T23:11:10.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.549 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141153+0000", 2024-05-15T23:11:10.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0834000000000001e-05 2024-05-15T23:11:10.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.549 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.550 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.550 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141270+0000", 2024-05-15T23:11:10.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000116906 2024-05-15T23:11:10.550 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.550 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.550 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.550 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.550 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:260080 2.d 2:bbc57010:::benchmark_data_smithi090_78160_object98722:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e164)", 2024-05-15T23:11:10.550 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:05.983898+0000", 2024-05-15T23:11:10.550 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 124.541587049, 2024-05-15T23:11:10.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.157602879, 2024-05-15T23:11:10.550 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.550 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.551 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.551 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.551 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:10.551 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:10.551 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 260080 2024-05-15T23:11:10.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.551 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983898+0000", 2024-05-15T23:11:10.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983898+0000", 2024-05-15T23:11:10.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983898+0000", 2024-05-15T23:11:10.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2600000000000002e-07 2024-05-15T23:11:10.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983903+0000", 2024-05-15T23:11:10.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2729999999999996e-06 2024-05-15T23:11:10.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983909+0000", 2024-05-15T23:11:10.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.246e-06 2024-05-15T23:11:10.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983915+0000", 2024-05-15T23:11:10.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3119999999999999e-06 2024-05-15T23:11:10.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.984026+0000", 2024-05-15T23:11:10.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000111121 2024-05-15T23:11:10.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:10.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.984035+0000", 2024-05-15T23:11:10.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5059999999999995e-06 2024-05-15T23:11:10.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.135545+0000", 2024-05-15T23:11:10.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15150972700000001 2024-05-15T23:11:10.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.135685+0000", 2024-05-15T23:11:10.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140292 2024-05-15T23:11:10.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3,9", 2024-05-15T23:11:10.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.135830+0000", 2024-05-15T23:11:10.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014494900000000001 2024-05-15T23:11:10.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.140772+0000", 2024-05-15T23:11:10.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049422040000000004 2024-05-15T23:11:10.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141042+0000", 2024-05-15T23:11:10.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00026987799999999998 2024-05-15T23:11:10.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141327+0000", 2024-05-15T23:11:10.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00028460600000000002 2024-05-15T23:11:10.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141388+0000", 2024-05-15T23:11:10.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1197000000000004e-05 2024-05-15T23:11:10.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141501+0000", 2024-05-15T23:11:10.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000112842 2024-05-15T23:11:10.558 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.558 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.558 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.558 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280201 2.d 2:b0d6d7ba:::benchmark_data_smithi090_78160_object118843:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:11:10.558 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205365+0000", 2024-05-15T23:11:10.559 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.320119761, 2024-05-15T23:11:10.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.057740659999999999, 2024-05-15T23:11:10.559 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.559 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.559 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.559 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.559 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:10.559 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:10.559 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280201 2024-05-15T23:11:10.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.559 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205365+0000", 2024-05-15T23:11:10.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205365+0000", 2024-05-15T23:11:10.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205366+0000", 2024-05-15T23:11:10.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3900000000000004e-07 2024-05-15T23:11:10.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205368+0000", 2024-05-15T23:11:10.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4710000000000001e-06 2024-05-15T23:11:10.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205375+0000", 2024-05-15T23:11:10.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7490000000000001e-06 2024-05-15T23:11:10.562 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.562 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.562 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205380+0000", 2024-05-15T23:11:10.562 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0329999999999999e-06 2024-05-15T23:11:10.562 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.562 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.562 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205678+0000", 2024-05-15T23:11:10.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00029813399999999997 2024-05-15T23:11:10.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:10.563 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205681+0000", 2024-05-15T23:11:10.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0199999999999999e-06 2024-05-15T23:11:10.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.564 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.564 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.256728+0000", 2024-05-15T23:11:10.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051046646000000001 2024-05-15T23:11:10.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.564 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.564 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.256820+0000", 2024-05-15T23:11:10.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2806999999999994e-05 2024-05-15T23:11:10.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:11:10.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.256939+0000", 2024-05-15T23:11:10.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000118594 2024-05-15T23:11:10.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.263035+0000", 2024-05-15T23:11:10.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0060960889999999998 2024-05-15T23:11:10.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.263043+0000", 2024-05-15T23:11:10.566 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8490000000000006e-06 2024-05-15T23:11:10.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.263070+0000", 2024-05-15T23:11:10.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7311999999999999e-05 2024-05-15T23:11:10.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.263106+0000", 2024-05-15T23:11:10.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5317000000000001e-05 2024-05-15T23:11:10.567 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.567 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.567 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.567 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280205 2.d 2:b412651e:::benchmark_data_smithi090_78160_object118847:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:11:10.567 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205391+0000", 2024-05-15T23:11:10.567 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.32009402, 2024-05-15T23:11:10.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065516668, 2024-05-15T23:11:10.568 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.568 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.568 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.568 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.568 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:10.568 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:10.568 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280205 2024-05-15T23:11:10.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.568 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.568 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.568 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205391+0000", 2024-05-15T23:11:10.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205391+0000", 2024-05-15T23:11:10.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205391+0000", 2024-05-15T23:11:10.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5600000000000005e-07 2024-05-15T23:11:10.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205394+0000", 2024-05-15T23:11:10.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9110000000000002e-06 2024-05-15T23:11:10.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205401+0000", 2024-05-15T23:11:10.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4540000000000004e-06 2024-05-15T23:11:10.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205407+0000", 2024-05-15T23:11:10.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4489999999999996e-06 2024-05-15T23:11:10.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205692+0000", 2024-05-15T23:11:10.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00028488999999999999 2024-05-15T23:11:10.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:10.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205695+0000", 2024-05-15T23:11:10.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0079999999999998e-06 2024-05-15T23:11:10.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.260213+0000", 2024-05-15T23:11:10.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054517822000000001 2024-05-15T23:11:10.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.260299+0000", 2024-05-15T23:11:10.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5719999999999999e-05 2024-05-15T23:11:10.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:11:10.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.260368+0000", 2024-05-15T23:11:10.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9080999999999998e-05 2024-05-15T23:11:10.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.270782+0000", 2024-05-15T23:11:10.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010414614000000001 2024-05-15T23:11:10.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.270827+0000", 2024-05-15T23:11:10.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4922000000000001e-05 2024-05-15T23:11:10.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.270863+0000", 2024-05-15T23:11:10.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5775999999999998e-05 2024-05-15T23:11:10.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.574 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.574 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.270907+0000", 2024-05-15T23:11:10.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4264999999999997e-05 2024-05-15T23:11:10.574 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.574 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.574 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.574 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280225 2.d 2:bc5231b5:::benchmark_data_smithi090_78160_object118867:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:11:10.574 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205415+0000", 2024-05-15T23:11:10.574 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.320069672, 2024-05-15T23:11:10.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065647528999999996, 2024-05-15T23:11:10.574 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.575 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.575 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.575 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.575 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:10.575 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:10.575 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280225 2024-05-15T23:11:10.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.575 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.575 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205415+0000", 2024-05-15T23:11:10.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.575 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205415+0000", 2024-05-15T23:11:10.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205416+0000", 2024-05-15T23:11:10.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5000000000000003e-07 2024-05-15T23:11:10.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205418+0000", 2024-05-15T23:11:10.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5959999999999998e-06 2024-05-15T23:11:10.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205424+0000", 2024-05-15T23:11:10.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2260000000000002e-06 2024-05-15T23:11:10.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205430+0000", 2024-05-15T23:11:10.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1370000000000001e-06 2024-05-15T23:11:10.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205706+0000", 2024-05-15T23:11:10.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00027671599999999998 2024-05-15T23:11:10.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205709+0000", 2024-05-15T23:11:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9979999999999999e-06 2024-05-15T23:11:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.263133+0000", 2024-05-15T23:11:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.057423921000000003 2024-05-15T23:11:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.263199+0000", 2024-05-15T23:11:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6115e-05 2024-05-15T23:11:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:11:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.263296+0000", 2024-05-15T23:11:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6730000000000004e-05 2024-05-15T23:11:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.270963+0000", 2024-05-15T23:11:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.007667447 2024-05-15T23:11:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.270976+0000", 2024-05-15T23:11:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2887e-05 2024-05-15T23:11:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.271023+0000", 2024-05-15T23:11:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6179e-05 2024-05-15T23:11:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.271063+0000", 2024-05-15T23:11:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0027000000000001e-05 2024-05-15T23:11:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280235 2.d 2:bc9a6377:::benchmark_data_smithi090_78160_object118877:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:11:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205436+0000", 2024-05-15T23:11:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.320049004, 2024-05-15T23:11:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.090379717999999998, 2024-05-15T23:11:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280235 2024-05-15T23:11:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.582 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205436+0000", 2024-05-15T23:11:10.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205436+0000", 2024-05-15T23:11:10.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205436+0000", 2024-05-15T23:11:10.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8100000000000003e-07 2024-05-15T23:11:10.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205439+0000", 2024-05-15T23:11:10.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3980000000000002e-06 2024-05-15T23:11:10.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205445+0000", 2024-05-15T23:11:10.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1240000000000002e-06 2024-05-15T23:11:10.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.584 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205450+0000", 2024-05-15T23:11:10.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7670000000000003e-06 2024-05-15T23:11:10.584 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.584 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.584 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.584 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205720+0000", 2024-05-15T23:11:10.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00027066699999999999 2024-05-15T23:11:10.584 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.584 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.584 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:10.584 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205723+0000", 2024-05-15T23:11:10.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9950000000000001e-06 2024-05-15T23:11:10.584 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.584 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.585 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.585 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.267133+0000", 2024-05-15T23:11:10.585 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061409686999999998 2024-05-15T23:11:10.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.585 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.585 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.585 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.267257+0000", 2024-05-15T23:11:10.585 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000124232 2024-05-15T23:11:10.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.585 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.585 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:11:10.585 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.267381+0000", 2024-05-15T23:11:10.585 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012427099999999999 2024-05-15T23:11:10.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.277587+0000", 2024-05-15T23:11:10.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010205624999999999 2024-05-15T23:11:10.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295730+0000", 2024-05-15T23:11:10.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018142587000000002 2024-05-15T23:11:10.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295766+0000", 2024-05-15T23:11:10.587 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5855999999999997e-05 2024-05-15T23:11:10.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.587 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.587 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.587 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295815+0000", 2024-05-15T23:11:10.587 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9928e-05 2024-05-15T23:11:10.587 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.587 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.587 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.587 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.588 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280240 2.d 2:be3a8fd7:::benchmark_data_smithi090_78160_object118882:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:11:10.588 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205456+0000", 2024-05-15T23:11:10.588 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.32002857499999, 2024-05-15T23:11:10.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.090458063000000005, 2024-05-15T23:11:10.588 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.588 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.588 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.588 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.588 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:10.589 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:10.589 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280240 2024-05-15T23:11:10.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.589 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.589 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.589 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205456+0000", 2024-05-15T23:11:10.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.589 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.589 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205456+0000", 2024-05-15T23:11:10.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.590 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.590 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205457+0000", 2024-05-15T23:11:10.590 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1799999999999995e-07 2024-05-15T23:11:10.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205459+0000", 2024-05-15T23:11:10.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6230000000000001e-06 2024-05-15T23:11:10.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205465+0000", 2024-05-15T23:11:10.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4779999999999999e-06 2024-05-15T23:11:10.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205469+0000", 2024-05-15T23:11:10.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.369e-06 2024-05-15T23:11:10.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205735+0000", 2024-05-15T23:11:10.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00026536000000000002 2024-05-15T23:11:10.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:10.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205738+0000", 2024-05-15T23:11:10.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.999e-06 2024-05-15T23:11:10.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.593 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.271117+0000", 2024-05-15T23:11:10.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065379197999999999 2024-05-15T23:11:10.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.593 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.271186+0000", 2024-05-15T23:11:10.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9676000000000002e-05 2024-05-15T23:11:10.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:11:10.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.271263+0000", 2024-05-15T23:11:10.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6917999999999998e-05 2024-05-15T23:11:10.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.282609+0000", 2024-05-15T23:11:10.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011346078000000001 2024-05-15T23:11:10.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295860+0000", 2024-05-15T23:11:10.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013250131 2024-05-15T23:11:10.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.595 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295888+0000", 2024-05-15T23:11:10.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8734999999999998e-05 2024-05-15T23:11:10.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.595 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295914+0000", 2024-05-15T23:11:10.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5979999999999999e-05 2024-05-15T23:11:10.595 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.595 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.595 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.595 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280244 2.d 2:b735b614:::benchmark_data_smithi090_78160_object118886:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:11:10.595 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205475+0000", 2024-05-15T23:11:10.596 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.320009935, 2024-05-15T23:11:10.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.090560130000000003, 2024-05-15T23:11:10.596 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.596 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.596 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.596 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.596 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:10.596 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:10.596 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280244 2024-05-15T23:11:10.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.596 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205475+0000", 2024-05-15T23:11:10.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205475+0000", 2024-05-15T23:11:10.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205475+0000", 2024-05-15T23:11:10.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9700000000000002e-07 2024-05-15T23:11:10.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205477+0000", 2024-05-15T23:11:10.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.916e-06 2024-05-15T23:11:10.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205482+0000", 2024-05-15T23:11:10.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2800000000000003e-06 2024-05-15T23:11:10.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205486+0000", 2024-05-15T23:11:10.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1049999999999997e-06 2024-05-15T23:11:10.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.599 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.599 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205749+0000", 2024-05-15T23:11:10.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00026279 2024-05-15T23:11:10.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205752+0000", 2024-05-15T23:11:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.999e-06 2024-05-15T23:11:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.274237+0000", 2024-05-15T23:11:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068484766000000002 2024-05-15T23:11:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.274355+0000", 2024-05-15T23:11:10.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000117552 2024-05-15T23:11:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:11:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.274478+0000", 2024-05-15T23:11:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012311200000000001 2024-05-15T23:11:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.293186+0000", 2024-05-15T23:11:10.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018708460999999999 2024-05-15T23:11:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295984+0000", 2024-05-15T23:11:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027980489999999999 2024-05-15T23:11:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296011+0000", 2024-05-15T23:11:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6934999999999999e-05 2024-05-15T23:11:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296035+0000", 2024-05-15T23:11:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3768e-05 2024-05-15T23:11:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.604 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280297 2.d 2:b81e8b87:::benchmark_data_smithi090_78160_object118939:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:11:10.604 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205491+0000", 2024-05-15T23:11:10.604 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.31999323399999, 2024-05-15T23:11:10.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.090641848999999997, 2024-05-15T23:11:10.604 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.604 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.605 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.605 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.605 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:10.605 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:10.605 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280297 2024-05-15T23:11:10.605 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.605 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.606 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.606 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205491+0000", 2024-05-15T23:11:10.606 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.606 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.606 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.606 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205491+0000", 2024-05-15T23:11:10.606 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.606 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.606 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.606 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205492+0000", 2024-05-15T23:11:10.606 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8599999999999998e-07 2024-05-15T23:11:10.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.607 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.607 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205494+0000", 2024-05-15T23:11:10.607 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0169999999999999e-06 2024-05-15T23:11:10.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.607 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.607 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205498+0000", 2024-05-15T23:11:10.607 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8449999999999996e-06 2024-05-15T23:11:10.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.607 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.607 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205501+0000", 2024-05-15T23:11:10.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8289999999999999e-06 2024-05-15T23:11:10.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205764+0000", 2024-05-15T23:11:10.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00026292699999999998 2024-05-15T23:11:10.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:10.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205767+0000", 2024-05-15T23:11:10.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0369999999999999e-06 2024-05-15T23:11:10.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.277627+0000", 2024-05-15T23:11:10.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.071859982000000003 2024-05-15T23:11:10.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.277714+0000", 2024-05-15T23:11:10.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6911999999999997e-05 2024-05-15T23:11:10.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:11:10.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.277783+0000", 2024-05-15T23:11:10.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9808999999999995e-05 2024-05-15T23:11:10.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.610 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.610 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.293189+0000", 2024-05-15T23:11:10.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015405762999999999 2024-05-15T23:11:10.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.610 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.610 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296078+0000", 2024-05-15T23:11:10.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028887740000000002 2024-05-15T23:11:10.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.610 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.610 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296112+0000", 2024-05-15T23:11:10.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4139e-05 2024-05-15T23:11:10.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.611 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.611 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296133+0000", 2024-05-15T23:11:10.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.1328999999999999e-05 2024-05-15T23:11:10.611 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.611 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.611 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.611 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280303 2.d 2:b682f83f:::benchmark_data_smithi090_78160_object118945:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:11:10.611 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205504+0000", 2024-05-15T23:11:10.611 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.319980907, 2024-05-15T23:11:10.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.090740782000000006, 2024-05-15T23:11:10.612 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.612 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.612 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.612 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.612 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:10.612 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:10.612 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280303 2024-05-15T23:11:10.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.612 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.612 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.612 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205504+0000", 2024-05-15T23:11:10.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205504+0000", 2024-05-15T23:11:10.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205504+0000", 2024-05-15T23:11:10.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3099999999999999e-07 2024-05-15T23:11:10.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205506+0000", 2024-05-15T23:11:10.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.482e-06 2024-05-15T23:11:10.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205509+0000", 2024-05-15T23:11:10.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3589999999999999e-06 2024-05-15T23:11:10.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205512+0000", 2024-05-15T23:11:10.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.253e-06 2024-05-15T23:11:10.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205778+0000", 2024-05-15T23:11:10.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000265565 2024-05-15T23:11:10.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:10.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205783+0000", 2024-05-15T23:11:10.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2639999999999999e-06 2024-05-15T23:11:10.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.280629+0000", 2024-05-15T23:11:10.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.074845940999999999 2024-05-15T23:11:10.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.280715+0000", 2024-05-15T23:11:10.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5680999999999995e-05 2024-05-15T23:11:10.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:11:10.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.280796+0000", 2024-05-15T23:11:10.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1108999999999999e-05 2024-05-15T23:11:10.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295925+0000", 2024-05-15T23:11:10.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.01512958 2024-05-15T23:11:10.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.617 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.617 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296185+0000", 2024-05-15T23:11:10.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00025999600000000002 2024-05-15T23:11:10.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.617 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.617 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296219+0000", 2024-05-15T23:11:10.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3732999999999997e-05 2024-05-15T23:11:10.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.617 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296245+0000", 2024-05-15T23:11:10.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5488000000000001e-05 2024-05-15T23:11:10.618 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.618 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.618 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.618 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280320 2.d 2:b17ccbcd:::benchmark_data_smithi090_78160_object118962:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:11:10.618 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205515+0000", 2024-05-15T23:11:10.618 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.319969738, 2024-05-15T23:11:10.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.090875773000000007, 2024-05-15T23:11:10.618 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.618 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.619 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.619 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.619 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:10.619 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:10.619 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280320 2024-05-15T23:11:10.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.619 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205515+0000", 2024-05-15T23:11:10.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205515+0000", 2024-05-15T23:11:10.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205515+0000", 2024-05-15T23:11:10.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9400000000000001e-07 2024-05-15T23:11:10.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205517+0000", 2024-05-15T23:11:10.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3179999999999999e-06 2024-05-15T23:11:10.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205520+0000", 2024-05-15T23:11:10.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2880000000000001e-06 2024-05-15T23:11:10.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205523+0000", 2024-05-15T23:11:10.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7049999999999999e-06 2024-05-15T23:11:10.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205795+0000", 2024-05-15T23:11:10.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000272172 2024-05-15T23:11:10.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:10.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205799+0000", 2024-05-15T23:11:10.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5079999999999999e-06 2024-05-15T23:11:10.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.282655+0000", 2024-05-15T23:11:10.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.076855993999999997 2024-05-15T23:11:10.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.282780+0000", 2024-05-15T23:11:10.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000124289 2024-05-15T23:11:10.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:11:10.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.282900+0000", 2024-05-15T23:11:10.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012050899999999999 2024-05-15T23:11:10.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295938+0000", 2024-05-15T23:11:10.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013038320000000001 2024-05-15T23:11:10.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296305+0000", 2024-05-15T23:11:10.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00036618399999999998 2024-05-15T23:11:10.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296355+0000", 2024-05-15T23:11:10.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0559000000000002e-05 2024-05-15T23:11:10.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296391+0000", 2024-05-15T23:11:10.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5633000000000003e-05 2024-05-15T23:11:10.625 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.625 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.625 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.625 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280333 2.d 2:bf76b750:::benchmark_data_smithi090_78160_object118975:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:11:10.625 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205525+0000", 2024-05-15T23:11:10.625 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.31995935400001, 2024-05-15T23:11:10.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.090955432000000003, 2024-05-15T23:11:10.625 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.625 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.625 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.625 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.626 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:10.626 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:10.626 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280333 2024-05-15T23:11:10.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.626 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.626 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205525+0000", 2024-05-15T23:11:10.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205525+0000", 2024-05-15T23:11:10.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205526+0000", 2024-05-15T23:11:10.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8099999999999999e-07 2024-05-15T23:11:10.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205527+0000", 2024-05-15T23:11:10.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.57e-06 2024-05-15T23:11:10.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205531+0000", 2024-05-15T23:11:10.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5949999999999999e-06 2024-05-15T23:11:10.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205533+0000", 2024-05-15T23:11:10.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6450000000000001e-06 2024-05-15T23:11:10.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205820+0000", 2024-05-15T23:11:10.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000286581 2024-05-15T23:11:10.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:10.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205826+0000", 2024-05-15T23:11:10.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.699e-06 2024-05-15T23:11:10.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.284753+0000", 2024-05-15T23:11:10.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.078927319999999995 2024-05-15T23:11:10.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.284856+0000", 2024-05-15T23:11:10.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000102534 2024-05-15T23:11:10.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:11:10.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.284969+0000", 2024-05-15T23:11:10.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000113175 2024-05-15T23:11:10.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295945+0000", 2024-05-15T23:11:10.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010976067000000001 2024-05-15T23:11:10.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296430+0000", 2024-05-15T23:11:10.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00048503899999999999 2024-05-15T23:11:10.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296459+0000", 2024-05-15T23:11:10.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8742e-05 2024-05-15T23:11:10.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296481+0000", 2024-05-15T23:11:10.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2184e-05 2024-05-15T23:11:10.635 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.635 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.635 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.635 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280340 2.d 2:ba3d05d6:::benchmark_data_smithi090_78160_object118982:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:11:10.635 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205536+0000", 2024-05-15T23:11:10.635 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.319948235, 2024-05-15T23:11:10.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10257904800000001, 2024-05-15T23:11:10.636 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.636 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.636 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.636 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.636 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:10.636 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:10.636 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280340 2024-05-15T23:11:10.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.637 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205536+0000", 2024-05-15T23:11:10.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205536+0000", 2024-05-15T23:11:10.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205537+0000", 2024-05-15T23:11:10.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1600000000000002e-07 2024-05-15T23:11:10.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205538+0000", 2024-05-15T23:11:10.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.291e-06 2024-05-15T23:11:10.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205541+0000", 2024-05-15T23:11:10.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2210000000000002e-06 2024-05-15T23:11:10.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205544+0000", 2024-05-15T23:11:10.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7580000000000001e-06 2024-05-15T23:11:10.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.640 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205848+0000", 2024-05-15T23:11:10.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000303751 2024-05-15T23:11:10.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:10.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205854+0000", 2024-05-15T23:11:10.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9569999999999999e-06 2024-05-15T23:11:10.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.286820+0000", 2024-05-15T23:11:10.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080966220000000005 2024-05-15T23:11:10.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.286927+0000", 2024-05-15T23:11:10.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000106597 2024-05-15T23:11:10.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:11:10.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.287037+0000", 2024-05-15T23:11:10.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011078000000000001 2024-05-15T23:11:10.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295951+0000", 2024-05-15T23:11:10.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0089135250000000003 2024-05-15T23:11:10.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.308004+0000", 2024-05-15T23:11:10.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.012053022 2024-05-15T23:11:10.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.308063+0000", 2024-05-15T23:11:10.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8640000000000001e-05 2024-05-15T23:11:10.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.308116+0000", 2024-05-15T23:11:10.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2970000000000003e-05 2024-05-15T23:11:10.644 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.644 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.644 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.644 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280398 2.d 2:b926c6fa:::benchmark_data_smithi090_78160_object119040:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:11:10.644 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205549+0000", 2024-05-15T23:11:10.644 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 102.319935577, 2024-05-15T23:11:10.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.102650407, 2024-05-15T23:11:10.645 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.645 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.645 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.645 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.645 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:10.645 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:10.645 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280398 2024-05-15T23:11:10.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.645 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205549+0000", 2024-05-15T23:11:10.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205549+0000", 2024-05-15T23:11:10.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205549+0000", 2024-05-15T23:11:10.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.41e-07 2024-05-15T23:11:10.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.646 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.646 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205551+0000", 2024-05-15T23:11:10.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.621e-06 2024-05-15T23:11:10.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205557+0000", 2024-05-15T23:11:10.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4600000000000002e-06 2024-05-15T23:11:10.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205562+0000", 2024-05-15T23:11:10.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8619999999999999e-06 2024-05-15T23:11:10.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205876+0000", 2024-05-15T23:11:10.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313295 2024-05-15T23:11:10.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:10.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205881+0000", 2024-05-15T23:11:10.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7379999999999999e-06 2024-05-15T23:11:10.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.288803+0000", 2024-05-15T23:11:10.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082921155999999996 2024-05-15T23:11:10.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.288910+0000", 2024-05-15T23:11:10.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00010718 2024-05-15T23:11:10.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:11:10.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.289026+0000", 2024-05-15T23:11:10.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000116693 2024-05-15T23:11:10.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295957+0000", 2024-05-15T23:11:10.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0069305670000000003 2024-05-15T23:11:10.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.308144+0000", 2024-05-15T23:11:10.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.012187144 2024-05-15T23:11:10.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.308174+0000", 2024-05-15T23:11:10.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9439000000000001e-05 2024-05-15T23:11:10.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.308200+0000", 2024-05-15T23:11:10.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5911e-05 2024-05-15T23:11:10.651 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.651 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.651 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.651 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.25135.0:40487 3.a e212/202 3:5125328d:::benchmark_data_smithi090_90974_object40486:head v 212'2427, mlcod=212'2425)", 2024-05-15T23:11:10.651 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:04.441741+0000", 2024-05-15T23:11:10.651 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 6.083743449, 2024-05-15T23:11:10.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.057175583000000002, 2024-05-15T23:11:10.651 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.651 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.651 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.651 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441741+0000", 2024-05-15T23:11:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441741+0000", 2024-05-15T23:11:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441743+0000", 2024-05-15T23:11:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3039999999999999e-06 2024-05-15T23:11:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.653 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.653 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441767+0000", 2024-05-15T23:11:10.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4227e-05 2024-05-15T23:11:10.653 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.653 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.653 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.653 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441776+0000", 2024-05-15T23:11:10.653 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3659999999999998e-06 2024-05-15T23:11:10.653 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.653 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.653 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.653 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441783+0000", 2024-05-15T23:11:10.654 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4099999999999996e-06 2024-05-15T23:11:10.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.654 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.654 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.654 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441831+0000", 2024-05-15T23:11:10.654 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8869000000000003e-05 2024-05-15T23:11:10.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.654 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.654 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.654 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441851+0000", 2024-05-15T23:11:10.654 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9483999999999998e-05 2024-05-15T23:11:10.654 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.498813+0000", 2024-05-15T23:11:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056961575 2024-05-15T23:11:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.498917+0000", 2024-05-15T23:11:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000104348 2024-05-15T23:11:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.25135.0:40488 3.e e212/202 3:7576ea27:::benchmark_data_smithi090_90974_object40487:head v 212'2508, mlcod=212'2506)", 2024-05-15T23:11:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:04.441824+0000", 2024-05-15T23:11:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 6.0836604320000003, 2024-05-15T23:11:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.059217157999999999, 2024-05-15T23:11:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441824+0000", 2024-05-15T23:11:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441824+0000", 2024-05-15T23:11:10.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441825+0000", 2024-05-15T23:11:10.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1299999999999999e-07 2024-05-15T23:11:10.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.657 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441880+0000", 2024-05-15T23:11:10.657 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4540000000000003e-05 2024-05-15T23:11:10.657 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.657 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.657 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441887+0000", 2024-05-15T23:11:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9290000000000001e-06 2024-05-15T23:11:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441892+0000", 2024-05-15T23:11:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3809999999999998e-06 2024-05-15T23:11:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441939+0000", 2024-05-15T23:11:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7014000000000001e-05 2024-05-15T23:11:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.441958+0000", 2024-05-15T23:11:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9153000000000001e-05 2024-05-15T23:11:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.501006+0000", 2024-05-15T23:11:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.059048367999999997 2024-05-15T23:11:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.501041+0000", 2024-05-15T23:11:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4959999999999997e-05 2024-05-15T23:11:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.25135.0:40497 3.1 e212/202 3:8af9f528:::benchmark_data_smithi090_90974_object40496:head v 212'2560, mlcod=212'2558)", 2024-05-15T23:11:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:04.444500+0000", 2024-05-15T23:11:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 6.0809845300000003, 2024-05-15T23:11:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.059964835000000001, 2024-05-15T23:11:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.444500+0000", 2024-05-15T23:11:10.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.661 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.661 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.661 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.661 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.444500+0000", 2024-05-15T23:11:10.661 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.662 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.662 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.662 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.662 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.444502+0000", 2024-05-15T23:11:10.662 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3659999999999999e-06 2024-05-15T23:11:10.662 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.662 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.662 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.662 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.444643+0000", 2024-05-15T23:11:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014187299999999999 2024-05-15T23:11:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.498737+0000", 2024-05-15T23:11:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054093328000000003 2024-05-15T23:11:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.498787+0000", 2024-05-15T23:11:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9802000000000002e-05 2024-05-15T23:11:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.664 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.498919+0000", 2024-05-15T23:11:10.664 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013240100000000001 2024-05-15T23:11:10.664 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.664 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.664 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.664 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.499014+0000", 2024-05-15T23:11:10.664 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4686999999999997e-05 2024-05-15T23:11:10.664 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.664 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.665 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.665 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.504426+0000", 2024-05-15T23:11:10.665 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0054125370000000003 2024-05-15T23:11:10.665 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.665 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.665 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.665 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:04.504465+0000", 2024-05-15T23:11:10.665 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8841000000000003e-05 2024-05-15T23:11:10.665 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.665 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.665 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.665 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.665 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.666 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.25135.0:42481 3.1 e212/202 3:8634d1bd:::benchmark_data_smithi090_90974_object42480:head v 212'2684, mlcod=212'2682)", 2024-05-15T23:11:10.666 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:05.095348+0000", 2024-05-15T23:11:10.666 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 5.4301363739999999, 2024-05-15T23:11:10.666 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073645720999999997, 2024-05-15T23:11:10.666 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.666 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.667 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.667 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.667 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.667 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.667 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.095348+0000", 2024-05-15T23:11:10.668 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.668 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.668 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.668 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.668 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.095348+0000", 2024-05-15T23:11:10.668 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.668 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.668 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.669 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.669 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.095349+0000", 2024-05-15T23:11:10.669 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2399999999999997e-07 2024-05-15T23:11:10.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.669 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.669 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.095421+0000", 2024-05-15T23:11:10.669 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1858000000000002e-05 2024-05-15T23:11:10.669 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.669 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.669 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.669 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165008+0000", 2024-05-15T23:11:10.670 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.069586711999999995 2024-05-15T23:11:10.670 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.670 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.670 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.670 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165019+0000", 2024-05-15T23:11:10.670 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1367e-05 2024-05-15T23:11:10.670 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.670 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.671 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.671 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165299+0000", 2024-05-15T23:11:10.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00027993099999999997 2024-05-15T23:11:10.671 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.671 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.671 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.671 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165321+0000", 2024-05-15T23:11:10.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2206000000000001e-05 2024-05-15T23:11:10.671 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.671 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.671 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.671 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.168961+0000", 2024-05-15T23:11:10.671 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036400920000000002 2024-05-15T23:11:10.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.672 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.672 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.168994+0000", 2024-05-15T23:11:10.672 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2731000000000002e-05 2024-05-15T23:11:10.672 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.672 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.672 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.672 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.672 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:42490 3.3 3:c12e4973:::benchmark_data_smithi090_90974_object42489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e212)", 2024-05-15T23:11:10.672 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:05.097523+0000", 2024-05-15T23:11:10.672 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 5.4279616080000004, 2024-05-15T23:11:10.672 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.071948781000000003, 2024-05-15T23:11:10.673 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:10.673 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:10.673 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:10.673 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:10.673 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:11:10.673 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:11:10.673 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 42490 2024-05-15T23:11:10.673 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.673 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:10.673 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.673 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:10.674 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.097523+0000", 2024-05-15T23:11:10.674 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.674 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.674 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.674 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:10.674 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.097523+0000", 2024-05-15T23:11:10.674 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:10.674 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.674 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.674 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:10.674 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.097525+0000", 2024-05-15T23:11:10.674 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4950000000000001e-06 2024-05-15T23:11:10.675 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.675 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.675 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:10.675 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.097583+0000", 2024-05-15T23:11:10.675 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8448e-05 2024-05-15T23:11:10.675 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.675 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.675 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:10.675 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165001+0000", 2024-05-15T23:11:10.675 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.067417801999999999 2024-05-15T23:11:10.676 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.676 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.676 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:10.676 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165017+0000", 2024-05-15T23:11:10.676 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6470999999999999e-05 2024-05-15T23:11:10.676 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.676 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.676 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:10.676 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165134+0000", 2024-05-15T23:11:10.677 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000116842 2024-05-15T23:11:10.677 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.677 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.677 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:10.677 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165274+0000", 2024-05-15T23:11:10.677 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139338 2024-05-15T23:11:10.677 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.677 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.677 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:11:10.678 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165413+0000", 2024-05-15T23:11:10.678 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013978300000000001 2024-05-15T23:11:10.678 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.678 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.678 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:10.678 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.169342+0000", 2024-05-15T23:11:10.678 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0039291669999999999 2024-05-15T23:11:10.679 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.679 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.679 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:10.679 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.169386+0000", 2024-05-15T23:11:10.679 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3609999999999998e-05 2024-05-15T23:11:10.679 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.679 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.679 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:10.679 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.169429+0000", 2024-05-15T23:11:10.679 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2505e-05 2024-05-15T23:11:10.680 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:10.680 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:10.680 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:10.680 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.169472+0000", 2024-05-15T23:11:10.680 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3319999999999999e-05 2024-05-15T23:11:10.680 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.680 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.680 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.680 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:10.680 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:10.680 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:11:10.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:10.682 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:11:10.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:11:10.783 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:10.821 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:10.846 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:10.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:10.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:10.921 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 16 16 58688 58672 229.14 164 0.00250127 0.00436261 2024-05-15T23:11:10.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:11:10.993 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:11.020 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:11:11.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:11.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:11:11.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:10 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:11:11.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:10 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:11:11.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:10 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:11:11.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:10 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:11:11.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:10 smithi084 ceph-mon[39784]: from='client.35213 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:11:11.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:10 smithi084 ceph-mon[39784]: osdmap e213: 12 total, 12 up, 12 in 2024-05-15T23:11:11.125 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:11:11.161 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:10 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:11:11.161 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:10 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:11:11.161 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:10 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:11:11.161 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:10 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:11:11.161 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:10 smithi088 ceph-mon[44699]: from='client.35213 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:11:11.161 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:10 smithi088 ceph-mon[44699]: osdmap e213: 12 total, 12 up, 12 in 2024-05-15T23:11:11.165 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:11.296 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:11:11.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:11.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:11:11.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:10 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:11:11.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:10 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:11:11.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:10 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:11:11.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:10 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:11:11.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:10 smithi090 ceph-mon[44434]: from='client.35213 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:11:11.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:10 smithi090 ceph-mon[44434]: osdmap e213: 12 total, 12 up, 12 in 2024-05-15T23:11:11.326 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T23:11:11.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:11:11.454 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:11.529 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:11:11.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:11.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:11:11.634 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:11:11.677 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:11.711 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:11:11.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:11.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:11:11.744 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:11:11.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:11:11.854 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:11.860 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:11 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:11:11.860 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:11 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:11:11.860 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:11 smithi084 ceph-mon[39784]: pgmap v609: 17 pgs: 17 active+clean; 3.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 194 MiB/s wr, 3.10k op/s 2024-05-15T23:11:11.860 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:11 smithi084 ceph-mon[39784]: osdmap e214: 12 total, 12 up, 12 in 2024-05-15T23:11:11.923 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 17 16 61219 61203 224.94 158.188 0.00867517 0.00444404 2024-05-15T23:11:12.011 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:11:12.014 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:11 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:11:12.014 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:11 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:11:12.015 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:11 smithi088 ceph-mon[44699]: pgmap v609: 17 pgs: 17 active+clean; 3.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 194 MiB/s wr, 3.10k op/s 2024-05-15T23:11:12.015 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:11 smithi088 ceph-mon[44699]: osdmap e214: 12 total, 12 up, 12 in 2024-05-15T23:11:12.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:12.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:11:12.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:11:12.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:11 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:11:12.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:11 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:11:12.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:11 smithi090 ceph-mon[44434]: pgmap v609: 17 pgs: 17 active+clean; 3.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 194 MiB/s wr, 3.10k op/s 2024-05-15T23:11:12.151 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:11 smithi090 ceph-mon[44434]: osdmap e214: 12 total, 12 up, 12 in 2024-05-15T23:11:12.156 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:12.184 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:11:12.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:12.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:11:12.289 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:11:12.327 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:12.352 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:11:12.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:12.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:11:12.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:11:12.506 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:12.550 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:11:12.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:12.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:11:12.655 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:11:12.690 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:12.923 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 18 16 63593 63577 220.687 148.375 0.00239903 0.0045284 2024-05-15T23:11:13.015 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:11:13.053 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:11:13.091 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:11:13.133 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:11:13.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:11:13.167 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:11:13.209 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:11:13.233 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:11:13.248 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:11:13.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:13.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:11:13.268 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:11:13.297 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:11:13.329 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:11:13.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:11:13.367 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:11:13.392 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:13.405 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:11:13.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:11:13.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:11:13.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:11:13.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:11:13.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:11:13.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:11:13.412 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:11:13.412 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:11:13.412 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:11:13.412 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:11:13.412 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:11:13.412 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:11:13.413 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:11:13.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:13.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:11:13.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:13.548 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:13.574 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:13.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:13.581 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:13.683 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:11:13.737 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:13.786 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:11:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:13.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:11:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:11:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:11:13.878 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2024-05-15T23:11:13.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:11:13.923 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 19 16 65782 65766 216.273 136.812 0.00421457 0.00462273 2024-05-15T23:11:13.950 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:14.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:13 smithi088 ceph-mon[44699]: pgmap v611: 17 pgs: 17 active+clean; 3.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 200 MiB/s wr, 3.20k op/s 2024-05-15T23:11:14.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2908851824' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:11:14.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3938973217' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T23:11:14.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:13 smithi088 ceph-mon[44699]: from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T23:11:14.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:13 smithi084 ceph-mon[39784]: pgmap v611: 17 pgs: 17 active+clean; 3.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 200 MiB/s wr, 3.20k op/s 2024-05-15T23:11:14.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2908851824' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:11:14.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3938973217' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T23:11:14.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:13 smithi084 ceph-mon[39784]: from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T23:11:14.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:13 smithi090 ceph-mon[44434]: pgmap v611: 17 pgs: 17 active+clean; 3.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 200 MiB/s wr, 3.20k op/s 2024-05-15T23:11:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2908851824' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:11:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3938973217' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T23:11:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:13 smithi090 ceph-mon[44434]: from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T23:11:14.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:14.507 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:11:14.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:14.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:11:14.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:11:14.643 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T23:11:14.655 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:14.682 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:11:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:14.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:11:14.789 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:11:14.831 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:14.926 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:11:14.923378 min lat: 0.000948627 max lat: 0.133086 avg lat: 0.00469282 2024-05-15T23:11:14.926 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:11:14.926 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 20 15 68196 68181 213.006 150.938 0.00303181 0.00469282 2024-05-15T23:11:15.045 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:11:15.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:15.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:15.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:15.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:15.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:14 smithi090 ceph-mon[44434]: from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T23:11:15.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:14 smithi090 ceph-mon[44434]: osdmap e215: 12 total, 12 up, 12 in 2024-05-15T23:11:15.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:15.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:15.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:15.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:15.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:14 smithi084 ceph-mon[39784]: from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T23:11:15.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:14 smithi084 ceph-mon[39784]: osdmap e215: 12 total, 12 up, 12 in 2024-05-15T23:11:15.078 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:15.090 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 3 1 2024-05-15T23:11:15.092 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:11:15.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:15.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:11:15.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:11:15.237 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:15.261 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:11:15.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:15.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:11:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:14 smithi088 ceph-mon[44699]: from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T23:11:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:14 smithi088 ceph-mon[44699]: osdmap e215: 12 total, 12 up, 12 in 2024-05-15T23:11:15.365 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:11:15.401 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:15.429 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:11:15.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:15.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:11:15.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:11:15.572 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:11:15.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:15.601 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:11:15.702 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:11:15.740 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:15.881 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:15 smithi090 ceph-mon[44434]: pgmap v613: 17 pgs: 17 active+clean; 3.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s wr, 2.82k op/s 2024-05-15T23:11:15.923 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 21 16 70434 70418 209.52 139.812 0.00267669 0.00477122 2024-05-15T23:11:16.008 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:11:16.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:16.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:11:16.011 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:15 smithi084 ceph-mon[39784]: pgmap v613: 17 pgs: 17 active+clean; 3.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s wr, 2.82k op/s 2024-05-15T23:11:16.039 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:16.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:16.138 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:15 smithi088 ceph-mon[44699]: pgmap v613: 17 pgs: 17 active+clean; 3.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s wr, 2.82k op/s 2024-05-15T23:11:16.150 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:16.179 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:16.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:16.184 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:16.285 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:11:16.331 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:11:16.332 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:16.459 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:11:16.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:16.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:11:16.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:11:16.618 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:16.847 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:11:16.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:16.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:11:16.923 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 22 16 72631 72615 206.238 137.312 0.00512756 0.00484717 2024-05-15T23:11:17.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:11:17.213 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:11:17.213 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:11:17.257 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:17.467 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:11:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:17.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:11:17.572 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:11:17.613 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:17.644 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:11:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:17.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:11:17.750 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:11:17.782 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:17 smithi088 ceph-mon[44699]: pgmap v614: 17 pgs: 17 active+clean; 4.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 174 MiB/s wr, 2.78k op/s 2024-05-15T23:11:17.782 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:17 smithi088 ceph-mon[44699]: osdmap e216: 12 total, 12 up, 12 in 2024-05-15T23:11:17.796 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:17.904 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:11:17.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:17.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:11:17.923 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 23 16 74832 74816 203.252 137.562 0.00704096 0.00491811 2024-05-15T23:11:17.940 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T23:11:18.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:11:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:17 smithi090 ceph-mon[44434]: pgmap v614: 17 pgs: 17 active+clean; 4.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 174 MiB/s wr, 2.78k op/s 2024-05-15T23:11:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:17 smithi090 ceph-mon[44434]: osdmap e216: 12 total, 12 up, 12 in 2024-05-15T23:11:18.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:17 smithi084 ceph-mon[39784]: pgmap v614: 17 pgs: 17 active+clean; 4.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 174 MiB/s wr, 2.78k op/s 2024-05-15T23:11:18.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:17 smithi084 ceph-mon[39784]: osdmap e216: 12 total, 12 up, 12 in 2024-05-15T23:11:18.059 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:18.319 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:11:18.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:18.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:11:18.350 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:11:18.425 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:11:18.467 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:18.493 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:11:18.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:18.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:11:18.602 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:11:18.641 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:18.675 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:18.776 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:11:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:18.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:11:18.800 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:11:18.862 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:18 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3486726456' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:11:18.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:11:18.924 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 24 16 76826 76810 199.976 124.625 0.00548409 0.00499872 2024-05-15T23:11:18.931 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:18 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3486726456' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:11:18.932 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:19.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:18 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3486726456' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:11:19.089 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:11:19.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:19.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:11:19.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:11:19.230 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:19.258 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:11:19.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:19.261 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:11:19.362 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:11:19.401 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:19.761 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:11:19.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:19.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:11:19.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:19 smithi084 ceph-mon[39784]: pgmap v616: 17 pgs: 17 active+clean; 4.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.13k op/s 2024-05-15T23:11:19.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:19 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:11:19.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3486726456' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:11:19.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:19 smithi084 ceph-mon[39784]: osdmap e217: 12 total, 12 up, 12 in 2024-05-15T23:11:19.866 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:11:19.901 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:19.925 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 25 16 78987 78971 197.368 135.062 0.00482699 0.00506499 2024-05-15T23:11:19.932 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:11:19.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:19.937 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:11:20.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:20.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:19 smithi090 ceph-mon[44434]: pgmap v616: 17 pgs: 17 active+clean; 4.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.13k op/s 2024-05-15T23:11:20.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:19 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:11:20.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3486726456' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:11:20.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:19 smithi090 ceph-mon[44434]: osdmap e217: 12 total, 12 up, 12 in 2024-05-15T23:11:20.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:19 smithi088 ceph-mon[44699]: pgmap v616: 17 pgs: 17 active+clean; 4.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.13k op/s 2024-05-15T23:11:20.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:19 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:11:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3486726456' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:11:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:19 smithi088 ceph-mon[44699]: osdmap e217: 12 total, 12 up, 12 in 2024-05-15T23:11:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:20.098 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:20.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:20.101 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:20.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:11:20.254 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:20.413 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:11:20.414 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:11:20.801 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:20 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/40170414' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:11:20.809 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-05-15T23:11:20.926 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 26 16 80986 80970 194.582 124.938 0.00407844 0.00513783 2024-05-15T23:11:21.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:20 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/40170414' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:11:21.096 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:20 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/40170414' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:11:21.490 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T23:11:21.806 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:21 smithi088 ceph-mon[44699]: pgmap v618: 17 pgs: 17 active+clean; 4.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 149 MiB/s wr, 2.38k op/s 2024-05-15T23:11:21.806 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:21 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/40170414' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:11:21.806 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:21 smithi088 ceph-mon[44699]: osdmap e218: 12 total, 12 up, 12 in 2024-05-15T23:11:21.806 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:21 smithi088 ceph-mon[44699]: osdmap e219: 12 total, 12 up, 12 in 2024-05-15T23:11:21.859 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:11:21.925 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 27 16 83082 83066 192.228 131 0.00317049 0.00520096 2024-05-15T23:11:22.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:21 smithi090 ceph-mon[44434]: pgmap v618: 17 pgs: 17 active+clean; 4.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 149 MiB/s wr, 2.38k op/s 2024-05-15T23:11:22.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:21 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/40170414' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:11:22.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:21 smithi090 ceph-mon[44434]: osdmap e218: 12 total, 12 up, 12 in 2024-05-15T23:11:22.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:21 smithi090 ceph-mon[44434]: osdmap e219: 12 total, 12 up, 12 in 2024-05-15T23:11:22.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:21 smithi084 ceph-mon[39784]: pgmap v618: 17 pgs: 17 active+clean; 4.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 149 MiB/s wr, 2.38k op/s 2024-05-15T23:11:22.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:21 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/40170414' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:11:22.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:21 smithi084 ceph-mon[39784]: osdmap e218: 12 total, 12 up, 12 in 2024-05-15T23:11:22.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:21 smithi084 ceph-mon[39784]: osdmap e219: 12 total, 12 up, 12 in 2024-05-15T23:11:22.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 4 1 2024-05-15T23:11:22.427 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:11:22.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:22.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:11:22.475 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:11:22.508 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:11:22.533 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:11:22.539 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:11:22.574 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:22.578 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:11:22.579 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:22.614 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:11:22.652 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:11:22.677 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:11:22.712 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:11:22.746 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:11:22.775 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:11:22.776 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:11:22.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:22.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:11:22.803 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:11:22.839 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:11:22.846 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:11:22.846 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:11:22.846 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:11:22.846 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:11:22.846 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:11:22.846 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:11:22.846 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:11:22.847 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:11:22.847 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:11:22.847 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:11:22.847 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:11:22.847 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:11:22.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2671434151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:11:22.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:22.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:22.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:22.880 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:11:22.925 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 28 16 84992 84976 189.626 119.375 0.0115325 0.00527172 2024-05-15T23:11:22.936 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2671434151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:11:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.139 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:11:23.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2671434151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:11:23.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.603 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:11:23.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:23.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:11:23.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:11:23.744 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:23 smithi084 ceph-mon[39784]: pgmap v621: 17 pgs: 17 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.24k op/s 2024-05-15T23:11:23.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.926 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 29 16 86592 86576 186.536 100 0.00791946 0.00535975 2024-05-15T23:11:23.945 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:11:23.945 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:11:23.945 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2024-05-15T23:11:23.945 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:11:23.969 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.969 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.969 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.969 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.970 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.970 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.970 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.970 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:23 smithi088 ceph-mon[44699]: pgmap v621: 17 pgs: 17 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.24k op/s 2024-05-15T23:11:23.970 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:23.970 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:24.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:24.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:23 smithi090 ceph-mon[44434]: pgmap v621: 17 pgs: 17 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.24k op/s 2024-05-15T23:11:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:24.753 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T23:11:24.926 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 30 16 88645 88629 184.595 128.312 0.00851344 0.00541592 2024-05-15T23:11:24.942 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:11:24.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:24.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:11:24.961 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:25.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:11:25.091 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:25.116 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:11:25.119 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:11:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:25.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:11:25.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:25.252 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:25.285 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:25.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:25.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:25.390 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:11:25.426 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:25.455 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:11:25.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:25.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:11:25.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:11:25.593 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:25.618 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:11:25.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:25.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:11:25.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:25.755 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:25.782 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:25.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:25.787 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:25.889 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:11:25.926 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 31 16 90462 90446 182.304 113.562 0.0135426 0.00548395 2024-05-15T23:11:25.959 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:26.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:25 smithi090 ceph-mon[44434]: pgmap v622: 17 pgs: 17 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.18k op/s 2024-05-15T23:11:26.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:25 smithi088 ceph-mon[44699]: pgmap v622: 17 pgs: 17 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.18k op/s 2024-05-15T23:11:26.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:25 smithi084 ceph-mon[39784]: pgmap v622: 17 pgs: 17 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.18k op/s 2024-05-15T23:11:26.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:26.167 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:11:26.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:11:26.268 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:11:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:26.518 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:11:26.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:26.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:11:26.622 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:11:26.660 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:26.736 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:11:26.800 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2024-05-15T23:11:26.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:26.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:11:26.906 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:11:26.926 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 32 16 92351 92335 180.296 118.062 0.0132333 0.00554439 2024-05-15T23:11:26.937 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:26.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:11:27.027 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:26 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:11:27.027 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:26 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:11:27.027 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:26 smithi084 ceph-mon[39784]: osdmap e220: 12 total, 12 up, 12 in 2024-05-15T23:11:27.027 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:26 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:11:27.028 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:26 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:11:27.028 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:26 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3209530233' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:11:27.028 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:26 smithi084 ceph-mon[39784]: from='client.25195 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:11:27.028 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:26 smithi084 ceph-mon[39784]: from='client.25195 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:11:27.028 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:26 smithi084 ceph-mon[39784]: osdmap e221: 12 total, 12 up, 12 in 2024-05-15T23:11:27.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:26 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:11:27.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:26 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:11:27.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:26 smithi088 ceph-mon[44699]: osdmap e220: 12 total, 12 up, 12 in 2024-05-15T23:11:27.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:26 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:11:27.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:26 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:11:27.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:26 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3209530233' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:11:27.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:26 smithi088 ceph-mon[44699]: from='client.25195 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:11:27.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:26 smithi088 ceph-mon[44699]: from='client.25195 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:11:27.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:26 smithi088 ceph-mon[44699]: osdmap e221: 12 total, 12 up, 12 in 2024-05-15T23:11:27.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:26 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:11:27.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:26 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:11:27.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:26 smithi090 ceph-mon[44434]: osdmap e220: 12 total, 12 up, 12 in 2024-05-15T23:11:27.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:26 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:11:27.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:26 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:11:27.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:26 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3209530233' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:11:27.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:26 smithi090 ceph-mon[44434]: from='client.25195 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:11:27.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:26 smithi090 ceph-mon[44434]: from='client.25195 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:11:27.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:26 smithi090 ceph-mon[44434]: osdmap e221: 12 total, 12 up, 12 in 2024-05-15T23:11:27.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:27 smithi084 ceph-mon[39784]: pgmap v624: 17 pgs: 17 active+clean; 5.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.17k op/s 2024-05-15T23:11:27.913 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:27.914 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 5 1 2024-05-15T23:11:27.926 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 33 16 94058 94042 178.066 106.688 0.0112904 0.00561443 2024-05-15T23:11:28.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:27 smithi090 ceph-mon[44434]: pgmap v624: 17 pgs: 17 active+clean; 5.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.17k op/s 2024-05-15T23:11:28.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:27 smithi088 ceph-mon[44699]: pgmap v624: 17 pgs: 17 active+clean; 5.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.17k op/s 2024-05-15T23:11:28.834 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:28.877 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:11:28.926 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 34 16 95819 95803 176.066 110.062 0.0043564 0.00567765 2024-05-15T23:11:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2638387934' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:28 smithi084 ceph-mon[39784]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:29.166 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2638387934' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:29.166 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:28 smithi088 ceph-mon[44699]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:29.204 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:11:29.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:29.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:11:29.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2638387934' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:29.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:28 smithi090 ceph-mon[44434]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:29.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:29.348 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:29.371 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:29.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:29.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:29.479 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:11:29.524 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:29.555 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:11:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:29.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:11:29.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:11:29.690 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:29.718 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:11:29.720 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:11:29.720 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:11:29.720 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:11:29.720 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:11:29.720 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:11:29.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:29.725 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:11:29.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:11:29.855 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:29.926 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 35 16 97487 97471 174.014 104.25 0.00544611 0.0057455 2024-05-15T23:11:30.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:29 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:11:30.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:29 smithi090 ceph-mon[44434]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:11:30.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:29 smithi090 ceph-mon[44434]: osdmap e222: 12 total, 12 up, 12 in 2024-05-15T23:11:30.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:11:30.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:11:30.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:11:30.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:11:30.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:11:30.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:11:30.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:29 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:11:30.094 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:29 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:11:30.094 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:29 smithi088 ceph-mon[44699]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:11:30.094 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:29 smithi088 ceph-mon[44699]: osdmap e222: 12 total, 12 up, 12 in 2024-05-15T23:11:30.094 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:29 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:11:30.094 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:29 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:11:30.094 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:29 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:11:30.094 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:29 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:11:30.095 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:29 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:11:30.095 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:29 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:11:30.095 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:29 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:11:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:29 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:11:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:29 smithi084 ceph-mon[39784]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:11:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:29 smithi084 ceph-mon[39784]: osdmap e222: 12 total, 12 up, 12 in 2024-05-15T23:11:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:11:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:11:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:11:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:11:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:11:30.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:11:30.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:29 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:11:30.486 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T23:11:30.879 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:11:30.926 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 36 16 99226 99210 172.199 108.688 0.0042292 0.00580586 2024-05-15T23:11:31.926 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 37 16 100923 100907 170.412 106.062 0.00846055 0.00586693 2024-05-15T23:11:32.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:31 smithi084 ceph-mon[39784]: pgmap v628: 17 pgs: 17 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 113 MiB/s wr, 1.81k op/s 2024-05-15T23:11:32.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:31 smithi084 ceph-mon[39784]: osdmap e223: 12 total, 12 up, 12 in 2024-05-15T23:11:32.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:31 smithi090 ceph-mon[44434]: pgmap v628: 17 pgs: 17 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 113 MiB/s wr, 1.81k op/s 2024-05-15T23:11:32.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:31 smithi090 ceph-mon[44434]: osdmap e223: 12 total, 12 up, 12 in 2024-05-15T23:11:32.317 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:31 smithi088 ceph-mon[44699]: pgmap v628: 17 pgs: 17 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 113 MiB/s wr, 1.81k op/s 2024-05-15T23:11:32.317 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:31 smithi088 ceph-mon[44699]: osdmap e223: 12 total, 12 up, 12 in 2024-05-15T23:11:32.926 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 38 16 102637 102621 168.747 107.125 0.00706931 0.00592449 2024-05-15T23:11:33.034 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:11:33.034 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:11:33.034 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:11:33.035 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:11:33.035 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.035 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16826 2.7 2:e191f622:::benchmark_data_smithi090_78160_object16825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.035 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.017535+0000", 2024-05-15T23:11:33.035 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.01290629800002, 2024-05-15T23:11:33.035 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.112188986, 2024-05-15T23:11:33.035 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.035 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.035 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.035 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.035 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.035 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.035 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16826 2024-05-15T23:11:33.035 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.036 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.036 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.036 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.036 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017535+0000", 2024-05-15T23:11:33.036 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.036 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.036 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.036 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.036 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017535+0000", 2024-05-15T23:11:33.036 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.036 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.036 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.036 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.036 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017538+0000", 2024-05-15T23:11:33.037 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0340000000000001e-06 2024-05-15T23:11:33.037 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.037 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.037 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.037 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017667+0000", 2024-05-15T23:11:33.037 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012813000000000001 2024-05-15T23:11:33.037 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.037 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.037 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.037 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017682+0000", 2024-05-15T23:11:33.037 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5699e-05 2024-05-15T23:11:33.037 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.037 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.037 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.038 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017694+0000", 2024-05-15T23:11:33.038 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2156e-05 2024-05-15T23:11:33.038 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.038 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.038 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.038 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017787+0000", 2024-05-15T23:11:33.038 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2972000000000004e-05 2024-05-15T23:11:33.038 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.038 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.038 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.038 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017809+0000", 2024-05-15T23:11:33.038 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2014e-05 2024-05-15T23:11:33.038 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.038 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.039 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.039 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.126600+0000", 2024-05-15T23:11:33.039 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.108790918 2024-05-15T23:11:33.039 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.039 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.039 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.039 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.126718+0000", 2024-05-15T23:11:33.039 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011804800000000001 2024-05-15T23:11:33.039 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.039 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.039 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.039 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.126855+0000", 2024-05-15T23:11:33.039 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000136546 2024-05-15T23:11:33.039 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.039 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.040 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.040 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129204+0000", 2024-05-15T23:11:33.040 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023486689999999998 2024-05-15T23:11:33.040 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.040 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.040 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.040 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129254+0000", 2024-05-15T23:11:33.040 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0640999999999998e-05 2024-05-15T23:11:33.040 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.040 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.040 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.040 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129380+0000", 2024-05-15T23:11:33.040 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012590400000000001 2024-05-15T23:11:33.040 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.041 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.041 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.041 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129430+0000", 2024-05-15T23:11:33.041 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0371999999999998e-05 2024-05-15T23:11:33.041 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.041 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.041 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.041 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129724+0000", 2024-05-15T23:11:33.041 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00029388299999999999 2024-05-15T23:11:33.041 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.041 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.041 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.041 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.041 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.042 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16861 2.7 2:e22f6f28:::benchmark_data_smithi090_78160_object16860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.042 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.017713+0000", 2024-05-15T23:11:33.042 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.01272819799999, 2024-05-15T23:11:33.042 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.113245968, 2024-05-15T23:11:33.042 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.042 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.042 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.042 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.042 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.042 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.042 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16861 2024-05-15T23:11:33.043 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.043 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.043 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.043 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.043 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017713+0000", 2024-05-15T23:11:33.043 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.043 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.043 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.043 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.043 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017713+0000", 2024-05-15T23:11:33.043 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.043 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.043 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.043 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.044 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017714+0000", 2024-05-15T23:11:33.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1299999999999999e-07 2024-05-15T23:11:33.044 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.044 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.044 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017745+0000", 2024-05-15T23:11:33.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0784999999999997e-05 2024-05-15T23:11:33.044 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.044 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.044 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017751+0000", 2024-05-15T23:11:33.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6319999999999996e-06 2024-05-15T23:11:33.044 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.044 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017755+0000", 2024-05-15T23:11:33.045 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1880000000000004e-06 2024-05-15T23:11:33.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.045 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017844+0000", 2024-05-15T23:11:33.045 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.9328999999999998e-05 2024-05-15T23:11:33.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.045 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017864+0000", 2024-05-15T23:11:33.045 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0159000000000001e-05 2024-05-15T23:11:33.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.127841+0000", 2024-05-15T23:11:33.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10997657399999999 2024-05-15T23:11:33.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.127934+0000", 2024-05-15T23:11:33.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3522000000000003e-05 2024-05-15T23:11:33.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.128048+0000", 2024-05-15T23:11:33.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000113461 2024-05-15T23:11:33.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129335+0000", 2024-05-15T23:11:33.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0012868059999999999 2024-05-15T23:11:33.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130512+0000", 2024-05-15T23:11:33.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001176781 2024-05-15T23:11:33.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130637+0000", 2024-05-15T23:11:33.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012557399999999999 2024-05-15T23:11:33.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130673+0000", 2024-05-15T23:11:33.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5803000000000003e-05 2024-05-15T23:11:33.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130959+0000", 2024-05-15T23:11:33.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00028654100000000002 2024-05-15T23:11:33.048 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.048 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.048 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.049 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16878 2.7 2:eadea6ce:::benchmark_data_smithi090_78160_object16877:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.049 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.017762+0000", 2024-05-15T23:11:33.049 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.01267981900003, 2024-05-15T23:11:33.049 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.113672233, 2024-05-15T23:11:33.049 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.049 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.049 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.049 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.049 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.049 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.049 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16878 2024-05-15T23:11:33.049 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.049 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017762+0000", 2024-05-15T23:11:33.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.050 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017762+0000", 2024-05-15T23:11:33.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.050 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017762+0000", 2024-05-15T23:11:33.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7000000000000005e-07 2024-05-15T23:11:33.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017796+0000", 2024-05-15T23:11:33.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3522999999999997e-05 2024-05-15T23:11:33.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017806+0000", 2024-05-15T23:11:33.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5580000000000006e-06 2024-05-15T23:11:33.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017810+0000", 2024-05-15T23:11:33.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3059999999999999e-06 2024-05-15T23:11:33.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017909+0000", 2024-05-15T23:11:33.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9451999999999996e-05 2024-05-15T23:11:33.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017921+0000", 2024-05-15T23:11:33.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1572e-05 2024-05-15T23:11:33.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.128568+0000", 2024-05-15T23:11:33.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.110646964 2024-05-15T23:11:33.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.128663+0000", 2024-05-15T23:11:33.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5342999999999998e-05 2024-05-15T23:11:33.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.128781+0000", 2024-05-15T23:11:33.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000117476 2024-05-15T23:11:33.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130466+0000", 2024-05-15T23:11:33.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00168541 2024-05-15T23:11:33.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.054 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.054 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130599+0000", 2024-05-15T23:11:33.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000132605 2024-05-15T23:11:33.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.054 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.054 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.131005+0000", 2024-05-15T23:11:33.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00040684899999999998 2024-05-15T23:11:33.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.054 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.054 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.055 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.131047+0000", 2024-05-15T23:11:33.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1885999999999998e-05 2024-05-15T23:11:33.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.055 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.055 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.131434+0000", 2024-05-15T23:11:33.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000386719 2024-05-15T23:11:33.055 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.055 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.055 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.055 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16883 2.7 2:e347a603:::benchmark_data_smithi090_78160_object16882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.055 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.017816+0000", 2024-05-15T23:11:33.055 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.01262597800002, 2024-05-15T23:11:33.056 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.114700729, 2024-05-15T23:11:33.056 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.056 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.056 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.056 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.056 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.056 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.056 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16883 2024-05-15T23:11:33.056 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.056 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.056 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.056 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.056 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017816+0000", 2024-05-15T23:11:33.056 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.056 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017816+0000", 2024-05-15T23:11:33.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017816+0000", 2024-05-15T23:11:33.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4899999999999995e-07 2024-05-15T23:11:33.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017890+0000", 2024-05-15T23:11:33.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3910000000000002e-05 2024-05-15T23:11:33.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017899+0000", 2024-05-15T23:11:33.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3389999999999993e-06 2024-05-15T23:11:33.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017906+0000", 2024-05-15T23:11:33.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1389999999999998e-06 2024-05-15T23:11:33.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017950+0000", 2024-05-15T23:11:33.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3900999999999999e-05 2024-05-15T23:11:33.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017954+0000", 2024-05-15T23:11:33.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5859999999999998e-06 2024-05-15T23:11:33.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129772+0000", 2024-05-15T23:11:33.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.111818023 2024-05-15T23:11:33.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129896+0000", 2024-05-15T23:11:33.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012374700000000001 2024-05-15T23:11:33.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.060 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.060 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.060 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130011+0000", 2024-05-15T23:11:33.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011510600000000001 2024-05-15T23:11:33.060 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.060 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.060 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.060 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.131450+0000", 2024-05-15T23:11:33.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001439271 2024-05-15T23:11:33.060 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.060 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.060 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.060 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.131966+0000", 2024-05-15T23:11:33.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00051586000000000004 2024-05-15T23:11:33.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.061 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.061 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.061 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132045+0000", 2024-05-15T23:11:33.061 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9083000000000002e-05 2024-05-15T23:11:33.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.061 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.061 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.061 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132107+0000", 2024-05-15T23:11:33.061 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1853999999999994e-05 2024-05-15T23:11:33.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.061 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.061 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.061 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132516+0000", 2024-05-15T23:11:33.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00040926100000000001 2024-05-15T23:11:33.062 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.062 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.062 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.062 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.062 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.062 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16908 2.7 2:eabdad50:::benchmark_data_smithi090_78160_object16907:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.062 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.017916+0000", 2024-05-15T23:11:33.062 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.01252532400002, 2024-05-15T23:11:33.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11791069899999999, 2024-05-15T23:11:33.062 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.062 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.062 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.062 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.062 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.063 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.063 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16908 2024-05-15T23:11:33.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.063 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017916+0000", 2024-05-15T23:11:33.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017916+0000", 2024-05-15T23:11:33.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017917+0000", 2024-05-15T23:11:33.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0100000000000004e-07 2024-05-15T23:11:33.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017986+0000", 2024-05-15T23:11:33.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8436999999999995e-05 2024-05-15T23:11:33.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017993+0000", 2024-05-15T23:11:33.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4719999999999998e-06 2024-05-15T23:11:33.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018000+0000", 2024-05-15T23:11:33.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5030000000000002e-06 2024-05-15T23:11:33.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018030+0000", 2024-05-15T23:11:33.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0035999999999998e-05 2024-05-15T23:11:33.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018036+0000", 2024-05-15T23:11:33.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5899999999999996e-06 2024-05-15T23:11:33.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.066 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.066 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132574+0000", 2024-05-15T23:11:33.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11453821 2024-05-15T23:11:33.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.066 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.066 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132718+0000", 2024-05-15T23:11:33.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014348599999999999 2024-05-15T23:11:33.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.066 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.066 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132919+0000", 2024-05-15T23:11:33.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000200776 2024-05-15T23:11:33.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135329+0000", 2024-05-15T23:11:33.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024106230000000002 2024-05-15T23:11:33.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135392+0000", 2024-05-15T23:11:33.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3010999999999997e-05 2024-05-15T23:11:33.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135410+0000", 2024-05-15T23:11:33.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8062000000000001e-05 2024-05-15T23:11:33.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135465+0000", 2024-05-15T23:11:33.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4889000000000003e-05 2024-05-15T23:11:33.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135827+0000", 2024-05-15T23:11:33.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00036180300000000001 2024-05-15T23:11:33.068 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.068 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.069 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.069 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16916 2.7 2:e8125f82:::benchmark_data_smithi090_78160_object16915:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.069 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018012+0000", 2024-05-15T23:11:33.069 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.01243011299999, 2024-05-15T23:11:33.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.120212466, 2024-05-15T23:11:33.069 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.069 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.069 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.069 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.069 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.069 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.069 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16916 2024-05-15T23:11:33.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.070 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018012+0000", 2024-05-15T23:11:33.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018012+0000", 2024-05-15T23:11:33.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018012+0000", 2024-05-15T23:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3200000000000004e-07 2024-05-15T23:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018073+0000", 2024-05-15T23:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0924999999999997e-05 2024-05-15T23:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018081+0000", 2024-05-15T23:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8120000000000008e-06 2024-05-15T23:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018086+0000", 2024-05-15T23:11:33.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8860000000000003e-06 2024-05-15T23:11:33.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018117+0000", 2024-05-15T23:11:33.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0602000000000003e-05 2024-05-15T23:11:33.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018121+0000", 2024-05-15T23:11:33.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3139999999999997e-06 2024-05-15T23:11:33.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.133639+0000", 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.115518282 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.133762+0000", 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012241699999999999 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.133892+0000", 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000130004 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135838+0000", 2024-05-15T23:11:33.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019465039999999999 2024-05-15T23:11:33.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135910+0000", 2024-05-15T23:11:33.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1577999999999998e-05 2024-05-15T23:11:33.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.137826+0000", 2024-05-15T23:11:33.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019163439999999999 2024-05-15T23:11:33.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.137885+0000", 2024-05-15T23:11:33.075 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8805999999999999e-05 2024-05-15T23:11:33.075 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.075 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.075 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.075 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138224+0000", 2024-05-15T23:11:33.075 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00033916000000000003 2024-05-15T23:11:33.075 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.075 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.075 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.075 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.075 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.075 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16920 2.7 2:ecf12bd4:::benchmark_data_smithi090_78160_object16919:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.075 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018095+0000", 2024-05-15T23:11:33.075 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.01234636300001, 2024-05-15T23:11:33.075 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.120514865, 2024-05-15T23:11:33.076 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.076 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.076 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.076 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.076 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.076 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.076 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16920 2024-05-15T23:11:33.076 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.076 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.076 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.076 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018095+0000", 2024-05-15T23:11:33.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.076 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018095+0000", 2024-05-15T23:11:33.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018096+0000", 2024-05-15T23:11:33.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1699999999999997e-07 2024-05-15T23:11:33.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018222+0000", 2024-05-15T23:11:33.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000125718 2024-05-15T23:11:33.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018230+0000", 2024-05-15T23:11:33.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9300000000000003e-06 2024-05-15T23:11:33.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018235+0000", 2024-05-15T23:11:33.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1619999999999999e-06 2024-05-15T23:11:33.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018269+0000", 2024-05-15T23:11:33.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4529000000000001e-05 2024-05-15T23:11:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018274+0000", 2024-05-15T23:11:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5970000000000002e-06 2024-05-15T23:11:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.134501+0000", 2024-05-15T23:11:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.116226648 2024-05-15T23:11:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.134644+0000", 2024-05-15T23:11:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014388500000000001 2024-05-15T23:11:33.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.080 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.080 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.080 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.134829+0000", 2024-05-15T23:11:33.080 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000184279 2024-05-15T23:11:33.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.080 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.080 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.080 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135983+0000", 2024-05-15T23:11:33.080 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0011546989999999999 2024-05-15T23:11:33.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.080 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.080 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.080 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.136000+0000", 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6526999999999999e-05 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138278+0000", 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0022781310000000001 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138316+0000", 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7691999999999998e-05 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138610+0000", 2024-05-15T23:11:33.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00029435099999999999 2024-05-15T23:11:33.082 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.082 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.082 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.082 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.082 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.082 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16933 2.7 2:e0794838:::benchmark_data_smithi090_78160_object16932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.082 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018244+0000", 2024-05-15T23:11:33.082 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.01219761499999, 2024-05-15T23:11:33.082 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12079975699999999, 2024-05-15T23:11:33.082 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.082 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.082 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.082 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.082 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.083 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.083 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16933 2024-05-15T23:11:33.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.083 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018244+0000", 2024-05-15T23:11:33.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018244+0000", 2024-05-15T23:11:33.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018245+0000", 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9899999999999999e-07 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018310+0000", 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5549999999999994e-05 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018317+0000", 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.063e-06 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018323+0000", 2024-05-15T23:11:33.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.04e-06 2024-05-15T23:11:33.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018350+0000", 2024-05-15T23:11:33.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7500000000000001e-05 2024-05-15T23:11:33.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018354+0000", 2024-05-15T23:11:33.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1500000000000001e-06 2024-05-15T23:11:33.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.136048+0000", 2024-05-15T23:11:33.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11769423900000001 2024-05-15T23:11:33.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.136216+0000", 2024-05-15T23:11:33.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167236 2024-05-15T23:11:33.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.136400+0000", 2024-05-15T23:11:33.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018410599999999999 2024-05-15T23:11:33.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.087 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.087 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138622+0000", 2024-05-15T23:11:33.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0022219190000000002 2024-05-15T23:11:33.087 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.087 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.087 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138682+0000", 2024-05-15T23:11:33.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0417000000000001e-05 2024-05-15T23:11:33.087 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.087 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.087 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138733+0000", 2024-05-15T23:11:33.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0949999999999998e-05 2024-05-15T23:11:33.087 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138768+0000", 2024-05-15T23:11:33.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4715000000000002e-05 2024-05-15T23:11:33.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139044+0000", 2024-05-15T23:11:33.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000276073 2024-05-15T23:11:33.088 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.088 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.088 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.088 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16935 2.7 2:e64d4890:::benchmark_data_smithi090_78160_object16934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.089 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018331+0000", 2024-05-15T23:11:33.089 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.012110479, 2024-05-15T23:11:33.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12112640099999999, 2024-05-15T23:11:33.089 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.089 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.089 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.089 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.089 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.089 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.089 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16935 2024-05-15T23:11:33.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.089 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.089 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.089 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018331+0000", 2024-05-15T23:11:33.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018331+0000", 2024-05-15T23:11:33.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018332+0000", 2024-05-15T23:11:33.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4900000000000005e-07 2024-05-15T23:11:33.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018394+0000", 2024-05-15T23:11:33.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1666000000000002e-05 2024-05-15T23:11:33.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018400+0000", 2024-05-15T23:11:33.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4999999999999996e-06 2024-05-15T23:11:33.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018404+0000", 2024-05-15T23:11:33.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2640000000000002e-06 2024-05-15T23:11:33.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018437+0000", 2024-05-15T23:11:33.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3002e-05 2024-05-15T23:11:33.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018442+0000", 2024-05-15T23:11:33.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3390000000000003e-06 2024-05-15T23:11:33.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.137007+0000", 2024-05-15T23:11:33.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.118565066 2024-05-15T23:11:33.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.137133+0000", 2024-05-15T23:11:33.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012647 2024-05-15T23:11:33.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.137305+0000", 2024-05-15T23:11:33.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017200399999999999 2024-05-15T23:11:33.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139055+0000", 2024-05-15T23:11:33.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001750049 2024-05-15T23:11:33.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139096+0000", 2024-05-15T23:11:33.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0997000000000001e-05 2024-05-15T23:11:33.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139142+0000", 2024-05-15T23:11:33.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5354e-05 2024-05-15T23:11:33.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139175+0000", 2024-05-15T23:11:33.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3386000000000002e-05 2024-05-15T23:11:33.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.095 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.095 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139458+0000", 2024-05-15T23:11:33.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00028255499999999998 2024-05-15T23:11:33.095 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.095 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.095 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.095 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16947 2.7 2:edcf61d7:::benchmark_data_smithi090_78160_object16946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.095 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018413+0000", 2024-05-15T23:11:33.095 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.012028843, 2024-05-15T23:11:33.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124689859, 2024-05-15T23:11:33.095 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.095 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.095 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.096 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.096 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.096 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.096 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16947 2024-05-15T23:11:33.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.096 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018413+0000", 2024-05-15T23:11:33.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018413+0000", 2024-05-15T23:11:33.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.097 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018414+0000", 2024-05-15T23:11:33.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0999999999999998e-07 2024-05-15T23:11:33.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.097 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018469+0000", 2024-05-15T23:11:33.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5869999999999999e-05 2024-05-15T23:11:33.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.097 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018476+0000", 2024-05-15T23:11:33.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.161e-06 2024-05-15T23:11:33.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.098 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.098 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018480+0000", 2024-05-15T23:11:33.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2069999999999997e-06 2024-05-15T23:11:33.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.098 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.098 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018510+0000", 2024-05-15T23:11:33.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0154999999999998e-05 2024-05-15T23:11:33.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.098 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.098 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018515+0000", 2024-05-15T23:11:33.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.074e-06 2024-05-15T23:11:33.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139510+0000", 2024-05-15T23:11:33.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.120994801 2024-05-15T23:11:33.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139649+0000", 2024-05-15T23:11:33.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013955200000000001 2024-05-15T23:11:33.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139826+0000", 2024-05-15T23:11:33.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017705999999999999 2024-05-15T23:11:33.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142625+0000", 2024-05-15T23:11:33.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027985340000000001 2024-05-15T23:11:33.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142693+0000", 2024-05-15T23:11:33.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8279000000000001e-05 2024-05-15T23:11:33.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142747+0000", 2024-05-15T23:11:33.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3838999999999997e-05 2024-05-15T23:11:33.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142788+0000", 2024-05-15T23:11:33.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0448999999999998e-05 2024-05-15T23:11:33.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143103+0000", 2024-05-15T23:11:33.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000315168 2024-05-15T23:11:33.101 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.101 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.101 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.102 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16960 2.7 2:e8496105:::benchmark_data_smithi090_78160_object16959:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.102 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018488+0000", 2024-05-15T23:11:33.102 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.01195320099998, 2024-05-15T23:11:33.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.126728282, 2024-05-15T23:11:33.102 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.102 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.102 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.102 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.102 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.102 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.102 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16960 2024-05-15T23:11:33.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.102 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018488+0000", 2024-05-15T23:11:33.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018488+0000", 2024-05-15T23:11:33.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018489+0000", 2024-05-15T23:11:33.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5199999999999996e-07 2024-05-15T23:11:33.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018549+0000", 2024-05-15T23:11:33.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9521999999999997e-05 2024-05-15T23:11:33.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018555+0000", 2024-05-15T23:11:33.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3620000000000002e-06 2024-05-15T23:11:33.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018559+0000", 2024-05-15T23:11:33.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1479999999999999e-06 2024-05-15T23:11:33.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018588+0000", 2024-05-15T23:11:33.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8538999999999999e-05 2024-05-15T23:11:33.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018592+0000", 2024-05-15T23:11:33.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1309999999999999e-06 2024-05-15T23:11:33.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.140349+0000", 2024-05-15T23:11:33.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.121757411 2024-05-15T23:11:33.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.140492+0000", 2024-05-15T23:11:33.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014264200000000001 2024-05-15T23:11:33.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.140606+0000", 2024-05-15T23:11:33.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000114437 2024-05-15T23:11:33.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143118+0000", 2024-05-15T23:11:33.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025118330000000002 2024-05-15T23:11:33.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143168+0000", 2024-05-15T23:11:33.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9667999999999999e-05 2024-05-15T23:11:33.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.144796+0000", 2024-05-15T23:11:33.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016276979999999999 2024-05-15T23:11:33.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.144849+0000", 2024-05-15T23:11:33.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3050000000000002e-05 2024-05-15T23:11:33.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.108 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145217+0000", 2024-05-15T23:11:33.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00036808899999999999 2024-05-15T23:11:33.108 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.108 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.108 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.108 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16971 2.7 2:ead74483:::benchmark_data_smithi090_78160_object16970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.108 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018569+0000", 2024-05-15T23:11:33.108 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.01187264100002, 2024-05-15T23:11:33.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12710465400000001, 2024-05-15T23:11:33.108 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.108 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.108 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.109 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.109 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.109 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.109 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16971 2024-05-15T23:11:33.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.109 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018569+0000", 2024-05-15T23:11:33.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018569+0000", 2024-05-15T23:11:33.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.110 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.110 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.110 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018570+0000", 2024-05-15T23:11:33.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1399999999999996e-07 2024-05-15T23:11:33.110 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.110 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.110 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018621+0000", 2024-05-15T23:11:33.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1483000000000002e-05 2024-05-15T23:11:33.110 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.110 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.110 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018627+0000", 2024-05-15T23:11:33.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2020000000000001e-06 2024-05-15T23:11:33.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018631+0000", 2024-05-15T23:11:33.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9149999999999998e-06 2024-05-15T23:11:33.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018662+0000", 2024-05-15T23:11:33.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0599000000000003e-05 2024-05-15T23:11:33.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018668+0000", 2024-05-15T23:11:33.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0990000000000004e-06 2024-05-15T23:11:33.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.141123+0000", 2024-05-15T23:11:33.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12245492700000001 2024-05-15T23:11:33.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.141282+0000", 2024-05-15T23:11:33.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015872099999999999 2024-05-15T23:11:33.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.141465+0000", 2024-05-15T23:11:33.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018367399999999999 2024-05-15T23:11:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143123+0000", 2024-05-15T23:11:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016577300000000001 2024-05-15T23:11:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143216+0000", 2024-05-15T23:11:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3317000000000001e-05 2024-05-15T23:11:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145289+0000", 2024-05-15T23:11:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020730890000000002 2024-05-15T23:11:33.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145338+0000", 2024-05-15T23:11:33.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8353000000000003e-05 2024-05-15T23:11:33.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145674+0000", 2024-05-15T23:11:33.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00033583099999999998 2024-05-15T23:11:33.114 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.114 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.114 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.115 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16975 2.7 2:e5d20b63:::benchmark_data_smithi090_78160_object16974:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.115 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018639+0000", 2024-05-15T23:11:33.115 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.01180309400002, 2024-05-15T23:11:33.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12747009100000001, 2024-05-15T23:11:33.115 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.115 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.115 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.115 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.115 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.115 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.115 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16975 2024-05-15T23:11:33.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.115 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.115 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.116 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018639+0000", 2024-05-15T23:11:33.116 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.116 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.116 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.116 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.116 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018639+0000", 2024-05-15T23:11:33.116 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.116 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.116 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.116 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.116 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018639+0000", 2024-05-15T23:11:33.116 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1600000000000001e-07 2024-05-15T23:11:33.116 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.116 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.116 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018690+0000", 2024-05-15T23:11:33.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1094000000000002e-05 2024-05-15T23:11:33.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.117 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018696+0000", 2024-05-15T23:11:33.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7540000000000003e-06 2024-05-15T23:11:33.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.117 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018700+0000", 2024-05-15T23:11:33.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9330000000000003e-06 2024-05-15T23:11:33.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018728+0000", 2024-05-15T23:11:33.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7447e-05 2024-05-15T23:11:33.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018732+0000", 2024-05-15T23:11:33.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.126e-06 2024-05-15T23:11:33.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.141950+0000", 2024-05-15T23:11:33.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12321836799999999 2024-05-15T23:11:33.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142073+0000", 2024-05-15T23:11:33.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012294000000000001 2024-05-15T23:11:33.119 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142192+0000", 2024-05-15T23:11:33.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000118751 2024-05-15T23:11:33.119 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143227+0000", 2024-05-15T23:11:33.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010353199999999999 2024-05-15T23:11:33.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143266+0000", 2024-05-15T23:11:33.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9242999999999997e-05 2024-05-15T23:11:33.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145771+0000", 2024-05-15T23:11:33.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025051489999999999 2024-05-15T23:11:33.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145808+0000", 2024-05-15T23:11:33.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6304999999999999e-05 2024-05-15T23:11:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146109+0000", 2024-05-15T23:11:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000300945 2024-05-15T23:11:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16985 2.7 2:e193a409:::benchmark_data_smithi090_78160_object16984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018708+0000", 2024-05-15T23:11:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.01173416300003, 2024-05-15T23:11:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.127922702, 2024-05-15T23:11:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.122 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.122 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.122 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.122 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.122 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.122 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16985 2024-05-15T23:11:33.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.122 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.122 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.122 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018708+0000", 2024-05-15T23:11:33.122 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.123 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.123 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018708+0000", 2024-05-15T23:11:33.123 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.123 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.123 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.123 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.123 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018708+0000", 2024-05-15T23:11:33.123 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6400000000000002e-07 2024-05-15T23:11:33.123 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.123 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.123 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.123 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018762+0000", 2024-05-15T23:11:33.123 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3594000000000002e-05 2024-05-15T23:11:33.123 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018768+0000", 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9869999999999996e-06 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018772+0000", 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9600000000000002e-06 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018797+0000", 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5701e-05 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018801+0000", 2024-05-15T23:11:33.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0280000000000002e-06 2024-05-15T23:11:33.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143297+0000", 2024-05-15T23:11:33.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124495859 2024-05-15T23:11:33.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143387+0000", 2024-05-15T23:11:33.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0144999999999998e-05 2024-05-15T23:11:33.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143541+0000", 2024-05-15T23:11:33.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015373200000000001 2024-05-15T23:11:33.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145726+0000", 2024-05-15T23:11:33.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0021849360000000002 2024-05-15T23:11:33.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145737+0000", 2024-05-15T23:11:33.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0769e-05 2024-05-15T23:11:33.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146249+0000", 2024-05-15T23:11:33.127 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00051189599999999997 2024-05-15T23:11:33.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146303+0000", 2024-05-15T23:11:33.127 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3795000000000002e-05 2024-05-15T23:11:33.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146630+0000", 2024-05-15T23:11:33.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00032763600000000002 2024-05-15T23:11:33.128 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.128 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.128 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.128 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16996 2.7 2:edffe2d2:::benchmark_data_smithi090_78160_object16995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.128 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018779+0000", 2024-05-15T23:11:33.128 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.01166217100001, 2024-05-15T23:11:33.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128263827, 2024-05-15T23:11:33.128 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.128 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.128 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.128 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.128 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.129 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.129 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16996 2024-05-15T23:11:33.129 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.129 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.129 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.129 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.129 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018779+0000", 2024-05-15T23:11:33.129 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.129 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.129 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.129 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.129 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018779+0000", 2024-05-15T23:11:33.129 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.129 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.129 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.130 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.130 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018780+0000", 2024-05-15T23:11:33.130 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1600000000000001e-07 2024-05-15T23:11:33.130 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.130 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.130 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.130 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018831+0000", 2024-05-15T23:11:33.130 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1023000000000003e-05 2024-05-15T23:11:33.130 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.130 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.130 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.130 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018837+0000", 2024-05-15T23:11:33.130 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0059999999999998e-06 2024-05-15T23:11:33.130 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018841+0000", 2024-05-15T23:11:33.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9480000000000001e-06 2024-05-15T23:11:33.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018868+0000", 2024-05-15T23:11:33.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6917999999999999e-05 2024-05-15T23:11:33.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018872+0000", 2024-05-15T23:11:33.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1239999999999998e-06 2024-05-15T23:11:33.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.144067+0000", 2024-05-15T23:11:33.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.125194783 2024-05-15T23:11:33.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.144155+0000", 2024-05-15T23:11:33.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8103999999999995e-05 2024-05-15T23:11:33.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.144283+0000", 2024-05-15T23:11:33.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012799099999999999 2024-05-15T23:11:33.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145742+0000", 2024-05-15T23:11:33.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0014585710000000001 2024-05-15T23:11:33.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146189+0000", 2024-05-15T23:11:33.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00044775400000000002 2024-05-15T23:11:33.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146681+0000", 2024-05-15T23:11:33.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00049144900000000003 2024-05-15T23:11:33.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.134 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.134 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146718+0000", 2024-05-15T23:11:33.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7146e-05 2024-05-15T23:11:33.134 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.134 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.134 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.147043+0000", 2024-05-15T23:11:33.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00032529400000000001 2024-05-15T23:11:33.134 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.134 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.134 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.134 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.134 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:17021 2.7 2:e966b9dc:::benchmark_data_smithi090_78160_object17020:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:11:33.135 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018848+0000", 2024-05-15T23:11:33.135 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 323.01159338600002, 2024-05-15T23:11:33.135 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.132449714, 2024-05-15T23:11:33.135 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.135 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.135 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.135 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.135 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.135 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.135 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 17021 2024-05-15T23:11:33.135 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.135 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.135 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.135 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.135 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018848+0000", 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018848+0000", 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018849+0000", 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0500000000000003e-07 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018901+0000", 2024-05-15T23:11:33.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1912e-05 2024-05-15T23:11:33.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.137 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.137 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018907+0000", 2024-05-15T23:11:33.137 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9490000000000001e-06 2024-05-15T23:11:33.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.137 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.137 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018911+0000", 2024-05-15T23:11:33.137 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8600000000000003e-06 2024-05-15T23:11:33.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.137 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.137 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018939+0000", 2024-05-15T23:11:33.138 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8167e-05 2024-05-15T23:11:33.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.138 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018943+0000", 2024-05-15T23:11:33.138 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2540000000000003e-06 2024-05-15T23:11:33.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.138 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.147139+0000", 2024-05-15T23:11:33.138 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12819576399999999 2024-05-15T23:11:33.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.147298+0000", 2024-05-15T23:11:33.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015885199999999999 2024-05-15T23:11:33.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:11:33.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.147472+0000", 2024-05-15T23:11:33.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017376300000000001 2024-05-15T23:11:33.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.150655+0000", 2024-05-15T23:11:33.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031836740000000001 2024-05-15T23:11:33.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.150707+0000", 2024-05-15T23:11:33.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1873000000000003e-05 2024-05-15T23:11:33.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.150771+0000", 2024-05-15T23:11:33.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4231000000000002e-05 2024-05-15T23:11:33.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.150833+0000", 2024-05-15T23:11:33.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2126999999999997e-05 2024-05-15T23:11:33.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.151298+0000", 2024-05-15T23:11:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00046458300000000002 2024-05-15T23:11:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:303922 2.7 2:e31405bd:::benchmark_data_smithi090_78160_object142564:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-05-15T23:11:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:54.872353+0000", 2024-05-15T23:11:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 98.158088938999995, 2024-05-15T23:11:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11004205, 2024-05-15T23:11:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 303922 2024-05-15T23:11:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872353+0000", 2024-05-15T23:11:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872353+0000", 2024-05-15T23:11:33.143 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.143 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.143 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.143 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.143 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872353+0000", 2024-05-15T23:11:33.143 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8999999999999998e-07 2024-05-15T23:11:33.143 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.143 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.143 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.143 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872354+0000", 2024-05-15T23:11:33.143 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4619999999999999e-06 2024-05-15T23:11:33.143 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.143 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.143 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872357+0000", 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5170000000000002e-06 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872360+0000", 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1839999999999999e-06 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872535+0000", 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017519200000000001 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.144 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872543+0000", 2024-05-15T23:11:33.145 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9440000000000005e-06 2024-05-15T23:11:33.145 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.145 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.145 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.145 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.953239+0000", 2024-05-15T23:11:33.145 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080695473000000004 2024-05-15T23:11:33.145 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.145 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.145 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.145 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.953356+0000", 2024-05-15T23:11:33.145 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000117167 2024-05-15T23:11:33.145 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.145 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.145 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:11:33.145 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.953478+0000", 2024-05-15T23:11:33.146 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012165800000000001 2024-05-15T23:11:33.146 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.146 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.146 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.146 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.962174+0000", 2024-05-15T23:11:33.146 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0086966879999999993 2024-05-15T23:11:33.146 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.146 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.146 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.146 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982345+0000", 2024-05-15T23:11:33.146 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020170325999999999 2024-05-15T23:11:33.146 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.146 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.146 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.147 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982373+0000", 2024-05-15T23:11:33.147 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8050000000000001e-05 2024-05-15T23:11:33.147 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.147 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.147 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.147 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982395+0000", 2024-05-15T23:11:33.147 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2099e-05 2024-05-15T23:11:33.147 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.147 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.147 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.147 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.147 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.147 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:303954 2.7 2:e13848e9:::benchmark_data_smithi090_78160_object142596:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-05-15T23:11:33.147 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:54.872363+0000", 2024-05-15T23:11:33.147 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 98.158078414000002, 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.110107154, 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 303954 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872363+0000", 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.148 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.149 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.149 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872363+0000", 2024-05-15T23:11:33.149 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.149 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.149 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.149 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.149 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872364+0000", 2024-05-15T23:11:33.149 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9299999999999999e-07 2024-05-15T23:11:33.149 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.149 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.149 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.149 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872365+0000", 2024-05-15T23:11:33.149 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.257e-06 2024-05-15T23:11:33.149 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.149 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.150 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.150 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872368+0000", 2024-05-15T23:11:33.150 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.954e-06 2024-05-15T23:11:33.150 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.150 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.150 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.150 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872371+0000", 2024-05-15T23:11:33.150 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3409999999999999e-06 2024-05-15T23:11:33.150 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.150 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.150 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.150 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872589+0000", 2024-05-15T23:11:33.150 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000217519 2024-05-15T23:11:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872596+0000", 2024-05-15T23:11:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0679999999999999e-06 2024-05-15T23:11:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.956312+0000", 2024-05-15T23:11:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.083716805000000005 2024-05-15T23:11:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.956440+0000", 2024-05-15T23:11:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000127627 2024-05-15T23:11:33.152 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.152 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.152 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:11:33.152 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.956548+0000", 2024-05-15T23:11:33.152 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000107718 2024-05-15T23:11:33.152 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.152 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.152 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.152 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.966609+0000", 2024-05-15T23:11:33.152 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010061383 2024-05-15T23:11:33.152 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.152 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.152 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.152 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982425+0000", 2024-05-15T23:11:33.153 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015816083000000002 2024-05-15T23:11:33.153 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.153 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.153 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.153 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982451+0000", 2024-05-15T23:11:33.153 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5479000000000001e-05 2024-05-15T23:11:33.153 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.153 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.153 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.153 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982470+0000", 2024-05-15T23:11:33.153 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9627e-05 2024-05-15T23:11:33.153 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.153 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.153 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.154 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.154 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.154 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:303997 2.7 2:e15001ba:::benchmark_data_smithi090_78160_object142639:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-05-15T23:11:33.154 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:54.872374+0000", 2024-05-15T23:11:33.154 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 98.158067207000002, 2024-05-15T23:11:33.154 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.110179877, 2024-05-15T23:11:33.154 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.154 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.154 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.154 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.154 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.154 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.154 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 303997 2024-05-15T23:11:33.154 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872374+0000", 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872374+0000", 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872375+0000", 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8999999999999998e-07 2024-05-15T23:11:33.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.156 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.156 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.156 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872376+0000", 2024-05-15T23:11:33.156 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.522e-06 2024-05-15T23:11:33.156 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.156 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.156 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.156 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872379+0000", 2024-05-15T23:11:33.156 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4339999999999999e-06 2024-05-15T23:11:33.156 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.156 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.156 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.156 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872382+0000", 2024-05-15T23:11:33.156 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.512e-06 2024-05-15T23:11:33.156 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.157 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.157 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.157 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872626+0000", 2024-05-15T23:11:33.157 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000243806 2024-05-15T23:11:33.157 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.157 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.157 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.157 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872630+0000", 2024-05-15T23:11:33.157 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6720000000000002e-06 2024-05-15T23:11:33.157 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.157 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.157 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.157 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.958946+0000", 2024-05-15T23:11:33.158 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.086315975000000003 2024-05-15T23:11:33.158 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.158 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.158 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.158 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.959046+0000", 2024-05-15T23:11:33.158 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000100605 2024-05-15T23:11:33.158 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.158 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.158 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:11:33.158 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.959124+0000", 2024-05-15T23:11:33.158 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8003999999999994e-05 2024-05-15T23:11:33.158 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.158 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.158 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.159 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.971147+0000", 2024-05-15T23:11:33.159 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.012022671 2024-05-15T23:11:33.159 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.159 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.159 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.159 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982500+0000", 2024-05-15T23:11:33.159 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011352944 2024-05-15T23:11:33.159 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.159 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.159 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.159 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982535+0000", 2024-05-15T23:11:33.159 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5188999999999998e-05 2024-05-15T23:11:33.159 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.159 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.159 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.160 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982554+0000", 2024-05-15T23:11:33.160 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9253e-05 2024-05-15T23:11:33.160 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.160 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.160 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.160 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.160 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.160 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:304014 2.7 2:e75fd2a5:::benchmark_data_smithi090_78160_object142656:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-05-15T23:11:33.160 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:54.872394+0000", 2024-05-15T23:11:33.160 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 98.158048170000001, 2024-05-15T23:11:33.160 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.110272975, 2024-05-15T23:11:33.160 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:33.160 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:33.160 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:33.161 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:33.161 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:11:33.161 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:33.161 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304014 2024-05-15T23:11:33.161 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.161 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:33.161 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.161 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:33.161 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872394+0000", 2024-05-15T23:11:33.161 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.161 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.161 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.161 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:33.161 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872394+0000", 2024-05-15T23:11:33.161 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:33.162 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.162 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.162 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:33.162 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872394+0000", 2024-05-15T23:11:33.162 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3500000000000002e-07 2024-05-15T23:11:33.162 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.162 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.162 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:33.162 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872396+0000", 2024-05-15T23:11:33.162 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8250000000000001e-06 2024-05-15T23:11:33.162 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.162 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.162 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:33.162 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872400+0000", 2024-05-15T23:11:33.162 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9330000000000003e-06 2024-05-15T23:11:33.163 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.163 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.163 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:33.163 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872403+0000", 2024-05-15T23:11:33.163 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0489999999999999e-06 2024-05-15T23:11:33.163 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.163 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.163 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.163 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872646+0000", 2024-05-15T23:11:33.163 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000243615 2024-05-15T23:11:33.163 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.163 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.163 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:11:33.163 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872650+0000", 2024-05-15T23:11:33.163 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6320000000000001e-06 2024-05-15T23:11:33.164 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.164 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.164 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:33.164 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.962216+0000", 2024-05-15T23:11:33.164 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.089566093999999999 2024-05-15T23:11:33.164 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.164 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.164 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:33.164 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.962289+0000", 2024-05-15T23:11:33.164 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3048999999999998e-05 2024-05-15T23:11:33.164 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.164 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.164 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:11:33.164 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.962360+0000", 2024-05-15T23:11:33.165 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0798999999999998e-05 2024-05-15T23:11:33.165 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.165 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.165 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:11:33.165 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.971153+0000", 2024-05-15T23:11:33.165 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0087932129999999994 2024-05-15T23:11:33.165 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.165 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.165 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:33.165 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982586+0000", 2024-05-15T23:11:33.165 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011432804 2024-05-15T23:11:33.165 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.165 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.165 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:11:33.166 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982630+0000", 2024-05-15T23:11:33.166 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4020000000000002e-05 2024-05-15T23:11:33.166 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:33.166 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:33.166 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:11:33.166 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982666+0000", 2024-05-15T23:11:33.166 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6606999999999997e-05 2024-05-15T23:11:33.166 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.166 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.166 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.166 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:33.166 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:33.166 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:11:33.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:32 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:11:33.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:32 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:11:33.167 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:32 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:11:33.167 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:32 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:11:33.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:32 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:11:33.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:32 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:11:33.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T23:11:33.926 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 39 16 104390 104374 167.229 109.562 0.00488216 0.00597859 2024-05-15T23:11:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:33 smithi088 ceph-mon[44699]: pgmap v630: 17 pgs: 17 active+clean; 6.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.86k op/s 2024-05-15T23:11:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:33 smithi088 ceph-mon[44699]: osdmap e224: 12 total, 12 up, 12 in 2024-05-15T23:11:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:33 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:11:34.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:33 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:11:34.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:33 smithi084 ceph-mon[39784]: pgmap v630: 17 pgs: 17 active+clean; 6.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.86k op/s 2024-05-15T23:11:34.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:33 smithi084 ceph-mon[39784]: osdmap e224: 12 total, 12 up, 12 in 2024-05-15T23:11:34.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:33 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:11:34.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:33 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:11:34.173 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:11:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:33 smithi090 ceph-mon[44434]: pgmap v630: 17 pgs: 17 active+clean; 6.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.86k op/s 2024-05-15T23:11:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:33 smithi090 ceph-mon[44434]: osdmap e224: 12 total, 12 up, 12 in 2024-05-15T23:11:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:33 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:11:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:33 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:11:34.927 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:11:34.927211 min lat: 0.000948627 max lat: 0.219607 avg lat: 0.0060369 2024-05-15T23:11:34.927 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:11:34.927 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 40 16 106028 106012 165.605 102.375 0.00468891 0.0060369 2024-05-15T23:11:35.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:34 smithi088 ceph-mon[44699]: osdmap e225: 12 total, 12 up, 12 in 2024-05-15T23:11:35.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:34 smithi084 ceph-mon[39784]: osdmap e225: 12 total, 12 up, 12 in 2024-05-15T23:11:35.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:34 smithi090 ceph-mon[44434]: osdmap e225: 12 total, 12 up, 12 in 2024-05-15T23:11:35.927 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 41 15 107782 107767 164.241 109.688 0.00683707 0.00608754 2024-05-15T23:11:36.283 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:11:36.283 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:11:36.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:36.283 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:108340 3.d 3:bbf2a8f1:::benchmark_data_smithi090_90974_object108339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e226)", 2024-05-15T23:11:36.283 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:36.279807+0000", 2024-05-15T23:11:36.283 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.001880254, 2024-05-15T23:11:36.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001914834, 2024-05-15T23:11:36.283 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:11:36.283 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:11:36.283 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:11:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:11:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:11:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:11:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 108340 2024-05-15T23:11:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:11:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:11:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:36.279807+0000", 2024-05-15T23:11:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:11:36.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:36.279807+0000", 2024-05-15T23:11:36.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:11:36.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:36.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:36.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:11:36.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:36.279814+0000", 2024-05-15T23:11:36.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2230000000000004e-06 2024-05-15T23:11:36.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:36.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:36.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:11:36.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:36.279867+0000", 2024-05-15T23:11:36.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2890999999999999e-05 2024-05-15T23:11:36.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:36.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:36.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:11:36.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:36.279878+0000", 2024-05-15T23:11:36.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1240000000000001e-05 2024-05-15T23:11:36.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:36.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:36.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:11:36.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:36.279885+0000", 2024-05-15T23:11:36.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8580000000000002e-06 2024-05-15T23:11:36.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:36.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:36.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:11:36.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:36.279942+0000", 2024-05-15T23:11:36.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7496e-05 2024-05-15T23:11:36.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:36.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:36.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:11:36.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:36.280080+0000", 2024-05-15T23:11:36.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013770699999999999 2024-05-15T23:11:36.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:11:36.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:11:36.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 5", 2024-05-15T23:11:36.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:36.280240+0000", 2024-05-15T23:11:36.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015987 2024-05-15T23:11:36.287 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:36.287 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:11:36.287 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:36.287 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:11:36.287 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2024-05-15T23:11:36.287 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 1 2024-05-15T23:11:36.287 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:11:36.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:35 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:11:36.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:35 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:11:36.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:35 smithi090 ceph-mon[44434]: pgmap v633: 17 pgs: 17 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 117 MiB/s wr, 1.87k op/s 2024-05-15T23:11:36.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:35 smithi090 ceph-mon[44434]: osdmap e226: 12 total, 12 up, 12 in 2024-05-15T23:11:36.305 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:35 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:11:36.305 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:35 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:11:36.305 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:35 smithi088 ceph-mon[44699]: pgmap v633: 17 pgs: 17 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 117 MiB/s wr, 1.87k op/s 2024-05-15T23:11:36.305 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:35 smithi088 ceph-mon[44699]: osdmap e226: 12 total, 12 up, 12 in 2024-05-15T23:11:36.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:35 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:11:36.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:35 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:11:36.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:35 smithi084 ceph-mon[39784]: pgmap v633: 17 pgs: 17 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 117 MiB/s wr, 1.87k op/s 2024-05-15T23:11:36.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:35 smithi084 ceph-mon[39784]: osdmap e226: 12 total, 12 up, 12 in 2024-05-15T23:11:36.742 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:11:36.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:36.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:11:36.849 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:11:36.886 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:36.927 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 42 16 109447 109431 162.806 104 0.00488303 0.00614101 2024-05-15T23:11:37.090 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:11:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:37.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:11:37.097 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T23:11:37.194 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:11:37.231 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:37.259 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:11:37.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:37.262 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:11:37.363 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:11:37.409 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:37.494 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:11:37.927 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 43 16 111059 111043 161.363 100.75 0.00394648 0.00619589 2024-05-15T23:11:38.040 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:37 smithi088 ceph-mon[44699]: pgmap v635: 17 pgs: 17 active+clean; 6.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.71k op/s 2024-05-15T23:11:38.040 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:37 smithi088 ceph-mon[44699]: osdmap e227: 12 total, 12 up, 12 in 2024-05-15T23:11:38.040 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:37 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:11:38.040 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:37 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:11:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:37 smithi090 ceph-mon[44434]: pgmap v635: 17 pgs: 17 active+clean; 6.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.71k op/s 2024-05-15T23:11:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:37 smithi090 ceph-mon[44434]: osdmap e227: 12 total, 12 up, 12 in 2024-05-15T23:11:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:37 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:11:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:37 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:11:38.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:37 smithi084 ceph-mon[39784]: pgmap v635: 17 pgs: 17 active+clean; 6.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.71k op/s 2024-05-15T23:11:38.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:37 smithi084 ceph-mon[39784]: osdmap e227: 12 total, 12 up, 12 in 2024-05-15T23:11:38.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:37 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:11:38.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:37 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:11:38.576 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:11:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:38.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:11:38.682 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:11:38.694 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:11:38.694 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:11:38.694 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:11:38.694 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:11:38.695 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:11:38.695 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:11:38.695 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:11:38.695 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:11:38.695 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:11:38.695 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:11:38.695 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:11:38.725 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:11:38.740 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:38.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:11:38.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:11:38.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:11:38.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:11:38.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:11:38.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:11:38.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:11:38.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:11:38.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:11:38.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:11:38.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:11:38.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:11:38.927 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 44 16 112642 112626 159.945 98.9375 0.00877938 0.00625106 2024-05-15T23:11:39.266 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:38 smithi088 ceph-mon[44699]: osdmap e228: 12 total, 12 up, 12 in 2024-05-15T23:11:39.266 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:38 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2196628928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:11:39.266 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:39.266 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:39.266 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:39.266 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:39.266 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:38 smithi090 ceph-mon[44434]: osdmap e228: 12 total, 12 up, 12 in 2024-05-15T23:11:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:38 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2196628928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:11:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:39.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:38 smithi084 ceph-mon[39784]: osdmap e228: 12 total, 12 up, 12 in 2024-05-15T23:11:39.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:38 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2196628928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:11:39.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:39.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:39.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:39.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:39.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:39.515 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:11:39.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:39.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:11:39.581 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:11:39.582 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:11:39.633 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:11:39.672 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:39.919 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-05-15T23:11:39.927 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 45 16 114197 114181 158.55 97.1875 0.00814865 0.00630584 2024-05-15T23:11:40.036 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:11:40.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:40.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:11:40.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.174 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:39 smithi090 ceph-mon[44434]: pgmap v638: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-05-15T23:11:40.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2619291771' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:11:40.176 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:40.202 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:40.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:40.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:40.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:11:40.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:39 smithi088 ceph-mon[44699]: pgmap v638: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-05-15T23:11:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2619291771' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:11:40.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:40.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:39 smithi084 ceph-mon[39784]: pgmap v638: 17 pgs: 17 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-05-15T23:11:40.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2619291771' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:11:40.346 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:40.386 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:11:40.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:40.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:11:40.437 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T23:11:40.493 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:11:40.539 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:40.579 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:11:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:40.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:11:40.684 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:11:40.720 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:40.745 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:11:40.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:40.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:11:40.808 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:11:40.839 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 6 1 2024-05-15T23:11:40.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:11:40.889 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:40.915 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:11:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:40.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:11:40.927 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 46 16 115758 115742 157.224 97.5625 0.00699378 0.00635877 2024-05-15T23:11:41.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:41.062 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:41.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:40 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2619291771' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:11:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:40 smithi088 ceph-mon[44699]: osdmap e229: 12 total, 12 up, 12 in 2024-05-15T23:11:41.085 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:41.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:41.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:40 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2619291771' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:11:41.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:40 smithi084 ceph-mon[39784]: osdmap e229: 12 total, 12 up, 12 in 2024-05-15T23:11:41.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:11:41.253 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:41.282 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:11:41.287 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:41.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:41.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:11:41.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:40 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2619291771' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:11:41.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:40 smithi090 ceph-mon[44434]: osdmap e229: 12 total, 12 up, 12 in 2024-05-15T23:11:41.390 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:11:41.427 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:41.568 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:11:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:41.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:11:41.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:41.707 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:41.738 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:41.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:41.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:41.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:41.879 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:41.906 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:41.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:41.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:41.928 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 47 16 117300 117284 155.93 96.375 0.00602153 0.00641176 2024-05-15T23:11:42.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:11:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:41 smithi090 ceph-mon[44434]: pgmap v640: 17 pgs: 17 active+clean; 6.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T23:11:42.062 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:42.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:11:42.179 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:11:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:42.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:41 smithi084 ceph-mon[39784]: pgmap v640: 17 pgs: 17 active+clean; 6.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T23:11:42.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:11:42.228 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:41 smithi088 ceph-mon[44699]: pgmap v640: 17 pgs: 17 active+clean; 6.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T23:11:42.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:42.328 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:42.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:42.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:42.465 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:11:42.508 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:42.613 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:11:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:42.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:11:42.721 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:11:42.762 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:42.793 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:11:42.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:42.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:11:42.897 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:11:42.928 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 48 16 118789 118773 154.62 93.0625 0.00738809 0.00646558 2024-05-15T23:11:42.937 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:42.948 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:11:42.949 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:11:43.076 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:11:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:43.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:11:43.184 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:43.216 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:43.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:11:43.241 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:43.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:43.245 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:43.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:11:43.383 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:43.415 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:11:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:43.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:11:43.478 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:43.519 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:11:43.552 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:43.710 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:11:43.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:43.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:11:43.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T23:11:43.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:11:43.850 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:43.873 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:11:43.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:43.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:11:43.928 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 49 16 120136 120120 153.182 84.1875 0.00459371 0.00652657 2024-05-15T23:11:43.930 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 1 (10000) 2024-05-15T23:11:43.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:11:44.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:43 smithi090 ceph-mon[44434]: pgmap v641: 17 pgs: 17 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.69k op/s 2024-05-15T23:11:44.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:43 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2716778412' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T23:11:44.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:43 smithi084 ceph-mon[39784]: pgmap v641: 17 pgs: 17 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.69k op/s 2024-05-15T23:11:44.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:43 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2716778412' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T23:11:44.042 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:44.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:43 smithi088 ceph-mon[44699]: pgmap v641: 17 pgs: 17 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.69k op/s 2024-05-15T23:11:44.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:43 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2716778412' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T23:11:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:11:44.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:44.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:11:44.123 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:11:44.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:44.217 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:44.420 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:44.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:44.423 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:44.524 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:11:44.569 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:44.798 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 7 1 2024-05-15T23:11:44.928 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 50 16 121624 121608 151.979 93 0.00590736 0.00657839 2024-05-15T23:11:45.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2716778412' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T23:11:45.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:44 smithi084 ceph-mon[39784]: osdmap e230: 12 total, 12 up, 12 in 2024-05-15T23:11:45.140 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:11:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:45.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:11:45.159 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:45.251 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:11:45.296 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:45.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2716778412' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T23:11:45.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:44 smithi090 ceph-mon[44434]: osdmap e230: 12 total, 12 up, 12 in 2024-05-15T23:11:45.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2716778412' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T23:11:45.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:44 smithi088 ceph-mon[44699]: osdmap e230: 12 total, 12 up, 12 in 2024-05-15T23:11:45.928 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 51 16 123129 123113 150.843 94.0625 0.00793823 0.00662796 2024-05-15T23:11:46.019 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:45 smithi084 ceph-mon[39784]: pgmap v643: 17 pgs: 17 active+clean; 7.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-05-15T23:11:46.214 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:11:46.215 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:11:46.243 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:45 smithi090 ceph-mon[44434]: pgmap v643: 17 pgs: 17 active+clean; 7.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-05-15T23:11:46.289 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:46.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:45 smithi088 ceph-mon[44699]: pgmap v643: 17 pgs: 17 active+clean; 7.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-05-15T23:11:46.661 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:11:46.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:46.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:11:46.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:11:46.815 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:46.928 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 52 16 124642 124626 149.761 94.5625 0.0149193 0.00667572 2024-05-15T23:11:46.949 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:11:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:46.955 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:11:46.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T23:11:47.057 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:11:47.130 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:47.161 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:11:47.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:47.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:11:47.266 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:11:47.304 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:47.336 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:11:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:47.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:11:47.410 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:11:47.439 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:11:47.492 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:47.529 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:11:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:47.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:11:47.639 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:11:47.685 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:47.710 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:11:47.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:47.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:11:47.817 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:11:47.858 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:47.882 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:11:47.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:47.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:11:47.928 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 53 16 126056 126040 148.602 88.375 0.00469825 0.00672857 2024-05-15T23:11:47.936 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:47 smithi090 ceph-mon[44434]: pgmap v644: 17 pgs: 17 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-05-15T23:11:47.936 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:47 smithi090 ceph-mon[44434]: osdmap e231: 12 total, 12 up, 12 in 2024-05-15T23:11:47.936 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:47 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1014693773' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:11:47.987 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:11:48.036 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:48.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:47 smithi088 ceph-mon[44699]: pgmap v644: 17 pgs: 17 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-05-15T23:11:48.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:47 smithi088 ceph-mon[44699]: osdmap e231: 12 total, 12 up, 12 in 2024-05-15T23:11:48.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:47 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1014693773' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:11:48.102 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:11:48.105 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:47 smithi084 ceph-mon[39784]: pgmap v644: 17 pgs: 17 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-05-15T23:11:48.105 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:47 smithi084 ceph-mon[39784]: osdmap e231: 12 total, 12 up, 12 in 2024-05-15T23:11:48.106 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:47 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1014693773' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:11:48.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:48.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:11:48.216 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:48.252 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:48.261 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:11:48.277 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:48.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:48.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:48.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:48.414 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:48.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:48 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:11:48.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1014693773' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:11:48.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:48 smithi084 ceph-mon[39784]: osdmap e232: 12 total, 12 up, 12 in 2024-05-15T23:11:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:48.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:48.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:48.928 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 54 16 127524 127508 147.549 91.75 0.0122179 0.00677598 2024-05-15T23:11:48.980 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:11:49.021 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:49.065 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:11:49.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:49.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:11:49.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:48 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:11:49.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1014693773' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:11:49.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:48 smithi088 ceph-mon[44699]: osdmap e232: 12 total, 12 up, 12 in 2024-05-15T23:11:49.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:48 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:11:49.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1014693773' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:11:49.152 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:48 smithi090 ceph-mon[44434]: osdmap e232: 12 total, 12 up, 12 in 2024-05-15T23:11:49.172 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:11:49.230 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:49.436 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:11:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:49.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:11:49.476 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:11:49.476 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:11:49.546 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:11:49.599 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:49.701 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:11:49.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:49.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:11:49.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:11:49.845 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:11:49.928 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 55 16 128974 128958 146.514 90.625 0.0064072 0.00682383 2024-05-15T23:11:49.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:49.932 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:11:50.033 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:11:50.068 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:50.238 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:49 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:11:50.238 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:49 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:11:50.238 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:49 smithi090 ceph-mon[44434]: pgmap v647: 17 pgs: 17 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-05-15T23:11:50.238 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:49 smithi090 ceph-mon[44434]: osdmap e233: 12 total, 12 up, 12 in 2024-05-15T23:11:50.252 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:11:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:50.257 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:11:50.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T23:11:50.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:49 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:11:50.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:49 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:11:50.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:49 smithi088 ceph-mon[44699]: pgmap v647: 17 pgs: 17 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-05-15T23:11:50.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:49 smithi088 ceph-mon[44699]: osdmap e233: 12 total, 12 up, 12 in 2024-05-15T23:11:50.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:11:50.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:49 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:11:50.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:49 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:11:50.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:49 smithi084 ceph-mon[39784]: pgmap v647: 17 pgs: 17 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-05-15T23:11:50.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:49 smithi084 ceph-mon[39784]: osdmap e233: 12 total, 12 up, 12 in 2024-05-15T23:11:50.391 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:11:50.420 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:50.435 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:11:50.452 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:11:50.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:50.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:11:50.471 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:11:50.502 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:11:50.533 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:11:50.566 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:11:50.568 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:11:50.601 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:11:50.613 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:50.632 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:11:50.654 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:11:50.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:50.661 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:11:50.668 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:11:50.675 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:11:50.708 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:11:50.744 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:11:50.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:11:50.785 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:11:50.786 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:11:50.786 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:11:50.786 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:11:50.787 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:11:50.787 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:11:50.787 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:11:50.787 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:11:50.787 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:11:50.787 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:11:50.787 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:11:50.787 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:11:50.787 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:11:50.834 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:50.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:11:50.928 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 56 16 130136 130120 145.195 72.625 0.00470638 0.00688588 2024-05-15T23:11:51.020 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-05-15T23:11:51.100 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:11:51.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:51.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:11:51.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:50 smithi084 ceph-mon[39784]: 3.7 scrub starts 2024-05-15T23:11:51.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/728579639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:11:51.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:51.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:50 smithi084 ceph-mon[39784]: 3.8 scrub starts 2024-05-15T23:11:51.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:51.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:51.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:51.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/244225273' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:11:51.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:11:51.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:50 smithi090 ceph-mon[44434]: 3.7 scrub starts 2024-05-15T23:11:51.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/728579639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:11:51.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:51.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:50 smithi090 ceph-mon[44434]: 3.8 scrub starts 2024-05-15T23:11:51.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:51.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:51.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:51.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/244225273' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:11:51.246 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:51.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:50 smithi088 ceph-mon[44699]: 3.7 scrub starts 2024-05-15T23:11:51.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/728579639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:11:51.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:51.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:50 smithi088 ceph-mon[44699]: 3.8 scrub starts 2024-05-15T23:11:51.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:51.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:51.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:51.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/244225273' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:11:51.928 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 57 16 131208 131192 143.823 67 0.00747942 0.00695103 2024-05-15T23:11:52.003 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:11:52.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:52.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:11:52.108 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:11:52.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:51 smithi084 ceph-mon[39784]: 3.4 scrub starts 2024-05-15T23:11:52.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:51 smithi084 ceph-mon[39784]: pgmap v649: 17 pgs: 17 active+clean; 7.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-05-15T23:11:52.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/244225273' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:11:52.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:51 smithi084 ceph-mon[39784]: osdmap e234: 12 total, 12 up, 12 in 2024-05-15T23:11:52.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:51 smithi084 ceph-mon[39784]: 3.a scrub starts 2024-05-15T23:11:52.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:51 smithi084 ceph-mon[39784]: 3.b scrub starts 2024-05-15T23:11:52.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:51 smithi084 ceph-mon[39784]: osdmap e235: 12 total, 12 up, 12 in 2024-05-15T23:11:52.180 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:52.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:51 smithi090 ceph-mon[44434]: 3.4 scrub starts 2024-05-15T23:11:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:51 smithi090 ceph-mon[44434]: pgmap v649: 17 pgs: 17 active+clean; 7.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-05-15T23:11:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/244225273' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:11:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:51 smithi090 ceph-mon[44434]: osdmap e234: 12 total, 12 up, 12 in 2024-05-15T23:11:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:51 smithi090 ceph-mon[44434]: 3.a scrub starts 2024-05-15T23:11:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:51 smithi090 ceph-mon[44434]: 3.b scrub starts 2024-05-15T23:11:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:51 smithi090 ceph-mon[44434]: osdmap e235: 12 total, 12 up, 12 in 2024-05-15T23:11:52.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:51 smithi088 ceph-mon[44699]: 3.4 scrub starts 2024-05-15T23:11:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:11:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:51 smithi088 ceph-mon[44699]: pgmap v649: 17 pgs: 17 active+clean; 7.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-05-15T23:11:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/244225273' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:11:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:51 smithi088 ceph-mon[44699]: osdmap e234: 12 total, 12 up, 12 in 2024-05-15T23:11:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:51 smithi088 ceph-mon[44699]: 3.a scrub starts 2024-05-15T23:11:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:51 smithi088 ceph-mon[44699]: 3.b scrub starts 2024-05-15T23:11:52.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:51 smithi088 ceph-mon[44699]: osdmap e235: 12 total, 12 up, 12 in 2024-05-15T23:11:52.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 8 1 2024-05-15T23:11:52.548 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:11:52.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:52.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:11:52.556 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:52.651 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:11:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:11:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:11:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:11:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109617 2.5 2:a3ae1bf3:::benchmark_data_smithi090_78160_object109616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:11:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.320545+0000", 2024-05-15T23:11:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 299.326725775, 2024-05-15T23:11:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14569204499999999, 2024-05-15T23:11:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109617 2024-05-15T23:11:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.320545+0000", 2024-05-15T23:11:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.320545+0000", 2024-05-15T23:11:52.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.320545+0000", 2024-05-15T23:11:52.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3099999999999996e-07 2024-05-15T23:11:52.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.320574+0000", 2024-05-15T23:11:52.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8470999999999999e-05 2024-05-15T23:11:52.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.320581+0000", 2024-05-15T23:11:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6200000000000001e-06 2024-05-15T23:11:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.320586+0000", 2024-05-15T23:11:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3240000000000002e-06 2024-05-15T23:11:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324708+0000", 2024-05-15T23:11:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004121988 2024-05-15T23:11:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324832+0000", 2024-05-15T23:11:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012456899999999999 2024-05-15T23:11:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:11:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324970+0000", 2024-05-15T23:11:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013749000000000001 2024-05-15T23:11:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329072+0000", 2024-05-15T23:11:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041016309999999997 2024-05-15T23:11:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466125+0000", 2024-05-15T23:11:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13705376399999999 2024-05-15T23:11:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466188+0000", 2024-05-15T23:11:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2574000000000001e-05 2024-05-15T23:11:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466237+0000", 2024-05-15T23:11:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8782999999999997e-05 2024-05-15T23:11:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109625 2.b 2:d2ace5aa:::benchmark_data_smithi090_78160_object109624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:11:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.324751+0000", 2024-05-15T23:11:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 299.32251906900001, 2024-05-15T23:11:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.141344727, 2024-05-15T23:11:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109625 2024-05-15T23:11:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324751+0000", 2024-05-15T23:11:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324751+0000", 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324753+0000", 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7460000000000001e-06 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324773+0000", 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9851e-05 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324780+0000", 2024-05-15T23:11:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.165e-06 2024-05-15T23:11:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324786+0000", 2024-05-15T23:11:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3900000000000001e-06 2024-05-15T23:11:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324834+0000", 2024-05-15T23:11:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8701e-05 2024-05-15T23:11:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324916+0000", 2024-05-15T23:11:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1674000000000004e-05 2024-05-15T23:11:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 11", 2024-05-15T23:11:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.324995+0000", 2024-05-15T23:11:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8987999999999997e-05 2024-05-15T23:11:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329078+0000", 2024-05-15T23:11:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040834210000000003 2024-05-15T23:11:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465993+0000", 2024-05-15T23:11:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13691505700000001 2024-05-15T23:11:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466052+0000", 2024-05-15T23:11:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8922000000000003e-05 2024-05-15T23:11:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466096+0000", 2024-05-15T23:11:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3812e-05 2024-05-15T23:11:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109626 2.f 2:f5db7710:::benchmark_data_smithi090_78160_object109625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:11:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.325464+0000", 2024-05-15T23:11:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 299.32180598000002, 2024-05-15T23:11:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14049392499999999, 2024-05-15T23:11:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109626 2024-05-15T23:11:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325464+0000", 2024-05-15T23:11:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325464+0000", 2024-05-15T23:11:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325466+0000", 2024-05-15T23:11:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3289999999999999e-06 2024-05-15T23:11:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325488+0000", 2024-05-15T23:11:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.196e-05 2024-05-15T23:11:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325497+0000", 2024-05-15T23:11:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6949999999999995e-06 2024-05-15T23:11:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325504+0000", 2024-05-15T23:11:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0199999999999997e-06 2024-05-15T23:11:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325556+0000", 2024-05-15T23:11:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1062e-05 2024-05-15T23:11:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325662+0000", 2024-05-15T23:11:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106178 2024-05-15T23:11:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 9", 2024-05-15T23:11:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.325779+0000", 2024-05-15T23:11:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117321 2024-05-15T23:11:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329086+0000", 2024-05-15T23:11:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033067629999999999 2024-05-15T23:11:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329613+0000", 2024-05-15T23:11:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052711200000000004 2024-05-15T23:11:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465911+0000", 2024-05-15T23:11:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.136297687 2024-05-15T23:11:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465958+0000", 2024-05-15T23:11:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7797999999999999e-05 2024-05-15T23:11:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.24745.0:109626 2.f e127/102 ondisk, result = 0)", 2024-05-15T23:11:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.329536+0000", 2024-05-15T23:11:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 299.31773487700002, 2024-05-15T23:11:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13643081000000001, 2024-05-15T23:11:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:11:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329536+0000", 2024-05-15T23:11:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329536+0000", 2024-05-15T23:11:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329537+0000", 2024-05-15T23:11:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.59e-06 2024-05-15T23:11:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329543+0000", 2024-05-15T23:11:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.891e-06 2024-05-15T23:11:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329553+0000", 2024-05-15T23:11:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0336999999999999e-05 2024-05-15T23:11:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329560+0000", 2024-05-15T23:11:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0690000000000004e-06 2024-05-15T23:11:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329602+0000", 2024-05-15T23:11:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1211999999999998e-05 2024-05-15T23:11:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329609+0000", 2024-05-15T23:11:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6809999999999998e-06 2024-05-15T23:11:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465966+0000", 2024-05-15T23:11:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13635702999999999 2024-05-15T23:11:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.24745.0:109617 2.5 e127/102 ondisk, result = 0)", 2024-05-15T23:11:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.329717+0000", 2024-05-15T23:11:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 299.31755296, 2024-05-15T23:11:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.136528596, 2024-05-15T23:11:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:11:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329717+0000", 2024-05-15T23:11:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329717+0000", 2024-05-15T23:11:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329719+0000", 2024-05-15T23:11:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.057e-06 2024-05-15T23:11:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.329722+0000", 2024-05-15T23:11:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8290000000000001e-06 2024-05-15T23:11:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465828+0000", 2024-05-15T23:11:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13610577099999999 2024-05-15T23:11:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465840+0000", 2024-05-15T23:11:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2140999999999999e-05 2024-05-15T23:11:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466110+0000", 2024-05-15T23:11:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027013500000000001 2024-05-15T23:11:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466119+0000", 2024-05-15T23:11:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4740000000000005e-06 2024-05-15T23:11:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466246+0000", 2024-05-15T23:11:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012718899999999999 2024-05-15T23:11:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109634 2.5 2:aefd519a:::benchmark_data_smithi090_78160_object109633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:11:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.331893+0000", 2024-05-15T23:11:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 299.31537717999998, 2024-05-15T23:11:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14172101100000001, 2024-05-15T23:11:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109634 2024-05-15T23:11:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.331893+0000", 2024-05-15T23:11:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.331893+0000", 2024-05-15T23:11:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.331895+0000", 2024-05-15T23:11:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0169999999999999e-06 2024-05-15T23:11:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.331918+0000", 2024-05-15T23:11:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3051999999999999e-05 2024-05-15T23:11:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465774+0000", 2024-05-15T23:11:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.133855329 2024-05-15T23:11:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465792+0000", 2024-05-15T23:11:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8848999999999999e-05 2024-05-15T23:11:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.467040+0000", 2024-05-15T23:11:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001247822 2024-05-15T23:11:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.467148+0000", 2024-05-15T23:11:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010783199999999999 2024-05-15T23:11:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:11:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.467278+0000", 2024-05-15T23:11:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013016500000000001 2024-05-15T23:11:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.471575+0000", 2024-05-15T23:11:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004296345 2024-05-15T23:11:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.473509+0000", 2024-05-15T23:11:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019343940000000001 2024-05-15T23:11:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.473566+0000", 2024-05-15T23:11:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7225000000000002e-05 2024-05-15T23:11:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.473614+0000", 2024-05-15T23:11:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7981e-05 2024-05-15T23:11:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:109635 2.6 e127/102 2:69c20386:::benchmark_data_smithi090_78160_object109634:head v 127'6850, mlcod=127'6848)", 2024-05-15T23:11:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.333210+0000", 2024-05-15T23:11:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 299.31406021599997, 2024-05-15T23:11:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13963065999999999, 2024-05-15T23:11:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.333210+0000", 2024-05-15T23:11:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.333210+0000", 2024-05-15T23:11:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.333212+0000", 2024-05-15T23:11:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5770000000000001e-06 2024-05-15T23:11:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.333249+0000", 2024-05-15T23:11:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7134000000000001e-05 2024-05-15T23:11:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466146+0000", 2024-05-15T23:11:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.132896659 2024-05-15T23:11:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466153+0000", 2024-05-15T23:11:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8399999999999995e-06 2024-05-15T23:11:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466367+0000", 2024-05-15T23:11:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021343600000000001 2024-05-15T23:11:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466387+0000", 2024-05-15T23:11:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9922e-05 2024-05-15T23:11:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.472790+0000", 2024-05-15T23:11:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064031050000000001 2024-05-15T23:11:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.472841+0000", 2024-05-15T23:11:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0986999999999998e-05 2024-05-15T23:11:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:111051 2.b 2:dae0a3ed:::benchmark_data_smithi090_78160_object111050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:11:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:54.386337+0000", 2024-05-15T23:11:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 298.26093383800003, 2024-05-15T23:11:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063655595999999995, 2024-05-15T23:11:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 111051 2024-05-15T23:11:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386337+0000", 2024-05-15T23:11:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386337+0000", 2024-05-15T23:11:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386339+0000", 2024-05-15T23:11:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.74e-06 2024-05-15T23:11:52.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386365+0000", 2024-05-15T23:11:52.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5542999999999999e-05 2024-05-15T23:11:52.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386381+0000", 2024-05-15T23:11:52.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6158999999999999e-05 2024-05-15T23:11:52.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386392+0000", 2024-05-15T23:11:52.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0584999999999999e-05 2024-05-15T23:11:52.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386453+0000", 2024-05-15T23:11:52.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1821000000000003e-05 2024-05-15T23:11:52.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.386576+0000", 2024-05-15T23:11:52.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122977 2024-05-15T23:11:52.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 11", 2024-05-15T23:11:52.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.399570+0000", 2024-05-15T23:11:52.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012993963000000001 2024-05-15T23:11:52.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.402445+0000", 2024-05-15T23:11:52.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028750550000000001 2024-05-15T23:11:52.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.449874+0000", 2024-05-15T23:11:52.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.047428234 2024-05-15T23:11:52.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.449941+0000", 2024-05-15T23:11:52.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7588000000000004e-05 2024-05-15T23:11:52.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:54.449992+0000", 2024-05-15T23:11:52.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0930999999999997e-05 2024-05-15T23:11:52.696 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.696 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.696 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.696 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:116822 2.5 2:a4a74574:::benchmark_data_smithi090_78160_object116821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:11:52.696 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:58.388393+0000", 2024-05-15T23:11:52.696 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 294.258877568, 2024-05-15T23:11:52.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049270092000000001, 2024-05-15T23:11:52.696 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.697 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.697 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.697 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 116822 2024-05-15T23:11:52.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.697 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388393+0000", 2024-05-15T23:11:52.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388393+0000", 2024-05-15T23:11:52.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388394+0000", 2024-05-15T23:11:52.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.607e-06 2024-05-15T23:11:52.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388415+0000", 2024-05-15T23:11:52.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0684000000000001e-05 2024-05-15T23:11:52.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388424+0000", 2024-05-15T23:11:52.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2480000000000002e-06 2024-05-15T23:11:52.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388433+0000", 2024-05-15T23:11:52.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-05-15T23:11:52.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388477+0000", 2024-05-15T23:11:52.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3893000000000001e-05 2024-05-15T23:11:52.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388580+0000", 2024-05-15T23:11:52.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102902 2024-05-15T23:11:52.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:11:52.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.388686+0000", 2024-05-15T23:11:52.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106543 2024-05-15T23:11:52.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.391560+0000", 2024-05-15T23:11:52.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028739429999999999 2024-05-15T23:11:52.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.437575+0000", 2024-05-15T23:11:52.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046014688999999998 2024-05-15T23:11:52.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.437635+0000", 2024-05-15T23:11:52.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9623000000000002e-05 2024-05-15T23:11:52.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.437663+0000", 2024-05-15T23:11:52.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8350000000000001e-05 2024-05-15T23:11:52.702 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.702 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.702 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.702 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:116838 2.5 2:a8d1e181:::benchmark_data_smithi090_78160_object116837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:11:52.702 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:58.403340+0000", 2024-05-15T23:11:52.702 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 294.24393056100001, 2024-05-15T23:11:52.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046982428, 2024-05-15T23:11:52.703 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.703 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.703 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.703 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 116838 2024-05-15T23:11:52.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.703 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403340+0000", 2024-05-15T23:11:52.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403340+0000", 2024-05-15T23:11:52.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403342+0000", 2024-05-15T23:11:52.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9709999999999998e-06 2024-05-15T23:11:52.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403366+0000", 2024-05-15T23:11:52.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4416000000000001e-05 2024-05-15T23:11:52.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403382+0000", 2024-05-15T23:11:52.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5381999999999999e-05 2024-05-15T23:11:52.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403390+0000", 2024-05-15T23:11:52.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5979999999999997e-06 2024-05-15T23:11:52.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403499+0000", 2024-05-15T23:11:52.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108922 2024-05-15T23:11:52.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403604+0000", 2024-05-15T23:11:52.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000105085 2024-05-15T23:11:52.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:11:52.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.403738+0000", 2024-05-15T23:11:52.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013375400000000001 2024-05-15T23:11:52.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.406702+0000", 2024-05-15T23:11:52.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029637930000000002 2024-05-15T23:11:52.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.450211+0000", 2024-05-15T23:11:52.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.043509054999999998 2024-05-15T23:11:52.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.450268+0000", 2024-05-15T23:11:52.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6793999999999999e-05 2024-05-15T23:11:52.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:58.450322+0000", 2024-05-15T23:11:52.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4657999999999997e-05 2024-05-15T23:11:52.708 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.708 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.708 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.708 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:190851 2.b 2:d6bfb2a4:::benchmark_data_smithi090_78160_object29493:head [delete] snapc 0=[] ondisk+write+known_if_redirected e148)", 2024-05-15T23:11:52.708 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:00.419416+0000", 2024-05-15T23:11:52.708 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 232.22785447199999, 2024-05-15T23:11:52.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045686048, 2024-05-15T23:11:52.708 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.709 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.709 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.709 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.709 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.709 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.709 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 190851 2024-05-15T23:11:52.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.709 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.419416+0000", 2024-05-15T23:11:52.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.419416+0000", 2024-05-15T23:11:52.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.419416+0000", 2024-05-15T23:11:52.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2600000000000002e-07 2024-05-15T23:11:52.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.419419+0000", 2024-05-15T23:11:52.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.565e-06 2024-05-15T23:11:52.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.419424+0000", 2024-05-15T23:11:52.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1399999999999999e-06 2024-05-15T23:11:52.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.419429+0000", 2024-05-15T23:11:52.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3149999999999999e-06 2024-05-15T23:11:52.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.433265+0000", 2024-05-15T23:11:52.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013835953999999999 2024-05-15T23:11:52.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.433400+0000", 2024-05-15T23:11:52.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013417999999999999 2024-05-15T23:11:52.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 11", 2024-05-15T23:11:52.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.433520+0000", 2024-05-15T23:11:52.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120078 2024-05-15T23:11:52.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.452068+0000", 2024-05-15T23:11:52.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018548179000000001 2024-05-15T23:11:52.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.465004+0000", 2024-05-15T23:11:52.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012936509000000001 2024-05-15T23:11:52.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.465057+0000", 2024-05-15T23:11:52.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2321999999999998e-05 2024-05-15T23:11:52.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:00.465102+0000", 2024-05-15T23:11:52.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.528e-05 2024-05-15T23:11:52.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.714 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.714 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:218350 2.b 2:d272971d:::benchmark_data_smithi090_78160_object56992:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:11:52.714 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:24.318097+0000", 2024-05-15T23:11:52.714 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 208.32917328400001, 2024-05-15T23:11:52.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055657801999999999, 2024-05-15T23:11:52.715 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.715 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.715 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.715 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218350 2024-05-15T23:11:52.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.715 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.318097+0000", 2024-05-15T23:11:52.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.318097+0000", 2024-05-15T23:11:52.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.318099+0000", 2024-05-15T23:11:52.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8789999999999999e-06 2024-05-15T23:11:52.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.318104+0000", 2024-05-15T23:11:52.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3680000000000001e-06 2024-05-15T23:11:52.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.318114+0000", 2024-05-15T23:11:52.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3819999999999993e-06 2024-05-15T23:11:52.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.318120+0000", 2024-05-15T23:11:52.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7100000000000001e-06 2024-05-15T23:11:52.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.319370+0000", 2024-05-15T23:11:52.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012499570000000001 2024-05-15T23:11:52.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.319479+0000", 2024-05-15T23:11:52.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108941 2024-05-15T23:11:52.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 11", 2024-05-15T23:11:52.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.319587+0000", 2024-05-15T23:11:52.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010780700000000001 2024-05-15T23:11:52.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.324031+0000", 2024-05-15T23:11:52.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044442750000000001 2024-05-15T23:11:52.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.337615+0000", 2024-05-15T23:11:52.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01358352 2024-05-15T23:11:52.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.337665+0000", 2024-05-15T23:11:52.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0319999999999999e-05 2024-05-15T23:11:52.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.373755+0000", 2024-05-15T23:11:52.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036089642999999998 2024-05-15T23:11:52.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.720 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.720 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:218364 2.5 2:ac4383fa:::benchmark_data_smithi090_78160_object57006:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:11:52.720 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:24.325312+0000", 2024-05-15T23:11:52.720 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 208.321958674, 2024-05-15T23:11:52.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.048743983999999997, 2024-05-15T23:11:52.720 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.720 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.720 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.720 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.721 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218364 2024-05-15T23:11:52.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.721 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325312+0000", 2024-05-15T23:11:52.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325312+0000", 2024-05-15T23:11:52.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325313+0000", 2024-05-15T23:11:52.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.536e-06 2024-05-15T23:11:52.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325319+0000", 2024-05-15T23:11:52.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4319999999999998e-06 2024-05-15T23:11:52.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325329+0000", 2024-05-15T23:11:52.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0564000000000001e-05 2024-05-15T23:11:52.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325337+0000", 2024-05-15T23:11:52.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5480000000000004e-06 2024-05-15T23:11:52.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325392+0000", 2024-05-15T23:11:52.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5476000000000002e-05 2024-05-15T23:11:52.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325493+0000", 2024-05-15T23:11:52.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100572 2024-05-15T23:11:52.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:11:52.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.325591+0000", 2024-05-15T23:11:52.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8259999999999998e-05 2024-05-15T23:11:52.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.333725+0000", 2024-05-15T23:11:52.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0081340900000000001 2024-05-15T23:11:52.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.373961+0000", 2024-05-15T23:11:52.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040235663999999997 2024-05-15T23:11:52.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.374013+0000", 2024-05-15T23:11:52.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1705e-05 2024-05-15T23:11:52.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.374056+0000", 2024-05-15T23:11:52.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3136999999999997e-05 2024-05-15T23:11:52.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.726 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.726 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:218365 2.5 2:a283dc57:::benchmark_data_smithi090_78160_object57007:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:11:52.726 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:24.327336+0000", 2024-05-15T23:11:52.726 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 208.31993425100001, 2024-05-15T23:11:52.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.051207694999999998, 2024-05-15T23:11:52.726 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.726 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.726 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.727 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218365 2024-05-15T23:11:52.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.727 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.327336+0000", 2024-05-15T23:11:52.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.327336+0000", 2024-05-15T23:11:52.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.327338+0000", 2024-05-15T23:11:52.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.86e-06 2024-05-15T23:11:52.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.327344+0000", 2024-05-15T23:11:52.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2169999999999999e-06 2024-05-15T23:11:52.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.327356+0000", 2024-05-15T23:11:52.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1292e-05 2024-05-15T23:11:52.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.327363+0000", 2024-05-15T23:11:52.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7200000000000006e-06 2024-05-15T23:11:52.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330074+0000", 2024-05-15T23:11:52.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002710559 2024-05-15T23:11:52.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330177+0000", 2024-05-15T23:11:52.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103081 2024-05-15T23:11:52.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:11:52.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330293+0000", 2024-05-15T23:11:52.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011625000000000001 2024-05-15T23:11:52.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.374099+0000", 2024-05-15T23:11:52.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.043805372000000002 2024-05-15T23:11:52.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.378434+0000", 2024-05-15T23:11:52.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043352629999999998 2024-05-15T23:11:52.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.378489+0000", 2024-05-15T23:11:52.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5680000000000002e-05 2024-05-15T23:11:52.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.378544+0000", 2024-05-15T23:11:52.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4401000000000003e-05 2024-05-15T23:11:52.732 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.732 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.733 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.733 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:218368 2.5 2:a696423a:::benchmark_data_smithi090_78160_object57010:head [delete] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-05-15T23:11:52.733 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:24.330243+0000", 2024-05-15T23:11:52.733 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 208.317027142, 2024-05-15T23:11:52.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.048395308999999997, 2024-05-15T23:11:52.733 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.733 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.733 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.734 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 218368 2024-05-15T23:11:52.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.734 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330243+0000", 2024-05-15T23:11:52.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330243+0000", 2024-05-15T23:11:52.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330244+0000", 2024-05-15T23:11:52.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6700000000000002e-07 2024-05-15T23:11:52.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330248+0000", 2024-05-15T23:11:52.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0160000000000002e-06 2024-05-15T23:11:52.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330255+0000", 2024-05-15T23:11:52.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.866e-06 2024-05-15T23:11:52.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.330261+0000", 2024-05-15T23:11:52.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9170000000000003e-06 2024-05-15T23:11:52.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.333760+0000", 2024-05-15T23:11:52.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034988649999999999 2024-05-15T23:11:52.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.333879+0000", 2024-05-15T23:11:52.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118651 2024-05-15T23:11:52.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:11:52.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.333991+0000", 2024-05-15T23:11:52.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011229499999999999 2024-05-15T23:11:52.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.374136+0000", 2024-05-15T23:11:52.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040145104000000001 2024-05-15T23:11:52.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.378560+0000", 2024-05-15T23:11:52.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044240140000000004 2024-05-15T23:11:52.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.378602+0000", 2024-05-15T23:11:52.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2450000000000002e-05 2024-05-15T23:11:52.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.378639+0000", 2024-05-15T23:11:52.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6164000000000001e-05 2024-05-15T23:11:52.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.739 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.739 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:245494 2.b 2:d7d1c86d:::benchmark_data_smithi090_78160_object84136:head [delete] snapc 0=[] ondisk+write+known_if_redirected e156)", 2024-05-15T23:11:52.739 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:49.782386+0000", 2024-05-15T23:11:52.739 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 182.86488441899999, 2024-05-15T23:11:52.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.047309813999999999, 2024-05-15T23:11:52.739 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.739 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.739 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.739 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.740 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.740 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.740 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 245494 2024-05-15T23:11:52.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.740 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:49.782386+0000", 2024-05-15T23:11:52.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:49.782386+0000", 2024-05-15T23:11:52.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:49.782387+0000", 2024-05-15T23:11:52.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4600000000000005e-07 2024-05-15T23:11:52.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:49.782389+0000", 2024-05-15T23:11:52.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1639999999999999e-06 2024-05-15T23:11:52.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:49.782394+0000", 2024-05-15T23:11:52.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.366e-06 2024-05-15T23:11:52.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:49.782399+0000", 2024-05-15T23:11:52.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9099999999999996e-06 2024-05-15T23:11:52.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:49.800052+0000", 2024-05-15T23:11:52.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017653111999999999 2024-05-15T23:11:52.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:49.800148+0000", 2024-05-15T23:11:52.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5514999999999996e-05 2024-05-15T23:11:52.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 11", 2024-05-15T23:11:52.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:49.800229+0000", 2024-05-15T23:11:52.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1839e-05 2024-05-15T23:11:52.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:49.805973+0000", 2024-05-15T23:11:52.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0057433980000000003 2024-05-15T23:11:52.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:49.829609+0000", 2024-05-15T23:11:52.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023635928 2024-05-15T23:11:52.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:49.829669+0000", 2024-05-15T23:11:52.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9811e-05 2024-05-15T23:11:52.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:49.829696+0000", 2024-05-15T23:11:52.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7225000000000001e-05 2024-05-15T23:11:52.744 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.745 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.745 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.745 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:260067 2.f 2:f21c3174:::benchmark_data_smithi090_78160_object98709:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e165)", 2024-05-15T23:11:52.745 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:09:05.984986+0000", 2024-05-15T23:11:52.745 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 166.66228435900001, 2024-05-15T23:11:52.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.146705111, 2024-05-15T23:11:52.745 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.745 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.745 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.745 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.745 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.746 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 260067 2024-05-15T23:11:52.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.746 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.984986+0000", 2024-05-15T23:11:52.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.984986+0000", 2024-05-15T23:11:52.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.984988+0000", 2024-05-15T23:11:52.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1940000000000001e-06 2024-05-15T23:11:52.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.984994+0000", 2024-05-15T23:11:52.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8850000000000004e-06 2024-05-15T23:11:52.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985005+0000", 2024-05-15T23:11:52.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1211e-05 2024-05-15T23:11:52.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985017+0000", 2024-05-15T23:11:52.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1963999999999999e-05 2024-05-15T23:11:52.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985079+0000", 2024-05-15T23:11:52.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1667000000000004e-05 2024-05-15T23:11:52.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:11:52.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985087+0000", 2024-05-15T23:11:52.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-05-15T23:11:52.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.119335+0000", 2024-05-15T23:11:52.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.134247755 2024-05-15T23:11:52.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.119431+0000", 2024-05-15T23:11:52.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6749000000000005e-05 2024-05-15T23:11:52.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,9", 2024-05-15T23:11:52.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.119514+0000", 2024-05-15T23:11:52.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2225999999999995e-05 2024-05-15T23:11:52.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.123182+0000", 2024-05-15T23:11:52.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036685569999999998 2024-05-15T23:11:52.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.131220+0000", 2024-05-15T23:11:52.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0080374480000000009 2024-05-15T23:11:52.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.131513+0000", 2024-05-15T23:11:52.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029322700000000002 2024-05-15T23:11:52.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.131580+0000", 2024-05-15T23:11:52.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7397999999999994e-05 2024-05-15T23:11:52.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.131691+0000", 2024-05-15T23:11:52.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110968 2024-05-15T23:11:52.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.752 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.752 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:260074 2.f 2:f554b67b:::benchmark_data_smithi090_78160_object98716:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165)", 2024-05-15T23:11:52.752 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:09:05.985036+0000", 2024-05-15T23:11:52.752 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 166.66223425000001, 2024-05-15T23:11:52.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.158383624, 2024-05-15T23:11:52.752 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.752 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.752 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.753 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 260074 2024-05-15T23:11:52.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.753 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985036+0000", 2024-05-15T23:11:52.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985036+0000", 2024-05-15T23:11:52.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985037+0000", 2024-05-15T23:11:52.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2900000000000002e-07 2024-05-15T23:11:52.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985041+0000", 2024-05-15T23:11:52.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6270000000000002e-06 2024-05-15T23:11:52.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985055+0000", 2024-05-15T23:11:52.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4717e-05 2024-05-15T23:11:52.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985062+0000", 2024-05-15T23:11:52.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5989999999999998e-06 2024-05-15T23:11:52.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985156+0000", 2024-05-15T23:11:52.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3927999999999999e-05 2024-05-15T23:11:52.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:11:52.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985166+0000", 2024-05-15T23:11:52.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0621999999999999e-05 2024-05-15T23:11:52.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.123230+0000", 2024-05-15T23:11:52.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.138063251 2024-05-15T23:11:52.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.123314+0000", 2024-05-15T23:11:52.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4631999999999999e-05 2024-05-15T23:11:52.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,9", 2024-05-15T23:11:52.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.123396+0000", 2024-05-15T23:11:52.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1866999999999994e-05 2024-05-15T23:11:52.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.131453+0000", 2024-05-15T23:11:52.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0080570399999999997 2024-05-15T23:11:52.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.131462+0000", 2024-05-15T23:11:52.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2050000000000008e-06 2024-05-15T23:11:52.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.143274+0000", 2024-05-15T23:11:52.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011811689 2024-05-15T23:11:52.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.143339+0000", 2024-05-15T23:11:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5204999999999997e-05 2024-05-15T23:11:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.143420+0000", 2024-05-15T23:11:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0413000000000004e-05 2024-05-15T23:11:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:260094 2.f 2:fe92e31a:::benchmark_data_smithi090_78160_object98736:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165)", 2024-05-15T23:11:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:09:05.985077+0000", 2024-05-15T23:11:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 166.662193317, 2024-05-15T23:11:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15857575400000001, 2024-05-15T23:11:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 260094 2024-05-15T23:11:52.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985077+0000", 2024-05-15T23:11:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985077+0000", 2024-05-15T23:11:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985078+0000", 2024-05-15T23:11:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4700000000000001e-07 2024-05-15T23:11:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985081+0000", 2024-05-15T23:11:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9500000000000001e-06 2024-05-15T23:11:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985087+0000", 2024-05-15T23:11:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3060000000000003e-06 2024-05-15T23:11:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985093+0000", 2024-05-15T23:11:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0939999999999996e-06 2024-05-15T23:11:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985199+0000", 2024-05-15T23:11:52.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000105697 2024-05-15T23:11:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:11:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:05.985208+0000", 2024-05-15T23:11:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8340000000000005e-06 2024-05-15T23:11:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.126940+0000", 2024-05-15T23:11:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14173261200000001 2024-05-15T23:11:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.127074+0000", 2024-05-15T23:11:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013398300000000001 2024-05-15T23:11:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,9", 2024-05-15T23:11:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.127221+0000", 2024-05-15T23:11:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014644699999999999 2024-05-15T23:11:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.131752+0000", 2024-05-15T23:11:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045312099999999999 2024-05-15T23:11:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.131772+0000", 2024-05-15T23:11:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9740000000000001e-05 2024-05-15T23:11:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.143547+0000", 2024-05-15T23:11:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011774866 2024-05-15T23:11:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.143594+0000", 2024-05-15T23:11:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6993000000000002e-05 2024-05-15T23:11:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.143653+0000", 2024-05-15T23:11:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9274999999999997e-05 2024-05-15T23:11:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:260098 2.f 2:f7214dda:::benchmark_data_smithi090_78160_object98740:head [delete] snapc 0=[] ondisk+write+known_if_redirected e165)", 2024-05-15T23:11:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:09:06.131827+0000", 2024-05-15T23:11:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 166.5154431, 2024-05-15T23:11:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046276237999999997, 2024-05-15T23:11:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:52.767 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:11:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:11:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:11:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:11:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 260098 2024-05-15T23:11:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.131827+0000", 2024-05-15T23:11:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.131827+0000", 2024-05-15T23:11:52.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.131829+0000", 2024-05-15T23:11:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5859999999999999e-06 2024-05-15T23:11:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.131834+0000", 2024-05-15T23:11:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4940000000000003e-06 2024-05-15T23:11:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.131841+0000", 2024-05-15T23:11:52.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6290000000000004e-06 2024-05-15T23:11:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.131846+0000", 2024-05-15T23:11:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.062e-06 2024-05-15T23:11:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.143763+0000", 2024-05-15T23:11:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011917297 2024-05-15T23:11:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:52.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.143904+0000", 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014107099999999999 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,9", 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.144028+0000", 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000123356 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.156496+0000", 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012468386999999999 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.156515+0000", 2024-05-15T23:11:52.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8714000000000001e-05 2024-05-15T23:11:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:11:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.177934+0000", 2024-05-15T23:11:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021418691 2024-05-15T23:11:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:11:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.177988+0000", 2024-05-15T23:11:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4537999999999999e-05 2024-05-15T23:11:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:11:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.178104+0000", 2024-05-15T23:11:52.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115413 2024-05-15T23:11:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:52.773 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:52.773 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:11:52.773 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:11:52.845 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:52.887 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:11:52.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:52.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:11:52.929 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 58 16 132368 132352 142.594 72.5 0.0111657 0.00701139 2024-05-15T23:11:52.991 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:11:53.034 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:53.057 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:11:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:53.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:11:53.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:52 smithi084 ceph-mon[39784]: 3.a scrub ok 2024-05-15T23:11:53.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:52 smithi084 ceph-mon[39784]: 3.8 scrub ok 2024-05-15T23:11:53.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:52 smithi084 ceph-mon[39784]: 3.7 scrub ok 2024-05-15T23:11:53.164 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:11:53.194 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:53.220 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:11:53.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:53.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:11:53.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:52 smithi090 ceph-mon[44434]: 3.a scrub ok 2024-05-15T23:11:53.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:52 smithi090 ceph-mon[44434]: 3.8 scrub ok 2024-05-15T23:11:53.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:52 smithi090 ceph-mon[44434]: 3.7 scrub ok 2024-05-15T23:11:53.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:52 smithi088 ceph-mon[44699]: 3.a scrub ok 2024-05-15T23:11:53.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:52 smithi088 ceph-mon[44699]: 3.8 scrub ok 2024-05-15T23:11:53.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:52 smithi088 ceph-mon[44699]: 3.7 scrub ok 2024-05-15T23:11:53.324 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:11:53.367 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:53.424 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:11:53.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:53.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:11:53.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T23:11:53.530 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:11:53.563 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:53.593 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:11:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:53.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:11:53.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:11:53.763 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:53.794 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:11:53.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:53.798 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:11:53.838 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:11:53.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:11:53.931 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 59 16 133728 133712 141.612 85 0.00626992 0.0070602 2024-05-15T23:11:53.941 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:11:53.957 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:53.957 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:53.987 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:11:53.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:53.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:11:54.013 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:54 smithi090 ceph-mon[44434]: 3.4 scrub ok 2024-05-15T23:11:54.013 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:54 smithi090 ceph-mon[44434]: pgmap v652: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-05-15T23:11:54.013 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:54 smithi090 ceph-mon[44434]: 3.b scrub ok 2024-05-15T23:11:54.095 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:11:54.128 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:11:54.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:54.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:11:54.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:11:54.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:54 smithi088 ceph-mon[44699]: 3.4 scrub ok 2024-05-15T23:11:54.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:54 smithi088 ceph-mon[44699]: pgmap v652: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-05-15T23:11:54.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:54 smithi088 ceph-mon[44699]: 3.b scrub ok 2024-05-15T23:11:54.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:54 smithi084 ceph-mon[39784]: 3.4 scrub ok 2024-05-15T23:11:54.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:54 smithi084 ceph-mon[39784]: pgmap v652: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-05-15T23:11:54.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:54 smithi084 ceph-mon[39784]: 3.b scrub ok 2024-05-15T23:11:54.329 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:54.931 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:11:54.931040 min lat: 0.000948627 max lat: 0.219607 avg lat: 0.00710991 2024-05-15T23:11:54.932 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:11:54.932 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 60 16 135043 135027 140.622 82.1875 0.00956473 0.00710991 2024-05-15T23:11:55.103 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:11:55.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:55.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:11:55.207 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:11:55.248 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:55.274 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:11:55.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:55.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:11:55.382 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:11:55.430 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:55.465 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:11:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:55.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:11:55.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:11:55.610 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:55.769 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:11:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:55.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:11:55.817 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:55.820 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:11:55.821 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:11:55.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:55.821 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25135.0:136245 3.0 e235/202 3:0f4dcbdf:::benchmark_data_smithi090_90974_object136244:head v 235'8599, mlcod=235'8597)", 2024-05-15T23:11:55.821 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:11:55.817113+0000", 2024-05-15T23:11:55.821 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0017675220000000001, 2024-05-15T23:11:55.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017984769999999999, 2024-05-15T23:11:55.821 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:11:55.821 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:11:55.821 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:11:55.821 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:11:55.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:55.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:11:55.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:11:55.817113+0000", 2024-05-15T23:11:55.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:55.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:55.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:55.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:11:55.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:11:55.817113+0000", 2024-05-15T23:11:55.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:11:55.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:55.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:55.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:11:55.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:11:55.817115+0000", 2024-05-15T23:11:55.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7010000000000001e-06 2024-05-15T23:11:55.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:55.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:55.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:11:55.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:11:55.817139+0000", 2024-05-15T23:11:55.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3682000000000001e-05 2024-05-15T23:11:55.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:55.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:55.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:11:55.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:11:55.817149+0000", 2024-05-15T23:11:55.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0591000000000001e-05 2024-05-15T23:11:55.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:55.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:55.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:11:55.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:11:55.817157+0000", 2024-05-15T23:11:55.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3780000000000003e-06 2024-05-15T23:11:55.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:55.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:55.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:11:55.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:11:55.817219+0000", 2024-05-15T23:11:55.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2339000000000001e-05 2024-05-15T23:11:55.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:11:55.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:11:55.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:11:55.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:11:55.817242+0000", 2024-05-15T23:11:55.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2541e-05 2024-05-15T23:11:55.824 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:55.824 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:11:55.824 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:55.824 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:11:55.824 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-05-15T23:11:55.824 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2024-05-15T23:11:55.824 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:11:55.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:11:55.931 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:55.932 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 61 16 136388 136372 139.694 84.0625 0.0108792 0.00715704 2024-05-15T23:11:56.041 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:11:56.059 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:11:56.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:56.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:11:56.165 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:11:56.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:56 smithi090 ceph-mon[44434]: pgmap v653: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-05-15T23:11:56.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/577350662' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:56.200 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:56 smithi090 ceph-mon[44434]: from='client.35285 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:56.202 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:56 smithi088 ceph-mon[44699]: pgmap v653: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-05-15T23:11:56.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/577350662' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:56.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:56 smithi088 ceph-mon[44699]: from='client.35285 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:56.358 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:56 smithi084 ceph-mon[39784]: pgmap v653: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-05-15T23:11:56.358 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/577350662' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:56.358 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:56 smithi084 ceph-mon[39784]: from='client.35285 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:11:56.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T23:11:56.655 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:11:56.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:56.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:11:56.760 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:11:56.821 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:56.875 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:11:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:56.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:11:56.931 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 62 16 137720 137704 138.784 83.25 0.00396064 0.00720316 2024-05-15T23:11:56.984 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:11:57.034 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:57 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:11:57.034 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:57 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:11:57.034 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:57 smithi090 ceph-mon[44434]: from='client.35285 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:11:57.034 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:57 smithi090 ceph-mon[44434]: osdmap e236: 12 total, 12 up, 12 in 2024-05-15T23:11:57.034 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:57 smithi090 ceph-mon[44434]: osdmap e237: 12 total, 12 up, 12 in 2024-05-15T23:11:57.054 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:57.068 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:11:57.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:57 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:11:57.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:57 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:11:57.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:57 smithi084 ceph-mon[39784]: from='client.35285 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:11:57.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:57 smithi084 ceph-mon[39784]: osdmap e236: 12 total, 12 up, 12 in 2024-05-15T23:11:57.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:57 smithi084 ceph-mon[39784]: osdmap e237: 12 total, 12 up, 12 in 2024-05-15T23:11:57.234 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:11:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:57.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:11:57.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:57 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:11:57.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:57 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:11:57.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:57 smithi088 ceph-mon[44699]: from='client.35285 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:11:57.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:57 smithi088 ceph-mon[44699]: osdmap e236: 12 total, 12 up, 12 in 2024-05-15T23:11:57.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:57 smithi088 ceph-mon[44699]: osdmap e237: 12 total, 12 up, 12 in 2024-05-15T23:11:57.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:11:57.384 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:57.415 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:11:57.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:57.417 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:11:57.519 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:11:57.549 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:57.571 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:11:57.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:57.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:11:57.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:11:57.722 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:57.924 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:11:57.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:57.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:11:57.931 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 63 16 139005 138989 137.856 80.3125 0.00903044 0.00725218 2024-05-15T23:11:58.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:11:58.052 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.8 to 1 (10000) 2024-05-15T23:11:58.087 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:58 smithi090 ceph-mon[44434]: pgmap v655: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-05-15T23:11:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:58 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3801187196' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:11:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:58 smithi090 ceph-mon[44434]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:11:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:58 smithi090 ceph-mon[44434]: 3.5 scrub starts 2024-05-15T23:11:58.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:58 smithi088 ceph-mon[44699]: pgmap v655: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-05-15T23:11:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:58 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3801187196' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:11:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:58 smithi088 ceph-mon[44699]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:11:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:58 smithi088 ceph-mon[44699]: 3.5 scrub starts 2024-05-15T23:11:58.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:58 smithi084 ceph-mon[39784]: pgmap v655: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-05-15T23:11:58.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:58 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3801187196' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:11:58.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:58 smithi084 ceph-mon[39784]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:11:58.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:58 smithi084 ceph-mon[39784]: 3.5 scrub starts 2024-05-15T23:11:58.341 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:11:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:58.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:11:58.447 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:11:58.490 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:58.521 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:11:58.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:58.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:11:58.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:11:58.663 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:58.932 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 64 16 140285 140269 136.952 80 0.0045408 0.00729999 2024-05-15T23:11:59.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:59 smithi084 ceph-mon[39784]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:11:59.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:59 smithi084 ceph-mon[39784]: osdmap e238: 12 total, 12 up, 12 in 2024-05-15T23:11:59.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:11:59 smithi084 ceph-mon[39784]: 3.c scrub starts 2024-05-15T23:11:59.226 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 11 1 2024-05-15T23:11:59.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:59 smithi090 ceph-mon[44434]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:11:59.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:59 smithi090 ceph-mon[44434]: osdmap e238: 12 total, 12 up, 12 in 2024-05-15T23:11:59.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:11:59 smithi090 ceph-mon[44434]: 3.c scrub starts 2024-05-15T23:11:59.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:59 smithi088 ceph-mon[44699]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:11:59.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:59 smithi088 ceph-mon[44699]: osdmap e238: 12 total, 12 up, 12 in 2024-05-15T23:11:59.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:11:59 smithi088 ceph-mon[44699]: 3.c scrub starts 2024-05-15T23:11:59.325 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:11:59.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:59.328 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:11:59.373 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:11:59.374 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:11:59.374 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:11:59.374 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:11:59.374 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:11:59.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:11:59.469 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:59.634 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:11:59.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:59.639 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:11:59.740 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:11:59.796 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:59.811 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:11:59.821 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:11:59.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:11:59.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:11:59.886 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:11:59.919 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:11:59.926 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:11:59.932 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 65 16 141409 141393 135.926 70.25 0.0119865 0.00735444 2024-05-15T23:11:59.957 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:11:59.957 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:11:59.979 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:11:59.994 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:11:59.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:00.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:12:00.006 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:12:00.047 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:12:00.085 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:12:00.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T23:12:00.100 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:12:00.115 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:12:00.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:00 smithi084 ceph-mon[39784]: pgmap v658: 17 pgs: 17 active+clean; 8.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T23:12:00.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:00 smithi084 ceph-mon[39784]: 3.5 scrub ok 2024-05-15T23:12:00.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:00 smithi084 ceph-mon[39784]: 3.e deep-scrub starts 2024-05-15T23:12:00.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:00 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1874937152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:00.134 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:00.146 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:12:00.167 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:12:00.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:00.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:12:00.180 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:12:00.216 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:12:00.251 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:12:00.273 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:12:00.283 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:12:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:12:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:12:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:12:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:12:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:12:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:12:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:12:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:12:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:12:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:12:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:12:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:12:00.305 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:00.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:00 smithi088 ceph-mon[44699]: pgmap v658: 17 pgs: 17 active+clean; 8.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T23:12:00.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:00 smithi088 ceph-mon[44699]: 3.5 scrub ok 2024-05-15T23:12:00.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:00 smithi088 ceph-mon[44699]: 3.e deep-scrub starts 2024-05-15T23:12:00.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:00 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1874937152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:00.330 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:12:00.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:00.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:12:00.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:12:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:12:00.480 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:00 smithi090 ceph-mon[44434]: pgmap v658: 17 pgs: 17 active+clean; 8.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T23:12:00.480 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:00 smithi090 ceph-mon[44434]: 3.5 scrub ok 2024-05-15T23:12:00.480 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:00 smithi090 ceph-mon[44434]: 3.e deep-scrub starts 2024-05-15T23:12:00.480 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:00 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1874937152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:00.481 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:00.504 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:12:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:00.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:12:00.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:12:00.645 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:00.752 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:12:00.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:00.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:12:00.857 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:12:00.891 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:00.918 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:12:00.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:00.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:12:00.932 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 66 16 142470 142454 134.871 66.3125 0.0110065 0.00741007 2024-05-15T23:12:01.024 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:12:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:01.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:01 smithi084 ceph-mon[39784]: 3.c scrub ok 2024-05-15T23:12:01.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.080 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:12:01.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:01.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:12:01.186 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:12:01.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:01 smithi088 ceph-mon[44699]: 3.c scrub ok 2024-05-15T23:12:01.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.218 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:01.254 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:12:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:01.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:12:01.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:01 smithi090 ceph-mon[44434]: 3.c scrub ok 2024-05-15T23:12:01.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:01.363 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:12:01.396 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:01.503 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:12:01.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:01.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:12:01.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:12:01.657 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:01.859 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:12:01.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:01.864 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:12:01.932 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 67 16 143573 143557 133.887 68.9375 0.00770751 0.00746749 2024-05-15T23:12:01.967 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:12:02.025 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:02.065 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:02.072 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:12:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:02.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:12:02.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:12:02.255 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:02.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:02 smithi084 ceph-mon[39784]: pgmap v659: 17 pgs: 17 active+clean; 8.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-05-15T23:12:02.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:02 smithi084 ceph-mon[39784]: 3.1 scrub starts 2024-05-15T23:12:02.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:02 smithi084 ceph-mon[39784]: osdmap e239: 12 total, 12 up, 12 in 2024-05-15T23:12:02.497 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:12:02.497 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:12:02.521 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:02 smithi090 ceph-mon[44434]: pgmap v659: 17 pgs: 17 active+clean; 8.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-05-15T23:12:02.521 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:02 smithi090 ceph-mon[44434]: 3.1 scrub starts 2024-05-15T23:12:02.521 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:02 smithi090 ceph-mon[44434]: osdmap e239: 12 total, 12 up, 12 in 2024-05-15T23:12:02.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:02 smithi088 ceph-mon[44699]: pgmap v659: 17 pgs: 17 active+clean; 8.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-05-15T23:12:02.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:02 smithi088 ceph-mon[44699]: 3.1 scrub starts 2024-05-15T23:12:02.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:02 smithi088 ceph-mon[44699]: osdmap e239: 12 total, 12 up, 12 in 2024-05-15T23:12:02.617 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:12:02.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:02.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:12:02.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:12:02.766 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:02.932 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 68 16 144633 144617 132.892 66.25 0.00447681 0.007522 2024-05-15T23:12:03.082 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.11 to 1 (10000) 2024-05-15T23:12:03.189 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:03 smithi084 ceph-mon[39784]: 3.1 scrub ok 2024-05-15T23:12:03.189 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3592841811' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:12:03.189 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:03 smithi084 ceph-mon[39784]: from='client.35300 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:12:03.189 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:03 smithi090 ceph-mon[44434]: 3.1 scrub ok 2024-05-15T23:12:03.189 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3592841811' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:12:03.190 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:03 smithi090 ceph-mon[44434]: from='client.35300 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:12:03.221 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:12:03.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T23:12:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:03.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:12:03.326 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:12:03.362 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:03 smithi088 ceph-mon[44699]: 3.1 scrub ok 2024-05-15T23:12:03.362 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3592841811' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:12:03.362 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:03 smithi088 ceph-mon[44699]: from='client.35300 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:12:03.363 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:03.394 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:12:03.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:03.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:12:03.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:12:03.561 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:03.599 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:12:03.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:03.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:12:03.652 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:12:03.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:12:03.746 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:03.817 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:12:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:03.824 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:12:03.925 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:12:03.932 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 69 16 145743 145727 131.972 69.375 0.00471419 0.00757386 2024-05-15T23:12:03.951 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:03.970 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:12:03.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:03.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:12:04.073 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:12:04.079 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:04 smithi090 ceph-mon[44434]: pgmap v661: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-05-15T23:12:04.079 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:04 smithi090 ceph-mon[44434]: from='client.35300 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:12:04.079 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:04 smithi090 ceph-mon[44434]: osdmap e240: 12 total, 12 up, 12 in 2024-05-15T23:12:04.108 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:04.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:04 smithi088 ceph-mon[44699]: pgmap v661: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-05-15T23:12:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:04 smithi088 ceph-mon[44699]: from='client.35300 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:12:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:04 smithi088 ceph-mon[44699]: osdmap e240: 12 total, 12 up, 12 in 2024-05-15T23:12:04.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:04 smithi084 ceph-mon[39784]: pgmap v661: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-05-15T23:12:04.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:04 smithi084 ceph-mon[39784]: from='client.35300 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:12:04.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:04 smithi084 ceph-mon[39784]: osdmap e240: 12 total, 12 up, 12 in 2024-05-15T23:12:04.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:12:04.932 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 70 15 146766 146751 131.001 64 0.0156582 0.0076294 2024-05-15T23:12:05.932 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 71 16 147802 147786 130.067 64.6875 0.0056115 0.00768579 2024-05-15T23:12:05.989 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:05.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 9 1 2024-05-15T23:12:06.274 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:06 smithi084 ceph-mon[39784]: pgmap v663: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-05-15T23:12:06.274 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:06 smithi084 ceph-mon[39784]: 3.6 deep-scrub starts 2024-05-15T23:12:06.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:06 smithi090 ceph-mon[44434]: pgmap v663: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-05-15T23:12:06.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:06 smithi090 ceph-mon[44434]: 3.6 deep-scrub starts 2024-05-15T23:12:06.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:06 smithi088 ceph-mon[44699]: pgmap v663: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-05-15T23:12:06.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:06 smithi088 ceph-mon[44699]: 3.6 deep-scrub starts 2024-05-15T23:12:06.670 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:12:06.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:06.673 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:12:06.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:12:06.810 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:06.841 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:12:06.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:06.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:12:06.932 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 72 16 148638 148622 128.986 52.25 0.00717172 0.00775091 2024-05-15T23:12:06.946 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:12:06.994 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:07.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:07 smithi090 ceph-mon[44434]: 3.3 scrub starts 2024-05-15T23:12:07.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:07 smithi090 ceph-mon[44434]: 3.0 scrub starts 2024-05-15T23:12:07.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:07 smithi090 ceph-mon[44434]: osdmap e241: 12 total, 12 up, 12 in 2024-05-15T23:12:07.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:07 smithi084 ceph-mon[39784]: 3.3 scrub starts 2024-05-15T23:12:07.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:07 smithi084 ceph-mon[39784]: 3.0 scrub starts 2024-05-15T23:12:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:07 smithi084 ceph-mon[39784]: osdmap e241: 12 total, 12 up, 12 in 2024-05-15T23:12:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:07 smithi088 ceph-mon[44699]: 3.3 scrub starts 2024-05-15T23:12:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:07 smithi088 ceph-mon[44699]: 3.0 scrub starts 2024-05-15T23:12:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:07 smithi088 ceph-mon[44699]: osdmap e241: 12 total, 12 up, 12 in 2024-05-15T23:12:07.876 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:12:07.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:07.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:12:07.914 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:07.934 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 73 16 149435 149419 127.898 49.8125 0.0047948 0.00781584 2024-05-15T23:12:07.987 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:12:08.047 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:08.113 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:12:08.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:08 smithi090 ceph-mon[44434]: pgmap v664: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-05-15T23:12:08.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:08 smithi090 ceph-mon[44434]: 3.2 scrub starts 2024-05-15T23:12:08.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3240305651' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:12:08.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:08 smithi090 ceph-mon[44434]: 3.f scrub starts 2024-05-15T23:12:08.119 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:08 smithi090 ceph-mon[44434]: 3.3 scrub ok 2024-05-15T23:12:08.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:08.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:12:08.127 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:08 smithi084 ceph-mon[39784]: pgmap v664: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-05-15T23:12:08.127 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:08 smithi084 ceph-mon[39784]: 3.2 scrub starts 2024-05-15T23:12:08.127 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3240305651' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:12:08.127 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:08 smithi084 ceph-mon[39784]: 3.f scrub starts 2024-05-15T23:12:08.127 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:08 smithi084 ceph-mon[39784]: 3.3 scrub ok 2024-05-15T23:12:08.142 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:12:08.220 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:12:08.252 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:08.277 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:12:08.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:08.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:12:08.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:12:08.419 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:08.451 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:12:08.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:08.455 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:12:08.505 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:12:08.506 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:12:08.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:12:08.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:08 smithi088 ceph-mon[44699]: pgmap v664: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-05-15T23:12:08.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:08 smithi088 ceph-mon[44699]: 3.2 scrub starts 2024-05-15T23:12:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3240305651' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:12:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:08 smithi088 ceph-mon[44699]: 3.f scrub starts 2024-05-15T23:12:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:08 smithi088 ceph-mon[44699]: 3.3 scrub ok 2024-05-15T23:12:08.603 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:08.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 74 16 150328 150312 126.924 55.8125 0.00514786 0.00787543 2024-05-15T23:12:08.963 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:12:08.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:08.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:12:09.070 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:12:09.101 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:09.126 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:12:09.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:09.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:12:09.214 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:09 smithi090 ceph-mon[44434]: 3.0 scrub ok 2024-05-15T23:12:09.214 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:09 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:12:09.214 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:09 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3240305651' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:12:09.214 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:09 smithi090 ceph-mon[44434]: osdmap e242: 12 total, 12 up, 12 in 2024-05-15T23:12:09.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:12:09.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:09 smithi084 ceph-mon[39784]: 3.0 scrub ok 2024-05-15T23:12:09.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:09 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:12:09.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:09 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3240305651' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:12:09.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:09 smithi084 ceph-mon[39784]: osdmap e242: 12 total, 12 up, 12 in 2024-05-15T23:12:09.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T23:12:09.273 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:09.301 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:12:09.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:09.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:12:09.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:12:09.444 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:09.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:09 smithi088 ceph-mon[44699]: 3.0 scrub ok 2024-05-15T23:12:09.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:09 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:12:09.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:09 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3240305651' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:12:09.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:09 smithi088 ceph-mon[44699]: osdmap e242: 12 total, 12 up, 12 in 2024-05-15T23:12:09.665 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:12:09.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 75 16 151251 151235 126.001 57.6875 0.0101126 0.00793225 2024-05-15T23:12:10.005 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:12:10.005 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:12:10.005 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:12:10.005 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:12:10.005 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:12:10.005 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:12:10.005 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:12:10.005 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:12:10.005 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:12:10.006 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:12:10.006 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:12:10.009 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:12:10.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:10.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:12:10.020 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:12:10.022 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:12:10.022 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:12:10.022 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:12:10.022 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:12:10.022 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:12:10.022 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:12:10.022 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:12:10.022 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:12:10.022 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:12:10.022 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:12:10.023 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:12:10.023 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:12:10.117 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:12:10.154 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:10 smithi084 ceph-mon[39784]: pgmap v667: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 9.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 876 op/s 2024-05-15T23:12:10.154 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:10 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2955750553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:10.154 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:10.154 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:10.155 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:10.155 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:10.155 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:10.158 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:10.206 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:10 smithi090 ceph-mon[44434]: pgmap v667: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 9.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 876 op/s 2024-05-15T23:12:10.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:10 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2955750553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:10.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:10.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:10.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:10.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:10.207 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:10.301 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:12:10.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:10.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:12:10.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:12:10.440 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:10.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:10 smithi088 ceph-mon[44699]: pgmap v667: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 9.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 876 op/s 2024-05-15T23:12:10.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:10 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2955750553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:10.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:10.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:10.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:10.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:10.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:10.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 76 16 152241 152225 125.157 61.875 0.0122157 0.00798831 2024-05-15T23:12:11.283 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.283 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.284 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.608 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:12:11.608 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:12:11.608 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:12:11.608 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:12:11.608 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:12:11.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:11.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 77 16 153157 153141 124.275 57.25 0.0164007 0.00804521 2024-05-15T23:12:11.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:12:12.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:12 smithi090 ceph-mon[44434]: pgmap v668: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 14 active+clean; 9.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 938 op/s 2024-05-15T23:12:12.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:12 smithi090 ceph-mon[44434]: osdmap e243: 12 total, 12 up, 12 in 2024-05-15T23:12:12.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T23:12:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:12 smithi084 ceph-mon[39784]: pgmap v668: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 14 active+clean; 9.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 938 op/s 2024-05-15T23:12:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:12 smithi084 ceph-mon[39784]: osdmap e243: 12 total, 12 up, 12 in 2024-05-15T23:12:12.479 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:12:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:12.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:12:12.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:12 smithi088 ceph-mon[44699]: pgmap v668: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 14 active+clean; 9.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 938 op/s 2024-05-15T23:12:12.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:12 smithi088 ceph-mon[44699]: osdmap e243: 12 total, 12 up, 12 in 2024-05-15T23:12:12.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:12:12.622 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:12.646 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:12:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:12.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:12:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:12:12.752 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:12:12.811 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:12.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 78 16 154045 154029 123.393 55.5 0.00545175 0.00810138 2024-05-15T23:12:13.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 79 15 154994 154979 122.583 59.375 0.0196905 0.0081561 2024-05-15T23:12:14.408 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:14 smithi090 ceph-mon[44434]: pgmap v670: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 967 op/s 2024-05-15T23:12:14.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:14 smithi088 ceph-mon[44699]: pgmap v670: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 967 op/s 2024-05-15T23:12:14.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:14 smithi084 ceph-mon[39784]: pgmap v670: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 967 op/s 2024-05-15T23:12:14.750 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:12:14.754 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:12:14.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:12:14.754 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:12:14.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.754 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:144857 2.0 e138/102 2:05078f07:::benchmark_data_smithi090_78160_object144856:head v 138'9168, mlcod=138'9166)", 2024-05-15T23:12:14.754 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:07:20.843168+0000", 2024-05-15T23:12:14.755 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 293.90390437299999, 2024-05-15T23:12:14.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030063666999999999, 2024-05-15T23:12:14.755 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.755 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.755 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.755 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.843168+0000", 2024-05-15T23:12:14.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.843168+0000", 2024-05-15T23:12:14.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.843171+0000", 2024-05-15T23:12:14.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.17e-06 2024-05-15T23:12:14.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.843251+0000", 2024-05-15T23:12:14.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0867999999999999e-05 2024-05-15T23:12:14.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.843270+0000", 2024-05-15T23:12:14.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8715e-05 2024-05-15T23:12:14.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.843279+0000", 2024-05-15T23:12:14.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3549999999999994e-06 2024-05-15T23:12:14.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.843340+0000", 2024-05-15T23:12:14.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0717999999999997e-05 2024-05-15T23:12:14.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.843377+0000", 2024-05-15T23:12:14.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7197000000000003e-05 2024-05-15T23:12:14.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.873174+0000", 2024-05-15T23:12:14.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029796798999999999 2024-05-15T23:12:14.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.873232+0000", 2024-05-15T23:12:14.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7844999999999999e-05 2024-05-15T23:12:14.759 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.759 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.759 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.759 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:144867 2.0 e138/102 2:075662e2:::benchmark_data_smithi090_78160_object144866:head v 138'9169, mlcod=138'9166)", 2024-05-15T23:12:14.759 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:07:20.843300+0000", 2024-05-15T23:12:14.760 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 293.90377228900002, 2024-05-15T23:12:14.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029967852, 2024-05-15T23:12:14.760 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.760 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.760 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.760 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.843300+0000", 2024-05-15T23:12:14.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.843300+0000", 2024-05-15T23:12:14.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.843301+0000", 2024-05-15T23:12:14.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.92e-07 2024-05-15T23:12:14.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.843335+0000", 2024-05-15T23:12:14.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4282000000000001e-05 2024-05-15T23:12:14.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.843357+0000", 2024-05-15T23:12:14.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1101e-05 2024-05-15T23:12:14.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.843370+0000", 2024-05-15T23:12:14.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3193000000000001e-05 2024-05-15T23:12:14.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.846799+0000", 2024-05-15T23:12:14.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034289580000000002 2024-05-15T23:12:14.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.846821+0000", 2024-05-15T23:12:14.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2819e-05 2024-05-15T23:12:14.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.873237+0000", 2024-05-15T23:12:14.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026415926999999999 2024-05-15T23:12:14.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.873268+0000", 2024-05-15T23:12:14.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0880000000000002e-05 2024-05-15T23:12:14.764 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.764 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.764 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.764 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:144868 2.0 e138/102 2:0775ffac:::benchmark_data_smithi090_78160_object144867:head v 138'9170, mlcod=138'9166)", 2024-05-15T23:12:14.764 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:07:20.844620+0000", 2024-05-15T23:12:14.764 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 293.90245239299998, 2024-05-15T23:12:14.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028678716, 2024-05-15T23:12:14.764 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.764 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.764 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.764 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.844620+0000", 2024-05-15T23:12:14.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.844620+0000", 2024-05-15T23:12:14.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.844622+0000", 2024-05-15T23:12:14.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5880000000000001e-06 2024-05-15T23:12:14.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.844676+0000", 2024-05-15T23:12:14.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4116999999999997e-05 2024-05-15T23:12:14.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.844689+0000", 2024-05-15T23:12:14.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2811e-05 2024-05-15T23:12:14.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.844696+0000", 2024-05-15T23:12:14.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9489999999999999e-06 2024-05-15T23:12:14.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.850903+0000", 2024-05-15T23:12:14.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.006207377 2024-05-15T23:12:14.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.850925+0000", 2024-05-15T23:12:14.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2286999999999999e-05 2024-05-15T23:12:14.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.873271+0000", 2024-05-15T23:12:14.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022345528999999999 2024-05-15T23:12:14.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.873299+0000", 2024-05-15T23:12:14.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8058000000000001e-05 2024-05-15T23:12:14.768 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.768 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.768 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.768 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:144874 2.0 e138/102 2:0c822d65:::benchmark_data_smithi090_78160_object144873:head v 138'9171, mlcod=138'9166)", 2024-05-15T23:12:14.768 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:07:20.847528+0000", 2024-05-15T23:12:14.769 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 293.89954432500002, 2024-05-15T23:12:14.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025797831, 2024-05-15T23:12:14.769 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.769 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.769 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.769 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.847528+0000", 2024-05-15T23:12:14.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.847528+0000", 2024-05-15T23:12:14.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.847530+0000", 2024-05-15T23:12:14.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5540000000000001e-06 2024-05-15T23:12:14.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.847597+0000", 2024-05-15T23:12:14.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7393999999999999e-05 2024-05-15T23:12:14.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.847612+0000", 2024-05-15T23:12:14.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4844000000000001e-05 2024-05-15T23:12:14.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.847620+0000", 2024-05-15T23:12:14.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8779999999999998e-06 2024-05-15T23:12:14.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.854787+0000", 2024-05-15T23:12:14.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.007167245 2024-05-15T23:12:14.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.854809+0000", 2024-05-15T23:12:14.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1869e-05 2024-05-15T23:12:14.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.873301+0000", 2024-05-15T23:12:14.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018491878 2024-05-15T23:12:14.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.873326+0000", 2024-05-15T23:12:14.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5168999999999999e-05 2024-05-15T23:12:14.772 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.773 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.773 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.773 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:144875 2.0 e138/102 2:0d47eaf4:::benchmark_data_smithi090_78160_object144874:head v 138'9172, mlcod=138'9166)", 2024-05-15T23:12:14.773 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:07:20.850282+0000", 2024-05-15T23:12:14.773 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 293.89679092900002, 2024-05-15T23:12:14.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02307294, 2024-05-15T23:12:14.773 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.773 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.773 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.773 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.850282+0000", 2024-05-15T23:12:14.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.850282+0000", 2024-05-15T23:12:14.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.850284+0000", 2024-05-15T23:12:14.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8759999999999999e-06 2024-05-15T23:12:14.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.850341+0000", 2024-05-15T23:12:14.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7584000000000003e-05 2024-05-15T23:12:14.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.850353+0000", 2024-05-15T23:12:14.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1986e-05 2024-05-15T23:12:14.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.850361+0000", 2024-05-15T23:12:14.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1440000000000003e-06 2024-05-15T23:12:14.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.858743+0000", 2024-05-15T23:12:14.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00838181 2024-05-15T23:12:14.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.858765+0000", 2024-05-15T23:12:14.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1994e-05 2024-05-15T23:12:14.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.873329+0000", 2024-05-15T23:12:14.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014563359 2024-05-15T23:12:14.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.873355+0000", 2024-05-15T23:12:14.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6186999999999998e-05 2024-05-15T23:12:14.777 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.777 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.777 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.777 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:144876 2.0 e138/102 2:078cd037:::benchmark_data_smithi090_78160_object144875:head v 138'9173, mlcod=138'9166)", 2024-05-15T23:12:14.777 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:07:20.852656+0000", 2024-05-15T23:12:14.777 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 293.89441669199999, 2024-05-15T23:12:14.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020725416999999999, 2024-05-15T23:12:14.778 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.778 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.778 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.778 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.852656+0000", 2024-05-15T23:12:14.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.852656+0000", 2024-05-15T23:12:14.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.852658+0000", 2024-05-15T23:12:14.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.59e-06 2024-05-15T23:12:14.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.852721+0000", 2024-05-15T23:12:14.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3250000000000006e-05 2024-05-15T23:12:14.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.852733+0000", 2024-05-15T23:12:14.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2296000000000001e-05 2024-05-15T23:12:14.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.852742+0000", 2024-05-15T23:12:14.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8340000000000005e-06 2024-05-15T23:12:14.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.862187+0000", 2024-05-15T23:12:14.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094447779999999992 2024-05-15T23:12:14.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.862214+0000", 2024-05-15T23:12:14.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7242000000000001e-05 2024-05-15T23:12:14.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.873357+0000", 2024-05-15T23:12:14.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011142569999999999 2024-05-15T23:12:14.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:20.873381+0000", 2024-05-15T23:12:14.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4856999999999999e-05 2024-05-15T23:12:14.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.781 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:170002 2.0 e143/102 2:0b74c617:::benchmark_data_smithi090_78160_object8644:head v 143'10815, mlcod=143'10813)", 2024-05-15T23:12:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:07:42.290397+0000", 2024-05-15T23:12:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 272.456675711, 2024-05-15T23:12:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021302808999999999, 2024-05-15T23:12:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:42.290397+0000", 2024-05-15T23:12:14.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:42.290397+0000", 2024-05-15T23:12:14.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:42.290399+0000", 2024-05-15T23:12:14.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3070000000000001e-06 2024-05-15T23:12:14.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:42.290414+0000", 2024-05-15T23:12:14.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4221e-05 2024-05-15T23:12:14.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:42.290431+0000", 2024-05-15T23:12:14.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7598999999999999e-05 2024-05-15T23:12:14.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:42.290442+0000", 2024-05-15T23:12:14.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0669000000000001e-05 2024-05-15T23:12:14.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:42.293765+0000", 2024-05-15T23:12:14.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003322705 2024-05-15T23:12:14.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:42.293793+0000", 2024-05-15T23:12:14.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8727000000000001e-05 2024-05-15T23:12:14.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:42.311666+0000", 2024-05-15T23:12:14.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017872961 2024-05-15T23:12:14.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:42.311700+0000", 2024-05-15T23:12:14.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.362e-05 2024-05-15T23:12:14.786 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.786 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.786 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.786 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:173010 2.0 e144/102 2:0822df11:::benchmark_data_smithi090_78160_object11652:head v 144'11015, mlcod=144'11013)", 2024-05-15T23:12:14.786 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:07:44.937368+0000", 2024-05-15T23:12:14.786 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 269.80970474399999, 2024-05-15T23:12:14.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019897294999999999, 2024-05-15T23:12:14.787 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.787 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.787 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.787 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.937368+0000", 2024-05-15T23:12:14.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.937368+0000", 2024-05-15T23:12:14.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.937370+0000", 2024-05-15T23:12:14.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4530000000000001e-06 2024-05-15T23:12:14.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.937381+0000", 2024-05-15T23:12:14.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0334e-05 2024-05-15T23:12:14.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.937397+0000", 2024-05-15T23:12:14.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5889e-05 2024-05-15T23:12:14.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.937406+0000", 2024-05-15T23:12:14.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6269999999999995e-06 2024-05-15T23:12:14.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.937474+0000", 2024-05-15T23:12:14.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7991999999999995e-05 2024-05-15T23:12:14.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.937509+0000", 2024-05-15T23:12:14.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4499999999999998e-05 2024-05-15T23:12:14.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.957211+0000", 2024-05-15T23:12:14.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019702193999999999 2024-05-15T23:12:14.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:44.957265+0000", 2024-05-15T23:12:14.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4305999999999998e-05 2024-05-15T23:12:14.790 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.790 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.791 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.791 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:219714 2.0 e151/102 2:08cd53cc:::benchmark_data_smithi090_78160_object58356:head v 151'13892, mlcod=151'13890)", 2024-05-15T23:12:14.791 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:25.590280+0000", 2024-05-15T23:12:14.791 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 229.15679226200001, 2024-05-15T23:12:14.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021311363, 2024-05-15T23:12:14.791 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.791 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.791 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.791 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:25.590280+0000", 2024-05-15T23:12:14.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:25.590280+0000", 2024-05-15T23:12:14.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:25.590282+0000", 2024-05-15T23:12:14.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.978e-06 2024-05-15T23:12:14.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:25.590292+0000", 2024-05-15T23:12:14.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4029999999999996e-06 2024-05-15T23:12:14.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:25.590306+0000", 2024-05-15T23:12:14.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3813e-05 2024-05-15T23:12:14.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:25.590315+0000", 2024-05-15T23:12:14.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0110000000000006e-06 2024-05-15T23:12:14.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:25.596752+0000", 2024-05-15T23:12:14.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064372359999999998 2024-05-15T23:12:14.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:25.596776+0000", 2024-05-15T23:12:14.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4136000000000001e-05 2024-05-15T23:12:14.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:25.611547+0000", 2024-05-15T23:12:14.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014771424999999999 2024-05-15T23:12:14.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:25.611592+0000", 2024-05-15T23:12:14.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4360999999999998e-05 2024-05-15T23:12:14.795 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.795 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.795 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.795 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:224658 2.0 e152/102 2:06f92d41:::benchmark_data_smithi090_78160_object63300:head v 152'14193, mlcod=152'14190)", 2024-05-15T23:12:14.795 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:30.031097+0000", 2024-05-15T23:12:14.795 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 224.715976238, 2024-05-15T23:12:14.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021207779, 2024-05-15T23:12:14.795 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.795 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.796 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.796 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:30.031097+0000", 2024-05-15T23:12:14.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:30.031097+0000", 2024-05-15T23:12:14.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:30.031097+0000", 2024-05-15T23:12:14.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.06e-07 2024-05-15T23:12:14.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:30.031102+0000", 2024-05-15T23:12:14.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5009999999999998e-06 2024-05-15T23:12:14.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:30.031107+0000", 2024-05-15T23:12:14.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0889999999999999e-06 2024-05-15T23:12:14.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:30.031111+0000", 2024-05-15T23:12:14.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5959999999999997e-06 2024-05-15T23:12:14.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:30.035052+0000", 2024-05-15T23:12:14.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039402880000000001 2024-05-15T23:12:14.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:30.035078+0000", 2024-05-15T23:12:14.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6842000000000002e-05 2024-05-15T23:12:14.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:30.052215+0000", 2024-05-15T23:12:14.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017136444000000001 2024-05-15T23:12:14.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:30.052304+0000", 2024-05-15T23:12:14.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9413000000000006e-05 2024-05-15T23:12:14.799 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.799 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.799 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.800 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:226815 2.0 e152/102 2:03974dc8:::benchmark_data_smithi090_78160_object65457:head v 152'14346, mlcod=152'14342)", 2024-05-15T23:12:14.800 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:31.913698+0000", 2024-05-15T23:12:14.800 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 222.83337467600001, 2024-05-15T23:12:14.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025629459, 2024-05-15T23:12:14.800 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.800 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.800 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.800 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.913698+0000", 2024-05-15T23:12:14.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.913698+0000", 2024-05-15T23:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.913700+0000", 2024-05-15T23:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7430000000000001e-06 2024-05-15T23:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.913709+0000", 2024-05-15T23:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7679999999999997e-06 2024-05-15T23:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.913721+0000", 2024-05-15T23:12:14.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2860000000000001e-05 2024-05-15T23:12:14.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.913729+0000", 2024-05-15T23:12:14.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9119999999999998e-06 2024-05-15T23:12:14.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.914666+0000", 2024-05-15T23:12:14.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093635400000000003 2024-05-15T23:12:14.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.914686+0000", 2024-05-15T23:12:14.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9996999999999998e-05 2024-05-15T23:12:14.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.939278+0000", 2024-05-15T23:12:14.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024592471000000001 2024-05-15T23:12:14.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.939328+0000", 2024-05-15T23:12:14.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9354000000000002e-05 2024-05-15T23:12:14.804 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.804 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.804 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.804 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:226816 2.0 e152/102 2:074acbf9:::benchmark_data_smithi090_78160_object65458:head v 152'14347, mlcod=152'14342)", 2024-05-15T23:12:14.804 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:31.915808+0000", 2024-05-15T23:12:14.804 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 222.83126447800001, 2024-05-15T23:12:14.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023547116999999999, 2024-05-15T23:12:14.804 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.804 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.804 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.915808+0000", 2024-05-15T23:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.915808+0000", 2024-05-15T23:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.915810+0000", 2024-05-15T23:12:14.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3289999999999999e-06 2024-05-15T23:12:14.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.915821+0000", 2024-05-15T23:12:14.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0934000000000001e-05 2024-05-15T23:12:14.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.915834+0000", 2024-05-15T23:12:14.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2024-05-15T23:12:14.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.915842+0000", 2024-05-15T23:12:14.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3289999999999996e-06 2024-05-15T23:12:14.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.918412+0000", 2024-05-15T23:12:14.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002570288 2024-05-15T23:12:14.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.918432+0000", 2024-05-15T23:12:14.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9650999999999999e-05 2024-05-15T23:12:14.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.939332+0000", 2024-05-15T23:12:14.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020900430000000001 2024-05-15T23:12:14.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:31.939355+0000", 2024-05-15T23:12:14.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3243000000000001e-05 2024-05-15T23:12:14.808 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.808 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.808 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.808 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:243373 2.0 e156/102 2:046078bd:::benchmark_data_smithi090_78160_object82015:head v 156'15362, mlcod=156'15360)", 2024-05-15T23:12:14.809 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:47.795909+0000", 2024-05-15T23:12:14.809 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 206.951163525, 2024-05-15T23:12:14.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025000378, 2024-05-15T23:12:14.809 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.809 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.809 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.809 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.795909+0000", 2024-05-15T23:12:14.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.795909+0000", 2024-05-15T23:12:14.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.795912+0000", 2024-05-15T23:12:14.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.537e-06 2024-05-15T23:12:14.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.795922+0000", 2024-05-15T23:12:14.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0097e-05 2024-05-15T23:12:14.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.795940+0000", 2024-05-15T23:12:14.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8028000000000001e-05 2024-05-15T23:12:14.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.795949+0000", 2024-05-15T23:12:14.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3409999999999992e-06 2024-05-15T23:12:14.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.796008+0000", 2024-05-15T23:12:14.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8977999999999997e-05 2024-05-15T23:12:14.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.796032+0000", 2024-05-15T23:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3334000000000001e-05 2024-05-15T23:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.820871+0000", 2024-05-15T23:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024839651000000001 2024-05-15T23:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.820910+0000", 2024-05-15T23:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8411999999999997e-05 2024-05-15T23:12:14.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.813 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.813 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:243375 2.0 e156/102 2:047537b2:::benchmark_data_smithi090_78160_object82017:head v 156'15363, mlcod=156'15360)", 2024-05-15T23:12:14.813 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:47.800726+0000", 2024-05-15T23:12:14.813 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 206.94634632399999, 2024-05-15T23:12:14.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020202105000000001, 2024-05-15T23:12:14.813 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.813 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.813 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.813 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.800726+0000", 2024-05-15T23:12:14.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.800726+0000", 2024-05-15T23:12:14.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.800728+0000", 2024-05-15T23:12:14.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9300000000000002e-06 2024-05-15T23:12:14.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.800739+0000", 2024-05-15T23:12:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0465000000000001e-05 2024-05-15T23:12:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.800751+0000", 2024-05-15T23:12:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.208e-05 2024-05-15T23:12:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.800759+0000", 2024-05-15T23:12:14.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5839999999999995e-06 2024-05-15T23:12:14.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.800813+0000", 2024-05-15T23:12:14.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3591000000000002e-05 2024-05-15T23:12:14.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.800833+0000", 2024-05-15T23:12:14.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9632000000000001e-05 2024-05-15T23:12:14.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.820914+0000", 2024-05-15T23:12:14.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020081239000000001 2024-05-15T23:12:14.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.820929+0000", 2024-05-15T23:12:14.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4584e-05 2024-05-15T23:12:14.817 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.817 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.817 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.817 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:248154 2.0 e157/102 2:00a330e0:::benchmark_data_smithi090_78160_object86796:head v 157'15678, mlcod=157'15676)", 2024-05-15T23:12:14.817 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:52.253541+0000", 2024-05-15T23:12:14.817 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 202.49353157300001, 2024-05-15T23:12:14.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027928676999999999, 2024-05-15T23:12:14.818 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.818 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.818 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.818 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.253541+0000", 2024-05-15T23:12:14.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.253541+0000", 2024-05-15T23:12:14.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.253543+0000", 2024-05-15T23:12:14.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1909999999999999e-06 2024-05-15T23:12:14.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.253553+0000", 2024-05-15T23:12:14.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6809999999999993e-06 2024-05-15T23:12:14.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.253567+0000", 2024-05-15T23:12:14.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4413000000000001e-05 2024-05-15T23:12:14.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.253576+0000", 2024-05-15T23:12:14.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2900000000000002e-06 2024-05-15T23:12:14.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.253657+0000", 2024-05-15T23:12:14.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1314000000000001e-05 2024-05-15T23:12:14.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.253680+0000", 2024-05-15T23:12:14.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2925000000000002e-05 2024-05-15T23:12:14.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.281425+0000", 2024-05-15T23:12:14.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027745140000000001 2024-05-15T23:12:14.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.281470+0000", 2024-05-15T23:12:14.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4722999999999999e-05 2024-05-15T23:12:14.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.821 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.822 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:248159 2.0 e157/102 2:0cfc9d0c:::benchmark_data_smithi090_78160_object86801:head v 157'15679, mlcod=157'15676)", 2024-05-15T23:12:14.822 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:52.260334+0000", 2024-05-15T23:12:14.822 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 202.48673865500001, 2024-05-15T23:12:14.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021163504999999999, 2024-05-15T23:12:14.822 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.822 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.822 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.822 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.260334+0000", 2024-05-15T23:12:14.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.260334+0000", 2024-05-15T23:12:14.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.260337+0000", 2024-05-15T23:12:14.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5229999999999998e-06 2024-05-15T23:12:14.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.260349+0000", 2024-05-15T23:12:14.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1967000000000001e-05 2024-05-15T23:12:14.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.260364+0000", 2024-05-15T23:12:14.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5534999999999999e-05 2024-05-15T23:12:14.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.260374+0000", 2024-05-15T23:12:14.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254999999999999e-05 2024-05-15T23:12:14.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.260442+0000", 2024-05-15T23:12:14.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7151999999999997e-05 2024-05-15T23:12:14.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.260466+0000", 2024-05-15T23:12:14.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4556e-05 2024-05-15T23:12:14.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.281475+0000", 2024-05-15T23:12:14.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021008428999999999 2024-05-15T23:12:14.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.281498+0000", 2024-05-15T23:12:14.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3088e-05 2024-05-15T23:12:14.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.826 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.826 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:248161 2.0 e157/102 2:0966c30f:::benchmark_data_smithi090_78160_object86803:head v 157'15680, mlcod=157'15676)", 2024-05-15T23:12:14.826 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:52.260618+0000", 2024-05-15T23:12:14.826 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 202.486455019, 2024-05-15T23:12:14.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020901231999999999, 2024-05-15T23:12:14.826 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.827 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.827 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.260618+0000", 2024-05-15T23:12:14.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.260618+0000", 2024-05-15T23:12:14.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.260619+0000", 2024-05-15T23:12:14.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.083e-06 2024-05-15T23:12:14.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.260625+0000", 2024-05-15T23:12:14.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1179999999999997e-06 2024-05-15T23:12:14.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.260635+0000", 2024-05-15T23:12:14.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0339e-05 2024-05-15T23:12:14.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.260642+0000", 2024-05-15T23:12:14.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5230000000000001e-06 2024-05-15T23:12:14.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.264074+0000", 2024-05-15T23:12:14.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003431786 2024-05-15T23:12:14.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.264095+0000", 2024-05-15T23:12:14.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0934999999999999e-05 2024-05-15T23:12:14.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.281499+0000", 2024-05-15T23:12:14.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017404611 2024-05-15T23:12:14.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:52.281519+0000", 2024-05-15T23:12:14.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9837e-05 2024-05-15T23:12:14.830 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.831 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.831 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.831 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:265768 2.0 e166/102 2:0aeab925:::benchmark_data_smithi090_78160_object104410:head v 166'16804, mlcod=166'16802)", 2024-05-15T23:12:14.831 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:09:11.983862+0000", 2024-05-15T23:12:14.831 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 182.76321108100001, 2024-05-15T23:12:14.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027959887999999999, 2024-05-15T23:12:14.831 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.831 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.831 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.831 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.983862+0000", 2024-05-15T23:12:14.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.983862+0000", 2024-05-15T23:12:14.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.983863+0000", 2024-05-15T23:12:14.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.294e-06 2024-05-15T23:12:14.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.983871+0000", 2024-05-15T23:12:14.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-05-15T23:12:14.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.983884+0000", 2024-05-15T23:12:14.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3648999999999999e-05 2024-05-15T23:12:14.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.983892+0000", 2024-05-15T23:12:14.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6550000000000004e-06 2024-05-15T23:12:14.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.983954+0000", 2024-05-15T23:12:14.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2498999999999999e-05 2024-05-15T23:12:14.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.983975+0000", 2024-05-15T23:12:14.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0628999999999998e-05 2024-05-15T23:12:14.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:12.011743+0000", 2024-05-15T23:12:14.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027767449999999999 2024-05-15T23:12:14.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:12.011822+0000", 2024-05-15T23:12:14.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8975999999999997e-05 2024-05-15T23:12:14.836 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.836 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.836 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.836 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:265776 2.0 e166/102 2:0f7011e0:::benchmark_data_smithi090_78160_object104418:head v 166'16805, mlcod=166'16802)", 2024-05-15T23:12:14.836 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:09:11.987678+0000", 2024-05-15T23:12:14.836 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 182.759395109, 2024-05-15T23:12:14.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024193077, 2024-05-15T23:12:14.836 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.836 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.836 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.836 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.987678+0000", 2024-05-15T23:12:14.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.987678+0000", 2024-05-15T23:12:14.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.987680+0000", 2024-05-15T23:12:14.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2879999999999999e-06 2024-05-15T23:12:14.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.987690+0000", 2024-05-15T23:12:14.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0506000000000001e-05 2024-05-15T23:12:14.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.987704+0000", 2024-05-15T23:12:14.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4070000000000001e-05 2024-05-15T23:12:14.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.987714+0000", 2024-05-15T23:12:14.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9510000000000001e-06 2024-05-15T23:12:14.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.987771+0000", 2024-05-15T23:12:14.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6533000000000002e-05 2024-05-15T23:12:14.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.987795+0000", 2024-05-15T23:12:14.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4488e-05 2024-05-15T23:12:14.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:12.011825+0000", 2024-05-15T23:12:14.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024029419999999999 2024-05-15T23:12:14.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:12.011871+0000", 2024-05-15T23:12:14.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5821000000000001e-05 2024-05-15T23:12:14.840 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.840 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.840 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.841 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:265784 2.0 e166/102 2:07c80b1a:::benchmark_data_smithi090_78160_object104426:head v 166'16806, mlcod=166'16802)", 2024-05-15T23:12:14.841 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:09:11.991430+0000", 2024-05-15T23:12:14.841 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 182.755642626, 2024-05-15T23:12:14.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020488798999999999, 2024-05-15T23:12:14.841 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:14.841 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:14.841 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:14.841 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:14.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:14.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.991430+0000", 2024-05-15T23:12:14.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.991430+0000", 2024-05-15T23:12:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.991432+0000", 2024-05-15T23:12:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.581e-06 2024-05-15T23:12:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.991448+0000", 2024-05-15T23:12:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6475000000000001e-05 2024-05-15T23:12:14.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:14.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.991460+0000", 2024-05-15T23:12:14.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1637999999999999e-05 2024-05-15T23:12:14.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:14.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:11.991468+0000", 2024-05-15T23:12:14.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9200000000000004e-06 2024-05-15T23:12:14.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:14.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:12.003254+0000", 2024-05-15T23:12:14.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011786662 2024-05-15T23:12:14.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:14.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:12.003282+0000", 2024-05-15T23:12:14.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7498e-05 2024-05-15T23:12:14.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:14.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:12.011874+0000", 2024-05-15T23:12:14.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085925189999999999 2024-05-15T23:12:14.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:14.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:14.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:14.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:12.011919+0000", 2024-05-15T23:12:14.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4505999999999997e-05 2024-05-15T23:12:14.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.845 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:14.845 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:14.845 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:12:14.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:12:14.935115 min lat: 0.000948627 max lat: 0.219607 avg lat: 0.0082098 2024-05-15T23:12:14.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:12:14.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 80 16 155896 155880 121.755 56.3125 0.00540286 0.0082098 2024-05-15T23:12:15.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T23:12:15.848 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:12:15.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 81 16 156844 156828 120.983 59.25 0.0147984 0.00826415 2024-05-15T23:12:16.383 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:16 smithi090 ceph-mon[44434]: pgmap v671: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 872 op/s 2024-05-15T23:12:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:16 smithi088 ceph-mon[44699]: pgmap v671: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 872 op/s 2024-05-15T23:12:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:16 smithi084 ceph-mon[39784]: pgmap v671: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 872 op/s 2024-05-15T23:12:16.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 82 15 157738 157723 120.19 55.9375 0.00981443 0.00831705 2024-05-15T23:12:17.693 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:17.728 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:12:17.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:17.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:12:17.800 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:12:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:12:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72783 3.e e243/202)", 2024-05-15T23:12:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.738416+0000", 2024-05-15T23:12:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.058687602999999998, 2024-05-15T23:12:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.058705519999999997, 2024-05-15T23:12:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.738416+0000", 2024-05-15T23:12:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.738416+0000", 2024-05-15T23:12:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.738417+0000", 2024-05-15T23:12:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2419999999999999e-06 2024-05-15T23:12:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.738422+0000", 2024-05-15T23:12:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9910000000000002e-06 2024-05-15T23:12:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.738437+0000", 2024-05-15T23:12:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4384e-05 2024-05-15T23:12:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.738445+0000", 2024-05-15T23:12:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6640000000000004e-06 2024-05-15T23:12:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72799 3.e e243/202)", 2024-05-15T23:12:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.793065+0000", 2024-05-15T23:12:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0040389520000000002, 2024-05-15T23:12:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040888219999999998, 2024-05-15T23:12:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.793065+0000", 2024-05-15T23:12:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.793065+0000", 2024-05-15T23:12:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.793066+0000", 2024-05-15T23:12:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3820000000000001e-06 2024-05-15T23:12:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.793075+0000", 2024-05-15T23:12:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6540000000000005e-06 2024-05-15T23:12:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.793086+0000", 2024-05-15T23:12:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1776e-05 2024-05-15T23:12:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.793094+0000", 2024-05-15T23:12:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2570000000000001e-06 2024-05-15T23:12:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72784 3.e e243/202)", 2024-05-15T23:12:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.743879+0000", 2024-05-15T23:12:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.053224665999999997, 2024-05-15T23:12:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.053301284999999997, 2024-05-15T23:12:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.743879+0000", 2024-05-15T23:12:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.743879+0000", 2024-05-15T23:12:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.743881+0000", 2024-05-15T23:12:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2689999999999998e-06 2024-05-15T23:12:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.743890+0000", 2024-05-15T23:12:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3739999999999998e-06 2024-05-15T23:12:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.743903+0000", 2024-05-15T23:12:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3519999999999999e-05 2024-05-15T23:12:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.743913+0000", 2024-05-15T23:12:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0456e-05 2024-05-15T23:12:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158484 3.e 3.34375eae (undecoded) ondisk+write+known_if_redirected e243)", 2024-05-15T23:12:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.794361+0000", 2024-05-15T23:12:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0027428769999999999, 2024-05-15T23:12:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028546169999999998, 2024-05-15T23:12:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158484 2024-05-15T23:12:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794361+0000", 2024-05-15T23:12:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794361+0000", 2024-05-15T23:12:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794363+0000", 2024-05-15T23:12:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.317e-06 2024-05-15T23:12:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794416+0000", 2024-05-15T23:12:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3442000000000001e-05 2024-05-15T23:12:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794428+0000", 2024-05-15T23:12:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.166e-05 2024-05-15T23:12:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794436+0000", 2024-05-15T23:12:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4869999999999996e-06 2024-05-15T23:12:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158485 3.8 3:1a6a90a4:::benchmark_data_smithi090_90974_object158484:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:12:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.794448+0000", 2024-05-15T23:12:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0026550800000000002, 2024-05-15T23:12:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028050620000000001, 2024-05-15T23:12:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:12:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158485 2024-05-15T23:12:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794448+0000", 2024-05-15T23:12:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794448+0000", 2024-05-15T23:12:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794449+0000", 2024-05-15T23:12:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6599999999999995e-07 2024-05-15T23:12:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794487+0000", 2024-05-15T23:12:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8253000000000001e-05 2024-05-15T23:12:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794494+0000", 2024-05-15T23:12:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6479999999999997e-06 2024-05-15T23:12:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794498+0000", 2024-05-15T23:12:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7239999999999998e-06 2024-05-15T23:12:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794533+0000", 2024-05-15T23:12:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4665000000000001e-05 2024-05-15T23:12:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794622+0000", 2024-05-15T23:12:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9798999999999999e-05 2024-05-15T23:12:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:12:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794716+0000", 2024-05-15T23:12:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3362999999999994e-05 2024-05-15T23:12:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72785 3.e e243/202)", 2024-05-15T23:12:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.749148+0000", 2024-05-15T23:12:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.047955224999999997, 2024-05-15T23:12:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.048145884999999999, 2024-05-15T23:12:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.749148+0000", 2024-05-15T23:12:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.749148+0000", 2024-05-15T23:12:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.749150+0000", 2024-05-15T23:12:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9400000000000001e-06 2024-05-15T23:12:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.749159+0000", 2024-05-15T23:12:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7970000000000006e-06 2024-05-15T23:12:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.749168+0000", 2024-05-15T23:12:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2830000000000007e-06 2024-05-15T23:12:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.749175+0000", 2024-05-15T23:12:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8519999999999998e-06 2024-05-15T23:12:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.25135.0:158479 3.8 e243/202)", 2024-05-15T23:12:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.794609+0000", 2024-05-15T23:12:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0024943529999999999, 2024-05-15T23:12:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027104059999999998, 2024-05-15T23:12:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794609+0000", 2024-05-15T23:12:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794609+0000", 2024-05-15T23:12:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794610+0000", 2024-05-15T23:12:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.176e-06 2024-05-15T23:12:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794615+0000", 2024-05-15T23:12:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5070000000000002e-06 2024-05-15T23:12:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794623+0000", 2024-05-15T23:12:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8290000000000008e-06 2024-05-15T23:12:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.794629+0000", 2024-05-15T23:12:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2870000000000001e-06 2024-05-15T23:12:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72786 3.e e243/202)", 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.754303+0000", 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.042800238999999997, 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.043041873000000001, 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.754303+0000", 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.754303+0000", 2024-05-15T23:12:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.754304+0000", 2024-05-15T23:12:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6799999999999999e-07 2024-05-15T23:12:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.754311+0000", 2024-05-15T23:12:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4470000000000003e-06 2024-05-15T23:12:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.754318+0000", 2024-05-15T23:12:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.238e-06 2024-05-15T23:12:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.754322+0000", 2024-05-15T23:12:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-05-15T23:12:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72787 3.e e243/202)", 2024-05-15T23:12:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.758653+0000", 2024-05-15T23:12:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.038450196999999998, 2024-05-15T23:12:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038717453999999998, 2024-05-15T23:12:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.758653+0000", 2024-05-15T23:12:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.758653+0000", 2024-05-15T23:12:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.758655+0000", 2024-05-15T23:12:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9970000000000001e-06 2024-05-15T23:12:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.758661+0000", 2024-05-15T23:12:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5130000000000004e-06 2024-05-15T23:12:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.758674+0000", 2024-05-15T23:12:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3577999999999999e-05 2024-05-15T23:12:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.758682+0000", 2024-05-15T23:12:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.61e-06 2024-05-15T23:12:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158423 3.e 3.28d3455e (undecoded) ondisk+write+known_if_redirected e243)", 2024-05-15T23:12:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.701674+0000", 2024-05-15T23:12:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.095429611999999997, 2024-05-15T23:12:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095725395000000005, 2024-05-15T23:12:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158423 2024-05-15T23:12:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701674+0000", 2024-05-15T23:12:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701674+0000", 2024-05-15T23:12:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701675+0000", 2024-05-15T23:12:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.424e-06 2024-05-15T23:12:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701694+0000", 2024-05-15T23:12:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8777e-05 2024-05-15T23:12:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701703+0000", 2024-05-15T23:12:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1090000000000004e-06 2024-05-15T23:12:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701710+0000", 2024-05-15T23:12:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0650000000000001e-06 2024-05-15T23:12:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158426 3.e 3.30ea24ae (undecoded) ondisk+write+known_if_redirected e243)", 2024-05-15T23:12:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.704015+0000", 2024-05-15T23:12:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.093088154000000006, 2024-05-15T23:12:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.093413971999999998, 2024-05-15T23:12:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158426 2024-05-15T23:12:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704015+0000", 2024-05-15T23:12:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704015+0000", 2024-05-15T23:12:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704017+0000", 2024-05-15T23:12:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7910000000000001e-06 2024-05-15T23:12:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704039+0000", 2024-05-15T23:12:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2243e-05 2024-05-15T23:12:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704053+0000", 2024-05-15T23:12:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3169e-05 2024-05-15T23:12:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704060+0000", 2024-05-15T23:12:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6799999999999993e-06 2024-05-15T23:12:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.25135.0:158411 3.e e243/202)", 2024-05-15T23:12:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.705460+0000", 2024-05-15T23:12:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.091643050000000004, 2024-05-15T23:12:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.091997788999999996, 2024-05-15T23:12:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.705460+0000", 2024-05-15T23:12:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.705460+0000", 2024-05-15T23:12:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.705462+0000", 2024-05-15T23:12:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.111e-06 2024-05-15T23:12:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.705469+0000", 2024-05-15T23:12:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6969999999999993e-06 2024-05-15T23:12:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.705479+0000", 2024-05-15T23:12:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3449999999999995e-06 2024-05-15T23:12:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.705485+0000", 2024-05-15T23:12:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5259999999999999e-06 2024-05-15T23:12:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72789 3.e e243/202)", 2024-05-15T23:12:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.763354+0000", 2024-05-15T23:12:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.033749924000000001, 2024-05-15T23:12:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034130222000000002, 2024-05-15T23:12:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.763354+0000", 2024-05-15T23:12:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.763354+0000", 2024-05-15T23:12:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.763355+0000", 2024-05-15T23:12:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5629999999999999e-06 2024-05-15T23:12:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.763361+0000", 2024-05-15T23:12:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8889999999999999e-06 2024-05-15T23:12:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.763374+0000", 2024-05-15T23:12:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3156000000000001e-05 2024-05-15T23:12:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.763381+0000", 2024-05-15T23:12:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0829999999999998e-06 2024-05-15T23:12:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158429 3.e 3.6f00102e (undecoded) ondisk+write+known_if_redirected e243)", 2024-05-15T23:12:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.708210+0000", 2024-05-15T23:12:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.088893788000000001, 2024-05-15T23:12:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089300624999999995, 2024-05-15T23:12:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158429 2024-05-15T23:12:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708210+0000", 2024-05-15T23:12:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708210+0000", 2024-05-15T23:12:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708212+0000", 2024-05-15T23:12:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4430000000000002e-06 2024-05-15T23:12:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708234+0000", 2024-05-15T23:12:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2195e-05 2024-05-15T23:12:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708245+0000", 2024-05-15T23:12:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1012999999999999e-05 2024-05-15T23:12:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708253+0000", 2024-05-15T23:12:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8879999999999997e-06 2024-05-15T23:12:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72775 3.e e243/202)", 2024-05-15T23:12:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.714888+0000", 2024-05-15T23:12:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.082215045, 2024-05-15T23:12:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.082661079999999998, 2024-05-15T23:12:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.714888+0000", 2024-05-15T23:12:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.714888+0000", 2024-05-15T23:12:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.714891+0000", 2024-05-15T23:12:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0889999999999998e-06 2024-05-15T23:12:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.714897+0000", 2024-05-15T23:12:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7789999999999998e-06 2024-05-15T23:12:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.714909+0000", 2024-05-15T23:12:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1445e-05 2024-05-15T23:12:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.714919+0000", 2024-05-15T23:12:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0468e-05 2024-05-15T23:12:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72791 3.e e243/202)", 2024-05-15T23:12:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.766831+0000", 2024-05-15T23:12:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.030272487000000001, 2024-05-15T23:12:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030765974000000001, 2024-05-15T23:12:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.766831+0000", 2024-05-15T23:12:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.766831+0000", 2024-05-15T23:12:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.766833+0000", 2024-05-15T23:12:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8199999999999999e-06 2024-05-15T23:12:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.766839+0000", 2024-05-15T23:12:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6609999999999997e-06 2024-05-15T23:12:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.766850+0000", 2024-05-15T23:12:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1707e-05 2024-05-15T23:12:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.766858+0000", 2024-05-15T23:12:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5140000000000004e-06 2024-05-15T23:12:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72776 3.e e243/202)", 2024-05-15T23:12:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.718019+0000", 2024-05-15T23:12:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.079084836000000006, 2024-05-15T23:12:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.079614905999999999, 2024-05-15T23:12:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.718019+0000", 2024-05-15T23:12:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.718019+0000", 2024-05-15T23:12:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.718020+0000", 2024-05-15T23:12:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4470000000000001e-06 2024-05-15T23:12:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.718025+0000", 2024-05-15T23:12:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9019999999999998e-06 2024-05-15T23:12:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.718035+0000", 2024-05-15T23:12:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5529999999999998e-06 2024-05-15T23:12:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.718040+0000", 2024-05-15T23:12:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4310000000000002e-06 2024-05-15T23:12:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158468 3.e 3.bf535e5e (undecoded) ondisk+write+known_if_redirected e243)", 2024-05-15T23:12:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.768558+0000", 2024-05-15T23:12:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.028545727, 2024-05-15T23:12:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029115746000000001, 2024-05-15T23:12:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158468 2024-05-15T23:12:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.768558+0000", 2024-05-15T23:12:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.768558+0000", 2024-05-15T23:12:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.768560+0000", 2024-05-15T23:12:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.323e-06 2024-05-15T23:12:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.768582+0000", 2024-05-15T23:12:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1546999999999999e-05 2024-05-15T23:12:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.768592+0000", 2024-05-15T23:12:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0698e-05 2024-05-15T23:12:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.768599+0000", 2024-05-15T23:12:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1399999999999997e-06 2024-05-15T23:12:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72793 3.e e243/202)", 2024-05-15T23:12:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.771616+0000", 2024-05-15T23:12:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.025487860000000001, 2024-05-15T23:12:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026090342999999998, 2024-05-15T23:12:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.771616+0000", 2024-05-15T23:12:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.771616+0000", 2024-05-15T23:12:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.771617+0000", 2024-05-15T23:12:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.658e-06 2024-05-15T23:12:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.771623+0000", 2024-05-15T23:12:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4790000000000004e-06 2024-05-15T23:12:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.771637+0000", 2024-05-15T23:12:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3739e-05 2024-05-15T23:12:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.771644+0000", 2024-05-15T23:12:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6729999999999992e-06 2024-05-15T23:12:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72779 3.e e243/202)", 2024-05-15T23:12:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.722650+0000", 2024-05-15T23:12:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.074453216000000003, 2024-05-15T23:12:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.075099951999999998, 2024-05-15T23:12:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.722650+0000", 2024-05-15T23:12:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.722650+0000", 2024-05-15T23:12:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.722652+0000", 2024-05-15T23:12:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.22e-06 2024-05-15T23:12:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.722664+0000", 2024-05-15T23:12:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2841e-05 2024-05-15T23:12:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.722673+0000", 2024-05-15T23:12:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6980000000000004e-06 2024-05-15T23:12:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.722680+0000", 2024-05-15T23:12:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.691e-06 2024-05-15T23:12:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72794 3.e e243/202)", 2024-05-15T23:12:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.776154+0000", 2024-05-15T23:12:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.020949479999999999, 2024-05-15T23:12:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021633917999999999, 2024-05-15T23:12:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.776154+0000", 2024-05-15T23:12:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.776154+0000", 2024-05-15T23:12:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.776155+0000", 2024-05-15T23:12:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3009999999999999e-06 2024-05-15T23:12:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.776161+0000", 2024-05-15T23:12:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1e-06 2024-05-15T23:12:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.776172+0000", 2024-05-15T23:12:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0482000000000001e-05 2024-05-15T23:12:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.776180+0000", 2024-05-15T23:12:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5040000000000002e-06 2024-05-15T23:12:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72795 3.e e243/202)", 2024-05-15T23:12:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.779763+0000", 2024-05-15T23:12:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.017340373999999999, 2024-05-15T23:12:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018060151999999999, 2024-05-15T23:12:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.779763+0000", 2024-05-15T23:12:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.779763+0000", 2024-05-15T23:12:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.779765+0000", 2024-05-15T23:12:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2560000000000001e-06 2024-05-15T23:12:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.779771+0000", 2024-05-15T23:12:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8200000000000002e-06 2024-05-15T23:12:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.779782+0000", 2024-05-15T23:12:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0710000000000001e-05 2024-05-15T23:12:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.779790+0000", 2024-05-15T23:12:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6709999999999999e-06 2024-05-15T23:12:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72780 3.e e243/202)", 2024-05-15T23:12:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.729696+0000", 2024-05-15T23:12:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.067407247000000003, 2024-05-15T23:12:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068152620999999997, 2024-05-15T23:12:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.729696+0000", 2024-05-15T23:12:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.729696+0000", 2024-05-15T23:12:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.729698+0000", 2024-05-15T23:12:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5209999999999999e-06 2024-05-15T23:12:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.729703+0000", 2024-05-15T23:12:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4739999999999996e-06 2024-05-15T23:12:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.729714+0000", 2024-05-15T23:12:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0315e-05 2024-05-15T23:12:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.729721+0000", 2024-05-15T23:12:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0380000000000002e-06 2024-05-15T23:12:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158479 3.8 3:16a7cb4f:::benchmark_data_smithi090_90974_object158478:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:12:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.784932+0000", 2024-05-15T23:12:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.012171095999999999, 2024-05-15T23:12:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012947527, 2024-05-15T23:12:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:12:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158479 2024-05-15T23:12:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.784932+0000", 2024-05-15T23:12:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.784932+0000", 2024-05-15T23:12:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.784934+0000", 2024-05-15T23:12:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3030000000000001e-06 2024-05-15T23:12:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.784953+0000", 2024-05-15T23:12:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9307999999999999e-05 2024-05-15T23:12:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.784964+0000", 2024-05-15T23:12:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0969e-05 2024-05-15T23:12:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.784972+0000", 2024-05-15T23:12:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9670000000000001e-06 2024-05-15T23:12:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.789380+0000", 2024-05-15T23:12:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044077029999999998 2024-05-15T23:12:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.789473+0000", 2024-05-15T23:12:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3153999999999996e-05 2024-05-15T23:12:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:12:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.789575+0000", 2024-05-15T23:12:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102595 2024-05-15T23:12:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.793361+0000", 2024-05-15T23:12:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037852179999999999 2024-05-15T23:12:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72781 3.e e243/202)", 2024-05-15T23:12:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.733769+0000", 2024-05-15T23:12:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.063334046000000005, 2024-05-15T23:12:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.064153717999999998, 2024-05-15T23:12:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.733769+0000", 2024-05-15T23:12:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.733769+0000", 2024-05-15T23:12:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.733772+0000", 2024-05-15T23:12:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2309999999999999e-06 2024-05-15T23:12:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.733782+0000", 2024-05-15T23:12:17.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0003000000000001e-05 2024-05-15T23:12:17.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.733794+0000", 2024-05-15T23:12:17.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2702e-05 2024-05-15T23:12:17.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.733803+0000", 2024-05-15T23:12:17.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0100000000000001e-06 2024-05-15T23:12:17.879 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.879 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.879 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.880 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72796 3.e e243/202)", 2024-05-15T23:12:17.880 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.788699+0000", 2024-05-15T23:12:17.880 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0084047289999999997, 2024-05-15T23:12:17.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092499379999999992, 2024-05-15T23:12:17.880 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.880 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.880 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.880 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.788699+0000", 2024-05-15T23:12:17.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.788699+0000", 2024-05-15T23:12:17.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.788701+0000", 2024-05-15T23:12:17.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.973e-06 2024-05-15T23:12:17.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.788706+0000", 2024-05-15T23:12:17.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8319999999999996e-06 2024-05-15T23:12:17.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.788716+0000", 2024-05-15T23:12:17.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0881e-05 2024-05-15T23:12:17.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.788724+0000", 2024-05-15T23:12:17.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9869999999999999e-06 2024-05-15T23:12:17.882 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.882 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.882 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.883 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.11.0:72798 3.e e243/202)", 2024-05-15T23:12:17.883 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.788748+0000", 2024-05-15T23:12:17.883 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0083558850000000004, 2024-05-15T23:12:17.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092265769999999997, 2024-05-15T23:12:17.883 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.883 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.883 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:12:17.883 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.788748+0000", 2024-05-15T23:12:17.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.788748+0000", 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.788748+0000", 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-07 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.788752+0000", 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4379999999999999e-06 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.788758+0000", 2024-05-15T23:12:17.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8799999999999996e-06 2024-05-15T23:12:17.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.788767+0000", 2024-05-15T23:12:17.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2369999999999998e-06 2024-05-15T23:12:17.885 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.885 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.885 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.885 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158411 3.e 3:721f625f:::benchmark_data_smithi090_90974_object158410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:12:17.885 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.688475+0000", 2024-05-15T23:12:17.885 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.108628048, 2024-05-15T23:12:17.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10952876, 2024-05-15T23:12:17.886 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:17.886 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:17.886 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:12:17.886 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:17.886 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:17.886 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:17.886 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158411 2024-05-15T23:12:17.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.886 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:17.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:17.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688475+0000", 2024-05-15T23:12:17.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:17.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688475+0000", 2024-05-15T23:12:17.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:17.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:17.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688478+0000", 2024-05-15T23:12:17.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6199999999999999e-06 2024-05-15T23:12:17.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:17.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688507+0000", 2024-05-15T23:12:17.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8671e-05 2024-05-15T23:12:17.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:17.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688524+0000", 2024-05-15T23:12:17.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6837000000000002e-05 2024-05-15T23:12:17.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:17.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688534+0000", 2024-05-15T23:12:17.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0513000000000001e-05 2024-05-15T23:12:17.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:17.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701293+0000", 2024-05-15T23:12:17.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012759111 2024-05-15T23:12:17.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:17.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701443+0000", 2024-05-15T23:12:17.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015017200000000001 2024-05-15T23:12:17.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:17.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:17.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:17.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701612+0000", 2024-05-15T23:12:17.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016817399999999999 2024-05-15T23:12:17.889 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.889 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:17.889 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.890 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:17.890 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-05-15T23:12:17.890 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 28 2024-05-15T23:12:17.890 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:12:17.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:12:17.934 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:17.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 83 16 158589 158573 119.382 53.125 0.00483995 0.00837221 2024-05-15T23:12:18.064 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:12:18.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:18.067 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:12:18.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:17 smithi084 ceph-mon[39784]: pgmap v672: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 913 op/s 2024-05-15T23:12:18.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:17 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:12:18.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:17 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:12:18.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:12:18.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:17 smithi090 ceph-mon[44434]: pgmap v672: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 913 op/s 2024-05-15T23:12:18.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:17 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:12:18.169 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:17 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:12:18.201 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:17 smithi088 ceph-mon[44699]: pgmap v672: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 913 op/s 2024-05-15T23:12:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:17 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:12:18.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:17 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:12:18.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T23:12:18.550 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:12:18.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:18.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:12:18.654 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:12:18.701 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:18.851 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.9 to 1 (10000) 2024-05-15T23:12:18.907 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:12:18.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 84 16 159455 159439 118.605 54.125 0.010676 0.00842916 2024-05-15T23:12:19.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:18 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:12:19.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:18 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:12:19.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:18 smithi084 ceph-mon[39784]: osdmap e244: 12 total, 12 up, 12 in 2024-05-15T23:12:19.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:18 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1721784884' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:12:19.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:18 smithi084 ceph-mon[39784]: from='client.35318 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:12:19.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:18 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:12:19.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:18 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:12:19.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:18 smithi090 ceph-mon[44434]: osdmap e244: 12 total, 12 up, 12 in 2024-05-15T23:12:19.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:18 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1721784884' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:12:19.175 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:18 smithi090 ceph-mon[44434]: from='client.35318 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:12:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:18 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:12:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:18 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:12:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:18 smithi088 ceph-mon[44699]: osdmap e244: 12 total, 12 up, 12 in 2024-05-15T23:12:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:18 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1721784884' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:12:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:18 smithi088 ceph-mon[44699]: from='client.35318 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:12:19.462 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:12:19.936 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 85 16 160381 160365 117.891 57.875 0.0100729 0.00847895 2024-05-15T23:12:20.068 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:12:20.069 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 10 1 2024-05-15T23:12:20.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:20.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:12:20.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:19 smithi084 ceph-mon[39784]: pgmap v674: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 893 op/s 2024-05-15T23:12:20.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:19 smithi084 ceph-mon[39784]: from='client.35318 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:12:20.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:19 smithi084 ceph-mon[39784]: osdmap e245: 12 total, 12 up, 12 in 2024-05-15T23:12:20.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:19 smithi090 ceph-mon[44434]: pgmap v674: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 893 op/s 2024-05-15T23:12:20.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:19 smithi090 ceph-mon[44434]: from='client.35318 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:12:20.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:19 smithi090 ceph-mon[44434]: osdmap e245: 12 total, 12 up, 12 in 2024-05-15T23:12:20.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:12:20.213 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:20.244 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:12:20.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:20.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:12:20.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:19 smithi088 ceph-mon[44699]: pgmap v674: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 893 op/s 2024-05-15T23:12:20.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:19 smithi088 ceph-mon[44699]: from='client.35318 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:12:20.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:19 smithi088 ceph-mon[44699]: osdmap e245: 12 total, 12 up, 12 in 2024-05-15T23:12:20.352 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:12:20.425 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:20.574 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:12:20.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:20.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:12:20.684 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:12:20.736 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:20.758 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:12:20.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:20.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:12:20.762 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:20.769 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:20.796 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:12:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:12:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:12:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:12:20.796 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:12:20.864 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:12:20.898 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:20.923 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:12:20.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:20.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:12:20.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 86 16 161158 161142 117.085 48.5625 0.00808244 0.00853884 2024-05-15T23:12:21.029 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:12:21.034 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:20 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:12:21.034 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:20 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:12:21.086 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:21.218 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:20 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:12:21.218 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:20 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:12:21.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:20 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:12:21.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:20 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:12:21.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T23:12:21.748 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:12:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:21.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:12:21.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:12:21.863 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:12:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:21.911 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:12:21.932 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:12:21.935 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 87 16 162015 161999 116.355 53.5625 0.0116608 0.00859141 2024-05-15T23:12:21.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:21.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:12:22.039 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:12:22.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:21 smithi084 ceph-mon[39784]: pgmap v676: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 827 op/s 2024-05-15T23:12:22.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:21 smithi084 ceph-mon[39784]: osdmap e246: 12 total, 12 up, 12 in 2024-05-15T23:12:22.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:21 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4127272945' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:12:22.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:21 smithi084 ceph-mon[39784]: from='client.35324 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:12:22.090 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:22.241 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:21 smithi090 ceph-mon[44434]: pgmap v676: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 827 op/s 2024-05-15T23:12:22.241 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:21 smithi090 ceph-mon[44434]: osdmap e246: 12 total, 12 up, 12 in 2024-05-15T23:12:22.241 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:21 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4127272945' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:12:22.242 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:21 smithi090 ceph-mon[44434]: from='client.35324 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:12:22.242 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:12:22.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:22.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:12:22.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:21 smithi088 ceph-mon[44699]: pgmap v676: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 9.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 827 op/s 2024-05-15T23:12:22.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:21 smithi088 ceph-mon[44699]: osdmap e246: 12 total, 12 up, 12 in 2024-05-15T23:12:22.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:21 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4127272945' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:12:22.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:21 smithi088 ceph-mon[44699]: from='client.35324 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:12:22.347 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:12:22.384 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:22.937 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 88 15 163196 163181 115.871 73.875 0.0177703 0.00862853 2024-05-15T23:12:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:22 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:12:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:22 smithi084 ceph-mon[39784]: from='client.35324 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:12:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:22 smithi084 ceph-mon[39784]: osdmap e247: 12 total, 12 up, 12 in 2024-05-15T23:12:23.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:22 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:12:23.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:22 smithi090 ceph-mon[44434]: from='client.35324 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:12:23.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:22 smithi090 ceph-mon[44434]: osdmap e247: 12 total, 12 up, 12 in 2024-05-15T23:12:23.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:22 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:12:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:22 smithi088 ceph-mon[44699]: from='client.35324 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:12:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:22 smithi088 ceph-mon[44699]: osdmap e247: 12 total, 12 up, 12 in 2024-05-15T23:12:23.937 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 89 16 164431 164415 115.436 77.125 0.00650702 0.00866136 2024-05-15T23:12:24.028 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:12:24.028 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:12:24.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:12:24.028 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:12:24.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.028 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109643 2.c 2:35bed8ca:::benchmark_data_smithi090_78160_object109642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:12:24.028 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.334876+0000", 2024-05-15T23:12:24.029 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 330.68908119499997, 2024-05-15T23:12:24.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.131093448, 2024-05-15T23:12:24.029 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.029 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.029 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.029 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.029 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:12:24.029 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:12:24.029 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109643 2024-05-15T23:12:24.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.029 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334876+0000", 2024-05-15T23:12:24.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334876+0000", 2024-05-15T23:12:24.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334878+0000", 2024-05-15T23:12:24.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5290000000000001e-06 2024-05-15T23:12:24.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334899+0000", 2024-05-15T23:12:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0951999999999999e-05 2024-05-15T23:12:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334909+0000", 2024-05-15T23:12:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308e-05 2024-05-15T23:12:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334917+0000", 2024-05-15T23:12:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3119999999999996e-06 2024-05-15T23:12:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.339148+0000", 2024-05-15T23:12:24.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004231004 2024-05-15T23:12:24.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.339226+0000", 2024-05-15T23:12:24.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8885000000000001e-05 2024-05-15T23:12:24.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:12:24.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.339304+0000", 2024-05-15T23:12:24.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7218000000000005e-05 2024-05-15T23:12:24.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.343262+0000", 2024-05-15T23:12:24.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003958746 2024-05-15T23:12:24.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351963+0000", 2024-05-15T23:12:24.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087008099999999998 2024-05-15T23:12:24.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.352004+0000", 2024-05-15T23:12:24.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0442999999999999e-05 2024-05-15T23:12:24.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465970+0000", 2024-05-15T23:12:24.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.113966241 2024-05-15T23:12:24.034 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.034 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.034 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.034 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109646 2.c 2:300500bb:::benchmark_data_smithi090_78160_object109645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:12:24.034 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.337707+0000", 2024-05-15T23:12:24.035 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 330.686250745, 2024-05-15T23:12:24.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128404672, 2024-05-15T23:12:24.035 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.035 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.035 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.035 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.035 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:12:24.035 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:12:24.035 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109646 2024-05-15T23:12:24.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.035 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337707+0000", 2024-05-15T23:12:24.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337707+0000", 2024-05-15T23:12:24.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337708+0000", 2024-05-15T23:12:24.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.511e-06 2024-05-15T23:12:24.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337728+0000", 2024-05-15T23:12:24.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0055999999999999e-05 2024-05-15T23:12:24.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337739+0000", 2024-05-15T23:12:24.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0471e-05 2024-05-15T23:12:24.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337746+0000", 2024-05-15T23:12:24.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4059999999999999e-06 2024-05-15T23:12:24.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.340943+0000", 2024-05-15T23:12:24.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031963859999999998 2024-05-15T23:12:24.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.341055+0000", 2024-05-15T23:12:24.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011188600000000001 2024-05-15T23:12:24.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:12:24.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.341175+0000", 2024-05-15T23:12:24.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120839 2024-05-15T23:12:24.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.343368+0000", 2024-05-15T23:12:24.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021920669999999998 2024-05-15T23:12:24.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466023+0000", 2024-05-15T23:12:24.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.122655237 2024-05-15T23:12:24.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466075+0000", 2024-05-15T23:12:24.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1910000000000003e-05 2024-05-15T23:12:24.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466112+0000", 2024-05-15T23:12:24.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6903000000000002e-05 2024-05-15T23:12:24.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.040 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.040 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.24745.0:109643 2.c e127/102 ondisk, result = 0)", 2024-05-15T23:12:24.040 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.351843+0000", 2024-05-15T23:12:24.040 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 330.67211453499999, 2024-05-15T23:12:24.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.114139029, 2024-05-15T23:12:24.040 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.040 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.041 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:12:24.041 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351843+0000", 2024-05-15T23:12:24.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351843+0000", 2024-05-15T23:12:24.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351843+0000", 2024-05-15T23:12:24.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9000000000000002e-07 2024-05-15T23:12:24.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351846+0000", 2024-05-15T23:12:24.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1179999999999999e-06 2024-05-15T23:12:24.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351849+0000", 2024-05-15T23:12:24.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8510000000000001e-06 2024-05-15T23:12:24.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351853+0000", 2024-05-15T23:12:24.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3890000000000001e-06 2024-05-15T23:12:24.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351950+0000", 2024-05-15T23:12:24.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7557000000000001e-05 2024-05-15T23:12:24.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351958+0000", 2024-05-15T23:12:24.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5310000000000004e-06 2024-05-15T23:12:24.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465982+0000", 2024-05-15T23:12:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.114024193 2024-05-15T23:12:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.24745.0:109646 2.c e127/102 ondisk, result = 0)", 2024-05-15T23:12:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.351916+0000", 2024-05-15T23:12:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 330.67204213999997, 2024-05-15T23:12:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.114202603, 2024-05-15T23:12:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:12:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351916+0000", 2024-05-15T23:12:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351916+0000", 2024-05-15T23:12:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351917+0000", 2024-05-15T23:12:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2759999999999999e-06 2024-05-15T23:12:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351921+0000", 2024-05-15T23:12:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1559999999999997e-06 2024-05-15T23:12:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351929+0000", 2024-05-15T23:12:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7500000000000003e-06 2024-05-15T23:12:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351934+0000", 2024-05-15T23:12:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.079e-06 2024-05-15T23:12:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466009+0000", 2024-05-15T23:12:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11407569300000001 2024-05-15T23:12:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466017+0000", 2024-05-15T23:12:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6350000000000006e-06 2024-05-15T23:12:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466118+0000", 2024-05-15T23:12:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010101400000000001 2024-05-15T23:12:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109656 2.a 2:509d81bb:::benchmark_data_smithi090_78160_object109655:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:12:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.354844+0000", 2024-05-15T23:12:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 330.66911347000001, 2024-05-15T23:12:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.119458562, 2024-05-15T23:12:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:12:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:12:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109656 2024-05-15T23:12:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.354844+0000", 2024-05-15T23:12:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.354844+0000", 2024-05-15T23:12:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.354848+0000", 2024-05-15T23:12:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3960000000000002e-06 2024-05-15T23:12:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.354888+0000", 2024-05-15T23:12:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0673999999999997e-05 2024-05-15T23:12:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466170+0000", 2024-05-15T23:12:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11128214 2024-05-15T23:12:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466180+0000", 2024-05-15T23:12:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6779999999999995e-06 2024-05-15T23:12:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466282+0000", 2024-05-15T23:12:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101661 2024-05-15T23:12:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466411+0000", 2024-05-15T23:12:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012923899999999999 2024-05-15T23:12:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:12:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466561+0000", 2024-05-15T23:12:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014973300000000001 2024-05-15T23:12:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.470610+0000", 2024-05-15T23:12:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040489970000000004 2024-05-15T23:12:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.474147+0000", 2024-05-15T23:12:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035371539999999998 2024-05-15T23:12:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.474210+0000", 2024-05-15T23:12:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3291000000000001e-05 2024-05-15T23:12:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.474303+0000", 2024-05-15T23:12:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2598999999999999e-05 2024-05-15T23:12:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:141617 3.e 3:72e350be:::benchmark_data_smithi090_90974_object141616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e238)", 2024-05-15T23:12:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:00.137574+0000", 2024-05-15T23:12:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.886384001, 2024-05-15T23:12:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.109237059, 2024-05-15T23:12:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 141617 2024-05-15T23:12:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.137574+0000", 2024-05-15T23:12:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.137574+0000", 2024-05-15T23:12:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.137576+0000", 2024-05-15T23:12:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4899999999999999e-06 2024-05-15T23:12:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.137605+0000", 2024-05-15T23:12:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8725e-05 2024-05-15T23:12:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.137622+0000", 2024-05-15T23:12:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7164999999999999e-05 2024-05-15T23:12:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.137631+0000", 2024-05-15T23:12:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2359999999999993e-06 2024-05-15T23:12:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.145013+0000", 2024-05-15T23:12:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073814149999999997 2024-05-15T23:12:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.145167+0000", 2024-05-15T23:12:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015469799999999999 2024-05-15T23:12:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.145342+0000", 2024-05-15T23:12:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017493000000000001 2024-05-15T23:12:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.149726+0000", 2024-05-15T23:12:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043835979999999998 2024-05-15T23:12:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.246689+0000", 2024-05-15T23:12:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.096963465999999998 2024-05-15T23:12:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.246748+0000", 2024-05-15T23:12:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8718999999999998e-05 2024-05-15T23:12:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.246811+0000", 2024-05-15T23:12:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2617e-05 2024-05-15T23:12:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:152314 3.e 3:7ab6d39c:::benchmark_data_smithi090_90974_object152313:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e242)", 2024-05-15T23:12:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:11.011469+0000", 2024-05-15T23:12:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.012488255999999, 2024-05-15T23:12:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11541359299999999, 2024-05-15T23:12:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 152314 2024-05-15T23:12:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011469+0000", 2024-05-15T23:12:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011469+0000", 2024-05-15T23:12:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011473+0000", 2024-05-15T23:12:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5880000000000002e-06 2024-05-15T23:12:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011500+0000", 2024-05-15T23:12:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6791000000000001e-05 2024-05-15T23:12:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011514+0000", 2024-05-15T23:12:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4518999999999999e-05 2024-05-15T23:12:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011524+0000", 2024-05-15T23:12:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0033e-05 2024-05-15T23:12:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017018+0000", 2024-05-15T23:12:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00549396 2024-05-15T23:12:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017157+0000", 2024-05-15T23:12:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139202 2024-05-15T23:12:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017343+0000", 2024-05-15T23:12:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018567099999999999 2024-05-15T23:12:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.026368+0000", 2024-05-15T23:12:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090247439999999995 2024-05-15T23:12:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.126798+0000", 2024-05-15T23:12:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10043023299999999 2024-05-15T23:12:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.126830+0000", 2024-05-15T23:12:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1899000000000001e-05 2024-05-15T23:12:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.126883+0000", 2024-05-15T23:12:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2952999999999999e-05 2024-05-15T23:12:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:152320 3.e 3:77b14055:::benchmark_data_smithi090_90974_object152319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e242)", 2024-05-15T23:12:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:11.017288+0000", 2024-05-15T23:12:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.006669939, 2024-05-15T23:12:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.109741271, 2024-05-15T23:12:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 152320 2024-05-15T23:12:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017288+0000", 2024-05-15T23:12:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017288+0000", 2024-05-15T23:12:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017290+0000", 2024-05-15T23:12:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.503e-06 2024-05-15T23:12:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017313+0000", 2024-05-15T23:12:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2501e-05 2024-05-15T23:12:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017322+0000", 2024-05-15T23:12:24.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.132e-06 2024-05-15T23:12:24.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017329+0000", 2024-05-15T23:12:24.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2320000000000004e-06 2024-05-15T23:12:24.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.022161+0000", 2024-05-15T23:12:24.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004831531 2024-05-15T23:12:24.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.022299+0000", 2024-05-15T23:12:24.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138214 2024-05-15T23:12:24.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:24.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.022423+0000", 2024-05-15T23:12:24.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012423800000000001 2024-05-15T23:12:24.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.126938+0000", 2024-05-15T23:12:24.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104514871 2024-05-15T23:12:24.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.126951+0000", 2024-05-15T23:12:24.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2934e-05 2024-05-15T23:12:24.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.126984+0000", 2024-05-15T23:12:24.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2982999999999999e-05 2024-05-15T23:12:24.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.127029+0000", 2024-05-15T23:12:24.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5132000000000001e-05 2024-05-15T23:12:24.073 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.073 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.073 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.073 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:153702 3.e 3:7dfbffe1:::benchmark_data_smithi090_90974_object153701:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:12:24.073 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:12.533977+0000", 2024-05-15T23:12:24.073 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.489980486, 2024-05-15T23:12:24.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.111764557, 2024-05-15T23:12:24.073 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.073 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 153702 2024-05-15T23:12:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.533977+0000", 2024-05-15T23:12:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.533977+0000", 2024-05-15T23:12:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.533979+0000", 2024-05-15T23:12:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0999999999999998e-06 2024-05-15T23:12:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.534003+0000", 2024-05-15T23:12:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3706999999999999e-05 2024-05-15T23:12:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.534014+0000", 2024-05-15T23:12:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0951000000000001e-05 2024-05-15T23:12:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.534022+0000", 2024-05-15T23:12:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4579999999999993e-06 2024-05-15T23:12:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.545636+0000", 2024-05-15T23:12:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011613573 2024-05-15T23:12:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.545735+0000", 2024-05-15T23:12:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9412000000000003e-05 2024-05-15T23:12:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.545840+0000", 2024-05-15T23:12:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010463099999999999 2024-05-15T23:12:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.550491+0000", 2024-05-15T23:12:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046514740000000001 2024-05-15T23:12:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.645639+0000", 2024-05-15T23:12:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095147332000000001 2024-05-15T23:12:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.645683+0000", 2024-05-15T23:12:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4459000000000002e-05 2024-05-15T23:12:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.645742+0000", 2024-05-15T23:12:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8459999999999999e-05 2024-05-15T23:12:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:154414 3.e 3:7f410bb8:::benchmark_data_smithi090_90974_object154413:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:12:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:13.323080+0000", 2024-05-15T23:12:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.700877288999999, 2024-05-15T23:12:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.116641645, 2024-05-15T23:12:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 154414 2024-05-15T23:12:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323080+0000", 2024-05-15T23:12:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323080+0000", 2024-05-15T23:12:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323082+0000", 2024-05-15T23:12:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5409999999999999e-06 2024-05-15T23:12:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323102+0000", 2024-05-15T23:12:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0557999999999999e-05 2024-05-15T23:12:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323111+0000", 2024-05-15T23:12:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9320000000000002e-06 2024-05-15T23:12:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323118+0000", 2024-05-15T23:12:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7209999999999997e-06 2024-05-15T23:12:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.343054+0000", 2024-05-15T23:12:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019935797000000002 2024-05-15T23:12:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.343173+0000", 2024-05-15T23:12:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118989 2024-05-15T23:12:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.343298+0000", 2024-05-15T23:12:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012522999999999999 2024-05-15T23:12:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.351419+0000", 2024-05-15T23:12:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0081205270000000006 2024-05-15T23:12:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.439647+0000", 2024-05-15T23:12:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.088228506999999998 2024-05-15T23:12:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.439683+0000", 2024-05-15T23:12:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5723999999999999e-05 2024-05-15T23:12:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.439722+0000", 2024-05-15T23:12:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9119000000000003e-05 2024-05-15T23:12:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158411 3.e 3:721f625f:::benchmark_data_smithi090_90974_object158410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:12:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.688475+0000", 2024-05-15T23:12:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.335482152, 2024-05-15T23:12:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11329204800000001, 2024-05-15T23:12:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158411 2024-05-15T23:12:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688475+0000", 2024-05-15T23:12:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688475+0000", 2024-05-15T23:12:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688478+0000", 2024-05-15T23:12:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6199999999999999e-06 2024-05-15T23:12:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688507+0000", 2024-05-15T23:12:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8671e-05 2024-05-15T23:12:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688524+0000", 2024-05-15T23:12:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6837000000000002e-05 2024-05-15T23:12:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688534+0000", 2024-05-15T23:12:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0513000000000001e-05 2024-05-15T23:12:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701293+0000", 2024-05-15T23:12:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012759111 2024-05-15T23:12:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701443+0000", 2024-05-15T23:12:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015017200000000001 2024-05-15T23:12:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701612+0000", 2024-05-15T23:12:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016817399999999999 2024-05-15T23:12:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.801638+0000", 2024-05-15T23:12:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100025981 2024-05-15T23:12:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.801691+0000", 2024-05-15T23:12:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3124999999999997e-05 2024-05-15T23:12:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.801724+0000", 2024-05-15T23:12:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3039e-05 2024-05-15T23:12:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.801768+0000", 2024-05-15T23:12:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3804999999999998e-05 2024-05-15T23:12:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158423 3.e 3:7aa2cb14:::benchmark_data_smithi090_90974_object158422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:12:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.701674+0000", 2024-05-15T23:12:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.3222837160000003, 2024-05-15T23:12:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.115640091, 2024-05-15T23:12:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158423 2024-05-15T23:12:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701674+0000", 2024-05-15T23:12:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701674+0000", 2024-05-15T23:12:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701675+0000", 2024-05-15T23:12:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.424e-06 2024-05-15T23:12:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701694+0000", 2024-05-15T23:12:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8777e-05 2024-05-15T23:12:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701703+0000", 2024-05-15T23:12:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1090000000000004e-06 2024-05-15T23:12:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701710+0000", 2024-05-15T23:12:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0650000000000001e-06 2024-05-15T23:12:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.803831+0000", 2024-05-15T23:12:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102121131 2024-05-15T23:12:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.803962+0000", 2024-05-15T23:12:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131042 2024-05-15T23:12:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.804116+0000", 2024-05-15T23:12:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015323299999999999 2024-05-15T23:12:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.817218+0000", 2024-05-15T23:12:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013102213 2024-05-15T23:12:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.817227+0000", 2024-05-15T23:12:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2949999999999999e-06 2024-05-15T23:12:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.817263+0000", 2024-05-15T23:12:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6003000000000001e-05 2024-05-15T23:12:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.817314+0000", 2024-05-15T23:12:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0798999999999999e-05 2024-05-15T23:12:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158426 3.e 3:7524570c:::benchmark_data_smithi090_90974_object158425:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:12:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.704015+0000", 2024-05-15T23:12:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.3199422580000002, 2024-05-15T23:12:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.118608484, 2024-05-15T23:12:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158426 2024-05-15T23:12:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704015+0000", 2024-05-15T23:12:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704015+0000", 2024-05-15T23:12:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704017+0000", 2024-05-15T23:12:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7910000000000001e-06 2024-05-15T23:12:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704039+0000", 2024-05-15T23:12:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2243e-05 2024-05-15T23:12:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704053+0000", 2024-05-15T23:12:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3169e-05 2024-05-15T23:12:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704060+0000", 2024-05-15T23:12:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6799999999999993e-06 2024-05-15T23:12:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.808093+0000", 2024-05-15T23:12:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10403225000000001 2024-05-15T23:12:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.808225+0000", 2024-05-15T23:12:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013245100000000001 2024-05-15T23:12:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.808375+0000", 2024-05-15T23:12:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014989300000000001 2024-05-15T23:12:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.817357+0000", 2024-05-15T23:12:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0089821039999999994 2024-05-15T23:12:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.822508+0000", 2024-05-15T23:12:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005150951 2024-05-15T23:12:24.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.822560+0000", 2024-05-15T23:12:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2488999999999998e-05 2024-05-15T23:12:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.822624+0000", 2024-05-15T23:12:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3462999999999999e-05 2024-05-15T23:12:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158429 3.e 3:740800f6:::benchmark_data_smithi090_90974_object158428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:12:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.708210+0000", 2024-05-15T23:12:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.3157478920000001, 2024-05-15T23:12:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11887426500000001, 2024-05-15T23:12:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158429 2024-05-15T23:12:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708210+0000", 2024-05-15T23:12:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708210+0000", 2024-05-15T23:12:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708212+0000", 2024-05-15T23:12:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4430000000000002e-06 2024-05-15T23:12:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708234+0000", 2024-05-15T23:12:24.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2195e-05 2024-05-15T23:12:24.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708245+0000", 2024-05-15T23:12:24.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1012999999999999e-05 2024-05-15T23:12:24.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708253+0000", 2024-05-15T23:12:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8879999999999997e-06 2024-05-15T23:12:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.813214+0000", 2024-05-15T23:12:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104960421 2024-05-15T23:12:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.813362+0000", 2024-05-15T23:12:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014787399999999999 2024-05-15T23:12:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.813490+0000", 2024-05-15T23:12:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012885199999999999 2024-05-15T23:12:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.822633+0000", 2024-05-15T23:12:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091427599999999998 2024-05-15T23:12:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.826999+0000", 2024-05-15T23:12:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00436572 2024-05-15T23:12:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.827035+0000", 2024-05-15T23:12:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6282000000000003e-05 2024-05-15T23:12:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.827084+0000", 2024-05-15T23:12:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8816999999999997e-05 2024-05-15T23:12:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:159320 3.e 3:722627b8:::benchmark_data_smithi090_90974_object159319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e244)", 2024-05-15T23:12:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:18.781348+0000", 2024-05-15T23:12:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.2426093910000002, 2024-05-15T23:12:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13170827900000001, 2024-05-15T23:12:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159320 2024-05-15T23:12:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781348+0000", 2024-05-15T23:12:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781348+0000", 2024-05-15T23:12:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781351+0000", 2024-05-15T23:12:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3410000000000001e-06 2024-05-15T23:12:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781377+0000", 2024-05-15T23:12:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6271e-05 2024-05-15T23:12:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781388+0000", 2024-05-15T23:12:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1141999999999999e-05 2024-05-15T23:12:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781396+0000", 2024-05-15T23:12:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2679999999999994e-06 2024-05-15T23:12:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.797611+0000", 2024-05-15T23:12:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016214605 2024-05-15T23:12:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.797693+0000", 2024-05-15T23:12:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2564000000000004e-05 2024-05-15T23:12:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.797790+0000", 2024-05-15T23:12:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6477000000000005e-05 2024-05-15T23:12:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.895923+0000", 2024-05-15T23:12:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.098133348999999995 2024-05-15T23:12:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.912951+0000", 2024-05-15T23:12:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017028192000000001 2024-05-15T23:12:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.912989+0000", 2024-05-15T23:12:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7867000000000002e-05 2024-05-15T23:12:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.913057+0000", 2024-05-15T23:12:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7203e-05 2024-05-15T23:12:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:159325 3.e 3:77a43cc1:::benchmark_data_smithi090_90974_object159324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e244)", 2024-05-15T23:12:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:18.785096+0000", 2024-05-15T23:12:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.2388617660000003, 2024-05-15T23:12:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12806985500000001, 2024-05-15T23:12:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159325 2024-05-15T23:12:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785096+0000", 2024-05-15T23:12:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785096+0000", 2024-05-15T23:12:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785098+0000", 2024-05-15T23:12:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.571e-06 2024-05-15T23:12:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785122+0000", 2024-05-15T23:12:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3864999999999999e-05 2024-05-15T23:12:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785135+0000", 2024-05-15T23:12:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3061000000000001e-05 2024-05-15T23:12:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785144+0000", 2024-05-15T23:12:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6589999999999996e-06 2024-05-15T23:12:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.801926+0000", 2024-05-15T23:12:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016782254 2024-05-15T23:12:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.802075+0000", 2024-05-15T23:12:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014822300000000001 2024-05-15T23:12:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.802255+0000", 2024-05-15T23:12:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018082999999999999 2024-05-15T23:12:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.895946+0000", 2024-05-15T23:12:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.093690839999999997 2024-05-15T23:12:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.913068+0000", 2024-05-15T23:12:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01712139 2024-05-15T23:12:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.913114+0000", 2024-05-15T23:12:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6802999999999999e-05 2024-05-15T23:12:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.913166+0000", 2024-05-15T23:12:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1359000000000001e-05 2024-05-15T23:12:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:159351 3.e 3:77148def:::benchmark_data_smithi090_90974_object159350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e244)", 2024-05-15T23:12:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:18.811652+0000", 2024-05-15T23:12:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.2123061039999996, 2024-05-15T23:12:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.112410336, 2024-05-15T23:12:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159351 2024-05-15T23:12:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.811652+0000", 2024-05-15T23:12:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.811652+0000", 2024-05-15T23:12:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.811652+0000", 2024-05-15T23:12:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1999999999999998e-07 2024-05-15T23:12:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.811669+0000", 2024-05-15T23:12:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6860000000000001e-05 2024-05-15T23:12:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.811675+0000", 2024-05-15T23:12:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5729999999999998e-06 2024-05-15T23:12:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.811681+0000", 2024-05-15T23:12:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9839999999999998e-06 2024-05-15T23:12:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.900818+0000", 2024-05-15T23:12:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089137076999999995 2024-05-15T23:12:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.900969+0000", 2024-05-15T23:12:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150679 2024-05-15T23:12:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.901104+0000", 2024-05-15T23:12:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013527499999999999 2024-05-15T23:12:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.918628+0000", 2024-05-15T23:12:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017524217000000002 2024-05-15T23:12:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.923944+0000", 2024-05-15T23:12:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053155939999999999 2024-05-15T23:12:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.924001+0000", 2024-05-15T23:12:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7816999999999999e-05 2024-05-15T23:12:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.924062+0000", 2024-05-15T23:12:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0439999999999997e-05 2024-05-15T23:12:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:160540 3.e 3:74ab87c2:::benchmark_data_smithi090_90974_object160539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-05-15T23:12:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:20.108799+0000", 2024-05-15T23:12:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9151581769999999, 2024-05-15T23:12:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105677878, 2024-05-15T23:12:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160540 2024-05-15T23:12:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.108799+0000", 2024-05-15T23:12:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.108799+0000", 2024-05-15T23:12:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.108801+0000", 2024-05-15T23:12:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0720000000000001e-06 2024-05-15T23:12:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.108822+0000", 2024-05-15T23:12:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1039e-05 2024-05-15T23:12:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.108831+0000", 2024-05-15T23:12:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6830000000000003e-06 2024-05-15T23:12:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.108838+0000", 2024-05-15T23:12:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4760000000000003e-06 2024-05-15T23:12:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.113383+0000", 2024-05-15T23:12:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045452019999999999 2024-05-15T23:12:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.113525+0000", 2024-05-15T23:12:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000141896 2024-05-15T23:12:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.113635+0000", 2024-05-15T23:12:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010969299999999999 2024-05-15T23:12:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.204569+0000", 2024-05-15T23:12:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090934849999999998 2024-05-15T23:12:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.214363+0000", 2024-05-15T23:12:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0097934130000000008 2024-05-15T23:12:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.214416+0000", 2024-05-15T23:12:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2989999999999999e-05 2024-05-15T23:12:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.214477+0000", 2024-05-15T23:12:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1563999999999995e-05 2024-05-15T23:12:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:160906 3.e 3:79e6de76:::benchmark_data_smithi090_90974_object160905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-05-15T23:12:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:20.576551+0000", 2024-05-15T23:12:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.4474067719999999, 2024-05-15T23:12:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.106230501, 2024-05-15T23:12:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160906 2024-05-15T23:12:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.576551+0000", 2024-05-15T23:12:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.576551+0000", 2024-05-15T23:12:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.576552+0000", 2024-05-15T23:12:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3200000000000003e-07 2024-05-15T23:12:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.576572+0000", 2024-05-15T23:12:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0543999999999998e-05 2024-05-15T23:12:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.576579+0000", 2024-05-15T23:12:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6730000000000003e-06 2024-05-15T23:12:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.576585+0000", 2024-05-15T23:12:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8510000000000004e-06 2024-05-15T23:12:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.583924+0000", 2024-05-15T23:12:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073390649999999997 2024-05-15T23:12:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.584079+0000", 2024-05-15T23:12:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000154797 2024-05-15T23:12:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.584260+0000", 2024-05-15T23:12:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018160499999999999 2024-05-15T23:12:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.592119+0000", 2024-05-15T23:12:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.007859029 2024-05-15T23:12:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.682657+0000", 2024-05-15T23:12:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090537754999999998 2024-05-15T23:12:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.682720+0000", 2024-05-15T23:12:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2472999999999996e-05 2024-05-15T23:12:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.682781+0000", 2024-05-15T23:12:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1777000000000002e-05 2024-05-15T23:12:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:161690 3.e 3:78fe5800:::benchmark_data_smithi090_90974_object161689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e246)", 2024-05-15T23:12:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:21.553373+0000", 2024-05-15T23:12:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.4705842599999999, 2024-05-15T23:12:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.107890657, 2024-05-15T23:12:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:12:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:12:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:12:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:12:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:12:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:12:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 161690 2024-05-15T23:12:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:12:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:12:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.553373+0000", 2024-05-15T23:12:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:12:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.553373+0000", 2024-05-15T23:12:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:12:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:12:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.553377+0000", 2024-05-15T23:12:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4170000000000001e-06 2024-05-15T23:12:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:12:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.553407+0000", 2024-05-15T23:12:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0602000000000003e-05 2024-05-15T23:12:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:12:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.553423+0000", 2024-05-15T23:12:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5721000000000001e-05 2024-05-15T23:12:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:12:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.553435+0000", 2024-05-15T23:12:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1758e-05 2024-05-15T23:12:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:12:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.556608+0000", 2024-05-15T23:12:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031734990000000002 2024-05-15T23:12:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:12:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.556713+0000", 2024-05-15T23:12:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010498299999999999 2024-05-15T23:12:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:12:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.556821+0000", 2024-05-15T23:12:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107743 2024-05-15T23:12:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:12:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.565377+0000", 2024-05-15T23:12:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085556590000000002 2024-05-15T23:12:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:12:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.661155+0000", 2024-05-15T23:12:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095778314000000003 2024-05-15T23:12:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:12:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.661189+0000", 2024-05-15T23:12:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3883000000000001e-05 2024-05-15T23:12:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:12:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:12:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:12:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.661264+0000", 2024-05-15T23:12:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5078000000000002e-05 2024-05-15T23:12:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:12:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:12:24.145 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:12:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:24.146 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:23 smithi090 ceph-mon[44434]: pgmap v679: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 886 op/s 2024-05-15T23:12:24.147 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:12:24.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:12:24.147 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:23 smithi084 ceph-mon[39784]: pgmap v679: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 886 op/s 2024-05-15T23:12:24.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:12:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:24.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:23 smithi088 ceph-mon[44699]: pgmap v679: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 886 op/s 2024-05-15T23:12:24.762 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:12:24.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:24.765 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:12:24.773 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T23:12:24.866 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:12:24.901 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:24.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:12:24.926 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:12:24.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:24.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:12:24.936 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total time run: 90.0159 2024-05-15T23:12:24.937 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total writes made: 165540 2024-05-15T23:12:24.937 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Write size: 65536 2024-05-15T23:12:24.937 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object size: 65536 2024-05-15T23:12:24.937 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Bandwidth (MB/sec): 114.938 2024-05-15T23:12:24.937 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Bandwidth: 66.0207 2024-05-15T23:12:24.937 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max bandwidth (MB/sec): 383.938 2024-05-15T23:12:24.937 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min bandwidth (MB/sec): 48.5625 2024-05-15T23:12:24.937 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average IOPS: 1839 2024-05-15T23:12:24.937 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev IOPS: 1056.33 2024-05-15T23:12:24.937 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max IOPS: 6143 2024-05-15T23:12:24.937 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min IOPS: 777 2024-05-15T23:12:24.937 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average Latency(s): 0.00869971 2024-05-15T23:12:24.938 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Latency(s): 0.00974329 2024-05-15T23:12:24.938 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max latency(s): 0.219607 2024-05-15T23:12:24.938 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min latency(s): 0.000948627 2024-05-15T23:12:24.944 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T23:12:24.950 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:12:24.985 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:12:25.022 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:12:25.030 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:12:25.071 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:12:25.119 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:25.120 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:12:25.147 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:12:25.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:25.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:12:25.155 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:12:25.192 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:12:25.233 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:12:25.252 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:12:25.263 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:12:25.270 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:25.285 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:25.288 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:12:25.311 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:12:25.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:25.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:12:25.326 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:12:25.335 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:12:25.336 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:12:25.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:12:25.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:12:25.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:12:25.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:12:25.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:12:25.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:12:25.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:12:25.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:12:25.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:12:25.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:12:25.414 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:12:25.452 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:25.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:12:25.476 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:12:25.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:25.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:12:25.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:12:25.696 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:25.864 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:12:25.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:25.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:12:25.919 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.10 to 1 (10000) 2024-05-15T23:12:25.972 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:12:26.013 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:26.038 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:12:26.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:26.045 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:12:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:25 smithi084 ceph-mon[39784]: pgmap v680: 17 pgs: 17 active+clean; 9.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s 2024-05-15T23:12:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:25 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3118588732' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:12:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:25 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3914689206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.146 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:12:26.199 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:26.237 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:12:26.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:26.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:12:26.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:25 smithi090 ceph-mon[44434]: pgmap v680: 17 pgs: 17 active+clean; 9.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s 2024-05-15T23:12:26.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:25 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3118588732' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:12:26.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:25 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3914689206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:26.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.310 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.310 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.310 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:25 smithi088 ceph-mon[44699]: pgmap v680: 17 pgs: 17 active+clean; 9.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s 2024-05-15T23:12:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:25 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3118588732' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:12:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:25 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3914689206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:26.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:12:26.378 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:26.405 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:12:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:26.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:12:26.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:12:26.580 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:27.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:27.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:27.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:27.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:26 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3118588732' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:12:27.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:26 smithi084 ceph-mon[39784]: osdmap e248: 12 total, 12 up, 12 in 2024-05-15T23:12:27.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:27.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:27.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:27.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:26 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3118588732' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:12:27.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:26 smithi090 ceph-mon[44434]: osdmap e248: 12 total, 12 up, 12 in 2024-05-15T23:12:27.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:27.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:27.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:27.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:26 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3118588732' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:12:27.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:26 smithi088 ceph-mon[44699]: osdmap e248: 12 total, 12 up, 12 in 2024-05-15T23:12:27.543 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:12:27.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:27.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:12:27.651 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:12:27.693 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:28.090 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:12:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:28.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:12:28.116 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-05-15T23:12:28.116 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-05-15T23:12:28.116 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-15T23:12:28.116 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump --format=json 2024-05-15T23:12:28.190 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:27 smithi084 ceph-mon[39784]: pgmap v682: 17 pgs: 17 active+clean; 10 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 63 MiB/s wr, 1.08k op/s 2024-05-15T23:12:28.198 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:12:28.258 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:28.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:27 smithi090 ceph-mon[44434]: pgmap v682: 17 pgs: 17 active+clean; 10 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 63 MiB/s wr, 1.08k op/s 2024-05-15T23:12:28.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:27 smithi088 ceph-mon[44699]: pgmap v682: 17 pgs: 17 active+clean; 10 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 63 MiB/s wr, 1.08k op/s 2024-05-15T23:12:28.366 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:12:28.367 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:28.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:28.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:12:28.472 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:12:28.509 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:28.726 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:12:28.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:28.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:12:28.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:12:28.831 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:12:28.858 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:28.887 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:12:28.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:28.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:12:28.992 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:12:29.019 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:29.041 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:12:29.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:29.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:12:29.144 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:12:29.146 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:12:29.146 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:12:29.174 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:29.204 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:12:29.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:29.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:12:29.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:12:29.350 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:29.379 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:12:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:29.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:12:29.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:12:29.526 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:29.554 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:12:29.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:29.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:12:29.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:12:29.704 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:29.731 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:12:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:29.737 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:12:29.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:12:29.869 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:29.897 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:12:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:29.900 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:12:30.001 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:12:30.034 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:29 smithi090 ceph-mon[44434]: pgmap v683: 17 pgs: 17 active+clean; 10 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 128 B/s rd, 53 MiB/s wr, 1.00k op/s 2024-05-15T23:12:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:12:30.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:29 smithi084 ceph-mon[39784]: pgmap v683: 17 pgs: 17 active+clean; 10 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 128 B/s rd, 53 MiB/s wr, 1.00k op/s 2024-05-15T23:12:30.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:12:30.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:29 smithi088 ceph-mon[44699]: pgmap v683: 17 pgs: 17 active+clean; 10 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 128 B/s rd, 53 MiB/s wr, 1.00k op/s 2024-05-15T23:12:30.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:29 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:12:30.559 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T23:12:30.838 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:30.850 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:12:31.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:31.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:12:32.006 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:31 smithi084 ceph-mon[39784]: pgmap v684: 17 pgs: 17 active+clean; 9.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 114 B/s rd, 38 MiB/s wr, 1.03k op/s 2024-05-15T23:12:32.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:12:32.154 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:32.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:31 smithi088 ceph-mon[44699]: pgmap v684: 17 pgs: 17 active+clean; 9.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 114 B/s rd, 38 MiB/s wr, 1.03k op/s 2024-05-15T23:12:32.289 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:12:32.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:32.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:12:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:31 smithi090 ceph-mon[44434]: pgmap v684: 17 pgs: 17 active+clean; 9.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 114 B/s rd, 38 MiB/s wr, 1.03k op/s 2024-05-15T23:12:32.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:12:32.441 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:32.666 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:12:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:32.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:12:32.770 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:12:32.815 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:32.845 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:12:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:32.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:12:32.955 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:12:32.969 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:12:32.999 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:33.024 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:12:33.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:33.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:12:33.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3839941016' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:12:33.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:32 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:12:33.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:32 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:12:33.132 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:12:33.169 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:33.199 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:12:33.205 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3839941016' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:12:33.206 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:32 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:12:33.206 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:32 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:12:33.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:33.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:12:33.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3839941016' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:12:33.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:32 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:12:33.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:32 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:12:33.308 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:12:33.350 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:33.388 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:12:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:33.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:12:33.494 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:12:33.541 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:33.576 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:12:33.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:33.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:12:33.714 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:12:33.753 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:33.912 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:12:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:33.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:12:34.022 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:12:34.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:33 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:12:34.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:33 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:12:34.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:33 smithi084 ceph-mon[39784]: pgmap v685: 17 pgs: 17 active+clean; 9.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 27 MiB/s wr, 1.15k op/s 2024-05-15T23:12:34.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:33 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:12:34.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:33 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3839941016' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:12:34.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:33 smithi084 ceph-mon[39784]: osdmap e249: 12 total, 12 up, 12 in 2024-05-15T23:12:34.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:33 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:12:34.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:33 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:12:34.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:12:34.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:12:34.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:33 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:12:34.084 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:33 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:12:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:33 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:12:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:33 smithi090 ceph-mon[44434]: pgmap v685: 17 pgs: 17 active+clean; 9.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 27 MiB/s wr, 1.15k op/s 2024-05-15T23:12:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:33 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:12:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:33 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3839941016' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:12:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:33 smithi090 ceph-mon[44434]: osdmap e249: 12 total, 12 up, 12 in 2024-05-15T23:12:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:33 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:12:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:33 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:12:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:12:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:12:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:33 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:12:34.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:33 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:12:34.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:33 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:12:34.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:33 smithi088 ceph-mon[44699]: pgmap v685: 17 pgs: 17 active+clean; 9.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 27 MiB/s wr, 1.15k op/s 2024-05-15T23:12:34.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:33 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:12:34.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:33 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3839941016' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:12:34.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:33 smithi088 ceph-mon[44699]: osdmap e249: 12 total, 12 up, 12 in 2024-05-15T23:12:34.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:33 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:12:34.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:33 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:12:34.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:12:34.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:12:34.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:33 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:12:34.945 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:12:34.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:34.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:12:35.049 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:12:35.090 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:35.112 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:12:35.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:35.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:12:35.204 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:35.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:12:35.246 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:35.276 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:12:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:35.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:12:35.384 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:12:35.445 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:35.559 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:12:35.560 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":250,"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","created":"2024-05-15T22:56:41.212493+0000","modified":"2024-05-15T23:12:34.937806+0000","last_up_change":"2024-05-15T23:03:56.576073+0000","last_in_change":"2024-05-15T23:10:24.109619+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T22:59:38.603575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.96207165718078613,"average_primary_affinity":0.95999997854232788,"average_primary_affinity_weighted":0.95999997854232788}},{"pool":3,"pool_name":"unique_pool_1","create_time":"2024-05-15T23:10:39.841228+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.2699999809265137,"score_stable":2.2699999809265137,"optimal_score":0.82999998331069946,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.96207165718078613,"average_primary_affinity":0.95999997854232788,"average_primary_affinity_weighted":0.95999997854232788}}],"osds":[{"osd":0,"uuid":"80159cc8-1b6c-4aa6-a114-966786d7be1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6801","nonce":1084680531}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6802","nonce":1084680531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6804","nonce":1084680531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6803","nonce":1084680531}]},"public_addr":"172.21.15.84:6801/1084680531","cluster_addr":"172.21.15.84:6802/1084680531","heartbeat_back_addr":"172.21.15.84:6804/1084680531","heartbeat_front_addr":"172.21.15.84:6803/1084680531","state":["exists","up"]},{"osd":1,"uuid":"9c2c76c9-ce0f-4c2d-9d96-c87a10187607","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6805","nonce":2140830229}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6806","nonce":2140830229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6808","nonce":2140830229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6807","nonce":2140830229}]},"public_addr":"172.21.15.84:6805/2140830229","cluster_addr":"172.21.15.84:6806/2140830229","heartbeat_back_addr":"172.21.15.84:6808/2140830229","heartbeat_front_addr":"172.21.15.84:6807/2140830229","state":["exists","up"]},{"osd":2,"uuid":"e8514daa-b5f9-4f2f-a432-53f4a4eb86bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6809","nonce":2699996290}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6810","nonce":2699996290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6812","nonce":2699996290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6811","nonce":2699996290}]},"public_addr":"172.21.15.84:6809/2699996290","cluster_addr":"172.21.15.84:6810/2699996290","heartbeat_back_addr":"172.21.15.84:6812/2699996290","heartbeat_front_addr":"172.21.15.84:6811/2699996290","state":["exists","up"]},{"osd":3,"uuid":"5d9965f9-c0b2-48ce-869c-582990985623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6813","nonce":2204039828}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6814","nonce":2204039828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6816","nonce":2204039828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6815","nonce":2204039828}]},"public_addr":"172.21.15.84:6813/2204039828","cluster_addr":"172.21.15.84:6814/2204039828","heartbeat_back_addr":"172.21.15.84:6816/2204039828","heartbeat_front_addr":"172.21.15.84:6815/2204039828","state":["exists","up"]},{"osd":4,"uuid":"4bc50523-8480-44d2-8d76-31479e839c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6800","nonce":2267955292}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":2267955292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":2267955292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":2267955292}]},"public_addr":"172.21.15.88:6800/2267955292","cluster_addr":"172.21.15.88:6801/2267955292","heartbeat_back_addr":"172.21.15.88:6803/2267955292","heartbeat_front_addr":"172.21.15.88:6802/2267955292","state":["exists","up"]},{"osd":5,"uuid":"d4db8d7f-afc7-43dc-81fe-f70c290d8b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":661041538}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":661041538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":661041538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":661041538}]},"public_addr":"172.21.15.88:6804/661041538","cluster_addr":"172.21.15.88:6805/661041538","heartbeat_back_addr":"172.21.15.88:6807/661041538","heartbeat_front_addr":"172.21.15.88:6806/661041538","state":["exists","up"]},{"osd":6,"uuid":"a2773454-2318-4a5b-8f31-0952b1b07f2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":380856869}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":380856869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":380856869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":380856869}]},"public_addr":"172.21.15.88:6808/380856869","cluster_addr":"172.21.15.88:6809/380856869","heartbeat_back_addr":"172.21.15.88:6811/380856869","heartbeat_front_addr":"172.21.15.88:6810/380856869","state":["exists","up"]},{"osd":7,"uuid":"028ca7b6-3e00-4466-a33f-b726cd60cb34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":1044709063}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":1044709063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":1044709063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":1044709063}]},"public_addr":"172.21.15.88:6812/1044709063","cluster_addr":"172.21.15.88:6813/1044709063","heartbeat_back_addr":"172.21.15.88:6815/1044709063","heartbeat_front_addr":"172.21.15.88:6814/1044709063","state":["exists","up"]},{"osd":8,"uuid":"02cab286-d487-4a66-982c-3cf28f55e851","up":1,"in":1,"weight":1,"primary_affinity":0.54486083984375,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6800","nonce":522459126}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6801","nonce":522459126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6803","nonce":522459126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6802","nonce":522459126}]},"public_addr":"172.21.15.90:6800/522459126","cluster_addr":"172.21.15.90:6801/522459126","heartbeat_back_addr":"172.21.15.90:6803/522459126","heartbeat_front_addr":"172.21.15.90:6802/522459126","state":["exists","up"]},{"osd":9,"uuid":"47d38bc7-39ba-4b35-af24-e282354dd613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6804","nonce":2794571133}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6805","nonce":2794571133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6807","nonce":2794571133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6806","nonce":2794571133}]},"public_addr":"172.21.15.90:6804/2794571133","cluster_addr":"172.21.15.90:6805/2794571133","heartbeat_back_addr":"172.21.15.90:6807/2794571133","heartbeat_front_addr":"172.21.15.90:6806/2794571133","state":["exists","up"]},{"osd":10,"uuid":"d7b2aeae-d012-41eb-9f91-e4c05c649473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6808","nonce":3735380744}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6809","nonce":3735380744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6811","nonce":3735380744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6810","nonce":3735380744}]},"public_addr":"172.21.15.90:6808/3735380744","cluster_addr":"172.21.15.90:6809/3735380744","heartbeat_back_addr":"172.21.15.90:6811/3735380744","heartbeat_front_addr":"172.21.15.90:6810/3735380744","state":["exists","up"]},{"osd":11,"uuid":"2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6812","nonce":2265615270}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6813","nonce":2265615270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6815","nonce":2265615270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6814","nonce":2265615270}]},"public_addr":"172.21.15.90:6812/2265615270","cluster_addr":"172.21.15.90:6813/2265615270","heartbeat_back_addr":"172.21.15.90:6815/2265615270","heartbeat_front_addr":"172.21.15.90:6814/2265615270","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:12:16.893801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:11:48.344656+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:11:37.349518+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:11:34.269957+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:12:32.736560+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:12:32.081381+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:11:32.990100+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:12:17.283556+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:12:20.626994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:12:34.624576+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:11:25.376274+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:11:25.900643+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/2581408231":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/1136984393":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/2443201864":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1252671888":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1451001371":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/4084814866":"2024-05-16T22:57:38.505263+0000","172.21.15.84:0/4076409285":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/2988977694":"2024-05-16T22:57:38.505263+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T23:12:35.610 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:12:35.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:35.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:12:35.718 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:12:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:35.789 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:12:35.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:35.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:12:35.893 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:12:35.923 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:35.945 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:12:35.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:35.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:12:35.959 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:12:36.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:12:36.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:35 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:12:36.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:35 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:12:36.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:35 smithi088 ceph-mon[44699]: pgmap v687: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s wr, 1.09k op/s 2024-05-15T23:12:36.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:35 smithi088 ceph-mon[44699]: osdmap e250: 12 total, 12 up, 12 in 2024-05-15T23:12:36.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:35 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3912224620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:12:36.076 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:12:36.079 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:36.102 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:12:36.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:36.105 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:12:36.115 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:12:36.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:35 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:12:36.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:35 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:12:36.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:35 smithi084 ceph-mon[39784]: pgmap v687: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s wr, 1.09k op/s 2024-05-15T23:12:36.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:35 smithi084 ceph-mon[39784]: osdmap e250: 12 total, 12 up, 12 in 2024-05-15T23:12:36.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:35 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3912224620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:12:36.147 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:12:36.196 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:12:36.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:12:36.226 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:12:36.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:35 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:12:36.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:35 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:12:36.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:35 smithi090 ceph-mon[44434]: pgmap v687: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s wr, 1.09k op/s 2024-05-15T23:12:36.238 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:35 smithi090 ceph-mon[44434]: osdmap e250: 12 total, 12 up, 12 in 2024-05-15T23:12:36.238 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:35 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3912224620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:12:36.239 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:36.259 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:12:36.264 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:12:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:36.267 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:12:36.303 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:12:36.345 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:12:36.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:12:36.374 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:12:36.408 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:12:36.411 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:36.437 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:12:36.456 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:12:36.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:36.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:12:36.475 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:12:36.486 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:12:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:12:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:12:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:12:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:12:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:12:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:12:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:12:36.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:12:36.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:12:36.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:12:36.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:12:36.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:12:36.584 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:36.615 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:12:36.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:36.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:12:36.722 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:12:36.752 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:36.778 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:12:36.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:36.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:12:36.883 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:12:36.922 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:37.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:36 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:12:37.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:36 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:12:37.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4288317814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:37.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:36 smithi088 ceph-mon[44699]: 3.9 scrub starts 2024-05-15T23:12:37.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:36 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:12:37.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:36 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:12:37.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4288317814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:37.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:36 smithi084 ceph-mon[39784]: 3.9 scrub starts 2024-05-15T23:12:37.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.313 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:36 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:12:37.313 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:36 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:12:37.313 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4288317814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:37.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:36 smithi090 ceph-mon[44434]: 3.9 scrub starts 2024-05-15T23:12:37.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.314 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:37.689 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:12:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:37.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:12:37.724 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-15T23:12:37.799 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:12:37.828 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:37.855 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:12:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:37.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:12:37.961 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:12:37.995 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:38.020 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:12:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:38.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:12:38.125 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:12:38.164 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:38.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:38.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:38.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:38.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:38.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:37 smithi090 ceph-mon[44434]: pgmap v689: 17 pgs: 17 active+clean; 9.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s wr, 1.31k op/s 2024-05-15T23:12:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:37 smithi090 ceph-mon[44434]: osdmap e251: 12 total, 12 up, 12 in 2024-05-15T23:12:38.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:37 smithi088 ceph-mon[44699]: pgmap v689: 17 pgs: 17 active+clean; 9.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s wr, 1.31k op/s 2024-05-15T23:12:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:37 smithi088 ceph-mon[44699]: osdmap e251: 12 total, 12 up, 12 in 2024-05-15T23:12:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:37 smithi084 ceph-mon[39784]: pgmap v689: 17 pgs: 17 active+clean; 9.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s wr, 1.31k op/s 2024-05-15T23:12:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:37 smithi084 ceph-mon[39784]: osdmap e251: 12 total, 12 up, 12 in 2024-05-15T23:12:38.829 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:12:38.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:38.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:12:38.938 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:12:38.981 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:12:39.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:39.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:12:39.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:12:39.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:38 smithi084 ceph-mon[39784]: 3.9 scrub ok 2024-05-15T23:12:39.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:39 smithi084 ceph-mon[39784]: 3.2 scrub starts 2024-05-15T23:12:39.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:39 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:12:39.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:39 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:12:39.152 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:39.179 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:12:39.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:39.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:12:39.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:12:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:39 smithi090 ceph-mon[44434]: 3.9 scrub ok 2024-05-15T23:12:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:39 smithi090 ceph-mon[44434]: 3.2 scrub starts 2024-05-15T23:12:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:39 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:12:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:39 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:12:39.304 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:39.320 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:39.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:39 smithi088 ceph-mon[44699]: 3.9 scrub ok 2024-05-15T23:12:39.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:39 smithi088 ceph-mon[44699]: 3.2 scrub starts 2024-05-15T23:12:39.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:39 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:12:39.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:39 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:12:39.347 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:12:39.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:39.350 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:12:39.451 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:12:39.481 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:39.501 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:12:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:39.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:12:39.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:12:39.635 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:39.656 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:12:39.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:39.659 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:12:39.760 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:12:39.796 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:39.828 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:12:39.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:39.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:12:39.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:12:39.968 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:12:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:40.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:12:40.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:40 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:12:40.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:40 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:12:40.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:40 smithi088 ceph-mon[44699]: pgmap v691: 17 pgs: 17 active+clean; 9.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T23:12:40.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:40 smithi088 ceph-mon[44699]: 3.f scrub starts 2024-05-15T23:12:40.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:40 smithi088 ceph-mon[44699]: 3.6 scrub starts 2024-05-15T23:12:40.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:40 smithi088 ceph-mon[44699]: osdmap e252: 12 total, 12 up, 12 in 2024-05-15T23:12:40.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:40 smithi088 ceph-mon[44699]: 3.d scrub starts 2024-05-15T23:12:40.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:12:40.102 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:12:40.102 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:12:40.103 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:12:40.103 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:12:40.103 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:12:40.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:40 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:12:40.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:40 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:12:40.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:40 smithi084 ceph-mon[39784]: pgmap v691: 17 pgs: 17 active+clean; 9.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T23:12:40.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:40 smithi084 ceph-mon[39784]: 3.f scrub starts 2024-05-15T23:12:40.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:40 smithi084 ceph-mon[39784]: 3.6 scrub starts 2024-05-15T23:12:40.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:40 smithi084 ceph-mon[39784]: osdmap e252: 12 total, 12 up, 12 in 2024-05-15T23:12:40.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:40 smithi084 ceph-mon[39784]: 3.d scrub starts 2024-05-15T23:12:40.163 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:40.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:40 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:12:40.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:40 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:12:40.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:40 smithi090 ceph-mon[44434]: pgmap v691: 17 pgs: 17 active+clean; 9.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T23:12:40.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:40 smithi090 ceph-mon[44434]: 3.f scrub starts 2024-05-15T23:12:40.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:40 smithi090 ceph-mon[44434]: 3.6 scrub starts 2024-05-15T23:12:40.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:40 smithi090 ceph-mon[44434]: osdmap e252: 12 total, 12 up, 12 in 2024-05-15T23:12:40.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:40 smithi090 ceph-mon[44434]: 3.d scrub starts 2024-05-15T23:12:41.031 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:12:41.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:41.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:12:41.139 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:12:41.177 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:41.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:12:41.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:41 smithi084 ceph-mon[39784]: 3.2 scrub ok 2024-05-15T23:12:41.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:41 smithi084 ceph-mon[39784]: 3.f scrub ok 2024-05-15T23:12:41.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:41 smithi084 ceph-mon[39784]: 3.e scrub starts 2024-05-15T23:12:41.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:41 smithi090 ceph-mon[44434]: 3.2 scrub ok 2024-05-15T23:12:41.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:41 smithi090 ceph-mon[44434]: 3.f scrub ok 2024-05-15T23:12:41.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:41 smithi090 ceph-mon[44434]: 3.e scrub starts 2024-05-15T23:12:41.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:41 smithi088 ceph-mon[44699]: 3.2 scrub ok 2024-05-15T23:12:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:41 smithi088 ceph-mon[44699]: 3.f scrub ok 2024-05-15T23:12:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:41 smithi088 ceph-mon[44699]: 3.e scrub starts 2024-05-15T23:12:41.392 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:12:41.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:41.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:12:41.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:12:41.559 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:41.620 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:12:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:41.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:12:41.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:12:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:41.795 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:12:41.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:41.798 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:12:41.899 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:12:41.932 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:41.952 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:12:41.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:41.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:12:42.055 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:12:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:42 smithi090 ceph-mon[44434]: pgmap v693: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:12:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:42 smithi090 ceph-mon[44434]: 3.6 scrub ok 2024-05-15T23:12:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:42 smithi090 ceph-mon[44434]: 3.d scrub ok 2024-05-15T23:12:42.103 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:42.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T23:12:42.137 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:12:42.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:42.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:12:42.244 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:12:42.275 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:12:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:42.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:12:42.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:42 smithi088 ceph-mon[44699]: pgmap v693: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:12:42.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:42 smithi088 ceph-mon[44699]: 3.6 scrub ok 2024-05-15T23:12:42.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:42 smithi088 ceph-mon[44699]: 3.d scrub ok 2024-05-15T23:12:42.361 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:42.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:42 smithi084 ceph-mon[39784]: pgmap v693: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:12:42.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:42 smithi084 ceph-mon[39784]: 3.6 scrub ok 2024-05-15T23:12:42.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:42 smithi084 ceph-mon[39784]: 3.d scrub ok 2024-05-15T23:12:42.412 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:12:42.481 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:42.687 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:12:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:42.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:12:42.792 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:12:42.819 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:42.843 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:12:42.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:42.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:12:42.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:12:42.991 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:43.018 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:12:43.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:43.021 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:12:43.050 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:12:43.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:43 smithi088 ceph-mon[44699]: 3.e scrub ok 2024-05-15T23:12:43.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:43 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2655186531' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:12:43.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:12:43.160 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:43.186 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:12:43.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:43.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:12:43.291 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:12:43.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:43 smithi090 ceph-mon[44434]: 3.e scrub ok 2024-05-15T23:12:43.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:43 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2655186531' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:12:43.339 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:43 smithi084 ceph-mon[39784]: 3.e scrub ok 2024-05-15T23:12:43.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:43 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2655186531' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:12:43.342 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:43.597 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:12:43.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:43.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:12:43.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:12:43.746 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:43.769 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:12:43.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:43.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:12:43.875 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:12:43.906 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:43.932 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:12:43.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:43.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:12:44.039 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:12:44.081 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:44.114 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:12:44.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:44.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:12:44.166 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:44.216 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:12:44.262 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:44.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:44 smithi090 ceph-mon[44434]: pgmap v694: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-05-15T23:12:44.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:44 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:12:44.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:44 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:12:44.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2655186531' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:12:44.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:44 smithi090 ceph-mon[44434]: osdmap e253: 12 total, 12 up, 12 in 2024-05-15T23:12:44.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:44 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:12:44.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:44 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:12:44.307 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:44 smithi084 ceph-mon[39784]: pgmap v694: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-05-15T23:12:44.307 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:44 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:12:44.307 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:44 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:12:44.307 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2655186531' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:12:44.307 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:44 smithi084 ceph-mon[39784]: osdmap e253: 12 total, 12 up, 12 in 2024-05-15T23:12:44.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:44 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:12:44.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:44 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:12:44.311 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:12:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:44 smithi088 ceph-mon[44699]: pgmap v694: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-05-15T23:12:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:44 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:12:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:44 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:12:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2655186531' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:12:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:44 smithi088 ceph-mon[44699]: osdmap e253: 12 total, 12 up, 12 in 2024-05-15T23:12:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:44 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:12:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:44 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:12:44.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:44.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:12:44.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:12:44.479 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:44.497 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:12:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:44.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:12:44.605 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:12:44.636 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:44.665 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:12:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:44.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:12:44.774 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:12:44.838 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:45.055 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:12:45.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:45.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:12:45.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:12:45.196 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:45.227 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:12:45.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:45.230 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:12:45.331 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:12:45.357 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:45.390 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:12:45.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:45.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:12:45.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:12:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:45.570 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:12:45.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:45.576 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:12:45.677 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:12:45.730 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:45.739 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:12:45.777 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:12:45.795 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:12:45.829 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:12:45.854 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:12:45.885 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:12:45.926 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:12:45.952 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:12:45.989 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:12:46.020 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:12:46.045 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:12:46.085 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:12:46.093 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:12:46.094 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:12:46.094 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:12:46.094 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:12:46.094 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:12:46.094 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:12:46.094 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:12:46.094 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:12:46.094 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:12:46.094 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:12:46.094 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:12:46.094 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:12:46.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:12:46.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:46 smithi090 ceph-mon[44434]: pgmap v696: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-05-15T23:12:46.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:46 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3740940888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:46.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:46.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:46 smithi088 ceph-mon[44699]: pgmap v696: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-05-15T23:12:46.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:46 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3740940888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:46.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:46.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:46 smithi084 ceph-mon[39784]: pgmap v696: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-05-15T23:12:46.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:46 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3740940888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:46.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:46.923 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:12:46.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:46.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:12:47.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:12:47.064 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:47.097 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:12:47.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:47.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:12:47.204 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:12:47.253 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:47.273 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:12:47.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:47.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:12:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.380 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:12:47.423 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:47.458 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:12:47.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:47.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:12:47.519 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:47.564 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:12:47.618 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:48.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:48 smithi084 ceph-mon[39784]: pgmap v697: 17 pgs: 17 active+clean; 8.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-05-15T23:12:48.273 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:12:48.273 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:12:48.280 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:12:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:48.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:12:48.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:48 smithi088 ceph-mon[44699]: pgmap v697: 17 pgs: 17 active+clean; 8.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-05-15T23:12:48.385 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:12:48.442 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:48.497 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:12:48.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:48.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:12:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:48 smithi090 ceph-mon[44434]: pgmap v697: 17 pgs: 17 active+clean; 8.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-05-15T23:12:48.613 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:12:48.654 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:48.676 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:12:48.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:48.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:12:48.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:12:48.809 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:48.839 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:12:48.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:48.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:12:48.947 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:12:48.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:12:48.977 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:12:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:49.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:12:49.109 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:12:49.150 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:50.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:50 smithi088 ceph-mon[44699]: pgmap v698: 17 pgs: 17 active+clean; 8.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:12:50.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:50 smithi084 ceph-mon[39784]: pgmap v698: 17 pgs: 17 active+clean; 8.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:12:50.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:50 smithi090 ceph-mon[44434]: pgmap v698: 17 pgs: 17 active+clean; 8.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:12:52.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:52 smithi084 ceph-mon[39784]: pgmap v699: 17 pgs: 17 active+clean; 8.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:12:52.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:52 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:12:52.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:52 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:12:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:52 smithi090 ceph-mon[44434]: pgmap v699: 17 pgs: 17 active+clean; 8.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:12:52.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:52 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:12:52.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:52 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:12:52.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:52 smithi088 ceph-mon[44699]: pgmap v699: 17 pgs: 17 active+clean; 8.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:12:52.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:52 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:12:52.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:52 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:12:53.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:53 smithi084 ceph-mon[39784]: osdmap e254: 12 total, 12 up, 12 in 2024-05-15T23:12:53.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:53 smithi090 ceph-mon[44434]: osdmap e254: 12 total, 12 up, 12 in 2024-05-15T23:12:53.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:53 smithi088 ceph-mon[44699]: osdmap e254: 12 total, 12 up, 12 in 2024-05-15T23:12:54.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:54 smithi084 ceph-mon[39784]: pgmap v701: 17 pgs: 17 active+clean; 8.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-05-15T23:12:54.406 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T23:12:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:54 smithi090 ceph-mon[44434]: pgmap v701: 17 pgs: 17 active+clean; 8.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-05-15T23:12:54.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:54 smithi088 ceph-mon[44699]: pgmap v701: 17 pgs: 17 active+clean; 8.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-05-15T23:12:54.629 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:54.647 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:12:54.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:54.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:12:54.752 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:12:54.798 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:54.970 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:12:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:54.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:12:55.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:12:55.112 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:55.134 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:12:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:55.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:12:55.238 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:12:55.280 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:55.290 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:55 smithi084 ceph-mon[39784]: 3.a scrub starts 2024-05-15T23:12:55.306 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:12:55.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:55.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:12:55.413 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:12:55.499 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:55.535 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:12:55.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:55.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:12:55.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:55 smithi090 ceph-mon[44434]: 3.a scrub starts 2024-05-15T23:12:55.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:55 smithi088 ceph-mon[44699]: 3.a scrub starts 2024-05-15T23:12:55.646 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:12:55.676 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:55.701 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:12:55.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:55.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:12:55.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:12:55.846 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:55.890 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:12:55.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:55.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:12:55.995 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:12:56.037 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:56.057 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:12:56.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:56.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:12:56.128 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:12:56.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:12:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:56.227 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:12:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:56.231 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:12:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:56 smithi088 ceph-mon[44699]: pgmap v702: 17 pgs: 17 active+clean; 8.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:12:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:56 smithi088 ceph-mon[44699]: 3.b scrub starts 2024-05-15T23:12:56.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:56 smithi088 ceph-mon[44699]: 3.a scrub ok 2024-05-15T23:12:56.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2333053427' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:12:56.331 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:12:56.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:56 smithi084 ceph-mon[39784]: pgmap v702: 17 pgs: 17 active+clean; 8.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:12:56.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:56 smithi084 ceph-mon[39784]: 3.b scrub starts 2024-05-15T23:12:56.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:56 smithi084 ceph-mon[39784]: 3.a scrub ok 2024-05-15T23:12:56.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2333053427' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:12:56.371 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:56.529 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:12:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:56.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:12:56.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:56 smithi090 ceph-mon[44434]: pgmap v702: 17 pgs: 17 active+clean; 8.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:12:56.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:56 smithi090 ceph-mon[44434]: 3.b scrub starts 2024-05-15T23:12:56.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:56 smithi090 ceph-mon[44434]: 3.a scrub ok 2024-05-15T23:12:56.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2333053427' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:12:56.638 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:12:56.680 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:56.701 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:12:56.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:56.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:12:56.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:12:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:56.875 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:12:56.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:56.900 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:12:57.001 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:12:57.032 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:57.060 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:12:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:57.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:12:57.167 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:12:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:57.218 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:12:57.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:57.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:12:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:57 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:12:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2333053427' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:12:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:57 smithi088 ceph-mon[44699]: osdmap e255: 12 total, 12 up, 12 in 2024-05-15T23:12:57.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:12:57.357 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:57.388 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:12:57.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:57 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:12:57.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2333053427' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:12:57.390 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:57 smithi090 ceph-mon[44434]: osdmap e255: 12 total, 12 up, 12 in 2024-05-15T23:12:57.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:57.391 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:12:57.492 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:12:57.497 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:57 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:12:57.497 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2333053427' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:12:57.497 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:57 smithi084 ceph-mon[39784]: osdmap e255: 12 total, 12 up, 12 in 2024-05-15T23:12:57.501 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:57.526 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:57.551 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:12:57.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:57.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:12:57.656 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:12:57.684 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:57.706 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:12:57.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:57.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:12:57.724 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-15T23:12:57.725 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:12:57.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:12:57.842 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:57.863 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:12:57.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:57.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:12:57.966 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:12:58.009 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:58.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:58 smithi090 ceph-mon[44434]: pgmap v704: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.37k op/s 2024-05-15T23:12:58.283 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:12:58.283 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:12:58.292 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:12:58.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:12:58.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:12:58.389 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:12:58.418 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:12:58.455 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:12:58.484 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:12:58.520 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:12:58.546 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:12:58.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:58 smithi088 ceph-mon[44699]: pgmap v704: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.37k op/s 2024-05-15T23:12:58.578 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:12:58.598 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:12:58.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:12:58.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:12:58.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:12:58.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:12:58.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:12:58.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:12:58.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:12:58.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:12:58.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:12:58.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:12:58.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:12:58.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:58 smithi084 ceph-mon[39784]: pgmap v704: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.37k op/s 2024-05-15T23:12:59.234 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:12:59.411 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:12:59.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:59.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:12:59.416 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:59 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/305636123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:59.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:59 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:12:59.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:12:59 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:12:59.451 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:12:59.517 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:12:59.543 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:59 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/305636123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:59.544 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.544 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.544 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.544 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.544 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.544 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.544 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.544 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.544 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.544 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:59 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:12:59.544 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:12:59 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:12:59.545 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:59 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/305636123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:12:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:12:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:59 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:12:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:12:59 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:12:59.567 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:12:59.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:59.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:12:59.671 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:12:59.722 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:12:59.938 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:12:59.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:12:59.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:13:00.047 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:00.090 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:00.117 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:00.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:00.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:00.274 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:00.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:00.278 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:00.379 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:00.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:00.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:00.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:00.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:00 smithi088 ceph-mon[44699]: pgmap v705: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:13:00.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:00 smithi088 ceph-mon[44699]: osdmap e256: 12 total, 12 up, 12 in 2024-05-15T23:13:00.418 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:00.423 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:00.423 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:00.423 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:00.423 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:00 smithi084 ceph-mon[39784]: pgmap v705: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:13:00.424 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:00 smithi084 ceph-mon[39784]: osdmap e256: 12 total, 12 up, 12 in 2024-05-15T23:13:00.440 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:00.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:00.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:00.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:13:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:00 smithi090 ceph-mon[44434]: pgmap v705: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:13:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:00 smithi090 ceph-mon[44434]: osdmap e256: 12 total, 12 up, 12 in 2024-05-15T23:13:00.575 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:00.598 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:13:00.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:00.600 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:13:00.702 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:00.730 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:00.765 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:00.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:00.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:00.805 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:13:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:13:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:13:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.25135.0:208855 3.6 e256/202 3:61b52239:::benchmark_data_smithi090_90974_object43312:head v 256'15326, mlcod=256'15322)", 2024-05-15T23:13:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:00.800998+0000", 2024-05-15T23:13:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.0015451040000000001, 2024-05-15T23:13:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015685709999999999, 2024-05-15T23:13:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:13:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:13:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-05-15T23:13:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:13:00.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:13:00.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:13:00.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:00.800998+0000", 2024-05-15T23:13:00.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:13:00.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:13:00.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:13:00.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:13:00.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:00.800998+0000", 2024-05-15T23:13:00.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:13:00.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:13:00.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:13:00.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:13:00.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:00.801000+0000", 2024-05-15T23:13:00.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6899999999999999e-06 2024-05-15T23:13:00.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:13:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:13:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:13:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:00.801009+0000", 2024-05-15T23:13:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3139999999999993e-06 2024-05-15T23:13:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:13:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:13:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:13:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:00.801022+0000", 2024-05-15T23:13:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2986999999999999e-05 2024-05-15T23:13:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:13:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:13:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:13:00.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:00.801030+0000", 2024-05-15T23:13:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1580000000000005e-06 2024-05-15T23:13:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:13:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:13:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:13:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:00.801127+0000", 2024-05-15T23:13:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7173e-05 2024-05-15T23:13:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:13:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:13:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:13:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:00.801148+0000", 2024-05-15T23:13:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0454000000000001e-05 2024-05-15T23:13:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:13:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:13:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:13:00.808 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:13:00.809 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:13:00.809 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 1 2024-05-15T23:13:00.809 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:13:00.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:00.921 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:01.099 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:01.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:01.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:01.203 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:13:01.237 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:01.265 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:13:01.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:01.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:13:01.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:13:01.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:13:01.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:13:01.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:13:01.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:13:01.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:13:01.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:13:01.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:01 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:13:01.407 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:01.431 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:13:01.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:01.433 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:13:01.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:13:01.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:13:01.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:13:01.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:13:01.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:13:01.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:13:01.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:01 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:13:01.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:13:01.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:13:01.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:13:01.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:13:01.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:13:01.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:13:01.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:13:01.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:01 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:13:01.587 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:01.966 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:13:02.060 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:13:02.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:02.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:13:02.087 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T23:13:02.174 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:02.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:02 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:13:02.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:02 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:13:02.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:02 smithi084 ceph-mon[39784]: pgmap v707: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:13:02.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:02 smithi084 ceph-mon[39784]: osdmap e257: 12 total, 12 up, 12 in 2024-05-15T23:13:02.275 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:02.537 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:02 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:13:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:02 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:13:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:02 smithi090 ceph-mon[44434]: pgmap v707: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:13:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:02 smithi090 ceph-mon[44434]: osdmap e257: 12 total, 12 up, 12 in 2024-05-15T23:13:02.553 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:02.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:02.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:02.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:02 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:13:02.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:02 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:13:02.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:02 smithi088 ceph-mon[44699]: pgmap v707: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:13:02.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:02 smithi088 ceph-mon[44699]: osdmap e257: 12 total, 12 up, 12 in 2024-05-15T23:13:02.659 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:02.689 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:02.715 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:02.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:02.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:02.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:13:02.849 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:02.882 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:13:02.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:02.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:13:02.986 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:03.016 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:03.037 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:03.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:03.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:03.141 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:03.172 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:03.202 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:03.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:03.308 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:03.345 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:03.428 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:03.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:03.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:03.535 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:03.593 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:03.744 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:03.745 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:03.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:03.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:03.888 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:03.893 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:13:03.909 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:13:03.914 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:03.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:03.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:04.018 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:04.068 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:04.378 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:04 smithi084 ceph-mon[39784]: pgmap v709: 17 pgs: 17 active+clean; 7.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:13:04.438 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:04.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:04 smithi090 ceph-mon[44434]: pgmap v709: 17 pgs: 17 active+clean; 7.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:13:04.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:04 smithi088 ceph-mon[44699]: pgmap v709: 17 pgs: 17 active+clean; 7.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:13:04.606 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:05.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:05 smithi084 ceph-mon[39784]: from='client.25285 v1:172.21.15.84:0/704936414' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:13:05.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:05 smithi090 ceph-mon[44434]: from='client.25285 v1:172.21.15.84:0/704936414' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:13:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:05 smithi088 ceph-mon[44699]: from='client.25285 v1:172.21.15.84:0/704936414' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:13:05.580 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:05.583 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":709,"stamp":"2024-05-15T23:13:02.819529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7997096504,"num_objects":122020,"num_object_clones":0,"num_object_copies":244040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122020,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":209187,"num_write_kb":10596711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160866,"ondisk_log_size":160866,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1124941824,"kb_used":17547332,"kb_used_data":15709768,"kb_used_omap":18,"kb_used_meta":1837229,"kb_avail":1107394492,"statfs":{"total":1151940427776,"available":1133971959808,"internally_reserved":0,"allocated":16086802432,"data_stored":16082752716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":1881322810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,7,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-554172416,"num_objects":-8456,"num_object_clones":0,"num_object_copies":-16912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8456,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.706605"},"pg_stats":[{"pgid":"3.f","version":"256'12902","reported_seq":13164,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-05-15T23:12:59.964360+0000","last_change":"2024-05-15T23:12:40.314892+0000","last_active":"2024-05-15T23:12:59.964360+0000","last_peered":"2024-05-15T23:12:59.964360+0000","last_clean":"2024-05-15T23:12:59.964360+0000","last_became_active":"2024-05-15T23:10:41.793048+0000","last_became_peered":"2024-05-15T23:10:41.793048+0000","last_unstale":"2024-05-15T23:12:59.964360+0000","last_undegraded":"2024-05-15T23:12:59.964360+0000","last_fullsized":"2024-05-15T23:12:59.964360+0000","mapping_epoch":202,"log_start":"212'2900","ondisk_log_start":"212'2900","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'11360","last_scrub_stamp":"2024-05-15T23:12:40.314742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:40.314742+0000","objects_scrubbed":9220,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:44.490079+0000","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499777536,"num_objects":7626,"num_object_clones":0,"num_object_copies":15252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12902,"num_write_kb":656896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"257'16611","reported_seq":17015,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-05-15T23:13:02.388754+0000","last_change":"2024-05-15T23:12:42.022558+0000","last_active":"2024-05-15T23:13:02.388754+0000","last_peered":"2024-05-15T23:13:02.388754+0000","last_clean":"2024-05-15T23:13:02.388754+0000","last_became_active":"2024-05-15T23:10:41.794521+0000","last_became_peered":"2024-05-15T23:10:41.794521+0000","last_unstale":"2024-05-15T23:13:02.388754+0000","last_undegraded":"2024-05-15T23:13:02.388754+0000","last_fullsized":"2024-05-15T23:13:02.388754+0000","mapping_epoch":202,"log_start":"225'6603","ondisk_log_start":"225'6603","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'15032","last_scrub_stamp":"2024-05-15T23:12:42.022424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:42.022424+0000","objects_scrubbed":9101,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:10.982043+0000","scrub_duration":1664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488898560,"num_objects":7460,"num_object_clones":0,"num_object_copies":14920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13140,"num_write_kb":659200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"256'13033","reported_seq":13253,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-05-15T23:12:59.942999+0000","last_change":"2024-05-15T23:12:41.500301+0000","last_active":"2024-05-15T23:12:59.942999+0000","last_peered":"2024-05-15T23:12:59.942999+0000","last_clean":"2024-05-15T23:12:59.942999+0000","last_became_active":"2024-05-15T23:10:41.794133+0000","last_became_peered":"2024-05-15T23:10:41.794133+0000","last_unstale":"2024-05-15T23:12:59.942999+0000","last_undegraded":"2024-05-15T23:12:59.942999+0000","last_fullsized":"2024-05-15T23:12:59.942999+0000","mapping_epoch":202,"log_start":"212'3000","ondisk_log_start":"212'3000","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'11587","last_scrub_stamp":"2024-05-15T23:12:41.500086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:41.500086+0000","objects_scrubbed":9204,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:51.521933+0000","scrub_duration":1568,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505020440,"num_objects":7707,"num_object_clones":0,"num_object_copies":15414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13033,"num_write_kb":663617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"257'13153","reported_seq":13369,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-05-15T23:13:01.794873+0000","last_change":"2024-05-15T23:11:59.909075+0000","last_active":"2024-05-15T23:13:01.794873+0000","last_peered":"2024-05-15T23:13:01.794873+0000","last_clean":"2024-05-15T23:13:01.794873+0000","last_became_active":"2024-05-15T23:10:41.795518+0000","last_became_peered":"2024-05-15T23:10:41.795518+0000","last_unstale":"2024-05-15T23:13:01.794873+0000","last_undegraded":"2024-05-15T23:13:01.794873+0000","last_fullsized":"2024-05-15T23:13:01.794873+0000","mapping_epoch":202,"log_start":"212'3100","ondisk_log_start":"212'3100","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'8858","last_scrub_stamp":"2024-05-15T23:11:59.908918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:11:59.908918+0000","objects_scrubbed":8807,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:29.204022+0000","scrub_duration":1539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493682688,"num_objects":7533,"num_object_clones":0,"num_object_copies":15066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13153,"num_write_kb":661952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"256'13150","reported_seq":13412,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-05-15T23:13:00.918295+0000","last_change":"2024-05-15T23:12:56.131288+0000","last_active":"2024-05-15T23:13:00.918295+0000","last_peered":"2024-05-15T23:13:00.918295+0000","last_clean":"2024-05-15T23:13:00.918295+0000","last_became_active":"2024-05-15T23:10:41.791796+0000","last_became_peered":"2024-05-15T23:10:41.791796+0000","last_unstale":"2024-05-15T23:13:00.918295+0000","last_undegraded":"2024-05-15T23:13:00.918295+0000","last_fullsized":"2024-05-15T23:13:00.918295+0000","mapping_epoch":202,"log_start":"212'3100","ondisk_log_start":"212'3100","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'8356","last_scrub_stamp":"2024-05-15T23:11:53.172114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:11:53.172114+0000","objects_scrubbed":4799,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:01.124759+0000","scrub_duration":2063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499646464,"num_objects":7624,"num_object_clones":0,"num_object_copies":15248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13150,"num_write_kb":664768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"257'13079","reported_seq":13378,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-05-15T23:13:02.518425+0000","last_change":"2024-05-15T23:12:55.282863+0000","last_active":"2024-05-15T23:13:02.518425+0000","last_peered":"2024-05-15T23:13:02.518425+0000","last_clean":"2024-05-15T23:13:02.518425+0000","last_became_active":"2024-05-15T23:10:41.794649+0000","last_became_peered":"2024-05-15T23:10:41.794649+0000","last_unstale":"2024-05-15T23:13:02.518425+0000","last_undegraded":"2024-05-15T23:13:02.518425+0000","last_fullsized":"2024-05-15T23:13:02.518425+0000","mapping_epoch":202,"log_start":"212'3000","ondisk_log_start":"212'3000","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12548","last_scrub_stamp":"2024-05-15T23:12:55.282531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:55.282531+0000","objects_scrubbed":8168,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:59.339140+0000","scrub_duration":1166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498794496,"num_objects":7611,"num_object_clones":0,"num_object_copies":15222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13079,"num_write_kb":662080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"256'13085","reported_seq":13307,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-05-15T23:12:59.535149+0000","last_change":"2024-05-15T23:12:37.885059+0000","last_active":"2024-05-15T23:12:59.535149+0000","last_peered":"2024-05-15T23:12:59.535149+0000","last_clean":"2024-05-15T23:12:59.535149+0000","last_became_active":"2024-05-15T23:10:41.793356+0000","last_became_peered":"2024-05-15T23:10:41.793356+0000","last_unstale":"2024-05-15T23:12:59.535149+0000","last_undegraded":"2024-05-15T23:12:59.535149+0000","last_fullsized":"2024-05-15T23:12:59.535149+0000","mapping_epoch":202,"log_start":"212'3000","ondisk_log_start":"212'3000","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'11462","last_scrub_stamp":"2024-05-15T23:12:37.884916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:37.884916+0000","objects_scrubbed":9564,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:05.736809+0000","scrub_duration":1718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":516620288,"num_objects":7883,"num_object_clones":0,"num_object_copies":15766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13085,"num_write_kb":670976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"257'13309","reported_seq":13514,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-05-15T23:13:02.392576+0000","last_change":"2024-05-15T23:11:52.185824+0000","last_active":"2024-05-15T23:13:02.392576+0000","last_peered":"2024-05-15T23:13:02.392576+0000","last_clean":"2024-05-15T23:13:02.392576+0000","last_became_active":"2024-05-15T23:10:41.794942+0000","last_became_peered":"2024-05-15T23:10:41.794942+0000","last_unstale":"2024-05-15T23:13:02.392576+0000","last_undegraded":"2024-05-15T23:13:02.392576+0000","last_fullsized":"2024-05-15T23:13:02.392576+0000","mapping_epoch":202,"log_start":"212'3300","ondisk_log_start":"212'3300","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'8246","last_scrub_stamp":"2024-05-15T23:11:52.185650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:11:52.185650+0000","objects_scrubbed":8194,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:06.253504+0000","scrub_duration":1758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":496697344,"num_objects":7579,"num_object_clones":0,"num_object_copies":15158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13309,"num_write_kb":668416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"256'13094","reported_seq":13303,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-05-15T23:13:00.832350+0000","last_change":"2024-05-15T23:11:52.189382+0000","last_active":"2024-05-15T23:13:00.832350+0000","last_peered":"2024-05-15T23:13:00.832350+0000","last_clean":"2024-05-15T23:13:00.832350+0000","last_became_active":"2024-05-15T23:10:41.792820+0000","last_became_peered":"2024-05-15T23:10:41.792820+0000","last_unstale":"2024-05-15T23:13:00.832350+0000","last_undegraded":"2024-05-15T23:13:00.832350+0000","last_fullsized":"2024-05-15T23:13:00.832350+0000","mapping_epoch":202,"log_start":"212'3000","ondisk_log_start":"212'3000","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'8266","last_scrub_stamp":"2024-05-15T23:11:52.189078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:11:52.189078+0000","objects_scrubbed":8190,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:05.829973+0000","scrub_duration":2080,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503054336,"num_objects":7676,"num_object_clones":0,"num_object_copies":15352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13094,"num_write_kb":664640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"256'15245","reported_seq":15585,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-05-15T23:12:59.741685+0000","last_change":"2024-05-15T23:12:40.833253+0000","last_active":"2024-05-15T23:12:59.741685+0000","last_peered":"2024-05-15T23:12:59.741685+0000","last_clean":"2024-05-15T23:12:59.741685+0000","last_became_active":"2024-05-15T23:10:41.794131+0000","last_became_peered":"2024-05-15T23:10:41.794131+0000","last_unstale":"2024-05-15T23:12:59.741685+0000","last_undegraded":"2024-05-15T23:12:59.741685+0000","last_fullsized":"2024-05-15T23:12:59.741685+0000","mapping_epoch":202,"log_start":"219'5202","ondisk_log_start":"219'5202","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'13799","last_scrub_stamp":"2024-05-15T23:12:40.833117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:40.833117+0000","objects_scrubbed":9233,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:09.080677+0000","scrub_duration":1917,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505479168,"num_objects":7713,"num_object_clones":0,"num_object_copies":15426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13003,"num_write_kb":662912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"257'13032","reported_seq":13248,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-05-15T23:13:01.782861+0000","last_change":"2024-05-15T23:11:59.197199+0000","last_active":"2024-05-15T23:13:01.782861+0000","last_peered":"2024-05-15T23:13:01.782861+0000","last_clean":"2024-05-15T23:13:01.782861+0000","last_became_active":"2024-05-15T23:10:41.796378+0000","last_became_peered":"2024-05-15T23:10:41.796378+0000","last_unstale":"2024-05-15T23:13:01.782861+0000","last_undegraded":"2024-05-15T23:13:01.782861+0000","last_fullsized":"2024-05-15T23:13:01.782861+0000","mapping_epoch":202,"log_start":"212'3000","ondisk_log_start":"212'3000","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'8785","last_scrub_stamp":"2024-05-15T23:11:59.197105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:11:59.197105+0000","objects_scrubbed":8717,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:05.475199+0000","scrub_duration":1842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489684992,"num_objects":7472,"num_object_clones":0,"num_object_copies":14944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13032,"num_write_kb":656128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"256'12935","reported_seq":13144,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-05-15T23:13:00.010617+0000","last_change":"2024-05-15T23:11:52.359289+0000","last_active":"2024-05-15T23:13:00.010617+0000","last_peered":"2024-05-15T23:13:00.010617+0000","last_clean":"2024-05-15T23:13:00.010617+0000","last_became_active":"2024-05-15T23:10:41.792966+0000","last_became_peered":"2024-05-15T23:10:41.792966+0000","last_unstale":"2024-05-15T23:13:00.010617+0000","last_undegraded":"2024-05-15T23:13:00.010617+0000","last_fullsized":"2024-05-15T23:13:00.010617+0000","mapping_epoch":202,"log_start":"212'2900","ondisk_log_start":"212'2900","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'8147","last_scrub_stamp":"2024-05-15T23:11:52.359167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:11:52.359167+0000","objects_scrubbed":8094,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:12.848386+0000","scrub_duration":1688,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494862336,"num_objects":7551,"num_object_clones":0,"num_object_copies":15102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12935,"num_write_kb":655552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"256'12901","reported_seq":13117,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-05-15T23:12:59.457867+0000","last_change":"2024-05-15T23:12:07.628488+0000","last_active":"2024-05-15T23:12:59.457867+0000","last_peered":"2024-05-15T23:12:59.457867+0000","last_clean":"2024-05-15T23:12:59.457867+0000","last_became_active":"2024-05-15T23:10:41.794364+0000","last_became_peered":"2024-05-15T23:10:41.794364+0000","last_unstale":"2024-05-15T23:12:59.457867+0000","last_undegraded":"2024-05-15T23:12:59.457867+0000","last_fullsized":"2024-05-15T23:12:59.457867+0000","mapping_epoch":202,"log_start":"212'2900","ondisk_log_start":"212'2900","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'9275","last_scrub_stamp":"2024-05-15T23:12:07.628308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:07.628308+0000","objects_scrubbed":9240,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:18.947219+0000","scrub_duration":1442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508493824,"num_objects":7759,"num_object_clones":0,"num_object_copies":15518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12901,"num_write_kb":661120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"256'12990","reported_seq":13205,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-05-15T23:12:59.752560+0000","last_change":"2024-05-15T23:12:02.421561+0000","last_active":"2024-05-15T23:12:59.752560+0000","last_peered":"2024-05-15T23:12:59.752560+0000","last_clean":"2024-05-15T23:12:59.752560+0000","last_became_active":"2024-05-15T23:10:41.794291+0000","last_became_peered":"2024-05-15T23:10:41.794291+0000","last_unstale":"2024-05-15T23:12:59.752560+0000","last_undegraded":"2024-05-15T23:12:59.752560+0000","last_fullsized":"2024-05-15T23:12:59.752560+0000","mapping_epoch":202,"log_start":"212'2900","ondisk_log_start":"212'2900","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'9012","last_scrub_stamp":"2024-05-15T23:12:02.421414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:02.421414+0000","objects_scrubbed":8965,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:24.413488+0000","scrub_duration":1406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502661120,"num_objects":7670,"num_object_clones":0,"num_object_copies":15340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12990,"num_write_kb":661120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"257'13277","reported_seq":13498,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-05-15T23:13:02.386807+0000","last_change":"2024-05-15T23:12:07.892209+0000","last_active":"2024-05-15T23:13:02.386807+0000","last_peered":"2024-05-15T23:13:02.386807+0000","last_clean":"2024-05-15T23:13:02.386807+0000","last_became_active":"2024-05-15T23:10:42.014274+0000","last_became_peered":"2024-05-15T23:10:42.014274+0000","last_unstale":"2024-05-15T23:13:02.386807+0000","last_undegraded":"2024-05-15T23:13:02.386807+0000","last_fullsized":"2024-05-15T23:13:02.386807+0000","mapping_epoch":202,"log_start":"212'3200","ondisk_log_start":"212'3200","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'9456","last_scrub_stamp":"2024-05-15T23:12:07.892057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:07.892057+0000","objects_scrubbed":9414,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:30.995632+0000","scrub_duration":1441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500498432,"num_objects":7637,"num_object_clones":0,"num_object_copies":15274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13277,"num_write_kb":669248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"256'12981","reported_seq":13259,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-05-15T23:13:00.902343+0000","last_change":"2024-05-15T23:12:39.938806+0000","last_active":"2024-05-15T23:13:00.902343+0000","last_peered":"2024-05-15T23:13:00.902343+0000","last_clean":"2024-05-15T23:13:00.902343+0000","last_became_active":"2024-05-15T23:10:42.014285+0000","last_became_peered":"2024-05-15T23:10:42.014285+0000","last_unstale":"2024-05-15T23:13:00.902343+0000","last_undegraded":"2024-05-15T23:13:00.902343+0000","last_fullsized":"2024-05-15T23:13:00.902343+0000","mapping_epoch":202,"log_start":"212'2900","ondisk_log_start":"212'2900","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'11402","last_scrub_stamp":"2024-05-15T23:12:39.938642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:39.938642+0000","objects_scrubbed":9163,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:40.708767+0000","scrub_duration":2007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":492634112,"num_objects":7517,"num_object_clones":0,"num_object_copies":15034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12981,"num_write_kb":655936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":507,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-05-15T23:12:59.160493+0000","last_change":"2024-05-15T23:12:43.939662+0000","last_active":"2024-05-15T23:12:59.160493+0000","last_peered":"2024-05-15T23:12:59.160493+0000","last_clean":"2024-05-15T23:12:59.160493+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:12:59.160493+0000","last_undegraded":"2024-05-15T23:12:59.160493+0000","last_fullsized":"2024-05-15T23:12:59.160493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:12:43.939526+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:12:43.939526+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:00.606486+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":7996506136,"num_objects":122018,"num_object_clones":0,"num_object_copies":244036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122018,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":209064,"num_write_kb":10594561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16010256384,"data_stored":16010248240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160772,"ondisk_log_size":160772,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351087,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3653296,"kb_used_data":3379176,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":90091856,"statfs":{"total":95995035648,"available":92254060544,"internally_reserved":0,"allocated":3460276224,"data_stored":3459930444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":280689092},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":63780,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93681372,"statfs":{"total":95995035648,"available":95929724928,"internally_reserved":0,"allocated":675840,"data_stored":339636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776252,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1173792,"kb_used_data":994068,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92571360,"statfs":{"total":95995035648,"available":94793072640,"internally_reserved":0,"allocated":1017925632,"data_stored":1017589428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":184023488},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005184,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":631840,"kb_used_data":506260,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93113312,"statfs":{"total":95995035648,"available":95348031488,"internally_reserved":0,"allocated":518410240,"data_stored":518074036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1121052,"kb_used_data":991316,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92624100,"statfs":{"total":95995035648,"available":94847078400,"internally_reserved":0,"allocated":1015107584,"data_stored":1014769760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":132839882},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":556904,"kb_used_data":489632,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93188248,"statfs":{"total":95995035648,"available":95424765952,"internally_reserved":0,"allocated":501383168,"data_stored":501040960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575012,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1640864,"kb_used_data":1479188,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92104288,"statfs":{"total":95995035648,"available":94314790912,"internally_reserved":0,"allocated":1514688512,"data_stored":1514352308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165542342},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738537,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2229648,"kb_used_data":1991432,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":91515504,"statfs":{"total":95995035648,"available":93711876096,"internally_reserved":0,"allocated":2039226368,"data_stored":2038896168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346078,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":112352,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93632800,"statfs":{"total":95995035648,"available":95879987200,"internally_reserved":0,"allocated":675840,"data_stored":339636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2662556,"kb_used_data":2432852,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":91082596,"statfs":{"total":95995035648,"available":93268578304,"internally_reserved":0,"allocated":2491240448,"data_stored":2490904244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":235207111},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986323,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2133840,"kb_used_data":1994376,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":91611312,"statfs":{"total":95995035648,"available":93809983488,"internally_reserved":0,"allocated":2042241024,"data_stored":2041910824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790094,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1567408,"kb_used_data":1450148,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92177744,"statfs":{"total":95995035648,"available":94390009856,"internally_reserved":0,"allocated":1484951552,"data_stored":1484605272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2019753984,"data_stored":2019753984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1507590144,"data_stored":1507590144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2488074240,"data_stored":2488074240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2032074752,"data_stored":2032074752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1478496256,"data_stored":1478492184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":498794496,"data_stored":498794496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1004802048,"data_stored":1004797976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":516554752,"data_stored":516554752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1007681536,"data_stored":1007681536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3456434176,"data_stored":3456434176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:13:05.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:13:05.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:05.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:05.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:05.719 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:05.750 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:05.757 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:05.858 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:13:05.895 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:05.923 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:13:05.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:05.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:13:06.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:13:06.065 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:06.090 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:13:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:06.094 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:13:06.194 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:06.227 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:06.258 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:06.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:06.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:06.307 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:06 smithi088 ceph-mon[44699]: pgmap v710: 17 pgs: 17 active+clean; 7.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:13:06.366 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:13:06.422 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:06.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:06 smithi090 ceph-mon[44434]: pgmap v710: 17 pgs: 17 active+clean; 7.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:13:06.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:06 smithi084 ceph-mon[39784]: pgmap v710: 17 pgs: 17 active+clean; 7.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:13:06.645 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:13:06.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:06.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:13:06.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:13:06.798 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:06.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:06.835 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:13:06.835 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:13:06.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:06.971 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:06.997 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:07.003 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:07.104 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:13:07.139 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:07.163 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:13:07.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:07.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:13:07.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:07.285 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:07.365 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:07.390 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:13:07.390 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:13:07.390 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:13:07.390 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:13:07.390 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:13:07.404 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:07.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:07.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:07.522 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:13:07.557 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:07.584 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:13:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:07.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:13:07.621 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:07.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:13:07.722 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:07.750 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:13:07.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:07.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:13:07.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:13:07.896 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:07.932 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:13:07.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:07.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:13:08.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:13:08.067 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:08.094 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:13:08.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:08.097 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:13:08.198 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:08.231 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:08.342 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:13:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:08 smithi084 ceph-mon[39784]: pgmap v711: 17 pgs: 17 active+clean; 7.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.33k op/s 2024-05-15T23:13:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/177755542' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:13:08.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:08 smithi084 ceph-mon[39784]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:13:08.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:08 smithi090 ceph-mon[44434]: pgmap v711: 17 pgs: 17 active+clean; 7.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.33k op/s 2024-05-15T23:13:08.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/177755542' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:13:08.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:08 smithi090 ceph-mon[44434]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:13:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:08 smithi088 ceph-mon[44699]: pgmap v711: 17 pgs: 17 active+clean; 7.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.33k op/s 2024-05-15T23:13:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/177755542' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:13:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:08 smithi088 ceph-mon[44699]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:13:08.908 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:08.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:09.016 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:09.059 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:09.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:09 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:13:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:09 smithi088 ceph-mon[44699]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:13:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:09 smithi088 ceph-mon[44699]: osdmap e258: 12 total, 12 up, 12 in 2024-05-15T23:13:09.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:09 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:13:09.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:09 smithi084 ceph-mon[39784]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:13:09.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:09 smithi084 ceph-mon[39784]: osdmap e258: 12 total, 12 up, 12 in 2024-05-15T23:13:09.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:09 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:13:09.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:09 smithi090 ceph-mon[44434]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:13:09.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:09 smithi090 ceph-mon[44434]: osdmap e258: 12 total, 12 up, 12 in 2024-05-15T23:13:10.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:10 smithi084 ceph-mon[39784]: pgmap v713: 17 pgs: 17 active+clean; 7.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:13:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:10 smithi090 ceph-mon[44434]: pgmap v713: 17 pgs: 17 active+clean; 7.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:13:10.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:10 smithi088 ceph-mon[44699]: pgmap v713: 17 pgs: 17 active+clean; 7.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:13:10.839 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:10.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:10.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:10.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T23:13:10.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:13:10.973 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:10.994 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:13:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:10.997 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:13:11.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:13:11.129 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:11.151 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:13:11.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:11.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:13:11.260 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:13:11.288 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:11.309 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:13:11.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:11.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:13:11.413 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:11.455 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:11.478 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:11.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:11.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:11.651 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:11.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:11.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:11.756 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:11.785 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:12.152 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:12.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:12.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:12.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:13:12.296 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:12.319 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:13:12.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:12.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:13:12.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:12.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:12 smithi084 ceph-mon[39784]: pgmap v714: 17 pgs: 17 active+clean; 6.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:13:12.462 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:12.488 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:12.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:12.491 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:12.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:12 smithi090 ceph-mon[44434]: pgmap v714: 17 pgs: 17 active+clean; 6.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:13:12.592 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:13:12.640 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:12.668 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:13:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:12.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:13:12.776 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:12.816 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:12 smithi088 ceph-mon[44699]: pgmap v714: 17 pgs: 17 active+clean; 6.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:13:12.823 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:12.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:12.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:12.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:13:13.061 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:13.120 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:13:13.121 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:13:13.132 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:13:13.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:13.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:13:13.158 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:13:13.162 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:13.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:13:13.240 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:13:13.247 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:13:13.276 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:13:13.290 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:13.293 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:13:13.316 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:13:13.336 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:13:13.387 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:13:13.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:13:13.444 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:13:13.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:13.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:13:13.448 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:13:13.488 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:13:13.500 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:13:13.500 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:13:13.500 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:13:13.501 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:13:13.501 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:13:13.501 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:13:13.501 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:13:13.501 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:13:13.501 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:13:13.501 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:13:13.501 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:13:13.501 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:13:13.549 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:13.592 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:13.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3146060249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:13:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:13.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:13:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3146060249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:13:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3146060249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:13:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:14 smithi084 ceph-mon[39784]: pgmap v715: 17 pgs: 17 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:13:14.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:14 smithi084 ceph-mon[39784]: from='client.15378 v1:172.21.15.84:0/591015008' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:13:14.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.600 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.600 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.600 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.600 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.600 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.600 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:14.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:14.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:14.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:14.760 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:14.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:14 smithi090 ceph-mon[44434]: pgmap v715: 17 pgs: 17 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:13:14.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:14 smithi090 ceph-mon[44434]: from='client.15378 v1:172.21.15.84:0/591015008' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:13:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.814 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:14 smithi088 ceph-mon[44699]: pgmap v715: 17 pgs: 17 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:13:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:14 smithi088 ceph-mon[44699]: from='client.15378 v1:172.21.15.84:0/591015008' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:13:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:14.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:14.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:14.937 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:14.966 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:14.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:14.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:15.095 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:15.135 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:15.178 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:15.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:15.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:15.211 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":715,"stamp":"2024-05-15T23:13:12.821441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7191200312,"num_objects":109723,"num_object_clones":0,"num_object_copies":219446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109723,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":221484,"num_write_kb":10596711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160963,"ondisk_log_size":160963,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1124941824,"kb_used":15984804,"kb_used_data":14127976,"kb_used_omap":18,"kb_used_meta":1856685,"kb_avail":1108957020,"statfs":{"total":1151940427776,"available":1135571988480,"internally_reserved":0,"allocated":14467047424,"data_stored":14462853012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":1901245754},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,4,6,10,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-805896192,"num_objects":-12297,"num_object_clones":0,"num_object_copies":-24594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12297,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001912"},"pg_stats":[{"pgid":"3.f","version":"258'13614","reported_seq":13880,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:09.968186+0000","last_change":"2024-05-15T23:12:40.314892+0000","last_active":"2024-05-15T23:13:09.968186+0000","last_peered":"2024-05-15T23:13:09.968186+0000","last_clean":"2024-05-15T23:13:09.968186+0000","last_became_active":"2024-05-15T23:10:41.793048+0000","last_became_peered":"2024-05-15T23:10:41.793048+0000","last_unstale":"2024-05-15T23:13:09.968186+0000","last_undegraded":"2024-05-15T23:13:09.968186+0000","last_fullsized":"2024-05-15T23:13:09.968186+0000","mapping_epoch":202,"log_start":"214'3600","ondisk_log_start":"214'3600","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'11360","last_scrub_stamp":"2024-05-15T23:12:40.314742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:40.314742+0000","objects_scrubbed":9220,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:44.490079+0000","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453115904,"num_objects":6914,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13614,"num_write_kb":656896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"258'17386","reported_seq":17792,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:12.374014+0000","last_change":"2024-05-15T23:12:42.022558+0000","last_active":"2024-05-15T23:13:12.374014+0000","last_peered":"2024-05-15T23:13:12.374014+0000","last_clean":"2024-05-15T23:13:12.374014+0000","last_became_active":"2024-05-15T23:10:41.794521+0000","last_became_peered":"2024-05-15T23:10:41.794521+0000","last_unstale":"2024-05-15T23:13:12.374014+0000","last_undegraded":"2024-05-15T23:13:12.374014+0000","last_fullsized":"2024-05-15T23:13:12.374014+0000","mapping_epoch":202,"log_start":"229'7303","ondisk_log_start":"229'7303","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'15032","last_scrub_stamp":"2024-05-15T23:12:42.022424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:42.022424+0000","objects_scrubbed":9101,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:10.982043+0000","scrub_duration":1664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438108160,"num_objects":6685,"num_object_clones":0,"num_object_copies":13370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13915,"num_write_kb":659200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"258'13793","reported_seq":14017,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:09.966386+0000","last_change":"2024-05-15T23:12:41.500301+0000","last_active":"2024-05-15T23:13:09.966386+0000","last_peered":"2024-05-15T23:13:09.966386+0000","last_clean":"2024-05-15T23:13:09.966386+0000","last_became_active":"2024-05-15T23:10:41.794133+0000","last_became_peered":"2024-05-15T23:10:41.794133+0000","last_unstale":"2024-05-15T23:13:09.966386+0000","last_undegraded":"2024-05-15T23:13:09.966386+0000","last_fullsized":"2024-05-15T23:13:09.966386+0000","mapping_epoch":202,"log_start":"214'3700","ondisk_log_start":"214'3700","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'11587","last_scrub_stamp":"2024-05-15T23:12:41.500086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:41.500086+0000","objects_scrubbed":9204,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:51.521933+0000","scrub_duration":1568,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455213080,"num_objects":6947,"num_object_clones":0,"num_object_copies":13894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13793,"num_write_kb":663617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"258'13939","reported_seq":14157,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:11.804139+0000","last_change":"2024-05-15T23:11:59.909075+0000","last_active":"2024-05-15T23:13:11.804139+0000","last_peered":"2024-05-15T23:13:11.804139+0000","last_clean":"2024-05-15T23:13:11.804139+0000","last_became_active":"2024-05-15T23:10:41.795518+0000","last_became_peered":"2024-05-15T23:10:41.795518+0000","last_unstale":"2024-05-15T23:13:11.804139+0000","last_undegraded":"2024-05-15T23:13:11.804139+0000","last_fullsized":"2024-05-15T23:13:11.804139+0000","mapping_epoch":202,"log_start":"214'3900","ondisk_log_start":"214'3900","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'8858","last_scrub_stamp":"2024-05-15T23:11:59.908918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:11:59.908918+0000","objects_scrubbed":8807,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:29.204022+0000","scrub_duration":1539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442171392,"num_objects":6747,"num_object_clones":0,"num_object_copies":13494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13939,"num_write_kb":661952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"258'13940","reported_seq":14202,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:10.932966+0000","last_change":"2024-05-15T23:12:56.131288+0000","last_active":"2024-05-15T23:13:10.932966+0000","last_peered":"2024-05-15T23:13:10.932966+0000","last_clean":"2024-05-15T23:13:10.932966+0000","last_became_active":"2024-05-15T23:10:41.791796+0000","last_became_peered":"2024-05-15T23:10:41.791796+0000","last_unstale":"2024-05-15T23:13:10.932966+0000","last_undegraded":"2024-05-15T23:13:10.932966+0000","last_fullsized":"2024-05-15T23:13:10.932966+0000","mapping_epoch":202,"log_start":"214'3900","ondisk_log_start":"214'3900","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'8356","last_scrub_stamp":"2024-05-15T23:11:53.172114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:11:53.172114+0000","objects_scrubbed":4799,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447873024,"num_objects":6834,"num_object_clones":0,"num_object_copies":13668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13940,"num_write_kb":664768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"258'13850","reported_seq":14151,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:12.512613+0000","last_change":"2024-05-15T23:12:55.282863+0000","last_active":"2024-05-15T23:13:12.512613+0000","last_peered":"2024-05-15T23:13:12.512613+0000","last_clean":"2024-05-15T23:13:12.512613+0000","last_became_active":"2024-05-15T23:10:41.794649+0000","last_became_peered":"2024-05-15T23:10:41.794649+0000","last_unstale":"2024-05-15T23:13:12.512613+0000","last_undegraded":"2024-05-15T23:13:12.512613+0000","last_fullsized":"2024-05-15T23:13:12.512613+0000","mapping_epoch":202,"log_start":"214'3800","ondisk_log_start":"214'3800","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12548","last_scrub_stamp":"2024-05-15T23:12:55.282531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:55.282531+0000","objects_scrubbed":8168,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:59.339140+0000","scrub_duration":1166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448266240,"num_objects":6840,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13850,"num_write_kb":662080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"258'13846","reported_seq":14072,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:09.491917+0000","last_change":"2024-05-15T23:12:37.885059+0000","last_active":"2024-05-15T23:13:09.491917+0000","last_peered":"2024-05-15T23:13:09.491917+0000","last_clean":"2024-05-15T23:13:09.491917+0000","last_became_active":"2024-05-15T23:10:41.793356+0000","last_became_peered":"2024-05-15T23:10:41.793356+0000","last_unstale":"2024-05-15T23:13:09.491917+0000","last_undegraded":"2024-05-15T23:13:09.491917+0000","last_fullsized":"2024-05-15T23:13:09.491917+0000","mapping_epoch":202,"log_start":"214'3800","ondisk_log_start":"214'3800","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'11462","last_scrub_stamp":"2024-05-15T23:12:37.884916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:37.884916+0000","objects_scrubbed":9564,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:05.736809+0000","scrub_duration":1718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466747392,"num_objects":7122,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13846,"num_write_kb":670976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"258'14081","reported_seq":14291,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:12.384137+0000","last_change":"2024-05-15T23:11:52.185824+0000","last_active":"2024-05-15T23:13:12.384137+0000","last_peered":"2024-05-15T23:13:12.384137+0000","last_clean":"2024-05-15T23:13:12.384137+0000","last_became_active":"2024-05-15T23:10:41.794942+0000","last_became_peered":"2024-05-15T23:10:41.794942+0000","last_unstale":"2024-05-15T23:13:12.384137+0000","last_undegraded":"2024-05-15T23:13:12.384137+0000","last_fullsized":"2024-05-15T23:13:12.384137+0000","mapping_epoch":202,"log_start":"214'4000","ondisk_log_start":"214'4000","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'8246","last_scrub_stamp":"2024-05-15T23:11:52.185650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:11:52.185650+0000","objects_scrubbed":8194,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446103552,"num_objects":6807,"num_object_clones":0,"num_object_copies":13614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14081,"num_write_kb":668416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"258'13862","reported_seq":14075,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:10.942455+0000","last_change":"2024-05-15T23:11:52.189382+0000","last_active":"2024-05-15T23:13:10.942455+0000","last_peered":"2024-05-15T23:13:10.942455+0000","last_clean":"2024-05-15T23:13:10.942455+0000","last_became_active":"2024-05-15T23:10:41.792820+0000","last_became_peered":"2024-05-15T23:10:41.792820+0000","last_unstale":"2024-05-15T23:13:10.942455+0000","last_undegraded":"2024-05-15T23:13:10.942455+0000","last_fullsized":"2024-05-15T23:13:10.942455+0000","mapping_epoch":202,"log_start":"214'3800","ondisk_log_start":"214'3800","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'8266","last_scrub_stamp":"2024-05-15T23:11:52.189078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:11:52.189078+0000","objects_scrubbed":8190,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2080,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452722688,"num_objects":6908,"num_object_clones":0,"num_object_copies":13816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13862,"num_write_kb":664640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"258'16004","reported_seq":16348,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:09.744760+0000","last_change":"2024-05-15T23:12:40.833253+0000","last_active":"2024-05-15T23:13:09.744760+0000","last_peered":"2024-05-15T23:13:09.744760+0000","last_clean":"2024-05-15T23:13:09.744760+0000","last_became_active":"2024-05-15T23:10:41.794131+0000","last_became_peered":"2024-05-15T23:10:41.794131+0000","last_unstale":"2024-05-15T23:13:09.744760+0000","last_undegraded":"2024-05-15T23:13:09.744760+0000","last_fullsized":"2024-05-15T23:13:09.744760+0000","mapping_epoch":202,"log_start":"222'6002","ondisk_log_start":"222'6002","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'13799","last_scrub_stamp":"2024-05-15T23:12:40.833117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:40.833117+0000","objects_scrubbed":9233,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:09.080677+0000","scrub_duration":1917,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455737344,"num_objects":6954,"num_object_clones":0,"num_object_copies":13908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13762,"num_write_kb":662912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"258'13805","reported_seq":14023,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:11.750936+0000","last_change":"2024-05-15T23:11:59.197199+0000","last_active":"2024-05-15T23:13:11.750936+0000","last_peered":"2024-05-15T23:13:11.750936+0000","last_clean":"2024-05-15T23:13:11.750936+0000","last_became_active":"2024-05-15T23:10:41.796378+0000","last_became_peered":"2024-05-15T23:10:41.796378+0000","last_unstale":"2024-05-15T23:13:11.750936+0000","last_undegraded":"2024-05-15T23:13:11.750936+0000","last_fullsized":"2024-05-15T23:13:11.750936+0000","mapping_epoch":202,"log_start":"214'3800","ondisk_log_start":"214'3800","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'8785","last_scrub_stamp":"2024-05-15T23:11:59.197105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:11:59.197105+0000","objects_scrubbed":8717,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:16.487303+0000","scrub_duration":1842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439025664,"num_objects":6699,"num_object_clones":0,"num_object_copies":13398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13805,"num_write_kb":656128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"258'13694","reported_seq":13907,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:10.025926+0000","last_change":"2024-05-15T23:11:52.359289+0000","last_active":"2024-05-15T23:13:10.025926+0000","last_peered":"2024-05-15T23:13:10.025926+0000","last_clean":"2024-05-15T23:13:10.025926+0000","last_became_active":"2024-05-15T23:10:41.792966+0000","last_became_peered":"2024-05-15T23:10:41.792966+0000","last_unstale":"2024-05-15T23:13:10.025926+0000","last_undegraded":"2024-05-15T23:13:10.025926+0000","last_fullsized":"2024-05-15T23:13:10.025926+0000","mapping_epoch":202,"log_start":"213'3600","ondisk_log_start":"213'3600","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'8147","last_scrub_stamp":"2024-05-15T23:11:52.359167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:11:52.359167+0000","objects_scrubbed":8094,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:12.848386+0000","scrub_duration":1688,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445120512,"num_objects":6792,"num_object_clones":0,"num_object_copies":13584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13694,"num_write_kb":655552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"258'13642","reported_seq":13864,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:09.466235+0000","last_change":"2024-05-15T23:12:07.628488+0000","last_active":"2024-05-15T23:13:09.466235+0000","last_peered":"2024-05-15T23:13:09.466235+0000","last_clean":"2024-05-15T23:13:09.466235+0000","last_became_active":"2024-05-15T23:10:41.794364+0000","last_became_peered":"2024-05-15T23:10:41.794364+0000","last_unstale":"2024-05-15T23:13:09.466235+0000","last_undegraded":"2024-05-15T23:13:09.466235+0000","last_fullsized":"2024-05-15T23:13:09.466235+0000","mapping_epoch":202,"log_start":"214'3600","ondisk_log_start":"214'3600","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'9275","last_scrub_stamp":"2024-05-15T23:12:07.628308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:07.628308+0000","objects_scrubbed":9240,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:18.947219+0000","scrub_duration":1442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459931648,"num_objects":7018,"num_object_clones":0,"num_object_copies":14036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13642,"num_write_kb":661120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"258'13743","reported_seq":13962,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:09.725851+0000","last_change":"2024-05-15T23:12:02.421561+0000","last_active":"2024-05-15T23:13:09.725851+0000","last_peered":"2024-05-15T23:13:09.725851+0000","last_clean":"2024-05-15T23:13:09.725851+0000","last_became_active":"2024-05-15T23:10:41.794291+0000","last_became_peered":"2024-05-15T23:10:41.794291+0000","last_unstale":"2024-05-15T23:13:09.725851+0000","last_undegraded":"2024-05-15T23:13:09.725851+0000","last_fullsized":"2024-05-15T23:13:09.725851+0000","mapping_epoch":202,"log_start":"214'3700","ondisk_log_start":"214'3700","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'9012","last_scrub_stamp":"2024-05-15T23:12:02.421414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:02.421414+0000","objects_scrubbed":8965,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:24.413488+0000","scrub_duration":1406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453312512,"num_objects":6917,"num_object_clones":0,"num_object_copies":13834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13743,"num_write_kb":661120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"258'14084","reported_seq":14307,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:12.407391+0000","last_change":"2024-05-15T23:12:07.892209+0000","last_active":"2024-05-15T23:13:12.407391+0000","last_peered":"2024-05-15T23:13:12.407391+0000","last_clean":"2024-05-15T23:13:12.407391+0000","last_became_active":"2024-05-15T23:10:42.014274+0000","last_became_peered":"2024-05-15T23:10:42.014274+0000","last_unstale":"2024-05-15T23:13:12.407391+0000","last_undegraded":"2024-05-15T23:13:12.407391+0000","last_fullsized":"2024-05-15T23:13:12.407391+0000","mapping_epoch":202,"log_start":"214'4000","ondisk_log_start":"214'4000","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'9456","last_scrub_stamp":"2024-05-15T23:12:07.892057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:07.892057+0000","objects_scrubbed":9414,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:30.995632+0000","scrub_duration":1441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447610880,"num_objects":6830,"num_object_clones":0,"num_object_copies":13660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14084,"num_write_kb":669248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"258'13791","reported_seq":14073,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:10.874569+0000","last_change":"2024-05-15T23:12:39.938806+0000","last_active":"2024-05-15T23:13:10.874569+0000","last_peered":"2024-05-15T23:13:10.874569+0000","last_clean":"2024-05-15T23:13:10.874569+0000","last_became_active":"2024-05-15T23:10:42.014285+0000","last_became_peered":"2024-05-15T23:10:42.014285+0000","last_unstale":"2024-05-15T23:13:10.874569+0000","last_undegraded":"2024-05-15T23:13:10.874569+0000","last_fullsized":"2024-05-15T23:13:10.874569+0000","mapping_epoch":202,"log_start":"213'3700","ondisk_log_start":"213'3700","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'11402","last_scrub_stamp":"2024-05-15T23:12:39.938642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:39.938642+0000","objects_scrubbed":9163,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:13:40.708767+0000","scrub_duration":2007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439549952,"num_objects":6707,"num_object_clones":0,"num_object_copies":13414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13791,"num_write_kb":655936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":511,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-05-15T23:13:08.323145+0000","last_change":"2024-05-15T23:12:43.939662+0000","last_active":"2024-05-15T23:13:08.323145+0000","last_peered":"2024-05-15T23:13:08.323145+0000","last_clean":"2024-05-15T23:13:08.323145+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:13:08.323145+0000","last_undegraded":"2024-05-15T23:13:08.323145+0000","last_fullsized":"2024-05-15T23:13:08.323145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:12:43.939526+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:12:43.939526+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:00.606486+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":7190609944,"num_objects":109721,"num_object_clones":0,"num_object_copies":219442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109721,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":221361,"num_write_kb":10594561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14394204160,"data_stored":14394196016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160869,"ondisk_log_size":160869,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351089,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3332092,"kb_used_data":3034356,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":90413060,"statfs":{"total":95995035648,"available":92582973440,"internally_reserved":0,"allocated":3107180544,"data_stored":3106828710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":304871876},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580022,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":63800,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93681352,"statfs":{"total":95995035648,"available":95929704448,"internally_reserved":0,"allocated":700416,"data_stored":352154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776254,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1077496,"kb_used_data":889452,"kb_used_omap":1,"kb_used_meta":188030,"kb_avail":92667656,"statfs":{"total":95995035648,"available":94891679744,"internally_reserved":0,"allocated":910798848,"data_stored":910450586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":192543168},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005186,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":587832,"kb_used_data":458092,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93157320,"statfs":{"total":95995035648,"available":95393095680,"internally_reserved":0,"allocated":469086208,"data_stored":468737946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019972,"kb_used_data":898552,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92725180,"statfs":{"total":95995035648,"available":94950584320,"internally_reserved":0,"allocated":920117248,"data_stored":919761362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":124320202},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495757,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":511796,"kb_used_data":440364,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93233356,"statfs":{"total":95995035648,"available":95470956544,"internally_reserved":0,"allocated":450932736,"data_stored":450584474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575014,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1489460,"kb_used_data":1336108,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92255692,"statfs":{"total":95995035648,"available":94469828608,"internally_reserved":0,"allocated":1368174592,"data_stored":1367826330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":157022662},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738539,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2023672,"kb_used_data":1793964,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":91721480,"statfs":{"total":95995035648,"available":93922795520,"internally_reserved":0,"allocated":1837019136,"data_stored":1836670874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":112376,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93632776,"statfs":{"total":95995035648,"available":95879962624,"internally_reserved":0,"allocated":700416,"data_stored":352154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2408628,"kb_used_data":2178924,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":91336524,"statfs":{"total":95995035648,"available":93528600576,"internally_reserved":0,"allocated":2231218176,"data_stored":2230869914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":235207111},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986325,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1931832,"kb_used_data":1796524,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":91813320,"statfs":{"total":95995035648,"available":94016839680,"internally_reserved":0,"allocated":1839640576,"data_stored":1839292314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790096,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1425848,"kb_used_data":1300272,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92319304,"statfs":{"total":95995035648,"available":94534967296,"internally_reserved":0,"allocated":1331478528,"data_stored":1331126194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1821966336,"data_stored":1821966336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1358626816,"data_stored":1358626816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2229731328,"data_stored":2229731328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1831796736,"data_stored":1831796736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1325666304,"data_stored":1325662232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":448135168,"data_stored":448135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":908333056,"data_stored":908328984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":466747392,"data_stored":466747392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":901120000,"data_stored":901120000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3102081024,"data_stored":3102081024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:13:15.212 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-15T23:13:15.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:15.320 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:15.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:15.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:15.455 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:13:15.514 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:16.055 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:16.073 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:13:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:16.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:13:16.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:13:16.225 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:16.247 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:13:16.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:16.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:13:16.351 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:16.381 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:16.405 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:16.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:16.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:16.509 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:16 smithi090 ceph-mon[44434]: pgmap v716: 17 pgs: 17 active+clean; 6.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:13:16.563 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:16.613 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:16 smithi084 ceph-mon[39784]: pgmap v716: 17 pgs: 17 active+clean; 6.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:13:16.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:16.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:16.644 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:13:16.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:16.771 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:16.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:16 smithi088 ceph-mon[44699]: pgmap v716: 17 pgs: 17 active+clean; 6.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:13:17.173 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:17.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:17.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:17.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:13:17.297 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:13:17.297 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:13:17.310 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:17.335 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:13:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:17.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:13:17.441 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:13:17.494 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:18.069 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:13:18.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:18.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:13:18.087 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T23:13:18.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:18.231 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:18.672 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:18 smithi084 ceph-mon[39784]: pgmap v717: 17 pgs: 17 active+clean; 6.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:13:18.690 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:18.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:18.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:18.743 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:18.793 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:18.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:18 smithi090 ceph-mon[44434]: pgmap v717: 17 pgs: 17 active+clean; 6.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:13:18.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:18 smithi088 ceph-mon[44699]: pgmap v717: 17 pgs: 17 active+clean; 6.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:13:18.840 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:19.620 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:19.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:19.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:19.726 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:19.767 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:19.788 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:19.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:19.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:19.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:19.933 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:19.962 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:19.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:20.066 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:20.103 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:20.300 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:20.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:20.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:20.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:13:20.456 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:20.650 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:20 smithi084 ceph-mon[39784]: pgmap v718: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-05-15T23:13:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:20 smithi090 ceph-mon[44434]: pgmap v718: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-05-15T23:13:20.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:20 smithi088 ceph-mon[44699]: pgmap v718: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-05-15T23:13:21.002 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:13:21.017 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:21.026 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:13:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:21.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:13:21.036 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:13:21.069 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:13:21.105 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:13:21.131 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:21.140 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:13:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:21.165 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:13:21.189 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:21.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:21.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:21.216 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:13:21.244 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:13:21.280 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:13:21.292 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:13:21.311 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:13:21.354 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:13:21.357 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:21.395 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:13:21.402 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:13:21.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:21.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:13:21.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:13:21.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:13:21.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:13:21.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:13:21.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:13:21.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:13:21.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:13:21.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:13:21.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:13:21.410 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:13:21.410 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:13:21.410 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:13:21.427 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:13:21.508 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:21.540 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:21 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:13:21.541 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:21 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:13:21.541 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:21 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2747841229' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:13:21.541 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:21.541 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:21.541 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:21 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3128729179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:13:21.543 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:21.564 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:21.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:21.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:21 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:13:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:21 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:13:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:21 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2747841229' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:13:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:21.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:21 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3128729179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:13:21.671 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:21.714 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:21.749 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:21.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:21 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:13:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:21 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:13:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:21 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2747841229' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:13:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:21.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:21.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:21 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3128729179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:13:21.855 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:13:21.893 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:21.918 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:13:21.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:21.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:13:22.021 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:22.095 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:22.365 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:22.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:22.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:22.475 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:13:22.516 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:22.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:22 smithi090 ceph-mon[44434]: pgmap v719: 17 pgs: 17 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:13:22.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:22 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:13:22.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3128729179' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:13:22.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:22 smithi090 ceph-mon[44434]: osdmap e259: 12 total, 12 up, 12 in 2024-05-15T23:13:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:22 smithi088 ceph-mon[44699]: pgmap v719: 17 pgs: 17 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:13:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:22 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:13:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3128729179' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:13:22.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:22 smithi088 ceph-mon[44699]: osdmap e259: 12 total, 12 up, 12 in 2024-05-15T23:13:22.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:22 smithi084 ceph-mon[39784]: pgmap v719: 17 pgs: 17 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:13:22.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:22 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:13:22.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:22.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3128729179' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:13:22.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:22 smithi084 ceph-mon[39784]: osdmap e259: 12 total, 12 up, 12 in 2024-05-15T23:13:23.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:13:23.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:13:23.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:23 smithi090 ceph-mon[44434]: osdmap e260: 12 total, 12 up, 12 in 2024-05-15T23:13:23.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:13:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:13:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:23 smithi088 ceph-mon[44699]: osdmap e260: 12 total, 12 up, 12 in 2024-05-15T23:13:23.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:13:23.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:13:23.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:23 smithi084 ceph-mon[39784]: osdmap e260: 12 total, 12 up, 12 in 2024-05-15T23:13:23.979 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:13:23.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:23.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:13:24.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:24.117 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:24.145 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:24.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:24.150 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:24.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:24.300 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:24.400 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:24.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:24.467 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:13:24.467 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:13:24.467 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-05-15T23:13:24.467 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:13:24.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:24.548 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:24.571 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:24.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:24.575 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:24.676 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:24.708 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:24 smithi090 ceph-mon[44434]: pgmap v722: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s 2024-05-15T23:13:24.711 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:24.734 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:24.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:24.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:24.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:24 smithi088 ceph-mon[44699]: pgmap v722: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s 2024-05-15T23:13:24.839 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:24.853 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:24 smithi084 ceph-mon[39784]: pgmap v722: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s 2024-05-15T23:13:24.869 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:24.905 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:24.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:24.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:25.011 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:25.060 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:25.086 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:25.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:25.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:25.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T23:13:25.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:25.224 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:25.259 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:25.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:25.263 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:25.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:13:25.393 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:25.422 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:13:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:25.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:13:25.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:13:25.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:25 smithi090 ceph-mon[44434]: 3.5 scrub starts 2024-05-15T23:13:25.556 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:25.577 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:13:25.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:25.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:13:25.583 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:25.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:13:25.680 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:25.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:25 smithi088 ceph-mon[44699]: 3.5 scrub starts 2024-05-15T23:13:25.715 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:25.741 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:25.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:25.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:25.848 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:25.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:25 smithi084 ceph-mon[39784]: 3.5 scrub starts 2024-05-15T23:13:25.879 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:25.896 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:13:25.900 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:25.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:26.004 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:26.044 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:26.070 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:26.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:26.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:26.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:26.203 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:26.226 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:26.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:26.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:26.335 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:26.369 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:26.389 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:26.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:26.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:13:26.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:26 smithi090 ceph-mon[44434]: pgmap v723: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:13:26.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:26 smithi090 ceph-mon[44434]: 3.5 scrub ok 2024-05-15T23:13:26.524 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:26 smithi090 ceph-mon[44434]: 3.b scrub starts 2024-05-15T23:13:26.527 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:26.558 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:13:26.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:26.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:13:26.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:26 smithi084 ceph-mon[39784]: pgmap v723: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:13:26.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:26 smithi084 ceph-mon[39784]: 3.5 scrub ok 2024-05-15T23:13:26.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:26 smithi084 ceph-mon[39784]: 3.b scrub starts 2024-05-15T23:13:26.663 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:13:26.694 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:26 smithi088 ceph-mon[44699]: pgmap v723: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:13:26.694 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:26 smithi088 ceph-mon[44699]: 3.5 scrub ok 2024-05-15T23:13:26.695 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:26 smithi088 ceph-mon[44699]: 3.b scrub starts 2024-05-15T23:13:26.697 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:26.723 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:13:26.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:26.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:13:26.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:27.316 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:27.324 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:27.347 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:27.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:27.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:27.452 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:13:27.501 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:27.534 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:13:27.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:27.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:13:27.643 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:13:27.673 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:27.710 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:13:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:27.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:13:27.815 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:27.865 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:28.539 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:13:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:13:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:13:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:13:28.540 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:13:28.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:28 smithi084 ceph-mon[39784]: pgmap v724: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s 2024-05-15T23:13:28.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:28 smithi084 ceph-mon[39784]: 3.b scrub ok 2024-05-15T23:13:28.775 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:28.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:28.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:28 smithi090 ceph-mon[44434]: pgmap v724: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s 2024-05-15T23:13:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:28 smithi090 ceph-mon[44434]: 3.b scrub ok 2024-05-15T23:13:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:28 smithi088 ceph-mon[44699]: pgmap v724: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s 2024-05-15T23:13:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:28 smithi088 ceph-mon[44699]: 3.b scrub ok 2024-05-15T23:13:28.880 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:13:28.910 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:28.935 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:13:28.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:28.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:13:29.041 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:13:29.072 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:29.102 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:13:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:29.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:13:29.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:13:29.244 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:29.271 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:13:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:29.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:13:29.376 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:13:29.422 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:29.477 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:13:29.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:29.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:13:29.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:29 smithi090 ceph-mon[44434]: 3.1 scrub starts 2024-05-15T23:13:29.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:29 smithi090 ceph-mon[44434]: 3.3 scrub starts 2024-05-15T23:13:29.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:13:29.649 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:29.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:29 smithi088 ceph-mon[44699]: 3.1 scrub starts 2024-05-15T23:13:29.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:29 smithi088 ceph-mon[44699]: 3.3 scrub starts 2024-05-15T23:13:29.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:29 smithi084 ceph-mon[39784]: 3.1 scrub starts 2024-05-15T23:13:29.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:29 smithi084 ceph-mon[39784]: 3.3 scrub starts 2024-05-15T23:13:30.019 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T23:13:30.125 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:13:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:30.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:13:30.231 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:30.266 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:30.283 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:30.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:30.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:30.387 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:13:30.422 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:30.444 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:13:30.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:30.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:13:30.548 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:30.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:30 smithi088 ceph-mon[44699]: 3.8 scrub ok 2024-05-15T23:13:30.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:30 smithi088 ceph-mon[44699]: pgmap v725: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:13:30.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:30 smithi088 ceph-mon[44699]: 3.1 scrub ok 2024-05-15T23:13:30.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:30 smithi088 ceph-mon[44699]: 3.c scrub starts 2024-05-15T23:13:30.595 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:30.619 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:30.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:30.623 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:30.724 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:30.736 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:30 smithi084 ceph-mon[39784]: 3.8 scrub ok 2024-05-15T23:13:30.736 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:30 smithi084 ceph-mon[39784]: pgmap v725: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:13:30.736 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:30 smithi084 ceph-mon[39784]: 3.1 scrub ok 2024-05-15T23:13:30.736 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:30 smithi084 ceph-mon[39784]: 3.c scrub starts 2024-05-15T23:13:30.753 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:30.773 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:30.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:30.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:30 smithi090 ceph-mon[44434]: 3.8 scrub ok 2024-05-15T23:13:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:30 smithi090 ceph-mon[44434]: pgmap v725: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:13:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:30 smithi090 ceph-mon[44434]: 3.1 scrub ok 2024-05-15T23:13:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:30 smithi090 ceph-mon[44434]: 3.c scrub starts 2024-05-15T23:13:30.877 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:13:30.907 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:30.941 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:13:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:30.945 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:13:31.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:13:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:31.099 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:13:31.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:31.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:13:31.161 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:13:31.203 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:31.210 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:13:31.235 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:31.246 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:13:31.264 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:31.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:31.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:31.280 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:13:31.321 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:13:31.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:13:31.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:31.389 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:13:31.408 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:31.421 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:13:31.428 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:31.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:31.438 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:31.452 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:13:31.489 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:13:31.505 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:31 smithi088 ceph-mon[44699]: 3.3 scrub ok 2024-05-15T23:13:31.505 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:31 smithi088 ceph-mon[44699]: 3.c scrub ok 2024-05-15T23:13:31.505 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:31 smithi088 ceph-mon[44699]: 3.7 scrub starts 2024-05-15T23:13:31.505 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1430549710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:13:31.517 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:13:31.539 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:31.561 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:13:31.570 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:31.571 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:13:31.571 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:13:31.571 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:13:31.571 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:13:31.571 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:13:31.571 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:13:31.571 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:13:31.571 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:13:31.572 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:13:31.572 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:13:31.572 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:13:31.572 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:13:31.598 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:31.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:31.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:31.703 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:13:31.757 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:31 smithi084 ceph-mon[39784]: 3.3 scrub ok 2024-05-15T23:13:31.757 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:31 smithi084 ceph-mon[39784]: 3.c scrub ok 2024-05-15T23:13:31.757 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:31 smithi084 ceph-mon[39784]: 3.7 scrub starts 2024-05-15T23:13:31.757 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1430549710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:13:31.761 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:31.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:31 smithi090 ceph-mon[44434]: 3.3 scrub ok 2024-05-15T23:13:31.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:31 smithi090 ceph-mon[44434]: 3.c scrub ok 2024-05-15T23:13:31.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:31 smithi090 ceph-mon[44434]: 3.7 scrub starts 2024-05-15T23:13:31.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1430549710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:13:31.850 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:13:31.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:31.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:13:31.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:13:31.998 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:32.026 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:13:32.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:32.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:13:32.134 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:32.164 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:32.187 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:32.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:32.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:13:32.345 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:32.801 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:32 smithi084 ceph-mon[39784]: pgmap v726: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:13:32.802 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.802 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.802 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.802 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.802 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.802 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.802 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.802 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.802 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.802 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.803 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.803 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.803 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:32 smithi084 ceph-mon[39784]: 3.4 scrub starts 2024-05-15T23:13:32.803 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:32 smithi084 ceph-mon[39784]: 3.7 scrub ok 2024-05-15T23:13:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:32 smithi090 ceph-mon[44434]: pgmap v726: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:13:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:32 smithi090 ceph-mon[44434]: 3.4 scrub starts 2024-05-15T23:13:32.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:32 smithi090 ceph-mon[44434]: 3.7 scrub ok 2024-05-15T23:13:32.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:32 smithi088 ceph-mon[44699]: pgmap v726: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:13:32.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:32.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:32 smithi088 ceph-mon[44699]: 3.4 scrub starts 2024-05-15T23:13:32.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:32 smithi088 ceph-mon[44699]: 3.7 scrub ok 2024-05-15T23:13:32.925 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:13:32.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:32.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:13:32.999 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:33.031 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:33.074 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:33.277 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:33.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:33.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:33.383 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:33.436 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:33.532 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:33.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:33.548 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:13:33.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:13:33.669 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:33.697 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:13:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:33.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:13:33.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:13:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:33 smithi090 ceph-mon[44434]: 3.4 scrub ok 2024-05-15T23:13:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:33 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/800015311' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:13:33.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:33 smithi088 ceph-mon[44699]: 3.4 scrub ok 2024-05-15T23:13:33.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:33 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/800015311' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:13:33.836 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:33.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:33 smithi084 ceph-mon[39784]: 3.4 scrub ok 2024-05-15T23:13:33.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:33 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/800015311' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:13:33.875 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:13:33.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:33.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:13:33.987 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:34.019 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:34.045 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:34.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:34.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:34.149 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:13:34.182 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:34.207 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:13:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:34.209 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:13:34.310 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:13:34.351 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:34.377 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:13:34.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:34.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:13:34.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:13:34.523 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:13:34.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:34.554 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:13:34.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:34 smithi084 ceph-mon[39784]: pgmap v727: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s 2024-05-15T23:13:34.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:34 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:13:34.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:34 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:13:34.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:34 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/800015311' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:13:34.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:34 smithi084 ceph-mon[39784]: osdmap e261: 12 total, 12 up, 12 in 2024-05-15T23:13:34.655 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:13:34.688 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:34.690 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:34 smithi088 ceph-mon[44699]: pgmap v727: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s 2024-05-15T23:13:34.690 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:34 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:13:34.690 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:34 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:13:34.691 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:34 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/800015311' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:13:34.691 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:34 smithi088 ceph-mon[44699]: osdmap e261: 12 total, 12 up, 12 in 2024-05-15T23:13:34.714 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:13:34.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:34.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:13:34.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:34 smithi090 ceph-mon[44434]: pgmap v727: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s 2024-05-15T23:13:34.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:34 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:13:34.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:34 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:13:34.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:34 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/800015311' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:13:34.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:34 smithi090 ceph-mon[44434]: osdmap e261: 12 total, 12 up, 12 in 2024-05-15T23:13:34.818 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:13:34.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:13:34.850 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:34.875 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:13:34.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:34.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:13:34.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:35.029 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:35.146 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:35.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:35.213 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-05-15T23:13:35.213 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-05-15T23:13:35.213 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-15T23:13:35.213 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.2 2024-05-15T23:13:35.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:13:35.286 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:35.309 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:13:35.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:35.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:13:35.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:35.451 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:35.457 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:13:35 smithi084 systemd[1]: Stopping Ceph osd.2 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:13:35.485 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:35.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:35.492 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:35.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:35.634 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:35.665 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:35.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:35.673 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:35.774 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:35.805 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:35.828 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:35.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:35.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:35.833 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:13:35.834 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:13:35.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:35.970 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:36.122 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:13:35 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-2[73385]: 2024-05-15T23:13:35.986+0000 7fdeb1dd5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:13:36.122 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:13:35 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-2[73385]: 2024-05-15T23:13:35.986+0000 7fdeb1dd5700 -1 osd.2 261 *** Got signal Terminated *** 2024-05-15T23:13:36.122 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:13:35 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-2[73385]: 2024-05-15T23:13:35.986+0000 7fdeb1dd5700 -1 osd.2 261 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:13:36.184 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:36.193 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:36.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:36.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:36.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:36.367 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:36.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:36.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:36.472 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:36.514 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:36.561 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:36.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:36.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:36.666 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:36.695 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:36.727 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:36.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:36.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:36 smithi084 ceph-mon[39784]: pgmap v729: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:13:36.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:36 smithi084 ceph-mon[39784]: 3.0 scrub starts 2024-05-15T23:13:36.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:36 smithi084 ceph-mon[39784]: osd.2 marked itself down and dead 2024-05-15T23:13:36.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:36 smithi090 ceph-mon[44434]: pgmap v729: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:13:36.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:36 smithi090 ceph-mon[44434]: 3.0 scrub starts 2024-05-15T23:13:36.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:36 smithi090 ceph-mon[44434]: osd.2 marked itself down and dead 2024-05-15T23:13:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:36 smithi088 ceph-mon[44699]: pgmap v729: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:13:36.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:36 smithi088 ceph-mon[44699]: 3.0 scrub starts 2024-05-15T23:13:36.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:36 smithi088 ceph-mon[44699]: osd.2 marked itself down and dead 2024-05-15T23:13:36.834 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:13:36.879 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:36.904 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:13:36.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:36.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:13:37.016 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:37.051 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:37.075 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:37.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:37.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:37.180 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:13:37.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:13:37.214 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:37.239 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:13:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:37.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:13:37.344 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:13:37.372 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:37.391 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:13:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:37.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:13:37.396 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T23:13:37.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:37.547 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:37.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:37 smithi084 ceph-mon[39784]: 3.0 scrub ok 2024-05-15T23:13:37.557 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:37 smithi084 ceph-mon[39784]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-15T23:13:37.557 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:37 smithi084 ceph-mon[39784]: osdmap e262: 12 total, 11 up, 12 in 2024-05-15T23:13:37.572 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:37.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:37.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:13:37.710 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:13:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:37.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:13:37.811 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:13:37.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:37 smithi088 ceph-mon[44699]: 3.0 scrub ok 2024-05-15T23:13:37.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:37 smithi088 ceph-mon[44699]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-15T23:13:37.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:37 smithi088 ceph-mon[44699]: osdmap e262: 12 total, 11 up, 12 in 2024-05-15T23:13:37.839 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:13:37.881 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:37.908 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:13:37.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:37.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:13:38.013 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:13:38.039 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:37 smithi090 ceph-mon[44434]: 3.0 scrub ok 2024-05-15T23:13:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:37 smithi090 ceph-mon[44434]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-15T23:13:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:37 smithi090 ceph-mon[44434]: osdmap e262: 12 total, 11 up, 12 in 2024-05-15T23:13:38.371 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:13:38 smithi084 bash[143712]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-2 2024-05-15T23:13:38.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:38 smithi088 ceph-mon[44699]: pgmap v731: 17 pgs: 17 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s 2024-05-15T23:13:38.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:38 smithi084 ceph-mon[39784]: pgmap v731: 17 pgs: 17 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s 2024-05-15T23:13:39.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:38 smithi090 ceph-mon[44434]: pgmap v731: 17 pgs: 17 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s 2024-05-15T23:13:39.498 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:13:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:39.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:13:39.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:39.620 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:39.660 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:39.731 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:13:39.774 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:13:39.813 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:13:39.838 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:13:39.871 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:13:39.905 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:13:39.931 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:13:39.962 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:13:40.003 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:13:40.042 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:13:40.075 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:13:40.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:13:40.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:13:40.076 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-05-15T23:13:40.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:13:40.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:13:40.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:13:40.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:13:40.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:13:40.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:13:40.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:13:40.077 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:13:40.077 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:13:40.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: pgmap v732: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-05-15T23:13:40.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2140719629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:13:40.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.567 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: pgmap v732: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-05-15T23:13:40.567 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2140719629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:13:40.567 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.567 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.567 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=53,logm=3} crc {auth=213311633,config=3313370793,health=3837777565,kv=2988908728,logm=2946944575}) 2024-05-15T23:13:40.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1364170095}) 2024-05-15T23:13:40.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1780885553}) 2024-05-15T23:13:40.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3266827154}) 2024-05-15T23:13:40.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1407075827}) 2024-05-15T23:13:40.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3915302827}) 2024-05-15T23:13:40.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=86624601}) 2024-05-15T23:13:40.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1273409153}) 2024-05-15T23:13:40.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2933815105}) 2024-05-15T23:13:40.568 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1432883674}) 2024-05-15T23:13:40.569 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2721550962}) 2024-05-15T23:13:40.569 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2199476607}) 2024-05-15T23:13:40.569 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2516775559}) 2024-05-15T23:13:40.569 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1121547100}) 2024-05-15T23:13:40.569 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=537628518}) 2024-05-15T23:13:40.569 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=615467093}) 2024-05-15T23:13:40.569 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3144439617}) 2024-05-15T23:13:40.569 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=687743846}) 2024-05-15T23:13:40.569 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2932681839}) 2024-05-15T23:13:40.569 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=584806940}) 2024-05-15T23:13:40.569 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1754184622}) 2024-05-15T23:13:40.569 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273407414}) 2024-05-15T23:13:40.569 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=678379763}) 2024-05-15T23:13:40.569 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1615304635}) 2024-05-15T23:13:40.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1993229269}) 2024-05-15T23:13:40.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3775378624}) 2024-05-15T23:13:40.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3400979435}) 2024-05-15T23:13:40.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=168382194}) 2024-05-15T23:13:40.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1269522871}) 2024-05-15T23:13:40.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3260536775}) 2024-05-15T23:13:40.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1496054833}) 2024-05-15T23:13:40.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2171225690}) 2024-05-15T23:13:40.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4151310979}) 2024-05-15T23:13:40.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=898383156}) 2024-05-15T23:13:40.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3618621887}) 2024-05-15T23:13:40.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3646502467}) 2024-05-15T23:13:40.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=87,mdsmap=3,mgr=10} crc {logm=1076480792,mdsmap=2840952586,mgr=2808086167}) 2024-05-15T23:13:40.570 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=79} crc {mgr=2146724547,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2034381531,mon_config_key=2055667162}) 2024-05-15T23:13:40.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1066237513}) 2024-05-15T23:13:40.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4191332992}) 2024-05-15T23:13:40.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=5,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=77} crc {mon_config_key=2750728387,monmap=1478146745,osd_metadata=1317173833,osd_pg_creating=4164115214,osdmap=2827546568}) 2024-05-15T23:13:40.571 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=46} crc {osdmap=2800123840}) 2024-05-15T23:13:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=53,logm=3} crc {auth=213311633,config=3313370793,health=3837777565,kv=2988908728,logm=2946944575}) 2024-05-15T23:13:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1364170095}) 2024-05-15T23:13:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1780885553}) 2024-05-15T23:13:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3266827154}) 2024-05-15T23:13:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1407075827}) 2024-05-15T23:13:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3915302827}) 2024-05-15T23:13:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=86624601}) 2024-05-15T23:13:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1273409153}) 2024-05-15T23:13:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2933815105}) 2024-05-15T23:13:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1432883674}) 2024-05-15T23:13:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2721550962}) 2024-05-15T23:13:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2199476607}) 2024-05-15T23:13:40.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2516775559}) 2024-05-15T23:13:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1121547100}) 2024-05-15T23:13:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=537628518}) 2024-05-15T23:13:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=615467093}) 2024-05-15T23:13:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3144439617}) 2024-05-15T23:13:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=687743846}) 2024-05-15T23:13:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2932681839}) 2024-05-15T23:13:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=584806940}) 2024-05-15T23:13:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1754184622}) 2024-05-15T23:13:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273407414}) 2024-05-15T23:13:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=678379763}) 2024-05-15T23:13:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1615304635}) 2024-05-15T23:13:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1993229269}) 2024-05-15T23:13:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3775378624}) 2024-05-15T23:13:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3400979435}) 2024-05-15T23:13:40.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=168382194}) 2024-05-15T23:13:40.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1269522871}) 2024-05-15T23:13:40.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3260536775}) 2024-05-15T23:13:40.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1496054833}) 2024-05-15T23:13:40.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2171225690}) 2024-05-15T23:13:40.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4151310979}) 2024-05-15T23:13:40.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=898383156}) 2024-05-15T23:13:40.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3618621887}) 2024-05-15T23:13:40.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3646502467}) 2024-05-15T23:13:40.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=87,mdsmap=3,mgr=10} crc {logm=1076480792,mdsmap=2840952586,mgr=2808086167}) 2024-05-15T23:13:40.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=79} crc {mgr=2146724547,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2034381531,mon_config_key=2055667162}) 2024-05-15T23:13:40.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1066237513}) 2024-05-15T23:13:40.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4191332992}) 2024-05-15T23:13:40.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=5,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=77} crc {mon_config_key=2750728387,monmap=1478146745,osd_metadata=1317173833,osd_pg_creating=4164115214,osdmap=2827546568}) 2024-05-15T23:13:40.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=46} crc {osdmap=2800123840}) 2024-05-15T23:13:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: pgmap v732: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-05-15T23:13:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2140719629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:13:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=53,logm=3} crc {auth=213311633,config=3313370793,health=3837777565,kv=2988908728,logm=2946944575}) 2024-05-15T23:13:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1364170095}) 2024-05-15T23:13:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1780885553}) 2024-05-15T23:13:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3266827154}) 2024-05-15T23:13:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1407075827}) 2024-05-15T23:13:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3915302827}) 2024-05-15T23:13:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=86624601}) 2024-05-15T23:13:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1273409153}) 2024-05-15T23:13:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2933815105}) 2024-05-15T23:13:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1432883674}) 2024-05-15T23:13:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2721550962}) 2024-05-15T23:13:40.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2199476607}) 2024-05-15T23:13:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2516775559}) 2024-05-15T23:13:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1121547100}) 2024-05-15T23:13:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=537628518}) 2024-05-15T23:13:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=615467093}) 2024-05-15T23:13:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3144439617}) 2024-05-15T23:13:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=687743846}) 2024-05-15T23:13:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2932681839}) 2024-05-15T23:13:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=584806940}) 2024-05-15T23:13:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1754184622}) 2024-05-15T23:13:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273407414}) 2024-05-15T23:13:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=678379763}) 2024-05-15T23:13:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1615304635}) 2024-05-15T23:13:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1993229269}) 2024-05-15T23:13:40.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3775378624}) 2024-05-15T23:13:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3400979435}) 2024-05-15T23:13:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=168382194}) 2024-05-15T23:13:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1269522871}) 2024-05-15T23:13:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3260536775}) 2024-05-15T23:13:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1496054833}) 2024-05-15T23:13:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2171225690}) 2024-05-15T23:13:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4151310979}) 2024-05-15T23:13:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=898383156}) 2024-05-15T23:13:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3618621887}) 2024-05-15T23:13:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3646502467}) 2024-05-15T23:13:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=87,mdsmap=3,mgr=10} crc {logm=1076480792,mdsmap=2840952586,mgr=2808086167}) 2024-05-15T23:13:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=79} crc {mgr=2146724547,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2034381531,mon_config_key=2055667162}) 2024-05-15T23:13:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1066237513}) 2024-05-15T23:13:40.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4191332992}) 2024-05-15T23:13:40.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=5,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=77} crc {mon_config_key=2750728387,monmap=1478146745,osd_metadata=1317173833,osd_pg_creating=4164115214,osdmap=2827546568}) 2024-05-15T23:13:40.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=46} crc {osdmap=2800123840}) 2024-05-15T23:13:41.420 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:41.426 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:13:41.426 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:13:41.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.427 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:258075 3.3 3:c19258d2:::benchmark_data_smithi090_90974_object92532:head [delete] snapc 0=[] ondisk+write+known_if_redirected e262)", 2024-05-15T23:13:41.427 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:13:41.412576+0000", 2024-05-15T23:13:41.427 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.0079638509999999992, 2024-05-15T23:13:41.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0079878989999999997, 2024-05-15T23:13:41.427 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:41.427 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:41.427 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:13:41.427 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:41.427 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:13:41.427 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:13:41.427 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 258075 2024-05-15T23:13:41.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.428 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:41.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:41.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.412576+0000", 2024-05-15T23:13:41.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:41.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:41.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.412576+0000", 2024-05-15T23:13:41.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:41.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:41.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.412578+0000", 2024-05-15T23:13:41.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.742e-06 2024-05-15T23:13:41.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:41.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.412582+0000", 2024-05-15T23:13:41.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6430000000000003e-06 2024-05-15T23:13:41.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:41.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.412591+0000", 2024-05-15T23:13:41.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.1230000000000006e-06 2024-05-15T23:13:41.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:41.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.412599+0000", 2024-05-15T23:13:41.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5109999999999997e-06 2024-05-15T23:13:41.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:41.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.412659+0000", 2024-05-15T23:13:41.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9608000000000003e-05 2024-05-15T23:13:41.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:41.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.412780+0000", 2024-05-15T23:13:41.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000121199 2024-05-15T23:13:41.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:13:41.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.412913+0000", 2024-05-15T23:13:41.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000133626 2024-05-15T23:13:41.432 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:41.432 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:41.432 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:41.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.433 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:258079 3.3 3:c16b7b88:::benchmark_data_smithi090_90974_object92536:head [delete] snapc 0=[] ondisk+write+known_if_redirected e262)", 2024-05-15T23:13:41.433 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:13:41.416251+0000", 2024-05-15T23:13:41.433 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.0042885340000000001, 2024-05-15T23:13:41.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0043638649999999998, 2024-05-15T23:13:41.433 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:41.433 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:41.433 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:13:41.434 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:41.434 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:13:41.434 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:13:41.434 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 258079 2024-05-15T23:13:41.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.434 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:41.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:41.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.416251+0000", 2024-05-15T23:13:41.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:41.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:41.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.416251+0000", 2024-05-15T23:13:41.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:41.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:41.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.416253+0000", 2024-05-15T23:13:41.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8920000000000001e-06 2024-05-15T23:13:41.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:41.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.416259+0000", 2024-05-15T23:13:41.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.339e-06 2024-05-15T23:13:41.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:41.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.416274+0000", 2024-05-15T23:13:41.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5121e-05 2024-05-15T23:13:41.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:41.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.416282+0000", 2024-05-15T23:13:41.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9500000000000001e-06 2024-05-15T23:13:41.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:41.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.416811+0000", 2024-05-15T23:13:41.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00052966199999999997 2024-05-15T23:13:41.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:41.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.416935+0000", 2024-05-15T23:13:41.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012338400000000001 2024-05-15T23:13:41.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:13:41.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.417054+0000", 2024-05-15T23:13:41.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000118956 2024-05-15T23:13:41.438 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:41.439 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:41.439 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:41.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.439 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.25135.0:258083 3.e e262/202 3:73420a66:::benchmark_data_smithi090_90974_object92540:head v 262'19595, mlcod=262'19593)", 2024-05-15T23:13:41.439 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:13:41.418557+0000", 2024-05-15T23:13:41.439 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.0019826449999999999, 2024-05-15T23:13:41.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0021014950000000001, 2024-05-15T23:13:41.439 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:41.439 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:41.439 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2024-05-15T23:13:41.440 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:41.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:41.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.418557+0000", 2024-05-15T23:13:41.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:41.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:41.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.418557+0000", 2024-05-15T23:13:41.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:41.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:41.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.418559+0000", 2024-05-15T23:13:41.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6440000000000001e-06 2024-05-15T23:13:41.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:41.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.418567+0000", 2024-05-15T23:13:41.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1669999999999999e-06 2024-05-15T23:13:41.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:41.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.418578+0000", 2024-05-15T23:13:41.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1499e-05 2024-05-15T23:13:41.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:41.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.418586+0000", 2024-05-15T23:13:41.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5719999999999997e-06 2024-05-15T23:13:41.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:41.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.418639+0000", 2024-05-15T23:13:41.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2553999999999999e-05 2024-05-15T23:13:41.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:41.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.418662+0000", 2024-05-15T23:13:41.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3223000000000001e-05 2024-05-15T23:13:41.443 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:41.443 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:41.443 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:41.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.443 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.25135.0:258085 3.1 e262/202 3:8daa7bac:::benchmark_data_smithi090_90974_object92542:head v 262'16154, mlcod=262'16152)", 2024-05-15T23:13:41.444 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:13:41.418934+0000", 2024-05-15T23:13:41.444 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.0016053720000000001, 2024-05-15T23:13:41.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001757567, 2024-05-15T23:13:41.444 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:41.444 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:41.444 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2024-05-15T23:13:41.444 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:41.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:41.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.418934+0000", 2024-05-15T23:13:41.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:41.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:41.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.418934+0000", 2024-05-15T23:13:41.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:41.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:41.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.418936+0000", 2024-05-15T23:13:41.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.19e-06 2024-05-15T23:13:41.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:41.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.418941+0000", 2024-05-15T23:13:41.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3929999999999999e-06 2024-05-15T23:13:41.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:41.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.418951+0000", 2024-05-15T23:13:41.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6369999999999994e-06 2024-05-15T23:13:41.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:41.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.418957+0000", 2024-05-15T23:13:41.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7909999999999999e-06 2024-05-15T23:13:41.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:41.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.419004+0000", 2024-05-15T23:13:41.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6097000000000003e-05 2024-05-15T23:13:41.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:41.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:41.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:41.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:13:41.419022+0000", 2024-05-15T23:13:41.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8658e-05 2024-05-15T23:13:41.447 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:41.447 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:41.447 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:41.447 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:41.447 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2024-05-15T23:13:41.447 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 4 2024-05-15T23:13:41.448 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:13:41.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:41.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:41.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:41.593 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:41.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:41.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:41.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:41.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:41.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:42.237 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T23:13:42.623 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:13:42.762 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:42.765 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:42 smithi084 ceph-mon[39784]: pgmap v733: 17 pgs: 17 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-05-15T23:13:42.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:42.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:42.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:42.907 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:42 smithi088 ceph-mon[44699]: pgmap v733: 17 pgs: 17 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-05-15T23:13:42.912 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:43.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:42 smithi090 ceph-mon[44434]: pgmap v733: 17 pgs: 17 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-05-15T23:13:43.130 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:43.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:43.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:43.158 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:13:43.159 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:13:43.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:43.281 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:43.322 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:43.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:43.429 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:43.481 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:43.730 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.2.service' 2024-05-15T23:13:43.741 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:43.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:43.798 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:13:43.798 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-15T23:13:43.798 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd down 2 2024-05-15T23:13:43.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:43.895 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:44.160 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:44.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:44.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:44.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:13:44.300 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:44.328 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:13:44.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:44.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:13:44.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:13:44.470 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:44.491 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:13:44.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:44.494 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:13:44.507 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:44.595 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:13:44.636 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:44 smithi088 ceph-mon[44699]: pgmap v734: 17 pgs: 17 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-05-15T23:13:44.647 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:44.694 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:13:44.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:44.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:13:44.704 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:13:44.704 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:13:44.704 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:13:44.704 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:13:44.704 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:13:44.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:13:44.834 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:44.857 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:13:44.859 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:44 smithi090 ceph-mon[44434]: pgmap v734: 17 pgs: 17 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-05-15T23:13:44.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:44.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:13:44.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:44 smithi084 ceph-mon[39784]: pgmap v734: 17 pgs: 17 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-05-15T23:13:44.961 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:45.012 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:45.108 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:45.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:45.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:45.160 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:13:45.216 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:13:45.251 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:45.456 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:13:45.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:45.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:13:45.463 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T23:13:45.559 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:45.613 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:45.621 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:13:45.637 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:45.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:45.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:45.742 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:13:45.778 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:45.795 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:13:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:45.798 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:13:45.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:45 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:13:45.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:45 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:13:45.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:45 smithi088 ceph-mon[44699]: osdmap e263: 12 total, 11 up, 12 in 2024-05-15T23:13:45.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3741677811' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:13:45.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:45 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:13:45.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:45 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:13:45.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:45 smithi084 ceph-mon[39784]: osdmap e263: 12 total, 11 up, 12 in 2024-05-15T23:13:45.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3741677811' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:13:45.862 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:13:45.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:45.956 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:45 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:13:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:45 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:13:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:45 smithi090 ceph-mon[44434]: osdmap e263: 12 total, 11 up, 12 in 2024-05-15T23:13:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3741677811' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:13:46.088 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:46.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:46.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:46.197 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:46.237 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:46.264 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:46.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:46.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:46.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:13:46.403 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:46.426 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:13:46.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:46.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:13:46.530 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:46.568 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:46.726 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:46.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:46 smithi090 ceph-mon[44434]: pgmap v736: 17 pgs: 17 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s 2024-05-15T23:13:46.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:46 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:13:46.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:46 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3741677811' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:13:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:46 smithi090 ceph-mon[44434]: osdmap e264: 12 total, 11 up, 12 in 2024-05-15T23:13:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:46 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:13:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:46 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:13:46.822 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:46.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:46 smithi088 ceph-mon[44699]: pgmap v736: 17 pgs: 17 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s 2024-05-15T23:13:46.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:46 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:13:46.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:46 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3741677811' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:13:46.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:46 smithi088 ceph-mon[44699]: osdmap e264: 12 total, 11 up, 12 in 2024-05-15T23:13:46.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:46 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:13:46.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:46 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:13:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:46.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:46.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:46.989 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:46 smithi084 ceph-mon[39784]: pgmap v736: 17 pgs: 17 active+clean; 4.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s 2024-05-15T23:13:46.990 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:46 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:13:46.990 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:46 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3741677811' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:13:46.990 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:46 smithi084 ceph-mon[39784]: osdmap e264: 12 total, 11 up, 12 in 2024-05-15T23:13:46.990 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:46 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:13:46.990 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:46 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:13:46.999 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:47.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2 is already down. 2024-05-15T23:13:47.262 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:47 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:13:47.066+0000 7fa0938b7700 -1 mon.a@0(leader).osd e265 definitely_dead 0 2024-05-15T23:13:47.468 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:47.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:47.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:47.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:13:47.609 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:13:47.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:47.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:13:47.652 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:47 smithi088 ceph-mon[44699]: osdmap e265: 12 total, 11 up, 12 in 2024-05-15T23:13:47.652 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:47 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1226587640' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-05-15T23:13:47.653 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:47 smithi088 ceph-mon[44699]: from='client.35417 ' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-05-15T23:13:47.742 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:13:47.778 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:47.801 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:13:47.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:47.807 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:13:47.908 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:13:47.949 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:47 smithi084 ceph-mon[39784]: osdmap e265: 12 total, 11 up, 12 in 2024-05-15T23:13:47.949 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:47 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1226587640' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-05-15T23:13:47.949 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:47 smithi084 ceph-mon[39784]: from='client.35417 ' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-05-15T23:13:47.952 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:48.013 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:13:48.013 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:13:48.014 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:13:48.014 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:13:48.014 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.014 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:260076 2.d 2:b949063d:::benchmark_data_smithi090_78160_object98718:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e164)", 2024-05-15T23:13:48.014 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:05.983807+0000", 2024-05-15T23:13:48.014 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 282.02579696100003, 2024-05-15T23:13:48.014 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15722057, 2024-05-15T23:13:48.014 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.014 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.014 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.014 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.014 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:13:48.014 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:13:48.015 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 260076 2024-05-15T23:13:48.015 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.015 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.015 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.015 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.015 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983807+0000", 2024-05-15T23:13:48.015 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.015 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.015 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.015 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.015 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983807+0000", 2024-05-15T23:13:48.015 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.015 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.015 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.015 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.016 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983810+0000", 2024-05-15T23:13:48.016 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.819e-06 2024-05-15T23:13:48.016 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.016 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.016 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.016 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983819+0000", 2024-05-15T23:13:48.016 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8270000000000004e-06 2024-05-15T23:13:48.016 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.016 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.016 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.016 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983834+0000", 2024-05-15T23:13:48.016 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5194000000000001e-05 2024-05-15T23:13:48.016 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.017 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.017 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.017 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983847+0000", 2024-05-15T23:13:48.017 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2447e-05 2024-05-15T23:13:48.017 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.017 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.017 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.017 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983937+0000", 2024-05-15T23:13:48.017 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0345000000000003e-05 2024-05-15T23:13:48.017 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.017 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.017 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:13:48.017 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983951+0000", 2024-05-15T23:13:48.017 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4389999999999999e-05 2024-05-15T23:13:48.018 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.018 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.018 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.018 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.125575+0000", 2024-05-15T23:13:48.018 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14162365199999999 2024-05-15T23:13:48.018 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.018 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.018 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.018 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.128147+0000", 2024-05-15T23:13:48.018 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025723819999999998 2024-05-15T23:13:48.018 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.018 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.018 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3,9", 2024-05-15T23:13:48.019 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.128256+0000", 2024-05-15T23:13:48.020 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00010836999999999999 2024-05-15T23:13:48.020 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.020 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.020 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.020 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.140453+0000", 2024-05-15T23:13:48.020 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.012197428999999999 2024-05-15T23:13:48.020 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.020 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.020 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.020 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.140473+0000", 2024-05-15T23:13:48.020 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9904000000000001e-05 2024-05-15T23:13:48.020 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.140829+0000", 2024-05-15T23:13:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00035547900000000002 2024-05-15T23:13:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.140896+0000", 2024-05-15T23:13:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7565999999999997e-05 2024-05-15T23:13:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141028+0000", 2024-05-15T23:13:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013176600000000001 2024-05-15T23:13:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.022 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.022 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.022 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.022 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.022 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:260078 2.d 2:b751e7be:::benchmark_data_smithi090_78160_object98720:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e164)", 2024-05-15T23:13:48.022 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:05.983869+0000", 2024-05-15T23:13:48.022 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 282.02573555399999, 2024-05-15T23:13:48.022 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15740128, 2024-05-15T23:13:48.022 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.022 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.022 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.022 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.022 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:13:48.022 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:13:48.023 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 260078 2024-05-15T23:13:48.023 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.023 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.023 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.023 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.023 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983869+0000", 2024-05-15T23:13:48.023 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.023 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.023 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.023 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.023 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983869+0000", 2024-05-15T23:13:48.023 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.023 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.023 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.024 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.024 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983870+0000", 2024-05-15T23:13:48.024 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1100000000000005e-07 2024-05-15T23:13:48.024 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.024 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.024 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.024 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983873+0000", 2024-05-15T23:13:48.024 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5149999999999998e-06 2024-05-15T23:13:48.024 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.024 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.024 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.025 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983880+0000", 2024-05-15T23:13:48.025 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8140000000000003e-06 2024-05-15T23:13:48.025 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.025 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.025 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.025 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983887+0000", 2024-05-15T23:13:48.025 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2989999999999999e-06 2024-05-15T23:13:48.025 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.025 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.025 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.025 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983990+0000", 2024-05-15T23:13:48.025 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000102845 2024-05-15T23:13:48.025 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.026 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.026 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:13:48.026 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983996+0000", 2024-05-15T23:13:48.026 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3300000000000004e-06 2024-05-15T23:13:48.026 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.026 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.026 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.026 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.131851+0000", 2024-05-15T23:13:48.026 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.14785506700000001 2024-05-15T23:13:48.026 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.026 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.026 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.026 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.131925+0000", 2024-05-15T23:13:48.026 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3078999999999996e-05 2024-05-15T23:13:48.026 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.027 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.027 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3,9", 2024-05-15T23:13:48.027 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.131997+0000", 2024-05-15T23:13:48.027 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2636e-05 2024-05-15T23:13:48.027 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.027 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.027 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.027 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.140704+0000", 2024-05-15T23:13:48.027 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0087072350000000007 2024-05-15T23:13:48.027 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.027 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.027 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.027 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.140721+0000", 2024-05-15T23:13:48.027 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6801999999999999e-05 2024-05-15T23:13:48.027 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.028 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.028 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.028 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141092+0000", 2024-05-15T23:13:48.028 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00037110700000000002 2024-05-15T23:13:48.028 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.028 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.028 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.028 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141153+0000", 2024-05-15T23:13:48.028 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0834000000000001e-05 2024-05-15T23:13:48.028 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.028 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.029 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.029 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141270+0000", 2024-05-15T23:13:48.029 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000116906 2024-05-15T23:13:48.029 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.029 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.029 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.029 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.029 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.029 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:260080 2.d 2:bbc57010:::benchmark_data_smithi090_78160_object98722:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e164)", 2024-05-15T23:13:48.029 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:05.983898+0000", 2024-05-15T23:13:48.029 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 282.02570664400002, 2024-05-15T23:13:48.029 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.157602879, 2024-05-15T23:13:48.029 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.030 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.030 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.030 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.030 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:13:48.030 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:13:48.030 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 260080 2024-05-15T23:13:48.030 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.030 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.030 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.030 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.030 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983898+0000", 2024-05-15T23:13:48.030 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.030 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.030 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.031 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.031 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983898+0000", 2024-05-15T23:13:48.031 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.031 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.031 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.031 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.031 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983898+0000", 2024-05-15T23:13:48.031 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2600000000000002e-07 2024-05-15T23:13:48.031 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.031 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.031 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.031 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983903+0000", 2024-05-15T23:13:48.031 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2729999999999996e-06 2024-05-15T23:13:48.031 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.032 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.032 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.032 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983909+0000", 2024-05-15T23:13:48.032 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.246e-06 2024-05-15T23:13:48.032 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.032 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.032 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.032 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.983915+0000", 2024-05-15T23:13:48.032 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3119999999999999e-06 2024-05-15T23:13:48.032 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.032 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.032 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.032 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.984026+0000", 2024-05-15T23:13:48.032 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000111121 2024-05-15T23:13:48.032 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.033 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.033 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:13:48.033 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.984035+0000", 2024-05-15T23:13:48.033 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5059999999999995e-06 2024-05-15T23:13:48.033 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.033 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.033 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.033 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.135545+0000", 2024-05-15T23:13:48.033 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15150972700000001 2024-05-15T23:13:48.033 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.033 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.033 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.033 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.135685+0000", 2024-05-15T23:13:48.033 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140292 2024-05-15T23:13:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3,9", 2024-05-15T23:13:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.135830+0000", 2024-05-15T23:13:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014494900000000001 2024-05-15T23:13:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.140772+0000", 2024-05-15T23:13:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049422040000000004 2024-05-15T23:13:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.034 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141042+0000", 2024-05-15T23:13:48.035 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00026987799999999998 2024-05-15T23:13:48.035 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.035 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.035 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.035 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141327+0000", 2024-05-15T23:13:48.035 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00028460600000000002 2024-05-15T23:13:48.035 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.035 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.035 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.035 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141388+0000", 2024-05-15T23:13:48.035 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1197000000000004e-05 2024-05-15T23:13:48.035 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.035 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.035 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.036 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.141501+0000", 2024-05-15T23:13:48.036 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000112842 2024-05-15T23:13:48.036 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.036 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.036 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.036 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.036 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.036 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280235 2.d 2:bc9a6377:::benchmark_data_smithi090_78160_object118877:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:13:48.036 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205436+0000", 2024-05-15T23:13:48.036 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 259.80416859899998, 2024-05-15T23:13:48.036 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.090379717999999998, 2024-05-15T23:13:48.036 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.036 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.036 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.037 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.037 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:13:48.037 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:13:48.037 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280235 2024-05-15T23:13:48.037 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.037 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.037 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.037 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.037 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205436+0000", 2024-05-15T23:13:48.037 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.037 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.037 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.037 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.037 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205436+0000", 2024-05-15T23:13:48.037 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.038 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.038 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.038 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.038 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205436+0000", 2024-05-15T23:13:48.038 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8100000000000003e-07 2024-05-15T23:13:48.038 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.038 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.038 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.038 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205439+0000", 2024-05-15T23:13:48.038 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3980000000000002e-06 2024-05-15T23:13:48.038 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.038 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.038 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.038 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205445+0000", 2024-05-15T23:13:48.039 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1240000000000002e-06 2024-05-15T23:13:48.039 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.039 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.039 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.039 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205450+0000", 2024-05-15T23:13:48.039 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7670000000000003e-06 2024-05-15T23:13:48.039 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.039 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.039 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.039 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205720+0000", 2024-05-15T23:13:48.039 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00027066699999999999 2024-05-15T23:13:48.040 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.040 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.040 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:13:48.040 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205723+0000", 2024-05-15T23:13:48.040 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9950000000000001e-06 2024-05-15T23:13:48.040 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.040 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.040 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.040 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.267133+0000", 2024-05-15T23:13:48.040 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.061409686999999998 2024-05-15T23:13:48.040 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.040 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.040 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.040 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.267257+0000", 2024-05-15T23:13:48.040 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000124232 2024-05-15T23:13:48.041 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.041 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.041 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:13:48.041 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.267381+0000", 2024-05-15T23:13:48.041 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012427099999999999 2024-05-15T23:13:48.041 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.041 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.041 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.041 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.277587+0000", 2024-05-15T23:13:48.041 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010205624999999999 2024-05-15T23:13:48.041 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.041 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.041 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.041 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295730+0000", 2024-05-15T23:13:48.041 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018142587000000002 2024-05-15T23:13:48.042 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.042 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.042 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.042 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295766+0000", 2024-05-15T23:13:48.042 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5855999999999997e-05 2024-05-15T23:13:48.042 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.042 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.042 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.042 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295815+0000", 2024-05-15T23:13:48.042 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9928e-05 2024-05-15T23:13:48.042 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.042 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.042 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.043 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.043 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.043 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280240 2.d 2:be3a8fd7:::benchmark_data_smithi090_78160_object118882:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:13:48.043 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205456+0000", 2024-05-15T23:13:48.043 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 259.80414817000002, 2024-05-15T23:13:48.043 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.090458063000000005, 2024-05-15T23:13:48.043 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.043 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.043 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.043 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.043 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:13:48.043 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:13:48.043 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280240 2024-05-15T23:13:48.043 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.043 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.044 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.044 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205456+0000", 2024-05-15T23:13:48.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.044 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.044 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.044 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205456+0000", 2024-05-15T23:13:48.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.044 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.044 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.044 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205457+0000", 2024-05-15T23:13:48.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1799999999999995e-07 2024-05-15T23:13:48.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.045 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205459+0000", 2024-05-15T23:13:48.045 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6230000000000001e-06 2024-05-15T23:13:48.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.045 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205465+0000", 2024-05-15T23:13:48.045 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4779999999999999e-06 2024-05-15T23:13:48.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.045 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205469+0000", 2024-05-15T23:13:48.045 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.369e-06 2024-05-15T23:13:48.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205735+0000", 2024-05-15T23:13:48.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00026536000000000002 2024-05-15T23:13:48.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:13:48.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205738+0000", 2024-05-15T23:13:48.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.999e-06 2024-05-15T23:13:48.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.271117+0000", 2024-05-15T23:13:48.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065379197999999999 2024-05-15T23:13:48.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.271186+0000", 2024-05-15T23:13:48.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9676000000000002e-05 2024-05-15T23:13:48.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:13:48.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.271263+0000", 2024-05-15T23:13:48.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6917999999999998e-05 2024-05-15T23:13:48.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.282609+0000", 2024-05-15T23:13:48.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011346078000000001 2024-05-15T23:13:48.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295860+0000", 2024-05-15T23:13:48.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013250131 2024-05-15T23:13:48.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295888+0000", 2024-05-15T23:13:48.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8734999999999998e-05 2024-05-15T23:13:48.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.049 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.049 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295914+0000", 2024-05-15T23:13:48.049 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5979999999999999e-05 2024-05-15T23:13:48.049 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.049 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.049 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.049 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.049 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280244 2.d 2:b735b614:::benchmark_data_smithi090_78160_object118886:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:13:48.049 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205475+0000", 2024-05-15T23:13:48.049 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 259.80412953000001, 2024-05-15T23:13:48.049 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.090560130000000003, 2024-05-15T23:13:48.049 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.050 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.050 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.050 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.050 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:13:48.050 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:13:48.050 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280244 2024-05-15T23:13:48.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.050 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.050 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205475+0000", 2024-05-15T23:13:48.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.050 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205475+0000", 2024-05-15T23:13:48.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205475+0000", 2024-05-15T23:13:48.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9700000000000002e-07 2024-05-15T23:13:48.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205477+0000", 2024-05-15T23:13:48.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.916e-06 2024-05-15T23:13:48.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205482+0000", 2024-05-15T23:13:48.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2800000000000003e-06 2024-05-15T23:13:48.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205486+0000", 2024-05-15T23:13:48.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1049999999999997e-06 2024-05-15T23:13:48.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205749+0000", 2024-05-15T23:13:48.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00026279 2024-05-15T23:13:48.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:13:48.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205752+0000", 2024-05-15T23:13:48.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.999e-06 2024-05-15T23:13:48.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.274237+0000", 2024-05-15T23:13:48.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068484766000000002 2024-05-15T23:13:48.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.274355+0000", 2024-05-15T23:13:48.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000117552 2024-05-15T23:13:48.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.054 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:13:48.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.274478+0000", 2024-05-15T23:13:48.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012311200000000001 2024-05-15T23:13:48.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.054 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.054 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.293186+0000", 2024-05-15T23:13:48.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018708460999999999 2024-05-15T23:13:48.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.054 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.054 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295984+0000", 2024-05-15T23:13:48.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027980489999999999 2024-05-15T23:13:48.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.055 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.055 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296011+0000", 2024-05-15T23:13:48.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6934999999999999e-05 2024-05-15T23:13:48.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.055 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.055 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296035+0000", 2024-05-15T23:13:48.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3768e-05 2024-05-15T23:13:48.055 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.055 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.055 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.056 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280297 2.d 2:b81e8b87:::benchmark_data_smithi090_78160_object118939:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:13:48.056 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205491+0000", 2024-05-15T23:13:48.056 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 259.80411282900002, 2024-05-15T23:13:48.056 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.090641848999999997, 2024-05-15T23:13:48.056 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.056 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.056 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.056 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.056 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:13:48.056 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:13:48.056 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280297 2024-05-15T23:13:48.056 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.056 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.056 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205491+0000", 2024-05-15T23:13:48.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205491+0000", 2024-05-15T23:13:48.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205492+0000", 2024-05-15T23:13:48.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8599999999999998e-07 2024-05-15T23:13:48.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205494+0000", 2024-05-15T23:13:48.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0169999999999999e-06 2024-05-15T23:13:48.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205498+0000", 2024-05-15T23:13:48.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8449999999999996e-06 2024-05-15T23:13:48.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205501+0000", 2024-05-15T23:13:48.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8289999999999999e-06 2024-05-15T23:13:48.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205764+0000", 2024-05-15T23:13:48.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00026292699999999998 2024-05-15T23:13:48.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:13:48.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205767+0000", 2024-05-15T23:13:48.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0369999999999999e-06 2024-05-15T23:13:48.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.277627+0000", 2024-05-15T23:13:48.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.071859982000000003 2024-05-15T23:13:48.060 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.060 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.060 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.060 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.277714+0000", 2024-05-15T23:13:48.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6911999999999997e-05 2024-05-15T23:13:48.060 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.060 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.060 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:13:48.060 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.277783+0000", 2024-05-15T23:13:48.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9808999999999995e-05 2024-05-15T23:13:48.060 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.060 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.060 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.060 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.293189+0000", 2024-05-15T23:13:48.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015405762999999999 2024-05-15T23:13:48.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.061 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.061 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.061 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296078+0000", 2024-05-15T23:13:48.061 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028887740000000002 2024-05-15T23:13:48.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.061 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.061 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.061 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296112+0000", 2024-05-15T23:13:48.061 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4139e-05 2024-05-15T23:13:48.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.061 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.061 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.061 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296133+0000", 2024-05-15T23:13:48.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.1328999999999999e-05 2024-05-15T23:13:48.062 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.062 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.062 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.062 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.062 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.062 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280303 2.d 2:b682f83f:::benchmark_data_smithi090_78160_object118945:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:13:48.062 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205504+0000", 2024-05-15T23:13:48.062 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 259.80410050199998, 2024-05-15T23:13:48.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.090740782000000006, 2024-05-15T23:13:48.062 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.062 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.062 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.062 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.063 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:13:48.063 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:13:48.063 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280303 2024-05-15T23:13:48.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.063 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205504+0000", 2024-05-15T23:13:48.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205504+0000", 2024-05-15T23:13:48.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205504+0000", 2024-05-15T23:13:48.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3099999999999999e-07 2024-05-15T23:13:48.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205506+0000", 2024-05-15T23:13:48.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.482e-06 2024-05-15T23:13:48.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205509+0000", 2024-05-15T23:13:48.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3589999999999999e-06 2024-05-15T23:13:48.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205512+0000", 2024-05-15T23:13:48.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.253e-06 2024-05-15T23:13:48.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205778+0000", 2024-05-15T23:13:48.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000265565 2024-05-15T23:13:48.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:13:48.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205783+0000", 2024-05-15T23:13:48.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2639999999999999e-06 2024-05-15T23:13:48.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.066 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.066 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.280629+0000", 2024-05-15T23:13:48.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.074845940999999999 2024-05-15T23:13:48.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.066 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.066 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.280715+0000", 2024-05-15T23:13:48.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5680999999999995e-05 2024-05-15T23:13:48.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.066 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:13:48.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.280796+0000", 2024-05-15T23:13:48.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1108999999999999e-05 2024-05-15T23:13:48.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295925+0000", 2024-05-15T23:13:48.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.01512958 2024-05-15T23:13:48.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296185+0000", 2024-05-15T23:13:48.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00025999600000000002 2024-05-15T23:13:48.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296219+0000", 2024-05-15T23:13:48.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3732999999999997e-05 2024-05-15T23:13:48.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296245+0000", 2024-05-15T23:13:48.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5488000000000001e-05 2024-05-15T23:13:48.068 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.068 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.068 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.068 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280320 2.d 2:b17ccbcd:::benchmark_data_smithi090_78160_object118962:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:13:48.068 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205515+0000", 2024-05-15T23:13:48.068 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 259.80408933299998, 2024-05-15T23:13:48.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.090875773000000007, 2024-05-15T23:13:48.069 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.069 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.069 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.069 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.069 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:13:48.069 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:13:48.069 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280320 2024-05-15T23:13:48.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.069 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.069 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205515+0000", 2024-05-15T23:13:48.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205515+0000", 2024-05-15T23:13:48.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205515+0000", 2024-05-15T23:13:48.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9400000000000001e-07 2024-05-15T23:13:48.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205517+0000", 2024-05-15T23:13:48.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3179999999999999e-06 2024-05-15T23:13:48.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205520+0000", 2024-05-15T23:13:48.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2880000000000001e-06 2024-05-15T23:13:48.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205523+0000", 2024-05-15T23:13:48.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7049999999999999e-06 2024-05-15T23:13:48.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205795+0000", 2024-05-15T23:13:48.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000272172 2024-05-15T23:13:48.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:13:48.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205799+0000", 2024-05-15T23:13:48.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5079999999999999e-06 2024-05-15T23:13:48.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.282655+0000", 2024-05-15T23:13:48.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.076855993999999997 2024-05-15T23:13:48.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.282780+0000", 2024-05-15T23:13:48.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000124289 2024-05-15T23:13:48.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:13:48.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.282900+0000", 2024-05-15T23:13:48.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012050899999999999 2024-05-15T23:13:48.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295938+0000", 2024-05-15T23:13:48.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013038320000000001 2024-05-15T23:13:48.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296305+0000", 2024-05-15T23:13:48.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00036618399999999998 2024-05-15T23:13:48.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296355+0000", 2024-05-15T23:13:48.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0559000000000002e-05 2024-05-15T23:13:48.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296391+0000", 2024-05-15T23:13:48.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5633000000000003e-05 2024-05-15T23:13:48.075 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.075 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.075 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.075 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.075 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.075 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280333 2.d 2:bf76b750:::benchmark_data_smithi090_78160_object118975:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:13:48.075 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205525+0000", 2024-05-15T23:13:48.075 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 259.80407894899997, 2024-05-15T23:13:48.075 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.090955432000000003, 2024-05-15T23:13:48.075 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.075 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.075 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.075 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.075 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:13:48.075 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:13:48.076 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280333 2024-05-15T23:13:48.076 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.076 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.076 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.076 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205525+0000", 2024-05-15T23:13:48.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.076 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.076 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.076 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205525+0000", 2024-05-15T23:13:48.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205526+0000", 2024-05-15T23:13:48.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8099999999999999e-07 2024-05-15T23:13:48.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205527+0000", 2024-05-15T23:13:48.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.57e-06 2024-05-15T23:13:48.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205531+0000", 2024-05-15T23:13:48.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5949999999999999e-06 2024-05-15T23:13:48.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205533+0000", 2024-05-15T23:13:48.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6450000000000001e-06 2024-05-15T23:13:48.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205820+0000", 2024-05-15T23:13:48.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000286581 2024-05-15T23:13:48.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:13:48.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205826+0000", 2024-05-15T23:13:48.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.699e-06 2024-05-15T23:13:48.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.284753+0000", 2024-05-15T23:13:48.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.078927319999999995 2024-05-15T23:13:48.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.284856+0000", 2024-05-15T23:13:48.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000102534 2024-05-15T23:13:48.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:13:48.080 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.284969+0000", 2024-05-15T23:13:48.080 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000113175 2024-05-15T23:13:48.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.080 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.080 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.080 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295945+0000", 2024-05-15T23:13:48.080 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010976067000000001 2024-05-15T23:13:48.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.080 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.080 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.080 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296430+0000", 2024-05-15T23:13:48.080 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00048503899999999999 2024-05-15T23:13:48.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.080 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.080 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296459+0000", 2024-05-15T23:13:48.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8742e-05 2024-05-15T23:13:48.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.081 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.296481+0000", 2024-05-15T23:13:48.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2184e-05 2024-05-15T23:13:48.081 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.081 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.081 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.081 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280340 2.d 2:ba3d05d6:::benchmark_data_smithi090_78160_object118982:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:13:48.081 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205536+0000", 2024-05-15T23:13:48.081 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 259.80406783000001, 2024-05-15T23:13:48.082 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10257904800000001, 2024-05-15T23:13:48.082 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.082 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.082 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.082 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.082 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:13:48.082 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:13:48.082 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280340 2024-05-15T23:13:48.082 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.082 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.082 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.082 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.082 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205536+0000", 2024-05-15T23:13:48.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205536+0000", 2024-05-15T23:13:48.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205537+0000", 2024-05-15T23:13:48.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1600000000000002e-07 2024-05-15T23:13:48.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205538+0000", 2024-05-15T23:13:48.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.291e-06 2024-05-15T23:13:48.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205541+0000", 2024-05-15T23:13:48.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2210000000000002e-06 2024-05-15T23:13:48.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205544+0000", 2024-05-15T23:13:48.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7580000000000001e-06 2024-05-15T23:13:48.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205848+0000", 2024-05-15T23:13:48.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000303751 2024-05-15T23:13:48.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:13:48.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205854+0000", 2024-05-15T23:13:48.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9569999999999999e-06 2024-05-15T23:13:48.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.286820+0000", 2024-05-15T23:13:48.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080966220000000005 2024-05-15T23:13:48.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.286927+0000", 2024-05-15T23:13:48.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000106597 2024-05-15T23:13:48.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:13:48.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.287037+0000", 2024-05-15T23:13:48.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011078000000000001 2024-05-15T23:13:48.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295951+0000", 2024-05-15T23:13:48.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0089135250000000003 2024-05-15T23:13:48.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.087 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.087 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.308004+0000", 2024-05-15T23:13:48.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.012053022 2024-05-15T23:13:48.087 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.087 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.087 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.308063+0000", 2024-05-15T23:13:48.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8640000000000001e-05 2024-05-15T23:13:48.087 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.087 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.308116+0000", 2024-05-15T23:13:48.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2970000000000003e-05 2024-05-15T23:13:48.088 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.088 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.088 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.088 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280398 2.d 2:b926c6fa:::benchmark_data_smithi090_78160_object119040:head [delete] snapc 0=[] ondisk+write+known_if_redirected e170)", 2024-05-15T23:13:48.088 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:28.205549+0000", 2024-05-15T23:13:48.088 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 259.80405517200001, 2024-05-15T23:13:48.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.102650407, 2024-05-15T23:13:48.088 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.088 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.088 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.088 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.089 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:13:48.089 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:13:48.089 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280398 2024-05-15T23:13:48.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.089 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.089 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.089 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.089 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205549+0000", 2024-05-15T23:13:48.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.089 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.089 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.089 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205549+0000", 2024-05-15T23:13:48.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205549+0000", 2024-05-15T23:13:48.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.41e-07 2024-05-15T23:13:48.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205551+0000", 2024-05-15T23:13:48.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.621e-06 2024-05-15T23:13:48.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205557+0000", 2024-05-15T23:13:48.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4600000000000002e-06 2024-05-15T23:13:48.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205562+0000", 2024-05-15T23:13:48.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8619999999999999e-06 2024-05-15T23:13:48.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205876+0000", 2024-05-15T23:13:48.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000313295 2024-05-15T23:13:48.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:13:48.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.205881+0000", 2024-05-15T23:13:48.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7379999999999999e-06 2024-05-15T23:13:48.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.288803+0000", 2024-05-15T23:13:48.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082921155999999996 2024-05-15T23:13:48.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.288910+0000", 2024-05-15T23:13:48.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00010718 2024-05-15T23:13:48.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:13:48.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.289026+0000", 2024-05-15T23:13:48.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000116693 2024-05-15T23:13:48.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.295957+0000", 2024-05-15T23:13:48.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0069305670000000003 2024-05-15T23:13:48.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.308144+0000", 2024-05-15T23:13:48.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.012187144 2024-05-15T23:13:48.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.308174+0000", 2024-05-15T23:13:48.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9439000000000001e-05 2024-05-15T23:13:48.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:28.308200+0000", 2024-05-15T23:13:48.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5911e-05 2024-05-15T23:13:48.094 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.094 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.094 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.095 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.25135.0:42481 3.1 e212/202 3:8634d1bd:::benchmark_data_smithi090_90974_object42480:head v 212'2684, mlcod=212'2682)", 2024-05-15T23:13:48.095 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:05.095348+0000", 2024-05-15T23:13:48.095 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 162.91425596900001, 2024-05-15T23:13:48.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073645720999999997, 2024-05-15T23:13:48.095 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.095 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.095 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.095 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.095 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.095 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.095348+0000", 2024-05-15T23:13:48.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.095348+0000", 2024-05-15T23:13:48.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.095349+0000", 2024-05-15T23:13:48.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2399999999999997e-07 2024-05-15T23:13:48.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.095421+0000", 2024-05-15T23:13:48.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1858000000000002e-05 2024-05-15T23:13:48.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.097 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165008+0000", 2024-05-15T23:13:48.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.069586711999999995 2024-05-15T23:13:48.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.097 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165019+0000", 2024-05-15T23:13:48.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1367e-05 2024-05-15T23:13:48.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.097 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165299+0000", 2024-05-15T23:13:48.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00027993099999999997 2024-05-15T23:13:48.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.098 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.098 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165321+0000", 2024-05-15T23:13:48.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2206000000000001e-05 2024-05-15T23:13:48.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.098 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.098 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.168961+0000", 2024-05-15T23:13:48.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036400920000000002 2024-05-15T23:13:48.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.098 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.098 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.168994+0000", 2024-05-15T23:13:48.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2731000000000002e-05 2024-05-15T23:13:48.099 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.099 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.099 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.099 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:42490 3.3 3:c12e4973:::benchmark_data_smithi090_90974_object42489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e212)", 2024-05-15T23:13:48.099 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:05.097523+0000", 2024-05-15T23:13:48.099 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 162.91208120300001, 2024-05-15T23:13:48.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.071948781000000003, 2024-05-15T23:13:48.099 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.099 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.099 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.099 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.100 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:13:48.100 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:13:48.100 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 42490 2024-05-15T23:13:48.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.100 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.097523+0000", 2024-05-15T23:13:48.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.097523+0000", 2024-05-15T23:13:48.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.097525+0000", 2024-05-15T23:13:48.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4950000000000001e-06 2024-05-15T23:13:48.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.097583+0000", 2024-05-15T23:13:48.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8448e-05 2024-05-15T23:13:48.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165001+0000", 2024-05-15T23:13:48.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.067417801999999999 2024-05-15T23:13:48.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165017+0000", 2024-05-15T23:13:48.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6470999999999999e-05 2024-05-15T23:13:48.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165134+0000", 2024-05-15T23:13:48.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000116842 2024-05-15T23:13:48.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165274+0000", 2024-05-15T23:13:48.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000139338 2024-05-15T23:13:48.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:13:48.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.165413+0000", 2024-05-15T23:13:48.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013978300000000001 2024-05-15T23:13:48.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.169342+0000", 2024-05-15T23:13:48.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0039291669999999999 2024-05-15T23:13:48.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.169386+0000", 2024-05-15T23:13:48.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3609999999999998e-05 2024-05-15T23:13:48.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.169429+0000", 2024-05-15T23:13:48.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2505e-05 2024-05-15T23:13:48.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:05.169472+0000", 2024-05-15T23:13:48.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3319999999999999e-05 2024-05-15T23:13:48.104 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.104 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.104 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.105 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:105648 3.3 3:c5b7db5c:::benchmark_data_smithi090_90974_object105647:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e225)", 2024-05-15T23:13:48.105 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:34.666818+0000", 2024-05-15T23:13:48.105 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 133.34278633400001, 2024-05-15T23:13:48.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075287804999999999, 2024-05-15T23:13:48.105 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.105 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.105 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.105 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.105 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:13:48.105 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:13:48.105 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 105648 2024-05-15T23:13:48.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.105 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:34.666818+0000", 2024-05-15T23:13:48.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:34.666818+0000", 2024-05-15T23:13:48.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:34.666820+0000", 2024-05-15T23:13:48.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.824e-06 2024-05-15T23:13:48.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:34.666921+0000", 2024-05-15T23:13:48.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000101105 2024-05-15T23:13:48.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:34.666931+0000", 2024-05-15T23:13:48.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0428000000000001e-05 2024-05-15T23:13:48.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:34.666938+0000", 2024-05-15T23:13:48.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4749999999999998e-06 2024-05-15T23:13:48.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.108 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:34.666979+0000", 2024-05-15T23:13:48.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1449999999999998e-05 2024-05-15T23:13:48.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.108 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:34.667092+0000", 2024-05-15T23:13:48.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000112687 2024-05-15T23:13:48.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.108 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:13:48.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:34.667206+0000", 2024-05-15T23:13:48.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000113686 2024-05-15T23:13:48.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:34.674827+0000", 2024-05-15T23:13:48.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0076214339999999998 2024-05-15T23:13:48.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:34.741999+0000", 2024-05-15T23:13:48.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.067171661999999993 2024-05-15T23:13:48.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:34.742058+0000", 2024-05-15T23:13:48.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9704999999999998e-05 2024-05-15T23:13:48.110 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.110 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.110 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:34.742106+0000", 2024-05-15T23:13:48.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7348999999999997e-05 2024-05-15T23:13:48.110 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.110 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.110 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.110 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.110 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:155079 3.3 3:cbcdd17b:::benchmark_data_smithi090_90974_object155078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:13:48.110 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:12:14.027240+0000", 2024-05-15T23:13:48.110 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 93.982364469000004, 2024-05-15T23:13:48.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075553051999999996, 2024-05-15T23:13:48.111 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.111 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.111 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.111 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.111 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:13:48.111 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:13:48.111 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 155079 2024-05-15T23:13:48.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.111 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.027240+0000", 2024-05-15T23:13:48.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.027240+0000", 2024-05-15T23:13:48.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.027242+0000", 2024-05-15T23:13:48.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.973e-06 2024-05-15T23:13:48.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.027302+0000", 2024-05-15T23:13:48.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0173999999999997e-05 2024-05-15T23:13:48.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.027313+0000", 2024-05-15T23:13:48.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1029999999999999e-05 2024-05-15T23:13:48.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.027319+0000", 2024-05-15T23:13:48.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7200000000000003e-06 2024-05-15T23:13:48.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.027381+0000", 2024-05-15T23:13:48.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1760000000000005e-05 2024-05-15T23:13:48.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.027528+0000", 2024-05-15T23:13:48.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000147744 2024-05-15T23:13:48.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:13:48.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.027617+0000", 2024-05-15T23:13:48.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.9234999999999995e-05 2024-05-15T23:13:48.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.034613+0000", 2024-05-15T23:13:48.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0069951559999999998 2024-05-15T23:13:48.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.115 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.115 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.102720+0000", 2024-05-15T23:13:48.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068106929999999996 2024-05-15T23:13:48.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.115 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.115 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.102756+0000", 2024-05-15T23:13:48.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6525e-05 2024-05-15T23:13:48.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.116 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.116 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.102793+0000", 2024-05-15T23:13:48.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6804999999999997e-05 2024-05-15T23:13:48.117 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.117 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.117 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.117 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:155081 3.3 3:c134dbab:::benchmark_data_smithi090_90974_object155080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:13:48.117 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:12:14.028965+0000", 2024-05-15T23:13:48.117 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 93.980639674000003, 2024-05-15T23:13:48.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073936462999999994, 2024-05-15T23:13:48.117 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.117 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.117 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.117 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.118 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:13:48.118 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:13:48.118 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 155081 2024-05-15T23:13:48.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.118 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.028965+0000", 2024-05-15T23:13:48.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.028965+0000", 2024-05-15T23:13:48.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.119 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.028966+0000", 2024-05-15T23:13:48.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.57e-06 2024-05-15T23:13:48.119 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.029031+0000", 2024-05-15T23:13:48.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5007999999999998e-05 2024-05-15T23:13:48.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.029042+0000", 2024-05-15T23:13:48.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0633e-05 2024-05-15T23:13:48.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.029049+0000", 2024-05-15T23:13:48.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.187e-06 2024-05-15T23:13:48.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.031088+0000", 2024-05-15T23:13:48.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020394039999999999 2024-05-15T23:13:48.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.121 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.121 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.031180+0000", 2024-05-15T23:13:48.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.1120999999999997e-05 2024-05-15T23:13:48.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.121 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:13:48.121 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.031326+0000", 2024-05-15T23:13:48.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000146298 2024-05-15T23:13:48.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.121 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.122 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.034776+0000", 2024-05-15T23:13:48.122 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034499610000000001 2024-05-15T23:13:48.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.122 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.122 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.102826+0000", 2024-05-15T23:13:48.122 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068049762 2024-05-15T23:13:48.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.122 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.122 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.102857+0000", 2024-05-15T23:13:48.122 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1560999999999998e-05 2024-05-15T23:13:48.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.123 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.123 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.102901+0000", 2024-05-15T23:13:48.123 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3958000000000002e-05 2024-05-15T23:13:48.123 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.123 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.123 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.123 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.123 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.123 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:155086 3.3 3:caf0214c:::benchmark_data_smithi090_90974_object155085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:13:48.123 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:12:14.037861+0000", 2024-05-15T23:13:48.123 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 93.971743420999999, 2024-05-15T23:13:48.123 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.081962179999999996, 2024-05-15T23:13:48.123 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.123 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.124 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.124 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.124 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:13:48.124 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:13:48.124 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 155086 2024-05-15T23:13:48.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.124 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.037861+0000", 2024-05-15T23:13:48.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.037861+0000", 2024-05-15T23:13:48.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.037863+0000", 2024-05-15T23:13:48.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.1720000000000001e-06 2024-05-15T23:13:48.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.037891+0000", 2024-05-15T23:13:48.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.828e-05 2024-05-15T23:13:48.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.037902+0000", 2024-05-15T23:13:48.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0678e-05 2024-05-15T23:13:48.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.037908+0000", 2024-05-15T23:13:48.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4110000000000001e-06 2024-05-15T23:13:48.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.037959+0000", 2024-05-15T23:13:48.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0161999999999998e-05 2024-05-15T23:13:48.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.038081+0000", 2024-05-15T23:13:48.127 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012285700000000001 2024-05-15T23:13:48.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:13:48.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.038225+0000", 2024-05-15T23:13:48.127 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014320700000000001 2024-05-15T23:13:48.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.042747+0000", 2024-05-15T23:13:48.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045225040000000001 2024-05-15T23:13:48.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.128 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.119717+0000", 2024-05-15T23:13:48.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.076970170000000004 2024-05-15T23:13:48.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.128 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.119773+0000", 2024-05-15T23:13:48.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5763999999999997e-05 2024-05-15T23:13:48.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.129 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.119823+0000", 2024-05-15T23:13:48.129 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9975000000000002e-05 2024-05-15T23:13:48.129 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.129 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.129 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.129 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.129 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.129 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:155090 3.3 3:c9d2b071:::benchmark_data_smithi090_90974_object155089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:13:48.129 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:12:14.046390+0000", 2024-05-15T23:13:48.129 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 93.963214050999994, 2024-05-15T23:13:48.129 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.077848137999999997, 2024-05-15T23:13:48.129 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.129 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.129 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.129 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.130 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:13:48.130 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:13:48.130 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 155090 2024-05-15T23:13:48.130 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.130 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.130 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.130 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.130 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.046390+0000", 2024-05-15T23:13:48.130 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.130 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.130 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.130 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.130 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.046390+0000", 2024-05-15T23:13:48.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.046392+0000", 2024-05-15T23:13:48.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6610000000000001e-06 2024-05-15T23:13:48.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.046430+0000", 2024-05-15T23:13:48.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7588e-05 2024-05-15T23:13:48.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.046441+0000", 2024-05-15T23:13:48.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1851000000000001e-05 2024-05-15T23:13:48.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.046450+0000", 2024-05-15T23:13:48.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3529999999999995e-06 2024-05-15T23:13:48.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.046505+0000", 2024-05-15T23:13:48.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5584999999999997e-05 2024-05-15T23:13:48.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.046620+0000", 2024-05-15T23:13:48.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000114923 2024-05-15T23:13:48.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:13:48.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.046766+0000", 2024-05-15T23:13:48.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145453 2024-05-15T23:13:48.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.051286+0000", 2024-05-15T23:13:48.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045200850000000001 2024-05-15T23:13:48.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.124133+0000", 2024-05-15T23:13:48.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.072847559000000006 2024-05-15T23:13:48.134 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.134 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.134 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.124188+0000", 2024-05-15T23:13:48.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5124999999999998e-05 2024-05-15T23:13:48.134 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.134 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.134 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.124238+0000", 2024-05-15T23:13:48.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9954999999999999e-05 2024-05-15T23:13:48.134 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.134 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.135 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.135 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.135 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.135 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:155094 3.3 3:ce603d19:::benchmark_data_smithi090_90974_object155093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:13:48.135 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:12:14.051753+0000", 2024-05-15T23:13:48.135 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 93.957851649000006, 2024-05-15T23:13:48.135 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.072614763999999998, 2024-05-15T23:13:48.135 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:13:48.135 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:13:48.135 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:13:48.135 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:13:48.135 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:13:48.135 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:13:48.135 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 155094 2024-05-15T23:13:48.135 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.136 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:13:48.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:13:48.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.051753+0000", 2024-05-15T23:13:48.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.136 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:13:48.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.051753+0000", 2024-05-15T23:13:48.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:13:48.136 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:13:48.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.051754+0000", 2024-05-15T23:13:48.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8160000000000001e-06 2024-05-15T23:13:48.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.137 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:13:48.137 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.051811+0000", 2024-05-15T23:13:48.137 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6808999999999998e-05 2024-05-15T23:13:48.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.137 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:13:48.137 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.051820+0000", 2024-05-15T23:13:48.137 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6659999999999997e-06 2024-05-15T23:13:48.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.137 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:13:48.137 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.051825+0000", 2024-05-15T23:13:48.138 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.468e-06 2024-05-15T23:13:48.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:13:48.138 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.051874+0000", 2024-05-15T23:13:48.138 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8369999999999999e-05 2024-05-15T23:13:48.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:13:48.138 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.051960+0000", 2024-05-15T23:13:48.138 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6218999999999996e-05 2024-05-15T23:13:48.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:13:48.138 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.052050+0000", 2024-05-15T23:13:48.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0261999999999997e-05 2024-05-15T23:13:48.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:13:48.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.055194+0000", 2024-05-15T23:13:48.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031433339999999998 2024-05-15T23:13:48.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:13:48.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.124289+0000", 2024-05-15T23:13:48.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.069095461999999996 2024-05-15T23:13:48.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:13:48.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.124333+0000", 2024-05-15T23:13:48.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3866999999999999e-05 2024-05-15T23:13:48.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:13:48.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:13:48.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:13:48.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:14.124367+0000", 2024-05-15T23:13:48.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4490999999999999e-05 2024-05-15T23:13:48.140 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.140 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.140 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.140 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:13:48.140 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:13:48.140 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:13:48.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:47 smithi090 ceph-mon[44434]: osdmap e265: 12 total, 11 up, 12 in 2024-05-15T23:13:48.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:47 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1226587640' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-05-15T23:13:48.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:47 smithi090 ceph-mon[44434]: from='client.35417 ' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-05-15T23:13:48.360 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 11, 9, 10] 2024-05-15T23:13:48.361 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd out 2 2024-05-15T23:13:48.794 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:48 smithi088 ceph-mon[44699]: pgmap v739: 17 pgs: 17 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:13:48.794 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:48 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:13:48.795 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:48 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:13:48.795 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:48 smithi088 ceph-mon[44699]: osdmap e266: 12 total, 11 up, 12 in 2024-05-15T23:13:48.824 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T23:13:49.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:48 smithi090 ceph-mon[44434]: pgmap v739: 17 pgs: 17 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:13:49.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:48 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:13:49.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:48 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:13:49.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:48 smithi090 ceph-mon[44434]: osdmap e266: 12 total, 11 up, 12 in 2024-05-15T23:13:49.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:48 smithi084 ceph-mon[39784]: pgmap v739: 17 pgs: 17 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:13:49.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:48 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:13:49.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:48 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:13:49.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:48 smithi084 ceph-mon[39784]: osdmap e266: 12 total, 11 up, 12 in 2024-05-15T23:13:49.189 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:13:49.293 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:49.297 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:13:49.312 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:13:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:49.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:13:49.418 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:49.451 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:49.474 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:49.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:49.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:49.577 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:49.612 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:49.714 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:49.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:49.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:13:49.858 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:49.886 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:13:49.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:49.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:13:49.991 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:50.045 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:50.090 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:50.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:50.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:50.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:50.238 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:50.262 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:50.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:50.295 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:13:50.329 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:13:50.360 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:13:50.375 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:13:50.386 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:13:50.408 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:50.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:13:50.449 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:13:50.484 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:13:50.517 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:13:50.543 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:13:50.589 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:13:50.614 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:13:50.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:50.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:13:50.621 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:13:50.621 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:13:50.621 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:13:50.621 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-05-15T23:13:50.622 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:13:50.622 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:13:50.622 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:13:50.622 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:13:50.622 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:13:50.622 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:13:50.622 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:13:50.622 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:13:50.622 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:13:50.721 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:50.757 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:50.872 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:50.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:50.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:50.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:50 smithi088 ceph-mon[44699]: pgmap v741: 17 pgs: 17 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:13:50.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3196260041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:13:50.976 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:51.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:50 smithi084 ceph-mon[39784]: pgmap v741: 17 pgs: 17 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:13:51.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3196260041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:13:51.022 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:51.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:50 smithi090 ceph-mon[44434]: pgmap v741: 17 pgs: 17 active+clean; 4.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:13:51.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3196260041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:13:51.287 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:13:51.287 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:13:51.796 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:51.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:51.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:51.806 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:13:51.806 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:13:51.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:13:51.906 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:51.942 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:51.942 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.943 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.965 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:13:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:51.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:13:51.981 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.982 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.982 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.982 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.982 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.982 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.982 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.982 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.982 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.982 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.982 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.996 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.996 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:51.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:13:52.013 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T23:13:52.069 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:13:52.103 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:52.136 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:13:52.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:52.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:13:52.241 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:13:52.283 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:52.323 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:13:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:52.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:13:52.392 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:13:52.435 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:13:52.491 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:52.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:52 smithi084 ceph-mon[39784]: pgmap v742: 17 pgs: 17 active+clean; 4.0 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-05-15T23:13:53.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:52 smithi090 ceph-mon[44434]: pgmap v742: 17 pgs: 17 active+clean; 4.0 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-05-15T23:13:53.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:52 smithi088 ceph-mon[44699]: pgmap v742: 17 pgs: 17 active+clean; 4.0 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-05-15T23:13:53.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:52 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:13:52.909+0000 7fa0938b7700 -1 mon.a@0(leader).osd e266 definitely_dead 0 2024-05-15T23:13:53.575 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:13:53.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:53.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:13:53.680 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:53.700 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.2. 2024-05-15T23:13:53.729 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:53.791 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:53.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:53.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:53.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:13:53.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:53 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:13:53.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:53 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:13:53.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:53 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/922049884' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-05-15T23:13:53.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:53 smithi084 ceph-mon[39784]: Client client.admin marked osd.2 out, after it was down for 16 seconds 2024-05-15T23:13:53.896 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:53.936 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:54.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:53 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:13:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:53 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:13:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:53 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/922049884' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-05-15T23:13:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:53 smithi090 ceph-mon[44434]: Client client.admin marked osd.2 out, after it was down for 16 seconds 2024-05-15T23:13:54.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:53 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:13:54.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:53 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:13:54.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:53 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/922049884' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-05-15T23:13:54.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:53 smithi088 ceph-mon[44699]: Client client.admin marked osd.2 out, after it was down for 16 seconds 2024-05-15T23:13:54.463 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:13:54.464 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:13:54.584 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-05-15T23:13:54.584 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi084.front.sepia.ceph.com 2024-05-15T23:13:54.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -n osd.2 --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-05-15T23:13:54.804 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:13:54.809 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:54 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:13:54.809 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:54 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:13:54.810 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:54 smithi084 ceph-mon[39784]: pgmap v743: 17 pgs: 17 active+clean; 3.8 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-05-15T23:13:54.810 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:54 smithi084 ceph-mon[39784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-15T23:13:54.810 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/922049884' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-05-15T23:13:54.810 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:54 smithi084 ceph-mon[39784]: osdmap e267: 12 total, 11 up, 11 in 2024-05-15T23:13:54.810 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:54 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:13:54.810 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:54 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:13:54.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:54.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:13:54.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:54 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:13:54.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:54 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:13:54.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:54 smithi088 ceph-mon[44699]: pgmap v743: 17 pgs: 17 active+clean; 3.8 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-05-15T23:13:54.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:54 smithi088 ceph-mon[44699]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-15T23:13:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/922049884' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-05-15T23:13:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:54 smithi088 ceph-mon[44699]: osdmap e267: 12 total, 11 up, 11 in 2024-05-15T23:13:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:54 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:13:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:54 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:13:54.861 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:13:54.911 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:13:54.942 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:55.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:54 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:13:55.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:54 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:13:55.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:54 smithi090 ceph-mon[44434]: pgmap v743: 17 pgs: 17 active+clean; 3.8 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-05-15T23:13:55.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:54 smithi090 ceph-mon[44434]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-15T23:13:55.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/922049884' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-05-15T23:13:55.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:54 smithi090 ceph-mon[44434]: osdmap e267: 12 total, 11 up, 11 in 2024-05-15T23:13:55.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:54 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:13:55.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:54 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:13:55.204 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/osd.2/config 2024-05-15T23:13:55.264 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T23:13:55.301 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:13:55.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:55.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:13:55.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:13:55.446 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:55.468 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:13:55.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:55.474 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:13:55.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:55.608 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:55.635 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:55.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:55.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:55.650 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:13:55.726 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:13:55.740 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:13:55.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:55 smithi084 ceph-mon[39784]: osdmap e268: 12 total, 11 up, 11 in 2024-05-15T23:13:55.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:55 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/933734409' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:13:55.781 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:55.809 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:13:55.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:55.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:13:55.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:13:55.948 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:55.974 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:13:55.975 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:55 smithi090 ceph-mon[44434]: osdmap e268: 12 total, 11 up, 11 in 2024-05-15T23:13:55.976 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:55 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/933734409' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:13:55.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:55.976 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:13:55.988 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:55 smithi088 ceph-mon[44699]: osdmap e268: 12 total, 11 up, 11 in 2024-05-15T23:13:55.989 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:55 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/933734409' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:13:56.077 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:13:56.114 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:56.193 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:13:56.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:56.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:13:56.297 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:13:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:56.590 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:13:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:56.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:13:56.694 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:13:56.738 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:56 smithi090 ceph-mon[44434]: pgmap v746: 17 pgs: 17 active+clean; 3.7 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 973 op/s 2024-05-15T23:13:57.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:56 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:13:57.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/933734409' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:13:57.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:56 smithi090 ceph-mon[44434]: osdmap e269: 12 total, 11 up, 11 in 2024-05-15T23:13:57.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:56 smithi088 ceph-mon[44699]: pgmap v746: 17 pgs: 17 active+clean; 3.7 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 973 op/s 2024-05-15T23:13:57.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:56 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:13:57.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/933734409' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:13:57.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:56 smithi088 ceph-mon[44699]: osdmap e269: 12 total, 11 up, 11 in 2024-05-15T23:13:57.099 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:13:57.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:57.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:13:57.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:56 smithi084 ceph-mon[39784]: pgmap v746: 17 pgs: 17 active+clean; 3.7 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 973 op/s 2024-05-15T23:13:57.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:56 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:13:57.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/933734409' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:13:57.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:56 smithi084 ceph-mon[39784]: osdmap e269: 12 total, 11 up, 11 in 2024-05-15T23:13:57.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:13:57.257 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:57.621 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:13:57.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:57.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:13:57.682 INFO:teuthology.orchestra.run.smithi084.stderr:Creating an OSD daemon form without an OSD FSID value 2024-05-15T23:13:57.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:13:57.762 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:57.785 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:13:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:57.790 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:13:57.805 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:13:57.805 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:13:57.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:13:57.931 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:57.959 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:13:57.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:57.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:13:58.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:13:58.108 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:58.151 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:13:58.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:13:58.153 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:13:58.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:13:58.296 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:13:58.591 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T23:13:58.696 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:13:58.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:13:58 smithi088 ceph-mon[44699]: pgmap v748: 17 pgs: 17 active+clean; 3.5 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:13:58.956 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:13:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:13:58 smithi090 ceph-mon[44434]: pgmap v748: 17 pgs: 17 active+clean; 3.5 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:13:59.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:13:58 smithi084 ceph-mon[39784]: pgmap v748: 17 pgs: 17 active+clean; 3.5 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:14:00.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:00 smithi088 ceph-mon[44699]: pgmap v749: 17 pgs: 17 active+clean; 3.4 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T23:14:00.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:14:01.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:00 smithi090 ceph-mon[44434]: pgmap v749: 17 pgs: 17 active+clean; 3.4 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T23:14:01.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:14:01.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:00 smithi084 ceph-mon[39784]: pgmap v749: 17 pgs: 17 active+clean; 3.4 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T23:14:01.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:14:01.202 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:14:01.202 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:14:01.271 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:01.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:01.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:01.413 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:01.499 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:01.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:01.603 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:01.636 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:01.986 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T23:14:02.002 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:14:02.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:02.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:14:02.108 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:14:02.144 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:02.167 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:14:02.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:02.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:14:02.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:02.346 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:02.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:02.359 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:14:02.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:14:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:02.522 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:14:02.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:02.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:14:02.629 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:02.685 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:02.719 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:02.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:02.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:02.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:14:02.860 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:02.888 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:14:02.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:02.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:14:02.943 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:02 smithi088 ceph-mon[44699]: pgmap v750: 17 pgs: 17 active+clean; 3.2 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:14:02.943 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:02 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:14:02.943 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:02 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:14:02.943 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:02 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:14:02.993 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:14:03.028 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:03.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:02 smithi090 ceph-mon[44434]: pgmap v750: 17 pgs: 17 active+clean; 3.2 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:14:03.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:02 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:14:03.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:02 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:14:03.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:02 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:14:03.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:02 smithi084 ceph-mon[39784]: pgmap v750: 17 pgs: 17 active+clean; 3.2 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:14:03.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:02 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:14:03.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:02 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:14:03.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:14:03.293 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:14:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:03.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:14:03.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:03.432 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:03.459 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:03.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:03.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:03.567 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:03.599 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:03.736 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:14:03.737 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:03 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:14:03.740 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:03.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:03.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:14:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:03.900 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:14:03.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:03.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:14:04.003 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:14:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:03 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:14:04.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:14:04.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:03 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:14:04.084 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:04.183 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:14:04.203 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:04.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:04.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:04.210 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:04.244 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:14:04.293 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:14:04.311 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:04.341 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:14:04.358 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:04.377 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:14:04.377 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:14:04.377 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:14:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:04.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:04.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:04.414 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:14:04.443 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:14:04.480 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:14:04.491 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:04.530 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:14:04.569 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:14:04.602 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:14:04.602 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:14:04.602 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:14:04.602 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-05-15T23:14:04.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:14:04.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:14:04.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:14:04.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:14:04.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:14:04.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:14:04.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:14:04.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:14:04.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:14:04.677 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:04.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:04.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:04.781 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:14:04.818 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:04 smithi084 ceph-mon[39784]: pgmap v751: 17 pgs: 17 active+clean; 3.0 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.40k op/s 2024-05-15T23:14:04.818 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:04 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:14:04.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:04 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:14:04.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:04 smithi084 ceph-mon[39784]: osdmap e270: 12 total, 11 up, 11 in 2024-05-15T23:14:04.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1266690952' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:04.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:04.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:04.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:04.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:04.836 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:05.051 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:14:05.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:04 smithi090 ceph-mon[44434]: pgmap v751: 17 pgs: 17 active+clean; 3.0 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.40k op/s 2024-05-15T23:14:05.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:04 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:14:05.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:04 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:14:05.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:04 smithi090 ceph-mon[44434]: osdmap e270: 12 total, 11 up, 11 in 2024-05-15T23:14:05.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1266690952' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:05.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:05.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:05.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:05.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:05.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:05.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:14:05.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:04 smithi088 ceph-mon[44699]: pgmap v751: 17 pgs: 17 active+clean; 3.0 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.40k op/s 2024-05-15T23:14:05.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:04 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:14:05.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:04 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:14:05.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:04 smithi088 ceph-mon[44699]: osdmap e270: 12 total, 11 up, 11 in 2024-05-15T23:14:05.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1266690952' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:05.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:05.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:05.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:05.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:05.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:05.185 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:05.209 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:05.210 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T23:14:05.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:05.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:05.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:05.346 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:05.374 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:05.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:05.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:05.480 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:05.542 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:05.580 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:05.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:05.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:05.598 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:14:05.687 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:05.743 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:05.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:05.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:05.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:05.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:05.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:05.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:05.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:05.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:05 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:14:05.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:05 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:14:06.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:06.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:06.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:06.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:06.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:06.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:06.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:06.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:05 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:14:06.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:05 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:14:06.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:06.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:06.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:06.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:06.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:06.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:06.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:06.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:05 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:14:06.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:05 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:14:06.600 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:06.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:06.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:06.639 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:14:06.640 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:14:06.715 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:06.748 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:06.780 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:06.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:06.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:06.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:06 smithi084 ceph-mon[39784]: pgmap v753: 17 pgs: 17 active+clean; 3.0 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:14:06.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:06 smithi084 ceph-mon[39784]: osdmap e271: 12 total, 11 up, 11 in 2024-05-15T23:14:06.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:06.915 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:06.951 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:06.954 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:07.008 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:06 smithi088 ceph-mon[44699]: pgmap v753: 17 pgs: 17 active+clean; 3.0 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:14:07.008 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:06 smithi088 ceph-mon[44699]: osdmap e271: 12 total, 11 up, 11 in 2024-05-15T23:14:07.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:06 smithi090 ceph-mon[44434]: pgmap v753: 17 pgs: 17 active+clean; 3.0 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:14:07.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:06 smithi090 ceph-mon[44434]: osdmap e271: 12 total, 11 up, 11 in 2024-05-15T23:14:07.056 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:07.099 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:07.124 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:07.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:07.131 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:07.231 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:07.280 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:07.386 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:07.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:07.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:07.413 INFO:tasks.thrashosds.thrasher:No PGs found for osd.2 2024-05-15T23:14:07.414 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-05-15T23:14:07.414 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-15T23:14:07.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:14:07.490 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:07.528 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:07.662 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:07.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:07.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:07.689 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:14:07.689 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:14:07.689 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:14:07.689 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:14:07.689 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.689 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16826 2.7 2:e191f622:::benchmark_data_smithi090_78160_object16825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.689 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.017535+0000", 2024-05-15T23:14:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.666663927, 2024-05-15T23:14:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.112188986, 2024-05-15T23:14:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16826 2024-05-15T23:14:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017535+0000", 2024-05-15T23:14:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017535+0000", 2024-05-15T23:14:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017538+0000", 2024-05-15T23:14:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0340000000000001e-06 2024-05-15T23:14:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017667+0000", 2024-05-15T23:14:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012813000000000001 2024-05-15T23:14:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017682+0000", 2024-05-15T23:14:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5699e-05 2024-05-15T23:14:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017694+0000", 2024-05-15T23:14:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2156e-05 2024-05-15T23:14:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.693 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.693 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017787+0000", 2024-05-15T23:14:07.693 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2972000000000004e-05 2024-05-15T23:14:07.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.693 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.693 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.693 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017809+0000", 2024-05-15T23:14:07.693 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2014e-05 2024-05-15T23:14:07.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.693 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.693 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.693 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.126600+0000", 2024-05-15T23:14:07.694 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.108790918 2024-05-15T23:14:07.694 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.694 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.694 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.694 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.126718+0000", 2024-05-15T23:14:07.694 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011804800000000001 2024-05-15T23:14:07.694 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.694 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.694 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.694 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.126855+0000", 2024-05-15T23:14:07.694 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000136546 2024-05-15T23:14:07.694 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.694 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.694 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.695 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129204+0000", 2024-05-15T23:14:07.695 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0023486689999999998 2024-05-15T23:14:07.695 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.695 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.695 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.695 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129254+0000", 2024-05-15T23:14:07.695 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0640999999999998e-05 2024-05-15T23:14:07.695 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.695 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.695 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.695 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129380+0000", 2024-05-15T23:14:07.695 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012590400000000001 2024-05-15T23:14:07.695 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.695 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.696 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.696 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129430+0000", 2024-05-15T23:14:07.696 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0371999999999998e-05 2024-05-15T23:14:07.696 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.696 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.696 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.696 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129724+0000", 2024-05-15T23:14:07.696 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00029388299999999999 2024-05-15T23:14:07.696 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.696 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.696 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.696 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.696 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.696 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16861 2.7 2:e22f6f28:::benchmark_data_smithi090_78160_object16860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.697 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.017713+0000", 2024-05-15T23:14:07.697 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.66648582699997, 2024-05-15T23:14:07.697 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.113245968, 2024-05-15T23:14:07.697 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.697 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.697 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.697 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.697 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.697 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.697 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16861 2024-05-15T23:14:07.697 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.697 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.697 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.697 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.698 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017713+0000", 2024-05-15T23:14:07.698 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.698 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.698 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.698 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.698 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017713+0000", 2024-05-15T23:14:07.698 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.698 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.698 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.698 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.698 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017714+0000", 2024-05-15T23:14:07.698 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1299999999999999e-07 2024-05-15T23:14:07.698 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.698 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.698 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.699 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017745+0000", 2024-05-15T23:14:07.699 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0784999999999997e-05 2024-05-15T23:14:07.699 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.699 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.699 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.699 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017751+0000", 2024-05-15T23:14:07.699 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6319999999999996e-06 2024-05-15T23:14:07.699 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.699 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.699 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.699 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017755+0000", 2024-05-15T23:14:07.699 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1880000000000004e-06 2024-05-15T23:14:07.699 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.699 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.700 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.700 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017844+0000", 2024-05-15T23:14:07.700 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.9328999999999998e-05 2024-05-15T23:14:07.700 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.700 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.700 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.700 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017864+0000", 2024-05-15T23:14:07.700 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0159000000000001e-05 2024-05-15T23:14:07.700 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.700 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.700 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.700 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.127841+0000", 2024-05-15T23:14:07.700 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10997657399999999 2024-05-15T23:14:07.700 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.700 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.701 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.701 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.127934+0000", 2024-05-15T23:14:07.701 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3522000000000003e-05 2024-05-15T23:14:07.701 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.701 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.701 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.701 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.128048+0000", 2024-05-15T23:14:07.701 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000113461 2024-05-15T23:14:07.701 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.701 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.701 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.701 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129335+0000", 2024-05-15T23:14:07.701 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0012868059999999999 2024-05-15T23:14:07.701 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.701 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.702 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.702 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130512+0000", 2024-05-15T23:14:07.702 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001176781 2024-05-15T23:14:07.702 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.702 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.702 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.702 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130637+0000", 2024-05-15T23:14:07.702 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012557399999999999 2024-05-15T23:14:07.702 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.702 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.702 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.702 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130673+0000", 2024-05-15T23:14:07.702 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5803000000000003e-05 2024-05-15T23:14:07.702 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.703 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.703 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.703 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130959+0000", 2024-05-15T23:14:07.703 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00028654100000000002 2024-05-15T23:14:07.703 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.703 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.703 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.703 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.703 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.703 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16878 2.7 2:eadea6ce:::benchmark_data_smithi090_78160_object16877:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.703 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.017762+0000", 2024-05-15T23:14:07.703 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.66643744800001, 2024-05-15T23:14:07.703 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.113672233, 2024-05-15T23:14:07.703 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.704 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.704 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.704 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.704 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.704 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.704 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16878 2024-05-15T23:14:07.704 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.704 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.704 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.704 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.704 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017762+0000", 2024-05-15T23:14:07.704 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.704 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.704 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.704 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.705 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017762+0000", 2024-05-15T23:14:07.705 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.705 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.705 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.705 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.705 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017762+0000", 2024-05-15T23:14:07.705 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7000000000000005e-07 2024-05-15T23:14:07.705 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.705 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.705 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.705 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017796+0000", 2024-05-15T23:14:07.705 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3522999999999997e-05 2024-05-15T23:14:07.705 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.705 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.706 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017806+0000", 2024-05-15T23:14:07.706 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5580000000000006e-06 2024-05-15T23:14:07.706 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.706 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.706 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017810+0000", 2024-05-15T23:14:07.706 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3059999999999999e-06 2024-05-15T23:14:07.706 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.706 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.706 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.706 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017909+0000", 2024-05-15T23:14:07.706 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9451999999999996e-05 2024-05-15T23:14:07.706 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.707 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.707 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.707 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017921+0000", 2024-05-15T23:14:07.707 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1572e-05 2024-05-15T23:14:07.707 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.707 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.707 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.707 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.128568+0000", 2024-05-15T23:14:07.707 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.110646964 2024-05-15T23:14:07.707 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.707 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.707 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.707 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.128663+0000", 2024-05-15T23:14:07.707 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5342999999999998e-05 2024-05-15T23:14:07.708 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.708 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.708 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.708 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.128781+0000", 2024-05-15T23:14:07.708 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000117476 2024-05-15T23:14:07.708 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.708 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.708 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.708 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130466+0000", 2024-05-15T23:14:07.708 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00168541 2024-05-15T23:14:07.708 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.708 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.708 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.709 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130599+0000", 2024-05-15T23:14:07.709 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000132605 2024-05-15T23:14:07.709 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.709 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.709 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.709 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.131005+0000", 2024-05-15T23:14:07.709 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00040684899999999998 2024-05-15T23:14:07.709 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.709 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.709 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.709 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.131047+0000", 2024-05-15T23:14:07.709 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1885999999999998e-05 2024-05-15T23:14:07.709 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.709 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.710 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.710 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.131434+0000", 2024-05-15T23:14:07.710 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000386719 2024-05-15T23:14:07.710 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.710 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.710 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.710 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.710 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.710 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16883 2.7 2:e347a603:::benchmark_data_smithi090_78160_object16882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.710 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.017816+0000", 2024-05-15T23:14:07.710 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.666383607, 2024-05-15T23:14:07.710 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.114700729, 2024-05-15T23:14:07.710 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.710 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.711 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.711 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.711 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.711 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.711 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16883 2024-05-15T23:14:07.711 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.711 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.711 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.711 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.711 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017816+0000", 2024-05-15T23:14:07.711 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.711 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.711 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.711 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.711 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017816+0000", 2024-05-15T23:14:07.712 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.712 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.712 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.712 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.712 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017816+0000", 2024-05-15T23:14:07.712 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4899999999999995e-07 2024-05-15T23:14:07.712 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.712 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.712 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.712 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017890+0000", 2024-05-15T23:14:07.712 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3910000000000002e-05 2024-05-15T23:14:07.712 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.712 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.712 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.713 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017899+0000", 2024-05-15T23:14:07.713 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3389999999999993e-06 2024-05-15T23:14:07.713 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.713 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.713 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.713 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017906+0000", 2024-05-15T23:14:07.713 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1389999999999998e-06 2024-05-15T23:14:07.713 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.713 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.713 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.713 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017950+0000", 2024-05-15T23:14:07.713 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3900999999999999e-05 2024-05-15T23:14:07.713 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.713 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.713 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.714 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017954+0000", 2024-05-15T23:14:07.714 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5859999999999998e-06 2024-05-15T23:14:07.714 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.714 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.714 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.714 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129772+0000", 2024-05-15T23:14:07.714 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.111818023 2024-05-15T23:14:07.714 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.714 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.714 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.714 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.129896+0000", 2024-05-15T23:14:07.714 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012374700000000001 2024-05-15T23:14:07.714 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.714 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.715 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.715 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.130011+0000", 2024-05-15T23:14:07.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011510600000000001 2024-05-15T23:14:07.715 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.715 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.715 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.715 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.131450+0000", 2024-05-15T23:14:07.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001439271 2024-05-15T23:14:07.715 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.715 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.715 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.715 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.131966+0000", 2024-05-15T23:14:07.715 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00051586000000000004 2024-05-15T23:14:07.715 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.715 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.716 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132045+0000", 2024-05-15T23:14:07.716 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9083000000000002e-05 2024-05-15T23:14:07.716 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.716 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132107+0000", 2024-05-15T23:14:07.716 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1853999999999994e-05 2024-05-15T23:14:07.716 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.716 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.716 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.716 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132516+0000", 2024-05-15T23:14:07.716 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00040926100000000001 2024-05-15T23:14:07.716 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.717 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.717 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.717 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.717 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.717 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16908 2.7 2:eabdad50:::benchmark_data_smithi090_78160_object16907:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.717 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.017916+0000", 2024-05-15T23:14:07.717 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.66628295300001, 2024-05-15T23:14:07.717 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11791069899999999, 2024-05-15T23:14:07.717 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.717 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.717 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.717 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.717 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.718 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.718 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16908 2024-05-15T23:14:07.718 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.718 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.718 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.718 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017916+0000", 2024-05-15T23:14:07.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.718 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.718 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.718 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.718 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017916+0000", 2024-05-15T23:14:07.718 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.719 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.719 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.719 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017917+0000", 2024-05-15T23:14:07.719 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0100000000000004e-07 2024-05-15T23:14:07.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.719 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.719 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.719 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017986+0000", 2024-05-15T23:14:07.719 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8436999999999995e-05 2024-05-15T23:14:07.719 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.719 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.719 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.017993+0000", 2024-05-15T23:14:07.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4719999999999998e-06 2024-05-15T23:14:07.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018000+0000", 2024-05-15T23:14:07.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5030000000000002e-06 2024-05-15T23:14:07.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.720 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.720 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018030+0000", 2024-05-15T23:14:07.720 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0035999999999998e-05 2024-05-15T23:14:07.720 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.720 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018036+0000", 2024-05-15T23:14:07.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5899999999999996e-06 2024-05-15T23:14:07.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132574+0000", 2024-05-15T23:14:07.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11453821 2024-05-15T23:14:07.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.721 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.721 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.721 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132718+0000", 2024-05-15T23:14:07.721 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014348599999999999 2024-05-15T23:14:07.721 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.132919+0000", 2024-05-15T23:14:07.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000200776 2024-05-15T23:14:07.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135329+0000", 2024-05-15T23:14:07.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024106230000000002 2024-05-15T23:14:07.722 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.722 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.722 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.722 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135392+0000", 2024-05-15T23:14:07.722 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3010999999999997e-05 2024-05-15T23:14:07.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135410+0000", 2024-05-15T23:14:07.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8062000000000001e-05 2024-05-15T23:14:07.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135465+0000", 2024-05-15T23:14:07.723 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4889000000000003e-05 2024-05-15T23:14:07.723 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.723 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.723 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.723 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135827+0000", 2024-05-15T23:14:07.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00036180300000000001 2024-05-15T23:14:07.724 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.724 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.724 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.724 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.724 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.724 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16916 2.7 2:e8125f82:::benchmark_data_smithi090_78160_object16915:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.724 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018012+0000", 2024-05-15T23:14:07.724 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.66618774199998, 2024-05-15T23:14:07.724 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.120212466, 2024-05-15T23:14:07.724 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.724 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.724 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.724 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.724 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16916 2024-05-15T23:14:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018012+0000", 2024-05-15T23:14:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018012+0000", 2024-05-15T23:14:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018012+0000", 2024-05-15T23:14:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3200000000000004e-07 2024-05-15T23:14:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018073+0000", 2024-05-15T23:14:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0924999999999997e-05 2024-05-15T23:14:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018081+0000", 2024-05-15T23:14:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8120000000000008e-06 2024-05-15T23:14:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018086+0000", 2024-05-15T23:14:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8860000000000003e-06 2024-05-15T23:14:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018117+0000", 2024-05-15T23:14:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0602000000000003e-05 2024-05-15T23:14:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018121+0000", 2024-05-15T23:14:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3139999999999997e-06 2024-05-15T23:14:07.727 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.133639+0000", 2024-05-15T23:14:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.115518282 2024-05-15T23:14:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.133762+0000", 2024-05-15T23:14:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012241699999999999 2024-05-15T23:14:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.133892+0000", 2024-05-15T23:14:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000130004 2024-05-15T23:14:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.729 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135838+0000", 2024-05-15T23:14:07.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019465039999999999 2024-05-15T23:14:07.729 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.729 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135910+0000", 2024-05-15T23:14:07.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1577999999999998e-05 2024-05-15T23:14:07.729 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.729 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.729 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.729 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.137826+0000", 2024-05-15T23:14:07.729 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019163439999999999 2024-05-15T23:14:07.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.730 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.730 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.137885+0000", 2024-05-15T23:14:07.730 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8805999999999999e-05 2024-05-15T23:14:07.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.730 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.730 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.730 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138224+0000", 2024-05-15T23:14:07.730 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00033916000000000003 2024-05-15T23:14:07.730 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.730 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.730 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.730 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.731 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16920 2.7 2:ecf12bd4:::benchmark_data_smithi090_78160_object16919:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.731 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018095+0000", 2024-05-15T23:14:07.731 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.66610399199999, 2024-05-15T23:14:07.731 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.120514865, 2024-05-15T23:14:07.731 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.731 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.731 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.731 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.731 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.731 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.731 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16920 2024-05-15T23:14:07.731 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.731 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.731 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018095+0000", 2024-05-15T23:14:07.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018095+0000", 2024-05-15T23:14:07.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.732 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.732 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.732 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018096+0000", 2024-05-15T23:14:07.732 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1699999999999997e-07 2024-05-15T23:14:07.732 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018222+0000", 2024-05-15T23:14:07.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000125718 2024-05-15T23:14:07.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018230+0000", 2024-05-15T23:14:07.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9300000000000003e-06 2024-05-15T23:14:07.733 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.733 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.733 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.733 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018235+0000", 2024-05-15T23:14:07.733 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1619999999999999e-06 2024-05-15T23:14:07.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018269+0000", 2024-05-15T23:14:07.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4529000000000001e-05 2024-05-15T23:14:07.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.734 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018274+0000", 2024-05-15T23:14:07.734 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5970000000000002e-06 2024-05-15T23:14:07.734 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.734 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.734 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.735 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.134501+0000", 2024-05-15T23:14:07.735 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.116226648 2024-05-15T23:14:07.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.735 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.735 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.735 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.134644+0000", 2024-05-15T23:14:07.735 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014388500000000001 2024-05-15T23:14:07.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.735 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.735 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.735 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.134829+0000", 2024-05-15T23:14:07.735 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000184279 2024-05-15T23:14:07.735 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.735 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.135983+0000", 2024-05-15T23:14:07.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0011546989999999999 2024-05-15T23:14:07.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.136000+0000", 2024-05-15T23:14:07.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6526999999999999e-05 2024-05-15T23:14:07.736 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.736 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.736 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.736 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138278+0000", 2024-05-15T23:14:07.736 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0022781310000000001 2024-05-15T23:14:07.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138316+0000", 2024-05-15T23:14:07.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7691999999999998e-05 2024-05-15T23:14:07.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.737 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.737 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138610+0000", 2024-05-15T23:14:07.737 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00029435099999999999 2024-05-15T23:14:07.737 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.737 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.737 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.737 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.737 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.738 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16933 2.7 2:e0794838:::benchmark_data_smithi090_78160_object16932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.738 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018244+0000", 2024-05-15T23:14:07.738 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.66595524399997, 2024-05-15T23:14:07.738 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12079975699999999, 2024-05-15T23:14:07.738 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.738 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.738 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.738 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.738 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.738 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.738 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16933 2024-05-15T23:14:07.738 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.738 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018244+0000", 2024-05-15T23:14:07.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018244+0000", 2024-05-15T23:14:07.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.739 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.739 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.739 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.739 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018245+0000", 2024-05-15T23:14:07.739 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9899999999999999e-07 2024-05-15T23:14:07.740 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.740 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.740 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.740 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018310+0000", 2024-05-15T23:14:07.740 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5549999999999994e-05 2024-05-15T23:14:07.740 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.740 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.740 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.740 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018317+0000", 2024-05-15T23:14:07.740 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.063e-06 2024-05-15T23:14:07.740 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.740 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.740 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.740 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018323+0000", 2024-05-15T23:14:07.741 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.04e-06 2024-05-15T23:14:07.741 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.741 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.741 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.741 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018350+0000", 2024-05-15T23:14:07.741 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7500000000000001e-05 2024-05-15T23:14:07.741 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.741 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.741 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.741 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018354+0000", 2024-05-15T23:14:07.741 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1500000000000001e-06 2024-05-15T23:14:07.741 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.741 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.741 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.742 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.136048+0000", 2024-05-15T23:14:07.742 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11769423900000001 2024-05-15T23:14:07.742 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.742 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.742 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.742 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.136216+0000", 2024-05-15T23:14:07.742 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000167236 2024-05-15T23:14:07.742 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.742 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.742 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.742 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.136400+0000", 2024-05-15T23:14:07.742 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018410599999999999 2024-05-15T23:14:07.742 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.743 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.743 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.743 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138622+0000", 2024-05-15T23:14:07.743 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0022219190000000002 2024-05-15T23:14:07.743 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.743 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.743 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.743 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138682+0000", 2024-05-15T23:14:07.743 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0417000000000001e-05 2024-05-15T23:14:07.743 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.743 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.743 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.744 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138733+0000", 2024-05-15T23:14:07.744 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0949999999999998e-05 2024-05-15T23:14:07.744 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.744 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.744 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.744 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.138768+0000", 2024-05-15T23:14:07.744 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4715000000000002e-05 2024-05-15T23:14:07.744 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.744 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.744 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.744 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139044+0000", 2024-05-15T23:14:07.744 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000276073 2024-05-15T23:14:07.744 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.745 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.745 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.745 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.745 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.745 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16935 2.7 2:e64d4890:::benchmark_data_smithi090_78160_object16934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.745 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018331+0000", 2024-05-15T23:14:07.745 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.66586810799998, 2024-05-15T23:14:07.745 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12112640099999999, 2024-05-15T23:14:07.745 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.745 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.745 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.745 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.745 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.745 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.746 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16935 2024-05-15T23:14:07.746 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.746 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.746 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.746 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.746 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018331+0000", 2024-05-15T23:14:07.746 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.746 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.746 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.746 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.746 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018331+0000", 2024-05-15T23:14:07.746 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.746 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.746 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.747 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.747 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018332+0000", 2024-05-15T23:14:07.747 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4900000000000005e-07 2024-05-15T23:14:07.747 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.747 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.747 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.747 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018394+0000", 2024-05-15T23:14:07.747 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1666000000000002e-05 2024-05-15T23:14:07.747 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.747 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.747 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.747 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018400+0000", 2024-05-15T23:14:07.747 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4999999999999996e-06 2024-05-15T23:14:07.747 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.748 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.748 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018404+0000", 2024-05-15T23:14:07.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2640000000000002e-06 2024-05-15T23:14:07.748 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.748 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.748 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018437+0000", 2024-05-15T23:14:07.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3002e-05 2024-05-15T23:14:07.748 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.748 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.748 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.748 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018442+0000", 2024-05-15T23:14:07.748 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3390000000000003e-06 2024-05-15T23:14:07.748 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.749 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.749 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.749 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.137007+0000", 2024-05-15T23:14:07.749 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.118565066 2024-05-15T23:14:07.749 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.749 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.749 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.749 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.137133+0000", 2024-05-15T23:14:07.749 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012647 2024-05-15T23:14:07.749 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.749 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.749 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.749 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.137305+0000", 2024-05-15T23:14:07.749 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017200399999999999 2024-05-15T23:14:07.750 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.750 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.750 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.750 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139055+0000", 2024-05-15T23:14:07.750 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001750049 2024-05-15T23:14:07.750 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.750 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.750 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.750 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139096+0000", 2024-05-15T23:14:07.750 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0997000000000001e-05 2024-05-15T23:14:07.750 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.750 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.750 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.750 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139142+0000", 2024-05-15T23:14:07.751 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5354e-05 2024-05-15T23:14:07.751 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.751 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.751 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.751 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139175+0000", 2024-05-15T23:14:07.751 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3386000000000002e-05 2024-05-15T23:14:07.751 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.751 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.751 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.751 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139458+0000", 2024-05-15T23:14:07.751 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00028255499999999998 2024-05-15T23:14:07.751 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.751 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.751 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.751 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.752 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.752 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16947 2.7 2:edcf61d7:::benchmark_data_smithi090_78160_object16946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.752 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018413+0000", 2024-05-15T23:14:07.752 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.66578647199998, 2024-05-15T23:14:07.752 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124689859, 2024-05-15T23:14:07.752 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.752 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.752 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.752 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.752 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.752 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.752 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16947 2024-05-15T23:14:07.752 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.752 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.752 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.753 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.753 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018413+0000", 2024-05-15T23:14:07.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.753 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.753 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.753 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.753 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018413+0000", 2024-05-15T23:14:07.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.753 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.753 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.753 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.753 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018414+0000", 2024-05-15T23:14:07.753 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0999999999999998e-07 2024-05-15T23:14:07.753 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.753 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.754 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.754 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018469+0000", 2024-05-15T23:14:07.754 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5869999999999999e-05 2024-05-15T23:14:07.754 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.754 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.754 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.754 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018476+0000", 2024-05-15T23:14:07.754 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.161e-06 2024-05-15T23:14:07.754 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.754 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.754 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.754 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018480+0000", 2024-05-15T23:14:07.754 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2069999999999997e-06 2024-05-15T23:14:07.754 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018510+0000", 2024-05-15T23:14:07.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0154999999999998e-05 2024-05-15T23:14:07.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018515+0000", 2024-05-15T23:14:07.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.074e-06 2024-05-15T23:14:07.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.755 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.755 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.755 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139510+0000", 2024-05-15T23:14:07.755 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.120994801 2024-05-15T23:14:07.755 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.756 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.756 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.756 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139649+0000", 2024-05-15T23:14:07.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013955200000000001 2024-05-15T23:14:07.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.756 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.756 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.756 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.139826+0000", 2024-05-15T23:14:07.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017705999999999999 2024-05-15T23:14:07.756 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.756 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.756 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.756 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142625+0000", 2024-05-15T23:14:07.756 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027985340000000001 2024-05-15T23:14:07.757 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.757 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.757 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.757 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142693+0000", 2024-05-15T23:14:07.757 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8279000000000001e-05 2024-05-15T23:14:07.757 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.757 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.757 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.757 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142747+0000", 2024-05-15T23:14:07.757 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3838999999999997e-05 2024-05-15T23:14:07.757 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.757 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.757 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.757 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142788+0000", 2024-05-15T23:14:07.758 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0448999999999998e-05 2024-05-15T23:14:07.758 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.758 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.758 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143103+0000", 2024-05-15T23:14:07.758 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000315168 2024-05-15T23:14:07.758 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.758 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.758 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.758 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.758 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.758 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16960 2.7 2:e8496105:::benchmark_data_smithi090_78160_object16959:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.758 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018488+0000", 2024-05-15T23:14:07.758 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.66571083000002, 2024-05-15T23:14:07.759 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.126728282, 2024-05-15T23:14:07.759 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.759 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.759 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.759 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.759 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.759 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.759 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16960 2024-05-15T23:14:07.759 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.759 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.759 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.759 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.759 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018488+0000", 2024-05-15T23:14:07.760 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.760 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.760 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.760 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.760 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018488+0000", 2024-05-15T23:14:07.760 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.760 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.760 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.760 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.760 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018489+0000", 2024-05-15T23:14:07.760 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5199999999999996e-07 2024-05-15T23:14:07.760 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.761 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.761 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.761 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018549+0000", 2024-05-15T23:14:07.761 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9521999999999997e-05 2024-05-15T23:14:07.761 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.761 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.761 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.761 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018555+0000", 2024-05-15T23:14:07.761 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3620000000000002e-06 2024-05-15T23:14:07.761 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.761 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.761 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.761 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018559+0000", 2024-05-15T23:14:07.761 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1479999999999999e-06 2024-05-15T23:14:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018588+0000", 2024-05-15T23:14:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8538999999999999e-05 2024-05-15T23:14:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018592+0000", 2024-05-15T23:14:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1309999999999999e-06 2024-05-15T23:14:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.140349+0000", 2024-05-15T23:14:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.121757411 2024-05-15T23:14:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.140492+0000", 2024-05-15T23:14:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014264200000000001 2024-05-15T23:14:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.140606+0000", 2024-05-15T23:14:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000114437 2024-05-15T23:14:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.763 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143118+0000", 2024-05-15T23:14:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025118330000000002 2024-05-15T23:14:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143168+0000", 2024-05-15T23:14:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9667999999999999e-05 2024-05-15T23:14:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.144796+0000", 2024-05-15T23:14:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016276979999999999 2024-05-15T23:14:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.144849+0000", 2024-05-15T23:14:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3050000000000002e-05 2024-05-15T23:14:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145217+0000", 2024-05-15T23:14:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00036808899999999999 2024-05-15T23:14:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.765 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16971 2.7 2:ead74483:::benchmark_data_smithi090_78160_object16970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018569+0000", 2024-05-15T23:14:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.66563027000001, 2024-05-15T23:14:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12710465400000001, 2024-05-15T23:14:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16971 2024-05-15T23:14:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.766 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018569+0000", 2024-05-15T23:14:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018569+0000", 2024-05-15T23:14:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018570+0000", 2024-05-15T23:14:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1399999999999996e-07 2024-05-15T23:14:07.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.768 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.768 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.768 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018621+0000", 2024-05-15T23:14:07.768 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1483000000000002e-05 2024-05-15T23:14:07.768 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.768 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.768 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.768 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018627+0000", 2024-05-15T23:14:07.768 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2020000000000001e-06 2024-05-15T23:14:07.768 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.768 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.768 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.768 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018631+0000", 2024-05-15T23:14:07.768 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9149999999999998e-06 2024-05-15T23:14:07.768 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018662+0000", 2024-05-15T23:14:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0599000000000003e-05 2024-05-15T23:14:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018668+0000", 2024-05-15T23:14:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0990000000000004e-06 2024-05-15T23:14:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.769 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.770 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.141123+0000", 2024-05-15T23:14:07.770 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12245492700000001 2024-05-15T23:14:07.770 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.141282+0000", 2024-05-15T23:14:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015872099999999999 2024-05-15T23:14:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.141465+0000", 2024-05-15T23:14:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00018367399999999999 2024-05-15T23:14:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.771 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143123+0000", 2024-05-15T23:14:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016577300000000001 2024-05-15T23:14:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143216+0000", 2024-05-15T23:14:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3317000000000001e-05 2024-05-15T23:14:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145289+0000", 2024-05-15T23:14:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020730890000000002 2024-05-15T23:14:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.772 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145338+0000", 2024-05-15T23:14:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8353000000000003e-05 2024-05-15T23:14:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145674+0000", 2024-05-15T23:14:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00033583099999999998 2024-05-15T23:14:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16975 2.7 2:e5d20b63:::benchmark_data_smithi090_78160_object16974:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018639+0000", 2024-05-15T23:14:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.665560723, 2024-05-15T23:14:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12747009100000001, 2024-05-15T23:14:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16975 2024-05-15T23:14:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018639+0000", 2024-05-15T23:14:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018639+0000", 2024-05-15T23:14:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018639+0000", 2024-05-15T23:14:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1600000000000001e-07 2024-05-15T23:14:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.775 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018690+0000", 2024-05-15T23:14:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1094000000000002e-05 2024-05-15T23:14:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018696+0000", 2024-05-15T23:14:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7540000000000003e-06 2024-05-15T23:14:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018700+0000", 2024-05-15T23:14:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9330000000000003e-06 2024-05-15T23:14:07.776 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018728+0000", 2024-05-15T23:14:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7447e-05 2024-05-15T23:14:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018732+0000", 2024-05-15T23:14:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.126e-06 2024-05-15T23:14:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.141950+0000", 2024-05-15T23:14:07.777 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12321836799999999 2024-05-15T23:14:07.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.778 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.778 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.778 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142073+0000", 2024-05-15T23:14:07.778 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012294000000000001 2024-05-15T23:14:07.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.778 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.778 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.778 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.142192+0000", 2024-05-15T23:14:07.778 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000118751 2024-05-15T23:14:07.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.778 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143227+0000", 2024-05-15T23:14:07.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010353199999999999 2024-05-15T23:14:07.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143266+0000", 2024-05-15T23:14:07.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9242999999999997e-05 2024-05-15T23:14:07.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.779 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.779 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.779 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145771+0000", 2024-05-15T23:14:07.779 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025051489999999999 2024-05-15T23:14:07.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145808+0000", 2024-05-15T23:14:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6304999999999999e-05 2024-05-15T23:14:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146109+0000", 2024-05-15T23:14:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000300945 2024-05-15T23:14:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.780 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16985 2.7 2:e193a409:::benchmark_data_smithi090_78160_object16984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018708+0000", 2024-05-15T23:14:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.66549179200001, 2024-05-15T23:14:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.127922702, 2024-05-15T23:14:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16985 2024-05-15T23:14:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.781 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018708+0000", 2024-05-15T23:14:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018708+0000", 2024-05-15T23:14:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018708+0000", 2024-05-15T23:14:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6400000000000002e-07 2024-05-15T23:14:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.782 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018762+0000", 2024-05-15T23:14:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3594000000000002e-05 2024-05-15T23:14:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018768+0000", 2024-05-15T23:14:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9869999999999996e-06 2024-05-15T23:14:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018772+0000", 2024-05-15T23:14:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9600000000000002e-06 2024-05-15T23:14:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.783 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018797+0000", 2024-05-15T23:14:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5701e-05 2024-05-15T23:14:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018801+0000", 2024-05-15T23:14:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0280000000000002e-06 2024-05-15T23:14:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143297+0000", 2024-05-15T23:14:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.124495859 2024-05-15T23:14:07.784 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143387+0000", 2024-05-15T23:14:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0144999999999998e-05 2024-05-15T23:14:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.143541+0000", 2024-05-15T23:14:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015373200000000001 2024-05-15T23:14:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145726+0000", 2024-05-15T23:14:07.785 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0021849360000000002 2024-05-15T23:14:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145737+0000", 2024-05-15T23:14:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0769e-05 2024-05-15T23:14:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146249+0000", 2024-05-15T23:14:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00051189599999999997 2024-05-15T23:14:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146303+0000", 2024-05-15T23:14:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3795000000000002e-05 2024-05-15T23:14:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146630+0000", 2024-05-15T23:14:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00032763600000000002 2024-05-15T23:14:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:16996 2.7 2:edffe2d2:::benchmark_data_smithi090_78160_object16995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018779+0000", 2024-05-15T23:14:07.787 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.6654198, 2024-05-15T23:14:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.128263827, 2024-05-15T23:14:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 16996 2024-05-15T23:14:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.788 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018779+0000", 2024-05-15T23:14:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018779+0000", 2024-05-15T23:14:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018780+0000", 2024-05-15T23:14:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1600000000000001e-07 2024-05-15T23:14:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.789 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.790 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.790 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018831+0000", 2024-05-15T23:14:07.790 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1023000000000003e-05 2024-05-15T23:14:07.790 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.790 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.790 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.790 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018837+0000", 2024-05-15T23:14:07.790 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0059999999999998e-06 2024-05-15T23:14:07.790 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.790 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.790 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.790 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018841+0000", 2024-05-15T23:14:07.790 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9480000000000001e-06 2024-05-15T23:14:07.790 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018868+0000", 2024-05-15T23:14:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6917999999999999e-05 2024-05-15T23:14:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018872+0000", 2024-05-15T23:14:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1239999999999998e-06 2024-05-15T23:14:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.144067+0000", 2024-05-15T23:14:07.791 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.125194783 2024-05-15T23:14:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.144155+0000", 2024-05-15T23:14:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8103999999999995e-05 2024-05-15T23:14:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.144283+0000", 2024-05-15T23:14:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012799099999999999 2024-05-15T23:14:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.145742+0000", 2024-05-15T23:14:07.792 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0014585710000000001 2024-05-15T23:14:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146189+0000", 2024-05-15T23:14:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00044775400000000002 2024-05-15T23:14:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146681+0000", 2024-05-15T23:14:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00049144900000000003 2024-05-15T23:14:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.146718+0000", 2024-05-15T23:14:07.793 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7146e-05 2024-05-15T23:14:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.147043+0000", 2024-05-15T23:14:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00032529400000000001 2024-05-15T23:14:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:17021 2.7 2:e966b9dc:::benchmark_data_smithi090_78160_object17020:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e109)", 2024-05-15T23:14:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:06:10.018848+0000", 2024-05-15T23:14:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 477.665351015, 2024-05-15T23:14:07.794 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.132449714, 2024-05-15T23:14:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 17021 2024-05-15T23:14:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018848+0000", 2024-05-15T23:14:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.795 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018848+0000", 2024-05-15T23:14:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018849+0000", 2024-05-15T23:14:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0500000000000003e-07 2024-05-15T23:14:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018901+0000", 2024-05-15T23:14:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1912e-05 2024-05-15T23:14:07.796 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018907+0000", 2024-05-15T23:14:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9490000000000001e-06 2024-05-15T23:14:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018911+0000", 2024-05-15T23:14:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8600000000000003e-06 2024-05-15T23:14:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018939+0000", 2024-05-15T23:14:07.797 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8167e-05 2024-05-15T23:14:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.018943+0000", 2024-05-15T23:14:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2540000000000003e-06 2024-05-15T23:14:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.147139+0000", 2024-05-15T23:14:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12819576399999999 2024-05-15T23:14:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.798 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.147298+0000", 2024-05-15T23:14:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015885199999999999 2024-05-15T23:14:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:14:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.147472+0000", 2024-05-15T23:14:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017376300000000001 2024-05-15T23:14:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.150655+0000", 2024-05-15T23:14:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031836740000000001 2024-05-15T23:14:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.799 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.150707+0000", 2024-05-15T23:14:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1873000000000003e-05 2024-05-15T23:14:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.150771+0000", 2024-05-15T23:14:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4231000000000002e-05 2024-05-15T23:14:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.150833+0000", 2024-05-15T23:14:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2126999999999997e-05 2024-05-15T23:14:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.800 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:06:10.151298+0000", 2024-05-15T23:14:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00046458300000000002 2024-05-15T23:14:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:303922 2.7 2:e31405bd:::benchmark_data_smithi090_78160_object142564:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-05-15T23:14:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:54.872353+0000", 2024-05-15T23:14:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 252.81184656799999, 2024-05-15T23:14:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11004205, 2024-05-15T23:14:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.801 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 303922 2024-05-15T23:14:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872353+0000", 2024-05-15T23:14:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.802 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872353+0000", 2024-05-15T23:14:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872353+0000", 2024-05-15T23:14:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8999999999999998e-07 2024-05-15T23:14:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872354+0000", 2024-05-15T23:14:07.803 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4619999999999999e-06 2024-05-15T23:14:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872357+0000", 2024-05-15T23:14:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5170000000000002e-06 2024-05-15T23:14:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872360+0000", 2024-05-15T23:14:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1839999999999999e-06 2024-05-15T23:14:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.804 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872535+0000", 2024-05-15T23:14:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017519200000000001 2024-05-15T23:14:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872543+0000", 2024-05-15T23:14:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9440000000000005e-06 2024-05-15T23:14:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.953239+0000", 2024-05-15T23:14:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.080695473000000004 2024-05-15T23:14:07.805 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.953356+0000", 2024-05-15T23:14:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000117167 2024-05-15T23:14:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:14:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.953478+0000", 2024-05-15T23:14:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012165800000000001 2024-05-15T23:14:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.806 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.962174+0000", 2024-05-15T23:14:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0086966879999999993 2024-05-15T23:14:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982345+0000", 2024-05-15T23:14:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.020170325999999999 2024-05-15T23:14:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982373+0000", 2024-05-15T23:14:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8050000000000001e-05 2024-05-15T23:14:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982395+0000", 2024-05-15T23:14:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2099e-05 2024-05-15T23:14:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:303954 2.7 2:e13848e9:::benchmark_data_smithi090_78160_object142596:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-05-15T23:14:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:54.872363+0000", 2024-05-15T23:14:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 252.811836043, 2024-05-15T23:14:07.808 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.110107154, 2024-05-15T23:14:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 303954 2024-05-15T23:14:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872363+0000", 2024-05-15T23:14:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872363+0000", 2024-05-15T23:14:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872364+0000", 2024-05-15T23:14:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9299999999999999e-07 2024-05-15T23:14:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.810 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872365+0000", 2024-05-15T23:14:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.257e-06 2024-05-15T23:14:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872368+0000", 2024-05-15T23:14:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.954e-06 2024-05-15T23:14:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872371+0000", 2024-05-15T23:14:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3409999999999999e-06 2024-05-15T23:14:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872589+0000", 2024-05-15T23:14:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000217519 2024-05-15T23:14:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872596+0000", 2024-05-15T23:14:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0679999999999999e-06 2024-05-15T23:14:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.812 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.956312+0000", 2024-05-15T23:14:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.083716805000000005 2024-05-15T23:14:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.956440+0000", 2024-05-15T23:14:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000127627 2024-05-15T23:14:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:14:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.956548+0000", 2024-05-15T23:14:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000107718 2024-05-15T23:14:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.966609+0000", 2024-05-15T23:14:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010061383 2024-05-15T23:14:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982425+0000", 2024-05-15T23:14:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015816083000000002 2024-05-15T23:14:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982451+0000", 2024-05-15T23:14:07.814 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5479000000000001e-05 2024-05-15T23:14:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982470+0000", 2024-05-15T23:14:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9627e-05 2024-05-15T23:14:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:303997 2.7 2:e15001ba:::benchmark_data_smithi090_78160_object142639:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-05-15T23:14:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:54.872374+0000", 2024-05-15T23:14:07.815 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 252.811824836, 2024-05-15T23:14:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.110179877, 2024-05-15T23:14:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 303997 2024-05-15T23:14:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872374+0000", 2024-05-15T23:14:07.816 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872374+0000", 2024-05-15T23:14:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872375+0000", 2024-05-15T23:14:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8999999999999998e-07 2024-05-15T23:14:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872376+0000", 2024-05-15T23:14:07.817 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.522e-06 2024-05-15T23:14:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872379+0000", 2024-05-15T23:14:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4339999999999999e-06 2024-05-15T23:14:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872382+0000", 2024-05-15T23:14:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.512e-06 2024-05-15T23:14:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.818 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872626+0000", 2024-05-15T23:14:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000243806 2024-05-15T23:14:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872630+0000", 2024-05-15T23:14:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6720000000000002e-06 2024-05-15T23:14:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.958946+0000", 2024-05-15T23:14:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.086315975000000003 2024-05-15T23:14:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.959046+0000", 2024-05-15T23:14:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000100605 2024-05-15T23:14:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:14:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.959124+0000", 2024-05-15T23:14:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8003999999999994e-05 2024-05-15T23:14:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.971147+0000", 2024-05-15T23:14:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.012022671 2024-05-15T23:14:07.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982500+0000", 2024-05-15T23:14:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011352944 2024-05-15T23:14:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982535+0000", 2024-05-15T23:14:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5188999999999998e-05 2024-05-15T23:14:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982554+0000", 2024-05-15T23:14:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9253e-05 2024-05-15T23:14:07.821 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:304014 2.7 2:e75fd2a5:::benchmark_data_smithi090_78160_object142656:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-05-15T23:14:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:54.872394+0000", 2024-05-15T23:14:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 252.81180579900001, 2024-05-15T23:14:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.110272975, 2024-05-15T23:14:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:14:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:14:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:14:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:14:07.822 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304014 2024-05-15T23:14:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:14:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:14:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872394+0000", 2024-05-15T23:14:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:14:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872394+0000", 2024-05-15T23:14:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:14:07.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:14:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872394+0000", 2024-05-15T23:14:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3500000000000002e-07 2024-05-15T23:14:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:14:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872396+0000", 2024-05-15T23:14:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8250000000000001e-06 2024-05-15T23:14:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:14:07.824 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872400+0000", 2024-05-15T23:14:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9330000000000003e-06 2024-05-15T23:14:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:14:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872403+0000", 2024-05-15T23:14:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0489999999999999e-06 2024-05-15T23:14:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872646+0000", 2024-05-15T23:14:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000243615 2024-05-15T23:14:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:14:07.825 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.872650+0000", 2024-05-15T23:14:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6320000000000001e-06 2024-05-15T23:14:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:14:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.962216+0000", 2024-05-15T23:14:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.089566093999999999 2024-05-15T23:14:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:14:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.962289+0000", 2024-05-15T23:14:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3048999999999998e-05 2024-05-15T23:14:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.826 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:14:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.962360+0000", 2024-05-15T23:14:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0798999999999998e-05 2024-05-15T23:14:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:14:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.971153+0000", 2024-05-15T23:14:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0087932129999999994 2024-05-15T23:14:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982586+0000", 2024-05-15T23:14:07.827 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011432804 2024-05-15T23:14:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:14:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982630+0000", 2024-05-15T23:14:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4020000000000002e-05 2024-05-15T23:14:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:14:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:14:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:14:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:54.982666+0000", 2024-05-15T23:14:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6606999999999997e-05 2024-05-15T23:14:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:14:07.828 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:14:07.829 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:14:07.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:14:07.829 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:14:07.862 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:07.873 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:07.889 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:14:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:07.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:14:07.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:08.030 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:07 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3713041075' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:14:08.031 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:08.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:07 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3713041075' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:14:08.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:07 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3713041075' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:14:08.162 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:08.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:08.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:08.269 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:08.307 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:08.330 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:08.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:08.437 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:08.468 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:08.469 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T23:14:08.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:14:08.777 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:08 smithi090 ceph-mon[44434]: pgmap v755: 17 pgs: 17 active+clean; 2.8 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:14:08.777 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:08 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:14:08.778 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3713041075' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:14:08.778 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:08 smithi090 ceph-mon[44434]: osdmap e272: 12 total, 11 up, 11 in 2024-05-15T23:14:08.783 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:08 smithi088 ceph-mon[44699]: pgmap v755: 17 pgs: 17 active+clean; 2.8 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:14:08.783 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:08 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:14:08.783 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3713041075' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:14:08.783 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:08 smithi088 ceph-mon[44699]: osdmap e272: 12 total, 11 up, 11 in 2024-05-15T23:14:08.854 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:14:08.934 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:08.935 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:08 smithi084 ceph-mon[39784]: pgmap v755: 17 pgs: 17 active+clean; 2.8 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:14:08.935 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:08 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:14:08.935 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3713041075' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:14:08.935 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:08 smithi084 ceph-mon[39784]: osdmap e272: 12 total, 11 up, 11 in 2024-05-15T23:14:08.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:08.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:09.038 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:14:09.088 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:09.118 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:14:09.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:09.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:14:09.224 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:09.253 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:09.275 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:09.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:09.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:09.379 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:09.418 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:09.768 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:09.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:09.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:09.872 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:09.903 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:09.930 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:09.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:09.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:10.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:10.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:09 smithi090 ceph-mon[44434]: 3.2 scrub starts 2024-05-15T23:14:10.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:09 smithi090 ceph-mon[44434]: 3.9 scrub starts 2024-05-15T23:14:10.066 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:10.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:09 smithi088 ceph-mon[44699]: 3.2 scrub starts 2024-05-15T23:14:10.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:09 smithi088 ceph-mon[44699]: 3.9 scrub starts 2024-05-15T23:14:10.090 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:10.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:10.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:09 smithi084 ceph-mon[39784]: 3.2 scrub starts 2024-05-15T23:14:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:09 smithi084 ceph-mon[39784]: 3.9 scrub starts 2024-05-15T23:14:10.129 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:10.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:10.263 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:10.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:10.370 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:14:10.417 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:10.673 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:14:10.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:10.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:14:10.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:10.811 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:10.834 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:10.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:10.837 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:10.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:14:10.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:10.966 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:10.980 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:10 smithi084 ceph-mon[39784]: pgmap v757: 17 pgs: 17 active+clean; 2.6 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 997 op/s 2024-05-15T23:14:10.981 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:10 smithi084 ceph-mon[39784]: 3.2 scrub ok 2024-05-15T23:14:10.981 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:10 smithi084 ceph-mon[39784]: 3.9 scrub ok 2024-05-15T23:14:10.981 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:14:10.981 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:14:10.982 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2024-05-15T23:14:10.982 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:14:10.995 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:11.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:11.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:11.012 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:10 smithi088 ceph-mon[44699]: pgmap v757: 17 pgs: 17 active+clean; 2.6 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 997 op/s 2024-05-15T23:14:11.012 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:10 smithi088 ceph-mon[44699]: 3.2 scrub ok 2024-05-15T23:14:11.012 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:10 smithi088 ceph-mon[44699]: 3.9 scrub ok 2024-05-15T23:14:11.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:10 smithi090 ceph-mon[44434]: pgmap v757: 17 pgs: 17 active+clean; 2.6 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 997 op/s 2024-05-15T23:14:11.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:10 smithi090 ceph-mon[44434]: 3.2 scrub ok 2024-05-15T23:14:11.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:10 smithi090 ceph-mon[44434]: 3.9 scrub ok 2024-05-15T23:14:11.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:11.131 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:11.163 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:11.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:11.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:11.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:11.302 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:11.325 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:11.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:11.329 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:11.414 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:14:11.421 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:14:11.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:11.467 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:11.496 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:11.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:11.600 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:12.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:11 smithi090 ceph-mon[44434]: 3.a scrub starts 2024-05-15T23:14:12.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:11 smithi090 ceph-mon[44434]: 3.6 scrub starts 2024-05-15T23:14:12.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:11 smithi090 ceph-mon[44434]: 3.a scrub ok 2024-05-15T23:14:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:11 smithi090 ceph-mon[44434]: 3.6 scrub ok 2024-05-15T23:14:12.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:11 smithi088 ceph-mon[44699]: 3.a scrub starts 2024-05-15T23:14:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:11 smithi088 ceph-mon[44699]: 3.6 scrub starts 2024-05-15T23:14:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:11 smithi088 ceph-mon[44699]: 3.a scrub ok 2024-05-15T23:14:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:11 smithi088 ceph-mon[44699]: 3.6 scrub ok 2024-05-15T23:14:12.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:11 smithi084 ceph-mon[39784]: 3.a scrub starts 2024-05-15T23:14:12.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:11 smithi084 ceph-mon[39784]: 3.6 scrub starts 2024-05-15T23:14:12.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:11 smithi084 ceph-mon[39784]: 3.a scrub ok 2024-05-15T23:14:12.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:11 smithi084 ceph-mon[39784]: 3.6 scrub ok 2024-05-15T23:14:12.857 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":758,"stamp":"2024-05-15T23:14:10.849362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2667446840,"num_objects":40696,"num_object_clones":0,"num_object_copies":81392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40696,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":290511,"num_write_kb":10596711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160790,"ondisk_log_size":160790,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":7147412,"kb_used_data":5263444,"kb_used_omap":17,"kb_used_meta":1883822,"kb_avail":1024049260,"statfs":{"total":1055945392128,"available":1048626442240,"internally_reserved":0,"allocated":5389766656,"data_stored":5385807277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":1929034613},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-564461568,"num_objects":-8613,"num_object_clones":0,"num_object_copies":-17226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8613,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.087068"},"pg_stats":[{"pgid":"3.f","version":"272'18176","reported_seq":18470,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:10.003440+0000","last_change":"2024-05-15T23:12:40.314892+0000","last_active":"2024-05-15T23:14:10.003440+0000","last_peered":"2024-05-15T23:14:10.003440+0000","last_clean":"2024-05-15T23:14:10.003440+0000","last_became_active":"2024-05-15T23:10:41.793048+0000","last_became_peered":"2024-05-15T23:10:41.793048+0000","last_unstale":"2024-05-15T23:14:10.003440+0000","last_undegraded":"2024-05-15T23:14:10.003440+0000","last_fullsized":"2024-05-15T23:14:10.003440+0000","mapping_epoch":202,"log_start":"233'8100","ondisk_log_start":"233'8100","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'11360","last_scrub_stamp":"2024-05-15T23:12:40.314742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:40.314742+0000","objects_scrubbed":9220,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154140672,"num_objects":2352,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18176,"num_write_kb":656896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"271'21526","reported_seq":21958,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T23:14:07.461942+0000","last_change":"2024-05-15T23:12:42.022558+0000","last_active":"2024-05-15T23:14:07.461942+0000","last_peered":"2024-05-15T23:14:07.461942+0000","last_clean":"2024-05-15T23:14:07.461942+0000","last_became_active":"2024-05-15T23:10:41.794521+0000","last_became_peered":"2024-05-15T23:10:41.794521+0000","last_unstale":"2024-05-15T23:14:07.461942+0000","last_undegraded":"2024-05-15T23:14:07.461942+0000","last_fullsized":"2024-05-15T23:14:07.461942+0000","mapping_epoch":202,"log_start":"243'11503","ondisk_log_start":"243'11503","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'15032","last_scrub_stamp":"2024-05-15T23:12:42.022424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:42.022424+0000","objects_scrubbed":9101,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:10.982043+0000","scrub_duration":1664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166789120,"num_objects":2545,"num_object_clones":0,"num_object_copies":5090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18055,"num_write_kb":659200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"272'18300","reported_seq":18552,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:09.969767+0000","last_change":"2024-05-15T23:12:41.500301+0000","last_active":"2024-05-15T23:14:09.969767+0000","last_peered":"2024-05-15T23:14:09.969767+0000","last_clean":"2024-05-15T23:14:09.969767+0000","last_became_active":"2024-05-15T23:10:41.794133+0000","last_became_peered":"2024-05-15T23:10:41.794133+0000","last_unstale":"2024-05-15T23:14:09.969767+0000","last_undegraded":"2024-05-15T23:14:09.969767+0000","last_fullsized":"2024-05-15T23:14:09.969767+0000","mapping_epoch":202,"log_start":"234'8200","ondisk_log_start":"234'8200","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'11587","last_scrub_stamp":"2024-05-15T23:12:41.500086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:41.500086+0000","objects_scrubbed":9204,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1568,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159842328,"num_objects":2440,"num_object_clones":0,"num_object_copies":4880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2440,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18300,"num_write_kb":663617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"271'18014","reported_seq":18319,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T23:14:06.819415+0000","last_change":"2024-05-15T23:13:30.345583+0000","last_active":"2024-05-15T23:14:06.819415+0000","last_peered":"2024-05-15T23:14:06.819415+0000","last_clean":"2024-05-15T23:14:06.819415+0000","last_became_active":"2024-05-15T23:10:41.795518+0000","last_became_peered":"2024-05-15T23:10:41.795518+0000","last_unstale":"2024-05-15T23:14:06.819415+0000","last_undegraded":"2024-05-15T23:14:06.819415+0000","last_fullsized":"2024-05-15T23:14:06.819415+0000","mapping_epoch":202,"log_start":"233'8000","ondisk_log_start":"233'8000","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15330","last_scrub_stamp":"2024-05-15T23:13:30.345428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:30.345428+0000","objects_scrubbed":5390,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:59.525761+0000","scrub_duration":974,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175112192,"num_objects":2672,"num_object_clones":0,"num_object_copies":5344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18014,"num_write_kb":661952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"271'18098","reported_seq":18455,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T23:14:06.026341+0000","last_change":"2024-05-15T23:13:27.508301+0000","last_active":"2024-05-15T23:14:06.026341+0000","last_peered":"2024-05-15T23:14:06.026341+0000","last_clean":"2024-05-15T23:14:06.026341+0000","last_became_active":"2024-05-15T23:10:41.791796+0000","last_became_peered":"2024-05-15T23:10:41.791796+0000","last_unstale":"2024-05-15T23:14:06.026341+0000","last_undegraded":"2024-05-15T23:14:06.026341+0000","last_fullsized":"2024-05-15T23:14:06.026341+0000","mapping_epoch":202,"log_start":"232'8000","ondisk_log_start":"232'8000","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15204","last_scrub_stamp":"2024-05-15T23:13:27.508024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:27.508024+0000","objects_scrubbed":5626,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:47.140576+0000","scrub_duration":1525,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175374336,"num_objects":2676,"num_object_clones":0,"num_object_copies":5352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18098,"num_write_kb":664768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"271'18034","reported_seq":18359,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T23:14:07.535233+0000","last_change":"2024-05-15T23:12:55.282863+0000","last_active":"2024-05-15T23:14:07.535233+0000","last_peered":"2024-05-15T23:14:07.535233+0000","last_clean":"2024-05-15T23:14:07.535233+0000","last_became_active":"2024-05-15T23:10:41.794649+0000","last_became_peered":"2024-05-15T23:10:41.794649+0000","last_unstale":"2024-05-15T23:14:07.535233+0000","last_undegraded":"2024-05-15T23:14:07.535233+0000","last_fullsized":"2024-05-15T23:14:07.535233+0000","mapping_epoch":202,"log_start":"233'8000","ondisk_log_start":"233'8000","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'12548","last_scrub_stamp":"2024-05-15T23:12:55.282531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:55.282531+0000","objects_scrubbed":8168,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:10.736493+0000","scrub_duration":1166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174063616,"num_objects":2656,"num_object_clones":0,"num_object_copies":5312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18034,"num_write_kb":662080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"272'18510","reported_seq":18780,"reported_epoch":272,"state":"active+clean+scrubbing","last_fresh":"2024-05-15T23:14:09.576434+0000","last_change":"2024-05-15T23:14:09.390930+0000","last_active":"2024-05-15T23:14:09.576434+0000","last_peered":"2024-05-15T23:14:09.576434+0000","last_clean":"2024-05-15T23:14:09.576434+0000","last_became_active":"2024-05-15T23:10:41.793356+0000","last_became_peered":"2024-05-15T23:10:41.793356+0000","last_unstale":"2024-05-15T23:14:09.576434+0000","last_undegraded":"2024-05-15T23:14:09.576434+0000","last_fullsized":"2024-05-15T23:14:09.576434+0000","mapping_epoch":202,"log_start":"235'8500","ondisk_log_start":"235'8500","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'11462","last_scrub_stamp":"2024-05-15T23:12:37.884916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:37.884916+0000","objects_scrubbed":1299,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 0s","scrub_duration":1718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161087488,"num_objects":2458,"num_object_clones":0,"num_object_copies":4916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18510,"num_write_kb":670976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"271'18234","reported_seq":18536,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T23:14:07.436769+0000","last_change":"2024-05-15T23:13:28.700587+0000","last_active":"2024-05-15T23:14:07.436769+0000","last_peered":"2024-05-15T23:14:07.436769+0000","last_clean":"2024-05-15T23:14:07.436769+0000","last_became_active":"2024-05-15T23:10:41.794942+0000","last_became_peered":"2024-05-15T23:10:41.794942+0000","last_unstale":"2024-05-15T23:14:07.436769+0000","last_undegraded":"2024-05-15T23:14:07.436769+0000","last_fullsized":"2024-05-15T23:14:07.436769+0000","mapping_epoch":202,"log_start":"234'8200","ondisk_log_start":"234'8200","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15293","last_scrub_stamp":"2024-05-15T23:13:28.700488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:28.700488+0000","objects_scrubbed":5642,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:41.348504+0000","scrub_duration":1245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173932544,"num_objects":2654,"num_object_clones":0,"num_object_copies":5308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18234,"num_write_kb":668416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"271'18084","reported_seq":18385,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T23:14:06.017275+0000","last_change":"2024-05-15T23:13:32.196550+0000","last_active":"2024-05-15T23:14:06.017275+0000","last_peered":"2024-05-15T23:14:06.017275+0000","last_clean":"2024-05-15T23:14:06.017275+0000","last_became_active":"2024-05-15T23:10:41.792820+0000","last_became_peered":"2024-05-15T23:10:41.792820+0000","last_unstale":"2024-05-15T23:14:06.017275+0000","last_undegraded":"2024-05-15T23:14:06.017275+0000","last_fullsized":"2024-05-15T23:14:06.017275+0000","mapping_epoch":202,"log_start":"232'8000","ondisk_log_start":"232'8000","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15478","last_scrub_stamp":"2024-05-15T23:13:32.196338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:32.196338+0000","objects_scrubbed":5325,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:32.374216+0000","scrub_duration":1194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176029696,"num_objects":2686,"num_object_clones":0,"num_object_copies":5372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18084,"num_write_kb":664640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"272'20543","reported_seq":20915,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:09.879762+0000","last_change":"2024-05-15T23:12:40.833253+0000","last_active":"2024-05-15T23:14:09.879762+0000","last_peered":"2024-05-15T23:14:09.879762+0000","last_clean":"2024-05-15T23:14:09.879762+0000","last_became_active":"2024-05-15T23:10:41.794131+0000","last_became_peered":"2024-05-15T23:10:41.794131+0000","last_unstale":"2024-05-15T23:14:09.879762+0000","last_undegraded":"2024-05-15T23:14:09.879762+0000","last_fullsized":"2024-05-15T23:14:09.879762+0000","mapping_epoch":202,"log_start":"243'10502","ondisk_log_start":"243'10502","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'13799","last_scrub_stamp":"2024-05-15T23:12:40.833117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:40.833117+0000","objects_scrubbed":9233,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1917,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158269440,"num_objects":2415,"num_object_clones":0,"num_object_copies":4830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18301,"num_write_kb":662912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"271'17905","reported_seq":18214,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T23:14:06.831149+0000","last_change":"2024-05-15T23:13:25.696329+0000","last_active":"2024-05-15T23:14:06.831149+0000","last_peered":"2024-05-15T23:14:06.831149+0000","last_clean":"2024-05-15T23:14:06.831149+0000","last_became_active":"2024-05-15T23:10:41.796378+0000","last_became_peered":"2024-05-15T23:10:41.796378+0000","last_unstale":"2024-05-15T23:14:06.831149+0000","last_undegraded":"2024-05-15T23:14:06.831149+0000","last_fullsized":"2024-05-15T23:14:06.831149+0000","mapping_epoch":202,"log_start":"232'7900","ondisk_log_start":"232'7900","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'14929","last_scrub_stamp":"2024-05-15T23:13:25.696132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:25.696132+0000","objects_scrubbed":5637,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:52.263125+0000","scrub_duration":1288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170328064,"num_objects":2599,"num_object_clones":0,"num_object_copies":5198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17905,"num_write_kb":656128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"272'18128","reported_seq":18428,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:10.022004+0000","last_change":"2024-05-15T23:13:32.720230+0000","last_active":"2024-05-15T23:14:10.022004+0000","last_peered":"2024-05-15T23:14:10.022004+0000","last_clean":"2024-05-15T23:14:10.022004+0000","last_became_active":"2024-05-15T23:10:41.792966+0000","last_became_peered":"2024-05-15T23:10:41.792966+0000","last_unstale":"2024-05-15T23:14:10.022004+0000","last_undegraded":"2024-05-15T23:14:10.022004+0000","last_fullsized":"2024-05-15T23:14:10.022004+0000","mapping_epoch":202,"log_start":"234'8100","ondisk_log_start":"234'8100","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15337","last_scrub_stamp":"2024-05-15T23:13:32.720051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:32.720051+0000","objects_scrubbed":5192,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:33.190944+0000","scrub_duration":1161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154533888,"num_objects":2358,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18128,"num_write_kb":655552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"272'18204","reported_seq":18518,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:09.515983+0000","last_change":"2024-05-15T23:13:30.204128+0000","last_active":"2024-05-15T23:14:09.515983+0000","last_peered":"2024-05-15T23:14:09.515983+0000","last_clean":"2024-05-15T23:14:09.515983+0000","last_became_active":"2024-05-15T23:10:41.794364+0000","last_became_peered":"2024-05-15T23:10:41.794364+0000","last_unstale":"2024-05-15T23:14:09.515983+0000","last_undegraded":"2024-05-15T23:14:09.515983+0000","last_fullsized":"2024-05-15T23:14:09.515983+0000","mapping_epoch":202,"log_start":"235'8200","ondisk_log_start":"235'8200","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15231","last_scrub_stamp":"2024-05-15T23:13:30.203931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:30.203931+0000","objects_scrubbed":5476,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:44.109942+0000","scrub_duration":1269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160956416,"num_objects":2456,"num_object_clones":0,"num_object_copies":4912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18204,"num_write_kb":661120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"272'18287","reported_seq":18599,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:09.823466+0000","last_change":"2024-05-15T23:13:29.043358+0000","last_active":"2024-05-15T23:14:09.823466+0000","last_peered":"2024-05-15T23:14:09.823466+0000","last_clean":"2024-05-15T23:14:09.823466+0000","last_became_active":"2024-05-15T23:10:41.794291+0000","last_became_peered":"2024-05-15T23:10:41.794291+0000","last_unstale":"2024-05-15T23:14:09.823466+0000","last_undegraded":"2024-05-15T23:14:09.823466+0000","last_fullsized":"2024-05-15T23:14:09.823466+0000","mapping_epoch":202,"log_start":"234'8200","ondisk_log_start":"234'8200","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15210","last_scrub_stamp":"2024-05-15T23:13:29.043207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:29.043207+0000","objects_scrubbed":5487,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:33.486232+0000","scrub_duration":1073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155516928,"num_objects":2373,"num_object_clones":0,"num_object_copies":4746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18287,"num_write_kb":661120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"271'18233","reported_seq":18542,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T23:14:07.472932+0000","last_change":"2024-05-15T23:13:36.477604+0000","last_active":"2024-05-15T23:14:07.472932+0000","last_peered":"2024-05-15T23:14:07.472932+0000","last_clean":"2024-05-15T23:14:07.472932+0000","last_became_active":"2024-05-15T23:10:42.014274+0000","last_became_peered":"2024-05-15T23:10:42.014274+0000","last_unstale":"2024-05-15T23:14:07.472932+0000","last_undegraded":"2024-05-15T23:14:07.472932+0000","last_fullsized":"2024-05-15T23:14:07.472932+0000","mapping_epoch":202,"log_start":"234'8200","ondisk_log_start":"234'8200","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'15877","last_scrub_stamp":"2024-05-15T23:13:36.477324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:36.477324+0000","objects_scrubbed":5072,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:15:05.465959+0000","scrub_duration":1039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175702016,"num_objects":2681,"num_object_clones":0,"num_object_copies":5362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18233,"num_write_kb":669248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"271'17825","reported_seq":18133,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-05-15T23:14:06.062136+0000","last_change":"2024-05-15T23:12:39.938806+0000","last_active":"2024-05-15T23:14:06.062136+0000","last_peered":"2024-05-15T23:14:06.062136+0000","last_clean":"2024-05-15T23:14:06.062136+0000","last_became_active":"2024-05-15T23:10:42.014285+0000","last_became_peered":"2024-05-15T23:10:42.014285+0000","last_unstale":"2024-05-15T23:14:06.062136+0000","last_undegraded":"2024-05-15T23:14:06.062136+0000","last_fullsized":"2024-05-15T23:14:06.062136+0000","mapping_epoch":202,"log_start":"231'7800","ondisk_log_start":"231'7800","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'11402","last_scrub_stamp":"2024-05-15T23:12:39.938642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:39.938642+0000","objects_scrubbed":9163,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:06.922162+0000","scrub_duration":2007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175177728,"num_objects":2673,"num_object_clones":0,"num_object_copies":5346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17825,"num_write_kb":655936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":539,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:07.792242+0000","last_change":"2024-05-15T23:12:43.939662+0000","last_active":"2024-05-15T23:14:07.792242+0000","last_peered":"2024-05-15T23:14:07.792242+0000","last_clean":"2024-05-15T23:14:07.792242+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:14:07.792242+0000","last_undegraded":"2024-05-15T23:14:07.792242+0000","last_fullsized":"2024-05-15T23:14:07.792242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:12:43.939526+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:12:43.939526+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:11.827428+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":2666856472,"num_objects":40694,"num_object_clones":0,"num_object_copies":81388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40694,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":290388,"num_write_kb":10594561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5300166656,"data_stored":5300158512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160696,"ondisk_log_size":160696,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351100,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1518348,"kb_used_data":1190080,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92226804,"statfs":{"total":95995035648,"available":94440247296,"internally_reserved":0,"allocated":1218641920,"data_stored":1218283645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":336132548},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":63828,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93681324,"statfs":{"total":95995035648,"available":95929675776,"internally_reserved":0,"allocated":724992,"data_stored":364235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776266,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":501132,"kb_used_data":311684,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93244020,"statfs":{"total":95995035648,"available":95481876480,"internally_reserved":0,"allocated":319164416,"data_stored":318803659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":193984960},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005198,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":292620,"kb_used_data":158724,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":93452532,"statfs":{"total":95995035648,"available":95695392768,"internally_reserved":0,"allocated":162533376,"data_stored":162172619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":137099717},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":435032,"kb_used_data":309456,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93310120,"statfs":{"total":95995035648,"available":95549562880,"internally_reserved":0,"allocated":316882944,"data_stored":316514563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128580042},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":246788,"kb_used_data":175352,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93498364,"statfs":{"total":95995035648,"available":95742324736,"internally_reserved":0,"allocated":179560448,"data_stored":179205745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575026,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622160,"kb_used_data":468804,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93122992,"statfs":{"total":95995035648,"available":95357943808,"internally_reserved":0,"allocated":480055296,"data_stored":479694539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":157022662},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738551,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":878544,"kb_used_data":630788,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92866608,"statfs":{"total":95995035648,"available":95095406592,"internally_reserved":0,"allocated":645926912,"data_stored":645566155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1117320,"kb_used_data":865400,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":92627832,"statfs":{"total":95995035648,"available":94850899968,"internally_reserved":0,"allocated":886169600,"data_stored":885814897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":257948103},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986337,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":812240,"kb_used_data":636676,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92932912,"statfs":{"total":95995035648,"available":95163301888,"internally_reserved":0,"allocated":651956224,"data_stored":651595467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790107,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":659400,"kb_used_data":515772,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93085752,"statfs":{"total":95995035648,"available":95319810048,"internally_reserved":0,"allocated":528150528,"data_stored":527791753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":629866496,"data_stored":629866496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":474021888,"data_stored":474021888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":879230976,"data_stored":879230976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":639893504,"data_stored":639893504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":520294400,"data_stored":520290328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":174063616,"data_stored":174063616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":313921536,"data_stored":313917464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":161087488,"data_stored":161087488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":314769408,"data_stored":314769408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1193017344,"data_stored":1193017344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:14:12.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:14:12.940 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:14:12.940 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:14:12.940 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:14:12.940 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:14:12.940 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:14:12.941 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:14:12.941 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:14:12.941 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:14:12.955 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:14:12.994 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:14:13.001 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:13.035 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:14:13.035 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:14:13.035 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:14:13.035 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-05-15T23:14:13.035 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:14:13.035 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:14:13.035 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:14:13.036 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:14:13.036 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:14:13.036 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:14:13.036 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:14:13.036 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:14:13.036 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:14:13.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:12 smithi084 ceph-mon[39784]: pgmap v758: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.5 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:14:13.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:12 smithi084 ceph-mon[39784]: from='client.25351 v1:172.21.15.84:0/1286703032' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:14:13.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:12 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3750320085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:13.060 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:13.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:12 smithi088 ceph-mon[44699]: pgmap v758: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.5 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:14:13.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:12 smithi088 ceph-mon[44699]: from='client.25351 v1:172.21.15.84:0/1286703032' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:14:13.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:12 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3750320085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:13.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:12 smithi090 ceph-mon[44434]: pgmap v758: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.5 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:14:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:12 smithi090 ceph-mon[44434]: from='client.25351 v1:172.21.15.84:0/1286703032' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:14:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:12 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3750320085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:13.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:13.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:13.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:13.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.202 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:14.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:14.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:14.308 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:14.339 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:15.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:14 smithi084 ceph-mon[39784]: pgmap v759: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.3 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.40k op/s 2024-05-15T23:14:15.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:14 smithi084 ceph-mon[39784]: 3.f scrub starts 2024-05-15T23:14:15.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:14 smithi084 ceph-mon[39784]: 3.e scrub starts 2024-05-15T23:14:15.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:14 smithi084 ceph-mon[39784]: 3.e scrub ok 2024-05-15T23:14:15.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:14 smithi090 ceph-mon[44434]: pgmap v759: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.3 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.40k op/s 2024-05-15T23:14:15.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:14 smithi090 ceph-mon[44434]: 3.f scrub starts 2024-05-15T23:14:15.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:14 smithi090 ceph-mon[44434]: 3.e scrub starts 2024-05-15T23:14:15.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:14 smithi090 ceph-mon[44434]: 3.e scrub ok 2024-05-15T23:14:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:14 smithi088 ceph-mon[44699]: pgmap v759: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.3 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.40k op/s 2024-05-15T23:14:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:14 smithi088 ceph-mon[44699]: 3.f scrub starts 2024-05-15T23:14:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:14 smithi088 ceph-mon[44699]: 3.e scrub starts 2024-05-15T23:14:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:14 smithi088 ceph-mon[44699]: 3.e scrub ok 2024-05-15T23:14:15.876 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:15.883 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:14:15.884 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:14:16.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:15 smithi084 ceph-mon[39784]: 3.f scrub ok 2024-05-15T23:14:16.235 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:15 smithi088 ceph-mon[44699]: 3.f scrub ok 2024-05-15T23:14:16.267 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T23:14:16.291 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:14:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:16.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:14:16.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:15 smithi090 ceph-mon[44434]: 3.f scrub ok 2024-05-15T23:14:16.396 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:16.434 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:16.462 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:16.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:16.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:16.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:14:16.607 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:16.632 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:14:16.635 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:14:16.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:16.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:14:16.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:14:16.778 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:16.805 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:14:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:16.808 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:14:16.909 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:16.983 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:17.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:16 smithi090 ceph-mon[44434]: pgmap v760: 17 pgs: 17 active+clean; 2.2 GiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-05-15T23:14:17.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:16 smithi084 ceph-mon[39784]: pgmap v760: 17 pgs: 17 active+clean; 2.2 GiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-05-15T23:14:17.169 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:17.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:17.268 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:16 smithi088 ceph-mon[44699]: pgmap v760: 17 pgs: 17 active+clean; 2.2 GiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-05-15T23:14:17.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:17.310 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:17.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:17.347 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:17.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:17.448 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:17.493 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:17.600 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:14:17.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:17.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:14:17.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:14:17.745 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:17.786 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:14:17.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:17.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:14:17.885 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:14:17.890 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:14:17.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:17.897 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:17 smithi088 ceph-mon[44699]: pgmap v761: 17 pgs: 17 active+clean; 2.1 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:14:17.897 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:17 smithi088 ceph-mon[44699]: 3.d scrub starts 2024-05-15T23:14:17.897 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:17 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3340942328' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:14:17.925 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:17.953 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:17.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:17.956 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:18.057 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:18.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:17 smithi084 ceph-mon[39784]: pgmap v761: 17 pgs: 17 active+clean; 2.1 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:14:18.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:17 smithi084 ceph-mon[39784]: 3.d scrub starts 2024-05-15T23:14:18.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:17 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3340942328' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:14:18.093 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:18.115 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:18.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:18.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:18.219 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:18.260 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:18.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:17 smithi090 ceph-mon[44434]: pgmap v761: 17 pgs: 17 active+clean; 2.1 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:14:18.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:17 smithi090 ceph-mon[44434]: 3.d scrub starts 2024-05-15T23:14:18.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:17 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3340942328' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:14:18.459 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:14:18.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:18.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:14:18.566 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:18.596 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:18.764 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:18.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:18.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:18.819 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:14:18.819 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:14:18.871 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:14:18.950 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:18.987 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:14:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:18.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:14:19.091 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:19.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:18 smithi084 ceph-mon[39784]: 3.d scrub ok 2024-05-15T23:14:19.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:18 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:14:19.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:18 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:14:19.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:18 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3340942328' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:14:19.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:18 smithi084 ceph-mon[39784]: osdmap e273: 12 total, 11 up, 11 in 2024-05-15T23:14:19.121 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:19.172 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:19.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:18 smithi088 ceph-mon[44699]: 3.d scrub ok 2024-05-15T23:14:19.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:18 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:14:19.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:18 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:14:19.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:18 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3340942328' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:14:19.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:18 smithi088 ceph-mon[44699]: osdmap e273: 12 total, 11 up, 11 in 2024-05-15T23:14:19.223 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:19.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:19.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:19.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:18 smithi090 ceph-mon[44434]: 3.d scrub ok 2024-05-15T23:14:19.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:18 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:14:19.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:18 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:14:19.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:18 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3340942328' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:14:19.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:18 smithi090 ceph-mon[44434]: osdmap e273: 12 total, 11 up, 11 in 2024-05-15T23:14:19.327 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:19.364 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:19.499 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:14:19.507 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:14:19.562 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:19.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:19.574 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T23:14:19.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:19.704 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:19.735 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:19.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:19.740 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:19.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:19.873 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:19.897 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:19.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:19.902 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:19.975 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:14:20.003 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:20.044 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:20.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:19 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:14:20.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:19 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:14:20.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:19 smithi090 ceph-mon[44434]: pgmap v763: 17 pgs: 17 active+clean; 1.9 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:14:20.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:19 smithi090 ceph-mon[44434]: osdmap e274: 12 total, 11 up, 11 in 2024-05-15T23:14:20.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:19 smithi090 ceph-mon[44434]: from='client.25360 v1:172.21.15.84:0/4281376128' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:14:20.235 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:19 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:14:20.235 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:19 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:14:20.235 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:19 smithi088 ceph-mon[44699]: pgmap v763: 17 pgs: 17 active+clean; 1.9 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:14:20.235 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:19 smithi088 ceph-mon[44699]: osdmap e274: 12 total, 11 up, 11 in 2024-05-15T23:14:20.235 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:19 smithi088 ceph-mon[44699]: from='client.25360 v1:172.21.15.84:0/4281376128' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:14:20.256 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:14:20.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:20.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:14:20.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:19 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:14:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:19 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:14:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:19 smithi084 ceph-mon[39784]: pgmap v763: 17 pgs: 17 active+clean; 1.9 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:14:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:19 smithi084 ceph-mon[39784]: osdmap e274: 12 total, 11 up, 11 in 2024-05-15T23:14:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:19 smithi084 ceph-mon[39784]: from='client.25360 v1:172.21.15.84:0/4281376128' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:14:20.400 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:20.433 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:20.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:20.573 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:20.601 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:20.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:20.604 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:20.705 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:20.735 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:20.836 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:20.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:20.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:14:20.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:21.017 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:21.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:20 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:14:21.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:20 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:14:21.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:20 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:14:21.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:20 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:14:21.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:20 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:14:21.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:20 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:14:21.395 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:21.397 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":764,"stamp":"2024-05-15T23:14:18.912238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2060386872,"num_objects":31433,"num_object_clones":0,"num_object_copies":62866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31433,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":299774,"num_write_kb":10596711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161153,"ondisk_log_size":161153,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":5971748,"kb_used_data":4104452,"kb_used_omap":17,"kb_used_meta":1867182,"kb_avail":1025224924,"statfs":{"total":1055945392128,"available":1049830322176,"internally_reserved":0,"allocated":4202958848,"data_stored":4198975253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":1911995253},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-607059968,"num_objects":-9263,"num_object_clones":0,"num_object_copies":-18526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9263,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.062875"},"pg_stats":[{"pgid":"3.f","version":"272'18546","reported_seq":18869,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:14.978660+0000","last_change":"2024-05-15T23:14:14.237693+0000","last_active":"2024-05-15T23:14:14.978660+0000","last_peered":"2024-05-15T23:14:14.978660+0000","last_clean":"2024-05-15T23:14:14.978660+0000","last_became_active":"2024-05-15T23:10:41.793048+0000","last_became_peered":"2024-05-15T23:10:41.793048+0000","last_unstale":"2024-05-15T23:14:14.978660+0000","last_undegraded":"2024-05-15T23:14:14.978660+0000","last_fullsized":"2024-05-15T23:14:14.978660+0000","mapping_epoch":202,"log_start":"236'8500","ondisk_log_start":"236'8500","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'18482","last_scrub_stamp":"2024-05-15T23:14:14.237542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:14:14.237542+0000","objects_scrubbed":2060,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:15:39.011631+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129892352,"num_objects":1982,"num_object_clones":0,"num_object_copies":3964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18546,"num_write_kb":656896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"272'22218","reported_seq":22681,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:17.502016+0000","last_change":"2024-05-15T23:14:14.307767+0000","last_active":"2024-05-15T23:14:17.502016+0000","last_peered":"2024-05-15T23:14:17.502016+0000","last_clean":"2024-05-15T23:14:17.502016+0000","last_became_active":"2024-05-15T23:10:41.794521+0000","last_became_peered":"2024-05-15T23:10:41.794521+0000","last_unstale":"2024-05-15T23:14:17.502016+0000","last_undegraded":"2024-05-15T23:14:17.502016+0000","last_fullsized":"2024-05-15T23:14:17.502016+0000","mapping_epoch":202,"log_start":"243'12203","ondisk_log_start":"243'12203","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'22008","last_scrub_stamp":"2024-05-15T23:14:14.307609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:14:14.307609+0000","objects_scrubbed":2082,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:15:40.782318+0000","scrub_duration":392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121438208,"num_objects":1853,"num_object_clones":0,"num_object_copies":3706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18747,"num_write_kb":659200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"272'18683","reported_seq":18931,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:15.001456+0000","last_change":"2024-05-15T23:12:41.500301+0000","last_active":"2024-05-15T23:14:15.001456+0000","last_peered":"2024-05-15T23:14:15.001456+0000","last_clean":"2024-05-15T23:14:15.001456+0000","last_became_active":"2024-05-15T23:10:41.794133+0000","last_became_peered":"2024-05-15T23:10:41.794133+0000","last_unstale":"2024-05-15T23:14:15.001456+0000","last_undegraded":"2024-05-15T23:14:15.001456+0000","last_fullsized":"2024-05-15T23:14:15.001456+0000","mapping_epoch":202,"log_start":"236'8600","ondisk_log_start":"236'8600","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'11587","last_scrub_stamp":"2024-05-15T23:12:41.500086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:12:41.500086+0000","objects_scrubbed":9204,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1568,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134742040,"num_objects":2057,"num_object_clones":0,"num_object_copies":4114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2057,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18683,"num_write_kb":663617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"272'18797","reported_seq":19104,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:16.828365+0000","last_change":"2024-05-15T23:13:30.345583+0000","last_active":"2024-05-15T23:14:16.828365+0000","last_peered":"2024-05-15T23:14:16.828365+0000","last_clean":"2024-05-15T23:14:16.828365+0000","last_became_active":"2024-05-15T23:10:41.795518+0000","last_became_peered":"2024-05-15T23:10:41.795518+0000","last_unstale":"2024-05-15T23:14:16.828365+0000","last_undegraded":"2024-05-15T23:14:16.828365+0000","last_fullsized":"2024-05-15T23:14:16.828365+0000","mapping_epoch":202,"log_start":"237'8700","ondisk_log_start":"237'8700","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15330","last_scrub_stamp":"2024-05-15T23:13:30.345428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:30.345428+0000","objects_scrubbed":5390,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:59.525761+0000","scrub_duration":974,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123797504,"num_objects":1889,"num_object_clones":0,"num_object_copies":3778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18797,"num_write_kb":661952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"272'18840","reported_seq":19199,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:16.053467+0000","last_change":"2024-05-15T23:13:27.508301+0000","last_active":"2024-05-15T23:14:16.053467+0000","last_peered":"2024-05-15T23:14:16.053467+0000","last_clean":"2024-05-15T23:14:16.053467+0000","last_became_active":"2024-05-15T23:10:41.791796+0000","last_became_peered":"2024-05-15T23:10:41.791796+0000","last_unstale":"2024-05-15T23:14:16.053467+0000","last_undegraded":"2024-05-15T23:14:16.053467+0000","last_fullsized":"2024-05-15T23:14:16.053467+0000","mapping_epoch":202,"log_start":"238'8800","ondisk_log_start":"238'8800","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15204","last_scrub_stamp":"2024-05-15T23:13:27.508024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:27.508024+0000","objects_scrubbed":5626,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:47.140576+0000","scrub_duration":1525,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126746624,"num_objects":1934,"num_object_clones":0,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18840,"num_write_kb":664768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"272'18776","reported_seq":19137,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:17.528278+0000","last_change":"2024-05-15T23:14:11.137375+0000","last_active":"2024-05-15T23:14:17.528278+0000","last_peered":"2024-05-15T23:14:17.528278+0000","last_clean":"2024-05-15T23:14:17.528278+0000","last_became_active":"2024-05-15T23:10:41.794649+0000","last_became_peered":"2024-05-15T23:10:41.794649+0000","last_unstale":"2024-05-15T23:14:17.528278+0000","last_undegraded":"2024-05-15T23:14:17.528278+0000","last_fullsized":"2024-05-15T23:14:17.528278+0000","mapping_epoch":202,"log_start":"238'8700","ondisk_log_start":"238'8700","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'18284","last_scrub_stamp":"2024-05-15T23:14:11.137255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:14:11.137255+0000","objects_scrubbed":2414,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:15:35.574591+0000","scrub_duration":389,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125435904,"num_objects":1914,"num_object_clones":0,"num_object_copies":3828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18776,"num_write_kb":662080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"272'18886","reported_seq":19172,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:14.530865+0000","last_change":"2024-05-15T23:14:09.769170+0000","last_active":"2024-05-15T23:14:14.530865+0000","last_peered":"2024-05-15T23:14:14.530865+0000","last_clean":"2024-05-15T23:14:14.530865+0000","last_became_active":"2024-05-15T23:10:41.793356+0000","last_became_peered":"2024-05-15T23:10:41.793356+0000","last_unstale":"2024-05-15T23:14:14.530865+0000","last_undegraded":"2024-05-15T23:14:14.530865+0000","last_fullsized":"2024-05-15T23:14:14.530865+0000","mapping_epoch":202,"log_start":"237'8800","ondisk_log_start":"237'8800","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'18518","last_scrub_stamp":"2024-05-15T23:14:09.768942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:14:09.768942+0000","objects_scrubbed":2458,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:15:30.942569+0000","scrub_duration":372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136445952,"num_objects":2082,"num_object_clones":0,"num_object_copies":4164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18886,"num_write_kb":670976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"272'18976","reported_seq":19279,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:17.509418+0000","last_change":"2024-05-15T23:13:28.700587+0000","last_active":"2024-05-15T23:14:17.509418+0000","last_peered":"2024-05-15T23:14:17.509418+0000","last_clean":"2024-05-15T23:14:17.509418+0000","last_became_active":"2024-05-15T23:10:41.794942+0000","last_became_peered":"2024-05-15T23:10:41.794942+0000","last_unstale":"2024-05-15T23:14:17.509418+0000","last_undegraded":"2024-05-15T23:14:17.509418+0000","last_fullsized":"2024-05-15T23:14:17.509418+0000","mapping_epoch":202,"log_start":"238'8900","ondisk_log_start":"238'8900","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15293","last_scrub_stamp":"2024-05-15T23:13:28.700488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:28.700488+0000","objects_scrubbed":5642,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:41.348504+0000","scrub_duration":1245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125304832,"num_objects":1912,"num_object_clones":0,"num_object_copies":3824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18976,"num_write_kb":668416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"272'18830","reported_seq":19133,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:16.068655+0000","last_change":"2024-05-15T23:13:32.196550+0000","last_active":"2024-05-15T23:14:16.068655+0000","last_peered":"2024-05-15T23:14:16.068655+0000","last_clean":"2024-05-15T23:14:16.068655+0000","last_became_active":"2024-05-15T23:10:41.792820+0000","last_became_peered":"2024-05-15T23:10:41.792820+0000","last_unstale":"2024-05-15T23:14:16.068655+0000","last_undegraded":"2024-05-15T23:14:16.068655+0000","last_fullsized":"2024-05-15T23:14:16.068655+0000","mapping_epoch":202,"log_start":"238'8800","ondisk_log_start":"238'8800","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15478","last_scrub_stamp":"2024-05-15T23:13:32.196338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:32.196338+0000","objects_scrubbed":5325,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:32.374216+0000","scrub_duration":1194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127139840,"num_objects":1940,"num_object_clones":0,"num_object_copies":3880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18830,"num_write_kb":664640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"272'20863","reported_seq":21266,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:14.261631+0000","last_change":"2024-05-15T23:14:11.708196+0000","last_active":"2024-05-15T23:14:14.261631+0000","last_peered":"2024-05-15T23:14:14.261631+0000","last_clean":"2024-05-15T23:14:14.261631+0000","last_became_active":"2024-05-15T23:10:41.794131+0000","last_became_peered":"2024-05-15T23:10:41.794131+0000","last_unstale":"2024-05-15T23:14:14.261631+0000","last_undegraded":"2024-05-15T23:14:14.261631+0000","last_fullsized":"2024-05-15T23:14:14.261631+0000","mapping_epoch":202,"log_start":"243'10802","ondisk_log_start":"243'10802","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'20685","last_scrub_stamp":"2024-05-15T23:14:11.708003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:14:11.708003+0000","objects_scrubbed":2313,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:15:18.091687+0000","scrub_duration":696,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137297920,"num_objects":2095,"num_object_clones":0,"num_object_copies":4190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18621,"num_write_kb":662912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"272'18680","reported_seq":18991,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:16.808877+0000","last_change":"2024-05-15T23:13:25.696329+0000","last_active":"2024-05-15T23:14:16.808877+0000","last_peered":"2024-05-15T23:14:16.808877+0000","last_clean":"2024-05-15T23:14:16.808877+0000","last_became_active":"2024-05-15T23:10:41.796378+0000","last_became_peered":"2024-05-15T23:10:41.796378+0000","last_unstale":"2024-05-15T23:14:16.808877+0000","last_undegraded":"2024-05-15T23:14:16.808877+0000","last_fullsized":"2024-05-15T23:14:16.808877+0000","mapping_epoch":202,"log_start":"237'8600","ondisk_log_start":"237'8600","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'14929","last_scrub_stamp":"2024-05-15T23:13:25.696132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:25.696132+0000","objects_scrubbed":5637,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:52.263125+0000","scrub_duration":1288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119537664,"num_objects":1824,"num_object_clones":0,"num_object_copies":3648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18680,"num_write_kb":656128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"272'18480","reported_seq":18780,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:15.085543+0000","last_change":"2024-05-15T23:13:32.720230+0000","last_active":"2024-05-15T23:14:15.085543+0000","last_peered":"2024-05-15T23:14:15.085543+0000","last_clean":"2024-05-15T23:14:15.085543+0000","last_became_active":"2024-05-15T23:10:41.792966+0000","last_became_peered":"2024-05-15T23:10:41.792966+0000","last_unstale":"2024-05-15T23:14:15.085543+0000","last_undegraded":"2024-05-15T23:14:15.085543+0000","last_fullsized":"2024-05-15T23:14:15.085543+0000","mapping_epoch":202,"log_start":"235'8400","ondisk_log_start":"235'8400","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15337","last_scrub_stamp":"2024-05-15T23:13:32.720051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:32.720051+0000","objects_scrubbed":5192,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:33.190944+0000","scrub_duration":1161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131465216,"num_objects":2006,"num_object_clones":0,"num_object_copies":4012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18480,"num_write_kb":655552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"272'18583","reported_seq":18896,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:14.560507+0000","last_change":"2024-05-15T23:13:30.204128+0000","last_active":"2024-05-15T23:14:14.560507+0000","last_peered":"2024-05-15T23:14:14.560507+0000","last_clean":"2024-05-15T23:14:14.560507+0000","last_became_active":"2024-05-15T23:10:41.794364+0000","last_became_peered":"2024-05-15T23:10:41.794364+0000","last_unstale":"2024-05-15T23:14:14.560507+0000","last_undegraded":"2024-05-15T23:14:14.560507+0000","last_fullsized":"2024-05-15T23:14:14.560507+0000","mapping_epoch":202,"log_start":"235'8500","ondisk_log_start":"235'8500","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15231","last_scrub_stamp":"2024-05-15T23:13:30.203931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:30.203931+0000","objects_scrubbed":5476,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:44.109942+0000","scrub_duration":1269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136118272,"num_objects":2077,"num_object_clones":0,"num_object_copies":4154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18583,"num_write_kb":661120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"272'18587","reported_seq":18899,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:14.256344+0000","last_change":"2024-05-15T23:13:29.043358+0000","last_active":"2024-05-15T23:14:14.256344+0000","last_peered":"2024-05-15T23:14:14.256344+0000","last_clean":"2024-05-15T23:14:14.256344+0000","last_became_active":"2024-05-15T23:10:41.794291+0000","last_became_peered":"2024-05-15T23:10:41.794291+0000","last_unstale":"2024-05-15T23:14:14.256344+0000","last_undegraded":"2024-05-15T23:14:14.256344+0000","last_fullsized":"2024-05-15T23:14:14.256344+0000","mapping_epoch":202,"log_start":"235'8500","ondisk_log_start":"235'8500","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15210","last_scrub_stamp":"2024-05-15T23:13:29.043207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:29.043207+0000","objects_scrubbed":5487,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:33.486232+0000","scrub_duration":1073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135856128,"num_objects":2073,"num_object_clones":0,"num_object_copies":4146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18587,"num_write_kb":661120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"272'19041","reported_seq":19352,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:17.450702+0000","last_change":"2024-05-15T23:13:36.477604+0000","last_active":"2024-05-15T23:14:17.450702+0000","last_peered":"2024-05-15T23:14:17.450702+0000","last_clean":"2024-05-15T23:14:17.450702+0000","last_became_active":"2024-05-15T23:10:42.014274+0000","last_became_peered":"2024-05-15T23:10:42.014274+0000","last_unstale":"2024-05-15T23:14:17.450702+0000","last_undegraded":"2024-05-15T23:14:17.450702+0000","last_fullsized":"2024-05-15T23:14:17.450702+0000","mapping_epoch":202,"log_start":"238'9000","ondisk_log_start":"238'9000","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'15877","last_scrub_stamp":"2024-05-15T23:13:36.477324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:13:36.477324+0000","objects_scrubbed":5072,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:15:05.465959+0000","scrub_duration":1039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122748928,"num_objects":1873,"num_object_clones":0,"num_object_copies":3746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19041,"num_write_kb":669248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"272'18578","reported_seq":18920,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:16.065504+0000","last_change":"2024-05-15T23:14:09.458972+0000","last_active":"2024-05-15T23:14:16.065504+0000","last_peered":"2024-05-15T23:14:16.065504+0000","last_clean":"2024-05-15T23:14:16.065504+0000","last_became_active":"2024-05-15T23:10:42.014285+0000","last_became_peered":"2024-05-15T23:10:42.014285+0000","last_unstale":"2024-05-15T23:14:16.065504+0000","last_undegraded":"2024-05-15T23:14:16.065504+0000","last_fullsized":"2024-05-15T23:14:16.065504+0000","mapping_epoch":202,"log_start":"236'8500","ondisk_log_start":"236'8500","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'18093","last_scrub_stamp":"2024-05-15T23:14:09.458753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:10:40.776599+0000","last_clean_scrub_stamp":"2024-05-15T23:14:09.458753+0000","objects_scrubbed":2424,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:15:14.178709+0000","scrub_duration":569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":1920,"num_object_clones":0,"num_object_copies":3840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18578,"num_write_kb":655936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":539,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-05-15T23:14:07.792242+0000","last_change":"2024-05-15T23:12:43.939662+0000","last_active":"2024-05-15T23:14:07.792242+0000","last_peered":"2024-05-15T23:14:07.792242+0000","last_clean":"2024-05-15T23:14:07.792242+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:14:07.792242+0000","last_undegraded":"2024-05-15T23:14:07.792242+0000","last_fullsized":"2024-05-15T23:14:07.792242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:12:43.939526+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:12:43.939526+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:14:11.827428+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":2059796504,"num_objects":31431,"num_object_clones":0,"num_object_copies":62862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31431,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":299651,"num_write_kb":10594561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4133691392,"data_stored":4133683248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161059,"ondisk_log_size":161059,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351102,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1163796,"kb_used_data":848012,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92581356,"statfs":{"total":95995035648,"available":94803308544,"internally_reserved":0,"allocated":868364288,"data_stored":867999959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":323353028},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":63828,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93681324,"statfs":{"total":95995035648,"available":95929675776,"internally_reserved":0,"allocated":724992,"data_stored":364235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776267,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":457868,"kb_used_data":264260,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93287284,"statfs":{"total":95995035648,"available":95526178816,"internally_reserved":0,"allocated":270602240,"data_stored":270241483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":198244800},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005199,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":251980,"kb_used_data":134724,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93493172,"statfs":{"total":95995035648,"available":95737008128,"internally_reserved":0,"allocated":137957376,"data_stored":137596619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":387160,"kb_used_data":261584,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93357992,"statfs":{"total":95995035648,"available":95598583808,"internally_reserved":0,"allocated":267862016,"data_stored":267493635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128580042},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":202508,"kb_used_data":126916,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93542644,"statfs":{"total":95995035648,"available":95787667456,"internally_reserved":0,"allocated":129961984,"data_stored":129601227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575027,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":555276,"kb_used_data":397764,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93189876,"statfs":{"total":95995035648,"available":95426433024,"internally_reserved":0,"allocated":407310336,"data_stored":406949579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":161282502},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738552,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":789900,"kb_used_data":537988,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":92955252,"statfs":{"total":95995035648,"available":95186178048,"internally_reserved":0,"allocated":550899712,"data_stored":550538955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":868300,"kb_used_data":620548,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92876852,"statfs":{"total":95995035648,"available":95105896448,"internally_reserved":0,"allocated":635441152,"data_stored":635080395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":253688263},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986338,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":729356,"kb_used_data":545476,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93015796,"statfs":{"total":95995035648,"available":95248175104,"internally_reserved":0,"allocated":558567424,"data_stored":558206667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790109,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":501776,"kb_used_data":366472,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93243376,"statfs":{"total":95995035648,"available":95481217024,"internally_reserved":0,"allocated":375267328,"data_stored":374902499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":546635776,"data_stored":546635776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401670144,"data_stored":401670144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":630325248,"data_stored":630325248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":545914880,"data_stored":545914880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":369037312,"data_stored":369033240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":125435904,"data_stored":125435904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":264638464,"data_stored":264634392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":136445952,"data_stored":136445952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":265289728,"data_stored":265289728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":848297984,"data_stored":848297984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:14:21.399 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-15T23:14:21.399 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-05-15T23:14:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:21.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:21.450 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:21.503 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:21.544 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:21.795 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:21.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:21.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:21.905 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:21.966 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:22.060 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:22.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:22.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:22.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:21 smithi088 ceph-mon[44699]: pgmap v765: 17 pgs: 17 active+clean; 1.8 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:14:22.135 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:14:22.135 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:14:22.163 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:22.206 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:22.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:14:22.270 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:14:22.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:21 smithi090 ceph-mon[44434]: pgmap v765: 17 pgs: 17 active+clean; 1.8 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:14:22.314 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:14:22.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:21 smithi084 ceph-mon[39784]: pgmap v765: 17 pgs: 17 active+clean; 1.8 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:14:22.359 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:14:22.390 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:14:22.424 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:14:22.453 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:14:22.496 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:14:22.523 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:14:22.554 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:14:22.687 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:14:22.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:14:22.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:14:22.688 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-05-15T23:14:22.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:14:22.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:14:22.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:14:22.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:14:22.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:14:22.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:14:22.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:14:22.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:14:22.689 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:14:22.868 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:22.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:22.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:22.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T23:14:22.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:23.011 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:23.038 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:23.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:23.041 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:23.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:23.196 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:14:23.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1866646786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:23.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:23.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:23.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:23.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1866646786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:23.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:23.351 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:23.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:23.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:23.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1866646786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:23.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:23.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:23.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:23.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:23.462 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:23.493 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:23.518 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:14:23.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:23.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:14:23.624 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:23.667 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:23.800 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:23.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:23.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:23.818 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:14:23.818 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:14:23.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:23.951 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:24.108 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:24.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:24.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:24.115 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.115 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.116 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.116 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.116 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.116 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.116 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.116 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:23 smithi090 ceph-mon[44434]: pgmap v766: 17 pgs: 17 active+clean; 1.6 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.31k op/s 2024-05-15T23:14:24.216 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:24.261 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:23 smithi088 ceph-mon[44699]: pgmap v766: 17 pgs: 17 active+clean; 1.6 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.31k op/s 2024-05-15T23:14:24.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:23 smithi084 ceph-mon[39784]: pgmap v766: 17 pgs: 17 active+clean; 1.6 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.31k op/s 2024-05-15T23:14:24.719 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:24.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:24.824 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:24.880 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:24.932 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:24.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:24.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:24.975 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:24 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/359871006' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:14:24.975 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:24 smithi090 ceph-mon[44434]: from='client.35468 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:14:24.998 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:14:25.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:25.083 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:25.184 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:25.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:25.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:25.290 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:25.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:24 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/359871006' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:14:25.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:24 smithi088 ceph-mon[44699]: from='client.35468 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:14:25.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:24 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/359871006' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:14:25.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:24 smithi084 ceph-mon[39784]: from='client.35468 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:14:25.346 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:14:25.346 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:14:25.346 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-05-15T23:14:25.347 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:14:25.347 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:25.819 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:14:25.905 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:25.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:25.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:26.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:26.043 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:26.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T23:14:26.084 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:26.097 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:26.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:25 smithi084 ceph-mon[39784]: pgmap v767: 17 pgs: 17 active+clean; 1.5 GiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:14:26.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:25 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:14:26.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:25 smithi084 ceph-mon[39784]: from='client.35468 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:14:26.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:25 smithi084 ceph-mon[39784]: osdmap e275: 12 total, 11 up, 11 in 2024-05-15T23:14:26.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:14:26.210 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:26.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:25 smithi090 ceph-mon[44434]: pgmap v767: 17 pgs: 17 active+clean; 1.5 GiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:14:26.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:25 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:14:26.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:25 smithi090 ceph-mon[44434]: from='client.35468 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:14:26.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:25 smithi090 ceph-mon[44434]: osdmap e275: 12 total, 11 up, 11 in 2024-05-15T23:14:26.238 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:26.269 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:14:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:26.272 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:14:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:25 smithi088 ceph-mon[44699]: pgmap v767: 17 pgs: 17 active+clean; 1.5 GiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:14:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:25 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:14:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:25 smithi088 ceph-mon[44699]: from='client.35468 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:14:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:25 smithi088 ceph-mon[44699]: osdmap e275: 12 total, 11 up, 11 in 2024-05-15T23:14:26.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:26.415 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:26.451 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:26.455 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:26.461 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:14:26.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:26.614 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:26.640 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:26.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:26.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:26.745 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:26.787 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:27.662 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:27.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:27.768 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:27.815 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:27.890 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:14:28.028 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:28.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:28.141 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:27 smithi090 ceph-mon[44434]: pgmap v769: 17 pgs: 17 active+clean; 1.3 GiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:14:28.143 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:14:28.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:27 smithi084 ceph-mon[39784]: pgmap v769: 17 pgs: 17 active+clean; 1.3 GiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:14:28.187 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:28.224 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:14:28.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:28.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:14:28.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:27 smithi088 ceph-mon[44699]: pgmap v769: 17 pgs: 17 active+clean; 1.3 GiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:14:28.333 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:28.394 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:28.423 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:28.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:28.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:28.434 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:28.461 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:14:28.488 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:14:28.489 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:14:28.489 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:14:28.489 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:14:28.489 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:14:28.496 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:14:28.530 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:28.533 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:14:28.558 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:28.561 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:14:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:28.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:28.598 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:14:28.639 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:14:28.674 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:14:28.697 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:28.707 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:14:28.741 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:14:28.757 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:28.770 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:14:28.788 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:28.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:28.816 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:14:28.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:14:28.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:14:28.817 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-05-15T23:14:28.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:14:28.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:14:28.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:14:28.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:14:28.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:14:28.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:14:28.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:14:28.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:14:28.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:14:28.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:28.931 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:29.137 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:29.137 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2785706742' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:29.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:29.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:29.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T23:14:29.249 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:14:29.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2785706742' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:29.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.279 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:29.304 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:14:29.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:29.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:14:29.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2785706742' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:29.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:29.453 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:29.490 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:29.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:29.555 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:14:29.596 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:29.641 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:29.666 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:29.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:29.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:29.771 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:29.812 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:29.834 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:29.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:29.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:29.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:14:29.986 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:29.997 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.998 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.998 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.998 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.998 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.998 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.998 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:29.998 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:29 smithi088 ceph-mon[44699]: pgmap v770: 17 pgs: 17 active+clean; 1.2 GiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:14:30.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:30.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:30.140 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:30.140 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:30.140 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:30.140 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:30.140 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:30.140 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:29 smithi090 ceph-mon[44434]: pgmap v770: 17 pgs: 17 active+clean; 1.2 GiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:14:30.222 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:14:30.223 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:14:30.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:30.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:30.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:30.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:30.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:30.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:30.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:30.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:29 smithi084 ceph-mon[39784]: pgmap v770: 17 pgs: 17 active+clean; 1.2 GiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:14:30.442 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:14:30.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:30.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:14:30.546 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:30.578 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:31.038 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:31.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:31.042 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:31.143 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:31.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:31 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:14:31.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:31 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:14:31.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:31 smithi084 ceph-mon[39784]: osdmap e276: 12 total, 11 up, 11 in 2024-05-15T23:14:31.180 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:31.227 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:31.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:31.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:31 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:14:31.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:31 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:14:31.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:31 smithi090 ceph-mon[44434]: osdmap e276: 12 total, 11 up, 11 in 2024-05-15T23:14:31.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:31 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:14:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:31 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:14:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:31 smithi088 ceph-mon[44699]: osdmap e276: 12 total, 11 up, 11 in 2024-05-15T23:14:31.332 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:14:31.382 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:31.418 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:14:31.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:31.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:14:31.527 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:31.557 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:31.587 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:14:31.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:31.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:14:31.648 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:14:31.648 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:14:31.694 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:31.726 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:31.752 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:14:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:31.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:14:31.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:31.891 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:31.920 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:31.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:31.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:32.025 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:32.026 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:32 smithi090 ceph-mon[44434]: pgmap v772: 17 pgs: 17 active+clean; 1.0 GiB data, 3.9 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-05-15T23:14:32.026 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/838108' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:14:32.051 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:14:32.061 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:32.092 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:32.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:32.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:32.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:32.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:14:32.244 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:32.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:32 smithi084 ceph-mon[39784]: pgmap v772: 17 pgs: 17 active+clean; 1.0 GiB data, 3.9 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-05-15T23:14:32.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/838108' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:14:32.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:32 smithi088 ceph-mon[44699]: pgmap v772: 17 pgs: 17 active+clean; 1.0 GiB data, 3.9 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-05-15T23:14:32.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/838108' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:14:32.412 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:32.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:32.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:32.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T23:14:32.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:32.565 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:32.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:32.618 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:32.719 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:32.768 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:32.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:32.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:32.858 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:14:32.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:32.958 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:33.035 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:33 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:14:33.035 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:33 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/838108' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:14:33.035 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:33 smithi090 ceph-mon[44434]: osdmap e277: 12 total, 11 up, 11 in 2024-05-15T23:14:33.053 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:33.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:33.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:33.304 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:33.326 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:33 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:14:33.326 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:33 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/838108' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:14:33.326 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:33 smithi088 ceph-mon[44699]: osdmap e277: 12 total, 11 up, 11 in 2024-05-15T23:14:33.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:33 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:14:33.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:33 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/838108' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:14:33.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:33 smithi084 ceph-mon[39784]: osdmap e277: 12 total, 11 up, 11 in 2024-05-15T23:14:33.386 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:33.393 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:33.494 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:33.552 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:33.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:33.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:33.657 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:33.689 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:33.718 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:14:33.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:33.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:14:33.823 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:33.859 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:33.883 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:33.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:33.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:33.992 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:34.031 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:34.054 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:34.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:34.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:34 smithi084 ceph-mon[39784]: pgmap v774: 17 pgs: 17 active+clean; 848 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.31k op/s 2024-05-15T23:14:34.160 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:34.202 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:34.223 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:34.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:34.295 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:34 smithi090 ceph-mon[44434]: pgmap v774: 17 pgs: 17 active+clean; 848 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.31k op/s 2024-05-15T23:14:34.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:34 smithi088 ceph-mon[44699]: pgmap v774: 17 pgs: 17 active+clean; 848 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.31k op/s 2024-05-15T23:14:34.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:34.377 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:34.403 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:34.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:34.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:34.537 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:34.673 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:34.677 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:34.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:14:34.811 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:34.840 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:14:34.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:34.843 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:14:34.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:34.971 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:14:34.992 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:35.017 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:14:35.017 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:14:35.027 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:35.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:35.035 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:35.136 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:35.172 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:35.195 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:35.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:35.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:35.259 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:14:35.300 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:14:35.301 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:14:35.330 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:35.348 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:14:35.359 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:14:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:35.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:14:35.385 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:14:35.390 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:35.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:14:35.456 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:14:35.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:14:35.486 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:14:35.516 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:14:35.521 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:35.550 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:14:35.582 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:14:35.583 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:14:35.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:35.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:14:35.618 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:14:35.618 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:14:35.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:14:35.619 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-05-15T23:14:35.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:14:35.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:14:35.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:14:35.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:14:35.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:14:35.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:14:35.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:14:35.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:14:35.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:14:35.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:35.739 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:35.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T23:14:35.772 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:35.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:35.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:35.878 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:35.912 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:35.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:35.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:36.050 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:36.079 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:36.102 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:36.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:36.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:36.153 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:14:36.205 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:36.236 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:36 smithi088 ceph-mon[44699]: pgmap v775: 17 pgs: 17 active+clean; 754 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:14:36.236 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1919460867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:36.236 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.236 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.236 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.236 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.236 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.237 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.238 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:36.264 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:36.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:36.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:36.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:36 smithi084 ceph-mon[39784]: pgmap v775: 17 pgs: 17 active+clean; 754 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:14:36.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1919460867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:36.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:14:36.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:36 smithi090 ceph-mon[44434]: pgmap v775: 17 pgs: 17 active+clean; 754 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:14:36.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1919460867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:36.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:36.400 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:36.465 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:14:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:36.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:14:36.569 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:36.601 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:36.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:36.731 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:36.759 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:36.781 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:14:36.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:36.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:14:36.886 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:14:36.928 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:36.959 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:14:36.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:36.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:14:37.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:37.112 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:37.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:37.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:37.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:37.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:37.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:37.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:37.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:37.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:37.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:37.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:37.674 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:37.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:37.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:37.779 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:37.813 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:37.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:37.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:38.034 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:38.086 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:38.272 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:14:38.272 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:14:38.272 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:14:38.272 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:14:38.272 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:14:38.300 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:38 smithi090 ceph-mon[44434]: pgmap v776: 17 pgs: 17 active+clean; 614 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:14:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:38 smithi088 ceph-mon[44699]: pgmap v776: 17 pgs: 17 active+clean; 614 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:14:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:38 smithi084 ceph-mon[39784]: pgmap v776: 17 pgs: 17 active+clean; 614 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:14:39.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T23:14:39.406 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:14:40.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:40 smithi090 ceph-mon[44434]: pgmap v777: 17 pgs: 17 active+clean; 472 MiB data, 2.8 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-05-15T23:14:40.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:40 smithi084 ceph-mon[39784]: pgmap v777: 17 pgs: 17 active+clean; 472 MiB data, 2.8 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-05-15T23:14:40.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:40 smithi088 ceph-mon[44699]: pgmap v777: 17 pgs: 17 active+clean; 472 MiB data, 2.8 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-05-15T23:14:41.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:41 smithi084 ceph-mon[39784]: pgmap v778: 17 pgs: 17 active+clean; 312 MiB data, 2.4 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:14:41.393 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:14:41.393 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:14:41.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:14:41.394 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:14:41.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.394 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:109644 2.7 e127/113 2:ea3584ac:::benchmark_data_smithi090_78160_object109643:head v 127'6843, mlcod=127'6840)", 2024-05-15T23:14:41.394 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.342608+0000", 2024-05-15T23:14:41.394 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 468.04740935900003, 2024-05-15T23:14:41.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13470748699999999, 2024-05-15T23:14:41.394 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.394 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.394 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.394 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.342608+0000", 2024-05-15T23:14:41.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.342608+0000", 2024-05-15T23:14:41.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.342609+0000", 2024-05-15T23:14:41.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.141e-06 2024-05-15T23:14:41.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.342666+0000", 2024-05-15T23:14:41.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7487e-05 2024-05-15T23:14:41.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.342675+0000", 2024-05-15T23:14:41.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9020000000000005e-06 2024-05-15T23:14:41.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.342680+0000", 2024-05-15T23:14:41.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.118e-06 2024-05-15T23:14:41.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466177+0000", 2024-05-15T23:14:41.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.123496724 2024-05-15T23:14:41.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466210+0000", 2024-05-15T23:14:41.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3429999999999997e-05 2024-05-15T23:14:41.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.477259+0000", 2024-05-15T23:14:41.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011048489999999999 2024-05-15T23:14:41.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.477315+0000", 2024-05-15T23:14:41.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6195e-05 2024-05-15T23:14:41.398 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.398 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.398 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.398 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:109647 2.7 e127/113 2:ef9645db:::benchmark_data_smithi090_78160_object109646:head v 127'6844, mlcod=127'6840)", 2024-05-15T23:14:41.398 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.343499+0000", 2024-05-15T23:14:41.398 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 468.04651838400002, 2024-05-15T23:14:41.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13389111400000001, 2024-05-15T23:14:41.399 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.399 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.399 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.399 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.343499+0000", 2024-05-15T23:14:41.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.343499+0000", 2024-05-15T23:14:41.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.343499+0000", 2024-05-15T23:14:41.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.61e-07 2024-05-15T23:14:41.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.343564+0000", 2024-05-15T23:14:41.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4554999999999994e-05 2024-05-15T23:14:41.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466159+0000", 2024-05-15T23:14:41.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12259522 2024-05-15T23:14:41.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466169+0000", 2024-05-15T23:14:41.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0261999999999999e-05 2024-05-15T23:14:41.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.470611+0000", 2024-05-15T23:14:41.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044414249999999997 2024-05-15T23:14:41.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.470636+0000", 2024-05-15T23:14:41.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5601000000000001e-05 2024-05-15T23:14:41.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.477361+0000", 2024-05-15T23:14:41.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0067251689999999996 2024-05-15T23:14:41.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.477390+0000", 2024-05-15T23:14:41.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8221e-05 2024-05-15T23:14:41.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.403 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.403 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.403 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:190066 2.f e148/102 2:f0cd21a5:::benchmark_data_smithi090_78160_object28708:head v 148'12016, mlcod=148'12014)", 2024-05-15T23:14:41.403 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:07:59.711270+0000", 2024-05-15T23:14:41.403 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 401.67874729599998, 2024-05-15T23:14:41.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032508072999999998, 2024-05-15T23:14:41.403 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.403 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.403 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.403 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.711270+0000", 2024-05-15T23:14:41.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.711270+0000", 2024-05-15T23:14:41.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.711272+0000", 2024-05-15T23:14:41.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1110000000000002e-06 2024-05-15T23:14:41.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.711280+0000", 2024-05-15T23:14:41.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.602e-06 2024-05-15T23:14:41.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.711289+0000", 2024-05-15T23:14:41.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9390000000000003e-06 2024-05-15T23:14:41.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.711299+0000", 2024-05-15T23:14:41.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4690000000000003e-06 2024-05-15T23:14:41.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.712444+0000", 2024-05-15T23:14:41.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011452210000000001 2024-05-15T23:14:41.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.712465+0000", 2024-05-15T23:14:41.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1316000000000001e-05 2024-05-15T23:14:41.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.743733+0000", 2024-05-15T23:14:41.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031267700000000002 2024-05-15T23:14:41.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.743778+0000", 2024-05-15T23:14:41.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4715000000000001e-05 2024-05-15T23:14:41.407 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.407 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.407 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.407 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:190070 2.7 e148/113 2:e98e57fa:::benchmark_data_smithi090_78160_object28712:head v 148'11905, mlcod=148'11902)", 2024-05-15T23:14:41.407 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:07:59.718016+0000", 2024-05-15T23:14:41.407 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 401.67200075300002, 2024-05-15T23:14:41.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030670510000000002, 2024-05-15T23:14:41.407 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.408 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.408 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.408 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.718016+0000", 2024-05-15T23:14:41.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.718016+0000", 2024-05-15T23:14:41.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.718018+0000", 2024-05-15T23:14:41.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.356e-06 2024-05-15T23:14:41.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.718024+0000", 2024-05-15T23:14:41.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7240000000000003e-06 2024-05-15T23:14:41.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.718032+0000", 2024-05-15T23:14:41.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9880000000000004e-06 2024-05-15T23:14:41.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.718038+0000", 2024-05-15T23:14:41.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9089999999999997e-06 2024-05-15T23:14:41.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.721269+0000", 2024-05-15T23:14:41.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032312270000000001 2024-05-15T23:14:41.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.721290+0000", 2024-05-15T23:14:41.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0630000000000001e-05 2024-05-15T23:14:41.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.748643+0000", 2024-05-15T23:14:41.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027353454999999999 2024-05-15T23:14:41.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.748687+0000", 2024-05-15T23:14:41.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3220999999999998e-05 2024-05-15T23:14:41.411 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.411 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.411 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.412 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:190072 2.f e148/102 2:f4642f8a:::benchmark_data_smithi090_78160_object28714:head v 148'12017, mlcod=148'12014)", 2024-05-15T23:14:41.412 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:07:59.721274+0000", 2024-05-15T23:14:41.412 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 401.66874314799998, 2024-05-15T23:14:41.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030545539999999999, 2024-05-15T23:14:41.412 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.412 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.412 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.412 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.721274+0000", 2024-05-15T23:14:41.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.721274+0000", 2024-05-15T23:14:41.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.721275+0000", 2024-05-15T23:14:41.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4780000000000001e-06 2024-05-15T23:14:41.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.721284+0000", 2024-05-15T23:14:41.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0559999999999994e-06 2024-05-15T23:14:41.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.721295+0000", 2024-05-15T23:14:41.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0915e-05 2024-05-15T23:14:41.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.721303+0000", 2024-05-15T23:14:41.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1079999999999993e-06 2024-05-15T23:14:41.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.723294+0000", 2024-05-15T23:14:41.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019903339999999999 2024-05-15T23:14:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.723310+0000", 2024-05-15T23:14:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5957e-05 2024-05-15T23:14:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.751774+0000", 2024-05-15T23:14:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028464817 2024-05-15T23:14:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:07:59.751819+0000", 2024-05-15T23:14:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4875e-05 2024-05-15T23:14:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:192780 2.d 2:bcee433c:::benchmark_data_smithi090_78160_object31422:head [delete] snapc 0=[] ondisk+write+known_if_redirected e148)", 2024-05-15T23:14:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:02.141499+0000", 2024-05-15T23:14:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 399.24851794900002, 2024-05-15T23:14:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030114086000000002, 2024-05-15T23:14:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:14:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:14:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:14:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192780 2024-05-15T23:14:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:02.141499+0000", 2024-05-15T23:14:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:02.141499+0000", 2024-05-15T23:14:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:02.141500+0000", 2024-05-15T23:14:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4640000000000001e-06 2024-05-15T23:14:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:02.141505+0000", 2024-05-15T23:14:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8289999999999998e-06 2024-05-15T23:14:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:02.141513+0000", 2024-05-15T23:14:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1799999999999996e-06 2024-05-15T23:14:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:02.141520+0000", 2024-05-15T23:14:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4119999999999998e-06 2024-05-15T23:14:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:02.157559+0000", 2024-05-15T23:14:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016038963 2024-05-15T23:14:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:02.157627+0000", 2024-05-15T23:14:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8330000000000005e-05 2024-05-15T23:14:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:14:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:02.157691+0000", 2024-05-15T23:14:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3966000000000004e-05 2024-05-15T23:14:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:14:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:02.171499+0000", 2024-05-15T23:14:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013807577 2024-05-15T23:14:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:14:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:02.171516+0000", 2024-05-15T23:14:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7453999999999999e-05 2024-05-15T23:14:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:02.171567+0000", 2024-05-15T23:14:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0992000000000002e-05 2024-05-15T23:14:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:02.171613+0000", 2024-05-15T23:14:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5918999999999999e-05 2024-05-15T23:14:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:216842 2.7 e150/113 2:e86fad87:::benchmark_data_smithi090_78160_object55484:head v 150'13540, mlcod=150'13538)", 2024-05-15T23:14:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:23.003298+0000", 2024-05-15T23:14:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 378.38671888699997, 2024-05-15T23:14:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030997403, 2024-05-15T23:14:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.003298+0000", 2024-05-15T23:14:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.003298+0000", 2024-05-15T23:14:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.003300+0000", 2024-05-15T23:14:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.491e-06 2024-05-15T23:14:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.003307+0000", 2024-05-15T23:14:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5020000000000003e-06 2024-05-15T23:14:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.003318+0000", 2024-05-15T23:14:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1341e-05 2024-05-15T23:14:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.003326+0000", 2024-05-15T23:14:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4440000000000002e-06 2024-05-15T23:14:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.003380+0000", 2024-05-15T23:14:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3723e-05 2024-05-15T23:14:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.003404+0000", 2024-05-15T23:14:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4148999999999999e-05 2024-05-15T23:14:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.034248+0000", 2024-05-15T23:14:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030844529999999998 2024-05-15T23:14:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.034295+0000", 2024-05-15T23:14:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7222999999999998e-05 2024-05-15T23:14:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:216844 2.7 e150/113 2:ed8e565c:::benchmark_data_smithi090_78160_object55486:head v 150'13541, mlcod=150'13538)", 2024-05-15T23:14:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:23.003565+0000", 2024-05-15T23:14:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 378.38645205199998, 2024-05-15T23:14:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030765443, 2024-05-15T23:14:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.003565+0000", 2024-05-15T23:14:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.003565+0000", 2024-05-15T23:14:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.003566+0000", 2024-05-15T23:14:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0109999999999999e-06 2024-05-15T23:14:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.003575+0000", 2024-05-15T23:14:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2280000000000004e-06 2024-05-15T23:14:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.003585+0000", 2024-05-15T23:14:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3710000000000006e-06 2024-05-15T23:14:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.003590+0000", 2024-05-15T23:14:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9440000000000002e-06 2024-05-15T23:14:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.012447+0000", 2024-05-15T23:14:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0088564400000000001 2024-05-15T23:14:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.012466+0000", 2024-05-15T23:14:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9137e-05 2024-05-15T23:14:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.034299+0000", 2024-05-15T23:14:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021832527000000001 2024-05-15T23:14:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:23.034330+0000", 2024-05-15T23:14:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1785000000000001e-05 2024-05-15T23:14:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:218265 2.f e150/102 2:fe74ff58:::benchmark_data_smithi090_78160_object56907:head v 150'13792, mlcod=150'13789)", 2024-05-15T23:14:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:24.252522+0000", 2024-05-15T23:14:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 377.13749489700001, 2024-05-15T23:14:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033046127000000002, 2024-05-15T23:14:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.252522+0000", 2024-05-15T23:14:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.252522+0000", 2024-05-15T23:14:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.252523+0000", 2024-05-15T23:14:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.426e-06 2024-05-15T23:14:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.252530+0000", 2024-05-15T23:14:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4930000000000003e-06 2024-05-15T23:14:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.252541+0000", 2024-05-15T23:14:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0878e-05 2024-05-15T23:14:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.252548+0000", 2024-05-15T23:14:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1330000000000001e-06 2024-05-15T23:14:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.256336+0000", 2024-05-15T23:14:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037883550000000002 2024-05-15T23:14:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.256357+0000", 2024-05-15T23:14:41.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0712000000000001e-05 2024-05-15T23:14:41.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.285522+0000", 2024-05-15T23:14:41.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029165083000000001 2024-05-15T23:14:41.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.285568+0000", 2024-05-15T23:14:41.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6047000000000001e-05 2024-05-15T23:14:41.434 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.435 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.435 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.435 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:218832 2.7 e151/113 2:e28812ea:::benchmark_data_smithi090_78160_object57474:head v 151'13672, mlcod=151'13667)", 2024-05-15T23:14:41.435 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:24.759548+0000", 2024-05-15T23:14:41.435 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 376.630468721, 2024-05-15T23:14:41.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040162389, 2024-05-15T23:14:41.435 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.435 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.435 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.435 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.759548+0000", 2024-05-15T23:14:41.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.759548+0000", 2024-05-15T23:14:41.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.759549+0000", 2024-05-15T23:14:41.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6699999999999999e-07 2024-05-15T23:14:41.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.759553+0000", 2024-05-15T23:14:41.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5639999999999998e-06 2024-05-15T23:14:41.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.759559+0000", 2024-05-15T23:14:41.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3059999999999997e-06 2024-05-15T23:14:41.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.759563+0000", 2024-05-15T23:14:41.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2729999999999996e-06 2024-05-15T23:14:41.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.763776+0000", 2024-05-15T23:14:41.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042129730000000001 2024-05-15T23:14:41.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.763801+0000", 2024-05-15T23:14:41.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5165e-05 2024-05-15T23:14:41.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.799650+0000", 2024-05-15T23:14:41.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035849363000000002 2024-05-15T23:14:41.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.799711+0000", 2024-05-15T23:14:41.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0278000000000001e-05 2024-05-15T23:14:41.439 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.439 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.439 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.439 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:218835 2.f e151/102 2:f87f350f:::benchmark_data_smithi090_78160_object57477:head v 151'13830, mlcod=151'13828)", 2024-05-15T23:14:41.439 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:24.765846+0000", 2024-05-15T23:14:41.440 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 376.62417120600003, 2024-05-15T23:14:41.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033905612000000002, 2024-05-15T23:14:41.440 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.440 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.440 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.440 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.765846+0000", 2024-05-15T23:14:41.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.765846+0000", 2024-05-15T23:14:41.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.765848+0000", 2024-05-15T23:14:41.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.792e-06 2024-05-15T23:14:41.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.765856+0000", 2024-05-15T23:14:41.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8489999999999995e-06 2024-05-15T23:14:41.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.765867+0000", 2024-05-15T23:14:41.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.044e-05 2024-05-15T23:14:41.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.765875+0000", 2024-05-15T23:14:41.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2190000000000004e-06 2024-05-15T23:14:41.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.767817+0000", 2024-05-15T23:14:41.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019416189999999999 2024-05-15T23:14:41.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.767837+0000", 2024-05-15T23:14:41.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0034000000000002e-05 2024-05-15T23:14:41.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.799715+0000", 2024-05-15T23:14:41.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031878788999999998 2024-05-15T23:14:41.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.799751+0000", 2024-05-15T23:14:41.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.587e-05 2024-05-15T23:14:41.443 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.443 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.444 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.444 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:218838 2.f e151/102 2:fefdbe18:::benchmark_data_smithi090_78160_object57480:head v 151'13831, mlcod=151'13828)", 2024-05-15T23:14:41.444 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:24.766166+0000", 2024-05-15T23:14:41.444 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 376.62385095600001, 2024-05-15T23:14:41.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033616974000000001, 2024-05-15T23:14:41.444 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.444 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.444 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.444 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.766166+0000", 2024-05-15T23:14:41.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.766166+0000", 2024-05-15T23:14:41.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.766167+0000", 2024-05-15T23:14:41.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3149999999999999e-06 2024-05-15T23:14:41.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.766174+0000", 2024-05-15T23:14:41.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6429999999999998e-06 2024-05-15T23:14:41.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.766183+0000", 2024-05-15T23:14:41.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3689999999999996e-06 2024-05-15T23:14:41.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.766190+0000", 2024-05-15T23:14:41.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5919999999999997e-06 2024-05-15T23:14:41.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.772772+0000", 2024-05-15T23:14:41.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0065822629999999997 2024-05-15T23:14:41.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.772796+0000", 2024-05-15T23:14:41.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3762999999999999e-05 2024-05-15T23:14:41.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.799753+0000", 2024-05-15T23:14:41.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026957525999999999 2024-05-15T23:14:41.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:24.799783+0000", 2024-05-15T23:14:41.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9502999999999999e-05 2024-05-15T23:14:41.448 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.448 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.448 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.448 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:243096 2.7 e156/113 2:ebf01465:::benchmark_data_smithi090_78160_object81738:head v 156'15225, mlcod=156'15221)", 2024-05-15T23:14:41.448 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:47.551622+0000", 2024-05-15T23:14:41.448 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 353.838394859, 2024-05-15T23:14:41.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032309602999999999, 2024-05-15T23:14:41.448 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.448 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.448 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.449 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.551622+0000", 2024-05-15T23:14:41.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.551622+0000", 2024-05-15T23:14:41.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.551623+0000", 2024-05-15T23:14:41.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.178e-06 2024-05-15T23:14:41.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.551630+0000", 2024-05-15T23:14:41.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6019999999999996e-06 2024-05-15T23:14:41.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.551640+0000", 2024-05-15T23:14:41.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0161e-05 2024-05-15T23:14:41.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.551646+0000", 2024-05-15T23:14:41.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4579999999999998e-06 2024-05-15T23:14:41.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.557950+0000", 2024-05-15T23:14:41.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063038490000000003 2024-05-15T23:14:41.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.557966+0000", 2024-05-15T23:14:41.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5319e-05 2024-05-15T23:14:41.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.583891+0000", 2024-05-15T23:14:41.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025925093999999999 2024-05-15T23:14:41.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:47.583932+0000", 2024-05-15T23:14:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0942000000000002e-05 2024-05-15T23:14:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:244224 2.f e156/102 2:f217fe16:::benchmark_data_smithi090_78160_object82866:head v 156'15430, mlcod=156'15425)", 2024-05-15T23:14:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:08:48.616297+0000", 2024-05-15T23:14:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 352.77372016300001, 2024-05-15T23:14:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030500169000000001, 2024-05-15T23:14:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.453 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.454 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:48.616297+0000", 2024-05-15T23:14:41.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:48.616297+0000", 2024-05-15T23:14:41.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:48.616297+0000", 2024-05-15T23:14:41.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4299999999999999e-07 2024-05-15T23:14:41.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:48.616301+0000", 2024-05-15T23:14:41.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9400000000000004e-06 2024-05-15T23:14:41.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:48.616305+0000", 2024-05-15T23:14:41.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9770000000000002e-06 2024-05-15T23:14:41.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:48.616309+0000", 2024-05-15T23:14:41.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7230000000000002e-06 2024-05-15T23:14:41.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:48.625236+0000", 2024-05-15T23:14:41.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0089271430000000002 2024-05-15T23:14:41.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:48.625257+0000", 2024-05-15T23:14:41.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0710999999999999e-05 2024-05-15T23:14:41.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:48.646764+0000", 2024-05-15T23:14:41.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021507082 2024-05-15T23:14:41.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:08:48.646797+0000", 2024-05-15T23:14:41.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3250000000000002e-05 2024-05-15T23:14:41.457 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.457 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.457 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.458 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:260098 2.f e165/164 2:f7214dda:::benchmark_data_smithi090_78160_object98740:head v 165'16425, mlcod=165'16422)", 2024-05-15T23:14:41.458 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:09:06.144127+0000", 2024-05-15T23:14:41.458 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 335.24588956299999, 2024-05-15T23:14:41.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030952984999999999, 2024-05-15T23:14:41.458 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.458 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.458 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.458 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.144127+0000", 2024-05-15T23:14:41.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.144127+0000", 2024-05-15T23:14:41.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.144129+0000", 2024-05-15T23:14:41.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7999999999999999e-06 2024-05-15T23:14:41.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.144137+0000", 2024-05-15T23:14:41.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9799999999999998e-06 2024-05-15T23:14:41.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.144152+0000", 2024-05-15T23:14:41.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4844999999999999e-05 2024-05-15T23:14:41.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.144160+0000", 2024-05-15T23:14:41.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1280000000000008e-06 2024-05-15T23:14:41.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.146799+0000", 2024-05-15T23:14:41.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026386439999999999 2024-05-15T23:14:41.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.146820+0000", 2024-05-15T23:14:41.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1036999999999999e-05 2024-05-15T23:14:41.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.175031+0000", 2024-05-15T23:14:41.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028210740000000002 2024-05-15T23:14:41.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:06.175080+0000", 2024-05-15T23:14:41.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9811000000000001e-05 2024-05-15T23:14:41.461 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.462 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.462 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.462 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:263635 2.f e166/164 2:fbd06981:::benchmark_data_smithi090_78160_object102277:head v 166'16653, mlcod=166'16650)", 2024-05-15T23:14:41.462 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:09:09.813259+0000", 2024-05-15T23:14:41.462 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 331.576757622, 2024-05-15T23:14:41.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034043792000000003, 2024-05-15T23:14:41.462 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.462 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.462 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.462 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.813259+0000", 2024-05-15T23:14:41.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.813259+0000", 2024-05-15T23:14:41.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.813261+0000", 2024-05-15T23:14:41.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7179999999999999e-06 2024-05-15T23:14:41.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.813270+0000", 2024-05-15T23:14:41.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.329e-06 2024-05-15T23:14:41.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.813285+0000", 2024-05-15T23:14:41.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4284000000000001e-05 2024-05-15T23:14:41.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.813293+0000", 2024-05-15T23:14:41.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.568e-06 2024-05-15T23:14:41.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.813367+0000", 2024-05-15T23:14:41.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3628999999999996e-05 2024-05-15T23:14:41.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.813391+0000", 2024-05-15T23:14:41.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3858000000000001e-05 2024-05-15T23:14:41.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.847255+0000", 2024-05-15T23:14:41.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033864656999999999 2024-05-15T23:14:41.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.847303+0000", 2024-05-15T23:14:41.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7749e-05 2024-05-15T23:14:41.466 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.466 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.466 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.466 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:263632 2.7 e166/164 2:ed6af388:::benchmark_data_smithi090_78160_object102274:head v 166'16517, mlcod=166'16512)", 2024-05-15T23:14:41.466 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:09:09.815634+0000", 2024-05-15T23:14:41.466 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 331.574382646, 2024-05-15T23:14:41.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035355859000000003, 2024-05-15T23:14:41.467 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.467 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.467 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.467 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.815634+0000", 2024-05-15T23:14:41.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.815634+0000", 2024-05-15T23:14:41.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.815636+0000", 2024-05-15T23:14:41.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.663e-06 2024-05-15T23:14:41.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.815645+0000", 2024-05-15T23:14:41.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0999999999999993e-06 2024-05-15T23:14:41.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.815660+0000", 2024-05-15T23:14:41.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5316e-05 2024-05-15T23:14:41.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.815669+0000", 2024-05-15T23:14:41.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4700000000000002e-06 2024-05-15T23:14:41.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.817077+0000", 2024-05-15T23:14:41.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001408227 2024-05-15T23:14:41.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.817102+0000", 2024-05-15T23:14:41.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4793000000000001e-05 2024-05-15T23:14:41.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.850957+0000", 2024-05-15T23:14:41.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033855112999999999 2024-05-15T23:14:41.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.850990+0000", 2024-05-15T23:14:41.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3176999999999998e-05 2024-05-15T23:14:41.470 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.470 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.470 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.471 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:263640 2.7 e166/164 2:e4ff2092:::benchmark_data_smithi090_78160_object102282:head v 166'16518, mlcod=166'16514)", 2024-05-15T23:14:41.471 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:09:09.816764+0000", 2024-05-15T23:14:41.471 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 331.57325265200001, 2024-05-15T23:14:41.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034243843000000003, 2024-05-15T23:14:41.471 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.471 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.471 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.471 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.816764+0000", 2024-05-15T23:14:41.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.816764+0000", 2024-05-15T23:14:41.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.816766+0000", 2024-05-15T23:14:41.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3480000000000001e-06 2024-05-15T23:14:41.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.816773+0000", 2024-05-15T23:14:41.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7079999999999997e-06 2024-05-15T23:14:41.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.816786+0000", 2024-05-15T23:14:41.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3111e-05 2024-05-15T23:14:41.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.816794+0000", 2024-05-15T23:14:41.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.644e-06 2024-05-15T23:14:41.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.822297+0000", 2024-05-15T23:14:41.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005502491 2024-05-15T23:14:41.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.822326+0000", 2024-05-15T23:14:41.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8966e-05 2024-05-15T23:14:41.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.850993+0000", 2024-05-15T23:14:41.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028667898000000001 2024-05-15T23:14:41.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.851008+0000", 2024-05-15T23:14:41.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4677e-05 2024-05-15T23:14:41.475 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.475 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.475 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.475 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:263641 2.f e166/164 2:fd6cbc2c:::benchmark_data_smithi090_78160_object102283:head v 166'16654, mlcod=166'16650)", 2024-05-15T23:14:41.475 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:09:09.817512+0000", 2024-05-15T23:14:41.475 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 331.57250470600002, 2024-05-15T23:14:41.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033528910000000002, 2024-05-15T23:14:41.475 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.475 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.475 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.475 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.817512+0000", 2024-05-15T23:14:41.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.817512+0000", 2024-05-15T23:14:41.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.817514+0000", 2024-05-15T23:14:41.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.344e-06 2024-05-15T23:14:41.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.817519+0000", 2024-05-15T23:14:41.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7620000000000001e-06 2024-05-15T23:14:41.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.817528+0000", 2024-05-15T23:14:41.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3480000000000004e-06 2024-05-15T23:14:41.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.817534+0000", 2024-05-15T23:14:41.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4200000000000004e-06 2024-05-15T23:14:41.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.822280+0000", 2024-05-15T23:14:41.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047456820000000002 2024-05-15T23:14:41.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.822301+0000", 2024-05-15T23:14:41.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1450999999999998e-05 2024-05-15T23:14:41.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.851024+0000", 2024-05-15T23:14:41.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028722329000000001 2024-05-15T23:14:41.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:09.851041+0000", 2024-05-15T23:14:41.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7574000000000001e-05 2024-05-15T23:14:41.479 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.479 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.479 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.479 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.24745.0:264754 2.d e166/164 2:ba0f3313:::benchmark_data_smithi090_78160_object103396:head v 166'16404, mlcod=166'16401)", 2024-05-15T23:14:41.479 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:09:10.969271+0000", 2024-05-15T23:14:41.479 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 330.42074552100001, 2024-05-15T23:14:41.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029728687, 2024-05-15T23:14:41.480 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:14:41.480 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:14:41.480 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:14:41.480 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:14:41.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:14:41.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:10.969271+0000", 2024-05-15T23:14:41.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:14:41.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:10.969271+0000", 2024-05-15T23:14:41.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:14:41.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:14:41.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:10.969273+0000", 2024-05-15T23:14:41.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3820000000000001e-06 2024-05-15T23:14:41.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:14:41.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:10.969281+0000", 2024-05-15T23:14:41.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5129999999999996e-06 2024-05-15T23:14:41.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:14:41.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:10.969290+0000", 2024-05-15T23:14:41.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1190000000000003e-06 2024-05-15T23:14:41.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:14:41.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:10.969298+0000", 2024-05-15T23:14:41.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5480000000000004e-06 2024-05-15T23:14:41.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:14:41.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:10.972668+0000", 2024-05-15T23:14:41.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033704910000000002 2024-05-15T23:14:41.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:14:41.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:10.972706+0000", 2024-05-15T23:14:41.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7206000000000003e-05 2024-05-15T23:14:41.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:14:41.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:10.998952+0000", 2024-05-15T23:14:41.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02624638 2024-05-15T23:14:41.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:14:41.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:14:41.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:14:41.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:09:10.999000+0000", 2024-05-15T23:14:41.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8047999999999998e-05 2024-05-15T23:14:41.483 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.483 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.483 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.483 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:14:41.483 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:14:41.483 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:14:41.484 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:41 smithi090 ceph-mon[44434]: pgmap v778: 17 pgs: 17 active+clean; 312 MiB data, 2.4 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:14:41.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:41 smithi088 ceph-mon[44699]: pgmap v778: 17 pgs: 17 active+clean; 312 MiB data, 2.4 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:14:42.105 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T23:14:42.271 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Removed 165540 objects 2024-05-15T23:14:42.271 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Clean up completed and total clean up time :137.327 2024-05-15T23:14:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:14:43.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:42 smithi090 ceph-mon[44434]: pgmap v779: 17 pgs: 17 active+clean; 108 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:14:43.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:42 smithi088 ceph-mon[44699]: pgmap v779: 17 pgs: 17 active+clean; 108 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:14:43.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:42 smithi084 ceph-mon[39784]: pgmap v779: 17 pgs: 17 active+clean; 108 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:14:43.622 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:14:43.623 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:14:44.131 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:14:44.131 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:14:44.131 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-05-15T23:14:44.131 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:14:44.158 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:44.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:44.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:44.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:44.291 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:44.414 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:44.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:44.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:44.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:44.549 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:44.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T23:14:44.896 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:14:44.898 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:44.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:45.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:45.032 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:45.146 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:14:45.154 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:45.156 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3177555895' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:14:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:45.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:45.258 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:45.288 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3177555895' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:14:45.291 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:45.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3177555895' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:14:45.312 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:14:45.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:45.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:14:45.416 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:45.447 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:45.467 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:45.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:45.572 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:45.602 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:45.622 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:14:45.624 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:14:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:45.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:14:45.726 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:45.759 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:45.779 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:45.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:45.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:45.895 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:45.910 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:45 smithi090 ceph-mon[44434]: pgmap v780: 17 pgs: 17 active+clean; 52 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:14:45.911 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:45 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:14:45.911 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3177555895' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:14:45.911 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:45 smithi090 ceph-mon[44434]: osdmap e278: 12 total, 11 up, 11 in 2024-05-15T23:14:45.911 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:45 smithi090 ceph-mon[44434]: 3.4 scrub starts 2024-05-15T23:14:45.911 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:45 smithi090 ceph-mon[44434]: 3.4 scrub ok 2024-05-15T23:14:45.911 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:45 smithi090 ceph-mon[44434]: 3.1 scrub starts 2024-05-15T23:14:45.911 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:45 smithi090 ceph-mon[44434]: 3.1 scrub ok 2024-05-15T23:14:45.912 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:45.931 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:45.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:45.936 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:46.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:46.066 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:46.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:45 smithi084 ceph-mon[39784]: pgmap v780: 17 pgs: 17 active+clean; 52 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:14:46.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:45 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:14:46.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3177555895' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:14:46.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:45 smithi084 ceph-mon[39784]: osdmap e278: 12 total, 11 up, 11 in 2024-05-15T23:14:46.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:45 smithi084 ceph-mon[39784]: 3.4 scrub starts 2024-05-15T23:14:46.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:45 smithi084 ceph-mon[39784]: 3.4 scrub ok 2024-05-15T23:14:46.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:45 smithi084 ceph-mon[39784]: 3.1 scrub starts 2024-05-15T23:14:46.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:45 smithi084 ceph-mon[39784]: 3.1 scrub ok 2024-05-15T23:14:46.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:45 smithi088 ceph-mon[44699]: pgmap v780: 17 pgs: 17 active+clean; 52 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:14:46.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:45 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:14:46.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3177555895' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:14:46.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:45 smithi088 ceph-mon[44699]: osdmap e278: 12 total, 11 up, 11 in 2024-05-15T23:14:46.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:45 smithi088 ceph-mon[44699]: 3.4 scrub starts 2024-05-15T23:14:46.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:45 smithi088 ceph-mon[44699]: 3.4 scrub ok 2024-05-15T23:14:46.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:45 smithi088 ceph-mon[44699]: 3.1 scrub starts 2024-05-15T23:14:46.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:45 smithi088 ceph-mon[44699]: 3.1 scrub ok 2024-05-15T23:14:46.413 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:46.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:46.416 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:46.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:46.545 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:46.644 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:46.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:46.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:46.672 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-05-15T23:14:46.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-05-15T23:14:46.747 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:46.775 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:46.795 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:46.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:46.798 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:46.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:14:46.914 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:14:46.914 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:14:46.914 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:14:46.915 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:14:46.915 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:14:46.925 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:47.172 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:14:47.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:47.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:14:47.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:14:47.302 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:47.452 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:14:47.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:47.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:14:47.555 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:14:47.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T23:14:47.589 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:47.610 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:14:47.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:47.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:14:47.688 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:47.696 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:14:47.700 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:14:47.713 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:47.722 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:14:47.742 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:14:47.745 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:47.762 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:14:47.766 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:47.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:47.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:47.782 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:14:47.802 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:14:47.816 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:14:47.821 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:14:47.844 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:14:47.861 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:14:47.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:47.880 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:14:47.898 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:47.902 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:14:47.902 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:14:47.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:14:47.903 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-05-15T23:14:47.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:14:47.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:14:47.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:14:47.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:14:47.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:14:47.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:14:47.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:14:47.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:14:47.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:14:48.008 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:47 smithi084 ceph-mon[39784]: 3.3 scrub starts 2024-05-15T23:14:48.009 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:47 smithi084 ceph-mon[39784]: 3.3 scrub ok 2024-05-15T23:14:48.009 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:47 smithi084 ceph-mon[39784]: pgmap v782: 17 pgs: 17 active+clean; 3.9 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 976 op/s 2024-05-15T23:14:48.009 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:47 smithi084 ceph-mon[39784]: 3.8 scrub starts 2024-05-15T23:14:48.009 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:47 smithi084 ceph-mon[39784]: 3.8 scrub ok 2024-05-15T23:14:48.009 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:47 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2673236226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:48.009 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:48.009 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:48.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:47 smithi090 ceph-mon[44434]: 3.3 scrub starts 2024-05-15T23:14:48.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:47 smithi090 ceph-mon[44434]: 3.3 scrub ok 2024-05-15T23:14:48.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:47 smithi090 ceph-mon[44434]: pgmap v782: 17 pgs: 17 active+clean; 3.9 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 976 op/s 2024-05-15T23:14:48.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:47 smithi090 ceph-mon[44434]: 3.8 scrub starts 2024-05-15T23:14:48.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:47 smithi090 ceph-mon[44434]: 3.8 scrub ok 2024-05-15T23:14:48.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:47 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2673236226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:48.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:48.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:47 smithi088 ceph-mon[44699]: 3.3 scrub starts 2024-05-15T23:14:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:47 smithi088 ceph-mon[44699]: 3.3 scrub ok 2024-05-15T23:14:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:47 smithi088 ceph-mon[44699]: pgmap v782: 17 pgs: 17 active+clean; 3.9 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 976 op/s 2024-05-15T23:14:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:47 smithi088 ceph-mon[44699]: 3.8 scrub starts 2024-05-15T23:14:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:47 smithi088 ceph-mon[44699]: 3.8 scrub ok 2024-05-15T23:14:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:47 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2673236226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:48.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:48.736 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:14:48.737 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:14:49.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.151 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:49.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:49.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:49.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:49.286 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:49.289 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:49.310 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:49.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:49.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:49.446 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:49.689 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:49.695 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:49.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:49.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:49.799 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:49.831 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:49.852 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:14:49.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:49.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:14:49.957 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:49.962 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:49 smithi084 ceph-mon[39784]: pgmap v783: 17 pgs: 17 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 754 op/s 2024-05-15T23:14:49.986 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:50.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:49 smithi090 ceph-mon[44434]: pgmap v783: 17 pgs: 17 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 754 op/s 2024-05-15T23:14:50.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:49 smithi088 ceph-mon[44699]: pgmap v783: 17 pgs: 17 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 754 op/s 2024-05-15T23:14:50.661 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:14:50.661 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:14:50.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:14:50.834 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:50.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:50.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:50.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:14:50.957 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_1' removed 2024-05-15T23:14:51.013 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/297650863' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:14:51.015 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:51.037 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:14:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:51.047 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:14:51.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/297650863' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:14:51.148 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:51.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/297650863' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:14:51.174 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:51.193 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:51.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:51.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:51.323 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:51.351 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:51.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:51.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:51.376 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T23:14:51.455 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:51.482 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:51.500 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:51.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:51.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:51.605 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:14:51.631 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:51.649 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:14:51.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:51.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:14:51.673 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:14:51.753 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:51.780 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:51.926 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:51.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:51.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:52.003 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:51 smithi084 ceph-mon[39784]: pgmap v784: 17 pgs: 17 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 498 op/s 2024-05-15T23:14:52.003 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/297650863' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:14:52.003 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:51 smithi084 ceph-mon[39784]: osdmap e279: 12 total, 11 up, 11 in 2024-05-15T23:14:52.029 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T23:14:52.030 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T23:14:52.030 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-05-15T23:14:52.030 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-05-15T23:14:52.031 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:14:52.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:51 smithi090 ceph-mon[44434]: pgmap v784: 17 pgs: 17 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 498 op/s 2024-05-15T23:14:52.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/297650863' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:14:52.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:51 smithi090 ceph-mon[44434]: osdmap e279: 12 total, 11 up, 11 in 2024-05-15T23:14:52.058 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:52.081 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:14:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:52.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:14:52.102 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:52.185 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:14:52.211 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:51 smithi088 ceph-mon[44699]: pgmap v784: 17 pgs: 17 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 498 op/s 2024-05-15T23:14:52.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/297650863' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:14:52.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:51 smithi088 ceph-mon[44699]: osdmap e279: 12 total, 11 up, 11 in 2024-05-15T23:14:52.213 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:52.233 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:14:52.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:52.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:14:52.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:14:52.365 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:52.417 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:14:52.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:52.420 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:14:52.521 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:52.554 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:52.575 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:52.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:52.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:52.679 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:52.708 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:53.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:52 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:14:53.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:52 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:14:53.132 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:52 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:14:53.132 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:52 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:14:53.156 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:53.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:53.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:53.262 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:53.291 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:53.311 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:53.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:53.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:53.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:52 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:14:53.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:52 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:14:53.415 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:53.430 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:14:53.430 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:14:53.447 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:53.467 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:53.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:53.571 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:53.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:53.599 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:53.672 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:53.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:53.674 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:53.775 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:53.802 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:53.925 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:14:54.034 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:53 smithi084 ceph-mon[39784]: pgmap v786: 1 pgs: 1 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:14:54.034 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:53 smithi084 ceph-mon[39784]: osdmap e280: 12 total, 11 up, 11 in 2024-05-15T23:14:54.035 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:53 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3770382773' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:14:54.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:53 smithi090 ceph-mon[44434]: pgmap v786: 1 pgs: 1 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:14:54.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:53 smithi090 ceph-mon[44434]: osdmap e280: 12 total, 11 up, 11 in 2024-05-15T23:14:54.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:53 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3770382773' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:14:54.103 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:54.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:54.108 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:54.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T23:14:54.208 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:54.239 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:53 smithi088 ceph-mon[44699]: pgmap v786: 1 pgs: 1 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:14:54.239 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:53 smithi088 ceph-mon[44699]: osdmap e280: 12 total, 11 up, 11 in 2024-05-15T23:14:54.239 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:53 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3770382773' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:14:54.242 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:54.264 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:14:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:54.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:14:54.368 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:54.397 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:54.417 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:54.417 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:14:54.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:54.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:54.520 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:14:54.548 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:54.567 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:14:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:54.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:14:54.671 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:54.699 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:54.718 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:54.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:54.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:54.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:54.849 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:54.887 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:54.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:54.889 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:54.990 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:14:55.018 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:55.037 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:14:55.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:55.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:14:55.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:54 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:14:55.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:54 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:14:55.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3770382773' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:14:55.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:54 smithi084 ceph-mon[39784]: osdmap e281: 12 total, 11 up, 11 in 2024-05-15T23:14:55.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:54 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:14:55.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:54 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:14:55.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:55.166 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:54 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:14:55.166 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:54 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:14:55.166 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3770382773' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:14:55.167 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:54 smithi090 ceph-mon[44434]: osdmap e281: 12 total, 11 up, 11 in 2024-05-15T23:14:55.167 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:54 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:14:55.167 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:54 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:14:55.168 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:55.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:54 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:14:55.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:54 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:14:55.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3770382773' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:14:55.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:54 smithi088 ceph-mon[44699]: osdmap e281: 12 total, 11 up, 11 in 2024-05-15T23:14:55.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:54 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:14:55.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:54 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:14:55.390 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:14:55.409 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:14:55.429 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:14:55.449 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:14:55.469 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:14:55.488 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:14:55.508 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:14:55.527 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:14:55.547 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:14:55.566 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:14:55.585 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:14:55.586 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:14:55.586 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:14:55.586 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-05-15T23:14:55.586 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:14:55.586 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:14:55.586 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:14:55.586 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:14:55.586 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:14:55.586 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:14:55.586 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:14:55.586 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:14:55.586 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:14:56.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:55 smithi084 ceph-mon[39784]: pgmap v789: 1 pgs: 1 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:14:56.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:55 smithi084 ceph-mon[39784]: osdmap e282: 12 total, 11 up, 11 in 2024-05-15T23:14:56.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:55 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2802107933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:56.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:56.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:56.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:56.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:56.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:56.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:55 smithi090 ceph-mon[44434]: pgmap v789: 1 pgs: 1 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:14:56.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:55 smithi090 ceph-mon[44434]: osdmap e282: 12 total, 11 up, 11 in 2024-05-15T23:14:56.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:55 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2802107933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:56.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:56.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:56.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:56.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:56.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:55 smithi088 ceph-mon[44699]: pgmap v789: 1 pgs: 1 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:14:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:55 smithi088 ceph-mon[44699]: osdmap e282: 12 total, 11 up, 11 in 2024-05-15T23:14:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:55 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2802107933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:14:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:56.483 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:14:56.484 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:14:56.517 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:56.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:56.520 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:56.621 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:14:56.651 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:56.724 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:14:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:56.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:14:56.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:56.857 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:56.934 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:56.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:56.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:57.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.070 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:57.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.170 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:57.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:57.274 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:14:57.304 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.304 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.304 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.304 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.304 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.304 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:14:57.306 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:57.328 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:14:57.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:57.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:14:57.432 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:14:57.462 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:57.483 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:14:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:57.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:14:57.587 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:57.615 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:57.634 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:14:57.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:57.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:14:57.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:14:57.770 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:57.787 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:14:57.791 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:14:57.792 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:14:57.792 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:14:57.792 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:14:57.793 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:14:57.793 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:14:57.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:57.794 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:14:57.895 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:57.924 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:57.943 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:57.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:57.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:57.949 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_2' created 2024-05-15T23:14:58.050 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:57 smithi090 ceph-mon[44434]: pgmap v791: 1 pgs: 1 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:14:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3415084974' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-05-15T23:14:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:57 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:14:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:57 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:14:58.076 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:57 smithi084 ceph-mon[39784]: pgmap v791: 1 pgs: 1 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:14:58.076 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3415084974' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-05-15T23:14:58.076 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:57 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:14:58.076 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:57 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:14:58.078 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:58.225 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:58.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:57 smithi088 ceph-mon[44699]: pgmap v791: 1 pgs: 1 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:14:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3415084974' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-05-15T23:14:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:57 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:14:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:57 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:14:58.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:58.356 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:58.479 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:58.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:58.483 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:58.485 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:14:58.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T23:14:58.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:14:58.610 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:58.628 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:14:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:58.631 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:14:58.657 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:58.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-05-15T23:14:58.732 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:58.758 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:58.792 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:14:58.957 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:58.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:59.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:14:59.090 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:59.110 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:14:59.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:59.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:14:59.198 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:58 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3415084974' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-05-15T23:14:59.199 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:58 smithi084 ceph-mon[39784]: osdmap e283: 12 total, 11 up, 11 in 2024-05-15T23:14:59.213 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:14:59.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:58 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3415084974' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-05-15T23:14:59.237 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:58 smithi090 ceph-mon[44434]: osdmap e283: 12 total, 11 up, 11 in 2024-05-15T23:14:59.239 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:59.292 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:14:59.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:59.295 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:14:59.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:58 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3415084974' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-05-15T23:14:59.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:58 smithi088 ceph-mon[44699]: osdmap e283: 12 total, 11 up, 11 in 2024-05-15T23:14:59.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:14:59.422 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:59.620 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:14:59.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:59.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:14:59.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:14:59.752 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:14:59.833 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:59.834 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:14:59.850 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:14:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:14:59.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:14:59.957 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:14:59.990 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:00.010 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:15:00.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:00.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:15:00.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:15:00.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:59 smithi084 ceph-mon[39784]: pgmap v793: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:15:00.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:59 smithi084 ceph-mon[39784]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:15:00.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:14:59 smithi084 ceph-mon[39784]: osdmap e284: 12 total, 11 up, 11 in 2024-05-15T23:15:00.141 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:00.289 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:15:00.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:59 smithi090 ceph-mon[44434]: pgmap v793: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:15:00.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:59 smithi090 ceph-mon[44434]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:15:00.291 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:14:59 smithi090 ceph-mon[44434]: osdmap e284: 12 total, 11 up, 11 in 2024-05-15T23:15:00.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:00.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:15:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:59 smithi088 ceph-mon[44699]: pgmap v793: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:15:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:59 smithi088 ceph-mon[44699]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:15:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:14:59 smithi088 ceph-mon[44699]: osdmap e284: 12 total, 11 up, 11 in 2024-05-15T23:15:00.393 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:15:00.425 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:00.446 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:15:00.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:00.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:15:00.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:15:00.579 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:00.581 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:15:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:15:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:15:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:15:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109643 2.c 2:35bed8ca:::benchmark_data_smithi090_78160_object109642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:15:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.334876+0000", 2024-05-15T23:15:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 487.24271917800002, 2024-05-15T23:15:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.131093448, 2024-05-15T23:15:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:15:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:15:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109643 2024-05-15T23:15:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334876+0000", 2024-05-15T23:15:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334876+0000", 2024-05-15T23:15:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334878+0000", 2024-05-15T23:15:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5290000000000001e-06 2024-05-15T23:15:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334899+0000", 2024-05-15T23:15:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0951999999999999e-05 2024-05-15T23:15:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334909+0000", 2024-05-15T23:15:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308e-05 2024-05-15T23:15:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.334917+0000", 2024-05-15T23:15:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3119999999999996e-06 2024-05-15T23:15:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.339148+0000", 2024-05-15T23:15:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004231004 2024-05-15T23:15:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.339226+0000", 2024-05-15T23:15:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8885000000000001e-05 2024-05-15T23:15:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:15:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.339304+0000", 2024-05-15T23:15:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7218000000000005e-05 2024-05-15T23:15:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.343262+0000", 2024-05-15T23:15:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003958746 2024-05-15T23:15:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351963+0000", 2024-05-15T23:15:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087008099999999998 2024-05-15T23:15:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.352004+0000", 2024-05-15T23:15:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0442999999999999e-05 2024-05-15T23:15:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465970+0000", 2024-05-15T23:15:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.113966241 2024-05-15T23:15:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109646 2.c 2:300500bb:::benchmark_data_smithi090_78160_object109645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:15:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.337707+0000", 2024-05-15T23:15:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 487.23988872799998, 2024-05-15T23:15:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128404672, 2024-05-15T23:15:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:15:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:15:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109646 2024-05-15T23:15:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337707+0000", 2024-05-15T23:15:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337707+0000", 2024-05-15T23:15:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337708+0000", 2024-05-15T23:15:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.511e-06 2024-05-15T23:15:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337728+0000", 2024-05-15T23:15:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0055999999999999e-05 2024-05-15T23:15:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337739+0000", 2024-05-15T23:15:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0471e-05 2024-05-15T23:15:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.337746+0000", 2024-05-15T23:15:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4059999999999999e-06 2024-05-15T23:15:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.340943+0000", 2024-05-15T23:15:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031963859999999998 2024-05-15T23:15:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.341055+0000", 2024-05-15T23:15:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011188600000000001 2024-05-15T23:15:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0", 2024-05-15T23:15:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.341175+0000", 2024-05-15T23:15:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120839 2024-05-15T23:15:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.343368+0000", 2024-05-15T23:15:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021920669999999998 2024-05-15T23:15:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466023+0000", 2024-05-15T23:15:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.122655237 2024-05-15T23:15:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466075+0000", 2024-05-15T23:15:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1910000000000003e-05 2024-05-15T23:15:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466112+0000", 2024-05-15T23:15:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6903000000000002e-05 2024-05-15T23:15:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.24745.0:109643 2.c e127/102 ondisk, result = 0)", 2024-05-15T23:15:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.351843+0000", 2024-05-15T23:15:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 487.22575251799998, 2024-05-15T23:15:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.114139029, 2024-05-15T23:15:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:15:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351843+0000", 2024-05-15T23:15:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351843+0000", 2024-05-15T23:15:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351843+0000", 2024-05-15T23:15:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9000000000000002e-07 2024-05-15T23:15:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351846+0000", 2024-05-15T23:15:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1179999999999999e-06 2024-05-15T23:15:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351849+0000", 2024-05-15T23:15:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8510000000000001e-06 2024-05-15T23:15:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351853+0000", 2024-05-15T23:15:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3890000000000001e-06 2024-05-15T23:15:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351950+0000", 2024-05-15T23:15:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7557000000000001e-05 2024-05-15T23:15:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351958+0000", 2024-05-15T23:15:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5310000000000004e-06 2024-05-15T23:15:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.465982+0000", 2024-05-15T23:15:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.114024193 2024-05-15T23:15:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.24745.0:109646 2.c e127/102 ondisk, result = 0)", 2024-05-15T23:15:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.351916+0000", 2024-05-15T23:15:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 487.22568012300002, 2024-05-15T23:15:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.114202603, 2024-05-15T23:15:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:15:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351916+0000", 2024-05-15T23:15:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351916+0000", 2024-05-15T23:15:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351917+0000", 2024-05-15T23:15:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2759999999999999e-06 2024-05-15T23:15:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351921+0000", 2024-05-15T23:15:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1559999999999997e-06 2024-05-15T23:15:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351929+0000", 2024-05-15T23:15:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7500000000000003e-06 2024-05-15T23:15:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.351934+0000", 2024-05-15T23:15:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.079e-06 2024-05-15T23:15:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466009+0000", 2024-05-15T23:15:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11407569300000001 2024-05-15T23:15:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466017+0000", 2024-05-15T23:15:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6350000000000006e-06 2024-05-15T23:15:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466118+0000", 2024-05-15T23:15:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010101400000000001 2024-05-15T23:15:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.24745.0:109656 2.a 2:509d81bb:::benchmark_data_smithi090_78160_object109655:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e127)", 2024-05-15T23:15:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:06:53.354844+0000", 2024-05-15T23:15:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 487.222751453, 2024-05-15T23:15:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.119458562, 2024-05-15T23:15:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.24745", 2024-05-15T23:15:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:15:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109656 2024-05-15T23:15:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.354844+0000", 2024-05-15T23:15:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.354844+0000", 2024-05-15T23:15:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.354848+0000", 2024-05-15T23:15:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3960000000000002e-06 2024-05-15T23:15:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.354888+0000", 2024-05-15T23:15:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0673999999999997e-05 2024-05-15T23:15:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466170+0000", 2024-05-15T23:15:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11128214 2024-05-15T23:15:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466180+0000", 2024-05-15T23:15:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6779999999999995e-06 2024-05-15T23:15:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466282+0000", 2024-05-15T23:15:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101661 2024-05-15T23:15:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466411+0000", 2024-05-15T23:15:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012923899999999999 2024-05-15T23:15:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:15:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.466561+0000", 2024-05-15T23:15:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014973300000000001 2024-05-15T23:15:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.470610+0000", 2024-05-15T23:15:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040489970000000004 2024-05-15T23:15:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.474147+0000", 2024-05-15T23:15:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0035371539999999998 2024-05-15T23:15:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.474210+0000", 2024-05-15T23:15:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3291000000000001e-05 2024-05-15T23:15:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:06:53.474303+0000", 2024-05-15T23:15:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2598999999999999e-05 2024-05-15T23:15:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:141617 3.e 3:72e350be:::benchmark_data_smithi090_90974_object141616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e238)", 2024-05-15T23:15:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:00.137574+0000", 2024-05-15T23:15:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 180.440021984, 2024-05-15T23:15:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.109237059, 2024-05-15T23:15:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:15:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:15:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 141617 2024-05-15T23:15:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.137574+0000", 2024-05-15T23:15:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.137574+0000", 2024-05-15T23:15:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.137576+0000", 2024-05-15T23:15:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4899999999999999e-06 2024-05-15T23:15:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.137605+0000", 2024-05-15T23:15:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8725e-05 2024-05-15T23:15:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.137622+0000", 2024-05-15T23:15:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7164999999999999e-05 2024-05-15T23:15:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.137631+0000", 2024-05-15T23:15:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2359999999999993e-06 2024-05-15T23:15:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.145013+0000", 2024-05-15T23:15:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073814149999999997 2024-05-15T23:15:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.145167+0000", 2024-05-15T23:15:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015469799999999999 2024-05-15T23:15:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:15:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.145342+0000", 2024-05-15T23:15:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017493000000000001 2024-05-15T23:15:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.149726+0000", 2024-05-15T23:15:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043835979999999998 2024-05-15T23:15:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.246689+0000", 2024-05-15T23:15:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.096963465999999998 2024-05-15T23:15:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.246748+0000", 2024-05-15T23:15:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8718999999999998e-05 2024-05-15T23:15:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:00.246811+0000", 2024-05-15T23:15:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2617e-05 2024-05-15T23:15:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:152314 3.e 3:7ab6d39c:::benchmark_data_smithi090_90974_object152313:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e242)", 2024-05-15T23:15:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:11.011469+0000", 2024-05-15T23:15:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 169.566126239, 2024-05-15T23:15:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11541359299999999, 2024-05-15T23:15:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:15:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:15:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 152314 2024-05-15T23:15:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011469+0000", 2024-05-15T23:15:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011469+0000", 2024-05-15T23:15:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011473+0000", 2024-05-15T23:15:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5880000000000002e-06 2024-05-15T23:15:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011500+0000", 2024-05-15T23:15:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6791000000000001e-05 2024-05-15T23:15:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011514+0000", 2024-05-15T23:15:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4518999999999999e-05 2024-05-15T23:15:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011524+0000", 2024-05-15T23:15:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0033e-05 2024-05-15T23:15:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017018+0000", 2024-05-15T23:15:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00549396 2024-05-15T23:15:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017157+0000", 2024-05-15T23:15:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139202 2024-05-15T23:15:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:15:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017343+0000", 2024-05-15T23:15:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018567099999999999 2024-05-15T23:15:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.026368+0000", 2024-05-15T23:15:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090247439999999995 2024-05-15T23:15:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.126798+0000", 2024-05-15T23:15:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10043023299999999 2024-05-15T23:15:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.126830+0000", 2024-05-15T23:15:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1899000000000001e-05 2024-05-15T23:15:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.126883+0000", 2024-05-15T23:15:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2952999999999999e-05 2024-05-15T23:15:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:152320 3.e 3:77b14055:::benchmark_data_smithi090_90974_object152319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e242)", 2024-05-15T23:15:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:11.017288+0000", 2024-05-15T23:15:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 169.56030792199999, 2024-05-15T23:15:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.109741271, 2024-05-15T23:15:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:15:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:15:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 152320 2024-05-15T23:15:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017288+0000", 2024-05-15T23:15:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017288+0000", 2024-05-15T23:15:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017290+0000", 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.503e-06 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017313+0000", 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2501e-05 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017322+0000", 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.132e-06 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017329+0000", 2024-05-15T23:15:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2320000000000004e-06 2024-05-15T23:15:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.022161+0000", 2024-05-15T23:15:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004831531 2024-05-15T23:15:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.022299+0000", 2024-05-15T23:15:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138214 2024-05-15T23:15:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:15:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.022423+0000", 2024-05-15T23:15:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012423800000000001 2024-05-15T23:15:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.126938+0000", 2024-05-15T23:15:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104514871 2024-05-15T23:15:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.126951+0000", 2024-05-15T23:15:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2934e-05 2024-05-15T23:15:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.126984+0000", 2024-05-15T23:15:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2982999999999999e-05 2024-05-15T23:15:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.127029+0000", 2024-05-15T23:15:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5132000000000001e-05 2024-05-15T23:15:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:153702 3.e 3:7dfbffe1:::benchmark_data_smithi090_90974_object153701:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:15:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:12.533977+0000", 2024-05-15T23:15:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 168.04361846899999, 2024-05-15T23:15:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.111764557, 2024-05-15T23:15:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:15:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:15:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 153702 2024-05-15T23:15:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.533977+0000", 2024-05-15T23:15:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.533977+0000", 2024-05-15T23:15:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.533979+0000", 2024-05-15T23:15:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0999999999999998e-06 2024-05-15T23:15:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.534003+0000", 2024-05-15T23:15:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3706999999999999e-05 2024-05-15T23:15:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.534014+0000", 2024-05-15T23:15:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0951000000000001e-05 2024-05-15T23:15:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.534022+0000", 2024-05-15T23:15:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4579999999999993e-06 2024-05-15T23:15:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.545636+0000", 2024-05-15T23:15:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011613573 2024-05-15T23:15:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.545735+0000", 2024-05-15T23:15:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9412000000000003e-05 2024-05-15T23:15:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:15:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.545840+0000", 2024-05-15T23:15:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010463099999999999 2024-05-15T23:15:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.550491+0000", 2024-05-15T23:15:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046514740000000001 2024-05-15T23:15:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.645639+0000", 2024-05-15T23:15:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095147332000000001 2024-05-15T23:15:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.645683+0000", 2024-05-15T23:15:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4459000000000002e-05 2024-05-15T23:15:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:12.645742+0000", 2024-05-15T23:15:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8459999999999999e-05 2024-05-15T23:15:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:154414 3.e 3:7f410bb8:::benchmark_data_smithi090_90974_object154413:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:15:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:13.323080+0000", 2024-05-15T23:15:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 167.25451527199999, 2024-05-15T23:15:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.116641645, 2024-05-15T23:15:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:15:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:15:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 154414 2024-05-15T23:15:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323080+0000", 2024-05-15T23:15:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323080+0000", 2024-05-15T23:15:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323082+0000", 2024-05-15T23:15:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5409999999999999e-06 2024-05-15T23:15:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323102+0000", 2024-05-15T23:15:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0557999999999999e-05 2024-05-15T23:15:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323111+0000", 2024-05-15T23:15:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9320000000000002e-06 2024-05-15T23:15:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323118+0000", 2024-05-15T23:15:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7209999999999997e-06 2024-05-15T23:15:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.343054+0000", 2024-05-15T23:15:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019935797000000002 2024-05-15T23:15:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.343173+0000", 2024-05-15T23:15:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118989 2024-05-15T23:15:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:15:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.343298+0000", 2024-05-15T23:15:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012522999999999999 2024-05-15T23:15:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.351419+0000", 2024-05-15T23:15:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0081205270000000006 2024-05-15T23:15:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.439647+0000", 2024-05-15T23:15:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.088228506999999998 2024-05-15T23:15:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.439683+0000", 2024-05-15T23:15:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5723999999999999e-05 2024-05-15T23:15:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.439722+0000", 2024-05-15T23:15:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9119000000000003e-05 2024-05-15T23:15:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158411 3.e 3:721f625f:::benchmark_data_smithi090_90974_object158410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:15:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.688475+0000", 2024-05-15T23:15:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 162.88912013500001, 2024-05-15T23:15:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11329204800000001, 2024-05-15T23:15:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:15:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:15:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158411 2024-05-15T23:15:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688475+0000", 2024-05-15T23:15:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688475+0000", 2024-05-15T23:15:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688478+0000", 2024-05-15T23:15:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6199999999999999e-06 2024-05-15T23:15:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688507+0000", 2024-05-15T23:15:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8671e-05 2024-05-15T23:15:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688524+0000", 2024-05-15T23:15:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6837000000000002e-05 2024-05-15T23:15:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688534+0000", 2024-05-15T23:15:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0513000000000001e-05 2024-05-15T23:15:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701293+0000", 2024-05-15T23:15:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012759111 2024-05-15T23:15:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701443+0000", 2024-05-15T23:15:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015017200000000001 2024-05-15T23:15:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:15:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701612+0000", 2024-05-15T23:15:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016817399999999999 2024-05-15T23:15:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.801638+0000", 2024-05-15T23:15:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100025981 2024-05-15T23:15:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.801691+0000", 2024-05-15T23:15:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3124999999999997e-05 2024-05-15T23:15:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.801724+0000", 2024-05-15T23:15:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3039e-05 2024-05-15T23:15:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.801768+0000", 2024-05-15T23:15:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3804999999999998e-05 2024-05-15T23:15:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158423 3.e 3:7aa2cb14:::benchmark_data_smithi090_90974_object158422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:15:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.701674+0000", 2024-05-15T23:15:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 162.875921699, 2024-05-15T23:15:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.115640091, 2024-05-15T23:15:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:15:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:15:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158423 2024-05-15T23:15:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701674+0000", 2024-05-15T23:15:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701674+0000", 2024-05-15T23:15:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701675+0000", 2024-05-15T23:15:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.424e-06 2024-05-15T23:15:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701694+0000", 2024-05-15T23:15:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8777e-05 2024-05-15T23:15:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701703+0000", 2024-05-15T23:15:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1090000000000004e-06 2024-05-15T23:15:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701710+0000", 2024-05-15T23:15:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0650000000000001e-06 2024-05-15T23:15:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.803831+0000", 2024-05-15T23:15:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102121131 2024-05-15T23:15:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.803962+0000", 2024-05-15T23:15:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131042 2024-05-15T23:15:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:15:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.804116+0000", 2024-05-15T23:15:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015323299999999999 2024-05-15T23:15:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.817218+0000", 2024-05-15T23:15:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013102213 2024-05-15T23:15:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.817227+0000", 2024-05-15T23:15:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2949999999999999e-06 2024-05-15T23:15:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.817263+0000", 2024-05-15T23:15:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6003000000000001e-05 2024-05-15T23:15:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.817314+0000", 2024-05-15T23:15:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0798999999999999e-05 2024-05-15T23:15:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158426 3.e 3:7524570c:::benchmark_data_smithi090_90974_object158425:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:15:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.704015+0000", 2024-05-15T23:15:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 162.87358024100001, 2024-05-15T23:15:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.118608484, 2024-05-15T23:15:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:15:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:15:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158426 2024-05-15T23:15:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704015+0000", 2024-05-15T23:15:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704015+0000", 2024-05-15T23:15:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704017+0000", 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7910000000000001e-06 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704039+0000", 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2243e-05 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704053+0000", 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3169e-05 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704060+0000", 2024-05-15T23:15:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6799999999999993e-06 2024-05-15T23:15:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.808093+0000", 2024-05-15T23:15:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10403225000000001 2024-05-15T23:15:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.808225+0000", 2024-05-15T23:15:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013245100000000001 2024-05-15T23:15:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:15:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.808375+0000", 2024-05-15T23:15:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014989300000000001 2024-05-15T23:15:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.817357+0000", 2024-05-15T23:15:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0089821039999999994 2024-05-15T23:15:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.822508+0000", 2024-05-15T23:15:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005150951 2024-05-15T23:15:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.822560+0000", 2024-05-15T23:15:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2488999999999998e-05 2024-05-15T23:15:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.822624+0000", 2024-05-15T23:15:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3462999999999999e-05 2024-05-15T23:15:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158429 3.e 3:740800f6:::benchmark_data_smithi090_90974_object158428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:15:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.708210+0000", 2024-05-15T23:15:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 162.86938587500001, 2024-05-15T23:15:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11887426500000001, 2024-05-15T23:15:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:15:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:15:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158429 2024-05-15T23:15:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708210+0000", 2024-05-15T23:15:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708210+0000", 2024-05-15T23:15:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708212+0000", 2024-05-15T23:15:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4430000000000002e-06 2024-05-15T23:15:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708234+0000", 2024-05-15T23:15:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2195e-05 2024-05-15T23:15:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708245+0000", 2024-05-15T23:15:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1012999999999999e-05 2024-05-15T23:15:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708253+0000", 2024-05-15T23:15:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8879999999999997e-06 2024-05-15T23:15:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.813214+0000", 2024-05-15T23:15:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104960421 2024-05-15T23:15:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.813362+0000", 2024-05-15T23:15:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014787399999999999 2024-05-15T23:15:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:15:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.813490+0000", 2024-05-15T23:15:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012885199999999999 2024-05-15T23:15:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.822633+0000", 2024-05-15T23:15:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091427599999999998 2024-05-15T23:15:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.826999+0000", 2024-05-15T23:15:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00436572 2024-05-15T23:15:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.827035+0000", 2024-05-15T23:15:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6282000000000003e-05 2024-05-15T23:15:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.827084+0000", 2024-05-15T23:15:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8816999999999997e-05 2024-05-15T23:15:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:159320 3.e 3:722627b8:::benchmark_data_smithi090_90974_object159319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e244)", 2024-05-15T23:15:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:18.781348+0000", 2024-05-15T23:15:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 161.79624737399999, 2024-05-15T23:15:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13170827900000001, 2024-05-15T23:15:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:15:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:15:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159320 2024-05-15T23:15:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781348+0000", 2024-05-15T23:15:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781348+0000", 2024-05-15T23:15:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781351+0000", 2024-05-15T23:15:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3410000000000001e-06 2024-05-15T23:15:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781377+0000", 2024-05-15T23:15:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6271e-05 2024-05-15T23:15:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781388+0000", 2024-05-15T23:15:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1141999999999999e-05 2024-05-15T23:15:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781396+0000", 2024-05-15T23:15:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2679999999999994e-06 2024-05-15T23:15:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.797611+0000", 2024-05-15T23:15:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016214605 2024-05-15T23:15:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.797693+0000", 2024-05-15T23:15:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2564000000000004e-05 2024-05-15T23:15:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:15:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.797790+0000", 2024-05-15T23:15:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6477000000000005e-05 2024-05-15T23:15:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.895923+0000", 2024-05-15T23:15:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.098133348999999995 2024-05-15T23:15:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.912951+0000", 2024-05-15T23:15:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017028192000000001 2024-05-15T23:15:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.912989+0000", 2024-05-15T23:15:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7867000000000002e-05 2024-05-15T23:15:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.913057+0000", 2024-05-15T23:15:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7203e-05 2024-05-15T23:15:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:159325 3.e 3:77a43cc1:::benchmark_data_smithi090_90974_object159324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e244)", 2024-05-15T23:15:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:18.785096+0000", 2024-05-15T23:15:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 161.792499749, 2024-05-15T23:15:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12806985500000001, 2024-05-15T23:15:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:15:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:15:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159325 2024-05-15T23:15:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785096+0000", 2024-05-15T23:15:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785096+0000", 2024-05-15T23:15:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785098+0000", 2024-05-15T23:15:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.571e-06 2024-05-15T23:15:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785122+0000", 2024-05-15T23:15:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3864999999999999e-05 2024-05-15T23:15:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785135+0000", 2024-05-15T23:15:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3061000000000001e-05 2024-05-15T23:15:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785144+0000", 2024-05-15T23:15:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6589999999999996e-06 2024-05-15T23:15:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.801926+0000", 2024-05-15T23:15:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016782254 2024-05-15T23:15:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.802075+0000", 2024-05-15T23:15:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014822300000000001 2024-05-15T23:15:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:15:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.802255+0000", 2024-05-15T23:15:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018082999999999999 2024-05-15T23:15:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.895946+0000", 2024-05-15T23:15:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.093690839999999997 2024-05-15T23:15:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.913068+0000", 2024-05-15T23:15:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01712139 2024-05-15T23:15:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.913114+0000", 2024-05-15T23:15:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6802999999999999e-05 2024-05-15T23:15:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.913166+0000", 2024-05-15T23:15:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1359000000000001e-05 2024-05-15T23:15:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:159351 3.e 3:77148def:::benchmark_data_smithi090_90974_object159350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e244)", 2024-05-15T23:15:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:18.811652+0000", 2024-05-15T23:15:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 161.76594408700001, 2024-05-15T23:15:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.112410336, 2024-05-15T23:15:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:15:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:15:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159351 2024-05-15T23:15:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.811652+0000", 2024-05-15T23:15:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.811652+0000", 2024-05-15T23:15:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.811652+0000", 2024-05-15T23:15:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1999999999999998e-07 2024-05-15T23:15:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.811669+0000", 2024-05-15T23:15:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6860000000000001e-05 2024-05-15T23:15:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.811675+0000", 2024-05-15T23:15:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5729999999999998e-06 2024-05-15T23:15:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.811681+0000", 2024-05-15T23:15:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9839999999999998e-06 2024-05-15T23:15:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.900818+0000", 2024-05-15T23:15:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089137076999999995 2024-05-15T23:15:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.900969+0000", 2024-05-15T23:15:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150679 2024-05-15T23:15:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:15:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.901104+0000", 2024-05-15T23:15:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013527499999999999 2024-05-15T23:15:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.918628+0000", 2024-05-15T23:15:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017524217000000002 2024-05-15T23:15:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.923944+0000", 2024-05-15T23:15:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053155939999999999 2024-05-15T23:15:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.924001+0000", 2024-05-15T23:15:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7816999999999999e-05 2024-05-15T23:15:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.924062+0000", 2024-05-15T23:15:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0439999999999997e-05 2024-05-15T23:15:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:160540 3.e 3:74ab87c2:::benchmark_data_smithi090_90974_object160539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-05-15T23:15:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:20.108799+0000", 2024-05-15T23:15:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 160.46879616000001, 2024-05-15T23:15:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105677878, 2024-05-15T23:15:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:15:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:15:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160540 2024-05-15T23:15:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.108799+0000", 2024-05-15T23:15:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.108799+0000", 2024-05-15T23:15:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.108801+0000", 2024-05-15T23:15:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0720000000000001e-06 2024-05-15T23:15:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.108822+0000", 2024-05-15T23:15:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1039e-05 2024-05-15T23:15:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.108831+0000", 2024-05-15T23:15:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6830000000000003e-06 2024-05-15T23:15:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.108838+0000", 2024-05-15T23:15:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4760000000000003e-06 2024-05-15T23:15:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.113383+0000", 2024-05-15T23:15:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045452019999999999 2024-05-15T23:15:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.113525+0000", 2024-05-15T23:15:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000141896 2024-05-15T23:15:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:15:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.113635+0000", 2024-05-15T23:15:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010969299999999999 2024-05-15T23:15:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.204569+0000", 2024-05-15T23:15:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090934849999999998 2024-05-15T23:15:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.214363+0000", 2024-05-15T23:15:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0097934130000000008 2024-05-15T23:15:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.214416+0000", 2024-05-15T23:15:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2989999999999999e-05 2024-05-15T23:15:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.214477+0000", 2024-05-15T23:15:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1563999999999995e-05 2024-05-15T23:15:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:160906 3.e 3:79e6de76:::benchmark_data_smithi090_90974_object160905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e245)", 2024-05-15T23:15:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:20.576551+0000", 2024-05-15T23:15:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 160.00104475500001, 2024-05-15T23:15:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.106230501, 2024-05-15T23:15:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:15:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:15:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160906 2024-05-15T23:15:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.576551+0000", 2024-05-15T23:15:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.576551+0000", 2024-05-15T23:15:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.576552+0000", 2024-05-15T23:15:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3200000000000003e-07 2024-05-15T23:15:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.576572+0000", 2024-05-15T23:15:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0543999999999998e-05 2024-05-15T23:15:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.576579+0000", 2024-05-15T23:15:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6730000000000003e-06 2024-05-15T23:15:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.576585+0000", 2024-05-15T23:15:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8510000000000004e-06 2024-05-15T23:15:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.583924+0000", 2024-05-15T23:15:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073390649999999997 2024-05-15T23:15:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.584079+0000", 2024-05-15T23:15:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000154797 2024-05-15T23:15:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:15:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.584260+0000", 2024-05-15T23:15:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018160499999999999 2024-05-15T23:15:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.592119+0000", 2024-05-15T23:15:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.007859029 2024-05-15T23:15:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.682657+0000", 2024-05-15T23:15:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090537754999999998 2024-05-15T23:15:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.682720+0000", 2024-05-15T23:15:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2472999999999996e-05 2024-05-15T23:15:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:20.682781+0000", 2024-05-15T23:15:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1777000000000002e-05 2024-05-15T23:15:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:161690 3.e 3:78fe5800:::benchmark_data_smithi090_90974_object161689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e246)", 2024-05-15T23:15:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:21.553373+0000", 2024-05-15T23:15:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 159.024222243, 2024-05-15T23:15:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.107890657, 2024-05-15T23:15:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:15:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:15:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:15:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:15:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:15:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:15:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 161690 2024-05-15T23:15:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:15:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:15:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.553373+0000", 2024-05-15T23:15:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:15:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.553373+0000", 2024-05-15T23:15:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:15:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:15:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.553377+0000", 2024-05-15T23:15:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4170000000000001e-06 2024-05-15T23:15:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:15:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.553407+0000", 2024-05-15T23:15:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0602000000000003e-05 2024-05-15T23:15:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:15:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.553423+0000", 2024-05-15T23:15:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5721000000000001e-05 2024-05-15T23:15:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:15:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.553435+0000", 2024-05-15T23:15:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1758e-05 2024-05-15T23:15:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:15:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.556608+0000", 2024-05-15T23:15:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031734990000000002 2024-05-15T23:15:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:15:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.556713+0000", 2024-05-15T23:15:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010498299999999999 2024-05-15T23:15:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:15:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.556821+0000", 2024-05-15T23:15:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107743 2024-05-15T23:15:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:15:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.565377+0000", 2024-05-15T23:15:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085556590000000002 2024-05-15T23:15:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:15:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.661155+0000", 2024-05-15T23:15:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095778314000000003 2024-05-15T23:15:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:15:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.661189+0000", 2024-05-15T23:15:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3883000000000001e-05 2024-05-15T23:15:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:15:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:15:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:15:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:21.661264+0000", 2024-05-15T23:15:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5078000000000002e-05 2024-05-15T23:15:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:15:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:15:00.697 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:15:00.702 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:15:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:00.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:15:00.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:15:00.837 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:01.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:00 smithi084 ceph-mon[39784]: osdmap e285: 12 total, 11 up, 11 in 2024-05-15T23:15:01.267 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:00 smithi090 ceph-mon[44434]: osdmap e285: 12 total, 11 up, 11 in 2024-05-15T23:15:01.269 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T23:15:01.288 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:15:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:01.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:15:01.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:00 smithi088 ceph-mon[44699]: osdmap e285: 12 total, 11 up, 11 in 2024-05-15T23:15:01.392 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:01.400 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-05-15T23:15:01.400 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-05-15T23:15:01.400 DEBUG:teuthology.orchestra.run.smithi084:osd.2> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.2.service 2024-05-15T23:15:01.403 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl start ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.2 2024-05-15T23:15:01.426 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:01.430 INFO:journalctl@ceph.osd.2.smithi084.stdout:-- Logs begin at Wed 2024-05-15 22:43:24 UTC. -- 2024-05-15T23:15:01.447 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:01.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:01.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:01.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:01.581 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:01.869 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:15:01 smithi084 systemd[1]: Starting Ceph osd.2 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:15:02.288 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:01 smithi084 ceph-mon[39784]: pgmap v796: 17 pgs: 7 unknown, 10 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:15:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:01 smithi090 ceph-mon[44434]: pgmap v796: 17 pgs: 7 unknown, 10 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:15:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:01 smithi088 ceph-mon[44699]: pgmap v796: 17 pgs: 7 unknown, 10 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:15:02.915 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:15:02.915 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:15:02.915 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:15:02.915 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:15:02.915 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:15:02.915 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:15:02.928 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:15:02.950 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:15:02.974 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:15:02.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:15:03.018 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:15:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:15:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:15:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-05-15T23:15:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:15:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:15:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:15:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:15:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:15:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:15:03.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:15:03.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:15:03.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:15:03.265 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3666400217' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:15:03.265 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:15:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3666400217' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:15:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:02 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:15:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3666400217' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:15:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:02 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:15:03.715 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:03.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:03.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:03.820 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:03.853 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:03.959 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:03.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:03.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:03.974 INFO:teuthology.orchestra.run.smithi084.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-05-15T23:15:04.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:04.091 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2958615239' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:15:04.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:15:04.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:15:04.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:15:04.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:15:04.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:15:04.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:15:04.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:15:04.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:15:04.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:15:04.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:15:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2958615239' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:15:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:15:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:15:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:15:04.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:15:04.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:15:04.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:15:04.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:15:04.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:15:04.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:15:04.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:15:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2958615239' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:15:04.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:15:04.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:15:04.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:15:04.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:15:04.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:15:04.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:15:04.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:15:04.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:15:04.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:15:04.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:15:04.869 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:15:04 smithi084 bash[152009]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-05-15T23:15:04.869 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:15:04 smithi084 bash[152009]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-15T23:15:04.869 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:15:04 smithi084 bash[152009]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-05-15T23:15:04.869 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:15:04 smithi084 bash[152009]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-05-15T23:15:04.870 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:15:04 smithi084 bash[152009]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-05-15T23:15:04.870 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:15:04 smithi084 bash[152009]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-05-15T23:15:04.870 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:15:04 smithi084 bash[152009]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-15T23:15:04.870 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:15:04 smithi084 bash[152009]: --> ceph-volume raw activate successful for osd ID: 2 2024-05-15T23:15:05.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2958615239' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:15:05.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:04 smithi084 ceph-mon[39784]: osdmap e286: 12 total, 11 up, 11 in 2024-05-15T23:15:05.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2768376542' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:15:05.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:04 smithi084 ceph-mon[39784]: from='client.25432 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:15:05.244 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:05.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:05.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2958615239' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:15:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:04 smithi090 ceph-mon[44434]: osdmap e286: 12 total, 11 up, 11 in 2024-05-15T23:15:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2768376542' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:15:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:04 smithi090 ceph-mon[44434]: from='client.25432 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:15:05.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2958615239' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:15:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:04 smithi088 ceph-mon[44699]: osdmap e286: 12 total, 11 up, 11 in 2024-05-15T23:15:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2768376542' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:15:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:04 smithi088 ceph-mon[44699]: from='client.25432 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:15:05.348 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:15:05.382 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:05.403 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:15:05.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:05.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:15:05.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:15:05.541 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:05.564 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:15:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:05.567 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:15:05.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:15:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:05.716 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:15:05.718 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:15:05.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:05.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:15:05.823 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:15:05.855 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:05.877 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:15:05.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:05.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:15:05.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:15:06.013 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:06.035 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:15:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:06.037 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:15:06.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:15:06.150 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:15:06.150 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:15:06.169 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:06.190 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:15:06.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:06.194 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:15:06.241 DEBUG:teuthology.orchestra.run.smithi090:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-05-15T23:15:06.245 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T23:15:06.245 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T23:15:06.295 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:05 smithi090 ceph-mon[44434]: pgmap v799: 17 pgs: 17 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:15:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:05 smithi090 ceph-mon[44434]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:15:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:05 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:15:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:05 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:15:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:05 smithi090 ceph-mon[44434]: from='client.25432 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:15:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:05 smithi090 ceph-mon[44434]: osdmap e287: 12 total, 11 up, 11 in 2024-05-15T23:15:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:05 smithi088 ceph-mon[44699]: pgmap v799: 17 pgs: 17 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:15:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:05 smithi088 ceph-mon[44699]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:15:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:05 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:15:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:05 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:15:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:05 smithi088 ceph-mon[44699]: from='client.25432 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:15:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:05 smithi088 ceph-mon[44699]: osdmap e287: 12 total, 11 up, 11 in 2024-05-15T23:15:06.328 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:06.341 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:hints = 1 2024-05-15T23:15:06.341 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T23:15:06.341 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object prefix: benchmark_data_smithi090_104448 2024-05-15T23:15:06.341 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:15:06.342 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 0 0 0 0 0 0 - 0 2024-05-15T23:15:06.358 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:06.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:06.378 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:05 smithi084 ceph-mon[39784]: pgmap v799: 17 pgs: 17 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:15:06.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:05 smithi084 ceph-mon[39784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:15:06.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:05 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:15:06.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:05 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:15:06.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:05 smithi084 ceph-mon[39784]: from='client.25432 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:15:06.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:05 smithi084 ceph-mon[39784]: osdmap e287: 12 total, 11 up, 11 in 2024-05-15T23:15:06.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:15:06.507 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:06.534 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:15:06.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:06.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:15:06.638 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:15:06.678 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:06.708 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:15:06.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:06.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:15:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:06.812 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:15:06.845 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:06.876 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:15:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:06.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:15:06.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:15:07.024 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:07.071 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:15:07.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:07.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:15:07.174 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:15:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:07.240 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:15:07.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:07.243 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:15:07.340 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 1 16 5902 5886 367.559 367.875 0.00284059 0.00271477 2024-05-15T23:15:07.344 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:15:07.381 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:07.410 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:15:07.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:07.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:15:07.514 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:07.556 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:07.765 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:07.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:07.871 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:15:07.912 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:07.938 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:15:07.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:07.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:15:08.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:08.081 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:07 smithi084 ceph-mon[39784]: pgmap v801: 17 pgs: 17 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:15:08.084 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:08.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:15:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:07 smithi090 ceph-mon[44434]: pgmap v801: 17 pgs: 17 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:15:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:07 smithi088 ceph-mon[44699]: pgmap v801: 17 pgs: 17 active+clean; 577 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-05-15T23:15:08.340 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 2 16 11539 11523 359.924 352.312 0.00319865 0.00277373 2024-05-15T23:15:08.579 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:08.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:08.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:08.670 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:15:08.685 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:15:08.720 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:08.748 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:15:08.749 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T23:15:08.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:08.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:15:08.817 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:15:08 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-2[152788]: 2024-05-15T23:15:08.663+0000 7f50f32237c0 -1 Falling back to public interface 2024-05-15T23:15:08.817 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:15:08 smithi084 bash[152718]: 8e6e5d35c3c81779b93ffa54dfbb9f5fc069fb0159c6693b9b4002217b2958e6 2024-05-15T23:15:08.818 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:15:08 smithi084 systemd[1]: Started Ceph osd.2 for 3999fb7e-130e-11ef-bc99-c7b262605968. 2024-05-15T23:15:08.853 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:15:08.904 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:08.992 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:15:08.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:08.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:15:09.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:08 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:15:09.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:08 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:15:09.098 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:09.137 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:09.183 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:09.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:09.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:09.287 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:09.292 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:08 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:15:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:08 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:15:09.318 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:08 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:15:09.318 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:08 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:15:09.321 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:09.340 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 3 16 15902 15886 330.846 272.688 0.00679199 0.00301768 2024-05-15T23:15:09.343 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:09.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:09.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:09.380 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:15:09.380 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:15:09.447 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:15:09.486 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:09.513 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:15:09.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:09.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:15:09.618 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:09.653 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:09.678 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:09.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:09.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:09.782 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:15:09.823 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:10.001 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:10 smithi088 ceph-mon[44699]: pgmap v802: 17 pgs: 17 active+clean; 174 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 22 MiB/s wr, 346 op/s 2024-05-15T23:15:10.001 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:10 smithi088 ceph-mon[44699]: osdmap e288: 12 total, 11 up, 11 in 2024-05-15T23:15:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:10 smithi084 ceph-mon[39784]: pgmap v802: 17 pgs: 17 active+clean; 174 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 22 MiB/s wr, 346 op/s 2024-05-15T23:15:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:10 smithi084 ceph-mon[39784]: osdmap e288: 12 total, 11 up, 11 in 2024-05-15T23:15:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:10 smithi090 ceph-mon[44434]: pgmap v802: 17 pgs: 17 active+clean; 174 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 22 MiB/s wr, 346 op/s 2024-05-15T23:15:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:10 smithi090 ceph-mon[44434]: osdmap e288: 12 total, 11 up, 11 in 2024-05-15T23:15:10.340 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 4 16 20584 20568 321.287 292.625 0.00174723 0.00311098 2024-05-15T23:15:11.341 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 5 15 24872 24857 310.639 268.062 0.00408515 0.00321705 2024-05-15T23:15:12.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:12 smithi090 ceph-mon[44434]: pgmap v804: 17 pgs: 17 active+clean; 818 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 118 MiB/s wr, 1.90k op/s 2024-05-15T23:15:12.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:12 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:15:12.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:12 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:15:12.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:12 smithi090 ceph-mon[44434]: osdmap e289: 12 total, 11 up, 11 in 2024-05-15T23:15:12.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:12 smithi088 ceph-mon[44699]: pgmap v804: 17 pgs: 17 active+clean; 818 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 118 MiB/s wr, 1.90k op/s 2024-05-15T23:15:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:12 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:15:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:12 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:15:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:12 smithi088 ceph-mon[44699]: osdmap e289: 12 total, 11 up, 11 in 2024-05-15T23:15:12.341 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 6 16 28770 28754 299.458 243.562 0.00472121 0.00333738 2024-05-15T23:15:12.369 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:15:12 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-2[152788]: 2024-05-15T23:15:12.171+0000 7f50f32237c0 -1 osd.2 261 log_to_monitors true 2024-05-15T23:15:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:12 smithi084 ceph-mon[39784]: pgmap v804: 17 pgs: 17 active+clean; 818 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 118 MiB/s wr, 1.90k op/s 2024-05-15T23:15:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:12 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:15:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:12 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:15:12.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:12 smithi084 ceph-mon[39784]: osdmap e289: 12 total, 11 up, 11 in 2024-05-15T23:15:12.486 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:15:12.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:12.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:15:12.513 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T23:15:12.519 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:12.595 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:15:12.630 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:12.657 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:15:12.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:12.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:15:12.760 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:15:12.796 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:12.821 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:15:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:12.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:15:12.926 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:12.963 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:12.990 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:12.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:12.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:13.095 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:15:13.138 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:13.240 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:13.244 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:15:13.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:13.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:15:13.254 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:15:13 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-2[152788]: 2024-05-15T23:15:13.038+0000 7f50e467a700 -1 osd.2 261 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T23:15:13.256 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:13 smithi084 ceph-mon[39784]: from='osd.2 v1:172.21.15.84:6809/941507493' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-15T23:15:13.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:13 smithi090 ceph-mon[44434]: from='osd.2 v1:172.21.15.84:6809/941507493' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-15T23:15:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:13 smithi088 ceph-mon[44699]: from='osd.2 v1:172.21.15.84:6809/941507493' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-15T23:15:13.341 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 7 16 32395 32379 289.032 226.562 0.00941284 0.00345744 2024-05-15T23:15:13.351 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:13.399 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:13.789 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:13.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:13.876 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:15:13.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:15:13.907 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:15:13.933 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:13.936 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:15:13.958 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:15:13.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:13.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:15:13.967 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:15:14.002 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:15:14.033 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:15:14.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:15:14.064 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:15:14.096 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:15:14.101 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:14.128 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:15:14.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:14.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:15:14.132 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:15:14.163 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:15:14.194 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:15:14.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:15:14.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:15:14.195 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-05-15T23:15:14.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:15:14.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:15:14.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:15:14.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:15:14.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:15:14.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:15:14.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:15:14.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:15:14.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:15:14.232 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:14.265 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:14.290 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:14.293 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:14 smithi088 ceph-mon[44699]: pgmap v806: 17 pgs: 17 active+clean; 1.4 GiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 182 MiB/s wr, 2.92k op/s 2024-05-15T23:15:14.293 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:14 smithi088 ceph-mon[44699]: from='osd.2 v1:172.21.15.84:6809/941507493' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-15T23:15:14.293 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:14 smithi088 ceph-mon[44699]: osdmap e290: 12 total, 11 up, 11 in 2024-05-15T23:15:14.293 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:14 smithi088 ceph-mon[44699]: from='osd.2 v1:172.21.15.84:6809/941507493' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T23:15:14.293 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:14 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:15:14.293 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:14 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2919347073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:15:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:14.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:14 smithi090 ceph-mon[44434]: pgmap v806: 17 pgs: 17 active+clean; 1.4 GiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 182 MiB/s wr, 2.92k op/s 2024-05-15T23:15:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:14 smithi090 ceph-mon[44434]: from='osd.2 v1:172.21.15.84:6809/941507493' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-15T23:15:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:14 smithi090 ceph-mon[44434]: osdmap e290: 12 total, 11 up, 11 in 2024-05-15T23:15:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:14 smithi090 ceph-mon[44434]: from='osd.2 v1:172.21.15.84:6809/941507493' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T23:15:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:14 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:15:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:14 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2919347073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:15:14.341 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 8 16 35808 35792 279.566 213.312 0.00331727 0.0035757 2024-05-15T23:15:14.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:14 smithi084 ceph-mon[39784]: pgmap v806: 17 pgs: 17 active+clean; 1.4 GiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 182 MiB/s wr, 2.92k op/s 2024-05-15T23:15:14.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:14 smithi084 ceph-mon[39784]: from='osd.2 v1:172.21.15.84:6809/941507493' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-15T23:15:14.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:14 smithi084 ceph-mon[39784]: osdmap e290: 12 total, 11 up, 11 in 2024-05-15T23:15:14.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:14 smithi084 ceph-mon[39784]: from='osd.2 v1:172.21.15.84:6809/941507493' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T23:15:14.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:14 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:15:14.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:14 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2919347073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:15:14.395 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:14.431 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:14.457 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:14.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:14.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:14.562 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:14.607 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:14.638 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:14.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:14.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:14.742 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:15:14.783 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:14.810 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:15:14.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:14.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:15:14.916 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:14.964 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:15.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:15 smithi084 ceph-mon[39784]: osd.2 v1:172.21.15.84:6809/941507493 boot 2024-05-15T23:15:15.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:15 smithi084 ceph-mon[39784]: osdmap e291: 12 total, 12 up, 11 in 2024-05-15T23:15:15.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:15 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:15:15.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:15 smithi090 ceph-mon[44434]: osd.2 v1:172.21.15.84:6809/941507493 boot 2024-05-15T23:15:15.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:15 smithi090 ceph-mon[44434]: osdmap e291: 12 total, 12 up, 11 in 2024-05-15T23:15:15.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:15 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:15:15.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:15 smithi088 ceph-mon[44699]: osd.2 v1:172.21.15.84:6809/941507493 boot 2024-05-15T23:15:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:15 smithi088 ceph-mon[44699]: osdmap e291: 12 total, 12 up, 11 in 2024-05-15T23:15:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:15 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T23:15:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:15.341 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 9 16 39190 39174 271.989 211.375 0.00641702 0.00367473 2024-05-15T23:15:15.347 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:15:15.348 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:15:15.608 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:15.745 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:15.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:15.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:15.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:15:15.893 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:15.921 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:15:15.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:15.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:15:16.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:15:16.072 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:16.100 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:15:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:16.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:15:16.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:16.254 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:16.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:16.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:16 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:15:16.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:16 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:15:16.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:16 smithi090 ceph-mon[44434]: pgmap v809: 17 pgs: 17 active+clean; 1.7 GiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 267 MiB/s wr, 4.28k op/s 2024-05-15T23:15:16.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:16 smithi090 ceph-mon[44434]: osdmap e292: 12 total, 12 up, 11 in 2024-05-15T23:15:16.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:16 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:15:16.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:16 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:15:16.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:16.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:16 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:15:16.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:16 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:15:16.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:16 smithi088 ceph-mon[44699]: pgmap v809: 17 pgs: 17 active+clean; 1.7 GiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 267 MiB/s wr, 4.28k op/s 2024-05-15T23:15:16.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:16 smithi088 ceph-mon[44699]: osdmap e292: 12 total, 12 up, 11 in 2024-05-15T23:15:16.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:16 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:15:16.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:16 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:15:16.341 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 10 16 42462 42446 265.24 204.5 0.00594928 0.00376799 2024-05-15T23:15:16.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:16.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:16 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:15:16.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:16 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:15:16.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:16 smithi084 ceph-mon[39784]: pgmap v809: 17 pgs: 17 active+clean; 1.7 GiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 267 MiB/s wr, 4.28k op/s 2024-05-15T23:15:16.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:16 smithi084 ceph-mon[39784]: osdmap e292: 12 total, 12 up, 11 in 2024-05-15T23:15:16.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:16 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:15:16.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:16 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:15:16.716 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:16.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:16.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:16.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:15:16.877 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:17.141 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:15:17.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:17.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:15:17.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:15:17.290 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:17.316 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:15:17.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:17.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:15:17.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:17 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:15:17.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:17 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:15:17.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:17 smithi088 ceph-mon[44699]: osdmap e293: 12 total, 12 up, 11 in 2024-05-15T23:15:17.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:17 smithi088 ceph-mon[44699]: osdmap e294: 12 total, 12 up, 11 in 2024-05-15T23:15:17.322 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:17 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:15:17.322 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:17 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:15:17.322 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:17 smithi090 ceph-mon[44434]: osdmap e293: 12 total, 12 up, 11 in 2024-05-15T23:15:17.322 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:17 smithi090 ceph-mon[44434]: osdmap e294: 12 total, 12 up, 11 in 2024-05-15T23:15:17.341 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 11 16 45627 45611 259.109 197.812 0.00587294 0.00385793 2024-05-15T23:15:17.349 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:15:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:17 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:15:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:17 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:15:17.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:17 smithi084 ceph-mon[39784]: osdmap e293: 12 total, 12 up, 11 in 2024-05-15T23:15:17.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:17 smithi084 ceph-mon[39784]: osdmap e294: 12 total, 12 up, 11 in 2024-05-15T23:15:17.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:15:17.470 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:17.497 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:15:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:17.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:15:17.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:15:17.643 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:17.663 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:15:17.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:17.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:15:17.767 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:15:17.807 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:17.835 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:15:17.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:17.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:15:17.939 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:15:17.983 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:18.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:18 smithi088 ceph-mon[44699]: pgmap v812: 17 pgs: 17 active+clean; 2.1 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 187 MiB/s wr, 2.98k op/s 2024-05-15T23:15:18.342 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 12 16 48419 48403 252.058 174.5 0.00505151 0.00396544 2024-05-15T23:15:18.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:18 smithi084 ceph-mon[39784]: pgmap v812: 17 pgs: 17 active+clean; 2.1 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 187 MiB/s wr, 2.98k op/s 2024-05-15T23:15:18.472 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:15:18.473 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:15:18.473 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-05-15T23:15:18.473 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:15:18.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:18 smithi090 ceph-mon[44434]: pgmap v812: 17 pgs: 17 active+clean; 2.1 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 187 MiB/s wr, 2.98k op/s 2024-05-15T23:15:18.938 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:15:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:18.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:15:19.042 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:15:19.082 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:15:19.089 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:19.120 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:15:19.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:19.124 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:15:19.225 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:19.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3689617637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:15:19.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:19 smithi084 ceph-mon[39784]: from='client.25456 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:15:19.278 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:19.342 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 13 16 51285 51269 246.447 179.125 0.00489917 0.00405601 2024-05-15T23:15:19.541 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:19.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:19.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3689617637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:15:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:19 smithi090 ceph-mon[44434]: from='client.25456 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:15:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3689617637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:15:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:19 smithi088 ceph-mon[44699]: from='client.25456 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:15:19.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:15:19.686 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:19.722 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:15:19.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:19.727 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:15:19.828 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:15:19.871 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:19.901 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:15:19.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:19.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:15:20.007 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:20.043 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:20.071 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:20.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:20.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:20.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:20 smithi084 ceph-mon[39784]: pgmap v814: 17 pgs: 17 active+clean; 2.5 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 235 MiB/s wr, 3.75k op/s 2024-05-15T23:15:20.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:20 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:15:20.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:20 smithi084 ceph-mon[39784]: from='client.25456 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:15:20.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:20 smithi084 ceph-mon[39784]: osdmap e295: 12 total, 12 up, 11 in 2024-05-15T23:15:20.230 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:20.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:20 smithi088 ceph-mon[44699]: pgmap v814: 17 pgs: 17 active+clean; 2.5 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 235 MiB/s wr, 3.75k op/s 2024-05-15T23:15:20.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:20 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:15:20.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:20 smithi088 ceph-mon[44699]: from='client.25456 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:15:20.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:20 smithi088 ceph-mon[44699]: osdmap e295: 12 total, 12 up, 11 in 2024-05-15T23:15:20.342 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 14 16 54122 54106 241.508 177.312 0.0032329 0.00413915 2024-05-15T23:15:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:20 smithi090 ceph-mon[44434]: pgmap v814: 17 pgs: 17 active+clean; 2.5 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 235 MiB/s wr, 3.75k op/s 2024-05-15T23:15:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:20 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:15:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:20 smithi090 ceph-mon[44434]: from='client.25456 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:15:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:20 smithi090 ceph-mon[44434]: osdmap e295: 12 total, 12 up, 11 in 2024-05-15T23:15:20.630 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:20.664 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:15:20.664 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-05-15T23:15:20.665 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T23:15:20.665 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T23:15:20.665 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-05-15T23:15:20.894 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:20.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:20.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:21.003 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:15:21.042 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:21.070 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:15:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:21.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:15:21.176 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:21.204 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:21.231 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:21.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:21.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:15:21.341 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 15 16 56695 56679 236.128 160.812 0.00457363 0.00423369 2024-05-15T23:15:21.389 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:21.437 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:15:21.437 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:15:21.437 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:15:21.437 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:15:21.437 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:15:22.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:15:22.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:22 smithi084 ceph-mon[39784]: pgmap v816: 17 pgs: 17 active+clean; 3.0 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 233 MiB/s wr, 3.73k op/s 2024-05-15T23:15:22.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:22 smithi084 ceph-mon[39784]: osdmap e296: 12 total, 12 up, 11 in 2024-05-15T23:15:22.342 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 16 16 59342 59326 231.709 165.438 0.00363658 0.00431355 2024-05-15T23:15:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:22 smithi090 ceph-mon[44434]: pgmap v816: 17 pgs: 17 active+clean; 3.0 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 233 MiB/s wr, 3.73k op/s 2024-05-15T23:15:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:22 smithi090 ceph-mon[44434]: osdmap e296: 12 total, 12 up, 11 in 2024-05-15T23:15:22.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:22 smithi088 ceph-mon[44699]: pgmap v816: 17 pgs: 17 active+clean; 3.0 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 233 MiB/s wr, 3.73k op/s 2024-05-15T23:15:22.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:22 smithi088 ceph-mon[44699]: osdmap e296: 12 total, 12 up, 11 in 2024-05-15T23:15:22.991 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:15:22.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:22.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:15:23.098 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:15:23.137 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:23.327 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:23.342 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 17 16 61707 61691 226.774 147.812 0.0022843 0.00440666 2024-05-15T23:15:23.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:15:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:15:23.392 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:15:23.420 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:15:23.449 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:15:23.482 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:15:23.514 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:15:23.546 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:15:23.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:15:23.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:15:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:15:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:15:23.576 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:15:23.608 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:15:23.640 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:15:23.667 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:15:23.688 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:15:23.727 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:15:23.737 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:15:23.737 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:15:23.737 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:15:23.737 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:15:23.737 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:15:23.737 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:15:23.737 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:15:23.738 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:15:23.738 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:15:23.738 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:15:23.738 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:15:23.738 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:15:23.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T23:15:24.300 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:15:24.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:24.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:15:24.342 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 18 16 63975 63959 222.05 141.75 0.009254 0.00450213 2024-05-15T23:15:24.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:24 smithi084 ceph-mon[39784]: pgmap v818: 17 pgs: 17 active+clean; 3.4 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 215 MiB/s wr, 3.44k op/s 2024-05-15T23:15:24.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:24 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/809318557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:15:24.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.410 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:15:24.440 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:24 smithi088 ceph-mon[44699]: pgmap v818: 17 pgs: 17 active+clean; 3.4 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 215 MiB/s wr, 3.44k op/s 2024-05-15T23:15:24.441 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:24 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/809318557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:15:24.441 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.441 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.441 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.441 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.441 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.441 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.443 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:24.468 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:15:24.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:24.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:15:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:24 smithi090 ceph-mon[44434]: pgmap v818: 17 pgs: 17 active+clean; 3.4 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 215 MiB/s wr, 3.44k op/s 2024-05-15T23:15:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:24 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/809318557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:15:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:24.571 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:24.625 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:24.971 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:24.979 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:24.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:24.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:25.085 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:15:25.132 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:25.244 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:15:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:25.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:15:25.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.342 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 19 16 66411 66395 218.376 152.25 0.00908185 0.0045779 2024-05-15T23:15:25.350 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:15:25.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.391 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.393 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:25.417 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:15:25.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:25.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:15:25.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:15:25.557 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:25.585 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:15:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:25.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:15:25.689 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:25.725 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:26.342 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:15:26.342000 min lat: 0.00102037 max lat: 0.0733665 avg lat: 0.00465045 2024-05-15T23:15:26.342 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:15:26.342 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 20 16 68812 68796 214.96 150.062 0.00297399 0.00465045 2024-05-15T23:15:26.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:26 smithi084 ceph-mon[39784]: pgmap v819: 17 pgs: 17 active+clean; 3.6 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 187 MiB/s wr, 2.99k op/s 2024-05-15T23:15:26.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:26 smithi090 ceph-mon[44434]: pgmap v819: 17 pgs: 17 active+clean; 3.6 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 187 MiB/s wr, 2.99k op/s 2024-05-15T23:15:26.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:26 smithi088 ceph-mon[44699]: pgmap v819: 17 pgs: 17 active+clean; 3.6 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 187 MiB/s wr, 2.99k op/s 2024-05-15T23:15:27.022 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:15:27.258 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:27.292 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:27.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:27.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:27.342 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 21 16 71152 71136 211.687 146.25 0.00629451 0.00472246 2024-05-15T23:15:27.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:15:27.455 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:27.571 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:15:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:27.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:15:27.682 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:15:27.740 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:28.342 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 22 16 73470 73454 208.65 144.875 0.00411706 0.00479131 2024-05-15T23:15:28.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:28 smithi084 ceph-mon[39784]: pgmap v820: 17 pgs: 17 active+clean; 3.9 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 174 MiB/s wr, 2.79k op/s 2024-05-15T23:15:28.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:28 smithi090 ceph-mon[44434]: pgmap v820: 17 pgs: 17 active+clean; 3.9 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 174 MiB/s wr, 2.79k op/s 2024-05-15T23:15:28.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:28 smithi088 ceph-mon[44699]: pgmap v820: 17 pgs: 17 active+clean; 3.9 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 174 MiB/s wr, 2.79k op/s 2024-05-15T23:15:28.571 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:15:28.571 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-15T23:15:28.571 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:15:29.342 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 23 15 75740 75725 205.749 141.938 0.00989896 0.00485878 2024-05-15T23:15:29.801 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:15:29.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:29.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:15:29.908 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:29.950 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:30.201 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:30.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:30.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:30.308 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:15:30.342 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 24 16 77908 77892 202.819 135.438 0.00364893 0.00492856 2024-05-15T23:15:30.345 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:30.377 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:15:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:30.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:30 smithi088 ceph-mon[44699]: pgmap v821: 17 pgs: 17 active+clean; 4.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 171 MiB/s wr, 2.73k op/s 2024-05-15T23:15:30.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:15:30.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:30 smithi084 ceph-mon[39784]: pgmap v821: 17 pgs: 17 active+clean; 4.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 171 MiB/s wr, 2.73k op/s 2024-05-15T23:15:30.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:15:30.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:30 smithi090 ceph-mon[44434]: pgmap v821: 17 pgs: 17 active+clean; 4.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 171 MiB/s wr, 2.73k op/s 2024-05-15T23:15:30.520 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:30.551 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:15:30.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:30.556 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:15:30.657 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:30.719 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:30.824 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:30.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:30.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:30.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:15:30.963 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:30.986 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:30.986 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:15:30.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:30.989 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:15:31.007 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:15:31.007 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:15:31.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:15:31.120 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:31.141 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:15:31.147 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:15:31.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:31.152 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:15:31.253 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:31.288 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:31.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1655972465' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:15:31.318 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:31.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:31.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:31.343 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 25 16 79887 79871 199.652 123.688 0.00926276 0.00500739 2024-05-15T23:15:31.352 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1655972465' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:15:31.425 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:31.456 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1655972465' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:15:31.458 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:31.490 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:31.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:31.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:31.643 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:32.343 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 26 16 82042 82026 197.153 134.688 0.010055 0.00507101 2024-05-15T23:15:32.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:32 smithi084 ceph-mon[39784]: pgmap v822: 17 pgs: 17 active+clean; 4.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 159 MiB/s wr, 2.55k op/s 2024-05-15T23:15:32.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:32 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:15:32.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1655972465' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:15:32.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:32 smithi084 ceph-mon[39784]: osdmap e297: 12 total, 12 up, 11 in 2024-05-15T23:15:32.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:32 smithi084 ceph-mon[39784]: osdmap e298: 12 total, 12 up, 11 in 2024-05-15T23:15:32.511 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:32.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:32.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:32.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:32 smithi090 ceph-mon[44434]: pgmap v822: 17 pgs: 17 active+clean; 4.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 159 MiB/s wr, 2.55k op/s 2024-05-15T23:15:32.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:32 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:15:32.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1655972465' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:15:32.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:32 smithi090 ceph-mon[44434]: osdmap e297: 12 total, 12 up, 11 in 2024-05-15T23:15:32.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:32 smithi090 ceph-mon[44434]: osdmap e298: 12 total, 12 up, 11 in 2024-05-15T23:15:32.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:32 smithi088 ceph-mon[44699]: pgmap v822: 17 pgs: 17 active+clean; 4.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 159 MiB/s wr, 2.55k op/s 2024-05-15T23:15:32.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:32 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:15:32.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1655972465' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:15:32.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:32 smithi088 ceph-mon[44699]: osdmap e297: 12 total, 12 up, 11 in 2024-05-15T23:15:32.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:32 smithi088 ceph-mon[44699]: osdmap e298: 12 total, 12 up, 11 in 2024-05-15T23:15:32.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:15:32.667 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:32.697 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:15:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:32.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:15:32.803 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:15:32.850 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:32.877 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:15:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:32.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:15:32.982 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:15:33.015 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:33.040 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:15:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:33.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:15:33.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:15:33.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T23:15:33.180 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:33.207 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:15:33.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:33.210 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:15:33.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:33.343 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 27 16 84133 84117 194.691 130.688 0.00367842 0.00513508 2024-05-15T23:15:33.365 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:33.402 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:33.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:33.508 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:15:33.544 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:33.570 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:15:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:33.574 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:15:33.675 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:15:33.723 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:33.934 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:15:33.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:33.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:15:34.042 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:15:34.084 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:34.109 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:15:34.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:34.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:15:34.213 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:15:34.234 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:15:34.243 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:34.265 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:15:34.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:34.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:15:34.274 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:15:34.281 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:34.309 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:15:34.339 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:15:34.343 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 28 16 86057 86041 192.032 120.25 0.00605046 0.00520596 2024-05-15T23:15:34.369 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:15:34.374 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:15:34.402 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:34 smithi088 ceph-mon[44699]: pgmap v825: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 158 MiB/s wr, 2.54k op/s 2024-05-15T23:15:34.402 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:34 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:15:34.402 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:34 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:15:34.402 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:34 smithi088 ceph-mon[44699]: osdmap e299: 12 total, 12 up, 11 in 2024-05-15T23:15:34.404 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:34.410 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:15:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:15:34.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:34.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:15:34.443 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:15:34.478 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:15:34.512 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:15:34.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:15:34.537 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:15:34.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:34 smithi090 ceph-mon[44434]: pgmap v825: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 158 MiB/s wr, 2.54k op/s 2024-05-15T23:15:34.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:34 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:15:34.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:34 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:15:34.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:34 smithi090 ceph-mon[44434]: osdmap e299: 12 total, 12 up, 11 in 2024-05-15T23:15:34.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:34 smithi084 ceph-mon[39784]: pgmap v825: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 158 MiB/s wr, 2.54k op/s 2024-05-15T23:15:34.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:34 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:15:34.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:34 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:15:34.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:34 smithi084 ceph-mon[39784]: osdmap e299: 12 total, 12 up, 11 in 2024-05-15T23:15:34.570 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:15:34.571 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:34.595 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:15:34.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:34.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:15:34.602 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:15:34.605 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:15:34.605 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:15:34.605 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:15:34.605 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:15:34.605 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:15:34.605 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:15:34.605 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:15:34.605 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:15:34.605 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:15:34.605 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:15:34.605 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:15:34.605 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:15:34.702 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:15:34.745 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:34.770 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:15:34.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:34.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:15:34.879 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:34.923 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:34.983 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:34.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:34.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:35.087 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:35.131 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:35.343 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 29 16 88022 88006 189.645 122.812 0.00402606 0.00527156 2024-05-15T23:15:35.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:35 smithi084 ceph-mon[39784]: from='client.15561 v1:172.21.15.84:0/1414345757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:15:35.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:35 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:15:35.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:35 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:15:35.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.396 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 11, 9, 10] out_osds: [2] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-05-15T23:15:35.397 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:15:35.397 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:15:35.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:35 smithi090 ceph-mon[44434]: from='client.15561 v1:172.21.15.84:0/1414345757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:15:35.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:35 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:15:35.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:35 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:15:35.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:35 smithi088 ceph-mon[44699]: from='client.15561 v1:172.21.15.84:0/1414345757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:15:35.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:35 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:15:35.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:35 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:15:35.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:35.599 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:35.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:35.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:35.705 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:15:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:35.760 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:15:35.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:35.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:15:35.866 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:15:35.898 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:35.919 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:15:35.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:35.927 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:15:35.970 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:15:36.028 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:15:36.061 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:36.083 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:15:36.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:36.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:15:36.189 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:36.249 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:36.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:36.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:36.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:36.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:36.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:36.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:36.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:36 smithi088 ceph-mon[44699]: pgmap v827: 17 pgs: 17 active+clean; 5.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-05-15T23:15:36.343 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 30 16 89915 89899 187.267 118.312 0.0125701 0.00533876 2024-05-15T23:15:36.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:36.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:36.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:36.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:36.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:36.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:36 smithi084 ceph-mon[39784]: pgmap v827: 17 pgs: 17 active+clean; 5.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-05-15T23:15:36.399 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:36.458 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:36.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:36.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:36.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:36.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:36 smithi090 ceph-mon[44434]: pgmap v827: 17 pgs: 17 active+clean; 5.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-05-15T23:15:36.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:15:36.590 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:36.618 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:15:36.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:36.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:15:36.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:15:36.761 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:36.789 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:15:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:36.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:15:36.896 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:15:36.935 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:36.966 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:15:36.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:36.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:15:37.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:15:37.094 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:37.106 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:37.128 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:15:37.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:37.131 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:15:37.155 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:37.187 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:15:37.188 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:15:37.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.188 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:91359 4.4 4:266051b4:::benchmark_data_smithi090_104448_object91358:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e300)", 2024-05-15T23:15:37.188 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:15:37.154295+0000", 2024-05-15T23:15:37.188 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.020215318999999999, 2024-05-15T23:15:37.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020256151999999999, 2024-05-15T23:15:37.188 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:15:37.188 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:15:37.188 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:15:37.188 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:15:37.188 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:15:37.188 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:15:37.189 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 91359 2024-05-15T23:15:37.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.189 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:15:37.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:15:37.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.154295+0000", 2024-05-15T23:15:37.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:15:37.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.154295+0000", 2024-05-15T23:15:37.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:15:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.154297+0000", 2024-05-15T23:15:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9520000000000001e-06 2024-05-15T23:15:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:15:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.154330+0000", 2024-05-15T23:15:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3427999999999999e-05 2024-05-15T23:15:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:15:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.154348+0000", 2024-05-15T23:15:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7363e-05 2024-05-15T23:15:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:15:37.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.154357+0000", 2024-05-15T23:15:37.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0759999999999992e-06 2024-05-15T23:15:37.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:15:37.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.155560+0000", 2024-05-15T23:15:37.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012035920000000001 2024-05-15T23:15:37.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:15:37.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.155690+0000", 2024-05-15T23:15:37.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000130152 2024-05-15T23:15:37.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3", 2024-05-15T23:15:37.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.155816+0000", 2024-05-15T23:15:37.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012561500000000001 2024-05-15T23:15:37.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:15:37.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.161521+0000", 2024-05-15T23:15:37.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0057048350000000001 2024-05-15T23:15:37.193 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.193 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:15:37.193 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.193 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:91369 4.4 4:276242bf:::benchmark_data_smithi090_104448_object91368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e300)", 2024-05-15T23:15:37.193 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:15:37.157923+0000", 2024-05-15T23:15:37.193 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.016587068999999999, 2024-05-15T23:15:37.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016749000999999999, 2024-05-15T23:15:37.193 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:15:37.193 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:15:37.193 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:15:37.193 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:15:37.194 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:15:37.194 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:15:37.194 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 91369 2024-05-15T23:15:37.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.194 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:15:37.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:15:37.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.157923+0000", 2024-05-15T23:15:37.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:15:37.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.157923+0000", 2024-05-15T23:15:37.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:15:37.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.157925+0000", 2024-05-15T23:15:37.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2350000000000002e-06 2024-05-15T23:15:37.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:15:37.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.157961+0000", 2024-05-15T23:15:37.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6164000000000001e-05 2024-05-15T23:15:37.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:15:37.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.157974+0000", 2024-05-15T23:15:37.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2323999999999999e-05 2024-05-15T23:15:37.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:15:37.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.157982+0000", 2024-05-15T23:15:37.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0369999999999995e-06 2024-05-15T23:15:37.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:15:37.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.158625+0000", 2024-05-15T23:15:37.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00064351999999999996 2024-05-15T23:15:37.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:15:37.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.158780+0000", 2024-05-15T23:15:37.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015484599999999999 2024-05-15T23:15:37.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3", 2024-05-15T23:15:37.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.158911+0000", 2024-05-15T23:15:37.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013124 2024-05-15T23:15:37.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:15:37.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.164625+0000", 2024-05-15T23:15:37.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0057133870000000003 2024-05-15T23:15:37.198 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.198 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:15:37.198 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.198 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:91380 4.4 4:26262ec7:::benchmark_data_smithi090_104448_object91379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e300)", 2024-05-15T23:15:37.198 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:15:37.164602+0000", 2024-05-15T23:15:37.198 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.009907908, 2024-05-15T23:15:37.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010170339, 2024-05-15T23:15:37.198 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:15:37.199 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:15:37.199 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:15:37.199 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:15:37.199 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:15:37.199 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:15:37.199 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 91380 2024-05-15T23:15:37.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.199 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:15:37.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:15:37.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.164602+0000", 2024-05-15T23:15:37.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:15:37.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.164602+0000", 2024-05-15T23:15:37.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:15:37.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.164604+0000", 2024-05-15T23:15:37.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1059999999999998e-06 2024-05-15T23:15:37.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:15:37.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.164647+0000", 2024-05-15T23:15:37.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2904999999999997e-05 2024-05-15T23:15:37.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:15:37.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.164658+0000", 2024-05-15T23:15:37.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1260999999999999e-05 2024-05-15T23:15:37.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:15:37.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.164667+0000", 2024-05-15T23:15:37.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7490000000000004e-06 2024-05-15T23:15:37.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:15:37.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.165139+0000", 2024-05-15T23:15:37.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00047224700000000002 2024-05-15T23:15:37.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:15:37.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.165245+0000", 2024-05-15T23:15:37.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105573 2024-05-15T23:15:37.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3", 2024-05-15T23:15:37.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.165341+0000", 2024-05-15T23:15:37.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5756999999999998e-05 2024-05-15T23:15:37.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:15:37.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.170840+0000", 2024-05-15T23:15:37.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054996300000000001 2024-05-15T23:15:37.203 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.203 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:15:37.203 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.203 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:91387 4.4 4:2f49894d:::benchmark_data_smithi090_104448_object91386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e300)", 2024-05-15T23:15:37.203 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:15:37.167413+0000", 2024-05-15T23:15:37.203 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.0070965890000000004, 2024-05-15T23:15:37.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074530289999999999, 2024-05-15T23:15:37.203 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:15:37.203 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:15:37.203 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:15:37.203 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:15:37.203 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:15:37.204 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:15:37.204 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 91387 2024-05-15T23:15:37.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.204 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:15:37.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:15:37.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.167413+0000", 2024-05-15T23:15:37.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:15:37.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.167413+0000", 2024-05-15T23:15:37.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:15:37.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.167416+0000", 2024-05-15T23:15:37.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0470000000000001e-06 2024-05-15T23:15:37.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:15:37.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.167459+0000", 2024-05-15T23:15:37.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3297999999999998e-05 2024-05-15T23:15:37.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:15:37.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.167470+0000", 2024-05-15T23:15:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0845e-05 2024-05-15T23:15:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:15:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.167478+0000", 2024-05-15T23:15:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1470000000000001e-06 2024-05-15T23:15:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:15:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.167554+0000", 2024-05-15T23:15:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6651999999999997e-05 2024-05-15T23:15:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:15:37.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.167722+0000", 2024-05-15T23:15:37.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016723500000000001 2024-05-15T23:15:37.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3", 2024-05-15T23:15:37.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.167894+0000", 2024-05-15T23:15:37.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017237599999999999 2024-05-15T23:15:37.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:15:37.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.171017+0000", 2024-05-15T23:15:37.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031232650000000001 2024-05-15T23:15:37.209 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.209 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:15:37.209 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.209 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.25441.0:91388 4.b e300/283 4:d4616beb:::benchmark_data_smithi090_104448_object91387:head v 300'5739, mlcod=300'5736)", 2024-05-15T23:15:37.209 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:15:37.170466+0000", 2024-05-15T23:15:37.209 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.004044265, 2024-05-15T23:15:37.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044878549999999998, 2024-05-15T23:15:37.209 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:15:37.209 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:15:37.209 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-05-15T23:15:37.209 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:15:37.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:15:37.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.170466+0000", 2024-05-15T23:15:37.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:15:37.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.170466+0000", 2024-05-15T23:15:37.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:15:37.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.170468+0000", 2024-05-15T23:15:37.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.807e-06 2024-05-15T23:15:37.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:15:37.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.170499+0000", 2024-05-15T23:15:37.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1210000000000001e-05 2024-05-15T23:15:37.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:15:37.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.170510+0000", 2024-05-15T23:15:37.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1174e-05 2024-05-15T23:15:37.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:15:37.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.170519+0000", 2024-05-15T23:15:37.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8810000000000002e-06 2024-05-15T23:15:37.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:15:37.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.170608+0000", 2024-05-15T23:15:37.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8962999999999995e-05 2024-05-15T23:15:37.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:15:37.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.170625+0000", 2024-05-15T23:15:37.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7612e-05 2024-05-15T23:15:37.212 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.212 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:15:37.212 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.213 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:91393 4.f 4:f6d03704:::benchmark_data_smithi090_104448_object91392:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e300)", 2024-05-15T23:15:37.213 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:15:37.170906+0000", 2024-05-15T23:15:37.213 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.0036039029999999999, 2024-05-15T23:15:37.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041252850000000002, 2024-05-15T23:15:37.213 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:15:37.213 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:15:37.213 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:15:37.213 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:15:37.213 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:15:37.213 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:15:37.213 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 91393 2024-05-15T23:15:37.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.213 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:15:37.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:15:37.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.170906+0000", 2024-05-15T23:15:37.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:15:37.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.170906+0000", 2024-05-15T23:15:37.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:15:37.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.170909+0000", 2024-05-15T23:15:37.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3910000000000001e-06 2024-05-15T23:15:37.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:15:37.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.170975+0000", 2024-05-15T23:15:37.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6622e-05 2024-05-15T23:15:37.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:15:37.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.170993+0000", 2024-05-15T23:15:37.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7785999999999999e-05 2024-05-15T23:15:37.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:15:37.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.171003+0000", 2024-05-15T23:15:37.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9280000000000005e-06 2024-05-15T23:15:37.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:15:37.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.171128+0000", 2024-05-15T23:15:37.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000124759 2024-05-15T23:15:37.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:15:37.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.171280+0000", 2024-05-15T23:15:37.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015243799999999999 2024-05-15T23:15:37.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3", 2024-05-15T23:15:37.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.171430+0000", 2024-05-15T23:15:37.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000149629 2024-05-15T23:15:37.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:15:37.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.175003+0000", 2024-05-15T23:15:37.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0035736790000000002 2024-05-15T23:15:37.217 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.217 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:15:37.217 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.217 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:91395 4.4 4:236080cb:::benchmark_data_smithi090_104448_object91394:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e300)", 2024-05-15T23:15:37.217 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:15:37.171128+0000", 2024-05-15T23:15:37.217 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.0033818250000000002, 2024-05-15T23:15:37.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039924649999999997, 2024-05-15T23:15:37.218 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:15:37.218 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:15:37.218 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:15:37.218 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:15:37.218 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:15:37.218 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:15:37.218 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 91395 2024-05-15T23:15:37.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.218 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:15:37.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:15:37.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.171128+0000", 2024-05-15T23:15:37.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:15:37.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.171128+0000", 2024-05-15T23:15:37.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:15:37.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.171130+0000", 2024-05-15T23:15:37.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.389e-06 2024-05-15T23:15:37.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:15:37.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.171209+0000", 2024-05-15T23:15:37.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.962e-05 2024-05-15T23:15:37.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:15:37.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.171217+0000", 2024-05-15T23:15:37.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7999999999999999e-06 2024-05-15T23:15:37.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:15:37.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.171226+0000", 2024-05-15T23:15:37.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5609999999999999e-06 2024-05-15T23:15:37.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:15:37.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.171271+0000", 2024-05-15T23:15:37.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5284999999999998e-05 2024-05-15T23:15:37.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:15:37.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.171384+0000", 2024-05-15T23:15:37.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113526 2024-05-15T23:15:37.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3", 2024-05-15T23:15:37.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.171518+0000", 2024-05-15T23:15:37.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133953 2024-05-15T23:15:37.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:15:37.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.174980+0000", 2024-05-15T23:15:37.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034612509999999998 2024-05-15T23:15:37.222 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.222 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:15:37.222 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.222 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.25441.0:91396 4.6 e300/283 4:62b6a55a:::benchmark_data_smithi090_104448_object91395:head v 300'5691, mlcod=300'5689)", 2024-05-15T23:15:37.222 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:15:37.174256+0000", 2024-05-15T23:15:37.222 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.00025434999999999998, 2024-05-15T23:15:37.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093959399999999998, 2024-05-15T23:15:37.222 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:15:37.222 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:15:37.222 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-05-15T23:15:37.223 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:15:37.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:15:37.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.174256+0000", 2024-05-15T23:15:37.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:15:37.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.174256+0000", 2024-05-15T23:15:37.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:15:37.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:15:37.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.174257+0000", 2024-05-15T23:15:37.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1799999999999999e-06 2024-05-15T23:15:37.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:15:37.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.174293+0000", 2024-05-15T23:15:37.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5856999999999999e-05 2024-05-15T23:15:37.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:15:37.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.174302+0000", 2024-05-15T23:15:37.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8710000000000003e-06 2024-05-15T23:15:37.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:15:37.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.174309+0000", 2024-05-15T23:15:37.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3390000000000004e-06 2024-05-15T23:15:37.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:15:37.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.174571+0000", 2024-05-15T23:15:37.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026170700000000002 2024-05-15T23:15:37.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:15:37.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:15:37.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:15:37.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:15:37.174589+0000", 2024-05-15T23:15:37.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8788e-05 2024-05-15T23:15:37.225 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.225 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:15:37.225 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.225 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:15:37.226 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:15:37.226 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 8 2024-05-15T23:15:37.226 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:15:37.232 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:15:37.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:37 smithi088 ceph-mon[44699]: pgmap v828: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-05-15T23:15:37.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:37 smithi088 ceph-mon[44699]: osdmap e300: 12 total, 12 up, 11 in 2024-05-15T23:15:37.287 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:37.315 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:15:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:37.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:15:37.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 31 16 91715 91699 184.848 112.5 0.00590831 0.00540859 2024-05-15T23:15:37.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:15:37.466 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:37 smithi084 ceph-mon[39784]: pgmap v828: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-05-15T23:15:37.466 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:37 smithi084 ceph-mon[39784]: osdmap e300: 12 total, 12 up, 11 in 2024-05-15T23:15:37.469 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:37 smithi090 ceph-mon[44434]: pgmap v828: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-05-15T23:15:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:37 smithi090 ceph-mon[44434]: osdmap e300: 12 total, 12 up, 11 in 2024-05-15T23:15:37.597 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:15:37.635 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:15:37.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:37.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:15:37.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:37.799 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:37.894 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:37.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:38.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:15:38.037 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:38.072 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:15:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:38.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:15:38.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:15:38.218 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:38.253 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:15:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:38.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:15:38.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 32 16 93539 93523 182.634 114 0.00555378 0.00547379 2024-05-15T23:15:38.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:38.394 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:39.153 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:39.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:39.192 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T23:15:39.263 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:15:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:39.317 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:15:39.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:39.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:15:39.344 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 33 16 95450 95434 180.719 119.438 0.00449803 0.00553235 2024-05-15T23:15:39.423 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:15:39.474 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:40.083 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:39 smithi084 ceph-mon[39784]: pgmap v830: 17 pgs: 17 active+clean; 5.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-05-15T23:15:40.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:39 smithi090 ceph-mon[44434]: pgmap v830: 17 pgs: 17 active+clean; 5.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-05-15T23:15:40.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:39 smithi088 ceph-mon[44699]: pgmap v830: 17 pgs: 17 active+clean; 5.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-05-15T23:15:40.344 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 34 16 97282 97266 178.771 114.5 0.0143024 0.00559238 2024-05-15T23:15:41.133 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:41.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 35 16 99084 99068 176.881 112.625 0.00398734 0.00565221 2024-05-15T23:15:41.364 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:15:41.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:41.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:15:41.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:41.523 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:41.728 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:41.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:41.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:41.845 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:15:41.887 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:41.911 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:15:41.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:41.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:15:41.958 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:15:42.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:15:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:42.092 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:15:42.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:42.096 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:15:42.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:15:42.199 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:15:42.202 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:15:42.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:41 smithi084 ceph-mon[39784]: pgmap v831: 17 pgs: 17 active+clean; 5.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-05-15T23:15:42.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:41 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1087875966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:15:42.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:41 smithi084 ceph-mon[39784]: from='client.25477 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:15:42.233 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:41 smithi090 ceph-mon[44434]: pgmap v831: 17 pgs: 17 active+clean; 5.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-05-15T23:15:42.234 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:41 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1087875966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:15:42.234 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:41 smithi090 ceph-mon[44434]: from='client.25477 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:15:42.236 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:42.264 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:15:42.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:42.271 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:15:42.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:41 smithi088 ceph-mon[44699]: pgmap v831: 17 pgs: 17 active+clean; 5.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-05-15T23:15:42.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:41 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1087875966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:15:42.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:41 smithi088 ceph-mon[44699]: from='client.25477 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:15:42.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 36 16 100735 100719 174.834 103.188 0.0106693 0.00571835 2024-05-15T23:15:42.372 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:42.425 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:42.497 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:42.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:42.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:42.604 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:15:42.638 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:42.670 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:15:42.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:42.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:15:42.772 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:15:42.808 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:42.837 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:15:42.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:42.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:15:42.943 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:15:42.952 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:42 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:15:42.952 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:42 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:15:42.952 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:42 smithi088 ceph-mon[44699]: from='client.25477 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:15:42.952 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:42 smithi088 ceph-mon[44699]: osdmap e301: 12 total, 12 up, 11 in 2024-05-15T23:15:42.952 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:42 smithi088 ceph-mon[44699]: from='client.15573 v1:172.21.15.84:0/661185271' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:15:42.952 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:42 smithi088 ceph-mon[44699]: pgmap v833: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 129 MiB/s wr, 2.07k op/s 2024-05-15T23:15:42.975 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:43.002 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:15:43.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:43.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:15:43.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:15:43.145 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:43.167 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:15:43.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:43.170 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:15:43.271 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:15:43.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:42 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:15:43.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:42 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:15:43.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:42 smithi090 ceph-mon[44434]: from='client.25477 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:15:43.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:42 smithi090 ceph-mon[44434]: osdmap e301: 12 total, 12 up, 11 in 2024-05-15T23:15:43.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:42 smithi090 ceph-mon[44434]: from='client.15573 v1:172.21.15.84:0/661185271' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:15:43.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:42 smithi090 ceph-mon[44434]: pgmap v833: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 129 MiB/s wr, 2.07k op/s 2024-05-15T23:15:43.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:42 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:15:43.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:42 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:15:43.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:42 smithi084 ceph-mon[39784]: from='client.25477 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:15:43.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:42 smithi084 ceph-mon[39784]: osdmap e301: 12 total, 12 up, 11 in 2024-05-15T23:15:43.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:42 smithi084 ceph-mon[39784]: from='client.15573 v1:172.21.15.84:0/661185271' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:15:43.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:42 smithi084 ceph-mon[39784]: pgmap v833: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 129 MiB/s wr, 2.07k op/s 2024-05-15T23:15:43.318 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:43.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 37 16 102547 102531 173.17 113.25 0.0154198 0.00577335 2024-05-15T23:15:43.399 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:15:43.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:43.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:15:43.447 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":832,"stamp":"2024-05-15T23:15:41.934001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6224544288,"num_objects":94972,"num_object_clones":0,"num_object_copies":189944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94972,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":95093,"num_write_kb":6080230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95064,"ondisk_log_size":95064,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":14537604,"kb_used_data":12226084,"kb_used_omap":17,"kb_used_meta":2311406,"kb_avail":1016659068,"statfs":{"total":1055945392128,"available":1041058885632,"internally_reserved":0,"allocated":12519510016,"data_stored":12514319960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":2366880629},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,8,3,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1037369344,"num_objects":15829,"num_object_clones":0,"num_object_copies":31658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15829,"num_write_kb":1013056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.798864"},"pg_stats":[{"pgid":"4.f","version":"300'5915","reported_seq":5957,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:40.131060+0000","last_change":"2024-05-15T23:14:58.959549+0000","last_active":"2024-05-15T23:15:40.131060+0000","last_peered":"2024-05-15T23:15:40.131060+0000","last_clean":"2024-05-15T23:15:40.131060+0000","last_became_active":"2024-05-15T23:14:58.959283+0000","last_became_peered":"2024-05-15T23:14:58.959283+0000","last_unstale":"2024-05-15T23:15:40.131060+0000","last_undegraded":"2024-05-15T23:15:40.131060+0000","last_fullsized":"2024-05-15T23:15:40.131060+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":5915,"log_dups_size":0,"ondisk_log_size":5915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:16:21.531737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387645440,"num_objects":5915,"num_object_clones":0,"num_object_copies":11830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5915,"num_write_kb":378560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"300'6143","reported_seq":6185,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:39.604522+0000","last_change":"2024-05-15T23:14:58.955607+0000","last_active":"2024-05-15T23:15:39.604522+0000","last_peered":"2024-05-15T23:15:39.604522+0000","last_clean":"2024-05-15T23:15:39.604522+0000","last_became_active":"2024-05-15T23:14:58.955357+0000","last_became_peered":"2024-05-15T23:14:58.955357+0000","last_unstale":"2024-05-15T23:15:39.604522+0000","last_undegraded":"2024-05-15T23:15:39.604522+0000","last_fullsized":"2024-05-15T23:15:39.604522+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":6143,"log_dups_size":0,"ondisk_log_size":6143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:16:21.313497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402587648,"num_objects":6143,"num_object_clones":0,"num_object_copies":12286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6143,"num_write_kb":393152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"300'5963","reported_seq":6006,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:39.579878+0000","last_change":"2024-05-15T23:14:58.955743+0000","last_active":"2024-05-15T23:15:39.579878+0000","last_peered":"2024-05-15T23:15:39.579878+0000","last_clean":"2024-05-15T23:15:39.579878+0000","last_became_active":"2024-05-15T23:14:58.955461+0000","last_became_peered":"2024-05-15T23:14:58.955461+0000","last_unstale":"2024-05-15T23:15:39.579878+0000","last_undegraded":"2024-05-15T23:15:39.579878+0000","last_fullsized":"2024-05-15T23:15:39.579878+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":5963,"log_dups_size":0,"ondisk_log_size":5963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:16:00.049985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390791168,"num_objects":5963,"num_object_clones":0,"num_object_copies":11926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5963,"num_write_kb":381632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"300'5910","reported_seq":5952,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:39.595839+0000","last_change":"2024-05-15T23:14:58.956558+0000","last_active":"2024-05-15T23:15:39.595839+0000","last_peered":"2024-05-15T23:15:39.595839+0000","last_clean":"2024-05-15T23:15:39.595839+0000","last_became_active":"2024-05-15T23:14:58.956216+0000","last_became_peered":"2024-05-15T23:14:58.956216+0000","last_unstale":"2024-05-15T23:15:39.595839+0000","last_undegraded":"2024-05-15T23:15:39.595839+0000","last_fullsized":"2024-05-15T23:15:39.595839+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":5910,"log_dups_size":0,"ondisk_log_size":5910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:16:18.992222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387317760,"num_objects":5910,"num_object_clones":0,"num_object_copies":11820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5910,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"300'5980","reported_seq":6023,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:39.345439+0000","last_change":"2024-05-15T23:14:58.959790+0000","last_active":"2024-05-15T23:15:39.345439+0000","last_peered":"2024-05-15T23:15:39.345439+0000","last_clean":"2024-05-15T23:15:39.345439+0000","last_became_active":"2024-05-15T23:14:58.959506+0000","last_became_peered":"2024-05-15T23:14:58.959506+0000","last_unstale":"2024-05-15T23:15:39.345439+0000","last_undegraded":"2024-05-15T23:15:39.345439+0000","last_fullsized":"2024-05-15T23:15:39.345439+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":5980,"log_dups_size":0,"ondisk_log_size":5980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:16:06.132179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391905280,"num_objects":5980,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5980,"num_write_kb":382720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"300'5823","reported_seq":5866,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:37.572477+0000","last_change":"2024-05-15T23:14:58.959996+0000","last_active":"2024-05-15T23:15:37.572477+0000","last_peered":"2024-05-15T23:15:37.572477+0000","last_clean":"2024-05-15T23:15:37.572477+0000","last_became_active":"2024-05-15T23:14:58.959781+0000","last_became_peered":"2024-05-15T23:14:58.959781+0000","last_unstale":"2024-05-15T23:15:37.572477+0000","last_undegraded":"2024-05-15T23:15:37.572477+0000","last_fullsized":"2024-05-15T23:15:37.572477+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":5823,"log_dups_size":0,"ondisk_log_size":5823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:16:10.002072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381616128,"num_objects":5823,"num_object_clones":0,"num_object_copies":11646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5823,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"300'5769","reported_seq":5812,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:37.598623+0000","last_change":"2024-05-15T23:14:58.958991+0000","last_active":"2024-05-15T23:15:37.598623+0000","last_peered":"2024-05-15T23:15:37.598623+0000","last_clean":"2024-05-15T23:15:37.598623+0000","last_became_active":"2024-05-15T23:14:58.958728+0000","last_became_peered":"2024-05-15T23:14:58.958728+0000","last_unstale":"2024-05-15T23:15:37.598623+0000","last_undegraded":"2024-05-15T23:15:37.598623+0000","last_fullsized":"2024-05-15T23:15:37.598623+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":5769,"log_dups_size":0,"ondisk_log_size":5769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:16:12.573741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378077184,"num_objects":5769,"num_object_clones":0,"num_object_copies":11538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5769,"num_write_kb":369216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"300'5759","reported_seq":5801,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:37.581749+0000","last_change":"2024-05-15T23:14:59.324305+0000","last_active":"2024-05-15T23:15:37.581749+0000","last_peered":"2024-05-15T23:15:37.581749+0000","last_clean":"2024-05-15T23:15:37.581749+0000","last_became_active":"2024-05-15T23:14:59.324101+0000","last_became_peered":"2024-05-15T23:14:59.324101+0000","last_unstale":"2024-05-15T23:15:37.581749+0000","last_undegraded":"2024-05-15T23:15:37.581749+0000","last_fullsized":"2024-05-15T23:15:37.581749+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":5759,"log_dups_size":0,"ondisk_log_size":5759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:16:27.584425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377421824,"num_objects":5759,"num_object_clones":0,"num_object_copies":11518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5759,"num_write_kb":368576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"300'6163","reported_seq":6202,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:40.134309+0000","last_change":"2024-05-15T23:14:58.959011+0000","last_active":"2024-05-15T23:15:40.134309+0000","last_peered":"2024-05-15T23:15:40.134309+0000","last_clean":"2024-05-15T23:15:40.134309+0000","last_became_active":"2024-05-15T23:14:58.958730+0000","last_became_peered":"2024-05-15T23:14:58.958730+0000","last_unstale":"2024-05-15T23:15:40.134309+0000","last_undegraded":"2024-05-15T23:15:40.134309+0000","last_fullsized":"2024-05-15T23:15:40.134309+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":6163,"log_dups_size":0,"ondisk_log_size":6163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:16:21.568740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403898368,"num_objects":6163,"num_object_clones":0,"num_object_copies":12326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":394432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"300'6047","reported_seq":6089,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:40.316583+0000","last_change":"2024-05-15T23:14:58.958921+0000","last_active":"2024-05-15T23:15:40.316583+0000","last_peered":"2024-05-15T23:15:40.316583+0000","last_clean":"2024-05-15T23:15:40.316583+0000","last_became_active":"2024-05-15T23:14:58.958613+0000","last_became_peered":"2024-05-15T23:14:58.958613+0000","last_unstale":"2024-05-15T23:15:40.316583+0000","last_undegraded":"2024-05-15T23:15:40.316583+0000","last_fullsized":"2024-05-15T23:15:40.316583+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":6047,"log_dups_size":0,"ondisk_log_size":6047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:15:58.850012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396296192,"num_objects":6047,"num_object_clones":0,"num_object_copies":12094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6047,"num_write_kb":387008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"300'6041","reported_seq":6084,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:40.140490+0000","last_change":"2024-05-15T23:14:58.959798+0000","last_active":"2024-05-15T23:15:40.140490+0000","last_peered":"2024-05-15T23:15:40.140490+0000","last_clean":"2024-05-15T23:15:40.140490+0000","last_became_active":"2024-05-15T23:14:58.959588+0000","last_became_peered":"2024-05-15T23:14:58.959588+0000","last_unstale":"2024-05-15T23:15:40.140490+0000","last_undegraded":"2024-05-15T23:15:40.140490+0000","last_fullsized":"2024-05-15T23:15:40.140490+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":6041,"log_dups_size":0,"ondisk_log_size":6041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:16:05.393623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395902976,"num_objects":6041,"num_object_clones":0,"num_object_copies":12082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6041,"num_write_kb":386624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"300'5851","reported_seq":5894,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:37.589112+0000","last_change":"2024-05-15T23:14:59.324389+0000","last_active":"2024-05-15T23:15:37.589112+0000","last_peered":"2024-05-15T23:15:37.589112+0000","last_clean":"2024-05-15T23:15:37.589112+0000","last_became_active":"2024-05-15T23:14:59.324138+0000","last_became_peered":"2024-05-15T23:14:59.324138+0000","last_unstale":"2024-05-15T23:15:37.589112+0000","last_undegraded":"2024-05-15T23:15:37.589112+0000","last_fullsized":"2024-05-15T23:15:37.589112+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":5851,"log_dups_size":0,"ondisk_log_size":5851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:16:19.864313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383451136,"num_objects":5851,"num_object_clones":0,"num_object_copies":11702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5851,"num_write_kb":374464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"300'5931","reported_seq":5974,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:40.310524+0000","last_change":"2024-05-15T23:14:59.080024+0000","last_active":"2024-05-15T23:15:40.310524+0000","last_peered":"2024-05-15T23:15:40.310524+0000","last_clean":"2024-05-15T23:15:40.310524+0000","last_became_active":"2024-05-15T23:14:59.079701+0000","last_became_peered":"2024-05-15T23:14:59.079701+0000","last_unstale":"2024-05-15T23:15:40.310524+0000","last_undegraded":"2024-05-15T23:15:40.310524+0000","last_fullsized":"2024-05-15T23:15:40.310524+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":5931,"log_dups_size":0,"ondisk_log_size":5931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:16:10.142413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388694016,"num_objects":5931,"num_object_clones":0,"num_object_copies":11862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5931,"num_write_kb":379584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"300'5723","reported_seq":5766,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:37.606940+0000","last_change":"2024-05-15T23:14:59.079624+0000","last_active":"2024-05-15T23:15:37.606940+0000","last_peered":"2024-05-15T23:15:37.606940+0000","last_clean":"2024-05-15T23:15:37.606940+0000","last_became_active":"2024-05-15T23:14:59.079345+0000","last_became_peered":"2024-05-15T23:14:59.079345+0000","last_unstale":"2024-05-15T23:15:37.606940+0000","last_undegraded":"2024-05-15T23:15:37.606940+0000","last_fullsized":"2024-05-15T23:15:37.606940+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":5723,"log_dups_size":0,"ondisk_log_size":5723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:16:14.739021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375062528,"num_objects":5723,"num_object_clones":0,"num_object_copies":11446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5723,"num_write_kb":366272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"300'6093","reported_seq":6136,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:40.538917+0000","last_change":"2024-05-15T23:14:59.324647+0000","last_active":"2024-05-15T23:15:40.538917+0000","last_peered":"2024-05-15T23:15:40.538917+0000","last_clean":"2024-05-15T23:15:40.538917+0000","last_became_active":"2024-05-15T23:14:59.324368+0000","last_became_peered":"2024-05-15T23:14:59.324368+0000","last_unstale":"2024-05-15T23:15:40.538917+0000","last_undegraded":"2024-05-15T23:15:40.538917+0000","last_fullsized":"2024-05-15T23:15:40.538917+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":6093,"log_dups_size":0,"ondisk_log_size":6093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:16:10.053890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399310848,"num_objects":6093,"num_object_clones":0,"num_object_copies":12186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6093,"num_write_kb":389952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"300'5859","reported_seq":5902,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:37.591405+0000","last_change":"2024-05-15T23:14:58.959100+0000","last_active":"2024-05-15T23:15:37.591405+0000","last_peered":"2024-05-15T23:15:37.591405+0000","last_clean":"2024-05-15T23:15:37.591405+0000","last_became_active":"2024-05-15T23:14:58.958832+0000","last_became_peered":"2024-05-15T23:14:58.958832+0000","last_unstale":"2024-05-15T23:15:37.591405+0000","last_undegraded":"2024-05-15T23:15:37.591405+0000","last_fullsized":"2024-05-15T23:15:37.591405+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:14:57.936933+0000","objects_scrubbed":0,"log_size":5859,"log_dups_size":0,"ondisk_log_size":5859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:15:58.721356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383975424,"num_objects":5859,"num_object_clones":0,"num_object_copies":11718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5859,"num_write_kb":374976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":612,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-05-15T23:15:36.904584+0000","last_change":"2024-05-15T23:15:34.313970+0000","last_active":"2024-05-15T23:15:36.904584+0000","last_peered":"2024-05-15T23:15:36.904584+0000","last_clean":"2024-05-15T23:15:36.904584+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:15:36.904584+0000","last_undegraded":"2024-05-15T23:15:36.904584+0000","last_fullsized":"2024-05-15T23:15:36.904584+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:15:34.313846+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:15:34.313846+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:16:56.153020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":6223953920,"num_objects":94970,"num_object_clones":0,"num_object_copies":189940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94970,"num_write_kb":6078080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12611682304,"data_stored":12611682304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94970,"ondisk_log_size":94970,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351118,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1806756,"kb_used_data":1454876,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91938396,"statfs":{"total":95995035648,"available":94144917504,"internally_reserved":0,"allocated":1489793024,"data_stored":1489323780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":360315332},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580052,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1284264,"kb_used_data":1180896,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92460888,"statfs":{"total":95995035648,"available":94679949312,"internally_reserved":0,"allocated":1209237504,"data_stored":1208765778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":105839040},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776284,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1368232,"kb_used_data":1152416,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92376920,"statfs":{"total":95995035648,"available":94593966080,"internally_reserved":0,"allocated":1180073984,"data_stored":1179602258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":220985792},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005216,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":144556,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93600596,"statfs":{"total":95995035648,"available":95847010304,"internally_reserved":0,"allocated":950272,"data_stored":478546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1314032,"kb_used_data":1148200,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92431120,"statfs":{"total":95995035648,"available":94649466880,"internally_reserved":0,"allocated":1175756800,"data_stored":1175281522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":169802186},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495786,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":837740,"kb_used_data":738528,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92907412,"statfs":{"total":95995035648,"available":95137189888,"internally_reserved":0,"allocated":756252672,"data_stored":755780946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575044,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2973096,"kb_used_data":2698976,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":90772056,"statfs":{"total":95995035648,"available":92950585344,"internally_reserved":0,"allocated":2763751424,"data_stored":2763279698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280689094},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738569,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":655464,"kb_used_data":381344,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93089688,"statfs":{"total":95995035648,"available":95323840512,"internally_reserved":0,"allocated":390496256,"data_stored":390024530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483142,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149882,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1836776,"kb_used_data":1548768,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91908376,"statfs":{"total":95995035648,"available":94114177024,"internally_reserved":0,"allocated":1585938432,"data_stored":1585466706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":294910407},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986355,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1364648,"kb_used_data":1144672,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92380504,"statfs":{"total":95995035648,"available":94597636096,"internally_reserved":0,"allocated":1172144128,"data_stored":1171672402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790126,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":952040,"kb_used_data":776480,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92793112,"statfs":{"total":95995035648,"available":95020146688,"internally_reserved":0,"allocated":795115520,"data_stored":794643794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":391970816,"data_stored":391970816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2776498176,"data_stored":2776498176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1585250304,"data_stored":1585250304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1180696576,"data_stored":1180696576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":807862272,"data_stored":807862272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":765591552,"data_stored":765591552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1191116800,"data_stored":1191116800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1181548544,"data_stored":1181548544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1217003520,"data_stored":1217003520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1514143744,"data_stored":1514143744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:15:43.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:43.555 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:43.810 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:43.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:43.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:43.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:15:43.955 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:43.980 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:15:43.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:43.983 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:15:44.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:15:44.127 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:44.156 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:15:44.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:44.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:15:44.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:44.288 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:44.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 38 16 104243 104227 171.402 106 0.00780775 0.00583286 2024-05-15T23:15:45.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:44 smithi090 ceph-mon[44434]: pgmap v834: 17 pgs: 17 active+clean; 6.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-05-15T23:15:45.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:44 smithi088 ceph-mon[44699]: pgmap v834: 17 pgs: 17 active+clean; 6.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-05-15T23:15:45.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 39 16 105907 105891 169.673 104 0.00768158 0.00589254 2024-05-15T23:15:45.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:44 smithi084 ceph-mon[39784]: pgmap v834: 17 pgs: 17 active+clean; 6.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-05-15T23:15:45.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:15:46.112 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:46.121 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:46.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:46.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:46.223 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:46.250 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:46.255 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:46.282 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:46.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:46.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:46.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:15:46.344942 min lat: 0.00102037 max lat: 0.0761315 avg lat: 0.0059536 2024-05-15T23:15:46.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:15:46.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 40 16 107488 107472 167.902 98.8125 0.00587673 0.0059536 2024-05-15T23:15:46.386 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:15:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:46.461 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:15:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:46.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:15:46.568 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:15:46.603 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:46.632 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:15:46.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:46.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:15:46.737 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:46.784 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:47.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 41 16 109127 109111 166.305 102.438 0.0102203 0.00601185 2024-05-15T23:15:48.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:47 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:15:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:47 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:15:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:47 smithi090 ceph-mon[44434]: osdmap e302: 12 total, 12 up, 11 in 2024-05-15T23:15:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:47 smithi090 ceph-mon[44434]: pgmap v836: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-05-15T23:15:48.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:47 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:15:48.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:47 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:15:48.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:47 smithi088 ceph-mon[44699]: osdmap e302: 12 total, 12 up, 11 in 2024-05-15T23:15:48.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:47 smithi088 ceph-mon[44699]: pgmap v836: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-05-15T23:15:48.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:47 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:15:48.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:47 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:15:48.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:47 smithi084 ceph-mon[39784]: osdmap e302: 12 total, 12 up, 11 in 2024-05-15T23:15:48.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:47 smithi084 ceph-mon[39784]: pgmap v836: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-05-15T23:15:48.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 42 16 110761 110745 164.777 102.125 0.0150778 0.00606709 2024-05-15T23:15:48.361 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:48.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:48.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:48.450 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 11, 9, 10] out_osds: [2] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-05-15T23:15:48.451 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 0 1 2024-05-15T23:15:48.472 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:15:48.540 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:49.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:48 smithi084 ceph-mon[39784]: pgmap v837: 17 pgs: 17 active+clean; 6.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-05-15T23:15:49.139 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:15:49.139 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:15:49.139 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:15:49.139 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:15:49.139 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:15:49.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:48 smithi090 ceph-mon[44434]: pgmap v837: 17 pgs: 17 active+clean; 6.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-05-15T23:15:49.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:48 smithi088 ceph-mon[44699]: pgmap v837: 17 pgs: 17 active+clean; 6.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-05-15T23:15:49.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 43 16 112342 112326 163.243 98.8125 0.00889847 0.00612453 2024-05-15T23:15:50.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 44 15 114072 114057 161.991 108.188 0.0119035 0.00617213 2024-05-15T23:15:50.955 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:15:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:50.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:15:51.059 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:15:51.090 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:51.118 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:15:51.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:51.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:15:51.222 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:15:51.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:50 smithi084 ceph-mon[39784]: pgmap v838: 17 pgs: 17 active+clean; 6.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-05-15T23:15:51.261 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:50 smithi088 ceph-mon[44699]: pgmap v838: 17 pgs: 17 active+clean; 6.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-05-15T23:15:51.263 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:51.296 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:15:51.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:50 smithi090 ceph-mon[44434]: pgmap v838: 17 pgs: 17 active+clean; 6.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-05-15T23:15:51.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:51.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:15:51.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 45 16 115701 115685 160.652 101.75 0.00702139 0.00622336 2024-05-15T23:15:51.404 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:51.439 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:51.462 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:51.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:51.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:51.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:51.631 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:51.696 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:51.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:51.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:51.717 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:15:51.761 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:15:51.788 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:15:51.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:51.818 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:15:51.849 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:15:51.857 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:51.886 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:15:51.918 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:15:51.949 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3698665725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:15:51.949 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:51.952 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:15:51.982 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:15:52.009 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:15:52.036 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:15:52.069 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:15:52.070 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:15:52.070 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:15:52.070 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:15:52.070 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:15:52.070 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:15:52.070 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:15:52.070 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:15:52.070 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:15:52.070 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:15:52.071 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:15:52.071 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:15:52.071 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:15:52.249 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T23:15:52.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3698665725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:15:52.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3698665725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:15:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:52.346 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 46 16 117244 117228 159.256 96.4375 0.00452415 0.0062778 2024-05-15T23:15:52.925 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:52.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:52.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:53.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:15:53.066 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:53.092 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:15:53.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:53.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:15:53.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:52 smithi084 ceph-mon[39784]: pgmap v839: 17 pgs: 17 active+clean; 7.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T23:15:53.197 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:15:53.226 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.226 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.227 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.227 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.227 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.227 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.227 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.227 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.227 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.227 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.227 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.227 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:52 smithi088 ceph-mon[44699]: pgmap v839: 17 pgs: 17 active+clean; 7.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T23:15:53.230 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:53.261 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:15:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:53.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:15:53.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:15:53.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:52 smithi090 ceph-mon[44434]: pgmap v839: 17 pgs: 17 active+clean; 7.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T23:15:53.346 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 47 16 118795 118779 157.93 96.9375 0.0109292 0.00633038 2024-05-15T23:15:53.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:15:53.418 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:53.448 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:15:53.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:53.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:15:53.508 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:53.562 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:53.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:15:53.622 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:53.979 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:15:54.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:53 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1976516896' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:15:54.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:53 smithi084 ceph-mon[39784]: from='client.35591 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:15:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:53 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1976516896' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:15:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:53 smithi090 ceph-mon[44434]: from='client.35591 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:15:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:53 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1976516896' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:15:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:53 smithi088 ceph-mon[44699]: from='client.35591 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:15:54.345 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 48 16 120320 120304 156.625 95.3125 0.0301327 0.0063836 2024-05-15T23:15:54.581 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:15:54.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:54.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:15:54.689 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:54.731 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:54.762 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:54.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:54.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:54.866 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:15:54.902 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:54.926 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:15:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:54.930 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:15:55.031 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:55.070 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:55.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:54 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:15:55.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:54 smithi088 ceph-mon[44699]: from='client.35591 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:15:55.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:54 smithi088 ceph-mon[44699]: osdmap e303: 12 total, 12 up, 11 in 2024-05-15T23:15:55.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:54 smithi088 ceph-mon[44699]: pgmap v841: 17 pgs: 17 active+clean; 7.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-05-15T23:15:55.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:54 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:15:55.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:54 smithi090 ceph-mon[44434]: from='client.35591 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:15:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:54 smithi090 ceph-mon[44434]: osdmap e303: 12 total, 12 up, 11 in 2024-05-15T23:15:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:54 smithi090 ceph-mon[44434]: pgmap v841: 17 pgs: 17 active+clean; 7.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-05-15T23:15:55.346 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 49 16 121909 121893 155.456 99.3125 0.00444643 0.00643108 2024-05-15T23:15:55.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:54 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:15:55.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:54 smithi084 ceph-mon[39784]: from='client.35591 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:15:55.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:54 smithi084 ceph-mon[39784]: osdmap e303: 12 total, 12 up, 11 in 2024-05-15T23:15:55.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:54 smithi084 ceph-mon[39784]: pgmap v841: 17 pgs: 17 active+clean; 7.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-05-15T23:15:55.464 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:15:55.533 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:55.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:55.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:55.646 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:55.698 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:55.740 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:55.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:55.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:55.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:55.894 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:56.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:56 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:15:56.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:56 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:15:56.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:56 smithi090 ceph-mon[44434]: osdmap e304: 12 total, 12 up, 11 in 2024-05-15T23:15:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:56 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:15:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:56 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:15:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:56 smithi088 ceph-mon[44699]: osdmap e304: 12 total, 12 up, 11 in 2024-05-15T23:15:56.346 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 50 16 123390 123374 154.198 92.5625 0.00796525 0.00648383 2024-05-15T23:15:56.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:56 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:15:56.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:56 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:15:56.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:56 smithi084 ceph-mon[39784]: osdmap e304: 12 total, 12 up, 11 in 2024-05-15T23:15:56.981 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:56.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:56.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:57.089 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:15:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:57.144 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:15:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:57.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:15:57.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:15:57.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:57 smithi084 ceph-mon[39784]: pgmap v843: 17 pgs: 17 active+clean; 7.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-05-15T23:15:57.299 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:57.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:57 smithi090 ceph-mon[44434]: pgmap v843: 17 pgs: 17 active+clean; 7.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-05-15T23:15:57.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:57 smithi088 ceph-mon[44699]: pgmap v843: 17 pgs: 17 active+clean; 7.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-05-15T23:15:57.346 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 51 16 124927 124911 153.058 96.0625 0.00559476 0.00653226 2024-05-15T23:15:57.715 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:15:57.715 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:15:57.977 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:15:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:57.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:15:57.995 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:15:58.046 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2024-05-15T23:15:58.086 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:58.125 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:58.149 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:58.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:58.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:58.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:15:58.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:58 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1123763962' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:15:58.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:58 smithi084 ceph-mon[39784]: from='client.25495 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:15:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:58 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1123763962' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:15:58.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:58 smithi090 ceph-mon[44434]: from='client.25495 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:15:58.303 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:58 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1123763962' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:15:58.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:58 smithi088 ceph-mon[44699]: from='client.25495 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:15:58.346 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 52 16 126320 126304 151.788 87.0625 0.00973147 0.00658684 2024-05-15T23:15:58.363 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:15:58.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:58.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:15:58.468 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:15:58.501 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:58.528 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:15:58.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:58.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:15:58.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:15:58.682 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:58.775 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:15:58.797 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:15:58.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:58.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:15:58.847 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T23:15:58.905 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:58.935 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:58.963 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:58.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:58.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:15:59.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:15:59.143 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:59.284 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:15:59.286 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:59 smithi084 ceph-mon[39784]: from='client.25495 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T23:15:59.286 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:59 smithi084 ceph-mon[39784]: osdmap e305: 12 total, 12 up, 11 in 2024-05-15T23:15:59.286 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:15:59 smithi084 ceph-mon[39784]: pgmap v845: 17 pgs: 17 active+clean; 7.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-05-15T23:15:59.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:59.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:15:59.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:59 smithi090 ceph-mon[44434]: from='client.25495 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T23:15:59.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:59 smithi090 ceph-mon[44434]: osdmap e305: 12 total, 12 up, 11 in 2024-05-15T23:15:59.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:15:59 smithi090 ceph-mon[44434]: pgmap v845: 17 pgs: 17 active+clean; 7.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-05-15T23:15:59.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:59 smithi088 ceph-mon[44699]: from='client.25495 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T23:15:59.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:59 smithi088 ceph-mon[44699]: osdmap e305: 12 total, 12 up, 11 in 2024-05-15T23:15:59.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:15:59 smithi088 ceph-mon[44699]: pgmap v845: 17 pgs: 17 active+clean; 7.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-05-15T23:15:59.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 1 1 2024-05-15T23:15:59.346 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 53 16 127734 127718 150.592 88.375 0.00386227 0.00663864 2024-05-15T23:15:59.389 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:15:59.419 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:15:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:59.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:15:59.552 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:15:59.585 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:59.609 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:15:59.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:59.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:15:59.715 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:15:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:59.779 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:15:59.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:59.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:15:59.887 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:15:59.921 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:15:59.950 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:15:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:15:59.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:16:00.060 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:16:00.095 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:00.185 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:00.186 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:16:00.186 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:00.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:00.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:16:00.235 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:00.292 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:16:00.338 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:00.346 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 54 16 129079 129063 149.36 84.0625 0.0242835 0.00669362 2024-05-15T23:16:01.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:00 smithi084 ceph-mon[39784]: pgmap v846: 17 pgs: 17 active+clean; 7.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-05-15T23:16:01.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:00 smithi090 ceph-mon[44434]: pgmap v846: 17 pgs: 17 active+clean; 7.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-05-15T23:16:01.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:00 smithi088 ceph-mon[44699]: pgmap v846: 17 pgs: 17 active+clean; 7.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-05-15T23:16:01.346 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 55 16 130534 130518 148.297 90.9375 0.00941817 0.00674161 2024-05-15T23:16:01.634 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:16:01.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:01.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:16:01.744 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:16:01.777 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:01.806 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:16:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:01.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:16:01.915 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:16:01.955 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:01.984 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:16:01.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:01.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:16:02.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:16:02.144 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:02.346 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 56 16 131981 131965 147.264 90.4375 0.00974241 0.00678883 2024-05-15T23:16:02.825 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:16:02.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:02.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:16:02.936 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:16:02.976 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:03.007 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:16:03.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:03.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:16:03.111 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:02 smithi084 ceph-mon[39784]: osdmap e306: 12 total, 12 up, 11 in 2024-05-15T23:16:03.111 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:02 smithi084 ceph-mon[39784]: pgmap v848: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-05-15T23:16:03.114 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:16:03.143 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:03.166 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:16:03.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:03.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:16:03.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:02 smithi088 ceph-mon[44699]: osdmap e306: 12 total, 12 up, 11 in 2024-05-15T23:16:03.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:02 smithi088 ceph-mon[44699]: pgmap v848: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-05-15T23:16:03.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:16:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:02 smithi090 ceph-mon[44434]: osdmap e306: 12 total, 12 up, 11 in 2024-05-15T23:16:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:02 smithi090 ceph-mon[44434]: pgmap v848: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-05-15T23:16:03.305 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:03.331 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:16:03.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:03.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:16:03.346 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 57 16 133420 133404 146.258 89.9375 0.00659776 0.00683587 2024-05-15T23:16:03.436 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:16:03.463 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:03.487 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:16:03.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:03.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:16:03.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:16:03.644 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:03.712 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:16:03.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:03.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:16:03.806 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:16:03.806 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:16:03.806 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:16:03.806 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:16:03.806 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:16:03.806 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:16:03.806 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:16:03.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:16:03.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:16:03.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:16:03.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:16:03.818 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:16:03.819 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:16:03.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:16:03.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:16:03.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:16:03.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:16:03.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:16:03.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:16:03.826 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:16:03.826 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:16:03.826 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:16:03.826 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:16:03.826 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:16:03.826 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:16:03.854 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:03.882 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:16:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:03.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:16:03.988 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:16:04.032 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:04.052 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:16:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:04.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:16:04.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:16:04.229 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/383227862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:04.229 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:04.229 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:04.229 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:04.229 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:04.231 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:04.294 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:16:04.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:04.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:16:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/383227862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/383227862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:04.348 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 58 16 134721 134705 145.136 81.3125 0.0112246 0.00688877 2024-05-15T23:16:04.397 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:16:04.453 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:05.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:04 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:16:05.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:04 smithi084 ceph-mon[39784]: pgmap v849: 17 pgs: 17 active+clean; 8.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 92 MiB/s wr, 1.46k op/s 2024-05-15T23:16:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:04 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:16:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:04 smithi090 ceph-mon[44434]: pgmap v849: 17 pgs: 17 active+clean; 8.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 92 MiB/s wr, 1.46k op/s 2024-05-15T23:16:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:04 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:16:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:04 smithi088 ceph-mon[44699]: pgmap v849: 17 pgs: 17 active+clean; 8.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 92 MiB/s wr, 1.46k op/s 2024-05-15T23:16:05.348 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 59 16 136022 136006 144.054 81.3125 0.0143461 0.00694009 2024-05-15T23:16:05.849 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:16:05.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:05.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:16:05.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:16:05.995 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:06.112 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:16:06.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:06.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:16:06.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:05 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:16:06.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:05 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:16:06.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:05 smithi084 ceph-mon[39784]: osdmap e307: 12 total, 12 up, 11 in 2024-05-15T23:16:06.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:05 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:16:06.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:05 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:16:06.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:05 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:16:06.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:05 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2849744231' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:16:06.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:05 smithi084 ceph-mon[39784]: from='client.35606 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:16:06.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:16:06.258 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:05 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:16:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:05 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:16:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:05 smithi090 ceph-mon[44434]: osdmap e307: 12 total, 12 up, 11 in 2024-05-15T23:16:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:05 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:16:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:05 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:16:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:05 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:16:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:05 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2849744231' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:16:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:05 smithi090 ceph-mon[44434]: from='client.35606 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:16:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:05 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:16:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:05 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:16:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:05 smithi088 ceph-mon[44699]: osdmap e307: 12 total, 12 up, 11 in 2024-05-15T23:16:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:05 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:16:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:05 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:16:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:05 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:16:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:05 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2849744231' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:16:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:05 smithi088 ceph-mon[44699]: from='client.35606 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:16:06.348 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:16:06.347548 min lat: 0.00102037 max lat: 0.215284 avg lat: 0.00698752 2024-05-15T23:16:06.348 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:16:06.348 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 60 16 137384 137368 143.072 85.125 0.00417554 0.00698752 2024-05-15T23:16:06.460 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:16:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:16:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:137513 4.0 4:0f89d55e:::benchmark_data_smithi090_104448_object137512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e308)", 2024-05-15T23:16:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:16:06.450572+0000", 2024-05-15T23:16:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.0059025909999999996, 2024-05-15T23:16:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059279420000000003, 2024-05-15T23:16:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:16:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:16:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:16:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 137513 2024-05-15T23:16:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.450572+0000", 2024-05-15T23:16:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.450572+0000", 2024-05-15T23:16:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.450574+0000", 2024-05-15T23:16:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.616e-06 2024-05-15T23:16:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.450609+0000", 2024-05-15T23:16:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4768000000000003e-05 2024-05-15T23:16:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.450622+0000", 2024-05-15T23:16:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3053e-05 2024-05-15T23:16:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.450630+0000", 2024-05-15T23:16:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4290000000000001e-06 2024-05-15T23:16:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.450707+0000", 2024-05-15T23:16:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7040999999999996e-05 2024-05-15T23:16:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-05-15T23:16:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.450714+0000", 2024-05-15T23:16:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9759999999999998e-06 2024-05-15T23:16:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.452701+0000", 2024-05-15T23:16:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019865239999999999 2024-05-15T23:16:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.452792+0000", 2024-05-15T23:16:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1531000000000002e-05 2024-05-15T23:16:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:16:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.452883+0000", 2024-05-15T23:16:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0283000000000003e-05 2024-05-15T23:16:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.25441.0:137510 4.c e308/283 4:39a681aa:::benchmark_data_smithi090_104448_object137509:head v 308'8597, mlcod=307'8593)", 2024-05-15T23:16:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:16:06.451482+0000", 2024-05-15T23:16:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.004992074, 2024-05-15T23:16:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050744509999999998, 2024-05-15T23:16:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-05-15T23:16:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.451482+0000", 2024-05-15T23:16:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.451482+0000", 2024-05-15T23:16:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.451482+0000", 2024-05-15T23:16:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3799999999999998e-07 2024-05-15T23:16:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.451524+0000", 2024-05-15T23:16:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1737999999999999e-05 2024-05-15T23:16:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.451536+0000", 2024-05-15T23:16:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1822e-05 2024-05-15T23:16:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.451544+0000", 2024-05-15T23:16:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4479999999999994e-06 2024-05-15T23:16:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.451808+0000", 2024-05-15T23:16:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026337000000000002 2024-05-15T23:16:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:16:06.451829+0000", 2024-05-15T23:16:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0996999999999999e-05 2024-05-15T23:16:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:16:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 2 2024-05-15T23:16:06.471 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:16:06.472 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2024-05-15T23:16:06.825 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:16:06.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:06.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:16:06.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:16:06.960 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:06.985 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:16:06.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:06.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:16:07.089 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:16:07.122 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:07.147 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:16:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:07.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:16:07.254 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:16:07.290 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:07.313 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:16:07.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:07.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:16:07.348 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 61 16 138739 138723 142.115 84.6875 0.00959538 0.00703546 2024-05-15T23:16:07.418 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:16:07.451 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:07.484 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:16:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:07.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:16:07.592 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:16:07.628 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:07.656 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:16:07.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:07.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:16:07.760 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:16:07.794 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:07 smithi088 ceph-mon[44699]: from='client.35606 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:16:07.795 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:07 smithi088 ceph-mon[44699]: osdmap e308: 12 total, 12 up, 11 in 2024-05-15T23:16:07.795 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:07 smithi088 ceph-mon[44699]: pgmap v852: 17 pgs: 17 active+clean; 8.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-05-15T23:16:07.796 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:07.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:07 smithi090 ceph-mon[44434]: from='client.35606 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:16:07.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:07 smithi090 ceph-mon[44434]: osdmap e308: 12 total, 12 up, 11 in 2024-05-15T23:16:07.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:07 smithi090 ceph-mon[44434]: pgmap v852: 17 pgs: 17 active+clean; 8.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-05-15T23:16:07.822 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:16:07.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:07.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:16:07.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:16:07.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:07 smithi084 ceph-mon[39784]: from='client.35606 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:16:07.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:07 smithi084 ceph-mon[39784]: osdmap e308: 12 total, 12 up, 11 in 2024-05-15T23:16:07.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:07 smithi084 ceph-mon[39784]: pgmap v852: 17 pgs: 17 active+clean; 8.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-05-15T23:16:07.929 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:16:07.963 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:07.990 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:16:07.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:07.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:16:08.095 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:16:08.103 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 3 1 2024-05-15T23:16:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:08.150 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:16:08.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:08.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:16:08.241 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T23:16:08.258 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:16:08.316 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:08.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 62 16 140017 140001 141.111 79.875 0.0274273 0.00708479 2024-05-15T23:16:08.352 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:08.455 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1832528558' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:16:08.487 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:16:08.657 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:16:08.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:08.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:16:08.724 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1832528558' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:16:08.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:16:08.796 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:08.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1832528558' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:16:09.348 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 63 16 141318 141302 140.162 81.3125 0.0251756 0.00713321 2024-05-15T23:16:09.551 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:16:09.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:09 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:16:09.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:09 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:16:09.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:09 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:16:09.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:09 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1832528558' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:16:09.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:09 smithi084 ceph-mon[39784]: osdmap e309: 12 total, 12 up, 11 in 2024-05-15T23:16:09.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:09 smithi084 ceph-mon[39784]: pgmap v854: 17 pgs: 17 active+clean; 8.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 72 MiB/s wr, 1.14k op/s 2024-05-15T23:16:09.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:09.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:16:09.660 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:16:09.710 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:09.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:09 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:16:09.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:09 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:16:09.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:09 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:16:09.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:09 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1832528558' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:16:09.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:09 smithi090 ceph-mon[44434]: osdmap e309: 12 total, 12 up, 11 in 2024-05-15T23:16:09.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:09 smithi090 ceph-mon[44434]: pgmap v854: 17 pgs: 17 active+clean; 8.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 72 MiB/s wr, 1.14k op/s 2024-05-15T23:16:09.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:09 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:16:09.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:09 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:16:09.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:09 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:16:09.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:09 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1832528558' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:16:09.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:09 smithi088 ceph-mon[44699]: osdmap e309: 12 total, 12 up, 11 in 2024-05-15T23:16:09.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:09 smithi088 ceph-mon[44699]: pgmap v854: 17 pgs: 17 active+clean; 8.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 72 MiB/s wr, 1.14k op/s 2024-05-15T23:16:10.348 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 64 16 142714 142698 139.335 87.25 0.0141479 0.00717585 2024-05-15T23:16:10.719 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:10.785 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:16:10.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:10.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:16:10.894 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:16:10.938 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:11.065 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:16:11.072 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:10 smithi084 ceph-mon[39784]: pgmap v855: 17 pgs: 17 active+clean; 8.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-05-15T23:16:11.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:11.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:16:11.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:16:11.232 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:11.301 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:16:11.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:10 smithi090 ceph-mon[44434]: pgmap v855: 17 pgs: 17 active+clean; 8.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-05-15T23:16:11.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:11.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:16:11.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:10 smithi088 ceph-mon[44699]: pgmap v855: 17 pgs: 17 active+clean; 8.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-05-15T23:16:11.348 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 65 16 144047 144031 138.473 83.3125 0.00944866 0.00722024 2024-05-15T23:16:11.410 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:16:11.456 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:11.915 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:16:11.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:11.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:16:12.019 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:16:12.034 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:16:12.034 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:16:12.034 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:16:12.034 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:16:12.034 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:16:12.035 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:16:12.035 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:16:12.035 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:16:12.035 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:16:12.053 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:16:12.055 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:12.082 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:16:12.083 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:16:12.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:12.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:16:12.112 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:16:12.123 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:16:12.123 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:16:12.123 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:16:12.123 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:16:12.123 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:16:12.124 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:16:12.124 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:16:12.124 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:16:12.124 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:16:12.124 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:16:12.124 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:16:12.124 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:16:12.164 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:16:12.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:11 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2735413910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:12.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:12.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:11 smithi084 ceph-mon[39784]: osdmap e310: 12 total, 12 up, 11 in 2024-05-15T23:16:12.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:16:12.233 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:12.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:11 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2735413910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:12.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:12.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:11 smithi090 ceph-mon[44434]: osdmap e310: 12 total, 12 up, 11 in 2024-05-15T23:16:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:11 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2735413910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:11 smithi088 ceph-mon[44699]: osdmap e310: 12 total, 12 up, 11 in 2024-05-15T23:16:12.348 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 66 16 145315 145299 137.576 79.25 0.00558118 0.00726731 2024-05-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:16:13.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:13.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:16:13.213 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:16:13.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:12 smithi084 ceph-mon[39784]: pgmap v857: 17 pgs: 17 active+clean; 8.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-05-15T23:16:13.267 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:13.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:12 smithi090 ceph-mon[44434]: pgmap v857: 17 pgs: 17 active+clean; 8.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-05-15T23:16:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:13.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:12 smithi088 ceph-mon[44699]: pgmap v857: 17 pgs: 17 active+clean; 8.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-05-15T23:16:13.348 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 67 16 146636 146620 136.755 82.5625 0.0169385 0.00731108 2024-05-15T23:16:13.730 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:16:13.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:13.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:16:13.837 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:16:13.837 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:16:13.839 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:16:13.891 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:14.057 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:16:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:14.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:16:14.167 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:16:14.199 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:14.229 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:16:14.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:14.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:16:14.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:14.348 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 68 16 147997 147981 135.994 85.0625 0.0148108 0.00735194 2024-05-15T23:16:14.376 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:14.404 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:14.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:14.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:14.509 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:16:14.546 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:14.575 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:16:14.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:14.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:16:14.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:14.712 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:14.739 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:14.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:14.743 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:14.845 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:16:14.879 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:14.908 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:16:14.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:14.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:16:15.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:16:15.051 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:15.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:14 smithi090 ceph-mon[44434]: pgmap v858: 17 pgs: 17 active+clean; 8.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-05-15T23:16:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:14 smithi088 ceph-mon[44699]: pgmap v858: 17 pgs: 17 active+clean; 8.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-05-15T23:16:15.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 69 16 149340 149324 135.24 83.9375 0.00642737 0.00739282 2024-05-15T23:16:15.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:14 smithi084 ceph-mon[39784]: pgmap v858: 17 pgs: 17 active+clean; 8.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-05-15T23:16:15.727 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:16:16.351 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 70 15 150640 150625 134.469 81.3125 0.00720494 0.00743528 2024-05-15T23:16:16.645 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:16:16.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:16.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:16:16.701 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:16.702 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T23:16:16.731 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:16.750 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:16:16.813 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:17.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:16 smithi090 ceph-mon[44434]: pgmap v859: 17 pgs: 17 active+clean; 9.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-05-15T23:16:17.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:16 smithi088 ceph-mon[44699]: pgmap v859: 17 pgs: 17 active+clean; 9.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-05-15T23:16:17.348 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 71 16 151805 151789 133.6 72.75 0.00674323 0.00748364 2024-05-15T23:16:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:16 smithi084 ceph-mon[39784]: pgmap v859: 17 pgs: 17 active+clean; 9.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-05-15T23:16:18.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 72 16 153118 153102 132.884 82.0625 0.013125 0.00752413 2024-05-15T23:16:19.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:18 smithi090 ceph-mon[44434]: pgmap v860: 17 pgs: 17 active+clean; 9.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-05-15T23:16:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:18 smithi088 ceph-mon[44699]: pgmap v860: 17 pgs: 17 active+clean; 9.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-05-15T23:16:19.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 73 16 154428 154412 132.185 81.875 0.00888865 0.00756386 2024-05-15T23:16:19.377 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:18 smithi084 ceph-mon[39784]: pgmap v860: 17 pgs: 17 active+clean; 9.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-05-15T23:16:20.014 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:16:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:20.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:16:20.121 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:16:20.170 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:20.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 74 16 155715 155699 131.486 80.4375 0.00984149 0.00760382 2024-05-15T23:16:20.756 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:21.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:20 smithi084 ceph-mon[39784]: pgmap v861: 17 pgs: 17 active+clean; 9.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-05-15T23:16:21.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:20 smithi090 ceph-mon[44434]: pgmap v861: 17 pgs: 17 active+clean; 9.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-05-15T23:16:21.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:20 smithi088 ceph-mon[44699]: pgmap v861: 17 pgs: 17 active+clean; 9.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-05-15T23:16:21.338 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:16:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:21.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:16:21.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 75 16 156996 156980 130.8 80.0625 0.00566517 0.0076432 2024-05-15T23:16:21.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:16:21.489 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:21.694 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:16:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:21.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:16:21.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:21.841 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:21.869 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:21.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:21.873 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:21.974 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:16:22.035 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:22.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 76 16 158183 158167 130.055 74.1875 0.0304 0.00768772 2024-05-15T23:16:22.475 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:16:22.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/939531566' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:16:22.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:22 smithi090 ceph-mon[44434]: from='client.35618 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:16:22.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:22 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:16:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/939531566' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:16:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:22 smithi088 ceph-mon[44699]: from='client.35618 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:16:22.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:22 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:16:22.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/939531566' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:16:22.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:22 smithi084 ceph-mon[39784]: from='client.35618 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:16:22.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:22 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:16:23.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 77 16 159438 159422 129.385 78.4375 0.017876 0.00772754 2024-05-15T23:16:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:23 smithi084 ceph-mon[39784]: from='client.35618 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:16:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:23 smithi084 ceph-mon[39784]: osdmap e311: 12 total, 12 up, 11 in 2024-05-15T23:16:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:23 smithi084 ceph-mon[39784]: pgmap v863: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-05-15T23:16:23.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:23 smithi090 ceph-mon[44434]: from='client.35618 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:16:23.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:23 smithi090 ceph-mon[44434]: osdmap e311: 12 total, 12 up, 11 in 2024-05-15T23:16:23.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:23 smithi090 ceph-mon[44434]: pgmap v863: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-05-15T23:16:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:23 smithi088 ceph-mon[44699]: from='client.35618 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:16:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:23 smithi088 ceph-mon[44699]: osdmap e311: 12 total, 12 up, 11 in 2024-05-15T23:16:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:23 smithi088 ceph-mon[44699]: pgmap v863: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-05-15T23:16:24.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 78 16 160650 160634 128.697 75.75 0.0104596 0.00776904 2024-05-15T23:16:24.510 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:16:24.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:24.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:16:24.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:24 smithi084 ceph-mon[39784]: 4.d scrub starts 2024-05-15T23:16:24.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:16:24.650 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:24.687 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:16:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:24.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:16:24.718 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:16:24.732 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:24.738 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:16:24.792 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:16:24.802 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:16:24.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:24 smithi090 ceph-mon[44434]: 4.d scrub starts 2024-05-15T23:16:24.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:24 smithi088 ceph-mon[44699]: 4.d scrub starts 2024-05-15T23:16:24.824 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:24.842 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:16:24.845 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:16:24.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:24.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:16:24.866 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:16:24.899 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:16:24.929 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:16:24.948 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:16:24.972 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:16:24.979 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:25.004 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:16:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:25.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:16:25.017 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:16:25.063 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:16:25.103 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:16:25.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:16:25.149 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:16:25.149 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:16:25.149 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:16:25.149 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:16:25.149 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:16:25.149 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:16:25.149 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:16:25.150 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:16:25.150 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:16:25.150 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:16:25.150 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:16:25.150 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:16:25.150 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:16:25.162 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:25.195 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:16:25.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:25.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:16:25.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:16:25.333 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:25.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 79 16 161749 161733 127.938 68.6875 0.0122568 0.00781464 2024-05-15T23:16:25.356 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:16:25.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:25.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:16:25.461 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:16:25.495 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:25.507 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-05-15T23:16:25.553 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:16:25.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:25.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:16:25.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:16:25.703 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:25 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:16:25.703 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:25 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:16:25.703 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:25 smithi090 ceph-mon[44434]: 4.d scrub ok 2024-05-15T23:16:25.703 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:25 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1394941860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:25.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:25 smithi090 ceph-mon[44434]: pgmap v864: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-05-15T23:16:25.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:25 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4089739157' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:16:25.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:25 smithi090 ceph-mon[44434]: from='client.25531 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:16:25.705 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:25.736 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:16:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:25.744 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:16:25.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:25 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:16:25.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:25 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:16:25.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:25 smithi088 ceph-mon[44699]: 4.d scrub ok 2024-05-15T23:16:25.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:25 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1394941860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:25.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:25 smithi088 ceph-mon[44699]: pgmap v864: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-05-15T23:16:25.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:25 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4089739157' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:16:25.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:25 smithi088 ceph-mon[44699]: from='client.25531 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:16:25.845 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:16:25.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:25 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:16:25.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:25 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:16:25.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:25 smithi084 ceph-mon[39784]: 4.d scrub ok 2024-05-15T23:16:25.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:25 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1394941860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:25.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:25 smithi084 ceph-mon[39784]: pgmap v864: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-05-15T23:16:25.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:25.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:25 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4089739157' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:16:25.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:25 smithi084 ceph-mon[39784]: from='client.25531 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:16:25.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:16:25.892 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:25.918 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:16:25.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:25.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:16:26.023 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:16:26.077 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:26.120 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:16:26.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:26.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:16:26.224 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:16:26.290 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:26.319 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:16:26.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:26.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:16:26.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:16:26.349164 min lat: 0.00102037 max lat: 0.215284 avg lat: 0.00786345 2024-05-15T23:16:26.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:16:26.350 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 80 16 162788 162772 127.15 64.9375 0.00652733 0.00786345 2024-05-15T23:16:26.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:26.462 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:26.491 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:26.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:26.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:26.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:26.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:26.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:26.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:26 smithi088 ceph-mon[44699]: 4.c scrub starts 2024-05-15T23:16:26.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:26.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:26 smithi088 ceph-mon[44699]: 4.5 scrub starts 2024-05-15T23:16:26.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:26 smithi088 ceph-mon[44699]: from='client.25531 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:16:26.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:26 smithi088 ceph-mon[44699]: osdmap e312: 12 total, 12 up, 11 in 2024-05-15T23:16:26.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:26 smithi088 ceph-mon[44699]: 4.0 scrub starts 2024-05-15T23:16:26.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:26 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:16:26.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:26 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:16:26.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:26 smithi088 ceph-mon[44699]: 4.4 scrub starts 2024-05-15T23:16:26.595 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:16:26.658 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:26.725 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:16:26.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:26.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:16:26.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:26.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:26.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:26.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:26 smithi090 ceph-mon[44434]: 4.c scrub starts 2024-05-15T23:16:26.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:26.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:26 smithi090 ceph-mon[44434]: 4.5 scrub starts 2024-05-15T23:16:26.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:26 smithi090 ceph-mon[44434]: from='client.25531 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:16:26.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:26 smithi090 ceph-mon[44434]: osdmap e312: 12 total, 12 up, 11 in 2024-05-15T23:16:26.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:26 smithi090 ceph-mon[44434]: 4.0 scrub starts 2024-05-15T23:16:26.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:26 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:16:26.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:26 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:16:26.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:26 smithi090 ceph-mon[44434]: 4.4 scrub starts 2024-05-15T23:16:26.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:16:26.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:26.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:26.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:26.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:26 smithi084 ceph-mon[39784]: 4.c scrub starts 2024-05-15T23:16:26.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:26.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:26 smithi084 ceph-mon[39784]: 4.5 scrub starts 2024-05-15T23:16:26.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:26 smithi084 ceph-mon[39784]: from='client.25531 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:16:26.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:26 smithi084 ceph-mon[39784]: osdmap e312: 12 total, 12 up, 11 in 2024-05-15T23:16:26.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:26 smithi084 ceph-mon[39784]: 4.0 scrub starts 2024-05-15T23:16:26.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:26 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:16:26.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:26 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:16:26.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:26 smithi084 ceph-mon[39784]: 4.4 scrub starts 2024-05-15T23:16:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:27.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 4 1 2024-05-15T23:16:27.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 81 16 163709 163693 126.291 57.5625 0.0181581 0.00791686 2024-05-15T23:16:27.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:27 smithi084 ceph-mon[39784]: 4.e scrub starts 2024-05-15T23:16:27.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:27 smithi084 ceph-mon[39784]: 4.a scrub starts 2024-05-15T23:16:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:27 smithi084 ceph-mon[39784]: osdmap e313: 12 total, 12 up, 11 in 2024-05-15T23:16:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:16:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:16:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:27 smithi084 ceph-mon[39784]: pgmap v867: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-05-15T23:16:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:27 smithi084 ceph-mon[39784]: osdmap e314: 12 total, 12 up, 11 in 2024-05-15T23:16:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:16:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:16:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:27 smithi084 ceph-mon[39784]: 4.6 scrub starts 2024-05-15T23:16:27.749 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:16:27.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:27.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:16:27.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:27 smithi090 ceph-mon[44434]: 4.e scrub starts 2024-05-15T23:16:27.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:27 smithi090 ceph-mon[44434]: 4.a scrub starts 2024-05-15T23:16:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:27 smithi090 ceph-mon[44434]: osdmap e313: 12 total, 12 up, 11 in 2024-05-15T23:16:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:16:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:16:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:27 smithi090 ceph-mon[44434]: pgmap v867: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-05-15T23:16:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:27 smithi090 ceph-mon[44434]: osdmap e314: 12 total, 12 up, 11 in 2024-05-15T23:16:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:16:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:16:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:27 smithi090 ceph-mon[44434]: 4.6 scrub starts 2024-05-15T23:16:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:27 smithi088 ceph-mon[44699]: 4.e scrub starts 2024-05-15T23:16:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:27 smithi088 ceph-mon[44699]: 4.a scrub starts 2024-05-15T23:16:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:27 smithi088 ceph-mon[44699]: osdmap e313: 12 total, 12 up, 11 in 2024-05-15T23:16:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:16:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:16:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:27 smithi088 ceph-mon[44699]: pgmap v867: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-05-15T23:16:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:27 smithi088 ceph-mon[44699]: osdmap e314: 12 total, 12 up, 11 in 2024-05-15T23:16:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:16:27.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:16:27.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:27 smithi088 ceph-mon[44699]: 4.6 scrub starts 2024-05-15T23:16:27.826 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:27.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:16:27.861 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:27.892 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:27.921 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:16:27.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:27.925 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:16:28.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:28.029 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:16:28.063 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:28.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:28.199 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:16:28.258 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:28.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 82 16 164636 164620 125.457 57.9375 0.0116899 0.00796938 2024-05-15T23:16:28.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:28 smithi084 ceph-mon[39784]: 4.5 scrub ok 2024-05-15T23:16:28.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:28 smithi084 ceph-mon[39784]: 4.c scrub ok 2024-05-15T23:16:28.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:28 smithi084 ceph-mon[39784]: 4.0 scrub ok 2024-05-15T23:16:28.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:28 smithi084 ceph-mon[39784]: 4.e scrub ok 2024-05-15T23:16:28.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:28 smithi084 ceph-mon[39784]: osdmap e315: 12 total, 12 up, 11 in 2024-05-15T23:16:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:28 smithi090 ceph-mon[44434]: 4.5 scrub ok 2024-05-15T23:16:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:28 smithi090 ceph-mon[44434]: 4.c scrub ok 2024-05-15T23:16:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:28 smithi090 ceph-mon[44434]: 4.0 scrub ok 2024-05-15T23:16:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:28 smithi090 ceph-mon[44434]: 4.e scrub ok 2024-05-15T23:16:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:28 smithi090 ceph-mon[44434]: osdmap e315: 12 total, 12 up, 11 in 2024-05-15T23:16:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:28 smithi088 ceph-mon[44699]: 4.5 scrub ok 2024-05-15T23:16:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:28 smithi088 ceph-mon[44699]: 4.c scrub ok 2024-05-15T23:16:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:28 smithi088 ceph-mon[44699]: 4.0 scrub ok 2024-05-15T23:16:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:28 smithi088 ceph-mon[44699]: 4.e scrub ok 2024-05-15T23:16:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:28 smithi088 ceph-mon[44699]: osdmap e315: 12 total, 12 up, 11 in 2024-05-15T23:16:29.044 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:16:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:29.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:16:29.162 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:16:29.201 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:29.227 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:16:29.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:29.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:16:29.334 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:16:29.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 83 16 165724 165708 124.765 68 0.0129069 0.00801357 2024-05-15T23:16:29.401 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:29.464 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:16:29.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:29.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:16:29.546 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:29 smithi088 ceph-mon[44699]: 4.b scrub starts 2024-05-15T23:16:29.546 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:29 smithi088 ceph-mon[44699]: 4.4 scrub ok 2024-05-15T23:16:29.547 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:29 smithi088 ceph-mon[44699]: 4.a scrub ok 2024-05-15T23:16:29.547 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:29 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:16:29.547 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:29 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:16:29.547 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:29 smithi088 ceph-mon[44699]: pgmap v870: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-05-15T23:16:29.547 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:29 smithi088 ceph-mon[44699]: 4.6 scrub ok 2024-05-15T23:16:29.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:16:29.615 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:29.643 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:16:29.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:29.647 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:16:29.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:16:29.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:29 smithi084 ceph-mon[39784]: 4.b scrub starts 2024-05-15T23:16:29.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:29 smithi084 ceph-mon[39784]: 4.4 scrub ok 2024-05-15T23:16:29.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:29 smithi084 ceph-mon[39784]: 4.a scrub ok 2024-05-15T23:16:29.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:29 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:16:29.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:29 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:16:29.761 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:29 smithi084 ceph-mon[39784]: pgmap v870: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-05-15T23:16:29.761 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:29 smithi084 ceph-mon[39784]: 4.6 scrub ok 2024-05-15T23:16:29.784 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:29 smithi090 ceph-mon[44434]: 4.b scrub starts 2024-05-15T23:16:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:29 smithi090 ceph-mon[44434]: 4.4 scrub ok 2024-05-15T23:16:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:29 smithi090 ceph-mon[44434]: 4.a scrub ok 2024-05-15T23:16:29.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:29 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:16:29.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:29 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:16:29.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:29 smithi090 ceph-mon[44434]: pgmap v870: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-05-15T23:16:29.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:29 smithi090 ceph-mon[44434]: 4.6 scrub ok 2024-05-15T23:16:29.808 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:16:29.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:29.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:16:29.865 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:16:29.866 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-05-15T23:16:29.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-05-15T23:16:29.866 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:16:29.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.866 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:179413 3.7 3:e1fef487:::benchmark_data_smithi090_90974_object13870:head [delete] snapc 0=[] ondisk+write+known_if_redirected e250)", 2024-05-15T23:16:29.866 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:12:36.500419+0000", 2024-05-15T23:16:29.866 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 232.27956761799999, 2024-05-15T23:16:29.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.10703578399999999, 2024-05-15T23:16:29.866 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.866 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.866 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.867 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.867 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.867 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 179413 2024-05-15T23:16:29.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.867 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:12:36.500419+0000", 2024-05-15T23:16:29.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:12:36.500419+0000", 2024-05-15T23:16:29.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:12:36.500421+0000", 2024-05-15T23:16:29.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0109999999999999e-06 2024-05-15T23:16:29.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:12:36.500427+0000", 2024-05-15T23:16:29.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1430000000000003e-06 2024-05-15T23:16:29.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:12:36.500439+0000", 2024-05-15T23:16:29.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1987999999999999e-05 2024-05-15T23:16:29.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:12:36.500446+0000", 2024-05-15T23:16:29.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1069999999999999e-06 2024-05-15T23:16:29.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:12:36.500502+0000", 2024-05-15T23:16:29.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5673e-05 2024-05-15T23:16:29.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:12:36.500593+0000", 2024-05-15T23:16:29.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1535999999999999e-05 2024-05-15T23:16:29.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 11", 2024-05-15T23:16:29.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:12:36.500673+0000", 2024-05-15T23:16:29.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0004000000000002e-05 2024-05-15T23:16:29.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:12:36.504194+0000", 2024-05-15T23:16:29.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0035214109999999999 2024-05-15T23:16:29.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:12:36.607381+0000", 2024-05-15T23:16:29.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.103186241 2024-05-15T23:16:29.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:12:36.607418+0000", 2024-05-15T23:16:29.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6962000000000003e-05 2024-05-15T23:16:29.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:12:36.607454+0000", 2024-05-15T23:16:29.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6708000000000002e-05 2024-05-15T23:16:29.875 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.875 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.876 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.876 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244624 3.b 3:d9663c5b:::benchmark_data_smithi090_90974_object79081:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:16:29.876 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.092561+0000", 2024-05-15T23:16:29.876 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 178.68742476400001, 2024-05-15T23:16:29.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.223617125, 2024-05-15T23:16:29.876 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.876 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.876 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.877 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.877 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.877 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.877 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244624 2024-05-15T23:16:29.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.877 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092561+0000", 2024-05-15T23:16:29.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092561+0000", 2024-05-15T23:16:29.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092563+0000", 2024-05-15T23:16:29.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.384e-06 2024-05-15T23:16:29.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092568+0000", 2024-05-15T23:16:29.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7679999999999999e-06 2024-05-15T23:16:29.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092576+0000", 2024-05-15T23:16:29.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4090000000000003e-06 2024-05-15T23:16:29.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092582+0000", 2024-05-15T23:16:29.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2310000000000001e-06 2024-05-15T23:16:29.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092632+0000", 2024-05-15T23:16:29.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0065000000000003e-05 2024-05-15T23:16:29.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092746+0000", 2024-05-15T23:16:29.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113791 2024-05-15T23:16:29.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:16:29.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092854+0000", 2024-05-15T23:16:29.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107921 2024-05-15T23:16:29.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.097906+0000", 2024-05-15T23:16:29.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050516399999999996 2024-05-15T23:16:29.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316079+0000", 2024-05-15T23:16:29.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.21817299800000001 2024-05-15T23:16:29.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316131+0000", 2024-05-15T23:16:29.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2525999999999998e-05 2024-05-15T23:16:29.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316179+0000", 2024-05-15T23:16:29.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7392000000000003e-05 2024-05-15T23:16:29.883 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.883 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.883 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.883 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244628 3.b 3:d9384fcf:::benchmark_data_smithi090_90974_object79085:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:16:29.883 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.098262+0000", 2024-05-15T23:16:29.884 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 178.68172412300001, 2024-05-15T23:16:29.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.21807148000000001, 2024-05-15T23:16:29.884 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.884 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.884 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.884 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.884 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.884 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.884 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244628 2024-05-15T23:16:29.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.884 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098262+0000", 2024-05-15T23:16:29.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098262+0000", 2024-05-15T23:16:29.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098264+0000", 2024-05-15T23:16:29.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.699e-06 2024-05-15T23:16:29.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098269+0000", 2024-05-15T23:16:29.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8670000000000002e-06 2024-05-15T23:16:29.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098282+0000", 2024-05-15T23:16:29.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2918999999999999e-05 2024-05-15T23:16:29.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098289+0000", 2024-05-15T23:16:29.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.554e-06 2024-05-15T23:16:29.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098335+0000", 2024-05-15T23:16:29.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6276999999999998e-05 2024-05-15T23:16:29.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098452+0000", 2024-05-15T23:16:29.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000117038 2024-05-15T23:16:29.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:16:29.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098560+0000", 2024-05-15T23:16:29.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000108009 2024-05-15T23:16:29.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.106584+0000", 2024-05-15T23:16:29.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080232729999999992 2024-05-15T23:16:29.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316237+0000", 2024-05-15T23:16:29.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.209653069 2024-05-15T23:16:29.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316290+0000", 2024-05-15T23:16:29.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2836e-05 2024-05-15T23:16:29.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316334+0000", 2024-05-15T23:16:29.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3939000000000001e-05 2024-05-15T23:16:29.890 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.890 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.890 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.890 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244632 3.b 3:d33b0449:::benchmark_data_smithi090_90974_object79089:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:16:29.890 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.102130+0000", 2024-05-15T23:16:29.890 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 178.67785626, 2024-05-15T23:16:29.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.21432957799999999, 2024-05-15T23:16:29.890 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.890 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.891 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.891 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.891 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.891 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.891 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244632 2024-05-15T23:16:29.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.891 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102130+0000", 2024-05-15T23:16:29.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102130+0000", 2024-05-15T23:16:29.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102133+0000", 2024-05-15T23:16:29.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6840000000000001e-06 2024-05-15T23:16:29.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102140+0000", 2024-05-15T23:16:29.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6629999999999993e-06 2024-05-15T23:16:29.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102155+0000", 2024-05-15T23:16:29.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5189e-05 2024-05-15T23:16:29.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102167+0000", 2024-05-15T23:16:29.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1058e-05 2024-05-15T23:16:29.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102585+0000", 2024-05-15T23:16:29.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000418742 2024-05-15T23:16:29.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102667+0000", 2024-05-15T23:16:29.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1692000000000003e-05 2024-05-15T23:16:29.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:16:29.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102738+0000", 2024-05-15T23:16:29.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0678000000000003e-05 2024-05-15T23:16:29.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.106775+0000", 2024-05-15T23:16:29.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040370099999999997 2024-05-15T23:16:29.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316378+0000", 2024-05-15T23:16:29.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.20960367299999999 2024-05-15T23:16:29.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316422+0000", 2024-05-15T23:16:29.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3850999999999997e-05 2024-05-15T23:16:29.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316460+0000", 2024-05-15T23:16:29.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7338000000000001e-05 2024-05-15T23:16:29.896 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.896 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.897 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.897 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244652 3.b 3:d38aa161:::benchmark_data_smithi090_90974_object79109:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:16:29.897 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.119285+0000", 2024-05-15T23:16:29.897 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 178.66070075299999, 2024-05-15T23:16:29.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.21651990300000001, 2024-05-15T23:16:29.897 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.897 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.897 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.897 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.897 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.898 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.898 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244652 2024-05-15T23:16:29.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.898 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119285+0000", 2024-05-15T23:16:29.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119285+0000", 2024-05-15T23:16:29.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119289+0000", 2024-05-15T23:16:29.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4999999999999999e-06 2024-05-15T23:16:29.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119299+0000", 2024-05-15T23:16:29.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9350000000000006e-06 2024-05-15T23:16:29.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119314+0000", 2024-05-15T23:16:29.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5163e-05 2024-05-15T23:16:29.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119326+0000", 2024-05-15T23:16:29.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1657e-05 2024-05-15T23:16:29.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119410+0000", 2024-05-15T23:16:29.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3991000000000003e-05 2024-05-15T23:16:29.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119611+0000", 2024-05-15T23:16:29.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00020102499999999999 2024-05-15T23:16:29.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:16:29.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119718+0000", 2024-05-15T23:16:29.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107137 2024-05-15T23:16:29.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.124475+0000", 2024-05-15T23:16:29.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047574499999999999 2024-05-15T23:16:29.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.335707+0000", 2024-05-15T23:16:29.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.211231271 2024-05-15T23:16:29.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.335758+0000", 2024-05-15T23:16:29.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1039999999999999e-05 2024-05-15T23:16:29.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.335805+0000", 2024-05-15T23:16:29.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7734000000000001e-05 2024-05-15T23:16:29.904 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.904 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.904 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.904 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244695 3.b 3:df81a8c7:::benchmark_data_smithi090_90974_object79152:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:16:29.904 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.161150+0000", 2024-05-15T23:16:29.904 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 178.618836058, 2024-05-15T23:16:29.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18517029900000001, 2024-05-15T23:16:29.904 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.905 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.905 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.905 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.905 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.905 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.905 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244695 2024-05-15T23:16:29.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.905 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161150+0000", 2024-05-15T23:16:29.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161150+0000", 2024-05-15T23:16:29.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161153+0000", 2024-05-15T23:16:29.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.587e-06 2024-05-15T23:16:29.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161161+0000", 2024-05-15T23:16:29.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0760000000000003e-06 2024-05-15T23:16:29.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161174+0000", 2024-05-15T23:16:29.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3524e-05 2024-05-15T23:16:29.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161184+0000", 2024-05-15T23:16:29.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5810000000000003e-06 2024-05-15T23:16:29.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161249+0000", 2024-05-15T23:16:29.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4918000000000004e-05 2024-05-15T23:16:29.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161379+0000", 2024-05-15T23:16:29.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013014400000000001 2024-05-15T23:16:29.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:16:29.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161522+0000", 2024-05-15T23:16:29.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014294500000000001 2024-05-15T23:16:29.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.165694+0000", 2024-05-15T23:16:29.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041721379999999997 2024-05-15T23:16:29.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346253+0000", 2024-05-15T23:16:29.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.180558575 2024-05-15T23:16:29.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346288+0000", 2024-05-15T23:16:29.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5845e-05 2024-05-15T23:16:29.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346320+0000", 2024-05-15T23:16:29.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1965999999999999e-05 2024-05-15T23:16:29.911 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.911 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.911 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.911 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244726 3.b 3:d016c9af:::benchmark_data_smithi090_90974_object79183:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:16:29.911 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.187811+0000", 2024-05-15T23:16:29.912 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 178.59217556499999, 2024-05-15T23:16:29.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15861062100000001, 2024-05-15T23:16:29.912 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.912 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.912 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.912 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.912 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.912 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.912 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244726 2024-05-15T23:16:29.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.912 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187811+0000", 2024-05-15T23:16:29.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187811+0000", 2024-05-15T23:16:29.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187812+0000", 2024-05-15T23:16:29.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.691e-06 2024-05-15T23:16:29.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187818+0000", 2024-05-15T23:16:29.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9630000000000003e-06 2024-05-15T23:16:29.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187828+0000", 2024-05-15T23:16:29.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3999999999999998e-06 2024-05-15T23:16:29.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187833+0000", 2024-05-15T23:16:29.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6879999999999996e-06 2024-05-15T23:16:29.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187898+0000", 2024-05-15T23:16:29.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4202000000000006e-05 2024-05-15T23:16:29.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.188038+0000", 2024-05-15T23:16:29.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014031100000000001 2024-05-15T23:16:29.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:16:29.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.188183+0000", 2024-05-15T23:16:29.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014492899999999999 2024-05-15T23:16:29.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.198038+0000", 2024-05-15T23:16:29.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0098547849999999996 2024-05-15T23:16:29.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346353+0000", 2024-05-15T23:16:29.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14831505 2024-05-15T23:16:29.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346382+0000", 2024-05-15T23:16:29.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9176e-05 2024-05-15T23:16:29.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346421+0000", 2024-05-15T23:16:29.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9425999999999998e-05 2024-05-15T23:16:29.917 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.917 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.918 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.918 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244727 3.b 3:d896ece8:::benchmark_data_smithi090_90974_object79184:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:16:29.918 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.187908+0000", 2024-05-15T23:16:29.918 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 178.59207817399999, 2024-05-15T23:16:29.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.158687891, 2024-05-15T23:16:29.918 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.918 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.918 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.918 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.918 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.918 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.918 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244727 2024-05-15T23:16:29.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.919 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187908+0000", 2024-05-15T23:16:29.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187908+0000", 2024-05-15T23:16:29.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187909+0000", 2024-05-15T23:16:29.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6099999999999999e-07 2024-05-15T23:16:29.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187912+0000", 2024-05-15T23:16:29.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.348e-06 2024-05-15T23:16:29.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187918+0000", 2024-05-15T23:16:29.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4519999999999996e-06 2024-05-15T23:16:29.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187922+0000", 2024-05-15T23:16:29.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8299999999999998e-06 2024-05-15T23:16:29.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.193291+0000", 2024-05-15T23:16:29.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053689599999999999 2024-05-15T23:16:29.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.193434+0000", 2024-05-15T23:16:29.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014339600000000001 2024-05-15T23:16:29.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:16:29.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.193548+0000", 2024-05-15T23:16:29.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011445799999999999 2024-05-15T23:16:29.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.198206+0000", 2024-05-15T23:16:29.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046579509999999996 2024-05-15T23:16:29.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346511+0000", 2024-05-15T23:16:29.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14830475400000001 2024-05-15T23:16:29.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346559+0000", 2024-05-15T23:16:29.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8217999999999998e-05 2024-05-15T23:16:29.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346596+0000", 2024-05-15T23:16:29.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6563000000000002e-05 2024-05-15T23:16:29.923 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.924 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.924 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.924 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244759 3.b 3:dc56f482:::benchmark_data_smithi090_90974_object79216:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:16:29.924 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.228332+0000", 2024-05-15T23:16:29.924 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 178.551654681, 2024-05-15T23:16:29.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12161722699999999, 2024-05-15T23:16:29.924 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.924 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.924 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.924 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.924 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.924 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.925 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244759 2024-05-15T23:16:29.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.925 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228332+0000", 2024-05-15T23:16:29.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228332+0000", 2024-05-15T23:16:29.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228334+0000", 2024-05-15T23:16:29.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2050000000000001e-06 2024-05-15T23:16:29.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228340+0000", 2024-05-15T23:16:29.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5359999999999998e-06 2024-05-15T23:16:29.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228355+0000", 2024-05-15T23:16:29.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4800999999999999e-05 2024-05-15T23:16:29.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228364+0000", 2024-05-15T23:16:29.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0880000000000001e-06 2024-05-15T23:16:29.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228430+0000", 2024-05-15T23:16:29.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6174999999999996e-05 2024-05-15T23:16:29.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228551+0000", 2024-05-15T23:16:29.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012065699999999999 2024-05-15T23:16:29.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:16:29.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228670+0000", 2024-05-15T23:16:29.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000118702 2024-05-15T23:16:29.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.236954+0000", 2024-05-15T23:16:29.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082840819999999999 2024-05-15T23:16:29.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.349859+0000", 2024-05-15T23:16:29.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.112905017 2024-05-15T23:16:29.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.349910+0000", 2024-05-15T23:16:29.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1613000000000002e-05 2024-05-15T23:16:29.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.349949+0000", 2024-05-15T23:16:29.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8350999999999999e-05 2024-05-15T23:16:29.930 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.930 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.930 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.930 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244761 3.b 3:dc7a3ef4:::benchmark_data_smithi090_90974_object79218:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:16:29.930 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.230451+0000", 2024-05-15T23:16:29.930 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 178.54953543100001, 2024-05-15T23:16:29.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.119620066, 2024-05-15T23:16:29.931 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.931 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.931 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.931 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.931 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244761 2024-05-15T23:16:29.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.931 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.230451+0000", 2024-05-15T23:16:29.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.230451+0000", 2024-05-15T23:16:29.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.230453+0000", 2024-05-15T23:16:29.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.773e-06 2024-05-15T23:16:29.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.230458+0000", 2024-05-15T23:16:29.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2259999999999996e-06 2024-05-15T23:16:29.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.230469+0000", 2024-05-15T23:16:29.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1364e-05 2024-05-15T23:16:29.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.230484+0000", 2024-05-15T23:16:29.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4549000000000001e-05 2024-05-15T23:16:29.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.232644+0000", 2024-05-15T23:16:29.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021598469999999999 2024-05-15T23:16:29.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.232769+0000", 2024-05-15T23:16:29.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012553900000000001 2024-05-15T23:16:29.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:16:29.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.232895+0000", 2024-05-15T23:16:29.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000125809 2024-05-15T23:16:29.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.237087+0000", 2024-05-15T23:16:29.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041917999999999999 2024-05-15T23:16:29.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.349988+0000", 2024-05-15T23:16:29.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.112901394 2024-05-15T23:16:29.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.350035+0000", 2024-05-15T23:16:29.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7160999999999998e-05 2024-05-15T23:16:29.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.350071+0000", 2024-05-15T23:16:29.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5604e-05 2024-05-15T23:16:29.937 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.937 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.937 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.938 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244770 3.b 3:d69b076a:::benchmark_data_smithi090_90974_object79227:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:16:29.938 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.240553+0000", 2024-05-15T23:16:29.938 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 178.53943313100001, 2024-05-15T23:16:29.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.109669028, 2024-05-15T23:16:29.938 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.938 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.938 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.938 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.938 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.938 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.939 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244770 2024-05-15T23:16:29.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.939 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240553+0000", 2024-05-15T23:16:29.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240553+0000", 2024-05-15T23:16:29.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240555+0000", 2024-05-15T23:16:29.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5e-06 2024-05-15T23:16:29.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240560+0000", 2024-05-15T23:16:29.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2689999999999999e-06 2024-05-15T23:16:29.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240569+0000", 2024-05-15T23:16:29.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1349999999999998e-06 2024-05-15T23:16:29.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240576+0000", 2024-05-15T23:16:29.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.737e-06 2024-05-15T23:16:29.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240610+0000", 2024-05-15T23:16:29.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4239000000000002e-05 2024-05-15T23:16:29.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240691+0000", 2024-05-15T23:16:29.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1198000000000004e-05 2024-05-15T23:16:29.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:16:29.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240762+0000", 2024-05-15T23:16:29.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0900000000000002e-05 2024-05-15T23:16:29.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.244680+0000", 2024-05-15T23:16:29.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039178169999999997 2024-05-15T23:16:29.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.350127+0000", 2024-05-15T23:16:29.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.10544740900000001 2024-05-15T23:16:29.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.350180+0000", 2024-05-15T23:16:29.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2967999999999998e-05 2024-05-15T23:16:29.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.350222+0000", 2024-05-15T23:16:29.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1856e-05 2024-05-15T23:16:29.945 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.945 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.945 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.945 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244787 3.b 3:d6752f13:::benchmark_data_smithi090_90974_object79244:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:16:29.945 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.261577+0000", 2024-05-15T23:16:29.945 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 178.518409577, 2024-05-15T23:16:29.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096225877000000001, 2024-05-15T23:16:29.946 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.946 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.946 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.946 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.946 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.946 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.946 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244787 2024-05-15T23:16:29.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.946 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261577+0000", 2024-05-15T23:16:29.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261577+0000", 2024-05-15T23:16:29.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261578+0000", 2024-05-15T23:16:29.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7910000000000001e-06 2024-05-15T23:16:29.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261585+0000", 2024-05-15T23:16:29.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3820000000000001e-06 2024-05-15T23:16:29.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261596+0000", 2024-05-15T23:16:29.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0929e-05 2024-05-15T23:16:29.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261604+0000", 2024-05-15T23:16:29.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.994e-06 2024-05-15T23:16:29.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.949 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261664+0000", 2024-05-15T23:16:29.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0652000000000001e-05 2024-05-15T23:16:29.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.949 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261797+0000", 2024-05-15T23:16:29.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013271100000000001 2024-05-15T23:16:29.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:16:29.949 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261926+0000", 2024-05-15T23:16:29.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128756 2024-05-15T23:16:29.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.266795+0000", 2024-05-15T23:16:29.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048690080000000002 2024-05-15T23:16:29.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.357732+0000", 2024-05-15T23:16:29.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090936930999999999 2024-05-15T23:16:29.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.357773+0000", 2024-05-15T23:16:29.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1495000000000002e-05 2024-05-15T23:16:29.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.357803+0000", 2024-05-15T23:16:29.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9227999999999999e-05 2024-05-15T23:16:29.951 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.951 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.951 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.951 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244797 3.b 3:da28cbb5:::benchmark_data_smithi090_90974_object79254:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:16:29.951 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.271847+0000", 2024-05-15T23:16:29.951 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 178.50813889, 2024-05-15T23:16:29.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.103864567, 2024-05-15T23:16:29.952 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.952 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.952 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.952 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.952 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.952 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.952 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244797 2024-05-15T23:16:29.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.952 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.271847+0000", 2024-05-15T23:16:29.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.271847+0000", 2024-05-15T23:16:29.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.271850+0000", 2024-05-15T23:16:29.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5450000000000002e-06 2024-05-15T23:16:29.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.271857+0000", 2024-05-15T23:16:29.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.43e-06 2024-05-15T23:16:29.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.954 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.954 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.271875+0000", 2024-05-15T23:16:29.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7337999999999999e-05 2024-05-15T23:16:29.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.954 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.954 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.271884+0000", 2024-05-15T23:16:29.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.696e-06 2024-05-15T23:16:29.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.271941+0000", 2024-05-15T23:16:29.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7000000000000003e-05 2024-05-15T23:16:29.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.272040+0000", 2024-05-15T23:16:29.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8319999999999994e-05 2024-05-15T23:16:29.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:16:29.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.272123+0000", 2024-05-15T23:16:29.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3572000000000006e-05 2024-05-15T23:16:29.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.276208+0000", 2024-05-15T23:16:29.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040850549999999998 2024-05-15T23:16:29.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.375616+0000", 2024-05-15T23:16:29.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.099407869999999995 2024-05-15T23:16:29.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.375672+0000", 2024-05-15T23:16:29.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5509e-05 2024-05-15T23:16:29.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.375712+0000", 2024-05-15T23:16:29.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0231999999999997e-05 2024-05-15T23:16:29.957 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.957 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.957 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.957 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:307618 3.7 3:ec218ab1:::benchmark_data_smithi090_90974_object142075:head [delete] snapc 0=[] ondisk+write+known_if_redirected e274)", 2024-05-15T23:16:29.958 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:14:22.587868+0000", 2024-05-15T23:16:29.958 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 126.19211819900001, 2024-05-15T23:16:29.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.102056827, 2024-05-15T23:16:29.958 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.958 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.958 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.958 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.958 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.958 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.958 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 307618 2024-05-15T23:16:29.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.959 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.959 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.959 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.959 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.587868+0000", 2024-05-15T23:16:29.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.959 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.959 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.959 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.587868+0000", 2024-05-15T23:16:29.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.960 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.587870+0000", 2024-05-15T23:16:29.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.669e-06 2024-05-15T23:16:29.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.960 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.587875+0000", 2024-05-15T23:16:29.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5090000000000001e-06 2024-05-15T23:16:29.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.587886+0000", 2024-05-15T23:16:29.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0413e-05 2024-05-15T23:16:29.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.587893+0000", 2024-05-15T23:16:29.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1149999999999996e-06 2024-05-15T23:16:29.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.587936+0000", 2024-05-15T23:16:29.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2963999999999997e-05 2024-05-15T23:16:29.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.588023+0000", 2024-05-15T23:16:29.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6965000000000005e-05 2024-05-15T23:16:29.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 11", 2024-05-15T23:16:29.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.588141+0000", 2024-05-15T23:16:29.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000117947 2024-05-15T23:16:29.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.591958+0000", 2024-05-15T23:16:29.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0038174569999999998 2024-05-15T23:16:29.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.689819+0000", 2024-05-15T23:16:29.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097860674999999994 2024-05-15T23:16:29.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.964 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.964 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.689874+0000", 2024-05-15T23:16:29.964 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5500000000000001e-05 2024-05-15T23:16:29.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.964 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.964 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.689925+0000", 2024-05-15T23:16:29.964 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0612999999999998e-05 2024-05-15T23:16:29.964 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.965 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.965 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.965 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.965 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.965 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:326342 3.2 3:4c240c5d:::benchmark_data_smithi090_90974_object160799:head [delete] snapc 0=[] ondisk+write+known_if_redirected e277)", 2024-05-15T23:16:29.965 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:14:38.287105+0000", 2024-05-15T23:16:29.965 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 110.492881083, 2024-05-15T23:16:29.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.098496609999999998, 2024-05-15T23:16:29.965 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.965 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.965 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.966 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.966 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.966 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.966 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 326342 2024-05-15T23:16:29.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.966 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287105+0000", 2024-05-15T23:16:29.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287105+0000", 2024-05-15T23:16:29.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287107+0000", 2024-05-15T23:16:29.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9069999999999999e-06 2024-05-15T23:16:29.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287115+0000", 2024-05-15T23:16:29.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1329999999999999e-06 2024-05-15T23:16:29.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287125+0000", 2024-05-15T23:16:29.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8190000000000003e-06 2024-05-15T23:16:29.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.969 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.969 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287134+0000", 2024-05-15T23:16:29.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5879999999999998e-06 2024-05-15T23:16:29.969 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.969 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.969 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.969 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287192+0000", 2024-05-15T23:16:29.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8838000000000001e-05 2024-05-15T23:16:29.969 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.969 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287306+0000", 2024-05-15T23:16:29.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113594 2024-05-15T23:16:29.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:16:29.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287418+0000", 2024-05-15T23:16:29.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111988 2024-05-15T23:16:29.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292266+0000", 2024-05-15T23:16:29.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048482689999999997 2024-05-15T23:16:29.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385538+0000", 2024-05-15T23:16:29.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093272043999999998 2024-05-15T23:16:29.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385571+0000", 2024-05-15T23:16:29.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2985999999999999e-05 2024-05-15T23:16:29.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385602+0000", 2024-05-15T23:16:29.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0443999999999999e-05 2024-05-15T23:16:29.972 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.972 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.972 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.973 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:326346 3.2 3:4e958513:::benchmark_data_smithi090_90974_object160803:head [delete] snapc 0=[] ondisk+write+known_if_redirected e277)", 2024-05-15T23:16:29.973 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:14:38.292280+0000", 2024-05-15T23:16:29.973 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 110.487706113, 2024-05-15T23:16:29.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093412213999999993, 2024-05-15T23:16:29.973 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.973 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.973 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.973 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.973 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.973 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.974 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 326346 2024-05-15T23:16:29.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.974 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292280+0000", 2024-05-15T23:16:29.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292280+0000", 2024-05-15T23:16:29.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292282+0000", 2024-05-15T23:16:29.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.494e-06 2024-05-15T23:16:29.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292288+0000", 2024-05-15T23:16:29.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2199999999999997e-06 2024-05-15T23:16:29.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292296+0000", 2024-05-15T23:16:29.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5520000000000005e-06 2024-05-15T23:16:29.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292303+0000", 2024-05-15T23:16:29.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.409e-06 2024-05-15T23:16:29.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292339+0000", 2024-05-15T23:16:29.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5806999999999998e-05 2024-05-15T23:16:29.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292440+0000", 2024-05-15T23:16:29.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101629 2024-05-15T23:16:29.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:16:29.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292548+0000", 2024-05-15T23:16:29.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107509 2024-05-15T23:16:29.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.381144+0000", 2024-05-15T23:16:29.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.088596757999999998 2024-05-15T23:16:29.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385638+0000", 2024-05-15T23:16:29.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044937809999999996 2024-05-15T23:16:29.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385667+0000", 2024-05-15T23:16:29.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9203000000000001e-05 2024-05-15T23:16:29.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385692+0000", 2024-05-15T23:16:29.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4851999999999999e-05 2024-05-15T23:16:29.979 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.979 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.979 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.979 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:326348 3.2 3:43de2e75:::benchmark_data_smithi090_90974_object160805:head [delete] snapc 0=[] ondisk+write+known_if_redirected e277)", 2024-05-15T23:16:29.979 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:14:38.292315+0000", 2024-05-15T23:16:29.979 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 110.48767149299999, 2024-05-15T23:16:29.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093457223000000006, 2024-05-15T23:16:29.979 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.979 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.979 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.979 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.980 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.980 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.980 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 326348 2024-05-15T23:16:29.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.980 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292315+0000", 2024-05-15T23:16:29.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292315+0000", 2024-05-15T23:16:29.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292315+0000", 2024-05-15T23:16:29.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.13e-07 2024-05-15T23:16:29.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292318+0000", 2024-05-15T23:16:29.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5799999999999999e-06 2024-05-15T23:16:29.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292323+0000", 2024-05-15T23:16:29.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9570000000000001e-06 2024-05-15T23:16:29.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292328+0000", 2024-05-15T23:16:29.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7729999999999999e-06 2024-05-15T23:16:29.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.296253+0000", 2024-05-15T23:16:29.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039255089999999998 2024-05-15T23:16:29.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.296350+0000", 2024-05-15T23:16:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6979999999999996e-05 2024-05-15T23:16:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:16:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.296444+0000", 2024-05-15T23:16:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3487999999999997e-05 2024-05-15T23:16:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385610+0000", 2024-05-15T23:16:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089166309999999999 2024-05-15T23:16:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385724+0000", 2024-05-15T23:16:29.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113776 2024-05-15T23:16:29.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385751+0000", 2024-05-15T23:16:29.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6945999999999999e-05 2024-05-15T23:16:29.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385772+0000", 2024-05-15T23:16:29.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1390999999999999e-05 2024-05-15T23:16:29.984 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.985 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.985 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.985 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:326350 3.2 3:4a89f156:::benchmark_data_smithi090_90974_object160807:head [delete] snapc 0=[] ondisk+write+known_if_redirected e277)", 2024-05-15T23:16:29.985 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:14:38.292651+0000", 2024-05-15T23:16:29.985 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 110.487334979, 2024-05-15T23:16:29.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093197133000000001, 2024-05-15T23:16:29.985 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.985 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.985 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.985 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.985 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.986 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.986 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 326350 2024-05-15T23:16:29.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.986 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292651+0000", 2024-05-15T23:16:29.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292651+0000", 2024-05-15T23:16:29.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292652+0000", 2024-05-15T23:16:29.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7600000000000006e-07 2024-05-15T23:16:29.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292656+0000", 2024-05-15T23:16:29.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7290000000000002e-06 2024-05-15T23:16:29.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292664+0000", 2024-05-15T23:16:29.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8690000000000004e-06 2024-05-15T23:16:29.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292670+0000", 2024-05-15T23:16:29.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0100000000000001e-06 2024-05-15T23:16:29.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.300238+0000", 2024-05-15T23:16:29.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0075680799999999996 2024-05-15T23:16:29.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.300343+0000", 2024-05-15T23:16:29.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105173 2024-05-15T23:16:29.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:16:29.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.300451+0000", 2024-05-15T23:16:29.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107803 2024-05-15T23:16:29.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385614+0000", 2024-05-15T23:16:29.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.085162784000000005 2024-05-15T23:16:29.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385804+0000", 2024-05-15T23:16:29.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019009800000000001 2024-05-15T23:16:29.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385829+0000", 2024-05-15T23:16:29.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5278000000000001e-05 2024-05-15T23:16:29.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385848+0000", 2024-05-15T23:16:29.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9332999999999999e-05 2024-05-15T23:16:29.991 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.991 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.991 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.992 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:326358 3.2 3:4bb7fff5:::benchmark_data_smithi090_90974_object160815:head [delete] snapc 0=[] ondisk+write+known_if_redirected e277)", 2024-05-15T23:16:29.992 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:14:38.297628+0000", 2024-05-15T23:16:29.992 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 110.482358166, 2024-05-15T23:16:29.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.088290534000000004, 2024-05-15T23:16:29.992 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:29.992 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:29.992 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:29.992 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:29.992 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:29.992 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:29.993 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 326358 2024-05-15T23:16:29.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.993 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:29.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:29.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.297628+0000", 2024-05-15T23:16:29.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:29.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.297628+0000", 2024-05-15T23:16:29.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:29.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.994 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:29.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.297629+0000", 2024-05-15T23:16:29.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.074e-06 2024-05-15T23:16:29.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.994 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:29.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.297636+0000", 2024-05-15T23:16:29.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.421e-06 2024-05-15T23:16:29.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:29.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.297646+0000", 2024-05-15T23:16:29.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0214000000000001e-05 2024-05-15T23:16:29.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:29.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.297652+0000", 2024-05-15T23:16:29.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5610000000000004e-06 2024-05-15T23:16:29.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:29.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.305101+0000", 2024-05-15T23:16:29.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074482250000000002 2024-05-15T23:16:29.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:29.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.305208+0000", 2024-05-15T23:16:29.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010787 2024-05-15T23:16:29.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:16:29.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.376051+0000", 2024-05-15T23:16:29.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.070842961999999995 2024-05-15T23:16:29.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:29.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385778+0000", 2024-05-15T23:16:29.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0097269499999999998 2024-05-15T23:16:29.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:29.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385875+0000", 2024-05-15T23:16:29.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6886999999999996e-05 2024-05-15T23:16:29.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:29.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385900+0000", 2024-05-15T23:16:29.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4769e-05 2024-05-15T23:16:29.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:29.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385919+0000", 2024-05-15T23:16:29.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8601e-05 2024-05-15T23:16:29.999 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.999 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:29.999 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:29.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:29.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:29.999 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:326363 3.2 3:4aae4c06:::benchmark_data_smithi090_90974_object160820:head [delete] snapc 0=[] ondisk+write+known_if_redirected e277)", 2024-05-15T23:16:29.999 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:14:38.302435+0000", 2024-05-15T23:16:29.999 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 110.477551217, 2024-05-15T23:16:30.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097756039000000003, 2024-05-15T23:16:30.000 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:16:30.000 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:16:30.000 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:30.000 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:16:30.000 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:16:30.000 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:30.000 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 326363 2024-05-15T23:16:30.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:30.000 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:16:30.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:30.001 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:16:30.001 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.302435+0000", 2024-05-15T23:16:30.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:30.001 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:30.001 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:30.001 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:16:30.001 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.302435+0000", 2024-05-15T23:16:30.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:16:30.001 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:30.001 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:30.001 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:16:30.002 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.302436+0000", 2024-05-15T23:16:30.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.35e-06 2024-05-15T23:16:30.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:30.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:30.002 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:16:30.002 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.302441+0000", 2024-05-15T23:16:30.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7509999999999999e-06 2024-05-15T23:16:30.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:30.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:30.002 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:16:30.002 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.302449+0000", 2024-05-15T23:16:30.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9869999999999999e-06 2024-05-15T23:16:30.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:30.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:30.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:16:30.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.302463+0000", 2024-05-15T23:16:30.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4363e-05 2024-05-15T23:16:30.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:30.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:30.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:16:30.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.386018+0000", 2024-05-15T23:16:30.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.083554470000000006 2024-05-15T23:16:30.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:30.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:30.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:16:30.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.386088+0000", 2024-05-15T23:16:30.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9783999999999995e-05 2024-05-15T23:16:30.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:30.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:30.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:16:30.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.386152+0000", 2024-05-15T23:16:30.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4420999999999999e-05 2024-05-15T23:16:30.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:30.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:30.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:30.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.400065+0000", 2024-05-15T23:16:30.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013912522 2024-05-15T23:16:30.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:30.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:30.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:16:30.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.400117+0000", 2024-05-15T23:16:30.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2476999999999999e-05 2024-05-15T23:16:30.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:30.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:30.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:16:30.006 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.400146+0000", 2024-05-15T23:16:30.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9354999999999999e-05 2024-05-15T23:16:30.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:16:30.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:16:30.006 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:16:30.006 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.400191+0000", 2024-05-15T23:16:30.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4558999999999998e-05 2024-05-15T23:16:30.006 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:30.006 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:30.006 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:30.006 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:16:30.006 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:16:30.007 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:16:30.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:16:30.044 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:30.068 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:16:30.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:30.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:16:30.174 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:16:30.202 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:30.226 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:16:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:30.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:16:30.335 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:16:30.349 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 84 16 166815 166799 124.092 68.1875 0.026246 0.00805684 2024-05-15T23:16:30.366 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:30.395 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:16:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:30.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:16:30.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:16:30.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:30 smithi090 ceph-mon[44434]: osdmap e316: 12 total, 12 up, 11 in 2024-05-15T23:16:30.547 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:30 smithi090 ceph-mon[44434]: 4.f scrub starts 2024-05-15T23:16:30.551 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:30.637 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:16:30.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:30.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:16:30.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:16:30.772 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:30.798 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:16:30.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:30.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:16:30.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:30 smithi088 ceph-mon[44699]: osdmap e316: 12 total, 12 up, 11 in 2024-05-15T23:16:30.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:30 smithi088 ceph-mon[44699]: 4.f scrub starts 2024-05-15T23:16:30.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:30 smithi084 ceph-mon[39784]: osdmap e316: 12 total, 12 up, 11 in 2024-05-15T23:16:30.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:30 smithi084 ceph-mon[39784]: 4.f scrub starts 2024-05-15T23:16:30.903 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:16:30.932 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:30.954 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:16:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:30.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:16:31.045 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T23:16:31.064 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:16:31.120 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:31.151 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:16:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:31.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:16:31.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:16:31.298 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:16:31.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:31.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:16:31.350 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 85 16 167923 167907 123.446 69.25 0.0159139 0.00809912 2024-05-15T23:16:31.394 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:16:31.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:16:31.473 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:16:31.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:31.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:16:31.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:16:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:31 smithi084 ceph-mon[39784]: 4.9 scrub starts 2024-05-15T23:16:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:31 smithi084 ceph-mon[39784]: 4.b scrub ok 2024-05-15T23:16:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:31 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:16:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:31 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:16:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:31 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:16:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:31 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:16:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:31 smithi084 ceph-mon[39784]: pgmap v872: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 10 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-05-15T23:16:31.665 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:31 smithi090 ceph-mon[44434]: 4.9 scrub starts 2024-05-15T23:16:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:31 smithi090 ceph-mon[44434]: 4.b scrub ok 2024-05-15T23:16:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:31 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:16:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:31 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:16:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:31 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:16:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:31 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:16:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:31 smithi090 ceph-mon[44434]: pgmap v872: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 10 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-05-15T23:16:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:31 smithi088 ceph-mon[44699]: 4.9 scrub starts 2024-05-15T23:16:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:31 smithi088 ceph-mon[44699]: 4.b scrub ok 2024-05-15T23:16:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:31 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:16:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:31 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:16:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:31 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:16:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:31 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:16:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:31 smithi088 ceph-mon[44699]: pgmap v872: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 10 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-05-15T23:16:31.920 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:32.043 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:16:32.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:32.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:16:32.155 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:16:32.189 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:16:32.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:32.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:16:32.350 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 86 16 169008 168992 122.799 67.8125 0.0148251 0.00814178 2024-05-15T23:16:32.409 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:16:32.470 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:32.534 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:16:32.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:32.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:16:32.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:32 smithi088 ceph-mon[44699]: osdmap e317: 12 total, 12 up, 11 in 2024-05-15T23:16:32.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:32 smithi088 ceph-mon[44699]: osdmap e318: 12 total, 12 up, 11 in 2024-05-15T23:16:32.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:32 smithi088 ceph-mon[44699]: 4.7 scrub starts 2024-05-15T23:16:32.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:32 smithi088 ceph-mon[44699]: 4.2 scrub starts 2024-05-15T23:16:32.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/464435936' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:16:32.644 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:16:32.680 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:32.689 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:32 smithi084 ceph-mon[39784]: osdmap e317: 12 total, 12 up, 11 in 2024-05-15T23:16:32.689 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:32 smithi084 ceph-mon[39784]: osdmap e318: 12 total, 12 up, 11 in 2024-05-15T23:16:32.689 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:32 smithi084 ceph-mon[39784]: 4.7 scrub starts 2024-05-15T23:16:32.689 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:32 smithi084 ceph-mon[39784]: 4.2 scrub starts 2024-05-15T23:16:32.689 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/464435936' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:16:32.939 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:16:32.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:32.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:16:32.964 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:16:33.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:16:33.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:32 smithi090 ceph-mon[44434]: osdmap e317: 12 total, 12 up, 11 in 2024-05-15T23:16:33.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:32 smithi090 ceph-mon[44434]: osdmap e318: 12 total, 12 up, 11 in 2024-05-15T23:16:33.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:32 smithi090 ceph-mon[44434]: 4.7 scrub starts 2024-05-15T23:16:33.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:32 smithi090 ceph-mon[44434]: 4.2 scrub starts 2024-05-15T23:16:33.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/464435936' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:16:33.081 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:33.283 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:16:33.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:33.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:16:33.350 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 87 16 169982 169966 122.087 60.875 0.0147479 0.0081892 2024-05-15T23:16:33.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:33.425 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:33.448 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:33.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:33.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:33.552 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:16:33.587 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:16:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:16:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:170202 4.d 4:b2f25503:::benchmark_data_smithi090_104448_object170201:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e319)", 2024-05-15T23:16:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:16:33.576171+0000", 2024-05-15T23:16:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.010020914000000001, 2024-05-15T23:16:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010053069, 2024-05-15T23:16:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:16:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:16:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:16:33.588 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 170202 2024-05-15T23:16:33.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.589 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:33.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.589 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:33.589 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.576171+0000", 2024-05-15T23:16:33.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:33.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.589 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:33.589 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.576171+0000", 2024-05-15T23:16:33.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:33.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.576173+0000", 2024-05-15T23:16:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8899999999999999e-06 2024-05-15T23:16:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.576263+0000", 2024-05-15T23:16:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.9604999999999993e-05 2024-05-15T23:16:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.576274+0000", 2024-05-15T23:16:33.590 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1034e-05 2024-05-15T23:16:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.576281+0000", 2024-05-15T23:16:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4590000000000001e-06 2024-05-15T23:16:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.579300+0000", 2024-05-15T23:16:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030186340000000001 2024-05-15T23:16:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.579405+0000", 2024-05-15T23:16:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000104922 2024-05-15T23:16:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:16:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.579522+0000", 2024-05-15T23:16:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000117766 2024-05-15T23:16:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.584606+0000", 2024-05-15T23:16:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005083204 2024-05-15T23:16:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:33.592 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:170205 4.c 4:31b4e8b1:::benchmark_data_smithi090_104448_object170204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e319)", 2024-05-15T23:16:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:16:33.576302+0000", 2024-05-15T23:16:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.0098904470000000001, 2024-05-15T23:16:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0099810939999999994, 2024-05-15T23:16:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:16:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:16:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:16:33.593 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 170205 2024-05-15T23:16:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.576302+0000", 2024-05-15T23:16:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.576302+0000", 2024-05-15T23:16:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.576302+0000", 2024-05-15T23:16:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5799999999999998e-07 2024-05-15T23:16:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.576334+0000", 2024-05-15T23:16:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1272000000000001e-05 2024-05-15T23:16:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.576342+0000", 2024-05-15T23:16:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.4740000000000005e-06 2024-05-15T23:16:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.576348+0000", 2024-05-15T23:16:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7479999999999999e-06 2024-05-15T23:16:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.576396+0000", 2024-05-15T23:16:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8185e-05 2024-05-15T23:16:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.576524+0000", 2024-05-15T23:16:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000127723 2024-05-15T23:16:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.576714+0000", 2024-05-15T23:16:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00019010199999999999 2024-05-15T23:16:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop_reply(client.25441.0:170205 4.c e319/283)", 2024-05-15T23:16:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:16:33.586298+0000", 2024-05-15T23:16:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 4294967295.9998941, 2024-05-15T23:16:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1193999999999999e-05, 2024-05-15T23:16:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:33.597 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "queued for pg", 2024-05-15T23:16:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.586298+0000", 2024-05-15T23:16:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.586298+0000", 2024-05-15T23:16:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.586299+0000", 2024-05-15T23:16:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5960000000000001e-06 2024-05-15T23:16:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.586305+0000", 2024-05-15T23:16:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0449999999999997e-06 2024-05-15T23:16:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.586317+0000", 2024-05-15T23:16:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1975e-05 2024-05-15T23:16:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:33.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:33.586325+0000", 2024-05-15T23:16:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6909999999999997e-06 2024-05-15T23:16:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2024-05-15T23:16:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 3 2024-05-15T23:16:33.600 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:16:33.601 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:33.626 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:16:33.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:33.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:16:33.732 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:16:33.771 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:33.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:33 smithi090 ceph-mon[44434]: 4.9 scrub ok 2024-05-15T23:16:33.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:33 smithi090 ceph-mon[44434]: pgmap v875: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 10 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T23:16:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:33 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:16:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:33 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:16:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:33 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/464435936' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:16:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:33 smithi090 ceph-mon[44434]: osdmap e319: 12 total, 12 up, 11 in 2024-05-15T23:16:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:33 smithi090 ceph-mon[44434]: 4.f scrub ok 2024-05-15T23:16:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:33 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2935881755' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:16:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:33 smithi090 ceph-mon[44434]: from='client.35639 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:16:33.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:33 smithi084 ceph-mon[39784]: 4.9 scrub ok 2024-05-15T23:16:33.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:33 smithi084 ceph-mon[39784]: pgmap v875: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 10 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T23:16:33.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:33 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:16:33.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:33 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:16:33.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:33 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/464435936' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:16:33.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:33 smithi084 ceph-mon[39784]: osdmap e319: 12 total, 12 up, 11 in 2024-05-15T23:16:33.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:33 smithi084 ceph-mon[39784]: 4.f scrub ok 2024-05-15T23:16:33.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:33 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2935881755' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:16:33.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:33 smithi084 ceph-mon[39784]: from='client.35639 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:16:33.879 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:16:33.882 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:33 smithi088 ceph-mon[44699]: 4.9 scrub ok 2024-05-15T23:16:33.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:33 smithi088 ceph-mon[44699]: pgmap v875: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 10 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T23:16:33.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:33 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:16:33.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:33 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:16:33.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:33 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/464435936' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:16:33.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:33 smithi088 ceph-mon[44699]: osdmap e319: 12 total, 12 up, 11 in 2024-05-15T23:16:33.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:33 smithi088 ceph-mon[44699]: 4.f scrub ok 2024-05-15T23:16:33.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:33 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2935881755' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:16:33.883 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:33 smithi088 ceph-mon[44699]: from='client.35639 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:16:33.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:33.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:16:33.956 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2024-05-15T23:16:33.984 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:16:34.145 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:34.350 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 88 16 171045 171029 121.455 66.4375 0.0176781 0.00823218 2024-05-15T23:16:34.501 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:16:34.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:34.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:16:34.529 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T23:16:34.606 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:16:34.669 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:34.702 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:16:34.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:34.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:16:34.811 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:16:34.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:34 smithi088 ceph-mon[44699]: 4.1 scrub starts 2024-05-15T23:16:34.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:34 smithi088 ceph-mon[44699]: from='client.35639 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:16:34.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:34 smithi088 ceph-mon[44699]: osdmap e320: 12 total, 12 up, 11 in 2024-05-15T23:16:34.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:34 smithi088 ceph-mon[44699]: 4.2 scrub ok 2024-05-15T23:16:34.858 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:34.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:34 smithi084 ceph-mon[39784]: 4.1 scrub starts 2024-05-15T23:16:34.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:34 smithi084 ceph-mon[39784]: from='client.35639 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:16:34.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:34 smithi084 ceph-mon[39784]: osdmap e320: 12 total, 12 up, 11 in 2024-05-15T23:16:34.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:34 smithi084 ceph-mon[39784]: 4.2 scrub ok 2024-05-15T23:16:34.894 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:16:34.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:34.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:16:34.953 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:16:34.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 5 1 2024-05-15T23:16:35.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:16:35.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:34 smithi090 ceph-mon[44434]: 4.1 scrub starts 2024-05-15T23:16:35.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:34 smithi090 ceph-mon[44434]: from='client.35639 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:16:35.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:34 smithi090 ceph-mon[44434]: osdmap e320: 12 total, 12 up, 11 in 2024-05-15T23:16:35.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:34 smithi090 ceph-mon[44434]: 4.2 scrub ok 2024-05-15T23:16:35.083 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:35.350 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 89 16 172140 172124 120.859 68.4375 0.00954729 0.00827314 2024-05-15T23:16:35.381 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:16:35.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:35.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:16:35.487 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:35.487 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:16:35.537 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:35.758 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:35 smithi084 ceph-mon[39784]: 4.7 scrub ok 2024-05-15T23:16:35.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:35 smithi088 ceph-mon[44699]: 4.7 scrub ok 2024-05-15T23:16:36.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:35 smithi090 ceph-mon[44434]: 4.7 scrub ok 2024-05-15T23:16:36.092 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:16:36.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:36.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:16:36.200 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:16:36.233 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:36.350 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 90 13 173235 173222 120.279 68.625 0.00979045 0.00831223 2024-05-15T23:16:36.442 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:16:36.485 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:16:36.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:36.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:16:36.588 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:16:36.616 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:36.753 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:36 smithi088 ceph-mon[44699]: 4.3 scrub starts 2024-05-15T23:16:36.753 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:36 smithi088 ceph-mon[44699]: pgmap v878: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 10 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 52 MiB/s wr, 824 op/s 2024-05-15T23:16:36.753 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:36 smithi088 ceph-mon[44699]: 4.1 scrub ok 2024-05-15T23:16:36.753 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3228670337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:36.753 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:36.765 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:16:36.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:36.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:16:36.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:16:36.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:36 smithi084 ceph-mon[39784]: 4.3 scrub starts 2024-05-15T23:16:36.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:36 smithi084 ceph-mon[39784]: pgmap v878: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 10 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 52 MiB/s wr, 824 op/s 2024-05-15T23:16:36.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:36 smithi084 ceph-mon[39784]: 4.1 scrub ok 2024-05-15T23:16:36.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3228670337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:36.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:36.898 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:36 smithi090 ceph-mon[44434]: 4.3 scrub starts 2024-05-15T23:16:36.898 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:36 smithi090 ceph-mon[44434]: pgmap v878: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 10 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 52 MiB/s wr, 824 op/s 2024-05-15T23:16:36.898 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:36 smithi090 ceph-mon[44434]: 4.1 scrub ok 2024-05-15T23:16:36.898 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3228670337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:36.898 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:36.900 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:36.922 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:16:36.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:36.925 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:16:37.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:37.030 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:16:37.030 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:16:37.054 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:37.073 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:37.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:37.076 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:37.135 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:16:37.135 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:16:37.136 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:16:37.136 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:16:37.136 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:16:37.136 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:16:37.136 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:16:37.136 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:16:37.136 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:16:37.136 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:16:37.136 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:16:37.136 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:16:37.136 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:16:37.136 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:16:37.137 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:16:37.137 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:16:37.137 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:16:37.137 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:16:37.137 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:16:37.137 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:16:37.137 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:16:37.137 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:16:37.137 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:16:37.137 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:16:37.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:37.204 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:37.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:37.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:37.328 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:16:37.350 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total time run: 90.0126 2024-05-15T23:16:37.350 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total writes made: 173235 2024-05-15T23:16:37.350 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Write size: 65536 2024-05-15T23:16:37.350 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object size: 65536 2024-05-15T23:16:37.351 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Bandwidth (MB/sec): 120.285 2024-05-15T23:16:37.351 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Bandwidth: 62.4112 2024-05-15T23:16:37.351 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max bandwidth (MB/sec): 367.875 2024-05-15T23:16:37.351 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min bandwidth (MB/sec): 57.5625 2024-05-15T23:16:37.351 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average IOPS: 1924 2024-05-15T23:16:37.351 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev IOPS: 998.579 2024-05-15T23:16:37.351 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max IOPS: 5886 2024-05-15T23:16:37.351 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min IOPS: 921 2024-05-15T23:16:37.351 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average Latency(s): 0.00831274 2024-05-15T23:16:37.351 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Latency(s): 0.00626264 2024-05-15T23:16:37.351 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max latency(s): 0.215284 2024-05-15T23:16:37.351 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min latency(s): 0.00102037 2024-05-15T23:16:37.357 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:37.359 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T23:16:37.441 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:16:37.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:37.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:16:37.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:37.582 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:37.604 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:37.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:37.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:37.709 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:16:37.749 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:37.800 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T23:16:37.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:37.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:37.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:37.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:37.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:37.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:37.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:37.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:37.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:37.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:37.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:37.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:37 smithi084 ceph-mon[39784]: 4.3 scrub ok 2024-05-15T23:16:37.886 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:16:37.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:37.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:16:37.994 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:16:38.025 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.025 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.025 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.025 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.025 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.026 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.026 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.026 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.026 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.026 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.026 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.026 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:37 smithi088 ceph-mon[44699]: 4.3 scrub ok 2024-05-15T23:16:38.028 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:38.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:37 smithi090 ceph-mon[44434]: 4.3 scrub ok 2024-05-15T23:16:38.074 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:16:38.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:38.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:16:38.174 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:16:38.183 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:16:38.214 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:38.474 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:38.481 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:16:38.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:38.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:16:38.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:16:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:38.627 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-05-15T23:16:38.648 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:16:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:38.657 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:16:38.738 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:38 smithi088 ceph-mon[44699]: pgmap v879: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-05-15T23:16:38.738 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:38 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3619974240' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:16:38.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:38.788 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:38.789 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:38 smithi084 ceph-mon[39784]: pgmap v879: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-05-15T23:16:38.789 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:38 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3619974240' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:16:38.810 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:38.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:38.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:38.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:16:38.959 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:39.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:38 smithi090 ceph-mon[44434]: pgmap v879: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-05-15T23:16:39.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:38 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3619974240' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:16:39.421 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:16:39.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:39.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:16:39.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:16:39.564 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:39.589 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:16:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:39.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:16:39.697 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:16:39.725 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3619974240' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:16:39.725 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:39 smithi084 ceph-mon[39784]: osdmap e321: 12 total, 12 up, 11 in 2024-05-15T23:16:39.731 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:39.937 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:16:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:39.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:16:40.001 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3619974240' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:16:40.001 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:39 smithi088 ceph-mon[44699]: osdmap e321: 12 total, 12 up, 11 in 2024-05-15T23:16:40.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:16:40.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3619974240' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:16:40.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:39 smithi090 ceph-mon[44434]: osdmap e321: 12 total, 12 up, 11 in 2024-05-15T23:16:40.095 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:40.122 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:16:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:40.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:16:40.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:16:40.266 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:40.288 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:16:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:40.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:16:40.331 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:16:40.331 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:16:40.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:16:40.434 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:40.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:16:40.639 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:40 smithi088 ceph-mon[44699]: pgmap v881: 17 pgs: 17 active+clean; 11 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 50 MiB/s wr, 818 op/s 2024-05-15T23:16:40.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 6 1 2024-05-15T23:16:40.791 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:16:40.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:40.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:16:40.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:40.946 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:40 smithi090 ceph-mon[44434]: pgmap v881: 17 pgs: 17 active+clean; 11 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 50 MiB/s wr, 818 op/s 2024-05-15T23:16:40.948 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:40.980 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:40.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:40.984 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:41.070 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:40 smithi084 ceph-mon[39784]: pgmap v881: 17 pgs: 17 active+clean; 11 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 50 MiB/s wr, 818 op/s 2024-05-15T23:16:41.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:16:41.111 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T23:16:41.124 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:41.149 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:16:41.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:41.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:16:41.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:16:41.298 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:41.388 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:16:41.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:41.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:16:41.430 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:41.453 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:41.456 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:16:41.498 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:16:41.532 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:41.655 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:41 smithi084 ceph-mon[39784]: 4.8 scrub starts 2024-05-15T23:16:41.657 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:16:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:41.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:16:41.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:16:41.802 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:41 smithi088 ceph-mon[44699]: 4.8 scrub starts 2024-05-15T23:16:41.802 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:41.828 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:16:41.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:41.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:16:41.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:16:41.960 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:41 smithi090 ceph-mon[44434]: 4.8 scrub starts 2024-05-15T23:16:41.962 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:41.987 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:16:41.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:41.990 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:16:42.091 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:16:42.128 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:42.386 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:16:42.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:42.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:16:42.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:42.522 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:42.550 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:42.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:42.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:16:42.688 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:16:42.696 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:42.726 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:16:42.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:42.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:16:42.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:42 smithi090 ceph-mon[44434]: pgmap v882: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 128 B/s rd, 44 MiB/s wr, 991 op/s 2024-05-15T23:16:42.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:42 smithi090 ceph-mon[44434]: 4.8 scrub ok 2024-05-15T23:16:42.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:42 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3143738759' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:16:42.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:42 smithi090 ceph-mon[44434]: from='client.25558 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:16:42.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:16:42.865 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:42.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:42 smithi084 ceph-mon[39784]: pgmap v882: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 128 B/s rd, 44 MiB/s wr, 991 op/s 2024-05-15T23:16:42.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:42 smithi084 ceph-mon[39784]: 4.8 scrub ok 2024-05-15T23:16:42.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:42 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3143738759' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:16:42.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:42 smithi084 ceph-mon[39784]: from='client.25558 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:16:42.891 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:16:42.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:42.896 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:16:42.975 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:42 smithi088 ceph-mon[44699]: pgmap v882: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 128 B/s rd, 44 MiB/s wr, 991 op/s 2024-05-15T23:16:42.975 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:42 smithi088 ceph-mon[44699]: 4.8 scrub ok 2024-05-15T23:16:42.975 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:42 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3143738759' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:16:42.976 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:42 smithi088 ceph-mon[44699]: from='client.25558 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:16:42.996 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:16:43.031 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:43.058 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:16:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:43.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:16:43.168 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:16:43.201 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:43.354 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:16:43.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:43.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:16:43.462 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:16:43.497 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:43.527 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:16:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:43.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:16:43.632 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:16:43.683 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:43.695 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:43 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:16:43.695 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:43 smithi088 ceph-mon[44699]: from='client.25558 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:16:43.695 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:43 smithi088 ceph-mon[44699]: osdmap e322: 12 total, 12 up, 11 in 2024-05-15T23:16:43.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:43 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:16:43.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:43 smithi084 ceph-mon[39784]: from='client.25558 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:16:43.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:43 smithi084 ceph-mon[39784]: osdmap e322: 12 total, 12 up, 11 in 2024-05-15T23:16:43.967 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:16:43.967 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:16:44.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:43 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:16:44.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:43 smithi090 ceph-mon[44434]: from='client.25558 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:16:44.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:43 smithi090 ceph-mon[44434]: osdmap e322: 12 total, 12 up, 11 in 2024-05-15T23:16:44.062 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:16:44.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:44.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:16:44.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:16:44.213 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:44.246 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:16:44.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:44.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:16:44.350 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:16:44.384 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:44.818 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T23:16:44.847 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:16:44.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:44.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:16:44.938 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:44 smithi084 ceph-mon[39784]: pgmap v884: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 127 B/s rd, 34 MiB/s wr, 1.12k op/s 2024-05-15T23:16:44.954 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:16:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:44.987 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:44 smithi088 ceph-mon[44699]: pgmap v884: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 127 B/s rd, 34 MiB/s wr, 1.12k op/s 2024-05-15T23:16:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:16:45.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:45.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:16:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:44 smithi090 ceph-mon[44434]: pgmap v884: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 127 B/s rd, 34 MiB/s wr, 1.12k op/s 2024-05-15T23:16:45.113 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:16:45.158 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:45.191 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:16:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:45.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:16:45.230 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:16:45.297 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:16:45.330 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:45.485 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:16:45.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:45.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:16:45.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:16:45.632 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:45.658 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:16:45.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:45.664 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:16:45.708 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2362134202' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:45.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:45.798 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:45.829 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:45.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:45.832 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:45.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:45.960 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2362134202' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:45.962 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:45.993 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:45.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:46.097 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:16:46.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2362134202' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:46.144 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:46.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:16:46.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:16:46.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:16:46.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:16:46.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:16:46.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:16:46.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:16:46.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:16:46.418 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:16:46.418 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:16:46.418 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:16:46.418 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:16:46.418 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:16:46.418 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:16:46.418 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:16:46.418 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:16:46.418 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:16:46.418 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:16:46.418 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:16:46.418 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:16:46.419 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:16:46.419 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:16:46.419 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:16:46.419 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:16:46.691 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:16:46.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:46 smithi088 ceph-mon[44699]: pgmap v885: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 127 B/s rd, 18 MiB/s wr, 1.05k op/s 2024-05-15T23:16:46.974 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:46.974 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:46.974 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:46.974 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:46.974 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:46.974 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:46.974 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:46.974 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:46.974 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:46.974 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:46.974 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:46 smithi090 ceph-mon[44434]: pgmap v885: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 127 B/s rd, 18 MiB/s wr, 1.05k op/s 2024-05-15T23:16:47.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:46 smithi084 ceph-mon[39784]: pgmap v885: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 127 B/s rd, 18 MiB/s wr, 1.05k op/s 2024-05-15T23:16:47.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:47.300 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:16:47.300 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:16:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:16:47.301 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:16:47.301 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:16:48.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:48.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:47 smithi090 ceph-mon[44434]: pgmap v886: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 123 B/s rd, 6.7 MiB/s wr, 1.09k op/s 2024-05-15T23:16:48.053 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:48.053 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:47 smithi088 ceph-mon[44699]: pgmap v886: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 123 B/s rd, 6.7 MiB/s wr, 1.09k op/s 2024-05-15T23:16:48.088 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T23:16:48.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:48.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:47 smithi084 ceph-mon[39784]: pgmap v886: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 123 B/s rd, 6.7 MiB/s wr, 1.09k op/s 2024-05-15T23:16:48.457 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:16:48.701 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:16:48.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:48.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:16:48.806 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:16:48.838 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:49.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:48 smithi090 ceph-mon[44434]: pgmap v887: 17 pgs: 17 active+clean; 9.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 102 B/s rd, 5.5 MiB/s wr, 1.15k op/s 2024-05-15T23:16:49.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:48 smithi088 ceph-mon[44699]: pgmap v887: 17 pgs: 17 active+clean; 9.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 102 B/s rd, 5.5 MiB/s wr, 1.15k op/s 2024-05-15T23:16:49.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:48 smithi084 ceph-mon[39784]: pgmap v887: 17 pgs: 17 active+clean; 9.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 102 B/s rd, 5.5 MiB/s wr, 1.15k op/s 2024-05-15T23:16:49.593 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:16:49.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:49.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:16:49.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:16:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:49.769 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:16:49.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:49.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:16:49.877 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:16:49.913 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:49.935 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:16:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:49.941 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:16:50.042 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:16:50.083 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:50.939 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:16:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:50.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:16:50.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:50 smithi088 ceph-mon[44699]: pgmap v888: 17 pgs: 17 active+clean; 9.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:16:51.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:16:51.079 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:51.176 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:16:51.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:51.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:16:51.198 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:50 smithi084 ceph-mon[39784]: pgmap v888: 17 pgs: 17 active+clean; 9.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:16:51.298 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:16:51.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:50 smithi090 ceph-mon[44434]: pgmap v888: 17 pgs: 17 active+clean; 9.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:16:51.334 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:51.376 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:16:51.376 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:16:51.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:16:51.376 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:16:51.376 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.376 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119417 2.3 2:cf4ffc0c:::benchmark_data_smithi090_78160_object119416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:16:51.376 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.178575+0000", 2024-05-15T23:16:51.376 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 591.19391587200005, 2024-05-15T23:16:51.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.079319406999999995, 2024-05-15T23:16:51.377 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.377 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.377 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.377 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.377 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:16:51.377 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:16:51.377 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119417 2024-05-15T23:16:51.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.377 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.377 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.377 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178575+0000", 2024-05-15T23:16:51.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178575+0000", 2024-05-15T23:16:51.378 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.378 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178577+0000", 2024-05-15T23:16:51.378 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9700000000000002e-06 2024-05-15T23:16:51.378 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178632+0000", 2024-05-15T23:16:51.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5040000000000002e-05 2024-05-15T23:16:51.379 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.379 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.379 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178642+0000", 2024-05-15T23:16:51.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0397999999999999e-05 2024-05-15T23:16:51.379 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.379 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.379 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178649+0000", 2024-05-15T23:16:51.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6019999999999996e-06 2024-05-15T23:16:51.379 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.379 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.380 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178700+0000", 2024-05-15T23:16:51.380 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1292000000000003e-05 2024-05-15T23:16:51.380 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.380 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.380 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178792+0000", 2024-05-15T23:16:51.380 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2165999999999998e-05 2024-05-15T23:16:51.380 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.380 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:16:51.380 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.178886+0000", 2024-05-15T23:16:51.380 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3111999999999999e-05 2024-05-15T23:16:51.380 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.181534+0000", 2024-05-15T23:16:51.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026488869999999999 2024-05-15T23:16:51.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.257736+0000", 2024-05-15T23:16:51.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.076201181000000007 2024-05-15T23:16:51.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.257843+0000", 2024-05-15T23:16:51.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00010730099999999999 2024-05-15T23:16:51.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.257894+0000", 2024-05-15T23:16:51.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1458000000000001e-05 2024-05-15T23:16:51.382 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.382 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.382 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.382 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119448 2.3 2:c91dd160:::benchmark_data_smithi090_78160_object119447:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:16:51.382 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.194884+0000", 2024-05-15T23:16:51.382 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 591.177606459, 2024-05-15T23:16:51.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.071121264000000003, 2024-05-15T23:16:51.382 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.383 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.383 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.383 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.383 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:16:51.383 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:16:51.383 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119448 2024-05-15T23:16:51.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.383 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194884+0000", 2024-05-15T23:16:51.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194884+0000", 2024-05-15T23:16:51.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194885+0000", 2024-05-15T23:16:51.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.04e-06 2024-05-15T23:16:51.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194946+0000", 2024-05-15T23:16:51.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0918000000000002e-05 2024-05-15T23:16:51.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194955+0000", 2024-05-15T23:16:51.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6929999999999996e-06 2024-05-15T23:16:51.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.194962+0000", 2024-05-15T23:16:51.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8689999999999998e-06 2024-05-15T23:16:51.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198056+0000", 2024-05-15T23:16:51.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030938900000000002 2024-05-15T23:16:51.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.386 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.386 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.386 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198180+0000", 2024-05-15T23:16:51.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012426000000000001 2024-05-15T23:16:51.386 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.386 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.386 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:16:51.386 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.198279+0000", 2024-05-15T23:16:51.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8665999999999994e-05 2024-05-15T23:16:51.386 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.386 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.386 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.386 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.200997+0000", 2024-05-15T23:16:51.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027182339999999999 2024-05-15T23:16:51.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.265881+0000", 2024-05-15T23:16:51.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.064884366999999998 2024-05-15T23:16:51.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.265938+0000", 2024-05-15T23:16:51.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6895999999999999e-05 2024-05-15T23:16:51.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.266006+0000", 2024-05-15T23:16:51.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7430999999999999e-05 2024-05-15T23:16:51.388 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.388 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.388 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.388 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:119727 2.3 2:ca0d49b0:::benchmark_data_smithi090_78160_object119726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-05-15T23:16:51.388 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:07:00.453742+0000", 2024-05-15T23:16:51.388 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 590.91874889799999, 2024-05-15T23:16:51.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070952512999999995, 2024-05-15T23:16:51.388 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.388 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.388 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.388 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.388 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:16:51.388 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:16:51.389 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 119727 2024-05-15T23:16:51.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.389 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.453742+0000", 2024-05-15T23:16:51.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.453742+0000", 2024-05-15T23:16:51.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.453744+0000", 2024-05-15T23:16:51.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.781e-06 2024-05-15T23:16:51.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.453812+0000", 2024-05-15T23:16:51.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8233999999999998e-05 2024-05-15T23:16:51.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.453823+0000", 2024-05-15T23:16:51.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1158999999999999e-05 2024-05-15T23:16:51.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.453829+0000", 2024-05-15T23:16:51.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.173e-06 2024-05-15T23:16:51.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.456675+0000", 2024-05-15T23:16:51.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002846094 2024-05-15T23:16:51.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.456802+0000", 2024-05-15T23:16:51.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012648100000000001 2024-05-15T23:16:51.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 2", 2024-05-15T23:16:51.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.456946+0000", 2024-05-15T23:16:51.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000144449 2024-05-15T23:16:51.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.461103+0000", 2024-05-15T23:16:51.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0041565229999999996 2024-05-15T23:16:51.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.524622+0000", 2024-05-15T23:16:51.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063519640000000002 2024-05-15T23:16:51.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.524668+0000", 2024-05-15T23:16:51.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5753000000000001e-05 2024-05-15T23:16:51.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:07:00.524694+0000", 2024-05-15T23:16:51.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6225999999999999e-05 2024-05-15T23:16:51.393 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.393 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.393 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.394 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85095 3.5 3:a5452822:::benchmark_data_smithi090_90974_object85094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.394 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:22.985580+0000", 2024-05-15T23:16:51.394 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.38691110299999, 2024-05-15T23:16:51.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21426990000000001, 2024-05-15T23:16:51.394 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.394 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.394 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.394 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.394 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.394 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.394 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85095 2024-05-15T23:16:51.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.395 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.985580+0000", 2024-05-15T23:16:51.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.985580+0000", 2024-05-15T23:16:51.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.985581+0000", 2024-05-15T23:16:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5340000000000001e-06 2024-05-15T23:16:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.985637+0000", 2024-05-15T23:16:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5818e-05 2024-05-15T23:16:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.985647+0000", 2024-05-15T23:16:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.011e-05 2024-05-15T23:16:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.985653+0000", 2024-05-15T23:16:51.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7100000000000004e-06 2024-05-15T23:16:51.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.985711+0000", 2024-05-15T23:16:51.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8579999999999998e-05 2024-05-15T23:16:51.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.985841+0000", 2024-05-15T23:16:51.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012981599999999999 2024-05-15T23:16:51.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.985983+0000", 2024-05-15T23:16:51.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000141433 2024-05-15T23:16:51.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.989753+0000", 2024-05-15T23:16:51.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037700400000000001 2024-05-15T23:16:51.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.199766+0000", 2024-05-15T23:16:51.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21001347100000001 2024-05-15T23:16:51.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.199805+0000", 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9059e-05 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.199850+0000", 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4329000000000002e-05 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85096 3.5 3:a46c191e:::benchmark_data_smithi090_90974_object85095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.399 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:22.986657+0000", 2024-05-15T23:16:51.400 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.38583374000001, 2024-05-15T23:16:51.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.219423535, 2024-05-15T23:16:51.400 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.400 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.400 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.400 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.400 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.400 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.400 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85096 2024-05-15T23:16:51.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.400 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.986657+0000", 2024-05-15T23:16:51.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.986657+0000", 2024-05-15T23:16:51.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.986658+0000", 2024-05-15T23:16:51.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0100000000000003e-07 2024-05-15T23:16:51.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.986720+0000", 2024-05-15T23:16:51.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1925000000000001e-05 2024-05-15T23:16:51.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.986726+0000", 2024-05-15T23:16:51.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4819999999999999e-06 2024-05-15T23:16:51.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.986731+0000", 2024-05-15T23:16:51.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5959999999999997e-06 2024-05-15T23:16:51.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.988047+0000", 2024-05-15T23:16:51.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001316106 2024-05-15T23:16:51.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.988168+0000", 2024-05-15T23:16:51.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0001209 2024-05-15T23:16:51.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.988318+0000", 2024-05-15T23:16:51.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149937 2024-05-15T23:16:51.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.991338+0000", 2024-05-15T23:16:51.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030202010000000001 2024-05-15T23:16:51.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.206021+0000", 2024-05-15T23:16:51.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21468338100000001 2024-05-15T23:16:51.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.206054+0000", 2024-05-15T23:16:51.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2679999999999999e-05 2024-05-15T23:16:51.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.405 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.206081+0000", 2024-05-15T23:16:51.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6426000000000001e-05 2024-05-15T23:16:51.405 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.405 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.405 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.405 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85100 3.5 3:a40fd042:::benchmark_data_smithi090_90974_object85099:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.405 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:22.989223+0000", 2024-05-15T23:16:51.405 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.383268198, 2024-05-15T23:16:51.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.216943568, 2024-05-15T23:16:51.405 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.405 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.405 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.406 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.406 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.406 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.406 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85100 2024-05-15T23:16:51.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.406 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.989223+0000", 2024-05-15T23:16:51.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.989223+0000", 2024-05-15T23:16:51.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.989224+0000", 2024-05-15T23:16:51.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4759999999999999e-06 2024-05-15T23:16:51.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.989274+0000", 2024-05-15T23:16:51.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9808999999999997e-05 2024-05-15T23:16:51.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.989286+0000", 2024-05-15T23:16:51.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1712000000000001e-05 2024-05-15T23:16:51.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.989293+0000", 2024-05-15T23:16:51.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8169999999999999e-06 2024-05-15T23:16:51.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.989779+0000", 2024-05-15T23:16:51.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00048537299999999999 2024-05-15T23:16:51.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.989904+0000", 2024-05-15T23:16:51.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000125425 2024-05-15T23:16:51.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.990045+0000", 2024-05-15T23:16:51.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014052100000000001 2024-05-15T23:16:51.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.991442+0000", 2024-05-15T23:16:51.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013976559999999999 2024-05-15T23:16:51.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.206109+0000", 2024-05-15T23:16:51.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21466665100000001 2024-05-15T23:16:51.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.206141+0000", 2024-05-15T23:16:51.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1579999999999999e-05 2024-05-15T23:16:51.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.206166+0000", 2024-05-15T23:16:51.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5548e-05 2024-05-15T23:16:51.411 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.411 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.411 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.411 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85109 3.5 3:af118299:::benchmark_data_smithi090_90974_object85108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.411 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:22.996267+0000", 2024-05-15T23:16:51.411 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.37622373400001, 2024-05-15T23:16:51.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21387199300000001, 2024-05-15T23:16:51.411 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.411 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.411 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.411 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.411 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.412 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.412 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85109 2024-05-15T23:16:51.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.412 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.996267+0000", 2024-05-15T23:16:51.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.996267+0000", 2024-05-15T23:16:51.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.996269+0000", 2024-05-15T23:16:51.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0839999999999999e-06 2024-05-15T23:16:51.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.996314+0000", 2024-05-15T23:16:51.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5237000000000001e-05 2024-05-15T23:16:51.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.996326+0000", 2024-05-15T23:16:51.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1209999999999999e-05 2024-05-15T23:16:51.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.996332+0000", 2024-05-15T23:16:51.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.117e-06 2024-05-15T23:16:51.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.996385+0000", 2024-05-15T23:16:51.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3130000000000001e-05 2024-05-15T23:16:51.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.996520+0000", 2024-05-15T23:16:51.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013498000000000001 2024-05-15T23:16:51.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.996665+0000", 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000145656 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.000825+0000", 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0041599920000000004 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.210038+0000", 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20921209700000001 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.210096+0000", 2024-05-15T23:16:51.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8023999999999999e-05 2024-05-15T23:16:51.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.210139+0000", 2024-05-15T23:16:51.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3466e-05 2024-05-15T23:16:51.416 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.416 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.416 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.416 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85112 3.5 3:a4ba3de9:::benchmark_data_smithi090_90974_object85111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.417 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:22.996995+0000", 2024-05-15T23:16:51.417 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.375495306, 2024-05-15T23:16:51.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21326978199999999, 2024-05-15T23:16:51.417 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.417 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.417 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.417 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.417 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.417 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.417 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85112 2024-05-15T23:16:51.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.417 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.996995+0000", 2024-05-15T23:16:51.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.996995+0000", 2024-05-15T23:16:51.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.996997+0000", 2024-05-15T23:16:51.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4929999999999999e-06 2024-05-15T23:16:51.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.997042+0000", 2024-05-15T23:16:51.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5349000000000002e-05 2024-05-15T23:16:51.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.997057+0000", 2024-05-15T23:16:51.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4617000000000001e-05 2024-05-15T23:16:51.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.997065+0000", 2024-05-15T23:16:51.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7519999999999996e-06 2024-05-15T23:16:51.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.999146+0000", 2024-05-15T23:16:51.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020813889999999999 2024-05-15T23:16:51.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.999248+0000", 2024-05-15T23:16:51.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000101884 2024-05-15T23:16:51.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.999336+0000", 2024-05-15T23:16:51.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8503000000000003e-05 2024-05-15T23:16:51.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.002477+0000", 2024-05-15T23:16:51.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031410069999999999 2024-05-15T23:16:51.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.210181+0000", 2024-05-15T23:16:51.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20770312699999999 2024-05-15T23:16:51.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.210229+0000", 2024-05-15T23:16:51.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8878999999999997e-05 2024-05-15T23:16:51.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.210265+0000", 2024-05-15T23:16:51.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5781999999999997e-05 2024-05-15T23:16:51.422 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.422 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.422 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.422 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85114 3.5 3:a4d1d3fa:::benchmark_data_smithi090_90974_object85113:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.422 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:22.998057+0000", 2024-05-15T23:16:51.422 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.37443333800002, 2024-05-15T23:16:51.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21233644800000001, 2024-05-15T23:16:51.422 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.422 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.423 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.423 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.423 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.423 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.423 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85114 2024-05-15T23:16:51.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.423 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.998057+0000", 2024-05-15T23:16:51.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.998057+0000", 2024-05-15T23:16:51.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.998059+0000", 2024-05-15T23:16:51.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.353e-06 2024-05-15T23:16:51.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.998123+0000", 2024-05-15T23:16:51.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3696999999999997e-05 2024-05-15T23:16:51.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.998132+0000", 2024-05-15T23:16:51.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7739999999999999e-06 2024-05-15T23:16:51.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:22.998139+0000", 2024-05-15T23:16:51.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9450000000000004e-06 2024-05-15T23:16:51.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.000846+0000", 2024-05-15T23:16:51.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027064910000000001 2024-05-15T23:16:51.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.000931+0000", 2024-05-15T23:16:51.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5224999999999998e-05 2024-05-15T23:16:51.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.001023+0000", 2024-05-15T23:16:51.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2052000000000006e-05 2024-05-15T23:16:51.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.002629+0000", 2024-05-15T23:16:51.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016058380000000001 2024-05-15T23:16:51.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.210303+0000", 2024-05-15T23:16:51.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20767428900000001 2024-05-15T23:16:51.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.210354+0000", 2024-05-15T23:16:51.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0522999999999997e-05 2024-05-15T23:16:51.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.210394+0000", 2024-05-15T23:16:51.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0261e-05 2024-05-15T23:16:51.427 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.427 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.427 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.428 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85124 3.c 3:354f1164:::benchmark_data_smithi090_90974_object85123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.428 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:23.007373+0000", 2024-05-15T23:16:51.428 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.36511812399999, 2024-05-15T23:16:51.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19255631200000001, 2024-05-15T23:16:51.428 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.428 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.428 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.428 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.428 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.428 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.428 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85124 2024-05-15T23:16:51.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.428 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.007373+0000", 2024-05-15T23:16:51.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.007373+0000", 2024-05-15T23:16:51.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.007376+0000", 2024-05-15T23:16:51.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.67e-06 2024-05-15T23:16:51.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.007427+0000", 2024-05-15T23:16:51.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0723999999999997e-05 2024-05-15T23:16:51.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.007439+0000", 2024-05-15T23:16:51.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1476e-05 2024-05-15T23:16:51.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.007445+0000", 2024-05-15T23:16:51.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2230000000000004e-06 2024-05-15T23:16:51.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.007502+0000", 2024-05-15T23:16:51.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7289e-05 2024-05-15T23:16:51.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.007624+0000", 2024-05-15T23:16:51.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012208400000000001 2024-05-15T23:16:51.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.007766+0000", 2024-05-15T23:16:51.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000142311 2024-05-15T23:16:51.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.012509+0000", 2024-05-15T23:16:51.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0047421080000000001 2024-05-15T23:16:51.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.199858+0000", 2024-05-15T23:16:51.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18734936899999999 2024-05-15T23:16:51.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.199898+0000", 2024-05-15T23:16:51.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0383999999999998e-05 2024-05-15T23:16:51.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.199929+0000", 2024-05-15T23:16:51.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0673999999999998e-05 2024-05-15T23:16:51.433 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.433 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.433 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.433 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85126 3.c 3:37b33103:::benchmark_data_smithi090_90974_object85125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.433 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:23.008200+0000", 2024-05-15T23:16:51.433 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.36429084500003, 2024-05-15T23:16:51.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.191822311, 2024-05-15T23:16:51.434 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.434 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.434 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.434 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.434 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.434 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.434 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85126 2024-05-15T23:16:51.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.434 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.008200+0000", 2024-05-15T23:16:51.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.008200+0000", 2024-05-15T23:16:51.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.008201+0000", 2024-05-15T23:16:51.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3e-06 2024-05-15T23:16:51.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.008239+0000", 2024-05-15T23:16:51.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7737999999999997e-05 2024-05-15T23:16:51.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.008249+0000", 2024-05-15T23:16:51.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0022e-05 2024-05-15T23:16:51.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.008255+0000", 2024-05-15T23:16:51.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4979999999999997e-06 2024-05-15T23:16:51.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.010396+0000", 2024-05-15T23:16:51.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002141586 2024-05-15T23:16:51.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.010484+0000", 2024-05-15T23:16:51.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7874000000000006e-05 2024-05-15T23:16:51.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.010634+0000", 2024-05-15T23:16:51.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014971200000000001 2024-05-15T23:16:51.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.012657+0000", 2024-05-15T23:16:51.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020233080000000001 2024-05-15T23:16:51.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.199968+0000", 2024-05-15T23:16:51.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.187310685 2024-05-15T23:16:51.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.199998+0000", 2024-05-15T23:16:51.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0516000000000001e-05 2024-05-15T23:16:51.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.200022+0000", 2024-05-15T23:16:51.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4071999999999999e-05 2024-05-15T23:16:51.438 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.438 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.439 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.439 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85135 3.5 3:a2cdbf1a:::benchmark_data_smithi090_90974_object85134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.439 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:23.013516+0000", 2024-05-15T23:16:51.439 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.35897494400001, 2024-05-15T23:16:51.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20317201900000001, 2024-05-15T23:16:51.439 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.439 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.439 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.439 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.439 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.439 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.439 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85135 2024-05-15T23:16:51.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.440 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.013516+0000", 2024-05-15T23:16:51.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.013516+0000", 2024-05-15T23:16:51.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.013524+0000", 2024-05-15T23:16:51.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7249999999999997e-06 2024-05-15T23:16:51.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.013582+0000", 2024-05-15T23:16:51.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8118000000000002e-05 2024-05-15T23:16:51.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.013595+0000", 2024-05-15T23:16:51.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3781000000000001e-05 2024-05-15T23:16:51.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.013604+0000", 2024-05-15T23:16:51.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.9830000000000002e-06 2024-05-15T23:16:51.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.013659+0000", 2024-05-15T23:16:51.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4619e-05 2024-05-15T23:16:51.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.013807+0000", 2024-05-15T23:16:51.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000148386 2024-05-15T23:16:51.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.013957+0000", 2024-05-15T23:16:51.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014942800000000001 2024-05-15T23:16:51.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.016235+0000", 2024-05-15T23:16:51.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0022785370000000002 2024-05-15T23:16:51.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.216616+0000", 2024-05-15T23:16:51.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20038014300000001 2024-05-15T23:16:51.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.216657+0000", 2024-05-15T23:16:51.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1050000000000002e-05 2024-05-15T23:16:51.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.216688+0000", 2024-05-15T23:16:51.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1248999999999998e-05 2024-05-15T23:16:51.444 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.444 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.444 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.444 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85139 3.5 3:a1604b12:::benchmark_data_smithi090_90974_object85138:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.444 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:23.016380+0000", 2024-05-15T23:16:51.445 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.35611113900001, 2024-05-15T23:16:51.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.200463801, 2024-05-15T23:16:51.445 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.445 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.445 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.445 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.445 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.445 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.445 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85139 2024-05-15T23:16:51.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.445 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.016380+0000", 2024-05-15T23:16:51.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.016380+0000", 2024-05-15T23:16:51.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.016381+0000", 2024-05-15T23:16:51.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5099999999999999e-06 2024-05-15T23:16:51.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.016434+0000", 2024-05-15T23:16:51.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3247e-05 2024-05-15T23:16:51.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.016447+0000", 2024-05-15T23:16:51.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2707e-05 2024-05-15T23:16:51.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.016461+0000", 2024-05-15T23:16:51.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3533000000000001e-05 2024-05-15T23:16:51.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.016530+0000", 2024-05-15T23:16:51.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9349000000000003e-05 2024-05-15T23:16:51.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.016647+0000", 2024-05-15T23:16:51.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011677500000000001 2024-05-15T23:16:51.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.016737+0000", 2024-05-15T23:16:51.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0711000000000006e-05 2024-05-15T23:16:51.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.018838+0000", 2024-05-15T23:16:51.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0021001510000000002 2024-05-15T23:16:51.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.216746+0000", 2024-05-15T23:16:51.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19790808700000001 2024-05-15T23:16:51.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.216800+0000", 2024-05-15T23:16:51.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4318999999999999e-05 2024-05-15T23:16:51.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.216843+0000", 2024-05-15T23:16:51.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3411999999999997e-05 2024-05-15T23:16:51.450 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.450 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.450 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.450 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85146 3.c 3:36cc5d8d:::benchmark_data_smithi090_90974_object85145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.450 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:23.021018+0000", 2024-05-15T23:16:51.450 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.35147230500002, 2024-05-15T23:16:51.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17908236699999999, 2024-05-15T23:16:51.450 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.450 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.450 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.451 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.451 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.451 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.451 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85146 2024-05-15T23:16:51.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.451 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021018+0000", 2024-05-15T23:16:51.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021018+0000", 2024-05-15T23:16:51.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021021+0000", 2024-05-15T23:16:51.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.058e-06 2024-05-15T23:16:51.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021075+0000", 2024-05-15T23:16:51.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4168e-05 2024-05-15T23:16:51.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021085+0000", 2024-05-15T23:16:51.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8719999999999997e-06 2024-05-15T23:16:51.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021092+0000", 2024-05-15T23:16:51.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8809999999999996e-06 2024-05-15T23:16:51.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021139+0000", 2024-05-15T23:16:51.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6557999999999997e-05 2024-05-15T23:16:51.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021250+0000", 2024-05-15T23:16:51.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000111397 2024-05-15T23:16:51.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021344+0000", 2024-05-15T23:16:51.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3517999999999995e-05 2024-05-15T23:16:51.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.025840+0000", 2024-05-15T23:16:51.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0044956409999999999 2024-05-15T23:16:51.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.200047+0000", 2024-05-15T23:16:51.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.174207271 2024-05-15T23:16:51.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.200076+0000", 2024-05-15T23:16:51.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9524000000000001e-05 2024-05-15T23:16:51.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.200101+0000", 2024-05-15T23:16:51.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4479000000000001e-05 2024-05-15T23:16:51.456 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.456 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.456 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.456 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85147 3.5 3:a745ef78:::benchmark_data_smithi090_90974_object85146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.456 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:23.021166+0000", 2024-05-15T23:16:51.456 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.351324729, 2024-05-15T23:16:51.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.195832425, 2024-05-15T23:16:51.456 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.456 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.456 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.456 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.456 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.457 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.457 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85147 2024-05-15T23:16:51.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.457 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021166+0000", 2024-05-15T23:16:51.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021166+0000", 2024-05-15T23:16:51.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021167+0000", 2024-05-15T23:16:51.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9600000000000008e-07 2024-05-15T23:16:51.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021250+0000", 2024-05-15T23:16:51.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3399999999999994e-05 2024-05-15T23:16:51.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021260+0000", 2024-05-15T23:16:51.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2490000000000007e-06 2024-05-15T23:16:51.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021267+0000", 2024-05-15T23:16:51.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7840000000000004e-06 2024-05-15T23:16:51.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021311+0000", 2024-05-15T23:16:51.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3386000000000001e-05 2024-05-15T23:16:51.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021403+0000", 2024-05-15T23:16:51.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.1858e-05 2024-05-15T23:16:51.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.460 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.021489+0000", 2024-05-15T23:16:51.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5982000000000004e-05 2024-05-15T23:16:51.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.460 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.023498+0000", 2024-05-15T23:16:51.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020093810000000002 2024-05-15T23:16:51.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.216900+0000", 2024-05-15T23:16:51.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19340175900000001 2024-05-15T23:16:51.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.216953+0000", 2024-05-15T23:16:51.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3488e-05 2024-05-15T23:16:51.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.216998+0000", 2024-05-15T23:16:51.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5142000000000002e-05 2024-05-15T23:16:51.461 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.461 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.462 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.462 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85150 3.c 3:3dee233c:::benchmark_data_smithi090_90974_object85149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.462 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:23.022714+0000", 2024-05-15T23:16:51.462 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.34977642699999, 2024-05-15T23:16:51.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17950811899999999, 2024-05-15T23:16:51.462 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.462 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.462 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.462 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.462 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.462 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.462 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85150 2024-05-15T23:16:51.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.463 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.022714+0000", 2024-05-15T23:16:51.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.022714+0000", 2024-05-15T23:16:51.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.022716+0000", 2024-05-15T23:16:51.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.235e-06 2024-05-15T23:16:51.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.022780+0000", 2024-05-15T23:16:51.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4247999999999999e-05 2024-05-15T23:16:51.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.022789+0000", 2024-05-15T23:16:51.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2439999999999999e-06 2024-05-15T23:16:51.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.022796+0000", 2024-05-15T23:16:51.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.148e-06 2024-05-15T23:16:51.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.023415+0000", 2024-05-15T23:16:51.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000619105 2024-05-15T23:16:51.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.023528+0000", 2024-05-15T23:16:51.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011230000000000001 2024-05-15T23:16:51.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.023673+0000", 2024-05-15T23:16:51.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014494900000000001 2024-05-15T23:16:51.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.026006+0000", 2024-05-15T23:16:51.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002333744 2024-05-15T23:16:51.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.202155+0000", 2024-05-15T23:16:51.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17614875999999999 2024-05-15T23:16:51.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.202190+0000", 2024-05-15T23:16:51.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5298e-05 2024-05-15T23:16:51.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.202222+0000", 2024-05-15T23:16:51.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2088000000000002e-05 2024-05-15T23:16:51.467 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.467 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.467 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.468 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85152 3.5 3:aa47e56b:::benchmark_data_smithi090_90974_object85151:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.468 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:23.025633+0000", 2024-05-15T23:16:51.468 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.34685771099998, 2024-05-15T23:16:51.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19151347799999999, 2024-05-15T23:16:51.468 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.468 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.468 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.468 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.468 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.468 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.468 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85152 2024-05-15T23:16:51.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.468 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.025633+0000", 2024-05-15T23:16:51.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.025633+0000", 2024-05-15T23:16:51.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.025635+0000", 2024-05-15T23:16:51.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4360000000000001e-06 2024-05-15T23:16:51.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.025691+0000", 2024-05-15T23:16:51.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6276000000000001e-05 2024-05-15T23:16:51.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.025701+0000", 2024-05-15T23:16:51.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0064000000000001e-05 2024-05-15T23:16:51.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.025709+0000", 2024-05-15T23:16:51.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8599999999999993e-06 2024-05-15T23:16:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.025750+0000", 2024-05-15T23:16:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1177999999999998e-05 2024-05-15T23:16:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.025887+0000", 2024-05-15T23:16:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013753800000000001 2024-05-15T23:16:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.026036+0000", 2024-05-15T23:16:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014856799999999999 2024-05-15T23:16:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.029075+0000", 2024-05-15T23:16:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030394620000000002 2024-05-15T23:16:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.217050+0000", 2024-05-15T23:16:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.187974841 2024-05-15T23:16:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.217104+0000", 2024-05-15T23:16:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3919000000000003e-05 2024-05-15T23:16:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.217147+0000", 2024-05-15T23:16:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2336000000000003e-05 2024-05-15T23:16:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85153 3.5 3:ad6a2f86:::benchmark_data_smithi090_90974_object85152:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:23.026221+0000", 2024-05-15T23:16:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.34626936900003, 2024-05-15T23:16:51.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19107407700000001, 2024-05-15T23:16:51.474 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.474 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.474 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.474 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.474 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.474 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.474 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85153 2024-05-15T23:16:51.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.474 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.026221+0000", 2024-05-15T23:16:51.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.026221+0000", 2024-05-15T23:16:51.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.026223+0000", 2024-05-15T23:16:51.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.271e-06 2024-05-15T23:16:51.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.026280+0000", 2024-05-15T23:16:51.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6910999999999998e-05 2024-05-15T23:16:51.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.026290+0000", 2024-05-15T23:16:51.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0226e-05 2024-05-15T23:16:51.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.026295+0000", 2024-05-15T23:16:51.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4709999999999998e-06 2024-05-15T23:16:51.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.027829+0000", 2024-05-15T23:16:51.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001533651 2024-05-15T23:16:51.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.027905+0000", 2024-05-15T23:16:51.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6091e-05 2024-05-15T23:16:51.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.027989+0000", 2024-05-15T23:16:51.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3669000000000001e-05 2024-05-15T23:16:51.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.029217+0000", 2024-05-15T23:16:51.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0012281379999999999 2024-05-15T23:16:51.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.217198+0000", 2024-05-15T23:16:51.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18798088099999999 2024-05-15T23:16:51.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.217257+0000", 2024-05-15T23:16:51.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9330000000000003e-05 2024-05-15T23:16:51.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.217296+0000", 2024-05-15T23:16:51.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8438e-05 2024-05-15T23:16:51.479 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.479 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.479 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.479 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:85177 3.c 3:36ae6b80:::benchmark_data_smithi090_90974_object85176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-05-15T23:16:51.479 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:11:23.098264+0000", 2024-05-15T23:16:51.479 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 328.27422695899998, 2024-05-15T23:16:51.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.104179858, 2024-05-15T23:16:51.479 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.480 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.480 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.480 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.480 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.480 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.480 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 85177 2024-05-15T23:16:51.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.480 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.098264+0000", 2024-05-15T23:16:51.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.098264+0000", 2024-05-15T23:16:51.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.098266+0000", 2024-05-15T23:16:51.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.858e-06 2024-05-15T23:16:51.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.098309+0000", 2024-05-15T23:16:51.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3427999999999998e-05 2024-05-15T23:16:51.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.098319+0000", 2024-05-15T23:16:51.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8879999999999992e-06 2024-05-15T23:16:51.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.098325+0000", 2024-05-15T23:16:51.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0120000000000002e-06 2024-05-15T23:16:51.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.098368+0000", 2024-05-15T23:16:51.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2880000000000003e-05 2024-05-15T23:16:51.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.098459+0000", 2024-05-15T23:16:51.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.1235000000000003e-05 2024-05-15T23:16:51.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.098548+0000", 2024-05-15T23:16:51.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8943000000000005e-05 2024-05-15T23:16:51.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.100633+0000", 2024-05-15T23:16:51.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020853859999999998 2024-05-15T23:16:51.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.202344+0000", 2024-05-15T23:16:51.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101710379 2024-05-15T23:16:51.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.202400+0000", 2024-05-15T23:16:51.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6542000000000002e-05 2024-05-15T23:16:51.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:11:23.202444+0000", 2024-05-15T23:16:51.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3306999999999997e-05 2024-05-15T23:16:51.484 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.485 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.485 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.485 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:181318 3.c 3:33ffaa91:::benchmark_data_smithi090_90974_object15775:head [delete] snapc 0=[] ondisk+write+known_if_redirected e251)", 2024-05-15T23:16:51.485 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:12:38.151696+0000", 2024-05-15T23:16:51.485 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 253.22079430400001, 2024-05-15T23:16:51.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.076763310000000001, 2024-05-15T23:16:51.485 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:51.485 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:51.485 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:16:51.485 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:51.485 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:16:51.485 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:16:51.485 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 181318 2024-05-15T23:16:51.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.486 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:51.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:51.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:38.151696+0000", 2024-05-15T23:16:51.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:51.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:38.151696+0000", 2024-05-15T23:16:51.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:51.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:51.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:38.151698+0000", 2024-05-15T23:16:51.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4410000000000001e-06 2024-05-15T23:16:51.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:51.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:38.151703+0000", 2024-05-15T23:16:51.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4419999999999997e-06 2024-05-15T23:16:51.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:51.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:38.151714+0000", 2024-05-15T23:16:51.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.043e-05 2024-05-15T23:16:51.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.488 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:51.488 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:38.151723+0000", 2024-05-15T23:16:51.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8550000000000008e-06 2024-05-15T23:16:51.488 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.488 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:51.488 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:38.156285+0000", 2024-05-15T23:16:51.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045624109999999997 2024-05-15T23:16:51.488 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.488 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:51.488 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:38.156408+0000", 2024-05-15T23:16:51.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000122912 2024-05-15T23:16:51.488 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:16:51.489 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:38.156532+0000", 2024-05-15T23:16:51.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000124051 2024-05-15T23:16:51.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:16:51.489 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:38.160425+0000", 2024-05-15T23:16:51.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003892632 2024-05-15T23:16:51.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:16:51.489 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:38.228392+0000", 2024-05-15T23:16:51.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.067967468000000003 2024-05-15T23:16:51.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.490 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:16:51.490 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:38.228430+0000", 2024-05-15T23:16:51.490 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7676999999999999e-05 2024-05-15T23:16:51.490 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:51.490 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:51.490 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:16:51.490 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:38.228460+0000", 2024-05-15T23:16:51.490 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9991000000000001e-05 2024-05-15T23:16:51.490 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.490 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.490 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.490 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:51.490 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:51.490 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:16:51.973 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:51.997 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:16:52.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:52.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:16:52.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:16:52.147 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:52.170 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:16:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:52.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:16:52.189 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T23:16:52.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:16:52.304 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:52.329 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:16:52.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:52.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:16:52.404 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 1 (10000) 2024-05-15T23:16:52.434 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:16:52.490 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:52.534 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:16:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:52 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1401836581' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T23:16:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:52 smithi090 ceph-mon[44434]: from='client.35657 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T23:16:52.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:52 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1401836581' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T23:16:52.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:52 smithi088 ceph-mon[44699]: from='client.35657 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T23:16:52.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:52 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1401836581' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T23:16:52.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:52 smithi084 ceph-mon[39784]: from='client.35657 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T23:16:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:53 smithi084 ceph-mon[39784]: from='client.35657 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T23:16:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:53 smithi084 ceph-mon[39784]: osdmap e323: 12 total, 12 up, 11 in 2024-05-15T23:16:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:53 smithi084 ceph-mon[39784]: pgmap v890: 17 pgs: 17 active+clean; 9.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:16:53.657 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:16:53.761 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:16:53.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:53.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:16:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:53 smithi090 ceph-mon[44434]: from='client.35657 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T23:16:53.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:53 smithi090 ceph-mon[44434]: osdmap e323: 12 total, 12 up, 11 in 2024-05-15T23:16:53.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:53 smithi090 ceph-mon[44434]: pgmap v890: 17 pgs: 17 active+clean; 9.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:16:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:53 smithi088 ceph-mon[44699]: from='client.35657 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T23:16:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:53 smithi088 ceph-mon[44699]: osdmap e323: 12 total, 12 up, 11 in 2024-05-15T23:16:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:53 smithi088 ceph-mon[44699]: pgmap v890: 17 pgs: 17 active+clean; 9.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:16:53.875 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:16:53.942 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:54.202 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:16:54.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:54.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:16:54.238 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 7 1 2024-05-15T23:16:54.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:54.337 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:54.358 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:54.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:54.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:54.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:16:54.524 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:54.626 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:16:54.626 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:16:54.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.626 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:194522 4.5 4:a9a99fa2:::benchmark_data_smithi090_104448_object21284:head [delete] snapc 0=[] ondisk+write+known_if_redirected e324)", 2024-05-15T23:16:54.626 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:16:54.618214+0000", 2024-05-15T23:16:54.626 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.006875674, 2024-05-15T23:16:54.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0069063249999999996, 2024-05-15T23:16:54.627 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:54.627 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:54.627 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:16:54.627 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:54.627 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:16:54.627 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:16:54.627 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 194522 2024-05-15T23:16:54.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.627 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:54.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:54.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.618214+0000", 2024-05-15T23:16:54.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:54.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:54.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.618214+0000", 2024-05-15T23:16:54.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:54.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:54.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.618218+0000", 2024-05-15T23:16:54.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.377e-06 2024-05-15T23:16:54.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:54.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.618226+0000", 2024-05-15T23:16:54.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6969999999999999e-06 2024-05-15T23:16:54.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:54.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.618246+0000", 2024-05-15T23:16:54.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9575000000000001e-05 2024-05-15T23:16:54.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:54.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.618256+0000", 2024-05-15T23:16:54.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0027000000000001e-05 2024-05-15T23:16:54.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:54.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.618344+0000", 2024-05-15T23:16:54.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8358999999999999e-05 2024-05-15T23:16:54.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:54.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.618479+0000", 2024-05-15T23:16:54.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013478399999999999 2024-05-15T23:16:54.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 10", 2024-05-15T23:16:54.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.618606+0000", 2024-05-15T23:16:54.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012678900000000001 2024-05-15T23:16:54.630 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:54.630 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:54.630 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:54.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.631 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:194528 4.5 4:aba9eb69:::benchmark_data_smithi090_104448_object21290:head [delete] snapc 0=[] ondisk+write+known_if_redirected e324)", 2024-05-15T23:16:54.631 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:16:54.618280+0000", 2024-05-15T23:16:54.631 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.0068098789999999996, 2024-05-15T23:16:54.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0069227079999999996, 2024-05-15T23:16:54.631 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:16:54.631 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:16:54.631 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:16:54.631 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:16:54.631 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:16:54.631 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:16:54.631 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 194528 2024-05-15T23:16:54.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.631 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:16:54.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:16:54.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.618280+0000", 2024-05-15T23:16:54.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:54.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:16:54.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.618280+0000", 2024-05-15T23:16:54.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:16:54.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:16:54.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.618281+0000", 2024-05-15T23:16:54.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7000000000000004e-07 2024-05-15T23:16:54.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:16:54.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.618284+0000", 2024-05-15T23:16:54.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.959e-06 2024-05-15T23:16:54.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:16:54.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.618289+0000", 2024-05-15T23:16:54.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2440000000000001e-06 2024-05-15T23:16:54.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:16:54.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.618294+0000", 2024-05-15T23:16:54.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7190000000000001e-06 2024-05-15T23:16:54.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:16:54.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.622214+0000", 2024-05-15T23:16:54.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0039197820000000001 2024-05-15T23:16:54.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:16:54.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.622290+0000", 2024-05-15T23:16:54.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6359000000000005e-05 2024-05-15T23:16:54.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:16:54.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:16:54.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 10", 2024-05-15T23:16:54.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:54.622364+0000", 2024-05-15T23:16:54.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4391e-05 2024-05-15T23:16:54.635 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:54.635 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:16:54.635 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:54.635 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:16:54.635 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2024-05-15T23:16:54.635 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 2 2024-05-15T23:16:54.635 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:16:54.639 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:16:54.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:54.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:16:54.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:16:54.789 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:54.813 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:16:54.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:54.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:16:54.844 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:54.903 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:54.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:16:55.006 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:55.216 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:16:55.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:55.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:16:55.320 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:16:55.377 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:55.418 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T23:16:55.430 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:16:55.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:55.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:16:55.446 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:16:55.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:16:55.574 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:55.597 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:16:55.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:55.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:16:55.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:55 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:16:55.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:55 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:16:55.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:55 smithi084 ceph-mon[39784]: osdmap e324: 12 total, 12 up, 11 in 2024-05-15T23:16:55.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:55 smithi084 ceph-mon[39784]: pgmap v892: 17 pgs: 17 active+clean; 9.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:16:55.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:55 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1500931913' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:16:55.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:16:55.753 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:55.790 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:16:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:55 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:16:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:55 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:16:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:55 smithi090 ceph-mon[44434]: osdmap e324: 12 total, 12 up, 11 in 2024-05-15T23:16:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:55 smithi090 ceph-mon[44434]: pgmap v892: 17 pgs: 17 active+clean; 9.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:16:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:55 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1500931913' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:16:55.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:55 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:16:55.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:55 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:16:55.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:55 smithi088 ceph-mon[44699]: osdmap e324: 12 total, 12 up, 11 in 2024-05-15T23:16:55.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:55 smithi088 ceph-mon[44699]: pgmap v892: 17 pgs: 17 active+clean; 9.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:16:55.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:55 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1500931913' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:16:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:56 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:16:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1500931913' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:16:56.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:56 smithi090 ceph-mon[44434]: osdmap e325: 12 total, 12 up, 11 in 2024-05-15T23:16:56.810 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:56 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:16:56.810 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1500931913' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:16:56.810 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:56 smithi084 ceph-mon[39784]: osdmap e325: 12 total, 12 up, 11 in 2024-05-15T23:16:56.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:56 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:16:56.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1500931913' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:16:56.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:56 smithi088 ceph-mon[44699]: osdmap e325: 12 total, 12 up, 11 in 2024-05-15T23:16:57.453 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:57 smithi084 ceph-mon[39784]: pgmap v894: 17 pgs: 17 active+clean; 9.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-05-15T23:16:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:57 smithi090 ceph-mon[44434]: pgmap v894: 17 pgs: 17 active+clean; 9.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-05-15T23:16:57.821 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:16:57.821 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:16:57.821 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:16:57.821 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:16:57.821 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:16:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:57 smithi088 ceph-mon[44699]: pgmap v894: 17 pgs: 17 active+clean; 9.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-05-15T23:16:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:58 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:16:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:58 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:16:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:58 smithi088 ceph-mon[44699]: osdmap e326: 12 total, 12 up, 11 in 2024-05-15T23:16:58.658 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T23:16:58.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:58 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:16:58.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:58 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:16:58.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:58 smithi090 ceph-mon[44434]: osdmap e326: 12 total, 12 up, 11 in 2024-05-15T23:16:58.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:16:58.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:58 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:16:58.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:58 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:16:58.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:58 smithi084 ceph-mon[39784]: osdmap e326: 12 total, 12 up, 11 in 2024-05-15T23:16:58.919 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:16:58.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:58.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:16:59.011 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:16:59.023 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:16:59.089 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:59.193 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:16:59.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:59.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:16:59.261 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:16:59.291 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:16:59.292 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:16:59.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:16:59.323 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:16:59.326 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:59.349 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:16:59.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:59.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:16:59.365 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:16:59.401 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:16:59.437 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:16:59.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:16:59.471 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:59 smithi084 ceph-mon[39784]: pgmap v896: 17 pgs: 17 active+clean; 9.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-05-15T23:16:59.471 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:59 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3506588098' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:16:59.474 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:16:59.481 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-05-15T23:16:59.493 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:59.510 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:16:59.517 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:16:59.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:59.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:16:59.537 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:16:59.545 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:59 smithi088 ceph-mon[44699]: pgmap v896: 17 pgs: 17 active+clean; 9.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-05-15T23:16:59.545 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:59 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3506588098' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:16:59.545 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:59 smithi088 ceph-mon[44699]: from='client.35663 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:16:59.545 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:59 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4227190219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:59.545 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:59.545 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:16:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:59.576 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:16:59.615 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:16:59.624 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:16:59.641 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:16:59.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:16:59.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:16:59.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:16:59.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:16:59.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:16:59.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:16:59.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:16:59.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:16:59.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:16:59.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:16:59.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:16:59.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:16:59.655 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:16:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:59 smithi090 ceph-mon[44434]: pgmap v896: 17 pgs: 17 active+clean; 9.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-05-15T23:16:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:59 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3506588098' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:16:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:59 smithi090 ceph-mon[44434]: from='client.35663 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:16:59.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:59 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4227190219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:59.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:59.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:16:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:59.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:59 smithi084 ceph-mon[39784]: from='client.35663 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:16:59.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:59 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4227190219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:16:59.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:59.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:16:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:16:59.907 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:16:59.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:16:59.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:17:00.011 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:17:00.058 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:00.415 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:17:00.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:00.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:17:00.526 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:17:00.579 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:00.660 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:17:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:00.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:17:00.772 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:17:00.784 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.784 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.784 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.784 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.785 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:00 smithi088 ceph-mon[44699]: from='client.35663 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:17:00.785 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:00 smithi088 ceph-mon[44699]: osdmap e327: 12 total, 12 up, 11 in 2024-05-15T23:17:00.785 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.785 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.785 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.785 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.785 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.785 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:00 smithi090 ceph-mon[44434]: from='client.35663 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:17:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:00 smithi090 ceph-mon[44434]: osdmap e327: 12 total, 12 up, 11 in 2024-05-15T23:17:00.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.823 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:00.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:00 smithi084 ceph-mon[39784]: from='client.35663 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:17:00.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:00 smithi084 ceph-mon[39784]: osdmap e327: 12 total, 12 up, 11 in 2024-05-15T23:17:00.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:00.961 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:17:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:00.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:17:01.068 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:17:01.102 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:01.105 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 8 1 2024-05-15T23:17:01.124 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:17:01.124 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:17:01.138 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:17:01.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:01.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:17:01.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:17:01.287 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:01.750 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:01 smithi084 ceph-mon[39784]: pgmap v898: 17 pgs: 17 active+clean; 8.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.42k op/s 2024-05-15T23:17:01.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:01 smithi090 ceph-mon[44434]: pgmap v898: 17 pgs: 17 active+clean; 8.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.42k op/s 2024-05-15T23:17:01.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:01 smithi088 ceph-mon[44699]: pgmap v898: 17 pgs: 17 active+clean; 8.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.42k op/s 2024-05-15T23:17:01.942 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T23:17:02.105 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:17:02.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:02.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:17:02.125 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:02.215 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:02.252 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:02.307 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:17:02.372 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:02.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:02.480 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:17:02.509 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:17:02.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:02.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:17:02.647 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:17:02.684 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:02.716 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:17:02.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:02.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:17:02.820 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:02.851 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:03.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:02 smithi084 ceph-mon[39784]: pgmap v899: 17 pgs: 17 active+clean; 8.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:17:03.213 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:03.218 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:02 smithi088 ceph-mon[44699]: pgmap v899: 17 pgs: 17 active+clean; 8.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:17:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:03.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:02 smithi090 ceph-mon[44434]: pgmap v899: 17 pgs: 17 active+clean; 8.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:17:03.321 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:17:03.353 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:03.810 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:17:03.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:03.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:17:03.920 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:17:03.958 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:17:03.962 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2846641351' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:17:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:03 smithi088 ceph-mon[44699]: from='client.35675 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:17:04.095 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:17:04.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:04.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:17:04.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:17:04.202 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:17:04.240 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2846641351' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:17:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:03 smithi090 ceph-mon[44434]: from='client.35675 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:17:04.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2846641351' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:17:04.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:03 smithi084 ceph-mon[39784]: from='client.35675 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:17:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:17:04.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:04.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:17:04.441 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:17:04.442 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:17:04.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:17:04.587 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:05.067 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:17:05.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:05.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:17:05.079 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:04 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:17:05.079 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:04 smithi088 ceph-mon[44699]: from='client.35675 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:17:05.079 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:04 smithi088 ceph-mon[44699]: osdmap e328: 12 total, 12 up, 11 in 2024-05-15T23:17:05.079 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:04 smithi088 ceph-mon[44699]: pgmap v901: 17 pgs: 17 active+clean; 8.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.35k op/s 2024-05-15T23:17:05.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:17:05.224 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:04 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:17:05.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:04 smithi090 ceph-mon[44434]: from='client.35675 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:17:05.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:04 smithi090 ceph-mon[44434]: osdmap e328: 12 total, 12 up, 11 in 2024-05-15T23:17:05.225 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:04 smithi090 ceph-mon[44434]: pgmap v901: 17 pgs: 17 active+clean; 8.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.35k op/s 2024-05-15T23:17:05.225 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:05.241 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:17:05.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:05.250 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:17:05.278 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:05.351 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:17:05.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:04 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:17:05.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:04 smithi084 ceph-mon[39784]: from='client.35675 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:17:05.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:04 smithi084 ceph-mon[39784]: osdmap e328: 12 total, 12 up, 11 in 2024-05-15T23:17:05.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:04 smithi084 ceph-mon[39784]: pgmap v901: 17 pgs: 17 active+clean; 8.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.35k op/s 2024-05-15T23:17:05.415 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:05.483 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T23:17:05.763 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:17:05.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:05.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:17:05.871 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:17:05.913 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:17:05.963 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:06.024 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.8 to 1 (10000) 2024-05-15T23:17:06.225 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:05 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3401262130' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:17:06.225 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:05 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:17:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:05 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3401262130' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:17:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:05 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:17:06.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:05 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3401262130' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:17:06.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:05 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:17:06.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:06 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3401262130' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:17:06.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:06 smithi084 ceph-mon[39784]: osdmap e329: 12 total, 12 up, 11 in 2024-05-15T23:17:06.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:17:06.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:17:06.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:17:06.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:06 smithi084 ceph-mon[39784]: pgmap v903: 17 pgs: 17 active+clean; 8.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.30k op/s 2024-05-15T23:17:07.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:17:07.065 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:17:07.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:07.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:17:07.174 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:17:07.225 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:07.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:06 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3401262130' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:17:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:06 smithi090 ceph-mon[44434]: osdmap e329: 12 total, 12 up, 11 in 2024-05-15T23:17:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:17:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:17:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:17:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:06 smithi090 ceph-mon[44434]: pgmap v903: 17 pgs: 17 active+clean; 8.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.30k op/s 2024-05-15T23:17:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:06 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3401262130' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:17:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:06 smithi088 ceph-mon[44699]: osdmap e329: 12 total, 12 up, 11 in 2024-05-15T23:17:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:17:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:17:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:17:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:06 smithi088 ceph-mon[44699]: pgmap v903: 17 pgs: 17 active+clean; 8.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.30k op/s 2024-05-15T23:17:07.680 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 11 1 2024-05-15T23:17:07.998 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:17:08.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:08.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:17:08.092 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:17:08.092 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:17:08.104 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:17:08.149 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:08.326 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:08.329 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:08.363 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:17:08.402 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:17:08.442 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:17:08.465 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:17:08.497 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:17:08.522 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:17:08.550 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:17:08.574 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:17:08.610 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:17:08.650 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:17:08.687 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:17:08.714 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:17:08.716 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:17:08.716 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:17:08.716 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:17:08.716 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:17:08.716 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:17:08.716 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:17:08.716 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:17:08.716 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:17:08.716 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:17:08.716 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:17:08.716 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:17:08.716 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:17:08.803 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:08 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:17:08.804 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:08 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:17:08.804 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3564935607' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:08.804 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:08.804 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:08.804 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:08.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:08 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:17:08.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:08 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:17:08.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3564935607' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:08.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:08.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:08.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:08.905 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:17:08.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:08.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:17:08.923 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T23:17:09.011 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:17:09.074 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:08 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:17:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:08 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:17:09.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3564935607' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:09.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:09.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:09.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:09.322 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:17:10.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:09 smithi088 ceph-mon[44699]: pgmap v904: 17 pgs: 17 active+clean; 8.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-05-15T23:17:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:09 smithi084 ceph-mon[39784]: pgmap v904: 17 pgs: 17 active+clean; 8.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-05-15T23:17:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:09 smithi090 ceph-mon[44434]: pgmap v904: 17 pgs: 17 active+clean; 8.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-05-15T23:17:11.149 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:17:11.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:11.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:17:11.253 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:10 smithi084 ceph-mon[39784]: pgmap v905: 17 pgs: 17 active+clean; 8.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:11.258 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:11.296 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:10 smithi088 ceph-mon[44699]: pgmap v905: 17 pgs: 17 active+clean; 8.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:11.300 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:11.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:10 smithi090 ceph-mon[44434]: pgmap v905: 17 pgs: 17 active+clean; 8.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:11.766 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:11.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:11.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:11.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:17:11.873 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:11.905 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:11.932 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:11.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:11.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:12.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:17:12.070 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:12.105 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:17:12.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:12.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:17:12.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:12.275 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:12.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:12.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:12.380 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:17:12.439 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:12.622 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:17:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:12.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:17:12.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:17:12.758 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:12.781 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:17:12.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:12.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:17:12.884 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:12.913 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:12.922 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:13.167 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:13 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:17:13.167 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:13 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:17:13.168 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:13 smithi088 ceph-mon[44699]: osdmap e330: 12 total, 12 up, 11 in 2024-05-15T23:17:13.168 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:13 smithi088 ceph-mon[44699]: pgmap v907: 17 pgs: 17 active+clean; 8.0 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.33k op/s 2024-05-15T23:17:13.168 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2825749483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:17:13.168 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:13 smithi088 ceph-mon[44699]: from='client.25597 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:17:13.168 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/356661751' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:17:13.170 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.11 to 1 (10000) 2024-05-15T23:17:13.193 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:17:13.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:13 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:17:13.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:13 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:17:13.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:13 smithi090 ceph-mon[44434]: osdmap e330: 12 total, 12 up, 11 in 2024-05-15T23:17:13.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:13 smithi090 ceph-mon[44434]: pgmap v907: 17 pgs: 17 active+clean; 8.0 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.33k op/s 2024-05-15T23:17:13.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2825749483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:17:13.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:13 smithi090 ceph-mon[44434]: from='client.25597 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:17:13.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/356661751' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:17:13.574 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:13 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:17:13.574 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:13 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:17:13.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:13 smithi084 ceph-mon[39784]: osdmap e330: 12 total, 12 up, 11 in 2024-05-15T23:17:13.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:13 smithi084 ceph-mon[39784]: pgmap v907: 17 pgs: 17 active+clean; 8.0 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.33k op/s 2024-05-15T23:17:13.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2825749483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:17:13.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:13 smithi084 ceph-mon[39784]: from='client.25597 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:17:13.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/356661751' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:17:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:14 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:17:14.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:14 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:17:14.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:14 smithi090 ceph-mon[44434]: from='client.25597 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:17:14.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:14 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/356661751' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:17:14.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:14 smithi090 ceph-mon[44434]: osdmap e331: 12 total, 12 up, 11 in 2024-05-15T23:17:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:14 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:17:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:14 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:17:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:14 smithi088 ceph-mon[44699]: from='client.25597 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:17:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:14 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/356661751' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:17:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:14 smithi088 ceph-mon[44699]: osdmap e331: 12 total, 12 up, 11 in 2024-05-15T23:17:14.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:14 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:17:14.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:14 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:17:14.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:14 smithi084 ceph-mon[39784]: from='client.25597 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:17:14.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:14 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/356661751' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:17:14.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:14 smithi084 ceph-mon[39784]: osdmap e331: 12 total, 12 up, 11 in 2024-05-15T23:17:14.678 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 9 1 2024-05-15T23:17:15.077 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:15.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:15.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:15.181 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:17:15.223 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:15 smithi088 ceph-mon[44699]: pgmap v909: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:15.527 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:17:15.527 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:17:15.527 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:17:15.527 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:17:15.527 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.527 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:260050 2.8 2:154eacae:::benchmark_data_smithi090_78160_object98692:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e165)", 2024-05-15T23:17:15.527 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:05.987901+0000", 2024-05-15T23:17:15.528 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.53471931500002, 2024-05-15T23:17:15.528 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15881131500000001, 2024-05-15T23:17:15.528 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.528 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.528 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2024-05-15T23:17:15.528 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.528 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:17:15.528 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:17:15.528 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 260050 2024-05-15T23:17:15.528 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.528 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.528 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.528 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.528 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.987901+0000", 2024-05-15T23:17:15.529 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.529 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.529 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.529 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.529 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.987901+0000", 2024-05-15T23:17:15.529 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.529 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.529 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.529 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.529 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.987903+0000", 2024-05-15T23:17:15.529 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3369999999999998e-06 2024-05-15T23:17:15.529 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.529 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.529 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.530 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.987911+0000", 2024-05-15T23:17:15.530 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9810000000000003e-06 2024-05-15T23:17:15.530 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.530 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.530 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.530 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.987925+0000", 2024-05-15T23:17:15.530 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3915e-05 2024-05-15T23:17:15.530 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.530 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.530 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.530 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.987938+0000", 2024-05-15T23:17:15.530 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3139000000000001e-05 2024-05-15T23:17:15.530 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.531 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.531 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.531 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988000+0000", 2024-05-15T23:17:15.531 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1684000000000001e-05 2024-05-15T23:17:15.531 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.531 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.531 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:17:15.531 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988011+0000", 2024-05-15T23:17:15.531 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.131e-05 2024-05-15T23:17:15.531 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.531 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.531 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.531 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.146633+0000", 2024-05-15T23:17:15.531 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.158621704 2024-05-15T23:17:15.531 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.532 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.532 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.532 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.146712+0000", 2024-05-15T23:17:15.532 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9245000000000004e-05 2024-05-15T23:17:15.532 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.532 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.532 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.532 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.532 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.532 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:260052 2.8 2:1c532154:::benchmark_data_smithi090_78160_object98694:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e165)", 2024-05-15T23:17:15.532 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:05.987960+0000", 2024-05-15T23:17:15.532 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.53465999899998, 2024-05-15T23:17:15.532 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.158988134, 2024-05-15T23:17:15.533 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.533 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.533 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2024-05-15T23:17:15.533 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.533 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:17:15.533 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:17:15.533 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 260052 2024-05-15T23:17:15.533 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.533 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.533 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.533 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.533 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.987960+0000", 2024-05-15T23:17:15.533 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.533 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.533 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.534 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.534 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.987960+0000", 2024-05-15T23:17:15.534 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.534 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.534 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.534 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.534 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.987961+0000", 2024-05-15T23:17:15.534 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0070000000000001e-06 2024-05-15T23:17:15.534 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.534 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.534 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.534 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.987966+0000", 2024-05-15T23:17:15.534 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.566e-06 2024-05-15T23:17:15.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.535 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.535 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.987973+0000", 2024-05-15T23:17:15.535 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9959999999999996e-06 2024-05-15T23:17:15.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.535 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.535 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.987979+0000", 2024-05-15T23:17:15.535 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6100000000000002e-06 2024-05-15T23:17:15.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.535 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.535 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988034+0000", 2024-05-15T23:17:15.535 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4882000000000001e-05 2024-05-15T23:17:15.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.536 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:17:15.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988041+0000", 2024-05-15T23:17:15.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8460000000000002e-06 2024-05-15T23:17:15.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.536 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.146900+0000", 2024-05-15T23:17:15.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15885940600000001 2024-05-15T23:17:15.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.536 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.146948+0000", 2024-05-15T23:17:15.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7821000000000002e-05 2024-05-15T23:17:15.537 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.537 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.537 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.537 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.537 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.537 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:260073 2.8 2:1c281eab:::benchmark_data_smithi090_78160_object98715:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165)", 2024-05-15T23:17:15.537 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:05.987997+0000", 2024-05-15T23:17:15.537 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.53462262800002, 2024-05-15T23:17:15.537 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.172987909, 2024-05-15T23:17:15.537 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.537 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.537 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.537 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.538 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:17:15.538 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:17:15.538 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 260073 2024-05-15T23:17:15.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.538 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.987997+0000", 2024-05-15T23:17:15.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.987997+0000", 2024-05-15T23:17:15.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.987998+0000", 2024-05-15T23:17:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2800000000000005e-07 2024-05-15T23:17:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988002+0000", 2024-05-15T23:17:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8010000000000001e-06 2024-05-15T23:17:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988009+0000", 2024-05-15T23:17:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.798e-06 2024-05-15T23:17:15.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.540 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988016+0000", 2024-05-15T23:17:15.540 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.024e-06 2024-05-15T23:17:15.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.540 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988076+0000", 2024-05-15T23:17:15.540 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0306000000000002e-05 2024-05-15T23:17:15.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.540 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:17:15.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988080+0000", 2024-05-15T23:17:15.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0820000000000001e-06 2024-05-15T23:17:15.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.146986+0000", 2024-05-15T23:17:15.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.158905514 2024-05-15T23:17:15.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.147092+0000", 2024-05-15T23:17:15.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000105818 2024-05-15T23:17:15.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:17:15.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.147174+0000", 2024-05-15T23:17:15.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2132000000000006e-05 2024-05-15T23:17:15.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.156391+0000", 2024-05-15T23:17:15.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0092170220000000001 2024-05-15T23:17:15.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.156424+0000", 2024-05-15T23:17:15.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2982999999999999e-05 2024-05-15T23:17:15.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.160880+0000", 2024-05-15T23:17:15.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0044564590000000003 2024-05-15T23:17:15.543 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.160925+0000", 2024-05-15T23:17:15.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4786000000000001e-05 2024-05-15T23:17:15.543 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.160985+0000", 2024-05-15T23:17:15.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0256e-05 2024-05-15T23:17:15.543 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.543 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.544 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.544 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:260075 2.8 2:1021f228:::benchmark_data_smithi090_78160_object98717:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165)", 2024-05-15T23:17:15.544 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:05.988025+0000", 2024-05-15T23:17:15.544 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.53459492399998, 2024-05-15T23:17:15.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.177878537, 2024-05-15T23:17:15.544 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.544 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.544 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.544 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.544 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:17:15.544 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:17:15.544 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 260075 2024-05-15T23:17:15.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.545 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988025+0000", 2024-05-15T23:17:15.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988025+0000", 2024-05-15T23:17:15.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988026+0000", 2024-05-15T23:17:15.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7199999999999998e-07 2024-05-15T23:17:15.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988029+0000", 2024-05-15T23:17:15.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1540000000000002e-06 2024-05-15T23:17:15.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988035+0000", 2024-05-15T23:17:15.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.626e-06 2024-05-15T23:17:15.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988040+0000", 2024-05-15T23:17:15.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9429999999999997e-06 2024-05-15T23:17:15.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.547 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988119+0000", 2024-05-15T23:17:15.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8344999999999996e-05 2024-05-15T23:17:15.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.547 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:17:15.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:05.988123+0000", 2024-05-15T23:17:15.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8029999999999998e-06 2024-05-15T23:17:15.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.151271+0000", 2024-05-15T23:17:15.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16314843100000001 2024-05-15T23:17:15.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.151394+0000", 2024-05-15T23:17:15.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012285200000000001 2024-05-15T23:17:15.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0,9", 2024-05-15T23:17:15.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.151541+0000", 2024-05-15T23:17:15.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000146644 2024-05-15T23:17:15.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.549 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.156437+0000", 2024-05-15T23:17:15.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0048967180000000004 2024-05-15T23:17:15.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.549 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.156483+0000", 2024-05-15T23:17:15.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5831999999999998e-05 2024-05-15T23:17:15.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.549 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.165761+0000", 2024-05-15T23:17:15.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0092774799999999994 2024-05-15T23:17:15.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.550 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.550 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.165829+0000", 2024-05-15T23:17:15.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8230000000000002e-05 2024-05-15T23:17:15.550 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.550 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.550 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:06.165904+0000", 2024-05-15T23:17:15.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4807000000000004e-05 2024-05-15T23:17:15.550 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.550 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.550 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.550 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.550 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280564 2.8 2:1f176eab:::benchmark_data_smithi090_78160_object119206:head [delete] snapc 0=[] ondisk+write+known_if_redirected e171)", 2024-05-15T23:17:15.551 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:29.216573+0000", 2024-05-15T23:17:15.551 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 466.30604657499998, 2024-05-15T23:17:15.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.083908037000000005, 2024-05-15T23:17:15.551 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.551 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.551 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.551 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.551 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:17:15.551 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:17:15.551 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280564 2024-05-15T23:17:15.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.551 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216573+0000", 2024-05-15T23:17:15.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216573+0000", 2024-05-15T23:17:15.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216574+0000", 2024-05-15T23:17:15.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6400000000000001e-07 2024-05-15T23:17:15.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216578+0000", 2024-05-15T23:17:15.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.771e-06 2024-05-15T23:17:15.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216584+0000", 2024-05-15T23:17:15.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2400000000000004e-06 2024-05-15T23:17:15.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216590+0000", 2024-05-15T23:17:15.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6540000000000004e-06 2024-05-15T23:17:15.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.287781+0000", 2024-05-15T23:17:15.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.071191219 2024-05-15T23:17:15.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.287849+0000", 2024-05-15T23:17:15.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.779e-05 2024-05-15T23:17:15.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:17:15.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.287918+0000", 2024-05-15T23:17:15.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9278999999999999e-05 2024-05-15T23:17:15.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.300336+0000", 2024-05-15T23:17:15.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.012417674 2024-05-15T23:17:15.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.300385+0000", 2024-05-15T23:17:15.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8928000000000003e-05 2024-05-15T23:17:15.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.300439+0000", 2024-05-15T23:17:15.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4214e-05 2024-05-15T23:17:15.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.300481+0000", 2024-05-15T23:17:15.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2404000000000003e-05 2024-05-15T23:17:15.556 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.556 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.556 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.556 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280584 2.8 2:1ceaac41:::benchmark_data_smithi090_78160_object119226:head [delete] snapc 0=[] ondisk+write+known_if_redirected e171)", 2024-05-15T23:17:15.556 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:29.216607+0000", 2024-05-15T23:17:15.556 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 466.30601307699999, 2024-05-15T23:17:15.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.083998592999999996, 2024-05-15T23:17:15.556 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.556 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.557 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.557 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.557 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:17:15.557 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:17:15.557 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280584 2024-05-15T23:17:15.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.557 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216607+0000", 2024-05-15T23:17:15.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.558 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216607+0000", 2024-05-15T23:17:15.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.558 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.558 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216607+0000", 2024-05-15T23:17:15.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8700000000000001e-07 2024-05-15T23:17:15.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.558 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.558 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216610+0000", 2024-05-15T23:17:15.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4499999999999998e-06 2024-05-15T23:17:15.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216616+0000", 2024-05-15T23:17:15.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7290000000000003e-06 2024-05-15T23:17:15.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216620+0000", 2024-05-15T23:17:15.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6349999999999999e-06 2024-05-15T23:17:15.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.289926+0000", 2024-05-15T23:17:15.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.073305513000000003 2024-05-15T23:17:15.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.289986+0000", 2024-05-15T23:17:15.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0275000000000001e-05 2024-05-15T23:17:15.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:17:15.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.290057+0000", 2024-05-15T23:17:15.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1033999999999998e-05 2024-05-15T23:17:15.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.300342+0000", 2024-05-15T23:17:15.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010284946 2024-05-15T23:17:15.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.300545+0000", 2024-05-15T23:17:15.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00020302499999999999 2024-05-15T23:17:15.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.300578+0000", 2024-05-15T23:17:15.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2704999999999999e-05 2024-05-15T23:17:15.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.300605+0000", 2024-05-15T23:17:15.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7894000000000001e-05 2024-05-15T23:17:15.562 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.562 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.562 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.562 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.562 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280590 2.8 2:1d4235ce:::benchmark_data_smithi090_78160_object119232:head [delete] snapc 0=[] ondisk+write+known_if_redirected e171)", 2024-05-15T23:17:15.562 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:29.216624+0000", 2024-05-15T23:17:15.562 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 466.30599570999999, 2024-05-15T23:17:15.562 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084125096999999996, 2024-05-15T23:17:15.562 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.562 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.562 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.562 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.562 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:17:15.563 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:17:15.563 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280590 2024-05-15T23:17:15.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.563 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.563 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216624+0000", 2024-05-15T23:17:15.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.563 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216624+0000", 2024-05-15T23:17:15.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.564 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.564 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216625+0000", 2024-05-15T23:17:15.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1300000000000001e-07 2024-05-15T23:17:15.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.564 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.564 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216627+0000", 2024-05-15T23:17:15.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9539999999999998e-06 2024-05-15T23:17:15.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.564 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.564 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216630+0000", 2024-05-15T23:17:15.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9160000000000001e-06 2024-05-15T23:17:15.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216633+0000", 2024-05-15T23:17:15.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0630000000000002e-06 2024-05-15T23:17:15.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.292375+0000", 2024-05-15T23:17:15.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075742432999999998 2024-05-15T23:17:15.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.292442+0000", 2024-05-15T23:17:15.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6519000000000005e-05 2024-05-15T23:17:15.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:17:15.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.292508+0000", 2024-05-15T23:17:15.566 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6798000000000007e-05 2024-05-15T23:17:15.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.300494+0000", 2024-05-15T23:17:15.566 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0079853500000000004 2024-05-15T23:17:15.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.300658+0000", 2024-05-15T23:17:15.566 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000164473 2024-05-15T23:17:15.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.300707+0000", 2024-05-15T23:17:15.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8590000000000001e-05 2024-05-15T23:17:15.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.300749+0000", 2024-05-15T23:17:15.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2588e-05 2024-05-15T23:17:15.567 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.567 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.567 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.568 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280623 2.8 2:185fc2f8:::benchmark_data_smithi090_78160_object119265:head [delete] snapc 0=[] ondisk+write+known_if_redirected e171)", 2024-05-15T23:17:15.568 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:29.216636+0000", 2024-05-15T23:17:15.568 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 466.30598378500002, 2024-05-15T23:17:15.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.095078678999999999, 2024-05-15T23:17:15.568 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.568 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.568 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.568 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.568 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:17:15.568 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:17:15.568 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280623 2024-05-15T23:17:15.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.568 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216636+0000", 2024-05-15T23:17:15.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216636+0000", 2024-05-15T23:17:15.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216637+0000", 2024-05-15T23:17:15.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7099999999999997e-07 2024-05-15T23:17:15.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216638+0000", 2024-05-15T23:17:15.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.691e-06 2024-05-15T23:17:15.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216641+0000", 2024-05-15T23:17:15.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.959e-06 2024-05-15T23:17:15.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216647+0000", 2024-05-15T23:17:15.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3070000000000002e-06 2024-05-15T23:17:15.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.295214+0000", 2024-05-15T23:17:15.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.078567674000000004 2024-05-15T23:17:15.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.295339+0000", 2024-05-15T23:17:15.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000124399 2024-05-15T23:17:15.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:17:15.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.295977+0000", 2024-05-15T23:17:15.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00063794399999999999 2024-05-15T23:17:15.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.308009+0000", 2024-05-15T23:17:15.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.012032774 2024-05-15T23:17:15.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.311655+0000", 2024-05-15T23:17:15.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003645218 2024-05-15T23:17:15.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.311688+0000", 2024-05-15T23:17:15.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3000999999999998e-05 2024-05-15T23:17:15.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.311715+0000", 2024-05-15T23:17:15.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7341000000000001e-05 2024-05-15T23:17:15.573 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.573 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.573 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.573 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280640 2.8 2:135280a4:::benchmark_data_smithi090_78160_object119282:head [delete] snapc 0=[] ondisk+write+known_if_redirected e171)", 2024-05-15T23:17:15.573 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:29.216670+0000", 2024-05-15T23:17:15.573 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 466.30594970800001, 2024-05-15T23:17:15.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.095163318999999996, 2024-05-15T23:17:15.573 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.574 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.574 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.574 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.574 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:17:15.574 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:17:15.574 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280640 2024-05-15T23:17:15.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.574 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.574 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.574 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216670+0000", 2024-05-15T23:17:15.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216670+0000", 2024-05-15T23:17:15.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.575 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216671+0000", 2024-05-15T23:17:15.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.92e-07 2024-05-15T23:17:15.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.575 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216674+0000", 2024-05-15T23:17:15.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2619999999999999e-06 2024-05-15T23:17:15.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.575 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216679+0000", 2024-05-15T23:17:15.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1279999999999999e-06 2024-05-15T23:17:15.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216685+0000", 2024-05-15T23:17:15.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5149999999999997e-06 2024-05-15T23:17:15.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.300794+0000", 2024-05-15T23:17:15.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084108901999999999 2024-05-15T23:17:15.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.300913+0000", 2024-05-15T23:17:15.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000119109 2024-05-15T23:17:15.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:17:15.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.301052+0000", 2024-05-15T23:17:15.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013924200000000001 2024-05-15T23:17:15.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.311630+0000", 2024-05-15T23:17:15.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010577813 2024-05-15T23:17:15.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.311762+0000", 2024-05-15T23:17:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000131886 2024-05-15T23:17:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.311794+0000", 2024-05-15T23:17:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1903999999999998e-05 2024-05-15T23:17:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.311834+0000", 2024-05-15T23:17:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9866000000000001e-05 2024-05-15T23:17:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.579 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.579 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.579 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:280670 2.8 2:10992190:::benchmark_data_smithi090_78160_object119312:head [delete] snapc 0=[] ondisk+write+known_if_redirected e171)", 2024-05-15T23:17:15.579 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:29.216691+0000", 2024-05-15T23:17:15.579 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 466.30592869200001, 2024-05-15T23:17:15.579 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.095212714000000004, 2024-05-15T23:17:15.579 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.579 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.579 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.579 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.579 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:17:15.579 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:17:15.579 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 280670 2024-05-15T23:17:15.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.580 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216691+0000", 2024-05-15T23:17:15.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216691+0000", 2024-05-15T23:17:15.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216692+0000", 2024-05-15T23:17:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9500000000000002e-07 2024-05-15T23:17:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216695+0000", 2024-05-15T23:17:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7650000000000002e-06 2024-05-15T23:17:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216701+0000", 2024-05-15T23:17:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4370000000000004e-06 2024-05-15T23:17:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.216706+0000", 2024-05-15T23:17:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.356e-06 2024-05-15T23:17:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.304125+0000", 2024-05-15T23:17:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.087419038000000004 2024-05-15T23:17:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.304199+0000", 2024-05-15T23:17:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3615999999999994e-05 2024-05-15T23:17:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 0", 2024-05-15T23:17:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.304280+0000", 2024-05-15T23:17:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0523000000000001e-05 2024-05-15T23:17:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.311723+0000", 2024-05-15T23:17:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074435289999999999 2024-05-15T23:17:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.311857+0000", 2024-05-15T23:17:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000134084 2024-05-15T23:17:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.584 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.584 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.311884+0000", 2024-05-15T23:17:15.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6356999999999999e-05 2024-05-15T23:17:15.584 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.584 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.584 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.584 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:29.311904+0000", 2024-05-15T23:17:15.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0414000000000001e-05 2024-05-15T23:17:15.584 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.584 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.584 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.584 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.584 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.585 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:301623 2.0 2:0ba37a79:::benchmark_data_smithi090_78160_object140265:head [delete] snapc 0=[] ondisk+write+known_if_redirected e179)", 2024-05-15T23:17:15.585 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:51.820852+0000", 2024-05-15T23:17:15.585 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 443.70176762, 2024-05-15T23:17:15.585 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.081728914, 2024-05-15T23:17:15.585 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.585 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.585 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.585 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.585 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:17:15.585 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:17:15.585 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 301623 2024-05-15T23:17:15.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.585 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.585 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.820852+0000", 2024-05-15T23:17:15.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.820852+0000", 2024-05-15T23:17:15.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.820853+0000", 2024-05-15T23:17:15.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5799999999999999e-07 2024-05-15T23:17:15.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.587 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.587 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.820856+0000", 2024-05-15T23:17:15.587 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1779999999999999e-06 2024-05-15T23:17:15.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.587 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.587 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.587 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.820862+0000", 2024-05-15T23:17:15.587 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.604e-06 2024-05-15T23:17:15.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.587 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.587 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.587 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.820867+0000", 2024-05-15T23:17:15.587 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3789999999999997e-06 2024-05-15T23:17:15.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.588 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.898687+0000", 2024-05-15T23:17:15.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.077820188999999998 2024-05-15T23:17:15.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.588 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.898784+0000", 2024-05-15T23:17:15.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7077999999999994e-05 2024-05-15T23:17:15.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.588 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:17:15.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.898851+0000", 2024-05-15T23:17:15.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6251e-05 2024-05-15T23:17:15.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.589 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.589 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.902397+0000", 2024-05-15T23:17:15.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0035460410000000002 2024-05-15T23:17:15.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.589 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.589 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.902497+0000", 2024-05-15T23:17:15.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00010021799999999999 2024-05-15T23:17:15.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.589 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.589 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.902541+0000", 2024-05-15T23:17:15.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4520000000000001e-05 2024-05-15T23:17:15.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.590 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.590 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.902581+0000", 2024-05-15T23:17:15.590 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9898000000000003e-05 2024-05-15T23:17:15.590 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.590 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.590 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.590 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.590 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.24745.0:301695 2.0 2.7911d430 (undecoded) ondisk+write+known_if_redirected e179)", 2024-05-15T23:17:15.590 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:09:51.820991+0000", 2024-05-15T23:17:15.590 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 443.70162856500002, 2024-05-15T23:17:15.590 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.081760200000000005, 2024-05-15T23:17:15.590 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.591 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.591 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2024-05-15T23:17:15.591 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.591 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.24745", 2024-05-15T23:17:15.591 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3753718171", 2024-05-15T23:17:15.591 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 301695 2024-05-15T23:17:15.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.591 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.820991+0000", 2024-05-15T23:17:15.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.820991+0000", 2024-05-15T23:17:15.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.820992+0000", 2024-05-15T23:17:15.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4999999999999998e-07 2024-05-15T23:17:15.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.820994+0000", 2024-05-15T23:17:15.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.739e-06 2024-05-15T23:17:15.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.593 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.820997+0000", 2024-05-15T23:17:15.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.45e-06 2024-05-15T23:17:15.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.593 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.821000+0000", 2024-05-15T23:17:15.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8389999999999998e-06 2024-05-15T23:17:15.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.593 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.902743+0000", 2024-05-15T23:17:15.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.081742771000000006 2024-05-15T23:17:15.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:09:51.902752+0000", 2024-05-15T23:17:15.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0510000000000002e-06 2024-05-15T23:17:15.594 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.594 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.594 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.594 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:179408 3.f 3:f0fc2f9c:::benchmark_data_smithi090_90974_object13865:head [delete] snapc 0=[] ondisk+write+known_if_redirected e250)", 2024-05-15T23:17:15.594 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:12:36.495231+0000", 2024-05-15T23:17:15.594 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 279.027388997, 2024-05-15T23:17:15.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.098344532999999998, 2024-05-15T23:17:15.594 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.594 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.594 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.595 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.595 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:17:15.595 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:17:15.595 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 179408 2024-05-15T23:17:15.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.595 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.595 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.495231+0000", 2024-05-15T23:17:15.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.595 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.495231+0000", 2024-05-15T23:17:15.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.495233+0000", 2024-05-15T23:17:15.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.103e-06 2024-05-15T23:17:15.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.495240+0000", 2024-05-15T23:17:15.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7859999999999999e-06 2024-05-15T23:17:15.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.495253+0000", 2024-05-15T23:17:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2998e-05 2024-05-15T23:17:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.495266+0000", 2024-05-15T23:17:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3264e-05 2024-05-15T23:17:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.495313+0000", 2024-05-15T23:17:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7354000000000001e-05 2024-05-15T23:17:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.495429+0000", 2024-05-15T23:17:15.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011514 2024-05-15T23:17:15.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:17:15.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.495541+0000", 2024-05-15T23:17:15.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000112829 2024-05-15T23:17:15.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.506899+0000", 2024-05-15T23:17:15.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011357119000000001 2024-05-15T23:17:15.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.599 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.593463+0000", 2024-05-15T23:17:15.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.086564408999999995 2024-05-15T23:17:15.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.599 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.599 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.593525+0000", 2024-05-15T23:17:15.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2349999999999998e-05 2024-05-15T23:17:15.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.599 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.599 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.593576+0000", 2024-05-15T23:17:15.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0180999999999999e-05 2024-05-15T23:17:15.599 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.599 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.600 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.600 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:179412 3.f 3:fb898cd8:::benchmark_data_smithi090_90974_object13869:head [delete] snapc 0=[] ondisk+write+known_if_redirected e250)", 2024-05-15T23:17:15.600 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:12:36.495667+0000", 2024-05-15T23:17:15.600 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 279.02695298600003, 2024-05-15T23:17:15.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10807837100000001, 2024-05-15T23:17:15.600 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.600 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.600 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.600 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.600 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:17:15.600 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:17:15.600 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 179412 2024-05-15T23:17:15.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.601 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.495667+0000", 2024-05-15T23:17:15.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.495667+0000", 2024-05-15T23:17:15.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.495668+0000", 2024-05-15T23:17:15.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.074e-06 2024-05-15T23:17:15.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.495672+0000", 2024-05-15T23:17:15.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9840000000000003e-06 2024-05-15T23:17:15.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.495680+0000", 2024-05-15T23:17:15.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2199999999999992e-06 2024-05-15T23:17:15.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.495687+0000", 2024-05-15T23:17:15.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4049999999999997e-06 2024-05-15T23:17:15.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.499241+0000", 2024-05-15T23:17:15.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0035539679999999998 2024-05-15T23:17:15.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.499324+0000", 2024-05-15T23:17:15.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2872000000000002e-05 2024-05-15T23:17:15.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:17:15.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.499395+0000", 2024-05-15T23:17:15.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1035e-05 2024-05-15T23:17:15.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.604 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.506901+0000", 2024-05-15T23:17:15.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0075068820000000003 2024-05-15T23:17:15.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.604 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.603638+0000", 2024-05-15T23:17:15.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.096736380999999996 2024-05-15T23:17:15.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.604 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.603698+0000", 2024-05-15T23:17:15.605 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0090000000000002e-05 2024-05-15T23:17:15.605 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.605 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.605 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.603745+0000", 2024-05-15T23:17:15.605 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7460000000000003e-05 2024-05-15T23:17:15.605 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.605 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.605 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.605 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.605 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:179414 3.f 3:f9ff7c1c:::benchmark_data_smithi090_90974_object13871:head [delete] snapc 0=[] ondisk+write+known_if_redirected e250)", 2024-05-15T23:17:15.605 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:12:36.497895+0000", 2024-05-15T23:17:15.605 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 279.02472525799999, 2024-05-15T23:17:15.605 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10595603000000001, 2024-05-15T23:17:15.606 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.606 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.606 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.606 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.606 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:17:15.606 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:17:15.606 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 179414 2024-05-15T23:17:15.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.606 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.606 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.606 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.606 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.497895+0000", 2024-05-15T23:17:15.606 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.607 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.607 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.497895+0000", 2024-05-15T23:17:15.607 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.607 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.607 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.497896+0000", 2024-05-15T23:17:15.607 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.31e-06 2024-05-15T23:17:15.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.607 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.607 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.497900+0000", 2024-05-15T23:17:15.607 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2640000000000002e-06 2024-05-15T23:17:15.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.497910+0000", 2024-05-15T23:17:15.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8889999999999997e-06 2024-05-15T23:17:15.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.497918+0000", 2024-05-15T23:17:15.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8280000000000003e-06 2024-05-15T23:17:15.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.503264+0000", 2024-05-15T23:17:15.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053463199999999999 2024-05-15T23:17:15.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.503341+0000", 2024-05-15T23:17:15.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6934999999999994e-05 2024-05-15T23:17:15.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:17:15.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.503427+0000", 2024-05-15T23:17:15.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5922999999999997e-05 2024-05-15T23:17:15.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.507050+0000", 2024-05-15T23:17:15.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036223710000000001 2024-05-15T23:17:15.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.610 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.610 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.603792+0000", 2024-05-15T23:17:15.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.096742660999999994 2024-05-15T23:17:15.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.610 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.610 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.603824+0000", 2024-05-15T23:17:15.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1956999999999999e-05 2024-05-15T23:17:15.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.610 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.611 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.603851+0000", 2024-05-15T23:17:15.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6571999999999999e-05 2024-05-15T23:17:15.611 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.611 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.611 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.611 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:179422 3.f 3:fe3bf2e1:::benchmark_data_smithi090_90974_object13879:head [delete] snapc 0=[] ondisk+write+known_if_redirected e250)", 2024-05-15T23:17:15.611 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:12:36.507077+0000", 2024-05-15T23:17:15.611 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 279.01554277899999, 2024-05-15T23:17:15.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.100647838, 2024-05-15T23:17:15.611 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.611 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.611 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.612 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.612 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:17:15.612 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:17:15.612 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 179422 2024-05-15T23:17:15.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.612 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.612 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.612 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.507077+0000", 2024-05-15T23:17:15.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.612 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.612 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.507077+0000", 2024-05-15T23:17:15.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.507079+0000", 2024-05-15T23:17:15.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.187e-06 2024-05-15T23:17:15.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.507085+0000", 2024-05-15T23:17:15.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8479999999999997e-06 2024-05-15T23:17:15.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.507098+0000", 2024-05-15T23:17:15.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2632e-05 2024-05-15T23:17:15.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.507107+0000", 2024-05-15T23:17:15.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4709999999999996e-06 2024-05-15T23:17:15.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.507144+0000", 2024-05-15T23:17:15.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7032000000000001e-05 2024-05-15T23:17:15.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.507228+0000", 2024-05-15T23:17:15.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3935000000000002e-05 2024-05-15T23:17:15.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:17:15.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.507298+0000", 2024-05-15T23:17:15.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9220999999999993e-05 2024-05-15T23:17:15.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.510629+0000", 2024-05-15T23:17:15.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003331774 2024-05-15T23:17:15.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.607667+0000", 2024-05-15T23:17:15.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.097037984999999993 2024-05-15T23:17:15.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.607699+0000", 2024-05-15T23:17:15.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1757000000000001e-05 2024-05-15T23:17:15.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.607725+0000", 2024-05-15T23:17:15.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5996e-05 2024-05-15T23:17:15.616 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.616 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.616 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.617 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:179430 3.f 3:fde0fea8:::benchmark_data_smithi090_90974_object13887:head [delete] snapc 0=[] ondisk+write+known_if_redirected e250)", 2024-05-15T23:17:15.617 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:12:36.524095+0000", 2024-05-15T23:17:15.617 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 278.998525046, 2024-05-15T23:17:15.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.083782858000000002, 2024-05-15T23:17:15.617 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.617 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.617 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.617 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.617 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:17:15.617 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:17:15.617 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 179430 2024-05-15T23:17:15.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.617 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.524095+0000", 2024-05-15T23:17:15.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.524095+0000", 2024-05-15T23:17:15.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.524098+0000", 2024-05-15T23:17:15.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.23e-06 2024-05-15T23:17:15.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.524108+0000", 2024-05-15T23:17:15.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.55e-06 2024-05-15T23:17:15.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.524131+0000", 2024-05-15T23:17:15.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3693000000000002e-05 2024-05-15T23:17:15.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.524152+0000", 2024-05-15T23:17:15.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0573000000000001e-05 2024-05-15T23:17:15.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.524282+0000", 2024-05-15T23:17:15.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012983600000000001 2024-05-15T23:17:15.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.524381+0000", 2024-05-15T23:17:15.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9516000000000001e-05 2024-05-15T23:17:15.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:17:15.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.524453+0000", 2024-05-15T23:17:15.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1421999999999995e-05 2024-05-15T23:17:15.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.530581+0000", 2024-05-15T23:17:15.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0061278340000000004 2024-05-15T23:17:15.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.607778+0000", 2024-05-15T23:17:15.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.077196949000000001 2024-05-15T23:17:15.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.607834+0000", 2024-05-15T23:17:15.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6842000000000002e-05 2024-05-15T23:17:15.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.607878+0000", 2024-05-15T23:17:15.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3412999999999999e-05 2024-05-15T23:17:15.622 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.622 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.622 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.622 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:179431 3.f 3:f6a30824:::benchmark_data_smithi090_90974_object13888:head [delete] snapc 0=[] ondisk+write+known_if_redirected e250)", 2024-05-15T23:17:15.622 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:12:36.524465+0000", 2024-05-15T23:17:15.623 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 278.99815468499997, 2024-05-15T23:17:15.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.083486352999999999, 2024-05-15T23:17:15.623 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.623 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.623 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.623 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.623 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:17:15.623 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:17:15.623 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 179431 2024-05-15T23:17:15.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.623 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.524465+0000", 2024-05-15T23:17:15.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.524465+0000", 2024-05-15T23:17:15.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.524466+0000", 2024-05-15T23:17:15.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.127e-06 2024-05-15T23:17:15.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.524471+0000", 2024-05-15T23:17:15.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4580000000000003e-06 2024-05-15T23:17:15.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.524481+0000", 2024-05-15T23:17:15.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0341e-05 2024-05-15T23:17:15.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.524488+0000", 2024-05-15T23:17:15.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.233e-06 2024-05-15T23:17:15.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.626 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.527815+0000", 2024-05-15T23:17:15.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033270299999999999 2024-05-15T23:17:15.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.626 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.527896+0000", 2024-05-15T23:17:15.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0561000000000003e-05 2024-05-15T23:17:15.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:17:15.626 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.527962+0000", 2024-05-15T23:17:15.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6310999999999996e-05 2024-05-15T23:17:15.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.530731+0000", 2024-05-15T23:17:15.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027682499999999999 2024-05-15T23:17:15.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.607906+0000", 2024-05-15T23:17:15.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.077175373000000005 2024-05-15T23:17:15.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.607931+0000", 2024-05-15T23:17:15.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5499000000000001e-05 2024-05-15T23:17:15.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:12:36.607952+0000", 2024-05-15T23:17:15.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0169999999999998e-05 2024-05-15T23:17:15.628 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.628 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.628 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.628 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:307600 3.f 3:f252ef07:::benchmark_data_smithi090_90974_object142057:head [delete] snapc 0=[] ondisk+write+known_if_redirected e274)", 2024-05-15T23:17:15.628 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:14:22.572825+0000", 2024-05-15T23:17:15.628 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 172.94979537200001, 2024-05-15T23:17:15.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11119865299999999, 2024-05-15T23:17:15.629 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.629 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.629 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.629 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.629 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:17:15.629 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:17:15.629 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 307600 2024-05-15T23:17:15.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.629 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.572825+0000", 2024-05-15T23:17:15.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.572825+0000", 2024-05-15T23:17:15.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.572826+0000", 2024-05-15T23:17:15.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.7799999999999999e-06 2024-05-15T23:17:15.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.572832+0000", 2024-05-15T23:17:15.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.502e-06 2024-05-15T23:17:15.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.572844+0000", 2024-05-15T23:17:15.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2091e-05 2024-05-15T23:17:15.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.572852+0000", 2024-05-15T23:17:15.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7559999999999999e-06 2024-05-15T23:17:15.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.572897+0000", 2024-05-15T23:17:15.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5071999999999998e-05 2024-05-15T23:17:15.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.572977+0000", 2024-05-15T23:17:15.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0011000000000004e-05 2024-05-15T23:17:15.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:17:15.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.573048+0000", 2024-05-15T23:17:15.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1168999999999996e-05 2024-05-15T23:17:15.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.576748+0000", 2024-05-15T23:17:15.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036998170000000002 2024-05-15T23:17:15.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.683922+0000", 2024-05-15T23:17:15.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10717386499999999 2024-05-15T23:17:15.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.683993+0000", 2024-05-15T23:17:15.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0989999999999996e-05 2024-05-15T23:17:15.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.684023+0000", 2024-05-15T23:17:15.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0599999999999998e-05 2024-05-15T23:17:15.633 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.633 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.634 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.634 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25135.0:307622 3.f 3:fa83700e:::benchmark_data_smithi090_90974_object142079:head [delete] snapc 0=[] ondisk+write+known_if_redirected e274)", 2024-05-15T23:17:15.634 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:14:22.596144+0000", 2024-05-15T23:17:15.634 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 172.926475783, 2024-05-15T23:17:15.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.098087131999999994, 2024-05-15T23:17:15.634 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:17:15.634 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:17:15.634 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:15.634 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:17:15.634 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25135", 2024-05-15T23:17:15.634 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:17:15.634 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 307622 2024-05-15T23:17:15.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.635 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:17:15.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:17:15.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.596144+0000", 2024-05-15T23:17:15.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:17:15.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.596144+0000", 2024-05-15T23:17:15.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:17:15.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:17:15.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.596146+0000", 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.978e-06 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.596152+0000", 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.412e-06 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.596164+0000", 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2490000000000001e-05 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.596174+0000", 2024-05-15T23:17:15.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5869999999999999e-06 2024-05-15T23:17:15.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:17:15.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.596240+0000", 2024-05-15T23:17:15.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6617000000000003e-05 2024-05-15T23:17:15.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:17:15.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.596370+0000", 2024-05-15T23:17:15.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012983600000000001 2024-05-15T23:17:15.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:17:15.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.596495+0000", 2024-05-15T23:17:15.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012448400000000001 2024-05-15T23:17:15.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:17:15.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.600280+0000", 2024-05-15T23:17:15.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037854350000000002 2024-05-15T23:17:15.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:15.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.694163+0000", 2024-05-15T23:17:15.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.093882660000000007 2024-05-15T23:17:15.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:17:15.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.694195+0000", 2024-05-15T23:17:15.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2744999999999999e-05 2024-05-15T23:17:15.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:17:15.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:17:15.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:17:15.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:14:22.694231+0000", 2024-05-15T23:17:15.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5887999999999999e-05 2024-05-15T23:17:15.639 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.639 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.639 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.639 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:17:15.639 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:17:15.639 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:17:15.640 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:15 smithi084 ceph-mon[39784]: pgmap v909: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:15.640 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:15 smithi090 ceph-mon[44434]: pgmap v909: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:15.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:15.641 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:17:15.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:17:15.742 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:17:15.794 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:15.827 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:17:15.828 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:15.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:15.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:17:15.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:15.967 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:15.988 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:15.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:15.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:16.092 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:16.149 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:16.314 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:16.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:16.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:16.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T23:17:16.422 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:17:16.464 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:16.490 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:17:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:16.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:17:16.594 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:17:16.639 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:16.640 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:16 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2933615759' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:16.668 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:17:16.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:17:16.687 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:17:16.769 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:17:16.803 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:16.827 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:17:16.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:16.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:17:16.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:16 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2933615759' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:16.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:17:16.978 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:17.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:16 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2933615759' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:17.086 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:17:17.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:17.094 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:17:17.195 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:17:17.237 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:17:17.237 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:17:17.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:17:17.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:17:17.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:17:17.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:17:17.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:17:17.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:17:17.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:17:17.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:17:17.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:17:17.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:17:17.239 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:17:17.239 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:17:17.239 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:17:17.239 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:17:17.239 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:17:17.239 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:17:17.239 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:17:17.239 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:17:17.239 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:17:17.239 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:17:17.239 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:17:17.239 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:17:17.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:17:17.259 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:17.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:17.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:17.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:17.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:17.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:17.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:17.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:17.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:17 smithi084 ceph-mon[39784]: pgmap v910: 17 pgs: 17 active+clean; 7.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:17.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:17.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:17 smithi090 ceph-mon[44434]: pgmap v910: 17 pgs: 17 active+clean; 7.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:17 smithi088 ceph-mon[44699]: pgmap v910: 17 pgs: 17 active+clean; 7.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:18.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.214 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:17:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:18.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:17:18.320 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:17:18.350 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:18.380 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:17:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:18.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:17:18.474 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:18.489 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:17:18.530 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:18.561 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:17:18.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:18.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:17:18.653 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.653 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.653 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.664 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:17:18.697 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:18.721 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:17:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:18.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:17:18.827 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:17:18.858 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:18.884 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:17:18.887 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:17:18.888 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:17:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:18.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:17:18.907 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.908 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.908 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.970 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.971 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.971 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:18.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:17:19.043 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:19.333 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:17:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:19.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:17:19.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:17:19.476 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:19.685 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:17:19.687 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.9 to 1 (10000) 2024-05-15T23:17:19.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:19.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:17:19.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T23:17:19.790 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:17:19.830 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:17:19.959 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:19 smithi084 ceph-mon[39784]: pgmap v911: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:17:19.959 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4109176012' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:17:19.959 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:19 smithi084 ceph-mon[39784]: from='client.35696 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:17:19.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:19.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:17:19.970 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:17:20.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:19 smithi090 ceph-mon[44434]: pgmap v911: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:17:20.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4109176012' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:17:20.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:19 smithi090 ceph-mon[44434]: from='client.35696 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:17:20.061 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:17:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:19 smithi088 ceph-mon[44699]: pgmap v911: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:17:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4109176012' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:17:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:19 smithi088 ceph-mon[44699]: from='client.35696 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:17:20.090 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:20.128 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:17:20.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:20.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:17:20.147 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:17:20.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:17:20.273 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:20.438 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:17:20.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:20.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:17:20.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:20.582 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:20.641 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:20.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:20.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:20.745 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:20.780 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:20.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:20 smithi084 ceph-mon[39784]: from='client.35696 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:17:20.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:20 smithi084 ceph-mon[39784]: osdmap e332: 12 total, 12 up, 11 in 2024-05-15T23:17:20.985 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:20.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:21.031 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 10 1 2024-05-15T23:17:21.034 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:21.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:20 smithi090 ceph-mon[44434]: from='client.35696 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:17:21.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:20 smithi090 ceph-mon[44434]: osdmap e332: 12 total, 12 up, 11 in 2024-05-15T23:17:21.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:20 smithi088 ceph-mon[44699]: from='client.35696 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:17:21.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:20 smithi088 ceph-mon[44699]: osdmap e332: 12 total, 12 up, 11 in 2024-05-15T23:17:21.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:17:21.158 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:21.246 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:17:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:21.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:17:21.357 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:21.395 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:21.555 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:21.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:21.560 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:21.661 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:17:21.684 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:21 smithi090 ceph-mon[44434]: pgmap v913: 17 pgs: 17 active+clean; 7.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:21.713 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:21.742 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:17:21.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:21.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:17:21.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:17:21.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:21 smithi084 ceph-mon[39784]: pgmap v913: 17 pgs: 17 active+clean; 7.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:21.903 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:22.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:21 smithi088 ceph-mon[44699]: pgmap v913: 17 pgs: 17 active+clean; 7.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:22.166 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:17:22.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:22.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:17:22.194 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:22.240 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:17:22.241 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:17:22.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:17:22.322 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:22.532 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:17:22.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:22.535 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:17:22.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:17:22.674 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:22.710 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:17:22.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3507673796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:17:22.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:22 smithi090 ceph-mon[44434]: from='client.35699 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:17:23.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3507673796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:17:23.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:22 smithi088 ceph-mon[44699]: from='client.35699 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:17:23.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T23:17:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3507673796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:17:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:22 smithi084 ceph-mon[39784]: from='client.35699 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:17:23.131 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:17:23.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:23.139 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:17:23.240 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:17:23.278 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:23.491 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:17:24.007 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:17:24.007 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:17:24.007 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:23 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:17:24.007 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:23 smithi090 ceph-mon[44434]: from='client.35699 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:17:24.007 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:23 smithi090 ceph-mon[44434]: osdmap e333: 12 total, 12 up, 11 in 2024-05-15T23:17:24.007 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:23 smithi090 ceph-mon[44434]: pgmap v915: 17 pgs: 17 active+clean; 7.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.32k op/s 2024-05-15T23:17:24.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:17:24.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:17:24.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:23 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:17:24.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:23 smithi088 ceph-mon[44699]: from='client.35699 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:17:24.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:23 smithi088 ceph-mon[44699]: osdmap e333: 12 total, 12 up, 11 in 2024-05-15T23:17:24.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:23 smithi088 ceph-mon[44699]: pgmap v915: 17 pgs: 17 active+clean; 7.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.32k op/s 2024-05-15T23:17:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:17:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:17:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:23 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:17:24.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:23 smithi084 ceph-mon[39784]: from='client.35699 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:17:24.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:23 smithi084 ceph-mon[39784]: osdmap e333: 12 total, 12 up, 11 in 2024-05-15T23:17:24.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:23 smithi084 ceph-mon[39784]: pgmap v915: 17 pgs: 17 active+clean; 7.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.32k op/s 2024-05-15T23:17:25.205 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:24 smithi090 ceph-mon[44434]: pgmap v916: 17 pgs: 17 active+clean; 7.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:25.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:24 smithi088 ceph-mon[44699]: pgmap v916: 17 pgs: 17 active+clean; 7.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:25.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:24 smithi084 ceph-mon[39784]: pgmap v916: 17 pgs: 17 active+clean; 7.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:25.575 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:17:25.575 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:17:26.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T23:17:26.737 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:17:26.877 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:17:26.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:26.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:17:26.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:17:27.016 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:27.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:26 smithi090 ceph-mon[44434]: pgmap v917: 17 pgs: 17 active+clean; 7.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:27.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:26 smithi088 ceph-mon[44699]: pgmap v917: 17 pgs: 17 active+clean; 7.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:27.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:26 smithi084 ceph-mon[39784]: pgmap v917: 17 pgs: 17 active+clean; 7.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:27.981 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:17:27.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:27.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:17:28.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:17:28.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:17:28.090 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:17:28.125 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:28.153 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:17:28.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:28.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:17:28.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:28.296 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:28.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:17:28.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:17:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:28.331 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:28.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:28.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:17:28.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:17:28.432 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:28.489 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:28.752 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:28.771 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:17:28.771 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:17:28.853 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:17:28.890 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:28.923 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:17:28.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:28.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:17:29.004 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:29 smithi090 ceph-mon[44434]: osdmap e334: 12 total, 12 up, 11 in 2024-05-15T23:17:29.004 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:29 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:17:29.005 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:29 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:17:29.005 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:29 smithi090 ceph-mon[44434]: pgmap v919: 17 pgs: 17 active+clean; 6.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-05-15T23:17:29.029 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:17:29.067 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:29.100 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:17:29.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:29.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:17:29.210 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:17:29.239 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:29 smithi088 ceph-mon[44699]: osdmap e334: 12 total, 12 up, 11 in 2024-05-15T23:17:29.239 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:29 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:17:29.240 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:29 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:17:29.240 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:29 smithi088 ceph-mon[44699]: pgmap v919: 17 pgs: 17 active+clean; 6.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-05-15T23:17:29.241 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:29.272 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:17:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:29.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:17:29.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:29 smithi084 ceph-mon[39784]: osdmap e334: 12 total, 12 up, 11 in 2024-05-15T23:17:29.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:29 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:17:29.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:29 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:17:29.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:29 smithi084 ceph-mon[39784]: pgmap v919: 17 pgs: 17 active+clean; 6.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-05-15T23:17:29.375 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:17:29.406 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:29.430 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:17:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:29.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:17:29.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T23:17:29.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:17:29.581 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:29.608 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:17:29.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:29.613 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:17:29.714 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:17:29.751 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:29.913 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:17:30.011 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:30 smithi084 ceph-mon[39784]: osdmap e335: 12 total, 12 up, 11 in 2024-05-15T23:17:30.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:30 smithi090 ceph-mon[44434]: osdmap e335: 12 total, 12 up, 11 in 2024-05-15T23:17:30.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:30 smithi088 ceph-mon[44699]: osdmap e335: 12 total, 12 up, 11 in 2024-05-15T23:17:31.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:31 smithi090 ceph-mon[44434]: pgmap v921: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:31 smithi088 ceph-mon[44699]: pgmap v921: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:31.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:31 smithi084 ceph-mon[39784]: pgmap v921: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:17:31.871 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:17:31.871 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:17:32.322 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:17:32.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:32.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:17:32.434 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:17:32.481 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:32.515 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:17:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:32.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:17:32.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T23:17:32.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:17:32.695 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:32.721 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:17:32.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:32.725 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:17:32.826 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:17:32.861 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:32.889 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:17:32.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:32.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:17:32.924 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:17:32.960 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:17:32.967 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:17:32.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:17:33.006 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:17:33.039 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:17:33.042 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:33.079 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:17:33.113 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:17:33.147 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:17:33.181 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:17:33.206 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:17:33.230 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:17:33.248 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1459741301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:33.248 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:32 smithi090 ceph-mon[44434]: pgmap v922: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.33k op/s 2024-05-15T23:17:33.263 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:17:33.292 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:17:33.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:17:33.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:17:33.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:17:33.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:17:33.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:17:33.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:17:33.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:17:33.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:17:33.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:17:33.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:17:33.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:17:33.295 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:17:33.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1459741301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:33.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:32 smithi084 ceph-mon[39784]: pgmap v922: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.33k op/s 2024-05-15T23:17:33.314 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:17:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:33.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:17:33.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1459741301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:33.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:32 smithi088 ceph-mon[44699]: pgmap v922: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.33k op/s 2024-05-15T23:17:33.420 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:17:33.459 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:33.482 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:17:33.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:33.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:17:33.490 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:17:33.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:33.662 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:33.963 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:33.963 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:33.963 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:33.963 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:33.963 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:33.963 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:33.964 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:33.964 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:33.964 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:33.964 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:33.964 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:33 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.125 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:34.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:34.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:34.230 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:34.265 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:34.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.269 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:33 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.298 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:34.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:34.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:33 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:34.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:34.449 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:34.575 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:34.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:34.582 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:34.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:34.732 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:34.757 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:34.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:34.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:34.865 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:34.867 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:34.894 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:34.918 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:34.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:34.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:34.927 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:17:34.927 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:17:34.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:17:34.927 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:17:34.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.928 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:105657 4.6 4:6213edb0:::benchmark_data_smithi090_104448_object105656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-05-15T23:17:34.928 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:15:45.162166+0000", 2024-05-15T23:17:34.928 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 109.759909349, 2024-05-15T23:17:34.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072861380000000003, 2024-05-15T23:17:34.928 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:34.928 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:34.928 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:34.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:34.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:34.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:34.928 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105657 2024-05-15T23:17:34.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.928 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:34.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:34.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.162166+0000", 2024-05-15T23:17:34.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:34.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.162166+0000", 2024-05-15T23:17:34.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:34.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.162168+0000", 2024-05-15T23:17:34.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2689999999999998e-06 2024-05-15T23:17:34.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:34.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.162192+0000", 2024-05-15T23:17:34.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3998999999999998e-05 2024-05-15T23:17:34.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:34.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.162210+0000", 2024-05-15T23:17:34.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8173999999999999e-05 2024-05-15T23:17:34.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:34.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.162219+0000", 2024-05-15T23:17:34.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6249999999999996e-06 2024-05-15T23:17:34.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:34.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.166576+0000", 2024-05-15T23:17:34.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043571959999999998 2024-05-15T23:17:34.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:34.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.166689+0000", 2024-05-15T23:17:34.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011356100000000001 2024-05-15T23:17:34.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:34.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.166813+0000", 2024-05-15T23:17:34.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012386499999999999 2024-05-15T23:17:34.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:34.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170042+0000", 2024-05-15T23:17:34.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032288899999999999 2024-05-15T23:17:34.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:34.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.234963+0000", 2024-05-15T23:17:34.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.064920448000000006 2024-05-15T23:17:34.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:34.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.234997+0000", 2024-05-15T23:17:34.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4371e-05 2024-05-15T23:17:34.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:34.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.235027+0000", 2024-05-15T23:17:34.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9981999999999999e-05 2024-05-15T23:17:34.934 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.934 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:34.934 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.934 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:105678 4.6 4:6cfaded3:::benchmark_data_smithi090_104448_object105677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-05-15T23:17:34.934 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:15:45.170335+0000", 2024-05-15T23:17:34.934 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 109.75173954, 2024-05-15T23:17:34.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.073724063000000006, 2024-05-15T23:17:34.934 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:34.935 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:34.935 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:34.935 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:34.935 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:34.935 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:34.935 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105678 2024-05-15T23:17:34.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.936 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:34.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:34.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170335+0000", 2024-05-15T23:17:34.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:34.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170335+0000", 2024-05-15T23:17:34.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:34.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170337+0000", 2024-05-15T23:17:34.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5099999999999999e-06 2024-05-15T23:17:34.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:34.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170360+0000", 2024-05-15T23:17:34.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3422999999999999e-05 2024-05-15T23:17:34.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:34.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170370+0000", 2024-05-15T23:17:34.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5140000000000007e-06 2024-05-15T23:17:34.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:34.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170377+0000", 2024-05-15T23:17:34.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4290000000000004e-06 2024-05-15T23:17:34.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:34.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170425+0000", 2024-05-15T23:17:34.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7633000000000003e-05 2024-05-15T23:17:34.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:34.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170531+0000", 2024-05-15T23:17:34.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010632 2024-05-15T23:17:34.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:34.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170653+0000", 2024-05-15T23:17:34.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012196 2024-05-15T23:17:34.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:34.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.173861+0000", 2024-05-15T23:17:34.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00320784 2024-05-15T23:17:34.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:34.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.243995+0000", 2024-05-15T23:17:34.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.070134028000000001 2024-05-15T23:17:34.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:34.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.244031+0000", 2024-05-15T23:17:34.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5862999999999999e-05 2024-05-15T23:17:34.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:34.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.244059+0000", 2024-05-15T23:17:34.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8543000000000001e-05 2024-05-15T23:17:34.941 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.941 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:34.941 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.941 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:105697 4.6 4:6cc800a4:::benchmark_data_smithi090_104448_object105696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-05-15T23:17:34.941 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:15:45.184192+0000", 2024-05-15T23:17:34.941 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 109.73788273700001, 2024-05-15T23:17:34.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.059962506999999998, 2024-05-15T23:17:34.942 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:34.942 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:34.942 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:34.942 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:34.942 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:34.942 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:34.942 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105697 2024-05-15T23:17:34.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.942 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:34.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:34.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.184192+0000", 2024-05-15T23:17:34.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:34.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.184192+0000", 2024-05-15T23:17:34.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:34.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.184198+0000", 2024-05-15T23:17:34.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.976e-06 2024-05-15T23:17:34.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:34.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.184221+0000", 2024-05-15T23:17:34.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2339000000000001e-05 2024-05-15T23:17:34.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:34.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.184231+0000", 2024-05-15T23:17:34.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0614999999999999e-05 2024-05-15T23:17:34.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:34.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.184238+0000", 2024-05-15T23:17:34.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-05-15T23:17:34.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:34.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.184297+0000", 2024-05-15T23:17:34.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9722999999999997e-05 2024-05-15T23:17:34.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:34.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.184423+0000", 2024-05-15T23:17:34.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012530500000000001 2024-05-15T23:17:34.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:34.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.184562+0000", 2024-05-15T23:17:34.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013914400000000001 2024-05-15T23:17:34.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:34.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.187678+0000", 2024-05-15T23:17:34.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031164090000000001 2024-05-15T23:17:34.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:34.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.244089+0000", 2024-05-15T23:17:34.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.056411124999999999 2024-05-15T23:17:34.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:34.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.244119+0000", 2024-05-15T23:17:34.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9711999999999999e-05 2024-05-15T23:17:34.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:34.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.244155+0000", 2024-05-15T23:17:34.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5599000000000003e-05 2024-05-15T23:17:34.948 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.948 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:34.948 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.948 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:106102 4.6 4:6008fb69:::benchmark_data_smithi090_104448_object106101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-05-15T23:17:34.948 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:15:45.453865+0000", 2024-05-15T23:17:34.948 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 109.468210051, 2024-05-15T23:17:34.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062887828000000007, 2024-05-15T23:17:34.948 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:34.949 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:34.949 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:34.949 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:34.949 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:34.949 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:34.949 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 106102 2024-05-15T23:17:34.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.949 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:34.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:34.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.453865+0000", 2024-05-15T23:17:34.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:34.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.453865+0000", 2024-05-15T23:17:34.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:34.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.453867+0000", 2024-05-15T23:17:34.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3190000000000002e-06 2024-05-15T23:17:34.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:34.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.453889+0000", 2024-05-15T23:17:34.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2174000000000001e-05 2024-05-15T23:17:34.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.453905+0000", 2024-05-15T23:17:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5685999999999999e-05 2024-05-15T23:17:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.453913+0000", 2024-05-15T23:17:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2150000000000001e-06 2024-05-15T23:17:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.453963+0000", 2024-05-15T23:17:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9524000000000002e-05 2024-05-15T23:17:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.454080+0000", 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011675699999999999 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.454227+0000", 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000147138 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.459497+0000", 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052701809999999997 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:34.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.516637+0000", 2024-05-15T23:17:34.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057139697000000003 2024-05-15T23:17:34.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:34.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.516704+0000", 2024-05-15T23:17:34.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7628999999999999e-05 2024-05-15T23:17:34.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:34.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.516753+0000", 2024-05-15T23:17:34.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8507999999999997e-05 2024-05-15T23:17:34.954 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.954 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:106104 4.6 4:6133851a:::benchmark_data_smithi090_104448_object106103:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:15:45.454833+0000", 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 109.467241544, 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062080499999999997, 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 106104 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.955 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.454833+0000", 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.454833+0000", 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.454835+0000", 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4890000000000001e-06 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:34.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.454856+0000", 2024-05-15T23:17:34.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1092999999999999e-05 2024-05-15T23:17:34.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:34.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.454867+0000", 2024-05-15T23:17:34.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.146e-05 2024-05-15T23:17:34.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:34.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.454875+0000", 2024-05-15T23:17:34.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.34e-06 2024-05-15T23:17:34.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:34.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.456975+0000", 2024-05-15T23:17:34.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021006570000000001 2024-05-15T23:17:34.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:34.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.457064+0000", 2024-05-15T23:17:34.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8852999999999998e-05 2024-05-15T23:17:34.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:34.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.457161+0000", 2024-05-15T23:17:34.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6696999999999999e-05 2024-05-15T23:17:34.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:34.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.459628+0000", 2024-05-15T23:17:34.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024666860000000001 2024-05-15T23:17:34.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:34.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.516812+0000", 2024-05-15T23:17:34.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057184048000000001 2024-05-15T23:17:34.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:34.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.516871+0000", 2024-05-15T23:17:34.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999998e-05 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.516914+0000", 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2917000000000003e-05 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:109402 4.6 4:6dc78191:::benchmark_data_smithi090_104448_object109401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e302)", 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:15:47.504458+0000", 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 107.417616864, 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062464471000000001, 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:34.960 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:34.961 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:34.961 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:34.961 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:34.961 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:34.961 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109402 2024-05-15T23:17:34.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.961 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:34.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:34.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.504458+0000", 2024-05-15T23:17:34.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:34.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.504458+0000", 2024-05-15T23:17:34.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:34.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.504460+0000", 2024-05-15T23:17:34.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5149999999999999e-06 2024-05-15T23:17:34.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:34.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.504478+0000", 2024-05-15T23:17:34.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8719000000000002e-05 2024-05-15T23:17:34.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:34.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.504488+0000", 2024-05-15T23:17:34.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0087e-05 2024-05-15T23:17:34.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:34.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.504497+0000", 2024-05-15T23:17:34.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3930000000000008e-06 2024-05-15T23:17:34.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:34.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.504553+0000", 2024-05-15T23:17:34.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5983999999999998e-05 2024-05-15T23:17:34.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:34.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.504675+0000", 2024-05-15T23:17:34.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012251800000000001 2024-05-15T23:17:34.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:34.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.504829+0000", 2024-05-15T23:17:34.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015396 2024-05-15T23:17:34.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:34.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.508262+0000", 2024-05-15T23:17:34.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034322900000000002 2024-05-15T23:17:34.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:34.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.566847+0000", 2024-05-15T23:17:34.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.058585210999999998 2024-05-15T23:17:34.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:34.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.566883+0000", 2024-05-15T23:17:34.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5772999999999998e-05 2024-05-15T23:17:34.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:34.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.566923+0000", 2024-05-15T23:17:34.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0021000000000002e-05 2024-05-15T23:17:34.965 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.966 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:34.966 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.966 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:109412 4.6 4:6a9dcd47:::benchmark_data_smithi090_104448_object109411:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e302)", 2024-05-15T23:17:34.966 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:15:47.511350+0000", 2024-05-15T23:17:34.966 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 107.41072498699999, 2024-05-15T23:17:34.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.059163551000000002, 2024-05-15T23:17:34.966 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:34.966 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:34.966 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:34.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:34.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:34.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:34.967 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109412 2024-05-15T23:17:34.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.967 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:34.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:34.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.511350+0000", 2024-05-15T23:17:34.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:34.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.511350+0000", 2024-05-15T23:17:34.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.511352+0000", 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3769999999999999e-06 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.511377+0000", 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4525e-05 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.511390+0000", 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3627e-05 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.511400+0000", 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4110000000000002e-06 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.511458+0000", 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8535000000000001e-05 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.511606+0000", 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014747999999999999 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:34.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.511762+0000", 2024-05-15T23:17:34.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015568 2024-05-15T23:17:34.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:34.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.515384+0000", 2024-05-15T23:17:34.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036219820000000002 2024-05-15T23:17:34.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:34.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.570419+0000", 2024-05-15T23:17:34.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055035351000000003 2024-05-15T23:17:34.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:34.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.570462+0000", 2024-05-15T23:17:34.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3010000000000003e-05 2024-05-15T23:17:34.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:34.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:47.570514+0000", 2024-05-15T23:17:34.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1573000000000002e-05 2024-05-15T23:17:34.971 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.971 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:34.971 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.971 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:185451 4.6 4:6aaf72ef:::benchmark_data_smithi090_104448_object12213:head [delete] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-05-15T23:17:34.972 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:16:47.294072+0000", 2024-05-15T23:17:34.972 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.628003409999998, 2024-05-15T23:17:34.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.066180623999999993, 2024-05-15T23:17:34.972 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:34.972 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:34.972 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:34.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:34.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:34.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:34.972 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 185451 2024-05-15T23:17:34.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.972 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:34.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294072+0000", 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294072+0000", 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294073+0000", 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.113e-06 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:34.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294077+0000", 2024-05-15T23:17:34.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0589999999999996e-06 2024-05-15T23:17:34.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:34.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294084+0000", 2024-05-15T23:17:34.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6559999999999992e-06 2024-05-15T23:17:34.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:34.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294089+0000", 2024-05-15T23:17:34.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7369999999999997e-06 2024-05-15T23:17:34.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:34.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294657+0000", 2024-05-15T23:17:34.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056806599999999999 2024-05-15T23:17:34.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:34.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294771+0000", 2024-05-15T23:17:34.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114119 2024-05-15T23:17:34.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:34.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294885+0000", 2024-05-15T23:17:34.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113989 2024-05-15T23:17:34.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:34.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.301849+0000", 2024-05-15T23:17:34.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069633079999999997 2024-05-15T23:17:34.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:34.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.360144+0000", 2024-05-15T23:17:34.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.058295670000000001 2024-05-15T23:17:34.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:34.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.360206+0000", 2024-05-15T23:17:34.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2120999999999997e-05 2024-05-15T23:17:34.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:34.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.360252+0000", 2024-05-15T23:17:34.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5785999999999998e-05 2024-05-15T23:17:34.977 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.977 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:34.977 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.977 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:185455 4.6 4:6e39477b:::benchmark_data_smithi090_104448_object12217:head [delete] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-05-15T23:17:34.977 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:16:47.295082+0000", 2024-05-15T23:17:34.977 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.626993257999999, 2024-05-15T23:17:34.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.065325242000000006, 2024-05-15T23:17:34.977 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:34.977 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:34.977 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:34.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:34.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:34.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:34.978 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 185455 2024-05-15T23:17:34.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.978 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:34.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:34.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.295082+0000", 2024-05-15T23:17:34.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.295082+0000", 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.295083+0000", 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1999999999999998e-07 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.295086+0000", 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3299999999999999e-06 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:34.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.295093+0000", 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6429999999999998e-06 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.295096+0000", 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7639999999999999e-06 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.298250+0000", 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031532740000000002 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.298322+0000", 2024-05-15T23:17:34.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2644000000000004e-05 2024-05-15T23:17:34.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:34.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.298407+0000", 2024-05-15T23:17:34.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5198999999999995e-05 2024-05-15T23:17:34.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:34.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.301988+0000", 2024-05-15T23:17:34.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003580654 2024-05-15T23:17:34.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:34.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.360317+0000", 2024-05-15T23:17:34.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.058329338000000001 2024-05-15T23:17:34.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:34.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.360365+0000", 2024-05-15T23:17:34.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7511e-05 2024-05-15T23:17:34.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:34.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.360407+0000", 2024-05-15T23:17:34.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1965000000000002e-05 2024-05-15T23:17:34.982 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.982 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:34.982 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.983 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:186951 4.6 4:6b5f4b58:::benchmark_data_smithi090_104448_object13713:head [delete] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-05-15T23:17:34.983 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:16:48.492723+0000", 2024-05-15T23:17:34.983 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.429352379000001, 2024-05-15T23:17:34.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068389040999999998, 2024-05-15T23:17:34.983 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:34.983 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:34.983 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:34.983 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:34.983 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:34.983 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:34.983 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 186951 2024-05-15T23:17:34.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.983 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:34.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:34.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492723+0000", 2024-05-15T23:17:34.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:34.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492723+0000", 2024-05-15T23:17:34.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:34.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492725+0000", 2024-05-15T23:17:34.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9700000000000002e-06 2024-05-15T23:17:34.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492732+0000", 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.216e-06 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492744+0000", 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2173e-05 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492753+0000", 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2439999999999999e-06 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492800+0000", 2024-05-15T23:17:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6848999999999998e-05 2024-05-15T23:17:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492909+0000", 2024-05-15T23:17:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010861699999999999 2024-05-15T23:17:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492981+0000", 2024-05-15T23:17:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2712000000000004e-05 2024-05-15T23:17:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.497073+0000", 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040911979999999999 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.561031+0000", 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063958584999999998 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.561066+0000", 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4675999999999998e-05 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.561112+0000", 2024-05-15T23:17:34.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5800999999999997e-05 2024-05-15T23:17:34.988 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.988 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:34.988 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.988 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:186966 4.6 4:669c6b05:::benchmark_data_smithi090_104448_object13728:head [delete] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-05-15T23:17:34.988 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:16:48.500957+0000", 2024-05-15T23:17:34.988 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.421118262, 2024-05-15T23:17:34.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068427644999999995, 2024-05-15T23:17:34.988 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:34.988 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:34.988 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:34.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:34.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:34.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:34.989 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 186966 2024-05-15T23:17:34.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.989 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:34.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:34.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.500957+0000", 2024-05-15T23:17:34.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:34.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.500957+0000", 2024-05-15T23:17:34.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:34.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.500959+0000", 2024-05-15T23:17:34.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0190000000000001e-06 2024-05-15T23:17:34.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:34.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.500965+0000", 2024-05-15T23:17:34.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9499999999999998e-06 2024-05-15T23:17:34.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:34.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.500976+0000", 2024-05-15T23:17:34.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1554e-05 2024-05-15T23:17:34.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:34.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.500987+0000", 2024-05-15T23:17:34.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0451999999999999e-05 2024-05-15T23:17:34.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:34.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.501037+0000", 2024-05-15T23:17:34.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9852999999999999e-05 2024-05-15T23:17:34.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:34.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.501121+0000", 2024-05-15T23:17:34.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4594999999999999e-05 2024-05-15T23:17:34.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:34.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.501202+0000", 2024-05-15T23:17:34.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0992e-05 2024-05-15T23:17:34.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:34.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.505005+0000", 2024-05-15T23:17:34.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038027619999999999 2024-05-15T23:17:34.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:34.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.569295+0000", 2024-05-15T23:17:34.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.064290150000000004 2024-05-15T23:17:34.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:34.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.569346+0000", 2024-05-15T23:17:34.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0605000000000001e-05 2024-05-15T23:17:34.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:34.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.569384+0000", 2024-05-15T23:17:34.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8713e-05 2024-05-15T23:17:34.994 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.994 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:34.994 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:34.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.994 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:186971 4.6 4:61dd3819:::benchmark_data_smithi090_104448_object13733:head [delete] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-05-15T23:17:34.994 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:16:48.509338+0000", 2024-05-15T23:17:34.994 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.412736998, 2024-05-15T23:17:34.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.060167646999999998, 2024-05-15T23:17:34.994 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:34.994 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:34.995 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:34.995 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:34.995 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:34.995 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:34.995 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 186971 2024-05-15T23:17:34.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.995 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:34.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:34.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.509338+0000", 2024-05-15T23:17:34.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:34.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.509338+0000", 2024-05-15T23:17:34.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:34.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:34.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.509341+0000", 2024-05-15T23:17:34.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6989999999999999e-06 2024-05-15T23:17:34.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:34.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.509346+0000", 2024-05-15T23:17:34.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6919999999999999e-06 2024-05-15T23:17:34.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:34.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.509359+0000", 2024-05-15T23:17:34.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3013999999999999e-05 2024-05-15T23:17:34.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:34.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.509368+0000", 2024-05-15T23:17:34.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4270000000000008e-06 2024-05-15T23:17:34.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:34.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.509427+0000", 2024-05-15T23:17:34.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9511e-05 2024-05-15T23:17:34.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:34.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.509551+0000", 2024-05-15T23:17:34.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012391400000000001 2024-05-15T23:17:34.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:34.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.509669+0000", 2024-05-15T23:17:34.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011785699999999999 2024-05-15T23:17:34.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:34.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.517305+0000", 2024-05-15T23:17:34.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.007635686 2024-05-15T23:17:34.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:34.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.569430+0000", 2024-05-15T23:17:34.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.052124787999999998 2024-05-15T23:17:34.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:34.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.569472+0000", 2024-05-15T23:17:34.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2313e-05 2024-05-15T23:17:34.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:34.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:34.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:35.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.569506+0000", 2024-05-15T23:17:35.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3745999999999998e-05 2024-05-15T23:17:35.000 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.000 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:35.000 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.000 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:186974 4.6 4:6ad8faf8:::benchmark_data_smithi090_104448_object13736:head [delete] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-05-15T23:17:35.000 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:16:48.509451+0000", 2024-05-15T23:17:35.000 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.412623607, 2024-05-15T23:17:35.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.060187744000000001, 2024-05-15T23:17:35.000 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:35.000 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:35.000 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:35.001 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:35.001 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:35.001 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:35.001 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 186974 2024-05-15T23:17:35.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.001 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:35.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:35.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.509451+0000", 2024-05-15T23:17:35.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:35.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.509451+0000", 2024-05-15T23:17:35.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:35.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.509453+0000", 2024-05-15T23:17:35.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6899999999999999e-06 2024-05-15T23:17:35.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:35.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.509458+0000", 2024-05-15T23:17:35.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1229999999999999e-06 2024-05-15T23:17:35.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.509468+0000", 2024-05-15T23:17:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9080000000000007e-06 2024-05-15T23:17:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.509475+0000", 2024-05-15T23:17:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3300000000000001e-06 2024-05-15T23:17:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:35.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.513340+0000", 2024-05-15T23:17:35.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038642469999999999 2024-05-15T23:17:35.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:35.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.513444+0000", 2024-05-15T23:17:35.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010477800000000001 2024-05-15T23:17:35.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:35.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.513521+0000", 2024-05-15T23:17:35.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6118000000000005e-05 2024-05-15T23:17:35.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.517388+0000", 2024-05-15T23:17:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003867082 2024-05-15T23:17:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.569555+0000", 2024-05-15T23:17:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.052167099000000001 2024-05-15T23:17:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.569602+0000", 2024-05-15T23:17:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7142999999999999e-05 2024-05-15T23:17:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.569639+0000", 2024-05-15T23:17:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7225999999999999e-05 2024-05-15T23:17:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:220159 4.6 4:629808a1:::benchmark_data_smithi090_104448_object46921:head [delete] snapc 0=[] ondisk+write+known_if_redirected e331)", 2024-05-15T23:17:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:15.535105+0000", 2024-05-15T23:17:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.386970172000002, 2024-05-15T23:17:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.064003843000000005, 2024-05-15T23:17:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 220159 2024-05-15T23:17:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535105+0000", 2024-05-15T23:17:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535105+0000", 2024-05-15T23:17:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535106+0000", 2024-05-15T23:17:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3549999999999999e-06 2024-05-15T23:17:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535111+0000", 2024-05-15T23:17:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5600000000000004e-06 2024-05-15T23:17:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535120+0000", 2024-05-15T23:17:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2830000000000007e-06 2024-05-15T23:17:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535127+0000", 2024-05-15T23:17:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3180000000000001e-06 2024-05-15T23:17:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535170+0000", 2024-05-15T23:17:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2799999999999997e-05 2024-05-15T23:17:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535251+0000", 2024-05-15T23:17:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1169999999999997e-05 2024-05-15T23:17:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535323+0000", 2024-05-15T23:17:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1778000000000003e-05 2024-05-15T23:17:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599006+0000", 2024-05-15T23:17:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063682697999999996 2024-05-15T23:17:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599022+0000", 2024-05-15T23:17:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5883e-05 2024-05-15T23:17:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599066+0000", 2024-05-15T23:17:35.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4277999999999999e-05 2024-05-15T23:17:35.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:35.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599109+0000", 2024-05-15T23:17:35.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2719999999999998e-05 2024-05-15T23:17:35.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.012 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:35.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.012 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:220163 4.6 4:6304f954:::benchmark_data_smithi090_104448_object46925:head [delete] snapc 0=[] ondisk+write+known_if_redirected e331)", 2024-05-15T23:17:35.012 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:15.535442+0000", 2024-05-15T23:17:35.012 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.386633031999999, 2024-05-15T23:17:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063832858000000006, 2024-05-15T23:17:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 220163 2024-05-15T23:17:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535442+0000", 2024-05-15T23:17:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535442+0000", 2024-05-15T23:17:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535443+0000", 2024-05-15T23:17:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.206e-06 2024-05-15T23:17:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535447+0000", 2024-05-15T23:17:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2289999999999997e-06 2024-05-15T23:17:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535455+0000", 2024-05-15T23:17:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4320000000000002e-06 2024-05-15T23:17:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535461+0000", 2024-05-15T23:17:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.908e-06 2024-05-15T23:17:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.538947+0000", 2024-05-15T23:17:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003486121 2024-05-15T23:17:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.539062+0000", 2024-05-15T23:17:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115658 2024-05-15T23:17:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.539174+0000", 2024-05-15T23:17:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011184999999999999 2024-05-15T23:17:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599117+0000", 2024-05-15T23:17:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.059942984999999997 2024-05-15T23:17:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599170+0000", 2024-05-15T23:17:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2212000000000001e-05 2024-05-15T23:17:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599232+0000", 2024-05-15T23:17:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2731000000000006e-05 2024-05-15T23:17:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599275+0000", 2024-05-15T23:17:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2525999999999999e-05 2024-05-15T23:17:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:220164 4.6 4:604e82dd:::benchmark_data_smithi090_104448_object46926:head [delete] snapc 0=[] ondisk+write+known_if_redirected e331)", 2024-05-15T23:17:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:15.535488+0000", 2024-05-15T23:17:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.386586678, 2024-05-15T23:17:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063931615999999997, 2024-05-15T23:17:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 220164 2024-05-15T23:17:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535488+0000", 2024-05-15T23:17:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535488+0000", 2024-05-15T23:17:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535489+0000", 2024-05-15T23:17:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4600000000000005e-07 2024-05-15T23:17:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535491+0000", 2024-05-15T23:17:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4200000000000001e-06 2024-05-15T23:17:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535496+0000", 2024-05-15T23:17:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6709999999999998e-06 2024-05-15T23:17:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535500+0000", 2024-05-15T23:17:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2119999999999997e-06 2024-05-15T23:17:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.542982+0000", 2024-05-15T23:17:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074823600000000004 2024-05-15T23:17:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.543082+0000", 2024-05-15T23:17:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9883000000000006e-05 2024-05-15T23:17:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.543152+0000", 2024-05-15T23:17:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9634000000000004e-05 2024-05-15T23:17:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599329+0000", 2024-05-15T23:17:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.056176822000000001 2024-05-15T23:17:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599342+0000", 2024-05-15T23:17:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2908000000000001e-05 2024-05-15T23:17:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599384+0000", 2024-05-15T23:17:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2701000000000003e-05 2024-05-15T23:17:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599420+0000", 2024-05-15T23:17:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5459000000000001e-05 2024-05-15T23:17:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:220171 4.6 4:62a9cace:::benchmark_data_smithi090_104448_object46933:head [delete] snapc 0=[] ondisk+write+known_if_redirected e331)", 2024-05-15T23:17:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:15.540845+0000", 2024-05-15T23:17:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.381229862000001, 2024-05-15T23:17:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.05869684, 2024-05-15T23:17:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 220171 2024-05-15T23:17:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.540845+0000", 2024-05-15T23:17:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.540845+0000", 2024-05-15T23:17:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.540847+0000", 2024-05-15T23:17:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0430000000000002e-06 2024-05-15T23:17:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.540853+0000", 2024-05-15T23:17:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7799999999999997e-06 2024-05-15T23:17:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.540863+0000", 2024-05-15T23:17:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0428000000000001e-05 2024-05-15T23:17:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.540871+0000", 2024-05-15T23:17:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7989999999999994e-06 2024-05-15T23:17:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.547235+0000", 2024-05-15T23:17:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063638770000000004 2024-05-15T23:17:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.547340+0000", 2024-05-15T23:17:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010529 2024-05-15T23:17:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.547442+0000", 2024-05-15T23:17:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101628 2024-05-15T23:17:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599427+0000", 2024-05-15T23:17:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.051985459999999997 2024-05-15T23:17:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:35.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599466+0000", 2024-05-15T23:17:35.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8722e-05 2024-05-15T23:17:35.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:35.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599516+0000", 2024-05-15T23:17:35.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9699e-05 2024-05-15T23:17:35.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:35.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599542+0000", 2024-05-15T23:17:35.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6114000000000001e-05 2024-05-15T23:17:35.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.030 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:35.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.031 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:220174 4.2 4:48f9bd14:::benchmark_data_smithi090_104448_object46936:head [delete] snapc 0=[] ondisk+write+known_if_redirected e331)", 2024-05-15T23:17:35.031 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:15.544676+0000", 2024-05-15T23:17:35.031 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.377399008000001, 2024-05-15T23:17:35.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062471345999999997, 2024-05-15T23:17:35.031 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:35.031 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:35.031 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:35.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:35.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:35.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:35.031 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 220174 2024-05-15T23:17:35.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.031 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.544676+0000", 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.544676+0000", 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.544678+0000", 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8980000000000001e-06 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:35.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.544684+0000", 2024-05-15T23:17:35.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0510000000000002e-06 2024-05-15T23:17:35.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:35.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.544694+0000", 2024-05-15T23:17:35.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9979999999999998e-06 2024-05-15T23:17:35.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:35.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.544701+0000", 2024-05-15T23:17:35.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2089999999999999e-06 2024-05-15T23:17:35.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:35.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.544750+0000", 2024-05-15T23:17:35.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8962999999999999e-05 2024-05-15T23:17:35.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:35.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.544858+0000", 2024-05-15T23:17:35.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107983 2024-05-15T23:17:35.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:17:35.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.544960+0000", 2024-05-15T23:17:35.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102249 2024-05-15T23:17:35.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.607078+0000", 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062117665000000002 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.607087+0000", 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9649999999999997e-06 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.607118+0000", 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0753000000000002e-05 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.607147+0000", 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9612e-05 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:220176 4.2 4:47a5bebc:::benchmark_data_smithi090_104448_object46938:head [delete] snapc 0=[] ondisk+write+known_if_redirected e331)", 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:15.546323+0000", 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.375751570999999, 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.064068929999999996, 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:35.036 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 220176 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.546323+0000", 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.546323+0000", 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:35.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.546325+0000", 2024-05-15T23:17:35.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.889e-06 2024-05-15T23:17:35.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:35.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.546331+0000", 2024-05-15T23:17:35.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4489999999999998e-06 2024-05-15T23:17:35.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:35.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.546341+0000", 2024-05-15T23:17:35.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0634e-05 2024-05-15T23:17:35.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:35.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.546349+0000", 2024-05-15T23:17:35.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.525e-06 2024-05-15T23:17:35.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:35.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.549631+0000", 2024-05-15T23:17:35.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032817900000000001 2024-05-15T23:17:35.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:35.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.549708+0000", 2024-05-15T23:17:35.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7069000000000004e-05 2024-05-15T23:17:35.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:17:35.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.598961+0000", 2024-05-15T23:17:35.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049253416000000001 2024-05-15T23:17:35.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:35.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.607209+0000", 2024-05-15T23:17:35.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0082474990000000001 2024-05-15T23:17:35.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:35.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.610338+0000", 2024-05-15T23:17:35.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031292210000000002 2024-05-15T23:17:35.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:35.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.610365+0000", 2024-05-15T23:17:35.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7161999999999998e-05 2024-05-15T23:17:35.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:35.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.610392+0000", 2024-05-15T23:17:35.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7276000000000001e-05 2024-05-15T23:17:35.041 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.041 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:35.041 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.041 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:220182 4.2 4:4383beef:::benchmark_data_smithi090_104448_object46944:head [delete] snapc 0=[] ondisk+write+known_if_redirected e331)", 2024-05-15T23:17:35.041 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:15.548488+0000", 2024-05-15T23:17:35.042 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.373587278999999, 2024-05-15T23:17:35.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.065174899999999994, 2024-05-15T23:17:35.042 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:35.042 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:35.042 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:17:35.042 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:35.042 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:35.042 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:35.042 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 220182 2024-05-15T23:17:35.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.042 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:35.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:35.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.548488+0000", 2024-05-15T23:17:35.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:35.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.548488+0000", 2024-05-15T23:17:35.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:35.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:35.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.548489+0000", 2024-05-15T23:17:35.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.668e-06 2024-05-15T23:17:35.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:35.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.548495+0000", 2024-05-15T23:17:35.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4410000000000001e-06 2024-05-15T23:17:35.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:35.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.548505+0000", 2024-05-15T23:17:35.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9399999999999997e-06 2024-05-15T23:17:35.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:35.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.548512+0000", 2024-05-15T23:17:35.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6979999999999998e-06 2024-05-15T23:17:35.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:35.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.603128+0000", 2024-05-15T23:17:35.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.054615489000000003 2024-05-15T23:17:35.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:35.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.603257+0000", 2024-05-15T23:17:35.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012957100000000001 2024-05-15T23:17:35.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:17:35.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.603366+0000", 2024-05-15T23:17:35.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010895600000000001 2024-05-15T23:17:35.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:17:35.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.613563+0000", 2024-05-15T23:17:35.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010197069 2024-05-15T23:17:35.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:35.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.613578+0000", 2024-05-15T23:17:35.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4358999999999999e-05 2024-05-15T23:17:35.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:17:35.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.613623+0000", 2024-05-15T23:17:35.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4727000000000001e-05 2024-05-15T23:17:35.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:35.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:35.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:17:35.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.613663+0000", 2024-05-15T23:17:35.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9981999999999998e-05 2024-05-15T23:17:35.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.047 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:35.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:35.047 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:35.047 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:17:35.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:35.079 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:35.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:35.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:34 smithi084 ceph-mon[39784]: pgmap v923: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:17:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:35.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:35.189 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:35.290 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:35.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:35.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:34 smithi090 ceph-mon[44434]: pgmap v923: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:17:35.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:35.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:34 smithi088 ceph-mon[44699]: pgmap v923: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:17:35.345 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:35.411 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:35.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:35.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:35.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:35.552 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:35.632 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:35.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:35.639 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:35.663 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T23:17:35.740 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:17:35.777 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:35.805 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:17:35.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:35.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:17:35.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:35.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:35 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:17:35.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:35 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:17:35.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:35 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:17:35.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:35 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:17:35.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:35 smithi090 ceph-mon[44434]: osdmap e336: 12 total, 12 up, 11 in 2024-05-15T23:17:35.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:35 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2146376798' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:17:35.995 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:35 smithi090 ceph-mon[44434]: from='client.35705 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:17:35.996 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.10 to 1 (10000) 2024-05-15T23:17:36.039 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:36.064 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:36.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:36.068 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:36.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:35 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:17:36.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:35 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:17:36.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:35 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:17:36.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:35 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:17:36.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:35 smithi088 ceph-mon[44699]: osdmap e336: 12 total, 12 up, 11 in 2024-05-15T23:17:36.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:35 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2146376798' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:17:36.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:35 smithi088 ceph-mon[44699]: from='client.35705 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:17:36.078 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:17:36.095 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:17:36.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:35 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:17:36.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:35 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:17:36.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:35 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:17:36.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:35 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:17:36.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:35 smithi084 ceph-mon[39784]: osdmap e336: 12 total, 12 up, 11 in 2024-05-15T23:17:36.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:35 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2146376798' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:17:36.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:35 smithi084 ceph-mon[39784]: from='client.35705 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:17:36.169 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:17:36.198 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:36.229 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:17:36.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:36.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:17:36.334 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:36.367 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:36.393 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:36.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:36.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:36.499 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:17:36.541 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:36.622 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:17:36.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:36.625 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:17:36.726 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:36.761 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:36.782 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:36.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:36.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:36.886 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:36.930 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:36.957 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:36.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:36.987 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:36 smithi090 ceph-mon[44434]: from='client.35705 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:17:36.987 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:36 smithi090 ceph-mon[44434]: osdmap e337: 12 total, 12 up, 11 in 2024-05-15T23:17:36.987 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:36 smithi090 ceph-mon[44434]: pgmap v926: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-05-15T23:17:37.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:37.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:36 smithi088 ceph-mon[44699]: from='client.35705 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:17:37.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:36 smithi088 ceph-mon[44699]: osdmap e337: 12 total, 12 up, 11 in 2024-05-15T23:17:37.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:36 smithi088 ceph-mon[44699]: pgmap v926: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-05-15T23:17:37.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:36 smithi084 ceph-mon[39784]: from='client.35705 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:17:37.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:36 smithi084 ceph-mon[39784]: osdmap e337: 12 total, 12 up, 11 in 2024-05-15T23:17:37.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:36 smithi084 ceph-mon[39784]: pgmap v926: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-05-15T23:17:37.105 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:37.197 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:37.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:37.249 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-05-15T23:17:37.249 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-05-15T23:17:37.249 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-15T23:17:37.249 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump --format=json 2024-05-15T23:17:37.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:37.338 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:37.421 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:37.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:37.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:37.527 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:17:37.559 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:37.584 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:17:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:37.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:17:37.688 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:17:37.781 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:37.817 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:17:37.824 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:37.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:17:37.928 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:37.964 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:37.994 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:37.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:38.097 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:38.162 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:38.204 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:38.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:38.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:38.215 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:38.242 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:38 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3728715497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:17:38.243 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:38 smithi088 ceph-mon[44699]: from='client.25633 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:17:38.271 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:17:38.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:17:38.348 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:38.508 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:17:38.510 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:38 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3728715497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:17:38.510 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:38 smithi084 ceph-mon[39784]: from='client.25633 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:17:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:38.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:17:38.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:38 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3728715497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:17:38.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:38 smithi090 ceph-mon[44434]: from='client.25633 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:17:38.611 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:17:38.644 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:38.669 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:17:38.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:38.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:17:38.715 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:17:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:17:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:248179 4.e e338/283 4:7f8a262f:::benchmark_data_smithi090_104448_object74941:head v 338'15771, mlcod=338'15768)", 2024-05-15T23:17:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:38.711713+0000", 2024-05-15T23:17:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0029854199999999999, 2024-05-15T23:17:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003010034, 2024-05-15T23:17:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:17:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.711713+0000", 2024-05-15T23:17:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.711713+0000", 2024-05-15T23:17:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.711714+0000", 2024-05-15T23:17:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1790000000000001e-06 2024-05-15T23:17:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.711720+0000", 2024-05-15T23:17:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9449999999999998e-06 2024-05-15T23:17:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.711728+0000", 2024-05-15T23:17:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6759999999999996e-06 2024-05-15T23:17:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.711735+0000", 2024-05-15T23:17:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1430000000000003e-06 2024-05-15T23:17:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.711780+0000", 2024-05-15T23:17:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5266999999999999e-05 2024-05-15T23:17:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.711799+0000", 2024-05-15T23:17:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8822999999999999e-05 2024-05-15T23:17:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:248175 4.7 e338/283 4:ef99ef98:::benchmark_data_smithi090_104448_object74937:head v 338'15583, mlcod=338'15580)", 2024-05-15T23:17:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:38.713078+0000", 2024-05-15T23:17:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.001619753, 2024-05-15T23:17:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016849090000000001, 2024-05-15T23:17:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:17:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.713078+0000", 2024-05-15T23:17:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.713078+0000", 2024-05-15T23:17:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.713079+0000", 2024-05-15T23:17:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.026e-06 2024-05-15T23:17:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.713086+0000", 2024-05-15T23:17:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.392e-06 2024-05-15T23:17:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.713095+0000", 2024-05-15T23:17:38.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8729999999999996e-06 2024-05-15T23:17:38.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:38.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:38.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.713101+0000", 2024-05-15T23:17:38.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6880000000000002e-06 2024-05-15T23:17:38.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:38.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:38.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.713141+0000", 2024-05-15T23:17:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9805999999999998e-05 2024-05-15T23:17:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:38.713159+0000", 2024-05-15T23:17:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7618e-05 2024-05-15T23:17:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-05-15T23:17:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 2 2024-05-15T23:17:38.723 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:17:38.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:17:38.827 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:38.979 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:17:38.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:38.982 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:17:39.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:17:39.123 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:39.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:39 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:17:39.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:39 smithi084 ceph-mon[39784]: from='client.25633 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:17:39.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:39 smithi084 ceph-mon[39784]: osdmap e338: 12 total, 12 up, 11 in 2024-05-15T23:17:39.249 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:39 smithi084 ceph-mon[39784]: pgmap v928: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T23:17:39.413 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:39 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:17:39.413 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:39 smithi090 ceph-mon[44434]: from='client.25633 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:17:39.413 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:39 smithi090 ceph-mon[44434]: osdmap e338: 12 total, 12 up, 11 in 2024-05-15T23:17:39.413 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:39 smithi090 ceph-mon[44434]: pgmap v928: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T23:17:39.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T23:17:39.486 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:17:39.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:39.492 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:17:39.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:39 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:17:39.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:39 smithi088 ceph-mon[44699]: from='client.25633 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:17:39.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:39 smithi088 ceph-mon[44699]: osdmap e338: 12 total, 12 up, 11 in 2024-05-15T23:17:39.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:39 smithi088 ceph-mon[44699]: pgmap v928: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T23:17:39.593 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:17:39.630 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:17:40.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:40 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:17:40.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:40 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:17:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:40 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:17:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:40 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:17:40.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:40 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:17:40.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:40 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:17:40.695 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:17:40.704 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:17:40.705 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:17:40.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:40.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:17:40.737 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:17:40.763 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:17:40.788 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:17:40.810 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:17:40.813 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:17:40.842 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:17:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:40.867 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:17:40.903 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:17:40.933 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:17:40.963 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:17:40.991 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:17:40.997 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:17:40.997 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:17:40.997 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:17:40.997 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:17:40.997 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:17:40.997 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:17:40.997 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:17:40.997 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:17:40.997 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:17:40.998 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:17:40.998 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:17:40.998 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:17:41.202 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:17:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:41.208 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:17:41.274 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:41 smithi084 ceph-mon[39784]: osdmap e339: 12 total, 12 up, 11 in 2024-05-15T23:17:41.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:17:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:41.449 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:41 smithi090 ceph-mon[44434]: osdmap e339: 12 total, 12 up, 11 in 2024-05-15T23:17:41.449 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:41 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3973283433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:41.449 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:41.449 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:41.449 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:41.449 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:41.449 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:41.449 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:41 smithi090 ceph-mon[44434]: pgmap v930: 17 pgs: 17 active+clean; 6.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.40k op/s 2024-05-15T23:17:41.467 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:17:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:41.472 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:17:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:41 smithi088 ceph-mon[44699]: osdmap e339: 12 total, 12 up, 11 in 2024-05-15T23:17:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:41 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3973283433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:41.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:41 smithi088 ceph-mon[44699]: pgmap v930: 17 pgs: 17 active+clean; 6.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.40k op/s 2024-05-15T23:17:41.573 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:17:41.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:41 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3973283433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:41.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:41.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:41.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:41.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:41.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:41.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:41 smithi084 ceph-mon[39784]: pgmap v930: 17 pgs: 17 active+clean; 6.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.40k op/s 2024-05-15T23:17:41.615 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:41.777 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:17:41.777 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:17:42.368 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:17:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:42.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:17:42.477 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:17:42.526 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:42.526 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.527 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.527 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.527 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.527 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.527 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.527 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.541 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.541 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.541 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.541 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.541 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.549 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:17:42.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:42.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:17:42.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T23:17:42.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:42.654 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:17:42.690 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:17:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:42.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:17:42.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:42.874 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:42.903 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:42.906 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:17:43.007 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:43.045 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:43.077 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:43.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:43.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:43.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:17:43.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:43.281 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:43.333 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:43.338 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:17:43.339 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":339,"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","created":"2024-05-15T22:56:41.212493+0000","modified":"2024-05-15T23:17:40.255074+0000","last_up_change":"2024-05-15T23:15:14.027117+0000","last_in_change":"2024-05-15T23:13:53.685360+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T22:59:38.603575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.96207165718078613,"average_primary_affinity":0.95999997854232788,"average_primary_affinity_weighted":0.95999997854232788}},{"pool":4,"pool_name":"unique_pool_2","create_time":"2024-05-15T23:14:57.158112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"286","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.0099999904632568,"score_stable":3.0099999904632568,"optimal_score":0.82999998331069946,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":0.96207165718078613,"average_primary_affinity":0.95999997854232788,"average_primary_affinity_weighted":0.95999997854232788}}],"osds":[{"osd":0,"uuid":"80159cc8-1b6c-4aa6-a114-966786d7be1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":283,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6801","nonce":1084680531}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6802","nonce":1084680531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6804","nonce":1084680531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6803","nonce":1084680531}]},"public_addr":"172.21.15.84:6801/1084680531","cluster_addr":"172.21.15.84:6802/1084680531","heartbeat_back_addr":"172.21.15.84:6804/1084680531","heartbeat_front_addr":"172.21.15.84:6803/1084680531","state":["exists","up"]},{"osd":1,"uuid":"9c2c76c9-ce0f-4c2d-9d96-c87a10187607","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":283,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6805","nonce":2140830229}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6806","nonce":2140830229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6808","nonce":2140830229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6807","nonce":2140830229}]},"public_addr":"172.21.15.84:6805/2140830229","cluster_addr":"172.21.15.84:6806/2140830229","heartbeat_back_addr":"172.21.15.84:6808/2140830229","heartbeat_front_addr":"172.21.15.84:6807/2140830229","state":["exists","up"]},{"osd":2,"uuid":"e8514daa-b5f9-4f2f-a432-53f4a4eb86bc","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":261,"up_from":291,"up_thru":190,"down_at":262,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6809","nonce":941507493}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6810","nonce":941507493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6812","nonce":941507493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6811","nonce":941507493}]},"public_addr":"172.21.15.84:6809/941507493","cluster_addr":"172.21.15.84:6810/941507493","heartbeat_back_addr":"172.21.15.84:6812/941507493","heartbeat_front_addr":"172.21.15.84:6811/941507493","state":["exists","up"]},{"osd":3,"uuid":"5d9965f9-c0b2-48ce-869c-582990985623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":283,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6813","nonce":2204039828}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6814","nonce":2204039828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6816","nonce":2204039828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6815","nonce":2204039828}]},"public_addr":"172.21.15.84:6813/2204039828","cluster_addr":"172.21.15.84:6814/2204039828","heartbeat_back_addr":"172.21.15.84:6816/2204039828","heartbeat_front_addr":"172.21.15.84:6815/2204039828","state":["exists","up"]},{"osd":4,"uuid":"4bc50523-8480-44d2-8d76-31479e839c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":283,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6800","nonce":2267955292}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":2267955292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":2267955292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":2267955292}]},"public_addr":"172.21.15.88:6800/2267955292","cluster_addr":"172.21.15.88:6801/2267955292","heartbeat_back_addr":"172.21.15.88:6803/2267955292","heartbeat_front_addr":"172.21.15.88:6802/2267955292","state":["exists","up"]},{"osd":5,"uuid":"d4db8d7f-afc7-43dc-81fe-f70c290d8b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":661041538}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":661041538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":661041538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":661041538}]},"public_addr":"172.21.15.88:6804/661041538","cluster_addr":"172.21.15.88:6805/661041538","heartbeat_back_addr":"172.21.15.88:6807/661041538","heartbeat_front_addr":"172.21.15.88:6806/661041538","state":["exists","up"]},{"osd":6,"uuid":"a2773454-2318-4a5b-8f31-0952b1b07f2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":283,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":380856869}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":380856869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":380856869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":380856869}]},"public_addr":"172.21.15.88:6808/380856869","cluster_addr":"172.21.15.88:6809/380856869","heartbeat_back_addr":"172.21.15.88:6811/380856869","heartbeat_front_addr":"172.21.15.88:6810/380856869","state":["exists","up"]},{"osd":7,"uuid":"028ca7b6-3e00-4466-a33f-b726cd60cb34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":1044709063}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":1044709063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":1044709063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":1044709063}]},"public_addr":"172.21.15.88:6812/1044709063","cluster_addr":"172.21.15.88:6813/1044709063","heartbeat_back_addr":"172.21.15.88:6815/1044709063","heartbeat_front_addr":"172.21.15.88:6814/1044709063","state":["exists","up"]},{"osd":8,"uuid":"02cab286-d487-4a66-982c-3cf28f55e851","up":1,"in":1,"weight":1,"primary_affinity":0.54486083984375,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6800","nonce":522459126}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6801","nonce":522459126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6803","nonce":522459126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6802","nonce":522459126}]},"public_addr":"172.21.15.90:6800/522459126","cluster_addr":"172.21.15.90:6801/522459126","heartbeat_back_addr":"172.21.15.90:6803/522459126","heartbeat_front_addr":"172.21.15.90:6802/522459126","state":["exists","up"]},{"osd":9,"uuid":"47d38bc7-39ba-4b35-af24-e282354dd613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":283,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6804","nonce":2794571133}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6805","nonce":2794571133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6807","nonce":2794571133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6806","nonce":2794571133}]},"public_addr":"172.21.15.90:6804/2794571133","cluster_addr":"172.21.15.90:6805/2794571133","heartbeat_back_addr":"172.21.15.90:6807/2794571133","heartbeat_front_addr":"172.21.15.90:6806/2794571133","state":["exists","up"]},{"osd":10,"uuid":"d7b2aeae-d012-41eb-9f91-e4c05c649473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6808","nonce":3735380744}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6809","nonce":3735380744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6811","nonce":3735380744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6810","nonce":3735380744}]},"public_addr":"172.21.15.90:6808/3735380744","cluster_addr":"172.21.15.90:6809/3735380744","heartbeat_back_addr":"172.21.15.90:6811/3735380744","heartbeat_front_addr":"172.21.15.90:6810/3735380744","state":["exists","up"]},{"osd":11,"uuid":"2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":283,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6812","nonce":2265615270}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6813","nonce":2265615270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6815","nonce":2265615270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6814","nonce":2265615270}]},"public_addr":"172.21.15.90:6812/2265615270","cluster_addr":"172.21.15.90:6813/2265615270","heartbeat_back_addr":"172.21.15.90:6815/2265615270","heartbeat_front_addr":"172.21.15.90:6814/2265615270","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:17:22.556471+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:16:30.674491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-05-15T23:17:34.773311+0000","dead_epoch":261},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:16:54.387907+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:17:39.528163+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:16:28.825744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:17:27.624583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:17:12.095272+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:17:27.999990+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:17:34.809710+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:16:26.805487+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:16:56.747082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/2581408231":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/1136984393":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/2443201864":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1252671888":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1451001371":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/4084814866":"2024-05-16T22:57:38.505263+0000","172.21.15.84:0/4076409285":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/2988977694":"2024-05-16T22:57:38.505263+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T23:17:43.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:43.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:43.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:43 smithi084 ceph-mon[39784]: pgmap v931: 17 pgs: 17 active+clean; 5.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:17:43.441 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:43.470 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:43.493 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:43.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:43 smithi090 ceph-mon[44434]: pgmap v931: 17 pgs: 17 active+clean; 5.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:17:43.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:43.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:43.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:43 smithi088 ceph-mon[44699]: pgmap v931: 17 pgs: 17 active+clean; 5.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-05-15T23:17:43.603 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:43.634 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:43.659 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:43.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:43.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:43.766 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:17:43.806 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:43.945 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:17:44.274 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:17:44.277 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-15T23:17:44.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:44.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:17:44.352 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:44.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:17:44.417 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:44.432 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:44.548 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/463642445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:17:44.555 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:17:44.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:44.558 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:17:44.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/463642445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:17:44.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/463642445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:17:44.659 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:17:44.712 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:44.886 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:17:44.887 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:17:45.071 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:17:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:45.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:17:45.178 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:45.207 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:45.234 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:45.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:45.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:45.339 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:17:45.386 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:45 smithi090 ceph-mon[44434]: pgmap v932: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:17:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:45 smithi088 ceph-mon[44699]: pgmap v932: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:17:45.597 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:17:45.606 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:45 smithi084 ceph-mon[39784]: pgmap v932: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:17:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:45.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:17:45.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T23:17:45.708 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:45.738 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:45.759 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:45.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:45.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:45.866 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:45.898 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:45.924 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:45.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:45.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:46.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:46.044 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:17:46.080 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:46.331 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:46.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:46 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:17:46.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:46 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:17:46.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:46 smithi090 ceph-mon[44434]: osdmap e340: 12 total, 12 up, 11 in 2024-05-15T23:17:46.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:46.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:46.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:17:46.486 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:46.568 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:17:46.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:46 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:17:46.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:46 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:17:46.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:46 smithi088 ceph-mon[44699]: osdmap e340: 12 total, 12 up, 11 in 2024-05-15T23:17:46.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:46.572 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:17:46.606 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:46 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:17:46.606 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:46 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:17:46.606 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:46 smithi084 ceph-mon[39784]: osdmap e340: 12 total, 12 up, 11 in 2024-05-15T23:17:46.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:46.720 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:46.938 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:46.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:46.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:47.047 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:17:47.090 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:47.207 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:17:47.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:47.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:17:47.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:17:47.354 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:47.393 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:17:47.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:47.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:17:47.500 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:17:47.532 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:47.555 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:17:47.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:47.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:17:47.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:47 smithi084 ceph-mon[39784]: pgmap v934: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:17:47.648 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:17:47.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:17:47.673 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:17:47.690 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:47 smithi090 ceph-mon[44434]: pgmap v934: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:17:47.693 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:47.705 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:17:47.727 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:17:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:47.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:17:47.731 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:17:47.763 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:17:47.804 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:17:47.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:47 smithi088 ceph-mon[44699]: pgmap v934: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:17:47.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:47.840 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:17:47.875 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:17:47.880 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:47.898 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:47.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:47.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:47.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:17:47.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:17:47.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:17:48.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:17:48.039 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:17:48.050 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:48.051 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:17:48.051 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:17:48.051 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:17:48.051 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:17:48.051 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:17:48.051 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:17:48.051 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:17:48.051 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:17:48.051 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:17:48.051 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:17:48.051 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:17:48.051 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:17:48.067 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:17:48.067 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:17:48.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.067 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.25441.0:259480 4.1 e340/283)", 2024-05-15T23:17:48.067 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:48.064390+0000", 2024-05-15T23:17:48.067 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.00099630000000000009, 2024-05-15T23:17:48.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001016241, 2024-05-15T23:17:48.067 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:48.067 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:48.067 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:17:48.067 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:48.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:48.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.064390+0000", 2024-05-15T23:17:48.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:48.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:48.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.064390+0000", 2024-05-15T23:17:48.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:48.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:48.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.064392+0000", 2024-05-15T23:17:48.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3750000000000001e-06 2024-05-15T23:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.064400+0000", 2024-05-15T23:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0590000000000003e-06 2024-05-15T23:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.064417+0000", 2024-05-15T23:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6820000000000002e-05 2024-05-15T23:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.064427+0000", 2024-05-15T23:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0020999999999999e-05 2024-05-15T23:17:48.070 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:48.070 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:48.070 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:48.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.070 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:259472 4.8 4:1a688319:::benchmark_data_smithi090_104448_object86234:head [delete] snapc 0=[] ondisk+write+known_if_redirected e340)", 2024-05-15T23:17:48.070 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:48.048397+0000", 2024-05-15T23:17:48.070 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.016989540000000001, 2024-05-15T23:17:48.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017048319999999999, 2024-05-15T23:17:48.070 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:48.070 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:48.070 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:17:48.070 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:48.070 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:48.070 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:48.071 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 259472 2024-05-15T23:17:48.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.071 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:48.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:48.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.048397+0000", 2024-05-15T23:17:48.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:48.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:48.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.048397+0000", 2024-05-15T23:17:48.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:48.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:48.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.048399+0000", 2024-05-15T23:17:48.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.627e-06 2024-05-15T23:17:48.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:48.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.048405+0000", 2024-05-15T23:17:48.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2609999999999999e-06 2024-05-15T23:17:48.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:48.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.048422+0000", 2024-05-15T23:17:48.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6560000000000001e-05 2024-05-15T23:17:48.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:48.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.048430+0000", 2024-05-15T23:17:48.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2199999999999992e-06 2024-05-15T23:17:48.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:48.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.048486+0000", 2024-05-15T23:17:48.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5671000000000003e-05 2024-05-15T23:17:48.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:48.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.048601+0000", 2024-05-15T23:17:48.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114684 2024-05-15T23:17:48.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:17:48.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.048706+0000", 2024-05-15T23:17:48.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000105213 2024-05-15T23:17:48.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:48.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.059126+0000", 2024-05-15T23:17:48.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010420228 2024-05-15T23:17:48.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:48.074 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:48.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:48.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.075 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:259474 4.8 4:177ea1e0:::benchmark_data_smithi090_104448_object86236:head [delete] snapc 0=[] ondisk+write+known_if_redirected e340)", 2024-05-15T23:17:48.075 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:48.053605+0000", 2024-05-15T23:17:48.075 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.011781052, 2024-05-15T23:17:48.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011890985999999999, 2024-05-15T23:17:48.075 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:48.075 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:48.075 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:17:48.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:48.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:48.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:48.075 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 259474 2024-05-15T23:17:48.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.076 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:48.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:48.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053605+0000", 2024-05-15T23:17:48.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:48.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:48.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053605+0000", 2024-05-15T23:17:48.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:48.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:48.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053607+0000", 2024-05-15T23:17:48.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1069999999999999e-06 2024-05-15T23:17:48.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:48.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053614+0000", 2024-05-15T23:17:48.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3759999999999996e-06 2024-05-15T23:17:48.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:48.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053623+0000", 2024-05-15T23:17:48.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9289999999999993e-06 2024-05-15T23:17:48.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:48.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053632+0000", 2024-05-15T23:17:48.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2519999999999999e-06 2024-05-15T23:17:48.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:48.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053696+0000", 2024-05-15T23:17:48.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4754999999999999e-05 2024-05-15T23:17:48.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:48.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053836+0000", 2024-05-15T23:17:48.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013946 2024-05-15T23:17:48.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:17:48.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053968+0000", 2024-05-15T23:17:48.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013168399999999999 2024-05-15T23:17:48.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:17:48.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.059377+0000", 2024-05-15T23:17:48.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054093079999999998 2024-05-15T23:17:48.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:48.079 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:48.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:48.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.080 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:259475 4.8 4:1be662e1:::benchmark_data_smithi090_104448_object86237:head [delete] snapc 0=[] ondisk+write+known_if_redirected e340)", 2024-05-15T23:17:48.080 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:48.053704+0000", 2024-05-15T23:17:48.080 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.011682428999999999, 2024-05-15T23:17:48.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011838553, 2024-05-15T23:17:48.080 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:48.080 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:48.080 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:17:48.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:48.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:48.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:48.080 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 259475 2024-05-15T23:17:48.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.080 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:48.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:48.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053704+0000", 2024-05-15T23:17:48.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:48.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:48.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053704+0000", 2024-05-15T23:17:48.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:48.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:48.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053705+0000", 2024-05-15T23:17:48.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7700000000000003e-07 2024-05-15T23:17:48.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:48.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053708+0000", 2024-05-15T23:17:48.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9090000000000002e-06 2024-05-15T23:17:48.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:48.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053714+0000", 2024-05-15T23:17:48.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5790000000000003e-06 2024-05-15T23:17:48.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:48.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053718+0000", 2024-05-15T23:17:48.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2679999999999996e-06 2024-05-15T23:17:48.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:48.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.062837+0000", 2024-05-15T23:17:48.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009118981 2024-05-15T23:17:48.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:48.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.062969+0000", 2024-05-15T23:17:48.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013145 2024-05-15T23:17:48.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:17:48.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.063094+0000", 2024-05-15T23:17:48.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125205 2024-05-15T23:17:48.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:48.084 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:48.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:48.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.084 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:259476 4.9 4:92032ba3:::benchmark_data_smithi090_104448_object86238:head [delete] snapc 0=[] ondisk+write+known_if_redirected e340)", 2024-05-15T23:17:48.084 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:48.053730+0000", 2024-05-15T23:17:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.011656270999999999, 2024-05-15T23:17:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011856024999999999, 2024-05-15T23:17:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:17:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 259476 2024-05-15T23:17:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053730+0000", 2024-05-15T23:17:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053730+0000", 2024-05-15T23:17:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053731+0000", 2024-05-15T23:17:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7000000000000004e-07 2024-05-15T23:17:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053733+0000", 2024-05-15T23:17:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7049999999999999e-06 2024-05-15T23:17:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053739+0000", 2024-05-15T23:17:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6060000000000002e-06 2024-05-15T23:17:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053744+0000", 2024-05-15T23:17:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8949999999999997e-06 2024-05-15T23:17:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:48.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.054355+0000", 2024-05-15T23:17:48.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061130500000000003 2024-05-15T23:17:48.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:48.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.054455+0000", 2024-05-15T23:17:48.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9998e-05 2024-05-15T23:17:48.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:48.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.054517+0000", 2024-05-15T23:17:48.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1962000000000001e-05 2024-05-15T23:17:48.088 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:48.089 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:48.089 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:48.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.089 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:259479 4.9 4:9e3045d4:::benchmark_data_smithi090_104448_object86241:head [delete] snapc 0=[] ondisk+write+known_if_redirected e340)", 2024-05-15T23:17:48.089 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:48.053846+0000", 2024-05-15T23:17:48.089 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.011539713, 2024-05-15T23:17:48.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011781185, 2024-05-15T23:17:48.089 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:48.089 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:48.089 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:17:48.089 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 259479 2024-05-15T23:17:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053846+0000", 2024-05-15T23:17:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053846+0000", 2024-05-15T23:17:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053848+0000", 2024-05-15T23:17:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.141e-06 2024-05-15T23:17:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053852+0000", 2024-05-15T23:17:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5759999999999999e-06 2024-05-15T23:17:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053861+0000", 2024-05-15T23:17:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8729999999999996e-06 2024-05-15T23:17:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.053868+0000", 2024-05-15T23:17:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8920000000000003e-06 2024-05-15T23:17:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.060337+0000", 2024-05-15T23:17:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064695129999999997 2024-05-15T23:17:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.060438+0000", 2024-05-15T23:17:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100176 2024-05-15T23:17:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:17:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.060537+0000", 2024-05-15T23:17:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9228999999999995e-05 2024-05-15T23:17:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:259480 4.1 4:8778651f:::benchmark_data_smithi090_104448_object86242:head [delete] snapc 0=[] ondisk+write+known_if_redirected e340)", 2024-05-15T23:17:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:48.054084+0000", 2024-05-15T23:17:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.011301657, 2024-05-15T23:17:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011584773, 2024-05-15T23:17:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:17:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:17:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:17:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:17:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:17:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:17:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 259480 2024-05-15T23:17:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:17:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:17:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.054084+0000", 2024-05-15T23:17:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:17:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.054084+0000", 2024-05-15T23:17:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:17:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:17:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.054086+0000", 2024-05-15T23:17:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.04e-06 2024-05-15T23:17:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:17:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.054090+0000", 2024-05-15T23:17:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1860000000000002e-06 2024-05-15T23:17:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:17:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.054098+0000", 2024-05-15T23:17:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1659999999999994e-06 2024-05-15T23:17:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:17:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.054104+0000", 2024-05-15T23:17:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.55e-06 2024-05-15T23:17:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:17:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.059374+0000", 2024-05-15T23:17:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052692889999999999 2024-05-15T23:17:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:17:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.059503+0000", 2024-05-15T23:17:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012888600000000001 2024-05-15T23:17:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:17:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:17:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:17:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:48.059621+0000", 2024-05-15T23:17:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118022 2024-05-15T23:17:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:17:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:17:48.099 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-05-15T23:17:48.099 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 7 2024-05-15T23:17:48.099 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:17:48.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:48.100 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:17:48.100 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:17:48.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:17:48.238 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:48.299 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:17:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:48.305 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:17:48.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2485534680' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:48.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.406 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:48.461 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:48.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2485534680' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:48.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2485534680' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:48.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:48.920 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:48.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:49.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:49.066 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:49.331 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:49 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2121361596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:17:49.332 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:49 smithi090 ceph-mon[44434]: pgmap v935: 17 pgs: 17 active+clean; 5.4 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:17:49.333 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:17:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:49 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2121361596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:17:49.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:49 smithi084 ceph-mon[39784]: pgmap v935: 17 pgs: 17 active+clean; 5.4 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:17:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:49.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:49 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2121361596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:17:49.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:49 smithi088 ceph-mon[44699]: pgmap v935: 17 pgs: 17 active+clean; 5.4 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:17:49.842 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:49.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T23:17:49.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:49.846 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:49.947 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:49.981 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:50.004 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:50.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:50.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:50.108 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:17:50.137 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:50.159 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:17:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:50.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:17:50.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:17:50.296 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:17:50.309 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:50.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:50 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:17:50.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2121361596' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:17:50.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:50 smithi084 ceph-mon[39784]: osdmap e341: 12 total, 12 up, 11 in 2024-05-15T23:17:50.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:50 smithi084 ceph-mon[39784]: 4.6 scrub starts 2024-05-15T23:17:50.435 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:17:50.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:50.441 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:17:50.542 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:17:50.578 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:50.597 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:17:50.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:50.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:17:50.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:17:50.734 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:50 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:17:50.734 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2121361596' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:17:50.734 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:50 smithi090 ceph-mon[44434]: osdmap e341: 12 total, 12 up, 11 in 2024-05-15T23:17:50.734 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:50 smithi090 ceph-mon[44434]: 4.6 scrub starts 2024-05-15T23:17:50.737 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:50.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:50 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:17:50.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2121361596' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:17:50.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:50 smithi088 ceph-mon[44699]: osdmap e341: 12 total, 12 up, 11 in 2024-05-15T23:17:50.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:50 smithi088 ceph-mon[44699]: 4.6 scrub starts 2024-05-15T23:17:50.881 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:17:51.265 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:51.352 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:51 smithi084 ceph-mon[39784]: 4.6 scrub ok 2024-05-15T23:17:51.352 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:51 smithi084 ceph-mon[39784]: pgmap v937: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:17:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:51 smithi090 ceph-mon[44434]: 4.6 scrub ok 2024-05-15T23:17:51.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:51 smithi090 ceph-mon[44434]: pgmap v937: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:17:51.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:51 smithi088 ceph-mon[44699]: 4.6 scrub ok 2024-05-15T23:17:51.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:51 smithi088 ceph-mon[44699]: pgmap v937: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:17:52.146 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:17:52.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:52 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:17:52.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:52 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:17:52.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:52 smithi084 ceph-mon[39784]: osdmap e342: 12 total, 12 up, 11 in 2024-05-15T23:17:52.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:52 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:17:52.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:52 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:17:52.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:52 smithi090 ceph-mon[44434]: osdmap e342: 12 total, 12 up, 11 in 2024-05-15T23:17:52.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:52 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:17:52.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:52 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:17:52.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:52 smithi088 ceph-mon[44699]: osdmap e342: 12 total, 12 up, 11 in 2024-05-15T23:17:53.299 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:17:53.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:53.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:17:53.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:53 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:17:53.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:53 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:17:53.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:53 smithi084 ceph-mon[39784]: osdmap e343: 12 total, 12 up, 11 in 2024-05-15T23:17:53.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:17:53.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:17:53.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:17:53.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:17:53.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:53 smithi084 ceph-mon[39784]: 4.e scrub starts 2024-05-15T23:17:53.403 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:17:53.419 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:53.468 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:53.477 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:17:53.529 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:17:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:53 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:17:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:53 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:17:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:53 smithi090 ceph-mon[44434]: osdmap e343: 12 total, 12 up, 11 in 2024-05-15T23:17:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:17:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:17:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:17:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:17:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:53 smithi090 ceph-mon[44434]: 4.e scrub starts 2024-05-15T23:17:53.554 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:17:53.586 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:17:53.624 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:17:53.664 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:17:53.703 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:17:53.741 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:17:53.775 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:17:53.809 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:17:53.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:53 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:17:53.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:53 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:17:53.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:53 smithi088 ceph-mon[44699]: osdmap e343: 12 total, 12 up, 11 in 2024-05-15T23:17:53.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:17:53.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:17:53.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:17:53.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:17:53.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:53 smithi088 ceph-mon[44699]: 4.e scrub starts 2024-05-15T23:17:53.835 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:17:53.839 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:17:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:53.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:17:53.870 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:17:53.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:17:53.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:17:53.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:17:53.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:17:53.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:17:53.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:17:53.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:17:53.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:17:53.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:17:53.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:17:53.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:17:53.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:17:53.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:17:53.979 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:54.016 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:17:54.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:54.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:17:54.120 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:54.156 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:54.221 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:54.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:54.327 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:54.372 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:54.398 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:17:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:54.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:17:54.505 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:17:54.543 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:54.569 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:17:54.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:54.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:17:54.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:54.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: 4.0 scrub starts 2024-05-15T23:17:54.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: 4.5 scrub starts 2024-05-15T23:17:54.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/59694751' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:54.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:17:54.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:17:54.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:17:54.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:17:54.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:17:54.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: 4.e scrub ok 2024-05-15T23:17:54.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:54 smithi084 ceph-mon[39784]: 4.f scrub starts 2024-05-15T23:17:54.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: 4.0 scrub starts 2024-05-15T23:17:54.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: 4.5 scrub starts 2024-05-15T23:17:54.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/59694751' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:54.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:17:54.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:17:54.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:17:54.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:17:54.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:17:54.710 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.710 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.710 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.710 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.710 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.710 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.710 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: 4.e scrub ok 2024-05-15T23:17:54.710 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:54 smithi090 ceph-mon[44434]: 4.f scrub starts 2024-05-15T23:17:54.711 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: 4.0 scrub starts 2024-05-15T23:17:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: 4.5 scrub starts 2024-05-15T23:17:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/59694751' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:17:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:17:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:17:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:17:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:17:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:17:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: 4.e scrub ok 2024-05-15T23:17:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:54 smithi088 ceph-mon[44699]: 4.f scrub starts 2024-05-15T23:17:54.841 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:54.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:54.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:54.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:54.993 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:55.023 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:55.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:55.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:55.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:17:55.160 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:55.184 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:17:55.184 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:17:55.201 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:17:55.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:55.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:17:55.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:55.350 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:55.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:55.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:55.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:55.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:55 smithi084 ceph-mon[39784]: 4.0 scrub ok 2024-05-15T23:17:55.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:55 smithi084 ceph-mon[39784]: 4.5 scrub ok 2024-05-15T23:17:55.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:55 smithi084 ceph-mon[39784]: 4.a scrub starts 2024-05-15T23:17:55.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:55 smithi084 ceph-mon[39784]: pgmap v941: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:17:55.637 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:55.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:55.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:55.742 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:17:55.770 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:55.770 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:55.770 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:55.770 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:55 smithi088 ceph-mon[44699]: 4.0 scrub ok 2024-05-15T23:17:55.770 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:55 smithi088 ceph-mon[44699]: 4.5 scrub ok 2024-05-15T23:17:55.770 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:55 smithi088 ceph-mon[44699]: 4.a scrub starts 2024-05-15T23:17:55.770 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:55 smithi088 ceph-mon[44699]: pgmap v941: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:17:55.770 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:55.794 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:17:55.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:55.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:17:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:17:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:55 smithi090 ceph-mon[44434]: 4.0 scrub ok 2024-05-15T23:17:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:55 smithi090 ceph-mon[44434]: 4.5 scrub ok 2024-05-15T23:17:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:55 smithi090 ceph-mon[44434]: 4.a scrub starts 2024-05-15T23:17:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:55 smithi090 ceph-mon[44434]: pgmap v941: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:17:55.901 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:17:55.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T23:17:55.933 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:55.961 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:17:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:55.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:17:56.067 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:17:56.100 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:56.122 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:17:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:56.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:17:56.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:56.283 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:56.305 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:17:56.305 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:56.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:56.310 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:56.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:56.452 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:56.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:56 smithi090 ceph-mon[44434]: 4.a scrub ok 2024-05-15T23:17:56.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:56 smithi090 ceph-mon[44434]: 4.f scrub ok 2024-05-15T23:17:56.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:56 smithi090 ceph-mon[44434]: 4.7 scrub starts 2024-05-15T23:17:56.586 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:56.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:56.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:56.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:56.746 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:56 smithi088 ceph-mon[44699]: 4.a scrub ok 2024-05-15T23:17:56.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:56 smithi088 ceph-mon[44699]: 4.f scrub ok 2024-05-15T23:17:56.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:56 smithi088 ceph-mon[44699]: 4.7 scrub starts 2024-05-15T23:17:56.862 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:56.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:56.866 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:56 smithi084 ceph-mon[39784]: 4.a scrub ok 2024-05-15T23:17:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:56 smithi084 ceph-mon[39784]: 4.f scrub ok 2024-05-15T23:17:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:56 smithi084 ceph-mon[39784]: 4.7 scrub starts 2024-05-15T23:17:56.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:57.019 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:57.233 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:57.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:57.239 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:57.340 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:17:57.410 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:57.437 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:17:57.462 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:17:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:57.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:17:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:57 smithi090 ceph-mon[44434]: 4.3 scrub starts 2024-05-15T23:17:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3848145367' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:17:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:57 smithi090 ceph-mon[44434]: 4.3 scrub ok 2024-05-15T23:17:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:57 smithi090 ceph-mon[44434]: pgmap v942: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:17:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:57 smithi090 ceph-mon[44434]: 4.c scrub starts 2024-05-15T23:17:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:57 smithi090 ceph-mon[44434]: 4.4 scrub starts 2024-05-15T23:17:57.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:17:57.603 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:57.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:17:57.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:57 smithi084 ceph-mon[39784]: 4.3 scrub starts 2024-05-15T23:17:57.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3848145367' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:17:57.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:57 smithi084 ceph-mon[39784]: 4.3 scrub ok 2024-05-15T23:17:57.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:57 smithi084 ceph-mon[39784]: pgmap v942: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:17:57.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:57 smithi084 ceph-mon[39784]: 4.c scrub starts 2024-05-15T23:17:57.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:57 smithi084 ceph-mon[39784]: 4.4 scrub starts 2024-05-15T23:17:57.755 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:17:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:57.763 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:17:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:57 smithi088 ceph-mon[44699]: 4.3 scrub starts 2024-05-15T23:17:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3848145367' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:17:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:57 smithi088 ceph-mon[44699]: 4.3 scrub ok 2024-05-15T23:17:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:57 smithi088 ceph-mon[44699]: pgmap v942: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:17:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:57 smithi088 ceph-mon[44699]: 4.c scrub starts 2024-05-15T23:17:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:57 smithi088 ceph-mon[44699]: 4.4 scrub starts 2024-05-15T23:17:57.864 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:57.897 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:57.974 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:57.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:58.079 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:17:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:58.132 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:17:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:58.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:17:58.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:17:58.272 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:58.306 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:17:58.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:58.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:17:58.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:17:58.418 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:58 smithi088 ceph-mon[44699]: 4.8 scrub starts 2024-05-15T23:17:58.419 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:58 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:17:58.419 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:58 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:17:58.419 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:58 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3848145367' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:17:58.419 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:58 smithi088 ceph-mon[44699]: osdmap e344: 12 total, 12 up, 11 in 2024-05-15T23:17:58.419 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:58 smithi088 ceph-mon[44699]: 4.7 scrub ok 2024-05-15T23:17:58.420 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:58 smithi088 ceph-mon[44699]: 4.8 scrub ok 2024-05-15T23:17:58.420 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:58 smithi088 ceph-mon[44699]: 4.b scrub starts 2024-05-15T23:17:58.420 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:58 smithi088 ceph-mon[44699]: 4.d scrub starts 2024-05-15T23:17:58.450 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:58.469 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:17:58.469 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:17:58.476 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:17:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:58.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:17:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:58 smithi090 ceph-mon[44434]: 4.8 scrub starts 2024-05-15T23:17:58.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:58 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:17:58.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:58 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:17:58.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:58 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3848145367' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:17:58.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:58 smithi090 ceph-mon[44434]: osdmap e344: 12 total, 12 up, 11 in 2024-05-15T23:17:58.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:58 smithi090 ceph-mon[44434]: 4.7 scrub ok 2024-05-15T23:17:58.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:58 smithi090 ceph-mon[44434]: 4.8 scrub ok 2024-05-15T23:17:58.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:58 smithi090 ceph-mon[44434]: 4.b scrub starts 2024-05-15T23:17:58.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:58 smithi090 ceph-mon[44434]: 4.d scrub starts 2024-05-15T23:17:58.580 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:58.624 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:58.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:58 smithi084 ceph-mon[39784]: 4.8 scrub starts 2024-05-15T23:17:58.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:58 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:17:58.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:58 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:17:58.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:58 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3848145367' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:17:58.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:58 smithi084 ceph-mon[39784]: osdmap e344: 12 total, 12 up, 11 in 2024-05-15T23:17:58.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:58 smithi084 ceph-mon[39784]: 4.7 scrub ok 2024-05-15T23:17:58.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:58 smithi084 ceph-mon[39784]: 4.8 scrub ok 2024-05-15T23:17:58.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:58 smithi084 ceph-mon[39784]: 4.b scrub starts 2024-05-15T23:17:58.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:58 smithi084 ceph-mon[39784]: 4.d scrub starts 2024-05-15T23:17:58.895 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:58.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:58.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:59.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:17:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:59.251 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:17:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:59.258 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:17:59.272 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T23:17:59.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:59.425 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:59.485 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:59.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:59 smithi090 ceph-mon[44434]: 4.c scrub ok 2024-05-15T23:17:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:59 smithi090 ceph-mon[44434]: 4.4 scrub ok 2024-05-15T23:17:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:59 smithi090 ceph-mon[44434]: 4.d scrub ok 2024-05-15T23:17:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:17:59 smithi090 ceph-mon[44434]: pgmap v944: 17 pgs: 17 active+clean; 4.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-05-15T23:17:59.558 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:17:59.597 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:17:59.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:59 smithi084 ceph-mon[39784]: 4.c scrub ok 2024-05-15T23:17:59.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:59 smithi084 ceph-mon[39784]: 4.4 scrub ok 2024-05-15T23:17:59.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:59 smithi084 ceph-mon[39784]: 4.d scrub ok 2024-05-15T23:17:59.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:17:59 smithi084 ceph-mon[39784]: pgmap v944: 17 pgs: 17 active+clean; 4.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-05-15T23:17:59.623 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:59.647 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:17:59.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:59.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:17:59.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:17:59.793 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:17:59.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:59 smithi088 ceph-mon[44699]: 4.c scrub ok 2024-05-15T23:17:59.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:59 smithi088 ceph-mon[44699]: 4.4 scrub ok 2024-05-15T23:17:59.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:59 smithi088 ceph-mon[44699]: 4.d scrub ok 2024-05-15T23:17:59.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:17:59 smithi088 ceph-mon[44699]: pgmap v944: 17 pgs: 17 active+clean; 4.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-05-15T23:17:59.826 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:17:59.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:17:59.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:17:59.933 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:17:59.977 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:00.000 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:18:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:00.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:18:00.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:00.152 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:00 smithi090 ceph-mon[44434]: 4.b scrub ok 2024-05-15T23:18:00.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:00 smithi088 ceph-mon[44699]: 4.b scrub ok 2024-05-15T23:18:00.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:00 smithi084 ceph-mon[39784]: 4.b scrub ok 2024-05-15T23:18:00.946 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:18:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:00.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:18:00.974 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:01.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:01.097 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:01.121 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:01.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:01.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:01.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:18:01.260 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:01.286 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:18:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:01.290 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:18:01.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:01.452 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:01.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:01.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:01.522 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:01 smithi084 ceph-mon[39784]: pgmap v945: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.32k op/s 2024-05-15T23:18:01.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:18:01.558 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:18:01.590 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:01.618 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:18:01.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:01.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:18:01.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:18:01.762 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:01.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:01 smithi090 ceph-mon[44434]: pgmap v945: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.32k op/s 2024-05-15T23:18:01.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:01 smithi088 ceph-mon[44699]: pgmap v945: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.32k op/s 2024-05-15T23:18:01.901 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:18:01.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:01.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:18:02.006 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:18:02.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:18:02.032 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:18:02.038 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:02.058 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:18:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:18:02.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:02.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:18:02.082 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:18:02.114 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:18:02.145 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:18:02.170 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:18:02.175 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:18:02.212 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:18:02.223 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:02.246 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:18:02.275 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:18:02.305 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:18:02.308 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:18:02.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:02.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:18:02.357 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:18:02.364 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:18:02.364 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:18:02.364 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:18:02.364 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:18:02.365 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:18:02.365 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:18:02.365 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:18:02.365 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:18:02.365 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:18:02.365 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:18:02.365 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:18:02.365 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:18:02.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:18:02.458 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:02.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:02 smithi084 ceph-mon[39784]: 4.9 scrub starts 2024-05-15T23:18:02.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1216784420' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:02.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:02.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:02.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:02.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:02.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:02 smithi090 ceph-mon[44434]: 4.9 scrub starts 2024-05-15T23:18:02.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1216784420' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:02.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:02.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:02.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:02.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:02.821 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:18:02.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:02 smithi088 ceph-mon[44699]: 4.9 scrub starts 2024-05-15T23:18:02.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1216784420' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:02.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:02.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:02.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:02.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:02.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:18:02.926 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:02.963 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:03.096 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:18:03.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:03.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:18:03.106 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:03.204 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:18:03.234 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:03.259 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:18:03.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:03.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:18:03.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:18:03.404 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:03.432 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:18:03.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:03.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:18:03.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:03 smithi088 ceph-mon[44699]: 4.9 scrub ok 2024-05-15T23:18:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:03 smithi088 ceph-mon[44699]: pgmap v946: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:18:03.607 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:03.631 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:03.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:03.742 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:18:03.769 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:03 smithi090 ceph-mon[44434]: 4.9 scrub ok 2024-05-15T23:18:03.770 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.770 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.770 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.770 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.770 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.770 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.770 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.770 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.770 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:03 smithi090 ceph-mon[44434]: pgmap v946: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:18:03.772 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:03.794 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:18:03.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:03.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:18:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:03 smithi084 ceph-mon[39784]: 4.9 scrub ok 2024-05-15T23:18:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:03 smithi084 ceph-mon[39784]: pgmap v946: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:18:03.900 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:18:03.964 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:04.278 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-15T23:18:04.278 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:18:04.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:04 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:18:04.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:04 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:18:04.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:04 smithi084 ceph-mon[39784]: osdmap e345: 12 total, 12 up, 11 in 2024-05-15T23:18:04.733 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:18:04.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:04.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:18:04.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:04 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:18:04.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:04 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:18:04.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:04 smithi090 ceph-mon[44434]: osdmap e345: 12 total, 12 up, 11 in 2024-05-15T23:18:04.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:04 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:18:04.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:04 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:18:04.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:04 smithi088 ceph-mon[44699]: osdmap e345: 12 total, 12 up, 11 in 2024-05-15T23:18:04.837 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:04.877 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:04.910 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:04.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:05.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:18:05.054 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:05.083 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:18:05.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:05.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:18:05.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:18:05.260 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:05.428 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:18:05.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:05.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:18:05.535 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:05.538 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:18:05.538 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:18:05.538 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-05-15T23:18:05.538 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:18:05.583 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:05.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:18:05.717 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:05 smithi084 ceph-mon[39784]: 4.1 scrub starts 2024-05-15T23:18:05.717 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:05 smithi084 ceph-mon[39784]: 4.2 scrub starts 2024-05-15T23:18:05.717 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:05 smithi084 ceph-mon[39784]: 4.1 scrub ok 2024-05-15T23:18:05.717 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:05 smithi084 ceph-mon[39784]: pgmap v948: 17 pgs: 17 active+clean; 4.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:18:05.718 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:05 smithi084 ceph-mon[39784]: 4.2 scrub ok 2024-05-15T23:18:05.741 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:05.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:05.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:05.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:05 smithi090 ceph-mon[44434]: 4.1 scrub starts 2024-05-15T23:18:05.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:05 smithi090 ceph-mon[44434]: 4.2 scrub starts 2024-05-15T23:18:05.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:05 smithi090 ceph-mon[44434]: 4.1 scrub ok 2024-05-15T23:18:05.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:05 smithi090 ceph-mon[44434]: pgmap v948: 17 pgs: 17 active+clean; 4.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:18:05.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:05 smithi090 ceph-mon[44434]: 4.2 scrub ok 2024-05-15T23:18:05.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:05 smithi088 ceph-mon[44699]: 4.1 scrub starts 2024-05-15T23:18:05.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:05 smithi088 ceph-mon[44699]: 4.2 scrub starts 2024-05-15T23:18:05.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:05 smithi088 ceph-mon[44699]: 4.1 scrub ok 2024-05-15T23:18:05.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:05 smithi088 ceph-mon[44699]: pgmap v948: 17 pgs: 17 active+clean; 4.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:18:05.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:05 smithi088 ceph-mon[44699]: 4.2 scrub ok 2024-05-15T23:18:05.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:18:05.882 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:05.913 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:18:05.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:05.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:18:06.021 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:18:06.056 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:06.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T23:18:06.434 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:18:06.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:06.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:18:06.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:18:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:06.608 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:18:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:06.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:18:06.717 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:06.777 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:06.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:06.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:06.841 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:06.867 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:06.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:18:06.915 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:06.944 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:18:06.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:06.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:18:07.049 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:18:07.080 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:07.111 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:18:07.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:07.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:18:07.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:18:07.258 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:07 smithi090 ceph-mon[44434]: pgmap v949: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:18:07.259 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:07.284 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:18:07.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:07.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:18:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:07 smithi088 ceph-mon[44699]: pgmap v949: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:18:07.376 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:07 smithi084 ceph-mon[39784]: pgmap v949: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:18:07.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:18:07.443 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:07.494 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:18:07.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:07.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:18:07.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:07.638 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:07.665 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:07.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:07.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:18:07.785 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:07.805 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:07.830 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:18:07.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:07.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:18:07.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:07.976 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:08.054 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:18:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/798633586' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:18:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:08 smithi090 ceph-mon[44434]: from='client.35750 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:18:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/798633586' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:18:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:08 smithi088 ceph-mon[44699]: from='client.35750 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:18:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/798633586' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:18:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:08 smithi084 ceph-mon[39784]: from='client.35750 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:18:08.644 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:18:08.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:08.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:18:08.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:08.789 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:08.819 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:08.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:08.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:08.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:08.969 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:09 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:18:09.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:09 smithi090 ceph-mon[44434]: from='client.35750 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:18:09.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:09 smithi090 ceph-mon[44434]: osdmap e346: 12 total, 12 up, 11 in 2024-05-15T23:18:09.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:09 smithi090 ceph-mon[44434]: pgmap v951: 17 pgs: 17 active+clean; 4.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T23:18:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:09 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:18:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:09 smithi088 ceph-mon[44699]: from='client.35750 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:18:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:09 smithi088 ceph-mon[44699]: osdmap e346: 12 total, 12 up, 11 in 2024-05-15T23:18:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:09 smithi088 ceph-mon[44699]: pgmap v951: 17 pgs: 17 active+clean; 4.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T23:18:09.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:09 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:18:09.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:09 smithi084 ceph-mon[39784]: from='client.35750 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:18:09.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:09 smithi084 ceph-mon[39784]: osdmap e346: 12 total, 12 up, 11 in 2024-05-15T23:18:09.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:09 smithi084 ceph-mon[39784]: pgmap v951: 17 pgs: 17 active+clean; 4.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T23:18:09.434 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:09.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:09.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:09.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:18:09.576 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:09.599 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:18:09.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:09.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:18:09.704 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:18:09.755 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:09.971 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:18:09.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:09.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:18:10.075 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:18:10.110 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:10.139 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:18:10.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:10.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:18:10.245 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:18:10.298 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:10.435 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:18:10.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:10.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:18:10.539 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:18:10.569 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:10.594 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:18:10.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:10.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:18:10.702 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:10.731 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:10.759 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:10.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:10.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:10.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:18:10.903 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:10.950 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:18:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:10.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:18:11.054 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:11.098 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:11.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:11 smithi090 ceph-mon[44434]: pgmap v952: 17 pgs: 17 active+clean; 3.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:18:11.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:11 smithi088 ceph-mon[44699]: pgmap v952: 17 pgs: 17 active+clean; 3.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:18:11.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:11 smithi084 ceph-mon[39784]: pgmap v952: 17 pgs: 17 active+clean; 3.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:18:11.478 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:18:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:11.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:18:11.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:18:11.622 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:11.652 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:18:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:11.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:18:11.757 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:11.788 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:11.816 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:11.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:11.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:18:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:11.980 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:18:11.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:11.986 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:18:12.003 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:18:12.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:18:12.121 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:12.150 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:18:12.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:12.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:18:12.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:12.286 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:12.308 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:12.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:12.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:12.413 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:12.444 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:12.477 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:12.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:12.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:12.515 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:18:12.516 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:18:12.516 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:18:12.516 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:18:12.516 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:18:12.583 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:18:12.615 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:12.639 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:18:12.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:12.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:18:12.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:18:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:18:12.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:12.801 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:18:12.902 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:18:12.933 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:12.954 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:18:12.954 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:18:12.954 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:18:12.954 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:18:12.957 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:18:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:12.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:18:12.975 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:18:13.013 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:18:13.046 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:18:13.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:12 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/977706098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:13.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:12 smithi090 ceph-mon[44434]: from='client.35753 v1:172.21.15.84:0/4278302273' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:18:13.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:12 smithi090 ceph-mon[44434]: pgmap v953: 17 pgs: 17 active+clean; 3.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.27k op/s 2024-05-15T23:18:13.060 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:18:13.061 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:18:13.062 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:18:13.071 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:18:13.095 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:18:13.099 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:13.121 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:18:13.125 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:18:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:13.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:18:13.153 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:18:13.181 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:18:13.188 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:18:13.188 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:18:13.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:18:13.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:18:13.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:18:13.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:18:13.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:18:13.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:18:13.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:18:13.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:18:13.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:18:13.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:18:13.213 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:12 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/977706098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:13.213 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:12 smithi084 ceph-mon[39784]: from='client.35753 v1:172.21.15.84:0/4278302273' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:18:13.213 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:12 smithi084 ceph-mon[39784]: pgmap v953: 17 pgs: 17 active+clean; 3.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.27k op/s 2024-05-15T23:18:13.233 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:18:13.262 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:13.289 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:18:13.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:13.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:18:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:12 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/977706098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:12 smithi088 ceph-mon[44699]: from='client.35753 v1:172.21.15.84:0/4278302273' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:18:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:12 smithi088 ceph-mon[44699]: pgmap v953: 17 pgs: 17 active+clean; 3.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.27k op/s 2024-05-15T23:18:13.394 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:13.432 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:13.505 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:13.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:13.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:13.610 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:13.651 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:13.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:13.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:13.776 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:18:13.823 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:13.974 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:13.975 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:13.975 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:13.975 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:13.975 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:13.975 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:13.975 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:13.975 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:13.975 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:13.975 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:13.975 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.084 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:18:14.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:14.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:18:14.111 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T23:18:14.189 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:18:14.217 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:14.241 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:18:14.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:14.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:18:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:14.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:18:14.395 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:14.979 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:18:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:14.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:18:15.021 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":952,"stamp":"2024-05-15T23:18:10.928047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4084073016,"num_objects":62312,"num_object_clones":0,"num_object_copies":124624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62312,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":284285,"num_write_kb":11089191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160756,"ondisk_log_size":160756,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":10887012,"kb_used_data":8003520,"kb_used_omap":17,"kb_used_meta":2883374,"kb_avail":1020309660,"statfs":{"total":1055945392128,"available":1044797091840,"internally_reserved":0,"allocated":8195604480,"data_stored":8191796468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":2952575861},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,14,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-635109376,"num_objects":-9691,"num_object_clones":0,"num_object_copies":-19382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9691,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004584"},"pg_stats":[{"pgid":"4.f","version":"346'17614","reported_seq":17919,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:10.623625+0000","last_change":"2024-05-15T23:17:55.492473+0000","last_active":"2024-05-15T23:18:10.623625+0000","last_peered":"2024-05-15T23:18:10.623625+0000","last_clean":"2024-05-15T23:18:10.623625+0000","last_became_active":"2024-05-15T23:14:58.959283+0000","last_became_peered":"2024-05-15T23:14:58.959283+0000","last_unstale":"2024-05-15T23:18:10.623625+0000","last_undegraded":"2024-05-15T23:18:10.623625+0000","last_fullsized":"2024-05-15T23:18:10.623625+0000","mapping_epoch":283,"log_start":"304'7600","ondisk_log_start":"304'7600","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'16517","last_scrub_stamp":"2024-05-15T23:17:55.492304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:55.492304+0000","objects_scrubbed":4953,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:02.351484+0000","scrub_duration":1175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250347520,"num_objects":3820,"num_object_clones":0,"num_object_copies":7640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17614,"num_write_kb":685888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"346'18098","reported_seq":18403,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:09.726712+0000","last_change":"2024-05-15T23:17:53.836854+0000","last_active":"2024-05-15T23:18:09.726712+0000","last_peered":"2024-05-15T23:18:09.726712+0000","last_clean":"2024-05-15T23:18:09.726712+0000","last_became_active":"2024-05-15T23:14:58.955357+0000","last_became_peered":"2024-05-15T23:14:58.955357+0000","last_unstale":"2024-05-15T23:18:09.726712+0000","last_undegraded":"2024-05-15T23:18:09.726712+0000","last_fullsized":"2024-05-15T23:18:09.726712+0000","mapping_epoch":283,"log_start":"304'8000","ondisk_log_start":"304'8000","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'16950","last_scrub_stamp":"2024-05-15T23:17:53.836683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:53.836683+0000","objects_scrubbed":5027,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:18:54.138019+0000","scrub_duration":813,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252575744,"num_objects":3854,"num_object_clones":0,"num_object_copies":7708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18098,"num_write_kb":702464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"346'17622","reported_seq":17920,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:09.727888+0000","last_change":"2024-05-15T23:17:58.683128+0000","last_active":"2024-05-15T23:18:09.727888+0000","last_peered":"2024-05-15T23:18:09.727888+0000","last_clean":"2024-05-15T23:18:09.727888+0000","last_became_active":"2024-05-15T23:14:58.955461+0000","last_became_peered":"2024-05-15T23:14:58.955461+0000","last_unstale":"2024-05-15T23:18:09.727888+0000","last_undegraded":"2024-05-15T23:18:09.727888+0000","last_fullsized":"2024-05-15T23:18:09.727888+0000","mapping_epoch":283,"log_start":"304'7600","ondisk_log_start":"304'7600","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'16817","last_scrub_stamp":"2024-05-15T23:17:58.682993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:58.682993+0000","objects_scrubbed":4600,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:15.051272+0000","scrub_duration":666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247398424,"num_objects":3776,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17622,"num_write_kb":684673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"346'17768","reported_seq":18070,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:09.653883+0000","last_change":"2024-05-15T23:17:58.124141+0000","last_active":"2024-05-15T23:18:09.653883+0000","last_peered":"2024-05-15T23:18:09.653883+0000","last_clean":"2024-05-15T23:18:09.653883+0000","last_became_active":"2024-05-15T23:14:58.956216+0000","last_became_peered":"2024-05-15T23:14:58.956216+0000","last_unstale":"2024-05-15T23:18:09.653883+0000","last_undegraded":"2024-05-15T23:18:09.653883+0000","last_fullsized":"2024-05-15T23:18:09.653883+0000","mapping_epoch":283,"log_start":"304'7700","ondisk_log_start":"304'7700","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'16873","last_scrub_stamp":"2024-05-15T23:17:58.124011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:58.124011+0000","objects_scrubbed":4825,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:21.394554+0000","scrub_duration":1073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254672896,"num_objects":3886,"num_object_clones":0,"num_object_copies":7772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17768,"num_write_kb":692928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"346'17837","reported_seq":18141,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:09.421139+0000","last_change":"2024-05-15T23:17:59.165263+0000","last_active":"2024-05-15T23:18:09.421139+0000","last_peered":"2024-05-15T23:18:09.421139+0000","last_clean":"2024-05-15T23:18:09.421139+0000","last_became_active":"2024-05-15T23:14:58.959506+0000","last_became_peered":"2024-05-15T23:14:58.959506+0000","last_unstale":"2024-05-15T23:18:09.421139+0000","last_undegraded":"2024-05-15T23:18:09.421139+0000","last_fullsized":"2024-05-15T23:18:09.421139+0000","mapping_epoch":283,"log_start":"304'7800","ondisk_log_start":"304'7800","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'17056","last_scrub_stamp":"2024-05-15T23:17:59.165113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:59.165113+0000","objects_scrubbed":4674,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:16.157458+0000","scrub_duration":1416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250150912,"num_objects":3817,"num_object_clones":0,"num_object_copies":7634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17837,"num_write_kb":692928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"345'17725","reported_seq":18028,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-05-15T23:18:07.663235+0000","last_change":"2024-05-15T23:17:55.102072+0000","last_active":"2024-05-15T23:18:07.663235+0000","last_peered":"2024-05-15T23:18:07.663235+0000","last_clean":"2024-05-15T23:18:07.663235+0000","last_became_active":"2024-05-15T23:14:58.959781+0000","last_became_peered":"2024-05-15T23:14:58.959781+0000","last_unstale":"2024-05-15T23:18:07.663235+0000","last_undegraded":"2024-05-15T23:18:07.663235+0000","last_fullsized":"2024-05-15T23:18:07.663235+0000","mapping_epoch":283,"log_start":"304'7700","ondisk_log_start":"304'7700","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'16853","last_scrub_stamp":"2024-05-15T23:17:55.101904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:55.101904+0000","objects_scrubbed":4963,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:19.324778+0000","scrub_duration":861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266272768,"num_objects":4063,"num_object_clones":0,"num_object_copies":8126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17725,"num_write_kb":697216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"345'17584","reported_seq":17883,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-05-15T23:18:07.696313+0000","last_change":"2024-05-15T23:18:01.854891+0000","last_active":"2024-05-15T23:18:07.696313+0000","last_peered":"2024-05-15T23:18:07.696313+0000","last_clean":"2024-05-15T23:18:07.696313+0000","last_became_active":"2024-05-15T23:14:58.958728+0000","last_became_peered":"2024-05-15T23:14:58.958728+0000","last_unstale":"2024-05-15T23:18:07.696313+0000","last_undegraded":"2024-05-15T23:18:07.696313+0000","last_fullsized":"2024-05-15T23:18:07.696313+0000","mapping_epoch":283,"log_start":"303'7500","ondisk_log_start":"303'7500","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'17148","last_scrub_stamp":"2024-05-15T23:18:01.854660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:18:01.854660+0000","objects_scrubbed":4492,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:23.649452+0000","scrub_duration":1098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263585792,"num_objects":4022,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17584,"num_write_kb":691392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"345'17659","reported_seq":17962,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-05-15T23:18:07.702115+0000","last_change":"2024-05-15T23:17:57.561478+0000","last_active":"2024-05-15T23:18:07.702115+0000","last_peered":"2024-05-15T23:18:07.702115+0000","last_clean":"2024-05-15T23:18:07.702115+0000","last_became_active":"2024-05-15T23:14:59.324101+0000","last_became_peered":"2024-05-15T23:14:59.324101+0000","last_unstale":"2024-05-15T23:18:07.702115+0000","last_undegraded":"2024-05-15T23:18:07.702115+0000","last_fullsized":"2024-05-15T23:18:07.702115+0000","mapping_epoch":283,"log_start":"304'7600","ondisk_log_start":"304'7600","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'16882","last_scrub_stamp":"2024-05-15T23:17:57.561373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:57.561373+0000","objects_scrubbed":4770,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:25.170594+0000","scrub_duration":733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260505600,"num_objects":3975,"num_object_clones":0,"num_object_copies":7950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17659,"num_write_kb":692288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"346'17938","reported_seq":18244,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:10.580524+0000","last_change":"2024-05-15T23:17:57.479856+0000","last_active":"2024-05-15T23:18:10.580524+0000","last_peered":"2024-05-15T23:18:10.580524+0000","last_clean":"2024-05-15T23:18:10.580524+0000","last_became_active":"2024-05-15T23:14:58.958730+0000","last_became_peered":"2024-05-15T23:14:58.958730+0000","last_unstale":"2024-05-15T23:18:10.580524+0000","last_undegraded":"2024-05-15T23:18:10.580524+0000","last_fullsized":"2024-05-15T23:18:10.580524+0000","mapping_epoch":283,"log_start":"304'7900","ondisk_log_start":"304'7900","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'17021","last_scrub_stamp":"2024-05-15T23:17:57.479570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:57.479570+0000","objects_scrubbed":4726,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:04.018226+0000","scrub_duration":1150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246677504,"num_objects":3764,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17938,"num_write_kb":694464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"346'18001","reported_seq":18313,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:10.429834+0000","last_change":"2024-05-15T23:17:50.662752+0000","last_active":"2024-05-15T23:18:10.429834+0000","last_peered":"2024-05-15T23:18:10.429834+0000","last_clean":"2024-05-15T23:18:10.429834+0000","last_became_active":"2024-05-15T23:14:58.958613+0000","last_became_peered":"2024-05-15T23:14:58.958613+0000","last_unstale":"2024-05-15T23:18:10.429834+0000","last_undegraded":"2024-05-15T23:18:10.429834+0000","last_fullsized":"2024-05-15T23:18:10.429834+0000","mapping_epoch":283,"log_start":"305'8000","ondisk_log_start":"305'8000","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'16511","last_scrub_stamp":"2024-05-15T23:17:50.662532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:50.662532+0000","objects_scrubbed":5403,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:18:57.454753+0000","scrub_duration":1208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253427712,"num_objects":3867,"num_object_clones":0,"num_object_copies":7734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18001,"num_write_kb":699776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"346'17640","reported_seq":17939,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:10.613486+0000","last_change":"2024-05-15T23:17:58.503743+0000","last_active":"2024-05-15T23:18:10.613486+0000","last_peered":"2024-05-15T23:18:10.613486+0000","last_clean":"2024-05-15T23:18:10.613486+0000","last_became_active":"2024-05-15T23:14:58.959588+0000","last_became_peered":"2024-05-15T23:14:58.959588+0000","last_unstale":"2024-05-15T23:18:10.613486+0000","last_undegraded":"2024-05-15T23:18:10.613486+0000","last_fullsized":"2024-05-15T23:18:10.613486+0000","mapping_epoch":283,"log_start":"304'7600","ondisk_log_start":"304'7600","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'16810","last_scrub_stamp":"2024-05-15T23:17:58.503579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:58.503579+0000","objects_scrubbed":4624,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:19.485194+0000","scrub_duration":1188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246546432,"num_objects":3762,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17640,"num_write_kb":684864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"345'17738","reported_seq":18044,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-05-15T23:18:07.707023+0000","last_change":"2024-05-15T23:17:56.650874+0000","last_active":"2024-05-15T23:18:07.707023+0000","last_peered":"2024-05-15T23:18:07.707023+0000","last_clean":"2024-05-15T23:18:07.707023+0000","last_became_active":"2024-05-15T23:14:59.324138+0000","last_became_peered":"2024-05-15T23:14:59.324138+0000","last_unstale":"2024-05-15T23:18:07.707023+0000","last_undegraded":"2024-05-15T23:18:07.707023+0000","last_fullsized":"2024-05-15T23:18:07.707023+0000","mapping_epoch":283,"log_start":"304'7700","ondisk_log_start":"304'7700","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'16967","last_scrub_stamp":"2024-05-15T23:17:56.650719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:56.650719+0000","objects_scrubbed":4848,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:19.007279+0000","scrub_duration":867,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264896512,"num_objects":4042,"num_object_clones":0,"num_object_copies":8084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17738,"num_write_kb":696960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"346'17643","reported_seq":17943,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:10.440006+0000","last_change":"2024-05-15T23:18:05.241047+0000","last_active":"2024-05-15T23:18:10.440006+0000","last_peered":"2024-05-15T23:18:10.440006+0000","last_clean":"2024-05-15T23:18:10.440006+0000","last_became_active":"2024-05-15T23:14:59.079701+0000","last_became_peered":"2024-05-15T23:14:59.079701+0000","last_unstale":"2024-05-15T23:18:10.440006+0000","last_undegraded":"2024-05-15T23:18:10.440006+0000","last_fullsized":"2024-05-15T23:18:10.440006+0000","mapping_epoch":283,"log_start":"304'7600","ondisk_log_start":"304'7600","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'17206","last_scrub_stamp":"2024-05-15T23:18:05.240792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:18:05.240792+0000","objects_scrubbed":4319,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:07.646094+0000","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252641280,"num_objects":3855,"num_object_clones":0,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17643,"num_write_kb":687936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"345'17661","reported_seq":17961,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-05-15T23:18:07.749445+0000","last_change":"2024-05-15T23:18:04.556346+0000","last_active":"2024-05-15T23:18:07.749445+0000","last_peered":"2024-05-15T23:18:07.749445+0000","last_clean":"2024-05-15T23:18:07.749445+0000","last_became_active":"2024-05-15T23:14:59.079345+0000","last_became_peered":"2024-05-15T23:14:59.079345+0000","last_unstale":"2024-05-15T23:18:07.749445+0000","last_undegraded":"2024-05-15T23:18:07.749445+0000","last_fullsized":"2024-05-15T23:18:07.749445+0000","mapping_epoch":283,"log_start":"303'7600","ondisk_log_start":"303'7600","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'17437","last_scrub_stamp":"2024-05-15T23:18:04.556086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:18:04.556086+0000","objects_scrubbed":4249,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:25.154900+0000","scrub_duration":740,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262340608,"num_objects":4003,"num_object_clones":0,"num_object_copies":8006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17661,"num_write_kb":693248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"346'17805","reported_seq":18110,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:10.695475+0000","last_change":"2024-05-15T23:17:54.035243+0000","last_active":"2024-05-15T23:18:10.695475+0000","last_peered":"2024-05-15T23:18:10.695475+0000","last_clean":"2024-05-15T23:18:10.695475+0000","last_became_active":"2024-05-15T23:14:59.324368+0000","last_became_peered":"2024-05-15T23:14:59.324368+0000","last_unstale":"2024-05-15T23:18:10.695475+0000","last_undegraded":"2024-05-15T23:18:10.695475+0000","last_fullsized":"2024-05-15T23:18:10.695475+0000","mapping_epoch":283,"log_start":"304'7800","ondisk_log_start":"304'7800","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'16574","last_scrub_stamp":"2024-05-15T23:17:54.035140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:54.035140+0000","objects_scrubbed":5053,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:00.556275+0000","scrub_duration":1240,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246874112,"num_objects":3767,"num_object_clones":0,"num_object_copies":7534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17805,"num_write_kb":690304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"345'17829","reported_seq":18131,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-05-15T23:18:07.685084+0000","last_change":"2024-05-15T23:17:54.077213+0000","last_active":"2024-05-15T23:18:07.685084+0000","last_peered":"2024-05-15T23:18:07.685084+0000","last_clean":"2024-05-15T23:18:07.685084+0000","last_became_active":"2024-05-15T23:14:58.958832+0000","last_became_peered":"2024-05-15T23:14:58.958832+0000","last_unstale":"2024-05-15T23:18:07.685084+0000","last_undegraded":"2024-05-15T23:18:07.685084+0000","last_fullsized":"2024-05-15T23:18:07.685084+0000","mapping_epoch":283,"log_start":"304'7800","ondisk_log_start":"304'7800","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'16858","last_scrub_stamp":"2024-05-15T23:17:54.076977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:54.076977+0000","objects_scrubbed":5039,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:04.883487+0000","scrub_duration":806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264568832,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17829,"num_write_kb":699712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":713,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:08.026310+0000","last_change":"2024-05-15T23:17:07.258646+0000","last_active":"2024-05-15T23:18:08.026310+0000","last_peered":"2024-05-15T23:18:08.026310+0000","last_clean":"2024-05-15T23:18:08.026310+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:18:08.026310+0000","last_undegraded":"2024-05-15T23:18:08.026310+0000","last_fullsized":"2024-05-15T23:18:08.026310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:17:07.258511+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:17:07.258511+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:18:08.293963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4083482648,"num_objects":62310,"num_object_clones":0,"num_object_copies":124620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62310,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":284162,"num_write_kb":11087041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8106024960,"data_stored":8106016816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351148,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1454504,"kb_used_data":1044320,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92290648,"statfs":{"total":95995035648,"available":94505623552,"internally_reserved":0,"allocated":1069383680,"data_stored":1069039716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":420018628},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580082,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887340,"kb_used_data":739556,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92857812,"statfs":{"total":95995035648,"available":95086399488,"internally_reserved":0,"allocated":757305344,"data_stored":756958898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776314,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1061364,"kb_used_data":733096,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92683788,"statfs":{"total":95995035648,"available":94908198912,"internally_reserved":0,"allocated":750690304,"data_stored":750339786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":336132544},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":144304,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93600848,"statfs":{"total":95995035648,"available":95847268352,"internally_reserved":0,"allocated":692224,"data_stored":345778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266882,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988084,"kb_used_data":740332,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92757068,"statfs":{"total":95995035648,"available":94983237632,"internally_reserved":0,"allocated":758099968,"data_stored":757749970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":253688266},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495816,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653792,"kb_used_data":524056,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93091360,"statfs":{"total":95995035648,"available":95325552640,"internally_reserved":0,"allocated":536633344,"data_stored":536292952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575074,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2045036,"kb_used_data":1711204,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91700116,"statfs":{"total":95995035648,"available":93900918784,"internally_reserved":0,"allocated":1752272896,"data_stored":1751926450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":341834182},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738600,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":539248,"kb_used_data":247076,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93205904,"statfs":{"total":95995035648,"available":95442845696,"internally_reserved":0,"allocated":253005824,"data_stored":252659378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483172,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1346732,"kb_used_data":986532,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92398420,"statfs":{"total":95995035648,"available":94615982080,"internally_reserved":0,"allocated":1010208768,"data_stored":1009862322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":368835015},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986385,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1032944,"kb_used_data":749096,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":92712208,"statfs":{"total":95995035648,"available":94937300992,"internally_reserved":0,"allocated":767074304,"data_stored":766723786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790156,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":733664,"kb_used_data":527576,"kb_used_omap":1,"kb_used_meta":206078,"kb_avail":93011488,"statfs":{"total":95995035648,"available":95243763712,"internally_reserved":0,"allocated":540237824,"data_stored":539897432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":211024325},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":250019840,"data_stored":250019840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1737490432,"data_stored":1737490432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":992739328,"data_stored":992739328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":754585600,"data_stored":754581528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":537985024,"data_stored":537985024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":530710528,"data_stored":530710528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":753074176,"data_stored":753074176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":749801472,"data_stored":749797400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":748421120,"data_stored":748421120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1051197440,"data_stored":1051197440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:18:15.023 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:18:15.023 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:15.085 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:18:15.119 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:15.142 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:18:15.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:15.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:18:15.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:18:15.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:15.280 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:15 smithi090 ceph-mon[44434]: pgmap v954: 17 pgs: 17 active+clean; 3.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:18:15.282 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:15.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:15.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:15 smithi084 ceph-mon[39784]: pgmap v954: 17 pgs: 17 active+clean; 3.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:18:15.315 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:18:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:15 smithi088 ceph-mon[44699]: pgmap v954: 17 pgs: 17 active+clean; 3.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:18:15.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:15.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:18:15.424 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:18:15.477 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:15.851 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:16.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:18:16.794 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:16.948 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:18:16.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:16.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:18:17.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:18:17.086 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:17.113 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:18:17.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:17.117 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:18:17.218 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:17.252 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:17.282 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:17.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:17.292 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:17.292 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:17 smithi090 ceph-mon[44434]: pgmap v955: 17 pgs: 17 active+clean; 3.4 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:18:17.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:17 smithi088 ceph-mon[44699]: pgmap v955: 17 pgs: 17 active+clean; 3.4 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:18:17.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:17 smithi084 ceph-mon[39784]: pgmap v955: 17 pgs: 17 active+clean; 3.4 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:18:17.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:18:17.430 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:17.462 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:18:17.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:17.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:18:17.569 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:17.617 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:18.323 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:18:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:18.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:18:18.429 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:18:18.458 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:18.480 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:18:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:18.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:18:18.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:18.646 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:18.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:18.649 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:18.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:18.783 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:18.813 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:18.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:18.819 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:18.920 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:18.987 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:19.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:19 smithi090 ceph-mon[44434]: pgmap v956: 17 pgs: 17 active+clean; 3.3 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:18:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:19 smithi088 ceph-mon[44699]: pgmap v956: 17 pgs: 17 active+clean; 3.3 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:18:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:19 smithi084 ceph-mon[39784]: pgmap v956: 17 pgs: 17 active+clean; 3.3 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:18:19.654 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:18:19.654 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:18:20.064 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:18:20.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:20 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3827043635' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:18:20.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:20 smithi090 ceph-mon[44434]: from='client.35762 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:18:20.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:20 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3827043635' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:18:20.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:20 smithi088 ceph-mon[44699]: from='client.35762 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:18:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:20 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3827043635' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:18:20.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:20 smithi084 ceph-mon[39784]: from='client.35762 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:18:20.444 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:18:20.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:20.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:18:20.549 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:20.599 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:20.712 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:18:20.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:20.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:18:20.816 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:20.867 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:21.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:21 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:18:21.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:21 smithi084 ceph-mon[39784]: from='client.35762 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:18:21.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:21 smithi084 ceph-mon[39784]: osdmap e347: 12 total, 12 up, 11 in 2024-05-15T23:18:21.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:21 smithi084 ceph-mon[39784]: pgmap v958: 17 pgs: 17 active+clean; 3.1 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:18:21.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:21 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:18:21.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:21 smithi090 ceph-mon[44434]: from='client.35762 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:18:21.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:21 smithi090 ceph-mon[44434]: osdmap e347: 12 total, 12 up, 11 in 2024-05-15T23:18:21.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:21 smithi090 ceph-mon[44434]: pgmap v958: 17 pgs: 17 active+clean; 3.1 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:18:21.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:21 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:18:21.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:21 smithi088 ceph-mon[44699]: from='client.35762 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:18:21.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:21 smithi088 ceph-mon[44699]: osdmap e347: 12 total, 12 up, 11 in 2024-05-15T23:18:21.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:21 smithi088 ceph-mon[44699]: pgmap v958: 17 pgs: 17 active+clean; 3.1 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:18:21.431 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:21.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:21.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:21.540 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:18:21.572 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:21.604 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:18:21.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:21.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:18:21.709 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:21.746 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:21.775 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:21.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:21.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:21.879 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:21.919 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:21.947 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:21.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:21.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:22.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:22.084 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:22.110 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:22.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:22.116 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:22.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T23:18:22.217 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:22.279 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:18:22.279 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:18:22.280 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:22.320 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:22.528 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:22.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:22.641 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:22.676 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:22.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:22.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:22.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:18:22.844 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:22.871 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:18:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:22.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:18:22.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:18:23.008 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:23.028 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:18:23.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:23.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:18:23.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:18:23.159 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:23.186 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:18:23.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:23.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:18:23.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:18:23.289 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:18:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:23.354 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:18:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:23.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:18:23.458 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:23.517 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:23.951 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":958,"stamp":"2024-05-15T23:18:20.936875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3280405048,"num_objects":50049,"num_object_clones":0,"num_object_copies":100098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50049,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":296548,"num_write_kb":11089191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160919,"ondisk_log_size":160919,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":9317104,"kb_used_data":6436360,"kb_used_omap":17,"kb_used_meta":2880622,"kb_avail":1021879568,"statfs":{"total":1055945392128,"available":1046404677632,"internally_reserved":0,"allocated":6590832640,"data_stored":6586988304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":2949757813},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-803667968,"num_objects":-12263,"num_object_clones":0,"num_object_copies":-24526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12263,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.008827"},"pg_stats":[{"pgid":"4.f","version":"347'18384","reported_seq":18691,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-05-15T23:18:20.652133+0000","last_change":"2024-05-15T23:17:55.492473+0000","last_active":"2024-05-15T23:18:20.652133+0000","last_peered":"2024-05-15T23:18:20.652133+0000","last_clean":"2024-05-15T23:18:20.652133+0000","last_became_active":"2024-05-15T23:14:58.959283+0000","last_became_peered":"2024-05-15T23:14:58.959283+0000","last_unstale":"2024-05-15T23:18:20.652133+0000","last_undegraded":"2024-05-15T23:18:20.652133+0000","last_fullsized":"2024-05-15T23:18:20.652133+0000","mapping_epoch":283,"log_start":"306'8300","ondisk_log_start":"306'8300","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'16517","last_scrub_stamp":"2024-05-15T23:17:55.492304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:55.492304+0000","objects_scrubbed":4953,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:02.351484+0000","scrub_duration":1175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199884800,"num_objects":3050,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18384,"num_write_kb":685888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"346'18843","reported_seq":19148,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:19.739350+0000","last_change":"2024-05-15T23:17:53.836854+0000","last_active":"2024-05-15T23:18:19.739350+0000","last_peered":"2024-05-15T23:18:19.739350+0000","last_clean":"2024-05-15T23:18:19.739350+0000","last_became_active":"2024-05-15T23:14:58.955357+0000","last_became_peered":"2024-05-15T23:14:58.955357+0000","last_unstale":"2024-05-15T23:18:19.739350+0000","last_undegraded":"2024-05-15T23:18:19.739350+0000","last_fullsized":"2024-05-15T23:18:19.739350+0000","mapping_epoch":283,"log_start":"308'8800","ondisk_log_start":"308'8800","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'16950","last_scrub_stamp":"2024-05-15T23:17:53.836683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:53.836683+0000","objects_scrubbed":5027,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:18:54.138019+0000","scrub_duration":813,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203751424,"num_objects":3109,"num_object_clones":0,"num_object_copies":6218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18843,"num_write_kb":702464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"346'18376","reported_seq":18674,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:19.644958+0000","last_change":"2024-05-15T23:17:58.683128+0000","last_active":"2024-05-15T23:18:19.644958+0000","last_peered":"2024-05-15T23:18:19.644958+0000","last_clean":"2024-05-15T23:18:19.644958+0000","last_became_active":"2024-05-15T23:14:58.955461+0000","last_became_peered":"2024-05-15T23:14:58.955461+0000","last_unstale":"2024-05-15T23:18:19.644958+0000","last_undegraded":"2024-05-15T23:18:19.644958+0000","last_fullsized":"2024-05-15T23:18:19.644958+0000","mapping_epoch":283,"log_start":"306'8300","ondisk_log_start":"306'8300","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'16817","last_scrub_stamp":"2024-05-15T23:17:58.682993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:58.682993+0000","objects_scrubbed":4600,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:15.051272+0000","scrub_duration":666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197984280,"num_objects":3022,"num_object_clones":0,"num_object_copies":6044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3022,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18376,"num_write_kb":684673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"346'18527","reported_seq":18829,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:19.728422+0000","last_change":"2024-05-15T23:17:58.124141+0000","last_active":"2024-05-15T23:18:19.728422+0000","last_peered":"2024-05-15T23:18:19.728422+0000","last_clean":"2024-05-15T23:18:19.728422+0000","last_became_active":"2024-05-15T23:14:58.956216+0000","last_became_peered":"2024-05-15T23:14:58.956216+0000","last_unstale":"2024-05-15T23:18:19.728422+0000","last_undegraded":"2024-05-15T23:18:19.728422+0000","last_fullsized":"2024-05-15T23:18:19.728422+0000","mapping_epoch":283,"log_start":"307'8500","ondisk_log_start":"307'8500","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'16873","last_scrub_stamp":"2024-05-15T23:17:58.124011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:58.124011+0000","objects_scrubbed":4825,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:21.394554+0000","scrub_duration":1073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204931072,"num_objects":3127,"num_object_clones":0,"num_object_copies":6254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18527,"num_write_kb":692928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"346'18627","reported_seq":18931,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:19.451023+0000","last_change":"2024-05-15T23:17:59.165263+0000","last_active":"2024-05-15T23:18:19.451023+0000","last_peered":"2024-05-15T23:18:19.451023+0000","last_clean":"2024-05-15T23:18:19.451023+0000","last_became_active":"2024-05-15T23:14:58.959506+0000","last_became_peered":"2024-05-15T23:14:58.959506+0000","last_unstale":"2024-05-15T23:18:19.451023+0000","last_undegraded":"2024-05-15T23:18:19.451023+0000","last_fullsized":"2024-05-15T23:18:19.451023+0000","mapping_epoch":283,"log_start":"307'8600","ondisk_log_start":"307'8600","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'17056","last_scrub_stamp":"2024-05-15T23:17:59.165113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:59.165113+0000","objects_scrubbed":4674,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:16.157458+0000","scrub_duration":1416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198377472,"num_objects":3027,"num_object_clones":0,"num_object_copies":6054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18627,"num_write_kb":692928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"346'18511","reported_seq":18816,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:17.682803+0000","last_change":"2024-05-15T23:17:55.102072+0000","last_active":"2024-05-15T23:18:17.682803+0000","last_peered":"2024-05-15T23:18:17.682803+0000","last_clean":"2024-05-15T23:18:17.682803+0000","last_became_active":"2024-05-15T23:14:58.959781+0000","last_became_peered":"2024-05-15T23:14:58.959781+0000","last_unstale":"2024-05-15T23:18:17.682803+0000","last_undegraded":"2024-05-15T23:18:17.682803+0000","last_fullsized":"2024-05-15T23:18:17.682803+0000","mapping_epoch":283,"log_start":"306'8500","ondisk_log_start":"306'8500","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'16853","last_scrub_stamp":"2024-05-15T23:17:55.101904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:55.101904+0000","objects_scrubbed":4963,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:19.324778+0000","scrub_duration":861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214761472,"num_objects":3277,"num_object_clones":0,"num_object_copies":6554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18511,"num_write_kb":697216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"346'18400","reported_seq":18701,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:17.775628+0000","last_change":"2024-05-15T23:18:01.854891+0000","last_active":"2024-05-15T23:18:17.775628+0000","last_peered":"2024-05-15T23:18:17.775628+0000","last_clean":"2024-05-15T23:18:17.775628+0000","last_became_active":"2024-05-15T23:14:58.958728+0000","last_became_peered":"2024-05-15T23:14:58.958728+0000","last_unstale":"2024-05-15T23:18:17.775628+0000","last_undegraded":"2024-05-15T23:18:17.775628+0000","last_fullsized":"2024-05-15T23:18:17.775628+0000","mapping_epoch":283,"log_start":"306'8300","ondisk_log_start":"306'8300","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'17148","last_scrub_stamp":"2024-05-15T23:18:01.854660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:18:01.854660+0000","objects_scrubbed":4492,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:23.649452+0000","scrub_duration":1098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210108416,"num_objects":3206,"num_object_clones":0,"num_object_copies":6412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18400,"num_write_kb":691392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"346'18366","reported_seq":18671,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:17.768288+0000","last_change":"2024-05-15T23:17:57.561478+0000","last_active":"2024-05-15T23:18:17.768288+0000","last_peered":"2024-05-15T23:18:17.768288+0000","last_clean":"2024-05-15T23:18:17.768288+0000","last_became_active":"2024-05-15T23:14:59.324101+0000","last_became_peered":"2024-05-15T23:14:59.324101+0000","last_unstale":"2024-05-15T23:18:17.768288+0000","last_undegraded":"2024-05-15T23:18:17.768288+0000","last_fullsized":"2024-05-15T23:18:17.768288+0000","mapping_epoch":283,"log_start":"306'8300","ondisk_log_start":"306'8300","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'16882","last_scrub_stamp":"2024-05-15T23:17:57.561373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:57.561373+0000","objects_scrubbed":4770,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:25.170594+0000","scrub_duration":733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214171648,"num_objects":3268,"num_object_clones":0,"num_object_copies":6536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18366,"num_write_kb":692288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"347'18722","reported_seq":19030,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-05-15T23:18:20.619639+0000","last_change":"2024-05-15T23:17:57.479856+0000","last_active":"2024-05-15T23:18:20.619639+0000","last_peered":"2024-05-15T23:18:20.619639+0000","last_clean":"2024-05-15T23:18:20.619639+0000","last_became_active":"2024-05-15T23:14:58.958730+0000","last_became_peered":"2024-05-15T23:14:58.958730+0000","last_unstale":"2024-05-15T23:18:20.619639+0000","last_undegraded":"2024-05-15T23:18:20.619639+0000","last_fullsized":"2024-05-15T23:18:20.619639+0000","mapping_epoch":283,"log_start":"308'8700","ondisk_log_start":"308'8700","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'17021","last_scrub_stamp":"2024-05-15T23:17:57.479570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:57.479570+0000","objects_scrubbed":4726,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:04.018226+0000","scrub_duration":1150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195297280,"num_objects":2980,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18722,"num_write_kb":694464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"347'18748","reported_seq":19060,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-05-15T23:18:20.444745+0000","last_change":"2024-05-15T23:17:50.662752+0000","last_active":"2024-05-15T23:18:20.444745+0000","last_peered":"2024-05-15T23:18:20.444745+0000","last_clean":"2024-05-15T23:18:20.444745+0000","last_became_active":"2024-05-15T23:14:58.958613+0000","last_became_peered":"2024-05-15T23:14:58.958613+0000","last_unstale":"2024-05-15T23:18:20.444745+0000","last_undegraded":"2024-05-15T23:18:20.444745+0000","last_fullsized":"2024-05-15T23:18:20.444745+0000","mapping_epoch":283,"log_start":"308'8700","ondisk_log_start":"308'8700","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'16511","last_scrub_stamp":"2024-05-15T23:17:50.662532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:50.662532+0000","objects_scrubbed":5403,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:18:57.454753+0000","scrub_duration":1208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204472320,"num_objects":3120,"num_object_clones":0,"num_object_copies":6240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18748,"num_write_kb":699776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"347'18368","reported_seq":18669,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-05-15T23:18:20.615150+0000","last_change":"2024-05-15T23:17:58.503743+0000","last_active":"2024-05-15T23:18:20.615150+0000","last_peered":"2024-05-15T23:18:20.615150+0000","last_clean":"2024-05-15T23:18:20.615150+0000","last_became_active":"2024-05-15T23:14:58.959588+0000","last_became_peered":"2024-05-15T23:14:58.959588+0000","last_unstale":"2024-05-15T23:18:20.615150+0000","last_undegraded":"2024-05-15T23:18:20.615150+0000","last_fullsized":"2024-05-15T23:18:20.615150+0000","mapping_epoch":283,"log_start":"306'8300","ondisk_log_start":"306'8300","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'16810","last_scrub_stamp":"2024-05-15T23:17:58.503579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:58.503579+0000","objects_scrubbed":4624,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:19.485194+0000","scrub_duration":1188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198836224,"num_objects":3034,"num_object_clones":0,"num_object_copies":6068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18368,"num_write_kb":684864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"346'18517","reported_seq":18825,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:17.697695+0000","last_change":"2024-05-15T23:17:56.650874+0000","last_active":"2024-05-15T23:18:17.697695+0000","last_peered":"2024-05-15T23:18:17.697695+0000","last_clean":"2024-05-15T23:18:17.697695+0000","last_became_active":"2024-05-15T23:14:59.324138+0000","last_became_peered":"2024-05-15T23:14:59.324138+0000","last_unstale":"2024-05-15T23:18:17.697695+0000","last_undegraded":"2024-05-15T23:18:17.697695+0000","last_fullsized":"2024-05-15T23:18:17.697695+0000","mapping_epoch":283,"log_start":"307'8500","ondisk_log_start":"307'8500","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'16967","last_scrub_stamp":"2024-05-15T23:17:56.650719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:56.650719+0000","objects_scrubbed":4848,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:19.007279+0000","scrub_duration":867,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213843968,"num_objects":3263,"num_object_clones":0,"num_object_copies":6526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18517,"num_write_kb":696960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"347'18407","reported_seq":18709,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-05-15T23:18:20.448160+0000","last_change":"2024-05-15T23:18:05.241047+0000","last_active":"2024-05-15T23:18:20.448160+0000","last_peered":"2024-05-15T23:18:20.448160+0000","last_clean":"2024-05-15T23:18:20.448160+0000","last_became_active":"2024-05-15T23:14:59.079701+0000","last_became_peered":"2024-05-15T23:14:59.079701+0000","last_unstale":"2024-05-15T23:18:20.448160+0000","last_undegraded":"2024-05-15T23:18:20.448160+0000","last_fullsized":"2024-05-15T23:18:20.448160+0000","mapping_epoch":283,"log_start":"307'8400","ondisk_log_start":"307'8400","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'17206","last_scrub_stamp":"2024-05-15T23:18:05.240792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:18:05.240792+0000","objects_scrubbed":4319,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:07.646094+0000","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202571776,"num_objects":3091,"num_object_clones":0,"num_object_copies":6182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18407,"num_write_kb":687936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"346'18451","reported_seq":18753,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:17.761829+0000","last_change":"2024-05-15T23:18:04.556346+0000","last_active":"2024-05-15T23:18:17.761829+0000","last_peered":"2024-05-15T23:18:17.761829+0000","last_clean":"2024-05-15T23:18:17.761829+0000","last_became_active":"2024-05-15T23:14:59.079345+0000","last_became_peered":"2024-05-15T23:14:59.079345+0000","last_unstale":"2024-05-15T23:18:17.761829+0000","last_undegraded":"2024-05-15T23:18:17.761829+0000","last_fullsized":"2024-05-15T23:18:17.761829+0000","mapping_epoch":283,"log_start":"306'8400","ondisk_log_start":"306'8400","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'17437","last_scrub_stamp":"2024-05-15T23:18:04.556086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:18:04.556086+0000","objects_scrubbed":4249,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:25.154900+0000","scrub_duration":740,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210567168,"num_objects":3213,"num_object_clones":0,"num_object_copies":6426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18451,"num_write_kb":693248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"347'18582","reported_seq":18889,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-05-15T23:18:20.695621+0000","last_change":"2024-05-15T23:17:54.035243+0000","last_active":"2024-05-15T23:18:20.695621+0000","last_peered":"2024-05-15T23:18:20.695621+0000","last_clean":"2024-05-15T23:18:20.695621+0000","last_became_active":"2024-05-15T23:14:59.324368+0000","last_became_peered":"2024-05-15T23:14:59.324368+0000","last_unstale":"2024-05-15T23:18:20.695621+0000","last_undegraded":"2024-05-15T23:18:20.695621+0000","last_fullsized":"2024-05-15T23:18:20.695621+0000","mapping_epoch":283,"log_start":"307'8500","ondisk_log_start":"307'8500","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'16574","last_scrub_stamp":"2024-05-15T23:17:54.035140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:54.035140+0000","objects_scrubbed":5053,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:00.556275+0000","scrub_duration":1240,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195952640,"num_objects":2990,"num_object_clones":0,"num_object_copies":5980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18582,"num_write_kb":690304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"346'18596","reported_seq":18900,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-05-15T23:18:17.665840+0000","last_change":"2024-05-15T23:17:54.077213+0000","last_active":"2024-05-15T23:18:17.665840+0000","last_peered":"2024-05-15T23:18:17.665840+0000","last_clean":"2024-05-15T23:18:17.665840+0000","last_became_active":"2024-05-15T23:14:58.958832+0000","last_became_peered":"2024-05-15T23:14:58.958832+0000","last_unstale":"2024-05-15T23:18:17.665840+0000","last_undegraded":"2024-05-15T23:18:17.665840+0000","last_fullsized":"2024-05-15T23:18:17.665840+0000","mapping_epoch":283,"log_start":"306'8500","ondisk_log_start":"306'8500","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'16858","last_scrub_stamp":"2024-05-15T23:17:54.076977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:14:57.936933+0000","last_clean_scrub_stamp":"2024-05-15T23:17:54.076977+0000","objects_scrubbed":5039,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:04.883487+0000","scrub_duration":806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214302720,"num_objects":3270,"num_object_clones":0,"num_object_copies":6540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18596,"num_write_kb":699712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":716,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-05-15T23:18:20.057459+0000","last_change":"2024-05-15T23:17:07.258646+0000","last_active":"2024-05-15T23:18:20.057459+0000","last_peered":"2024-05-15T23:18:20.057459+0000","last_clean":"2024-05-15T23:18:20.057459+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:18:20.057459+0000","last_undegraded":"2024-05-15T23:18:20.057459+0000","last_fullsized":"2024-05-15T23:18:20.057459+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:17:07.258511+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:17:07.258511+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:18:21.804913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":3279814680,"num_objects":50047,"num_object_clones":0,"num_object_copies":100094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50047,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":296425,"num_write_kb":11087041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6497312768,"data_stored":6497304624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351150,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1271540,"kb_used_data":848876,"kb_used_omap":1,"kb_used_meta":422654,"kb_avail":92473612,"statfs":{"total":95995035648,"available":94692978688,"internally_reserved":0,"allocated":869249024,"data_stored":868899006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":432798148},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580084,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":735468,"kb_used_data":591844,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93009684,"statfs":{"total":95995035648,"available":95241916416,"internally_reserved":0,"allocated":606048256,"data_stored":605701810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776316,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":913088,"kb_used_data":588980,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92832064,"statfs":{"total":95995035648,"available":95060033536,"internally_reserved":0,"allocated":603115520,"data_stored":602758948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":331872704},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005248,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":144304,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93600848,"statfs":{"total":95995035648,"available":95847268352,"internally_reserved":0,"allocated":692224,"data_stored":345778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266884,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":851380,"kb_used_data":595308,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":92893772,"statfs":{"total":95995035648,"available":95123222528,"internally_reserved":0,"allocated":609595392,"data_stored":609245394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":262207946},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495818,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":560364,"kb_used_data":425060,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93184788,"statfs":{"total":95995035648,"available":95421222912,"internally_reserved":0,"allocated":435261440,"data_stored":434914994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":138541510},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575076,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1702652,"kb_used_data":1368816,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92042500,"statfs":{"total":95995035648,"available":94251520000,"internally_reserved":0,"allocated":1401667584,"data_stored":1401315084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":341834182},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738602,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":476524,"kb_used_data":196836,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":93268628,"statfs":{"total":95995035648,"available":95507075072,"internally_reserved":0,"allocated":201560064,"data_stored":201213618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483174,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149914,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1144892,"kb_used_data":788848,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92600260,"statfs":{"total":95995035648,"available":94822666240,"internally_reserved":0,"allocated":807780352,"data_stored":807427852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":364575175},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986387,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895664,"kb_used_data":603496,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92849488,"statfs":{"total":95995035648,"available":95077875712,"internally_reserved":0,"allocated":617979904,"data_stored":617629386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790158,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":621228,"kb_used_data":427620,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93123924,"statfs":{"total":95995035648,"available":95358898176,"internally_reserved":0,"allocated":437882880,"data_stored":437536434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":198377472,"data_stored":198377472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1384906752,"data_stored":1384906752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":792854528,"data_stored":792854528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":606670848,"data_stored":606666776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":435552256,"data_stored":435552256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":428867584,"data_stored":428867584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":602144768,"data_stored":602144768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":602017792,"data_stored":602013720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":597229568,"data_stored":597229568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":848691200,"data_stored":848691200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:18:23.952 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-15T23:18:24.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:23 smithi090 ceph-mon[44434]: from='client.15735 v1:172.21.15.84:0/1901393127' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:18:24.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:23 smithi090 ceph-mon[44434]: pgmap v959: 17 pgs: 17 active+clean; 2.9 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:18:24.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:23 smithi084 ceph-mon[39784]: from='client.15735 v1:172.21.15.84:0/1901393127' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:18:24.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:23 smithi084 ceph-mon[39784]: pgmap v959: 17 pgs: 17 active+clean; 2.9 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:18:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:23 smithi088 ceph-mon[44699]: from='client.15735 v1:172.21.15.84:0/1901393127' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:18:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:23 smithi088 ceph-mon[44699]: pgmap v959: 17 pgs: 17 active+clean; 2.9 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:18:24.394 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:24.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:24.403 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:24.499 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:18:24.526 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:18:24.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:24.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:18:24.662 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:24.694 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:24.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:24.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:24.824 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:18:24.868 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:24.896 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:18:24.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:24.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:18:25.001 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:18:25.027 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:25.056 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:18:25.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:25.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:18:25.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:25.194 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:25.231 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:25.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:24 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:18:25.236 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:24 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:18:25.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:25.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:25.282 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:24 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:18:25.282 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:24 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:18:25.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:24 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:18:25.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:24 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:18:25.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:18:25.372 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:25.381 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:18:25.387 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:18:25.398 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:18:25.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:25.406 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:18:25.414 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:18:25.446 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:18:25.479 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:18:25.507 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:25.522 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:18:25.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:18:25.564 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:25.594 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:18:25.617 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:18:25.646 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:18:25.677 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:18:25.704 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:18:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:25.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:18:25.708 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:18:25.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:18:25.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:18:25.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:18:25.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:18:25.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:18:25.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:18:25.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:18:25.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:18:25.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:18:25.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:18:25.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:18:25.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:18:25.808 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:25.839 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:25.859 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:25.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:25.963 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:18:25.989 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:26.008 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:18:26.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:26.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:18:26.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:25 smithi084 ceph-mon[39784]: pgmap v960: 17 pgs: 17 active+clean; 2.8 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:18:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:25 smithi084 ceph-mon[39784]: osdmap e348: 12 total, 12 up, 11 in 2024-05-15T23:18:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:25 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3355074282' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.155 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:26.197 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:18:26.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:26.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:18:26.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:25 smithi090 ceph-mon[44434]: pgmap v960: 17 pgs: 17 active+clean; 2.8 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:18:26.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:25 smithi090 ceph-mon[44434]: osdmap e348: 12 total, 12 up, 11 in 2024-05-15T23:18:26.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:25 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3355074282' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:26.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.305 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:25 smithi088 ceph-mon[44699]: pgmap v960: 17 pgs: 17 active+clean; 2.8 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:18:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:25 smithi088 ceph-mon[44699]: osdmap e348: 12 total, 12 up, 11 in 2024-05-15T23:18:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:25 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3355074282' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:26.342 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:26.376 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:26.378 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:26.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:26.537 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:26.998 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:27.001 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:27.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:27.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:27.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:18:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:27.184 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:18:27.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:27.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:18:27.289 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:18:27.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:27.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:27.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:27.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:27.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:27.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:27.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:27.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:27.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:27.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:27.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:27.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:27.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:27.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:27.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:27.337 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:27.592 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:18:27.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:27.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:18:27.697 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:18:27.697 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:18:27.697 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-05-15T23:18:27.697 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:18:27.697 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:27.753 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:27.967 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:18:27.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:27.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:18:28.073 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:18:28.109 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:28.133 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:18:28.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:28.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:18:28.241 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:18:28.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:27 smithi084 ceph-mon[39784]: pgmap v962: 17 pgs: 17 active+clean; 2.7 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:18:28.281 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:28.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:27 smithi090 ceph-mon[44434]: pgmap v962: 17 pgs: 17 active+clean; 2.7 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:18:28.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:27 smithi088 ceph-mon[44699]: pgmap v962: 17 pgs: 17 active+clean; 2.7 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:18:28.416 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:18:28.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:28.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:18:28.523 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:28.569 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:29.050 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:18:29.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:29 smithi090 ceph-mon[44434]: pgmap v963: 17 pgs: 17 active+clean; 2.5 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:18:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:29 smithi088 ceph-mon[44699]: pgmap v963: 17 pgs: 17 active+clean; 2.5 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:18:29.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:29 smithi084 ceph-mon[39784]: pgmap v963: 17 pgs: 17 active+clean; 2.5 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:18:29.425 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:29.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:29.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:29.447 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T23:18:29.535 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:18:29.587 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:29.662 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:18:29.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:29.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:18:29.768 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:29.829 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:30.005 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:30 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:18:30.005 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:30 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:18:30.062 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:18:30.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:30.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:18:30.115 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:30.159 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:30.175 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:18:30.232 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:30.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:30 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:18:30.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:30 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:18:30.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:30 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:18:30.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:30 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:18:31.038 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:18:31.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:31 smithi090 ceph-mon[44434]: osdmap e349: 12 total, 12 up, 11 in 2024-05-15T23:18:31.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2820632826' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:18:31.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:31 smithi090 ceph-mon[44434]: from='client.35774 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:18:31.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:31 smithi090 ceph-mon[44434]: pgmap v965: 17 pgs: 17 active+clean; 2.3 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:18:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:31 smithi088 ceph-mon[44699]: osdmap e349: 12 total, 12 up, 11 in 2024-05-15T23:18:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2820632826' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:18:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:31 smithi088 ceph-mon[44699]: from='client.35774 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:18:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:31 smithi088 ceph-mon[44699]: pgmap v965: 17 pgs: 17 active+clean; 2.3 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:18:31.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:31 smithi084 ceph-mon[39784]: osdmap e349: 12 total, 12 up, 11 in 2024-05-15T23:18:31.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2820632826' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:18:31.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:31 smithi084 ceph-mon[39784]: from='client.35774 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:18:31.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:31 smithi084 ceph-mon[39784]: pgmap v965: 17 pgs: 17 active+clean; 2.3 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:18:32.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:32 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:18:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:32 smithi090 ceph-mon[44434]: from='client.35774 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:18:32.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:32 smithi090 ceph-mon[44434]: osdmap e350: 12 total, 12 up, 11 in 2024-05-15T23:18:32.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:32 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:18:32.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:32 smithi088 ceph-mon[44699]: from='client.35774 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:18:32.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:32 smithi088 ceph-mon[44699]: osdmap e350: 12 total, 12 up, 11 in 2024-05-15T23:18:32.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:32 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:18:32.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:32 smithi084 ceph-mon[39784]: from='client.35774 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:18:32.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:32 smithi084 ceph-mon[39784]: osdmap e350: 12 total, 12 up, 11 in 2024-05-15T23:18:32.415 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:18:32.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:32.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:18:32.522 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:32.565 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:32.824 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:32.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:32.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:32.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:32.969 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:32.995 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:32.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:32.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:33.100 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:18:33.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:33 smithi084 ceph-mon[39784]: pgmap v967: 17 pgs: 17 active+clean; 2.2 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.31k op/s 2024-05-15T23:18:33.133 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:33.153 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:18:33.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:33.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:18:33.258 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:18:33.291 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:33 smithi088 ceph-mon[44699]: pgmap v967: 17 pgs: 17 active+clean; 2.2 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.31k op/s 2024-05-15T23:18:33.292 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:33.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:33 smithi090 ceph-mon[44434]: pgmap v967: 17 pgs: 17 active+clean; 2.2 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.31k op/s 2024-05-15T23:18:33.321 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:18:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:33.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:18:33.428 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:18:33.473 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:33.931 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:18:33.931 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:18:33.931 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:18:33.931 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:18:33.932 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:18:33.932 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:18:33.932 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:18:33.932 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:18:33.932 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:18:33.932 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:18:33.932 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:18:33.932 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:18:33.932 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:18:33.932 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:18:33.932 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:18:33.932 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:18:33.932 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:18:33.933 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:18:33.933 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:18:33.933 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:18:33.933 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:18:33.933 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:18:33.933 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:18:33.933 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:18:33.944 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:18:33.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:33.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:18:34.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:34.086 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:34.114 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:34.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:34.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:34.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:34 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3514444352' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:34.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:34.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:34.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:34.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:18:34.261 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:34.288 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:18:34.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:34.294 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:18:34.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:34 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3514444352' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:34.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:34.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:34.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:34.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:34 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3514444352' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:34.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:34.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:34.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:34.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:18:34.435 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:34.469 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:18:34.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:34.478 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:18:34.579 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:18:34.612 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:34.636 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:18:34.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:34.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:18:34.740 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:34.776 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:34.800 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:34.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:34.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:34.871 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:18:34.903 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:34.976 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:35.027 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:18:35.027 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:18:35.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:35 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:18:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:35 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:18:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:35 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:18:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:35 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:18:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:35 smithi090 ceph-mon[44434]: pgmap v968: 17 pgs: 17 active+clean; 2.1 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:18:35.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:35 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:18:35.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:35 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:18:35.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:35 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:18:35.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:35 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:18:35.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:35 smithi088 ceph-mon[44699]: pgmap v968: 17 pgs: 17 active+clean; 2.1 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:18:35.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:35 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:18:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:35 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:18:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:35.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:35 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:18:35.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:35 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:18:35.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:35 smithi084 ceph-mon[39784]: pgmap v968: 17 pgs: 17 active+clean; 2.1 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:18:36.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:36 smithi090 ceph-mon[44434]: osdmap e351: 12 total, 12 up, 11 in 2024-05-15T23:18:36.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:36 smithi088 ceph-mon[44699]: osdmap e351: 12 total, 12 up, 11 in 2024-05-15T23:18:36.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:36 smithi084 ceph-mon[39784]: osdmap e351: 12 total, 12 up, 11 in 2024-05-15T23:18:36.973 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:36.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:36.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:37.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:37.114 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:37.140 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:37.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:37.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:37.246 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:37.279 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:38.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:37 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:18:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:37 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:18:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:37 smithi090 ceph-mon[44434]: pgmap v970: 17 pgs: 17 active+clean; 1.9 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:18:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:37 smithi090 ceph-mon[44434]: osdmap e352: 12 total, 12 up, 11 in 2024-05-15T23:18:38.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:37 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:18:38.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:37 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:18:38.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:37 smithi088 ceph-mon[44699]: pgmap v970: 17 pgs: 17 active+clean; 1.9 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:18:38.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:37 smithi088 ceph-mon[44699]: osdmap e352: 12 total, 12 up, 11 in 2024-05-15T23:18:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:37 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:18:38.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:37 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:18:38.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:37 smithi084 ceph-mon[39784]: pgmap v970: 17 pgs: 17 active+clean; 1.9 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:18:38.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:37 smithi084 ceph-mon[39784]: osdmap e352: 12 total, 12 up, 11 in 2024-05-15T23:18:38.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:18:40.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:39 smithi090 ceph-mon[44434]: pgmap v972: 17 pgs: 17 active+clean; 1.8 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:18:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:39 smithi088 ceph-mon[44699]: pgmap v972: 17 pgs: 17 active+clean; 1.8 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:18:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:39 smithi084 ceph-mon[39784]: pgmap v972: 17 pgs: 17 active+clean; 1.8 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:18:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:18:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:18:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: osdmap e353: 12 total, 12 up, 11 in 2024-05-15T23:18:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=53,logm=2} crc {auth=1884927487,config=3313370793,health=1997216375,kv=1550167019,logm=1943647153}) 2024-05-15T23:18:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3027780643}) 2024-05-15T23:18:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3762635035}) 2024-05-15T23:18:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=672900680}) 2024-05-15T23:18:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3906884001}) 2024-05-15T23:18:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2329766205}) 2024-05-15T23:18:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1381152336}) 2024-05-15T23:18:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3600092471}) 2024-05-15T23:18:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1174693991}) 2024-05-15T23:18:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2536281528}) 2024-05-15T23:18:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=215922433}) 2024-05-15T23:18:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=696617330}) 2024-05-15T23:18:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562507268}) 2024-05-15T23:18:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1712230194}) 2024-05-15T23:18:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=711005070}) 2024-05-15T23:18:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2085043503}) 2024-05-15T23:18:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2109388669}) 2024-05-15T23:18:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1495540415}) 2024-05-15T23:18:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4210325749}) 2024-05-15T23:18:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=15333161}) 2024-05-15T23:18:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4176161930}) 2024-05-15T23:18:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3577741817}) 2024-05-15T23:18:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2529855490}) 2024-05-15T23:18:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228240039}) 2024-05-15T23:18:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1239524408}) 2024-05-15T23:18:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3438005918}) 2024-05-15T23:18:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2460665530}) 2024-05-15T23:18:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2784804718}) 2024-05-15T23:18:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1362181512}) 2024-05-15T23:18:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131853439}) 2024-05-15T23:18:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1018723740}) 2024-05-15T23:18:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1814337928}) 2024-05-15T23:18:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3859234329}) 2024-05-15T23:18:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2256294131}) 2024-05-15T23:18:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1809192130}) 2024-05-15T23:18:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3014887397}) 2024-05-15T23:18:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=968833803}) 2024-05-15T23:18:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2946241368}) 2024-05-15T23:18:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1478089891}) 2024-05-15T23:18:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2920873621}) 2024-05-15T23:18:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1532372314}) 2024-05-15T23:18:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3301866225}) 2024-05-15T23:18:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=374664787}) 2024-05-15T23:18:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2930140739}) 2024-05-15T23:18:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1679727910}) 2024-05-15T23:18:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373629717}) 2024-05-15T23:18:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=951397870}) 2024-05-15T23:18:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=27,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=32} crc {logm=2803561838,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2501968666,mon_config_key=2206670375}) 2024-05-15T23:18:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4080946797}) 2024-05-15T23:18:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=582975532}) 2024-05-15T23:18:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=52,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=30} crc {mon_config_key=2798165237,monmap=1478146745,osd_metadata=1212471037,osd_pg_creating=1465284689,osdmap=3186797609}) 2024-05-15T23:18:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=97} crc {osdmap=228210922}) 2024-05-15T23:18:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:41 smithi090 ceph-mon[44434]: pgmap v974: 17 pgs: 17 active+clean; 1.6 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.42k op/s 2024-05-15T23:18:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:18:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:18:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: osdmap e353: 12 total, 12 up, 11 in 2024-05-15T23:18:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=53,logm=2} crc {auth=1884927487,config=3313370793,health=1997216375,kv=1550167019,logm=1943647153}) 2024-05-15T23:18:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3027780643}) 2024-05-15T23:18:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3762635035}) 2024-05-15T23:18:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=672900680}) 2024-05-15T23:18:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3906884001}) 2024-05-15T23:18:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2329766205}) 2024-05-15T23:18:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1381152336}) 2024-05-15T23:18:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3600092471}) 2024-05-15T23:18:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1174693991}) 2024-05-15T23:18:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2536281528}) 2024-05-15T23:18:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=215922433}) 2024-05-15T23:18:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=696617330}) 2024-05-15T23:18:41.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562507268}) 2024-05-15T23:18:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1712230194}) 2024-05-15T23:18:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=711005070}) 2024-05-15T23:18:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2085043503}) 2024-05-15T23:18:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2109388669}) 2024-05-15T23:18:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1495540415}) 2024-05-15T23:18:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4210325749}) 2024-05-15T23:18:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=15333161}) 2024-05-15T23:18:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4176161930}) 2024-05-15T23:18:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3577741817}) 2024-05-15T23:18:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2529855490}) 2024-05-15T23:18:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228240039}) 2024-05-15T23:18:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1239524408}) 2024-05-15T23:18:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3438005918}) 2024-05-15T23:18:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2460665530}) 2024-05-15T23:18:41.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2784804718}) 2024-05-15T23:18:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1362181512}) 2024-05-15T23:18:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131853439}) 2024-05-15T23:18:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1018723740}) 2024-05-15T23:18:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1814337928}) 2024-05-15T23:18:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3859234329}) 2024-05-15T23:18:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2256294131}) 2024-05-15T23:18:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1809192130}) 2024-05-15T23:18:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3014887397}) 2024-05-15T23:18:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=968833803}) 2024-05-15T23:18:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2946241368}) 2024-05-15T23:18:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1478089891}) 2024-05-15T23:18:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2920873621}) 2024-05-15T23:18:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1532372314}) 2024-05-15T23:18:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3301866225}) 2024-05-15T23:18:41.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=374664787}) 2024-05-15T23:18:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2930140739}) 2024-05-15T23:18:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1679727910}) 2024-05-15T23:18:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373629717}) 2024-05-15T23:18:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=951397870}) 2024-05-15T23:18:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=27,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=32} crc {logm=2803561838,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2501968666,mon_config_key=2206670375}) 2024-05-15T23:18:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4080946797}) 2024-05-15T23:18:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=582975532}) 2024-05-15T23:18:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=52,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=30} crc {mon_config_key=2798165237,monmap=1478146745,osd_metadata=1212471037,osd_pg_creating=1465284689,osdmap=3186797609}) 2024-05-15T23:18:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=97} crc {osdmap=228210922}) 2024-05-15T23:18:41.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:41 smithi088 ceph-mon[44699]: pgmap v974: 17 pgs: 17 active+clean; 1.6 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.42k op/s 2024-05-15T23:18:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:18:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:18:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: osdmap e353: 12 total, 12 up, 11 in 2024-05-15T23:18:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=53,logm=2} crc {auth=1884927487,config=3313370793,health=1997216375,kv=1550167019,logm=1943647153}) 2024-05-15T23:18:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3027780643}) 2024-05-15T23:18:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3762635035}) 2024-05-15T23:18:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=672900680}) 2024-05-15T23:18:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3906884001}) 2024-05-15T23:18:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2329766205}) 2024-05-15T23:18:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1381152336}) 2024-05-15T23:18:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3600092471}) 2024-05-15T23:18:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1174693991}) 2024-05-15T23:18:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2536281528}) 2024-05-15T23:18:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=215922433}) 2024-05-15T23:18:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=696617330}) 2024-05-15T23:18:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562507268}) 2024-05-15T23:18:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1712230194}) 2024-05-15T23:18:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=711005070}) 2024-05-15T23:18:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2085043503}) 2024-05-15T23:18:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2109388669}) 2024-05-15T23:18:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1495540415}) 2024-05-15T23:18:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4210325749}) 2024-05-15T23:18:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=15333161}) 2024-05-15T23:18:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4176161930}) 2024-05-15T23:18:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3577741817}) 2024-05-15T23:18:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2529855490}) 2024-05-15T23:18:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228240039}) 2024-05-15T23:18:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1239524408}) 2024-05-15T23:18:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3438005918}) 2024-05-15T23:18:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2460665530}) 2024-05-15T23:18:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2784804718}) 2024-05-15T23:18:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1362181512}) 2024-05-15T23:18:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131853439}) 2024-05-15T23:18:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1018723740}) 2024-05-15T23:18:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1814337928}) 2024-05-15T23:18:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3859234329}) 2024-05-15T23:18:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2256294131}) 2024-05-15T23:18:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1809192130}) 2024-05-15T23:18:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3014887397}) 2024-05-15T23:18:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=968833803}) 2024-05-15T23:18:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2946241368}) 2024-05-15T23:18:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1478089891}) 2024-05-15T23:18:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2920873621}) 2024-05-15T23:18:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1532372314}) 2024-05-15T23:18:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3301866225}) 2024-05-15T23:18:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=374664787}) 2024-05-15T23:18:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2930140739}) 2024-05-15T23:18:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1679727910}) 2024-05-15T23:18:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373629717}) 2024-05-15T23:18:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=951397870}) 2024-05-15T23:18:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=27,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=32} crc {logm=2803561838,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2501968666,mon_config_key=2206670375}) 2024-05-15T23:18:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4080946797}) 2024-05-15T23:18:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=582975532}) 2024-05-15T23:18:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=52,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=30} crc {mon_config_key=2798165237,monmap=1478146745,osd_metadata=1212471037,osd_pg_creating=1465284689,osdmap=3186797609}) 2024-05-15T23:18:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=97} crc {osdmap=228210922}) 2024-05-15T23:18:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:41 smithi084 ceph-mon[39784]: pgmap v974: 17 pgs: 17 active+clean; 1.6 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.42k op/s 2024-05-15T23:18:42.395 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T23:18:42.410 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:42.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:42.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:42.432 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:42.439 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:42.517 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:18:42.548 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:42.571 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:18:42.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:42.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:18:42.675 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:42.734 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:42.808 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:18:42.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:42.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:18:42.913 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:42.964 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:43.251 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:43.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:43.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:43.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:43 smithi088 ceph-mon[44699]: pgmap v975: 17 pgs: 17 active+clean; 1.4 GiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.38k op/s 2024-05-15T23:18:43.357 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:43.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:43 smithi084 ceph-mon[39784]: pgmap v975: 17 pgs: 17 active+clean; 1.4 GiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.38k op/s 2024-05-15T23:18:43.396 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:43.419 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:43.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:43.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:43.527 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:18:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:43 smithi090 ceph-mon[44434]: pgmap v975: 17 pgs: 17 active+clean; 1.4 GiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.38k op/s 2024-05-15T23:18:43.572 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:43.603 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:18:43.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:43.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:18:43.708 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:18:43.743 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:43.770 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:18:43.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:43.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:18:43.874 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:18:43.919 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:43.948 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:18:43.953 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-05-15T23:18:43.953 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-05-15T23:18:43.953 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-15T23:18:43.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.0 2024-05-15T23:18:43.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:43.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:18:44.057 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:44.097 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:44.120 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:44.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:44.226 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:44.281 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:18:43 smithi084 systemd[1]: Stopping Ceph osd.0 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:18:44.285 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:45.461 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:45.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:45.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:45.615 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:45.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:45 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:18:45.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:45 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:18:45.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:45 smithi084 ceph-mon[39784]: osdmap e354: 12 total, 12 up, 11 in 2024-05-15T23:18:45.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:45 smithi084 ceph-mon[39784]: pgmap v977: 17 pgs: 17 active+clean; 1.3 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-05-15T23:18:45.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:45 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:18:45.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:45 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:18:45.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:18:45.988 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:18:46.019 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:18:46.048 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:18:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:45 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:18:46.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:45 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:18:46.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:45 smithi090 ceph-mon[44434]: osdmap e354: 12 total, 12 up, 11 in 2024-05-15T23:18:46.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:45 smithi090 ceph-mon[44434]: pgmap v977: 17 pgs: 17 active+clean; 1.3 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-05-15T23:18:46.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:45 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:18:46.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:45 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:18:46.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:45 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:18:46.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:45 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:18:46.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:45 smithi088 ceph-mon[44699]: osdmap e354: 12 total, 12 up, 11 in 2024-05-15T23:18:46.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:45 smithi088 ceph-mon[44699]: pgmap v977: 17 pgs: 17 active+clean; 1.3 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-05-15T23:18:46.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:45 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:18:46.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:45 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:18:46.081 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:18:46.113 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:18:46.141 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:18:46.184 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:18:46.192 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:18:46.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:46.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:18:46.230 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:18:46.273 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:18:46.303 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:18:46.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:46.330 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:18:46.334 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:18:46.334 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:18:46.334 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:18:46.334 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:18:46.334 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:18:46.334 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:18:46.334 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:18:46.334 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:18:46.334 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:18:46.334 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:18:46.335 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:18:46.335 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:18:46.351 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:18:46 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-0[57754]: 2024-05-15T23:18:46.002+0000 7f19c7a37700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:18:46.351 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:18:46 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-0[57754]: 2024-05-15T23:18:46.002+0000 7f19c7a37700 -1 osd.0 355 *** Got signal Terminated *** 2024-05-15T23:18:46.351 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:18:46 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-0[57754]: 2024-05-15T23:18:46.002+0000 7f19c7a37700 -1 osd.0 355 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:18:46.355 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:46.665 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:46.828 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:46.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:46.948 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:46.972 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:46 smithi088 ceph-mon[44699]: osdmap e355: 12 total, 12 up, 11 in 2024-05-15T23:18:46.972 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:46 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2309602171' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:46.972 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:46.972 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:46 smithi088 ceph-mon[44699]: osd.0 marked itself down and dead 2024-05-15T23:18:46.972 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:46.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:46.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:46.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:46.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:46.974 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:46.993 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:46.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:47.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:46 smithi090 ceph-mon[44434]: osdmap e355: 12 total, 12 up, 11 in 2024-05-15T23:18:47.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:46 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2309602171' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:46 smithi090 ceph-mon[44434]: osd.0 marked itself down and dead 2024-05-15T23:18:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.097 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:47.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:46 smithi084 ceph-mon[39784]: osdmap e355: 12 total, 12 up, 11 in 2024-05-15T23:18:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:46 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2309602171' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:46 smithi084 ceph-mon[39784]: osd.0 marked itself down and dead 2024-05-15T23:18:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.129 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:47.725 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:18:47.870 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:18:47 smithi084 bash[174076]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-0 2024-05-15T23:18:47.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:47.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:47 smithi084 ceph-mon[39784]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-15T23:18:47.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:47 smithi084 ceph-mon[39784]: osdmap e356: 12 total, 11 up, 11 in 2024-05-15T23:18:47.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:47 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3155310646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:18:47.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:47 smithi084 ceph-mon[39784]: from='client.35789 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:18:47.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:47 smithi084 ceph-mon[39784]: pgmap v980: 17 pgs: 1 stale+active+clean, 16 active+clean; 1.2 GiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-05-15T23:18:47.982 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:47.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:47.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:47 smithi090 ceph-mon[44434]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-15T23:18:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:47 smithi090 ceph-mon[44434]: osdmap e356: 12 total, 11 up, 11 in 2024-05-15T23:18:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:47 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3155310646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:18:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:47 smithi090 ceph-mon[44434]: from='client.35789 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:18:48.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:47 smithi090 ceph-mon[44434]: pgmap v980: 17 pgs: 1 stale+active+clean, 16 active+clean; 1.2 GiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-05-15T23:18:48.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:48.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:47 smithi088 ceph-mon[44699]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-15T23:18:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:47 smithi088 ceph-mon[44699]: osdmap e356: 12 total, 11 up, 11 in 2024-05-15T23:18:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:47 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3155310646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:18:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:47 smithi088 ceph-mon[44699]: from='client.35789 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:18:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:47 smithi088 ceph-mon[44699]: pgmap v980: 17 pgs: 1 stale+active+clean, 16 active+clean; 1.2 GiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-05-15T23:18:48.087 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:18:48.117 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:48.136 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:18:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:48.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:18:48.240 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:48.272 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:48.397 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:48.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:48.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:48.501 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:48.534 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:48.556 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:48.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:48.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:48.660 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:18:48.693 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:48.714 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:18:48.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:48.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:18:48.818 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:18:48.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:48 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:18:48.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:48 smithi088 ceph-mon[44699]: from='client.35789 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:18:48.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:48 smithi088 ceph-mon[44699]: osdmap e357: 12 total, 11 up, 11 in 2024-05-15T23:18:48.849 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:48.869 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:18:48.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:48.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:18:48.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:49.003 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:48 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:18:49.003 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:48 smithi090 ceph-mon[44434]: from='client.35789 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:18:49.004 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:48 smithi090 ceph-mon[44434]: osdmap e357: 12 total, 11 up, 11 in 2024-05-15T23:18:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:49.030 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:49.033 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:48 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:18:49.033 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:48 smithi084 ceph-mon[39784]: from='client.35789 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:18:49.033 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:48 smithi084 ceph-mon[39784]: osdmap e357: 12 total, 11 up, 11 in 2024-05-15T23:18:49.130 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:18:49.163 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:49.184 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:18:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:49.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:18:49.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:18:49.289 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:18:49.318 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:49.337 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:18:49.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:49.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:18:49.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:18:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:49.496 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:18:49.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:49.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:18:49.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:18:49.628 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:49.776 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:18:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:49.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:18:49.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:49 smithi090 ceph-mon[44434]: pgmap v982: 17 pgs: 1 stale+active+clean, 16 active+clean; 1.1 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:18:49.880 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:18:49.913 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:49.934 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:18:49.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:49.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:18:50.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:50.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:49 smithi084 ceph-mon[39784]: pgmap v982: 17 pgs: 1 stale+active+clean, 16 active+clean; 1.1 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:18:50.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:49 smithi088 ceph-mon[44699]: pgmap v982: 17 pgs: 1 stale+active+clean, 16 active+clean; 1.1 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.26k op/s 2024-05-15T23:18:50.071 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:50.093 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:50.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:50.096 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:50.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:18:50.227 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:50.247 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:18:50.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:50.250 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:18:50.351 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:18:50.384 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:50.404 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:18:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:50.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:18:50.508 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:18:50.538 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:50.557 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:18:50.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:50.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:18:50.586 INFO:teuthology.orchestra.run.smithi084.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-05-15T23:18:50.661 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:50.693 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:50.767 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:50.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:50.871 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:18:50.904 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:50.905 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:18:50.926 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:18:50.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:50.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:18:51.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:18:51.062 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:51.083 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:18:51.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:51.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:18:51.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:18:51.220 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:51.241 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:18:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:51.244 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:18:51.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:51 smithi090 ceph-mon[44434]: pgmap v983: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 914 op/s; 1035/32642 objects degraded (3.171%) 2024-05-15T23:18:51.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:51 smithi090 ceph-mon[44434]: Health check failed: Degraded data redundancy: 1035/32642 objects degraded (3.171%), 1 pg degraded (PG_DEGRADED) 2024-05-15T23:18:51.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:51 smithi088 ceph-mon[44699]: pgmap v983: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 914 op/s; 1035/32642 objects degraded (3.171%) 2024-05-15T23:18:51.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:51 smithi088 ceph-mon[44699]: Health check failed: Degraded data redundancy: 1035/32642 objects degraded (3.171%), 1 pg degraded (PG_DEGRADED) 2024-05-15T23:18:51.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:18:51.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:51 smithi084 ceph-mon[39784]: pgmap v983: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 914 op/s; 1035/32642 objects degraded (3.171%) 2024-05-15T23:18:51.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:51 smithi084 ceph-mon[39784]: Health check failed: Degraded data redundancy: 1035/32642 objects degraded (3.171%), 1 pg degraded (PG_DEGRADED) 2024-05-15T23:18:51.373 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T23:18:51.373 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T23:18:51.375 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:51.395 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:18:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:51.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:18:51.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:18:51.526 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:51.546 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:18:51.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:51.549 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:18:51.650 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:18:51.683 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:51.699 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:51.704 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:18:51.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:51.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:18:51.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:18:51.838 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:51.858 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:18:51.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:51.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:18:51.962 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:18:51.995 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:52.016 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:18:52.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:52.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:18:52.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.0.service' 2024-05-15T23:18:52.119 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:18:52.133 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:18:52.134 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-15T23:18:52.134 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd down 0 2024-05-15T23:18:52.139 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:52.149 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:52.168 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:18:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:52.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:18:52.272 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:18:52.305 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:52.326 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:18:52.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:52.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:18:52.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:18:52.460 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:52.480 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:18:52.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:52.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:18:52.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:18:52.612 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:52.632 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:18:52.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:52.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:18:52.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:18:52.779 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:52.796 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:18:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:52.798 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:18:52.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:52.929 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:54.077 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:18:54.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:54.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:18:54.115 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:54.180 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:18:54.207 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:54.226 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:18:54.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:54.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:18:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:53 smithi090 ceph-mon[44434]: pgmap v984: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 648 op/s; 1035/32642 objects degraded (3.171%) 2024-05-15T23:18:54.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:53 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:18:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:53 smithi088 ceph-mon[44699]: pgmap v984: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 648 op/s; 1035/32642 objects degraded (3.171%) 2024-05-15T23:18:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:53 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:18:54.329 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:54.356 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:53 smithi084 ceph-mon[39784]: pgmap v984: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 648 op/s; 1035/32642 objects degraded (3.171%) 2024-05-15T23:18:54.356 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:53 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:18:54.358 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:54.977 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:18:55.114 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:55.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:55.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:55.213 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:18:55.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:18:55.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:18:55.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:18:55.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:18:55.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:18:55.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:18:55.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:55 smithi084 ceph-mon[39784]: 4.e scrub starts 2024-05-15T23:18:55.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:55 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1037745308' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:18:55.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:55 smithi084 ceph-mon[39784]: pgmap v985: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 382 op/s; 1035/32642 objects degraded (3.171%) 2024-05-15T23:18:55.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:55 smithi084 ceph-mon[39784]: 4.e scrub ok 2024-05-15T23:18:55.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:18:55.253 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:55.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:55 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:18:55.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:55 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:18:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:55 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:18:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:55 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:18:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:55 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:18:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:55 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:18:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:55 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:18:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:55 smithi090 ceph-mon[44434]: 4.e scrub starts 2024-05-15T23:18:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:55 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1037745308' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:18:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:55 smithi090 ceph-mon[44434]: pgmap v985: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 382 op/s; 1035/32642 objects degraded (3.171%) 2024-05-15T23:18:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:55 smithi090 ceph-mon[44434]: 4.e scrub ok 2024-05-15T23:18:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:55 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:18:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:55 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:18:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:55 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:18:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:55 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:18:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:55 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:18:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:55 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:18:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:55 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:18:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:55 smithi088 ceph-mon[44699]: 4.e scrub starts 2024-05-15T23:18:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:55 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1037745308' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:18:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:55 smithi088 ceph-mon[44699]: pgmap v985: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 382 op/s; 1035/32642 objects degraded (3.171%) 2024-05-15T23:18:55.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:55 smithi088 ceph-mon[44699]: 4.e scrub ok 2024-05-15T23:18:55.704 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:18:56.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:56 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:18:56.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1037745308' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:18:56.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:56 smithi090 ceph-mon[44434]: osdmap e358: 12 total, 11 up, 11 in 2024-05-15T23:18:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:56 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:18:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1037745308' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:18:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:56 smithi088 ceph-mon[44699]: osdmap e358: 12 total, 11 up, 11 in 2024-05-15T23:18:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:56 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:18:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1037745308' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:18:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:56 smithi084 ceph-mon[39784]: osdmap e358: 12 total, 11 up, 11 in 2024-05-15T23:18:57.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:57 smithi084 ceph-mon[39784]: pgmap v987: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 341 op/s; 1035/32642 objects degraded (3.171%) 2024-05-15T23:18:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:57 smithi090 ceph-mon[44434]: pgmap v987: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 341 op/s; 1035/32642 objects degraded (3.171%) 2024-05-15T23:18:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:57 smithi088 ceph-mon[44699]: pgmap v987: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 341 op/s; 1035/32642 objects degraded (3.171%) 2024-05-15T23:18:57.603 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:18:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:57.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:18:57.707 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:57.737 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:18:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:58 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:18:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:58 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:18:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:58 smithi090 ceph-mon[44434]: osdmap e359: 12 total, 11 up, 11 in 2024-05-15T23:18:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:58 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3349828234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:58 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:18:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:58 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:18:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:58 smithi088 ceph-mon[44699]: osdmap e359: 12 total, 11 up, 11 in 2024-05-15T23:18:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:58 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3349828234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:58 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:18:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:58 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:18:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:58 smithi084 ceph-mon[39784]: osdmap e359: 12 total, 11 up, 11 in 2024-05-15T23:18:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:58 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3349828234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:18:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:58.779 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0 is already down. 2024-05-15T23:18:58.790 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:18:58.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:18:58.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:18:58.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:18:58.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:18:58.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:18:58.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:18:58.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:18:58.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:18:58.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:18:58.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:18:58.849 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:18:58.849 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-05-15T23:18:58.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:18:58.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:18:58.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:18:58.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:18:58.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:18:58.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:18:58.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:18:58.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:18:58.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:18:58.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:18:58.850 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:18:59.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:58 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:18:58.773+0000 7fa0938b7700 -1 mon.a@0(leader).osd e359 definitely_dead 0 2024-05-15T23:18:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:59 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3962809977' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["0"]}]: dispatch 2024-05-15T23:18:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:18:59 smithi090 ceph-mon[44434]: pgmap v989: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 1035/32642 objects degraded (3.171%) 2024-05-15T23:18:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:59 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3962809977' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["0"]}]: dispatch 2024-05-15T23:18:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:18:59 smithi088 ceph-mon[44699]: pgmap v989: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 1035/32642 objects degraded (3.171%) 2024-05-15T23:18:59.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:18:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:59 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3962809977' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["0"]}]: dispatch 2024-05-15T23:18:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:18:59 smithi084 ceph-mon[39784]: pgmap v989: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 1035/32642 objects degraded (3.171%) 2024-05-15T23:18:59.777 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:18:59.777 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:18:59.786 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:18:59.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:18:59.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:18:59.889 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:18:59.918 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:00.100 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 3, 4, 5, 6, 7, 8, 11, 9, 10] 2024-05-15T23:19:00.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd out 0 2024-05-15T23:19:00.370 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:00.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:00.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:00.382 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:19:00.382 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:19:00.396 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:00.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:00.506 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:00.604 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:00.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:00.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:00.610 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:00.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:00.743 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:00.764 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:00.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:00.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:00.869 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:00.903 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:00.925 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:00.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:01.029 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:01.058 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:01.078 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:01.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:01.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:01.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:01 smithi084 ceph-mon[39784]: pgmap v990: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 1035/32642 objects degraded (3.171%) 2024-05-15T23:19:01.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:01.212 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:01.240 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:01.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:01.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:01.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:01 smithi090 ceph-mon[44434]: pgmap v990: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 1035/32642 objects degraded (3.171%) 2024-05-15T23:19:01.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:01 smithi088 ceph-mon[44699]: pgmap v990: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 1035/32642 objects degraded (3.171%) 2024-05-15T23:19:01.344 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:01.374 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:01.394 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:01.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:01.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:01.531 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:01.552 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:01.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:01.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:01.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:01.685 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:01.704 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:01.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:01.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:01.779 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:19:01.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:01.849 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:02.254 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:02.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:02.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:02.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:02.385 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:03.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T23:19:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:02 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:19:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:02 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:19:03.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:02 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:19:03.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:02 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:19:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:02 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:19:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:02 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:19:03.765 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:03.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:03.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:03.902 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:03.923 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:03.927 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:04.028 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:04.061 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:04.081 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:04.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:04.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:03 smithi084 ceph-mon[39784]: pgmap v991: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 1035/32642 objects degraded (3.171%) 2024-05-15T23:19:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:03 smithi084 ceph-mon[39784]: osdmap e360: 12 total, 11 up, 11 in 2024-05-15T23:19:04.185 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:04.215 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:04.252 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:04.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:04.270 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:03 smithi090 ceph-mon[44434]: pgmap v991: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 1035/32642 objects degraded (3.171%) 2024-05-15T23:19:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:03 smithi090 ceph-mon[44434]: osdmap e360: 12 total, 11 up, 11 in 2024-05-15T23:19:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:03 smithi088 ceph-mon[44699]: pgmap v991: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 1035/32642 objects degraded (3.171%) 2024-05-15T23:19:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:03 smithi088 ceph-mon[44699]: osdmap e360: 12 total, 11 up, 11 in 2024-05-15T23:19:04.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:04.389 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:04.409 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:04.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:04.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:04.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:04.523 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:04 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:19:04.240+0000 7fa0938b7700 -1 mon.a@0(leader).osd e361 definitely_dead 0 2024-05-15T23:19:04.543 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:04.563 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:04.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:04.566 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:04.667 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:04.695 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:04.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:04.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:04.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:04.845 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:04.864 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:04.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:04.867 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:04.967 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:04.998 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.0. 2024-05-15T23:19:05.001 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:19:05.001 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:05.023 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:05.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:05.127 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:05.157 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:05.177 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:05.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:05.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:04 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:19:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:04 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:19:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:04 smithi090 ceph-mon[44434]: osdmap e361: 12 total, 11 up, 11 in 2024-05-15T23:19:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1570512816' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-05-15T23:19:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:04 smithi090 ceph-mon[44434]: Client client.admin marked osd.0 out, after it was down for 17 seconds 2024-05-15T23:19:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/176967340' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:19:05.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:04 smithi090 ceph-mon[44434]: from='client.25744 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:19:05.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:04 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:19:05.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:04 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:19:05.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:04 smithi088 ceph-mon[44699]: osdmap e361: 12 total, 11 up, 11 in 2024-05-15T23:19:05.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1570512816' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-05-15T23:19:05.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:04 smithi088 ceph-mon[44699]: Client client.admin marked osd.0 out, after it was down for 17 seconds 2024-05-15T23:19:05.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/176967340' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:19:05.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:04 smithi088 ceph-mon[44699]: from='client.25744 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:19:05.310 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:05.330 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:05.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:05.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:05.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:04 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:19:05.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:04 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:19:05.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:04 smithi084 ceph-mon[39784]: osdmap e361: 12 total, 11 up, 11 in 2024-05-15T23:19:05.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1570512816' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-05-15T23:19:05.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:04 smithi084 ceph-mon[39784]: Client client.admin marked osd.0 out, after it was down for 17 seconds 2024-05-15T23:19:05.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/176967340' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:19:05.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:04 smithi084 ceph-mon[39784]: from='client.25744 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:19:05.434 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:05.463 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:05.911 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:05.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:05.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:06.015 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:06.050 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:06.071 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:06.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:06.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:06.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:06.208 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:06.230 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:06.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:06.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:06 smithi090 ceph-mon[44434]: pgmap v994: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 1035/32642 objects degraded (3.171%) 2024-05-15T23:19:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:06 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:19:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:06 smithi090 ceph-mon[44434]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-15T23:19:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:06 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1570512816' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-05-15T23:19:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:06 smithi090 ceph-mon[44434]: from='client.25744 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:19:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:06 smithi090 ceph-mon[44434]: osdmap e362: 12 total, 11 up, 10 in 2024-05-15T23:19:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:06 smithi088 ceph-mon[44699]: pgmap v994: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 1035/32642 objects degraded (3.171%) 2024-05-15T23:19:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:06 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:19:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:06 smithi088 ceph-mon[44699]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-15T23:19:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:06 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1570512816' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-05-15T23:19:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:06 smithi088 ceph-mon[44699]: from='client.25744 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:19:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:06 smithi088 ceph-mon[44699]: osdmap e362: 12 total, 11 up, 10 in 2024-05-15T23:19:06.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:06.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:05 smithi084 ceph-mon[39784]: pgmap v994: 17 pgs: 1 undersized+degraded+peered, 16 active+clean; 1020 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 1035/32642 objects degraded (3.171%) 2024-05-15T23:19:06.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:05 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:19:06.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:06 smithi084 ceph-mon[39784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-15T23:19:06.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:06 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1570512816' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-05-15T23:19:06.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:06 smithi084 ceph-mon[39784]: from='client.25744 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:19:06.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:06 smithi084 ceph-mon[39784]: osdmap e362: 12 total, 11 up, 10 in 2024-05-15T23:19:06.365 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:06.375 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:06.385 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:06.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:06.387 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:06.488 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:06.522 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:06.543 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:06.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:06.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:06.647 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:19:06.676 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:06.696 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:19:06.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:06.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:19:06.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:19:06.830 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:07.149 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2024-05-15T23:19:07.149 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi084.front.sepia.ceph.com 2024-05-15T23:19:07.149 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -n osd.0 --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-05-15T23:19:07.241 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:19:07.241 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:19:07.241 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:19:07.241 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:19:07.241 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:19:07.241 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:19:07.241 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:19:07.241 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:19:07.241 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:19:07.242 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:19:07.242 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:19:07.242 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-05-15T23:19:07.242 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:19:07.242 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:19:07.242 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:19:07.242 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:19:07.242 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:19:07.242 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:19:07.242 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:19:07.242 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:19:07.242 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:19:07.243 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:19:07.243 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:19:07.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:07 smithi090 ceph-mon[44434]: osdmap e363: 12 total, 11 up, 10 in 2024-05-15T23:19:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:07 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2882045702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:07 smithi090 ceph-mon[44434]: pgmap v997: 17 pgs: 1 peering, 16 active+clean; 1020 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail 2024-05-15T23:19:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:07 smithi088 ceph-mon[44699]: osdmap e363: 12 total, 11 up, 10 in 2024-05-15T23:19:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:07 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2882045702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:07 smithi088 ceph-mon[44699]: pgmap v997: 17 pgs: 1 peering, 16 active+clean; 1020 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail 2024-05-15T23:19:07.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:07 smithi084 ceph-mon[39784]: osdmap e363: 12 total, 11 up, 10 in 2024-05-15T23:19:07.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:07 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2882045702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:07.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:07 smithi084 ceph-mon[39784]: pgmap v997: 17 pgs: 1 peering, 16 active+clean; 1020 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail 2024-05-15T23:19:07.441 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/osd.0/config 2024-05-15T23:19:08.003 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:19:08.003 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:19:08.236 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.236 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.236 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.236 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.236 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:08 smithi084 ceph-mon[39784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1035/32642 objects degraded (3.171%), 1 pg degraded) 2024-05-15T23:19:08.236 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.237 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:08 smithi084 ceph-mon[39784]: 4.b starting backfill to osd.4 from (0'0,0'0] MAX to 355'20619 2024-05-15T23:19:08.237 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:08 smithi084 ceph-mon[39784]: osdmap e364: 12 total, 11 up, 10 in 2024-05-15T23:19:08.237 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.237 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.237 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.237 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.237 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:08 smithi090 ceph-mon[44434]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1035/32642 objects degraded (3.171%), 1 pg degraded) 2024-05-15T23:19:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:08 smithi090 ceph-mon[44434]: 4.b starting backfill to osd.4 from (0'0,0'0] MAX to 355'20619 2024-05-15T23:19:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:08 smithi090 ceph-mon[44434]: osdmap e364: 12 total, 11 up, 10 in 2024-05-15T23:19:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:08 smithi088 ceph-mon[44699]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1035/32642 objects degraded (3.171%), 1 pg degraded) 2024-05-15T23:19:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:08 smithi088 ceph-mon[44699]: 4.b starting backfill to osd.4 from (0'0,0'0] MAX to 355'20619 2024-05-15T23:19:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:08 smithi088 ceph-mon[44699]: osdmap e364: 12 total, 11 up, 10 in 2024-05-15T23:19:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:08.398 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:19:09.280 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:19:09.282 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:09.282 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:09 smithi084 ceph-mon[39784]: pgmap v999: 17 pgs: 1 peering, 16 active+clean; 1020 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail 2024-05-15T23:19:09.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:09.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:19:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:09 smithi090 ceph-mon[44434]: pgmap v999: 17 pgs: 1 peering, 16 active+clean; 1020 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail 2024-05-15T23:19:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:09 smithi088 ceph-mon[44699]: pgmap v999: 17 pgs: 1 peering, 16 active+clean; 1020 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail 2024-05-15T23:19:09.384 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:19:09.411 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:09.438 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:19:09.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:09.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:19:09.549 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:09.581 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:09.602 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:09.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:09.605 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:09.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:09.733 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:10.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:19:10.289 INFO:teuthology.orchestra.run.smithi084.stderr:Creating an OSD daemon form without an OSD FSID value 2024-05-15T23:19:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:10 smithi090 ceph-mon[44434]: osdmap e365: 12 total, 11 up, 10 in 2024-05-15T23:19:10.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:10 smithi088 ceph-mon[44699]: osdmap e365: 12 total, 11 up, 10 in 2024-05-15T23:19:10.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:10 smithi084 ceph-mon[39784]: osdmap e365: 12 total, 11 up, 10 in 2024-05-15T23:19:10.687 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:10.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:10.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:10.792 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:10.825 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:10.851 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:10.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:10.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:10.957 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:11.001 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:11.023 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:11.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:11.132 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:11.164 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:11 smithi088 ceph-mon[44699]: osdmap e366: 12 total, 11 up, 10 in 2024-05-15T23:19:11.164 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:11 smithi088 ceph-mon[44699]: pgmap v1002: 17 pgs: 17 active+clean; 1000 MiB data, 4.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 67 op/s; 13 MiB/s, 208 objects/s recovering 2024-05-15T23:19:11.166 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:11.196 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:11.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:11.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:11.277 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:11 smithi084 ceph-mon[39784]: osdmap e366: 12 total, 11 up, 10 in 2024-05-15T23:19:11.277 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:11 smithi084 ceph-mon[39784]: pgmap v1002: 17 pgs: 17 active+clean; 1000 MiB data, 4.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 67 op/s; 13 MiB/s, 208 objects/s recovering 2024-05-15T23:19:11.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:11.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:11 smithi090 ceph-mon[44434]: osdmap e366: 12 total, 11 up, 10 in 2024-05-15T23:19:11.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:11 smithi090 ceph-mon[44434]: pgmap v1002: 17 pgs: 17 active+clean; 1000 MiB data, 4.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 67 op/s; 13 MiB/s, 208 objects/s recovering 2024-05-15T23:19:11.333 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:11.381 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:11.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:11.485 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:11.521 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:11.560 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:11.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:11.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:11.665 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:11.699 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:11.733 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:11.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:11.811 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:19:11.811 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:19:11.811 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:19:11.811 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:19:11.811 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:19:11.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:11.882 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:11.910 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:11.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:11.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:12.014 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:12.055 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:12.085 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:12.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:12.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:12.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:12 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:19:12.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:12 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:19:12.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:12.222 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:12 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:19:12.222 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:12 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:19:12.224 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:12.250 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:12.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:12.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:12 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:19:12.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:12 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:19:12.359 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:12.390 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:12.420 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:12.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:12.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:12.525 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:12.567 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:12.607 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:12.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:12.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:12.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:12.747 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:12.777 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:12.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:12.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:12.914 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:12.948 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:12.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:12.955 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:13.056 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:13.099 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:13.168 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:13.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:13.212 INFO:teuthology.orchestra.run.smithi084.stdout:4.b 2024-05-15T23:19:13.217 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T23:19:13.287 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:13.314 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:13 smithi088 ceph-mon[44699]: osdmap e367: 12 total, 11 up, 10 in 2024-05-15T23:19:13.315 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:13 smithi088 ceph-mon[44699]: pgmap v1004: 17 pgs: 17 active+clean; 937 MiB data, 4.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 225 op/s; 11 MiB/s, 173 objects/s recovering 2024-05-15T23:19:13.317 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:13.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:13 smithi084 ceph-mon[39784]: osdmap e367: 12 total, 11 up, 10 in 2024-05-15T23:19:13.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:13 smithi084 ceph-mon[39784]: pgmap v1004: 17 pgs: 17 active+clean; 937 MiB data, 4.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 225 op/s; 11 MiB/s, 173 objects/s recovering 2024-05-15T23:19:13.351 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:13.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:13.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:13.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:13 smithi090 ceph-mon[44434]: osdmap e367: 12 total, 11 up, 10 in 2024-05-15T23:19:13.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:13 smithi090 ceph-mon[44434]: pgmap v1004: 17 pgs: 17 active+clean; 937 MiB data, 4.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 225 op/s; 11 MiB/s, 173 objects/s recovering 2024-05-15T23:19:13.492 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:13.514 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:13.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:13.518 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:13.619 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:19:13.654 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:13.686 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:19:13.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:13.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:19:13.792 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:13.795 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:13.864 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:13.991 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:13.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:13.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:14.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:14.128 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:14.152 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:14.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:14.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:14.289 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:14.314 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:14.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:14.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:14.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:14.457 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:15.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:15 smithi090 ceph-mon[44434]: pgmap v1005: 17 pgs: 17 active+clean; 873 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 392 op/s; 11 MiB/s, 172 objects/s recovering 2024-05-15T23:19:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:15 smithi088 ceph-mon[44699]: pgmap v1005: 17 pgs: 17 active+clean; 873 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 392 op/s; 11 MiB/s, 172 objects/s recovering 2024-05-15T23:19:15.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:15 smithi084 ceph-mon[39784]: pgmap v1005: 17 pgs: 17 active+clean; 873 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 392 op/s; 11 MiB/s, 172 objects/s recovering 2024-05-15T23:19:15.723 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:15.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:15.828 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:15.860 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:15.883 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:15.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:15.989 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:16.022 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:16.058 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:16.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:16.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:16.167 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:16.194 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -n osd.0 --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 4.b --file /var/log/ceph/exp.4.b.0 2024-05-15T23:19:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:16.229 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:16.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:16.333 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:16.395 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:16.738 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:19:16.760 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:19:16.785 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:19:16.827 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:19:16.853 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:19:16.881 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:19:16.909 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:19:16.940 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:19:16.967 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:19:16.995 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:19:17.019 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:19:17.020 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-05-15T23:19:17.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:19:17.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:19:17.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:19:17.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:19:17.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:19:17.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:19:17.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:19:17.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:19:17.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:19:17.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:19:17.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:19:17.035 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:16 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/690306282' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:17.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:16 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/690306282' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:17.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:16 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/690306282' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:17.424 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/osd.0/config 2024-05-15T23:19:17.860 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:17.860 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:17.860 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:17.861 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:17.861 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:17.861 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:17.861 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:17.861 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:17 smithi084 ceph-mon[39784]: pgmap v1006: 17 pgs: 17 active+clean; 607 MiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 833 op/s; 8.1 MiB/s, 130 objects/s recovering 2024-05-15T23:19:17.861 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:17.861 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:17.861 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:17.870 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:17.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:17.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:17.905 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:19:17.906 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:19:17.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:18.023 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.023 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.023 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.023 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.023 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.023 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.023 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.023 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:17 smithi090 ceph-mon[44434]: pgmap v1006: 17 pgs: 17 active+clean; 607 MiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 833 op/s; 8.1 MiB/s, 130 objects/s recovering 2024-05-15T23:19:18.024 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.024 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.024 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.025 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:18.052 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:18.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:18.060 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:17 smithi088 ceph-mon[44699]: pgmap v1006: 17 pgs: 17 active+clean; 607 MiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 833 op/s; 8.1 MiB/s, 130 objects/s recovering 2024-05-15T23:19:18.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.161 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:18.190 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:18.219 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:18.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:18.324 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:18.361 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:18.391 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:18.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:18.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:18.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:18.526 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:18.550 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:18.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:18.556 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:18.657 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:18.662 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:18.739 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:18.757 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:18.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:19.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:19.209 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:19.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:19.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:19.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:19.344 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:19.372 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:19.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:19.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:19.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:19.526 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:19.553 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:19.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:19.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:19.660 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:19:19.697 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:19.770 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2751509649' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:19:19.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:19 smithi090 ceph-mon[44434]: from='client.25759 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:19:19.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:19 smithi090 ceph-mon[44434]: pgmap v1007: 17 pgs: 17 active+clean; 559 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 824 op/s; 7.2 MiB/s, 115 objects/s recovering 2024-05-15T23:19:19.788 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:19:19.907 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:19:20.038 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2751509649' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:19:20.038 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:19 smithi084 ceph-mon[39784]: from='client.25759 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:19:20.038 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:19 smithi084 ceph-mon[39784]: pgmap v1007: 17 pgs: 17 active+clean; 559 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 824 op/s; 7.2 MiB/s, 115 objects/s recovering 2024-05-15T23:19:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2751509649' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:19:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:19 smithi088 ceph-mon[44699]: from='client.25759 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:19:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:19 smithi088 ceph-mon[44699]: pgmap v1007: 17 pgs: 17 active+clean; 559 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 824 op/s; 7.2 MiB/s, 115 objects/s recovering 2024-05-15T23:19:20.355 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:19:20.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:20.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:19:20.425 INFO:teuthology.orchestra.run.smithi084.stderr:Creating an OSD daemon form without an OSD FSID value 2024-05-15T23:19:20.462 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:20.508 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:20.532 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:20.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:20.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:20.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:20.690 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:20.722 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:20.725 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:20.773 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:20 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:19:20.773 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:20 smithi088 ceph-mon[44699]: from='client.25759 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:19:20.773 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:20 smithi088 ceph-mon[44699]: osdmap e368: 12 total, 11 up, 10 in 2024-05-15T23:19:20.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:19:20.858 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:21.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:20 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:19:21.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:20 smithi090 ceph-mon[44434]: from='client.25759 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:19:21.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:20 smithi090 ceph-mon[44434]: osdmap e368: 12 total, 11 up, 10 in 2024-05-15T23:19:21.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:20 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:19:21.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:20 smithi084 ceph-mon[39784]: from='client.25759 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:19:21.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:20 smithi084 ceph-mon[39784]: osdmap e368: 12 total, 11 up, 10 in 2024-05-15T23:19:22.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:21 smithi090 ceph-mon[44434]: pgmap v1009: 17 pgs: 17 active+clean; 314 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:19:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:21 smithi088 ceph-mon[44699]: pgmap v1009: 17 pgs: 17 active+clean; 314 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:19:22.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:21 smithi084 ceph-mon[39784]: pgmap v1009: 17 pgs: 17 active+clean; 314 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:19:22.614 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:19:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:22.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:19:22.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:22.759 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:22.782 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:22.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:22.801 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:19:22.802 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:19:22.891 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:22.919 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:22.949 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:22.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:22.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:23.011 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:23.020 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:23 smithi090 ceph-mon[44434]: 4.3 scrub starts 2024-05-15T23:19:23.020 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:23 smithi090 ceph-mon[44434]: 4.c scrub starts 2024-05-15T23:19:23.020 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:23 smithi090 ceph-mon[44434]: 4.3 scrub ok 2024-05-15T23:19:23.020 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:23 smithi090 ceph-mon[44434]: 4.c scrub ok 2024-05-15T23:19:23.020 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:23 smithi090 ceph-mon[44434]: 4.a scrub starts 2024-05-15T23:19:23.020 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:23 smithi090 ceph-mon[44434]: 4.a scrub ok 2024-05-15T23:19:23.021 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:23 smithi090 ceph-mon[44434]: pgmap v1010: 17 pgs: 17 active+clean; 165 MiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:19:23.055 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:23.085 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:23.112 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:23.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:23.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:23.167 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Removed 173235 objects 2024-05-15T23:19:23.167 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Clean up completed and total clean up time :165.807 2024-05-15T23:19:23.220 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:23.263 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:23.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:23.266 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:23 smithi088 ceph-mon[44699]: 4.3 scrub starts 2024-05-15T23:19:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:23 smithi088 ceph-mon[44699]: 4.c scrub starts 2024-05-15T23:19:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:23 smithi088 ceph-mon[44699]: 4.3 scrub ok 2024-05-15T23:19:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:23 smithi088 ceph-mon[44699]: 4.c scrub ok 2024-05-15T23:19:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:23 smithi088 ceph-mon[44699]: 4.a scrub starts 2024-05-15T23:19:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:23 smithi088 ceph-mon[44699]: 4.a scrub ok 2024-05-15T23:19:23.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:23 smithi088 ceph-mon[44699]: pgmap v1010: 17 pgs: 17 active+clean; 165 MiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:19:23.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:19:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:23 smithi084 ceph-mon[39784]: 4.3 scrub starts 2024-05-15T23:19:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:23 smithi084 ceph-mon[39784]: 4.c scrub starts 2024-05-15T23:19:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:23 smithi084 ceph-mon[39784]: 4.3 scrub ok 2024-05-15T23:19:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:23 smithi084 ceph-mon[39784]: 4.c scrub ok 2024-05-15T23:19:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:23 smithi084 ceph-mon[39784]: 4.a scrub starts 2024-05-15T23:19:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:23 smithi084 ceph-mon[39784]: 4.a scrub ok 2024-05-15T23:19:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:23 smithi084 ceph-mon[39784]: pgmap v1010: 17 pgs: 17 active+clean; 165 MiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:19:23.397 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:23.522 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:19:23.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:23.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:19:23.532 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T23:19:23.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:23.653 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:23.753 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:23.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:23.789 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:19:23.857 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:19:23.883 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:23.914 INFO:teuthology.orchestra.run.smithi084.stderr:Exporting 4.b info 4.b( v 355'20619 (318'10600,355'20619] local-lis/les=283/284 n=1035 ec=283/283 lis/c=283/283 les/c/f=284/284/0 sis=283) 2024-05-15T23:19:23.961 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d012317e:::benchmark_data_smithi090_104448_object158482:head# 2024-05-15T23:19:23.961 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0129e53:::benchmark_data_smithi090_104448_object171208:head# 2024-05-15T23:19:23.962 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d01799a4:::benchmark_data_smithi090_104448_object170254:head# 2024-05-15T23:19:23.962 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0199fca:::benchmark_data_smithi090_104448_object166694:head# 2024-05-15T23:19:23.962 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d01c2eba:::benchmark_data_smithi090_104448_object173055:head# 2024-05-15T23:19:23.963 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0228b4f:::benchmark_data_smithi090_104448_object171832:head# 2024-05-15T23:19:23.963 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d039794a:::benchmark_data_smithi090_104448_object168370:head# 2024-05-15T23:19:23.964 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d041cc54:::benchmark_data_smithi090_104448_object162077:head# 2024-05-15T23:19:23.964 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d042ed42:::benchmark_data_smithi090_104448_object164503:head# 2024-05-15T23:19:23.964 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d048394e:::benchmark_data_smithi090_104448_object169206:head# 2024-05-15T23:19:23.965 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d04bd93d:::benchmark_data_smithi090_104448_object162358:head# 2024-05-15T23:19:23.965 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d04ebc3d:::benchmark_data_smithi090_104448_object161958:head# 2024-05-15T23:19:23.966 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d056fa0f:::benchmark_data_smithi090_104448_object159835:head# 2024-05-15T23:19:23.966 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0570e84:::benchmark_data_smithi090_104448_object157201:head# 2024-05-15T23:19:23.966 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0583124:::benchmark_data_smithi090_104448_object162507:head# 2024-05-15T23:19:23.967 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d05a82f5:::benchmark_data_smithi090_104448_object167662:head# 2024-05-15T23:19:23.967 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0693e91:::benchmark_data_smithi090_104448_object167715:head# 2024-05-15T23:19:23.968 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d06cf267:::benchmark_data_smithi090_104448_object171642:head# 2024-05-15T23:19:23.968 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d077e1bb:::benchmark_data_smithi090_104448_object158950:head# 2024-05-15T23:19:23.969 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0839531:::benchmark_data_smithi090_104448_object162232:head# 2024-05-15T23:19:23.969 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d08665b5:::benchmark_data_smithi090_104448_object170017:head# 2024-05-15T23:19:23.969 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0893b05:::benchmark_data_smithi090_104448_object159959:head# 2024-05-15T23:19:23.969 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d08a3c96:::benchmark_data_smithi090_104448_object164396:head# 2024-05-15T23:19:23.970 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d090e161:::benchmark_data_smithi090_104448_object164996:head# 2024-05-15T23:19:23.970 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0938174:::benchmark_data_smithi090_104448_object161965:head# 2024-05-15T23:19:23.971 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0961fe2:::benchmark_data_smithi090_104448_object160775:head# 2024-05-15T23:19:23.971 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d097132d:::benchmark_data_smithi090_104448_object168619:head# 2024-05-15T23:19:23.971 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0a3a586:::benchmark_data_smithi090_104448_object167456:head# 2024-05-15T23:19:23.972 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0ae2ef2:::benchmark_data_smithi090_104448_object156795:head# 2024-05-15T23:19:23.972 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0b5b01f:::benchmark_data_smithi090_104448_object162424:head# 2024-05-15T23:19:23.972 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0b7dd29:::benchmark_data_smithi090_104448_object168193:head# 2024-05-15T23:19:23.973 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0ba4865:::benchmark_data_smithi090_104448_object170236:head# 2024-05-15T23:19:23.973 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0c0a793:::benchmark_data_smithi090_104448_object162087:head# 2024-05-15T23:19:23.974 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0c1412a:::benchmark_data_smithi090_104448_object165902:head# 2024-05-15T23:19:23.975 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0c2a2bd:::benchmark_data_smithi090_104448_object169146:head# 2024-05-15T23:19:23.975 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0d1eb55:::benchmark_data_smithi090_104448_object163076:head# 2024-05-15T23:19:23.975 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0d451e6:::benchmark_data_smithi090_104448_object166219:head# 2024-05-15T23:19:23.975 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0d632e9:::benchmark_data_smithi090_104448_object161322:head# 2024-05-15T23:19:23.976 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0d71efd:::benchmark_data_smithi090_104448_object167586:head# 2024-05-15T23:19:23.976 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0d836eb:::benchmark_data_smithi090_104448_object171791:head# 2024-05-15T23:19:23.977 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0d8f85c:::benchmark_data_smithi090_104448_object172772:head# 2024-05-15T23:19:23.977 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0d9e179:::benchmark_data_smithi090_104448_object158467:head# 2024-05-15T23:19:23.977 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0dc8ddf:::benchmark_data_smithi090_104448_object156993:head# 2024-05-15T23:19:23.978 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0e3f610:::benchmark_data_smithi090_104448_object162718:head# 2024-05-15T23:19:23.978 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0e5e66a:::benchmark_data_smithi090_104448_object168295:head# 2024-05-15T23:19:23.978 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0e63550:::benchmark_data_smithi090_104448_object166312:head# 2024-05-15T23:19:23.979 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0ed8768:::benchmark_data_smithi090_104448_object171929:head# 2024-05-15T23:19:23.979 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0edc9e5:::benchmark_data_smithi090_104448_object160465:head# 2024-05-15T23:19:23.980 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0eeb3e0:::benchmark_data_smithi090_104448_object168084:head# 2024-05-15T23:19:23.980 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0f12942:::benchmark_data_smithi090_104448_object161880:head# 2024-05-15T23:19:23.980 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d0f75f82:::benchmark_data_smithi090_104448_object165506:head# 2024-05-15T23:19:23.981 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1027389:::benchmark_data_smithi090_104448_object165741:head# 2024-05-15T23:19:23.981 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1097834:::benchmark_data_smithi090_104448_object168326:head# 2024-05-15T23:19:23.982 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d11472fe:::benchmark_data_smithi090_104448_object166226:head# 2024-05-15T23:19:23.982 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1149b84:::benchmark_data_smithi090_104448_object165733:head# 2024-05-15T23:19:23.982 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d115d225:::benchmark_data_smithi090_104448_object157765:head# 2024-05-15T23:19:23.983 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d116d390:::benchmark_data_smithi090_104448_object167799:head# 2024-05-15T23:19:23.983 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1175bda:::benchmark_data_smithi090_104448_object168282:head# 2024-05-15T23:19:23.983 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1192fa5:::benchmark_data_smithi090_104448_object157520:head# 2024-05-15T23:19:23.984 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d11c54b3:::benchmark_data_smithi090_104448_object159274:head# 2024-05-15T23:19:23.984 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d11f79e1:::benchmark_data_smithi090_104448_object171402:head# 2024-05-15T23:19:23.986 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d121acda:::benchmark_data_smithi090_104448_object168195:head# 2024-05-15T23:19:23.986 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d121c251:::benchmark_data_smithi090_104448_object167519:head# 2024-05-15T23:19:23.986 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1258048:::benchmark_data_smithi090_104448_object167134:head# 2024-05-15T23:19:23.986 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1282744:::benchmark_data_smithi090_104448_object159399:head# 2024-05-15T23:19:23.987 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d12bcf63:::benchmark_data_smithi090_104448_object170235:head# 2024-05-15T23:19:23.990 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1320bf1:::benchmark_data_smithi090_104448_object168129:head# 2024-05-15T23:19:23.990 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d13507c5:::benchmark_data_smithi090_104448_object171158:head# 2024-05-15T23:19:23.990 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d135c7fc:::benchmark_data_smithi090_104448_object165857:head# 2024-05-15T23:19:23.990 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d13deac4:::benchmark_data_smithi090_104448_object168338:head# 2024-05-15T23:19:23.990 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d141d64d:::benchmark_data_smithi090_104448_object169258:head# 2024-05-15T23:19:23.990 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1429e38:::benchmark_data_smithi090_104448_object161825:head# 2024-05-15T23:19:23.991 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d145c5a7:::benchmark_data_smithi090_104448_object160365:head# 2024-05-15T23:19:23.991 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d146563a:::benchmark_data_smithi090_104448_object165048:head# 2024-05-15T23:19:23.991 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1482457:::benchmark_data_smithi090_104448_object164923:head# 2024-05-15T23:19:23.991 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d14910b7:::benchmark_data_smithi090_104448_object157570:head# 2024-05-15T23:19:23.994 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d14c5e3f:::benchmark_data_smithi090_104448_object168817:head# 2024-05-15T23:19:23.994 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d14dbb44:::benchmark_data_smithi090_104448_object172364:head# 2024-05-15T23:19:23.994 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1526fad:::benchmark_data_smithi090_104448_object159713:head# 2024-05-15T23:19:23.994 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1588610:::benchmark_data_smithi090_104448_object170810:head# 2024-05-15T23:19:23.994 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d15a5c5a:::benchmark_data_smithi090_104448_object159916:head# 2024-05-15T23:19:23.995 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d15b8033:::benchmark_data_smithi090_104448_object160042:head# 2024-05-15T23:19:23.995 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d15d549d:::benchmark_data_smithi090_104448_object164987:head# 2024-05-15T23:19:23.995 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d168249a:::benchmark_data_smithi090_104448_object172581:head# 2024-05-15T23:19:23.995 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d16c5e0f:::benchmark_data_smithi090_104448_object170002:head# 2024-05-15T23:19:23.995 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d173044e:::benchmark_data_smithi090_104448_object167910:head# 2024-05-15T23:19:23.999 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d177f22b:::benchmark_data_smithi090_104448_object158736:head# 2024-05-15T23:19:23.999 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1799a91:::benchmark_data_smithi090_104448_object163736:head# 2024-05-15T23:19:23.999 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d17a6d3c:::benchmark_data_smithi090_104448_object163013:head# 2024-05-15T23:19:23.999 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d17fe946:::benchmark_data_smithi090_104448_object169494:head# 2024-05-15T23:19:23.999 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d18a36a2:::benchmark_data_smithi090_104448_object170166:head# 2024-05-15T23:19:23.999 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d18dfb42:::benchmark_data_smithi090_104448_object165395:head# 2024-05-15T23:19:24.000 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d18f78dc:::benchmark_data_smithi090_104448_object169138:head# 2024-05-15T23:19:24.000 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d193853f:::benchmark_data_smithi090_104448_object160989:head# 2024-05-15T23:19:24.000 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1949da2:::benchmark_data_smithi090_104448_object156666:head# 2024-05-15T23:19:24.000 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1964575:::benchmark_data_smithi090_104448_object162885:head# 2024-05-15T23:19:24.000 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1a2e87b:::benchmark_data_smithi090_104448_object170674:head# 2024-05-15T23:19:24.001 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1a615b9:::benchmark_data_smithi090_104448_object160170:head# 2024-05-15T23:19:24.001 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1ae37f4:::benchmark_data_smithi090_104448_object162720:head# 2024-05-15T23:19:24.002 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:19:24.005 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1aeb713:::benchmark_data_smithi090_104448_object167354:head# 2024-05-15T23:19:24.005 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1b08e72:::benchmark_data_smithi090_104448_object156828:head# 2024-05-15T23:19:24.005 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1b1cce3:::benchmark_data_smithi090_104448_object166176:head# 2024-05-15T23:19:24.006 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1b36775:::benchmark_data_smithi090_104448_object169969:head# 2024-05-15T23:19:24.006 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1b6f2c0:::benchmark_data_smithi090_104448_object160507:head# 2024-05-15T23:19:24.006 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1b9a9e4:::benchmark_data_smithi090_104448_object158400:head# 2024-05-15T23:19:24.006 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1c0345d:::benchmark_data_smithi090_104448_object172926:head# 2024-05-15T23:19:24.006 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1c1b097:::benchmark_data_smithi090_104448_object172759:head# 2024-05-15T23:19:24.006 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1c6b12a:::benchmark_data_smithi090_104448_object172406:head# 2024-05-15T23:19:24.007 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1d378c1:::benchmark_data_smithi090_104448_object165117:head# 2024-05-15T23:19:24.007 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1d9037d:::benchmark_data_smithi090_104448_object162268:head# 2024-05-15T23:19:24.007 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1d906ae:::benchmark_data_smithi090_104448_object161815:head# 2024-05-15T23:19:24.007 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1e2f73f:::benchmark_data_smithi090_104448_object167882:head# 2024-05-15T23:19:24.007 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1e73507:::benchmark_data_smithi090_104448_object160643:head# 2024-05-15T23:19:24.008 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1f072e5:::benchmark_data_smithi090_104448_object169053:head# 2024-05-15T23:19:24.008 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1f64215:::benchmark_data_smithi090_104448_object167965:head# 2024-05-15T23:19:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:24.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:19:24.014 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1fd2aca:::benchmark_data_smithi090_104448_object163957:head# 2024-05-15T23:19:24.015 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d1ff3578:::benchmark_data_smithi090_104448_object160341:head# 2024-05-15T23:19:24.015 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d200111a:::benchmark_data_smithi090_104448_object159497:head# 2024-05-15T23:19:24.015 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d20b3944:::benchmark_data_smithi090_104448_object162345:head# 2024-05-15T23:19:24.015 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d21212eb:::benchmark_data_smithi090_104448_object160049:head# 2024-05-15T23:19:24.015 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d21611a6:::benchmark_data_smithi090_104448_object157193:head# 2024-05-15T23:19:24.015 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d21621a9:::benchmark_data_smithi090_104448_object171828:head# 2024-05-15T23:19:24.016 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d218a703:::benchmark_data_smithi090_104448_object162960:head# 2024-05-15T23:19:24.016 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d21afba7:::benchmark_data_smithi090_104448_object160982:head# 2024-05-15T23:19:24.016 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d21c6eca:::benchmark_data_smithi090_104448_object162286:head# 2024-05-15T23:19:24.016 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d21e7538:::benchmark_data_smithi090_104448_object159684:head# 2024-05-15T23:19:24.016 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d21f7964:::benchmark_data_smithi090_104448_object169019:head# 2024-05-15T23:19:24.017 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d220f212:::benchmark_data_smithi090_104448_object159797:head# 2024-05-15T23:19:24.017 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2228c98:::benchmark_data_smithi090_104448_object161310:head# 2024-05-15T23:19:24.017 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d228b558:::benchmark_data_smithi090_104448_object160693:head# 2024-05-15T23:19:24.017 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2309af3:::benchmark_data_smithi090_104448_object171357:head# 2024-05-15T23:19:24.017 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2413c37:::benchmark_data_smithi090_104448_object161188:head# 2024-05-15T23:19:24.017 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d244b84a:::benchmark_data_smithi090_104448_object161136:head# 2024-05-15T23:19:24.018 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d24b6b92:::benchmark_data_smithi090_104448_object170445:head# 2024-05-15T23:19:24.018 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d24d49c2:::benchmark_data_smithi090_104448_object170949:head# 2024-05-15T23:19:24.018 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d24fea30:::benchmark_data_smithi090_104448_object171694:head# 2024-05-15T23:19:24.022 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2525bf2:::benchmark_data_smithi090_104448_object165267:head# 2024-05-15T23:19:24.022 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d25bffd4:::benchmark_data_smithi090_104448_object158249:head# 2024-05-15T23:19:24.022 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d25dfb9c:::benchmark_data_smithi090_104448_object158463:head# 2024-05-15T23:19:24.023 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d260a1a0:::benchmark_data_smithi090_104448_object164214:head# 2024-05-15T23:19:24.023 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d261216d:::benchmark_data_smithi090_104448_object162168:head# 2024-05-15T23:19:24.023 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d26496d3:::benchmark_data_smithi090_104448_object169301:head# 2024-05-15T23:19:24.023 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d269185b:::benchmark_data_smithi090_104448_object163562:head# 2024-05-15T23:19:24.023 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d26a2bd3:::benchmark_data_smithi090_104448_object160755:head# 2024-05-15T23:19:24.023 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d26ce10e:::benchmark_data_smithi090_104448_object172545:head# 2024-05-15T23:19:24.024 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d26dfd76:::benchmark_data_smithi090_104448_object163516:head# 2024-05-15T23:19:24.024 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d26fefad:::benchmark_data_smithi090_104448_object171295:head# 2024-05-15T23:19:24.024 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2718a37:::benchmark_data_smithi090_104448_object160104:head# 2024-05-15T23:19:24.024 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2719342:::benchmark_data_smithi090_104448_object163094:head# 2024-05-15T23:19:24.024 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d275b212:::benchmark_data_smithi090_104448_object157551:head# 2024-05-15T23:19:24.025 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d27763d2:::benchmark_data_smithi090_104448_object162493:head# 2024-05-15T23:19:24.025 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d279b3c6:::benchmark_data_smithi090_104448_object168209:head# 2024-05-15T23:19:24.025 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d27d69f2:::benchmark_data_smithi090_104448_object157837:head# 2024-05-15T23:19:24.029 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d281a8f0:::benchmark_data_smithi090_104448_object170317:head# 2024-05-15T23:19:24.029 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d28237c6:::benchmark_data_smithi090_104448_object161524:head# 2024-05-15T23:19:24.029 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d28552a4:::benchmark_data_smithi090_104448_object170968:head# 2024-05-15T23:19:24.029 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d28557ea:::benchmark_data_smithi090_104448_object165296:head# 2024-05-15T23:19:24.030 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d290b60d:::benchmark_data_smithi090_104448_object167612:head# 2024-05-15T23:19:24.030 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d292e10e:::benchmark_data_smithi090_104448_object168425:head# 2024-05-15T23:19:24.030 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d298df97:::benchmark_data_smithi090_104448_object169648:head# 2024-05-15T23:19:24.030 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d29c0f1f:::benchmark_data_smithi090_104448_object159476:head# 2024-05-15T23:19:24.030 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d29f7185:::benchmark_data_smithi090_104448_object164363:head# 2024-05-15T23:19:24.030 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2aacc6b:::benchmark_data_smithi090_104448_object167065:head# 2024-05-15T23:19:24.031 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2ab8a76:::benchmark_data_smithi090_104448_object168409:head# 2024-05-15T23:19:24.031 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2b80c75:::benchmark_data_smithi090_104448_object162606:head# 2024-05-15T23:19:24.031 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2bb4ae1:::benchmark_data_smithi090_104448_object164696:head# 2024-05-15T23:19:24.031 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2bc0b62:::benchmark_data_smithi090_104448_object159494:head# 2024-05-15T23:19:24.031 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2bd42d7:::benchmark_data_smithi090_104448_object158371:head# 2024-05-15T23:19:24.031 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2bd4ff5:::benchmark_data_smithi090_104448_object165219:head# 2024-05-15T23:19:24.032 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2bdee71:::benchmark_data_smithi090_104448_object159288:head# 2024-05-15T23:19:24.036 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2bf1140:::benchmark_data_smithi090_104448_object159534:head# 2024-05-15T23:19:24.036 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2bf1343:::benchmark_data_smithi090_104448_object172042:head# 2024-05-15T23:19:24.036 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2c177ac:::benchmark_data_smithi090_104448_object168611:head# 2024-05-15T23:19:24.036 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2c9b329:::benchmark_data_smithi090_104448_object164850:head# 2024-05-15T23:19:24.037 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2cd8aab:::benchmark_data_smithi090_104448_object171688:head# 2024-05-15T23:19:24.037 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2d3e00d:::benchmark_data_smithi090_104448_object165964:head# 2024-05-15T23:19:24.037 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2d56b7b:::benchmark_data_smithi090_104448_object169202:head# 2024-05-15T23:19:24.037 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2d88ee9:::benchmark_data_smithi090_104448_object168499:head# 2024-05-15T23:19:24.037 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2e1555f:::benchmark_data_smithi090_104448_object160634:head# 2024-05-15T23:19:24.037 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2e3918d:::benchmark_data_smithi090_104448_object172090:head# 2024-05-15T23:19:24.038 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2e4322b:::benchmark_data_smithi090_104448_object166653:head# 2024-05-15T23:19:24.038 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2e60e52:::benchmark_data_smithi090_104448_object165319:head# 2024-05-15T23:19:24.038 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2e80c33:::benchmark_data_smithi090_104448_object156642:head# 2024-05-15T23:19:24.038 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2ea56e7:::benchmark_data_smithi090_104448_object157781:head# 2024-05-15T23:19:24.038 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2ec8489:::benchmark_data_smithi090_104448_object158876:head# 2024-05-15T23:19:24.039 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2ed7724:::benchmark_data_smithi090_104448_object157906:head# 2024-05-15T23:19:24.039 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2f28ba0:::benchmark_data_smithi090_104448_object161749:head# 2024-05-15T23:19:24.039 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2f721c2:::benchmark_data_smithi090_104448_object157747:head# 2024-05-15T23:19:24.043 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2f8cbe1:::benchmark_data_smithi090_104448_object158229:head# 2024-05-15T23:19:24.043 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2f92d09:::benchmark_data_smithi090_104448_object169219:head# 2024-05-15T23:19:24.043 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2f98a7e:::benchmark_data_smithi090_104448_object164065:head# 2024-05-15T23:19:24.043 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d2ff9504:::benchmark_data_smithi090_104448_object166208:head# 2024-05-15T23:19:24.043 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d300f2f0:::benchmark_data_smithi090_104448_object167886:head# 2024-05-15T23:19:24.043 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d301c459:::benchmark_data_smithi090_104448_object163634:head# 2024-05-15T23:19:24.044 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3034e12:::benchmark_data_smithi090_104448_object156652:head# 2024-05-15T23:19:24.044 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3082d20:::benchmark_data_smithi090_104448_object156632:head# 2024-05-15T23:19:24.044 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d309c7e0:::benchmark_data_smithi090_104448_object167740:head# 2024-05-15T23:19:24.044 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d309d78d:::benchmark_data_smithi090_104448_object163152:head# 2024-05-15T23:19:24.044 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d30a45dd:::benchmark_data_smithi090_104448_object168727:head# 2024-05-15T23:19:24.045 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d30b5b9d:::benchmark_data_smithi090_104448_object170801:head# 2024-05-15T23:19:24.045 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d30ed7be:::benchmark_data_smithi090_104448_object167285:head# 2024-05-15T23:19:24.045 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d30fc751:::benchmark_data_smithi090_104448_object166955:head# 2024-05-15T23:19:24.045 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d310ef2a:::benchmark_data_smithi090_104448_object170954:head# 2024-05-15T23:19:24.045 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d31e3990:::benchmark_data_smithi090_104448_object169004:head# 2024-05-15T23:19:24.050 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d32164e7:::benchmark_data_smithi090_104448_object159973:head# 2024-05-15T23:19:24.050 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3243ad4:::benchmark_data_smithi090_104448_object161176:head# 2024-05-15T23:19:24.050 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d324ccb1:::benchmark_data_smithi090_104448_object165560:head# 2024-05-15T23:19:24.051 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3259749:::benchmark_data_smithi090_104448_object170822:head# 2024-05-15T23:19:24.051 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3276831:::benchmark_data_smithi090_104448_object170767:head# 2024-05-15T23:19:24.051 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d338960d:::benchmark_data_smithi090_104448_object168702:head# 2024-05-15T23:19:24.051 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d33e3598:::benchmark_data_smithi090_104448_object171530:head# 2024-05-15T23:19:24.051 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d33e61d6:::benchmark_data_smithi090_104448_object162904:head# 2024-05-15T23:19:24.051 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3431634:::benchmark_data_smithi090_104448_object171507:head# 2024-05-15T23:19:24.052 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3442165:::benchmark_data_smithi090_104448_object163041:head# 2024-05-15T23:19:24.052 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d357fe12:::benchmark_data_smithi090_104448_object169264:head# 2024-05-15T23:19:24.052 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d358b745:::benchmark_data_smithi090_104448_object162303:head# 2024-05-15T23:19:24.052 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d35c6874:::benchmark_data_smithi090_104448_object170138:head# 2024-05-15T23:19:24.052 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3699d13:::benchmark_data_smithi090_104448_object172286:head# 2024-05-15T23:19:24.052 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d36b575a:::benchmark_data_smithi090_104448_object158085:head# 2024-05-15T23:19:24.052 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d36ed135:::benchmark_data_smithi090_104448_object161795:head# 2024-05-15T23:19:24.052 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d371fc28:::benchmark_data_smithi090_104448_object172169:head# 2024-05-15T23:19:24.052 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3796d5c:::benchmark_data_smithi090_104448_object157200:head# 2024-05-15T23:19:24.054 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d37ceef4:::benchmark_data_smithi090_104448_object164350:head# 2024-05-15T23:19:24.054 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d37db4e4:::benchmark_data_smithi090_104448_object167891:head# 2024-05-15T23:19:24.054 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d38a8753:::benchmark_data_smithi090_104448_object167118:head# 2024-05-15T23:19:24.054 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d38c295b:::benchmark_data_smithi090_104448_object167919:head# 2024-05-15T23:19:24.054 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d38c5d80:::benchmark_data_smithi090_104448_object171848:head# 2024-05-15T23:19:24.054 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d38ca859:::benchmark_data_smithi090_104448_object158819:head# 2024-05-15T23:19:24.054 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d38dbe37:::benchmark_data_smithi090_104448_object158472:head# 2024-05-15T23:19:24.054 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d391b6a5:::benchmark_data_smithi090_104448_object164080:head# 2024-05-15T23:19:24.054 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3b1558f:::benchmark_data_smithi090_104448_object160311:head# 2024-05-15T23:19:24.055 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3b20104:::benchmark_data_smithi090_104448_object169804:head# 2024-05-15T23:19:24.055 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3b234bd:::benchmark_data_smithi090_104448_object160817:head# 2024-05-15T23:19:24.056 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3b32a50:::benchmark_data_smithi090_104448_object164828:head# 2024-05-15T23:19:24.056 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3b9bb12:::benchmark_data_smithi090_104448_object167773:head# 2024-05-15T23:19:24.056 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3bed8f1:::benchmark_data_smithi090_104448_object158911:head# 2024-05-15T23:19:24.057 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3cb90ce:::benchmark_data_smithi090_104448_object163919:head# 2024-05-15T23:19:24.057 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3cedf5d:::benchmark_data_smithi090_104448_object172780:head# 2024-05-15T23:19:24.057 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3d51da4:::benchmark_data_smithi090_104448_object171304:head# 2024-05-15T23:19:24.058 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3e33ed6:::benchmark_data_smithi090_104448_object163296:head# 2024-05-15T23:19:24.058 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3e34c0b:::benchmark_data_smithi090_104448_object157111:head# 2024-05-15T23:19:24.059 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3e41108:::benchmark_data_smithi090_104448_object171932:head# 2024-05-15T23:19:24.059 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3e615e8:::benchmark_data_smithi090_104448_object167238:head# 2024-05-15T23:19:24.060 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3e7b442:::benchmark_data_smithi090_104448_object168397:head# 2024-05-15T23:19:24.060 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3f83433:::benchmark_data_smithi090_104448_object168358:head# 2024-05-15T23:19:24.060 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d3f9c5fa:::benchmark_data_smithi090_104448_object171027:head# 2024-05-15T23:19:24.061 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4004e24:::benchmark_data_smithi090_104448_object160272:head# 2024-05-15T23:19:24.061 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d40780c8:::benchmark_data_smithi090_104448_object161361:head# 2024-05-15T23:19:24.062 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d40817b4:::benchmark_data_smithi090_104448_object172029:head# 2024-05-15T23:19:24.062 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d408db06:::benchmark_data_smithi090_104448_object160560:head# 2024-05-15T23:19:24.062 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d40e53b4:::benchmark_data_smithi090_104448_object164685:head# 2024-05-15T23:19:24.063 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d40f41ec:::benchmark_data_smithi090_104448_object163375:head# 2024-05-15T23:19:24.063 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4145a86:::benchmark_data_smithi090_104448_object170905:head# 2024-05-15T23:19:24.064 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d418a316:::benchmark_data_smithi090_104448_object162990:head# 2024-05-15T23:19:24.064 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d41a2c4d:::benchmark_data_smithi090_104448_object157119:head# 2024-05-15T23:19:24.064 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4231968:::benchmark_data_smithi090_104448_object162700:head# 2024-05-15T23:19:24.065 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d42443c8:::benchmark_data_smithi090_104448_object161541:head# 2024-05-15T23:19:24.065 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d42505f9:::benchmark_data_smithi090_104448_object162761:head# 2024-05-15T23:19:24.066 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4282efb:::benchmark_data_smithi090_104448_object169579:head# 2024-05-15T23:19:24.067 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d431da9b:::benchmark_data_smithi090_104448_object163778:head# 2024-05-15T23:19:24.067 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4343c1c:::benchmark_data_smithi090_104448_object167942:head# 2024-05-15T23:19:24.067 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d43c5c3f:::benchmark_data_smithi090_104448_object164986:head# 2024-05-15T23:19:24.068 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d43e206c:::benchmark_data_smithi090_104448_object159332:head# 2024-05-15T23:19:24.069 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d443091d:::benchmark_data_smithi090_104448_object158436:head# 2024-05-15T23:19:24.070 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4445142:::benchmark_data_smithi090_104448_object157501:head# 2024-05-15T23:19:24.070 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d445aa71:::benchmark_data_smithi090_104448_object158387:head# 2024-05-15T23:19:24.070 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d44aae27:::benchmark_data_smithi090_104448_object168336:head# 2024-05-15T23:19:24.070 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d44b2bef:::benchmark_data_smithi090_104448_object170489:head# 2024-05-15T23:19:24.071 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d44df32a:::benchmark_data_smithi090_104448_object160537:head# 2024-05-15T23:19:24.071 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d44ed37c:::benchmark_data_smithi090_104448_object158597:head# 2024-05-15T23:19:24.071 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d453e761:::benchmark_data_smithi090_104448_object169343:head# 2024-05-15T23:19:24.072 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4563ea6:::benchmark_data_smithi090_104448_object163555:head# 2024-05-15T23:19:24.072 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4568dc6:::benchmark_data_smithi090_104448_object163189:head# 2024-05-15T23:19:24.073 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d45a12cf:::benchmark_data_smithi090_104448_object160092:head# 2024-05-15T23:19:24.073 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d45aadc9:::benchmark_data_smithi090_104448_object164455:head# 2024-05-15T23:19:24.074 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d466cc48:::benchmark_data_smithi090_104448_object169587:head# 2024-05-15T23:19:24.074 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d46ba954:::benchmark_data_smithi090_104448_object158914:head# 2024-05-15T23:19:24.075 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d46d2716:::benchmark_data_smithi090_104448_object160519:head# 2024-05-15T23:19:24.075 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d46da3e6:::benchmark_data_smithi090_104448_object168201:head# 2024-05-15T23:19:24.076 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4715ad6:::benchmark_data_smithi090_104448_object169948:head# 2024-05-15T23:19:24.076 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d47c418e:::benchmark_data_smithi090_104448_object168061:head# 2024-05-15T23:19:24.076 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d47e6eee:::benchmark_data_smithi090_104448_object166349:head# 2024-05-15T23:19:24.077 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d48c6c5f:::benchmark_data_smithi090_104448_object166964:head# 2024-05-15T23:19:24.077 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d48da413:::benchmark_data_smithi090_104448_object162078:head# 2024-05-15T23:19:24.077 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4903b4d:::benchmark_data_smithi090_104448_object161811:head# 2024-05-15T23:19:24.078 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d490c099:::benchmark_data_smithi090_104448_object160355:head# 2024-05-15T23:19:24.078 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d499c830:::benchmark_data_smithi090_104448_object167302:head# 2024-05-15T23:19:24.079 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d49f2084:::benchmark_data_smithi090_104448_object161437:head# 2024-05-15T23:19:24.079 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4a79530:::benchmark_data_smithi090_104448_object157667:head# 2024-05-15T23:19:24.079 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4a99d83:::benchmark_data_smithi090_104448_object158564:head# 2024-05-15T23:19:24.080 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4aba819:::benchmark_data_smithi090_104448_object168218:head# 2024-05-15T23:19:24.080 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4af59dc:::benchmark_data_smithi090_104448_object157199:head# 2024-05-15T23:19:24.081 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4afdd65:::benchmark_data_smithi090_104448_object161101:head# 2024-05-15T23:19:24.081 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4b6860d:::benchmark_data_smithi090_104448_object166056:head# 2024-05-15T23:19:24.081 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4b78bec:::benchmark_data_smithi090_104448_object158413:head# 2024-05-15T23:19:24.082 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4ba8d8d:::benchmark_data_smithi090_104448_object172693:head# 2024-05-15T23:19:24.082 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4bd34b5:::benchmark_data_smithi090_104448_object172775:head# 2024-05-15T23:19:24.083 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4c00974:::benchmark_data_smithi090_104448_object161568:head# 2024-05-15T23:19:24.085 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4c03aa4:::benchmark_data_smithi090_104448_object160963:head# 2024-05-15T23:19:24.085 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4c61ef9:::benchmark_data_smithi090_104448_object158186:head# 2024-05-15T23:19:24.085 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4c87194:::benchmark_data_smithi090_104448_object168013:head# 2024-05-15T23:19:24.086 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4c89c21:::benchmark_data_smithi090_104448_object172448:head# 2024-05-15T23:19:24.086 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4ca6fff:::benchmark_data_smithi090_104448_object157527:head# 2024-05-15T23:19:24.086 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4ce78e3:::benchmark_data_smithi090_104448_object164026:head# 2024-05-15T23:19:24.087 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4d3e6da:::benchmark_data_smithi090_104448_object166077:head# 2024-05-15T23:19:24.087 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4d45449:::benchmark_data_smithi090_104448_object165171:head# 2024-05-15T23:19:24.088 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4dbd6ef:::benchmark_data_smithi090_104448_object163278:head# 2024-05-15T23:19:24.089 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4e388db:::benchmark_data_smithi090_104448_object172804:head# 2024-05-15T23:19:24.089 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4e7d6c4:::benchmark_data_smithi090_104448_object160413:head# 2024-05-15T23:19:24.089 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4ebfd5f:::benchmark_data_smithi090_104448_object172019:head# 2024-05-15T23:19:24.090 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4ec69b4:::benchmark_data_smithi090_104448_object164787:head# 2024-05-15T23:19:24.090 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4f4c89d:::benchmark_data_smithi090_104448_object169463:head# 2024-05-15T23:19:24.091 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4fc48aa:::benchmark_data_smithi090_104448_object167021:head# 2024-05-15T23:19:24.091 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d4fd13e5:::benchmark_data_smithi090_104448_object172863:head# 2024-05-15T23:19:24.092 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5005a30:::benchmark_data_smithi090_104448_object159081:head# 2024-05-15T23:19:24.092 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5016951:::benchmark_data_smithi090_104448_object166682:head# 2024-05-15T23:19:24.093 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5066628:::benchmark_data_smithi090_104448_object162103:head# 2024-05-15T23:19:24.093 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d508b0e0:::benchmark_data_smithi090_104448_object160201:head# 2024-05-15T23:19:24.093 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5091958:::benchmark_data_smithi090_104448_object159506:head# 2024-05-15T23:19:24.094 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d50ea3f1:::benchmark_data_smithi090_104448_object164254:head# 2024-05-15T23:19:24.094 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d50f67ac:::benchmark_data_smithi090_104448_object171261:head# 2024-05-15T23:19:24.094 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d512f4dc:::benchmark_data_smithi090_104448_object167158:head# 2024-05-15T23:19:24.095 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d515beab:::benchmark_data_smithi090_104448_object169671:head# 2024-05-15T23:19:24.095 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d518bb2b:::benchmark_data_smithi090_104448_object158031:head# 2024-05-15T23:19:24.096 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d51a6aaa:::benchmark_data_smithi090_104448_object157860:head# 2024-05-15T23:19:24.096 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d51cdc9e:::benchmark_data_smithi090_104448_object157092:head# 2024-05-15T23:19:24.097 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d52701ef:::benchmark_data_smithi090_104448_object167538:head# 2024-05-15T23:19:24.097 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5276139:::benchmark_data_smithi090_104448_object167431:head# 2024-05-15T23:19:24.098 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d52b675b:::benchmark_data_smithi090_104448_object161763:head# 2024-05-15T23:19:24.098 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d52fb296:::benchmark_data_smithi090_104448_object163257:head# 2024-05-15T23:19:24.099 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d533e1d3:::benchmark_data_smithi090_104448_object165791:head# 2024-05-15T23:19:24.099 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d534956c:::benchmark_data_smithi090_104448_object162744:head# 2024-05-15T23:19:24.099 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d537eb57:::benchmark_data_smithi090_104448_object164289:head# 2024-05-15T23:19:24.100 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d53cb61d:::benchmark_data_smithi090_104448_object160098:head# 2024-05-15T23:19:24.100 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d54039bb:::benchmark_data_smithi090_104448_object161154:head# 2024-05-15T23:19:24.101 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5494a14:::benchmark_data_smithi090_104448_object163590:head# 2024-05-15T23:19:24.101 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d54978e7:::benchmark_data_smithi090_104448_object162574:head# 2024-05-15T23:19:24.103 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5501512:::benchmark_data_smithi090_104448_object169839:head# 2024-05-15T23:19:24.103 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d550b236:::benchmark_data_smithi090_104448_object163928:head# 2024-05-15T23:19:24.103 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d554b191:::benchmark_data_smithi090_104448_object162131:head# 2024-05-15T23:19:24.104 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5577678:::benchmark_data_smithi090_104448_object165028:head# 2024-05-15T23:19:24.104 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d55bd617:::benchmark_data_smithi090_104448_object172322:head# 2024-05-15T23:19:24.105 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d55ccdc8:::benchmark_data_smithi090_104448_object164818:head# 2024-05-15T23:19:24.105 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d55d13c4:::benchmark_data_smithi090_104448_object162317:head# 2024-05-15T23:19:24.106 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d55eb2a3:::benchmark_data_smithi090_104448_object165098:head# 2024-05-15T23:19:24.106 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5643d26:::benchmark_data_smithi090_104448_object167033:head# 2024-05-15T23:19:24.107 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d56a672b:::benchmark_data_smithi090_104448_object161707:head# 2024-05-15T23:19:24.107 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d56c431b:::benchmark_data_smithi090_104448_object159406:head# 2024-05-15T23:19:24.107 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d570b156:::benchmark_data_smithi090_104448_object162580:head# 2024-05-15T23:19:24.108 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d570cb4e:::benchmark_data_smithi090_104448_object160249:head# 2024-05-15T23:19:24.108 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d576a357:::benchmark_data_smithi090_104448_object164025:head# 2024-05-15T23:19:24.109 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5770819:::benchmark_data_smithi090_104448_object159389:head# 2024-05-15T23:19:24.109 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d57c29da:::benchmark_data_smithi090_104448_object162581:head# 2024-05-15T23:19:24.110 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d57e7419:::benchmark_data_smithi090_104448_object168651:head# 2024-05-15T23:19:24.110 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d58000b4:::benchmark_data_smithi090_104448_object158175:head# 2024-05-15T23:19:24.111 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5883893:::benchmark_data_smithi090_104448_object169540:head# 2024-05-15T23:19:24.113 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:24.115 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d588aef7:::benchmark_data_smithi090_104448_object171619:head# 2024-05-15T23:19:24.115 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d58ed79a:::benchmark_data_smithi090_104448_object165711:head# 2024-05-15T23:19:24.115 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d58eebb6:::benchmark_data_smithi090_104448_object168909:head# 2024-05-15T23:19:24.115 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5929c92:::benchmark_data_smithi090_104448_object172374:head# 2024-05-15T23:19:24.115 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d594e0eb:::benchmark_data_smithi090_104448_object168883:head# 2024-05-15T23:19:24.115 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d59630c2:::benchmark_data_smithi090_104448_object163477:head# 2024-05-15T23:19:24.115 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d59cd130:::benchmark_data_smithi090_104448_object157198:head# 2024-05-15T23:19:24.116 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d59e8bd0:::benchmark_data_smithi090_104448_object171056:head# 2024-05-15T23:19:24.116 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5a2a206:::benchmark_data_smithi090_104448_object165477:head# 2024-05-15T23:19:24.116 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5a345d3:::benchmark_data_smithi090_104448_object170360:head# 2024-05-15T23:19:24.117 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5a47930:::benchmark_data_smithi090_104448_object157217:head# 2024-05-15T23:19:24.117 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5aa3b89:::benchmark_data_smithi090_104448_object163890:head# 2024-05-15T23:19:24.117 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5ace42c:::benchmark_data_smithi090_104448_object163091:head# 2024-05-15T23:19:24.117 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5b3320a:::benchmark_data_smithi090_104448_object171634:head# 2024-05-15T23:19:24.119 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5b784f1:::benchmark_data_smithi090_104448_object163766:head# 2024-05-15T23:19:24.119 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5ba50ab:::benchmark_data_smithi090_104448_object158477:head# 2024-05-15T23:19:24.119 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5bf13ba:::benchmark_data_smithi090_104448_object167885:head# 2024-05-15T23:19:24.119 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5c11c4c:::benchmark_data_smithi090_104448_object172537:head# 2024-05-15T23:19:24.119 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5c1913f:::benchmark_data_smithi090_104448_object169971:head# 2024-05-15T23:19:24.120 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5c2eb6a:::benchmark_data_smithi090_104448_object171260:head# 2024-05-15T23:19:24.120 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5c45007:::benchmark_data_smithi090_104448_object165049:head# 2024-05-15T23:19:24.121 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5c4a687:::benchmark_data_smithi090_104448_object162844:head# 2024-05-15T23:19:24.121 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5caf2d1:::benchmark_data_smithi090_104448_object165577:head# 2024-05-15T23:19:24.122 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5ce0a84:::benchmark_data_smithi090_104448_object164751:head# 2024-05-15T23:19:24.122 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5d27cdd:::benchmark_data_smithi090_104448_object167924:head# 2024-05-15T23:19:24.123 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5d4846a:::benchmark_data_smithi090_104448_object168596:head# 2024-05-15T23:19:24.123 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5d93a15:::benchmark_data_smithi090_104448_object158016:head# 2024-05-15T23:19:24.124 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5dd1419:::benchmark_data_smithi090_104448_object171168:head# 2024-05-15T23:19:24.124 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5ddc952:::benchmark_data_smithi090_104448_object161679:head# 2024-05-15T23:19:24.125 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5dddd42:::benchmark_data_smithi090_104448_object164778:head# 2024-05-15T23:19:24.125 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5e02329:::benchmark_data_smithi090_104448_object159837:head# 2024-05-15T23:19:24.125 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5e63e09:::benchmark_data_smithi090_104448_object168782:head# 2024-05-15T23:19:24.126 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5e79d50:::benchmark_data_smithi090_104448_object166553:head# 2024-05-15T23:19:24.126 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5f042ca:::benchmark_data_smithi090_104448_object169720:head# 2024-05-15T23:19:24.127 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5fd888c:::benchmark_data_smithi090_104448_object157589:head# 2024-05-15T23:19:24.127 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d5ff305b:::benchmark_data_smithi090_104448_object167254:head# 2024-05-15T23:19:24.128 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d601bf82:::benchmark_data_smithi090_104448_object158577:head# 2024-05-15T23:19:24.128 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6069d26:::benchmark_data_smithi090_104448_object170990:head# 2024-05-15T23:19:24.129 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d608dabb:::benchmark_data_smithi090_104448_object159161:head# 2024-05-15T23:19:24.129 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d60cfcf9:::benchmark_data_smithi090_104448_object162088:head# 2024-05-15T23:19:24.130 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d612d9b6:::benchmark_data_smithi090_104448_object157156:head# 2024-05-15T23:19:24.130 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6134c92:::benchmark_data_smithi090_104448_object165320:head# 2024-05-15T23:19:24.130 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d61359d5:::benchmark_data_smithi090_104448_object167339:head# 2024-05-15T23:19:24.131 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d61a1a06:::benchmark_data_smithi090_104448_object157104:head# 2024-05-15T23:19:24.131 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d61e6dfd:::benchmark_data_smithi090_104448_object158098:head# 2024-05-15T23:19:24.132 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d620ff59:::benchmark_data_smithi090_104448_object161381:head# 2024-05-15T23:19:24.132 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d62a0338:::benchmark_data_smithi090_104448_object170742:head# 2024-05-15T23:19:24.133 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d62ab3af:::benchmark_data_smithi090_104448_object171550:head# 2024-05-15T23:19:24.133 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6323154:::benchmark_data_smithi090_104448_object160142:head# 2024-05-15T23:19:24.133 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6347525:::benchmark_data_smithi090_104448_object165862:head# 2024-05-15T23:19:24.134 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d634ab47:::benchmark_data_smithi090_104448_object159339:head# 2024-05-15T23:19:24.134 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d63e9182:::benchmark_data_smithi090_104448_object169514:head# 2024-05-15T23:19:24.135 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d648f14b:::benchmark_data_smithi090_104448_object169537:head# 2024-05-15T23:19:24.135 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d64cb720:::benchmark_data_smithi090_104448_object172051:head# 2024-05-15T23:19:24.136 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6537421:::benchmark_data_smithi090_104448_object173171:head# 2024-05-15T23:19:24.136 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d65418ce:::benchmark_data_smithi090_104448_object165069:head# 2024-05-15T23:19:24.136 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d65ae19d:::benchmark_data_smithi090_104448_object160542:head# 2024-05-15T23:19:24.137 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d65e97e4:::benchmark_data_smithi090_104448_object164449:head# 2024-05-15T23:19:24.137 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6717ff5:::benchmark_data_smithi090_104448_object159738:head# 2024-05-15T23:19:24.138 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d677c2c7:::benchmark_data_smithi090_104448_object173023:head# 2024-05-15T23:19:24.138 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d67a4a38:::benchmark_data_smithi090_104448_object164961:head# 2024-05-15T23:19:24.139 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6817035:::benchmark_data_smithi090_104448_object162804:head# 2024-05-15T23:19:24.139 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6819305:::benchmark_data_smithi090_104448_object164186:head# 2024-05-15T23:19:24.139 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d686ff31:::benchmark_data_smithi090_104448_object158656:head# 2024-05-15T23:19:24.140 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d687fb03:::benchmark_data_smithi090_104448_object163456:head# 2024-05-15T23:19:24.140 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6919a96:::benchmark_data_smithi090_104448_object157755:head# 2024-05-15T23:19:24.141 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d694bd68:::benchmark_data_smithi090_104448_object168451:head# 2024-05-15T23:19:24.141 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d697064d:::benchmark_data_smithi090_104448_object163615:head# 2024-05-15T23:19:24.141 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d69db45a:::benchmark_data_smithi090_104448_object172179:head# 2024-05-15T23:19:24.142 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d69dce88:::benchmark_data_smithi090_104448_object166678:head# 2024-05-15T23:19:24.142 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d69e4bde:::benchmark_data_smithi090_104448_object163301:head# 2024-05-15T23:19:24.143 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:24.143 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6a60284:::benchmark_data_smithi090_104448_object171247:head# 2024-05-15T23:19:24.143 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6aacc44:::benchmark_data_smithi090_104448_object157686:head# 2024-05-15T23:19:24.144 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6b02f9e:::benchmark_data_smithi090_104448_object160497:head# 2024-05-15T23:19:24.144 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6b95976:::benchmark_data_smithi090_104448_object163974:head# 2024-05-15T23:19:24.145 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6b9d407:::benchmark_data_smithi090_104448_object161010:head# 2024-05-15T23:19:24.145 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6ba2e5d:::benchmark_data_smithi090_104448_object163528:head# 2024-05-15T23:19:24.145 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6bdb41d:::benchmark_data_smithi090_104448_object171714:head# 2024-05-15T23:19:24.146 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6be6697:::benchmark_data_smithi090_104448_object167166:head# 2024-05-15T23:19:24.146 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6c50d6c:::benchmark_data_smithi090_104448_object159715:head# 2024-05-15T23:19:24.147 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6c581c4:::benchmark_data_smithi090_104448_object166121:head# 2024-05-15T23:19:24.147 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6cbbcec:::benchmark_data_smithi090_104448_object165184:head# 2024-05-15T23:19:24.147 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6ccee1a:::benchmark_data_smithi090_104448_object161676:head# 2024-05-15T23:19:24.148 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6d3ba65:::benchmark_data_smithi090_104448_object164664:head# 2024-05-15T23:19:24.148 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6d51fce:::benchmark_data_smithi090_104448_object164669:head# 2024-05-15T23:19:24.148 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6d6ea98:::benchmark_data_smithi090_104448_object172089:head# 2024-05-15T23:19:24.149 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6e5f51d:::benchmark_data_smithi090_104448_object159195:head# 2024-05-15T23:19:24.149 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6e7ed25:::benchmark_data_smithi090_104448_object171959:head# 2024-05-15T23:19:24.150 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6ed852b:::benchmark_data_smithi090_104448_object172196:head# 2024-05-15T23:19:24.150 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6efcc7b:::benchmark_data_smithi090_104448_object160106:head# 2024-05-15T23:19:24.151 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6f1f87a:::benchmark_data_smithi090_104448_object172622:head# 2024-05-15T23:19:24.151 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6f9f523:::benchmark_data_smithi090_104448_object160435:head# 2024-05-15T23:19:24.151 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6fd2308:::benchmark_data_smithi090_104448_object167903:head# 2024-05-15T23:19:24.152 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6fdefaa:::benchmark_data_smithi090_104448_object162139:head# 2024-05-15T23:19:24.152 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d6fefbb2:::benchmark_data_smithi090_104448_object160242:head# 2024-05-15T23:19:24.153 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d703f043:::benchmark_data_smithi090_104448_object162857:head# 2024-05-15T23:19:24.153 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d707081e:::benchmark_data_smithi090_104448_object164235:head# 2024-05-15T23:19:24.153 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d70d736d:::benchmark_data_smithi090_104448_object168719:head# 2024-05-15T23:19:24.154 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d70ec930:::benchmark_data_smithi090_104448_object169626:head# 2024-05-15T23:19:24.154 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d710fc33:::benchmark_data_smithi090_104448_object162976:head# 2024-05-15T23:19:24.155 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d718e4ec:::benchmark_data_smithi090_104448_object162699:head# 2024-05-15T23:19:24.155 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d719ace3:::benchmark_data_smithi090_104448_object167941:head# 2024-05-15T23:19:24.156 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d71baf99:::benchmark_data_smithi090_104448_object168182:head# 2024-05-15T23:19:24.156 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7226626:::benchmark_data_smithi090_104448_object169472:head# 2024-05-15T23:19:24.157 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d723ab17:::benchmark_data_smithi090_104448_object159550:head# 2024-05-15T23:19:24.157 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7263470:::benchmark_data_smithi090_104448_object158012:head# 2024-05-15T23:19:24.157 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d72b0451:::benchmark_data_smithi090_104448_object165545:head# 2024-05-15T23:19:24.158 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7354e7c:::benchmark_data_smithi090_104448_object171589:head# 2024-05-15T23:19:24.159 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7385574:::benchmark_data_smithi090_104448_object160765:head# 2024-05-15T23:19:24.159 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d739420c:::benchmark_data_smithi090_104448_object168588:head# 2024-05-15T23:19:24.159 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d73e9683:::benchmark_data_smithi090_104448_object160312:head# 2024-05-15T23:19:24.160 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d744d8ab:::benchmark_data_smithi090_104448_object163417:head# 2024-05-15T23:19:24.160 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d74b3c2b:::benchmark_data_smithi090_104448_object168373:head# 2024-05-15T23:19:24.161 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d74b7d43:::benchmark_data_smithi090_104448_object171296:head# 2024-05-15T23:19:24.161 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d74e2d71:::benchmark_data_smithi090_104448_object164798:head# 2024-05-15T23:19:24.161 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d75073d2:::benchmark_data_smithi090_104448_object172285:head# 2024-05-15T23:19:24.162 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d75609ae:::benchmark_data_smithi090_104448_object173161:head# 2024-05-15T23:19:24.162 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d75751dd:::benchmark_data_smithi090_104448_object167460:head# 2024-05-15T23:19:24.162 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d75a4498:::benchmark_data_smithi090_104448_object165819:head# 2024-05-15T23:19:24.163 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d75d4a7d:::benchmark_data_smithi090_104448_object171862:head# 2024-05-15T23:19:24.163 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7624448:::benchmark_data_smithi090_104448_object164951:head# 2024-05-15T23:19:24.164 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d766200b:::benchmark_data_smithi090_104448_object170839:head# 2024-05-15T23:19:24.164 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7672bfa:::benchmark_data_smithi090_104448_object169139:head# 2024-05-15T23:19:24.165 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d768189e:::benchmark_data_smithi090_104448_object172338:head# 2024-05-15T23:19:24.165 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7693660:::benchmark_data_smithi090_104448_object162523:head# 2024-05-15T23:19:24.166 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d76fb235:::benchmark_data_smithi090_104448_object171900:head# 2024-05-15T23:19:24.166 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d770a9b2:::benchmark_data_smithi090_104448_object161616:head# 2024-05-15T23:19:24.167 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d770d899:::benchmark_data_smithi090_104448_object169838:head# 2024-05-15T23:19:24.167 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7757421:::benchmark_data_smithi090_104448_object157051:head# 2024-05-15T23:19:24.168 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7840993:::benchmark_data_smithi090_104448_object167599:head# 2024-05-15T23:19:24.168 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7922f38:::benchmark_data_smithi090_104448_object172578:head# 2024-05-15T23:19:24.169 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d794c8c9:::benchmark_data_smithi090_104448_object165600:head# 2024-05-15T23:19:24.169 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7996a7b:::benchmark_data_smithi090_104448_object167433:head# 2024-05-15T23:19:24.169 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7a0adcc:::benchmark_data_smithi090_104448_object172838:head# 2024-05-15T23:19:24.170 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7a12619:::benchmark_data_smithi090_104448_object158921:head# 2024-05-15T23:19:24.170 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7a28c1a:::benchmark_data_smithi090_104448_object161564:head# 2024-05-15T23:19:24.170 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7a6732d:::benchmark_data_smithi090_104448_object158192:head# 2024-05-15T23:19:24.171 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7ab2da1:::benchmark_data_smithi090_104448_object160786:head# 2024-05-15T23:19:24.171 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7ac251f:::benchmark_data_smithi090_104448_object172341:head# 2024-05-15T23:19:24.171 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7b087b7:::benchmark_data_smithi090_104448_object160953:head# 2024-05-15T23:19:24.172 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7b79145:::benchmark_data_smithi090_104448_object167685:head# 2024-05-15T23:19:24.172 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7b944f2:::benchmark_data_smithi090_104448_object171096:head# 2024-05-15T23:19:24.172 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7bd6311:::benchmark_data_smithi090_104448_object160400:head# 2024-05-15T23:19:24.173 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7c35047:::benchmark_data_smithi090_104448_object164089:head# 2024-05-15T23:19:24.174 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7c35a59:::benchmark_data_smithi090_104448_object165763:head# 2024-05-15T23:19:24.174 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7c4f4d4:::benchmark_data_smithi090_104448_object170676:head# 2024-05-15T23:19:24.174 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7cb84c3:::benchmark_data_smithi090_104448_object157145:head# 2024-05-15T23:19:24.175 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7cd2c11:::benchmark_data_smithi090_104448_object161842:head# 2024-05-15T23:19:24.175 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7d00295:::benchmark_data_smithi090_104448_object159449:head# 2024-05-15T23:19:24.176 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7d55181:::benchmark_data_smithi090_104448_object162143:head# 2024-05-15T23:19:24.176 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7dce0cb:::benchmark_data_smithi090_104448_object172382:head# 2024-05-15T23:19:24.177 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7def6e1:::benchmark_data_smithi090_104448_object171439:head# 2024-05-15T23:19:24.177 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7e6a26a:::benchmark_data_smithi090_104448_object156737:head# 2024-05-15T23:19:24.177 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7eb1759:::benchmark_data_smithi090_104448_object170276:head# 2024-05-15T23:19:24.178 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7eb9639:::benchmark_data_smithi090_104448_object164801:head# 2024-05-15T23:19:24.179 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7f20691:::benchmark_data_smithi090_104448_object156675:head# 2024-05-15T23:19:24.179 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7f68dec:::benchmark_data_smithi090_104448_object170707:head# 2024-05-15T23:19:24.180 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7fcdd89:::benchmark_data_smithi090_104448_object173061:head# 2024-05-15T23:19:24.180 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7fdba6a:::benchmark_data_smithi090_104448_object167428:head# 2024-05-15T23:19:24.181 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d7fffa6f:::benchmark_data_smithi090_104448_object156963:head# 2024-05-15T23:19:24.181 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d809c21c:::benchmark_data_smithi090_104448_object160513:head# 2024-05-15T23:19:24.181 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d809f21b:::benchmark_data_smithi090_104448_object157623:head# 2024-05-15T23:19:24.182 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d80d3ea0:::benchmark_data_smithi090_104448_object159768:head# 2024-05-15T23:19:24.182 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d80d40db:::benchmark_data_smithi090_104448_object170120:head# 2024-05-15T23:19:24.182 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d80f0737:::benchmark_data_smithi090_104448_object167665:head# 2024-05-15T23:19:24.183 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d80f3c16:::benchmark_data_smithi090_104448_object163993:head# 2024-05-15T23:19:24.183 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d812890d:::benchmark_data_smithi090_104448_object164044:head# 2024-05-15T23:19:24.184 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d812e65c:::benchmark_data_smithi090_104448_object157244:head# 2024-05-15T23:19:24.184 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8133ef0:::benchmark_data_smithi090_104448_object159204:head# 2024-05-15T23:19:24.184 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d81a0b19:::benchmark_data_smithi090_104448_object165357:head# 2024-05-15T23:19:24.185 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d81a4b90:::benchmark_data_smithi090_104448_object163198:head# 2024-05-15T23:19:24.186 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d81c64e4:::benchmark_data_smithi090_104448_object170693:head# 2024-05-15T23:19:24.186 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d81e5082:::benchmark_data_smithi090_104448_object164999:head# 2024-05-15T23:19:24.187 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d81fc717:::benchmark_data_smithi090_104448_object161622:head# 2024-05-15T23:19:24.187 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d827cc56:::benchmark_data_smithi090_104448_object160838:head# 2024-05-15T23:19:24.187 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d82c3cff:::benchmark_data_smithi090_104448_object161086:head# 2024-05-15T23:19:24.188 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d832b245:::benchmark_data_smithi090_104448_object166843:head# 2024-05-15T23:19:24.188 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d833e706:::benchmark_data_smithi090_104448_object158285:head# 2024-05-15T23:19:24.189 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d837ce25:::benchmark_data_smithi090_104448_object159920:head# 2024-05-15T23:19:24.189 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d83a3802:::benchmark_data_smithi090_104448_object169254:head# 2024-05-15T23:19:24.189 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d83b3279:::benchmark_data_smithi090_104448_object166695:head# 2024-05-15T23:19:24.190 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d83e752d:::benchmark_data_smithi090_104448_object156813:head# 2024-05-15T23:19:24.190 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d83f74db:::benchmark_data_smithi090_104448_object166152:head# 2024-05-15T23:19:24.191 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8483ffa:::benchmark_data_smithi090_104448_object157395:head# 2024-05-15T23:19:24.191 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8557fea:::benchmark_data_smithi090_104448_object158788:head# 2024-05-15T23:19:24.192 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d861a787:::benchmark_data_smithi090_104448_object170333:head# 2024-05-15T23:19:24.192 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8650ef7:::benchmark_data_smithi090_104448_object173218:head# 2024-05-15T23:19:24.192 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d866bcb7:::benchmark_data_smithi090_104448_object163412:head# 2024-05-15T23:19:24.193 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d86b782d:::benchmark_data_smithi090_104448_object158347:head# 2024-05-15T23:19:24.195 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d86ccbc5:::benchmark_data_smithi090_104448_object159469:head# 2024-05-15T23:19:24.195 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8714e80:::benchmark_data_smithi090_104448_object167103:head# 2024-05-15T23:19:24.195 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d871b731:::benchmark_data_smithi090_104448_object169489:head# 2024-05-15T23:19:24.195 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d872ed9e:::benchmark_data_smithi090_104448_object162166:head# 2024-05-15T23:19:24.195 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d873572a:::benchmark_data_smithi090_104448_object161467:head# 2024-05-15T23:19:24.196 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8778e15:::benchmark_data_smithi090_104448_object168224:head# 2024-05-15T23:19:24.196 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8781282:::benchmark_data_smithi090_104448_object159181:head# 2024-05-15T23:19:24.196 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d87e2eb6:::benchmark_data_smithi090_104448_object166880:head# 2024-05-15T23:19:24.197 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d88262cc:::benchmark_data_smithi090_104448_object170683:head# 2024-05-15T23:19:24.197 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d88ca399:::benchmark_data_smithi090_104448_object169761:head# 2024-05-15T23:19:24.198 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d88f1433:::benchmark_data_smithi090_104448_object160836:head# 2024-05-15T23:19:24.198 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d893a2db:::benchmark_data_smithi090_104448_object168210:head# 2024-05-15T23:19:24.199 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8952069:::benchmark_data_smithi090_104448_object170897:head# 2024-05-15T23:19:24.199 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8955541:::benchmark_data_smithi090_104448_object170967:head# 2024-05-15T23:19:24.200 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8a06b1c:::benchmark_data_smithi090_104448_object171413:head# 2024-05-15T23:19:24.200 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8a55007:::benchmark_data_smithi090_104448_object156726:head# 2024-05-15T23:19:24.201 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8a83563:::benchmark_data_smithi090_104448_object166692:head# 2024-05-15T23:19:24.201 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8a9aa6d:::benchmark_data_smithi090_104448_object170852:head# 2024-05-15T23:19:24.201 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8aae4a8:::benchmark_data_smithi090_104448_object171464:head# 2024-05-15T23:19:24.202 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8b64453:::benchmark_data_smithi090_104448_object159034:head# 2024-05-15T23:19:24.202 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8b88f81:::benchmark_data_smithi090_104448_object167277:head# 2024-05-15T23:19:24.203 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8beec0c:::benchmark_data_smithi090_104448_object161382:head# 2024-05-15T23:19:24.203 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8c2cb2e:::benchmark_data_smithi090_104448_object165808:head# 2024-05-15T23:19:24.204 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8c34a9c:::benchmark_data_smithi090_104448_object170697:head# 2024-05-15T23:19:24.204 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8c7ff8c:::benchmark_data_smithi090_104448_object167300:head# 2024-05-15T23:19:24.205 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8c88e88:::benchmark_data_smithi090_104448_object169361:head# 2024-05-15T23:19:24.205 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8cc952d:::benchmark_data_smithi090_104448_object170834:head# 2024-05-15T23:19:24.205 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8d061b5:::benchmark_data_smithi090_104448_object169347:head# 2024-05-15T23:19:24.206 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8d2e1e8:::benchmark_data_smithi090_104448_object169880:head# 2024-05-15T23:19:24.207 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8db0e73:::benchmark_data_smithi090_104448_object167157:head# 2024-05-15T23:19:24.207 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8db62ee:::benchmark_data_smithi090_104448_object163328:head# 2024-05-15T23:19:24.207 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8dc8dca:::benchmark_data_smithi090_104448_object167383:head# 2024-05-15T23:19:24.208 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8def457:::benchmark_data_smithi090_104448_object169306:head# 2024-05-15T23:19:24.208 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8e72a56:::benchmark_data_smithi090_104448_object168536:head# 2024-05-15T23:19:24.209 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8e8c722:::benchmark_data_smithi090_104448_object157800:head# 2024-05-15T23:19:24.209 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8ed46ae:::benchmark_data_smithi090_104448_object161639:head# 2024-05-15T23:19:24.209 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8ee2159:::benchmark_data_smithi090_104448_object165837:head# 2024-05-15T23:19:24.210 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8f3166b:::benchmark_data_smithi090_104448_object163256:head# 2024-05-15T23:19:24.210 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d8ff7676:::benchmark_data_smithi090_104448_object164258:head# 2024-05-15T23:19:24.210 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d900d8e0:::benchmark_data_smithi090_104448_object166438:head# 2024-05-15T23:19:24.211 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d904bccf:::benchmark_data_smithi090_104448_object163233:head# 2024-05-15T23:19:24.211 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9082322:::benchmark_data_smithi090_104448_object162478:head# 2024-05-15T23:19:24.211 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9088071:::benchmark_data_smithi090_104448_object172056:head# 2024-05-15T23:19:24.212 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d90949df:::benchmark_data_smithi090_104448_object158805:head# 2024-05-15T23:19:24.212 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d90956c0:::benchmark_data_smithi090_104448_object163154:head# 2024-05-15T23:19:24.213 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d911ad89:::benchmark_data_smithi090_104448_object170295:head# 2024-05-15T23:19:24.214 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9143dc7:::benchmark_data_smithi090_104448_object157977:head# 2024-05-15T23:19:24.214 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d915305f:::benchmark_data_smithi090_104448_object162209:head# 2024-05-15T23:19:24.214 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d91a2e64:::benchmark_data_smithi090_104448_object157834:head# 2024-05-15T23:19:24.214 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d91be9af:::benchmark_data_smithi090_104448_object167092:head# 2024-05-15T23:19:24.215 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d91e46cd:::benchmark_data_smithi090_104448_object168236:head# 2024-05-15T23:19:24.215 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d91ee1ea:::benchmark_data_smithi090_104448_object159233:head# 2024-05-15T23:19:24.216 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d92170cb:::benchmark_data_smithi090_104448_object165562:head# 2024-05-15T23:19:24.218 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9232673:::benchmark_data_smithi090_104448_object162125:head# 2024-05-15T23:19:24.218 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d923330d:::benchmark_data_smithi090_104448_object164353:head# 2024-05-15T23:19:24.218 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9245f93:::benchmark_data_smithi090_104448_object172982:head# 2024-05-15T23:19:24.220 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d92d626a:::benchmark_data_smithi090_104448_object173211:head# 2024-05-15T23:19:24.220 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d932fade:::benchmark_data_smithi090_104448_object168776:head# 2024-05-15T23:19:24.220 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d935ddb6:::benchmark_data_smithi090_104448_object158671:head# 2024-05-15T23:19:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d938fbb6:::benchmark_data_smithi090_104448_object164638:head# 2024-05-15T23:19:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d93a75f7:::benchmark_data_smithi090_104448_object171795:head# 2024-05-15T23:19:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d93edd9b:::benchmark_data_smithi090_104448_object161254:head# 2024-05-15T23:19:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d93fab08:::benchmark_data_smithi090_104448_object170862:head# 2024-05-15T23:19:24.222 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9456850:::benchmark_data_smithi090_104448_object164205:head# 2024-05-15T23:19:24.222 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d948f307:::benchmark_data_smithi090_104448_object166415:head# 2024-05-15T23:19:24.223 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d95750d4:::benchmark_data_smithi090_104448_object171838:head# 2024-05-15T23:19:24.223 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d96d00d3:::benchmark_data_smithi090_104448_object164219:head# 2024-05-15T23:19:24.224 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d96d0973:::benchmark_data_smithi090_104448_object171870:head# 2024-05-15T23:19:24.224 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d96ea22a:::benchmark_data_smithi090_104448_object159509:head# 2024-05-15T23:19:24.225 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d96ec0ae:::benchmark_data_smithi090_104448_object169569:head# 2024-05-15T23:19:24.225 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d980b4fb:::benchmark_data_smithi090_104448_object167196:head# 2024-05-15T23:19:24.226 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9901deb:::benchmark_data_smithi090_104448_object168212:head# 2024-05-15T23:19:24.226 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d99152d7:::benchmark_data_smithi090_104448_object161691:head# 2024-05-15T23:19:24.226 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d99d714e:::benchmark_data_smithi090_104448_object162319:head# 2024-05-15T23:19:24.227 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9a1768e:::benchmark_data_smithi090_104448_object167039:head# 2024-05-15T23:19:24.228 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9a1ff29:::benchmark_data_smithi090_104448_object160278:head# 2024-05-15T23:19:24.228 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9a45238:::benchmark_data_smithi090_104448_object171376:head# 2024-05-15T23:19:24.228 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9a6ecf0:::benchmark_data_smithi090_104448_object165817:head# 2024-05-15T23:19:24.229 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9af4333:::benchmark_data_smithi090_104448_object164815:head# 2024-05-15T23:19:24.229 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9b1d7a3:::benchmark_data_smithi090_104448_object163655:head# 2024-05-15T23:19:24.229 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9b35de0:::benchmark_data_smithi090_104448_object166631:head# 2024-05-15T23:19:24.230 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9bcaac9:::benchmark_data_smithi090_104448_object161794:head# 2024-05-15T23:19:24.230 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9bf8e42:::benchmark_data_smithi090_104448_object158097:head# 2024-05-15T23:19:24.231 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9c3af7f:::benchmark_data_smithi090_104448_object164468:head# 2024-05-15T23:19:24.231 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9c4f976:::benchmark_data_smithi090_104448_object162323:head# 2024-05-15T23:19:24.231 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9c8740d:::benchmark_data_smithi090_104448_object159183:head# 2024-05-15T23:19:24.232 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9ce7c73:::benchmark_data_smithi090_104448_object164994:head# 2024-05-15T23:19:24.232 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9cfcdc0:::benchmark_data_smithi090_104448_object167446:head# 2024-05-15T23:19:24.233 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9d742d1:::benchmark_data_smithi090_104448_object167221:head# 2024-05-15T23:19:24.233 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9d7d7fb:::benchmark_data_smithi090_104448_object161909:head# 2024-05-15T23:19:24.234 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9da44eb:::benchmark_data_smithi090_104448_object164249:head# 2024-05-15T23:19:24.234 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9e27484:::benchmark_data_smithi090_104448_object161725:head# 2024-05-15T23:19:24.234 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9e30ca6:::benchmark_data_smithi090_104448_object172012:head# 2024-05-15T23:19:24.235 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9e4c4d0:::benchmark_data_smithi090_104448_object157808:head# 2024-05-15T23:19:24.235 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9e57835:::benchmark_data_smithi090_104448_object157502:head# 2024-05-15T23:19:24.236 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9ed92d4:::benchmark_data_smithi090_104448_object157116:head# 2024-05-15T23:19:24.236 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9edeacb:::benchmark_data_smithi090_104448_object161778:head# 2024-05-15T23:19:24.237 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9ef1d91:::benchmark_data_smithi090_104448_object171496:head# 2024-05-15T23:19:24.237 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9f7ff47:::benchmark_data_smithi090_104448_object164463:head# 2024-05-15T23:19:24.238 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9f99b11:::benchmark_data_smithi090_104448_object171546:head# 2024-05-15T23:19:24.238 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:d9fc37ab:::benchmark_data_smithi090_104448_object160159:head# 2024-05-15T23:19:24.239 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da0e460b:::benchmark_data_smithi090_104448_object169164:head# 2024-05-15T23:19:24.239 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da1294b4:::benchmark_data_smithi090_104448_object159887:head# 2024-05-15T23:19:24.240 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da1ca531:::benchmark_data_smithi090_104448_object172278:head# 2024-05-15T23:19:24.240 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da1cbb7d:::benchmark_data_smithi090_104448_object170509:head# 2024-05-15T23:19:24.240 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:24.241 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da283037:::benchmark_data_smithi090_104448_object159091:head# 2024-05-15T23:19:24.241 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da28ec05:::benchmark_data_smithi090_104448_object162126:head# 2024-05-15T23:19:24.241 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da298127:::benchmark_data_smithi090_104448_object164675:head# 2024-05-15T23:19:24.242 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da2d183c:::benchmark_data_smithi090_104448_object170426:head# 2024-05-15T23:19:24.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:24.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:24.243 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da2dd8b1:::benchmark_data_smithi090_104448_object158701:head# 2024-05-15T23:19:24.244 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da2de93f:::benchmark_data_smithi090_104448_object169689:head# 2024-05-15T23:19:24.244 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da2dfaac:::benchmark_data_smithi090_104448_object162109:head# 2024-05-15T23:19:24.244 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da30f6e8:::benchmark_data_smithi090_104448_object163820:head# 2024-05-15T23:19:24.244 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da30fa70:::benchmark_data_smithi090_104448_object167537:head# 2024-05-15T23:19:24.244 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da36eea9:::benchmark_data_smithi090_104448_object168963:head# 2024-05-15T23:19:24.245 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da3bc558:::benchmark_data_smithi090_104448_object157376:head# 2024-05-15T23:19:24.245 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da3e00e1:::benchmark_data_smithi090_104448_object168478:head# 2024-05-15T23:19:24.246 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da3f85be:::benchmark_data_smithi090_104448_object172000:head# 2024-05-15T23:19:24.246 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da42f909:::benchmark_data_smithi090_104448_object162996:head# 2024-05-15T23:19:24.247 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da4710a0:::benchmark_data_smithi090_104448_object161043:head# 2024-05-15T23:19:24.247 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da47e66e:::benchmark_data_smithi090_104448_object167274:head# 2024-05-15T23:19:24.248 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da4be431:::benchmark_data_smithi090_104448_object169563:head# 2024-05-15T23:19:24.248 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da4c4a70:::benchmark_data_smithi090_104448_object163510:head# 2024-05-15T23:19:24.249 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da585115:::benchmark_data_smithi090_104448_object161465:head# 2024-05-15T23:19:24.249 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da59c020:::benchmark_data_smithi090_104448_object163487:head# 2024-05-15T23:19:24.249 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da5bf086:::benchmark_data_smithi090_104448_object161441:head# 2024-05-15T23:19:24.250 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da6ee82e:::benchmark_data_smithi090_104448_object171137:head# 2024-05-15T23:19:24.250 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da70685e:::benchmark_data_smithi090_104448_object169479:head# 2024-05-15T23:19:24.251 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da752ed2:::benchmark_data_smithi090_104448_object167958:head# 2024-05-15T23:19:24.251 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da78427d:::benchmark_data_smithi090_104448_object172861:head# 2024-05-15T23:19:24.252 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da7b18ff:::benchmark_data_smithi090_104448_object170944:head# 2024-05-15T23:19:24.252 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da7cc4f2:::benchmark_data_smithi090_104448_object159102:head# 2024-05-15T23:19:24.252 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da7e537a:::benchmark_data_smithi090_104448_object165928:head# 2024-05-15T23:19:24.253 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da7e8d73:::benchmark_data_smithi090_104448_object157907:head# 2024-05-15T23:19:24.253 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da85010d:::benchmark_data_smithi090_104448_object166266:head# 2024-05-15T23:19:24.254 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da8a10f6:::benchmark_data_smithi090_104448_object167301:head# 2024-05-15T23:19:24.254 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da8c5f17:::benchmark_data_smithi090_104448_object172131:head# 2024-05-15T23:19:24.254 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da9288e2:::benchmark_data_smithi090_104448_object158935:head# 2024-05-15T23:19:24.255 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da9b00fe:::benchmark_data_smithi090_104448_object167485:head# 2024-05-15T23:19:24.255 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da9bb4cf:::benchmark_data_smithi090_104448_object171677:head# 2024-05-15T23:19:24.255 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da9cb2e8:::benchmark_data_smithi090_104448_object166332:head# 2024-05-15T23:19:24.256 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da9dce5a:::benchmark_data_smithi090_104448_object169738:head# 2024-05-15T23:19:24.256 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:da9f7078:::benchmark_data_smithi090_104448_object171693:head# 2024-05-15T23:19:24.257 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:daa3589e:::benchmark_data_smithi090_104448_object161314:head# 2024-05-15T23:19:24.257 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:daa57a8a:::benchmark_data_smithi090_104448_object165342:head# 2024-05-15T23:19:24.258 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dab21112:::benchmark_data_smithi090_104448_object159292:head# 2024-05-15T23:19:24.258 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dab4c9e5:::benchmark_data_smithi090_104448_object165838:head# 2024-05-15T23:19:24.259 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dab96f7f:::benchmark_data_smithi090_104448_object169697:head# 2024-05-15T23:19:24.259 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dac24c02:::benchmark_data_smithi090_104448_object164484:head# 2024-05-15T23:19:24.259 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dad1dd60:::benchmark_data_smithi090_104448_object172245:head# 2024-05-15T23:19:24.260 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dad361b9:::benchmark_data_smithi090_104448_object156655:head# 2024-05-15T23:19:24.260 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dad6377f:::benchmark_data_smithi090_104448_object163986:head# 2024-05-15T23:19:24.261 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dad6cf58:::benchmark_data_smithi090_104448_object157879:head# 2024-05-15T23:19:24.261 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dadb4b83:::benchmark_data_smithi090_104448_object168865:head# 2024-05-15T23:19:24.261 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dae2cf27:::benchmark_data_smithi090_104448_object159073:head# 2024-05-15T23:19:24.262 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dae3789e:::benchmark_data_smithi090_104448_object166701:head# 2024-05-15T23:19:24.262 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dae3bfc0:::benchmark_data_smithi090_104448_object162575:head# 2024-05-15T23:19:24.263 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:daee60eb:::benchmark_data_smithi090_104448_object169872:head# 2024-05-15T23:19:24.263 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:daeeff35:::benchmark_data_smithi090_104448_object165912:head# 2024-05-15T23:19:24.264 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:daf0abef:::benchmark_data_smithi090_104448_object157794:head# 2024-05-15T23:19:24.264 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:daf1aa1c:::benchmark_data_smithi090_104448_object161865:head# 2024-05-15T23:19:24.265 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:daf5a897:::benchmark_data_smithi090_104448_object168238:head# 2024-05-15T23:19:24.265 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:daf6718e:::benchmark_data_smithi090_104448_object167375:head# 2024-05-15T23:19:24.265 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dafaada1:::benchmark_data_smithi090_104448_object165689:head# 2024-05-15T23:19:24.266 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dafb146e:::benchmark_data_smithi090_104448_object157405:head# 2024-05-15T23:19:24.266 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dafd5363:::benchmark_data_smithi090_104448_object157267:head# 2024-05-15T23:19:24.266 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db010433:::benchmark_data_smithi090_104448_object171018:head# 2024-05-15T23:19:24.267 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db07922d:::benchmark_data_smithi090_104448_object157571:head# 2024-05-15T23:19:24.267 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db1287b4:::benchmark_data_smithi090_104448_object169086:head# 2024-05-15T23:19:24.268 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db12d1a2:::benchmark_data_smithi090_104448_object162444:head# 2024-05-15T23:19:24.268 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db175c29:::benchmark_data_smithi090_104448_object158670:head# 2024-05-15T23:19:24.268 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db23bc0d:::benchmark_data_smithi090_104448_object160685:head# 2024-05-15T23:19:24.269 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db270727:::benchmark_data_smithi090_104448_object161325:head# 2024-05-15T23:19:24.269 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db289958:::benchmark_data_smithi090_104448_object167084:head# 2024-05-15T23:19:24.270 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db2bff47:::benchmark_data_smithi090_104448_object166458:head# 2024-05-15T23:19:24.270 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db30b2a0:::benchmark_data_smithi090_104448_object164380:head# 2024-05-15T23:19:24.270 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db328b96:::benchmark_data_smithi090_104448_object172814:head# 2024-05-15T23:19:24.271 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db358dc0:::benchmark_data_smithi090_104448_object165256:head# 2024-05-15T23:19:24.271 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db3cbb61:::benchmark_data_smithi090_104448_object159371:head# 2024-05-15T23:19:24.271 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db3ce536:::benchmark_data_smithi090_104448_object167859:head# 2024-05-15T23:19:24.272 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db411411:::benchmark_data_smithi090_104448_object160617:head# 2024-05-15T23:19:24.272 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db425b27:::benchmark_data_smithi090_104448_object157188:head# 2024-05-15T23:19:24.272 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db47a5f8:::benchmark_data_smithi090_104448_object166688:head# 2024-05-15T23:19:24.273 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db491060:::benchmark_data_smithi090_104448_object170412:head# 2024-05-15T23:19:24.273 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db4b8e72:::benchmark_data_smithi090_104448_object166209:head# 2024-05-15T23:19:24.274 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db4fa25e:::benchmark_data_smithi090_104448_object162381:head# 2024-05-15T23:19:24.274 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db5059c0:::benchmark_data_smithi090_104448_object160424:head# 2024-05-15T23:19:24.274 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db576216:::benchmark_data_smithi090_104448_object166290:head# 2024-05-15T23:19:24.275 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db5d7e8d:::benchmark_data_smithi090_104448_object158143:head# 2024-05-15T23:19:24.275 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db5dd314:::benchmark_data_smithi090_104448_object171398:head# 2024-05-15T23:19:24.275 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db5fd73d:::benchmark_data_smithi090_104448_object163994:head# 2024-05-15T23:19:24.276 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db643ae8:::benchmark_data_smithi090_104448_object166666:head# 2024-05-15T23:19:24.276 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db64b30f:::benchmark_data_smithi090_104448_object173153:head# 2024-05-15T23:19:24.277 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db66807a:::benchmark_data_smithi090_104448_object171485:head# 2024-05-15T23:19:24.277 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db6dfce9:::benchmark_data_smithi090_104448_object158492:head# 2024-05-15T23:19:24.278 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db766d86:::benchmark_data_smithi090_104448_object172253:head# 2024-05-15T23:19:24.278 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db7c95b3:::benchmark_data_smithi090_104448_object158686:head# 2024-05-15T23:19:24.279 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db7f88a8:::benchmark_data_smithi090_104448_object161093:head# 2024-05-15T23:19:24.279 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db81d6df:::benchmark_data_smithi090_104448_object161225:head# 2024-05-15T23:19:24.279 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db8d439a:::benchmark_data_smithi090_104448_object156839:head# 2024-05-15T23:19:24.280 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db8f1ae7:::benchmark_data_smithi090_104448_object170119:head# 2024-05-15T23:19:24.280 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db8fb233:::benchmark_data_smithi090_104448_object158641:head# 2024-05-15T23:19:24.281 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db955315:::benchmark_data_smithi090_104448_object163460:head# 2024-05-15T23:19:24.281 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:db9b171f:::benchmark_data_smithi090_104448_object159771:head# 2024-05-15T23:19:24.282 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dba23c65:::benchmark_data_smithi090_104448_object170504:head# 2024-05-15T23:19:24.282 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbaf0fcd:::benchmark_data_smithi090_104448_object158843:head# 2024-05-15T23:19:24.282 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbbfa1fe:::benchmark_data_smithi090_104448_object161988:head# 2024-05-15T23:19:24.283 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbc37c10:::benchmark_data_smithi090_104448_object165904:head# 2024-05-15T23:19:24.283 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbc5e53e:::benchmark_data_smithi090_104448_object168038:head# 2024-05-15T23:19:24.283 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbc6348d:::benchmark_data_smithi090_104448_object160820:head# 2024-05-15T23:19:24.284 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbc8ec81:::benchmark_data_smithi090_104448_object160009:head# 2024-05-15T23:19:24.284 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbca6b11:::benchmark_data_smithi090_104448_object158765:head# 2024-05-15T23:19:24.285 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbcc1d2e:::benchmark_data_smithi090_104448_object171857:head# 2024-05-15T23:19:24.285 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbcd46ec:::benchmark_data_smithi090_104448_object168498:head# 2024-05-15T23:19:24.286 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbce9a9b:::benchmark_data_smithi090_104448_object169733:head# 2024-05-15T23:19:24.286 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbcf35b8:::benchmark_data_smithi090_104448_object163625:head# 2024-05-15T23:19:24.286 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbd12397:::benchmark_data_smithi090_104448_object173004:head# 2024-05-15T23:19:24.287 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbd7d533:::benchmark_data_smithi090_104448_object168676:head# 2024-05-15T23:19:24.287 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbdbc061:::benchmark_data_smithi090_104448_object164557:head# 2024-05-15T23:19:24.288 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbddb243:::benchmark_data_smithi090_104448_object166902:head# 2024-05-15T23:19:24.288 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbea4876:::benchmark_data_smithi090_104448_object157329:head# 2024-05-15T23:19:24.288 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbee79c0:::benchmark_data_smithi090_104448_object168766:head# 2024-05-15T23:19:24.289 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbf23b17:::benchmark_data_smithi090_104448_object170374:head# 2024-05-15T23:19:24.289 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbf723a7:::benchmark_data_smithi090_104448_object164012:head# 2024-05-15T23:19:24.290 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbfa64f1:::benchmark_data_smithi090_104448_object169071:head# 2024-05-15T23:19:24.290 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dbfbfded:::benchmark_data_smithi090_104448_object156803:head# 2024-05-15T23:19:24.290 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc03ed47:::benchmark_data_smithi090_104448_object172212:head# 2024-05-15T23:19:24.291 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc064532:::benchmark_data_smithi090_104448_object167828:head# 2024-05-15T23:19:24.291 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc0c95af:::benchmark_data_smithi090_104448_object164829:head# 2024-05-15T23:19:24.291 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc0cfacb:::benchmark_data_smithi090_104448_object163832:head# 2024-05-15T23:19:24.292 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc0daa4a:::benchmark_data_smithi090_104448_object163743:head# 2024-05-15T23:19:24.292 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc13f871:::benchmark_data_smithi090_104448_object169982:head# 2024-05-15T23:19:24.293 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc183bb3:::benchmark_data_smithi090_104448_object171120:head# 2024-05-15T23:19:24.293 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc1c4c3b:::benchmark_data_smithi090_104448_object164612:head# 2024-05-15T23:19:24.294 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc1d75ca:::benchmark_data_smithi090_104448_object158291:head# 2024-05-15T23:19:24.294 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc214fc7:::benchmark_data_smithi090_104448_object163426:head# 2024-05-15T23:19:24.295 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc233de2:::benchmark_data_smithi090_104448_object158660:head# 2024-05-15T23:19:24.295 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc289b47:::benchmark_data_smithi090_104448_object166947:head# 2024-05-15T23:19:24.295 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc28f98d:::benchmark_data_smithi090_104448_object158673:head# 2024-05-15T23:19:24.296 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc2b99cf:::benchmark_data_smithi090_104448_object167000:head# 2024-05-15T23:19:24.296 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc301609:::benchmark_data_smithi090_104448_object172806:head# 2024-05-15T23:19:24.297 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc31fcba:::benchmark_data_smithi090_104448_object160022:head# 2024-05-15T23:19:24.297 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc35c7af:::benchmark_data_smithi090_104448_object159963:head# 2024-05-15T23:19:24.298 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc3a63a8:::benchmark_data_smithi090_104448_object170579:head# 2024-05-15T23:19:24.298 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc3ca0ac:::benchmark_data_smithi090_104448_object165728:head# 2024-05-15T23:19:24.299 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc41080a:::benchmark_data_smithi090_104448_object170740:head# 2024-05-15T23:19:24.299 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc45b01e:::benchmark_data_smithi090_104448_object172417:head# 2024-05-15T23:19:24.300 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc559667:::benchmark_data_smithi090_104448_object167545:head# 2024-05-15T23:19:24.300 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc589cce:::benchmark_data_smithi090_104448_object161209:head# 2024-05-15T23:19:24.300 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc5afd84:::benchmark_data_smithi090_104448_object167523:head# 2024-05-15T23:19:24.301 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc620463:::benchmark_data_smithi090_104448_object157931:head# 2024-05-15T23:19:24.301 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc63e241:::benchmark_data_smithi090_104448_object164659:head# 2024-05-15T23:19:24.302 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc677387:::benchmark_data_smithi090_104448_object168841:head# 2024-05-15T23:19:24.302 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc69f5fd:::benchmark_data_smithi090_104448_object159453:head# 2024-05-15T23:19:24.303 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc6c02eb:::benchmark_data_smithi090_104448_object159017:head# 2024-05-15T23:19:24.303 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc6ed56d:::benchmark_data_smithi090_104448_object157660:head# 2024-05-15T23:19:24.304 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc77524f:::benchmark_data_smithi090_104448_object171095:head# 2024-05-15T23:19:24.304 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc84bc93:::benchmark_data_smithi090_104448_object173021:head# 2024-05-15T23:19:24.304 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc89a898:::benchmark_data_smithi090_104448_object171246:head# 2024-05-15T23:19:24.305 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc8bd4ae:::benchmark_data_smithi090_104448_object163194:head# 2024-05-15T23:19:24.305 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc8d8b85:::benchmark_data_smithi090_104448_object162462:head# 2024-05-15T23:19:24.305 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc8ec595:::benchmark_data_smithi090_104448_object168434:head# 2024-05-15T23:19:24.306 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc8fc75a:::benchmark_data_smithi090_104448_object158873:head# 2024-05-15T23:19:24.306 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc91b079:::benchmark_data_smithi090_104448_object159250:head# 2024-05-15T23:19:24.307 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dc9488fd:::benchmark_data_smithi090_104448_object167376:head# 2024-05-15T23:19:24.307 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dca2f879:::benchmark_data_smithi090_104448_object166245:head# 2024-05-15T23:19:24.307 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dca6a81c:::benchmark_data_smithi090_104448_object166611:head# 2024-05-15T23:19:24.308 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dca9ba8f:::benchmark_data_smithi090_104448_object158557:head# 2024-05-15T23:19:24.308 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcaa49a2:::benchmark_data_smithi090_104448_object164061:head# 2024-05-15T23:19:24.309 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcad1434:::benchmark_data_smithi090_104448_object163854:head# 2024-05-15T23:19:24.309 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcb12e8f:::benchmark_data_smithi090_104448_object168922:head# 2024-05-15T23:19:24.309 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcb175ad:::benchmark_data_smithi090_104448_object159156:head# 2024-05-15T23:19:24.310 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcb50846:::benchmark_data_smithi090_104448_object168249:head# 2024-05-15T23:19:24.310 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcb60dbd:::benchmark_data_smithi090_104448_object167014:head# 2024-05-15T23:19:24.310 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcb6b582:::benchmark_data_smithi090_104448_object162920:head# 2024-05-15T23:19:24.311 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcb6ed11:::benchmark_data_smithi090_104448_object159186:head# 2024-05-15T23:19:24.311 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcbb44de:::benchmark_data_smithi090_104448_object165660:head# 2024-05-15T23:19:24.312 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcbbf0a2:::benchmark_data_smithi090_104448_object159930:head# 2024-05-15T23:19:24.312 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcc05549:::benchmark_data_smithi090_104448_object161448:head# 2024-05-15T23:19:24.312 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcc0b834:::benchmark_data_smithi090_104448_object170708:head# 2024-05-15T23:19:24.313 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcc8a7c1:::benchmark_data_smithi090_104448_object160886:head# 2024-05-15T23:19:24.313 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcc9c32b:::benchmark_data_smithi090_104448_object161663:head# 2024-05-15T23:19:24.313 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcce6ba6:::benchmark_data_smithi090_104448_object161351:head# 2024-05-15T23:19:24.314 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcd0a5c2:::benchmark_data_smithi090_104448_object167706:head# 2024-05-15T23:19:24.318 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcd1112b:::benchmark_data_smithi090_104448_object160189:head# 2024-05-15T23:19:24.319 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcd19574:::benchmark_data_smithi090_104448_object160147:head# 2024-05-15T23:19:24.319 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcd2d052:::benchmark_data_smithi090_104448_object161402:head# 2024-05-15T23:19:24.319 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcd8100c:::benchmark_data_smithi090_104448_object171140:head# 2024-05-15T23:19:24.319 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcdf7377:::benchmark_data_smithi090_104448_object158060:head# 2024-05-15T23:19:24.319 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dce1cbea:::benchmark_data_smithi090_104448_object161812:head# 2024-05-15T23:19:24.319 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dceed1a1:::benchmark_data_smithi090_104448_object170357:head# 2024-05-15T23:19:24.319 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcf544c4:::benchmark_data_smithi090_104448_object161681:head# 2024-05-15T23:19:24.319 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcf9633e:::benchmark_data_smithi090_104448_object169946:head# 2024-05-15T23:19:24.319 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcf99bab:::benchmark_data_smithi090_104448_object158035:head# 2024-05-15T23:19:24.319 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dcfec32b:::benchmark_data_smithi090_104448_object161132:head# 2024-05-15T23:19:24.319 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd004d1d:::benchmark_data_smithi090_104448_object163639:head# 2024-05-15T23:19:24.320 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd0627de:::benchmark_data_smithi090_104448_object161598:head# 2024-05-15T23:19:24.320 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd08e6ad:::benchmark_data_smithi090_104448_object163619:head# 2024-05-15T23:19:24.320 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd09b5d9:::benchmark_data_smithi090_104448_object168863:head# 2024-05-15T23:19:24.320 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd0b4f93:::benchmark_data_smithi090_104448_object166143:head# 2024-05-15T23:19:24.321 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd0f483f:::benchmark_data_smithi090_104448_object163752:head# 2024-05-15T23:19:24.321 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd1438b0:::benchmark_data_smithi090_104448_object162486:head# 2024-05-15T23:19:24.322 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd147ebb:::benchmark_data_smithi090_104448_object167186:head# 2024-05-15T23:19:24.322 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd152ec9:::benchmark_data_smithi090_104448_object159993:head# 2024-05-15T23:19:24.322 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd176c11:::benchmark_data_smithi090_104448_object164263:head# 2024-05-15T23:19:24.323 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd1c83b5:::benchmark_data_smithi090_104448_object158672:head# 2024-05-15T23:19:24.323 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd1f383e:::benchmark_data_smithi090_104448_object172045:head# 2024-05-15T23:19:24.324 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd203e1e:::benchmark_data_smithi090_104448_object168634:head# 2024-05-15T23:19:24.324 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd441e33:::benchmark_data_smithi090_104448_object157743:head# 2024-05-15T23:19:24.324 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd443b56:::benchmark_data_smithi090_104448_object164498:head# 2024-05-15T23:19:24.325 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd4a6bba:::benchmark_data_smithi090_104448_object158176:head# 2024-05-15T23:19:24.325 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd5134db:::benchmark_data_smithi090_104448_object172590:head# 2024-05-15T23:19:24.326 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd57e547:::benchmark_data_smithi090_104448_object169155:head# 2024-05-15T23:19:24.326 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd637e63:::benchmark_data_smithi090_104448_object172655:head# 2024-05-15T23:19:24.327 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd642628:::benchmark_data_smithi090_104448_object160433:head# 2024-05-15T23:19:24.328 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd64c2f7:::benchmark_data_smithi090_104448_object166526:head# 2024-05-15T23:19:24.328 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd68cbcc:::benchmark_data_smithi090_104448_object167623:head# 2024-05-15T23:19:24.328 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd695232:::benchmark_data_smithi090_104448_object169269:head# 2024-05-15T23:19:24.329 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd6b4ba7:::benchmark_data_smithi090_104448_object169833:head# 2024-05-15T23:19:24.329 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd6f9f46:::benchmark_data_smithi090_104448_object163646:head# 2024-05-15T23:19:24.330 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd70ea81:::benchmark_data_smithi090_104448_object165327:head# 2024-05-15T23:19:24.330 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd7232cd:::benchmark_data_smithi090_104448_object170919:head# 2024-05-15T23:19:24.330 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd7b624b:::benchmark_data_smithi090_104448_object160372:head# 2024-05-15T23:19:24.331 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd85b05b:::benchmark_data_smithi090_104448_object172046:head# 2024-05-15T23:19:24.331 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd8745f4:::benchmark_data_smithi090_104448_object167872:head# 2024-05-15T23:19:24.332 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd89b13d:::benchmark_data_smithi090_104448_object167234:head# 2024-05-15T23:19:24.332 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd89cad8:::benchmark_data_smithi090_104448_object161701:head# 2024-05-15T23:19:24.332 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd8eebfd:::benchmark_data_smithi090_104448_object162689:head# 2024-05-15T23:19:24.333 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dd9155fe:::benchmark_data_smithi090_104448_object165734:head# 2024-05-15T23:19:24.333 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dda16d04:::benchmark_data_smithi090_104448_object165522:head# 2024-05-15T23:19:24.334 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dda68a26:::benchmark_data_smithi090_104448_object161420:head# 2024-05-15T23:19:24.334 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ddaf70ee:::benchmark_data_smithi090_104448_object159947:head# 2024-05-15T23:19:24.334 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ddb04a0a:::benchmark_data_smithi090_104448_object168365:head# 2024-05-15T23:19:24.335 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ddb17e86:::benchmark_data_smithi090_104448_object162965:head# 2024-05-15T23:19:24.335 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ddbe7603:::benchmark_data_smithi090_104448_object167691:head# 2024-05-15T23:19:24.336 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ddbfddf6:::benchmark_data_smithi090_104448_object163756:head# 2024-05-15T23:19:24.336 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ddc1c731:::benchmark_data_smithi090_104448_object161106:head# 2024-05-15T23:19:24.336 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ddc6df24:::benchmark_data_smithi090_104448_object161327:head# 2024-05-15T23:19:24.337 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ddc6e8c3:::benchmark_data_smithi090_104448_object161504:head# 2024-05-15T23:19:24.337 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ddcb5754:::benchmark_data_smithi090_104448_object166016:head# 2024-05-15T23:19:24.337 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ddd0d606:::benchmark_data_smithi090_104448_object166725:head# 2024-05-15T23:19:24.338 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ddd39ff5:::benchmark_data_smithi090_104448_object167321:head# 2024-05-15T23:19:24.338 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dddbaef2:::benchmark_data_smithi090_104448_object168042:head# 2024-05-15T23:19:24.339 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dddeb8e4:::benchmark_data_smithi090_104448_object168962:head# 2024-05-15T23:19:24.339 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dde4a6df:::benchmark_data_smithi090_104448_object159199:head# 2024-05-15T23:19:24.339 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dded3d8e:::benchmark_data_smithi090_104448_object157007:head# 2024-05-15T23:19:24.340 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ddf4fd7f:::benchmark_data_smithi090_104448_object167593:head# 2024-05-15T23:19:24.340 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ddfad0d2:::benchmark_data_smithi090_104448_object158871:head# 2024-05-15T23:19:24.341 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ddfb3462:::benchmark_data_smithi090_104448_object170973:head# 2024-05-15T23:19:24.341 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ddfbd370:::benchmark_data_smithi090_104448_object160167:head# 2024-05-15T23:19:24.342 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de034602:::benchmark_data_smithi090_104448_object159979:head# 2024-05-15T23:19:24.342 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de0507f8:::benchmark_data_smithi090_104448_object164435:head# 2024-05-15T23:19:24.343 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de076b60:::benchmark_data_smithi090_104448_object160132:head# 2024-05-15T23:19:24.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:24.344 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de08b65f:::benchmark_data_smithi090_104448_object163783:head# 2024-05-15T23:19:24.347 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de0a0abc:::benchmark_data_smithi090_104448_object157849:head# 2024-05-15T23:19:24.347 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de0cf46e:::benchmark_data_smithi090_104448_object172146:head# 2024-05-15T23:19:24.347 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de106580:::benchmark_data_smithi090_104448_object165701:head# 2024-05-15T23:19:24.347 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de10807e:::benchmark_data_smithi090_104448_object158512:head# 2024-05-15T23:19:24.347 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de124199:::benchmark_data_smithi090_104448_object167939:head# 2024-05-15T23:19:24.348 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de144918:::benchmark_data_smithi090_104448_object169601:head# 2024-05-15T23:19:24.348 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de184b79:::benchmark_data_smithi090_104448_object158237:head# 2024-05-15T23:19:24.348 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de1b9e8f:::benchmark_data_smithi090_104448_object160263:head# 2024-05-15T23:19:24.348 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de207208:::benchmark_data_smithi090_104448_object170773:head# 2024-05-15T23:19:24.349 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de21d431:::benchmark_data_smithi090_104448_object165681:head# 2024-05-15T23:19:24.349 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de250d9d:::benchmark_data_smithi090_104448_object170549:head# 2024-05-15T23:19:24.349 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de26b12b:::benchmark_data_smithi090_104448_object167486:head# 2024-05-15T23:19:24.350 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de295ccf:::benchmark_data_smithi090_104448_object160725:head# 2024-05-15T23:19:24.350 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de2a2a22:::benchmark_data_smithi090_104448_object163940:head# 2024-05-15T23:19:24.351 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de2e6125:::benchmark_data_smithi090_104448_object171139:head# 2024-05-15T23:19:24.351 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de2f4253:::benchmark_data_smithi090_104448_object159459:head# 2024-05-15T23:19:24.351 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de3545f9:::benchmark_data_smithi090_104448_object165760:head# 2024-05-15T23:19:24.352 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de376250:::benchmark_data_smithi090_104448_object170240:head# 2024-05-15T23:19:24.352 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de38cd47:::benchmark_data_smithi090_104448_object163843:head# 2024-05-15T23:19:24.353 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de3af0f0:::benchmark_data_smithi090_104448_object168349:head# 2024-05-15T23:19:24.353 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de3f4d58:::benchmark_data_smithi090_104448_object167264:head# 2024-05-15T23:19:24.353 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de3faedd:::benchmark_data_smithi090_104448_object170282:head# 2024-05-15T23:19:24.354 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de480914:::benchmark_data_smithi090_104448_object159758:head# 2024-05-15T23:19:24.354 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de4a4643:::benchmark_data_smithi090_104448_object170105:head# 2024-05-15T23:19:24.354 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de4b72d7:::benchmark_data_smithi090_104448_object168937:head# 2024-05-15T23:19:24.355 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de4f2dca:::benchmark_data_smithi090_104448_object162927:head# 2024-05-15T23:19:24.355 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de505a8b:::benchmark_data_smithi090_104448_object158043:head# 2024-05-15T23:19:24.355 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de53da8b:::benchmark_data_smithi090_104448_object167992:head# 2024-05-15T23:19:24.356 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de595409:::benchmark_data_smithi090_104448_object160706:head# 2024-05-15T23:19:24.356 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de59beff:::benchmark_data_smithi090_104448_object162390:head# 2024-05-15T23:19:24.357 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de5c134f:::benchmark_data_smithi090_104448_object170373:head# 2024-05-15T23:19:24.357 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de60bb4b:::benchmark_data_smithi090_104448_object166177:head# 2024-05-15T23:19:24.357 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de64e2ef:::benchmark_data_smithi090_104448_object156780:head# 2024-05-15T23:19:24.358 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de6b9e38:::benchmark_data_smithi090_104448_object165333:head# 2024-05-15T23:19:24.358 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de6c9ca8:::benchmark_data_smithi090_104448_object160688:head# 2024-05-15T23:19:24.358 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de6cc179:::benchmark_data_smithi090_104448_object166311:head# 2024-05-15T23:19:24.359 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de7a5a41:::benchmark_data_smithi090_104448_object171708:head# 2024-05-15T23:19:24.359 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de7afed5:::benchmark_data_smithi090_104448_object172628:head# 2024-05-15T23:19:24.359 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de81a362:::benchmark_data_smithi090_104448_object164708:head# 2024-05-15T23:19:24.360 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de88e094:::benchmark_data_smithi090_104448_object162570:head# 2024-05-15T23:19:24.360 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de987865:::benchmark_data_smithi090_104448_object165631:head# 2024-05-15T23:19:24.361 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de9a97f1:::benchmark_data_smithi090_104448_object168888:head# 2024-05-15T23:19:24.361 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de9ab80e:::benchmark_data_smithi090_104448_object158056:head# 2024-05-15T23:19:24.361 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:de9bb8f0:::benchmark_data_smithi090_104448_object172474:head# 2024-05-15T23:19:24.362 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dea2cdef:::benchmark_data_smithi090_104448_object156931:head# 2024-05-15T23:19:24.362 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dea44fb7:::benchmark_data_smithi090_104448_object168948:head# 2024-05-15T23:19:24.363 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:deaceb2c:::benchmark_data_smithi090_104448_object162045:head# 2024-05-15T23:19:24.363 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:deb1ca6a:::benchmark_data_smithi090_104448_object168686:head# 2024-05-15T23:19:24.363 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:deb77611:::benchmark_data_smithi090_104448_object171293:head# 2024-05-15T23:19:24.364 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dec29a9c:::benchmark_data_smithi090_104448_object168904:head# 2024-05-15T23:19:24.364 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dec4068c:::benchmark_data_smithi090_104448_object172651:head# 2024-05-15T23:19:24.365 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dec9b76b:::benchmark_data_smithi090_104448_object162039:head# 2024-05-15T23:19:24.365 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:decdb4ee:::benchmark_data_smithi090_104448_object166180:head# 2024-05-15T23:19:24.366 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ded23a54:::benchmark_data_smithi090_104448_object169608:head# 2024-05-15T23:19:24.366 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ded29b35:::benchmark_data_smithi090_104448_object172534:head# 2024-05-15T23:19:24.366 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ded2b364:::benchmark_data_smithi090_104448_object165876:head# 2024-05-15T23:19:24.367 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ded2ce50:::benchmark_data_smithi090_104448_object168411:head# 2024-05-15T23:19:24.367 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ded42f31:::benchmark_data_smithi090_104448_object172026:head# 2024-05-15T23:19:24.367 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ded4e470:::benchmark_data_smithi090_104448_object160896:head# 2024-05-15T23:19:24.368 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ded558da:::benchmark_data_smithi090_104448_object169061:head# 2024-05-15T23:19:24.368 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:ded77225:::benchmark_data_smithi090_104448_object168362:head# 2024-05-15T23:19:24.369 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dedb7bf6:::benchmark_data_smithi090_104448_object165383:head# 2024-05-15T23:19:24.369 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dee3fc74:::benchmark_data_smithi090_104448_object167381:head# 2024-05-15T23:19:24.371 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dee57081:::benchmark_data_smithi090_104448_object164476:head# 2024-05-15T23:19:24.371 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dee7ba1d:::benchmark_data_smithi090_104448_object162817:head# 2024-05-15T23:19:24.372 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dee95e02:::benchmark_data_smithi090_104448_object170088:head# 2024-05-15T23:19:24.372 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:deecd005:::benchmark_data_smithi090_104448_object166254:head# 2024-05-15T23:19:24.372 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:24.373 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:def277aa:::benchmark_data_smithi090_104448_object172881:head# 2024-05-15T23:19:24.373 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:def48251:::benchmark_data_smithi090_104448_object168518:head# 2024-05-15T23:19:24.373 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:defbdf5c:::benchmark_data_smithi090_104448_object163586:head# 2024-05-15T23:19:24.374 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df006192:::benchmark_data_smithi090_104448_object165311:head# 2024-05-15T23:19:24.374 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df072858:::benchmark_data_smithi090_104448_object161743:head# 2024-05-15T23:19:24.374 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df0b8bf4:::benchmark_data_smithi090_104448_object171472:head# 2024-05-15T23:19:24.375 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df0e0d34:::benchmark_data_smithi090_104448_object164047:head# 2024-05-15T23:19:24.375 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df0e12b0:::benchmark_data_smithi090_104448_object167356:head# 2024-05-15T23:19:24.376 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df10a7dd:::benchmark_data_smithi090_104448_object163987:head# 2024-05-15T23:19:24.376 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df124242:::benchmark_data_smithi090_104448_object157175:head# 2024-05-15T23:19:24.376 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df1306e6:::benchmark_data_smithi090_104448_object167982:head# 2024-05-15T23:19:24.376 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df15f35b:::benchmark_data_smithi090_104448_object166429:head# 2024-05-15T23:19:24.377 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df177ba7:::benchmark_data_smithi090_104448_object171950:head# 2024-05-15T23:19:24.377 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df1a37d9:::benchmark_data_smithi090_104448_object172566:head# 2024-05-15T23:19:24.377 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df1fd80f:::benchmark_data_smithi090_104448_object162050:head# 2024-05-15T23:19:24.378 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df245c6a:::benchmark_data_smithi090_104448_object172488:head# 2024-05-15T23:19:24.378 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df3019cf:::benchmark_data_smithi090_104448_object172690:head# 2024-05-15T23:19:24.378 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df31daf2:::benchmark_data_smithi090_104448_object163341:head# 2024-05-15T23:19:24.379 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df36db9d:::benchmark_data_smithi090_104448_object163997:head# 2024-05-15T23:19:24.379 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df4363d4:::benchmark_data_smithi090_104448_object167549:head# 2024-05-15T23:19:24.379 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df43aff1:::benchmark_data_smithi090_104448_object168600:head# 2024-05-15T23:19:24.380 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df4e11c3:::benchmark_data_smithi090_104448_object169989:head# 2024-05-15T23:19:24.380 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df55a67a:::benchmark_data_smithi090_104448_object165744:head# 2024-05-15T23:19:24.381 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df5b99ce:::benchmark_data_smithi090_104448_object163883:head# 2024-05-15T23:19:24.381 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df5cd600:::benchmark_data_smithi090_104448_object157181:head# 2024-05-15T23:19:24.381 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df5f11de:::benchmark_data_smithi090_104448_object161715:head# 2024-05-15T23:19:24.382 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df5f1ffa:::benchmark_data_smithi090_104448_object160679:head# 2024-05-15T23:19:24.382 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df600256:::benchmark_data_smithi090_104448_object160714:head# 2024-05-15T23:19:24.382 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df67f9f6:::benchmark_data_smithi090_104448_object162152:head# 2024-05-15T23:19:24.383 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df6aa5a3:::benchmark_data_smithi090_104448_object170562:head# 2024-05-15T23:19:24.383 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df6b3bc0:::benchmark_data_smithi090_104448_object170570:head# 2024-05-15T23:19:24.384 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df6fbf98:::benchmark_data_smithi090_104448_object172082:head# 2024-05-15T23:19:24.384 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df724616:::benchmark_data_smithi090_104448_object166076:head# 2024-05-15T23:19:24.385 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df751915:::benchmark_data_smithi090_104448_object156945:head# 2024-05-15T23:19:24.385 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df768e20:::benchmark_data_smithi090_104448_object162403:head# 2024-05-15T23:19:24.386 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df77497c:::benchmark_data_smithi090_104448_object168131:head# 2024-05-15T23:19:24.386 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df79363f:::benchmark_data_smithi090_104448_object165539:head# 2024-05-15T23:19:24.387 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df7ad08b:::benchmark_data_smithi090_104448_object158219:head# 2024-05-15T23:19:24.387 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df8151fb:::benchmark_data_smithi090_104448_object164545:head# 2024-05-15T23:19:24.388 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df907817:::benchmark_data_smithi090_104448_object167198:head# 2024-05-15T23:19:24.388 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df90815e:::benchmark_data_smithi090_104448_object159027:head# 2024-05-15T23:19:24.388 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df90c2c8:::benchmark_data_smithi090_104448_object159197:head# 2024-05-15T23:19:24.389 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df91c8e4:::benchmark_data_smithi090_104448_object162489:head# 2024-05-15T23:19:24.390 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df9316e7:::benchmark_data_smithi090_104448_object169972:head# 2024-05-15T23:19:24.390 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df9456b2:::benchmark_data_smithi090_104448_object160203:head# 2024-05-15T23:19:24.390 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df994973:::benchmark_data_smithi090_104448_object164167:head# 2024-05-15T23:19:24.391 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df9af3c8:::benchmark_data_smithi090_104448_object164526:head# 2024-05-15T23:19:24.391 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df9d425d:::benchmark_data_smithi090_104448_object167353:head# 2024-05-15T23:19:24.392 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:df9fda65:::benchmark_data_smithi090_104448_object160052:head# 2024-05-15T23:19:24.392 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfa1ec8e:::benchmark_data_smithi090_104448_object158495:head# 2024-05-15T23:19:24.392 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfa3ba4a:::benchmark_data_smithi090_104448_object159007:head# 2024-05-15T23:19:24.393 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfab4fe4:::benchmark_data_smithi090_104448_object162798:head# 2024-05-15T23:19:24.393 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfcac9e0:::benchmark_data_smithi090_104448_object158849:head# 2024-05-15T23:19:24.394 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfcec2da:::benchmark_data_smithi090_104448_object161419:head# 2024-05-15T23:19:24.394 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfcef5c1:::benchmark_data_smithi090_104448_object158000:head# 2024-05-15T23:19:24.394 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfd03484:::benchmark_data_smithi090_104448_object160478:head# 2024-05-15T23:19:24.395 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfd19c18:::benchmark_data_smithi090_104448_object172180:head# 2024-05-15T23:19:24.395 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfd28c5b:::benchmark_data_smithi090_104448_object160888:head# 2024-05-15T23:19:24.395 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfd69a7d:::benchmark_data_smithi090_104448_object168461:head# 2024-05-15T23:19:24.396 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfd6f441:::benchmark_data_smithi090_104448_object168796:head# 2024-05-15T23:19:24.396 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfdfd416:::benchmark_data_smithi090_104448_object168032:head# 2024-05-15T23:19:24.397 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfe2e25b:::benchmark_data_smithi090_104448_object172137:head# 2024-05-15T23:19:24.397 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfe594dd:::benchmark_data_smithi090_104448_object160936:head# 2024-05-15T23:19:24.397 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfeac963:::benchmark_data_smithi090_104448_object172354:head# 2024-05-15T23:19:24.398 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfebacbb:::benchmark_data_smithi090_104448_object172014:head# 2024-05-15T23:19:24.398 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfef5ad8:::benchmark_data_smithi090_104448_object161064:head# 2024-05-15T23:19:24.399 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dfef6a90:::benchmark_data_smithi090_104448_object172965:head# 2024-05-15T23:19:24.399 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dff10968:::benchmark_data_smithi090_104448_object162031:head# 2024-05-15T23:19:24.399 INFO:teuthology.orchestra.run.smithi084.stderr:Read #4:dff12bc3:::benchmark_data_smithi090_104448_object166003:head# 2024-05-15T23:19:24.400 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:24.403 INFO:teuthology.orchestra.run.smithi084.stderr:Export successful 2024-05-15T23:19:24.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:24.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:24.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:19:24.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:24.540 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:24.561 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:24.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:24.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:24 smithi088 ceph-mon[44699]: 4.6 scrub starts 2024-05-15T23:19:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:24 smithi088 ceph-mon[44699]: 4.6 scrub ok 2024-05-15T23:19:24.666 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:24.693 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:24 smithi084 ceph-mon[39784]: 4.6 scrub starts 2024-05-15T23:19:24.693 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:24 smithi084 ceph-mon[39784]: 4.6 scrub ok 2024-05-15T23:19:24.697 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:24.718 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:24.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:24.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:24.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:24 smithi090 ceph-mon[44434]: 4.6 scrub starts 2024-05-15T23:19:24.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:24 smithi090 ceph-mon[44434]: 4.6 scrub ok 2024-05-15T23:19:24.822 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:19:24.852 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:25.051 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:19:25.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:25.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:19:25.155 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:19:25.183 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:25.305 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:19:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:25.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:19:25.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:25.442 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:25.463 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:25.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:25.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:25.499 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:25.523 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -n osd.0 --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 4.b 2024-05-15T23:19:25.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:25 smithi088 ceph-mon[44699]: 4.9 scrub starts 2024-05-15T23:19:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:25 smithi088 ceph-mon[44699]: 4.9 scrub ok 2024-05-15T23:19:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:25 smithi088 ceph-mon[44699]: pgmap v1011: 17 pgs: 17 active+clean; 118 MiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:19:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:25 smithi088 ceph-mon[44699]: 4.d scrub starts 2024-05-15T23:19:25.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:25 smithi088 ceph-mon[44699]: 4.d scrub ok 2024-05-15T23:19:25.598 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:25.614 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:19:25.614 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:19:25.618 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:25.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:25.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:25.692 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:25 smithi084 ceph-mon[39784]: 4.9 scrub starts 2024-05-15T23:19:25.692 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:25 smithi084 ceph-mon[39784]: 4.9 scrub ok 2024-05-15T23:19:25.692 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:25 smithi084 ceph-mon[39784]: pgmap v1011: 17 pgs: 17 active+clean; 118 MiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:19:25.692 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:25 smithi084 ceph-mon[39784]: 4.d scrub starts 2024-05-15T23:19:25.692 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:25 smithi084 ceph-mon[39784]: 4.d scrub ok 2024-05-15T23:19:25.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:25.749 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:25 smithi090 ceph-mon[44434]: 4.9 scrub starts 2024-05-15T23:19:25.749 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:25 smithi090 ceph-mon[44434]: 4.9 scrub ok 2024-05-15T23:19:25.749 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:25 smithi090 ceph-mon[44434]: pgmap v1011: 17 pgs: 17 active+clean; 118 MiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:19:25.750 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:25 smithi090 ceph-mon[44434]: 4.d scrub starts 2024-05-15T23:19:25.750 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:25 smithi090 ceph-mon[44434]: 4.d scrub ok 2024-05-15T23:19:25.751 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:25.771 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:25.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:25.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:25.875 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:25.904 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:25.923 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:25.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:25.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:25.969 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/osd.0/config 2024-05-15T23:19:26.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:26.055 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:26.075 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:26.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:26.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:26.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:26.210 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:26.213 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:19:26.230 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:26.232 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:19:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:26.233 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:26.251 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:19:26.270 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:19:26.289 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:19:26.308 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:19:26.328 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:19:26.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:26.348 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:19:26.364 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:26.367 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:19:26.376 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T23:19:26.385 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:26.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:26.388 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:26.388 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:19:26.411 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:19:26.411 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-05-15T23:19:26.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:19:26.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:19:26.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:19:26.412 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:19:26.412 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:19:26.412 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:19:26.412 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:19:26.412 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:19:26.412 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:19:26.412 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:19:26.412 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:19:26.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:26.515 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:26 smithi090 ceph-mon[44434]: 4.0 scrub starts 2024-05-15T23:19:26.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:26 smithi090 ceph-mon[44434]: 4.0 scrub ok 2024-05-15T23:19:26.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:26 smithi090 ceph-mon[44434]: 4.5 scrub starts 2024-05-15T23:19:26.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:26 smithi090 ceph-mon[44434]: 4.5 scrub ok 2024-05-15T23:19:26.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:26 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2612385032' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:26.516 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:26.517 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:26.536 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:26.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:26.539 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:26.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:26 smithi084 ceph-mon[39784]: 4.0 scrub starts 2024-05-15T23:19:26.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:26 smithi084 ceph-mon[39784]: 4.0 scrub ok 2024-05-15T23:19:26.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:26 smithi084 ceph-mon[39784]: 4.5 scrub starts 2024-05-15T23:19:26.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:26 smithi084 ceph-mon[39784]: 4.5 scrub ok 2024-05-15T23:19:26.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:26 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2612385032' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:26.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:26.637 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:19:26.640 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:19:26.665 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:26 smithi088 ceph-mon[44699]: 4.0 scrub starts 2024-05-15T23:19:26.666 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:26 smithi088 ceph-mon[44699]: 4.0 scrub ok 2024-05-15T23:19:26.666 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:26 smithi088 ceph-mon[44699]: 4.5 scrub starts 2024-05-15T23:19:26.666 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:26 smithi088 ceph-mon[44699]: 4.5 scrub ok 2024-05-15T23:19:26.666 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:26 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2612385032' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:26.666 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:26.669 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:27.417 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:19:27.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:27.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:19:27.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:27.555 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:27.576 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:27.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:27.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:27 smithi084 ceph-mon[39784]: 4.2 scrub starts 2024-05-15T23:19:27.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:27 smithi084 ceph-mon[39784]: 4.2 scrub ok 2024-05-15T23:19:27.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:27 smithi084 ceph-mon[39784]: pgmap v1012: 17 pgs: 17 active+clean; 4.7 MiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 963 op/s 2024-05-15T23:19:27.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:27.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:27 smithi090 ceph-mon[44434]: 4.2 scrub starts 2024-05-15T23:19:27.710 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:27 smithi090 ceph-mon[44434]: 4.2 scrub ok 2024-05-15T23:19:27.710 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:27 smithi090 ceph-mon[44434]: pgmap v1012: 17 pgs: 17 active+clean; 4.7 MiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 963 op/s 2024-05-15T23:19:27.711 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:27.732 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:27.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:27.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:27.751 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:19:27.751 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:19:27.756 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.756 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.756 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.756 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.756 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.757 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.757 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.757 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.757 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.757 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:27.757 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:27 smithi088 ceph-mon[44699]: 4.2 scrub starts 2024-05-15T23:19:27.757 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:27 smithi088 ceph-mon[44699]: 4.2 scrub ok 2024-05-15T23:19:27.757 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:27 smithi088 ceph-mon[44699]: pgmap v1012: 17 pgs: 17 active+clean; 4.7 MiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 963 op/s 2024-05-15T23:19:27.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:27.865 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:27.885 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:27.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:27.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:19:28.019 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:28.047 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:19:28.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:28.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:19:28.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:28.181 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:28.203 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:28.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:28.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:28.335 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:28.355 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:28.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:28.358 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:28.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:28.486 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:28.505 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:28.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:28.508 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:28.572 INFO:teuthology.orchestra.run.smithi084.stderr:Creating an OSD daemon form without an OSD FSID value 2024-05-15T23:19:28.601 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-05-15T23:19:28.602 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-05-15T23:19:28.609 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:28.744 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:28.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:28.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:28.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:28.880 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:28.900 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:28.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:28.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:29.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:29.033 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:29.052 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:29.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:29.058 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:19:29.058 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:19:29.059 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:19:29.059 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:19:29.059 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:19:29.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:29.181 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:29 smithi090 ceph-mon[44434]: pgmap v1013: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 894 op/s 2024-05-15T23:19:29.183 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:29.202 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:29.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:29.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:29.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:29 smithi088 ceph-mon[44699]: pgmap v1013: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 894 op/s 2024-05-15T23:19:29.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:29 smithi084 ceph-mon[39784]: pgmap v1013: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 894 op/s 2024-05-15T23:19:29.338 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:29.752 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T23:19:29.752 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:19:29.887 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:29.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:29.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:29.925 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:29.995 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:30.017 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:19:30.024 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:30.045 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:30.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:30.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:30.058 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:19:30.149 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:30.176 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:30.297 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:30 smithi088 ceph-mon[44699]: 4.1 scrub starts 2024-05-15T23:19:30.297 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:30 smithi088 ceph-mon[44699]: 4.1 scrub ok 2024-05-15T23:19:30.297 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:30 smithi088 ceph-mon[44699]: 4.f scrub starts 2024-05-15T23:19:30.297 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:30 smithi088 ceph-mon[44699]: 4.f scrub ok 2024-05-15T23:19:30.297 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:30 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:19:30.297 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:30 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:19:30.297 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:30 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2785296192' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:19:30.298 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:30.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:30.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:30.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:30 smithi084 ceph-mon[39784]: 4.1 scrub starts 2024-05-15T23:19:30.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:30 smithi084 ceph-mon[39784]: 4.1 scrub ok 2024-05-15T23:19:30.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:30 smithi084 ceph-mon[39784]: 4.f scrub starts 2024-05-15T23:19:30.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:30 smithi084 ceph-mon[39784]: 4.f scrub ok 2024-05-15T23:19:30.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:30 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:19:30.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:30 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:19:30.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:30 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2785296192' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:19:30.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:30.431 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:30.453 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:30.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:30.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:30 smithi090 ceph-mon[44434]: 4.1 scrub starts 2024-05-15T23:19:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:30 smithi090 ceph-mon[44434]: 4.1 scrub ok 2024-05-15T23:19:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:30 smithi090 ceph-mon[44434]: 4.f scrub starts 2024-05-15T23:19:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:30 smithi090 ceph-mon[44434]: 4.f scrub ok 2024-05-15T23:19:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:30 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:19:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:30 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:19:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:30 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2785296192' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:19:30.557 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:30.584 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:30.933 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:30.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:30.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:31.037 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:31.069 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:31 smithi088 ceph-mon[44699]: 4.8 scrub starts 2024-05-15T23:19:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:31 smithi088 ceph-mon[44699]: 4.8 scrub ok 2024-05-15T23:19:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:31 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:19:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:31 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:19:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2785296192' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:19:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:31 smithi088 ceph-mon[44699]: osdmap e369: 12 total, 11 up, 10 in 2024-05-15T23:19:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:31 smithi088 ceph-mon[44699]: 4.7 deep-scrub starts 2024-05-15T23:19:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:31 smithi088 ceph-mon[44699]: 4.7 deep-scrub ok 2024-05-15T23:19:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:31 smithi088 ceph-mon[44699]: pgmap v1015: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 501 op/s 2024-05-15T23:19:31.322 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:31.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:31 smithi084 ceph-mon[39784]: 4.8 scrub starts 2024-05-15T23:19:31.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:31 smithi084 ceph-mon[39784]: 4.8 scrub ok 2024-05-15T23:19:31.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:31 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:19:31.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:31 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:19:31.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2785296192' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:19:31.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:31 smithi084 ceph-mon[39784]: osdmap e369: 12 total, 11 up, 10 in 2024-05-15T23:19:31.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:31 smithi084 ceph-mon[39784]: 4.7 deep-scrub starts 2024-05-15T23:19:31.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:31 smithi084 ceph-mon[39784]: 4.7 deep-scrub ok 2024-05-15T23:19:31.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:31 smithi084 ceph-mon[39784]: pgmap v1015: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 501 op/s 2024-05-15T23:19:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:31.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:31.327 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:31.427 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:31.457 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:31 smithi090 ceph-mon[44434]: 4.8 scrub starts 2024-05-15T23:19:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:31 smithi090 ceph-mon[44434]: 4.8 scrub ok 2024-05-15T23:19:31.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:31 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:19:31.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:31 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:19:31.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2785296192' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:19:31.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:31 smithi090 ceph-mon[44434]: osdmap e369: 12 total, 11 up, 10 in 2024-05-15T23:19:31.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:31 smithi090 ceph-mon[44434]: 4.7 deep-scrub starts 2024-05-15T23:19:31.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:31 smithi090 ceph-mon[44434]: 4.7 deep-scrub ok 2024-05-15T23:19:31.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:31 smithi090 ceph-mon[44434]: pgmap v1015: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 501 op/s 2024-05-15T23:19:31.581 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:31.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:31.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:31.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:31.717 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:31.738 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:31.742 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:31.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:31.872 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:31.973 INFO:teuthology.orchestra.run.smithi084.stdout: marking collection for removal 2024-05-15T23:19:31.973 INFO:teuthology.orchestra.run.smithi084.stdout:setting '_remove' omap key 2024-05-15T23:19:31.980 INFO:teuthology.orchestra.run.smithi084.stdout:finish_remove_pgs 4.b_head removing 4.b 2024-05-15T23:19:32.003 INFO:teuthology.orchestra.run.smithi084.stdout:Remove successful 2024-05-15T23:19:32.028 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:19:32.028 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:19:32.028 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:19:32.028 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:19:32.028 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.028 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278178 4.d 4:b4a56e92:::benchmark_data_smithi090_104448_object104940:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:19:32.028 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.204118+0000", 2024-05-15T23:19:32.028 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 87.820543528000002, 2024-05-15T23:19:32.028 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22432159900000001, 2024-05-15T23:19:32.028 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.028 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.028 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.028 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.029 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.029 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.029 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278178 2024-05-15T23:19:32.029 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.029 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.029 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.029 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.029 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.204118+0000", 2024-05-15T23:19:32.029 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.029 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.029 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.029 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.029 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.204118+0000", 2024-05-15T23:19:32.029 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.029 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.030 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.030 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.030 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.204118+0000", 2024-05-15T23:19:32.030 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.13e-07 2024-05-15T23:19:32.030 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.030 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.030 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.030 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.204121+0000", 2024-05-15T23:19:32.030 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.367e-06 2024-05-15T23:19:32.030 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.030 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.030 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.030 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.204125+0000", 2024-05-15T23:19:32.030 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5000000000000001e-06 2024-05-15T23:19:32.031 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.031 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.031 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.031 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.204130+0000", 2024-05-15T23:19:32.031 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.848e-06 2024-05-15T23:19:32.031 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.031 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.031 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.031 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.208629+0000", 2024-05-15T23:19:32.031 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004498747 2024-05-15T23:19:32.031 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.031 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.031 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.032 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.208753+0000", 2024-05-15T23:19:32.032 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000124733 2024-05-15T23:19:32.032 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.032 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.032 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:19:32.032 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.208844+0000", 2024-05-15T23:19:32.032 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0866999999999996e-05 2024-05-15T23:19:32.032 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.032 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.032 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.032 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.217585+0000", 2024-05-15T23:19:32.032 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008741034 2024-05-15T23:19:32.032 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.032 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.032 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.033 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.428366+0000", 2024-05-15T23:19:32.033 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21078121999999999 2024-05-15T23:19:32.033 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.033 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.033 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.033 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.428399+0000", 2024-05-15T23:19:32.033 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2113999999999998e-05 2024-05-15T23:19:32.033 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.033 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.033 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.033 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.428439+0000", 2024-05-15T23:19:32.033 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0556000000000003e-05 2024-05-15T23:19:32.033 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.033 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.034 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.034 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.034 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.034 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278185 4.d 4:bd70842c:::benchmark_data_smithi090_104448_object104947:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:19:32.034 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.209728+0000", 2024-05-15T23:19:32.034 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 87.814933353000001, 2024-05-15T23:19:32.034 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21882157399999999, 2024-05-15T23:19:32.034 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.034 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.034 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.034 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.034 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.034 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.034 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278185 2024-05-15T23:19:32.034 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.209728+0000", 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.209728+0000", 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.209730+0000", 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.86e-06 2024-05-15T23:19:32.035 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.209735+0000", 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4380000000000003e-06 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.209748+0000", 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3152999999999999e-05 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.209756+0000", 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6559999999999992e-06 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.036 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.212660+0000", 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029041710000000001 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.212788+0000", 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000128169 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.212901+0000", 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000113251 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.037 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.217621+0000", 2024-05-15T23:19:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0047191280000000004 2024-05-15T23:19:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.428471+0000", 2024-05-15T23:19:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21085094200000001 2024-05-15T23:19:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.428509+0000", 2024-05-15T23:19:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7642999999999999e-05 2024-05-15T23:19:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.038 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.428549+0000", 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0163000000000002e-05 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278196 4.d 4:b76d4705:::benchmark_data_smithi090_104448_object104958:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.226357+0000", 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 87.798303747000006, 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208817265, 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.039 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278196 2024-05-15T23:19:32.040 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.040 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.040 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.040 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.040 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226357+0000", 2024-05-15T23:19:32.040 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.040 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.040 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.040 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.040 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226357+0000", 2024-05-15T23:19:32.040 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.040 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.040 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.040 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.040 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226360+0000", 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3420000000000002e-06 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226368+0000", 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2900000000000002e-06 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226382+0000", 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3608000000000001e-05 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226390+0000", 2024-05-15T23:19:32.041 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1149999999999994e-06 2024-05-15T23:19:32.042 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.042 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.042 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.042 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226440+0000", 2024-05-15T23:19:32.042 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0476000000000003e-05 2024-05-15T23:19:32.042 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.042 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.042 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.042 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226533+0000", 2024-05-15T23:19:32.042 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3282999999999994e-05 2024-05-15T23:19:32.042 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.042 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.042 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:19:32.042 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226648+0000", 2024-05-15T23:19:32.042 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000114387 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.236296+0000", 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096484080000000007 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.435080+0000", 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19878395500000001 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.435121+0000", 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0774e-05 2024-05-15T23:19:32.043 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.435175+0000", 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3627e-05 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278201 4.d 4:b6101031:::benchmark_data_smithi090_104448_object104963:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.226528+0000", 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 87.798133160000006, 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208752454, 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.044 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.045 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.045 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.045 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.045 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278201 2024-05-15T23:19:32.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.045 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.045 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226528+0000", 2024-05-15T23:19:32.045 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.045 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226528+0000", 2024-05-15T23:19:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226529+0000", 2024-05-15T23:19:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8199999999999999e-07 2024-05-15T23:19:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226532+0000", 2024-05-15T23:19:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7519999999999998e-06 2024-05-15T23:19:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226539+0000", 2024-05-15T23:19:32.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4169999999999997e-06 2024-05-15T23:19:32.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226544+0000", 2024-05-15T23:19:32.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6289999999999998e-06 2024-05-15T23:19:32.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.231428+0000", 2024-05-15T23:19:32.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0048830419999999998 2024-05-15T23:19:32.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.231522+0000", 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4277999999999994e-05 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.231615+0000", 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3298e-05 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.237393+0000", 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0057775719999999999 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.435216+0000", 2024-05-15T23:19:32.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.197823205 2024-05-15T23:19:32.049 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.049 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.049 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.435244+0000", 2024-05-15T23:19:32.049 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8017e-05 2024-05-15T23:19:32.049 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.049 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.049 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.435280+0000", 2024-05-15T23:19:32.049 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6461999999999998e-05 2024-05-15T23:19:32.049 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.049 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.049 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.049 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.050 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278218 4.d 4:b6a9e0eb:::benchmark_data_smithi090_104448_object104980:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:19:32.050 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.238248+0000", 2024-05-15T23:19:32.050 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 87.786412566999999, 2024-05-15T23:19:32.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.201633439, 2024-05-15T23:19:32.050 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.050 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.050 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.050 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.050 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.050 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.050 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278218 2024-05-15T23:19:32.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.050 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.050 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238248+0000", 2024-05-15T23:19:32.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238248+0000", 2024-05-15T23:19:32.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238250+0000", 2024-05-15T23:19:32.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3990000000000001e-06 2024-05-15T23:19:32.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238254+0000", 2024-05-15T23:19:32.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4139999999999996e-06 2024-05-15T23:19:32.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238263+0000", 2024-05-15T23:19:32.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0880000000000001e-06 2024-05-15T23:19:32.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238269+0000", 2024-05-15T23:19:32.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7309999999999998e-06 2024-05-15T23:19:32.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238323+0000", 2024-05-15T23:19:32.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4277999999999998e-05 2024-05-15T23:19:32.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238456+0000", 2024-05-15T23:19:32.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013282999999999999 2024-05-15T23:19:32.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:19:32.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238550+0000", 2024-05-15T23:19:32.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3916e-05 2024-05-15T23:19:32.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.243359+0000", 2024-05-15T23:19:32.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0048091829999999999 2024-05-15T23:19:32.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.054 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.054 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.439791+0000", 2024-05-15T23:19:32.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19643179 2024-05-15T23:19:32.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.054 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.054 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.439835+0000", 2024-05-15T23:19:32.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3438e-05 2024-05-15T23:19:32.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.439882+0000", 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7372e-05 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278247 4.d 4:bf4844ca:::benchmark_data_smithi090_104448_object105009:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.269137+0000", 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 87.755524195000007, 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17087540700000001, 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.055 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.056 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.056 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.056 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.056 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278247 2024-05-15T23:19:32.056 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.056 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.056 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.056 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.056 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269137+0000", 2024-05-15T23:19:32.056 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.056 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.056 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.056 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.056 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269137+0000", 2024-05-15T23:19:32.056 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269139+0000", 2024-05-15T23:19:32.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.1390000000000002e-06 2024-05-15T23:19:32.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269145+0000", 2024-05-15T23:19:32.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.711e-06 2024-05-15T23:19:32.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269158+0000", 2024-05-15T23:19:32.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3174e-05 2024-05-15T23:19:32.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269167+0000", 2024-05-15T23:19:32.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6249999999999996e-06 2024-05-15T23:19:32.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269226+0000", 2024-05-15T23:19:32.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9342999999999997e-05 2024-05-15T23:19:32.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269313+0000", 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7183999999999997e-05 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269432+0000", 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000118987 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.274514+0000", 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005082128 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.439926+0000", 2024-05-15T23:19:32.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16541231200000001 2024-05-15T23:19:32.060 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.060 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.060 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.060 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.439978+0000", 2024-05-15T23:19:32.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1944999999999998e-05 2024-05-15T23:19:32.060 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.060 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.060 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.060 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.440012+0000", 2024-05-15T23:19:32.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3859000000000002e-05 2024-05-15T23:19:32.060 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.060 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.060 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.060 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278301 4.d 4:b78c14fc:::benchmark_data_smithi090_104448_object105063:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.324748+0000", 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 87.699913338000002, 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12313596, 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278301 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.061 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.062 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324748+0000", 2024-05-15T23:19:32.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.062 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.062 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.062 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.062 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324748+0000", 2024-05-15T23:19:32.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.062 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.062 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.062 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.062 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324749+0000", 2024-05-15T23:19:32.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4419999999999999e-06 2024-05-15T23:19:32.062 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.062 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324754+0000", 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6369999999999998e-06 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324764+0000", 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0465999999999999e-05 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324772+0000", 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4649999999999997e-06 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324833+0000", 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0776999999999998e-05 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324973+0000", 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140378 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.325060+0000", 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6712000000000006e-05 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.328886+0000", 2024-05-15T23:19:32.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0038261649999999999 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.447814+0000", 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11892828699999999 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.447845+0000", 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0970999999999999e-05 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.447884+0000", 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8659999999999999e-05 2024-05-15T23:19:32.065 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278311 4.d 4:b37c2762:::benchmark_data_smithi090_104448_object105073:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.338775+0000", 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 87.685886302, 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11404146900000001, 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278311 2024-05-15T23:19:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338775+0000", 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338775+0000", 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338777+0000", 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8729999999999999e-06 2024-05-15T23:19:32.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338782+0000", 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4389999999999999e-06 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338793+0000", 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1182000000000001e-05 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338800+0000", 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4459999999999998e-06 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338887+0000", 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7468999999999999e-05 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338969+0000", 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1698000000000003e-05 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.339040+0000", 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1340000000000005e-05 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.347909+0000", 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0088689029999999992 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.452710+0000", 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.104800421 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.452763+0000", 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3227999999999999e-05 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.452816+0000", 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3470000000000001e-05 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278317 4.d 4:b931cefd:::benchmark_data_smithi090_104448_object105079:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.340282+0000", 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 87.684379406000005, 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.112684298, 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.071 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.072 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278317 2024-05-15T23:19:32.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.072 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.340282+0000", 2024-05-15T23:19:32.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.340282+0000", 2024-05-15T23:19:32.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.340283+0000", 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5689999999999999e-06 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.340289+0000", 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8039999999999998e-06 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.340301+0000", 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1783999999999999e-05 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.340308+0000", 2024-05-15T23:19:32.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5750000000000003e-06 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.343824+0000", 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0035159119999999999 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.343942+0000", 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000118126 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.344062+0000", 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000119957 2024-05-15T23:19:32.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.075 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.075 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.075 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.348054+0000", 2024-05-15T23:19:32.075 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003991199 2024-05-15T23:19:32.075 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.075 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.075 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.075 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.452864+0000", 2024-05-15T23:19:32.075 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10481017300000001 2024-05-15T23:19:32.075 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.075 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.075 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.075 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.452916+0000", 2024-05-15T23:19:32.075 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1983e-05 2024-05-15T23:19:32.075 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.076 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.076 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.452966+0000", 2024-05-15T23:19:32.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0216000000000002e-05 2024-05-15T23:19:32.076 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.076 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.076 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.076 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.076 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278324 4.d 4:b32aacc2:::benchmark_data_smithi090_104448_object105086:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:19:32.076 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.349495+0000", 2024-05-15T23:19:32.076 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 87.675166035000004, 2024-05-15T23:19:32.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11114384200000001, 2024-05-15T23:19:32.076 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278324 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349495+0000", 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349495+0000", 2024-05-15T23:19:32.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349496+0000", 2024-05-15T23:19:32.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5700000000000002e-07 2024-05-15T23:19:32.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349499+0000", 2024-05-15T23:19:32.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8779999999999998e-06 2024-05-15T23:19:32.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349504+0000", 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.519e-06 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349509+0000", 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6609999999999999e-06 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349570+0000", 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1458999999999996e-05 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349647+0000", 2024-05-15T23:19:32.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6907000000000001e-05 2024-05-15T23:19:32.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.080 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.080 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:19:32.080 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349717+0000", 2024-05-15T23:19:32.080 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9311000000000001e-05 2024-05-15T23:19:32.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.080 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.080 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.080 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.353196+0000", 2024-05-15T23:19:32.080 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034794919999999998 2024-05-15T23:19:32.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.080 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.080 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.080 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.460561+0000", 2024-05-15T23:19:32.080 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107364949 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.460591+0000", 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9745999999999999e-05 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.460639+0000", 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8162999999999999e-05 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.081 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278331 4.d 4:be6484c2:::benchmark_data_smithi090_104448_object105093:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:19:32.082 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.359850+0000", 2024-05-15T23:19:32.082 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 87.664811026999999, 2024-05-15T23:19:32.082 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11042561400000001, 2024-05-15T23:19:32.082 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.082 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.082 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.082 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.082 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.082 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.082 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278331 2024-05-15T23:19:32.082 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.082 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.082 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.082 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.082 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359850+0000", 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359850+0000", 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359851+0000", 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.344e-06 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359856+0000", 2024-05-15T23:19:32.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2629999999999997e-06 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359864+0000", 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1249999999999993e-06 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359869+0000", 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2650000000000004e-06 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359908+0000", 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8668000000000003e-05 2024-05-15T23:19:32.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359986+0000", 2024-05-15T23:19:32.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7886000000000006e-05 2024-05-15T23:19:32.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:19:32.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.360051+0000", 2024-05-15T23:19:32.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5531999999999995e-05 2024-05-15T23:19:32.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.364149+0000", 2024-05-15T23:19:32.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0040982019999999996 2024-05-15T23:19:32.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.470176+0000", 2024-05-15T23:19:32.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.106026286 2024-05-15T23:19:32.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.470229+0000", 2024-05-15T23:19:32.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3693000000000002e-05 2024-05-15T23:19:32.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.470276+0000", 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6350000000000002e-05 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278342 4.d 4:b6201b5e:::benchmark_data_smithi090_104448_object105104:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.371607+0000", 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 87.653053744000005, 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11209290099999999, 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.087 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.088 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278342 2024-05-15T23:19:32.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.088 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371607+0000", 2024-05-15T23:19:32.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371607+0000", 2024-05-15T23:19:32.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.089 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371609+0000", 2024-05-15T23:19:32.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3430000000000001e-06 2024-05-15T23:19:32.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.089 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.089 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.089 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371613+0000", 2024-05-15T23:19:32.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6469999999999997e-06 2024-05-15T23:19:32.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.089 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.089 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.089 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371623+0000", 2024-05-15T23:19:32.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0131e-05 2024-05-15T23:19:32.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.089 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.089 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371631+0000", 2024-05-15T23:19:32.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9789999999999993e-06 2024-05-15T23:19:32.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371694+0000", 2024-05-15T23:19:32.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2917999999999996e-05 2024-05-15T23:19:32.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371811+0000", 2024-05-15T23:19:32.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000116381 2024-05-15T23:19:32.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:19:32.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371904+0000", 2024-05-15T23:19:32.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3083000000000003e-05 2024-05-15T23:19:32.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.375745+0000", 2024-05-15T23:19:32.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00384074 2024-05-15T23:19:32.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.483632+0000", 2024-05-15T23:19:32.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107886998 2024-05-15T23:19:32.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.483664+0000", 2024-05-15T23:19:32.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2793000000000003e-05 2024-05-15T23:19:32.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.483700+0000", 2024-05-15T23:19:32.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5887999999999999e-05 2024-05-15T23:19:32.092 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.092 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.092 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.092 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278342 4.d 4:b6201b5e:::benchmark_data_smithi090_104448_object105104:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e345)", 2024-05-15T23:19:32.092 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.385798+0000", 2024-05-15T23:19:32.092 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 87.638863256999997, 2024-05-15T23:19:32.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.097873819000000001, 2024-05-15T23:19:32.093 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.093 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.093 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "waiting for ondisk", 2024-05-15T23:19:32.093 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.093 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.093 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.093 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278342 2024-05-15T23:19:32.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.093 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385798+0000", 2024-05-15T23:19:32.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385798+0000", 2024-05-15T23:19:32.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385798+0000", 2024-05-15T23:19:32.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0999999999999999e-07 2024-05-15T23:19:32.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385800+0000", 2024-05-15T23:19:32.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.1880000000000001e-06 2024-05-15T23:19:32.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.095 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.095 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385807+0000", 2024-05-15T23:19:32.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3169999999999998e-06 2024-05-15T23:19:32.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.095 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.095 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385815+0000", 2024-05-15T23:19:32.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1839999999999999e-06 2024-05-15T23:19:32.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.095 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.386530+0000", 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071482399999999997 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for ondisk", 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.386557+0000", 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6783000000000001e-05 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.483672+0000", 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.097115013 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.097 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278350 4.d 4:b848d388:::benchmark_data_smithi090_104448_object105112:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e345)", 2024-05-15T23:19:32.097 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.385820+0000", 2024-05-15T23:19:32.097 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 87.638840900000005, 2024-05-15T23:19:32.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.097980265999999996, 2024-05-15T23:19:32.097 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.097 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.097 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.097 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.097 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.097 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.097 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278350 2024-05-15T23:19:32.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.097 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.098 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385820+0000", 2024-05-15T23:19:32.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.098 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.098 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385820+0000", 2024-05-15T23:19:32.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.098 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.098 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385821+0000", 2024-05-15T23:19:32.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6400000000000003e-07 2024-05-15T23:19:32.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385823+0000", 2024-05-15T23:19:32.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.255e-06 2024-05-15T23:19:32.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385826+0000", 2024-05-15T23:19:32.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5109999999999999e-06 2024-05-15T23:19:32.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385829+0000", 2024-05-15T23:19:32.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8040000000000002e-06 2024-05-15T23:19:32.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.386580+0000", 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075106100000000002 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.386697+0000", 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011701099999999999 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.386827+0000", 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000130004 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.391672+0000", 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0048445520000000002 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.483736+0000", 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092064423000000006 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.483771+0000", 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4900999999999997e-05 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.483800+0000", 2024-05-15T23:19:32.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9280000000000001e-05 2024-05-15T23:19:32.102 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.102 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.102 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.102 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:331989 4.c 4:3f19a99f:::benchmark_data_smithi090_104448_object158751:head [delete] snapc 0=[] ondisk+write+known_if_redirected e366)", 2024-05-15T23:19:32.102 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:19:11.540074+0000", 2024-05-15T23:19:32.102 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 20.484587446999999, 2024-05-15T23:19:32.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046779924000000001, 2024-05-15T23:19:32.102 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.102 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.102 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.102 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.102 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.103 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.103 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 331989 2024-05-15T23:19:32.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.103 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.540074+0000", 2024-05-15T23:19:32.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.540074+0000", 2024-05-15T23:19:32.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.540075+0000", 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.658e-06 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.540081+0000", 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0589999999999999e-06 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.540091+0000", 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.73e-06 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.540099+0000", 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6389999999999992e-06 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.540165+0000", 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5814000000000004e-05 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.540304+0000", 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013974699999999999 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:19:32.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.540428+0000", 2024-05-15T23:19:32.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012384700000000001 2024-05-15T23:19:32.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.549489+0000", 2024-05-15T23:19:32.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0090611900000000002 2024-05-15T23:19:32.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.586771+0000", 2024-05-15T23:19:32.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.037281485000000003 2024-05-15T23:19:32.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.586816+0000", 2024-05-15T23:19:32.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4783999999999997e-05 2024-05-15T23:19:32.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.586854+0000", 2024-05-15T23:19:32.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7970999999999999e-05 2024-05-15T23:19:32.107 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.107 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.107 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.107 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:331990 4.c 4:37ace327:::benchmark_data_smithi090_104448_object158752:head [delete] snapc 0=[] ondisk+write+known_if_redirected e366)", 2024-05-15T23:19:32.107 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:19:11.540127+0000", 2024-05-15T23:19:32.107 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 20.484534309000001, 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046880667000000001, 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 331990 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.540127+0000", 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.540127+0000", 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.540127+0000", 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9400000000000005e-07 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.540130+0000", 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3e-06 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.540145+0000", 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4859e-05 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.540155+0000", 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0614999999999999e-05 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.545439+0000", 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0052834249999999996 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.110 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.545554+0000", 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000114895 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.545669+0000", 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011578099999999999 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.553475+0000", 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.007805485 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.586912+0000", 2024-05-15T23:19:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.033437675 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.586959+0000", 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6653999999999998e-05 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.587007+0000", 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8284000000000001e-05 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:331997 4.c 4:34f3f745:::benchmark_data_smithi090_104448_object158759:head [delete] snapc 0=[] ondisk+write+known_if_redirected e366)", 2024-05-15T23:19:32.112 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:19:11.546820+0000", 2024-05-15T23:19:32.113 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 20.477840827000001, 2024-05-15T23:19:32.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.040302452000000002, 2024-05-15T23:19:32.113 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:32.113 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:32.113 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:32.113 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:32.113 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:32.113 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:32.113 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 331997 2024-05-15T23:19:32.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.113 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:32.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:32.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.546820+0000", 2024-05-15T23:19:32.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:32.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.546820+0000", 2024-05-15T23:19:32.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:32.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:32.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.546822+0000", 2024-05-15T23:19:32.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.1629999999999999e-06 2024-05-15T23:19:32.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:32.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.546829+0000", 2024-05-15T23:19:32.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2230000000000004e-06 2024-05-15T23:19:32.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.115 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:32.115 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.546841+0000", 2024-05-15T23:19:32.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1939e-05 2024-05-15T23:19:32.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.115 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:32.115 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.546848+0000", 2024-05-15T23:19:32.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9109999999999993e-06 2024-05-15T23:19:32.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.115 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:32.115 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.549519+0000", 2024-05-15T23:19:32.116 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0026705159999999999 2024-05-15T23:19:32.116 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.116 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.116 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:32.116 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.549603+0000", 2024-05-15T23:19:32.116 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.4246000000000006e-05 2024-05-15T23:19:32.116 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.116 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.116 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:19:32.116 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.549677+0000", 2024-05-15T23:19:32.116 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3479000000000006e-05 2024-05-15T23:19:32.116 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.116 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.116 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:32.116 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.553604+0000", 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0039272079999999997 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.587055+0000", 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.033450756999999998 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.587084+0000", 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9456999999999999e-05 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:19:11.587123+0000", 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8553000000000002e-05 2024-05-15T23:19:32.117 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.118 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.118 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.118 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:32.118 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:32.118 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:19:32.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:32 smithi084 ceph-mon[39784]: 4.4 scrub starts 2024-05-15T23:19:32.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:32 smithi084 ceph-mon[39784]: 4.4 scrub ok 2024-05-15T23:19:32.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:32 smithi088 ceph-mon[44699]: 4.4 scrub starts 2024-05-15T23:19:32.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:32 smithi088 ceph-mon[44699]: 4.4 scrub ok 2024-05-15T23:19:32.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:32 smithi090 ceph-mon[44434]: 4.4 scrub starts 2024-05-15T23:19:32.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:32 smithi090 ceph-mon[44434]: 4.4 scrub ok 2024-05-15T23:19:32.722 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:32.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:32.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:32.731 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T23:19:32.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:32.853 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:32.906 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:32.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:32.991 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:19:33.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:33.024 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:19:33.043 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:33.064 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:33.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:33.067 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:33.168 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:33.198 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:33.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:33 smithi090 ceph-mon[44434]: pgmap v1016: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 262 op/s 2024-05-15T23:19:33.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:33 smithi084 ceph-mon[39784]: pgmap v1016: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 262 op/s 2024-05-15T23:19:33.442 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:33 smithi088 ceph-mon[44699]: pgmap v1016: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 262 op/s 2024-05-15T23:19:33.890 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:19:33.891 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:19:33.891 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:19:33.891 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:19:33.891 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:19:33.891 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:19:33.891 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:19:33.899 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:19:33.918 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:19:33.939 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:19:33.959 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:19:33.959 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-05-15T23:19:33.959 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:19:33.959 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:19:33.959 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:19:33.959 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:19:33.959 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:19:33.959 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:19:33.959 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:19:33.959 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:19:33.960 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:19:33.960 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:19:33.960 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:19:34.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:34 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/15725216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:34.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:34.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:34.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:34 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/15725216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:34.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:34.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:34.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:34 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/15725216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:34.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:34.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:34.741 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:19:34.741 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:19:34.748 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:34.751 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:19:34.752 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:19:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:34.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:34.853 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:19:34.882 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:34.902 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:19:34.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:34.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:19:35.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:35.040 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:35.060 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:35.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:35.086 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_2' removed 2024-05-15T23:19:35.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:35.194 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:35.214 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:35.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:35.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:35.256 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -n osd.0 --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.b.0 2024-05-15T23:19:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:35 smithi090 ceph-mon[44434]: 4.b scrub starts 2024-05-15T23:19:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:35 smithi090 ceph-mon[44434]: 4.b scrub ok 2024-05-15T23:19:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:35 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3746726853' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:19:35.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:35 smithi090 ceph-mon[44434]: pgmap v1017: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 187 op/s 2024-05-15T23:19:35.319 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:35.346 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.346 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.346 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.346 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.346 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.346 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.346 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.346 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.346 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.346 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:35 smithi088 ceph-mon[44699]: 4.b scrub starts 2024-05-15T23:19:35.346 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:35 smithi088 ceph-mon[44699]: 4.b scrub ok 2024-05-15T23:19:35.346 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:35 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3746726853' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:19:35.347 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:35 smithi088 ceph-mon[44699]: pgmap v1017: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 187 op/s 2024-05-15T23:19:35.349 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:35.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:35 smithi084 ceph-mon[39784]: 4.b scrub starts 2024-05-15T23:19:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:35 smithi084 ceph-mon[39784]: 4.b scrub ok 2024-05-15T23:19:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:35 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3746726853' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:19:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:35 smithi084 ceph-mon[39784]: pgmap v1017: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 187 op/s 2024-05-15T23:19:35.498 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:35.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:35.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:35.503 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T23:19:35.603 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:19:35.630 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:35.801 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:19:36.207 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T23:19:36.208 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T23:19:36.208 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-05-15T23:19:36.208 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-05-15T23:19:36.216 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/osd.0/config 2024-05-15T23:19:36.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3746726853' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:19:36.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:36 smithi088 ceph-mon[44699]: osdmap e370: 12 total, 11 up, 10 in 2024-05-15T23:19:36.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3746726853' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:19:36.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:36 smithi084 ceph-mon[39784]: osdmap e370: 12 total, 11 up, 10 in 2024-05-15T23:19:36.478 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:19:36.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:36.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:19:36.483 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3746726853' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:19:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:36 smithi090 ceph-mon[44434]: osdmap e370: 12 total, 11 up, 10 in 2024-05-15T23:19:36.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:36.615 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:36.637 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:36.641 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:36.742 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:36.752 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:19:36.768 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:37.119 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:37.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:37.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:37.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:37.256 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:37.278 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:37.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:37.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:37.412 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:37.433 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:37.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:37.537 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:37.585 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:37.588 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:37.644 INFO:teuthology.orchestra.run.smithi084.stderr:Creating an OSD daemon form without an OSD FSID value 2024-05-15T23:19:37.689 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:37.714 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:19:37.714 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:19:37.720 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:37.740 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:37.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:37.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:37.844 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:19:37.871 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:38.019 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:19:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:38.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:19:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:37 smithi084 ceph-mon[39784]: pgmap v1019: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:38.123 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:38.150 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:38.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:37 smithi090 ceph-mon[44434]: pgmap v1019: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:37 smithi088 ceph-mon[44699]: pgmap v1019: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:38.398 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:38.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:38.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:38.401 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T23:19:38.407 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:38.502 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:19:38.528 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:38.546 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:19:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:38.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:19:38.651 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:38.678 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:38.698 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:38.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:38.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:38.709 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:19:38.801 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:38.828 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:38.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:38.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:39.079 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:39.109 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:39.147 INFO:teuthology.orchestra.run.smithi084.stdout:get_pg_num_history pg_num_history pg_num_history(e355 pg_nums {1={14=1},4={283=16}} deleted_pools ) 2024-05-15T23:19:39.147 INFO:teuthology.orchestra.run.smithi084.stdout:Importing pgid 4.b 2024-05-15T23:19:39.157 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d012317e:::benchmark_data_smithi090_104448_object158482:head# 2024-05-15T23:19:39.157 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.157 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0129e53:::benchmark_data_smithi090_104448_object171208:head# 2024-05-15T23:19:39.157 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.157 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d01799a4:::benchmark_data_smithi090_104448_object170254:head# 2024-05-15T23:19:39.157 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.157 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0199fca:::benchmark_data_smithi090_104448_object166694:head# 2024-05-15T23:19:39.157 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.157 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d01c2eba:::benchmark_data_smithi090_104448_object173055:head# 2024-05-15T23:19:39.157 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.157 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0228b4f:::benchmark_data_smithi090_104448_object171832:head# 2024-05-15T23:19:39.158 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.159 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d039794a:::benchmark_data_smithi090_104448_object168370:head# 2024-05-15T23:19:39.159 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.159 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d041cc54:::benchmark_data_smithi090_104448_object162077:head# 2024-05-15T23:19:39.159 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.159 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d042ed42:::benchmark_data_smithi090_104448_object164503:head# 2024-05-15T23:19:39.159 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.159 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d048394e:::benchmark_data_smithi090_104448_object169206:head# 2024-05-15T23:19:39.159 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.159 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d04bd93d:::benchmark_data_smithi090_104448_object162358:head# 2024-05-15T23:19:39.159 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.160 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d04ebc3d:::benchmark_data_smithi090_104448_object161958:head# 2024-05-15T23:19:39.165 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.165 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d056fa0f:::benchmark_data_smithi090_104448_object159835:head# 2024-05-15T23:19:39.166 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.166 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0570e84:::benchmark_data_smithi090_104448_object157201:head# 2024-05-15T23:19:39.166 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.166 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0583124:::benchmark_data_smithi090_104448_object162507:head# 2024-05-15T23:19:39.166 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.166 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d05a82f5:::benchmark_data_smithi090_104448_object167662:head# 2024-05-15T23:19:39.166 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.166 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0693e91:::benchmark_data_smithi090_104448_object167715:head# 2024-05-15T23:19:39.166 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.166 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d06cf267:::benchmark_data_smithi090_104448_object171642:head# 2024-05-15T23:19:39.166 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.166 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d077e1bb:::benchmark_data_smithi090_104448_object158950:head# 2024-05-15T23:19:39.166 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.167 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0839531:::benchmark_data_smithi090_104448_object162232:head# 2024-05-15T23:19:39.167 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.167 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d08665b5:::benchmark_data_smithi090_104448_object170017:head# 2024-05-15T23:19:39.167 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.167 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0893b05:::benchmark_data_smithi090_104448_object159959:head# 2024-05-15T23:19:39.167 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.167 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d08a3c96:::benchmark_data_smithi090_104448_object164396:head# 2024-05-15T23:19:39.167 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.167 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d090e161:::benchmark_data_smithi090_104448_object164996:head# 2024-05-15T23:19:39.167 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.167 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0938174:::benchmark_data_smithi090_104448_object161965:head# 2024-05-15T23:19:39.167 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.167 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0961fe2:::benchmark_data_smithi090_104448_object160775:head# 2024-05-15T23:19:39.167 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.168 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d097132d:::benchmark_data_smithi090_104448_object168619:head# 2024-05-15T23:19:39.168 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.168 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0a3a586:::benchmark_data_smithi090_104448_object167456:head# 2024-05-15T23:19:39.168 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.168 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0ae2ef2:::benchmark_data_smithi090_104448_object156795:head# 2024-05-15T23:19:39.168 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.168 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0b5b01f:::benchmark_data_smithi090_104448_object162424:head# 2024-05-15T23:19:39.168 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.168 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0b7dd29:::benchmark_data_smithi090_104448_object168193:head# 2024-05-15T23:19:39.168 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.168 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0ba4865:::benchmark_data_smithi090_104448_object170236:head# 2024-05-15T23:19:39.168 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.168 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0c0a793:::benchmark_data_smithi090_104448_object162087:head# 2024-05-15T23:19:39.168 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.168 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0c1412a:::benchmark_data_smithi090_104448_object165902:head# 2024-05-15T23:19:39.169 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.169 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0c2a2bd:::benchmark_data_smithi090_104448_object169146:head# 2024-05-15T23:19:39.169 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.169 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0d1eb55:::benchmark_data_smithi090_104448_object163076:head# 2024-05-15T23:19:39.169 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.169 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0d451e6:::benchmark_data_smithi090_104448_object166219:head# 2024-05-15T23:19:39.169 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.169 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0d632e9:::benchmark_data_smithi090_104448_object161322:head# 2024-05-15T23:19:39.169 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.169 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0d71efd:::benchmark_data_smithi090_104448_object167586:head# 2024-05-15T23:19:39.169 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.169 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0d836eb:::benchmark_data_smithi090_104448_object171791:head# 2024-05-15T23:19:39.169 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.169 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0d8f85c:::benchmark_data_smithi090_104448_object172772:head# 2024-05-15T23:19:39.170 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.170 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0d9e179:::benchmark_data_smithi090_104448_object158467:head# 2024-05-15T23:19:39.170 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.170 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0dc8ddf:::benchmark_data_smithi090_104448_object156993:head# 2024-05-15T23:19:39.170 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.170 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0e3f610:::benchmark_data_smithi090_104448_object162718:head# 2024-05-15T23:19:39.170 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.170 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0e5e66a:::benchmark_data_smithi090_104448_object168295:head# 2024-05-15T23:19:39.170 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.170 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0e63550:::benchmark_data_smithi090_104448_object166312:head# 2024-05-15T23:19:39.170 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.177 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0ed8768:::benchmark_data_smithi090_104448_object171929:head# 2024-05-15T23:19:39.177 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.177 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0edc9e5:::benchmark_data_smithi090_104448_object160465:head# 2024-05-15T23:19:39.177 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.177 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0eeb3e0:::benchmark_data_smithi090_104448_object168084:head# 2024-05-15T23:19:39.177 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.177 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0f12942:::benchmark_data_smithi090_104448_object161880:head# 2024-05-15T23:19:39.177 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.177 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d0f75f82:::benchmark_data_smithi090_104448_object165506:head# 2024-05-15T23:19:39.177 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.177 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1027389:::benchmark_data_smithi090_104448_object165741:head# 2024-05-15T23:19:39.178 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.178 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1097834:::benchmark_data_smithi090_104448_object168326:head# 2024-05-15T23:19:39.178 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.178 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d11472fe:::benchmark_data_smithi090_104448_object166226:head# 2024-05-15T23:19:39.178 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.178 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1149b84:::benchmark_data_smithi090_104448_object165733:head# 2024-05-15T23:19:39.178 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.178 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d115d225:::benchmark_data_smithi090_104448_object157765:head# 2024-05-15T23:19:39.178 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.178 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d116d390:::benchmark_data_smithi090_104448_object167799:head# 2024-05-15T23:19:39.178 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.178 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1175bda:::benchmark_data_smithi090_104448_object168282:head# 2024-05-15T23:19:39.178 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.178 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1192fa5:::benchmark_data_smithi090_104448_object157520:head# 2024-05-15T23:19:39.179 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.179 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d11c54b3:::benchmark_data_smithi090_104448_object159274:head# 2024-05-15T23:19:39.179 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.179 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d11f79e1:::benchmark_data_smithi090_104448_object171402:head# 2024-05-15T23:19:39.179 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.179 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d121acda:::benchmark_data_smithi090_104448_object168195:head# 2024-05-15T23:19:39.179 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.179 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d121c251:::benchmark_data_smithi090_104448_object167519:head# 2024-05-15T23:19:39.179 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.179 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1258048:::benchmark_data_smithi090_104448_object167134:head# 2024-05-15T23:19:39.179 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.179 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1282744:::benchmark_data_smithi090_104448_object159399:head# 2024-05-15T23:19:39.179 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.179 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d12bcf63:::benchmark_data_smithi090_104448_object170235:head# 2024-05-15T23:19:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1320bf1:::benchmark_data_smithi090_104448_object168129:head# 2024-05-15T23:19:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d13507c5:::benchmark_data_smithi090_104448_object171158:head# 2024-05-15T23:19:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d135c7fc:::benchmark_data_smithi090_104448_object165857:head# 2024-05-15T23:19:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d13deac4:::benchmark_data_smithi090_104448_object168338:head# 2024-05-15T23:19:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d141d64d:::benchmark_data_smithi090_104448_object169258:head# 2024-05-15T23:19:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1429e38:::benchmark_data_smithi090_104448_object161825:head# 2024-05-15T23:19:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d145c5a7:::benchmark_data_smithi090_104448_object160365:head# 2024-05-15T23:19:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.181 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d146563a:::benchmark_data_smithi090_104448_object165048:head# 2024-05-15T23:19:39.181 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.181 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1482457:::benchmark_data_smithi090_104448_object164923:head# 2024-05-15T23:19:39.181 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.181 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d14910b7:::benchmark_data_smithi090_104448_object157570:head# 2024-05-15T23:19:39.181 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.181 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d14c5e3f:::benchmark_data_smithi090_104448_object168817:head# 2024-05-15T23:19:39.181 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.181 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d14dbb44:::benchmark_data_smithi090_104448_object172364:head# 2024-05-15T23:19:39.181 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.181 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1526fad:::benchmark_data_smithi090_104448_object159713:head# 2024-05-15T23:19:39.181 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.181 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1588610:::benchmark_data_smithi090_104448_object170810:head# 2024-05-15T23:19:39.181 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.182 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d15a5c5a:::benchmark_data_smithi090_104448_object159916:head# 2024-05-15T23:19:39.182 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.182 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d15b8033:::benchmark_data_smithi090_104448_object160042:head# 2024-05-15T23:19:39.182 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.182 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d15d549d:::benchmark_data_smithi090_104448_object164987:head# 2024-05-15T23:19:39.182 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.187 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d168249a:::benchmark_data_smithi090_104448_object172581:head# 2024-05-15T23:19:39.187 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.187 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d16c5e0f:::benchmark_data_smithi090_104448_object170002:head# 2024-05-15T23:19:39.187 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.187 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d173044e:::benchmark_data_smithi090_104448_object167910:head# 2024-05-15T23:19:39.187 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.187 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d177f22b:::benchmark_data_smithi090_104448_object158736:head# 2024-05-15T23:19:39.187 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.187 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1799a91:::benchmark_data_smithi090_104448_object163736:head# 2024-05-15T23:19:39.187 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.187 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d17a6d3c:::benchmark_data_smithi090_104448_object163013:head# 2024-05-15T23:19:39.187 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.187 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d17fe946:::benchmark_data_smithi090_104448_object169494:head# 2024-05-15T23:19:39.188 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.188 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d18a36a2:::benchmark_data_smithi090_104448_object170166:head# 2024-05-15T23:19:39.188 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.188 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d18dfb42:::benchmark_data_smithi090_104448_object165395:head# 2024-05-15T23:19:39.188 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.188 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d18f78dc:::benchmark_data_smithi090_104448_object169138:head# 2024-05-15T23:19:39.188 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.188 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d193853f:::benchmark_data_smithi090_104448_object160989:head# 2024-05-15T23:19:39.188 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.188 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1949da2:::benchmark_data_smithi090_104448_object156666:head# 2024-05-15T23:19:39.188 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.188 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1964575:::benchmark_data_smithi090_104448_object162885:head# 2024-05-15T23:19:39.188 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.188 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1a2e87b:::benchmark_data_smithi090_104448_object170674:head# 2024-05-15T23:19:39.189 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.189 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1a615b9:::benchmark_data_smithi090_104448_object160170:head# 2024-05-15T23:19:39.189 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.189 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1ae37f4:::benchmark_data_smithi090_104448_object162720:head# 2024-05-15T23:19:39.189 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.189 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1aeb713:::benchmark_data_smithi090_104448_object167354:head# 2024-05-15T23:19:39.189 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.189 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1b08e72:::benchmark_data_smithi090_104448_object156828:head# 2024-05-15T23:19:39.189 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.189 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1b1cce3:::benchmark_data_smithi090_104448_object166176:head# 2024-05-15T23:19:39.189 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.189 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1b36775:::benchmark_data_smithi090_104448_object169969:head# 2024-05-15T23:19:39.189 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.189 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1b6f2c0:::benchmark_data_smithi090_104448_object160507:head# 2024-05-15T23:19:39.190 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.190 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1b9a9e4:::benchmark_data_smithi090_104448_object158400:head# 2024-05-15T23:19:39.190 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.190 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1c0345d:::benchmark_data_smithi090_104448_object172926:head# 2024-05-15T23:19:39.190 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.190 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1c1b097:::benchmark_data_smithi090_104448_object172759:head# 2024-05-15T23:19:39.190 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.190 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1c6b12a:::benchmark_data_smithi090_104448_object172406:head# 2024-05-15T23:19:39.190 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.190 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1d378c1:::benchmark_data_smithi090_104448_object165117:head# 2024-05-15T23:19:39.190 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.190 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1d9037d:::benchmark_data_smithi090_104448_object162268:head# 2024-05-15T23:19:39.190 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.191 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1d906ae:::benchmark_data_smithi090_104448_object161815:head# 2024-05-15T23:19:39.191 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.195 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1e2f73f:::benchmark_data_smithi090_104448_object167882:head# 2024-05-15T23:19:39.195 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.195 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1e73507:::benchmark_data_smithi090_104448_object160643:head# 2024-05-15T23:19:39.195 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.195 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1f072e5:::benchmark_data_smithi090_104448_object169053:head# 2024-05-15T23:19:39.196 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.196 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1f64215:::benchmark_data_smithi090_104448_object167965:head# 2024-05-15T23:19:39.196 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.196 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1fd2aca:::benchmark_data_smithi090_104448_object163957:head# 2024-05-15T23:19:39.196 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.196 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d1ff3578:::benchmark_data_smithi090_104448_object160341:head# 2024-05-15T23:19:39.196 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.196 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d200111a:::benchmark_data_smithi090_104448_object159497:head# 2024-05-15T23:19:39.196 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.196 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d20b3944:::benchmark_data_smithi090_104448_object162345:head# 2024-05-15T23:19:39.196 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.196 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d21212eb:::benchmark_data_smithi090_104448_object160049:head# 2024-05-15T23:19:39.196 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.197 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d21611a6:::benchmark_data_smithi090_104448_object157193:head# 2024-05-15T23:19:39.197 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.197 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d21621a9:::benchmark_data_smithi090_104448_object171828:head# 2024-05-15T23:19:39.197 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.197 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d218a703:::benchmark_data_smithi090_104448_object162960:head# 2024-05-15T23:19:39.197 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.197 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d21afba7:::benchmark_data_smithi090_104448_object160982:head# 2024-05-15T23:19:39.197 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.197 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d21c6eca:::benchmark_data_smithi090_104448_object162286:head# 2024-05-15T23:19:39.197 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.197 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d21e7538:::benchmark_data_smithi090_104448_object159684:head# 2024-05-15T23:19:39.197 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.197 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d21f7964:::benchmark_data_smithi090_104448_object169019:head# 2024-05-15T23:19:39.197 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.198 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d220f212:::benchmark_data_smithi090_104448_object159797:head# 2024-05-15T23:19:39.198 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.198 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2228c98:::benchmark_data_smithi090_104448_object161310:head# 2024-05-15T23:19:39.198 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.198 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d228b558:::benchmark_data_smithi090_104448_object160693:head# 2024-05-15T23:19:39.198 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.198 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2309af3:::benchmark_data_smithi090_104448_object171357:head# 2024-05-15T23:19:39.198 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.198 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2413c37:::benchmark_data_smithi090_104448_object161188:head# 2024-05-15T23:19:39.198 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.198 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d244b84a:::benchmark_data_smithi090_104448_object161136:head# 2024-05-15T23:19:39.198 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.198 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d24b6b92:::benchmark_data_smithi090_104448_object170445:head# 2024-05-15T23:19:39.199 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.199 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d24d49c2:::benchmark_data_smithi090_104448_object170949:head# 2024-05-15T23:19:39.199 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.199 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d24fea30:::benchmark_data_smithi090_104448_object171694:head# 2024-05-15T23:19:39.199 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.199 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2525bf2:::benchmark_data_smithi090_104448_object165267:head# 2024-05-15T23:19:39.199 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.199 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d25bffd4:::benchmark_data_smithi090_104448_object158249:head# 2024-05-15T23:19:39.199 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.199 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d25dfb9c:::benchmark_data_smithi090_104448_object158463:head# 2024-05-15T23:19:39.199 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.199 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d260a1a0:::benchmark_data_smithi090_104448_object164214:head# 2024-05-15T23:19:39.199 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.199 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d261216d:::benchmark_data_smithi090_104448_object162168:head# 2024-05-15T23:19:39.199 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.200 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d26496d3:::benchmark_data_smithi090_104448_object169301:head# 2024-05-15T23:19:39.200 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.200 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d269185b:::benchmark_data_smithi090_104448_object163562:head# 2024-05-15T23:19:39.200 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.200 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d26a2bd3:::benchmark_data_smithi090_104448_object160755:head# 2024-05-15T23:19:39.200 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.200 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d26ce10e:::benchmark_data_smithi090_104448_object172545:head# 2024-05-15T23:19:39.200 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.200 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d26dfd76:::benchmark_data_smithi090_104448_object163516:head# 2024-05-15T23:19:39.200 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.200 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d26fefad:::benchmark_data_smithi090_104448_object171295:head# 2024-05-15T23:19:39.200 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.200 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2718a37:::benchmark_data_smithi090_104448_object160104:head# 2024-05-15T23:19:39.200 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.201 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2719342:::benchmark_data_smithi090_104448_object163094:head# 2024-05-15T23:19:39.201 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.211 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d275b212:::benchmark_data_smithi090_104448_object157551:head# 2024-05-15T23:19:39.211 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.212 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d27763d2:::benchmark_data_smithi090_104448_object162493:head# 2024-05-15T23:19:39.212 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.212 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d279b3c6:::benchmark_data_smithi090_104448_object168209:head# 2024-05-15T23:19:39.212 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.212 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d27d69f2:::benchmark_data_smithi090_104448_object157837:head# 2024-05-15T23:19:39.212 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.212 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d281a8f0:::benchmark_data_smithi090_104448_object170317:head# 2024-05-15T23:19:39.212 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.212 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d28237c6:::benchmark_data_smithi090_104448_object161524:head# 2024-05-15T23:19:39.212 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.212 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d28552a4:::benchmark_data_smithi090_104448_object170968:head# 2024-05-15T23:19:39.212 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.212 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d28557ea:::benchmark_data_smithi090_104448_object165296:head# 2024-05-15T23:19:39.213 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.213 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d290b60d:::benchmark_data_smithi090_104448_object167612:head# 2024-05-15T23:19:39.213 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.213 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d292e10e:::benchmark_data_smithi090_104448_object168425:head# 2024-05-15T23:19:39.213 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.213 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d298df97:::benchmark_data_smithi090_104448_object169648:head# 2024-05-15T23:19:39.213 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.213 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d29c0f1f:::benchmark_data_smithi090_104448_object159476:head# 2024-05-15T23:19:39.213 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.213 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d29f7185:::benchmark_data_smithi090_104448_object164363:head# 2024-05-15T23:19:39.213 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.213 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2aacc6b:::benchmark_data_smithi090_104448_object167065:head# 2024-05-15T23:19:39.213 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.213 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2ab8a76:::benchmark_data_smithi090_104448_object168409:head# 2024-05-15T23:19:39.213 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.214 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2b80c75:::benchmark_data_smithi090_104448_object162606:head# 2024-05-15T23:19:39.214 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.214 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2bb4ae1:::benchmark_data_smithi090_104448_object164696:head# 2024-05-15T23:19:39.214 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.214 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2bc0b62:::benchmark_data_smithi090_104448_object159494:head# 2024-05-15T23:19:39.214 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.214 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2bd42d7:::benchmark_data_smithi090_104448_object158371:head# 2024-05-15T23:19:39.214 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.214 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2bd4ff5:::benchmark_data_smithi090_104448_object165219:head# 2024-05-15T23:19:39.214 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.214 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2bdee71:::benchmark_data_smithi090_104448_object159288:head# 2024-05-15T23:19:39.214 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.214 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2bf1140:::benchmark_data_smithi090_104448_object159534:head# 2024-05-15T23:19:39.215 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.215 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2bf1343:::benchmark_data_smithi090_104448_object172042:head# 2024-05-15T23:19:39.215 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.215 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2c177ac:::benchmark_data_smithi090_104448_object168611:head# 2024-05-15T23:19:39.215 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.215 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2c9b329:::benchmark_data_smithi090_104448_object164850:head# 2024-05-15T23:19:39.215 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.215 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2cd8aab:::benchmark_data_smithi090_104448_object171688:head# 2024-05-15T23:19:39.215 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.215 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2d3e00d:::benchmark_data_smithi090_104448_object165964:head# 2024-05-15T23:19:39.215 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.215 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2d56b7b:::benchmark_data_smithi090_104448_object169202:head# 2024-05-15T23:19:39.215 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.215 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2d88ee9:::benchmark_data_smithi090_104448_object168499:head# 2024-05-15T23:19:39.216 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.216 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2e1555f:::benchmark_data_smithi090_104448_object160634:head# 2024-05-15T23:19:39.216 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.216 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2e3918d:::benchmark_data_smithi090_104448_object172090:head# 2024-05-15T23:19:39.216 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.216 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2e4322b:::benchmark_data_smithi090_104448_object166653:head# 2024-05-15T23:19:39.216 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.216 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2e60e52:::benchmark_data_smithi090_104448_object165319:head# 2024-05-15T23:19:39.216 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.216 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2e80c33:::benchmark_data_smithi090_104448_object156642:head# 2024-05-15T23:19:39.216 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.216 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2ea56e7:::benchmark_data_smithi090_104448_object157781:head# 2024-05-15T23:19:39.216 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.216 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2ec8489:::benchmark_data_smithi090_104448_object158876:head# 2024-05-15T23:19:39.216 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.217 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2ed7724:::benchmark_data_smithi090_104448_object157906:head# 2024-05-15T23:19:39.217 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.217 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2f28ba0:::benchmark_data_smithi090_104448_object161749:head# 2024-05-15T23:19:39.217 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.217 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2f721c2:::benchmark_data_smithi090_104448_object157747:head# 2024-05-15T23:19:39.217 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.217 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2f8cbe1:::benchmark_data_smithi090_104448_object158229:head# 2024-05-15T23:19:39.217 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.217 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2f92d09:::benchmark_data_smithi090_104448_object169219:head# 2024-05-15T23:19:39.217 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.217 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2f98a7e:::benchmark_data_smithi090_104448_object164065:head# 2024-05-15T23:19:39.217 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.217 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d2ff9504:::benchmark_data_smithi090_104448_object166208:head# 2024-05-15T23:19:39.217 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.217 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d300f2f0:::benchmark_data_smithi090_104448_object167886:head# 2024-05-15T23:19:39.218 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.218 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d301c459:::benchmark_data_smithi090_104448_object163634:head# 2024-05-15T23:19:39.218 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.218 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3034e12:::benchmark_data_smithi090_104448_object156652:head# 2024-05-15T23:19:39.218 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.218 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3082d20:::benchmark_data_smithi090_104448_object156632:head# 2024-05-15T23:19:39.218 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.218 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d309c7e0:::benchmark_data_smithi090_104448_object167740:head# 2024-05-15T23:19:39.218 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.218 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d309d78d:::benchmark_data_smithi090_104448_object163152:head# 2024-05-15T23:19:39.218 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.218 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d30a45dd:::benchmark_data_smithi090_104448_object168727:head# 2024-05-15T23:19:39.218 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.218 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d30b5b9d:::benchmark_data_smithi090_104448_object170801:head# 2024-05-15T23:19:39.219 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.219 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d30ed7be:::benchmark_data_smithi090_104448_object167285:head# 2024-05-15T23:19:39.219 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.219 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d30fc751:::benchmark_data_smithi090_104448_object166955:head# 2024-05-15T23:19:39.219 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.219 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d310ef2a:::benchmark_data_smithi090_104448_object170954:head# 2024-05-15T23:19:39.219 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.219 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d31e3990:::benchmark_data_smithi090_104448_object169004:head# 2024-05-15T23:19:39.219 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.219 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d32164e7:::benchmark_data_smithi090_104448_object159973:head# 2024-05-15T23:19:39.219 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.219 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3243ad4:::benchmark_data_smithi090_104448_object161176:head# 2024-05-15T23:19:39.219 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.219 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d324ccb1:::benchmark_data_smithi090_104448_object165560:head# 2024-05-15T23:19:39.219 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3259749:::benchmark_data_smithi090_104448_object170822:head# 2024-05-15T23:19:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3276831:::benchmark_data_smithi090_104448_object170767:head# 2024-05-15T23:19:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d338960d:::benchmark_data_smithi090_104448_object168702:head# 2024-05-15T23:19:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d33e3598:::benchmark_data_smithi090_104448_object171530:head# 2024-05-15T23:19:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d33e61d6:::benchmark_data_smithi090_104448_object162904:head# 2024-05-15T23:19:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3431634:::benchmark_data_smithi090_104448_object171507:head# 2024-05-15T23:19:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3442165:::benchmark_data_smithi090_104448_object163041:head# 2024-05-15T23:19:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d357fe12:::benchmark_data_smithi090_104448_object169264:head# 2024-05-15T23:19:39.221 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.221 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d358b745:::benchmark_data_smithi090_104448_object162303:head# 2024-05-15T23:19:39.221 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.221 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d35c6874:::benchmark_data_smithi090_104448_object170138:head# 2024-05-15T23:19:39.221 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.249 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3699d13:::benchmark_data_smithi090_104448_object172286:head# 2024-05-15T23:19:39.249 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.249 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d36b575a:::benchmark_data_smithi090_104448_object158085:head# 2024-05-15T23:19:39.249 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.249 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d36ed135:::benchmark_data_smithi090_104448_object161795:head# 2024-05-15T23:19:39.249 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.250 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d371fc28:::benchmark_data_smithi090_104448_object172169:head# 2024-05-15T23:19:39.250 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.250 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3796d5c:::benchmark_data_smithi090_104448_object157200:head# 2024-05-15T23:19:39.250 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.250 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d37ceef4:::benchmark_data_smithi090_104448_object164350:head# 2024-05-15T23:19:39.250 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.250 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d37db4e4:::benchmark_data_smithi090_104448_object167891:head# 2024-05-15T23:19:39.250 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.250 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d38a8753:::benchmark_data_smithi090_104448_object167118:head# 2024-05-15T23:19:39.250 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.250 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d38c295b:::benchmark_data_smithi090_104448_object167919:head# 2024-05-15T23:19:39.250 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.250 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d38c5d80:::benchmark_data_smithi090_104448_object171848:head# 2024-05-15T23:19:39.250 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.251 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d38ca859:::benchmark_data_smithi090_104448_object158819:head# 2024-05-15T23:19:39.251 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.251 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d38dbe37:::benchmark_data_smithi090_104448_object158472:head# 2024-05-15T23:19:39.251 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.251 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d391b6a5:::benchmark_data_smithi090_104448_object164080:head# 2024-05-15T23:19:39.251 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.251 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3b1558f:::benchmark_data_smithi090_104448_object160311:head# 2024-05-15T23:19:39.251 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.251 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3b20104:::benchmark_data_smithi090_104448_object169804:head# 2024-05-15T23:19:39.251 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.251 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3b234bd:::benchmark_data_smithi090_104448_object160817:head# 2024-05-15T23:19:39.251 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3b32a50:::benchmark_data_smithi090_104448_object164828:head# 2024-05-15T23:19:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3b9bb12:::benchmark_data_smithi090_104448_object167773:head# 2024-05-15T23:19:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3bed8f1:::benchmark_data_smithi090_104448_object158911:head# 2024-05-15T23:19:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3cb90ce:::benchmark_data_smithi090_104448_object163919:head# 2024-05-15T23:19:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3cedf5d:::benchmark_data_smithi090_104448_object172780:head# 2024-05-15T23:19:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3d51da4:::benchmark_data_smithi090_104448_object171304:head# 2024-05-15T23:19:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3e33ed6:::benchmark_data_smithi090_104448_object163296:head# 2024-05-15T23:19:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3e34c0b:::benchmark_data_smithi090_104448_object157111:head# 2024-05-15T23:19:39.253 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.253 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3e41108:::benchmark_data_smithi090_104448_object171932:head# 2024-05-15T23:19:39.253 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.253 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3e615e8:::benchmark_data_smithi090_104448_object167238:head# 2024-05-15T23:19:39.253 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.253 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3e7b442:::benchmark_data_smithi090_104448_object168397:head# 2024-05-15T23:19:39.253 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.253 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3f83433:::benchmark_data_smithi090_104448_object168358:head# 2024-05-15T23:19:39.253 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.253 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d3f9c5fa:::benchmark_data_smithi090_104448_object171027:head# 2024-05-15T23:19:39.253 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.253 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4004e24:::benchmark_data_smithi090_104448_object160272:head# 2024-05-15T23:19:39.253 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.253 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d40780c8:::benchmark_data_smithi090_104448_object161361:head# 2024-05-15T23:19:39.254 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.254 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d40817b4:::benchmark_data_smithi090_104448_object172029:head# 2024-05-15T23:19:39.254 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.254 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d408db06:::benchmark_data_smithi090_104448_object160560:head# 2024-05-15T23:19:39.254 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.254 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d40e53b4:::benchmark_data_smithi090_104448_object164685:head# 2024-05-15T23:19:39.254 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.254 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d40f41ec:::benchmark_data_smithi090_104448_object163375:head# 2024-05-15T23:19:39.254 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.254 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4145a86:::benchmark_data_smithi090_104448_object170905:head# 2024-05-15T23:19:39.254 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.254 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d418a316:::benchmark_data_smithi090_104448_object162990:head# 2024-05-15T23:19:39.254 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.254 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d41a2c4d:::benchmark_data_smithi090_104448_object157119:head# 2024-05-15T23:19:39.255 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.255 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4231968:::benchmark_data_smithi090_104448_object162700:head# 2024-05-15T23:19:39.255 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.255 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d42443c8:::benchmark_data_smithi090_104448_object161541:head# 2024-05-15T23:19:39.255 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.255 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d42505f9:::benchmark_data_smithi090_104448_object162761:head# 2024-05-15T23:19:39.255 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.255 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4282efb:::benchmark_data_smithi090_104448_object169579:head# 2024-05-15T23:19:39.255 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.255 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d431da9b:::benchmark_data_smithi090_104448_object163778:head# 2024-05-15T23:19:39.255 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.255 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4343c1c:::benchmark_data_smithi090_104448_object167942:head# 2024-05-15T23:19:39.255 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.255 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d43c5c3f:::benchmark_data_smithi090_104448_object164986:head# 2024-05-15T23:19:39.256 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.256 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d43e206c:::benchmark_data_smithi090_104448_object159332:head# 2024-05-15T23:19:39.256 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.256 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d443091d:::benchmark_data_smithi090_104448_object158436:head# 2024-05-15T23:19:39.256 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.256 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4445142:::benchmark_data_smithi090_104448_object157501:head# 2024-05-15T23:19:39.256 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.256 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d445aa71:::benchmark_data_smithi090_104448_object158387:head# 2024-05-15T23:19:39.256 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.256 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d44aae27:::benchmark_data_smithi090_104448_object168336:head# 2024-05-15T23:19:39.256 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.256 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d44b2bef:::benchmark_data_smithi090_104448_object170489:head# 2024-05-15T23:19:39.256 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.257 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d44df32a:::benchmark_data_smithi090_104448_object160537:head# 2024-05-15T23:19:39.257 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.257 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d44ed37c:::benchmark_data_smithi090_104448_object158597:head# 2024-05-15T23:19:39.257 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.257 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d453e761:::benchmark_data_smithi090_104448_object169343:head# 2024-05-15T23:19:39.257 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.257 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4563ea6:::benchmark_data_smithi090_104448_object163555:head# 2024-05-15T23:19:39.257 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.257 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4568dc6:::benchmark_data_smithi090_104448_object163189:head# 2024-05-15T23:19:39.257 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.257 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d45a12cf:::benchmark_data_smithi090_104448_object160092:head# 2024-05-15T23:19:39.257 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.257 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d45aadc9:::benchmark_data_smithi090_104448_object164455:head# 2024-05-15T23:19:39.257 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.257 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d466cc48:::benchmark_data_smithi090_104448_object169587:head# 2024-05-15T23:19:39.258 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.258 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d46ba954:::benchmark_data_smithi090_104448_object158914:head# 2024-05-15T23:19:39.258 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.258 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d46d2716:::benchmark_data_smithi090_104448_object160519:head# 2024-05-15T23:19:39.258 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.258 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d46da3e6:::benchmark_data_smithi090_104448_object168201:head# 2024-05-15T23:19:39.258 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.258 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4715ad6:::benchmark_data_smithi090_104448_object169948:head# 2024-05-15T23:19:39.258 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.258 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d47c418e:::benchmark_data_smithi090_104448_object168061:head# 2024-05-15T23:19:39.258 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.258 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d47e6eee:::benchmark_data_smithi090_104448_object166349:head# 2024-05-15T23:19:39.258 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.258 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d48c6c5f:::benchmark_data_smithi090_104448_object166964:head# 2024-05-15T23:19:39.259 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.259 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d48da413:::benchmark_data_smithi090_104448_object162078:head# 2024-05-15T23:19:39.259 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.259 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4903b4d:::benchmark_data_smithi090_104448_object161811:head# 2024-05-15T23:19:39.259 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.259 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d490c099:::benchmark_data_smithi090_104448_object160355:head# 2024-05-15T23:19:39.259 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.259 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d499c830:::benchmark_data_smithi090_104448_object167302:head# 2024-05-15T23:19:39.259 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.259 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d49f2084:::benchmark_data_smithi090_104448_object161437:head# 2024-05-15T23:19:39.259 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.259 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4a79530:::benchmark_data_smithi090_104448_object157667:head# 2024-05-15T23:19:39.259 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.259 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4a99d83:::benchmark_data_smithi090_104448_object158564:head# 2024-05-15T23:19:39.259 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.260 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4aba819:::benchmark_data_smithi090_104448_object168218:head# 2024-05-15T23:19:39.260 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.260 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4af59dc:::benchmark_data_smithi090_104448_object157199:head# 2024-05-15T23:19:39.260 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.260 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4afdd65:::benchmark_data_smithi090_104448_object161101:head# 2024-05-15T23:19:39.260 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.260 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4b6860d:::benchmark_data_smithi090_104448_object166056:head# 2024-05-15T23:19:39.260 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.260 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4b78bec:::benchmark_data_smithi090_104448_object158413:head# 2024-05-15T23:19:39.260 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.260 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4ba8d8d:::benchmark_data_smithi090_104448_object172693:head# 2024-05-15T23:19:39.260 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.260 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4bd34b5:::benchmark_data_smithi090_104448_object172775:head# 2024-05-15T23:19:39.260 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.261 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4c00974:::benchmark_data_smithi090_104448_object161568:head# 2024-05-15T23:19:39.261 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.261 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4c03aa4:::benchmark_data_smithi090_104448_object160963:head# 2024-05-15T23:19:39.261 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.261 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4c61ef9:::benchmark_data_smithi090_104448_object158186:head# 2024-05-15T23:19:39.261 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.261 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4c87194:::benchmark_data_smithi090_104448_object168013:head# 2024-05-15T23:19:39.261 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.261 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4c89c21:::benchmark_data_smithi090_104448_object172448:head# 2024-05-15T23:19:39.261 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.261 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4ca6fff:::benchmark_data_smithi090_104448_object157527:head# 2024-05-15T23:19:39.261 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.261 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4ce78e3:::benchmark_data_smithi090_104448_object164026:head# 2024-05-15T23:19:39.261 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.262 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4d3e6da:::benchmark_data_smithi090_104448_object166077:head# 2024-05-15T23:19:39.262 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.262 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4d45449:::benchmark_data_smithi090_104448_object165171:head# 2024-05-15T23:19:39.262 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.262 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4dbd6ef:::benchmark_data_smithi090_104448_object163278:head# 2024-05-15T23:19:39.262 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.262 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4e388db:::benchmark_data_smithi090_104448_object172804:head# 2024-05-15T23:19:39.262 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.262 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4e7d6c4:::benchmark_data_smithi090_104448_object160413:head# 2024-05-15T23:19:39.262 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.262 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4ebfd5f:::benchmark_data_smithi090_104448_object172019:head# 2024-05-15T23:19:39.262 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.262 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4ec69b4:::benchmark_data_smithi090_104448_object164787:head# 2024-05-15T23:19:39.262 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.263 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4f4c89d:::benchmark_data_smithi090_104448_object169463:head# 2024-05-15T23:19:39.263 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.263 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4fc48aa:::benchmark_data_smithi090_104448_object167021:head# 2024-05-15T23:19:39.263 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.263 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d4fd13e5:::benchmark_data_smithi090_104448_object172863:head# 2024-05-15T23:19:39.263 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.263 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5005a30:::benchmark_data_smithi090_104448_object159081:head# 2024-05-15T23:19:39.263 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.263 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5016951:::benchmark_data_smithi090_104448_object166682:head# 2024-05-15T23:19:39.263 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.263 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5066628:::benchmark_data_smithi090_104448_object162103:head# 2024-05-15T23:19:39.263 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.263 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d508b0e0:::benchmark_data_smithi090_104448_object160201:head# 2024-05-15T23:19:39.263 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.264 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5091958:::benchmark_data_smithi090_104448_object159506:head# 2024-05-15T23:19:39.264 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.264 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d50ea3f1:::benchmark_data_smithi090_104448_object164254:head# 2024-05-15T23:19:39.264 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.264 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d50f67ac:::benchmark_data_smithi090_104448_object171261:head# 2024-05-15T23:19:39.264 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.264 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d512f4dc:::benchmark_data_smithi090_104448_object167158:head# 2024-05-15T23:19:39.264 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.264 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d515beab:::benchmark_data_smithi090_104448_object169671:head# 2024-05-15T23:19:39.264 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.264 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d518bb2b:::benchmark_data_smithi090_104448_object158031:head# 2024-05-15T23:19:39.264 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.264 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d51a6aaa:::benchmark_data_smithi090_104448_object157860:head# 2024-05-15T23:19:39.264 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.265 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d51cdc9e:::benchmark_data_smithi090_104448_object157092:head# 2024-05-15T23:19:39.265 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.265 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d52701ef:::benchmark_data_smithi090_104448_object167538:head# 2024-05-15T23:19:39.265 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.265 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5276139:::benchmark_data_smithi090_104448_object167431:head# 2024-05-15T23:19:39.265 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.265 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d52b675b:::benchmark_data_smithi090_104448_object161763:head# 2024-05-15T23:19:39.265 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.265 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d52fb296:::benchmark_data_smithi090_104448_object163257:head# 2024-05-15T23:19:39.265 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.265 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d533e1d3:::benchmark_data_smithi090_104448_object165791:head# 2024-05-15T23:19:39.265 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.265 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d534956c:::benchmark_data_smithi090_104448_object162744:head# 2024-05-15T23:19:39.265 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.265 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d537eb57:::benchmark_data_smithi090_104448_object164289:head# 2024-05-15T23:19:39.266 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.266 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d53cb61d:::benchmark_data_smithi090_104448_object160098:head# 2024-05-15T23:19:39.266 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.266 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d54039bb:::benchmark_data_smithi090_104448_object161154:head# 2024-05-15T23:19:39.266 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.266 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5494a14:::benchmark_data_smithi090_104448_object163590:head# 2024-05-15T23:19:39.266 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.266 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d54978e7:::benchmark_data_smithi090_104448_object162574:head# 2024-05-15T23:19:39.266 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.266 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5501512:::benchmark_data_smithi090_104448_object169839:head# 2024-05-15T23:19:39.266 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.266 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d550b236:::benchmark_data_smithi090_104448_object163928:head# 2024-05-15T23:19:39.266 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.266 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d554b191:::benchmark_data_smithi090_104448_object162131:head# 2024-05-15T23:19:39.267 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.267 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5577678:::benchmark_data_smithi090_104448_object165028:head# 2024-05-15T23:19:39.267 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.267 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d55bd617:::benchmark_data_smithi090_104448_object172322:head# 2024-05-15T23:19:39.267 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.267 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d55ccdc8:::benchmark_data_smithi090_104448_object164818:head# 2024-05-15T23:19:39.267 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.267 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d55d13c4:::benchmark_data_smithi090_104448_object162317:head# 2024-05-15T23:19:39.267 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.267 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d55eb2a3:::benchmark_data_smithi090_104448_object165098:head# 2024-05-15T23:19:39.267 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.267 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5643d26:::benchmark_data_smithi090_104448_object167033:head# 2024-05-15T23:19:39.267 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.267 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d56a672b:::benchmark_data_smithi090_104448_object161707:head# 2024-05-15T23:19:39.268 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.268 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d56c431b:::benchmark_data_smithi090_104448_object159406:head# 2024-05-15T23:19:39.268 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.268 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d570b156:::benchmark_data_smithi090_104448_object162580:head# 2024-05-15T23:19:39.268 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.268 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d570cb4e:::benchmark_data_smithi090_104448_object160249:head# 2024-05-15T23:19:39.268 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.268 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d576a357:::benchmark_data_smithi090_104448_object164025:head# 2024-05-15T23:19:39.268 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.268 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5770819:::benchmark_data_smithi090_104448_object159389:head# 2024-05-15T23:19:39.268 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.268 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d57c29da:::benchmark_data_smithi090_104448_object162581:head# 2024-05-15T23:19:39.268 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.268 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d57e7419:::benchmark_data_smithi090_104448_object168651:head# 2024-05-15T23:19:39.268 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d58000b4:::benchmark_data_smithi090_104448_object158175:head# 2024-05-15T23:19:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5883893:::benchmark_data_smithi090_104448_object169540:head# 2024-05-15T23:19:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d588aef7:::benchmark_data_smithi090_104448_object171619:head# 2024-05-15T23:19:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d58ed79a:::benchmark_data_smithi090_104448_object165711:head# 2024-05-15T23:19:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d58eebb6:::benchmark_data_smithi090_104448_object168909:head# 2024-05-15T23:19:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5929c92:::benchmark_data_smithi090_104448_object172374:head# 2024-05-15T23:19:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d594e0eb:::benchmark_data_smithi090_104448_object168883:head# 2024-05-15T23:19:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.269 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d59630c2:::benchmark_data_smithi090_104448_object163477:head# 2024-05-15T23:19:39.270 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.270 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d59cd130:::benchmark_data_smithi090_104448_object157198:head# 2024-05-15T23:19:39.270 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.270 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d59e8bd0:::benchmark_data_smithi090_104448_object171056:head# 2024-05-15T23:19:39.270 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.270 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5a2a206:::benchmark_data_smithi090_104448_object165477:head# 2024-05-15T23:19:39.270 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.270 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5a345d3:::benchmark_data_smithi090_104448_object170360:head# 2024-05-15T23:19:39.270 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.270 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5a47930:::benchmark_data_smithi090_104448_object157217:head# 2024-05-15T23:19:39.270 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.270 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5aa3b89:::benchmark_data_smithi090_104448_object163890:head# 2024-05-15T23:19:39.270 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.270 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5ace42c:::benchmark_data_smithi090_104448_object163091:head# 2024-05-15T23:19:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5b3320a:::benchmark_data_smithi090_104448_object171634:head# 2024-05-15T23:19:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5b784f1:::benchmark_data_smithi090_104448_object163766:head# 2024-05-15T23:19:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5ba50ab:::benchmark_data_smithi090_104448_object158477:head# 2024-05-15T23:19:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5bf13ba:::benchmark_data_smithi090_104448_object167885:head# 2024-05-15T23:19:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5c11c4c:::benchmark_data_smithi090_104448_object172537:head# 2024-05-15T23:19:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5c1913f:::benchmark_data_smithi090_104448_object169971:head# 2024-05-15T23:19:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5c2eb6a:::benchmark_data_smithi090_104448_object171260:head# 2024-05-15T23:19:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5c45007:::benchmark_data_smithi090_104448_object165049:head# 2024-05-15T23:19:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5c4a687:::benchmark_data_smithi090_104448_object162844:head# 2024-05-15T23:19:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5caf2d1:::benchmark_data_smithi090_104448_object165577:head# 2024-05-15T23:19:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5ce0a84:::benchmark_data_smithi090_104448_object164751:head# 2024-05-15T23:19:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5d27cdd:::benchmark_data_smithi090_104448_object167924:head# 2024-05-15T23:19:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5d4846a:::benchmark_data_smithi090_104448_object168596:head# 2024-05-15T23:19:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5d93a15:::benchmark_data_smithi090_104448_object158016:head# 2024-05-15T23:19:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5dd1419:::benchmark_data_smithi090_104448_object171168:head# 2024-05-15T23:19:39.273 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.273 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5ddc952:::benchmark_data_smithi090_104448_object161679:head# 2024-05-15T23:19:39.273 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.273 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5dddd42:::benchmark_data_smithi090_104448_object164778:head# 2024-05-15T23:19:39.273 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.273 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5e02329:::benchmark_data_smithi090_104448_object159837:head# 2024-05-15T23:19:39.273 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.273 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5e63e09:::benchmark_data_smithi090_104448_object168782:head# 2024-05-15T23:19:39.273 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.273 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5e79d50:::benchmark_data_smithi090_104448_object166553:head# 2024-05-15T23:19:39.313 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.313 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5f042ca:::benchmark_data_smithi090_104448_object169720:head# 2024-05-15T23:19:39.313 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.314 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5fd888c:::benchmark_data_smithi090_104448_object157589:head# 2024-05-15T23:19:39.314 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.314 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d5ff305b:::benchmark_data_smithi090_104448_object167254:head# 2024-05-15T23:19:39.314 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.314 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d601bf82:::benchmark_data_smithi090_104448_object158577:head# 2024-05-15T23:19:39.314 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.314 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6069d26:::benchmark_data_smithi090_104448_object170990:head# 2024-05-15T23:19:39.314 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.314 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d608dabb:::benchmark_data_smithi090_104448_object159161:head# 2024-05-15T23:19:39.314 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.314 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d60cfcf9:::benchmark_data_smithi090_104448_object162088:head# 2024-05-15T23:19:39.314 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.314 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d612d9b6:::benchmark_data_smithi090_104448_object157156:head# 2024-05-15T23:19:39.314 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6134c92:::benchmark_data_smithi090_104448_object165320:head# 2024-05-15T23:19:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d61359d5:::benchmark_data_smithi090_104448_object167339:head# 2024-05-15T23:19:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d61a1a06:::benchmark_data_smithi090_104448_object157104:head# 2024-05-15T23:19:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d61e6dfd:::benchmark_data_smithi090_104448_object158098:head# 2024-05-15T23:19:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d620ff59:::benchmark_data_smithi090_104448_object161381:head# 2024-05-15T23:19:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d62a0338:::benchmark_data_smithi090_104448_object170742:head# 2024-05-15T23:19:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d62ab3af:::benchmark_data_smithi090_104448_object171550:head# 2024-05-15T23:19:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.316 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6323154:::benchmark_data_smithi090_104448_object160142:head# 2024-05-15T23:19:39.316 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.316 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6347525:::benchmark_data_smithi090_104448_object165862:head# 2024-05-15T23:19:39.316 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.316 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d634ab47:::benchmark_data_smithi090_104448_object159339:head# 2024-05-15T23:19:39.316 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.316 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d63e9182:::benchmark_data_smithi090_104448_object169514:head# 2024-05-15T23:19:39.316 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.316 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d648f14b:::benchmark_data_smithi090_104448_object169537:head# 2024-05-15T23:19:39.316 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.316 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d64cb720:::benchmark_data_smithi090_104448_object172051:head# 2024-05-15T23:19:39.316 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.316 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6537421:::benchmark_data_smithi090_104448_object173171:head# 2024-05-15T23:19:39.316 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.317 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d65418ce:::benchmark_data_smithi090_104448_object165069:head# 2024-05-15T23:19:39.317 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.317 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d65ae19d:::benchmark_data_smithi090_104448_object160542:head# 2024-05-15T23:19:39.317 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.317 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d65e97e4:::benchmark_data_smithi090_104448_object164449:head# 2024-05-15T23:19:39.317 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.317 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6717ff5:::benchmark_data_smithi090_104448_object159738:head# 2024-05-15T23:19:39.317 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.317 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d677c2c7:::benchmark_data_smithi090_104448_object173023:head# 2024-05-15T23:19:39.317 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.317 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d67a4a38:::benchmark_data_smithi090_104448_object164961:head# 2024-05-15T23:19:39.317 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.317 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6817035:::benchmark_data_smithi090_104448_object162804:head# 2024-05-15T23:19:39.317 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.317 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6819305:::benchmark_data_smithi090_104448_object164186:head# 2024-05-15T23:19:39.318 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.318 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d686ff31:::benchmark_data_smithi090_104448_object158656:head# 2024-05-15T23:19:39.318 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.318 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d687fb03:::benchmark_data_smithi090_104448_object163456:head# 2024-05-15T23:19:39.318 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.318 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6919a96:::benchmark_data_smithi090_104448_object157755:head# 2024-05-15T23:19:39.318 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.318 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d694bd68:::benchmark_data_smithi090_104448_object168451:head# 2024-05-15T23:19:39.318 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.318 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d697064d:::benchmark_data_smithi090_104448_object163615:head# 2024-05-15T23:19:39.318 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.318 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d69db45a:::benchmark_data_smithi090_104448_object172179:head# 2024-05-15T23:19:39.318 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.319 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d69dce88:::benchmark_data_smithi090_104448_object166678:head# 2024-05-15T23:19:39.319 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.319 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d69e4bde:::benchmark_data_smithi090_104448_object163301:head# 2024-05-15T23:19:39.319 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.319 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6a60284:::benchmark_data_smithi090_104448_object171247:head# 2024-05-15T23:19:39.319 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.319 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6aacc44:::benchmark_data_smithi090_104448_object157686:head# 2024-05-15T23:19:39.319 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.319 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6b02f9e:::benchmark_data_smithi090_104448_object160497:head# 2024-05-15T23:19:39.319 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.319 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6b95976:::benchmark_data_smithi090_104448_object163974:head# 2024-05-15T23:19:39.319 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.319 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6b9d407:::benchmark_data_smithi090_104448_object161010:head# 2024-05-15T23:19:39.319 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6ba2e5d:::benchmark_data_smithi090_104448_object163528:head# 2024-05-15T23:19:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6bdb41d:::benchmark_data_smithi090_104448_object171714:head# 2024-05-15T23:19:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6be6697:::benchmark_data_smithi090_104448_object167166:head# 2024-05-15T23:19:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6c50d6c:::benchmark_data_smithi090_104448_object159715:head# 2024-05-15T23:19:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6c581c4:::benchmark_data_smithi090_104448_object166121:head# 2024-05-15T23:19:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6cbbcec:::benchmark_data_smithi090_104448_object165184:head# 2024-05-15T23:19:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6ccee1a:::benchmark_data_smithi090_104448_object161676:head# 2024-05-15T23:19:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6d3ba65:::benchmark_data_smithi090_104448_object164664:head# 2024-05-15T23:19:39.321 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.321 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6d51fce:::benchmark_data_smithi090_104448_object164669:head# 2024-05-15T23:19:39.321 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.321 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6d6ea98:::benchmark_data_smithi090_104448_object172089:head# 2024-05-15T23:19:39.321 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.321 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6e5f51d:::benchmark_data_smithi090_104448_object159195:head# 2024-05-15T23:19:39.321 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.321 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6e7ed25:::benchmark_data_smithi090_104448_object171959:head# 2024-05-15T23:19:39.321 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.321 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6ed852b:::benchmark_data_smithi090_104448_object172196:head# 2024-05-15T23:19:39.321 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.321 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6efcc7b:::benchmark_data_smithi090_104448_object160106:head# 2024-05-15T23:19:39.321 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.321 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6f1f87a:::benchmark_data_smithi090_104448_object172622:head# 2024-05-15T23:19:39.322 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.322 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6f9f523:::benchmark_data_smithi090_104448_object160435:head# 2024-05-15T23:19:39.322 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.322 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6fd2308:::benchmark_data_smithi090_104448_object167903:head# 2024-05-15T23:19:39.322 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.322 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6fdefaa:::benchmark_data_smithi090_104448_object162139:head# 2024-05-15T23:19:39.322 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.322 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d6fefbb2:::benchmark_data_smithi090_104448_object160242:head# 2024-05-15T23:19:39.322 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.322 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d703f043:::benchmark_data_smithi090_104448_object162857:head# 2024-05-15T23:19:39.322 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.322 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d707081e:::benchmark_data_smithi090_104448_object164235:head# 2024-05-15T23:19:39.322 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.322 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d70d736d:::benchmark_data_smithi090_104448_object168719:head# 2024-05-15T23:19:39.323 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.323 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d70ec930:::benchmark_data_smithi090_104448_object169626:head# 2024-05-15T23:19:39.323 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.323 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d710fc33:::benchmark_data_smithi090_104448_object162976:head# 2024-05-15T23:19:39.323 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.323 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d718e4ec:::benchmark_data_smithi090_104448_object162699:head# 2024-05-15T23:19:39.323 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.323 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d719ace3:::benchmark_data_smithi090_104448_object167941:head# 2024-05-15T23:19:39.323 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.323 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d71baf99:::benchmark_data_smithi090_104448_object168182:head# 2024-05-15T23:19:39.323 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.323 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7226626:::benchmark_data_smithi090_104448_object169472:head# 2024-05-15T23:19:39.323 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.323 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d723ab17:::benchmark_data_smithi090_104448_object159550:head# 2024-05-15T23:19:39.323 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.324 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7263470:::benchmark_data_smithi090_104448_object158012:head# 2024-05-15T23:19:39.324 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.324 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d72b0451:::benchmark_data_smithi090_104448_object165545:head# 2024-05-15T23:19:39.324 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.324 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7354e7c:::benchmark_data_smithi090_104448_object171589:head# 2024-05-15T23:19:39.324 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.324 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7385574:::benchmark_data_smithi090_104448_object160765:head# 2024-05-15T23:19:39.324 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.324 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d739420c:::benchmark_data_smithi090_104448_object168588:head# 2024-05-15T23:19:39.324 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.324 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d73e9683:::benchmark_data_smithi090_104448_object160312:head# 2024-05-15T23:19:39.324 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.324 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d744d8ab:::benchmark_data_smithi090_104448_object163417:head# 2024-05-15T23:19:39.324 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.324 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d74b3c2b:::benchmark_data_smithi090_104448_object168373:head# 2024-05-15T23:19:39.325 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.325 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d74b7d43:::benchmark_data_smithi090_104448_object171296:head# 2024-05-15T23:19:39.325 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.325 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d74e2d71:::benchmark_data_smithi090_104448_object164798:head# 2024-05-15T23:19:39.325 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.325 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d75073d2:::benchmark_data_smithi090_104448_object172285:head# 2024-05-15T23:19:39.325 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.325 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d75609ae:::benchmark_data_smithi090_104448_object173161:head# 2024-05-15T23:19:39.325 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.325 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d75751dd:::benchmark_data_smithi090_104448_object167460:head# 2024-05-15T23:19:39.325 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.325 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d75a4498:::benchmark_data_smithi090_104448_object165819:head# 2024-05-15T23:19:39.325 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.325 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d75d4a7d:::benchmark_data_smithi090_104448_object171862:head# 2024-05-15T23:19:39.326 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.326 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7624448:::benchmark_data_smithi090_104448_object164951:head# 2024-05-15T23:19:39.326 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.326 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d766200b:::benchmark_data_smithi090_104448_object170839:head# 2024-05-15T23:19:39.326 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.326 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7672bfa:::benchmark_data_smithi090_104448_object169139:head# 2024-05-15T23:19:39.326 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.326 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d768189e:::benchmark_data_smithi090_104448_object172338:head# 2024-05-15T23:19:39.326 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.326 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7693660:::benchmark_data_smithi090_104448_object162523:head# 2024-05-15T23:19:39.326 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.326 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d76fb235:::benchmark_data_smithi090_104448_object171900:head# 2024-05-15T23:19:39.326 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.326 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d770a9b2:::benchmark_data_smithi090_104448_object161616:head# 2024-05-15T23:19:39.327 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.327 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d770d899:::benchmark_data_smithi090_104448_object169838:head# 2024-05-15T23:19:39.327 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.327 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7757421:::benchmark_data_smithi090_104448_object157051:head# 2024-05-15T23:19:39.327 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.327 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7840993:::benchmark_data_smithi090_104448_object167599:head# 2024-05-15T23:19:39.327 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.327 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7922f38:::benchmark_data_smithi090_104448_object172578:head# 2024-05-15T23:19:39.327 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.327 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d794c8c9:::benchmark_data_smithi090_104448_object165600:head# 2024-05-15T23:19:39.327 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.327 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7996a7b:::benchmark_data_smithi090_104448_object167433:head# 2024-05-15T23:19:39.327 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.327 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7a0adcc:::benchmark_data_smithi090_104448_object172838:head# 2024-05-15T23:19:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7a12619:::benchmark_data_smithi090_104448_object158921:head# 2024-05-15T23:19:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7a28c1a:::benchmark_data_smithi090_104448_object161564:head# 2024-05-15T23:19:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7a6732d:::benchmark_data_smithi090_104448_object158192:head# 2024-05-15T23:19:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7ab2da1:::benchmark_data_smithi090_104448_object160786:head# 2024-05-15T23:19:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7ac251f:::benchmark_data_smithi090_104448_object172341:head# 2024-05-15T23:19:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7b087b7:::benchmark_data_smithi090_104448_object160953:head# 2024-05-15T23:19:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7b79145:::benchmark_data_smithi090_104448_object167685:head# 2024-05-15T23:19:39.328 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.329 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7b944f2:::benchmark_data_smithi090_104448_object171096:head# 2024-05-15T23:19:39.329 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.329 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7bd6311:::benchmark_data_smithi090_104448_object160400:head# 2024-05-15T23:19:39.329 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.329 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7c35047:::benchmark_data_smithi090_104448_object164089:head# 2024-05-15T23:19:39.329 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.329 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7c35a59:::benchmark_data_smithi090_104448_object165763:head# 2024-05-15T23:19:39.329 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.329 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7c4f4d4:::benchmark_data_smithi090_104448_object170676:head# 2024-05-15T23:19:39.329 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.329 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7cb84c3:::benchmark_data_smithi090_104448_object157145:head# 2024-05-15T23:19:39.329 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.329 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7cd2c11:::benchmark_data_smithi090_104448_object161842:head# 2024-05-15T23:19:39.329 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.330 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7d00295:::benchmark_data_smithi090_104448_object159449:head# 2024-05-15T23:19:39.330 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.330 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7d55181:::benchmark_data_smithi090_104448_object162143:head# 2024-05-15T23:19:39.330 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.330 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7dce0cb:::benchmark_data_smithi090_104448_object172382:head# 2024-05-15T23:19:39.330 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.330 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7def6e1:::benchmark_data_smithi090_104448_object171439:head# 2024-05-15T23:19:39.330 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.330 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7e6a26a:::benchmark_data_smithi090_104448_object156737:head# 2024-05-15T23:19:39.330 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.330 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7eb1759:::benchmark_data_smithi090_104448_object170276:head# 2024-05-15T23:19:39.330 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.330 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7eb9639:::benchmark_data_smithi090_104448_object164801:head# 2024-05-15T23:19:39.331 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.331 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7f20691:::benchmark_data_smithi090_104448_object156675:head# 2024-05-15T23:19:39.331 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.331 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7f68dec:::benchmark_data_smithi090_104448_object170707:head# 2024-05-15T23:19:39.331 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.331 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7fcdd89:::benchmark_data_smithi090_104448_object173061:head# 2024-05-15T23:19:39.331 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.331 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7fdba6a:::benchmark_data_smithi090_104448_object167428:head# 2024-05-15T23:19:39.331 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.331 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d7fffa6f:::benchmark_data_smithi090_104448_object156963:head# 2024-05-15T23:19:39.331 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.331 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d809c21c:::benchmark_data_smithi090_104448_object160513:head# 2024-05-15T23:19:39.331 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.331 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d809f21b:::benchmark_data_smithi090_104448_object157623:head# 2024-05-15T23:19:39.332 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.332 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d80d3ea0:::benchmark_data_smithi090_104448_object159768:head# 2024-05-15T23:19:39.332 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.332 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d80d40db:::benchmark_data_smithi090_104448_object170120:head# 2024-05-15T23:19:39.332 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.332 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d80f0737:::benchmark_data_smithi090_104448_object167665:head# 2024-05-15T23:19:39.332 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.332 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d80f3c16:::benchmark_data_smithi090_104448_object163993:head# 2024-05-15T23:19:39.332 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.332 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d812890d:::benchmark_data_smithi090_104448_object164044:head# 2024-05-15T23:19:39.332 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.332 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d812e65c:::benchmark_data_smithi090_104448_object157244:head# 2024-05-15T23:19:39.332 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.333 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8133ef0:::benchmark_data_smithi090_104448_object159204:head# 2024-05-15T23:19:39.333 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.333 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d81a0b19:::benchmark_data_smithi090_104448_object165357:head# 2024-05-15T23:19:39.333 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.333 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d81a4b90:::benchmark_data_smithi090_104448_object163198:head# 2024-05-15T23:19:39.333 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.333 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d81c64e4:::benchmark_data_smithi090_104448_object170693:head# 2024-05-15T23:19:39.333 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.333 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d81e5082:::benchmark_data_smithi090_104448_object164999:head# 2024-05-15T23:19:39.333 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.333 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d81fc717:::benchmark_data_smithi090_104448_object161622:head# 2024-05-15T23:19:39.333 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.333 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d827cc56:::benchmark_data_smithi090_104448_object160838:head# 2024-05-15T23:19:39.333 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d82c3cff:::benchmark_data_smithi090_104448_object161086:head# 2024-05-15T23:19:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d832b245:::benchmark_data_smithi090_104448_object166843:head# 2024-05-15T23:19:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d833e706:::benchmark_data_smithi090_104448_object158285:head# 2024-05-15T23:19:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d837ce25:::benchmark_data_smithi090_104448_object159920:head# 2024-05-15T23:19:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d83a3802:::benchmark_data_smithi090_104448_object169254:head# 2024-05-15T23:19:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d83b3279:::benchmark_data_smithi090_104448_object166695:head# 2024-05-15T23:19:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d83e752d:::benchmark_data_smithi090_104448_object156813:head# 2024-05-15T23:19:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d83f74db:::benchmark_data_smithi090_104448_object166152:head# 2024-05-15T23:19:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8483ffa:::benchmark_data_smithi090_104448_object157395:head# 2024-05-15T23:19:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8557fea:::benchmark_data_smithi090_104448_object158788:head# 2024-05-15T23:19:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d861a787:::benchmark_data_smithi090_104448_object170333:head# 2024-05-15T23:19:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8650ef7:::benchmark_data_smithi090_104448_object173218:head# 2024-05-15T23:19:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d866bcb7:::benchmark_data_smithi090_104448_object163412:head# 2024-05-15T23:19:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d86b782d:::benchmark_data_smithi090_104448_object158347:head# 2024-05-15T23:19:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d86ccbc5:::benchmark_data_smithi090_104448_object159469:head# 2024-05-15T23:19:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8714e80:::benchmark_data_smithi090_104448_object167103:head# 2024-05-15T23:19:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d871b731:::benchmark_data_smithi090_104448_object169489:head# 2024-05-15T23:19:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d872ed9e:::benchmark_data_smithi090_104448_object162166:head# 2024-05-15T23:19:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d873572a:::benchmark_data_smithi090_104448_object161467:head# 2024-05-15T23:19:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8778e15:::benchmark_data_smithi090_104448_object168224:head# 2024-05-15T23:19:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8781282:::benchmark_data_smithi090_104448_object159181:head# 2024-05-15T23:19:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d87e2eb6:::benchmark_data_smithi090_104448_object166880:head# 2024-05-15T23:19:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d88262cc:::benchmark_data_smithi090_104448_object170683:head# 2024-05-15T23:19:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d88ca399:::benchmark_data_smithi090_104448_object169761:head# 2024-05-15T23:19:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d88f1433:::benchmark_data_smithi090_104448_object160836:head# 2024-05-15T23:19:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d893a2db:::benchmark_data_smithi090_104448_object168210:head# 2024-05-15T23:19:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8952069:::benchmark_data_smithi090_104448_object170897:head# 2024-05-15T23:19:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8955541:::benchmark_data_smithi090_104448_object170967:head# 2024-05-15T23:19:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8a06b1c:::benchmark_data_smithi090_104448_object171413:head# 2024-05-15T23:19:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8a55007:::benchmark_data_smithi090_104448_object156726:head# 2024-05-15T23:19:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8a83563:::benchmark_data_smithi090_104448_object166692:head# 2024-05-15T23:19:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8a9aa6d:::benchmark_data_smithi090_104448_object170852:head# 2024-05-15T23:19:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8aae4a8:::benchmark_data_smithi090_104448_object171464:head# 2024-05-15T23:19:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8b64453:::benchmark_data_smithi090_104448_object159034:head# 2024-05-15T23:19:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8b88f81:::benchmark_data_smithi090_104448_object167277:head# 2024-05-15T23:19:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8beec0c:::benchmark_data_smithi090_104448_object161382:head# 2024-05-15T23:19:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8c2cb2e:::benchmark_data_smithi090_104448_object165808:head# 2024-05-15T23:19:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8c34a9c:::benchmark_data_smithi090_104448_object170697:head# 2024-05-15T23:19:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8c7ff8c:::benchmark_data_smithi090_104448_object167300:head# 2024-05-15T23:19:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8c88e88:::benchmark_data_smithi090_104448_object169361:head# 2024-05-15T23:19:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8cc952d:::benchmark_data_smithi090_104448_object170834:head# 2024-05-15T23:19:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8d061b5:::benchmark_data_smithi090_104448_object169347:head# 2024-05-15T23:19:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8d2e1e8:::benchmark_data_smithi090_104448_object169880:head# 2024-05-15T23:19:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8db0e73:::benchmark_data_smithi090_104448_object167157:head# 2024-05-15T23:19:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8db62ee:::benchmark_data_smithi090_104448_object163328:head# 2024-05-15T23:19:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8dc8dca:::benchmark_data_smithi090_104448_object167383:head# 2024-05-15T23:19:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8def457:::benchmark_data_smithi090_104448_object169306:head# 2024-05-15T23:19:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8e72a56:::benchmark_data_smithi090_104448_object168536:head# 2024-05-15T23:19:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8e8c722:::benchmark_data_smithi090_104448_object157800:head# 2024-05-15T23:19:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8ed46ae:::benchmark_data_smithi090_104448_object161639:head# 2024-05-15T23:19:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8ee2159:::benchmark_data_smithi090_104448_object165837:head# 2024-05-15T23:19:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8f3166b:::benchmark_data_smithi090_104448_object163256:head# 2024-05-15T23:19:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d8ff7676:::benchmark_data_smithi090_104448_object164258:head# 2024-05-15T23:19:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d900d8e0:::benchmark_data_smithi090_104448_object166438:head# 2024-05-15T23:19:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d904bccf:::benchmark_data_smithi090_104448_object163233:head# 2024-05-15T23:19:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9082322:::benchmark_data_smithi090_104448_object162478:head# 2024-05-15T23:19:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9088071:::benchmark_data_smithi090_104448_object172056:head# 2024-05-15T23:19:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d90949df:::benchmark_data_smithi090_104448_object158805:head# 2024-05-15T23:19:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d90956c0:::benchmark_data_smithi090_104448_object163154:head# 2024-05-15T23:19:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d911ad89:::benchmark_data_smithi090_104448_object170295:head# 2024-05-15T23:19:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9143dc7:::benchmark_data_smithi090_104448_object157977:head# 2024-05-15T23:19:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d915305f:::benchmark_data_smithi090_104448_object162209:head# 2024-05-15T23:19:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d91a2e64:::benchmark_data_smithi090_104448_object157834:head# 2024-05-15T23:19:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d91be9af:::benchmark_data_smithi090_104448_object167092:head# 2024-05-15T23:19:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.343 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d91e46cd:::benchmark_data_smithi090_104448_object168236:head# 2024-05-15T23:19:39.343 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.343 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d91ee1ea:::benchmark_data_smithi090_104448_object159233:head# 2024-05-15T23:19:39.343 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.343 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d92170cb:::benchmark_data_smithi090_104448_object165562:head# 2024-05-15T23:19:39.343 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.343 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9232673:::benchmark_data_smithi090_104448_object162125:head# 2024-05-15T23:19:39.343 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.343 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d923330d:::benchmark_data_smithi090_104448_object164353:head# 2024-05-15T23:19:39.343 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.343 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9245f93:::benchmark_data_smithi090_104448_object172982:head# 2024-05-15T23:19:39.343 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.343 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d92d626a:::benchmark_data_smithi090_104448_object173211:head# 2024-05-15T23:19:39.343 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.343 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d932fade:::benchmark_data_smithi090_104448_object168776:head# 2024-05-15T23:19:39.344 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.344 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d935ddb6:::benchmark_data_smithi090_104448_object158671:head# 2024-05-15T23:19:39.344 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.344 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d938fbb6:::benchmark_data_smithi090_104448_object164638:head# 2024-05-15T23:19:39.344 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.344 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d93a75f7:::benchmark_data_smithi090_104448_object171795:head# 2024-05-15T23:19:39.344 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.344 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d93edd9b:::benchmark_data_smithi090_104448_object161254:head# 2024-05-15T23:19:39.344 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.344 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d93fab08:::benchmark_data_smithi090_104448_object170862:head# 2024-05-15T23:19:39.344 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.344 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9456850:::benchmark_data_smithi090_104448_object164205:head# 2024-05-15T23:19:39.344 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.344 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d948f307:::benchmark_data_smithi090_104448_object166415:head# 2024-05-15T23:19:39.344 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.345 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d95750d4:::benchmark_data_smithi090_104448_object171838:head# 2024-05-15T23:19:39.345 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.345 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d96d00d3:::benchmark_data_smithi090_104448_object164219:head# 2024-05-15T23:19:39.345 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.345 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d96d0973:::benchmark_data_smithi090_104448_object171870:head# 2024-05-15T23:19:39.345 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.345 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d96ea22a:::benchmark_data_smithi090_104448_object159509:head# 2024-05-15T23:19:39.345 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.345 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d96ec0ae:::benchmark_data_smithi090_104448_object169569:head# 2024-05-15T23:19:39.345 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.345 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d980b4fb:::benchmark_data_smithi090_104448_object167196:head# 2024-05-15T23:19:39.345 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.345 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9901deb:::benchmark_data_smithi090_104448_object168212:head# 2024-05-15T23:19:39.345 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.346 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d99152d7:::benchmark_data_smithi090_104448_object161691:head# 2024-05-15T23:19:39.346 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.346 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d99d714e:::benchmark_data_smithi090_104448_object162319:head# 2024-05-15T23:19:39.346 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.346 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9a1768e:::benchmark_data_smithi090_104448_object167039:head# 2024-05-15T23:19:39.346 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.346 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9a1ff29:::benchmark_data_smithi090_104448_object160278:head# 2024-05-15T23:19:39.346 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.346 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9a45238:::benchmark_data_smithi090_104448_object171376:head# 2024-05-15T23:19:39.346 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.346 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9a6ecf0:::benchmark_data_smithi090_104448_object165817:head# 2024-05-15T23:19:39.346 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.346 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9af4333:::benchmark_data_smithi090_104448_object164815:head# 2024-05-15T23:19:39.346 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.347 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9b1d7a3:::benchmark_data_smithi090_104448_object163655:head# 2024-05-15T23:19:39.347 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.347 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9b35de0:::benchmark_data_smithi090_104448_object166631:head# 2024-05-15T23:19:39.347 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.347 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9bcaac9:::benchmark_data_smithi090_104448_object161794:head# 2024-05-15T23:19:39.347 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.347 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9bf8e42:::benchmark_data_smithi090_104448_object158097:head# 2024-05-15T23:19:39.347 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.347 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9c3af7f:::benchmark_data_smithi090_104448_object164468:head# 2024-05-15T23:19:39.347 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.347 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9c4f976:::benchmark_data_smithi090_104448_object162323:head# 2024-05-15T23:19:39.347 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.347 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9c8740d:::benchmark_data_smithi090_104448_object159183:head# 2024-05-15T23:19:39.347 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.347 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9ce7c73:::benchmark_data_smithi090_104448_object164994:head# 2024-05-15T23:19:39.348 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.348 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9cfcdc0:::benchmark_data_smithi090_104448_object167446:head# 2024-05-15T23:19:39.348 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.348 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9d742d1:::benchmark_data_smithi090_104448_object167221:head# 2024-05-15T23:19:39.348 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.348 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9d7d7fb:::benchmark_data_smithi090_104448_object161909:head# 2024-05-15T23:19:39.348 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.348 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:39 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:19:39.348 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:39 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:19:39.349 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:39 smithi084 ceph-mon[39784]: osdmap e371: 12 total, 11 up, 10 in 2024-05-15T23:19:39.349 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:39 smithi084 ceph-mon[39784]: pgmap v1021: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:39.351 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:39 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:19:39.351 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:39 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:19:39.351 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:39 smithi088 ceph-mon[44699]: osdmap e371: 12 total, 11 up, 10 in 2024-05-15T23:19:39.351 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:39 smithi088 ceph-mon[44699]: pgmap v1021: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:39.410 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9da44eb:::benchmark_data_smithi090_104448_object164249:head# 2024-05-15T23:19:39.410 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.410 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9e27484:::benchmark_data_smithi090_104448_object161725:head# 2024-05-15T23:19:39.411 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.411 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9e30ca6:::benchmark_data_smithi090_104448_object172012:head# 2024-05-15T23:19:39.411 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.411 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9e4c4d0:::benchmark_data_smithi090_104448_object157808:head# 2024-05-15T23:19:39.411 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.411 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9e57835:::benchmark_data_smithi090_104448_object157502:head# 2024-05-15T23:19:39.411 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.411 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9ed92d4:::benchmark_data_smithi090_104448_object157116:head# 2024-05-15T23:19:39.411 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.411 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9edeacb:::benchmark_data_smithi090_104448_object161778:head# 2024-05-15T23:19:39.411 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.411 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9ef1d91:::benchmark_data_smithi090_104448_object171496:head# 2024-05-15T23:19:39.411 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.411 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9f7ff47:::benchmark_data_smithi090_104448_object164463:head# 2024-05-15T23:19:39.412 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.412 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9f99b11:::benchmark_data_smithi090_104448_object171546:head# 2024-05-15T23:19:39.412 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.412 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:d9fc37ab:::benchmark_data_smithi090_104448_object160159:head# 2024-05-15T23:19:39.412 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.412 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da0e460b:::benchmark_data_smithi090_104448_object169164:head# 2024-05-15T23:19:39.412 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.412 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da1294b4:::benchmark_data_smithi090_104448_object159887:head# 2024-05-15T23:19:39.412 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.412 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da1ca531:::benchmark_data_smithi090_104448_object172278:head# 2024-05-15T23:19:39.412 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.412 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da1cbb7d:::benchmark_data_smithi090_104448_object170509:head# 2024-05-15T23:19:39.412 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da283037:::benchmark_data_smithi090_104448_object159091:head# 2024-05-15T23:19:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da28ec05:::benchmark_data_smithi090_104448_object162126:head# 2024-05-15T23:19:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da298127:::benchmark_data_smithi090_104448_object164675:head# 2024-05-15T23:19:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da2d183c:::benchmark_data_smithi090_104448_object170426:head# 2024-05-15T23:19:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da2dd8b1:::benchmark_data_smithi090_104448_object158701:head# 2024-05-15T23:19:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da2de93f:::benchmark_data_smithi090_104448_object169689:head# 2024-05-15T23:19:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da2dfaac:::benchmark_data_smithi090_104448_object162109:head# 2024-05-15T23:19:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da30f6e8:::benchmark_data_smithi090_104448_object163820:head# 2024-05-15T23:19:39.414 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.414 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da30fa70:::benchmark_data_smithi090_104448_object167537:head# 2024-05-15T23:19:39.414 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.414 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da36eea9:::benchmark_data_smithi090_104448_object168963:head# 2024-05-15T23:19:39.414 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.414 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da3bc558:::benchmark_data_smithi090_104448_object157376:head# 2024-05-15T23:19:39.414 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.414 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da3e00e1:::benchmark_data_smithi090_104448_object168478:head# 2024-05-15T23:19:39.414 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.414 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da3f85be:::benchmark_data_smithi090_104448_object172000:head# 2024-05-15T23:19:39.414 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.414 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da42f909:::benchmark_data_smithi090_104448_object162996:head# 2024-05-15T23:19:39.414 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.414 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da4710a0:::benchmark_data_smithi090_104448_object161043:head# 2024-05-15T23:19:39.415 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.415 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da47e66e:::benchmark_data_smithi090_104448_object167274:head# 2024-05-15T23:19:39.415 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.415 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da4be431:::benchmark_data_smithi090_104448_object169563:head# 2024-05-15T23:19:39.415 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.415 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da4c4a70:::benchmark_data_smithi090_104448_object163510:head# 2024-05-15T23:19:39.415 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.415 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da585115:::benchmark_data_smithi090_104448_object161465:head# 2024-05-15T23:19:39.415 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.415 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da59c020:::benchmark_data_smithi090_104448_object163487:head# 2024-05-15T23:19:39.415 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.415 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da5bf086:::benchmark_data_smithi090_104448_object161441:head# 2024-05-15T23:19:39.415 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.415 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da6ee82e:::benchmark_data_smithi090_104448_object171137:head# 2024-05-15T23:19:39.416 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.416 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da70685e:::benchmark_data_smithi090_104448_object169479:head# 2024-05-15T23:19:39.416 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.416 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da752ed2:::benchmark_data_smithi090_104448_object167958:head# 2024-05-15T23:19:39.416 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.416 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da78427d:::benchmark_data_smithi090_104448_object172861:head# 2024-05-15T23:19:39.416 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.416 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da7b18ff:::benchmark_data_smithi090_104448_object170944:head# 2024-05-15T23:19:39.416 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.416 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da7cc4f2:::benchmark_data_smithi090_104448_object159102:head# 2024-05-15T23:19:39.416 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.416 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da7e537a:::benchmark_data_smithi090_104448_object165928:head# 2024-05-15T23:19:39.416 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.416 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da7e8d73:::benchmark_data_smithi090_104448_object157907:head# 2024-05-15T23:19:39.416 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.417 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da85010d:::benchmark_data_smithi090_104448_object166266:head# 2024-05-15T23:19:39.417 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.417 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da8a10f6:::benchmark_data_smithi090_104448_object167301:head# 2024-05-15T23:19:39.417 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.417 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da8c5f17:::benchmark_data_smithi090_104448_object172131:head# 2024-05-15T23:19:39.417 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.417 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da9288e2:::benchmark_data_smithi090_104448_object158935:head# 2024-05-15T23:19:39.417 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.417 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da9b00fe:::benchmark_data_smithi090_104448_object167485:head# 2024-05-15T23:19:39.417 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.417 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da9bb4cf:::benchmark_data_smithi090_104448_object171677:head# 2024-05-15T23:19:39.417 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.417 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da9cb2e8:::benchmark_data_smithi090_104448_object166332:head# 2024-05-15T23:19:39.417 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.418 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da9dce5a:::benchmark_data_smithi090_104448_object169738:head# 2024-05-15T23:19:39.418 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.418 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:da9f7078:::benchmark_data_smithi090_104448_object171693:head# 2024-05-15T23:19:39.418 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.418 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:daa3589e:::benchmark_data_smithi090_104448_object161314:head# 2024-05-15T23:19:39.418 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.418 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:daa57a8a:::benchmark_data_smithi090_104448_object165342:head# 2024-05-15T23:19:39.418 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.418 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dab21112:::benchmark_data_smithi090_104448_object159292:head# 2024-05-15T23:19:39.418 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.418 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dab4c9e5:::benchmark_data_smithi090_104448_object165838:head# 2024-05-15T23:19:39.418 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.418 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dab96f7f:::benchmark_data_smithi090_104448_object169697:head# 2024-05-15T23:19:39.418 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.419 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dac24c02:::benchmark_data_smithi090_104448_object164484:head# 2024-05-15T23:19:39.419 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.419 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dad1dd60:::benchmark_data_smithi090_104448_object172245:head# 2024-05-15T23:19:39.419 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.419 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dad361b9:::benchmark_data_smithi090_104448_object156655:head# 2024-05-15T23:19:39.419 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.419 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dad6377f:::benchmark_data_smithi090_104448_object163986:head# 2024-05-15T23:19:39.419 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.419 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dad6cf58:::benchmark_data_smithi090_104448_object157879:head# 2024-05-15T23:19:39.419 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.419 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dadb4b83:::benchmark_data_smithi090_104448_object168865:head# 2024-05-15T23:19:39.419 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.419 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dae2cf27:::benchmark_data_smithi090_104448_object159073:head# 2024-05-15T23:19:39.419 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.419 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dae3789e:::benchmark_data_smithi090_104448_object166701:head# 2024-05-15T23:19:39.420 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.420 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dae3bfc0:::benchmark_data_smithi090_104448_object162575:head# 2024-05-15T23:19:39.420 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.420 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:daee60eb:::benchmark_data_smithi090_104448_object169872:head# 2024-05-15T23:19:39.420 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.420 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:daeeff35:::benchmark_data_smithi090_104448_object165912:head# 2024-05-15T23:19:39.420 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.420 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:daf0abef:::benchmark_data_smithi090_104448_object157794:head# 2024-05-15T23:19:39.420 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.420 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:daf1aa1c:::benchmark_data_smithi090_104448_object161865:head# 2024-05-15T23:19:39.420 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.420 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:daf5a897:::benchmark_data_smithi090_104448_object168238:head# 2024-05-15T23:19:39.420 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.420 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:daf6718e:::benchmark_data_smithi090_104448_object167375:head# 2024-05-15T23:19:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dafaada1:::benchmark_data_smithi090_104448_object165689:head# 2024-05-15T23:19:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dafb146e:::benchmark_data_smithi090_104448_object157405:head# 2024-05-15T23:19:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dafd5363:::benchmark_data_smithi090_104448_object157267:head# 2024-05-15T23:19:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db010433:::benchmark_data_smithi090_104448_object171018:head# 2024-05-15T23:19:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db07922d:::benchmark_data_smithi090_104448_object157571:head# 2024-05-15T23:19:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db1287b4:::benchmark_data_smithi090_104448_object169086:head# 2024-05-15T23:19:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db12d1a2:::benchmark_data_smithi090_104448_object162444:head# 2024-05-15T23:19:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.422 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db175c29:::benchmark_data_smithi090_104448_object158670:head# 2024-05-15T23:19:39.422 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.422 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db23bc0d:::benchmark_data_smithi090_104448_object160685:head# 2024-05-15T23:19:39.422 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.422 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db270727:::benchmark_data_smithi090_104448_object161325:head# 2024-05-15T23:19:39.422 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.422 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db289958:::benchmark_data_smithi090_104448_object167084:head# 2024-05-15T23:19:39.422 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.422 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db2bff47:::benchmark_data_smithi090_104448_object166458:head# 2024-05-15T23:19:39.422 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.422 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db30b2a0:::benchmark_data_smithi090_104448_object164380:head# 2024-05-15T23:19:39.422 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.422 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db328b96:::benchmark_data_smithi090_104448_object172814:head# 2024-05-15T23:19:39.422 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.423 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db358dc0:::benchmark_data_smithi090_104448_object165256:head# 2024-05-15T23:19:39.423 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.423 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db3cbb61:::benchmark_data_smithi090_104448_object159371:head# 2024-05-15T23:19:39.423 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.423 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db3ce536:::benchmark_data_smithi090_104448_object167859:head# 2024-05-15T23:19:39.423 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.423 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db411411:::benchmark_data_smithi090_104448_object160617:head# 2024-05-15T23:19:39.423 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.423 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db425b27:::benchmark_data_smithi090_104448_object157188:head# 2024-05-15T23:19:39.423 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.423 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db47a5f8:::benchmark_data_smithi090_104448_object166688:head# 2024-05-15T23:19:39.423 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.423 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db491060:::benchmark_data_smithi090_104448_object170412:head# 2024-05-15T23:19:39.423 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.423 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db4b8e72:::benchmark_data_smithi090_104448_object166209:head# 2024-05-15T23:19:39.424 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.424 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db4fa25e:::benchmark_data_smithi090_104448_object162381:head# 2024-05-15T23:19:39.424 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.424 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db5059c0:::benchmark_data_smithi090_104448_object160424:head# 2024-05-15T23:19:39.424 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.424 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db576216:::benchmark_data_smithi090_104448_object166290:head# 2024-05-15T23:19:39.424 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.424 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db5d7e8d:::benchmark_data_smithi090_104448_object158143:head# 2024-05-15T23:19:39.424 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.424 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db5dd314:::benchmark_data_smithi090_104448_object171398:head# 2024-05-15T23:19:39.424 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.424 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db5fd73d:::benchmark_data_smithi090_104448_object163994:head# 2024-05-15T23:19:39.424 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.425 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db643ae8:::benchmark_data_smithi090_104448_object166666:head# 2024-05-15T23:19:39.425 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.425 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db64b30f:::benchmark_data_smithi090_104448_object173153:head# 2024-05-15T23:19:39.425 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.425 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db66807a:::benchmark_data_smithi090_104448_object171485:head# 2024-05-15T23:19:39.425 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.425 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db6dfce9:::benchmark_data_smithi090_104448_object158492:head# 2024-05-15T23:19:39.425 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.425 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db766d86:::benchmark_data_smithi090_104448_object172253:head# 2024-05-15T23:19:39.425 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.425 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db7c95b3:::benchmark_data_smithi090_104448_object158686:head# 2024-05-15T23:19:39.425 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.425 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db7f88a8:::benchmark_data_smithi090_104448_object161093:head# 2024-05-15T23:19:39.425 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.425 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db81d6df:::benchmark_data_smithi090_104448_object161225:head# 2024-05-15T23:19:39.426 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.426 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db8d439a:::benchmark_data_smithi090_104448_object156839:head# 2024-05-15T23:19:39.426 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.426 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db8f1ae7:::benchmark_data_smithi090_104448_object170119:head# 2024-05-15T23:19:39.426 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.426 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db8fb233:::benchmark_data_smithi090_104448_object158641:head# 2024-05-15T23:19:39.426 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.426 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db955315:::benchmark_data_smithi090_104448_object163460:head# 2024-05-15T23:19:39.426 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.426 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:db9b171f:::benchmark_data_smithi090_104448_object159771:head# 2024-05-15T23:19:39.426 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.426 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dba23c65:::benchmark_data_smithi090_104448_object170504:head# 2024-05-15T23:19:39.426 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbaf0fcd:::benchmark_data_smithi090_104448_object158843:head# 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbbfa1fe:::benchmark_data_smithi090_104448_object161988:head# 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbc37c10:::benchmark_data_smithi090_104448_object165904:head# 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbc5e53e:::benchmark_data_smithi090_104448_object168038:head# 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbc6348d:::benchmark_data_smithi090_104448_object160820:head# 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbc8ec81:::benchmark_data_smithi090_104448_object160009:head# 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbca6b11:::benchmark_data_smithi090_104448_object158765:head# 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbcc1d2e:::benchmark_data_smithi090_104448_object171857:head# 2024-05-15T23:19:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.428 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbcd46ec:::benchmark_data_smithi090_104448_object168498:head# 2024-05-15T23:19:39.428 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.428 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbce9a9b:::benchmark_data_smithi090_104448_object169733:head# 2024-05-15T23:19:39.428 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.428 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbcf35b8:::benchmark_data_smithi090_104448_object163625:head# 2024-05-15T23:19:39.428 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.428 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbd12397:::benchmark_data_smithi090_104448_object173004:head# 2024-05-15T23:19:39.428 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.428 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbd7d533:::benchmark_data_smithi090_104448_object168676:head# 2024-05-15T23:19:39.428 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.428 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbdbc061:::benchmark_data_smithi090_104448_object164557:head# 2024-05-15T23:19:39.428 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.428 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbddb243:::benchmark_data_smithi090_104448_object166902:head# 2024-05-15T23:19:39.428 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.429 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbea4876:::benchmark_data_smithi090_104448_object157329:head# 2024-05-15T23:19:39.429 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.429 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbee79c0:::benchmark_data_smithi090_104448_object168766:head# 2024-05-15T23:19:39.429 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.429 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbf23b17:::benchmark_data_smithi090_104448_object170374:head# 2024-05-15T23:19:39.429 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.429 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbf723a7:::benchmark_data_smithi090_104448_object164012:head# 2024-05-15T23:19:39.429 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.429 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbfa64f1:::benchmark_data_smithi090_104448_object169071:head# 2024-05-15T23:19:39.429 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.429 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dbfbfded:::benchmark_data_smithi090_104448_object156803:head# 2024-05-15T23:19:39.429 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.429 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc03ed47:::benchmark_data_smithi090_104448_object172212:head# 2024-05-15T23:19:39.429 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.429 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc064532:::benchmark_data_smithi090_104448_object167828:head# 2024-05-15T23:19:39.430 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.430 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc0c95af:::benchmark_data_smithi090_104448_object164829:head# 2024-05-15T23:19:39.430 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.430 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc0cfacb:::benchmark_data_smithi090_104448_object163832:head# 2024-05-15T23:19:39.430 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.430 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc0daa4a:::benchmark_data_smithi090_104448_object163743:head# 2024-05-15T23:19:39.430 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.430 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc13f871:::benchmark_data_smithi090_104448_object169982:head# 2024-05-15T23:19:39.430 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.430 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc183bb3:::benchmark_data_smithi090_104448_object171120:head# 2024-05-15T23:19:39.430 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.430 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc1c4c3b:::benchmark_data_smithi090_104448_object164612:head# 2024-05-15T23:19:39.430 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.431 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc1d75ca:::benchmark_data_smithi090_104448_object158291:head# 2024-05-15T23:19:39.431 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.431 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc214fc7:::benchmark_data_smithi090_104448_object163426:head# 2024-05-15T23:19:39.431 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.431 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc233de2:::benchmark_data_smithi090_104448_object158660:head# 2024-05-15T23:19:39.431 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.431 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc289b47:::benchmark_data_smithi090_104448_object166947:head# 2024-05-15T23:19:39.431 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.431 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc28f98d:::benchmark_data_smithi090_104448_object158673:head# 2024-05-15T23:19:39.431 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.431 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc2b99cf:::benchmark_data_smithi090_104448_object167000:head# 2024-05-15T23:19:39.431 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.431 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc301609:::benchmark_data_smithi090_104448_object172806:head# 2024-05-15T23:19:39.431 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc31fcba:::benchmark_data_smithi090_104448_object160022:head# 2024-05-15T23:19:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc35c7af:::benchmark_data_smithi090_104448_object159963:head# 2024-05-15T23:19:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc3a63a8:::benchmark_data_smithi090_104448_object170579:head# 2024-05-15T23:19:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc3ca0ac:::benchmark_data_smithi090_104448_object165728:head# 2024-05-15T23:19:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc41080a:::benchmark_data_smithi090_104448_object170740:head# 2024-05-15T23:19:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc45b01e:::benchmark_data_smithi090_104448_object172417:head# 2024-05-15T23:19:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc559667:::benchmark_data_smithi090_104448_object167545:head# 2024-05-15T23:19:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.432 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc589cce:::benchmark_data_smithi090_104448_object161209:head# 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc5afd84:::benchmark_data_smithi090_104448_object167523:head# 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc620463:::benchmark_data_smithi090_104448_object157931:head# 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc63e241:::benchmark_data_smithi090_104448_object164659:head# 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc677387:::benchmark_data_smithi090_104448_object168841:head# 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc69f5fd:::benchmark_data_smithi090_104448_object159453:head# 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc6c02eb:::benchmark_data_smithi090_104448_object159017:head# 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc6ed56d:::benchmark_data_smithi090_104448_object157660:head# 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc77524f:::benchmark_data_smithi090_104448_object171095:head# 2024-05-15T23:19:39.434 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.434 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc84bc93:::benchmark_data_smithi090_104448_object173021:head# 2024-05-15T23:19:39.434 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.434 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc89a898:::benchmark_data_smithi090_104448_object171246:head# 2024-05-15T23:19:39.434 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.434 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc8bd4ae:::benchmark_data_smithi090_104448_object163194:head# 2024-05-15T23:19:39.434 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.434 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc8d8b85:::benchmark_data_smithi090_104448_object162462:head# 2024-05-15T23:19:39.434 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.434 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc8ec595:::benchmark_data_smithi090_104448_object168434:head# 2024-05-15T23:19:39.434 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.434 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc8fc75a:::benchmark_data_smithi090_104448_object158873:head# 2024-05-15T23:19:39.434 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.434 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc91b079:::benchmark_data_smithi090_104448_object159250:head# 2024-05-15T23:19:39.434 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dc9488fd:::benchmark_data_smithi090_104448_object167376:head# 2024-05-15T23:19:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dca2f879:::benchmark_data_smithi090_104448_object166245:head# 2024-05-15T23:19:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dca6a81c:::benchmark_data_smithi090_104448_object166611:head# 2024-05-15T23:19:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dca9ba8f:::benchmark_data_smithi090_104448_object158557:head# 2024-05-15T23:19:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcaa49a2:::benchmark_data_smithi090_104448_object164061:head# 2024-05-15T23:19:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcad1434:::benchmark_data_smithi090_104448_object163854:head# 2024-05-15T23:19:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcb12e8f:::benchmark_data_smithi090_104448_object168922:head# 2024-05-15T23:19:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcb175ad:::benchmark_data_smithi090_104448_object159156:head# 2024-05-15T23:19:39.436 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.436 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcb50846:::benchmark_data_smithi090_104448_object168249:head# 2024-05-15T23:19:39.436 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.436 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcb60dbd:::benchmark_data_smithi090_104448_object167014:head# 2024-05-15T23:19:39.436 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.436 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcb6b582:::benchmark_data_smithi090_104448_object162920:head# 2024-05-15T23:19:39.436 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.436 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcb6ed11:::benchmark_data_smithi090_104448_object159186:head# 2024-05-15T23:19:39.436 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.436 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcbb44de:::benchmark_data_smithi090_104448_object165660:head# 2024-05-15T23:19:39.436 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.436 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcbbf0a2:::benchmark_data_smithi090_104448_object159930:head# 2024-05-15T23:19:39.436 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.436 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcc05549:::benchmark_data_smithi090_104448_object161448:head# 2024-05-15T23:19:39.437 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.437 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcc0b834:::benchmark_data_smithi090_104448_object170708:head# 2024-05-15T23:19:39.437 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.437 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcc8a7c1:::benchmark_data_smithi090_104448_object160886:head# 2024-05-15T23:19:39.437 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.437 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcc9c32b:::benchmark_data_smithi090_104448_object161663:head# 2024-05-15T23:19:39.437 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.437 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcce6ba6:::benchmark_data_smithi090_104448_object161351:head# 2024-05-15T23:19:39.437 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.437 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcd0a5c2:::benchmark_data_smithi090_104448_object167706:head# 2024-05-15T23:19:39.437 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.437 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcd1112b:::benchmark_data_smithi090_104448_object160189:head# 2024-05-15T23:19:39.437 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.437 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcd19574:::benchmark_data_smithi090_104448_object160147:head# 2024-05-15T23:19:39.438 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.438 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcd2d052:::benchmark_data_smithi090_104448_object161402:head# 2024-05-15T23:19:39.438 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.438 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcd8100c:::benchmark_data_smithi090_104448_object171140:head# 2024-05-15T23:19:39.438 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.438 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcdf7377:::benchmark_data_smithi090_104448_object158060:head# 2024-05-15T23:19:39.438 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.438 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dce1cbea:::benchmark_data_smithi090_104448_object161812:head# 2024-05-15T23:19:39.438 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.438 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dceed1a1:::benchmark_data_smithi090_104448_object170357:head# 2024-05-15T23:19:39.438 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.438 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcf544c4:::benchmark_data_smithi090_104448_object161681:head# 2024-05-15T23:19:39.438 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.439 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcf9633e:::benchmark_data_smithi090_104448_object169946:head# 2024-05-15T23:19:39.439 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.439 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcf99bab:::benchmark_data_smithi090_104448_object158035:head# 2024-05-15T23:19:39.439 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.439 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dcfec32b:::benchmark_data_smithi090_104448_object161132:head# 2024-05-15T23:19:39.439 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.439 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd004d1d:::benchmark_data_smithi090_104448_object163639:head# 2024-05-15T23:19:39.439 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.439 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd0627de:::benchmark_data_smithi090_104448_object161598:head# 2024-05-15T23:19:39.439 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.439 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd08e6ad:::benchmark_data_smithi090_104448_object163619:head# 2024-05-15T23:19:39.439 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.439 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd09b5d9:::benchmark_data_smithi090_104448_object168863:head# 2024-05-15T23:19:39.439 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.439 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd0b4f93:::benchmark_data_smithi090_104448_object166143:head# 2024-05-15T23:19:39.440 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.440 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd0f483f:::benchmark_data_smithi090_104448_object163752:head# 2024-05-15T23:19:39.440 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.440 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd1438b0:::benchmark_data_smithi090_104448_object162486:head# 2024-05-15T23:19:39.440 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.440 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd147ebb:::benchmark_data_smithi090_104448_object167186:head# 2024-05-15T23:19:39.440 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.440 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd152ec9:::benchmark_data_smithi090_104448_object159993:head# 2024-05-15T23:19:39.440 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.440 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd176c11:::benchmark_data_smithi090_104448_object164263:head# 2024-05-15T23:19:39.440 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.440 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd1c83b5:::benchmark_data_smithi090_104448_object158672:head# 2024-05-15T23:19:39.440 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.440 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd1f383e:::benchmark_data_smithi090_104448_object172045:head# 2024-05-15T23:19:39.440 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd203e1e:::benchmark_data_smithi090_104448_object168634:head# 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd441e33:::benchmark_data_smithi090_104448_object157743:head# 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd443b56:::benchmark_data_smithi090_104448_object164498:head# 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd4a6bba:::benchmark_data_smithi090_104448_object158176:head# 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd5134db:::benchmark_data_smithi090_104448_object172590:head# 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd57e547:::benchmark_data_smithi090_104448_object169155:head# 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd637e63:::benchmark_data_smithi090_104448_object172655:head# 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd642628:::benchmark_data_smithi090_104448_object160433:head# 2024-05-15T23:19:39.441 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.442 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd64c2f7:::benchmark_data_smithi090_104448_object166526:head# 2024-05-15T23:19:39.442 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.442 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd68cbcc:::benchmark_data_smithi090_104448_object167623:head# 2024-05-15T23:19:39.442 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.442 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd695232:::benchmark_data_smithi090_104448_object169269:head# 2024-05-15T23:19:39.442 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.442 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd6b4ba7:::benchmark_data_smithi090_104448_object169833:head# 2024-05-15T23:19:39.442 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.442 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd6f9f46:::benchmark_data_smithi090_104448_object163646:head# 2024-05-15T23:19:39.442 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.442 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd70ea81:::benchmark_data_smithi090_104448_object165327:head# 2024-05-15T23:19:39.442 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.442 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd7232cd:::benchmark_data_smithi090_104448_object170919:head# 2024-05-15T23:19:39.442 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.443 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd7b624b:::benchmark_data_smithi090_104448_object160372:head# 2024-05-15T23:19:39.443 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.443 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd85b05b:::benchmark_data_smithi090_104448_object172046:head# 2024-05-15T23:19:39.443 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.443 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd8745f4:::benchmark_data_smithi090_104448_object167872:head# 2024-05-15T23:19:39.443 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.443 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd89b13d:::benchmark_data_smithi090_104448_object167234:head# 2024-05-15T23:19:39.443 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.443 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd89cad8:::benchmark_data_smithi090_104448_object161701:head# 2024-05-15T23:19:39.443 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.443 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd8eebfd:::benchmark_data_smithi090_104448_object162689:head# 2024-05-15T23:19:39.443 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.443 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dd9155fe:::benchmark_data_smithi090_104448_object165734:head# 2024-05-15T23:19:39.443 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.443 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dda16d04:::benchmark_data_smithi090_104448_object165522:head# 2024-05-15T23:19:39.444 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.444 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dda68a26:::benchmark_data_smithi090_104448_object161420:head# 2024-05-15T23:19:39.444 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.444 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ddaf70ee:::benchmark_data_smithi090_104448_object159947:head# 2024-05-15T23:19:39.444 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.444 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ddb04a0a:::benchmark_data_smithi090_104448_object168365:head# 2024-05-15T23:19:39.444 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.444 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ddb17e86:::benchmark_data_smithi090_104448_object162965:head# 2024-05-15T23:19:39.444 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.444 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ddbe7603:::benchmark_data_smithi090_104448_object167691:head# 2024-05-15T23:19:39.444 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.444 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ddbfddf6:::benchmark_data_smithi090_104448_object163756:head# 2024-05-15T23:19:39.444 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.444 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ddc1c731:::benchmark_data_smithi090_104448_object161106:head# 2024-05-15T23:19:39.444 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.445 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ddc6df24:::benchmark_data_smithi090_104448_object161327:head# 2024-05-15T23:19:39.445 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.445 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ddc6e8c3:::benchmark_data_smithi090_104448_object161504:head# 2024-05-15T23:19:39.445 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.445 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ddcb5754:::benchmark_data_smithi090_104448_object166016:head# 2024-05-15T23:19:39.445 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.445 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ddd0d606:::benchmark_data_smithi090_104448_object166725:head# 2024-05-15T23:19:39.445 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.445 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ddd39ff5:::benchmark_data_smithi090_104448_object167321:head# 2024-05-15T23:19:39.445 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.445 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dddbaef2:::benchmark_data_smithi090_104448_object168042:head# 2024-05-15T23:19:39.445 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.445 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dddeb8e4:::benchmark_data_smithi090_104448_object168962:head# 2024-05-15T23:19:39.445 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.445 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dde4a6df:::benchmark_data_smithi090_104448_object159199:head# 2024-05-15T23:19:39.446 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.446 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dded3d8e:::benchmark_data_smithi090_104448_object157007:head# 2024-05-15T23:19:39.446 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.446 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ddf4fd7f:::benchmark_data_smithi090_104448_object167593:head# 2024-05-15T23:19:39.446 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.446 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ddfad0d2:::benchmark_data_smithi090_104448_object158871:head# 2024-05-15T23:19:39.446 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.446 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ddfb3462:::benchmark_data_smithi090_104448_object170973:head# 2024-05-15T23:19:39.446 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.446 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ddfbd370:::benchmark_data_smithi090_104448_object160167:head# 2024-05-15T23:19:39.446 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.446 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de034602:::benchmark_data_smithi090_104448_object159979:head# 2024-05-15T23:19:39.446 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.446 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de0507f8:::benchmark_data_smithi090_104448_object164435:head# 2024-05-15T23:19:39.446 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.447 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de076b60:::benchmark_data_smithi090_104448_object160132:head# 2024-05-15T23:19:39.447 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.447 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de08b65f:::benchmark_data_smithi090_104448_object163783:head# 2024-05-15T23:19:39.447 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.447 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de0a0abc:::benchmark_data_smithi090_104448_object157849:head# 2024-05-15T23:19:39.447 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.447 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de0cf46e:::benchmark_data_smithi090_104448_object172146:head# 2024-05-15T23:19:39.447 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.447 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de106580:::benchmark_data_smithi090_104448_object165701:head# 2024-05-15T23:19:39.447 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.447 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de10807e:::benchmark_data_smithi090_104448_object158512:head# 2024-05-15T23:19:39.447 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.447 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de124199:::benchmark_data_smithi090_104448_object167939:head# 2024-05-15T23:19:39.448 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.448 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de144918:::benchmark_data_smithi090_104448_object169601:head# 2024-05-15T23:19:39.448 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.448 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de184b79:::benchmark_data_smithi090_104448_object158237:head# 2024-05-15T23:19:39.448 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.448 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de1b9e8f:::benchmark_data_smithi090_104448_object160263:head# 2024-05-15T23:19:39.448 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.448 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de207208:::benchmark_data_smithi090_104448_object170773:head# 2024-05-15T23:19:39.448 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.448 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de21d431:::benchmark_data_smithi090_104448_object165681:head# 2024-05-15T23:19:39.448 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.448 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de250d9d:::benchmark_data_smithi090_104448_object170549:head# 2024-05-15T23:19:39.448 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.448 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de26b12b:::benchmark_data_smithi090_104448_object167486:head# 2024-05-15T23:19:39.448 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.449 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de295ccf:::benchmark_data_smithi090_104448_object160725:head# 2024-05-15T23:19:39.449 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.449 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de2a2a22:::benchmark_data_smithi090_104448_object163940:head# 2024-05-15T23:19:39.449 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.449 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de2e6125:::benchmark_data_smithi090_104448_object171139:head# 2024-05-15T23:19:39.449 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.449 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de2f4253:::benchmark_data_smithi090_104448_object159459:head# 2024-05-15T23:19:39.449 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.449 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de3545f9:::benchmark_data_smithi090_104448_object165760:head# 2024-05-15T23:19:39.449 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.449 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de376250:::benchmark_data_smithi090_104448_object170240:head# 2024-05-15T23:19:39.449 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.449 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de38cd47:::benchmark_data_smithi090_104448_object163843:head# 2024-05-15T23:19:39.449 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.449 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de3af0f0:::benchmark_data_smithi090_104448_object168349:head# 2024-05-15T23:19:39.450 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.450 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de3f4d58:::benchmark_data_smithi090_104448_object167264:head# 2024-05-15T23:19:39.450 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.450 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de3faedd:::benchmark_data_smithi090_104448_object170282:head# 2024-05-15T23:19:39.450 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.450 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de480914:::benchmark_data_smithi090_104448_object159758:head# 2024-05-15T23:19:39.450 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.450 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de4a4643:::benchmark_data_smithi090_104448_object170105:head# 2024-05-15T23:19:39.450 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.450 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de4b72d7:::benchmark_data_smithi090_104448_object168937:head# 2024-05-15T23:19:39.450 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.450 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de4f2dca:::benchmark_data_smithi090_104448_object162927:head# 2024-05-15T23:19:39.450 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.451 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de505a8b:::benchmark_data_smithi090_104448_object158043:head# 2024-05-15T23:19:39.451 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.451 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de53da8b:::benchmark_data_smithi090_104448_object167992:head# 2024-05-15T23:19:39.451 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.451 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de595409:::benchmark_data_smithi090_104448_object160706:head# 2024-05-15T23:19:39.451 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.451 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de59beff:::benchmark_data_smithi090_104448_object162390:head# 2024-05-15T23:19:39.454 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.454 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de5c134f:::benchmark_data_smithi090_104448_object170373:head# 2024-05-15T23:19:39.454 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.454 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de60bb4b:::benchmark_data_smithi090_104448_object166177:head# 2024-05-15T23:19:39.454 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.454 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de64e2ef:::benchmark_data_smithi090_104448_object156780:head# 2024-05-15T23:19:39.454 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.454 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de6b9e38:::benchmark_data_smithi090_104448_object165333:head# 2024-05-15T23:19:39.455 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.455 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de6c9ca8:::benchmark_data_smithi090_104448_object160688:head# 2024-05-15T23:19:39.455 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.455 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de6cc179:::benchmark_data_smithi090_104448_object166311:head# 2024-05-15T23:19:39.455 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.455 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de7a5a41:::benchmark_data_smithi090_104448_object171708:head# 2024-05-15T23:19:39.455 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.455 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de7afed5:::benchmark_data_smithi090_104448_object172628:head# 2024-05-15T23:19:39.455 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.455 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de81a362:::benchmark_data_smithi090_104448_object164708:head# 2024-05-15T23:19:39.455 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.455 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de88e094:::benchmark_data_smithi090_104448_object162570:head# 2024-05-15T23:19:39.455 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.455 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de987865:::benchmark_data_smithi090_104448_object165631:head# 2024-05-15T23:19:39.456 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.456 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de9a97f1:::benchmark_data_smithi090_104448_object168888:head# 2024-05-15T23:19:39.456 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.456 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de9ab80e:::benchmark_data_smithi090_104448_object158056:head# 2024-05-15T23:19:39.456 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.456 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:de9bb8f0:::benchmark_data_smithi090_104448_object172474:head# 2024-05-15T23:19:39.456 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.456 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dea2cdef:::benchmark_data_smithi090_104448_object156931:head# 2024-05-15T23:19:39.456 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.456 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dea44fb7:::benchmark_data_smithi090_104448_object168948:head# 2024-05-15T23:19:39.456 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.456 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:deaceb2c:::benchmark_data_smithi090_104448_object162045:head# 2024-05-15T23:19:39.456 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.456 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:deb1ca6a:::benchmark_data_smithi090_104448_object168686:head# 2024-05-15T23:19:39.457 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.457 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:deb77611:::benchmark_data_smithi090_104448_object171293:head# 2024-05-15T23:19:39.457 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.457 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dec29a9c:::benchmark_data_smithi090_104448_object168904:head# 2024-05-15T23:19:39.457 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.457 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dec4068c:::benchmark_data_smithi090_104448_object172651:head# 2024-05-15T23:19:39.457 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.457 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dec9b76b:::benchmark_data_smithi090_104448_object162039:head# 2024-05-15T23:19:39.457 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.457 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:decdb4ee:::benchmark_data_smithi090_104448_object166180:head# 2024-05-15T23:19:39.457 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.457 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ded23a54:::benchmark_data_smithi090_104448_object169608:head# 2024-05-15T23:19:39.457 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.457 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ded29b35:::benchmark_data_smithi090_104448_object172534:head# 2024-05-15T23:19:39.458 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.458 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ded2b364:::benchmark_data_smithi090_104448_object165876:head# 2024-05-15T23:19:39.458 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.458 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ded2ce50:::benchmark_data_smithi090_104448_object168411:head# 2024-05-15T23:19:39.458 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.458 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ded42f31:::benchmark_data_smithi090_104448_object172026:head# 2024-05-15T23:19:39.458 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.458 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ded4e470:::benchmark_data_smithi090_104448_object160896:head# 2024-05-15T23:19:39.458 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.458 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ded558da:::benchmark_data_smithi090_104448_object169061:head# 2024-05-15T23:19:39.458 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.458 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:ded77225:::benchmark_data_smithi090_104448_object168362:head# 2024-05-15T23:19:39.458 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.458 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dedb7bf6:::benchmark_data_smithi090_104448_object165383:head# 2024-05-15T23:19:39.459 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.459 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dee3fc74:::benchmark_data_smithi090_104448_object167381:head# 2024-05-15T23:19:39.459 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.459 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dee57081:::benchmark_data_smithi090_104448_object164476:head# 2024-05-15T23:19:39.459 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.459 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dee7ba1d:::benchmark_data_smithi090_104448_object162817:head# 2024-05-15T23:19:39.459 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.459 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dee95e02:::benchmark_data_smithi090_104448_object170088:head# 2024-05-15T23:19:39.459 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.459 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:deecd005:::benchmark_data_smithi090_104448_object166254:head# 2024-05-15T23:19:39.459 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.459 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:def277aa:::benchmark_data_smithi090_104448_object172881:head# 2024-05-15T23:19:39.459 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.459 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:def48251:::benchmark_data_smithi090_104448_object168518:head# 2024-05-15T23:19:39.460 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.460 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:defbdf5c:::benchmark_data_smithi090_104448_object163586:head# 2024-05-15T23:19:39.460 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.460 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df006192:::benchmark_data_smithi090_104448_object165311:head# 2024-05-15T23:19:39.460 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.460 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df072858:::benchmark_data_smithi090_104448_object161743:head# 2024-05-15T23:19:39.460 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.460 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df0b8bf4:::benchmark_data_smithi090_104448_object171472:head# 2024-05-15T23:19:39.460 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.460 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df0e0d34:::benchmark_data_smithi090_104448_object164047:head# 2024-05-15T23:19:39.460 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.460 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df0e12b0:::benchmark_data_smithi090_104448_object167356:head# 2024-05-15T23:19:39.460 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.460 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df10a7dd:::benchmark_data_smithi090_104448_object163987:head# 2024-05-15T23:19:39.460 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.461 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df124242:::benchmark_data_smithi090_104448_object157175:head# 2024-05-15T23:19:39.461 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.461 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df1306e6:::benchmark_data_smithi090_104448_object167982:head# 2024-05-15T23:19:39.461 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.461 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df15f35b:::benchmark_data_smithi090_104448_object166429:head# 2024-05-15T23:19:39.461 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.461 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df177ba7:::benchmark_data_smithi090_104448_object171950:head# 2024-05-15T23:19:39.461 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.461 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df1a37d9:::benchmark_data_smithi090_104448_object172566:head# 2024-05-15T23:19:39.461 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.461 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df1fd80f:::benchmark_data_smithi090_104448_object162050:head# 2024-05-15T23:19:39.461 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.461 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df245c6a:::benchmark_data_smithi090_104448_object172488:head# 2024-05-15T23:19:39.461 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.462 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df3019cf:::benchmark_data_smithi090_104448_object172690:head# 2024-05-15T23:19:39.462 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.462 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df31daf2:::benchmark_data_smithi090_104448_object163341:head# 2024-05-15T23:19:39.462 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.462 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df36db9d:::benchmark_data_smithi090_104448_object163997:head# 2024-05-15T23:19:39.462 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.462 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df4363d4:::benchmark_data_smithi090_104448_object167549:head# 2024-05-15T23:19:39.462 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.462 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df43aff1:::benchmark_data_smithi090_104448_object168600:head# 2024-05-15T23:19:39.462 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.462 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df4e11c3:::benchmark_data_smithi090_104448_object169989:head# 2024-05-15T23:19:39.462 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.462 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df55a67a:::benchmark_data_smithi090_104448_object165744:head# 2024-05-15T23:19:39.462 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.462 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df5b99ce:::benchmark_data_smithi090_104448_object163883:head# 2024-05-15T23:19:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df5cd600:::benchmark_data_smithi090_104448_object157181:head# 2024-05-15T23:19:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df5f11de:::benchmark_data_smithi090_104448_object161715:head# 2024-05-15T23:19:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df5f1ffa:::benchmark_data_smithi090_104448_object160679:head# 2024-05-15T23:19:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df600256:::benchmark_data_smithi090_104448_object160714:head# 2024-05-15T23:19:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df67f9f6:::benchmark_data_smithi090_104448_object162152:head# 2024-05-15T23:19:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df6aa5a3:::benchmark_data_smithi090_104448_object170562:head# 2024-05-15T23:19:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df6b3bc0:::benchmark_data_smithi090_104448_object170570:head# 2024-05-15T23:19:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.464 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df6fbf98:::benchmark_data_smithi090_104448_object172082:head# 2024-05-15T23:19:39.464 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.464 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df724616:::benchmark_data_smithi090_104448_object166076:head# 2024-05-15T23:19:39.464 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.464 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df751915:::benchmark_data_smithi090_104448_object156945:head# 2024-05-15T23:19:39.464 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.464 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df768e20:::benchmark_data_smithi090_104448_object162403:head# 2024-05-15T23:19:39.464 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.464 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df77497c:::benchmark_data_smithi090_104448_object168131:head# 2024-05-15T23:19:39.464 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.464 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df79363f:::benchmark_data_smithi090_104448_object165539:head# 2024-05-15T23:19:39.464 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.464 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df7ad08b:::benchmark_data_smithi090_104448_object158219:head# 2024-05-15T23:19:39.464 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.465 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df8151fb:::benchmark_data_smithi090_104448_object164545:head# 2024-05-15T23:19:39.465 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.465 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df907817:::benchmark_data_smithi090_104448_object167198:head# 2024-05-15T23:19:39.465 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.465 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df90815e:::benchmark_data_smithi090_104448_object159027:head# 2024-05-15T23:19:39.465 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.465 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df90c2c8:::benchmark_data_smithi090_104448_object159197:head# 2024-05-15T23:19:39.465 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.465 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df91c8e4:::benchmark_data_smithi090_104448_object162489:head# 2024-05-15T23:19:39.465 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.465 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df9316e7:::benchmark_data_smithi090_104448_object169972:head# 2024-05-15T23:19:39.465 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.465 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df9456b2:::benchmark_data_smithi090_104448_object160203:head# 2024-05-15T23:19:39.465 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.466 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df994973:::benchmark_data_smithi090_104448_object164167:head# 2024-05-15T23:19:39.466 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.466 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df9af3c8:::benchmark_data_smithi090_104448_object164526:head# 2024-05-15T23:19:39.466 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.466 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df9d425d:::benchmark_data_smithi090_104448_object167353:head# 2024-05-15T23:19:39.466 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.466 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:df9fda65:::benchmark_data_smithi090_104448_object160052:head# 2024-05-15T23:19:39.466 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.466 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfa1ec8e:::benchmark_data_smithi090_104448_object158495:head# 2024-05-15T23:19:39.466 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.466 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfa3ba4a:::benchmark_data_smithi090_104448_object159007:head# 2024-05-15T23:19:39.466 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.466 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfab4fe4:::benchmark_data_smithi090_104448_object162798:head# 2024-05-15T23:19:39.466 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.467 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfcac9e0:::benchmark_data_smithi090_104448_object158849:head# 2024-05-15T23:19:39.467 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.467 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfcec2da:::benchmark_data_smithi090_104448_object161419:head# 2024-05-15T23:19:39.467 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.467 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfcef5c1:::benchmark_data_smithi090_104448_object158000:head# 2024-05-15T23:19:39.467 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.467 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfd03484:::benchmark_data_smithi090_104448_object160478:head# 2024-05-15T23:19:39.467 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.467 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfd19c18:::benchmark_data_smithi090_104448_object172180:head# 2024-05-15T23:19:39.467 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.467 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfd28c5b:::benchmark_data_smithi090_104448_object160888:head# 2024-05-15T23:19:39.467 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.467 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfd69a7d:::benchmark_data_smithi090_104448_object168461:head# 2024-05-15T23:19:39.467 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfd6f441:::benchmark_data_smithi090_104448_object168796:head# 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfdfd416:::benchmark_data_smithi090_104448_object168032:head# 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfe2e25b:::benchmark_data_smithi090_104448_object172137:head# 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfe594dd:::benchmark_data_smithi090_104448_object160936:head# 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfeac963:::benchmark_data_smithi090_104448_object172354:head# 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfebacbb:::benchmark_data_smithi090_104448_object172014:head# 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfef5ad8:::benchmark_data_smithi090_104448_object161064:head# 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dfef6a90:::benchmark_data_smithi090_104448_object172965:head# 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.468 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dff10968:::benchmark_data_smithi090_104448_object162031:head# 2024-05-15T23:19:39.469 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.469 INFO:teuthology.orchestra.run.smithi084.stdout:Write #4:dff12bc3:::benchmark_data_smithi090_104448_object166003:head# 2024-05-15T23:19:39.469 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 0=[]:{} 2024-05-15T23:19:39.469 INFO:teuthology.orchestra.run.smithi084.stdout:write_pg epoch 355 info 4.b( v 355'20619 (318'10600,355'20619] local-lis/les=283/284 n=1035 ec=283/283 lis/c=283/283 les/c/f=284/284/0 sis=283) 2024-05-15T23:19:39.469 INFO:teuthology.orchestra.run.smithi084.stdout:Import successful 2024-05-15T23:19:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:39 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:19:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:39 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:19:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:39 smithi090 ceph-mon[44434]: osdmap e371: 12 total, 11 up, 10 in 2024-05-15T23:19:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:39 smithi090 ceph-mon[44434]: pgmap v1021: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:39.739 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:39.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:39.843 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:39.875 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:39.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:39.900 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:39.964 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:40.001 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:19:40.031 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:40.182 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:19:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:40.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:19:40.267 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:19:40.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:40.291 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:40 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1204674879' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:19:40.291 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:40 smithi084 ceph-mon[39784]: from='client.35861 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:19:40.320 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:40.341 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:40.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:40.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:40.430 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:40 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1204674879' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:19:40.431 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:40 smithi088 ceph-mon[44699]: from='client.35861 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:19:40.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:40.456 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:19:40.456 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:19:40.475 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:40.495 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:40.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:40.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:40 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1204674879' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:19:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:40 smithi090 ceph-mon[44434]: from='client.35861 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:19:40.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:40.628 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:40.644 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/exp.4.b.0 2024-05-15T23:19:40.647 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:40.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:40.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:40.681 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-05-15T23:19:40.682 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-15T23:19:40.682 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:19:40.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:40.779 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:40.799 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:40.802 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:40.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:40.930 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:40.952 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:41.053 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:41.077 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T23:19:41.084 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:41.105 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:41.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:41.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:41.209 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:41.235 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:41.253 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:41.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:41.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:41 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:19:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:41 smithi088 ceph-mon[44699]: from='client.35861 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:19:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:41 smithi088 ceph-mon[44699]: osdmap e372: 12 total, 11 up, 10 in 2024-05-15T23:19:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:41 smithi088 ceph-mon[44699]: pgmap v1023: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:41.354 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:19:41.356 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:19:41.384 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:41.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:41 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:19:41.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:41 smithi090 ceph-mon[44434]: from='client.35861 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:19:41.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:41 smithi090 ceph-mon[44434]: osdmap e372: 12 total, 11 up, 10 in 2024-05-15T23:19:41.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:41 smithi090 ceph-mon[44434]: pgmap v1023: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:19:41.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:41.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:19:41.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:41 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:19:41.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:41 smithi084 ceph-mon[39784]: from='client.35861 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:19:41.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:41 smithi084 ceph-mon[39784]: osdmap e372: 12 total, 11 up, 10 in 2024-05-15T23:19:41.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:41 smithi084 ceph-mon[39784]: pgmap v1023: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:41.686 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:41.713 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:41.810 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:41.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:41.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:41.946 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:41.967 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:41.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:41.971 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:42.072 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:42.150 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:42.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:42.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:42.287 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:42.350 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:42 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:19:42.350 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:42 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:19:42.350 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:42 smithi088 ceph-mon[44699]: osdmap e373: 12 total, 11 up, 10 in 2024-05-15T23:19:42.430 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:42.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:42.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:42.535 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:42.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:42 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:19:42.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:42 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:19:42.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:42 smithi090 ceph-mon[44434]: osdmap e373: 12 total, 11 up, 10 in 2024-05-15T23:19:42.562 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:42.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:42 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:19:42.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:42 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:19:42.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:42 smithi084 ceph-mon[39784]: osdmap e373: 12 total, 11 up, 10 in 2024-05-15T23:19:42.659 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:42.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:42.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:42.763 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:42.790 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:42.938 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:42.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:43.043 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:43.071 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:43.076 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:19:43.095 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:19:43.097 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:43.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:43.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:43.114 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:19:43.134 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:19:43.154 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:19:43.173 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:19:43.193 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:19:43.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:43.212 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:19:43.232 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:19:43.234 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:43.235 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:19:43.235 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:19:43.235 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2024-05-15T23:19:43.236 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:19:43.253 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:19:43.256 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:43.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:43.259 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:43.276 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:19:43.276 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-05-15T23:19:43.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:19:43.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:19:43.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:19:43.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:19:43.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:19:43.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:19:43.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:19:43.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:19:43.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:19:43.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:19:43.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:19:43.360 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:43.390 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:43.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:43 smithi090 ceph-mon[44434]: pgmap v1025: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:43.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:43 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/203687656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:43.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:43 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:43.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:43 smithi088 ceph-mon[44699]: pgmap v1025: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:43.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:43 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/203687656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:43.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:43 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:43 smithi084 ceph-mon[39784]: pgmap v1025: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:43 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/203687656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:43.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:43 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:43.989 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:19:43.990 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:19:44.006 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T23:19:44.267 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:19:44.279 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:44 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.279 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:44 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.279 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:44 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.280 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:44 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.280 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:44 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.280 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:44 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.280 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:44 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.280 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:44 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.280 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:44 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.280 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:44 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.464 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:19:44.490 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:44 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:44 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:44 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:44 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:44 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:44 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:44 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:44 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:44 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:44 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:44 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:44 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:44 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:44 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:44 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:44 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:44 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:44 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:44 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:44 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:44.711 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:44.742 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:44.970 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:44.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:44.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:45.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:45.107 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:45.128 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:45.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:45.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:45.283 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:45.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:45.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:45.388 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:19:45.417 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:45 smithi090 ceph-mon[44434]: pgmap v1026: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:45 smithi088 ceph-mon[44699]: pgmap v1026: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:45.615 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:19:45.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:45.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:19:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:45 smithi084 ceph-mon[39784]: pgmap v1026: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:45.720 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:45.750 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:45.991 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:19:46.100 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:46.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:46.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:46.185 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:19:46.185 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:19:46.185 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:19:46.185 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:19:46.185 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:19:46.204 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:46.233 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:46.297 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_3' created 2024-05-15T23:19:46.436 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:46.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:46.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:46.540 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:19:46.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:46 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:19:46.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:46 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:19:46.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:46 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3189701106' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-05-15T23:19:46.555 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:46 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:19:46.555 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:46 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:19:46.555 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:46 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3189701106' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-05-15T23:19:46.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:46 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:19:46.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:46 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:19:46.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:46 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3189701106' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-05-15T23:19:46.568 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:46.586 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:19:46.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:46.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:19:46.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:19:46.717 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:46.795 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T23:19:47.055 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:19:47.521 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:47 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3189701106' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-05-15T23:19:47.521 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:47 smithi088 ceph-mon[44699]: osdmap e374: 12 total, 11 up, 10 in 2024-05-15T23:19:47.521 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:47 smithi088 ceph-mon[44699]: pgmap v1028: 17 pgs: 5 creating+peering, 11 unknown, 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:47 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3189701106' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-05-15T23:19:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:47 smithi090 ceph-mon[44434]: osdmap e374: 12 total, 11 up, 10 in 2024-05-15T23:19:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:47 smithi090 ceph-mon[44434]: pgmap v1028: 17 pgs: 5 creating+peering, 11 unknown, 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:47.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:47 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3189701106' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-05-15T23:19:47.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:47 smithi084 ceph-mon[39784]: osdmap e374: 12 total, 11 up, 10 in 2024-05-15T23:19:47.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:47 smithi084 ceph-mon[39784]: pgmap v1028: 17 pgs: 5 creating+peering, 11 unknown, 1 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:47.966 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:19:47.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:47.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:19:48.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:48.106 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:48.126 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:48.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:48.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:48.230 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:48.258 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:48.349 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-05-15T23:19:48.357 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:48.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:48.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:48.460 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:48.487 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:48.506 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:48.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:48.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:48 smithi090 ceph-mon[44434]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:19:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:48 smithi090 ceph-mon[44434]: osdmap e375: 12 total, 11 up, 10 in 2024-05-15T23:19:48.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:48 smithi084 ceph-mon[39784]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:19:48.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:48 smithi084 ceph-mon[39784]: osdmap e375: 12 total, 11 up, 10 in 2024-05-15T23:19:48.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:48 smithi088 ceph-mon[44699]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:19:48.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:48 smithi088 ceph-mon[44699]: osdmap e375: 12 total, 11 up, 10 in 2024-05-15T23:19:48.609 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:48.635 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:48.883 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:48.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:48.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:49.014 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:19:49.014 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:19:49.014 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:19:49.014 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:19:49.014 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.014 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:135069 4.a 4:54e24872:::benchmark_data_smithi090_104448_object135068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e306)", 2024-05-15T23:19:49.014 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:16:04.604696+0000", 2024-05-15T23:19:49.014 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 224.40577465800001, 2024-05-15T23:19:49.014 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.215105308, 2024-05-15T23:19:49.014 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.014 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.015 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.015 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.015 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.015 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.015 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 135069 2024-05-15T23:19:49.015 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.015 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.015 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.015 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.015 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604696+0000", 2024-05-15T23:19:49.015 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.015 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.015 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.015 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.015 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604696+0000", 2024-05-15T23:19:49.016 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.016 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.016 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.016 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.016 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604699+0000", 2024-05-15T23:19:49.016 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.514e-06 2024-05-15T23:19:49.016 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.016 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.016 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.016 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604753+0000", 2024-05-15T23:19:49.016 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4558999999999997e-05 2024-05-15T23:19:49.016 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.016 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.016 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.017 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604765+0000", 2024-05-15T23:19:49.017 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2041999999999999e-05 2024-05-15T23:19:49.017 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.017 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.017 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.017 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604773+0000", 2024-05-15T23:19:49.017 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4259999999999997e-06 2024-05-15T23:19:49.017 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.017 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.017 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.017 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604817+0000", 2024-05-15T23:19:49.017 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4700999999999998e-05 2024-05-15T23:19:49.017 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.018 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.018 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.018 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604918+0000", 2024-05-15T23:19:49.018 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00010052000000000001 2024-05-15T23:19:49.018 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.018 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.018 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.018 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.605016+0000", 2024-05-15T23:19:49.018 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8197999999999997e-05 2024-05-15T23:19:49.018 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.018 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.018 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.019 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.609170+0000", 2024-05-15T23:19:49.019 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0041537689999999999 2024-05-15T23:19:49.019 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.019 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.019 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.019 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.819738+0000", 2024-05-15T23:19:49.019 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21056836700000001 2024-05-15T23:19:49.019 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.019 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.019 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.019 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.819773+0000", 2024-05-15T23:19:49.019 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5132999999999997e-05 2024-05-15T23:19:49.020 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.020 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.020 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.020 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.819801+0000", 2024-05-15T23:19:49.020 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8079e-05 2024-05-15T23:19:49.020 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.020 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.020 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.020 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.020 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.020 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:135097 4.a 4:5c1029ec:::benchmark_data_smithi090_104448_object135096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e306)", 2024-05-15T23:19:49.020 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:16:04.626796+0000", 2024-05-15T23:19:49.020 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 224.383674579, 2024-05-15T23:19:49.020 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19805182700000001, 2024-05-15T23:19:49.020 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.021 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.021 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.021 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.021 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.021 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.021 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 135097 2024-05-15T23:19:49.021 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.021 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.021 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.021 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.021 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.626796+0000", 2024-05-15T23:19:49.021 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.021 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.021 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.022 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.022 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.626796+0000", 2024-05-15T23:19:49.022 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.022 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.022 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.022 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.022 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.626800+0000", 2024-05-15T23:19:49.022 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7179999999999998e-06 2024-05-15T23:19:49.022 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.022 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.022 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.022 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.626842+0000", 2024-05-15T23:19:49.022 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2415e-05 2024-05-15T23:19:49.022 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.023 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.023 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.023 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.626858+0000", 2024-05-15T23:19:49.023 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6186000000000001e-05 2024-05-15T23:19:49.023 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.023 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.023 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.023 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.626866+0000", 2024-05-15T23:19:49.023 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0150000000000003e-06 2024-05-15T23:19:49.023 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.023 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.023 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.023 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.626921+0000", 2024-05-15T23:19:49.023 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4092000000000003e-05 2024-05-15T23:19:49.024 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.024 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.024 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.024 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.627016+0000", 2024-05-15T23:19:49.024 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5414000000000005e-05 2024-05-15T23:19:49.024 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.024 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.024 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.024 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.627110+0000", 2024-05-15T23:19:49.024 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4067000000000006e-05 2024-05-15T23:19:49.024 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.024 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.024 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.024 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.631312+0000", 2024-05-15T23:19:49.025 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042020879999999997 2024-05-15T23:19:49.025 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.025 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.025 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.025 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.824777+0000", 2024-05-15T23:19:49.025 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19346450400000001 2024-05-15T23:19:49.025 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.025 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.025 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.025 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.824811+0000", 2024-05-15T23:19:49.025 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4106000000000002e-05 2024-05-15T23:19:49.025 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.025 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.025 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.025 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.824848+0000", 2024-05-15T23:19:49.026 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7221999999999997e-05 2024-05-15T23:19:49.026 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.026 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.026 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.026 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.026 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.026 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:135103 4.a 4:59f15f3d:::benchmark_data_smithi090_104448_object135102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e306)", 2024-05-15T23:19:49.026 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:16:04.632612+0000", 2024-05-15T23:19:49.026 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 224.37785872399999, 2024-05-15T23:19:49.026 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.192344814, 2024-05-15T23:19:49.026 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.026 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.026 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.026 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.026 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.027 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.027 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 135103 2024-05-15T23:19:49.027 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.027 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.027 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.027 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.027 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632612+0000", 2024-05-15T23:19:49.027 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.027 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.027 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.027 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.027 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632612+0000", 2024-05-15T23:19:49.028 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.028 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.028 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.028 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.028 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632614+0000", 2024-05-15T23:19:49.028 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2050000000000001e-06 2024-05-15T23:19:49.028 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.028 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.028 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.028 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632653+0000", 2024-05-15T23:19:49.028 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8399000000000003e-05 2024-05-15T23:19:49.028 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.028 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.028 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.028 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632668+0000", 2024-05-15T23:19:49.029 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5554999999999999e-05 2024-05-15T23:19:49.029 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.029 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.029 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.029 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632679+0000", 2024-05-15T23:19:49.029 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0550999999999999e-05 2024-05-15T23:19:49.029 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.029 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.029 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.029 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632737+0000", 2024-05-15T23:19:49.029 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8569000000000001e-05 2024-05-15T23:19:49.029 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.029 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.029 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.029 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632860+0000", 2024-05-15T23:19:49.030 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012253599999999999 2024-05-15T23:19:49.030 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.030 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.030 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.030 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.633022+0000", 2024-05-15T23:19:49.030 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016184899999999999 2024-05-15T23:19:49.030 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.030 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.030 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.030 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.637320+0000", 2024-05-15T23:19:49.030 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042986190000000001 2024-05-15T23:19:49.030 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.030 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.030 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.031 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.824896+0000", 2024-05-15T23:19:49.031 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18757555200000001 2024-05-15T23:19:49.031 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.031 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.031 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.031 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.824930+0000", 2024-05-15T23:19:49.031 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4551999999999997e-05 2024-05-15T23:19:49.031 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.031 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.031 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.031 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.824957+0000", 2024-05-15T23:19:49.031 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6427e-05 2024-05-15T23:19:49.031 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.031 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.031 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.032 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.032 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.032 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:135124 4.a 4:5c4f84a3:::benchmark_data_smithi090_104448_object135123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e306)", 2024-05-15T23:19:49.032 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:16:04.645139+0000", 2024-05-15T23:19:49.032 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 224.36533168700001, 2024-05-15T23:19:49.032 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17994866500000001, 2024-05-15T23:19:49.032 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.032 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.032 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.032 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.032 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.032 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.032 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 135124 2024-05-15T23:19:49.032 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645139+0000", 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645139+0000", 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645140+0000", 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.1999999999999998e-07 2024-05-15T23:19:49.033 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.034 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.034 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.034 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645233+0000", 2024-05-15T23:19:49.034 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2946000000000001e-05 2024-05-15T23:19:49.034 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.034 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.034 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.034 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645240+0000", 2024-05-15T23:19:49.034 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2200000000000003e-06 2024-05-15T23:19:49.034 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.034 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.034 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.034 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645246+0000", 2024-05-15T23:19:49.034 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0270000000000001e-06 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645295+0000", 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9301000000000001e-05 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645412+0000", 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011633699999999999 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645553+0000", 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014087799999999999 2024-05-15T23:19:49.035 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.036 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.036 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.036 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.649870+0000", 2024-05-15T23:19:49.036 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0043172430000000001 2024-05-15T23:19:49.036 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.036 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.036 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.036 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.824997+0000", 2024-05-15T23:19:49.036 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17512725000000001 2024-05-15T23:19:49.036 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.036 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.036 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.036 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.825049+0000", 2024-05-15T23:19:49.036 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2145999999999998e-05 2024-05-15T23:19:49.036 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.825088+0000", 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8396999999999998e-05 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304280 4.a 4:5b65c4c5:::benchmark_data_smithi090_104448_object131042:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.815719+0000", 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 83.194751698000005, 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.213512642, 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.037 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.038 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.038 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.038 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.038 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304280 2024-05-15T23:19:49.038 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.038 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.038 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.038 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.038 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815719+0000", 2024-05-15T23:19:49.038 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.038 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.038 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.038 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.038 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815719+0000", 2024-05-15T23:19:49.039 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.039 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.039 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.039 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.039 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815721+0000", 2024-05-15T23:19:49.039 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0279999999999999e-06 2024-05-15T23:19:49.039 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.039 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.039 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.039 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815728+0000", 2024-05-15T23:19:49.039 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7270000000000001e-06 2024-05-15T23:19:49.039 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.039 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.039 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.040 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815737+0000", 2024-05-15T23:19:49.040 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6150000000000003e-06 2024-05-15T23:19:49.040 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.040 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.040 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.040 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815744+0000", 2024-05-15T23:19:49.040 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2589999999999997e-06 2024-05-15T23:19:49.040 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.040 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.040 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.040 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815805+0000", 2024-05-15T23:19:49.040 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1169999999999999e-05 2024-05-15T23:19:49.040 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.040 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.041 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.041 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815916+0000", 2024-05-15T23:19:49.041 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011129300000000001 2024-05-15T23:19:49.041 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.041 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.041 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.041 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816037+0000", 2024-05-15T23:19:49.041 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000120598 2024-05-15T23:19:49.041 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.041 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.041 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.041 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.824204+0000", 2024-05-15T23:19:49.041 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008167545 2024-05-15T23:19:49.041 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.041 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.042 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.042 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.029121+0000", 2024-05-15T23:19:49.042 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20491626399999999 2024-05-15T23:19:49.042 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.042 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.042 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.042 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.029181+0000", 2024-05-15T23:19:49.042 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0053999999999998e-05 2024-05-15T23:19:49.042 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.042 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.042 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.042 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.029232+0000", 2024-05-15T23:19:49.042 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1088999999999998e-05 2024-05-15T23:19:49.042 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304283 4.a 4:58e8bfc1:::benchmark_data_smithi090_104448_object131045:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:19:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.816913+0000", 2024-05-15T23:19:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 83.193558150000001, 2024-05-15T23:19:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22379715, 2024-05-15T23:19:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304283 2024-05-15T23:19:49.044 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.044 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.044 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.044 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816913+0000", 2024-05-15T23:19:49.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.044 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.044 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.044 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816913+0000", 2024-05-15T23:19:49.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.044 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.045 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816914+0000", 2024-05-15T23:19:49.045 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.519e-06 2024-05-15T23:19:49.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.045 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816919+0000", 2024-05-15T23:19:49.045 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.977e-06 2024-05-15T23:19:49.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.045 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816928+0000", 2024-05-15T23:19:49.045 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0310000000000004e-06 2024-05-15T23:19:49.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816935+0000", 2024-05-15T23:19:49.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3610000000000003e-06 2024-05-15T23:19:49.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.819984+0000", 2024-05-15T23:19:49.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030482909999999998 2024-05-15T23:19:49.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.820096+0000", 2024-05-15T23:19:49.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000112496 2024-05-15T23:19:49.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.820173+0000", 2024-05-15T23:19:49.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7179000000000001e-05 2024-05-15T23:19:49.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.825431+0000", 2024-05-15T23:19:49.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0052575199999999999 2024-05-15T23:19:49.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.040648+0000", 2024-05-15T23:19:49.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21521739400000001 2024-05-15T23:19:49.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.040681+0000", 2024-05-15T23:19:49.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2372000000000001e-05 2024-05-15T23:19:49.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.040710+0000", 2024-05-15T23:19:49.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9009999999999998e-05 2024-05-15T23:19:49.048 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.048 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.048 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.048 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304296 4.a 4:59aa04f1:::benchmark_data_smithi090_104448_object131058:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.828982+0000", 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 83.181488619999996, 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22203131700000001, 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304296 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.828982+0000", 2024-05-15T23:19:49.049 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.050 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.828982+0000", 2024-05-15T23:19:49.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.050 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.828984+0000", 2024-05-15T23:19:49.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.243e-06 2024-05-15T23:19:49.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.050 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.828992+0000", 2024-05-15T23:19:49.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4479999999999997e-06 2024-05-15T23:19:49.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.829004+0000", 2024-05-15T23:19:49.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1747e-05 2024-05-15T23:19:49.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.829012+0000", 2024-05-15T23:19:49.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.9590000000000001e-06 2024-05-15T23:19:49.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.829077+0000", 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4916e-05 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.829221+0000", 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014339600000000001 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.829352+0000", 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000130996 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.834093+0000", 2024-05-15T23:19:49.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0047415189999999996 2024-05-15T23:19:49.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.050919+0000", 2024-05-15T23:19:49.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21682558900000001 2024-05-15T23:19:49.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.050970+0000", 2024-05-15T23:19:49.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0618000000000002e-05 2024-05-15T23:19:49.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051013+0000", 2024-05-15T23:19:49.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3886e-05 2024-05-15T23:19:49.054 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.054 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.054 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.054 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.054 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304319 4.a 4:5860cfdb:::benchmark_data_smithi090_104448_object131081:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:19:49.054 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.849831+0000", 2024-05-15T23:19:49.054 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 83.160639576999998, 2024-05-15T23:19:49.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20133589099999999, 2024-05-15T23:19:49.054 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.055 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.055 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.055 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.055 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.055 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.055 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304319 2024-05-15T23:19:49.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.055 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.055 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.055 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849831+0000", 2024-05-15T23:19:49.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.055 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.056 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849831+0000", 2024-05-15T23:19:49.056 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.056 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.056 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.056 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.056 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849834+0000", 2024-05-15T23:19:49.056 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.875e-06 2024-05-15T23:19:49.056 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.056 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.056 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.056 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849842+0000", 2024-05-15T23:19:49.056 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.4659999999999999e-06 2024-05-15T23:19:49.056 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.056 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.056 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849858+0000", 2024-05-15T23:19:49.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5536000000000002e-05 2024-05-15T23:19:49.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849868+0000", 2024-05-15T23:19:49.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7610000000000002e-06 2024-05-15T23:19:49.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849918+0000", 2024-05-15T23:19:49.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0219000000000001e-05 2024-05-15T23:19:49.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.850026+0000", 2024-05-15T23:19:49.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000108256 2024-05-15T23:19:49.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.850126+0000", 2024-05-15T23:19:49.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9479000000000001e-05 2024-05-15T23:19:49.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.854202+0000", 2024-05-15T23:19:49.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004076774 2024-05-15T23:19:49.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051073+0000", 2024-05-15T23:19:49.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19687078499999999 2024-05-15T23:19:49.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051124+0000", 2024-05-15T23:19:49.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0436999999999998e-05 2024-05-15T23:19:49.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051167+0000", 2024-05-15T23:19:49.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3303000000000002e-05 2024-05-15T23:19:49.060 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.060 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.060 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.060 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.060 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.060 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304341 4.a 4:53ff7ba8:::benchmark_data_smithi090_104448_object131103:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:19:49.060 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.865333+0000", 2024-05-15T23:19:49.060 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 83.145137427999998, 2024-05-15T23:19:49.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18599020299999999, 2024-05-15T23:19:49.060 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.060 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.060 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.060 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.060 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.060 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.061 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304341 2024-05-15T23:19:49.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.061 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.061 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.061 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.061 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865333+0000", 2024-05-15T23:19:49.061 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.061 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.061 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.061 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865333+0000", 2024-05-15T23:19:49.061 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.062 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.062 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.062 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865334+0000", 2024-05-15T23:19:49.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0999999999999998e-07 2024-05-15T23:19:49.062 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.062 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.062 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.062 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865337+0000", 2024-05-15T23:19:49.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7470000000000001e-06 2024-05-15T23:19:49.062 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.062 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.062 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.062 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865341+0000", 2024-05-15T23:19:49.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4789999999999998e-06 2024-05-15T23:19:49.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865344+0000", 2024-05-15T23:19:49.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1030000000000002e-06 2024-05-15T23:19:49.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865395+0000", 2024-05-15T23:19:49.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0340999999999998e-05 2024-05-15T23:19:49.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865511+0000", 2024-05-15T23:19:49.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000116851 2024-05-15T23:19:49.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865618+0000", 2024-05-15T23:19:49.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000106659 2024-05-15T23:19:49.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.870311+0000", 2024-05-15T23:19:49.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0046930080000000002 2024-05-15T23:19:49.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051226+0000", 2024-05-15T23:19:49.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.180914669 2024-05-15T23:19:49.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051279+0000", 2024-05-15T23:19:49.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3637999999999997e-05 2024-05-15T23:19:49.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051323+0000", 2024-05-15T23:19:49.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4097999999999997e-05 2024-05-15T23:19:49.065 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.066 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.066 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.066 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304347 4.a 4:5813fd51:::benchmark_data_smithi090_104448_object131109:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:19:49.066 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.873270+0000", 2024-05-15T23:19:49.066 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 83.137201136000002, 2024-05-15T23:19:49.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17820270799999999, 2024-05-15T23:19:49.066 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.066 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.066 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.066 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.066 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.066 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.067 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304347 2024-05-15T23:19:49.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.067 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873270+0000", 2024-05-15T23:19:49.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873270+0000", 2024-05-15T23:19:49.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873270+0000", 2024-05-15T23:19:49.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6700000000000003e-07 2024-05-15T23:19:49.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873274+0000", 2024-05-15T23:19:49.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5300000000000001e-06 2024-05-15T23:19:49.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873281+0000", 2024-05-15T23:19:49.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3300000000000001e-06 2024-05-15T23:19:49.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.069 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873288+0000", 2024-05-15T23:19:49.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8569999999999997e-06 2024-05-15T23:19:49.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.069 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873335+0000", 2024-05-15T23:19:49.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7434e-05 2024-05-15T23:19:49.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873420+0000", 2024-05-15T23:19:49.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5000000000000006e-05 2024-05-15T23:19:49.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873494+0000", 2024-05-15T23:19:49.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3901999999999998e-05 2024-05-15T23:19:49.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.070 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.070 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882687+0000", 2024-05-15T23:19:49.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091921250000000006 2024-05-15T23:19:49.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051380+0000", 2024-05-15T23:19:49.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168693808 2024-05-15T23:19:49.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051431+0000", 2024-05-15T23:19:49.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1150000000000003e-05 2024-05-15T23:19:49.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051472+0000", 2024-05-15T23:19:49.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0805e-05 2024-05-15T23:19:49.072 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.072 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.072 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.072 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304351 4.a 4:582c10c7:::benchmark_data_smithi090_104448_object131113:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:19:49.072 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.874105+0000", 2024-05-15T23:19:49.072 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 83.136365966, 2024-05-15T23:19:49.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18030819300000001, 2024-05-15T23:19:49.072 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.072 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.072 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.072 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.073 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.073 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.073 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304351 2024-05-15T23:19:49.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.073 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874105+0000", 2024-05-15T23:19:49.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874105+0000", 2024-05-15T23:19:49.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874106+0000", 2024-05-15T23:19:49.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2950000000000001e-06 2024-05-15T23:19:49.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874111+0000", 2024-05-15T23:19:49.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8080000000000004e-06 2024-05-15T23:19:49.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874126+0000", 2024-05-15T23:19:49.075 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4915e-05 2024-05-15T23:19:49.075 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.075 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.075 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.075 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874133+0000", 2024-05-15T23:19:49.075 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4760000000000001e-06 2024-05-15T23:19:49.075 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.075 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.075 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.075 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.877862+0000", 2024-05-15T23:19:49.075 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037288489999999998 2024-05-15T23:19:49.076 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.076 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.076 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.877973+0000", 2024-05-15T23:19:49.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011109599999999999 2024-05-15T23:19:49.076 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.076 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.076 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.878075+0000", 2024-05-15T23:19:49.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000102221 2024-05-15T23:19:49.076 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.076 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.076 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882814+0000", 2024-05-15T23:19:49.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004738669 2024-05-15T23:19:49.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054322+0000", 2024-05-15T23:19:49.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171508251 2024-05-15T23:19:49.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054373+0000", 2024-05-15T23:19:49.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0822999999999998e-05 2024-05-15T23:19:49.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054413+0000", 2024-05-15T23:19:49.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9789999999999997e-05 2024-05-15T23:19:49.078 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.078 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.078 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.078 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304363 4.a 4:571d2066:::benchmark_data_smithi090_104448_object131125:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:19:49.078 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.882815+0000", 2024-05-15T23:19:49.078 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 83.127655324000003, 2024-05-15T23:19:49.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17174726100000001, 2024-05-15T23:19:49.078 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.078 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.079 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.079 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.079 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.079 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.079 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304363 2024-05-15T23:19:49.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.079 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882815+0000", 2024-05-15T23:19:49.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.080 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882815+0000", 2024-05-15T23:19:49.080 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.080 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.080 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.080 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882817+0000", 2024-05-15T23:19:49.080 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8950000000000001e-06 2024-05-15T23:19:49.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.080 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.080 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.080 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882823+0000", 2024-05-15T23:19:49.080 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3410000000000002e-06 2024-05-15T23:19:49.080 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.081 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882835+0000", 2024-05-15T23:19:49.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2853999999999999e-05 2024-05-15T23:19:49.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.081 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882848+0000", 2024-05-15T23:19:49.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2695e-05 2024-05-15T23:19:49.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.081 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882894+0000", 2024-05-15T23:19:49.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5383999999999998e-05 2024-05-15T23:19:49.082 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.082 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.082 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.082 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883006+0000", 2024-05-15T23:19:49.082 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011286299999999999 2024-05-15T23:19:49.082 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.082 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.082 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.082 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883119+0000", 2024-05-15T23:19:49.082 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011275199999999999 2024-05-15T23:19:49.082 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.082 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.082 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.891222+0000", 2024-05-15T23:19:49.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0081027779999999997 2024-05-15T23:19:49.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054470+0000", 2024-05-15T23:19:49.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.163247948 2024-05-15T23:19:49.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.083 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054521+0000", 2024-05-15T23:19:49.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0760999999999997e-05 2024-05-15T23:19:49.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054563+0000", 2024-05-15T23:19:49.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1990000000000003e-05 2024-05-15T23:19:49.084 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.084 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.084 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.084 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304365 4.a 4:5a74db35:::benchmark_data_smithi090_104448_object131127:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:19:49.084 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.883969+0000", 2024-05-15T23:19:49.084 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 83.126501556999997, 2024-05-15T23:19:49.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17074013900000001, 2024-05-15T23:19:49.084 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.085 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.085 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.085 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.085 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.085 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.085 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304365 2024-05-15T23:19:49.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.085 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883969+0000", 2024-05-15T23:19:49.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883969+0000", 2024-05-15T23:19:49.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883971+0000", 2024-05-15T23:19:49.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.567e-06 2024-05-15T23:19:49.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883976+0000", 2024-05-15T23:19:49.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0209999999999999e-06 2024-05-15T23:19:49.087 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.087 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.087 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883988+0000", 2024-05-15T23:19:49.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.256e-05 2024-05-15T23:19:49.087 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.087 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.087 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883996+0000", 2024-05-15T23:19:49.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8779999999999998e-06 2024-05-15T23:19:49.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.887685+0000", 2024-05-15T23:19:49.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036892100000000001 2024-05-15T23:19:49.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.887768+0000", 2024-05-15T23:19:49.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2952000000000001e-05 2024-05-15T23:19:49.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.089 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.887841+0000", 2024-05-15T23:19:49.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2880999999999996e-05 2024-05-15T23:19:49.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.089 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.089 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.089 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.891325+0000", 2024-05-15T23:19:49.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034839340000000002 2024-05-15T23:19:49.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.089 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.089 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.089 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054613+0000", 2024-05-15T23:19:49.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.163287922 2024-05-15T23:19:49.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054668+0000", 2024-05-15T23:19:49.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4787999999999998e-05 2024-05-15T23:19:49.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054709+0000", 2024-05-15T23:19:49.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1425999999999999e-05 2024-05-15T23:19:49.090 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.090 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.090 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.091 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304372 4.a 4:519d11bc:::benchmark_data_smithi090_104448_object131134:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:19:49.091 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.892294+0000", 2024-05-15T23:19:49.091 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 83.118176255999998, 2024-05-15T23:19:49.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17457294600000001, 2024-05-15T23:19:49.091 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.091 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.091 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.091 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.091 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.091 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.092 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304372 2024-05-15T23:19:49.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.092 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892294+0000", 2024-05-15T23:19:49.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892294+0000", 2024-05-15T23:19:49.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892296+0000", 2024-05-15T23:19:49.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8479999999999999e-06 2024-05-15T23:19:49.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892303+0000", 2024-05-15T23:19:49.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3899999999999998e-06 2024-05-15T23:19:49.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892313+0000", 2024-05-15T23:19:49.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9599999999999995e-06 2024-05-15T23:19:49.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892320+0000", 2024-05-15T23:19:49.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0419999999999997e-06 2024-05-15T23:19:49.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892359+0000", 2024-05-15T23:19:49.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9001999999999997e-05 2024-05-15T23:19:49.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.095 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.095 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892450+0000", 2024-05-15T23:19:49.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0978999999999997e-05 2024-05-15T23:19:49.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.095 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.095 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892524+0000", 2024-05-15T23:19:49.095 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3943999999999995e-05 2024-05-15T23:19:49.095 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.896062+0000", 2024-05-15T23:19:49.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0035380670000000002 2024-05-15T23:19:49.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.096 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.066773+0000", 2024-05-15T23:19:49.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.170711062 2024-05-15T23:19:49.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.096 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.096 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.066825+0000", 2024-05-15T23:19:49.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1969999999999999e-05 2024-05-15T23:19:49.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.097 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.066867+0000", 2024-05-15T23:19:49.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2682000000000002e-05 2024-05-15T23:19:49.097 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.097 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.097 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.097 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313500 4.a 4:54957ed1:::benchmark_data_smithi090_104448_object140262:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:19:49.097 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.433300+0000", 2024-05-15T23:19:49.098 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 75.577170508999998, 2024-05-15T23:19:49.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.232290565, 2024-05-15T23:19:49.098 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.098 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.098 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.098 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.098 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.098 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.098 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313500 2024-05-15T23:19:49.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.098 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.098 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.098 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433300+0000", 2024-05-15T23:19:49.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433300+0000", 2024-05-15T23:19:49.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433301+0000", 2024-05-15T23:19:49.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7600000000000006e-07 2024-05-15T23:19:49.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433305+0000", 2024-05-15T23:19:49.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2880000000000003e-06 2024-05-15T23:19:49.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433311+0000", 2024-05-15T23:19:49.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1279999999999999e-06 2024-05-15T23:19:49.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433314+0000", 2024-05-15T23:19:49.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8130000000000002e-06 2024-05-15T23:19:49.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433357+0000", 2024-05-15T23:19:49.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.214e-05 2024-05-15T23:19:49.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433465+0000", 2024-05-15T23:19:49.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000108762 2024-05-15T23:19:49.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433582+0000", 2024-05-15T23:19:49.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011637 2024-05-15T23:19:49.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.438171+0000", 2024-05-15T23:19:49.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045897919999999997 2024-05-15T23:19:49.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665486+0000", 2024-05-15T23:19:49.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22731428400000001 2024-05-15T23:19:49.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665546+0000", 2024-05-15T23:19:49.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9988999999999998e-05 2024-05-15T23:19:49.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665591+0000", 2024-05-15T23:19:49.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5022999999999999e-05 2024-05-15T23:19:49.103 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.103 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.103 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.103 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313526 4.a 4:537fd6b5:::benchmark_data_smithi090_104448_object140288:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:19:49.104 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.459071+0000", 2024-05-15T23:19:49.104 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 75.551399356999994, 2024-05-15T23:19:49.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20662114200000001, 2024-05-15T23:19:49.104 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.104 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.104 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.104 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.104 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.104 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.104 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313526 2024-05-15T23:19:49.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.104 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459071+0000", 2024-05-15T23:19:49.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459071+0000", 2024-05-15T23:19:49.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459073+0000", 2024-05-15T23:19:49.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8980000000000001e-06 2024-05-15T23:19:49.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459079+0000", 2024-05-15T23:19:49.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2730000000000002e-06 2024-05-15T23:19:49.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459090+0000", 2024-05-15T23:19:49.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1619e-05 2024-05-15T23:19:49.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459097+0000", 2024-05-15T23:19:49.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4189999999999999e-06 2024-05-15T23:19:49.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459148+0000", 2024-05-15T23:19:49.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1921e-05 2024-05-15T23:19:49.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459248+0000", 2024-05-15T23:19:49.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9388000000000005e-05 2024-05-15T23:19:49.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459325+0000", 2024-05-15T23:19:49.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7024e-05 2024-05-15T23:19:49.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.108 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.462998+0000", 2024-05-15T23:19:49.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036728120000000001 2024-05-15T23:19:49.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.108 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665627+0000", 2024-05-15T23:19:49.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.202629165 2024-05-15T23:19:49.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.108 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665657+0000", 2024-05-15T23:19:49.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9984999999999998e-05 2024-05-15T23:19:49.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665692+0000", 2024-05-15T23:19:49.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5638e-05 2024-05-15T23:19:49.109 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.109 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.109 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.109 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313534 4.a 4:53d3c98e:::benchmark_data_smithi090_104448_object140296:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:19:49.109 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.467942+0000", 2024-05-15T23:19:49.109 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 75.542528257000001, 2024-05-15T23:19:49.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20189164800000001, 2024-05-15T23:19:49.110 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.110 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.110 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.110 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.110 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.110 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.110 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313534 2024-05-15T23:19:49.110 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.110 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.110 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.110 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467942+0000", 2024-05-15T23:19:49.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467942+0000", 2024-05-15T23:19:49.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467944+0000", 2024-05-15T23:19:49.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1459999999999999e-06 2024-05-15T23:19:49.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467948+0000", 2024-05-15T23:19:49.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8389999999999997e-06 2024-05-15T23:19:49.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467957+0000", 2024-05-15T23:19:49.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0790000000000001e-06 2024-05-15T23:19:49.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467962+0000", 2024-05-15T23:19:49.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5180000000000004e-06 2024-05-15T23:19:49.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.468001+0000", 2024-05-15T23:19:49.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9119999999999998e-05 2024-05-15T23:19:49.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.468074+0000", 2024-05-15T23:19:49.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2698000000000001e-05 2024-05-15T23:19:49.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.468139+0000", 2024-05-15T23:19:49.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.512e-05 2024-05-15T23:19:49.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.471787+0000", 2024-05-15T23:19:49.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036482730000000001 2024-05-15T23:19:49.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669769+0000", 2024-05-15T23:19:49.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19798183499999999 2024-05-15T23:19:49.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669803+0000", 2024-05-15T23:19:49.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3670999999999997e-05 2024-05-15T23:19:49.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.115 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.115 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669834+0000", 2024-05-15T23:19:49.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1349000000000001e-05 2024-05-15T23:19:49.115 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.115 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.115 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.115 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313558 4.a 4:526840ac:::benchmark_data_smithi090_104448_object140320:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:19:49.115 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.489279+0000", 2024-05-15T23:19:49.115 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 75.521191764999998, 2024-05-15T23:19:49.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18070008900000001, 2024-05-15T23:19:49.115 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.116 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.116 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.116 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.116 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.116 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.116 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313558 2024-05-15T23:19:49.116 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.116 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.116 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.116 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.116 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489279+0000", 2024-05-15T23:19:49.116 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.116 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.116 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.116 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489279+0000", 2024-05-15T23:19:49.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.117 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489281+0000", 2024-05-15T23:19:49.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.784e-06 2024-05-15T23:19:49.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.117 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.117 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489286+0000", 2024-05-15T23:19:49.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1699999999999996e-06 2024-05-15T23:19:49.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.117 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.117 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489296+0000", 2024-05-15T23:19:49.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.005e-05 2024-05-15T23:19:49.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489302+0000", 2024-05-15T23:19:49.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5309999999999998e-06 2024-05-15T23:19:49.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489348+0000", 2024-05-15T23:19:49.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5028000000000003e-05 2024-05-15T23:19:49.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489429+0000", 2024-05-15T23:19:49.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1233e-05 2024-05-15T23:19:49.119 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489499+0000", 2024-05-15T23:19:49.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0450000000000005e-05 2024-05-15T23:19:49.119 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.493430+0000", 2024-05-15T23:19:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0039305049999999999 2024-05-15T23:19:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669881+0000", 2024-05-15T23:19:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17645111499999999 2024-05-15T23:19:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669936+0000", 2024-05-15T23:19:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5328999999999998e-05 2024-05-15T23:19:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.121 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669979+0000", 2024-05-15T23:19:49.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2894e-05 2024-05-15T23:19:49.121 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.121 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.121 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.121 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313566 4.a 4:56b72cc5:::benchmark_data_smithi090_104448_object140328:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:19:49.121 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.498107+0000", 2024-05-15T23:19:49.121 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 75.512363472000004, 2024-05-15T23:19:49.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18568280400000001, 2024-05-15T23:19:49.121 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.121 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.121 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.122 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.122 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.122 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.122 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313566 2024-05-15T23:19:49.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.122 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.122 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.122 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498107+0000", 2024-05-15T23:19:49.122 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.122 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.122 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498107+0000", 2024-05-15T23:19:49.123 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.123 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.123 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.123 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.123 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498110+0000", 2024-05-15T23:19:49.123 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3140000000000002e-06 2024-05-15T23:19:49.123 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.123 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.123 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.123 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498117+0000", 2024-05-15T23:19:49.123 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1690000000000003e-06 2024-05-15T23:19:49.123 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.123 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.123 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498128+0000", 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1362000000000001e-05 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498136+0000", 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5769999999999996e-06 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498184+0000", 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8631000000000002e-05 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498280+0000", 2024-05-15T23:19:49.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5958999999999994e-05 2024-05-15T23:19:49.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498356+0000", 2024-05-15T23:19:49.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5574000000000005e-05 2024-05-15T23:19:49.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.503160+0000", 2024-05-15T23:19:49.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0048046499999999997 2024-05-15T23:19:49.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683729+0000", 2024-05-15T23:19:49.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.180568066 2024-05-15T23:19:49.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683760+0000", 2024-05-15T23:19:49.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1798000000000003e-05 2024-05-15T23:19:49.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683790+0000", 2024-05-15T23:19:49.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9703999999999999e-05 2024-05-15T23:19:49.126 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.126 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.126 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.127 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313575 4.a 4:5e037716:::benchmark_data_smithi090_104448_object140337:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:19:49.127 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.507823+0000", 2024-05-15T23:19:49.127 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 75.502647902999996, 2024-05-15T23:19:49.127 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17607811600000001, 2024-05-15T23:19:49.127 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:19:49.127 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:19:49.127 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:19:49.127 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:19:49.127 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:19:49.127 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:19:49.127 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313575 2024-05-15T23:19:49.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.128 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:19:49.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:19:49.128 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507823+0000", 2024-05-15T23:19:49.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:19:49.128 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507823+0000", 2024-05-15T23:19:49.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:19:49.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:19:49.129 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507825+0000", 2024-05-15T23:19:49.129 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3099999999999999e-06 2024-05-15T23:19:49.129 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.129 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.129 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:19:49.129 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507831+0000", 2024-05-15T23:19:49.129 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0120000000000002e-06 2024-05-15T23:19:49.129 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.129 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.129 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:19:49.129 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507843+0000", 2024-05-15T23:19:49.129 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1572e-05 2024-05-15T23:19:49.129 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.129 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.129 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:19:49.130 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507851+0000", 2024-05-15T23:19:49.130 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3159999999999997e-06 2024-05-15T23:19:49.130 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.130 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.130 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:19:49.130 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507900+0000", 2024-05-15T23:19:49.130 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9444999999999998e-05 2024-05-15T23:19:49.130 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.130 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.130 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:19:49.130 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507981+0000", 2024-05-15T23:19:49.130 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0547e-05 2024-05-15T23:19:49.130 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.130 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:19:49.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.508068+0000", 2024-05-15T23:19:49.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6699000000000005e-05 2024-05-15T23:19:49.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:19:49.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.513050+0000", 2024-05-15T23:19:49.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049819829999999997 2024-05-15T23:19:49.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:19:49.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683820+0000", 2024-05-15T23:19:49.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17076991899999999 2024-05-15T23:19:49.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:19:49.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683858+0000", 2024-05-15T23:19:49.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7917999999999998e-05 2024-05-15T23:19:49.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:19:49.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:19:49.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:19:49.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683901+0000", 2024-05-15T23:19:49.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3395e-05 2024-05-15T23:19:49.132 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.132 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.132 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.132 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:19:49.132 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:19:49.132 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:19:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:49.134 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:49.134 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:49.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:49.235 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:49.255 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:49.265 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:49.284 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:49.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:49.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:49.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:19:49.421 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:49.443 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:19:49.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:49.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:19:49.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:49 smithi088 ceph-mon[44699]: osdmap e376: 12 total, 11 up, 10 in 2024-05-15T23:19:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:49 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:19:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:49 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:19:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:49 smithi088 ceph-mon[44699]: pgmap v1031: 17 pgs: 5 creating+peering, 9 unknown, 3 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:49.574 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:49 smithi090 ceph-mon[44434]: osdmap e376: 12 total, 11 up, 10 in 2024-05-15T23:19:49.574 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:49 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:19:49.574 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:49 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:19:49.574 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:49 smithi090 ceph-mon[44434]: pgmap v1031: 17 pgs: 5 creating+peering, 9 unknown, 3 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:49.576 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:49.596 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:49.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:49.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:49 smithi084 ceph-mon[39784]: osdmap e376: 12 total, 11 up, 10 in 2024-05-15T23:19:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:49 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:19:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:49 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:19:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:49 smithi084 ceph-mon[39784]: pgmap v1031: 17 pgs: 5 creating+peering, 9 unknown, 3 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:49.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:49.730 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:49.778 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T23:19:49.928 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:49.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:49.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:50.032 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:50.043 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:19:50.058 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:50.261 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:50.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:50.369 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:50.398 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:50.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:50 smithi088 ceph-mon[44699]: osdmap e377: 12 total, 11 up, 10 in 2024-05-15T23:19:50.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:50 smithi088 ceph-mon[44699]: from='client.15837 v1:172.21.15.84:0/2024985403' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:19:50.596 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:50.598 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:50 smithi084 ceph-mon[39784]: osdmap e377: 12 total, 11 up, 10 in 2024-05-15T23:19:50.598 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:50 smithi084 ceph-mon[39784]: from='client.15837 v1:172.21.15.84:0/2024985403' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:19:50.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:50.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:50.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:50.713 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:19:50.713 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:19:50.730 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:50 smithi090 ceph-mon[44434]: osdmap e377: 12 total, 11 up, 10 in 2024-05-15T23:19:50.731 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:50 smithi090 ceph-mon[44434]: from='client.15837 v1:172.21.15.84:0/2024985403' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:19:50.733 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:50.754 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:50.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:50.757 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:50.858 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:19:50.885 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:51.234 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:19:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:51.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:19:51.319 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:19:51.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:19:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:51.363 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1032,"stamp":"2024-05-15T23:19:49.308519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":4,"kb":937451520,"kb_used":2740200,"kb_used_data":8356,"kb_used_omap":15,"kb_used_meta":2731440,"kb_avail":934711320,"statfs":{"total":959950356480,"available":957144391680,"internally_reserved":0,"allocated":8556544,"data_stored":4777535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15952,"internal_metadata":2796994992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.346825"},"pg_stats":[{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":374,"state":"creating+peering","last_fresh":"2024-05-15T23:19:46.299800+0000","last_change":"2024-05-15T23:19:46.299800+0000","last_active":"2024-05-15T23:19:46.285287+0000","last_peered":"2024-05-15T23:19:46.285287+0000","last_clean":"2024-05-15T23:19:46.285287+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-15T23:19:46.299800+0000","last_undegraded":"2024-05-15T23:19:46.299800+0000","last_fullsized":"2024-05-15T23:19:46.299800+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-15T23:19:46.285287+0000","last_change":"2024-05-15T23:19:46.285287+0000","last_active":"2024-05-15T23:19:46.285287+0000","last_peered":"2024-05-15T23:19:46.285287+0000","last_clean":"2024-05-15T23:19:46.285287+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-15T23:19:46.285287+0000","last_undegraded":"2024-05-15T23:19:46.285287+0000","last_fullsized":"2024-05-15T23:19:46.285287+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":374,"state":"creating+peering","last_fresh":"2024-05-15T23:19:46.759523+0000","last_change":"2024-05-15T23:19:46.759523+0000","last_active":"2024-05-15T23:19:46.285287+0000","last_peered":"2024-05-15T23:19:46.285287+0000","last_clean":"2024-05-15T23:19:46.285287+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-15T23:19:46.759523+0000","last_undegraded":"2024-05-15T23:19:46.759523+0000","last_fullsized":"2024-05-15T23:19:46.759523+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-15T23:19:46.285287+0000","last_change":"2024-05-15T23:19:46.285287+0000","last_active":"2024-05-15T23:19:46.285287+0000","last_peered":"2024-05-15T23:19:46.285287+0000","last_clean":"2024-05-15T23:19:46.285287+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-15T23:19:46.285287+0000","last_undegraded":"2024-05-15T23:19:46.285287+0000","last_fullsized":"2024-05-15T23:19:46.285287+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-15T23:19:46.285287+0000","last_change":"2024-05-15T23:19:46.285287+0000","last_active":"2024-05-15T23:19:46.285287+0000","last_peered":"2024-05-15T23:19:46.285287+0000","last_clean":"2024-05-15T23:19:46.285287+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-15T23:19:46.285287+0000","last_undegraded":"2024-05-15T23:19:46.285287+0000","last_fullsized":"2024-05-15T23:19:46.285287+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-15T23:19:46.285287+0000","last_change":"2024-05-15T23:19:46.285287+0000","last_active":"2024-05-15T23:19:46.285287+0000","last_peered":"2024-05-15T23:19:46.285287+0000","last_clean":"2024-05-15T23:19:46.285287+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-15T23:19:46.285287+0000","last_undegraded":"2024-05-15T23:19:46.285287+0000","last_fullsized":"2024-05-15T23:19:46.285287+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-15T23:19:46.285287+0000","last_change":"2024-05-15T23:19:46.285287+0000","last_active":"2024-05-15T23:19:46.285287+0000","last_peered":"2024-05-15T23:19:46.285287+0000","last_clean":"2024-05-15T23:19:46.285287+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-15T23:19:46.285287+0000","last_undegraded":"2024-05-15T23:19:46.285287+0000","last_fullsized":"2024-05-15T23:19:46.285287+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":374,"state":"creating+peering","last_fresh":"2024-05-15T23:19:46.759439+0000","last_change":"2024-05-15T23:19:46.759439+0000","last_active":"2024-05-15T23:19:46.285287+0000","last_peered":"2024-05-15T23:19:46.285287+0000","last_clean":"2024-05-15T23:19:46.285287+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-15T23:19:46.759439+0000","last_undegraded":"2024-05-15T23:19:46.759439+0000","last_fullsized":"2024-05-15T23:19:46.759439+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":374,"state":"creating+peering","last_fresh":"2024-05-15T23:19:46.299470+0000","last_change":"2024-05-15T23:19:46.299470+0000","last_active":"2024-05-15T23:19:46.285287+0000","last_peered":"2024-05-15T23:19:46.285287+0000","last_clean":"2024-05-15T23:19:46.285287+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-15T23:19:46.299470+0000","last_undegraded":"2024-05-15T23:19:46.299470+0000","last_fullsized":"2024-05-15T23:19:46.299470+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":374,"state":"creating+peering","last_fresh":"2024-05-15T23:19:46.299422+0000","last_change":"2024-05-15T23:19:46.299422+0000","last_active":"2024-05-15T23:19:46.285287+0000","last_peered":"2024-05-15T23:19:46.285287+0000","last_clean":"2024-05-15T23:19:46.285287+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-15T23:19:46.299422+0000","last_undegraded":"2024-05-15T23:19:46.299422+0000","last_fullsized":"2024-05-15T23:19:46.299422+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-15T23:19:46.285287+0000","last_change":"2024-05-15T23:19:46.285287+0000","last_active":"2024-05-15T23:19:46.285287+0000","last_peered":"2024-05-15T23:19:46.285287+0000","last_clean":"2024-05-15T23:19:46.285287+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-15T23:19:46.285287+0000","last_undegraded":"2024-05-15T23:19:46.285287+0000","last_fullsized":"2024-05-15T23:19:46.285287+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-05-15T23:19:47.307080+0000","last_change":"2024-05-15T23:19:47.306838+0000","last_active":"2024-05-15T23:19:47.307080+0000","last_peered":"2024-05-15T23:19:47.307080+0000","last_clean":"2024-05-15T23:19:47.307080+0000","last_became_active":"2024-05-15T23:19:47.306523+0000","last_became_peered":"2024-05-15T23:19:47.306523+0000","last_unstale":"2024-05-15T23:19:47.307080+0000","last_undegraded":"2024-05-15T23:19:47.307080+0000","last_fullsized":"2024-05-15T23:19:47.307080+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:21:01.345394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-05-15T23:19:47.305105+0000","last_change":"2024-05-15T23:19:47.304803+0000","last_active":"2024-05-15T23:19:47.305105+0000","last_peered":"2024-05-15T23:19:47.305105+0000","last_clean":"2024-05-15T23:19:47.305105+0000","last_became_active":"2024-05-15T23:19:47.304522+0000","last_became_peered":"2024-05-15T23:19:47.304522+0000","last_unstale":"2024-05-15T23:19:47.305105+0000","last_undegraded":"2024-05-15T23:19:47.305105+0000","last_fullsized":"2024-05-15T23:19:47.305105+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:20:49.397557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-15T23:19:46.285287+0000","last_change":"2024-05-15T23:19:46.285287+0000","last_active":"2024-05-15T23:19:46.285287+0000","last_peered":"2024-05-15T23:19:46.285287+0000","last_clean":"2024-05-15T23:19:46.285287+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-15T23:19:46.285287+0000","last_undegraded":"2024-05-15T23:19:46.285287+0000","last_fullsized":"2024-05-15T23:19:46.285287+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-15T23:19:46.285287+0000","last_change":"2024-05-15T23:19:46.285287+0000","last_active":"2024-05-15T23:19:46.285287+0000","last_peered":"2024-05-15T23:19:46.285287+0000","last_clean":"2024-05-15T23:19:46.285287+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-15T23:19:46.285287+0000","last_undegraded":"2024-05-15T23:19:46.285287+0000","last_fullsized":"2024-05-15T23:19:46.285287+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-15T23:19:46.285287+0000","last_change":"2024-05-15T23:19:46.285287+0000","last_active":"2024-05-15T23:19:46.285287+0000","last_peered":"2024-05-15T23:19:46.285287+0000","last_clean":"2024-05-15T23:19:46.285287+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-15T23:19:46.285287+0000","last_undegraded":"2024-05-15T23:19:46.285287+0000","last_fullsized":"2024-05-15T23:19:46.285287+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":776,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-05-15T23:19:42.166789+0000","last_change":"2024-05-15T23:18:33.705243+0000","last_active":"2024-05-15T23:19:42.166789+0000","last_peered":"2024-05-15T23:19:42.166789+0000","last_clean":"2024-05-15T23:19:42.166789+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:19:42.166789+0000","last_undegraded":"2024-05-15T23:19:42.166789+0000","last_fullsized":"2024-05-15T23:19:42.166789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:18:33.705118+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:18:33.705118+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:19:50.341622+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351168,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":425348,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93319804,"statfs":{"total":95995035648,"available":95559479296,"internally_reserved":0,"allocated":1302528,"data_stored":911452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":434239940},"hb_peers":[1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580101,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":166592,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93578560,"statfs":{"total":95995035648,"available":95824445440,"internally_reserved":0,"allocated":770048,"data_stored":382544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":169802176},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776333,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":324864,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93420288,"statfs":{"total":95995035648,"available":95662374912,"internally_reserved":0,"allocated":770048,"data_stored":382544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":331872704},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005265,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":144384,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93600768,"statfs":{"total":95995035648,"available":95847186432,"internally_reserved":0,"allocated":770048,"data_stored":382544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":262916,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93482236,"statfs":{"total":95995035648,"available":95725809664,"internally_reserved":0,"allocated":1302528,"data_stored":911472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":267909578},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148568,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93596584,"statfs":{"total":95995035648,"available":95842902016,"internally_reserved":0,"allocated":671744,"data_stored":332816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151321030},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575093,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":360960,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93384192,"statfs":{"total":95995035648,"available":95625412608,"internally_reserved":0,"allocated":770048,"data_stored":382544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":368835014},"hb_peers":[2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483192,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149931,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":352640,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93392512,"statfs":{"total":95995035648,"available":95633932288,"internally_reserved":0,"allocated":770048,"data_stored":382544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":360315335},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986404,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":329024,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93416128,"statfs":{"total":95995035648,"available":95658115072,"internally_reserved":0,"allocated":770048,"data_stored":382544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790176,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":224904,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":93520248,"statfs":{"total":95995035648,"available":95764733952,"internally_reserved":0,"allocated":659456,"data_stored":326531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":229505477},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:19:51.365 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:19:51.373 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:51.394 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:19:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:51.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:19:51.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:51.527 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:51.530 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2971201569' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:19:51.530 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:51 smithi088 ceph-mon[44699]: pgmap v1033: 17 pgs: 5 creating+peering, 12 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:51.546 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:51.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:51.550 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:51.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2971201569' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:19:51.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:51 smithi084 ceph-mon[39784]: pgmap v1033: 17 pgs: 5 creating+peering, 12 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:51.650 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:19:51.680 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2971201569' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:19:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:51 smithi090 ceph-mon[44434]: pgmap v1033: 17 pgs: 5 creating+peering, 12 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:51.813 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:19:51.814 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:19:51.814 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2024-05-15T23:19:51.814 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:19:51.828 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:19:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:51.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:19:51.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:51.965 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:51.987 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:51.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:51.991 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:52.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:52.120 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:52.140 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:52.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:52.143 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:52.244 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:52.272 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:52.490 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:52 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:19:52.491 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:52 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2971201569' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:19:52.491 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:52 smithi088 ceph-mon[44699]: osdmap e378: 12 total, 11 up, 10 in 2024-05-15T23:19:52.515 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T23:19:52.521 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:52.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:52.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:52.565 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:52 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:19:52.565 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:52 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2971201569' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:19:52.565 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:52 smithi084 ceph-mon[39784]: osdmap e378: 12 total, 11 up, 10 in 2024-05-15T23:19:52.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:52.653 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:52 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:19:52.654 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:52 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2971201569' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:19:52.654 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:52 smithi090 ceph-mon[44434]: osdmap e378: 12 total, 11 up, 10 in 2024-05-15T23:19:52.656 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:52.678 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:52.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:52.682 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:52.779 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:19:52.782 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:52.810 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:53.058 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:53.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:53.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:53.162 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:19:53.188 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:53.286 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:19:53.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:53.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:19:53.389 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:19:53.416 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:53.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:53 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:19:53.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:53 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:19:53.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:53 smithi088 ceph-mon[44699]: pgmap v1035: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:53 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:19:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:53 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:19:53.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:53 smithi084 ceph-mon[39784]: pgmap v1035: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:53.764 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:19:53.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:53.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:19:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:53 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:19:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:53 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:19:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:53 smithi090 ceph-mon[44434]: pgmap v1035: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:53.868 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:53.900 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:53.973 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:53.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:53.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:54.078 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:54.106 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:54.218 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:54.232 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:19:54.252 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:19:54.271 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:19:54.292 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:19:54.311 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:19:54.332 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:19:54.352 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:19:54.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:54 smithi084 ceph-mon[39784]: osdmap e379: 12 total, 11 up, 10 in 2024-05-15T23:19:54.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4003586529' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:54.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:54.372 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:19:54.393 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:19:54.413 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:19:54.435 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:19:54.435 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-05-15T23:19:54.435 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:19:54.435 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:19:54.435 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:19:54.435 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:19:54.435 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:19:54.436 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:19:54.436 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:19:54.436 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:19:54.436 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:19:54.436 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:19:54.436 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:19:54.456 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:54.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:54.555 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:54 smithi088 ceph-mon[44699]: osdmap e379: 12 total, 11 up, 10 in 2024-05-15T23:19:54.555 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4003586529' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:54.555 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:54.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:54.567 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:19:54.588 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:54.686 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:54.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:54.790 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:54.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:54 smithi090 ceph-mon[44434]: osdmap e379: 12 total, 11 up, 10 in 2024-05-15T23:19:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4003586529' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:19:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:54.820 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:54.870 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:19:54.871 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:19:54.873 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:54.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:54.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:54.977 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:55.003 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:55.025 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:55.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:55.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:55.129 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:19:55.156 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:55.326 INFO:teuthology.orchestra.run.smithi084.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-05-15T23:19:55.504 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:19:55.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:55 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:19:55.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:55 smithi088 ceph-mon[44699]: pgmap v1037: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:55.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:55 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/206809111' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:19:55.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:55.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:19:55.531 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T23:19:55.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:55.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:55 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:19:55.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:55 smithi084 ceph-mon[39784]: pgmap v1037: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:55.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:55 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/206809111' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:19:55.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:55.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:55 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:19:55.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:55 smithi090 ceph-mon[44434]: pgmap v1037: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:55.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:55 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/206809111' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:19:55.636 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:55.653 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:55.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:55.757 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:55.788 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:55.798 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:19:55.798 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:19:55.838 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:19:56.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:56.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:56.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/206809111' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:19:56.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:56 smithi084 ceph-mon[39784]: osdmap e380: 12 total, 11 up, 10 in 2024-05-15T23:19:56.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:56 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:19:56.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:56 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:19:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/206809111' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:19:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:56 smithi088 ceph-mon[44699]: osdmap e380: 12 total, 11 up, 10 in 2024-05-15T23:19:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:56 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:19:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:56 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:19:56.779 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:19:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/206809111' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:19:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:56 smithi090 ceph-mon[44434]: osdmap e380: 12 total, 11 up, 10 in 2024-05-15T23:19:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:56 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:19:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:56 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:19:57.038 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:57.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:57.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:57.143 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:19:57.171 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:57.190 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:19:57.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:57.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:19:57.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:57.327 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:57.348 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:57.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:57.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:19:57.482 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:57.502 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:19:57.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:57.505 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:19:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:57 smithi088 ceph-mon[44699]: osdmap e381: 12 total, 11 up, 10 in 2024-05-15T23:19:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:19:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:19:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:19:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:57 smithi088 ceph-mon[44699]: pgmap v1040: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:57.606 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:19:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:57 smithi084 ceph-mon[39784]: osdmap e381: 12 total, 11 up, 10 in 2024-05-15T23:19:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:19:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:19:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:19:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:57 smithi084 ceph-mon[39784]: pgmap v1040: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:57.626 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:19:57.626 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:19:57.639 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:57.658 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:19:57.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:57.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:19:57.761 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:19:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:57.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:19:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:57 smithi090 ceph-mon[44434]: osdmap e381: 12 total, 11 up, 10 in 2024-05-15T23:19:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:19:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:19:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:19:57.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:57 smithi090 ceph-mon[44434]: pgmap v1040: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:19:57.995 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:19:58.338 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:19:58.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:58.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:19:58.357 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T23:19:58.442 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:58.470 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:58.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:58 smithi084 ceph-mon[39784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:19:58.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:58 smithi084 ceph-mon[39784]: from='client.15861 v1:172.21.15.84:0/3045169785' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:19:58.552 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1040,"stamp":"2024-05-15T23:19:56.964162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":2739688,"kb_used_data":8092,"kb_used_omap":15,"kb_used_meta":2731440,"kb_avail":934711832,"statfs":{"total":959950356480,"available":957144915968,"internally_reserved":0,"allocated":8286208,"data_stored":4645996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15952,"internal_metadata":2796994992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.649143"},"pg_stats":[{"pgid":"5.f","version":"0'0","reported_seq":21,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-05-15T23:19:56.078465+0000","last_change":"2024-05-15T23:19:47.305945+0000","last_active":"2024-05-15T23:19:56.078465+0000","last_peered":"2024-05-15T23:19:56.078465+0000","last_clean":"2024-05-15T23:19:56.078465+0000","last_became_active":"2024-05-15T23:19:47.305689+0000","last_became_peered":"2024-05-15T23:19:47.305689+0000","last_unstale":"2024-05-15T23:19:56.078465+0000","last_undegraded":"2024-05-15T23:19:56.078465+0000","last_fullsized":"2024-05-15T23:19:56.078465+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:20:58.315888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":17,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-05-15T23:19:53.599815+0000","last_change":"2024-05-15T23:19:47.325799+0000","last_active":"2024-05-15T23:19:53.599815+0000","last_peered":"2024-05-15T23:19:53.599815+0000","last_clean":"2024-05-15T23:19:53.599815+0000","last_became_active":"2024-05-15T23:19:47.325414+0000","last_became_peered":"2024-05-15T23:19:47.325414+0000","last_unstale":"2024-05-15T23:19:53.599815+0000","last_undegraded":"2024-05-15T23:19:53.599815+0000","last_fullsized":"2024-05-15T23:19:53.599815+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:21:08.797414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":23,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-05-15T23:19:56.329910+0000","last_change":"2024-05-15T23:19:47.306160+0000","last_active":"2024-05-15T23:19:56.329910+0000","last_peered":"2024-05-15T23:19:56.329910+0000","last_clean":"2024-05-15T23:19:56.329910+0000","last_became_active":"2024-05-15T23:19:47.305860+0000","last_became_peered":"2024-05-15T23:19:47.305860+0000","last_unstale":"2024-05-15T23:19:56.329910+0000","last_undegraded":"2024-05-15T23:19:56.329910+0000","last_fullsized":"2024-05-15T23:19:56.329910+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:20:58.120635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":20,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-05-15T23:19:55.323734+0000","last_change":"2024-05-15T23:19:47.306425+0000","last_active":"2024-05-15T23:19:55.323734+0000","last_peered":"2024-05-15T23:19:55.323734+0000","last_clean":"2024-05-15T23:19:55.323734+0000","last_became_active":"2024-05-15T23:19:47.306168+0000","last_became_peered":"2024-05-15T23:19:47.306168+0000","last_unstale":"2024-05-15T23:19:55.323734+0000","last_undegraded":"2024-05-15T23:19:55.323734+0000","last_fullsized":"2024-05-15T23:19:55.323734+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:20:53.202575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":17,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-05-15T23:19:53.599811+0000","last_change":"2024-05-15T23:19:47.307437+0000","last_active":"2024-05-15T23:19:53.599811+0000","last_peered":"2024-05-15T23:19:53.599811+0000","last_clean":"2024-05-15T23:19:53.599811+0000","last_became_active":"2024-05-15T23:19:47.307061+0000","last_became_peered":"2024-05-15T23:19:47.307061+0000","last_unstale":"2024-05-15T23:19:53.599811+0000","last_undegraded":"2024-05-15T23:19:53.599811+0000","last_fullsized":"2024-05-15T23:19:53.599811+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:21:01.341051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":18,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-05-15T23:19:53.599970+0000","last_change":"2024-05-15T23:19:47.325657+0000","last_active":"2024-05-15T23:19:53.599970+0000","last_peered":"2024-05-15T23:19:53.599970+0000","last_clean":"2024-05-15T23:19:53.599970+0000","last_became_active":"2024-05-15T23:19:47.325371+0000","last_became_peered":"2024-05-15T23:19:47.325371+0000","last_unstale":"2024-05-15T23:19:53.599970+0000","last_undegraded":"2024-05-15T23:19:53.599970+0000","last_fullsized":"2024-05-15T23:19:53.599970+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:21:13.833896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":17,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-05-15T23:19:53.599487+0000","last_change":"2024-05-15T23:19:47.301587+0000","last_active":"2024-05-15T23:19:53.599487+0000","last_peered":"2024-05-15T23:19:53.599487+0000","last_clean":"2024-05-15T23:19:53.599487+0000","last_became_active":"2024-05-15T23:19:47.301349+0000","last_became_peered":"2024-05-15T23:19:47.301349+0000","last_unstale":"2024-05-15T23:19:53.599487+0000","last_undegraded":"2024-05-15T23:19:53.599487+0000","last_fullsized":"2024-05-15T23:19:53.599487+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:21:01.063035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":23,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-05-15T23:19:56.329905+0000","last_change":"2024-05-15T23:19:47.304832+0000","last_active":"2024-05-15T23:19:56.329905+0000","last_peered":"2024-05-15T23:19:56.329905+0000","last_clean":"2024-05-15T23:19:56.329905+0000","last_became_active":"2024-05-15T23:19:47.304547+0000","last_became_peered":"2024-05-15T23:19:47.304547+0000","last_unstale":"2024-05-15T23:19:56.329905+0000","last_undegraded":"2024-05-15T23:19:56.329905+0000","last_fullsized":"2024-05-15T23:19:56.329905+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:21:12.061432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":21,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-05-15T23:19:56.078513+0000","last_change":"2024-05-15T23:19:47.306634+0000","last_active":"2024-05-15T23:19:56.078513+0000","last_peered":"2024-05-15T23:19:56.078513+0000","last_clean":"2024-05-15T23:19:56.078513+0000","last_became_active":"2024-05-15T23:19:47.306331+0000","last_became_peered":"2024-05-15T23:19:47.306331+0000","last_unstale":"2024-05-15T23:19:56.078513+0000","last_undegraded":"2024-05-15T23:19:56.078513+0000","last_fullsized":"2024-05-15T23:19:56.078513+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:21:00.056654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":21,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-05-15T23:19:56.078465+0000","last_change":"2024-05-15T23:19:47.306611+0000","last_active":"2024-05-15T23:19:56.078465+0000","last_peered":"2024-05-15T23:19:56.078465+0000","last_clean":"2024-05-15T23:19:56.078465+0000","last_became_active":"2024-05-15T23:19:47.306349+0000","last_became_peered":"2024-05-15T23:19:47.306349+0000","last_unstale":"2024-05-15T23:19:56.078465+0000","last_undegraded":"2024-05-15T23:19:56.078465+0000","last_fullsized":"2024-05-15T23:19:56.078465+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:21:11.136300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":20,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-05-15T23:19:55.323720+0000","last_change":"2024-05-15T23:19:47.305988+0000","last_active":"2024-05-15T23:19:55.323720+0000","last_peered":"2024-05-15T23:19:55.323720+0000","last_clean":"2024-05-15T23:19:55.323720+0000","last_became_active":"2024-05-15T23:19:47.305685+0000","last_became_peered":"2024-05-15T23:19:47.305685+0000","last_unstale":"2024-05-15T23:19:55.323720+0000","last_undegraded":"2024-05-15T23:19:55.323720+0000","last_fullsized":"2024-05-15T23:19:55.323720+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:21:05.086484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-05-15T23:19:52.077481+0000","last_change":"2024-05-15T23:19:47.306838+0000","last_active":"2024-05-15T23:19:52.077481+0000","last_peered":"2024-05-15T23:19:52.077481+0000","last_clean":"2024-05-15T23:19:52.077481+0000","last_became_active":"2024-05-15T23:19:47.306523+0000","last_became_peered":"2024-05-15T23:19:47.306523+0000","last_unstale":"2024-05-15T23:19:52.077481+0000","last_undegraded":"2024-05-15T23:19:52.077481+0000","last_fullsized":"2024-05-15T23:19:52.077481+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:21:10.443744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":16,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-05-15T23:19:52.077801+0000","last_change":"2024-05-15T23:19:47.304803+0000","last_active":"2024-05-15T23:19:52.077801+0000","last_peered":"2024-05-15T23:19:52.077801+0000","last_clean":"2024-05-15T23:19:52.077801+0000","last_became_active":"2024-05-15T23:19:47.304522+0000","last_became_peered":"2024-05-15T23:19:47.304522+0000","last_unstale":"2024-05-15T23:19:52.077801+0000","last_undegraded":"2024-05-15T23:19:52.077801+0000","last_fullsized":"2024-05-15T23:19:52.077801+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:20:49.336544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":18,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-05-15T23:19:53.318197+0000","last_change":"2024-05-15T23:19:47.517124+0000","last_active":"2024-05-15T23:19:53.318197+0000","last_peered":"2024-05-15T23:19:53.318197+0000","last_clean":"2024-05-15T23:19:53.318197+0000","last_became_active":"2024-05-15T23:19:47.516865+0000","last_became_peered":"2024-05-15T23:19:47.516865+0000","last_unstale":"2024-05-15T23:19:53.318197+0000","last_undegraded":"2024-05-15T23:19:53.318197+0000","last_fullsized":"2024-05-15T23:19:53.318197+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:21:11.276570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":18,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-05-15T23:19:53.599957+0000","last_change":"2024-05-15T23:19:47.305723+0000","last_active":"2024-05-15T23:19:53.599957+0000","last_peered":"2024-05-15T23:19:53.599957+0000","last_clean":"2024-05-15T23:19:53.599957+0000","last_became_active":"2024-05-15T23:19:47.305459+0000","last_became_peered":"2024-05-15T23:19:47.305459+0000","last_unstale":"2024-05-15T23:19:53.599957+0000","last_undegraded":"2024-05-15T23:19:53.599957+0000","last_fullsized":"2024-05-15T23:19:53.599957+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:20:57.455850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":18,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-05-15T23:19:53.599981+0000","last_change":"2024-05-15T23:19:47.307499+0000","last_active":"2024-05-15T23:19:53.599981+0000","last_peered":"2024-05-15T23:19:53.599981+0000","last_clean":"2024-05-15T23:19:53.599981+0000","last_became_active":"2024-05-15T23:19:47.307149+0000","last_became_peered":"2024-05-15T23:19:47.307149+0000","last_unstale":"2024-05-15T23:19:53.599981+0000","last_undegraded":"2024-05-15T23:19:53.599981+0000","last_fullsized":"2024-05-15T23:19:53.599981+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:20:54.248491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":787,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-05-15T23:19:53.599487+0000","last_change":"2024-05-15T23:18:33.705243+0000","last_active":"2024-05-15T23:19:53.599487+0000","last_peered":"2024-05-15T23:19:53.599487+0000","last_clean":"2024-05-15T23:19:53.599487+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:19:53.599487+0000","last_undegraded":"2024-05-15T23:19:53.599487+0000","last_fullsized":"2024-05-15T23:19:53.599487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:18:33.705118+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:18:33.705118+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351170,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":425328,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93319824,"statfs":{"total":95995035648,"available":95559499776,"internally_reserved":0,"allocated":1277952,"data_stored":899893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":434239940},"hb_peers":[1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580103,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":166540,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93578612,"statfs":{"total":95995035648,"available":95824498688,"internally_reserved":0,"allocated":720896,"data_stored":358127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":169802176},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":324812,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93420340,"statfs":{"total":95995035648,"available":95662428160,"internally_reserved":0,"allocated":720896,"data_stored":358127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":331872704},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005267,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":144320,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93600832,"statfs":{"total":95995035648,"available":95847251968,"internally_reserved":0,"allocated":708608,"data_stored":351856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266903,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":262872,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93482280,"statfs":{"total":95995035648,"available":95725854720,"internally_reserved":0,"allocated":1253376,"data_stored":887055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":267909578},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148480,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93596672,"statfs":{"total":95995035648,"available":95842992128,"internally_reserved":0,"allocated":708608,"data_stored":351856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151321030},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575095,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":360908,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93384244,"statfs":{"total":95995035648,"available":95625465856,"internally_reserved":0,"allocated":720896,"data_stored":358127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":368835014},"hb_peers":[2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483193,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":352592,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93392560,"statfs":{"total":95995035648,"available":95633981440,"internally_reserved":0,"allocated":720896,"data_stored":358127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":360315335},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986406,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":328976,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93416176,"statfs":{"total":95995035648,"available":95658164224,"internally_reserved":0,"allocated":720896,"data_stored":358127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790178,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":224860,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":93520292,"statfs":{"total":95995035648,"available":95764779008,"internally_reserved":0,"allocated":733184,"data_stored":364701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":229505477},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:19:58.553 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-15T23:19:58.553 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-05-15T23:19:58.628 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:19:58.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:58 smithi090 ceph-mon[44434]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:19:58.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:58 smithi090 ceph-mon[44434]: from='client.15861 v1:172.21.15.84:0/3045169785' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:19:58.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:58 smithi088 ceph-mon[44699]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:19:58.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:58 smithi088 ceph-mon[44699]: from='client.15861 v1:172.21.15.84:0/3045169785' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:19:58.950 DEBUG:teuthology.orchestra.run.smithi090:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-05-15T23:19:58.953 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T23:19:58.953 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T23:19:59.009 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:19:59.018 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:19:59.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:59.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:19:59.045 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:hints = 1 2024-05-15T23:19:59.045 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T23:19:59.045 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object prefix: benchmark_data_smithi090_116470 2024-05-15T23:19:59.045 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:19:59.045 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 0 0 0 0 0 0 - 0 2024-05-15T23:19:59.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:19:59.164 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:59.524 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:19:59.525 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:19:59 smithi090 ceph-mon[44434]: pgmap v1041: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:59.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:19:59.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:19:59 smithi084 ceph-mon[39784]: pgmap v1041: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:59.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:19:59.657 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:19:59.809 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:19:59.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:19:59.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:19:59.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:19:59 smithi088 ceph-mon[44699]: pgmap v1041: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-05-15T23:19:59.912 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:19:59.944 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:00.045 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 1 16 5669 5653 353.294 353.312 0.00305382 0.00282549 2024-05-15T23:20:00.295 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:00.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:00.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:00.341 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:00 smithi090 ceph-mon[44434]: overall HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-05-15T23:20:00.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:20:00.449 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:00.479 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:20:00.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:00.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:20:00.583 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:00.616 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:00 smithi088 ceph-mon[44699]: overall HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-05-15T23:20:00.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:00 smithi084 ceph-mon[39784]: overall HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-05-15T23:20:00.621 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:00.647 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:00.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:00.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:00.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:20:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:20:00.782 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:00.857 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:00.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:00.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:00.962 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:00.992 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:01.045 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 2 16 10484 10468 327.104 300.938 0.00470442 0.00305295 2024-05-15T23:20:01.093 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:01.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:01.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:01.199 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:20:01.238 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:20:01.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:01.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:20:01.377 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:01.413 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:01.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:01 smithi090 ceph-mon[44434]: pgmap v1042: 17 pgs: 17 active+clean; 248 MiB data, 2.9 GiB used, 891 GiB / 894 GiB avail; 32 MiB/s wr, 518 op/s 2024-05-15T23:20:01.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:01 smithi088 ceph-mon[44699]: pgmap v1042: 17 pgs: 17 active+clean; 248 MiB data, 2.9 GiB used, 891 GiB / 894 GiB avail; 32 MiB/s wr, 518 op/s 2024-05-15T23:20:01.573 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:01.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:01.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:01.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:01 smithi084 ceph-mon[39784]: pgmap v1042: 17 pgs: 17 active+clean; 248 MiB data, 2.9 GiB used, 891 GiB / 894 GiB avail; 32 MiB/s wr, 518 op/s 2024-05-15T23:20:01.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T23:20:01.679 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:01.747 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:01.784 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:01.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:01.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:01.870 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:20:01.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:01.893 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:20:01.916 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:20:01.937 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:20:01.949 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:01.960 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:20:01.983 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:20:01.987 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:01.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:01.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:02.007 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:20:02.033 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:20:02.047 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 3 16 14835 14819 308.594 271.938 0.00533122 0.00323564 2024-05-15T23:20:02.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:20:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:20:02.080 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:20:02.092 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:20:02.108 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:20:02.108 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-05-15T23:20:02.108 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:20:02.108 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:20:02.108 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:20:02.108 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:20:02.108 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:20:02.108 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:20:02.108 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:20:02.109 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:20:02.109 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:20:02.109 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:20:02.109 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:20:02.132 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:02.157 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:20:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:02.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:20:02.262 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:20:02.305 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:02.339 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:20:02.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:02.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:20:02.359 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:20:02.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3760829907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:02.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3466868557' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:20:02.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:02.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:02.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:02.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:02.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:20:02.484 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:02.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3760829907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:02.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3466868557' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:20:02.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:02.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:02.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:02.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:02.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3760829907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:02.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3466868557' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:20:02.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:02.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:02.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:02.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:02.594 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:20:02.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:02.597 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:20:02.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:20:02.735 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:20:02.986 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:20:03.049 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 4 15 18852 18837 294.224 251.125 0.00377887 0.00339591 2024-05-15T23:20:03.289 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:20:03.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:03.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:20:03.394 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:03.423 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:03.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:03 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:20:03.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3466868557' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:20:03.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:03 smithi090 ceph-mon[44434]: osdmap e382: 12 total, 11 up, 10 in 2024-05-15T23:20:03.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:03 smithi090 ceph-mon[44434]: pgmap v1044: 17 pgs: 17 active+clean; 670 MiB data, 3.6 GiB used, 890 GiB / 894 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-05-15T23:20:03.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:03 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:20:03.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3466868557' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:20:03.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:03 smithi084 ceph-mon[39784]: osdmap e382: 12 total, 11 up, 10 in 2024-05-15T23:20:03.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:03 smithi084 ceph-mon[39784]: pgmap v1044: 17 pgs: 17 active+clean; 670 MiB data, 3.6 GiB used, 890 GiB / 894 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-05-15T23:20:03.672 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:03.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:03.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:03.778 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:03.807 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:03.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:03.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:03 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:20:03.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3466868557' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:20:03.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:03 smithi088 ceph-mon[44699]: osdmap e382: 12 total, 11 up, 10 in 2024-05-15T23:20:03.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:03 smithi088 ceph-mon[44699]: pgmap v1044: 17 pgs: 17 active+clean; 670 MiB data, 3.6 GiB used, 890 GiB / 894 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-05-15T23:20:03.828 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:03.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:03.932 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:03.960 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:03.981 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:03.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:03.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:04.047 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 5 16 22804 22788 284.761 246.938 0.00543041 0.00350945 2024-05-15T23:20:04.086 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:04.135 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:04.167 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:04.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:04.272 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:04.308 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:04.343 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:04.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:04.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:04.448 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:04.476 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:20:04.477 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:20:04.477 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:20:04.477 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:20:04.477 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:20:04.489 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:04.517 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:04.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:04.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:04.621 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:04.663 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:04.692 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:04.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:04.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:04.797 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:04.835 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:04.865 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:04.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:04.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:04.969 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:20:04.987 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:20:05.009 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:05.038 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:20:05.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:05.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:20:05.047 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 6 16 26267 26251 273.373 216.438 0.00200512 0.00365402 2024-05-15T23:20:05.141 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:05.171 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:05.199 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:05.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:05.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:05.233 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:05 smithi090 ceph-mon[44434]: pgmap v1045: 17 pgs: 17 active+clean; 752 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-05-15T23:20:05.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T23:20:05.301 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:05.304 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:05 smithi088 ceph-mon[44699]: pgmap v1045: 17 pgs: 17 active+clean; 752 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-05-15T23:20:05.355 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:05.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:05 smithi084 ceph-mon[39784]: pgmap v1045: 17 pgs: 17 active+clean; 752 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-05-15T23:20:05.390 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:05.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:05.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:05.494 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:20:05.537 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:05.561 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:20:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:05.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:20:05.665 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:20:05.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:20:05.702 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:05.728 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:05.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:05.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:05.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:05.878 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:05.961 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:05.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:05.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:06.048 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 7 15 29533 29518 263.465 204.188 0.00329725 0.00379421 2024-05-15T23:20:06.060 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:06 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:20:06.060 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:06 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:20:06.066 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:06.105 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:06.134 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:06.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:06.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:06.231 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:06 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:20:06.231 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:06 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:20:06.239 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:06.269 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:06.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:06 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:20:06.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:06 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:20:06.322 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:06.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:06.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:06.431 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:06.459 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:06.708 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:06.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:06.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:06.768 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:06.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:20:06.854 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:07.049 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 8 16 32860 32844 256.516 207.875 0.00191579 0.00389587 2024-05-15T23:20:07.224 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:20:07.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:07.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:20:07.260 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:07 smithi084 ceph-mon[39784]: pgmap v1046: 17 pgs: 17 active+clean; 1.5 GiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-05-15T23:20:07.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:07 smithi088 ceph-mon[44699]: pgmap v1046: 17 pgs: 17 active+clean; 1.5 GiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-05-15T23:20:07.328 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:07.358 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:07.480 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:07 smithi090 ceph-mon[44434]: pgmap v1046: 17 pgs: 17 active+clean; 1.5 GiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-05-15T23:20:07.603 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:20:07.603 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:20:07.603 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:20:07.605 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:20:07.628 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:20:07.654 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:20:07.678 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:20:07.705 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:20:07.728 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:20:07.752 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:20:07.776 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:20:07.776 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-05-15T23:20:07.776 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:20:07.776 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:20:07.776 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:20:07.776 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:20:07.776 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:20:07.776 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:20:07.776 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:20:07.776 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:20:07.776 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:20:07.776 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:20:07.777 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:20:07.808 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:20:07.808 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:20:08.048 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 9 16 35949 35933 249.465 193.062 0.00680576 0.00400633 2024-05-15T23:20:08.240 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/510894469' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:08.240 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:08.240 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:08.240 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:08.240 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/510894469' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:08.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:08.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:08.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/510894469' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:08.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.010 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:09.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:09.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:20:09.040 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:20:09.048 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 10 16 39020 39004 243.712 191.938 0.00326239 0.00409687 2024-05-15T23:20:09.116 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:20:09.161 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:09.191 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:20:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:09.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:20:09.297 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:09.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:09 smithi084 ceph-mon[39784]: pgmap v1047: 17 pgs: 17 active+clean; 1.8 GiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 187 MiB/s wr, 2.99k op/s 2024-05-15T23:20:09.326 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:09.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:09 smithi090 ceph-mon[44434]: pgmap v1047: 17 pgs: 17 active+clean; 1.8 GiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 187 MiB/s wr, 2.99k op/s 2024-05-15T23:20:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:09.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:09 smithi088 ceph-mon[44699]: pgmap v1047: 17 pgs: 17 active+clean; 1.8 GiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 187 MiB/s wr, 2.99k op/s 2024-05-15T23:20:09.574 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:09.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:09.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:09.679 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:09.733 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:09.893 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:09.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:09.902 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:09.943 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T23:20:10.003 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:20:10.048 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 11 15 41884 41869 237.831 179.062 0.0322904 0.0041996 2024-05-15T23:20:10.064 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:10.094 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:20:10.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:10.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:20:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:10 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:20:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:10 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:20:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:10 smithi084 ceph-mon[39784]: osdmap e383: 12 total, 11 up, 10 in 2024-05-15T23:20:10.200 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:10.250 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:10.281 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:10.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:10.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:10 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:20:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:10 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:20:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:10 smithi090 ceph-mon[44434]: osdmap e383: 12 total, 11 up, 10 in 2024-05-15T23:20:10.372 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:20:10.386 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:20:10.419 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:10 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:20:10.419 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:10 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:20:10.419 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:10 smithi088 ceph-mon[44699]: osdmap e383: 12 total, 11 up, 10 in 2024-05-15T23:20:10.422 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:10.456 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:20:10.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:10.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:20:10.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:10.598 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:10.619 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:10.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:10.724 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:10.772 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:10.791 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:10.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:10.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:10.895 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:20:10.938 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:10.967 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:20:10.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:10.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:20:11.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:20:11.048 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 12 16 44472 44456 231.486 161.688 0.00441624 0.00431811 2024-05-15T23:20:11.075 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:11.096 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:20:11.116 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:11.139 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:11.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:11.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:11.274 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:11.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:11 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1670117872' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:20:11.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:11 smithi084 ceph-mon[39784]: pgmap v1049: 17 pgs: 17 active+clean; 2.3 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 217 MiB/s wr, 3.48k op/s 2024-05-15T23:20:11.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:11 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1670117872' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:20:11.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:11 smithi090 ceph-mon[44434]: pgmap v1049: 17 pgs: 17 active+clean; 2.3 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 217 MiB/s wr, 3.48k op/s 2024-05-15T23:20:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:11 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1670117872' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:20:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:11 smithi088 ceph-mon[44699]: pgmap v1049: 17 pgs: 17 active+clean; 2.3 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 217 MiB/s wr, 3.48k op/s 2024-05-15T23:20:11.963 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:12.048 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 13 16 47109 47093 226.357 164.812 0.00313313 0.00441623 2024-05-15T23:20:12.267 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:12 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:20:12.267 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:12 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:20:12.267 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:12 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1670117872' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:20:12.267 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:12 smithi084 ceph-mon[39784]: osdmap e384: 12 total, 11 up, 10 in 2024-05-15T23:20:12.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:12 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:20:12.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:12 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:20:12.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:12 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1670117872' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:20:12.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:12 smithi090 ceph-mon[44434]: osdmap e384: 12 total, 11 up, 10 in 2024-05-15T23:20:12.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:12 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:20:12.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:12 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:20:12.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:12 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1670117872' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:20:12.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:12 smithi088 ceph-mon[44699]: osdmap e384: 12 total, 11 up, 10 in 2024-05-15T23:20:13.048 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 14 16 49524 49508 220.969 150.938 0.00506629 0.00452328 2024-05-15T23:20:13.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:13 smithi084 ceph-mon[39784]: pgmap v1051: 17 pgs: 17 active+clean; 2.7 GiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 255 MiB/s wr, 4.08k op/s 2024-05-15T23:20:13.431 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:13.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:13 smithi090 ceph-mon[44434]: pgmap v1051: 17 pgs: 17 active+clean; 2.7 GiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 255 MiB/s wr, 4.08k op/s 2024-05-15T23:20:13.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:13.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:13.535 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:20:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:13 smithi088 ceph-mon[44699]: pgmap v1051: 17 pgs: 17 active+clean; 2.7 GiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 255 MiB/s wr, 4.08k op/s 2024-05-15T23:20:13.576 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:13.603 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:20:13.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:13.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:20:13.708 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:13.760 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:13.792 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:13.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:13.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:13.897 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:13.919 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:20:13.934 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:13.963 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:13.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:13.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:14.003 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:20:14.026 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:20:14.048 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:20:14.050 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 15 16 51823 51807 215.796 143.688 0.00376328 0.00463228 2024-05-15T23:20:14.068 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:14.074 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:20:14.103 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:20:14.113 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:14.129 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:20:14.142 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:14.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:14.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:14.155 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:20:14.182 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:20:14.210 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:20:14.237 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:20:14.247 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:14.254 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:14.265 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:20:14.266 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-05-15T23:20:14.266 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:20:14.266 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:20:14.266 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:20:14.266 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:20:14.266 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:20:14.266 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:20:14.266 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:20:14.266 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:20:14.266 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:20:14.266 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:20:14.266 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:20:14.277 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:14 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1062579861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:14.281 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:14.311 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:14.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:14.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:14.415 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:20:14.458 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:14 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1062579861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:14.461 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:14.478 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:14 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1062579861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:14.488 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:20:14.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:14.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:20:14.592 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:14.622 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:14.748 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:14.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:14.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:20:14.896 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:14.940 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:20:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:14.947 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:20:15.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:20:15.053 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 16 16 54134 54118 211.336 144.438 0.00890784 0.00472999 2024-05-15T23:20:15.093 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:15.124 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:20:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:20:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:54266 5.b 5:dec00f32:::benchmark_data_smithi090_116470_object54265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e384)", 2024-05-15T23:20:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:15.118624+0000", 2024-05-15T23:20:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0035557520000000001, 2024-05-15T23:20:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036052520000000002, 2024-05-15T23:20:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:20:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54266 2024-05-15T23:20:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.118624+0000", 2024-05-15T23:20:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.118624+0000", 2024-05-15T23:20:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.118625+0000", 2024-05-15T23:20:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0499999999999999e-06 2024-05-15T23:20:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.118643+0000", 2024-05-15T23:20:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8439000000000001e-05 2024-05-15T23:20:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.118651+0000", 2024-05-15T23:20:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0909999999999993e-06 2024-05-15T23:20:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.118658+0000", 2024-05-15T23:20:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8809999999999998e-06 2024-05-15T23:20:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.119351+0000", 2024-05-15T23:20:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069313100000000004 2024-05-15T23:20:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.119481+0000", 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012999600000000001 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.119624+0000", 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014272300000000001 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.121633+0000", 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020087220000000001 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-05-15T23:20:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2024-05-15T23:20:15.129 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:20:15.129 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:20:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:15.134 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:20:15.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:20:15.270 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:15.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:15 smithi084 ceph-mon[39784]: pgmap v1052: 17 pgs: 17 active+clean; 2.8 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-05-15T23:20:15.430 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:20:15.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:15 smithi090 ceph-mon[44434]: pgmap v1052: 17 pgs: 17 active+clean; 2.8 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-05-15T23:20:15.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:15.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:20:15.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:20:15.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:15.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:15 smithi088 ceph-mon[44699]: pgmap v1052: 17 pgs: 17 active+clean; 2.8 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-05-15T23:20:15.582 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:20:15.756 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:20:15.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T23:20:15.944 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:20:15.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:15.947 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:20:16.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:16.051 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 17 16 56284 56268 206.809 134.375 0.00713353 0.00483337 2024-05-15T23:20:16.087 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:16.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:16.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:16.256 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:16.280 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:16.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:16.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:16.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:16 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:20:16.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:16 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:20:16.339 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:20:16.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:16 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:20:16.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:16 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:20:16.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:20:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:16.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:16 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:20:16.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:16 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:20:16.676 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:20:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:16.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:20:16.787 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:16.820 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:16.844 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:16.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:16.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:16.948 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:20:16.996 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:17.050 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 18 16 58478 58462 202.938 137.125 0.00609409 0.00492587 2024-05-15T23:20:17.253 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:20:17.255 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:17 smithi090 ceph-mon[44434]: osdmap e385: 12 total, 11 up, 10 in 2024-05-15T23:20:17.255 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:17 smithi090 ceph-mon[44434]: pgmap v1054: 17 pgs: 17 active+clean; 3.2 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 183 MiB/s wr, 2.92k op/s 2024-05-15T23:20:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:17.256 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:20:17.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:17 smithi084 ceph-mon[39784]: osdmap e385: 12 total, 11 up, 10 in 2024-05-15T23:20:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:17 smithi084 ceph-mon[39784]: pgmap v1054: 17 pgs: 17 active+clean; 3.2 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 183 MiB/s wr, 2.92k op/s 2024-05-15T23:20:17.385 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:17.406 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:17.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:17.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:17.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:17.537 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:17.559 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:17.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:17.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:17.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:17 smithi088 ceph-mon[44699]: osdmap e385: 12 total, 11 up, 10 in 2024-05-15T23:20:17.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:17 smithi088 ceph-mon[44699]: pgmap v1054: 17 pgs: 17 active+clean; 3.2 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 183 MiB/s wr, 2.92k op/s 2024-05-15T23:20:17.663 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:17.714 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:17.746 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:17.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:17.757 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:20:17.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:20:17.900 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:18.050 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 19 15 60689 60674 199.533 138.25 0.00512841 0.00501048 2024-05-15T23:20:18.054 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:18.115 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:20:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:18.119 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:20:18.130 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:20:18.220 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:20:18.265 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:18.289 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:20:18.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:18.292 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:20:18.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:18 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2042664953' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:20:18.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:18.431 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:18.457 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:20:18.457 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:20:18.457 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:20:18.457 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:20:18.457 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:20:18.482 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:18 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2042664953' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:20:18.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:18 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2042664953' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:20:18.629 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:18.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:18.734 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:20:18.783 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:18.815 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:20:18.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:18.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:20:18.921 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:18.949 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:19.050 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:20:19.049964 min lat: 0.000997126 max lat: 0.0901907 avg lat: 0.00509888 2024-05-15T23:20:19.050 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:20:19.050 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 20 16 62762 62746 196.032 129.5 0.00436051 0.00509888 2024-05-15T23:20:19.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:19 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:20:19.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2042664953' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:20:19.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:19 smithi090 ceph-mon[44434]: osdmap e386: 12 total, 11 up, 10 in 2024-05-15T23:20:19.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:19 smithi090 ceph-mon[44434]: pgmap v1056: 17 pgs: 17 active+clean; 3.5 GiB data, 9.5 GiB used, 884 GiB / 894 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-05-15T23:20:19.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T23:20:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:19 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:20:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2042664953' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:20:19.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:19 smithi084 ceph-mon[39784]: osdmap e386: 12 total, 11 up, 10 in 2024-05-15T23:20:19.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:19 smithi084 ceph-mon[39784]: pgmap v1056: 17 pgs: 17 active+clean; 3.5 GiB data, 9.5 GiB used, 884 GiB / 894 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-05-15T23:20:19.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:19 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:20:19.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2042664953' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:20:19.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:19 smithi088 ceph-mon[44699]: osdmap e386: 12 total, 11 up, 10 in 2024-05-15T23:20:19.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:19 smithi088 ceph-mon[44699]: pgmap v1056: 17 pgs: 17 active+clean; 3.5 GiB data, 9.5 GiB used, 884 GiB / 894 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-05-15T23:20:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:20:20.050 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 21 15 64860 64845 192.944 131.188 0.0046653 0.00518071 2024-05-15T23:20:21.052 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 22 16 67020 67004 190.292 134.938 0.0083447 0.00525364 2024-05-15T23:20:21.148 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:20:21.240 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:21 smithi090 ceph-mon[44434]: pgmap v1057: 17 pgs: 17 active+clean; 3.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-05-15T23:20:21.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:21 smithi088 ceph-mon[44699]: pgmap v1057: 17 pgs: 17 active+clean; 3.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-05-15T23:20:21.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:21 smithi084 ceph-mon[39784]: pgmap v1057: 17 pgs: 17 active+clean; 3.8 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-05-15T23:20:21.583 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:20:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:20:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:20:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:20:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:105657 4.6 4:6213edb0:::benchmark_data_smithi090_104448_object105656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-05-15T23:20:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:15:45.162166+0000", 2024-05-15T23:20:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 276.41673908899998, 2024-05-15T23:20:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072861380000000003, 2024-05-15T23:20:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105657 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.162166+0000", 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.162166+0000", 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.162168+0000", 2024-05-15T23:20:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2689999999999998e-06 2024-05-15T23:20:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.162192+0000", 2024-05-15T23:20:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3998999999999998e-05 2024-05-15T23:20:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.162210+0000", 2024-05-15T23:20:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8173999999999999e-05 2024-05-15T23:20:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.162219+0000", 2024-05-15T23:20:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6249999999999996e-06 2024-05-15T23:20:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.166576+0000", 2024-05-15T23:20:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043571959999999998 2024-05-15T23:20:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.166689+0000", 2024-05-15T23:20:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011356100000000001 2024-05-15T23:20:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:20:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.166813+0000", 2024-05-15T23:20:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012386499999999999 2024-05-15T23:20:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170042+0000", 2024-05-15T23:20:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032288899999999999 2024-05-15T23:20:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.234963+0000", 2024-05-15T23:20:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.064920448000000006 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.234997+0000", 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4371e-05 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.235027+0000", 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9981999999999999e-05 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:105678 4.6 4:6cfaded3:::benchmark_data_smithi090_104448_object105677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-05-15T23:20:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:15:45.170335+0000", 2024-05-15T23:20:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 276.40856927999999, 2024-05-15T23:20:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.073724063000000006, 2024-05-15T23:20:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:20:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:20:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105678 2024-05-15T23:20:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170335+0000", 2024-05-15T23:20:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170335+0000", 2024-05-15T23:20:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170337+0000", 2024-05-15T23:20:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5099999999999999e-06 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170360+0000", 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3422999999999999e-05 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170370+0000", 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5140000000000007e-06 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170377+0000", 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4290000000000004e-06 2024-05-15T23:20:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170425+0000", 2024-05-15T23:20:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7633000000000003e-05 2024-05-15T23:20:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170531+0000", 2024-05-15T23:20:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010632 2024-05-15T23:20:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:20:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.170653+0000", 2024-05-15T23:20:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012196 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.173861+0000", 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00320784 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.243995+0000", 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.070134028000000001 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.244031+0000", 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5862999999999999e-05 2024-05-15T23:20:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:15:45.244059+0000", 2024-05-15T23:20:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8543000000000001e-05 2024-05-15T23:20:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:185451 4.6 4:6aaf72ef:::benchmark_data_smithi090_104448_object12213:head [delete] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-05-15T23:20:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:16:47.294072+0000", 2024-05-15T23:20:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 214.28483315, 2024-05-15T23:20:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.066180623999999993, 2024-05-15T23:20:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:20:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:20:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 185451 2024-05-15T23:20:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294072+0000", 2024-05-15T23:20:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294072+0000", 2024-05-15T23:20:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294073+0000", 2024-05-15T23:20:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.113e-06 2024-05-15T23:20:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294077+0000", 2024-05-15T23:20:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0589999999999996e-06 2024-05-15T23:20:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294084+0000", 2024-05-15T23:20:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6559999999999992e-06 2024-05-15T23:20:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294089+0000", 2024-05-15T23:20:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7369999999999997e-06 2024-05-15T23:20:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294657+0000", 2024-05-15T23:20:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056806599999999999 2024-05-15T23:20:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294771+0000", 2024-05-15T23:20:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114119 2024-05-15T23:20:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:20:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.294885+0000", 2024-05-15T23:20:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113989 2024-05-15T23:20:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.301849+0000", 2024-05-15T23:20:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069633079999999997 2024-05-15T23:20:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.360144+0000", 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.058295670000000001 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.360206+0000", 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2120999999999997e-05 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.360252+0000", 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5785999999999998e-05 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:185455 4.6 4:6e39477b:::benchmark_data_smithi090_104448_object12217:head [delete] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-05-15T23:20:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:16:47.295082+0000", 2024-05-15T23:20:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 214.28382299800001, 2024-05-15T23:20:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.065325242000000006, 2024-05-15T23:20:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:20:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:20:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 185455 2024-05-15T23:20:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.295082+0000", 2024-05-15T23:20:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.295082+0000", 2024-05-15T23:20:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.295083+0000", 2024-05-15T23:20:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1999999999999998e-07 2024-05-15T23:20:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.295086+0000", 2024-05-15T23:20:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3299999999999999e-06 2024-05-15T23:20:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.295093+0000", 2024-05-15T23:20:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6429999999999998e-06 2024-05-15T23:20:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.295096+0000", 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7639999999999999e-06 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.298250+0000", 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031532740000000002 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.298322+0000", 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2644000000000004e-05 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.298407+0000", 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5198999999999995e-05 2024-05-15T23:20:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.301988+0000", 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003580654 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.360317+0000", 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.058329338000000001 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.360365+0000", 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7511e-05 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.360407+0000", 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1965000000000002e-05 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:186951 4.6 4:6b5f4b58:::benchmark_data_smithi090_104448_object13713:head [delete] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:16:48.492723+0000", 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 213.086182119, 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068389040999999998, 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:20:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:20:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 186951 2024-05-15T23:20:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492723+0000", 2024-05-15T23:20:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492723+0000", 2024-05-15T23:20:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492725+0000", 2024-05-15T23:20:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9700000000000002e-06 2024-05-15T23:20:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492732+0000", 2024-05-15T23:20:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.216e-06 2024-05-15T23:20:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492744+0000", 2024-05-15T23:20:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2173e-05 2024-05-15T23:20:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492753+0000", 2024-05-15T23:20:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2439999999999999e-06 2024-05-15T23:20:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492800+0000", 2024-05-15T23:20:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6848999999999998e-05 2024-05-15T23:20:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492909+0000", 2024-05-15T23:20:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010861699999999999 2024-05-15T23:20:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:20:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.492981+0000", 2024-05-15T23:20:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2712000000000004e-05 2024-05-15T23:20:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.497073+0000", 2024-05-15T23:20:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040911979999999999 2024-05-15T23:20:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.561031+0000", 2024-05-15T23:20:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063958584999999998 2024-05-15T23:20:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.561066+0000", 2024-05-15T23:20:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4675999999999998e-05 2024-05-15T23:20:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.561112+0000", 2024-05-15T23:20:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5800999999999997e-05 2024-05-15T23:20:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:186966 4.6 4:669c6b05:::benchmark_data_smithi090_104448_object13728:head [delete] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-05-15T23:20:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:16:48.500957+0000", 2024-05-15T23:20:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 213.077948002, 2024-05-15T23:20:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068427644999999995, 2024-05-15T23:20:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:20:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:20:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 186966 2024-05-15T23:20:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.500957+0000", 2024-05-15T23:20:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.500957+0000", 2024-05-15T23:20:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.500959+0000", 2024-05-15T23:20:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0190000000000001e-06 2024-05-15T23:20:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.500965+0000", 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9499999999999998e-06 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.500976+0000", 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1554e-05 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.500987+0000", 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0451999999999999e-05 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.501037+0000", 2024-05-15T23:20:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9852999999999999e-05 2024-05-15T23:20:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.501121+0000", 2024-05-15T23:20:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4594999999999999e-05 2024-05-15T23:20:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:20:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.501202+0000", 2024-05-15T23:20:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0992e-05 2024-05-15T23:20:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.505005+0000", 2024-05-15T23:20:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038027619999999999 2024-05-15T23:20:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.569295+0000", 2024-05-15T23:20:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.064290150000000004 2024-05-15T23:20:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.569346+0000", 2024-05-15T23:20:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0605000000000001e-05 2024-05-15T23:20:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:48.569384+0000", 2024-05-15T23:20:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8713e-05 2024-05-15T23:20:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:220159 4.6 4:629808a1:::benchmark_data_smithi090_104448_object46921:head [delete] snapc 0=[] ondisk+write+known_if_redirected e331)", 2024-05-15T23:20:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:15.535105+0000", 2024-05-15T23:20:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 186.043799912, 2024-05-15T23:20:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.064003843000000005, 2024-05-15T23:20:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:20:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:20:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 220159 2024-05-15T23:20:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535105+0000", 2024-05-15T23:20:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535105+0000", 2024-05-15T23:20:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535106+0000", 2024-05-15T23:20:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3549999999999999e-06 2024-05-15T23:20:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535111+0000", 2024-05-15T23:20:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5600000000000004e-06 2024-05-15T23:20:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535120+0000", 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2830000000000007e-06 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535127+0000", 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3180000000000001e-06 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535170+0000", 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2799999999999997e-05 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535251+0000", 2024-05-15T23:20:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1169999999999997e-05 2024-05-15T23:20:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:20:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535323+0000", 2024-05-15T23:20:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1778000000000003e-05 2024-05-15T23:20:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599006+0000", 2024-05-15T23:20:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063682697999999996 2024-05-15T23:20:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599022+0000", 2024-05-15T23:20:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5883e-05 2024-05-15T23:20:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599066+0000", 2024-05-15T23:20:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4277999999999999e-05 2024-05-15T23:20:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599109+0000", 2024-05-15T23:20:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2719999999999998e-05 2024-05-15T23:20:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:220163 4.6 4:6304f954:::benchmark_data_smithi090_104448_object46925:head [delete] snapc 0=[] ondisk+write+known_if_redirected e331)", 2024-05-15T23:20:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:15.535442+0000", 2024-05-15T23:20:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 186.043462772, 2024-05-15T23:20:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063832858000000006, 2024-05-15T23:20:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:20:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:20:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 220163 2024-05-15T23:20:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535442+0000", 2024-05-15T23:20:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535442+0000", 2024-05-15T23:20:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535443+0000", 2024-05-15T23:20:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.206e-06 2024-05-15T23:20:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535447+0000", 2024-05-15T23:20:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2289999999999997e-06 2024-05-15T23:20:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535455+0000", 2024-05-15T23:20:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4320000000000002e-06 2024-05-15T23:20:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535461+0000", 2024-05-15T23:20:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.908e-06 2024-05-15T23:20:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.538947+0000", 2024-05-15T23:20:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003486121 2024-05-15T23:20:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.539062+0000", 2024-05-15T23:20:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115658 2024-05-15T23:20:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:20:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.539174+0000", 2024-05-15T23:20:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011184999999999999 2024-05-15T23:20:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599117+0000", 2024-05-15T23:20:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.059942984999999997 2024-05-15T23:20:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599170+0000", 2024-05-15T23:20:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2212000000000001e-05 2024-05-15T23:20:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599232+0000", 2024-05-15T23:20:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2731000000000006e-05 2024-05-15T23:20:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599275+0000", 2024-05-15T23:20:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2525999999999999e-05 2024-05-15T23:20:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:220164 4.6 4:604e82dd:::benchmark_data_smithi090_104448_object46926:head [delete] snapc 0=[] ondisk+write+known_if_redirected e331)", 2024-05-15T23:20:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:15.535488+0000", 2024-05-15T23:20:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 186.04341641799999, 2024-05-15T23:20:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063931615999999997, 2024-05-15T23:20:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:20:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:20:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 220164 2024-05-15T23:20:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535488+0000", 2024-05-15T23:20:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535488+0000", 2024-05-15T23:20:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535489+0000", 2024-05-15T23:20:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4600000000000005e-07 2024-05-15T23:20:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535491+0000", 2024-05-15T23:20:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4200000000000001e-06 2024-05-15T23:20:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535496+0000", 2024-05-15T23:20:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6709999999999998e-06 2024-05-15T23:20:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.535500+0000", 2024-05-15T23:20:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2119999999999997e-06 2024-05-15T23:20:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.542982+0000", 2024-05-15T23:20:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074823600000000004 2024-05-15T23:20:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.543082+0000", 2024-05-15T23:20:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9883000000000006e-05 2024-05-15T23:20:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:20:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.543152+0000", 2024-05-15T23:20:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9634000000000004e-05 2024-05-15T23:20:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599329+0000", 2024-05-15T23:20:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.056176822000000001 2024-05-15T23:20:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599342+0000", 2024-05-15T23:20:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2908000000000001e-05 2024-05-15T23:20:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599384+0000", 2024-05-15T23:20:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2701000000000003e-05 2024-05-15T23:20:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.599420+0000", 2024-05-15T23:20:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5459000000000001e-05 2024-05-15T23:20:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:220176 4.2 4:47a5bebc:::benchmark_data_smithi090_104448_object46938:head [delete] snapc 0=[] ondisk+write+known_if_redirected e331)", 2024-05-15T23:20:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:15.546323+0000", 2024-05-15T23:20:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 186.032581311, 2024-05-15T23:20:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.064068929999999996, 2024-05-15T23:20:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:20:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:20:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 220176 2024-05-15T23:20:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.546323+0000", 2024-05-15T23:20:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.546323+0000", 2024-05-15T23:20:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.546325+0000", 2024-05-15T23:20:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.889e-06 2024-05-15T23:20:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.546331+0000", 2024-05-15T23:20:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4489999999999998e-06 2024-05-15T23:20:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.546341+0000", 2024-05-15T23:20:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0634e-05 2024-05-15T23:20:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.546349+0000", 2024-05-15T23:20:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.525e-06 2024-05-15T23:20:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.549631+0000", 2024-05-15T23:20:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032817900000000001 2024-05-15T23:20:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.549708+0000", 2024-05-15T23:20:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7069000000000004e-05 2024-05-15T23:20:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.598961+0000", 2024-05-15T23:20:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049253416000000001 2024-05-15T23:20:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.607209+0000", 2024-05-15T23:20:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0082474990000000001 2024-05-15T23:20:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.610338+0000", 2024-05-15T23:20:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031292210000000002 2024-05-15T23:20:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.610365+0000", 2024-05-15T23:20:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7161999999999998e-05 2024-05-15T23:20:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.610392+0000", 2024-05-15T23:20:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7276000000000001e-05 2024-05-15T23:20:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:220182 4.2 4:4383beef:::benchmark_data_smithi090_104448_object46944:head [delete] snapc 0=[] ondisk+write+known_if_redirected e331)", 2024-05-15T23:20:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:15.548488+0000", 2024-05-15T23:20:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 186.030417019, 2024-05-15T23:20:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.065174899999999994, 2024-05-15T23:20:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:20:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:20:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 220182 2024-05-15T23:20:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.548488+0000", 2024-05-15T23:20:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.548488+0000", 2024-05-15T23:20:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.548489+0000", 2024-05-15T23:20:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.668e-06 2024-05-15T23:20:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.548495+0000", 2024-05-15T23:20:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4410000000000001e-06 2024-05-15T23:20:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.548505+0000", 2024-05-15T23:20:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9399999999999997e-06 2024-05-15T23:20:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.548512+0000", 2024-05-15T23:20:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6979999999999998e-06 2024-05-15T23:20:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.603128+0000", 2024-05-15T23:20:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.054615489000000003 2024-05-15T23:20:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.603257+0000", 2024-05-15T23:20:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012957100000000001 2024-05-15T23:20:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.603366+0000", 2024-05-15T23:20:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010895600000000001 2024-05-15T23:20:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.613563+0000", 2024-05-15T23:20:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010197069 2024-05-15T23:20:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.613578+0000", 2024-05-15T23:20:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4358999999999999e-05 2024-05-15T23:20:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.613623+0000", 2024-05-15T23:20:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4727000000000001e-05 2024-05-15T23:20:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:15.613663+0000", 2024-05-15T23:20:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9981999999999998e-05 2024-05-15T23:20:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:262513 4.6 4:61b88191:::benchmark_data_smithi090_104448_object89275:head [delete] snapc 0=[] ondisk+write+known_if_redirected e341)", 2024-05-15T23:20:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:50.543613+0000", 2024-05-15T23:20:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 151.03529126699999, 2024-05-15T23:20:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.084090350999999994, 2024-05-15T23:20:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:20:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:20:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 262513 2024-05-15T23:20:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.543613+0000", 2024-05-15T23:20:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.543613+0000", 2024-05-15T23:20:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.543615+0000", 2024-05-15T23:20:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8479999999999999e-06 2024-05-15T23:20:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.543620+0000", 2024-05-15T23:20:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5990000000000004e-06 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.543628+0000", 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4360000000000002e-06 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.543635+0000", 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.114e-06 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.543717+0000", 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.14e-05 2024-05-15T23:20:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.543826+0000", 2024-05-15T23:20:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010948200000000001 2024-05-15T23:20:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:20:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.543941+0000", 2024-05-15T23:20:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114924 2024-05-15T23:20:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.623400+0000", 2024-05-15T23:20:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.079459074000000005 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.627589+0000", 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041883839999999999 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.627641+0000", 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1835000000000001e-05 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.627704+0000", 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3255000000000003e-05 2024-05-15T23:20:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.25441.0:262513 4.6 e341/283 ondisk, result = 0)", 2024-05-15T23:20:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:50.548939+0000", 2024-05-15T23:20:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 151.029965616, 2024-05-15T23:20:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.074468150999999996, 2024-05-15T23:20:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:20:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.548939+0000", 2024-05-15T23:20:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.548939+0000", 2024-05-15T23:20:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.548941+0000", 2024-05-15T23:20:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6109999999999999e-06 2024-05-15T23:20:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.548947+0000", 2024-05-15T23:20:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7769999999999997e-06 2024-05-15T23:20:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.548958+0000", 2024-05-15T23:20:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0743e-05 2024-05-15T23:20:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.548966+0000", 2024-05-15T23:20:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5290000000000003e-06 2024-05-15T23:20:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.623389+0000", 2024-05-15T23:20:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.074422947000000003 2024-05-15T23:20:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.623396+0000", 2024-05-15T23:20:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-05-15T23:20:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.623407+0000", 2024-05-15T23:20:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1082e-05 2024-05-15T23:20:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:262523 4.d e341/283 4:ba3cefb0:::benchmark_data_smithi090_104448_object89285:head v 341'16264, mlcod=341'16262)", 2024-05-15T23:20:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:50.551392+0000", 2024-05-15T23:20:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 151.02751278900001, 2024-05-15T23:20:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068299992000000004, 2024-05-15T23:20:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.551392+0000", 2024-05-15T23:20:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.551392+0000", 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.551394+0000", 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.883e-06 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.551402+0000", 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2900000000000002e-06 2024-05-15T23:20:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.551413+0000", 2024-05-15T23:20:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1175999999999999e-05 2024-05-15T23:20:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.551422+0000", 2024-05-15T23:20:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3110000000000006e-06 2024-05-15T23:20:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.551474+0000", 2024-05-15T23:20:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2827000000000001e-05 2024-05-15T23:20:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.551494+0000", 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0064e-05 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.619649+0000", 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068154207999999994 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.619692+0000", 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3232999999999998e-05 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:262530 4.d e341/283 4:b4dcfca7:::benchmark_data_smithi090_104448_object89292:head v 341'16265, mlcod=341'16262)", 2024-05-15T23:20:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:50.556898+0000", 2024-05-15T23:20:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 151.02200659499999, 2024-05-15T23:20:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.080643230999999996, 2024-05-15T23:20:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.556898+0000", 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.556898+0000", 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.556900+0000", 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.714e-06 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.556907+0000", 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2119999999999997e-06 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.619639+0000", 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062731561000000005 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.619654+0000", 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5770999999999999e-05 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.619718+0000", 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3565999999999994e-05 2024-05-15T23:20:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.619738+0000", 2024-05-15T23:20:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9731000000000001e-05 2024-05-15T23:20:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.637496+0000", 2024-05-15T23:20:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017758673999999999 2024-05-15T23:20:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:50.637541+0000", 2024-05-15T23:20:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5002e-05 2024-05-15T23:20:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:278342 4.d e345/283 4:b6201b5e:::benchmark_data_smithi090_104448_object105104:head v 345'17241, mlcod=345'17228)", 2024-05-15T23:20:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:18:04.410098+0000", 2024-05-15T23:20:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 137.16880621000001, 2024-05-15T23:20:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.073334785, 2024-05-15T23:20:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410098+0000", 2024-05-15T23:20:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410098+0000", 2024-05-15T23:20:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410099+0000", 2024-05-15T23:20:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9600000000000001e-07 2024-05-15T23:20:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410101+0000", 2024-05-15T23:20:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7130000000000001e-06 2024-05-15T23:20:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410104+0000", 2024-05-15T23:20:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.384e-06 2024-05-15T23:20:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410106+0000", 2024-05-15T23:20:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.266e-06 2024-05-15T23:20:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.452543+0000", 2024-05-15T23:20:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042436966999999999 2024-05-15T23:20:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.452563+0000", 2024-05-15T23:20:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0429999999999999e-05 2024-05-15T23:20:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.483392+0000", 2024-05-15T23:20:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030828324000000001 2024-05-15T23:20:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.483433+0000", 2024-05-15T23:20:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1505000000000003e-05 2024-05-15T23:20:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:278350 4.d e345/283 4:b848d388:::benchmark_data_smithi090_104448_object105112:head v 345'17242, mlcod=345'17228)", 2024-05-15T23:20:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:18:04.410109+0000", 2024-05-15T23:20:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 137.16879596999999, 2024-05-15T23:20:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.073363612999999994, 2024-05-15T23:20:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410109+0000", 2024-05-15T23:20:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410109+0000", 2024-05-15T23:20:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410109+0000", 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8e-07 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410113+0000", 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410116+0000", 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5119999999999998e-06 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410118+0000", 2024-05-15T23:20:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4739999999999999e-06 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.456500+0000", 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046381299000000001 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.456524+0000", 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4530000000000001e-05 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.483444+0000", 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026919381999999999 2024-05-15T23:20:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.483472+0000", 2024-05-15T23:20:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8710000000000001e-05 2024-05-15T23:20:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:278362 4.d e345/283 4:bc505f25:::benchmark_data_smithi090_104448_object105124:head v 345'17243, mlcod=345'17228)", 2024-05-15T23:20:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:18:04.410121+0000", 2024-05-15T23:20:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 137.16878348099999, 2024-05-15T23:20:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.082983913000000006, 2024-05-15T23:20:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410121+0000", 2024-05-15T23:20:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410121+0000", 2024-05-15T23:20:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410121+0000", 2024-05-15T23:20:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.92e-07 2024-05-15T23:20:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410124+0000", 2024-05-15T23:20:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6910000000000002e-06 2024-05-15T23:20:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410126+0000", 2024-05-15T23:20:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2979999999999999e-06 2024-05-15T23:20:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.410129+0000", 2024-05-15T23:20:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3269999999999999e-06 2024-05-15T23:20:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.460308+0000", 2024-05-15T23:20:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050179012000000002 2024-05-15T23:20:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.460332+0000", 2024-05-15T23:20:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4328000000000002e-05 2024-05-15T23:20:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.493065+0000", 2024-05-15T23:20:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032733050999999999 2024-05-15T23:20:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.493105+0000", 2024-05-15T23:20:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0014e-05 2024-05-15T23:20:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:278370 4.d e345/283 4:bbd446e7:::benchmark_data_smithi090_104448_object105132:head v 345'17244, mlcod=345'17230)", 2024-05-15T23:20:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:18:04.428886+0000", 2024-05-15T23:20:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 137.15001908599999, 2024-05-15T23:20:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.064244843999999995, 2024-05-15T23:20:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.428886+0000", 2024-05-15T23:20:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.428886+0000", 2024-05-15T23:20:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.428887+0000", 2024-05-15T23:20:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.497e-06 2024-05-15T23:20:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.428893+0000", 2024-05-15T23:20:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3940000000000001e-06 2024-05-15T23:20:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.428903+0000", 2024-05-15T23:20:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1400000000000006e-06 2024-05-15T23:20:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.428910+0000", 2024-05-15T23:20:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1069999999999999e-06 2024-05-15T23:20:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.465150+0000", 2024-05-15T23:20:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036240104000000002 2024-05-15T23:20:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.465173+0000", 2024-05-15T23:20:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3574000000000002e-05 2024-05-15T23:20:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.493108+0000", 2024-05-15T23:20:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027934798 2024-05-15T23:20:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:04.493130+0000", 2024-05-15T23:20:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2229999999999999e-05 2024-05-15T23:20:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25441.0:281230 4.6 4:6edb229a:::benchmark_data_smithi090_104448_object107992:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:20:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:18:06.826710+0000", 2024-05-15T23:20:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 134.752195175, 2024-05-15T23:20:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.064344461000000006, 2024-05-15T23:20:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25441", 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 281230 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:06.826710+0000", 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:06.826710+0000", 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:06.826711+0000", 2024-05-15T23:20:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.877e-06 2024-05-15T23:20:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:06.826717+0000", 2024-05-15T23:20:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1250000000000001e-06 2024-05-15T23:20:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:06.826727+0000", 2024-05-15T23:20:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.093e-05 2024-05-15T23:20:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:06.826734+0000", 2024-05-15T23:20:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1800000000000001e-06 2024-05-15T23:20:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:06.826817+0000", 2024-05-15T23:20:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3405999999999994e-05 2024-05-15T23:20:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:06.826953+0000", 2024-05-15T23:20:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013599099999999999 2024-05-15T23:20:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:06.827593+0000", 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063991500000000002 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:06.831703+0000", 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041100900000000003 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:06.890953+0000", 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.059250406999999998 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:06.890999+0000", 2024-05-15T23:20:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5677000000000003e-05 2024-05-15T23:20:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:06.891054+0000", 2024-05-15T23:20:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4863e-05 2024-05-15T23:20:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:21.684 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:20:21.701 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:21.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:21.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:21.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:21.833 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:22.052 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 23 16 68981 68965 187.347 122.562 0.0036886 0.00533625 2024-05-15T23:20:22.295 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:22.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:22.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T23:20:22.338 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:22.401 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:22.450 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:22.475 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:22.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:22.589 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:22.625 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:22.695 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:20:23.049 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:20:23.052 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 24 16 70935 70919 184.63 122.125 0.00565973 0.00541481 2024-05-15T23:20:23.070 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:20:23.093 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:20:23.116 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:20:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:20:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:20:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:23 smithi084 ceph-mon[39784]: osdmap e387: 12 total, 11 up, 10 in 2024-05-15T23:20:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:23 smithi084 ceph-mon[39784]: pgmap v1059: 17 pgs: 17 active+clean; 4.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-05-15T23:20:23.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:23 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3747183167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:23.138 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:20:23.165 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:20:23.188 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:20:23.210 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:20:23.235 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:20:23.253 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:20:23.253 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:20:23.253 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:23 smithi090 ceph-mon[44434]: osdmap e387: 12 total, 11 up, 10 in 2024-05-15T23:20:23.254 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:23 smithi090 ceph-mon[44434]: pgmap v1059: 17 pgs: 17 active+clean; 4.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-05-15T23:20:23.254 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:23 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3747183167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:23.259 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:20:23.286 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:20:23.286 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-05-15T23:20:23.286 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:20:23.286 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:20:23.286 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:20:23.286 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:20:23.286 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:20:23.287 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:20:23.287 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:20:23.287 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:20:23.287 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:20:23.287 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:20:23.287 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:20:23.476 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:23.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:20:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:20:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:23 smithi088 ceph-mon[44699]: osdmap e387: 12 total, 11 up, 10 in 2024-05-15T23:20:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:23 smithi088 ceph-mon[44699]: pgmap v1059: 17 pgs: 17 active+clean; 4.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-05-15T23:20:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:23 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3747183167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:23.583 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:23.622 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:23.660 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:23.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:23.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:23.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:23.808 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:24.010 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:24.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:24.014 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:24.052 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 25 16 72877 72861 182.1 121.375 0.00410835 0.00549042 2024-05-15T23:20:24.115 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:24.145 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:24.244 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:24.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:24.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:24.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:24.387 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:24.634 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:20:24.634 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:20:24.642 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:24.646 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:24.747 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:20:24.754 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:20:24.754 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:20:24.794 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:24.901 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:20:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:25.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:20:25.053 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 26 15 74610 74595 179.264 108.375 0.0121819 0.00557711 2024-05-15T23:20:25.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:20:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:25.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:25 smithi090 ceph-mon[44434]: pgmap v1060: 17 pgs: 17 active+clean; 4.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-05-15T23:20:25.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:25 smithi084 ceph-mon[39784]: pgmap v1060: 17 pgs: 17 active+clean; 4.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-05-15T23:20:25.409 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:20:25.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T23:20:25.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:25.412 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:20:25.514 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:20:25.562 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:25.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:25 smithi088 ceph-mon[44699]: pgmap v1060: 17 pgs: 17 active+clean; 4.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-05-15T23:20:25.591 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:20:25.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:25.598 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:20:25.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:25.730 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:25.749 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:25.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:25.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:25.839 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:20:25.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:20:25.885 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:26.053 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 27 16 76467 76451 176.921 116 0.00947805 0.00565081 2024-05-15T23:20:26.101 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:20:26.141 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:20:26.145 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:26 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2556651326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:20:26.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:26.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:20:26.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:20:26.293 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:26.360 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:20:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:26.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:20:26.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:26 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2556651326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:20:26.465 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:26.507 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:26 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2556651326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:20:26.510 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:26.542 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:26.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:26.651 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:26.692 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:26.755 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:20:27.053 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 28 16 78283 78267 174.656 113.5 0.00861193 0.00572402 2024-05-15T23:20:27.057 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:27.146 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:27.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:27.152 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:27.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:27.280 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:27.304 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:27.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:27.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:27 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:20:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:27 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2556651326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:20:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:27 smithi084 ceph-mon[39784]: osdmap e388: 12 total, 11 up, 10 in 2024-05-15T23:20:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:27 smithi084 ceph-mon[39784]: pgmap v1062: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-05-15T23:20:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:27 smithi084 ceph-mon[39784]: osdmap e389: 12 total, 11 up, 10 in 2024-05-15T23:20:27.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:20:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:27 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:20:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:27 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2556651326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:20:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:27 smithi090 ceph-mon[44434]: osdmap e388: 12 total, 11 up, 10 in 2024-05-15T23:20:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:27 smithi090 ceph-mon[44434]: pgmap v1062: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-05-15T23:20:27.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:27 smithi090 ceph-mon[44434]: osdmap e389: 12 total, 11 up, 10 in 2024-05-15T23:20:27.445 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:27.470 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:20:27.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:27.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:20:27.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:27 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:20:27.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:27 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2556651326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:20:27.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:27 smithi088 ceph-mon[44699]: osdmap e388: 12 total, 11 up, 10 in 2024-05-15T23:20:27.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:27 smithi088 ceph-mon[44699]: pgmap v1062: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-05-15T23:20:27.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:27 smithi088 ceph-mon[44699]: osdmap e389: 12 total, 11 up, 10 in 2024-05-15T23:20:27.577 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:20:27.628 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:27.654 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:20:27.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:27.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:20:27.761 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:27.792 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:20:27.792 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:20:27.813 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:27.839 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:27.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:27.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:27.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:27.972 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:28.053 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 29 16 80089 80073 172.526 112.875 0.0044496 0.00579445 2024-05-15T23:20:28.522 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:28.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:28.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:28.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T23:20:28.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:20:28.664 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:28.694 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:20:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:28.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:20:28.800 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:28.831 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:28.860 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:28.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:28.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:28.964 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:20:28.967 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:20:29.009 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:20:29.010 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:29.052 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:20:29.053 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 30 16 81852 81836 170.448 110.188 0.0135932 0.0058649 2024-05-15T23:20:29.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:29.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:20:29.156 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:20:29.157 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:20:29.176 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:20:29.195 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:29.199 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:20:29.225 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:20:29.259 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:20:29.280 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:20:29.284 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:20:29.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:29.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:20:29.312 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:29.315 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:20:29.341 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:20:29.365 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:20:29.386 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:29.389 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:20:29.412 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:20:29.413 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-05-15T23:20:29.413 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:20:29.413 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:20:29.413 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:20:29.413 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:20:29.413 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:20:29.413 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:20:29.413 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:20:29.413 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:20:29.413 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:20:29.413 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:20:29.415 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:29.472 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:29.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:29.531 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:29 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:20:29.533 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:29 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:20:29.533 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:29 smithi090 ceph-mon[44434]: osdmap e390: 12 total, 11 up, 10 in 2024-05-15T23:20:29.533 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:29 smithi090 ceph-mon[44434]: pgmap v1065: 17 pgs: 17 active+clean; 4.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-05-15T23:20:29.533 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:29 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/166434713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:29.533 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:29.578 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:29.619 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:29.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:29 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:20:29.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:29 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:20:29.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:29 smithi088 ceph-mon[44699]: osdmap e390: 12 total, 11 up, 10 in 2024-05-15T23:20:29.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:29 smithi088 ceph-mon[44699]: pgmap v1065: 17 pgs: 17 active+clean; 4.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-05-15T23:20:29.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:29 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/166434713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:29.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:29.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:29 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:20:29.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:29 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:20:29.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:29 smithi084 ceph-mon[39784]: osdmap e390: 12 total, 11 up, 10 in 2024-05-15T23:20:29.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:29 smithi084 ceph-mon[39784]: pgmap v1065: 17 pgs: 17 active+clean; 4.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-05-15T23:20:29.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:29 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/166434713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:29.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:29.874 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:29.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:29.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:29.979 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:30.019 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:30.053 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 31 16 83652 83636 168.579 112.5 0.00727381 0.00592982 2024-05-15T23:20:30.058 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:30.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:30.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:30.164 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:30.222 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:30.250 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:30.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:30.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:20:30.278 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:20:30.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:30.388 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:30.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:30.424 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:30.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:20:30.565 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:30.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.741 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:20:30.744 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.744 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.744 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.745 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.745 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.745 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.745 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.745 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.745 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.745 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:30.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:20:30.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:30.847 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:30.889 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:30.914 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:30.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:30.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:31.020 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:31.053 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 32 16 85349 85333 166.625 106.062 0.00407642 0.0059994 2024-05-15T23:20:31.060 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:31.077 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:20:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:20:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:20:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:20:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:166456 4.5 e316/283 4:ad18b346:::benchmark_data_smithi090_104448_object166455:head v 316'10530, mlcod=316'10528)", 2024-05-15T23:20:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:16:29.998883+0000", 2024-05-15T23:20:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 241.06793018900001, 2024-05-15T23:20:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022506923000000002, 2024-05-15T23:20:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:29.998883+0000", 2024-05-15T23:20:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:29.998883+0000", 2024-05-15T23:20:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:29.998885+0000", 2024-05-15T23:20:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5990000000000001e-06 2024-05-15T23:20:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:29.998936+0000", 2024-05-15T23:20:31.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1650000000000002e-05 2024-05-15T23:20:31.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:29.998946+0000", 2024-05-15T23:20:31.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0112000000000001e-05 2024-05-15T23:20:31.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:29.998954+0000", 2024-05-15T23:20:31.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9060000000000002e-06 2024-05-15T23:20:31.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:29.999009+0000", 2024-05-15T23:20:31.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4568999999999999e-05 2024-05-15T23:20:31.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:29.999025+0000", 2024-05-15T23:20:31.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5943e-05 2024-05-15T23:20:31.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:30.021332+0000", 2024-05-15T23:20:31.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022307467000000001 2024-05-15T23:20:31.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:30.021390+0000", 2024-05-15T23:20:31.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7676999999999997e-05 2024-05-15T23:20:31.082 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.082 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.082 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.082 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:185488 4.e e322/283 4:738cfb4b:::benchmark_data_smithi090_104448_object12250:head v 322'11746, mlcod=322'11743)", 2024-05-15T23:20:31.082 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:16:47.362155+0000", 2024-05-15T23:20:31.082 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 223.704658443, 2024-05-15T23:20:31.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023139817, 2024-05-15T23:20:31.082 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.083 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.083 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.083 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.362155+0000", 2024-05-15T23:20:31.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.362155+0000", 2024-05-15T23:20:31.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.362156+0000", 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.052e-06 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.362162+0000", 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3409999999999999e-06 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.362173+0000", 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0276e-05 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.362179+0000", 2024-05-15T23:20:31.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5289999999999997e-06 2024-05-15T23:20:31.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.366002+0000", 2024-05-15T23:20:31.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038227119999999998 2024-05-15T23:20:31.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.366024+0000", 2024-05-15T23:20:31.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2189999999999999e-05 2024-05-15T23:20:31.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.385248+0000", 2024-05-15T23:20:31.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019223593000000001 2024-05-15T23:20:31.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:47.385295+0000", 2024-05-15T23:20:31.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7123999999999998e-05 2024-05-15T23:20:31.086 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.086 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.086 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.087 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:197007 4.5 e325/283 4:aaffd228:::benchmark_data_smithi090_104448_object23769:head v 325'12477, mlcod=325'12473)", 2024-05-15T23:20:31.087 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:16:56.663881+0000", 2024-05-15T23:20:31.087 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 214.40293273099999, 2024-05-15T23:20:31.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022446674, 2024-05-15T23:20:31.087 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.087 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.087 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.087 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:56.663881+0000", 2024-05-15T23:20:31.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:56.663881+0000", 2024-05-15T23:20:31.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:56.663882+0000", 2024-05-15T23:20:31.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.403e-06 2024-05-15T23:20:31.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:56.663890+0000", 2024-05-15T23:20:31.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8790000000000003e-06 2024-05-15T23:20:31.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:56.663899+0000", 2024-05-15T23:20:31.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6500000000000002e-06 2024-05-15T23:20:31.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:56.663907+0000", 2024-05-15T23:20:31.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8189999999999998e-06 2024-05-15T23:20:31.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:56.664172+0000", 2024-05-15T23:20:31.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026425100000000001 2024-05-15T23:20:31.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:56.664189+0000", 2024-05-15T23:20:31.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.698e-05 2024-05-15T23:20:31.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:56.686269+0000", 2024-05-15T23:20:31.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022080853000000001 2024-05-15T23:20:31.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:56.686327+0000", 2024-05-15T23:20:31.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7839e-05 2024-05-15T23:20:31.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.091 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.092 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:200507 4.7 e327/283 4:e7b54af0:::benchmark_data_smithi090_104448_object27269:head v 327'12576, mlcod=326'12571)", 2024-05-15T23:20:31.092 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:16:59.495840+0000", 2024-05-15T23:20:31.092 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 211.57097366400001, 2024-05-15T23:20:31.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025642952, 2024-05-15T23:20:31.092 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.092 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.092 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.092 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:59.495840+0000", 2024-05-15T23:20:31.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:59.495840+0000", 2024-05-15T23:20:31.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:59.495840+0000", 2024-05-15T23:20:31.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6400000000000001e-07 2024-05-15T23:20:31.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:59.495846+0000", 2024-05-15T23:20:31.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7910000000000001e-06 2024-05-15T23:20:31.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:59.495852+0000", 2024-05-15T23:20:31.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0610000000000001e-06 2024-05-15T23:20:31.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:59.495857+0000", 2024-05-15T23:20:31.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0839999999999999e-06 2024-05-15T23:20:31.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:59.499522+0000", 2024-05-15T23:20:31.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036642689999999999 2024-05-15T23:20:31.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:59.499548+0000", 2024-05-15T23:20:31.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6619999999999999e-05 2024-05-15T23:20:31.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:59.521433+0000", 2024-05-15T23:20:31.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021884577999999998 2024-05-15T23:20:31.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:16:59.521483+0000", 2024-05-15T23:20:31.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9784999999999999e-05 2024-05-15T23:20:31.096 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.096 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.096 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.097 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:241709 4.e e335/283 4:79828809:::benchmark_data_smithi090_104448_object68471:head v 335'15354, mlcod=335'15350)", 2024-05-15T23:20:31.097 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:33.276406+0000", 2024-05-15T23:20:31.097 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.79040745500001, 2024-05-15T23:20:31.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020655844999999999, 2024-05-15T23:20:31.097 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.097 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.097 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.097 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:33.276406+0000", 2024-05-15T23:20:31.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:33.276406+0000", 2024-05-15T23:20:31.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:33.276407+0000", 2024-05-15T23:20:31.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5519999999999999e-06 2024-05-15T23:20:31.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:33.276417+0000", 2024-05-15T23:20:31.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0019e-05 2024-05-15T23:20:31.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:33.276428+0000", 2024-05-15T23:20:31.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0628999999999999e-05 2024-05-15T23:20:31.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:33.276435+0000", 2024-05-15T23:20:31.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6510000000000004e-06 2024-05-15T23:20:31.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:33.280204+0000", 2024-05-15T23:20:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037690900000000001 2024-05-15T23:20:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:33.280227+0000", 2024-05-15T23:20:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3068e-05 2024-05-15T23:20:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:33.297021+0000", 2024-05-15T23:20:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016794007999999999 2024-05-15T23:20:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:33.297062+0000", 2024-05-15T23:20:31.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0828000000000003e-05 2024-05-15T23:20:31.101 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.101 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.101 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.101 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:249447 4.5 e338/283 4:adc73874:::benchmark_data_smithi090_104448_object76209:head v 338'15773, mlcod=338'15770)", 2024-05-15T23:20:31.101 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:39.766521+0000", 2024-05-15T23:20:31.101 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 171.30029259700001, 2024-05-15T23:20:31.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026177769, 2024-05-15T23:20:31.101 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.101 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.102 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.102 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.766521+0000", 2024-05-15T23:20:31.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.766521+0000", 2024-05-15T23:20:31.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.766523+0000", 2024-05-15T23:20:31.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.052e-06 2024-05-15T23:20:31.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.766535+0000", 2024-05-15T23:20:31.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1885000000000001e-05 2024-05-15T23:20:31.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.766549+0000", 2024-05-15T23:20:31.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4333e-05 2024-05-15T23:20:31.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.766556+0000", 2024-05-15T23:20:31.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0110000000000003e-06 2024-05-15T23:20:31.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.766615+0000", 2024-05-15T23:20:31.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9378e-05 2024-05-15T23:20:31.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.766642+0000", 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6455e-05 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.792665+0000", 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026022983999999999 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.792698+0000", 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3670999999999997e-05 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.106 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:249449 4.5 e338/283 4:a5c36248:::benchmark_data_smithi090_104448_object76211:head v 338'15774, mlcod=338'15770)", 2024-05-15T23:20:31.106 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:39.766572+0000", 2024-05-15T23:20:31.106 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 171.300241382, 2024-05-15T23:20:31.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026161831999999999, 2024-05-15T23:20:31.106 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.106 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.106 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.106 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.766572+0000", 2024-05-15T23:20:31.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.766572+0000", 2024-05-15T23:20:31.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.766573+0000", 2024-05-15T23:20:31.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3399999999999999e-07 2024-05-15T23:20:31.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.766577+0000", 2024-05-15T23:20:31.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5709999999999999e-06 2024-05-15T23:20:31.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.768222+0000", 2024-05-15T23:20:31.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001644979 2024-05-15T23:20:31.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.768231+0000", 2024-05-15T23:20:31.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3249999999999997e-06 2024-05-15T23:20:31.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.771333+0000", 2024-05-15T23:20:31.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031012660000000001 2024-05-15T23:20:31.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.771355+0000", 2024-05-15T23:20:31.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2152000000000001e-05 2024-05-15T23:20:31.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.792703+0000", 2024-05-15T23:20:31.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021348615000000001 2024-05-15T23:20:31.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.792734+0000", 2024-05-15T23:20:31.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.029e-05 2024-05-15T23:20:31.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.110 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.110 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:249451 4.5 e338/283 4:a8790ee7:::benchmark_data_smithi090_104448_object76213:head v 338'15775, mlcod=338'15770)", 2024-05-15T23:20:31.111 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:39.770139+0000", 2024-05-15T23:20:31.111 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 171.29667389900001, 2024-05-15T23:20:31.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022622100999999999, 2024-05-15T23:20:31.111 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.111 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.111 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.111 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.770139+0000", 2024-05-15T23:20:31.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.770139+0000", 2024-05-15T23:20:31.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.770141+0000", 2024-05-15T23:20:31.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4500000000000001e-06 2024-05-15T23:20:31.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.770148+0000", 2024-05-15T23:20:31.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0849999999999999e-06 2024-05-15T23:20:31.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.770159+0000", 2024-05-15T23:20:31.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1245e-05 2024-05-15T23:20:31.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.770166+0000", 2024-05-15T23:20:31.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0040000000000002e-06 2024-05-15T23:20:31.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.775258+0000", 2024-05-15T23:20:31.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050916829999999996 2024-05-15T23:20:31.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.775279+0000", 2024-05-15T23:20:31.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1549999999999999e-05 2024-05-15T23:20:31.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.792740+0000", 2024-05-15T23:20:31.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017460749000000001 2024-05-15T23:20:31.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:39.792762+0000", 2024-05-15T23:20:31.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1335000000000002e-05 2024-05-15T23:20:31.114 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.114 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:261391 4.7 e341/283 4:e076fc85:::benchmark_data_smithi090_104448_object88153:head v 341'16451, mlcod=341'16445)", 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:17:49.583221+0000", 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 161.48359226700001, 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020640911000000001, 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:49.583221+0000", 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:49.583221+0000", 2024-05-15T23:20:31.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:49.583222+0000", 2024-05-15T23:20:31.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4300000000000001e-06 2024-05-15T23:20:31.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:49.583232+0000", 2024-05-15T23:20:31.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1900000000000001e-06 2024-05-15T23:20:31.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:49.583248+0000", 2024-05-15T23:20:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6041000000000001e-05 2024-05-15T23:20:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:49.583257+0000", 2024-05-15T23:20:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8589999999999994e-06 2024-05-15T23:20:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:49.590851+0000", 2024-05-15T23:20:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075946920000000001 2024-05-15T23:20:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:49.590875+0000", 2024-05-15T23:20:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.376e-05 2024-05-15T23:20:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:49.603813+0000", 2024-05-15T23:20:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012937717 2024-05-15T23:20:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:17:49.603862+0000", 2024-05-15T23:20:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9221999999999997e-05 2024-05-15T23:20:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25441.0:284114 4.e e346/283 4:777ad63d:::benchmark_data_smithi090_104448_object110876:head v 346'18057, mlcod=346'18054)", 2024-05-15T23:20:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:18:09.251650+0000", 2024-05-15T23:20:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 141.81516371399999, 2024-05-15T23:20:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023000434, 2024-05-15T23:20:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:09.251650+0000", 2024-05-15T23:20:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:09.251650+0000", 2024-05-15T23:20:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:09.251650+0000", 2024-05-15T23:20:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4799999999999997e-07 2024-05-15T23:20:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:09.251656+0000", 2024-05-15T23:20:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.699e-06 2024-05-15T23:20:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:09.251662+0000", 2024-05-15T23:20:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.784e-06 2024-05-15T23:20:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:09.251666+0000", 2024-05-15T23:20:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0709999999999996e-06 2024-05-15T23:20:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:09.256102+0000", 2024-05-15T23:20:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044363149999999997 2024-05-15T23:20:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:09.256125+0000", 2024-05-15T23:20:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2640999999999999e-05 2024-05-15T23:20:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:09.274609+0000", 2024-05-15T23:20:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018483920000000001 2024-05-15T23:20:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:18:09.274650+0000", 2024-05-15T23:20:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1155999999999997e-05 2024-05-15T23:20:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:55877 5.4 5:212bcbac:::benchmark_data_smithi090_116470_object55876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e384)", 2024-05-15T23:20:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:15.811589+0000", 2024-05-15T23:20:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.255224744, 2024-05-15T23:20:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.065675088000000006, 2024-05-15T23:20:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55877 2024-05-15T23:20:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.811589+0000", 2024-05-15T23:20:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.811589+0000", 2024-05-15T23:20:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.811590+0000", 2024-05-15T23:20:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.841e-06 2024-05-15T23:20:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.811610+0000", 2024-05-15T23:20:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9171e-05 2024-05-15T23:20:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.811618+0000", 2024-05-15T23:20:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7339999999999997e-06 2024-05-15T23:20:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.811624+0000", 2024-05-15T23:20:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2519999999999999e-06 2024-05-15T23:20:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.811676+0000", 2024-05-15T23:20:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2383999999999999e-05 2024-05-15T23:20:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.811803+0000", 2024-05-15T23:20:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127188 2024-05-15T23:20:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.811895+0000", 2024-05-15T23:20:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1756999999999996e-05 2024-05-15T23:20:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.815072+0000", 2024-05-15T23:20:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003177446 2024-05-15T23:20:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.877163+0000", 2024-05-15T23:20:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062090510000000002 2024-05-15T23:20:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.877222+0000", 2024-05-15T23:20:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9321999999999999e-05 2024-05-15T23:20:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.877264+0000", 2024-05-15T23:20:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1483000000000002e-05 2024-05-15T23:20:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:55880 5.4 5:2f30a8db:::benchmark_data_smithi090_116470_object55879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e384)", 2024-05-15T23:20:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:15.812203+0000", 2024-05-15T23:20:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.254610535999999, 2024-05-15T23:20:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.066590694000000006, 2024-05-15T23:20:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.130 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.130 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.130 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:31.130 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:31.130 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:31.130 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55880 2024-05-15T23:20:31.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.130 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.812203+0000", 2024-05-15T23:20:31.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.812203+0000", 2024-05-15T23:20:31.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.812204+0000", 2024-05-15T23:20:31.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3650000000000001e-06 2024-05-15T23:20:31.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.812225+0000", 2024-05-15T23:20:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1067999999999999e-05 2024-05-15T23:20:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.812236+0000", 2024-05-15T23:20:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0321e-05 2024-05-15T23:20:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.812242+0000", 2024-05-15T23:20:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.888e-06 2024-05-15T23:20:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.813292+0000", 2024-05-15T23:20:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010496800000000001 2024-05-15T23:20:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.813411+0000", 2024-05-15T23:20:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118483 2024-05-15T23:20:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.813537+0000", 2024-05-15T23:20:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012629699999999999 2024-05-15T23:20:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.815937+0000", 2024-05-15T23:20:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002399675 2024-05-15T23:20:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.878715+0000", 2024-05-15T23:20:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062778137999999997 2024-05-15T23:20:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.878755+0000", 2024-05-15T23:20:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0222000000000002e-05 2024-05-15T23:20:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.878793+0000", 2024-05-15T23:20:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8556999999999997e-05 2024-05-15T23:20:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:55891 5.4 5:234c5327:::benchmark_data_smithi090_116470_object55890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e384)", 2024-05-15T23:20:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:15.818117+0000", 2024-05-15T23:20:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.248696592, 2024-05-15T23:20:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.060861315999999999, 2024-05-15T23:20:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55891 2024-05-15T23:20:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.818117+0000", 2024-05-15T23:20:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.818117+0000", 2024-05-15T23:20:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.818118+0000", 2024-05-15T23:20:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.283e-06 2024-05-15T23:20:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.818136+0000", 2024-05-15T23:20:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8406999999999998e-05 2024-05-15T23:20:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.818145+0000", 2024-05-15T23:20:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9239999999999996e-06 2024-05-15T23:20:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.818153+0000", 2024-05-15T23:20:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2169999999999997e-06 2024-05-15T23:20:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.818192+0000", 2024-05-15T23:20:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9368e-05 2024-05-15T23:20:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.818291+0000", 2024-05-15T23:20:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8777999999999995e-05 2024-05-15T23:20:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.818377+0000", 2024-05-15T23:20:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6303000000000004e-05 2024-05-15T23:20:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.819739+0000", 2024-05-15T23:20:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013618370000000001 2024-05-15T23:20:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.878876+0000", 2024-05-15T23:20:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.059137568000000001 2024-05-15T23:20:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.878928+0000", 2024-05-15T23:20:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1854999999999997e-05 2024-05-15T23:20:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:15.878978+0000", 2024-05-15T23:20:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9775999999999999e-05 2024-05-15T23:20:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:56376 5.a 5:5121708b:::benchmark_data_smithi090_116470_object56375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e384)", 2024-05-15T23:20:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:16.099686+0000", 2024-05-15T23:20:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.967126835, 2024-05-15T23:20:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021800219999999999, 2024-05-15T23:20:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56376 2024-05-15T23:20:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.099686+0000", 2024-05-15T23:20:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.099686+0000", 2024-05-15T23:20:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.099687+0000", 2024-05-15T23:20:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0800000000000004e-07 2024-05-15T23:20:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.099709+0000", 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1569999999999998e-05 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.099715+0000", 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9900000000000002e-06 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.099720+0000", 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4149999999999998e-06 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.099769+0000", 2024-05-15T23:20:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9014999999999997e-05 2024-05-15T23:20:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.099869+0000", 2024-05-15T23:20:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9668999999999997e-05 2024-05-15T23:20:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 8", 2024-05-15T23:20:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.099979+0000", 2024-05-15T23:20:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011061900000000001 2024-05-15T23:20:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.106652+0000", 2024-05-15T23:20:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0066724210000000004 2024-05-15T23:20:31.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:31.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.121383+0000", 2024-05-15T23:20:31.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014731531000000001 2024-05-15T23:20:31.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.121441+0000", 2024-05-15T23:20:31.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8032999999999998e-05 2024-05-15T23:20:31.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.121487+0000", 2024-05-15T23:20:31.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5249e-05 2024-05-15T23:20:31.147 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.147 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.147 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.147 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:56377 5.a 5:5dc1dab5:::benchmark_data_smithi090_116470_object56376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e384)", 2024-05-15T23:20:31.147 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:16.099733+0000", 2024-05-15T23:20:31.147 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.967080727000001, 2024-05-15T23:20:31.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021895965, 2024-05-15T23:20:31.147 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.147 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.147 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.148 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:31.148 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:31.148 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:31.148 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56377 2024-05-15T23:20:31.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.148 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.099733+0000", 2024-05-15T23:20:31.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.099733+0000", 2024-05-15T23:20:31.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.099733+0000", 2024-05-15T23:20:31.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0200000000000001e-07 2024-05-15T23:20:31.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.099754+0000", 2024-05-15T23:20:31.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0738999999999999e-05 2024-05-15T23:20:31.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.099760+0000", 2024-05-15T23:20:31.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5999999999999997e-06 2024-05-15T23:20:31.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.099764+0000", 2024-05-15T23:20:31.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.69e-06 2024-05-15T23:20:31.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.104360+0000", 2024-05-15T23:20:31.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045957020000000001 2024-05-15T23:20:31.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.104496+0000", 2024-05-15T23:20:31.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136267 2024-05-15T23:20:31.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 8", 2024-05-15T23:20:31.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.104643+0000", 2024-05-15T23:20:31.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000146704 2024-05-15T23:20:31.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:31.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.106766+0000", 2024-05-15T23:20:31.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021231150000000001 2024-05-15T23:20:31.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:31.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.121534+0000", 2024-05-15T23:20:31.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014767984 2024-05-15T23:20:31.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.121590+0000", 2024-05-15T23:20:31.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6412999999999997e-05 2024-05-15T23:20:31.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:16.121629+0000", 2024-05-15T23:20:31.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8049000000000001e-05 2024-05-15T23:20:31.153 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.153 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.153 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.153 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:62334 5.4 5:2448de6f:::benchmark_data_smithi090_116470_object62333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-05-15T23:20:31.153 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:18.830328+0000", 2024-05-15T23:20:31.154 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 12.236485148, 2024-05-15T23:20:31.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038133013, 2024-05-15T23:20:31.154 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.154 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.154 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.154 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:31.154 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:31.154 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:31.154 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 62334 2024-05-15T23:20:31.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.154 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:18.830328+0000", 2024-05-15T23:20:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:18.830328+0000", 2024-05-15T23:20:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:18.830330+0000", 2024-05-15T23:20:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.708e-06 2024-05-15T23:20:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:18.830350+0000", 2024-05-15T23:20:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.003e-05 2024-05-15T23:20:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:18.830361+0000", 2024-05-15T23:20:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.062e-05 2024-05-15T23:20:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:18.830366+0000", 2024-05-15T23:20:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8329999999999999e-06 2024-05-15T23:20:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:18.830418+0000", 2024-05-15T23:20:31.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1700000000000003e-05 2024-05-15T23:20:31.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:18.830548+0000", 2024-05-15T23:20:31.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130295 2024-05-15T23:20:31.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:31.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:18.830682+0000", 2024-05-15T23:20:31.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133528 2024-05-15T23:20:31.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:31.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:18.832631+0000", 2024-05-15T23:20:31.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001948696 2024-05-15T23:20:31.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:31.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:18.868398+0000", 2024-05-15T23:20:31.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035767396 2024-05-15T23:20:31.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:18.868433+0000", 2024-05-15T23:20:31.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5336000000000002e-05 2024-05-15T23:20:31.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:18.868461+0000", 2024-05-15T23:20:31.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7871000000000001e-05 2024-05-15T23:20:31.159 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.159 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.159 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.159 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:67707 5.0 5:03a98c88:::benchmark_data_smithi090_116470_object67706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-05-15T23:20:31.160 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:21.388184+0000", 2024-05-15T23:20:31.160 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.6786296420000006, 2024-05-15T23:20:31.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029224805, 2024-05-15T23:20:31.160 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.160 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.160 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.160 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:31.160 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:31.160 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:31.160 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 67707 2024-05-15T23:20:31.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.160 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.388184+0000", 2024-05-15T23:20:31.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.388184+0000", 2024-05-15T23:20:31.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.388185+0000", 2024-05-15T23:20:31.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2219999999999999e-06 2024-05-15T23:20:31.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.388208+0000", 2024-05-15T23:20:31.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2926999999999999e-05 2024-05-15T23:20:31.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.388215+0000", 2024-05-15T23:20:31.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5850000000000002e-06 2024-05-15T23:20:31.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.388221+0000", 2024-05-15T23:20:31.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3750000000000002e-06 2024-05-15T23:20:31.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.388274+0000", 2024-05-15T23:20:31.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3140999999999998e-05 2024-05-15T23:20:31.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.388396+0000", 2024-05-15T23:20:31.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012225699999999999 2024-05-15T23:20:31.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:20:31.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.388533+0000", 2024-05-15T23:20:31.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013726400000000001 2024-05-15T23:20:31.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:31.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.390355+0000", 2024-05-15T23:20:31.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001821914 2024-05-15T23:20:31.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:31.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.417291+0000", 2024-05-15T23:20:31.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026935880999999998 2024-05-15T23:20:31.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.417344+0000", 2024-05-15T23:20:31.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2939999999999998e-05 2024-05-15T23:20:31.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.417408+0000", 2024-05-15T23:20:31.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4299000000000002e-05 2024-05-15T23:20:31.165 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.165 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.165 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.165 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:68477 5.4 5:266d1104:::benchmark_data_smithi090_116470_object68476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-05-15T23:20:31.166 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:21.775936+0000", 2024-05-15T23:20:31.166 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.290876913, 2024-05-15T23:20:31.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049925247999999998, 2024-05-15T23:20:31.166 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.166 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.166 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.166 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:31.166 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:31.166 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:31.166 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 68477 2024-05-15T23:20:31.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.166 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.775936+0000", 2024-05-15T23:20:31.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.775936+0000", 2024-05-15T23:20:31.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.775938+0000", 2024-05-15T23:20:31.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4130000000000001e-06 2024-05-15T23:20:31.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.775959+0000", 2024-05-15T23:20:31.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0825000000000001e-05 2024-05-15T23:20:31.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.775971+0000", 2024-05-15T23:20:31.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2479e-05 2024-05-15T23:20:31.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.775978+0000", 2024-05-15T23:20:31.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2559999999999996e-06 2024-05-15T23:20:31.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.776033+0000", 2024-05-15T23:20:31.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4922e-05 2024-05-15T23:20:31.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.776148+0000", 2024-05-15T23:20:31.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011459 2024-05-15T23:20:31.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:31.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.776281+0000", 2024-05-15T23:20:31.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013359799999999999 2024-05-15T23:20:31.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:31.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.778249+0000", 2024-05-15T23:20:31.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019676509999999999 2024-05-15T23:20:31.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:31.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.825805+0000", 2024-05-15T23:20:31.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.047556072999999997 2024-05-15T23:20:31.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.825837+0000", 2024-05-15T23:20:31.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1810000000000002e-05 2024-05-15T23:20:31.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:21.825862+0000", 2024-05-15T23:20:31.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4630999999999998e-05 2024-05-15T23:20:31.171 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.171 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.171 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.171 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:69872 5.0 5:094313bd:::benchmark_data_smithi090_116470_object69871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e387)", 2024-05-15T23:20:31.171 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:22.507810+0000", 2024-05-15T23:20:31.172 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.5590031100000008, 2024-05-15T23:20:31.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024153197000000001, 2024-05-15T23:20:31.172 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.172 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.172 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.172 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:31.172 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:31.172 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:31.172 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 69872 2024-05-15T23:20:31.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.172 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:22.507810+0000", 2024-05-15T23:20:31.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:22.507810+0000", 2024-05-15T23:20:31.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:22.507811+0000", 2024-05-15T23:20:31.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8899999999999999e-07 2024-05-15T23:20:31.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:22.507830+0000", 2024-05-15T23:20:31.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9255000000000001e-05 2024-05-15T23:20:31.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:22.507836+0000", 2024-05-15T23:20:31.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4659999999999998e-06 2024-05-15T23:20:31.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:22.507840+0000", 2024-05-15T23:20:31.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6260000000000003e-06 2024-05-15T23:20:31.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:22.519235+0000", 2024-05-15T23:20:31.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011395058 2024-05-15T23:20:31.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:22.519381+0000", 2024-05-15T23:20:31.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000146021 2024-05-15T23:20:31.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:20:31.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:22.519527+0000", 2024-05-15T23:20:31.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000145315 2024-05-15T23:20:31.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:31.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:22.522335+0000", 2024-05-15T23:20:31.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002808311 2024-05-15T23:20:31.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:31.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:22.531917+0000", 2024-05-15T23:20:31.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095823260000000004 2024-05-15T23:20:31.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:22.531942+0000", 2024-05-15T23:20:31.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4579999999999998e-05 2024-05-15T23:20:31.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:22.531963+0000", 2024-05-15T23:20:31.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1650000000000001e-05 2024-05-15T23:20:31.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.177 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.178 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:83678 5.4 5:279d14eb:::benchmark_data_smithi090_116470_object83677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e390)", 2024-05-15T23:20:31.178 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:30.066468+0000", 2024-05-15T23:20:31.178 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.0003457680000001, 2024-05-15T23:20:31.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021634613, 2024-05-15T23:20:31.178 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:31.178 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:31.178 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:31.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:31.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:31.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:31.178 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 83678 2024-05-15T23:20:31.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.179 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:31.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:31.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:30.066468+0000", 2024-05-15T23:20:31.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:31.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:30.066468+0000", 2024-05-15T23:20:31.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:31.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:31.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:30.066469+0000", 2024-05-15T23:20:31.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.11e-06 2024-05-15T23:20:31.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:31.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:30.066484+0000", 2024-05-15T23:20:31.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5753e-05 2024-05-15T23:20:31.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:31.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:30.066492+0000", 2024-05-15T23:20:31.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9070000000000007e-06 2024-05-15T23:20:31.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:31.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:30.066497+0000", 2024-05-15T23:20:31.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0320000000000003e-06 2024-05-15T23:20:31.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:31.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:30.068021+0000", 2024-05-15T23:20:31.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015234109999999999 2024-05-15T23:20:31.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:31.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:30.068153+0000", 2024-05-15T23:20:31.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013185900000000001 2024-05-15T23:20:31.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:31.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:30.068302+0000", 2024-05-15T23:20:31.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149338 2024-05-15T23:20:31.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:31.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:30.070739+0000", 2024-05-15T23:20:31.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024370099999999999 2024-05-15T23:20:31.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:31.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:30.088028+0000", 2024-05-15T23:20:31.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017288705000000001 2024-05-15T23:20:31.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:31.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:30.088062+0000", 2024-05-15T23:20:31.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4774000000000003e-05 2024-05-15T23:20:31.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:31.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:31.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:31.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:30.088102+0000", 2024-05-15T23:20:31.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9714e-05 2024-05-15T23:20:31.184 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.184 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.184 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.184 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:31.184 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:31.184 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:20:31.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:31.185 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:31.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:31.286 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:31.316 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:31.391 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:31.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:31.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:31.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:20:31.543 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:31 smithi084 ceph-mon[39784]: pgmap v1066: 17 pgs: 17 active+clean; 5.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-05-15T23:20:31.792 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:31 smithi090 ceph-mon[44434]: pgmap v1066: 17 pgs: 17 active+clean; 5.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-05-15T23:20:31.806 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:20:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:31.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:20:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:31 smithi088 ceph-mon[44699]: pgmap v1066: 17 pgs: 17 active+clean; 5.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-05-15T23:20:31.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T23:20:31.921 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:20:31.961 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:31.991 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:20:31.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:31.995 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:20:32.053 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 33 16 87069 87053 164.833 107.5 0.00549071 0.00606539 2024-05-15T23:20:32.096 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:32.133 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:32.159 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:32.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:32.163 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:20:32.264 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:32.278 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:20:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:32.339 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:32.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:32.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:32.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:20:32.492 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:32.551 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:20:32.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:32.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:20:32.578 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:32.656 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:32.716 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:32.750 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:32.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:32.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:32.857 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:20:32.915 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:32.946 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:20:32.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:32.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:20:33.010 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:33 smithi088 ceph-mon[44699]: osdmap e391: 12 total, 11 up, 10 in 2024-05-15T23:20:33.011 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:33 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/171657615' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:20:33.031 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:20:33.052 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:33.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:33 smithi090 ceph-mon[44434]: osdmap e391: 12 total, 11 up, 10 in 2024-05-15T23:20:33.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:33 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/171657615' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:20:33.054 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 34 16 88679 88663 162.945 100.625 0.00506705 0.00613605 2024-05-15T23:20:33.083 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:33.109 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:33.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:33 smithi084 ceph-mon[39784]: osdmap e391: 12 total, 11 up, 10 in 2024-05-15T23:20:33.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:33 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/171657615' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:20:33.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:33.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:33.215 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:33.243 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:34.054 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 35 16 90393 90377 161.35 107.125 0.0109245 0.00619631 2024-05-15T23:20:34.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:34 smithi084 ceph-mon[39784]: pgmap v1068: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-05-15T23:20:34.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:34 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:20:34.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:34 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/171657615' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:20:34.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:34 smithi084 ceph-mon[39784]: osdmap e392: 12 total, 11 up, 10 in 2024-05-15T23:20:34.291 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:34.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:34.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:34 smithi090 ceph-mon[44434]: pgmap v1068: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-05-15T23:20:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:34 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:20:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:34 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/171657615' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:20:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:34 smithi090 ceph-mon[44434]: osdmap e392: 12 total, 11 up, 10 in 2024-05-15T23:20:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:34 smithi088 ceph-mon[44699]: pgmap v1068: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-05-15T23:20:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:34 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:20:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:34 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/171657615' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:20:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:34 smithi088 ceph-mon[44699]: osdmap e392: 12 total, 11 up, 10 in 2024-05-15T23:20:34.396 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:34.426 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:34.454 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:34.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:34.562 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:20:34.578 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:20:34.579 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:20:34.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.579 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:90775 5.d 5:b3f85306:::benchmark_data_smithi090_116470_object90774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e392)", 2024-05-15T23:20:34.579 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:34.279052+0000", 2024-05-15T23:20:34.579 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.071089854999999993, 2024-05-15T23:20:34.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.071119909999999995, 2024-05-15T23:20:34.579 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:34.579 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:34.579 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:20:34.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:34.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:34.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:34.580 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90775 2024-05-15T23:20:34.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.580 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:34.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:34.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279052+0000", 2024-05-15T23:20:34.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:34.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:34.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279052+0000", 2024-05-15T23:20:34.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:34.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:34.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279053+0000", 2024-05-15T23:20:34.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4499999999999996e-07 2024-05-15T23:20:34.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:34.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279073+0000", 2024-05-15T23:20:34.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0598000000000002e-05 2024-05-15T23:20:34.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:34.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279080+0000", 2024-05-15T23:20:34.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0679999999999999e-06 2024-05-15T23:20:34.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:34.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279087+0000", 2024-05-15T23:20:34.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7309999999999996e-06 2024-05-15T23:20:34.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:34.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.282248+0000", 2024-05-15T23:20:34.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031613000000000001 2024-05-15T23:20:34.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:34.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.282376+0000", 2024-05-15T23:20:34.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799 2024-05-15T23:20:34.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:34.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.282511+0000", 2024-05-15T23:20:34.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134544 2024-05-15T23:20:34.583 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:34.583 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:34.583 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:34.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.583 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:90784 5.d 5:bc54d435:::benchmark_data_smithi090_116470_object90783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e392)", 2024-05-15T23:20:34.583 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:34.288478+0000", 2024-05-15T23:20:34.584 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.061663699000000002, 2024-05-15T23:20:34.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061752828000000003, 2024-05-15T23:20:34.584 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:34.584 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:34.584 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:20:34.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:34.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:34.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:34.584 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90784 2024-05-15T23:20:34.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.584 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:34.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:34.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288478+0000", 2024-05-15T23:20:34.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:34.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:34.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288478+0000", 2024-05-15T23:20:34.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:34.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:34.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288481+0000", 2024-05-15T23:20:34.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5459999999999998e-06 2024-05-15T23:20:34.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:34.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288504+0000", 2024-05-15T23:20:34.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3923000000000001e-05 2024-05-15T23:20:34.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:34.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288516+0000", 2024-05-15T23:20:34.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1364e-05 2024-05-15T23:20:34.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:34.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288524+0000", 2024-05-15T23:20:34.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0090000000000007e-06 2024-05-15T23:20:34.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:34.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288581+0000", 2024-05-15T23:20:34.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.749e-05 2024-05-15T23:20:34.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:34.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288709+0000", 2024-05-15T23:20:34.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012748100000000001 2024-05-15T23:20:34.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:34.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288841+0000", 2024-05-15T23:20:34.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131958 2024-05-15T23:20:34.588 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:34.588 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:34.588 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:34.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.588 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.25840.0:90775 5.d e392/374)", 2024-05-15T23:20:34.588 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:34.288603+0000", 2024-05-15T23:20:34.588 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.06153844, 2024-05-15T23:20:34.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061669333999999999, 2024-05-15T23:20:34.588 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:34.588 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:34.588 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:20:34.588 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:34.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:34.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288603+0000", 2024-05-15T23:20:34.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:34.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:34.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288603+0000", 2024-05-15T23:20:34.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:34.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:34.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288604+0000", 2024-05-15T23:20:34.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2240000000000001e-06 2024-05-15T23:20:34.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:34.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288609+0000", 2024-05-15T23:20:34.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5920000000000002e-06 2024-05-15T23:20:34.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:34.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288617+0000", 2024-05-15T23:20:34.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6839999999999996e-06 2024-05-15T23:20:34.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:34.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:34.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:34.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288623+0000", 2024-05-15T23:20:34.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3110000000000002e-06 2024-05-15T23:20:34.591 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:34.591 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:34.591 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:34.591 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:34.591 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-05-15T23:20:34.591 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 3 2024-05-15T23:20:34.591 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:20:34.603 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:34.631 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:20:34.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:34.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:20:34.734 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:34.788 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:34.829 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:34.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:34.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:34.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:20:34.983 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:35.055 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 36 16 91908 91892 159.492 94.6875 0.00876837 0.00626803 2024-05-15T23:20:35.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:35 smithi088 ceph-mon[44699]: pgmap v1070: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-05-15T23:20:35.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:35 smithi084 ceph-mon[39784]: pgmap v1070: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-05-15T23:20:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:35 smithi090 ceph-mon[44434]: pgmap v1070: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-05-15T23:20:35.340 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:20:35.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:35.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:20:35.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T23:20:35.448 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:35.480 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:35.485 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:20:35.500 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:35.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:35.509 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:20:35.531 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:20:35.556 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:20:35.582 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:20:35.604 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:35.605 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:20:35.632 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:20:35.654 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:20:35.662 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:35.676 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:20:35.691 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:35.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:35.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:35.701 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:20:35.722 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:20:35.722 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-05-15T23:20:35.722 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:20:35.722 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:20:35.723 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:20:35.723 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:20:35.723 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:20:35.723 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:20:35.723 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:20:35.723 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:20:35.723 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:20:35.723 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:20:35.723 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:20:35.772 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:20:35.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:20:35.834 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:36.025 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/634854560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:36.025 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:36.025 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:36.025 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:36.025 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:36.055 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 37 16 93528 93512 157.918 101.25 0.00851359 0.00633109 2024-05-15T23:20:36.089 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:20:36.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:36.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/634854560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:36.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:36.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:36.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:36.094 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:36.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:20:36.195 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:36.224 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:36.298 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:20:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/634854560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:36.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:36.573 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:36.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:36.592 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:20:36.593 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:20:36.634 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:36.676 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:36.703 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:36.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:36.906 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:36.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:37.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:20:37.038 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.038 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.038 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.038 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.038 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.038 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.038 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.038 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:37 smithi090 ceph-mon[44434]: pgmap v1071: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-05-15T23:20:37.038 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:37 smithi090 ceph-mon[44434]: osdmap e393: 12 total, 11 up, 10 in 2024-05-15T23:20:37.044 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:37.055 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 38 16 95148 95132 156.427 101.25 0.00875638 0.00639133 2024-05-15T23:20:37.090 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:20:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:37.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:20:37.194 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:20:37.242 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:37.272 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:20:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:37.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:20:37.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:37 smithi088 ceph-mon[44699]: pgmap v1071: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-05-15T23:20:37.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:37 smithi088 ceph-mon[44699]: osdmap e393: 12 total, 11 up, 10 in 2024-05-15T23:20:37.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:37.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:37 smithi084 ceph-mon[39784]: pgmap v1071: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-05-15T23:20:37.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:37 smithi084 ceph-mon[39784]: osdmap e393: 12 total, 11 up, 10 in 2024-05-15T23:20:37.378 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:37.424 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:37.458 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:37.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:37.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:37.611 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:37.716 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:37.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:37.720 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:37.776 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:20:37.776 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:20:37.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:37.850 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:38.055 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 39 16 96680 96664 154.871 95.75 0.00538458 0.00645466 2024-05-15T23:20:38.070 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:38.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:38.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:38.174 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:38.202 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:38.352 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:38.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:38.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:38.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:20:38.497 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:38.522 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:20:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:38.525 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:20:38.554 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-05-15T23:20:38.554 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2024-05-15T23:20:38.554 DEBUG:teuthology.orchestra.run.smithi084:osd.0> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.0.service 2024-05-15T23:20:38.556 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl start ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.0 2024-05-15T23:20:38.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T23:20:38.588 INFO:journalctl@ceph.osd.0.smithi084.stdout:-- Logs begin at Wed 2024-05-15 22:43:24 UTC. -- 2024-05-15T23:20:38.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:20:38.626 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:38.663 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:38.698 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:38.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:38.807 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:38.836 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:38.869 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:20:38 smithi084 systemd[1]: Starting Ceph osd.0 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:20:38.974 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:20:39.051 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:20:39.055 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:20:39.054896 min lat: 0.000997126 max lat: 0.211796 avg lat: 0.00651727 2024-05-15T23:20:39.055 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:20:39.055 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 40 16 98221 98205 153.407 96.3125 0.0147125 0.00651727 2024-05-15T23:20:39.089 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:39.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:39.194 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:39.224 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:39.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:39 smithi090 ceph-mon[44434]: pgmap v1073: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-05-15T23:20:39.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1971784467' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:20:39.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:39 smithi088 ceph-mon[44699]: pgmap v1073: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-05-15T23:20:39.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1971784467' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:20:39.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:39 smithi084 ceph-mon[39784]: pgmap v1073: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-05-15T23:20:39.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1971784467' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:20:40.055 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 41 16 99794 99778 152.063 98.3125 0.00621359 0.00657491 2024-05-15T23:20:40.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:40 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:20:40.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:40 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:20:40.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:40 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1971784467' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:20:40.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:40 smithi088 ceph-mon[44699]: osdmap e394: 12 total, 11 up, 10 in 2024-05-15T23:20:40.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:40 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:20:40.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:40 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:20:40.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:40 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1971784467' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:20:40.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:40 smithi084 ceph-mon[39784]: osdmap e394: 12 total, 11 up, 10 in 2024-05-15T23:20:40.370 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:20:39 smithi084 bash[188039]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-05-15T23:20:40.370 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:20:39 smithi084 bash[188039]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-15T23:20:40.370 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:20:39 smithi084 bash[188039]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-05-15T23:20:40.370 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:20:39 smithi084 bash[188039]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-05-15T23:20:40.370 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:20:39 smithi084 bash[188039]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-05-15T23:20:40.370 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:20:39 smithi084 bash[188039]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-05-15T23:20:40.370 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:20:39 smithi084 bash[188039]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-15T23:20:40.371 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:20:39 smithi084 bash[188039]: --> ceph-volume raw activate successful for osd ID: 0 2024-05-15T23:20:40.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:40 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:20:40.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:40 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:20:40.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:40 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1971784467' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:20:40.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:40 smithi090 ceph-mon[44434]: osdmap e394: 12 total, 11 up, 10 in 2024-05-15T23:20:40.874 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:40.876 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:40.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:40.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:40.981 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:41.026 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:41.055 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:41.055 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 42 15 101304 101289 150.691 94.4375 0.00855021 0.00663293 2024-05-15T23:20:41.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:41.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:41.159 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:41.180 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:41 smithi084 ceph-mon[39784]: pgmap v1075: 17 pgs: 17 active+clean; 6.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-05-15T23:20:41.206 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:41.249 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:41.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:41.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:41 smithi088 ceph-mon[44699]: pgmap v1075: 17 pgs: 17 active+clean; 6.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-05-15T23:20:41.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:20:41.386 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:41 smithi090 ceph-mon[44434]: pgmap v1075: 17 pgs: 17 active+clean; 6.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-05-15T23:20:41.389 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:41.847 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T23:20:42.056 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 43 16 102818 102802 149.386 94.5625 0.00965366 0.00669247 2024-05-15T23:20:42.120 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:20:41 smithi084 bash[188441]: 327fb3e3af2291245445803d80ecda2d9f790a91c50a2da81c60e5df7599bc95 2024-05-15T23:20:42.120 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:20:41 smithi084 systemd[1]: Started Ceph osd.0 for 3999fb7e-130e-11ef-bc99-c7b262605968. 2024-05-15T23:20:42.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:20:42.240 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:20:42.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:42.243 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:20:42.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:42.373 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:42.588 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:20:42 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-0[188475]: 2024-05-15T23:20:42.316+0000 7f66db10f7c0 -1 Falling back to public interface 2024-05-15T23:20:43.024 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:43.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:43.055 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 44 16 104316 104300 148.119 93.625 0.003947 0.00674981 2024-05-15T23:20:43.098 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:43.117 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:43.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:43.169 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:43.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:43 smithi090 ceph-mon[44434]: pgmap v1076: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-05-15T23:20:43.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:43 smithi088 ceph-mon[44699]: pgmap v1076: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-05-15T23:20:43.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:43 smithi084 ceph-mon[39784]: pgmap v1076: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-05-15T23:20:43.679 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:20:43.698 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:20:43.717 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:20:43.744 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:20:43.768 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:20:43.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:20:43.825 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:20:43.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:20:43.870 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:20:43.897 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:20:43.925 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:20:43.925 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-05-15T23:20:43.925 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:20:43.925 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:20:43.925 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:20:43.925 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:20:43.925 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:20:43.925 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:20:43.925 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:20:43.926 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:20:43.926 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:20:43.926 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:20:43.926 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:20:44.056 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 45 16 105668 105652 146.705 84.5 0.0108477 0.00681459 2024-05-15T23:20:44.152 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:44 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:20:44.152 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:44 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:20:44.152 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:44 smithi084 ceph-mon[39784]: osdmap e395: 12 total, 11 up, 10 in 2024-05-15T23:20:44.152 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2899428418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:44.152 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:44 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:44.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:44 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:20:44.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:44 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:20:44.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:44 smithi090 ceph-mon[44434]: osdmap e395: 12 total, 11 up, 10 in 2024-05-15T23:20:44.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2899428418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:44.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:44 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:44 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:20:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:44 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:20:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:44 smithi088 ceph-mon[44699]: osdmap e395: 12 total, 11 up, 10 in 2024-05-15T23:20:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2899428418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:44.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:44 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.056 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 46 16 107104 107088 145.467 89.75 0.0038588 0.00687273 2024-05-15T23:20:45.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:45 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:45 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:45 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:45 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:45 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:45 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:45 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:45 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:45 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:45 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:45 smithi090 ceph-mon[44434]: pgmap v1078: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-05-15T23:20:45.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:45 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:45 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:45 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:45 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:45 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:45 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:45 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:45 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:45 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:45 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:45 smithi088 ceph-mon[44699]: pgmap v1078: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-05-15T23:20:45.369 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:20:45 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-0[188475]: 2024-05-15T23:20:45.218+0000 7f66db10f7c0 -1 osd.0 355 log_to_monitors true 2024-05-15T23:20:45.431 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:45 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.431 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:45 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.431 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:45 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.431 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:45 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:45 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:45 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:45 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:45 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:45 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:45 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:45.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:45 smithi084 ceph-mon[39784]: pgmap v1078: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-05-15T23:20:46.056 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 47 16 108503 108487 144.232 87.4375 0.00791424 0.00693183 2024-05-15T23:20:46.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:46 smithi084 ceph-mon[39784]: from='osd.0 v1:172.21.15.84:6801/3077815271' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-15T23:20:46.150 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:20:46.151 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:20:46.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:46 smithi090 ceph-mon[44434]: from='osd.0 v1:172.21.15.84:6801/3077815271' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-15T23:20:46.310 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:20:46 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-0[188475]: 2024-05-15T23:20:46.056+0000 7f66c855e700 -1 osd.0 396 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T23:20:46.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:46 smithi088 ceph-mon[44699]: from='osd.0 v1:172.21.15.84:6801/3077815271' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-15T23:20:47.056 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 48 16 109771 109755 142.879 79.25 0.0194071 0.00699738 2024-05-15T23:20:47.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:47 smithi084 ceph-mon[39784]: from='osd.0 v1:172.21.15.84:6801/3077815271' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-15T23:20:47.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:47 smithi084 ceph-mon[39784]: osdmap e396: 12 total, 11 up, 10 in 2024-05-15T23:20:47.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:47 smithi084 ceph-mon[39784]: from='osd.0 v1:172.21.15.84:6801/3077815271' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T23:20:47.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:47 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T23:20:47.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:47 smithi084 ceph-mon[39784]: pgmap v1080: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-05-15T23:20:47.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:47 smithi090 ceph-mon[44434]: from='osd.0 v1:172.21.15.84:6801/3077815271' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-15T23:20:47.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:47 smithi090 ceph-mon[44434]: osdmap e396: 12 total, 11 up, 10 in 2024-05-15T23:20:47.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:47 smithi090 ceph-mon[44434]: from='osd.0 v1:172.21.15.84:6801/3077815271' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T23:20:47.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:47 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T23:20:47.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:47 smithi090 ceph-mon[44434]: pgmap v1080: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-05-15T23:20:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:47 smithi088 ceph-mon[44699]: from='osd.0 v1:172.21.15.84:6801/3077815271' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-15T23:20:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:47 smithi088 ceph-mon[44699]: osdmap e396: 12 total, 11 up, 10 in 2024-05-15T23:20:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:47 smithi088 ceph-mon[44699]: from='osd.0 v1:172.21.15.84:6801/3077815271' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-05-15T23:20:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:47 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T23:20:47.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:47 smithi088 ceph-mon[44699]: pgmap v1080: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-05-15T23:20:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:20:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:20:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-05-15T23:20:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:20:48.054 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:20:48.056 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 49 16 111223 111207 141.815 90.75 0.00484246 0.00704919 2024-05-15T23:20:48.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:20:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:48 smithi088 ceph-mon[44699]: osd.0 v1:172.21.15.84:6801/3077815271 boot 2024-05-15T23:20:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:48 smithi088 ceph-mon[44699]: osdmap e397: 12 total, 12 up, 10 in 2024-05-15T23:20:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:48 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T23:20:48.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3658948742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:20:48.346 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:48 smithi090 ceph-mon[44434]: osd.0 v1:172.21.15.84:6801/3077815271 boot 2024-05-15T23:20:48.346 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:48 smithi090 ceph-mon[44434]: osdmap e397: 12 total, 12 up, 10 in 2024-05-15T23:20:48.346 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:48 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T23:20:48.346 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3658948742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:20:48.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:48 smithi084 ceph-mon[39784]: osd.0 v1:172.21.15.84:6801/3077815271 boot 2024-05-15T23:20:48.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:48 smithi084 ceph-mon[39784]: osdmap e397: 12 total, 12 up, 10 in 2024-05-15T23:20:48.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:48 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T23:20:48.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3658948742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:20:48.399 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:48.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:48.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:48.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:48.520 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:20:48.521 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-15T23:20:48.521 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2024-05-15T23:20:48.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 5 1 2024-05-15T23:20:48.540 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:49.056 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 50 16 112554 112538 140.642 83.1875 0.00444614 0.00710788 2024-05-15T23:20:49.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:49.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:49 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:20:49.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:49 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:20:49.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:49 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:20:49.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:49 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3658948742' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:20:49.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:49 smithi084 ceph-mon[39784]: osdmap e398: 12 total, 12 up, 10 in 2024-05-15T23:20:49.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:49 smithi084 ceph-mon[39784]: pgmap v1083: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-05-15T23:20:49.191 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:49.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:49.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:49.215 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:49.296 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:49.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:49.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:49 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:20:49.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:49 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:20:49.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:49 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:20:49.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:49 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3658948742' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:20:49.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:49 smithi088 ceph-mon[44699]: osdmap e398: 12 total, 12 up, 10 in 2024-05-15T23:20:49.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:49 smithi088 ceph-mon[44699]: pgmap v1083: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-05-15T23:20:49.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:49.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:49 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:20:49.338 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:49 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:20:49.339 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:49 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:20:49.339 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:49 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3658948742' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:20:49.339 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:49 smithi090 ceph-mon[44434]: osdmap e398: 12 total, 12 up, 10 in 2024-05-15T23:20:49.339 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:49 smithi090 ceph-mon[44434]: pgmap v1083: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-05-15T23:20:49.346 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:49.893 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:49.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:49.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:49.999 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:20:50.019 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:50.037 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:50.056 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 51 16 113838 113822 139.458 80.25 0.00790677 0.00716846 2024-05-15T23:20:50.062 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:20:50.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:50.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:20:50.167 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:50.216 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:50.303 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:50.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:50.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:50.355 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:20:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:20:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:20:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:20:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:152314 3.e 3:7ab6d39c:::benchmark_data_smithi090_90974_object152313:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e242)", 2024-05-15T23:20:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:11.011469+0000", 2024-05-15T23:20:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 519.33648830899995, 2024-05-15T23:20:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11541359299999999, 2024-05-15T23:20:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:20:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:20:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 152314 2024-05-15T23:20:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011469+0000", 2024-05-15T23:20:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011469+0000", 2024-05-15T23:20:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011473+0000", 2024-05-15T23:20:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5880000000000002e-06 2024-05-15T23:20:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011500+0000", 2024-05-15T23:20:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6791000000000001e-05 2024-05-15T23:20:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011514+0000", 2024-05-15T23:20:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4518999999999999e-05 2024-05-15T23:20:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.011524+0000", 2024-05-15T23:20:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0033e-05 2024-05-15T23:20:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017018+0000", 2024-05-15T23:20:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00549396 2024-05-15T23:20:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017157+0000", 2024-05-15T23:20:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139202 2024-05-15T23:20:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:20:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.017343+0000", 2024-05-15T23:20:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018567099999999999 2024-05-15T23:20:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.026368+0000", 2024-05-15T23:20:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090247439999999995 2024-05-15T23:20:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.126798+0000", 2024-05-15T23:20:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10043023299999999 2024-05-15T23:20:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.126830+0000", 2024-05-15T23:20:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1899000000000001e-05 2024-05-15T23:20:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:11.126883+0000", 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2952999999999999e-05 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:154414 3.e 3:7f410bb8:::benchmark_data_smithi090_90974_object154413:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:13.323080+0000", 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 517.02487734199997, 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.116641645, 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:20:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:20:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 154414 2024-05-15T23:20:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323080+0000", 2024-05-15T23:20:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323080+0000", 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323082+0000", 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5409999999999999e-06 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323102+0000", 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0557999999999999e-05 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323111+0000", 2024-05-15T23:20:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9320000000000002e-06 2024-05-15T23:20:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.323118+0000", 2024-05-15T23:20:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7209999999999997e-06 2024-05-15T23:20:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.343054+0000", 2024-05-15T23:20:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019935797000000002 2024-05-15T23:20:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.343173+0000", 2024-05-15T23:20:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118989 2024-05-15T23:20:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:20:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.343298+0000", 2024-05-15T23:20:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012522999999999999 2024-05-15T23:20:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.351419+0000", 2024-05-15T23:20:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0081205270000000006 2024-05-15T23:20:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.439647+0000", 2024-05-15T23:20:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.088228506999999998 2024-05-15T23:20:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.439683+0000", 2024-05-15T23:20:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5723999999999999e-05 2024-05-15T23:20:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:13.439722+0000", 2024-05-15T23:20:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9119000000000003e-05 2024-05-15T23:20:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158411 3.e 3:721f625f:::benchmark_data_smithi090_90974_object158410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:20:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.688475+0000", 2024-05-15T23:20:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 512.65948220500002, 2024-05-15T23:20:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11329204800000001, 2024-05-15T23:20:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:20:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:20:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158411 2024-05-15T23:20:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688475+0000", 2024-05-15T23:20:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688475+0000", 2024-05-15T23:20:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688478+0000", 2024-05-15T23:20:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6199999999999999e-06 2024-05-15T23:20:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688507+0000", 2024-05-15T23:20:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8671e-05 2024-05-15T23:20:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688524+0000", 2024-05-15T23:20:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6837000000000002e-05 2024-05-15T23:20:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.688534+0000", 2024-05-15T23:20:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0513000000000001e-05 2024-05-15T23:20:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701293+0000", 2024-05-15T23:20:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012759111 2024-05-15T23:20:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701443+0000", 2024-05-15T23:20:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015017200000000001 2024-05-15T23:20:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:20:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701612+0000", 2024-05-15T23:20:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016817399999999999 2024-05-15T23:20:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.801638+0000", 2024-05-15T23:20:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100025981 2024-05-15T23:20:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.801691+0000", 2024-05-15T23:20:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3124999999999997e-05 2024-05-15T23:20:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.801724+0000", 2024-05-15T23:20:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3039e-05 2024-05-15T23:20:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.801768+0000", 2024-05-15T23:20:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3804999999999998e-05 2024-05-15T23:20:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158423 3.e 3:7aa2cb14:::benchmark_data_smithi090_90974_object158422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:20:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.701674+0000", 2024-05-15T23:20:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 512.64628376899998, 2024-05-15T23:20:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.115640091, 2024-05-15T23:20:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:20:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:20:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158423 2024-05-15T23:20:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701674+0000", 2024-05-15T23:20:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701674+0000", 2024-05-15T23:20:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701675+0000", 2024-05-15T23:20:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.424e-06 2024-05-15T23:20:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701694+0000", 2024-05-15T23:20:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8777e-05 2024-05-15T23:20:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701703+0000", 2024-05-15T23:20:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1090000000000004e-06 2024-05-15T23:20:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.701710+0000", 2024-05-15T23:20:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0650000000000001e-06 2024-05-15T23:20:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.803831+0000", 2024-05-15T23:20:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102121131 2024-05-15T23:20:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.803962+0000", 2024-05-15T23:20:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131042 2024-05-15T23:20:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:20:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.804116+0000", 2024-05-15T23:20:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015323299999999999 2024-05-15T23:20:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.817218+0000", 2024-05-15T23:20:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013102213 2024-05-15T23:20:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.817227+0000", 2024-05-15T23:20:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2949999999999999e-06 2024-05-15T23:20:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.817263+0000", 2024-05-15T23:20:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6003000000000001e-05 2024-05-15T23:20:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.817314+0000", 2024-05-15T23:20:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0798999999999999e-05 2024-05-15T23:20:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158426 3.e 3:7524570c:::benchmark_data_smithi090_90974_object158425:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:20:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.704015+0000", 2024-05-15T23:20:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 512.64394231100005, 2024-05-15T23:20:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.118608484, 2024-05-15T23:20:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:20:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:20:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158426 2024-05-15T23:20:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704015+0000", 2024-05-15T23:20:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704015+0000", 2024-05-15T23:20:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704017+0000", 2024-05-15T23:20:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7910000000000001e-06 2024-05-15T23:20:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704039+0000", 2024-05-15T23:20:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2243e-05 2024-05-15T23:20:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704053+0000", 2024-05-15T23:20:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3169e-05 2024-05-15T23:20:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.704060+0000", 2024-05-15T23:20:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6799999999999993e-06 2024-05-15T23:20:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.808093+0000", 2024-05-15T23:20:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10403225000000001 2024-05-15T23:20:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.808225+0000", 2024-05-15T23:20:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013245100000000001 2024-05-15T23:20:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:20:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.808375+0000", 2024-05-15T23:20:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014989300000000001 2024-05-15T23:20:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.817357+0000", 2024-05-15T23:20:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0089821039999999994 2024-05-15T23:20:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.822508+0000", 2024-05-15T23:20:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005150951 2024-05-15T23:20:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.822560+0000", 2024-05-15T23:20:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2488999999999998e-05 2024-05-15T23:20:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.822624+0000", 2024-05-15T23:20:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3462999999999999e-05 2024-05-15T23:20:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:158429 3.e 3:740800f6:::benchmark_data_smithi090_90974_object158428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-05-15T23:20:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:17.708210+0000", 2024-05-15T23:20:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 512.63974794499995, 2024-05-15T23:20:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11887426500000001, 2024-05-15T23:20:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:20:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:20:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158429 2024-05-15T23:20:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708210+0000", 2024-05-15T23:20:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708210+0000", 2024-05-15T23:20:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708212+0000", 2024-05-15T23:20:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4430000000000002e-06 2024-05-15T23:20:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708234+0000", 2024-05-15T23:20:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2195e-05 2024-05-15T23:20:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708245+0000", 2024-05-15T23:20:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1012999999999999e-05 2024-05-15T23:20:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.708253+0000", 2024-05-15T23:20:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8879999999999997e-06 2024-05-15T23:20:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.813214+0000", 2024-05-15T23:20:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104960421 2024-05-15T23:20:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.813362+0000", 2024-05-15T23:20:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014787399999999999 2024-05-15T23:20:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:20:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.813490+0000", 2024-05-15T23:20:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012885199999999999 2024-05-15T23:20:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.822633+0000", 2024-05-15T23:20:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091427599999999998 2024-05-15T23:20:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.826999+0000", 2024-05-15T23:20:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00436572 2024-05-15T23:20:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.827035+0000", 2024-05-15T23:20:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6282000000000003e-05 2024-05-15T23:20:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:17.827084+0000", 2024-05-15T23:20:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8816999999999997e-05 2024-05-15T23:20:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:159320 3.e 3:722627b8:::benchmark_data_smithi090_90974_object159319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e244)", 2024-05-15T23:20:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:18.781348+0000", 2024-05-15T23:20:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 511.56660944399999, 2024-05-15T23:20:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13170827900000001, 2024-05-15T23:20:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:20:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:20:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159320 2024-05-15T23:20:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781348+0000", 2024-05-15T23:20:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781348+0000", 2024-05-15T23:20:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781351+0000", 2024-05-15T23:20:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3410000000000001e-06 2024-05-15T23:20:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781377+0000", 2024-05-15T23:20:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6271e-05 2024-05-15T23:20:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781388+0000", 2024-05-15T23:20:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1141999999999999e-05 2024-05-15T23:20:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.781396+0000", 2024-05-15T23:20:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2679999999999994e-06 2024-05-15T23:20:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.797611+0000", 2024-05-15T23:20:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016214605 2024-05-15T23:20:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.797693+0000", 2024-05-15T23:20:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2564000000000004e-05 2024-05-15T23:20:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:20:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.797790+0000", 2024-05-15T23:20:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6477000000000005e-05 2024-05-15T23:20:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.895923+0000", 2024-05-15T23:20:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.098133348999999995 2024-05-15T23:20:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.912951+0000", 2024-05-15T23:20:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017028192000000001 2024-05-15T23:20:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.912989+0000", 2024-05-15T23:20:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7867000000000002e-05 2024-05-15T23:20:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.913057+0000", 2024-05-15T23:20:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7203e-05 2024-05-15T23:20:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25135.0:159325 3.e 3:77a43cc1:::benchmark_data_smithi090_90974_object159324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e244)", 2024-05-15T23:20:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:12:18.785096+0000", 2024-05-15T23:20:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 511.56286181899998, 2024-05-15T23:20:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12806985500000001, 2024-05-15T23:20:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25135", 2024-05-15T23:20:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:20:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159325 2024-05-15T23:20:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785096+0000", 2024-05-15T23:20:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785096+0000", 2024-05-15T23:20:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785098+0000", 2024-05-15T23:20:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.571e-06 2024-05-15T23:20:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785122+0000", 2024-05-15T23:20:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3864999999999999e-05 2024-05-15T23:20:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785135+0000", 2024-05-15T23:20:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3061000000000001e-05 2024-05-15T23:20:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.785144+0000", 2024-05-15T23:20:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6589999999999996e-06 2024-05-15T23:20:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.801926+0000", 2024-05-15T23:20:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016782254 2024-05-15T23:20:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.802075+0000", 2024-05-15T23:20:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014822300000000001 2024-05-15T23:20:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:20:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.802255+0000", 2024-05-15T23:20:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018082999999999999 2024-05-15T23:20:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.895946+0000", 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.093690839999999997 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.913068+0000", 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01712139 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.913114+0000", 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6802999999999999e-05 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:12:18.913166+0000", 2024-05-15T23:20:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1359000000000001e-05 2024-05-15T23:20:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:90775 5.d 5:b3f85306:::benchmark_data_smithi090_116470_object90774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e392)", 2024-05-15T23:20:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:34.279052+0000", 2024-05-15T23:20:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.068905889, 2024-05-15T23:20:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13313823499999999, 2024-05-15T23:20:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90775 2024-05-15T23:20:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279052+0000", 2024-05-15T23:20:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279052+0000", 2024-05-15T23:20:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279053+0000", 2024-05-15T23:20:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4499999999999996e-07 2024-05-15T23:20:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279073+0000", 2024-05-15T23:20:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0598000000000002e-05 2024-05-15T23:20:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279080+0000", 2024-05-15T23:20:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0679999999999999e-06 2024-05-15T23:20:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279087+0000", 2024-05-15T23:20:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7309999999999996e-06 2024-05-15T23:20:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.282248+0000", 2024-05-15T23:20:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031613000000000001 2024-05-15T23:20:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.282376+0000", 2024-05-15T23:20:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799 2024-05-15T23:20:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.282511+0000", 2024-05-15T23:20:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134544 2024-05-15T23:20:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412011+0000", 2024-05-15T23:20:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12950052200000001 2024-05-15T23:20:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412093+0000", 2024-05-15T23:20:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1329999999999996e-05 2024-05-15T23:20:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412141+0000", 2024-05-15T23:20:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8327e-05 2024-05-15T23:20:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412190+0000", 2024-05-15T23:20:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8980000000000002e-05 2024-05-15T23:20:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:90784 5.d 5:bc54d435:::benchmark_data_smithi090_116470_object90783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e392)", 2024-05-15T23:20:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:34.288478+0000", 2024-05-15T23:20:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.059479733, 2024-05-15T23:20:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12653699900000001, 2024-05-15T23:20:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90784 2024-05-15T23:20:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288478+0000", 2024-05-15T23:20:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288478+0000", 2024-05-15T23:20:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288481+0000", 2024-05-15T23:20:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5459999999999998e-06 2024-05-15T23:20:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288504+0000", 2024-05-15T23:20:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3923000000000001e-05 2024-05-15T23:20:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288516+0000", 2024-05-15T23:20:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1364e-05 2024-05-15T23:20:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288524+0000", 2024-05-15T23:20:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0090000000000007e-06 2024-05-15T23:20:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288581+0000", 2024-05-15T23:20:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.749e-05 2024-05-15T23:20:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288709+0000", 2024-05-15T23:20:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012748100000000001 2024-05-15T23:20:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288841+0000", 2024-05-15T23:20:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131958 2024-05-15T23:20:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412079+0000", 2024-05-15T23:20:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.123238218 2024-05-15T23:20:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.414956+0000", 2024-05-15T23:20:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028775430000000002 2024-05-15T23:20:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.414987+0000", 2024-05-15T23:20:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0614999999999997e-05 2024-05-15T23:20:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.415015+0000", 2024-05-15T23:20:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7852e-05 2024-05-15T23:20:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.25840.0:90775 5.d e392/374)", 2024-05-15T23:20:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:34.288603+0000", 2024-05-15T23:20:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.059354473999999, 2024-05-15T23:20:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12341743199999999, 2024-05-15T23:20:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:20:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288603+0000", 2024-05-15T23:20:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288603+0000", 2024-05-15T23:20:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288604+0000", 2024-05-15T23:20:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2240000000000001e-06 2024-05-15T23:20:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288609+0000", 2024-05-15T23:20:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5920000000000002e-06 2024-05-15T23:20:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288617+0000", 2024-05-15T23:20:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6839999999999996e-06 2024-05-15T23:20:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288623+0000", 2024-05-15T23:20:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3110000000000002e-06 2024-05-15T23:20:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.411996+0000", 2024-05-15T23:20:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12337332199999999 2024-05-15T23:20:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412005+0000", 2024-05-15T23:20:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3389999999999993e-06 2024-05-15T23:20:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412021+0000", 2024-05-15T23:20:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.596e-05 2024-05-15T23:20:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.25840.0:90784 5.d e392/374 ondisk, result = 0)", 2024-05-15T23:20:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:34.291602+0000", 2024-05-15T23:20:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.05635543, 2024-05-15T23:20:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12060193600000001, 2024-05-15T23:20:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:20:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.291602+0000", 2024-05-15T23:20:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.291602+0000", 2024-05-15T23:20:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.291604+0000", 2024-05-15T23:20:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.731e-06 2024-05-15T23:20:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.291612+0000", 2024-05-15T23:20:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-05-15T23:20:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.408963+0000", 2024-05-15T23:20:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11735071 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.408978+0000", 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4948e-05 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412063+0000", 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003084669 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412070+0000", 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5449999999999998e-06 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412204+0000", 2024-05-15T23:20:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013396799999999999 2024-05-15T23:20:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:109154 5.8 5:121e6b3a:::benchmark_data_smithi090_116470_object109153:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-05-15T23:20:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:46.486383+0000", 2024-05-15T23:20:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.861574579, 2024-05-15T23:20:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.127318981, 2024-05-15T23:20:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109154 2024-05-15T23:20:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.486383+0000", 2024-05-15T23:20:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.486383+0000", 2024-05-15T23:20:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.486385+0000", 2024-05-15T23:20:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6899999999999999e-06 2024-05-15T23:20:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.486408+0000", 2024-05-15T23:20:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2920000000000001e-05 2024-05-15T23:20:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.486416+0000", 2024-05-15T23:20:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8979999999999996e-06 2024-05-15T23:20:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.486422+0000", 2024-05-15T23:20:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9429999999999997e-06 2024-05-15T23:20:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.489130+0000", 2024-05-15T23:20:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027081520000000001 2024-05-15T23:20:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.489252+0000", 2024-05-15T23:20:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012243200000000001 2024-05-15T23:20:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:20:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.489332+0000", 2024-05-15T23:20:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0045999999999999e-05 2024-05-15T23:20:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492263+0000", 2024-05-15T23:20:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029307510000000001 2024-05-15T23:20:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.613598+0000", 2024-05-15T23:20:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.121335048 2024-05-15T23:20:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.613652+0000", 2024-05-15T23:20:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4079000000000002e-05 2024-05-15T23:20:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.613702+0000", 2024-05-15T23:20:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0022000000000003e-05 2024-05-15T23:20:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:109165 5.8 5:197248c9:::benchmark_data_smithi090_116470_object109164:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-05-15T23:20:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:46.492898+0000", 2024-05-15T23:20:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.8550592130000001, 2024-05-15T23:20:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12576668099999999, 2024-05-15T23:20:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109165 2024-05-15T23:20:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492898+0000", 2024-05-15T23:20:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492898+0000", 2024-05-15T23:20:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492900+0000", 2024-05-15T23:20:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9180000000000001e-06 2024-05-15T23:20:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492922+0000", 2024-05-15T23:20:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.122e-05 2024-05-15T23:20:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492931+0000", 2024-05-15T23:20:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5910000000000002e-06 2024-05-15T23:20:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492938+0000", 2024-05-15T23:20:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9990000000000002e-06 2024-05-15T23:20:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492979+0000", 2024-05-15T23:20:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0571999999999997e-05 2024-05-15T23:20:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.609709+0000", 2024-05-15T23:20:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11673001 2024-05-15T23:20:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:20:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.609876+0000", 2024-05-15T23:20:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016714599999999999 2024-05-15T23:20:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.617471+0000", 2024-05-15T23:20:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075946499999999997 2024-05-15T23:20:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.618564+0000", 2024-05-15T23:20:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010938440000000001 2024-05-15T23:20:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.618622+0000", 2024-05-15T23:20:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7992999999999999e-05 2024-05-15T23:20:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.618665+0000", 2024-05-15T23:20:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2738000000000003e-05 2024-05-15T23:20:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.25840.0:109154 5.8 e396/374 ondisk, result = 0)", 2024-05-15T23:20:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:46.493242+0000", 2024-05-15T23:20:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.8547161069999998, 2024-05-15T23:20:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120466937, 2024-05-15T23:20:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:20:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.493242+0000", 2024-05-15T23:20:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.493242+0000", 2024-05-15T23:20:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.493242+0000", 2024-05-15T23:20:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9699999999999995e-07 2024-05-15T23:20:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.493246+0000", 2024-05-15T23:20:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0180000000000003e-06 2024-05-15T23:20:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.609486+0000", 2024-05-15T23:20:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11623934299999999 2024-05-15T23:20:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.609511+0000", 2024-05-15T23:20:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5476000000000001e-05 2024-05-15T23:20:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.613584+0000", 2024-05-15T23:20:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040730100000000002 2024-05-15T23:20:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.613592+0000", 2024-05-15T23:20:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1270000000000003e-06 2024-05-15T23:20:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.613709+0000", 2024-05-15T23:20:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116166 2024-05-15T23:20:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:109219 5.d 5:b88eda13:::benchmark_data_smithi090_116470_object109218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-05-15T23:20:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:46.637687+0000", 2024-05-15T23:20:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.7102711880000001, 2024-05-15T23:20:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11331556499999999, 2024-05-15T23:20:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109219 2024-05-15T23:20:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637687+0000", 2024-05-15T23:20:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637687+0000", 2024-05-15T23:20:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637688+0000", 2024-05-15T23:20:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.855e-06 2024-05-15T23:20:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637712+0000", 2024-05-15T23:20:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3506000000000002e-05 2024-05-15T23:20:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637722+0000", 2024-05-15T23:20:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0464e-05 2024-05-15T23:20:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637731+0000", 2024-05-15T23:20:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3310000000000004e-06 2024-05-15T23:20:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637787+0000", 2024-05-15T23:20:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5942000000000001e-05 2024-05-15T23:20:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637900+0000", 2024-05-15T23:20:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113751 2024-05-15T23:20:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.638026+0000", 2024-05-15T23:20:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125532 2024-05-15T23:20:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.641729+0000", 2024-05-15T23:20:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037033399999999998 2024-05-15T23:20:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.750899+0000", 2024-05-15T23:20:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.109169718 2024-05-15T23:20:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.750940+0000", 2024-05-15T23:20:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1186999999999997e-05 2024-05-15T23:20:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.751002+0000", 2024-05-15T23:20:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1939000000000004e-05 2024-05-15T23:20:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:109223 5.d 5:b95a4800:::benchmark_data_smithi090_116470_object109222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-05-15T23:20:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:46.642357+0000", 2024-05-15T23:20:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.705601148, 2024-05-15T23:20:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11458536699999999, 2024-05-15T23:20:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109223 2024-05-15T23:20:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642357+0000", 2024-05-15T23:20:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642357+0000", 2024-05-15T23:20:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642359+0000", 2024-05-15T23:20:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1950000000000002e-06 2024-05-15T23:20:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642382+0000", 2024-05-15T23:20:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3337999999999999e-05 2024-05-15T23:20:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642394+0000", 2024-05-15T23:20:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2024-05-15T23:20:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642402+0000", 2024-05-15T23:20:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9659999999999996e-06 2024-05-15T23:20:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642454+0000", 2024-05-15T23:20:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2698999999999998e-05 2024-05-15T23:20:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642540+0000", 2024-05-15T23:20:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5768999999999998e-05 2024-05-15T23:20:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642625+0000", 2024-05-15T23:20:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4457999999999997e-05 2024-05-15T23:20:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.751046+0000", 2024-05-15T23:20:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.108421383 2024-05-15T23:20:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.756850+0000", 2024-05-15T23:20:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0058037239999999997 2024-05-15T23:20:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.756898+0000", 2024-05-15T23:20:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8031999999999997e-05 2024-05-15T23:20:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.756942+0000", 2024-05-15T23:20:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4057999999999997e-05 2024-05-15T23:20:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:109227 5.d 5:bc936f0d:::benchmark_data_smithi090_116470_object109226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-05-15T23:20:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:46.644710+0000", 2024-05-15T23:20:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.7032474689999999, 2024-05-15T23:20:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11603258299999999, 2024-05-15T23:20:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109227 2024-05-15T23:20:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.644710+0000", 2024-05-15T23:20:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.644710+0000", 2024-05-15T23:20:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.644711+0000", 2024-05-15T23:20:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.054e-06 2024-05-15T23:20:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.644772+0000", 2024-05-15T23:20:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129000000000004e-05 2024-05-15T23:20:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.644781+0000", 2024-05-15T23:20:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5259999999999993e-06 2024-05-15T23:20:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.644787+0000", 2024-05-15T23:20:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5080000000000002e-06 2024-05-15T23:20:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.747813+0000", 2024-05-15T23:20:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10302538999999999 2024-05-15T23:20:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.747905+0000", 2024-05-15T23:20:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1855999999999996e-05 2024-05-15T23:20:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:20:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.747993+0000", 2024-05-15T23:20:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8801000000000006e-05 2024-05-15T23:20:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.751088+0000", 2024-05-15T23:20:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030946799999999998 2024-05-15T23:20:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.760647+0000", 2024-05-15T23:20:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095584699999999995 2024-05-15T23:20:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.760698+0000", 2024-05-15T23:20:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0893999999999998e-05 2024-05-15T23:20:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.760743+0000", 2024-05-15T23:20:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5275000000000003e-05 2024-05-15T23:20:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:112241 5.8 5:180485a9:::benchmark_data_smithi090_116470_object112240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e398)", 2024-05-15T23:20:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:48.753350+0000", 2024-05-15T23:20:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.5946075319999999, 2024-05-15T23:20:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.112608941, 2024-05-15T23:20:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 112241 2024-05-15T23:20:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.753350+0000", 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.753350+0000", 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.753351+0000", 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3300000000000001e-07 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.753368+0000", 2024-05-15T23:20:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6728e-05 2024-05-15T23:20:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.753374+0000", 2024-05-15T23:20:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1e-06 2024-05-15T23:20:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.753378+0000", 2024-05-15T23:20:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3749999999999996e-06 2024-05-15T23:20:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.758202+0000", 2024-05-15T23:20:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004824226 2024-05-15T23:20:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.758286+0000", 2024-05-15T23:20:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3924000000000005e-05 2024-05-15T23:20:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:20:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.758422+0000", 2024-05-15T23:20:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578 2024-05-15T23:20:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.857856+0000", 2024-05-15T23:20:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099434211999999994 2024-05-15T23:20:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.865858+0000", 2024-05-15T23:20:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0080013389999999997 2024-05-15T23:20:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.865914+0000", 2024-05-15T23:20:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6514000000000001e-05 2024-05-15T23:20:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.865959+0000", 2024-05-15T23:20:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4910000000000002e-05 2024-05-15T23:20:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:112264 5.8 5:16d6920f:::benchmark_data_smithi090_116470_object112263:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e398)", 2024-05-15T23:20:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:48.767258+0000", 2024-05-15T23:20:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.5806995129999999, 2024-05-15T23:20:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11474749300000001, 2024-05-15T23:20:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:20:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:20:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:20:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:20:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:20:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:20:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 112264 2024-05-15T23:20:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:20:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:20:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.767258+0000", 2024-05-15T23:20:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:20:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.767258+0000", 2024-05-15T23:20:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:20:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:20:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.767260+0000", 2024-05-15T23:20:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8169999999999999e-06 2024-05-15T23:20:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:20:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.767280+0000", 2024-05-15T23:20:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0117999999999999e-05 2024-05-15T23:20:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:20:50.462 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.856514+0000", 2024-05-15T23:20:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089234340999999995 2024-05-15T23:20:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:20:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.856525+0000", 2024-05-15T23:20:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0795000000000001e-05 2024-05-15T23:20:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:20:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.857904+0000", 2024-05-15T23:20:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001379187 2024-05-15T23:20:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:20:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.858039+0000", 2024-05-15T23:20:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013446899999999999 2024-05-15T23:20:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:20:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.858168+0000", 2024-05-15T23:20:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128944 2024-05-15T23:20:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:20:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.865971+0000", 2024-05-15T23:20:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0078029570000000001 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.881910+0000", 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015938933999999998 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.881962+0000", 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1924e-05 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.882006+0000", 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4007000000000001e-05 2024-05-15T23:20:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:20:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:20:50.466 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:20:50.490 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:50.529 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:50.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:50.633 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:50.660 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:51.056 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 52 16 115215 115199 138.431 86.0625 0.016589 0.00722156 2024-05-15T23:20:51.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T23:20:51.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:20:51.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:51 smithi088 ceph-mon[44699]: pgmap v1084: 17 pgs: 17 active+clean; 6.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-05-15T23:20:51.352 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:51 smithi084 ceph-mon[39784]: pgmap v1084: 17 pgs: 17 active+clean; 6.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-05-15T23:20:51.390 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:51.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:51.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:51.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:20:51.528 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:51 smithi090 ceph-mon[44434]: pgmap v1084: 17 pgs: 17 active+clean; 6.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-05-15T23:20:51.531 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:51.554 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:20:51.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:51.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:20:51.661 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:51.692 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:51.942 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:51.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:52.046 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:20:52.056 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 53 16 116621 116605 137.477 87.875 0.00448801 0.00727239 2024-05-15T23:20:52.077 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:52.200 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:20:52.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:52.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:20:52.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:20:52.344 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:52.375 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:20:52.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:52.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:20:52.482 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:20:52.540 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:52.568 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:20:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:52.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:20:52.580 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:52.624 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:20:52.644 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:20:52.647 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:52 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2374614077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:52.667 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:20:52.674 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:20:52.691 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:20:52.719 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:20:52.727 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:52.750 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:20:52.754 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:20:52.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:52.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:20:52.775 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:20:52.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:20:52.823 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:20:52.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:52 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2374614077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:52.844 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:20:52.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:52.872 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:20:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:52.896 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:20:52.898 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-05-15T23:20:52.898 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:20:52.898 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:20:52.898 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:20:52.898 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:20:52.898 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:20:52.898 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:20:52.898 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:20:52.898 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:20:52.898 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:20:52.899 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:20:52.899 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:20:52.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:52 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2374614077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:20:52.967 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:52.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:52.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:53.059 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 54 16 117987 117971 136.507 85.375 0.00918754 0.00732372 2024-05-15T23:20:53.074 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:20:53.108 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:53.183 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:20:53.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:53.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:20:53.289 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:20:53.337 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:53.393 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:20:53.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:53.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:20:53.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:53.528 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:53.655 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.655 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.655 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.655 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.655 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.655 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.656 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:53 smithi088 ceph-mon[44699]: pgmap v1085: 17 pgs: 17 active+clean; 7.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 88 MiB/s wr, 1.42k op/s 2024-05-15T23:20:53.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:53.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:53 smithi084 ceph-mon[39784]: pgmap v1085: 17 pgs: 17 active+clean; 7.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 88 MiB/s wr, 1.42k op/s 2024-05-15T23:20:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:54.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:53 smithi090 ceph-mon[44434]: pgmap v1085: 17 pgs: 17 active+clean; 7.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 88 MiB/s wr, 1.42k op/s 2024-05-15T23:20:54.058 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 55 16 119366 119350 135.592 86.1875 0.00507416 0.00737333 2024-05-15T23:20:55.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:55.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:55.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:55.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3002137921' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:20:55.059 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 56 15 120722 120707 134.685 84.8125 0.0237908 0.00742287 2024-05-15T23:20:55.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:55.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:55.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:55.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3002137921' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:20:55.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:55.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:55.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:20:55.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3002137921' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:20:55.877 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:55.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:55.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:20:55.976 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-05-15T23:20:55.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:20:56.014 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:55 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3002137921' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:20:56.014 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:55 smithi090 ceph-mon[44434]: osdmap e399: 12 total, 12 up, 10 in 2024-05-15T23:20:56.014 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:55 smithi090 ceph-mon[44434]: pgmap v1087: 17 pgs: 17 active+clean; 7.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-05-15T23:20:56.016 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:20:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:56.043 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:20:56.059 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 57 16 122089 122073 133.82 85.375 0.00849591 0.00747134 2024-05-15T23:20:56.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:55 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3002137921' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:20:56.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:55 smithi088 ceph-mon[44699]: osdmap e399: 12 total, 12 up, 10 in 2024-05-15T23:20:56.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:55 smithi088 ceph-mon[44699]: pgmap v1087: 17 pgs: 17 active+clean; 7.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-05-15T23:20:56.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:55 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3002137921' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:20:56.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:55 smithi084 ceph-mon[39784]: osdmap e399: 12 total, 12 up, 10 in 2024-05-15T23:20:56.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:55 smithi084 ceph-mon[39784]: pgmap v1087: 17 pgs: 17 active+clean; 7.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-05-15T23:20:56.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:56.174 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:20:56.180 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:56.204 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:56.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:56.207 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:56.308 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:20:56.356 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:56.386 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:20:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:56.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:20:56.497 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:20:56.554 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:56.586 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:20:56.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:56.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:20:56.695 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:20:56.758 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:56.789 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:20:56.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:56.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:20:56.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:20:56.927 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:56.955 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:20:56.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:56.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:20:57.059 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 58 16 123361 123345 132.884 79.5 0.00713545 0.00752356 2024-05-15T23:20:57.062 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:20:57.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:56 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:20:57.122 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:57.153 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:20:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:57.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:20:57.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:56 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:20:57.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:20:57.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:56 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:20:57.324 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:57.348 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:20:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:57.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:20:57.442 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:20:57.453 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:57.508 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:57.543 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:57.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:57.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:57.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:20:57.684 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:57.711 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:20:57.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:57.715 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:20:57.816 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:20:57.864 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:20:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:57.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:20:58.009 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:20:58.059 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 59 15 124575 124560 131.919 75.9375 0.00855678 0.00757866 2024-05-15T23:20:58.062 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:20:58.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:58.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:20:58.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:20:58.207 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:57 smithi084 ceph-mon[39784]: pgmap v1088: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-05-15T23:20:58.207 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:57 smithi084 ceph-mon[39784]: osdmap e400: 12 total, 12 up, 10 in 2024-05-15T23:20:58.245 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:57 smithi090 ceph-mon[44434]: pgmap v1088: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-05-15T23:20:58.245 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:57 smithi090 ceph-mon[44434]: osdmap e400: 12 total, 12 up, 10 in 2024-05-15T23:20:58.252 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:58.269 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:20:58.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:58.273 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:20:58.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:57 smithi088 ceph-mon[44699]: pgmap v1088: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-05-15T23:20:58.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:57 smithi088 ceph-mon[44699]: osdmap e400: 12 total, 12 up, 10 in 2024-05-15T23:20:58.374 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:58.403 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:59.059 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:20:59.058684 min lat: 0.000997126 max lat: 0.211796 avg lat: 0.0076271 2024-05-15T23:20:59.059 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:20:59.060 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 60 16 125891 125875 131.09 82.1875 0.0108489 0.0076271 2024-05-15T23:20:59.263 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:59.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:59.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:59.281 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:20:59.282 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:20:59.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:59.401 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:59.652 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:59.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:59.756 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:20:59.787 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:59.810 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:20:59.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:59.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:20:59.914 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:20:59.946 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:20:59.973 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:20:59.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:20:59.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:00.059 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 61 16 127186 127170 130.268 80.9375 0.0079999 0.00767493 2024-05-15T23:21:00.077 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:00.118 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:00.158 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:00.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:00.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:00.269 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:00.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:59 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:21:00.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:59 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:21:00.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:59 smithi090 ceph-mon[44434]: osdmap e401: 12 total, 12 up, 10 in 2024-05-15T23:21:00.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:20:59 smithi090 ceph-mon[44434]: pgmap v1091: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-05-15T23:21:00.311 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:59 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:21:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:59 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:21:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:59 smithi088 ceph-mon[44699]: osdmap e401: 12 total, 12 up, 10 in 2024-05-15T23:21:00.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:20:59 smithi088 ceph-mon[44699]: pgmap v1091: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-05-15T23:21:00.340 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:00.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:00.347 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:00.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:59 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:21:00.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:59 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:21:00.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:59 smithi084 ceph-mon[39784]: osdmap e401: 12 total, 12 up, 10 in 2024-05-15T23:21:00.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:20:59 smithi084 ceph-mon[39784]: pgmap v1091: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-05-15T23:21:00.447 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:00.509 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:00.539 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:00.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:00.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:00.565 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:00.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T23:21:00.645 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:21:00.674 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:00.875 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:21:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:00.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:21:00.973 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:21:00.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:21:01.010 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:01.059 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 62 16 128479 128463 129.47 80.8125 0.0117279 0.00772224 2024-05-15T23:21:01.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:00 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1711635977' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:21:01.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:00 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1711635977' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:21:01.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:00 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1711635977' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:21:02.060 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 63 16 129784 129768 128.709 81.5625 0.00791847 0.00776804 2024-05-15T23:21:02.071 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 11, 9, 10] out_osds: [2, 0] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2, 0] 2024-05-15T23:21:02.071 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:21:02.072 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [1, 3, 4, 5, 6, 7, 8, 11, 9, 10] 2024-05-15T23:21:02.072 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd out 6 2024-05-15T23:21:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:01 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:21:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:01 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1711635977' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:21:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:01 smithi090 ceph-mon[44434]: osdmap e402: 12 total, 12 up, 10 in 2024-05-15T23:21:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:01 smithi090 ceph-mon[44434]: pgmap v1093: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 105 MiB/s wr, 1.67k op/s 2024-05-15T23:21:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:01 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:21:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:01 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:21:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:01 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:21:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:01 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1711635977' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:21:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:01 smithi088 ceph-mon[44699]: osdmap e402: 12 total, 12 up, 10 in 2024-05-15T23:21:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:01 smithi088 ceph-mon[44699]: pgmap v1093: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 105 MiB/s wr, 1.67k op/s 2024-05-15T23:21:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:01 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:21:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:01 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:21:02.355 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:01 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:21:02.355 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:01 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1711635977' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:21:02.355 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:01 smithi084 ceph-mon[39784]: osdmap e402: 12 total, 12 up, 10 in 2024-05-15T23:21:02.355 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:01 smithi084 ceph-mon[39784]: pgmap v1093: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 105 MiB/s wr, 1.67k op/s 2024-05-15T23:21:02.355 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:01 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:21:02.355 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:01 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:21:02.361 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:21:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:02.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:21:02.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:21:02.497 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:02.834 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:02.850 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:21:02.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:02.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:21:02.958 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:21:03.030 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:03.059 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 64 16 131095 131079 127.979 81.9375 0.0177464 0.00781221 2024-05-15T23:21:03.061 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:21:03.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:03.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:21:03.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:02 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:21:03.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:02 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:21:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:02 smithi084 ceph-mon[39784]: osdmap e403: 12 total, 12 up, 10 in 2024-05-15T23:21:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:02 smithi084 ceph-mon[39784]: osdmap e404: 12 total, 12 up, 10 in 2024-05-15T23:21:03.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:03.204 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:03.231 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:03.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:03.239 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:02 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:21:03.239 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:02 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:21:03.240 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:02 smithi090 ceph-mon[44434]: osdmap e403: 12 total, 12 up, 10 in 2024-05-15T23:21:03.240 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:02 smithi090 ceph-mon[44434]: osdmap e404: 12 total, 12 up, 10 in 2024-05-15T23:21:03.309 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:03.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:02 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:21:03.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:02 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:21:03.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:02 smithi088 ceph-mon[44699]: osdmap e403: 12 total, 12 up, 10 in 2024-05-15T23:21:03.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:02 smithi088 ceph-mon[44699]: osdmap e404: 12 total, 12 up, 10 in 2024-05-15T23:21:03.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:21:03.372 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:03.524 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:21:03.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:03.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:21:03.630 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:21:03.684 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:03.732 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:21:03.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:03.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:21:03.838 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:21:03.879 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:03.908 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:21:03.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:03.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:21:03.984 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:03 smithi088 ceph-mon[44699]: pgmap v1096: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-05-15T23:21:04.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:21:04.060 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 65 16 132335 132319 127.202 77.5 0.0120256 0.00786002 2024-05-15T23:21:04.063 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:04.093 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:21:04.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:04.098 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:21:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:03 smithi084 ceph-mon[39784]: pgmap v1096: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-05-15T23:21:04.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:04.238 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:03 smithi090 ceph-mon[44434]: pgmap v1096: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-05-15T23:21:04.239 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:04.259 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:04.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:04.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:04.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:21:04.391 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:21:04.397 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:21:04.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:04.427 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:21:04.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:04.559 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:04.584 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:04.592 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:04.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:04.740 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:04.771 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:04.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:04.875 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:21:04.903 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:05.054 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:21:05.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:05.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:21:05.059 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 66 16 133578 133562 126.452 77.6875 0.00550028 0.00790693 2024-05-15T23:21:05.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:05.197 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:05.227 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:05.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:05.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:05.375 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:05.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:05.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:05.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:21:05.537 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:05.611 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:21:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:05.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:21:05.720 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:21:05.749 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:06.060 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 67 16 134829 134813 125.731 78.1875 0.0184818 0.00795176 2024-05-15T23:21:06.099 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:21:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:06.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:21:06.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:06.240 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:06.269 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:06.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:06.272 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:06.272 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:05 smithi090 ceph-mon[44434]: pgmap v1097: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-05-15T23:21:06.272 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:05 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:21:06.272 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:05 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:21:06.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:05 smithi084 ceph-mon[39784]: pgmap v1097: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-05-15T23:21:06.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:05 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:21:06.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:05 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:21:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:05 smithi088 ceph-mon[44699]: pgmap v1097: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-05-15T23:21:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:05 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:21:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:05 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:21:06.372 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:21:06.472 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:06.515 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:21:06.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:06.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:21:06.558 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:21:06.582 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:21:06.606 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:21:06.618 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:21:06.631 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:21:06.648 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:06.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:21:06.689 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:21:06.731 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:21:06.752 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:21:06.778 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:21:06.800 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:21:06.826 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:21:06.849 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:21:06.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:06.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:21:06.863 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:21:06.865 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:21:06.865 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:21:06.865 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:21:06.865 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:21:06.866 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:21:06.866 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:21:06.866 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:21:06.866 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:21:06.866 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:21:06.866 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:21:06.866 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:21:06.866 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:21:06.879 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:06.955 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:21:07.013 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:07.049 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:21:07.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:07.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:21:07.060 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 68 16 136007 135991 124.965 73.625 0.0100656 0.00800081 2024-05-15T23:21:07.155 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:07.187 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:07.227 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:07.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:07.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:07.258 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:06 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:21:07.258 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:06 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:21:07.258 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:06 smithi084 ceph-mon[39784]: osdmap e405: 12 total, 12 up, 10 in 2024-05-15T23:21:07.258 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:21:07.258 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:21:07.259 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:21:07.259 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:06 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3080778288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:21:07.259 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:06 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:07.259 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:06 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:07.259 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:07 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:21:07.103+0000 7fa0938b7700 -1 mon.a@0(leader).osd e406 definitely_dead 0 2024-05-15T23:21:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:06 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:21:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:06 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:21:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:06 smithi090 ceph-mon[44434]: osdmap e405: 12 total, 12 up, 10 in 2024-05-15T23:21:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:21:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:21:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:21:07.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:06 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3080778288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:21:07.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:06 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:07.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:06 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:07 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:21:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:07 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:21:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:07 smithi088 ceph-mon[44699]: osdmap e405: 12 total, 12 up, 10 in 2024-05-15T23:21:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:21:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:21:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:21:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:07 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3080778288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:21:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:07.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:07.333 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:07.367 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:07.391 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:07.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:07.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:21:07.529 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:07.788 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:21:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:21:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:21:07.789 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:21:07.789 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:21:08.019 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.6. 2024-05-15T23:21:08.060 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 69 16 137252 137236 124.282 77.8125 0.00400441 0.00804378 2024-05-15T23:21:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:08 smithi090 ceph-mon[44434]: pgmap v1099: 17 pgs: 17 active+clean; 8.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-05-15T23:21:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:08 smithi090 ceph-mon[44434]: osdmap e406: 12 total, 12 up, 10 in 2024-05-15T23:21:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1686161043' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-05-15T23:21:08.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:08 smithi090 ceph-mon[44434]: Client client.admin marked osd.6 out, while it was still marked up 2024-05-15T23:21:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:08 smithi088 ceph-mon[44699]: pgmap v1099: 17 pgs: 17 active+clean; 8.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-05-15T23:21:08.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:08 smithi088 ceph-mon[44699]: osdmap e406: 12 total, 12 up, 10 in 2024-05-15T23:21:08.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1686161043' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-05-15T23:21:08.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:08 smithi088 ceph-mon[44699]: Client client.admin marked osd.6 out, while it was still marked up 2024-05-15T23:21:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:08.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:08 smithi084 ceph-mon[39784]: pgmap v1099: 17 pgs: 17 active+clean; 8.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-05-15T23:21:08.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:08 smithi084 ceph-mon[39784]: osdmap e406: 12 total, 12 up, 10 in 2024-05-15T23:21:08.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1686161043' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-05-15T23:21:08.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:08 smithi084 ceph-mon[39784]: Client client.admin marked osd.6 out, while it was still marked up 2024-05-15T23:21:09.060 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 70 16 137290 137274 122.54 2.375 0.00279262 0.00804339 2024-05-15T23:21:09.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:09 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1686161043' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-05-15T23:21:09.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:09 smithi084 ceph-mon[39784]: osdmap e407: 12 total, 12 up, 9 in 2024-05-15T23:21:09.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:09 smithi084 ceph-mon[39784]: pgmap v1102: 17 pgs: 17 active+clean; 8.2 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T23:21:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:09 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1686161043' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-05-15T23:21:09.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:09 smithi090 ceph-mon[44434]: osdmap e407: 12 total, 12 up, 9 in 2024-05-15T23:21:09.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:09 smithi090 ceph-mon[44434]: pgmap v1102: 17 pgs: 17 active+clean; 8.2 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T23:21:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:09 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1686161043' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-05-15T23:21:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:09 smithi088 ceph-mon[44699]: osdmap e407: 12 total, 12 up, 9 in 2024-05-15T23:21:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:09 smithi088 ceph-mon[44699]: pgmap v1102: 17 pgs: 17 active+clean; 8.2 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-05-15T23:21:09.782 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:21:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:09.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:21:09.887 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:21:09.917 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:10.044 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:21:10.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:10.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:21:10.060 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 71 16 137866 137850 121.322 36 0.00762677 0.00823949 2024-05-15T23:21:10.153 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:10 smithi090 ceph-mon[44434]: osdmap e408: 12 total, 12 up, 9 in 2024-05-15T23:21:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:10 smithi088 ceph-mon[44699]: osdmap e408: 12 total, 12 up, 9 in 2024-05-15T23:21:10.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:10 smithi084 ceph-mon[39784]: osdmap e408: 12 total, 12 up, 9 in 2024-05-15T23:21:10.722 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T23:21:10.729 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:10.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:10.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:10.834 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:21:10.871 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:10.899 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:21:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:10.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:21:11.006 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:21:11.040 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:11.060 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 72 16 138759 138743 120.412 55.8125 0.0102974 0.0083019 2024-05-15T23:21:11.067 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:21:11.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:11.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:21:11.173 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:21:11.201 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:11.226 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:21:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:11.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:21:11.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:11 smithi090 ceph-mon[44434]: pgmap v1104: 17 pgs: 2 active+recovering, 15 active+clean; 8.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 65 MiB/s wr, 1.03k op/s; 17163/274804 objects misplaced (6.246%); 462 KiB/s, 7 objects/s recovering 2024-05-15T23:21:11.326 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:11 smithi088 ceph-mon[44699]: pgmap v1104: 17 pgs: 2 active+recovering, 15 active+clean; 8.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 65 MiB/s wr, 1.03k op/s; 17163/274804 objects misplaced (6.246%); 462 KiB/s, 7 objects/s recovering 2024-05-15T23:21:11.331 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:21:11.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:11 smithi084 ceph-mon[39784]: pgmap v1104: 17 pgs: 2 active+recovering, 15 active+clean; 8.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 65 MiB/s wr, 1.03k op/s; 17163/274804 objects misplaced (6.246%); 462 KiB/s, 7 objects/s recovering 2024-05-15T23:21:11.375 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:11.386 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:11.410 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:21:11.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:11.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:21:11.515 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:11.566 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:11.618 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:11.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:11.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:11.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:21:11.750 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:11.790 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:21:11.826 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:21:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:11.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:21:11.934 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:21:11.988 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:12.018 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:21:12.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:12.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:21:12.026 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:21:12.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:12 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:21:12.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:12 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:21:12.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:12 smithi084 ceph-mon[39784]: osdmap e409: 12 total, 12 up, 9 in 2024-05-15T23:21:12.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:12 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/903073569' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:21:12.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:12 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:21:12.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:12 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/903073569' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:21:12.038 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:12 smithi084 ceph-mon[39784]: osdmap e410: 12 total, 12 up, 9 in 2024-05-15T23:21:12.060 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 73 16 139568 139552 119.455 50.5625 0.0316908 0.00836883 2024-05-15T23:21:12.127 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:21:12.177 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:12.209 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:21:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:12.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:21:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:12 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:21:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:12 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:21:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:12 smithi090 ceph-mon[44434]: osdmap e409: 12 total, 12 up, 9 in 2024-05-15T23:21:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:12 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/903073569' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:21:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:12 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:21:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:12 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/903073569' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:21:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:12 smithi090 ceph-mon[44434]: osdmap e410: 12 total, 12 up, 9 in 2024-05-15T23:21:12.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:12.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:12 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:21:12.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:12 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:21:12.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:12 smithi088 ceph-mon[44699]: osdmap e409: 12 total, 12 up, 9 in 2024-05-15T23:21:12.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:12 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/903073569' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:21:12.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:12 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:21:12.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:12 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/903073569' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:21:12.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:12 smithi088 ceph-mon[44699]: osdmap e410: 12 total, 12 up, 9 in 2024-05-15T23:21:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:12.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:12.370 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:12.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:12.501 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:12.533 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:12.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:12.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:12.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:12.688 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:12.715 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:12.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:12.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:12.824 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:12.854 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:13.003 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:13.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:13.060 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 74 16 140321 140305 118.477 47.0625 0.0979648 0.00843603 2024-05-15T23:21:13.107 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:21:13.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:13 smithi084 ceph-mon[39784]: 5.1 scrub starts 2024-05-15T23:21:13.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:13 smithi084 ceph-mon[39784]: pgmap v1107: 17 pgs: 3 active+recovering, 14 active+clean; 8.5 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 47 MiB/s wr, 745 op/s; 25783/277352 objects misplaced (9.296%); 1.3 MiB/s, 21 objects/s recovering 2024-05-15T23:21:13.182 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:13.213 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:21:13.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:13.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:21:13.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:13 smithi090 ceph-mon[44434]: 5.1 scrub starts 2024-05-15T23:21:13.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:13 smithi090 ceph-mon[44434]: pgmap v1107: 17 pgs: 3 active+recovering, 14 active+clean; 8.5 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 47 MiB/s wr, 745 op/s; 25783/277352 objects misplaced (9.296%); 1.3 MiB/s, 21 objects/s recovering 2024-05-15T23:21:13.317 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:21:13.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:13 smithi088 ceph-mon[44699]: 5.1 scrub starts 2024-05-15T23:21:13.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:13 smithi088 ceph-mon[44699]: pgmap v1107: 17 pgs: 3 active+recovering, 14 active+clean; 8.5 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 47 MiB/s wr, 745 op/s; 25783/277352 objects misplaced (9.296%); 1.3 MiB/s, 21 objects/s recovering 2024-05-15T23:21:13.364 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:13.401 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:21:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:13.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:21:13.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:21:13.548 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:13.578 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:13.585 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:21:13.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:13.590 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:21:13.691 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:21:13.744 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:21:13.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:13.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:21:13.868 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 8, 11, 9, 10] out_osds: [2, 0, 6] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2, 0] 2024-05-15T23:21:13.868 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 1 1 2024-05-15T23:21:13.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:13.912 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:13.945 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:13.953 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:14.051 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:14 smithi088 ceph-mon[44699]: 5.b scrub starts 2024-05-15T23:21:14.054 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:21:14.060 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 75 16 141122 141106 117.565 50.0625 0.0231033 0.00850419 2024-05-15T23:21:14.088 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:14.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:14 smithi084 ceph-mon[39784]: 5.b scrub starts 2024-05-15T23:21:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:14 smithi090 ceph-mon[44434]: 5.b scrub starts 2024-05-15T23:21:14.926 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:21:14.926 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-05-15T23:21:14.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-05-15T23:21:14.926 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2024-05-15T23:21:14.926 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:21:15.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:21:15.060 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 76 16 141939 141923 116.69 51.0625 0.00748921 0.00856606 2024-05-15T23:21:15.290 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:15 smithi084 ceph-mon[39784]: 5.1 scrub ok 2024-05-15T23:21:15.290 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:15 smithi084 ceph-mon[39784]: 5.c scrub starts 2024-05-15T23:21:15.290 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:15 smithi084 ceph-mon[39784]: pgmap v1108: 17 pgs: 3 active+recovering, 14 active+clean; 8.5 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 41 MiB/s wr, 659 op/s; 25783/277864 objects misplaced (9.279%); 1.1 MiB/s, 17 objects/s recovering 2024-05-15T23:21:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:15 smithi088 ceph-mon[44699]: 5.1 scrub ok 2024-05-15T23:21:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:15 smithi088 ceph-mon[44699]: 5.c scrub starts 2024-05-15T23:21:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:15 smithi088 ceph-mon[44699]: pgmap v1108: 17 pgs: 3 active+recovering, 14 active+clean; 8.5 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 41 MiB/s wr, 659 op/s; 25783/277864 objects misplaced (9.279%); 1.1 MiB/s, 17 objects/s recovering 2024-05-15T23:21:15.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:15 smithi090 ceph-mon[44434]: 5.1 scrub ok 2024-05-15T23:21:15.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:15 smithi090 ceph-mon[44434]: 5.c scrub starts 2024-05-15T23:21:15.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:15 smithi090 ceph-mon[44434]: pgmap v1108: 17 pgs: 3 active+recovering, 14 active+clean; 8.5 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 41 MiB/s wr, 659 op/s; 25783/277864 objects misplaced (9.279%); 1.1 MiB/s, 17 objects/s recovering 2024-05-15T23:21:15.610 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:15.840 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:21:15.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:15.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:21:15.847 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:15.948 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:21:15.981 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:16.061 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 77 16 142812 142796 115.883 54.5625 0.0163431 0.00862714 2024-05-15T23:21:16.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:16 smithi084 ceph-mon[39784]: 5.c scrub ok 2024-05-15T23:21:16.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:16 smithi088 ceph-mon[44699]: 5.c scrub ok 2024-05-15T23:21:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:16 smithi090 ceph-mon[44434]: 5.c scrub ok 2024-05-15T23:21:16.683 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:21:16.704 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:21:16.723 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:21:16.744 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:21:16.771 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:21:16.804 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:21:16.837 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:21:16.864 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:21:16.888 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:21:16.910 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:21:16.933 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:21:16.955 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:21:16.956 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:21:16.956 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:21:16.956 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:21:16.956 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:21:16.956 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:21:16.956 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:21:16.956 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:21:16.957 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:21:16.957 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:21:16.957 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:21:16.957 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:21:16.957 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:21:17.061 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 78 16 143686 143670 115.098 54.625 0.0157041 0.00868654 2024-05-15T23:21:17.133 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:21:17.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:17.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:21:17.237 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:21:17.287 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:17.320 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:21:17.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:17.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:17 smithi088 ceph-mon[44699]: 5.b scrub ok 2024-05-15T23:21:17.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:17 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2246805239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:21:17.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:17.326 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:17 smithi088 ceph-mon[44699]: pgmap v1109: 17 pgs: 1 active+clean+scrubbing, 3 active+recovering, 13 active+clean; 8.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 49 MiB/s wr, 784 op/s; 25575/282440 objects misplaced (9.055%); 2.5 MiB/s, 39 objects/s recovering 2024-05-15T23:21:17.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:21:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:17 smithi084 ceph-mon[39784]: 5.b scrub ok 2024-05-15T23:21:17.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:17 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2246805239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:21:17.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:17.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:17 smithi084 ceph-mon[39784]: pgmap v1109: 17 pgs: 1 active+clean+scrubbing, 3 active+recovering, 13 active+clean; 8.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 49 MiB/s wr, 784 op/s; 25575/282440 objects misplaced (9.055%); 2.5 MiB/s, 39 objects/s recovering 2024-05-15T23:21:17.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:17.475 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:17 smithi090 ceph-mon[44434]: 5.b scrub ok 2024-05-15T23:21:17.475 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:17 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2246805239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:21:17.476 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:17.476 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:17 smithi090 ceph-mon[44434]: pgmap v1109: 17 pgs: 1 active+clean+scrubbing, 3 active+recovering, 13 active+clean; 8.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 49 MiB/s wr, 784 op/s; 25575/282440 objects misplaced (9.055%); 2.5 MiB/s, 39 objects/s recovering 2024-05-15T23:21:17.479 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:17.516 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:17.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:17.523 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:17.624 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:17.664 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:17.707 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:17.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:17.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:17.812 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:17.868 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:17.904 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:17.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:17.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:18.011 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:21:18.046 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:18.061 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 79 16 144439 144423 114.236 47.0625 0.0114807 0.00875161 2024-05-15T23:21:18.079 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:21:18.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:18.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:21:18.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:21:18.220 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:18.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:18 smithi088 ceph-mon[44699]: 5.e scrub starts 2024-05-15T23:21:18.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:18 smithi084 ceph-mon[39784]: 5.e scrub starts 2024-05-15T23:21:18.372 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:21:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:18.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:21:18.396 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T23:21:18.477 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:18.519 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:18.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:18.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:18 smithi090 ceph-mon[44434]: 5.e scrub starts 2024-05-15T23:21:18.555 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:18.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:18.661 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:18.692 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:19.061 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:21:19.060371 min lat: 0.000997126 max lat: 1.49715 avg lat: 0.00881118 2024-05-15T23:21:19.061 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:21:19.061 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 80 16 145281 145265 113.466 52.625 0.00991216 0.00881118 2024-05-15T23:21:19.255 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:19.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:19.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:19.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:21:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:19 smithi084 ceph-mon[39784]: 5.e scrub ok 2024-05-15T23:21:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:19 smithi084 ceph-mon[39784]: pgmap v1110: 17 pgs: 1 active+clean+scrubbing, 3 active+recovering, 13 active+clean; 8.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 42 MiB/s wr, 673 op/s; 25453/285584 objects misplaced (8.913%); 3.1 MiB/s, 49 objects/s recovering 2024-05-15T23:21:19.390 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:19 smithi090 ceph-mon[44434]: 5.e scrub ok 2024-05-15T23:21:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:19 smithi090 ceph-mon[44434]: pgmap v1110: 17 pgs: 1 active+clean+scrubbing, 3 active+recovering, 13 active+clean; 8.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 42 MiB/s wr, 673 op/s; 25453/285584 objects misplaced (8.913%); 3.1 MiB/s, 49 objects/s recovering 2024-05-15T23:21:19.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:19 smithi088 ceph-mon[44699]: 5.e scrub ok 2024-05-15T23:21:19.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:19 smithi088 ceph-mon[44699]: pgmap v1110: 17 pgs: 1 active+clean+scrubbing, 3 active+recovering, 13 active+clean; 8.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 42 MiB/s wr, 673 op/s; 25453/285584 objects misplaced (8.913%); 3.1 MiB/s, 49 objects/s recovering 2024-05-15T23:21:19.592 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:21:19.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:19.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:21:19.698 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:21:19.729 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:19.820 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:21:20.061 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 81 16 146115 146099 112.709 52.125 0.016162 0.00887086 2024-05-15T23:21:20.087 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:21:20.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:20.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:21:20.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:21:20.231 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:20.257 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:21:20.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:20.260 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:21:20.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:20 smithi084 ceph-mon[39784]: 5.5 scrub starts 2024-05-15T23:21:20.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:20 smithi090 ceph-mon[44434]: 5.5 scrub starts 2024-05-15T23:21:20.395 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:20.424 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:20.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:20.431 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:20.532 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:20 smithi088 ceph-mon[44699]: 5.5 scrub starts 2024-05-15T23:21:20.587 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:20.617 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:20.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:20.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:20.723 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:21:20.759 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:20.793 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:21:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:20.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:21:20.840 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:20.897 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:20.926 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:20.954 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:20.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:20.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:21.060 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:21.061 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 82 16 146928 146912 111.954 50.8125 0.00777668 0.00893007 2024-05-15T23:21:21.074 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:21.097 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:21.299 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:21.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:21.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:21.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:21 smithi090 ceph-mon[44434]: 5.5 scrub ok 2024-05-15T23:21:21.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:21 smithi090 ceph-mon[44434]: pgmap v1111: 17 pgs: 3 active+recovering, 14 active+clean; 8.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 50 MiB/s wr, 802 op/s; 25235/290776 objects misplaced (8.679%); 3.9 MiB/s, 62 objects/s recovering 2024-05-15T23:21:21.355 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:21 smithi084 ceph-mon[39784]: 5.5 scrub ok 2024-05-15T23:21:21.355 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:21 smithi084 ceph-mon[39784]: pgmap v1111: 17 pgs: 3 active+recovering, 14 active+clean; 8.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 50 MiB/s wr, 802 op/s; 25235/290776 objects misplaced (8.679%); 3.9 MiB/s, 62 objects/s recovering 2024-05-15T23:21:21.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:21.437 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:21.467 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:21.568 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:21:21.577 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:21 smithi088 ceph-mon[44699]: 5.5 scrub ok 2024-05-15T23:21:21.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:21 smithi088 ceph-mon[44699]: pgmap v1111: 17 pgs: 3 active+recovering, 14 active+clean; 8.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 50 MiB/s wr, 802 op/s; 25235/290776 objects misplaced (8.679%); 3.9 MiB/s, 62 objects/s recovering 2024-05-15T23:21:21.622 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:21.651 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:21:21.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:21.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:21:21.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:21.805 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:21.828 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:21.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:21.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:21.938 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:21:21.984 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:22.017 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:21:22.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:22.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:21:22.061 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 83 16 147772 147756 111.241 52.75 0.0153944 0.00898696 2024-05-15T23:21:22.125 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:22.135 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2024-05-15T23:21:22.135 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:21:22.158 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:22.188 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:22.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:22.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:22.293 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:22.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2039146551' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:21:22.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/276709911' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:21:22.370 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:22.409 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:22.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:22.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:22.521 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:21:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2039146551' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:21:22.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/276709911' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:21:22.569 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:22.606 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:21:22.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:22.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:21:22.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2039146551' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:21:22.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/276709911' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:21:22.710 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:22.741 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:23.061 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 84 16 148554 148538 110.499 48.875 0.00549301 0.00904717 2024-05-15T23:21:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:21:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:21:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:23 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:21:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:23 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:21:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:23 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2039146551' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:21:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:23 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/276709911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:21:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:23 smithi090 ceph-mon[44434]: osdmap e411: 12 total, 12 up, 9 in 2024-05-15T23:21:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:23 smithi090 ceph-mon[44434]: pgmap v1113: 17 pgs: 3 active+recovering, 14 active+clean; 9.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 51 MiB/s wr, 823 op/s; 25102/293826 objects misplaced (8.543%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:21:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:21:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:21:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:23 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:21:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:23 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:21:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:23 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2039146551' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:21:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:23 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/276709911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:21:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:23 smithi088 ceph-mon[44699]: osdmap e411: 12 total, 12 up, 9 in 2024-05-15T23:21:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:23 smithi088 ceph-mon[44699]: pgmap v1113: 17 pgs: 3 active+recovering, 14 active+clean; 9.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 51 MiB/s wr, 823 op/s; 25102/293826 objects misplaced (8.543%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:21:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:21:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:21:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:23 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:21:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:23 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:21:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:23 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2039146551' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:21:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:23 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/276709911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:21:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:23 smithi084 ceph-mon[39784]: osdmap e411: 12 total, 12 up, 9 in 2024-05-15T23:21:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:23 smithi084 ceph-mon[39784]: pgmap v1113: 17 pgs: 3 active+recovering, 14 active+clean; 9.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 51 MiB/s wr, 823 op/s; 25102/293826 objects misplaced (8.543%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:21:24.061 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 85 16 149371 149355 109.799 51.0625 0.025817 0.00910402 2024-05-15T23:21:24.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 3 1 2024-05-15T23:21:24.893 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:24.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:24.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:24.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:21:25.027 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:25.061 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 86 16 150178 150162 109.109 50.4375 0.00503534 0.0091628 2024-05-15T23:21:25.151 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:21:25.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:25.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:21:25.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:21:25.276 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:21:25.307 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:25.340 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:21:25.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:25.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:21:25.444 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:25.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:25 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:21:25.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:25 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:21:25.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:25 smithi084 ceph-mon[39784]: osdmap e412: 12 total, 12 up, 9 in 2024-05-15T23:21:25.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:25 smithi084 ceph-mon[39784]: pgmap v1115: 17 pgs: 3 active+recovering, 14 active+clean; 9.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 46 MiB/s wr, 740 op/s; 25102/294296 objects misplaced (8.530%); 3.7 MiB/s, 59 objects/s recovering 2024-05-15T23:21:25.484 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:25.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:25 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:21:25.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:25 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:21:25.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:25 smithi090 ceph-mon[44434]: osdmap e412: 12 total, 12 up, 9 in 2024-05-15T23:21:25.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:25 smithi090 ceph-mon[44434]: pgmap v1115: 17 pgs: 3 active+recovering, 14 active+clean; 9.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 46 MiB/s wr, 740 op/s; 25102/294296 objects misplaced (8.530%); 3.7 MiB/s, 59 objects/s recovering 2024-05-15T23:21:25.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:25 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:21:25.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:25 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:21:25.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:25 smithi088 ceph-mon[44699]: osdmap e412: 12 total, 12 up, 9 in 2024-05-15T23:21:25.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:25 smithi088 ceph-mon[44699]: pgmap v1115: 17 pgs: 3 active+recovering, 14 active+clean; 9.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 46 MiB/s wr, 740 op/s; 25102/294296 objects misplaced (8.530%); 3.7 MiB/s, 59 objects/s recovering 2024-05-15T23:21:25.641 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:21:25.642 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:21:25.642 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:21:25.642 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:21:25.642 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:21:25.642 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:21:25.652 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:21:25.679 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:21:25.701 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:21:25.722 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:21:25.744 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:21:25.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:21:25.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:21:25.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:21:25.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:21:25.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:21:25.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:21:25.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:21:25.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:21:25.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:21:25.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:21:25.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:21:25.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:21:25.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:21:25.923 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:21:25.923 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:21:25.923 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-05-15T23:21:25.923 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:21:26.061 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 87 16 151009 150993 108.452 51.9375 0.0160467 0.00921907 2024-05-15T23:21:26.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:26 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1450984020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:21:26.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:26.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:26.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:26.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:26.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:26 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1450984020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:21:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:26.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:26 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1450984020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:21:26.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:26.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:26.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:26.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:26.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.061 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 88 16 151866 151850 107.828 53.5625 0.00623883 0.00927058 2024-05-15T23:21:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:27 smithi090 ceph-mon[44434]: pgmap v1116: 17 pgs: 3 active+recovering, 14 active+clean; 9.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 52 MiB/s wr, 838 op/s; 24906/298996 objects misplaced (8.330%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:21:27.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:27 smithi088 ceph-mon[44699]: pgmap v1116: 17 pgs: 3 active+recovering, 14 active+clean; 9.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 52 MiB/s wr, 838 op/s; 24906/298996 objects misplaced (8.330%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:21:27.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:27.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:27 smithi084 ceph-mon[39784]: pgmap v1116: 17 pgs: 3 active+recovering, 14 active+clean; 9.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 52 MiB/s wr, 838 op/s; 24906/298996 objects misplaced (8.330%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:21:28.061 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 89 16 152693 152677 107.197 51.6875 0.0236808 0.00932642 2024-05-15T23:21:29.062 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 90 11 153557 153546 106.61 54.3125 0.00720556 0.00937702 2024-05-15T23:21:29.143 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:29.175 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:29.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T23:21:29.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:29 smithi088 ceph-mon[44699]: pgmap v1117: 17 pgs: 3 active+recovering, 14 active+clean; 9.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 44 MiB/s wr, 699 op/s; 24770/301966 objects misplaced (8.203%); 3.6 MiB/s, 58 objects/s recovering 2024-05-15T23:21:29.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:29 smithi084 ceph-mon[39784]: pgmap v1117: 17 pgs: 3 active+recovering, 14 active+clean; 9.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 44 MiB/s wr, 699 op/s; 24770/301966 objects misplaced (8.203%); 3.6 MiB/s, 58 objects/s recovering 2024-05-15T23:21:29.435 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:29.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:29.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:21:29.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:29 smithi090 ceph-mon[44434]: pgmap v1117: 17 pgs: 3 active+recovering, 14 active+clean; 9.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 44 MiB/s wr, 699 op/s; 24770/301966 objects misplaced (8.203%); 3.6 MiB/s, 58 objects/s recovering 2024-05-15T23:21:29.567 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:29.586 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:21:29.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:29.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:21:29.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:21:29.719 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:29.845 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:21:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:29.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:21:29.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:29.975 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:29.994 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:29.997 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:30.062 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total time run: 90.0251 2024-05-15T23:21:30.062 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total writes made: 153557 2024-05-15T23:21:30.062 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Write size: 65536 2024-05-15T23:21:30.062 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object size: 65536 2024-05-15T23:21:30.062 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Bandwidth (MB/sec): 106.607 2024-05-15T23:21:30.062 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Bandwidth: 60.4023 2024-05-15T23:21:30.062 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max bandwidth (MB/sec): 353.312 2024-05-15T23:21:30.063 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min bandwidth (MB/sec): 2.375 2024-05-15T23:21:30.063 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average IOPS: 1705 2024-05-15T23:21:30.063 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev IOPS: 966.437 2024-05-15T23:21:30.063 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max IOPS: 5653 2024-05-15T23:21:30.063 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min IOPS: 38 2024-05-15T23:21:30.063 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average Latency(s): 0.00937883 2024-05-15T23:21:30.063 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Latency(s): 0.0164287 2024-05-15T23:21:30.063 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max latency(s): 1.49715 2024-05-15T23:21:30.063 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min latency(s): 0.000997126 2024-05-15T23:21:30.069 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T23:21:30.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:21:30.134 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:30.163 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:21:30.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:30.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:21:30.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:30.303 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:30.354 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:30.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:30.362 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:30.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:30.501 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:30.529 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:30.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:30.640 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:21:30.689 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:30.722 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:21:30.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:30.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:21:30.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:30.854 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:31.040 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:21:31.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:31 smithi084 ceph-mon[39784]: pgmap v1118: 17 pgs: 3 active+recovering, 14 active+clean; 9.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 42 MiB/s wr, 697 op/s; 24501/305494 objects misplaced (8.020%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:21:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:31 smithi090 ceph-mon[44434]: pgmap v1118: 17 pgs: 3 active+recovering, 14 active+clean; 9.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 42 MiB/s wr, 697 op/s; 24501/305494 objects misplaced (8.020%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:21:31.584 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:31 smithi088 ceph-mon[44699]: pgmap v1118: 17 pgs: 3 active+recovering, 14 active+clean; 9.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 42 MiB/s wr, 697 op/s; 24501/305494 objects misplaced (8.020%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:21:31.702 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:31.705 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:31.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:31.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:31.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:21:31.844 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:31.871 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:21:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:31.874 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:21:31.975 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:21:32.008 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:32.658 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:21:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:32.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:21:32.762 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:21:32.814 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:32.863 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:21:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:32.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:21:32.972 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:21:33.009 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:33.032 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:21:33.049 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:21:33.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:33.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:21:33.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:33 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1027541652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:21:33.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:33 smithi088 ceph-mon[44699]: pgmap v1119: 17 pgs: 3 active+recovering, 14 active+clean; 9.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 102 B/s rd, 42 MiB/s wr, 749 op/s; 24335/305432 objects misplaced (7.967%); 4.8 MiB/s, 76 objects/s recovering 2024-05-15T23:21:33.157 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:33.235 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:33.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:33.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:33 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1027541652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:21:33.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:33 smithi090 ceph-mon[44434]: pgmap v1119: 17 pgs: 3 active+recovering, 14 active+clean; 9.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 102 B/s rd, 42 MiB/s wr, 749 op/s; 24335/305432 objects misplaced (7.967%); 4.8 MiB/s, 76 objects/s recovering 2024-05-15T23:21:33.339 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:33.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:33 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1027541652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:21:33.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:33 smithi084 ceph-mon[39784]: pgmap v1119: 17 pgs: 3 active+recovering, 14 active+clean; 9.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 102 B/s rd, 42 MiB/s wr, 749 op/s; 24335/305432 objects misplaced (7.967%); 4.8 MiB/s, 76 objects/s recovering 2024-05-15T23:21:33.371 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:33.498 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:33.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:33.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:33.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:33.641 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:33.679 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:33.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:33.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:33.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:33.826 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:33.846 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:33.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:33.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:33.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:33.985 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:34.019 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:34.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:34.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:34.039 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-05-15T23:21:34.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:21:34.151 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:34 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:21:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:34 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1027541652' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:21:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:34 smithi090 ceph-mon[44434]: osdmap e413: 12 total, 12 up, 9 in 2024-05-15T23:21:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:34 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/961181426' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:21:34.306 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:21:34.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:34 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:21:34.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:34 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1027541652' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:21:34.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:34 smithi084 ceph-mon[39784]: osdmap e413: 12 total, 12 up, 9 in 2024-05-15T23:21:34.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:34 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/961181426' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:21:34.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:34.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:21:34.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:34 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:21:34.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:34 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1027541652' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:21:34.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:34 smithi088 ceph-mon[44699]: osdmap e413: 12 total, 12 up, 9 in 2024-05-15T23:21:34.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:34 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/961181426' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:21:34.410 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:21:34.439 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:34.843 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:35.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:35 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/961181426' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:21:35.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:35 smithi084 ceph-mon[39784]: osdmap e414: 12 total, 12 up, 9 in 2024-05-15T23:21:35.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:35 smithi084 ceph-mon[39784]: pgmap v1122: 17 pgs: 3 active+recovering, 14 active+clean; 9.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 127 B/s rd, 32 MiB/s wr, 642 op/s; 24335/304958 objects misplaced (7.980%); 4.5 MiB/s, 71 objects/s recovering 2024-05-15T23:21:35.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:35 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/961181426' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:21:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:35 smithi090 ceph-mon[44434]: osdmap e414: 12 total, 12 up, 9 in 2024-05-15T23:21:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:35 smithi090 ceph-mon[44434]: pgmap v1122: 17 pgs: 3 active+recovering, 14 active+clean; 9.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 127 B/s rd, 32 MiB/s wr, 642 op/s; 24335/304958 objects misplaced (7.980%); 4.5 MiB/s, 71 objects/s recovering 2024-05-15T23:21:35.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:35 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/961181426' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:21:35.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:35 smithi088 ceph-mon[44699]: osdmap e414: 12 total, 12 up, 9 in 2024-05-15T23:21:35.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:35 smithi088 ceph-mon[44699]: pgmap v1122: 17 pgs: 3 active+recovering, 14 active+clean; 9.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 127 B/s rd, 32 MiB/s wr, 642 op/s; 24335/304958 objects misplaced (7.980%); 4.5 MiB/s, 71 objects/s recovering 2024-05-15T23:21:35.894 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:21:35.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:35.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:21:35.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 4 1 2024-05-15T23:21:35.999 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:36.044 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:36.073 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:36.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:36.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:36.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:21:36.159 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:21:36.160 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:21:36.160 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:21:36.160 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:21:36.160 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:21:36.178 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:21:36.233 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:37.294 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:21:37.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:37 smithi084 ceph-mon[39784]: pgmap v1123: 17 pgs: 3 active+recovering, 14 active+clean; 9.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 127 B/s rd, 20 MiB/s wr, 753 op/s; 24125/300212 objects misplaced (8.036%); 5.0 MiB/s, 80 objects/s recovering 2024-05-15T23:21:37.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:37.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:21:37.401 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:37.432 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:37.539 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:37.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:37.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:37.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:37 smithi090 ceph-mon[44434]: pgmap v1123: 17 pgs: 3 active+recovering, 14 active+clean; 9.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 127 B/s rd, 20 MiB/s wr, 753 op/s; 24125/300212 objects misplaced (8.036%); 5.0 MiB/s, 80 objects/s recovering 2024-05-15T23:21:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:37 smithi088 ceph-mon[44699]: pgmap v1123: 17 pgs: 3 active+recovering, 14 active+clean; 9.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 127 B/s rd, 20 MiB/s wr, 753 op/s; 24125/300212 objects misplaced (8.036%); 5.0 MiB/s, 80 objects/s recovering 2024-05-15T23:21:37.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:37.675 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:37.715 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:37.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:37.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:37.822 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:21:37.870 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:37.915 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:21:37.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:37.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:21:38.023 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:21:38.056 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:38.189 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:21:38.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:38.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:21:38.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T23:21:38.295 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:21:38.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:21:38.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:21:38.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:21:38.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:21:38.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:21:38.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:21:38.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:21:38.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:21:38.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:21:38.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:21:38.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:21:38.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:21:38.320 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:21:38.320 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:21:38.320 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:21:38.320 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:21:38.320 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:21:38.320 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:21:38.320 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:21:38.320 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:21:38.320 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:21:38.320 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:21:38.320 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:21:38.320 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:21:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:38 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2152170781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:21:38.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:38 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2152170781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:21:38.329 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:38.531 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:21:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:38.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:21:38.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:38 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2152170781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:21:38.578 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:38.592 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:38.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:21:38.672 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:38.700 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:21:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:38.705 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:21:38.806 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:38.863 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:38.915 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:38.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:38.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:39.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:21:39.054 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:39.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:39 smithi088 ceph-mon[44699]: pgmap v1124: 17 pgs: 3 active+recovering, 14 active+clean; 9.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 127 B/s rd, 5.0 MiB/s wr, 680 op/s; 23993/297188 objects misplaced (8.073%); 4.0 MiB/s, 63 objects/s recovering 2024-05-15T23:21:39.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:39 smithi084 ceph-mon[39784]: pgmap v1124: 17 pgs: 3 active+recovering, 14 active+clean; 9.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 127 B/s rd, 5.0 MiB/s wr, 680 op/s; 23993/297188 objects misplaced (8.073%); 4.0 MiB/s, 63 objects/s recovering 2024-05-15T23:21:39.505 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:21:39.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:39.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:21:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:39 smithi090 ceph-mon[44434]: pgmap v1124: 17 pgs: 3 active+recovering, 14 active+clean; 9.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 127 B/s rd, 5.0 MiB/s wr, 680 op/s; 23993/297188 objects misplaced (8.073%); 4.0 MiB/s, 63 objects/s recovering 2024-05-15T23:21:39.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:39.638 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:39.660 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:39.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:39.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:39.764 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:39.792 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:39.990 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:39.997 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:39.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:40.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:40.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:40.131 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:40.152 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:40.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:40.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:21:40.286 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:40.308 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:21:40.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:40.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:21:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:40.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:21:40.447 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:40.448 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:21:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:40.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:21:40.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:40.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:21:40.616 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:41.167 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:21:41.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:41.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:21:41.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:21:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:21:41.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:41.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:21:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:41 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:21:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:41 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:21:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:41 smithi084 ceph-mon[39784]: osdmap e415: 12 total, 12 up, 9 in 2024-05-15T23:21:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:41 smithi084 ceph-mon[39784]: pgmap v1126: 17 pgs: 3 active+recovering, 14 active+clean; 8.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 806 op/s; 23729/292602 objects misplaced (8.110%); 4.8 MiB/s, 76 objects/s recovering 2024-05-15T23:21:41.424 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:21:41.454 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:41.508 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:21:41.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:21:41.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:41.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:21:41.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:41 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:21:41.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:41 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:21:41.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:41 smithi090 ceph-mon[44434]: osdmap e415: 12 total, 12 up, 9 in 2024-05-15T23:21:41.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:41 smithi090 ceph-mon[44434]: pgmap v1126: 17 pgs: 3 active+recovering, 14 active+clean; 8.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 806 op/s; 23729/292602 objects misplaced (8.110%); 4.8 MiB/s, 76 objects/s recovering 2024-05-15T23:21:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:41 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:21:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:41 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:21:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:41 smithi088 ceph-mon[44699]: osdmap e415: 12 total, 12 up, 9 in 2024-05-15T23:21:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:41 smithi088 ceph-mon[44699]: pgmap v1126: 17 pgs: 3 active+recovering, 14 active+clean; 8.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 806 op/s; 23729/292602 objects misplaced (8.110%); 4.8 MiB/s, 76 objects/s recovering 2024-05-15T23:21:41.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:21:41.653 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:41.681 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:21:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:41.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:21:41.793 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:21:41.820 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:42.269 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:21:42.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:42.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:21:42.372 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:42.428 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:42.465 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:42.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:42.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:42.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:42.606 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:42.639 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:42.644 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:42.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:21:42.782 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:42.804 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:21:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:42.808 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:21:42.909 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:21:42.957 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:42.985 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:21:42.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:42.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:21:43.036 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:43 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:21:43.037 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:43 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:21:43.037 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:43 smithi090 ceph-mon[44434]: pgmap v1127: 17 pgs: 3 active+recovering, 14 active+clean; 8.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 859 op/s; 23572/290044 objects misplaced (8.127%); 5.3 MiB/s, 85 objects/s recovering 2024-05-15T23:21:43.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:21:43.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:43 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:21:43.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:43 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:21:43.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:43 smithi084 ceph-mon[39784]: pgmap v1127: 17 pgs: 3 active+recovering, 14 active+clean; 8.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 859 op/s; 23572/290044 objects misplaced (8.127%); 5.3 MiB/s, 85 objects/s recovering 2024-05-15T23:21:43.132 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:43.156 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:21:43.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:43.160 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:21:43.261 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:43.303 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:43 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:21:43.303 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:43 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:21:43.304 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:43 smithi088 ceph-mon[44699]: pgmap v1127: 17 pgs: 3 active+recovering, 14 active+clean; 8.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 859 op/s; 23572/290044 objects misplaced (8.127%); 5.3 MiB/s, 85 objects/s recovering 2024-05-15T23:21:43.306 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:43.339 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:43.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:43.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:43.453 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:21:43.484 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:43.508 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:21:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:43.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:21:43.617 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:21:43.674 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:43.709 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:21:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:43.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:21:43.714 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:43.785 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:21:43.785 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:21:43.814 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:21:43.847 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:44.044 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:21:44.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:44 smithi090 ceph-mon[44434]: osdmap e416: 12 total, 12 up, 9 in 2024-05-15T23:21:44.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3855763956' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:21:44.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:44 smithi088 ceph-mon[44699]: osdmap e416: 12 total, 12 up, 9 in 2024-05-15T23:21:44.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3855763956' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:21:44.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:44 smithi084 ceph-mon[39784]: osdmap e416: 12 total, 12 up, 9 in 2024-05-15T23:21:44.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3855763956' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:21:44.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T23:21:45.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:45 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:21:45.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3855763956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:21:45.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:45 smithi090 ceph-mon[44434]: osdmap e417: 12 total, 12 up, 9 in 2024-05-15T23:21:45.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:45 smithi090 ceph-mon[44434]: pgmap v1130: 17 pgs: 3 active+recovering, 14 active+clean; 8.8 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 629 op/s; 23572/289628 objects misplaced (8.139%); 4.4 MiB/s, 70 objects/s recovering 2024-05-15T23:21:45.329 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:45 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:21:45.329 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3855763956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:21:45.329 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:45 smithi088 ceph-mon[44699]: osdmap e417: 12 total, 12 up, 9 in 2024-05-15T23:21:45.329 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:45 smithi088 ceph-mon[44699]: pgmap v1130: 17 pgs: 3 active+recovering, 14 active+clean; 8.8 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 629 op/s; 23572/289628 objects misplaced (8.139%); 4.4 MiB/s, 70 objects/s recovering 2024-05-15T23:21:45.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:45 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:21:45.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3855763956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:21:45.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:45 smithi084 ceph-mon[39784]: osdmap e417: 12 total, 12 up, 9 in 2024-05-15T23:21:45.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:45 smithi084 ceph-mon[39784]: pgmap v1130: 17 pgs: 3 active+recovering, 14 active+clean; 8.8 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 629 op/s; 23572/289628 objects misplaced (8.139%); 4.4 MiB/s, 70 objects/s recovering 2024-05-15T23:21:46.404 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:21:46.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:46.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:21:46.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:46.543 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:46.992 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:46.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:46.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:47.033 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:47.070 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2024-05-15T23:21:47.098 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:21:47.126 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:47.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:47 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:21:47.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:47 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:21:47.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:47 smithi084 ceph-mon[39784]: osdmap e418: 12 total, 12 up, 9 in 2024-05-15T23:21:47.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:47 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/738955903' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:21:47.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:47 smithi084 ceph-mon[39784]: from='client.25987 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:21:47.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:47 smithi084 ceph-mon[39784]: pgmap v1132: 17 pgs: 3 active+recovering, 14 active+clean; 8.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 583 op/s; 23207/285600 objects misplaced (8.126%); 5.4 MiB/s, 86 objects/s recovering 2024-05-15T23:21:47.375 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:21:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:47.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:21:47.480 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:21:47.519 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:47 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:21:47.519 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:47 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:21:47.519 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:47 smithi088 ceph-mon[44699]: osdmap e418: 12 total, 12 up, 9 in 2024-05-15T23:21:47.519 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:47 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/738955903' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:21:47.519 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:47 smithi088 ceph-mon[44699]: from='client.25987 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:21:47.519 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:47 smithi088 ceph-mon[44699]: pgmap v1132: 17 pgs: 3 active+recovering, 14 active+clean; 8.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 583 op/s; 23207/285600 objects misplaced (8.126%); 5.4 MiB/s, 86 objects/s recovering 2024-05-15T23:21:47.521 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:47 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:21:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:47 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:21:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:47 smithi090 ceph-mon[44434]: osdmap e418: 12 total, 12 up, 9 in 2024-05-15T23:21:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:47 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/738955903' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:21:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:47 smithi090 ceph-mon[44434]: from='client.25987 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:21:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:47 smithi090 ceph-mon[44434]: pgmap v1132: 17 pgs: 3 active+recovering, 14 active+clean; 8.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 583 op/s; 23207/285600 objects misplaced (8.126%); 5.4 MiB/s, 86 objects/s recovering 2024-05-15T23:21:47.558 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:21:47.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:47.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:21:47.665 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:21:47.716 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:47.757 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:21:47.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:47.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:21:47.872 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:47.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:21:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:47.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:47.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:47.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 5 1 2024-05-15T23:21:48.041 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:21:48.077 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:48.100 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:21:48.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:48.105 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:21:48.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:48 smithi084 ceph-mon[39784]: from='client.25987 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:21:48.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:48 smithi084 ceph-mon[39784]: osdmap e419: 12 total, 12 up, 9 in 2024-05-15T23:21:48.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:48.239 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:48 smithi090 ceph-mon[44434]: from='client.25987 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:21:48.239 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:48 smithi090 ceph-mon[44434]: osdmap e419: 12 total, 12 up, 9 in 2024-05-15T23:21:48.243 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:48.270 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:48.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:48.273 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:48.374 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:21:48.403 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:48.425 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:21:48.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:48.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:21:48.530 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:21:48.558 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:48.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:48 smithi088 ceph-mon[44699]: from='client.25987 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:21:48.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:48 smithi088 ceph-mon[44699]: osdmap e419: 12 total, 12 up, 9 in 2024-05-15T23:21:48.807 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:21:48.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:48.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:21:48.911 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:21:48.954 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:48.980 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:21:48.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:48.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:21:49.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:21:49.112 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:49.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:49 smithi088 ceph-mon[44699]: pgmap v1134: 17 pgs: 3 active+recovering, 14 active+clean; 8.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 598 op/s; 23055/282912 objects misplaced (8.149%); 5.4 MiB/s, 86 objects/s recovering 2024-05-15T23:21:49.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:49 smithi084 ceph-mon[39784]: pgmap v1134: 17 pgs: 3 active+recovering, 14 active+clean; 8.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 598 op/s; 23055/282912 objects misplaced (8.149%); 5.4 MiB/s, 86 objects/s recovering 2024-05-15T23:21:49.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:49 smithi090 ceph-mon[44434]: pgmap v1134: 17 pgs: 3 active+recovering, 14 active+clean; 8.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 598 op/s; 23055/282912 objects misplaced (8.149%); 5.4 MiB/s, 86 objects/s recovering 2024-05-15T23:21:49.667 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:21:49.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:49.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:21:49.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:21:49.802 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:49.825 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:21:49.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:49.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:21:49.931 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:49.965 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:50.033 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:21:50.033 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:21:50.033 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:21:50.033 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:21:50.033 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:21:50.033 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:21:50.033 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:21:50.034 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:21:50.034 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:21:50.058 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:21:50.081 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:21:50.103 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:21:50.104 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:21:50.104 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:21:50.104 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:21:50.105 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:21:50.105 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:21:50.105 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:21:50.105 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:21:50.105 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:21:50.105 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:21:50.105 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:21:50.105 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:21:50.105 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:21:50.289 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:21:50.289 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:21:50.289 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-05-15T23:21:50.289 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:21:50.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/535623261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:21:50.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/535623261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:21:50.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/535623261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:21:51.126 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:51.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:51.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:51.260 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:51.280 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:51.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:51.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:51 smithi084 ceph-mon[39784]: pgmap v1135: 17 pgs: 3 active+recovering, 14 active+clean; 8.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 810 op/s; 22662/278788 objects misplaced (8.129%); 8.2 MiB/s, 130 objects/s recovering 2024-05-15T23:21:51.386 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:21:51.414 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:51.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:51 smithi090 ceph-mon[44434]: pgmap v1135: 17 pgs: 3 active+recovering, 14 active+clean; 8.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 810 op/s; 22662/278788 objects misplaced (8.129%); 8.2 MiB/s, 130 objects/s recovering 2024-05-15T23:21:51.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:21:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:51 smithi088 ceph-mon[44699]: pgmap v1135: 17 pgs: 3 active+recovering, 14 active+clean; 8.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 810 op/s; 22662/278788 objects misplaced (8.129%); 8.2 MiB/s, 130 objects/s recovering 2024-05-15T23:21:51.595 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:51.597 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:51.612 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:21:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:51.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:21:51.637 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T23:21:51.718 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:51.752 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:51.875 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:51.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:51.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:51.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:52.019 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:52.041 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:52.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:52.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:52.149 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:21:52.185 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:52.206 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:21:52.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:52.214 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:21:52.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:52 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:21:52.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:52 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:21:52.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:21:52.355 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:52.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:52 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:21:52.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:52 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:21:52.380 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:21:52.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:52.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:21:52.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:21:52.515 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:52.537 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:21:52.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:52.542 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:21:52.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:52 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:21:52.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:52 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:21:52.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:52.673 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:53.127 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:21:53.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:53 smithi084 ceph-mon[39784]: osdmap e420: 12 total, 12 up, 9 in 2024-05-15T23:21:53.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:53 smithi084 ceph-mon[39784]: pgmap v1137: 17 pgs: 3 active+recovering, 14 active+clean; 8.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 665 op/s; 22488/276374 objects misplaced (8.137%); 6.5 MiB/s, 103 objects/s recovering 2024-05-15T23:21:53.525 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:53.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:53.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:53 smithi090 ceph-mon[44434]: osdmap e420: 12 total, 12 up, 9 in 2024-05-15T23:21:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:53 smithi090 ceph-mon[44434]: pgmap v1137: 17 pgs: 3 active+recovering, 14 active+clean; 8.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 665 op/s; 22488/276374 objects misplaced (8.137%); 6.5 MiB/s, 103 objects/s recovering 2024-05-15T23:21:53.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:53 smithi088 ceph-mon[44699]: osdmap e420: 12 total, 12 up, 9 in 2024-05-15T23:21:53.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:53 smithi088 ceph-mon[44699]: pgmap v1137: 17 pgs: 3 active+recovering, 14 active+clean; 8.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 665 op/s; 22488/276374 objects misplaced (8.137%); 6.5 MiB/s, 103 objects/s recovering 2024-05-15T23:21:53.629 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:21:53.691 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:53.717 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:21:53.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:53.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:21:53.820 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:53.851 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:54.133 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:54.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:54.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:54.237 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:21:54.271 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:54.301 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:21:54.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:54.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:21:54.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:54 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:21:54.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:54 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:21:54.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:54 smithi084 ceph-mon[39784]: osdmap e421: 12 total, 12 up, 9 in 2024-05-15T23:21:54.410 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:21:54.440 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:21:54.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:54.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:21:54.498 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:54.509 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:21:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:54 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:21:54.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:54 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:21:54.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:54 smithi090 ceph-mon[44434]: osdmap e421: 12 total, 12 up, 9 in 2024-05-15T23:21:54.568 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:21:54.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:54 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:21:54.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:54 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:21:54.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:54 smithi088 ceph-mon[44699]: osdmap e421: 12 total, 12 up, 9 in 2024-05-15T23:21:54.634 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:54.658 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:21:54.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:54.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:21:54.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:21:54.797 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:54.822 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:21:54.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:54.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:21:54.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:54.959 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:54.985 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:54.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:54.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:55.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:21:55.120 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:55.129 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-05-15T23:21:55.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:55 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3544960905' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:21:55.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:55 smithi084 ceph-mon[39784]: pgmap v1139: 17 pgs: 3 active+recovering, 14 active+clean; 8.4 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 552 op/s; 22488/275922 objects misplaced (8.150%); 5.7 MiB/s, 90 objects/s recovering 2024-05-15T23:21:55.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:55 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3544960905' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:21:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:55 smithi090 ceph-mon[44434]: pgmap v1139: 17 pgs: 3 active+recovering, 14 active+clean; 8.4 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 552 op/s; 22488/275922 objects misplaced (8.150%); 5.7 MiB/s, 90 objects/s recovering 2024-05-15T23:21:55.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:55 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3544960905' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:21:55.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:55 smithi088 ceph-mon[44699]: pgmap v1139: 17 pgs: 3 active+recovering, 14 active+clean; 8.4 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 552 op/s; 22488/275922 objects misplaced (8.150%); 5.7 MiB/s, 90 objects/s recovering 2024-05-15T23:21:56.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3544960905' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:21:56.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:56 smithi090 ceph-mon[44434]: osdmap e422: 12 total, 12 up, 9 in 2024-05-15T23:21:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3544960905' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:21:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:56 smithi088 ceph-mon[44699]: osdmap e422: 12 total, 12 up, 9 in 2024-05-15T23:21:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3544960905' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:21:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:56 smithi084 ceph-mon[39784]: osdmap e422: 12 total, 12 up, 9 in 2024-05-15T23:21:56.672 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:21:56.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:56.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:21:56.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:56.808 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:56.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:56.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:56.934 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:56.977 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:57.004 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:57.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:57.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:57.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:57.140 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:57 smithi090 ceph-mon[44434]: pgmap v1141: 17 pgs: 3 active+recovering, 14 active+clean; 8.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 589 op/s; 22131/271712 objects misplaced (8.145%); 5.5 MiB/s, 88 objects/s recovering 2024-05-15T23:21:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:57.202 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:57.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:57.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:21:57.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:57 smithi088 ceph-mon[44699]: pgmap v1141: 17 pgs: 3 active+recovering, 14 active+clean; 8.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 589 op/s; 22131/271712 objects misplaced (8.145%); 5.5 MiB/s, 88 objects/s recovering 2024-05-15T23:21:57.354 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:57.405 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:21:57.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:57.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:21:57.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:57 smithi084 ceph-mon[39784]: pgmap v1141: 17 pgs: 3 active+recovering, 14 active+clean; 8.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 589 op/s; 22131/271712 objects misplaced (8.145%); 5.5 MiB/s, 88 objects/s recovering 2024-05-15T23:21:57.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:21:57.544 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:57.695 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:21:57.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:57.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:21:57.799 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:21:57.834 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:57.859 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:21:57.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:57.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:21:57.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:21:57.996 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:58.022 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:21:58.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:58.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:21:58.128 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:21:58.137 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:21:58.137 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:21:58.137 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:21:58.137 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:21:58.137 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:21:58.164 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:58.188 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:21:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:58.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:21:58.292 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:21:58.295 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:21:58.344 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:58.367 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:21:58.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:58 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2601856191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:21:58.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:58.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:21:58.471 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:21:58.502 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:58.503 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:58 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2601856191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:21:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:21:58.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 7 1 2024-05-15T23:21:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:58.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:21:58.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:58 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2601856191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:21:58.654 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:21:58.683 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:58.709 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:21:58.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:58.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:21:58.813 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:21:58.847 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:58.870 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:21:58.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:21:58.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:21:58.974 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:21:59.002 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:21:59.297 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:59 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:21:59.560 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:59 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:21:59.560 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:59 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2601856191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:21:59.560 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:59 smithi090 ceph-mon[44434]: osdmap e423: 12 total, 12 up, 9 in 2024-05-15T23:21:59.560 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:21:59 smithi090 ceph-mon[44434]: pgmap v1143: 17 pgs: 3 active+recovering, 14 active+clean; 8.2 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 604 op/s; 21963/269124 objects misplaced (8.161%); 5.5 MiB/s, 87 objects/s recovering 2024-05-15T23:21:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:59 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2601856191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:21:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:59 smithi088 ceph-mon[44699]: osdmap e423: 12 total, 12 up, 9 in 2024-05-15T23:21:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:21:59 smithi088 ceph-mon[44699]: pgmap v1143: 17 pgs: 3 active+recovering, 14 active+clean; 8.2 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 604 op/s; 21963/269124 objects misplaced (8.161%); 5.5 MiB/s, 87 objects/s recovering 2024-05-15T23:21:59.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:59 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:21:59.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:59 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2601856191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:21:59.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:59 smithi084 ceph-mon[39784]: osdmap e423: 12 total, 12 up, 9 in 2024-05-15T23:21:59.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:21:59 smithi084 ceph-mon[39784]: pgmap v1143: 17 pgs: 3 active+recovering, 14 active+clean; 8.2 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 604 op/s; 21963/269124 objects misplaced (8.161%); 5.5 MiB/s, 87 objects/s recovering 2024-05-15T23:22:00.053 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:22:00.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:00.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:22:00.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:00.199 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:00.224 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:00.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:00.227 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:00.328 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:00.375 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:00.407 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:00.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:00.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:00.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:00.567 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:00.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:00.570 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:00.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:22:00.701 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:00.722 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:22:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:00.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:22:00.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:00.858 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:00.886 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:00.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:00.889 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:00.991 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:22:01.059 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:01.096 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:22:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:01.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:22:01.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:22:01.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:22:01.258 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:00 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1314189506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:01.262 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:00 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1314189506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:01.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:00 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1314189506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:01.378 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:22:01.378 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:22:01.378 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:22:01.378 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:22:01.378 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:22:01.378 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:22:01.378 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:22:01.378 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:22:01.378 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:22:01.378 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:22:01.379 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:22:01.379 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:22:01.379 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:22:01.379 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:22:01.379 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:22:01.379 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:22:01.379 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:22:01.379 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:22:01.379 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:22:01.379 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:22:01.379 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:22:01.379 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:22:01.380 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:22:01.380 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:22:01.981 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:02.018 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T23:22:02.020 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:22:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:02.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:22:02.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:02.160 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:02.183 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:02.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:02.187 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:02.234 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:01 smithi084 ceph-mon[39784]: pgmap v1144: 17 pgs: 3 active+recovering, 14 active+clean; 8.1 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 733 op/s; 21563/264820 objects misplaced (8.143%); 7.3 MiB/s, 117 objects/s recovering 2024-05-15T23:22:02.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.288 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:22:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:01 smithi090 ceph-mon[44434]: pgmap v1144: 17 pgs: 3 active+recovering, 14 active+clean; 8.1 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 733 op/s; 21563/264820 objects misplaced (8.143%); 7.3 MiB/s, 117 objects/s recovering 2024-05-15T23:22:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:01 smithi088 ceph-mon[44699]: pgmap v1144: 17 pgs: 3 active+recovering, 14 active+clean; 8.1 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 733 op/s; 21563/264820 objects misplaced (8.143%); 7.3 MiB/s, 117 objects/s recovering 2024-05-15T23:22:02.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:02.329 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:02.368 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:22:02.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:02.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:22:02.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:22:02.504 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:02.754 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:22:02.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:02.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:22:02.858 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:02.910 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:02.933 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:02.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:03.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:03.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:03.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:03 smithi084 ceph-mon[39784]: pgmap v1145: 17 pgs: 3 active+recovering, 14 active+clean; 8.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 858 op/s; 21391/262184 objects misplaced (8.159%); 8.6 MiB/s, 137 objects/s recovering 2024-05-15T23:22:03.074 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:03.102 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:03.109 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:03.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:03.265 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:03.267 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:03.268 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:03 smithi090 ceph-mon[44434]: pgmap v1145: 17 pgs: 3 active+recovering, 14 active+clean; 8.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 858 op/s; 21391/262184 objects misplaced (8.159%); 8.6 MiB/s, 137 objects/s recovering 2024-05-15T23:22:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:03 smithi088 ceph-mon[44699]: pgmap v1145: 17 pgs: 3 active+recovering, 14 active+clean; 8.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 858 op/s; 21391/262184 objects misplaced (8.159%); 8.6 MiB/s, 137 objects/s recovering 2024-05-15T23:22:03.369 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:03.411 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:03.435 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:03.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:03.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:03.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:03.567 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:03.593 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:03.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:03.596 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:03.697 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:22:03.726 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:04.175 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:22:04.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:04.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:22:04.279 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:22:04.307 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:04.779 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:22:05.258 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:05 smithi088 ceph-mon[44699]: pgmap v1146: 17 pgs: 3 active+recovering, 14 active+clean; 8.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 715 op/s; 21391/261796 objects misplaced (8.171%); 6.9 MiB/s, 111 objects/s recovering 2024-05-15T23:22:05.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:05 smithi084 ceph-mon[39784]: pgmap v1146: 17 pgs: 3 active+recovering, 14 active+clean; 8.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 715 op/s; 21391/261796 objects misplaced (8.171%); 6.9 MiB/s, 111 objects/s recovering 2024-05-15T23:22:05.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:05 smithi090 ceph-mon[44434]: pgmap v1146: 17 pgs: 3 active+recovering, 14 active+clean; 8.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 715 op/s; 21391/261796 objects misplaced (8.171%); 6.9 MiB/s, 111 objects/s recovering 2024-05-15T23:22:05.991 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:22:06.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:06.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:22:06.092 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-05-15T23:22:06.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:22:06.134 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:06.233 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:22:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:06.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:22:06.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:06 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:22:06.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:06 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:22:06.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:06 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2847140817' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:22:06.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:22:06.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:22:06.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:22:06.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:22:06.337 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:22:06.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:06 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:22:06.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:06 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:22:06.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:06 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2847140817' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:22:06.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:22:06.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:22:06.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:22:06.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:22:06.384 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:06.420 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:22:06.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:06.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:22:06.528 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:06.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:06 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:22:06.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:06 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:22:06.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:06 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2847140817' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:22:06.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:22:06.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:22:06.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:22:06.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:22:06.570 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:06.599 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:06.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:06.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:06.703 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:22:06.732 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:06.930 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:22:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:06.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:22:06.958 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:07.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:07.075 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:07.106 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:07.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:07.111 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:07.211 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:07.248 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:07.277 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:07.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:07.283 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:07 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2847140817' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:22:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:07 smithi084 ceph-mon[39784]: osdmap e424: 12 total, 12 up, 9 in 2024-05-15T23:22:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:22:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:22:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:22:07.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:07 smithi084 ceph-mon[39784]: pgmap v1148: 17 pgs: 3 active+recovering, 14 active+clean; 7.9 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 640 op/s; 20982/257968 objects misplaced (8.134%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T23:22:07.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:07.418 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:07 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2847140817' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:22:07.418 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:07 smithi090 ceph-mon[44434]: osdmap e424: 12 total, 12 up, 9 in 2024-05-15T23:22:07.418 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:22:07.418 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:22:07.418 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:22:07.418 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:07 smithi090 ceph-mon[44434]: pgmap v1148: 17 pgs: 3 active+recovering, 14 active+clean; 7.9 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 640 op/s; 20982/257968 objects misplaced (8.134%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T23:22:07.419 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:07.447 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:07.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:07.457 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:07.559 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:22:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:07 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2847140817' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:22:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:07 smithi088 ceph-mon[44699]: osdmap e424: 12 total, 12 up, 9 in 2024-05-15T23:22:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:22:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:22:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:22:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:07 smithi088 ceph-mon[44699]: pgmap v1148: 17 pgs: 3 active+recovering, 14 active+clean; 7.9 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 640 op/s; 20982/257968 objects misplaced (8.134%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T23:22:07.604 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:07.649 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:22:07.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:07.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:22:07.756 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:22:07.787 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:08.055 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:22:08.055 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:22:08.552 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:22:08.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/654810414' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:22:08.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:08 smithi090 ceph-mon[44434]: from='client.36038 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:22:08.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/654810414' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:22:08.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:08 smithi088 ceph-mon[44699]: from='client.36038 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:22:08.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/654810414' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:22:08.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:08 smithi084 ceph-mon[39784]: from='client.36038 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:22:09.121 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 8 1 2024-05-15T23:22:09.456 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T23:22:09.734 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:09.747 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:22:09.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:09.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:22:09.752 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:09 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:22:09.752 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:09 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:22:09.752 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:09 smithi084 ceph-mon[39784]: from='client.36038 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:22:09.752 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:09 smithi084 ceph-mon[39784]: osdmap e425: 12 total, 12 up, 9 in 2024-05-15T23:22:09.752 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:09 smithi084 ceph-mon[39784]: pgmap v1150: 17 pgs: 3 active+recovering, 14 active+clean; 7.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 590 op/s; 20808/255370 objects misplaced (8.148%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T23:22:09.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:09 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:22:09.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:09 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:22:09.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:09 smithi088 ceph-mon[44699]: from='client.36038 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:22:09.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:09 smithi088 ceph-mon[44699]: osdmap e425: 12 total, 12 up, 9 in 2024-05-15T23:22:09.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:09 smithi088 ceph-mon[44699]: pgmap v1150: 17 pgs: 3 active+recovering, 14 active+clean; 7.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 590 op/s; 20808/255370 objects misplaced (8.148%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T23:22:09.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:09.896 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:09 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:22:09.896 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:09 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:22:09.896 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:09 smithi090 ceph-mon[44434]: from='client.36038 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:22:09.896 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:09 smithi090 ceph-mon[44434]: osdmap e425: 12 total, 12 up, 9 in 2024-05-15T23:22:09.896 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:09 smithi090 ceph-mon[44434]: pgmap v1150: 17 pgs: 3 active+recovering, 14 active+clean; 7.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 590 op/s; 20808/255370 objects misplaced (8.148%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T23:22:09.899 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:09.930 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:09.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:10.035 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:22:10.065 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:11.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:11 smithi088 ceph-mon[44699]: pgmap v1151: 17 pgs: 3 active+recovering, 14 active+clean; 7.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 707 op/s; 20424/250868 objects misplaced (8.141%); 7.6 MiB/s, 120 objects/s recovering 2024-05-15T23:22:11.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:11 smithi084 ceph-mon[39784]: pgmap v1151: 17 pgs: 3 active+recovering, 14 active+clean; 7.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 707 op/s; 20424/250868 objects misplaced (8.141%); 7.6 MiB/s, 120 objects/s recovering 2024-05-15T23:22:11.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:11 smithi090 ceph-mon[44434]: pgmap v1151: 17 pgs: 3 active+recovering, 14 active+clean; 7.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 707 op/s; 20424/250868 objects misplaced (8.141%); 7.6 MiB/s, 120 objects/s recovering 2024-05-15T23:22:11.716 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:22:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:11.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:22:11.823 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:11.877 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:11.915 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:11.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:11.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:12.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:22:12.070 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:12.092 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:22:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:12.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:22:12.119 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:12.195 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:22:12.225 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:12.481 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:22:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:12.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:22:12.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:12.618 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:12.638 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:12.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:12.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:12.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:12.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:12.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:22:12.962 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:12.988 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:22:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:12.992 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:22:13.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:13 smithi084 ceph-mon[39784]: pgmap v1152: 17 pgs: 3 active+recovering, 14 active+clean; 7.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 854 op/s; 20256/248124 objects misplaced (8.164%); 8.9 MiB/s, 141 objects/s recovering 2024-05-15T23:22:13.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:13.129 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:13.160 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:13.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:13.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:13.184 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:22:13.267 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:22:13.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:13 smithi090 ceph-mon[44434]: pgmap v1152: 17 pgs: 3 active+recovering, 14 active+clean; 7.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 854 op/s; 20256/248124 objects misplaced (8.164%); 8.9 MiB/s, 141 objects/s recovering 2024-05-15T23:22:13.313 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:13 smithi088 ceph-mon[44699]: pgmap v1152: 17 pgs: 3 active+recovering, 14 active+clean; 7.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 854 op/s; 20256/248124 objects misplaced (8.164%); 8.9 MiB/s, 141 objects/s recovering 2024-05-15T23:22:13.314 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:13.347 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:22:13.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:13.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:22:13.452 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:22:13.483 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:13.502 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:22:13.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:13.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:22:13.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:13.639 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:13.673 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:13.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:13.676 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:13.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:22:13.824 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:13.848 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:22:13.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:13.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:22:13.954 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:13.989 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:14.015 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:14.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:14.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:14.043 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.8 to 1 (10000) 2024-05-15T23:22:14.044 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:14 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/804564976' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:14.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:14 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/804564976' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:14.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:14 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1970670739' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:22:14.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:14.118 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:22:14.150 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:14.181 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:22:14.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:14.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:22:14.187 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:22:14.187 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:22:14.187 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:22:14.187 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:22:14.187 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:22:14.187 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:22:14.187 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:22:14.187 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:22:14.187 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:22:14.189 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:22:14.189 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:22:14.189 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:22:14.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:22:14.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:22:14.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:22:14.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:22:14.190 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:22:14.190 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:22:14.190 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:22:14.190 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:22:14.190 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:22:14.190 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:22:14.190 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:22:14.190 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:22:14.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:22:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:14 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1970670739' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:22:14.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:14.315 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:14 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/804564976' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:14 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1970670739' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:22:14.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:14.389 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:22:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:14.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:22:14.492 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:14.540 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:14.563 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:14.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:14.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:14.667 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:14.722 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:14.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:14.828 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:14.881 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:14.905 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:14.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:14.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:15.013 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:22:15.049 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:15.076 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:22:15.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:15.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:22:15.184 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:22:15.210 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:15 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1970670739' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:22:15.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:15 smithi084 ceph-mon[39784]: osdmap e426: 12 total, 12 up, 9 in 2024-05-15T23:22:15.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:15 smithi084 ceph-mon[39784]: pgmap v1154: 17 pgs: 3 active+recovering, 14 active+clean; 7.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 639 op/s; 20256/247730 objects misplaced (8.177%); 5.7 MiB/s, 90 objects/s recovering 2024-05-15T23:22:15.213 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:15.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.310 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.310 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.310 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:15 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1970670739' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:22:15.310 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:15 smithi090 ceph-mon[44434]: osdmap e426: 12 total, 12 up, 9 in 2024-05-15T23:22:15.310 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.310 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.310 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:15 smithi090 ceph-mon[44434]: pgmap v1154: 17 pgs: 3 active+recovering, 14 active+clean; 7.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 639 op/s; 20256/247730 objects misplaced (8.177%); 5.7 MiB/s, 90 objects/s recovering 2024-05-15T23:22:15.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:15 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1970670739' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:22:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:15 smithi088 ceph-mon[44699]: osdmap e426: 12 total, 12 up, 9 in 2024-05-15T23:22:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:15.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:15 smithi088 ceph-mon[44699]: pgmap v1154: 17 pgs: 3 active+recovering, 14 active+clean; 7.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 639 op/s; 20256/247730 objects misplaced (8.177%); 5.7 MiB/s, 90 objects/s recovering 2024-05-15T23:22:15.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 11 1 2024-05-15T23:22:16.646 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:16.656 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:16.663 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:22:16.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:16.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:22:16.698 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:22:16.699 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:22:16.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:16.699 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "MOSDPGPush(5.3 426/407 [PushOp(5:c975f973:::benchmark_data_smithi090_116470_object33430:head, version: 382'2084, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c975f973:::benchmark_data_smithi090_116470_object33430:head@382'2084, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-05-15T23:22:16.699 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:22:16.696743+0000", 2024-05-15T23:22:16.699 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.00055522099999999995, 2024-05-15T23:22:16.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00060356199999999998, 2024-05-15T23:22:16.699 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:16.699 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:16.699 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-05-15T23:22:16.699 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:16.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:16.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:16.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:16.696743+0000", 2024-05-15T23:22:16.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:16.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:16.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:16.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:16.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:16.696743+0000", 2024-05-15T23:22:16.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:16.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:16.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:16.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:16.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:16.696745+0000", 2024-05-15T23:22:16.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.029e-06 2024-05-15T23:22:16.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:16.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:16.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:16.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:16.696798+0000", 2024-05-15T23:22:16.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3149000000000002e-05 2024-05-15T23:22:16.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:16.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:16.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:16.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:16.696813+0000", 2024-05-15T23:22:16.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5676999999999999e-05 2024-05-15T23:22:16.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:16.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:16.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:16.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:16.696825+0000", 2024-05-15T23:22:16.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1440999999999999e-05 2024-05-15T23:22:16.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:16.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:16.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:16.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:16.696888+0000", 2024-05-15T23:22:16.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3579999999999998e-05 2024-05-15T23:22:16.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:16.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:16.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:16.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:16.696899+0000", 2024-05-15T23:22:16.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0236e-05 2024-05-15T23:22:16.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:16.702 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:16.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:16.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:16.702 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:22:16.702 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 1 2024-05-15T23:22:16.703 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:22:16.768 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:16.826 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:16.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:16.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:16.969 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:16.991 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:16.994 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:17.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:17.127 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:17.147 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:17.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:17.150 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:17.251 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:17.265 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:17 smithi084 ceph-mon[39784]: pgmap v1155: 17 pgs: 3 active+recovering, 14 active+clean; 7.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 685 op/s; 19856/243788 objects misplaced (8.145%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T23:22:17.283 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:17.316 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:17.320 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:17 smithi090 ceph-mon[44434]: pgmap v1155: 17 pgs: 3 active+recovering, 14 active+clean; 7.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 685 op/s; 19856/243788 objects misplaced (8.145%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T23:22:17.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:17 smithi088 ceph-mon[44699]: pgmap v1155: 17 pgs: 3 active+recovering, 14 active+clean; 7.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 685 op/s; 19856/243788 objects misplaced (8.145%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T23:22:17.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:17.322 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:17.423 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:17.466 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:17.502 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:17.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:17.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:17.626 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:17.659 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:17.702 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:17.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:17.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:22:17.837 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:18.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:22:18.390 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:22:18.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:18.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:22:18.396 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T23:22:18.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:18.525 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:18.554 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:18.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:18.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:18.658 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:22:18.715 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:18.782 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:22:18.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:18.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:22:18.895 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:18.951 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:18.989 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:18.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:18.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:19.093 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:22:19.122 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:19.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:19 smithi088 ceph-mon[44699]: pgmap v1156: 17 pgs: 3 active+recovering, 14 active+clean; 7.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 712 op/s; 19671/241124 objects misplaced (8.158%); 7.1 MiB/s, 113 objects/s recovering 2024-05-15T23:22:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:19 smithi084 ceph-mon[39784]: pgmap v1156: 17 pgs: 3 active+recovering, 14 active+clean; 7.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 712 op/s; 19671/241124 objects misplaced (8.158%); 7.1 MiB/s, 113 objects/s recovering 2024-05-15T23:22:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:19 smithi090 ceph-mon[44434]: pgmap v1156: 17 pgs: 3 active+recovering, 14 active+clean; 7.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 712 op/s; 19671/241124 objects misplaced (8.158%); 7.1 MiB/s, 113 objects/s recovering 2024-05-15T23:22:19.973 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:22:19.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:19.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:22:19.983 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:20.082 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:22:20.114 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:20.138 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:22:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:20.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:22:20.227 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:20 smithi084 ceph-mon[39784]: 5.1 scrub starts 2024-05-15T23:22:20.246 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:22:20.287 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:20.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:20 smithi090 ceph-mon[44434]: 5.1 scrub starts 2024-05-15T23:22:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:20 smithi088 ceph-mon[44699]: 5.1 scrub starts 2024-05-15T23:22:21.097 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:21 smithi084 ceph-mon[39784]: 5.1 scrub ok 2024-05-15T23:22:21.097 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:21 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1470875325' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:22:21.097 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:21 smithi084 ceph-mon[39784]: pgmap v1157: 17 pgs: 3 active+recovering, 14 active+clean; 7.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 691 op/s; 19225/237034 objects misplaced (8.111%); 7.5 MiB/s, 119 objects/s recovering 2024-05-15T23:22:21.199 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.11 to 1 (10000) 2024-05-15T23:22:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:21 smithi090 ceph-mon[44434]: 5.1 scrub ok 2024-05-15T23:22:21.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:21 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1470875325' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:22:21.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:21 smithi090 ceph-mon[44434]: pgmap v1157: 17 pgs: 3 active+recovering, 14 active+clean; 7.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 691 op/s; 19225/237034 objects misplaced (8.111%); 7.5 MiB/s, 119 objects/s recovering 2024-05-15T23:22:21.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:21 smithi088 ceph-mon[44699]: 5.1 scrub ok 2024-05-15T23:22:21.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:21 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1470875325' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:22:21.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:21 smithi088 ceph-mon[44699]: pgmap v1157: 17 pgs: 3 active+recovering, 14 active+clean; 7.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 691 op/s; 19225/237034 objects misplaced (8.111%); 7.5 MiB/s, 119 objects/s recovering 2024-05-15T23:22:21.968 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:22:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:21.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:22:21.995 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 9 1 2024-05-15T23:22:22.072 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:22:22.115 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:22.167 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:22:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:22.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:22:22.273 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:22:22.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1470875325' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:22:22.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:22 smithi084 ceph-mon[39784]: osdmap e427: 12 total, 12 up, 9 in 2024-05-15T23:22:22.324 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:22.360 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:22:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:22.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:22:22.368 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1470875325' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:22:22.368 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:22 smithi088 ceph-mon[44699]: osdmap e427: 12 total, 12 up, 9 in 2024-05-15T23:22:22.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:22:22.495 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:22.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1470875325' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:22:22.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:22 smithi090 ceph-mon[44434]: osdmap e427: 12 total, 12 up, 9 in 2024-05-15T23:22:22.663 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:22:22.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:22.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:22:22.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:22:22.796 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:22.994 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:22:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:22.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:22:23.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:23.145 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:23.183 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:23.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:23.187 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:23.288 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:22:23.329 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:23.359 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:22:23.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:23.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:22:23.366 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:23 smithi088 ceph-mon[44699]: pgmap v1159: 17 pgs: 3 active+recovering, 14 active+clean; 7.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 769 op/s; 19033/234336 objects misplaced (8.122%); 8.5 MiB/s, 136 objects/s recovering 2024-05-15T23:22:23.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:23 smithi084 ceph-mon[39784]: pgmap v1159: 17 pgs: 3 active+recovering, 14 active+clean; 7.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 769 op/s; 19033/234336 objects misplaced (8.122%); 8.5 MiB/s, 136 objects/s recovering 2024-05-15T23:22:23.394 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:22:23.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:22:23.442 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:22:23.463 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:22:23.467 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:22:23.504 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:22:23.513 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:23.531 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:22:23.547 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:22:23.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:23.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:22:23.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:23 smithi090 ceph-mon[44434]: pgmap v1159: 17 pgs: 3 active+recovering, 14 active+clean; 7.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 769 op/s; 19033/234336 objects misplaced (8.122%); 8.5 MiB/s, 136 objects/s recovering 2024-05-15T23:22:23.561 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:22:23.588 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:23.605 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:22:23.636 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:23.638 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:22:23.652 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:22:23.674 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:22:23.697 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:23.708 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:22:23.743 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:22:23.745 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:22:23.745 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:22:23.745 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:22:23.745 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:22:23.745 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:22:23.745 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:22:23.746 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:22:23.746 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:22:23.746 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:22:23.746 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:22:23.746 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:22:23.746 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:22:23.913 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:22:23.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:23.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:22:24.020 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:22:24.081 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:24.123 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:22:24.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:22:24.127 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:22:24.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:24.287 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:24.290 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: 5.e scrub starts 2024-05-15T23:22:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:22:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:22:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: osdmap e428: 12 total, 12 up, 9 in 2024-05-15T23:22:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: 5.5 scrub starts 2024-05-15T23:22:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3505706116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:22:24.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:22:24.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:24 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4259965118' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:22:24.390 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:22:24.422 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: 5.e scrub starts 2024-05-15T23:22:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:22:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:22:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: osdmap e428: 12 total, 12 up, 9 in 2024-05-15T23:22:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: 5.5 scrub starts 2024-05-15T23:22:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3505706116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:22:24.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:22:24.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:24 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4259965118' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:22:24.577 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: 5.e scrub starts 2024-05-15T23:22:24.577 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:22:24.577 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:22:24.577 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: osdmap e428: 12 total, 12 up, 9 in 2024-05-15T23:22:24.577 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: 5.5 scrub starts 2024-05-15T23:22:24.577 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3505706116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:22:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:22:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:24 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4259965118' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:22:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:25 smithi084 ceph-mon[39784]: 5.e scrub ok 2024-05-15T23:22:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:25 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:22:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:25 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4259965118' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:22:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:25 smithi084 ceph-mon[39784]: osdmap e429: 12 total, 12 up, 9 in 2024-05-15T23:22:25.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:25 smithi084 ceph-mon[39784]: pgmap v1162: 17 pgs: 3 active+recovering, 14 active+clean; 7.1 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 603 op/s; 19033/233884 objects misplaced (8.138%); 6.6 MiB/s, 106 objects/s recovering 2024-05-15T23:22:25.477 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:22:25.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:25.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:22:25.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:25.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:25.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:25.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:25 smithi090 ceph-mon[44434]: 5.e scrub ok 2024-05-15T23:22:25.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:25.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:25.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:25 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:22:25.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:25 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4259965118' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:22:25.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:25 smithi090 ceph-mon[44434]: osdmap e429: 12 total, 12 up, 9 in 2024-05-15T23:22:25.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:25 smithi090 ceph-mon[44434]: pgmap v1162: 17 pgs: 3 active+recovering, 14 active+clean; 7.1 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 603 op/s; 19033/233884 objects misplaced (8.138%); 6.6 MiB/s, 106 objects/s recovering 2024-05-15T23:22:25.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:25.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:25.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:25.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:25 smithi088 ceph-mon[44699]: 5.e scrub ok 2024-05-15T23:22:25.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:25.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:25.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:25 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:22:25.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:25 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4259965118' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:22:25.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:25 smithi088 ceph-mon[44699]: osdmap e429: 12 total, 12 up, 9 in 2024-05-15T23:22:25.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:25 smithi088 ceph-mon[44699]: pgmap v1162: 17 pgs: 3 active+recovering, 14 active+clean; 7.1 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 603 op/s; 19033/233884 objects misplaced (8.138%); 6.6 MiB/s, 106 objects/s recovering 2024-05-15T23:22:25.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:22:25.610 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:26.361 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:22:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:26.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:22:26.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:26.503 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:26.526 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:26.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:26.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:26.631 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:22:26.677 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:26.725 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:22:26.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:26.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:22:26.834 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:22:26.887 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:26.922 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:22:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:26.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:22:27.032 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:22:27.063 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:27.261 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:22:27.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:27.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:22:27.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:27 smithi088 ceph-mon[44699]: pgmap v1163: 17 pgs: 3 active+recovering, 14 active+clean; 7.0 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 594 op/s; 18608/229896 objects misplaced (8.094%); 6.4 MiB/s, 102 objects/s recovering 2024-05-15T23:22:27.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:22:27.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:27 smithi084 ceph-mon[39784]: pgmap v1163: 17 pgs: 3 active+recovering, 14 active+clean; 7.0 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 594 op/s; 18608/229896 objects misplaced (8.094%); 6.4 MiB/s, 102 objects/s recovering 2024-05-15T23:22:27.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:27 smithi090 ceph-mon[44434]: pgmap v1163: 17 pgs: 3 active+recovering, 14 active+clean; 7.0 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 594 op/s; 18608/229896 objects misplaced (8.094%); 6.4 MiB/s, 102 objects/s recovering 2024-05-15T23:22:27.403 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:27.427 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:22:27.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:27.430 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:22:27.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:22:27.530 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:22:27.585 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:27.626 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:22:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:27.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:22:27.734 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:22:27.767 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:28.220 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:22:28.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:28.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:22:28.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:28.364 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:28.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:28.402 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:28.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:28.576 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:28.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:28.579 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:28.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:28.709 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:28.737 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:28.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:28.780 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:28.783 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.9 to 1 (10000) 2024-05-15T23:22:28.788 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:22:28.788 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:22:28.841 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:22:28.887 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:28.916 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:22:28.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:28.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:22:29.028 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:22:29.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3855729131' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:22:29.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3855729131' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:22:29.075 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:29.077 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3855729131' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:22:29.111 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:22:29.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:29.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:22:29.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:29.254 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:29.282 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:29.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:22:29.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:29.286 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:29.387 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:22:29.436 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:29.482 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:22:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:29.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:22:29.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:29.633 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:29.670 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:29.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:29.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:29.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:29.804 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:29 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3855729131' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:22:29.804 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:29 smithi088 ceph-mon[44699]: osdmap e430: 12 total, 12 up, 9 in 2024-05-15T23:22:29.804 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:29 smithi088 ceph-mon[44699]: pgmap v1165: 17 pgs: 3 active+recovering, 14 active+clean; 6.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 604 op/s; 18365/227080 objects misplaced (8.087%); 7.0 MiB/s, 111 objects/s recovering 2024-05-15T23:22:29.816 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:29.837 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:29.841 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:29.942 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:29.978 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:29 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3855729131' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:22:29.978 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:29 smithi090 ceph-mon[44434]: osdmap e430: 12 total, 12 up, 9 in 2024-05-15T23:22:29.978 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:29 smithi090 ceph-mon[44434]: pgmap v1165: 17 pgs: 3 active+recovering, 14 active+clean; 6.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 604 op/s; 18365/227080 objects misplaced (8.087%); 7.0 MiB/s, 111 objects/s recovering 2024-05-15T23:22:29.980 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:30.009 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:30.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:30.016 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:30.117 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:29 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3855729131' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:22:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:29 smithi084 ceph-mon[39784]: osdmap e430: 12 total, 12 up, 9 in 2024-05-15T23:22:30.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:29 smithi084 ceph-mon[39784]: pgmap v1165: 17 pgs: 3 active+recovering, 14 active+clean; 6.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 604 op/s; 18365/227080 objects misplaced (8.087%); 7.0 MiB/s, 111 objects/s recovering 2024-05-15T23:22:30.167 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:30.200 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:30.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:30.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:30.309 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:22:30.365 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:30.403 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:22:30.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:30.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:22:30.432 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 10 1 2024-05-15T23:22:30.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:30.541 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:30.568 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:30.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:30.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:30.672 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:22:30.703 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:30.725 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:22:30.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:30.729 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:22:30.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:30.869 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:30.899 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:30.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:30.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:31.005 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:31.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:30 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:22:31.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:30 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:22:31.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:30 smithi090 ceph-mon[44434]: osdmap e431: 12 total, 12 up, 9 in 2024-05-15T23:22:31.067 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:30 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:22:31.067 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:30 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:22:31.067 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:30 smithi088 ceph-mon[44699]: osdmap e431: 12 total, 12 up, 9 in 2024-05-15T23:22:31.076 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:31.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:30 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:22:31.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:30 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:22:31.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:30 smithi084 ceph-mon[39784]: osdmap e431: 12 total, 12 up, 9 in 2024-05-15T23:22:31.126 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:31.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:31.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:31.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:31.264 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:31.290 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:31.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:31.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:31.325 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T23:22:31.395 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:22:31.432 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:31.465 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:22:31.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:31.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:22:31.560 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:31.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:31.607 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:31.650 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:31.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:31.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:31.757 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:22:31.787 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:31.810 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:31 smithi088 ceph-mon[44699]: pgmap v1167: 17 pgs: 3 active+recovering, 14 active+clean; 6.8 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 859 op/s; 17909/222530 objects misplaced (8.048%); 10 MiB/s, 163 objects/s recovering 2024-05-15T23:22:31.831 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:22:31.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:31.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:22:31.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:31.970 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:31 smithi090 ceph-mon[44434]: pgmap v1167: 17 pgs: 3 active+recovering, 14 active+clean; 6.8 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 859 op/s; 17909/222530 objects misplaced (8.048%); 10 MiB/s, 163 objects/s recovering 2024-05-15T23:22:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:31.992 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:31.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:31.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:32.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:22:32.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:31 smithi084 ceph-mon[39784]: pgmap v1167: 17 pgs: 3 active+recovering, 14 active+clean; 6.8 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 859 op/s; 17909/222530 objects misplaced (8.048%); 10 MiB/s, 163 objects/s recovering 2024-05-15T23:22:32.130 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:32.204 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:22:32.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:32.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:22:32.282 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:32.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:32.341 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:32.362 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:32.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:32.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:32.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:22:32.495 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:32.694 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:22:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:32.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:22:32.799 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:32.828 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:32.854 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:32.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:32.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:32.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:33.004 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:33.022 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:33.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:33.050 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:22:33.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:32 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:22:33.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:32 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:22:33.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:32 smithi088 ceph-mon[44699]: osdmap e432: 12 total, 12 up, 9 in 2024-05-15T23:22:33.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1800470147' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:22:33.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:32 smithi088 ceph-mon[44699]: from='client.26050 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:22:33.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:32 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:22:33.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:32 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:22:33.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:32 smithi084 ceph-mon[39784]: osdmap e432: 12 total, 12 up, 9 in 2024-05-15T23:22:33.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1800470147' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:22:33.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:32 smithi084 ceph-mon[39784]: from='client.26050 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:22:33.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:33.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:32 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:22:33.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:32 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:22:33.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:32 smithi090 ceph-mon[44434]: osdmap e432: 12 total, 12 up, 9 in 2024-05-15T23:22:33.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1800470147' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:22:33.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:32 smithi090 ceph-mon[44434]: from='client.26050 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:22:33.172 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:33.198 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:33.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:33.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:33.301 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:22:33.331 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:33.362 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:22:33.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:33.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:22:33.473 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:22:33.514 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:33.542 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:22:33.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:33.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:22:33.651 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:22:33.679 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:33.815 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:33 smithi088 ceph-mon[44699]: pgmap v1169: 17 pgs: 3 active+recovering, 14 active+clean; 6.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 843 op/s; 17666/219770 objects misplaced (8.038%); 9.8 MiB/s, 156 objects/s recovering 2024-05-15T23:22:33.815 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:33 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:22:33.815 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:33 smithi088 ceph-mon[44699]: from='client.26050 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:22:33.816 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:33 smithi088 ceph-mon[44699]: osdmap e433: 12 total, 12 up, 9 in 2024-05-15T23:22:34.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:33 smithi084 ceph-mon[39784]: pgmap v1169: 17 pgs: 3 active+recovering, 14 active+clean; 6.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 843 op/s; 17666/219770 objects misplaced (8.038%); 9.8 MiB/s, 156 objects/s recovering 2024-05-15T23:22:34.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:33 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:22:34.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:33 smithi084 ceph-mon[39784]: from='client.26050 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:22:34.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:33 smithi084 ceph-mon[39784]: osdmap e433: 12 total, 12 up, 9 in 2024-05-15T23:22:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:33 smithi090 ceph-mon[44434]: pgmap v1169: 17 pgs: 3 active+recovering, 14 active+clean; 6.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 843 op/s; 17666/219770 objects misplaced (8.038%); 9.8 MiB/s, 156 objects/s recovering 2024-05-15T23:22:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:33 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:22:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:33 smithi090 ceph-mon[44434]: from='client.26050 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:22:34.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:33 smithi090 ceph-mon[44434]: osdmap e433: 12 total, 12 up, 9 in 2024-05-15T23:22:34.497 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:35.028 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:22:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:35.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:22:35.133 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:22:35.163 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:35.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:35 smithi084 ceph-mon[39784]: pgmap v1171: 17 pgs: 3 active+recovering, 14 active+clean; 6.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 647 op/s; 17666/219312 objects misplaced (8.055%); 7.3 MiB/s, 116 objects/s recovering 2024-05-15T23:22:35.314 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:22:35.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:35.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:22:35.417 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:22:35.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:35 smithi088 ceph-mon[44699]: pgmap v1171: 17 pgs: 3 active+recovering, 14 active+clean; 6.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 647 op/s; 17666/219312 objects misplaced (8.055%); 7.3 MiB/s, 116 objects/s recovering 2024-05-15T23:22:35.465 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:35.502 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:22:35.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:35.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:22:35.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:35 smithi090 ceph-mon[44434]: pgmap v1171: 17 pgs: 3 active+recovering, 14 active+clean; 6.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 647 op/s; 17666/219312 objects misplaced (8.055%); 7.3 MiB/s, 116 objects/s recovering 2024-05-15T23:22:35.608 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:22:35.652 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:35.698 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:22:35.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:35.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:22:35.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:35.846 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:35.873 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:35.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:35.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:35.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:22:36.004 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:36.025 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:22:36.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:36.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:22:36.131 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:22:36.173 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:36.223 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:22:36.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:36.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:22:36.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:22:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:36.399 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:22:36.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:36.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:22:36.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/994436219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:36.581 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:36.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:36.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:36.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/994436219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:36.626 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:22:36.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:22:36.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:22:36.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:22:36.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:22:36.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:22:36.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:22:36.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:22:36.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:22:36.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:22:36.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:22:36.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:22:36.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:22:36.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:22:36.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:22:36.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:22:36.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:22:36.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:22:36.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:22:36.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:22:36.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:22:36.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:22:36.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:22:36.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:22:36.685 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:22:36.739 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:36.791 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:22:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:36.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:22:36.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/994436219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:36.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:36.938 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:36.971 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:36.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:36.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:37.022 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:22:37.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:37.109 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:37.145 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:37.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:37.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:37.254 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:22:37.283 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:37.309 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:22:37.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:37.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:22:37.415 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:37.451 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:37.497 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:37 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:22:37.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:37 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:22:37.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:37 smithi088 ceph-mon[44699]: pgmap v1172: 17 pgs: 3 active+recovering, 14 active+clean; 6.6 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 829 op/s; 17166/215150 objects misplaced (7.979%); 10 MiB/s, 166 objects/s recovering 2024-05-15T23:22:37.598 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:22:37.641 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:37.669 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:22:37.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:37.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:22:37.774 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:22:37.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:37 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:22:37.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:37 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:22:37.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:37 smithi084 ceph-mon[39784]: pgmap v1172: 17 pgs: 3 active+recovering, 14 active+clean; 6.6 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 829 op/s; 17166/215150 objects misplaced (7.979%); 10 MiB/s, 166 objects/s recovering 2024-05-15T23:22:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:37 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:22:37.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:37 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:22:37.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:37.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:37 smithi090 ceph-mon[44434]: pgmap v1172: 17 pgs: 3 active+recovering, 14 active+clean; 6.6 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 829 op/s; 17166/215150 objects misplaced (7.979%); 10 MiB/s, 166 objects/s recovering 2024-05-15T23:22:37.806 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:37.829 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:22:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:37.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:22:37.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:37.966 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:37.987 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:37.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:38.092 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:22:38.132 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:38.166 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:22:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:38.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:22:38.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:22:38.320 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:38.346 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:22:38.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:38.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:22:38.403 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:38.403 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:38.403 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:38 smithi088 ceph-mon[44699]: osdmap e434: 12 total, 12 up, 9 in 2024-05-15T23:22:38.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:38.524 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:38.527 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:38.585 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:22:38.585 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-05-15T23:22:38.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-05-15T23:22:38.586 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:22:38.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.586 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244624 3.b 3:d9663c5b:::benchmark_data_smithi090_90974_object79081:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:22:38.586 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.092561+0000", 2024-05-15T23:22:38.586 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 548.48878118599998, 2024-05-15T23:22:38.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.223617125, 2024-05-15T23:22:38.586 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.586 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.586 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.586 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.586 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.587 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.587 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244624 2024-05-15T23:22:38.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.587 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092561+0000", 2024-05-15T23:22:38.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092561+0000", 2024-05-15T23:22:38.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092563+0000", 2024-05-15T23:22:38.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.384e-06 2024-05-15T23:22:38.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092568+0000", 2024-05-15T23:22:38.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7679999999999999e-06 2024-05-15T23:22:38.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092576+0000", 2024-05-15T23:22:38.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4090000000000003e-06 2024-05-15T23:22:38.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092582+0000", 2024-05-15T23:22:38.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2310000000000001e-06 2024-05-15T23:22:38.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092632+0000", 2024-05-15T23:22:38.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0065000000000003e-05 2024-05-15T23:22:38.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092746+0000", 2024-05-15T23:22:38.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113791 2024-05-15T23:22:38.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:22:38.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.092854+0000", 2024-05-15T23:22:38.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107921 2024-05-15T23:22:38.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.097906+0000", 2024-05-15T23:22:38.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050516399999999996 2024-05-15T23:22:38.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316079+0000", 2024-05-15T23:22:38.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.21817299800000001 2024-05-15T23:22:38.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316131+0000", 2024-05-15T23:22:38.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2525999999999998e-05 2024-05-15T23:22:38.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316179+0000", 2024-05-15T23:22:38.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7392000000000003e-05 2024-05-15T23:22:38.592 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.592 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.592 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.592 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244628 3.b 3:d9384fcf:::benchmark_data_smithi090_90974_object79085:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:22:38.592 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.098262+0000", 2024-05-15T23:22:38.592 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 548.48308054500001, 2024-05-15T23:22:38.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.21807148000000001, 2024-05-15T23:22:38.592 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.592 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.592 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.593 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.593 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.593 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244628 2024-05-15T23:22:38.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.593 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098262+0000", 2024-05-15T23:22:38.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098262+0000", 2024-05-15T23:22:38.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098264+0000", 2024-05-15T23:22:38.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.699e-06 2024-05-15T23:22:38.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098269+0000", 2024-05-15T23:22:38.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8670000000000002e-06 2024-05-15T23:22:38.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098282+0000", 2024-05-15T23:22:38.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2918999999999999e-05 2024-05-15T23:22:38.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098289+0000", 2024-05-15T23:22:38.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.554e-06 2024-05-15T23:22:38.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098335+0000", 2024-05-15T23:22:38.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6276999999999998e-05 2024-05-15T23:22:38.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098452+0000", 2024-05-15T23:22:38.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000117038 2024-05-15T23:22:38.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:22:38.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.098560+0000", 2024-05-15T23:22:38.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000108009 2024-05-15T23:22:38.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.106584+0000", 2024-05-15T23:22:38.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080232729999999992 2024-05-15T23:22:38.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316237+0000", 2024-05-15T23:22:38.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.209653069 2024-05-15T23:22:38.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316290+0000", 2024-05-15T23:22:38.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2836e-05 2024-05-15T23:22:38.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316334+0000", 2024-05-15T23:22:38.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3939000000000001e-05 2024-05-15T23:22:38.597 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.598 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.598 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.598 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244632 3.b 3:d33b0449:::benchmark_data_smithi090_90974_object79089:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:22:38.598 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.102130+0000", 2024-05-15T23:22:38.598 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 548.47921268200002, 2024-05-15T23:22:38.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.21432957799999999, 2024-05-15T23:22:38.598 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.598 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.598 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.598 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.598 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.599 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.599 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244632 2024-05-15T23:22:38.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.599 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102130+0000", 2024-05-15T23:22:38.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102130+0000", 2024-05-15T23:22:38.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102133+0000", 2024-05-15T23:22:38.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6840000000000001e-06 2024-05-15T23:22:38.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102140+0000", 2024-05-15T23:22:38.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6629999999999993e-06 2024-05-15T23:22:38.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102155+0000", 2024-05-15T23:22:38.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5189e-05 2024-05-15T23:22:38.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102167+0000", 2024-05-15T23:22:38.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1058e-05 2024-05-15T23:22:38.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102585+0000", 2024-05-15T23:22:38.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000418742 2024-05-15T23:22:38.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102667+0000", 2024-05-15T23:22:38.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1692000000000003e-05 2024-05-15T23:22:38.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:22:38.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.102738+0000", 2024-05-15T23:22:38.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0678000000000003e-05 2024-05-15T23:22:38.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.106775+0000", 2024-05-15T23:22:38.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040370099999999997 2024-05-15T23:22:38.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316378+0000", 2024-05-15T23:22:38.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.20960367299999999 2024-05-15T23:22:38.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316422+0000", 2024-05-15T23:22:38.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3850999999999997e-05 2024-05-15T23:22:38.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.316460+0000", 2024-05-15T23:22:38.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7338000000000001e-05 2024-05-15T23:22:38.603 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.603 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244652 3.b 3:d38aa161:::benchmark_data_smithi090_90974_object79109:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:22:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.119285+0000", 2024-05-15T23:22:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 548.46205717500004, 2024-05-15T23:22:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.21651990300000001, 2024-05-15T23:22:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244652 2024-05-15T23:22:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.605 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119285+0000", 2024-05-15T23:22:38.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119285+0000", 2024-05-15T23:22:38.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119289+0000", 2024-05-15T23:22:38.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4999999999999999e-06 2024-05-15T23:22:38.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119299+0000", 2024-05-15T23:22:38.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9350000000000006e-06 2024-05-15T23:22:38.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119314+0000", 2024-05-15T23:22:38.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5163e-05 2024-05-15T23:22:38.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119326+0000", 2024-05-15T23:22:38.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1657e-05 2024-05-15T23:22:38.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119410+0000", 2024-05-15T23:22:38.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3991000000000003e-05 2024-05-15T23:22:38.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119611+0000", 2024-05-15T23:22:38.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00020102499999999999 2024-05-15T23:22:38.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:22:38.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.119718+0000", 2024-05-15T23:22:38.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107137 2024-05-15T23:22:38.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.124475+0000", 2024-05-15T23:22:38.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047574499999999999 2024-05-15T23:22:38.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.335707+0000", 2024-05-15T23:22:38.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.211231271 2024-05-15T23:22:38.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.335758+0000", 2024-05-15T23:22:38.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1039999999999999e-05 2024-05-15T23:22:38.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.335805+0000", 2024-05-15T23:22:38.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7734000000000001e-05 2024-05-15T23:22:38.609 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.609 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.609 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.609 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244695 3.b 3:df81a8c7:::benchmark_data_smithi090_90974_object79152:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:22:38.609 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.161150+0000", 2024-05-15T23:22:38.610 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 548.42019247999997, 2024-05-15T23:22:38.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18517029900000001, 2024-05-15T23:22:38.610 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.610 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.610 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.610 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.610 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.610 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.610 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244695 2024-05-15T23:22:38.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.610 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161150+0000", 2024-05-15T23:22:38.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161150+0000", 2024-05-15T23:22:38.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161153+0000", 2024-05-15T23:22:38.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.587e-06 2024-05-15T23:22:38.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161161+0000", 2024-05-15T23:22:38.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0760000000000003e-06 2024-05-15T23:22:38.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161174+0000", 2024-05-15T23:22:38.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3524e-05 2024-05-15T23:22:38.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161184+0000", 2024-05-15T23:22:38.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5810000000000003e-06 2024-05-15T23:22:38.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161249+0000", 2024-05-15T23:22:38.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4918000000000004e-05 2024-05-15T23:22:38.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161379+0000", 2024-05-15T23:22:38.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013014400000000001 2024-05-15T23:22:38.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:22:38.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.161522+0000", 2024-05-15T23:22:38.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014294500000000001 2024-05-15T23:22:38.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.165694+0000", 2024-05-15T23:22:38.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041721379999999997 2024-05-15T23:22:38.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346253+0000", 2024-05-15T23:22:38.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.180558575 2024-05-15T23:22:38.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346288+0000", 2024-05-15T23:22:38.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5845e-05 2024-05-15T23:22:38.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346320+0000", 2024-05-15T23:22:38.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1965999999999999e-05 2024-05-15T23:22:38.615 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.615 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.615 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.615 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244726 3.b 3:d016c9af:::benchmark_data_smithi090_90974_object79183:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:22:38.615 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.187811+0000", 2024-05-15T23:22:38.615 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 548.39353198699996, 2024-05-15T23:22:38.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15861062100000001, 2024-05-15T23:22:38.616 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.616 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.616 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.616 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.616 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.616 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.616 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244726 2024-05-15T23:22:38.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.616 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187811+0000", 2024-05-15T23:22:38.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187811+0000", 2024-05-15T23:22:38.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187812+0000", 2024-05-15T23:22:38.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.691e-06 2024-05-15T23:22:38.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187818+0000", 2024-05-15T23:22:38.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9630000000000003e-06 2024-05-15T23:22:38.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187828+0000", 2024-05-15T23:22:38.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3999999999999998e-06 2024-05-15T23:22:38.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187833+0000", 2024-05-15T23:22:38.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6879999999999996e-06 2024-05-15T23:22:38.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187898+0000", 2024-05-15T23:22:38.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4202000000000006e-05 2024-05-15T23:22:38.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.188038+0000", 2024-05-15T23:22:38.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014031100000000001 2024-05-15T23:22:38.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:22:38.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.188183+0000", 2024-05-15T23:22:38.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014492899999999999 2024-05-15T23:22:38.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.198038+0000", 2024-05-15T23:22:38.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0098547849999999996 2024-05-15T23:22:38.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346353+0000", 2024-05-15T23:22:38.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14831505 2024-05-15T23:22:38.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346382+0000", 2024-05-15T23:22:38.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9176e-05 2024-05-15T23:22:38.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346421+0000", 2024-05-15T23:22:38.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9425999999999998e-05 2024-05-15T23:22:38.621 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.621 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.621 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.621 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244727 3.b 3:d896ece8:::benchmark_data_smithi090_90974_object79184:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:22:38.621 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.187908+0000", 2024-05-15T23:22:38.621 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 548.39343459600002, 2024-05-15T23:22:38.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.158687891, 2024-05-15T23:22:38.621 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.621 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.621 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.622 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.622 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.622 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.622 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244727 2024-05-15T23:22:38.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.622 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187908+0000", 2024-05-15T23:22:38.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187908+0000", 2024-05-15T23:22:38.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187909+0000", 2024-05-15T23:22:38.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6099999999999999e-07 2024-05-15T23:22:38.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187912+0000", 2024-05-15T23:22:38.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.348e-06 2024-05-15T23:22:38.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187918+0000", 2024-05-15T23:22:38.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4519999999999996e-06 2024-05-15T23:22:38.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.187922+0000", 2024-05-15T23:22:38.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8299999999999998e-06 2024-05-15T23:22:38.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.193291+0000", 2024-05-15T23:22:38.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053689599999999999 2024-05-15T23:22:38.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.193434+0000", 2024-05-15T23:22:38.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014339600000000001 2024-05-15T23:22:38.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:22:38.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.193548+0000", 2024-05-15T23:22:38.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011445799999999999 2024-05-15T23:22:38.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.198206+0000", 2024-05-15T23:22:38.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046579509999999996 2024-05-15T23:22:38.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346511+0000", 2024-05-15T23:22:38.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14830475400000001 2024-05-15T23:22:38.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346559+0000", 2024-05-15T23:22:38.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8217999999999998e-05 2024-05-15T23:22:38.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.346596+0000", 2024-05-15T23:22:38.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6563000000000002e-05 2024-05-15T23:22:38.627 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.627 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.627 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.627 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244759 3.b 3:dc56f482:::benchmark_data_smithi090_90974_object79216:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:22:38.627 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.228332+0000", 2024-05-15T23:22:38.627 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 548.35301110299997, 2024-05-15T23:22:38.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12161722699999999, 2024-05-15T23:22:38.627 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.627 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.627 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.627 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.627 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.628 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.628 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244759 2024-05-15T23:22:38.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.628 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228332+0000", 2024-05-15T23:22:38.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228332+0000", 2024-05-15T23:22:38.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228334+0000", 2024-05-15T23:22:38.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2050000000000001e-06 2024-05-15T23:22:38.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228340+0000", 2024-05-15T23:22:38.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5359999999999998e-06 2024-05-15T23:22:38.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228355+0000", 2024-05-15T23:22:38.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4800999999999999e-05 2024-05-15T23:22:38.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228364+0000", 2024-05-15T23:22:38.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0880000000000001e-06 2024-05-15T23:22:38.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228430+0000", 2024-05-15T23:22:38.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6174999999999996e-05 2024-05-15T23:22:38.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228551+0000", 2024-05-15T23:22:38.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012065699999999999 2024-05-15T23:22:38.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:22:38.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.228670+0000", 2024-05-15T23:22:38.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000118702 2024-05-15T23:22:38.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.236954+0000", 2024-05-15T23:22:38.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082840819999999999 2024-05-15T23:22:38.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.349859+0000", 2024-05-15T23:22:38.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.112905017 2024-05-15T23:22:38.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.349910+0000", 2024-05-15T23:22:38.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1613000000000002e-05 2024-05-15T23:22:38.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.349949+0000", 2024-05-15T23:22:38.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8350999999999999e-05 2024-05-15T23:22:38.632 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.632 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.632 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244761 3.b 3:dc7a3ef4:::benchmark_data_smithi090_90974_object79218:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:22:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.230451+0000", 2024-05-15T23:22:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 548.35089185300001, 2024-05-15T23:22:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.119620066, 2024-05-15T23:22:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244761 2024-05-15T23:22:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.230451+0000", 2024-05-15T23:22:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.230451+0000", 2024-05-15T23:22:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.230453+0000", 2024-05-15T23:22:38.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.773e-06 2024-05-15T23:22:38.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.230458+0000", 2024-05-15T23:22:38.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2259999999999996e-06 2024-05-15T23:22:38.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.230469+0000", 2024-05-15T23:22:38.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1364e-05 2024-05-15T23:22:38.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.230484+0000", 2024-05-15T23:22:38.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4549000000000001e-05 2024-05-15T23:22:38.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.232644+0000", 2024-05-15T23:22:38.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021598469999999999 2024-05-15T23:22:38.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.232769+0000", 2024-05-15T23:22:38.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012553900000000001 2024-05-15T23:22:38.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:22:38.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.232895+0000", 2024-05-15T23:22:38.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000125809 2024-05-15T23:22:38.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.237087+0000", 2024-05-15T23:22:38.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041917999999999999 2024-05-15T23:22:38.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.349988+0000", 2024-05-15T23:22:38.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.112901394 2024-05-15T23:22:38.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.350035+0000", 2024-05-15T23:22:38.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7160999999999998e-05 2024-05-15T23:22:38.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.350071+0000", 2024-05-15T23:22:38.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5604e-05 2024-05-15T23:22:38.638 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.638 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.638 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.638 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244770 3.b 3:d69b076a:::benchmark_data_smithi090_90974_object79227:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:22:38.639 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.240553+0000", 2024-05-15T23:22:38.639 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 548.34078955300004, 2024-05-15T23:22:38.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.109669028, 2024-05-15T23:22:38.639 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.639 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.639 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.639 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.639 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.639 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.639 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244770 2024-05-15T23:22:38.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.639 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240553+0000", 2024-05-15T23:22:38.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240553+0000", 2024-05-15T23:22:38.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240555+0000", 2024-05-15T23:22:38.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5e-06 2024-05-15T23:22:38.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240560+0000", 2024-05-15T23:22:38.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2689999999999999e-06 2024-05-15T23:22:38.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240569+0000", 2024-05-15T23:22:38.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1349999999999998e-06 2024-05-15T23:22:38.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240576+0000", 2024-05-15T23:22:38.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.737e-06 2024-05-15T23:22:38.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240610+0000", 2024-05-15T23:22:38.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4239000000000002e-05 2024-05-15T23:22:38.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240691+0000", 2024-05-15T23:22:38.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1198000000000004e-05 2024-05-15T23:22:38.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:22:38.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.240762+0000", 2024-05-15T23:22:38.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0900000000000002e-05 2024-05-15T23:22:38.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.244680+0000", 2024-05-15T23:22:38.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039178169999999997 2024-05-15T23:22:38.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.350127+0000", 2024-05-15T23:22:38.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.10544740900000001 2024-05-15T23:22:38.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.350180+0000", 2024-05-15T23:22:38.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2967999999999998e-05 2024-05-15T23:22:38.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.350222+0000", 2024-05-15T23:22:38.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1856e-05 2024-05-15T23:22:38.644 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.644 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.644 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.644 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244787 3.b 3:d6752f13:::benchmark_data_smithi090_90974_object79244:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:22:38.644 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.261577+0000", 2024-05-15T23:22:38.644 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 548.31976599899997, 2024-05-15T23:22:38.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.096225877000000001, 2024-05-15T23:22:38.645 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.645 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.645 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.645 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.645 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.645 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.645 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244787 2024-05-15T23:22:38.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.645 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261577+0000", 2024-05-15T23:22:38.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261577+0000", 2024-05-15T23:22:38.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261578+0000", 2024-05-15T23:22:38.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7910000000000001e-06 2024-05-15T23:22:38.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261585+0000", 2024-05-15T23:22:38.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3820000000000001e-06 2024-05-15T23:22:38.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261596+0000", 2024-05-15T23:22:38.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0929e-05 2024-05-15T23:22:38.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261604+0000", 2024-05-15T23:22:38.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.994e-06 2024-05-15T23:22:38.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261664+0000", 2024-05-15T23:22:38.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0652000000000001e-05 2024-05-15T23:22:38.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261797+0000", 2024-05-15T23:22:38.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013271100000000001 2024-05-15T23:22:38.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:22:38.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.261926+0000", 2024-05-15T23:22:38.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128756 2024-05-15T23:22:38.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.266795+0000", 2024-05-15T23:22:38.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048690080000000002 2024-05-15T23:22:38.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.357732+0000", 2024-05-15T23:22:38.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090936930999999999 2024-05-15T23:22:38.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.357773+0000", 2024-05-15T23:22:38.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1495000000000002e-05 2024-05-15T23:22:38.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.357803+0000", 2024-05-15T23:22:38.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9227999999999999e-05 2024-05-15T23:22:38.650 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.650 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.650 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.650 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:244797 3.b 3:da28cbb5:::benchmark_data_smithi090_90974_object79254:head [delete] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-05-15T23:22:38.650 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:13:30.271847+0000", 2024-05-15T23:22:38.650 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 548.30949531199997, 2024-05-15T23:22:38.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.103864567, 2024-05-15T23:22:38.650 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.651 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.651 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.651 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.651 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.651 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.651 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 244797 2024-05-15T23:22:38.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.651 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.271847+0000", 2024-05-15T23:22:38.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.271847+0000", 2024-05-15T23:22:38.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.271850+0000", 2024-05-15T23:22:38.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5450000000000002e-06 2024-05-15T23:22:38.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.271857+0000", 2024-05-15T23:22:38.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.43e-06 2024-05-15T23:22:38.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.271875+0000", 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7337999999999999e-05 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.271884+0000", 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.696e-06 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.271941+0000", 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7000000000000003e-05 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.272040+0000", 2024-05-15T23:22:38.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8319999999999994e-05 2024-05-15T23:22:38.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0", 2024-05-15T23:22:38.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.272123+0000", 2024-05-15T23:22:38.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3572000000000006e-05 2024-05-15T23:22:38.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.276208+0000", 2024-05-15T23:22:38.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040850549999999998 2024-05-15T23:22:38.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.375616+0000", 2024-05-15T23:22:38.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.099407869999999995 2024-05-15T23:22:38.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.375672+0000", 2024-05-15T23:22:38.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5509e-05 2024-05-15T23:22:38.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:13:30.375712+0000", 2024-05-15T23:22:38.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0231999999999997e-05 2024-05-15T23:22:38.655 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.656 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.656 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.656 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:307618 3.7 3:ec218ab1:::benchmark_data_smithi090_90974_object142075:head [delete] snapc 0=[] ondisk+write+known_if_redirected e274)", 2024-05-15T23:22:38.656 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:14:22.587868+0000", 2024-05-15T23:22:38.656 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 495.99347462100002, 2024-05-15T23:22:38.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.102056827, 2024-05-15T23:22:38.656 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.656 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.656 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.656 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.656 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.656 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.656 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 307618 2024-05-15T23:22:38.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.657 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.587868+0000", 2024-05-15T23:22:38.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.587868+0000", 2024-05-15T23:22:38.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.587870+0000", 2024-05-15T23:22:38.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.669e-06 2024-05-15T23:22:38.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.587875+0000", 2024-05-15T23:22:38.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5090000000000001e-06 2024-05-15T23:22:38.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.587886+0000", 2024-05-15T23:22:38.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0413e-05 2024-05-15T23:22:38.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.587893+0000", 2024-05-15T23:22:38.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1149999999999996e-06 2024-05-15T23:22:38.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.587936+0000", 2024-05-15T23:22:38.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2963999999999997e-05 2024-05-15T23:22:38.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.588023+0000", 2024-05-15T23:22:38.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6965000000000005e-05 2024-05-15T23:22:38.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 11", 2024-05-15T23:22:38.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.588141+0000", 2024-05-15T23:22:38.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000117947 2024-05-15T23:22:38.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.591958+0000", 2024-05-15T23:22:38.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0038174569999999998 2024-05-15T23:22:38.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.689819+0000", 2024-05-15T23:22:38.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097860674999999994 2024-05-15T23:22:38.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.689874+0000", 2024-05-15T23:22:38.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5500000000000001e-05 2024-05-15T23:22:38.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:22.689925+0000", 2024-05-15T23:22:38.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0612999999999998e-05 2024-05-15T23:22:38.663 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.663 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.663 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.663 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:326342 3.2 3:4c240c5d:::benchmark_data_smithi090_90974_object160799:head [delete] snapc 0=[] ondisk+write+known_if_redirected e277)", 2024-05-15T23:22:38.663 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:14:38.287105+0000", 2024-05-15T23:22:38.663 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 480.29423750500001, 2024-05-15T23:22:38.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.098496609999999998, 2024-05-15T23:22:38.663 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.663 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.663 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.663 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.663 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 326342 2024-05-15T23:22:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287105+0000", 2024-05-15T23:22:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287105+0000", 2024-05-15T23:22:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287107+0000", 2024-05-15T23:22:38.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9069999999999999e-06 2024-05-15T23:22:38.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287115+0000", 2024-05-15T23:22:38.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1329999999999999e-06 2024-05-15T23:22:38.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287125+0000", 2024-05-15T23:22:38.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8190000000000003e-06 2024-05-15T23:22:38.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287134+0000", 2024-05-15T23:22:38.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5879999999999998e-06 2024-05-15T23:22:38.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287192+0000", 2024-05-15T23:22:38.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8838000000000001e-05 2024-05-15T23:22:38.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287306+0000", 2024-05-15T23:22:38.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113594 2024-05-15T23:22:38.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:22:38.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.287418+0000", 2024-05-15T23:22:38.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111988 2024-05-15T23:22:38.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292266+0000", 2024-05-15T23:22:38.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048482689999999997 2024-05-15T23:22:38.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385538+0000", 2024-05-15T23:22:38.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093272043999999998 2024-05-15T23:22:38.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385571+0000", 2024-05-15T23:22:38.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2985999999999999e-05 2024-05-15T23:22:38.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385602+0000", 2024-05-15T23:22:38.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0443999999999999e-05 2024-05-15T23:22:38.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.668 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.669 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:326346 3.2 3:4e958513:::benchmark_data_smithi090_90974_object160803:head [delete] snapc 0=[] ondisk+write+known_if_redirected e277)", 2024-05-15T23:22:38.669 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:14:38.292280+0000", 2024-05-15T23:22:38.669 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 480.28906253500003, 2024-05-15T23:22:38.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093412213999999993, 2024-05-15T23:22:38.669 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.669 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.669 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.669 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.669 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.669 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.670 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 326346 2024-05-15T23:22:38.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.670 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292280+0000", 2024-05-15T23:22:38.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292280+0000", 2024-05-15T23:22:38.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292282+0000", 2024-05-15T23:22:38.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.494e-06 2024-05-15T23:22:38.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292288+0000", 2024-05-15T23:22:38.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2199999999999997e-06 2024-05-15T23:22:38.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292296+0000", 2024-05-15T23:22:38.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5520000000000005e-06 2024-05-15T23:22:38.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292303+0000", 2024-05-15T23:22:38.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.409e-06 2024-05-15T23:22:38.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292339+0000", 2024-05-15T23:22:38.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5806999999999998e-05 2024-05-15T23:22:38.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292440+0000", 2024-05-15T23:22:38.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101629 2024-05-15T23:22:38.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:22:38.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292548+0000", 2024-05-15T23:22:38.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107509 2024-05-15T23:22:38.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.381144+0000", 2024-05-15T23:22:38.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.088596757999999998 2024-05-15T23:22:38.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385638+0000", 2024-05-15T23:22:38.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044937809999999996 2024-05-15T23:22:38.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385667+0000", 2024-05-15T23:22:38.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9203000000000001e-05 2024-05-15T23:22:38.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385692+0000", 2024-05-15T23:22:38.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4851999999999999e-05 2024-05-15T23:22:38.674 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.674 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.675 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.675 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:326348 3.2 3:43de2e75:::benchmark_data_smithi090_90974_object160805:head [delete] snapc 0=[] ondisk+write+known_if_redirected e277)", 2024-05-15T23:22:38.675 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:14:38.292315+0000", 2024-05-15T23:22:38.675 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 480.28902791500002, 2024-05-15T23:22:38.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093457223000000006, 2024-05-15T23:22:38.675 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.675 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.675 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.675 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.675 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.675 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.675 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 326348 2024-05-15T23:22:38.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.676 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292315+0000", 2024-05-15T23:22:38.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292315+0000", 2024-05-15T23:22:38.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292315+0000", 2024-05-15T23:22:38.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.13e-07 2024-05-15T23:22:38.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292318+0000", 2024-05-15T23:22:38.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5799999999999999e-06 2024-05-15T23:22:38.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292323+0000", 2024-05-15T23:22:38.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9570000000000001e-06 2024-05-15T23:22:38.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292328+0000", 2024-05-15T23:22:38.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7729999999999999e-06 2024-05-15T23:22:38.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.296253+0000", 2024-05-15T23:22:38.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039255089999999998 2024-05-15T23:22:38.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.296350+0000", 2024-05-15T23:22:38.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6979999999999996e-05 2024-05-15T23:22:38.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:22:38.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.296444+0000", 2024-05-15T23:22:38.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3487999999999997e-05 2024-05-15T23:22:38.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385610+0000", 2024-05-15T23:22:38.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089166309999999999 2024-05-15T23:22:38.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385724+0000", 2024-05-15T23:22:38.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113776 2024-05-15T23:22:38.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385751+0000", 2024-05-15T23:22:38.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6945999999999999e-05 2024-05-15T23:22:38.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385772+0000", 2024-05-15T23:22:38.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1390999999999999e-05 2024-05-15T23:22:38.681 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.681 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.681 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.681 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:326350 3.2 3:4a89f156:::benchmark_data_smithi090_90974_object160807:head [delete] snapc 0=[] ondisk+write+known_if_redirected e277)", 2024-05-15T23:22:38.681 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:14:38.292651+0000", 2024-05-15T23:22:38.681 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 480.28869140099999, 2024-05-15T23:22:38.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.093197133000000001, 2024-05-15T23:22:38.681 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.681 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.681 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.682 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.682 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.682 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.682 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 326350 2024-05-15T23:22:38.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.682 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292651+0000", 2024-05-15T23:22:38.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292651+0000", 2024-05-15T23:22:38.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292652+0000", 2024-05-15T23:22:38.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7600000000000006e-07 2024-05-15T23:22:38.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292656+0000", 2024-05-15T23:22:38.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7290000000000002e-06 2024-05-15T23:22:38.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292664+0000", 2024-05-15T23:22:38.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8690000000000004e-06 2024-05-15T23:22:38.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.292670+0000", 2024-05-15T23:22:38.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0100000000000001e-06 2024-05-15T23:22:38.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.300238+0000", 2024-05-15T23:22:38.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0075680799999999996 2024-05-15T23:22:38.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.300343+0000", 2024-05-15T23:22:38.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105173 2024-05-15T23:22:38.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:22:38.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.300451+0000", 2024-05-15T23:22:38.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107803 2024-05-15T23:22:38.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385614+0000", 2024-05-15T23:22:38.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.085162784000000005 2024-05-15T23:22:38.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385804+0000", 2024-05-15T23:22:38.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019009800000000001 2024-05-15T23:22:38.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385829+0000", 2024-05-15T23:22:38.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5278000000000001e-05 2024-05-15T23:22:38.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385848+0000", 2024-05-15T23:22:38.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9332999999999999e-05 2024-05-15T23:22:38.687 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.687 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.687 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.687 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:326358 3.2 3:4bb7fff5:::benchmark_data_smithi090_90974_object160815:head [delete] snapc 0=[] ondisk+write+known_if_redirected e277)", 2024-05-15T23:22:38.687 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:14:38.297628+0000", 2024-05-15T23:22:38.687 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 480.28371458800001, 2024-05-15T23:22:38.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.088290534000000004, 2024-05-15T23:22:38.688 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.688 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.688 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.688 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.688 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.688 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.688 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 326358 2024-05-15T23:22:38.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.688 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.297628+0000", 2024-05-15T23:22:38.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.297628+0000", 2024-05-15T23:22:38.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.297629+0000", 2024-05-15T23:22:38.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.074e-06 2024-05-15T23:22:38.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.297636+0000", 2024-05-15T23:22:38.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.421e-06 2024-05-15T23:22:38.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.297646+0000", 2024-05-15T23:22:38.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0214000000000001e-05 2024-05-15T23:22:38.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.297652+0000", 2024-05-15T23:22:38.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5610000000000004e-06 2024-05-15T23:22:38.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.305101+0000", 2024-05-15T23:22:38.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074482250000000002 2024-05-15T23:22:38.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.305208+0000", 2024-05-15T23:22:38.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010787 2024-05-15T23:22:38.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:22:38.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.376051+0000", 2024-05-15T23:22:38.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.070842961999999995 2024-05-15T23:22:38.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385778+0000", 2024-05-15T23:22:38.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0097269499999999998 2024-05-15T23:22:38.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385875+0000", 2024-05-15T23:22:38.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6886999999999996e-05 2024-05-15T23:22:38.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385900+0000", 2024-05-15T23:22:38.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4769e-05 2024-05-15T23:22:38.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.385919+0000", 2024-05-15T23:22:38.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8601e-05 2024-05-15T23:22:38.693 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.693 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.693 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.694 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25135.0:326363 3.2 3:4aae4c06:::benchmark_data_smithi090_90974_object160820:head [delete] snapc 0=[] ondisk+write+known_if_redirected e277)", 2024-05-15T23:22:38.694 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:14:38.302435+0000", 2024-05-15T23:22:38.694 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 480.27890763900001, 2024-05-15T23:22:38.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.097756039000000003, 2024-05-15T23:22:38.694 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.694 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.694 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.694 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:22:38.694 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25135", 2024-05-15T23:22:38.694 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1313024645", 2024-05-15T23:22:38.694 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 326363 2024-05-15T23:22:38.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.695 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.302435+0000", 2024-05-15T23:22:38.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.302435+0000", 2024-05-15T23:22:38.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.302436+0000", 2024-05-15T23:22:38.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.35e-06 2024-05-15T23:22:38.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.302441+0000", 2024-05-15T23:22:38.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7509999999999999e-06 2024-05-15T23:22:38.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.302449+0000", 2024-05-15T23:22:38.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9869999999999999e-06 2024-05-15T23:22:38.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.302463+0000", 2024-05-15T23:22:38.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4363e-05 2024-05-15T23:22:38.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.386018+0000", 2024-05-15T23:22:38.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.083554470000000006 2024-05-15T23:22:38.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.386088+0000", 2024-05-15T23:22:38.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9783999999999995e-05 2024-05-15T23:22:38.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:22:38.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.386152+0000", 2024-05-15T23:22:38.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4420999999999999e-05 2024-05-15T23:22:38.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:22:38.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.400065+0000", 2024-05-15T23:22:38.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013912522 2024-05-15T23:22:38.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:22:38.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.400117+0000", 2024-05-15T23:22:38.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2476999999999999e-05 2024-05-15T23:22:38.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.400146+0000", 2024-05-15T23:22:38.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9354999999999999e-05 2024-05-15T23:22:38.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:14:38.400191+0000", 2024-05-15T23:22:38.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4558999999999998e-05 2024-05-15T23:22:38.700 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.700 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.700 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.700 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.25840.0:201645 5.3 e433/407 5:cf8f44ac:::benchmark_data_smithi090_116470_object48085:head v 433'12631, mlcod=384'2991)", 2024-05-15T23:22:38.700 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:22:37.187043+0000", 2024-05-15T23:22:38.700 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.394299336, 2024-05-15T23:22:38.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.03675867, 2024-05-15T23:22:38.700 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:22:38.700 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:22:38.701 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:22:38.701 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:22:38.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:22:38.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:37.187043+0000", 2024-05-15T23:22:38.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:22:38.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:37.187043+0000", 2024-05-15T23:22:38.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:22:38.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:22:38.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:37.187045+0000", 2024-05-15T23:22:38.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.393e-06 2024-05-15T23:22:38.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:22:38.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:37.187056+0000", 2024-05-15T23:22:38.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0868e-05 2024-05-15T23:22:38.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:22:38.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:37.187066+0000", 2024-05-15T23:22:38.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0533000000000001e-05 2024-05-15T23:22:38.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:22:38.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:37.187072+0000", 2024-05-15T23:22:38.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1700000000000002e-06 2024-05-15T23:22:38.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:22:38.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:37.195021+0000", 2024-05-15T23:22:38.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0079483030000000003 2024-05-15T23:22:38.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:22:38.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:37.195049+0000", 2024-05-15T23:22:38.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8824999999999999e-05 2024-05-15T23:22:38.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:22:38.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:37.223750+0000", 2024-05-15T23:22:38.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028700465000000001 2024-05-15T23:22:38.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:22:38.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:22:38.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:22:38.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:22:37.223802+0000", 2024-05-15T23:22:38.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2113000000000001e-05 2024-05-15T23:22:38.704 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.704 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.704 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.704 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:22:38.705 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:22:38.705 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:22:38.705 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:22:38.777 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:38.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:38.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:38.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:38 smithi090 ceph-mon[44434]: osdmap e434: 12 total, 12 up, 9 in 2024-05-15T23:22:38.816 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:22:38.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:38.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:22:38.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:38.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:38.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:38 smithi084 ceph-mon[39784]: osdmap e434: 12 total, 12 up, 9 in 2024-05-15T23:22:38.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:22:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:38.987 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:22:38.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:38.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:22:39.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:22:39.121 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:39.142 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:22:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:39.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:22:39.252 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:39.340 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:39.389 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.10 to 1 (10000) 2024-05-15T23:22:39.390 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:39.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:39.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:39.499 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:22:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:39.556 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:22:39.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:39.561 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:22:39.619 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T23:22:39.662 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:22:39.688 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:39 smithi084 ceph-mon[39784]: pgmap v1174: 17 pgs: 3 active+recovering, 14 active+clean; 6.5 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 697 op/s; 16954/212504 objects misplaced (7.978%); 8.3 MiB/s, 132 objects/s recovering 2024-05-15T23:22:39.688 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3999206852' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:22:39.690 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:39.734 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:39 smithi088 ceph-mon[44699]: pgmap v1174: 17 pgs: 3 active+recovering, 14 active+clean; 6.5 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 697 op/s; 16954/212504 objects misplaced (7.978%); 8.3 MiB/s, 132 objects/s recovering 2024-05-15T23:22:39.734 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3999206852' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:22:39.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:39 smithi090 ceph-mon[44434]: pgmap v1174: 17 pgs: 3 active+recovering, 14 active+clean; 6.5 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 697 op/s; 16954/212504 objects misplaced (7.978%); 8.3 MiB/s, 132 objects/s recovering 2024-05-15T23:22:39.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3999206852' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:22:40.291 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:22:40.417 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-05-15T23:22:40.417 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-05-15T23:22:40.418 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-15T23:22:40.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump --format=json 2024-05-15T23:22:40.438 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:22:40.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:40.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:22:40.542 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:22:40.681 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:40 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3999206852' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:22:40.681 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:40 smithi088 ceph-mon[44699]: osdmap e435: 12 total, 12 up, 9 in 2024-05-15T23:22:40.689 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:40.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:40 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3999206852' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:22:40.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:40 smithi084 ceph-mon[39784]: osdmap e435: 12 total, 12 up, 9 in 2024-05-15T23:22:40.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:40 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3999206852' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:22:40.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:40 smithi090 ceph-mon[44434]: osdmap e435: 12 total, 12 up, 9 in 2024-05-15T23:22:41.120 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:41.677 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:41 smithi084 ceph-mon[39784]: pgmap v1176: 17 pgs: 3 active+recovering, 14 active+clean; 6.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 754 op/s; 16497/207776 objects misplaced (7.940%); 9.2 MiB/s, 146 objects/s recovering 2024-05-15T23:22:41.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:41 smithi090 ceph-mon[44434]: pgmap v1176: 17 pgs: 3 active+recovering, 14 active+clean; 6.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 754 op/s; 16497/207776 objects misplaced (7.940%); 9.2 MiB/s, 146 objects/s recovering 2024-05-15T23:22:41.956 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:41.994 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:41 smithi088 ceph-mon[44699]: pgmap v1176: 17 pgs: 3 active+recovering, 14 active+clean; 6.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 754 op/s; 16497/207776 objects misplaced (7.940%); 9.2 MiB/s, 146 objects/s recovering 2024-05-15T23:22:42.730 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:42 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:22:42.731 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:42 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:22:42.731 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:42 smithi084 ceph-mon[39784]: osdmap e436: 12 total, 12 up, 9 in 2024-05-15T23:22:42.731 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:42 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:22:42.731 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:42 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:22:42.753 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:22:42.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:42 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:22:42.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:42 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:22:42.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:42 smithi090 ceph-mon[44434]: osdmap e436: 12 total, 12 up, 9 in 2024-05-15T23:22:42.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:42 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:22:42.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:42 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:22:42.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:42.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:22:42.819 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:42 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:22:42.819 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:42 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:22:42.819 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:42 smithi088 ceph-mon[44699]: osdmap e436: 12 total, 12 up, 9 in 2024-05-15T23:22:42.819 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:42 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:22:42.819 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:42 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:22:42.919 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:22:43.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:22:43.529 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:43.576 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:22:43.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:43.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:22:43.689 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:43.736 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:43 smithi088 ceph-mon[44699]: osdmap e437: 12 total, 12 up, 9 in 2024-05-15T23:22:43.736 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:43 smithi088 ceph-mon[44699]: pgmap v1179: 17 pgs: 3 active+recovering, 14 active+clean; 6.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 671 op/s; 16279/204950 objects misplaced (7.943%); 7.5 MiB/s, 120 objects/s recovering 2024-05-15T23:22:43.747 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:43.781 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:43.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:43.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:43.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:43 smithi084 ceph-mon[39784]: osdmap e437: 12 total, 12 up, 9 in 2024-05-15T23:22:43.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:43 smithi084 ceph-mon[39784]: pgmap v1179: 17 pgs: 3 active+recovering, 14 active+clean; 6.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 671 op/s; 16279/204950 objects misplaced (7.943%); 7.5 MiB/s, 120 objects/s recovering 2024-05-15T23:22:43.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:43 smithi090 ceph-mon[44434]: osdmap e437: 12 total, 12 up, 9 in 2024-05-15T23:22:43.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:43 smithi090 ceph-mon[44434]: pgmap v1179: 17 pgs: 3 active+recovering, 14 active+clean; 6.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 671 op/s; 16279/204950 objects misplaced (7.943%); 7.5 MiB/s, 120 objects/s recovering 2024-05-15T23:22:43.888 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:22:43.933 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:43.961 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:22:43.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:43.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:22:44.066 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:22:44.095 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:44.321 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:22:44.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:44.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:22:44.356 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:44.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:44.456 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:44.479 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1677600406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:22:44.479 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:44 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:22:44.479 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:44 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:22:44.485 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:22:44.496 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:44.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:44.500 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:44.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:22:44.631 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:44.672 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:22:44.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:44.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:22:44.737 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1677600406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:22:44.737 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:44 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:22:44.737 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:44 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:22:44.776 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:22:44.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1677600406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:22:44.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:44 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:22:44.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:44 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:22:44.822 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:45 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:22:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1677600406' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:22:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:45 smithi090 ceph-mon[44434]: osdmap e438: 12 total, 12 up, 9 in 2024-05-15T23:22:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:45 smithi090 ceph-mon[44434]: pgmap v1181: 17 pgs: 3 active+recovering, 14 active+clean; 6.2 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 715 op/s; 16279/204476 objects misplaced (7.961%); 7.5 MiB/s, 120 objects/s recovering 2024-05-15T23:22:45.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:45 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:22:45.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1677600406' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:22:45.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:45 smithi088 ceph-mon[44699]: osdmap e438: 12 total, 12 up, 9 in 2024-05-15T23:22:45.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:45 smithi088 ceph-mon[44699]: pgmap v1181: 17 pgs: 3 active+recovering, 14 active+clean; 6.2 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 715 op/s; 16279/204476 objects misplaced (7.961%); 7.5 MiB/s, 120 objects/s recovering 2024-05-15T23:22:45.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:45 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:22:45.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1677600406' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:22:45.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:45 smithi084 ceph-mon[39784]: osdmap e438: 12 total, 12 up, 9 in 2024-05-15T23:22:45.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:45 smithi084 ceph-mon[39784]: pgmap v1181: 17 pgs: 3 active+recovering, 14 active+clean; 6.2 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 715 op/s; 16279/204476 objects misplaced (7.961%); 7.5 MiB/s, 120 objects/s recovering 2024-05-15T23:22:46.011 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:22:46.012 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":438,"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","created":"2024-05-15T22:56:41.212493+0000","modified":"2024-05-15T23:22:44.472111+0000","last_up_change":"2024-05-15T23:20:47.040869+0000","last_in_change":"2024-05-15T23:21:08.011852+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":29,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T22:59:38.603575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.96207165718078613,"average_primary_affinity":0.95999997854232788,"average_primary_affinity_weighted":0.95999997854232788}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2024-05-15T23:19:45.847140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"380","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":0.75,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.96207165718078613,"average_primary_affinity":0.95999997854232788,"average_primary_affinity_weighted":0.95999997854232788}}],"osds":[{"osd":0,"uuid":"80159cc8-1b6c-4aa6-a114-966786d7be1e","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":355,"up_from":397,"up_thru":283,"down_at":356,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6801","nonce":3077815271}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6802","nonce":3077815271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6804","nonce":3077815271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6803","nonce":3077815271}]},"public_addr":"172.21.15.84:6801/3077815271","cluster_addr":"172.21.15.84:6802/3077815271","heartbeat_back_addr":"172.21.15.84:6804/3077815271","heartbeat_front_addr":"172.21.15.84:6803/3077815271","state":["exists","up"]},{"osd":1,"uuid":"9c2c76c9-ce0f-4c2d-9d96-c87a10187607","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":374,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6805","nonce":2140830229}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6806","nonce":2140830229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6808","nonce":2140830229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6807","nonce":2140830229}]},"public_addr":"172.21.15.84:6805/2140830229","cluster_addr":"172.21.15.84:6806/2140830229","heartbeat_back_addr":"172.21.15.84:6808/2140830229","heartbeat_front_addr":"172.21.15.84:6807/2140830229","state":["exists","up"]},{"osd":2,"uuid":"e8514daa-b5f9-4f2f-a432-53f4a4eb86bc","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":261,"up_from":291,"up_thru":190,"down_at":262,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6809","nonce":941507493}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6810","nonce":941507493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6812","nonce":941507493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6811","nonce":941507493}]},"public_addr":"172.21.15.84:6809/941507493","cluster_addr":"172.21.15.84:6810/941507493","heartbeat_back_addr":"172.21.15.84:6812/941507493","heartbeat_front_addr":"172.21.15.84:6811/941507493","state":["exists","up"]},{"osd":3,"uuid":"5d9965f9-c0b2-48ce-869c-582990985623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":283,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6813","nonce":2204039828}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6814","nonce":2204039828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6816","nonce":2204039828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6815","nonce":2204039828}]},"public_addr":"172.21.15.84:6813/2204039828","cluster_addr":"172.21.15.84:6814/2204039828","heartbeat_back_addr":"172.21.15.84:6816/2204039828","heartbeat_front_addr":"172.21.15.84:6815/2204039828","state":["exists","up"]},{"osd":4,"uuid":"4bc50523-8480-44d2-8d76-31479e839c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":374,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6800","nonce":2267955292}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":2267955292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":2267955292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":2267955292}]},"public_addr":"172.21.15.88:6800/2267955292","cluster_addr":"172.21.15.88:6801/2267955292","heartbeat_back_addr":"172.21.15.88:6803/2267955292","heartbeat_front_addr":"172.21.15.88:6802/2267955292","state":["exists","up"]},{"osd":5,"uuid":"d4db8d7f-afc7-43dc-81fe-f70c290d8b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":407,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":661041538}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":661041538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":661041538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":661041538}]},"public_addr":"172.21.15.88:6804/661041538","cluster_addr":"172.21.15.88:6805/661041538","heartbeat_back_addr":"172.21.15.88:6807/661041538","heartbeat_front_addr":"172.21.15.88:6806/661041538","state":["exists","up"]},{"osd":6,"uuid":"a2773454-2318-4a5b-8f31-0952b1b07f2f","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":374,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":380856869}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":380856869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":380856869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":380856869}]},"public_addr":"172.21.15.88:6808/380856869","cluster_addr":"172.21.15.88:6809/380856869","heartbeat_back_addr":"172.21.15.88:6811/380856869","heartbeat_front_addr":"172.21.15.88:6810/380856869","state":["exists","up"]},{"osd":7,"uuid":"028ca7b6-3e00-4466-a33f-b726cd60cb34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":407,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":1044709063}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":1044709063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":1044709063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":1044709063}]},"public_addr":"172.21.15.88:6812/1044709063","cluster_addr":"172.21.15.88:6813/1044709063","heartbeat_back_addr":"172.21.15.88:6815/1044709063","heartbeat_front_addr":"172.21.15.88:6814/1044709063","state":["exists","up"]},{"osd":8,"uuid":"02cab286-d487-4a66-982c-3cf28f55e851","up":1,"in":1,"weight":1,"primary_affinity":0.54486083984375,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6800","nonce":522459126}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6801","nonce":522459126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6803","nonce":522459126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6802","nonce":522459126}]},"public_addr":"172.21.15.90:6800/522459126","cluster_addr":"172.21.15.90:6801/522459126","heartbeat_back_addr":"172.21.15.90:6803/522459126","heartbeat_front_addr":"172.21.15.90:6802/522459126","state":["exists","up"]},{"osd":9,"uuid":"47d38bc7-39ba-4b35-af24-e282354dd613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":374,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6804","nonce":2794571133}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6805","nonce":2794571133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6807","nonce":2794571133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6806","nonce":2794571133}]},"public_addr":"172.21.15.90:6804/2794571133","cluster_addr":"172.21.15.90:6805/2794571133","heartbeat_back_addr":"172.21.15.90:6807/2794571133","heartbeat_front_addr":"172.21.15.90:6806/2794571133","state":["exists","up"]},{"osd":10,"uuid":"d7b2aeae-d012-41eb-9f91-e4c05c649473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":407,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6808","nonce":3735380744}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6809","nonce":3735380744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6811","nonce":3735380744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6810","nonce":3735380744}]},"public_addr":"172.21.15.90:6808/3735380744","cluster_addr":"172.21.15.90:6809/3735380744","heartbeat_back_addr":"172.21.15.90:6811/3735380744","heartbeat_front_addr":"172.21.15.90:6810/3735380744","state":["exists","up"]},{"osd":11,"uuid":"2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":374,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6812","nonce":2265615270}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6813","nonce":2265615270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6815","nonce":2265615270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6814","nonce":2265615270}]},"public_addr":"172.21.15.90:6812/2265615270","cluster_addr":"172.21.15.90:6813/2265615270","heartbeat_back_addr":"172.21.15.90:6815/2265615270","heartbeat_front_addr":"172.21.15.90:6814/2265615270","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-05-15T23:21:53.055863+0000","dead_epoch":355},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:21:46.002722+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-05-15T23:22:41.890004+0000","dead_epoch":261},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:21:51.273878+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:22:36.612048+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:22:43.794479+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-05-15T23:22:05.435172+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:22:23.739239+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:22:41.085974+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:22:31.722396+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:22:23.065611+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:22:28.901646+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/2581408231":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/1136984393":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/2443201864":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1252671888":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1451001371":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/4084814866":"2024-05-16T22:57:38.505263+0000","172.21.15.84:0/4076409285":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/2988977694":"2024-05-16T22:57:38.505263+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T23:22:46.754 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:46 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2350198306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:22:46.784 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-15T23:22:46.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:46 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2350198306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:22:46.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:46 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2350198306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:22:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:22:47.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:47.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:22:47.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:22:47.233 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:47.467 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:22:47.492 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:47 smithi088 ceph-mon[44699]: 5.b scrub starts 2024-05-15T23:22:47.582 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:22:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:47.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:22:47.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:22:47.741 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:47.771 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:22:47.776 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:47 smithi090 ceph-mon[44434]: 5.b scrub starts 2024-05-15T23:22:47.776 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:47 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:22:47.776 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:47 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:22:47.776 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:47 smithi090 ceph-mon[44434]: pgmap v1182: 17 pgs: 3 active+recovering, 14 active+clean; 6.1 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 637 op/s; 15816/200126 objects misplaced (7.903%); 7.1 MiB/s, 113 objects/s recovering 2024-05-15T23:22:47.777 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:47 smithi090 ceph-mon[44434]: 5.5 scrub starts 2024-05-15T23:22:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:47.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:22:47.828 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:47 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:22:47.828 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:47 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:22:47.828 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:47 smithi088 ceph-mon[44699]: pgmap v1182: 17 pgs: 3 active+recovering, 14 active+clean; 6.1 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 637 op/s; 15816/200126 objects misplaced (7.903%); 7.1 MiB/s, 113 objects/s recovering 2024-05-15T23:22:47.828 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:47 smithi088 ceph-mon[44699]: 5.5 scrub starts 2024-05-15T23:22:47.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:47 smithi084 ceph-mon[39784]: 5.b scrub starts 2024-05-15T23:22:47.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:47 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:22:47.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:47 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:22:47.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:47 smithi084 ceph-mon[39784]: pgmap v1182: 17 pgs: 3 active+recovering, 14 active+clean; 6.1 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 637 op/s; 15816/200126 objects misplaced (7.903%); 7.1 MiB/s, 113 objects/s recovering 2024-05-15T23:22:47.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:47 smithi084 ceph-mon[39784]: 5.5 scrub starts 2024-05-15T23:22:47.878 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:22:47.914 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:48.172 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:22:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:48.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:22:48.174 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:22:48.188 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:48.198 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:22:48.227 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:22:48.248 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:22:48.275 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:48.275 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:22:48.296 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:22:48.313 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:48.321 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:22:48.345 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:22:48.372 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:22:48.392 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:22:48.413 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:22:48.436 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:22:48.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:22:48.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:22:48.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:22:48.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:22:48.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:22:48.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:22:48.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:22:48.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:22:48.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:22:48.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:22:48.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:22:48.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:22:48.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:48 smithi088 ceph-mon[44699]: osdmap e439: 12 total, 12 up, 9 in 2024-05-15T23:22:48.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:48 smithi088 ceph-mon[44699]: 5.5 scrub ok 2024-05-15T23:22:48.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2343385102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:48.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:48.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:48.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:48.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:48 smithi088 ceph-mon[44699]: 5.b scrub ok 2024-05-15T23:22:48.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:48.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:48 smithi090 ceph-mon[44434]: osdmap e439: 12 total, 12 up, 9 in 2024-05-15T23:22:48.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:48 smithi090 ceph-mon[44434]: 5.5 scrub ok 2024-05-15T23:22:48.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2343385102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:48.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:48.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:48.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:48.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:48 smithi090 ceph-mon[44434]: 5.b scrub ok 2024-05-15T23:22:48.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:48.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:48 smithi084 ceph-mon[39784]: osdmap e439: 12 total, 12 up, 9 in 2024-05-15T23:22:48.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:48 smithi084 ceph-mon[39784]: 5.5 scrub ok 2024-05-15T23:22:48.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2343385102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:48.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:48.820 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:48.820 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:48.820 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:48 smithi084 ceph-mon[39784]: 5.b scrub ok 2024-05-15T23:22:48.820 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:49 smithi090 ceph-mon[44434]: pgmap v1184: 17 pgs: 3 active+recovering, 14 active+clean; 6.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 579 op/s; 15567/197376 objects misplaced (7.887%); 6.8 MiB/s, 109 objects/s recovering 2024-05-15T23:22:49.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:49 smithi088 ceph-mon[44699]: pgmap v1184: 17 pgs: 3 active+recovering, 14 active+clean; 6.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 579 op/s; 15567/197376 objects misplaced (7.887%); 6.8 MiB/s, 109 objects/s recovering 2024-05-15T23:22:49.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:49.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:49 smithi084 ceph-mon[39784]: pgmap v1184: 17 pgs: 3 active+recovering, 14 active+clean; 6.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 579 op/s; 15567/197376 objects misplaced (7.887%); 6.8 MiB/s, 109 objects/s recovering 2024-05-15T23:22:50.097 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:50.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:50.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:50.205 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:22:50.253 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:50.291 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:22:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:50.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:22:50.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:50.427 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:50.452 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:50.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:50.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:50.556 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:22:50.577 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:22:50.590 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:50.616 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:22:50.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:50.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:22:50.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:50.754 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1840533069' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:22:50.762 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:50.779 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:50.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:50.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:50.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1840533069' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:22:50.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1840533069' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:22:50.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:50.922 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:50.952 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:50.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:50.955 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:51.056 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:22:51.484 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:22:51.484 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:22:51.562 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:51.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:51 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:22:51.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:51 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:22:51.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1840533069' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:22:51.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:51 smithi088 ceph-mon[44699]: osdmap e440: 12 total, 12 up, 9 in 2024-05-15T23:22:51.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:51 smithi088 ceph-mon[44699]: pgmap v1186: 17 pgs: 3 active+recovering, 14 active+clean; 5.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 940 op/s; 15094/192690 objects misplaced (7.833%); 11 MiB/s, 181 objects/s recovering 2024-05-15T23:22:51.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:51 smithi088 ceph-mon[44699]: 5.c scrub starts 2024-05-15T23:22:51.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:51 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:22:51.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:51 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:22:51.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1840533069' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:22:51.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:51 smithi084 ceph-mon[39784]: osdmap e440: 12 total, 12 up, 9 in 2024-05-15T23:22:51.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:51 smithi084 ceph-mon[39784]: pgmap v1186: 17 pgs: 3 active+recovering, 14 active+clean; 5.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 940 op/s; 15094/192690 objects misplaced (7.833%); 11 MiB/s, 181 objects/s recovering 2024-05-15T23:22:51.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:51 smithi084 ceph-mon[39784]: 5.c scrub starts 2024-05-15T23:22:52.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:51 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:22:52.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:51 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:22:52.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1840533069' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:22:52.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:51 smithi090 ceph-mon[44434]: osdmap e440: 12 total, 12 up, 9 in 2024-05-15T23:22:52.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:51 smithi090 ceph-mon[44434]: pgmap v1186: 17 pgs: 3 active+recovering, 14 active+clean; 5.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 940 op/s; 15094/192690 objects misplaced (7.833%); 11 MiB/s, 181 objects/s recovering 2024-05-15T23:22:52.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:51 smithi090 ceph-mon[44434]: 5.c scrub starts 2024-05-15T23:22:52.475 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:22:52.776 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:53.248 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:22:53.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:53.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:22:53.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:53 smithi090 ceph-mon[44434]: 5.c scrub ok 2024-05-15T23:22:53.309 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:53 smithi090 ceph-mon[44434]: pgmap v1187: 17 pgs: 3 active+recovering, 14 active+clean; 5.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 912 op/s; 14859/189870 objects misplaced (7.826%); 11 MiB/s, 177 objects/s recovering 2024-05-15T23:22:53.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:53 smithi084 ceph-mon[39784]: 5.c scrub ok 2024-05-15T23:22:53.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:53 smithi084 ceph-mon[39784]: pgmap v1187: 17 pgs: 3 active+recovering, 14 active+clean; 5.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 912 op/s; 14859/189870 objects misplaced (7.826%); 11 MiB/s, 177 objects/s recovering 2024-05-15T23:22:53.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:53 smithi088 ceph-mon[44699]: 5.c scrub ok 2024-05-15T23:22:53.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:53 smithi088 ceph-mon[44699]: pgmap v1187: 17 pgs: 3 active+recovering, 14 active+clean; 5.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 912 op/s; 14859/189870 objects misplaced (7.826%); 11 MiB/s, 177 objects/s recovering 2024-05-15T23:22:53.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:22:53.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:22:53.401 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:53.422 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:22:53.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:53.427 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:22:53.528 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:22:53.580 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:54.219 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:54.711 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:22:54.715 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T23:22:54.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:54.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:22:54.777 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:22:54.801 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:22:54.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:22:54.830 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:22:54.850 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:54.857 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:22:54.884 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:22:54.907 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:22:54.928 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:22:54.951 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:22:54.979 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:22:55.002 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:22:55.028 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:22:55.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:22:55.056 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:22:55.056 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:22:55.057 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:22:55.057 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:22:55.057 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:22:55.057 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:22:55.057 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:22:55.057 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:22:55.057 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:22:55.057 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:22:55.057 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:22:55.057 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:22:55.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/116086235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:55.082 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/116086235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/116086235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:22:55.408 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:22:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:55 smithi084 ceph-mon[39784]: pgmap v1188: 17 pgs: 3 active+recovering, 14 active+clean; 5.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 668 op/s; 14859/189428 objects misplaced (7.844%); 7.5 MiB/s, 119 objects/s recovering 2024-05-15T23:22:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:55.904 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:22:55.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:55.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:22:56.004 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.004 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.004 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.004 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.004 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.005 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.005 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.005 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:55 smithi088 ceph-mon[44699]: pgmap v1188: 17 pgs: 3 active+recovering, 14 active+clean; 5.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 668 op/s; 14859/189428 objects misplaced (7.844%); 7.5 MiB/s, 119 objects/s recovering 2024-05-15T23:22:56.005 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.005 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.005 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:22:56.063 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:56.093 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:22:56.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:56.100 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:22:56.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:55 smithi090 ceph-mon[44434]: pgmap v1188: 17 pgs: 3 active+recovering, 14 active+clean; 5.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 668 op/s; 14859/189428 objects misplaced (7.844%); 7.5 MiB/s, 119 objects/s recovering 2024-05-15T23:22:56.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.104 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.200 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:22:56.229 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:56.827 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.827 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:56.980 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:22:56.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:56.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:22:57.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:57.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:57.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:22:57.117 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:57.118 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:22:57.120 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:57.151 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:22:57.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:57.155 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:22:57.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:22:57.283 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:57.303 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:22:57.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:57.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:22:57.406 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:22:57.440 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:57.505 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:22:57.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:57.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:22:57.611 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:22:57.747 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:57.840 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:22:58.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:57 smithi088 ceph-mon[44699]: pgmap v1189: 17 pgs: 3 active+recovering, 14 active+clean; 5.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 642 op/s; 14349/185160 objects misplaced (7.750%); 8.0 MiB/s, 128 objects/s recovering 2024-05-15T23:22:58.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3761105992' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:22:58.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:57 smithi084 ceph-mon[39784]: pgmap v1189: 17 pgs: 3 active+recovering, 14 active+clean; 5.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 642 op/s; 14349/185160 objects misplaced (7.750%); 8.0 MiB/s, 128 objects/s recovering 2024-05-15T23:22:58.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3761105992' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:22:58.222 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:22:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:22:58.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:22:58.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:57 smithi090 ceph-mon[44434]: pgmap v1189: 17 pgs: 3 active+recovering, 14 active+clean; 5.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 642 op/s; 14349/185160 objects misplaced (7.750%); 8.0 MiB/s, 128 objects/s recovering 2024-05-15T23:22:58.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3761105992' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:22:58.329 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:22:58.391 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:22:58.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:58 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:22:58.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:58 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3761105992' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:22:58.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:58 smithi084 ceph-mon[39784]: osdmap e441: 12 total, 12 up, 9 in 2024-05-15T23:22:58.959 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:22:59.252 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:22:59.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:58 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:22:59.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:58 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3761105992' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:22:59.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:58 smithi090 ceph-mon[44434]: osdmap e441: 12 total, 12 up, 9 in 2024-05-15T23:22:59.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:58 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:22:59.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:58 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3761105992' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:22:59.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:58 smithi088 ceph-mon[44699]: osdmap e441: 12 total, 12 up, 9 in 2024-05-15T23:22:59.413 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:22:59.413 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:22:59.413 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:22:59.414 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:22:59.414 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:23:00.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:59 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:23:00.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:59 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:23:00.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:22:59 smithi084 ceph-mon[39784]: pgmap v1191: 17 pgs: 3 active+recovering, 14 active+clean; 5.6 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 759 op/s; 14114/182266 objects misplaced (7.744%); 9.1 MiB/s, 145 objects/s recovering 2024-05-15T23:23:00.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:59 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:23:00.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:59 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:23:00.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:22:59 smithi090 ceph-mon[44434]: pgmap v1191: 17 pgs: 3 active+recovering, 14 active+clean; 5.6 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 759 op/s; 14114/182266 objects misplaced (7.744%); 9.1 MiB/s, 145 objects/s recovering 2024-05-15T23:23:00.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:59 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:23:00.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:59 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:23:00.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:22:59 smithi088 ceph-mon[44699]: pgmap v1191: 17 pgs: 3 active+recovering, 14 active+clean; 5.6 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 759 op/s; 14114/182266 objects misplaced (7.744%); 9.1 MiB/s, 145 objects/s recovering 2024-05-15T23:23:00.381 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:23:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:00.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:23:00.486 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:23:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:00.649 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:23:00.676 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:23:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:00.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:23:00.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:23:00.830 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:01.035 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:23:01.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:01.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:23:01.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:00 smithi088 ceph-mon[44699]: osdmap e442: 12 total, 12 up, 9 in 2024-05-15T23:23:01.097 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:01.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:00 smithi084 ceph-mon[39784]: osdmap e442: 12 total, 12 up, 9 in 2024-05-15T23:23:01.140 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:23:01.169 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:00 smithi090 ceph-mon[44434]: osdmap e442: 12 total, 12 up, 9 in 2024-05-15T23:23:01.312 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T23:23:01.622 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:23:01.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:01.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:23:01.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:01.764 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:01.789 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:01.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:23:01.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:01.924 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:01.967 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:01.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:01.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:23:02.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:01 smithi090 ceph-mon[44434]: pgmap v1193: 17 pgs: 3 active+recovering, 14 active+clean; 5.4 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 795 op/s; 13606/177136 objects misplaced (7.681%); 9.8 MiB/s, 156 objects/s recovering 2024-05-15T23:23:02.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:23:02.111 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:02.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:01 smithi084 ceph-mon[39784]: pgmap v1193: 17 pgs: 3 active+recovering, 14 active+clean; 5.4 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 795 op/s; 13606/177136 objects misplaced (7.681%); 9.8 MiB/s, 156 objects/s recovering 2024-05-15T23:23:02.133 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:23:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:02.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:23:02.154 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:23:02.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:23:02.276 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:01 smithi088 ceph-mon[44699]: pgmap v1193: 17 pgs: 3 active+recovering, 14 active+clean; 5.4 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 795 op/s; 13606/177136 objects misplaced (7.681%); 9.8 MiB/s, 156 objects/s recovering 2024-05-15T23:23:02.479 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:23:02.479 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:23:02.479 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:23:02.479 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:23:02.479 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:23:02.479 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:23:02.480 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:23:02.480 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:23:02.480 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:23:02.480 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:23:02.480 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:23:02.480 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:23:02.480 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:23:02.480 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:23:02.480 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:23:02.480 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:23:02.481 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:23:02.481 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:23:02.481 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:23:02.481 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:23:02.481 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:23:02.481 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:23:02.481 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:23:02.481 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:23:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3369491143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:02 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:02 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:23:03.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:02 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:23:03.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3369491143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:03.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:02 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:02 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:23:03.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:02 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:23:03.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3369491143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:03.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:02 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:02 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:23:03.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:02 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:23:03.726 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:23:03.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:03.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:23:03.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:23:03.877 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:03.905 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:23:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:03.910 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:23:03.983 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:03.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:03 smithi084 ceph-mon[39784]: pgmap v1194: 17 pgs: 3 active+recovering, 14 active+clean; 5.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 950 op/s; 13361/174222 objects misplaced (7.669%); 12 MiB/s, 187 objects/s recovering 2024-05-15T23:23:03.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:03 smithi084 ceph-mon[39784]: osdmap e443: 12 total, 12 up, 9 in 2024-05-15T23:23:04.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:23:04.044 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:04.077 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:23:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:04.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:23:04.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:04.210 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:04.210 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:03 smithi090 ceph-mon[44434]: pgmap v1194: 17 pgs: 3 active+recovering, 14 active+clean; 5.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 950 op/s; 13361/174222 objects misplaced (7.669%); 12 MiB/s, 187 objects/s recovering 2024-05-15T23:23:04.210 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:03 smithi090 ceph-mon[44434]: osdmap e443: 12 total, 12 up, 9 in 2024-05-15T23:23:04.215 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:04.237 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:04.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:04.240 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:23:04.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:04.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:03 smithi088 ceph-mon[44699]: pgmap v1194: 17 pgs: 3 active+recovering, 14 active+clean; 5.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 950 op/s; 13361/174222 objects misplaced (7.669%); 12 MiB/s, 187 objects/s recovering 2024-05-15T23:23:04.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:03 smithi088 ceph-mon[44699]: osdmap e443: 12 total, 12 up, 9 in 2024-05-15T23:23:04.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:23:04.377 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:04.405 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:23:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:04.411 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:23:04.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:23:04.550 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:04.579 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:23:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:04.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:23:04.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:23:04.726 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:04.749 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:23:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:04.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:23:04.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:23:04.886 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:23:04.906 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:04.935 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:23:04.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:04.947 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:23:05.048 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:23:05.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/439374531' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:23:05.252 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:05.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:23:05.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/439374531' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:23:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/439374531' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:23:05.857 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:23:05.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:05.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:23:05.965 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:23:05.993 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:06.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:05 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:23:06.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:05 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/439374531' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:23:06.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:05 smithi090 ceph-mon[44434]: osdmap e444: 12 total, 12 up, 9 in 2024-05-15T23:23:06.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:05 smithi090 ceph-mon[44434]: pgmap v1197: 17 pgs: 3 active+recovering, 14 active+clean; 5.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 713 op/s; 13361/173706 objects misplaced (7.692%); 7.8 MiB/s, 125 objects/s recovering 2024-05-15T23:23:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:05 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:23:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:05 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/439374531' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:23:06.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:05 smithi088 ceph-mon[44699]: osdmap e444: 12 total, 12 up, 9 in 2024-05-15T23:23:06.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:05 smithi088 ceph-mon[44699]: pgmap v1197: 17 pgs: 3 active+recovering, 14 active+clean; 5.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 713 op/s; 13361/173706 objects misplaced (7.692%); 7.8 MiB/s, 125 objects/s recovering 2024-05-15T23:23:06.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:05 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:23:06.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:05 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/439374531' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:23:06.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:05 smithi084 ceph-mon[39784]: osdmap e444: 12 total, 12 up, 9 in 2024-05-15T23:23:06.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:05 smithi084 ceph-mon[39784]: pgmap v1197: 17 pgs: 3 active+recovering, 14 active+clean; 5.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 713 op/s; 13361/173706 objects misplaced (7.692%); 7.8 MiB/s, 125 objects/s recovering 2024-05-15T23:23:06.784 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-15T23:23:06.785 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:23:07.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:06 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:23:07.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:06 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:23:07.145 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:23:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:07.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:23:07.197 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:07.226 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:07.248 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:23:07.293 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:07.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:06 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:23:07.367 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:23:07.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:07.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:23:07.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:07.507 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:07.530 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:07.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:07.533 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:23:07.576 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:23:07.577 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:23:07.633 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:23:07.663 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:07.686 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:23:07.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:07.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:23:07.794 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:23:07.853 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:07.918 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:07 smithi088 ceph-mon[44699]: pgmap v1198: 17 pgs: 3 active+recovering, 14 active+clean; 5.2 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 914 op/s; 12886/169182 objects misplaced (7.617%); 11 MiB/s, 171 objects/s recovering 2024-05-15T23:23:08.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:07 smithi084 ceph-mon[39784]: pgmap v1198: 17 pgs: 3 active+recovering, 14 active+clean; 5.2 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 914 op/s; 12886/169182 objects misplaced (7.617%); 11 MiB/s, 171 objects/s recovering 2024-05-15T23:23:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:07 smithi090 ceph-mon[44434]: pgmap v1198: 17 pgs: 3 active+recovering, 14 active+clean; 5.2 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 914 op/s; 12886/169182 objects misplaced (7.617%); 11 MiB/s, 171 objects/s recovering 2024-05-15T23:23:08.330 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:23:08.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:08.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:23:08.439 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:23:08.498 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:08.859 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T23:23:08.863 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:23:08.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:08.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:23:08.969 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:23:09.066 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:09.131 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:23:09.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:09.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:23:09.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:23:09.179 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:09 smithi084 ceph-mon[39784]: pgmap v1199: 17 pgs: 3 active+recovering, 14 active+clean; 5.1 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 684 op/s; 12648/166186 objects misplaced (7.611%); 7.5 MiB/s, 119 objects/s recovering 2024-05-15T23:23:09.245 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:23:09.273 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:09.385 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:09 smithi088 ceph-mon[44699]: pgmap v1199: 17 pgs: 3 active+recovering, 14 active+clean; 5.1 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 684 op/s; 12648/166186 objects misplaced (7.611%); 7.5 MiB/s, 119 objects/s recovering 2024-05-15T23:23:09.401 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:23:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:09.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:23:09.506 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:23:09.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:09 smithi090 ceph-mon[44434]: pgmap v1199: 17 pgs: 3 active+recovering, 14 active+clean; 5.1 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 684 op/s; 12648/166186 objects misplaced (7.611%); 7.5 MiB/s, 119 objects/s recovering 2024-05-15T23:23:09.592 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:10.376 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:23:10.398 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:23:10.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:23:10.441 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:23:10.465 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:23:10.493 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:23:10.523 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:23:10.550 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:23:10.576 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:23:10.598 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:23:10.626 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:23:10.651 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:23:10.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:23:10.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:23:10.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:23:10.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:23:10.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:23:10.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:23:10.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:23:10.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:23:10.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:23:10.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:23:10.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:23:10.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:23:10.679 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:10 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2296563763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:10.711 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:10.784 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:23:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:10 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2296563763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:10.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:10 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2296563763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:10.855 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:23:10.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:10.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:23:10.966 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:23:10.983 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:23:10.985 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:23:10.999 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:11.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:11 smithi088 ceph-mon[44699]: from='client.16185 v1:172.21.15.84:0/1673600798' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:11.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:11 smithi088 ceph-mon[44699]: pgmap v1200: 17 pgs: 3 active+recovering, 14 active+clean; 4.9 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 798 op/s; 12146/161446 objects misplaced (7.523%); 9.5 MiB/s, 151 objects/s recovering 2024-05-15T23:23:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:11 smithi090 ceph-mon[44434]: from='client.16185 v1:172.21.15.84:0/1673600798' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:11.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:11 smithi090 ceph-mon[44434]: pgmap v1200: 17 pgs: 3 active+recovering, 14 active+clean; 4.9 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 798 op/s; 12146/161446 objects misplaced (7.523%); 9.5 MiB/s, 151 objects/s recovering 2024-05-15T23:23:11.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:11.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:11 smithi084 ceph-mon[39784]: from='client.16185 v1:172.21.15.84:0/1673600798' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:11.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:11 smithi084 ceph-mon[39784]: pgmap v1200: 17 pgs: 3 active+recovering, 14 active+clean; 4.9 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 798 op/s; 12146/161446 objects misplaced (7.523%); 9.5 MiB/s, 151 objects/s recovering 2024-05-15T23:23:12.351 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:23:12.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:12.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:23:12.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:23:12.492 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:12.516 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:23:12.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:12.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:23:12.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:12.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:23:12.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:23:12.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:12.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:23:12.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:23:12.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:12.667 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:12.694 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:12.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:12.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:23:12.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:23:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:12.705 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:23:12.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:23:12.840 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:12.941 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:23:12.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:12.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:23:13.050 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:23:13.087 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:13.382 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1199,"stamp":"2024-05-15T23:23:09.001275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5445976632,"num_objects":83093,"num_object_clones":0,"num_object_copies":166186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12648,"num_objects_unfound":0,"num_objects_dirty":83093,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":224148,"num_write_kb":9829799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13247,"num_bytes_recovered":869348960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161219,"ondisk_log_size":161219,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":13203892,"kb_used_data":9898120,"kb_used_omap":14,"kb_used_meta":3305585,"kb_avail":830502476,"statfs":{"total":863955320832,"available":850434535424,"internally_reserved":0,"allocated":10135674880,"data_stored":10132101150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14358,"internal_metadata":3384920042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-358809600,"num_objects":-5475,"num_object_clones":0,"num_object_copies":-10950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-958,"num_objects_unfound":0,"num_objects_dirty":-5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5475,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":958,"num_bytes_recovered":62783488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001760"},"pg_stats":[{"pgid":"5.f","version":"444'13877","reported_seq":14027,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:07.473646+0000","last_change":"2024-05-15T23:19:47.305945+0000","last_active":"2024-05-15T23:23:07.473646+0000","last_peered":"2024-05-15T23:23:07.473646+0000","last_clean":"2024-05-15T23:23:07.473646+0000","last_became_active":"2024-05-15T23:19:47.305689+0000","last_became_peered":"2024-05-15T23:19:47.305689+0000","last_unstale":"2024-05-15T23:23:07.473646+0000","last_undegraded":"2024-05-15T23:23:07.473646+0000","last_fullsized":"2024-05-15T23:23:07.473646+0000","mapping_epoch":374,"log_start":"386'3800","ondisk_log_start":"386'3800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339673088,"num_objects":5183,"num_object_clones":0,"num_object_copies":10366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13877,"num_write_kb":609920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"444'13773","reported_seq":14098,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:05.551015+0000","last_change":"2024-05-15T23:22:23.637106+0000","last_active":"2024-05-15T23:23:05.551015+0000","last_peered":"2024-05-15T23:23:05.551015+0000","last_clean":"2024-05-15T23:23:05.551015+0000","last_became_active":"2024-05-15T23:19:47.325414+0000","last_became_peered":"2024-05-15T23:19:47.325414+0000","last_unstale":"2024-05-15T23:23:05.551015+0000","last_undegraded":"2024-05-15T23:23:05.551015+0000","last_fullsized":"2024-05-15T23:23:05.551015+0000","mapping_epoch":374,"log_start":"385'3700","ondisk_log_start":"385'3700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'11848","last_scrub_stamp":"2024-05-15T23:22:23.636778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:23.636778+0000","objects_scrubbed":7095,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:23:39.191980+0000","scrub_duration":997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337444864,"num_objects":5149,"num_object_clones":0,"num_object_copies":10298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13773,"num_write_kb":605504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"444'14264","reported_seq":14414,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:06.999490+0000","last_change":"2024-05-15T23:19:47.306160+0000","last_active":"2024-05-15T23:23:06.999490+0000","last_peered":"2024-05-15T23:23:06.999490+0000","last_clean":"2024-05-15T23:23:06.999490+0000","last_became_active":"2024-05-15T23:19:47.305860+0000","last_became_peered":"2024-05-15T23:19:47.305860+0000","last_unstale":"2024-05-15T23:23:06.999490+0000","last_undegraded":"2024-05-15T23:23:06.999490+0000","last_fullsized":"2024-05-15T23:23:06.999490+0000","mapping_epoch":374,"log_start":"386'4200","ondisk_log_start":"386'4200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339542040,"num_objects":5182,"num_object_clones":0,"num_object_copies":10364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5182,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14264,"num_write_kb":622209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"444'14089","reported_seq":14402,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:05.824847+0000","last_change":"2024-05-15T23:22:51.995640+0000","last_active":"2024-05-15T23:23:05.824847+0000","last_peered":"2024-05-15T23:23:05.824847+0000","last_clean":"2024-05-15T23:23:05.824847+0000","last_became_active":"2024-05-15T23:19:47.306168+0000","last_became_peered":"2024-05-15T23:19:47.306168+0000","last_unstale":"2024-05-15T23:23:05.824847+0000","last_undegraded":"2024-05-15T23:23:05.824847+0000","last_fullsized":"2024-05-15T23:23:05.824847+0000","mapping_epoch":374,"log_start":"386'4000","ondisk_log_start":"386'4000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'13335","last_scrub_stamp":"2024-05-15T23:22:51.995473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:51.995473+0000","objects_scrubbed":6018,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:13.168527+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342687744,"num_objects":5229,"num_object_clones":0,"num_object_copies":10458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14089,"num_write_kb":618176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"444'13986","reported_seq":14301,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:05.529503+0000","last_change":"2024-05-15T23:22:48.338084+0000","last_active":"2024-05-15T23:23:05.529503+0000","last_peered":"2024-05-15T23:23:05.529503+0000","last_clean":"2024-05-15T23:23:05.529503+0000","last_became_active":"2024-05-15T23:19:47.307061+0000","last_became_peered":"2024-05-15T23:19:47.307061+0000","last_unstale":"2024-05-15T23:23:05.529503+0000","last_undegraded":"2024-05-15T23:23:05.529503+0000","last_fullsized":"2024-05-15T23:23:05.529503+0000","mapping_epoch":374,"log_start":"386'3900","ondisk_log_start":"386'3900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'13172","last_scrub_stamp":"2024-05-15T23:22:48.337866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:48.337866+0000","objects_scrubbed":6060,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:11.906769+0000","scrub_duration":1844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341180416,"num_objects":5206,"num_object_clones":0,"num_object_copies":10412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13986,"num_write_kb":614144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"444'13970","reported_seq":14120,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:05.486298+0000","last_change":"2024-05-15T23:19:47.325657+0000","last_active":"2024-05-15T23:23:05.486298+0000","last_peered":"2024-05-15T23:23:05.486298+0000","last_clean":"2024-05-15T23:23:05.486298+0000","last_became_active":"2024-05-15T23:19:47.325371+0000","last_became_peered":"2024-05-15T23:19:47.325371+0000","last_unstale":"2024-05-15T23:23:05.486298+0000","last_undegraded":"2024-05-15T23:23:05.486298+0000","last_fullsized":"2024-05-15T23:23:05.486298+0000","mapping_epoch":374,"log_start":"386'3900","ondisk_log_start":"386'3900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343539712,"num_objects":5242,"num_object_clones":0,"num_object_copies":10484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13970,"num_write_kb":614784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"444'13998","reported_seq":22992,"reported_epoch":444,"state":"active+recovering","last_fresh":"2024-05-15T23:23:05.894976+0000","last_change":"2024-05-15T23:21:09.568311+0000","last_active":"2024-05-15T23:23:05.894976+0000","last_peered":"2024-05-15T23:23:05.894976+0000","last_clean":"2024-05-15T23:21:08.023957+0000","last_became_active":"2024-05-15T23:21:09.454639+0000","last_became_peered":"2024-05-15T23:21:09.454639+0000","last_unstale":"2024-05-15T23:23:05.894976+0000","last_undegraded":"2024-05-15T23:23:05.894976+0000","last_fullsized":"2024-05-15T23:23:05.894976+0000","mapping_epoch":407,"log_start":"386'3900","ondisk_log_start":"386'3900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337903616,"num_objects":5156,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4149,"num_objects_unfound":0,"num_objects_dirty":5156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13998,"num_write_kb":612928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4421,"num_bytes_recovered":289734656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":4149},{"shards":"7,11","objects":1007}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"444'14071","reported_seq":14221,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:07.043533+0000","last_change":"2024-05-15T23:19:47.304832+0000","last_active":"2024-05-15T23:23:07.043533+0000","last_peered":"2024-05-15T23:23:07.043533+0000","last_clean":"2024-05-15T23:23:07.043533+0000","last_became_active":"2024-05-15T23:19:47.304547+0000","last_became_peered":"2024-05-15T23:19:47.304547+0000","last_unstale":"2024-05-15T23:23:07.043533+0000","last_undegraded":"2024-05-15T23:23:07.043533+0000","last_fullsized":"2024-05-15T23:23:07.043533+0000","mapping_epoch":374,"log_start":"386'4000","ondisk_log_start":"386'4000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336396288,"num_objects":5133,"num_object_clones":0,"num_object_copies":10266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14071,"num_write_kb":614528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"444'13966","reported_seq":14116,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:07.433595+0000","last_change":"2024-05-15T23:19:47.306634+0000","last_active":"2024-05-15T23:23:07.433595+0000","last_peered":"2024-05-15T23:23:07.433595+0000","last_clean":"2024-05-15T23:23:07.433595+0000","last_became_active":"2024-05-15T23:19:47.306331+0000","last_became_peered":"2024-05-15T23:19:47.306331+0000","last_unstale":"2024-05-15T23:23:07.433595+0000","last_undegraded":"2024-05-15T23:23:07.433595+0000","last_fullsized":"2024-05-15T23:23:07.433595+0000","mapping_epoch":374,"log_start":"386'3900","ondisk_log_start":"386'3900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333578240,"num_objects":5090,"num_object_clones":0,"num_object_copies":10180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13966,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"444'14141","reported_seq":14290,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:07.451162+0000","last_change":"2024-05-15T23:19:47.306611+0000","last_active":"2024-05-15T23:23:07.451162+0000","last_peered":"2024-05-15T23:23:07.451162+0000","last_clean":"2024-05-15T23:23:07.451162+0000","last_became_active":"2024-05-15T23:19:47.306349+0000","last_became_peered":"2024-05-15T23:19:47.306349+0000","last_unstale":"2024-05-15T23:23:07.451162+0000","last_undegraded":"2024-05-15T23:23:07.451162+0000","last_fullsized":"2024-05-15T23:23:07.451162+0000","mapping_epoch":374,"log_start":"386'4100","ondisk_log_start":"386'4100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338755584,"num_objects":5169,"num_object_clones":0,"num_object_copies":10338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14141,"num_write_kb":617920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"444'14156","reported_seq":14536,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:05.809026+0000","last_change":"2024-05-15T23:22:48.067681+0000","last_active":"2024-05-15T23:23:05.809026+0000","last_peered":"2024-05-15T23:23:05.809026+0000","last_clean":"2024-05-15T23:23:05.809026+0000","last_became_active":"2024-05-15T23:19:47.305685+0000","last_became_peered":"2024-05-15T23:19:47.305685+0000","last_unstale":"2024-05-15T23:23:05.809026+0000","last_undegraded":"2024-05-15T23:23:05.809026+0000","last_fullsized":"2024-05-15T23:23:05.809026+0000","mapping_epoch":374,"log_start":"386'4100","ondisk_log_start":"386'4100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'13277","last_scrub_stamp":"2024-05-15T23:22:48.067507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:48.067507+0000","objects_scrubbed":6142,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:23:49.704509+0000","scrub_duration":929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343408640,"num_objects":5240,"num_object_clones":0,"num_object_copies":10480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14156,"num_write_kb":620672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"444'14059","reported_seq":23049,"reported_epoch":444,"state":"active+recovering","last_fresh":"2024-05-15T23:23:07.475826+0000","last_change":"2024-05-15T23:21:09.470461+0000","last_active":"2024-05-15T23:23:07.475826+0000","last_peered":"2024-05-15T23:23:07.475826+0000","last_clean":"2024-05-15T23:21:07.989871+0000","last_became_active":"2024-05-15T23:21:09.400731+0000","last_became_peered":"2024-05-15T23:21:09.400731+0000","last_unstale":"2024-05-15T23:23:07.475826+0000","last_undegraded":"2024-05-15T23:23:07.475826+0000","last_fullsized":"2024-05-15T23:23:07.475826+0000","mapping_epoch":407,"log_start":"386'4000","ondisk_log_start":"386'4000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343212032,"num_objects":5237,"num_object_clones":0,"num_object_copies":10474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4271,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14059,"num_write_kb":617472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4419,"num_bytes_recovered":289603584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"3,5","objects":966},{"shards":"5,6","objects":4271}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"444'13982","reported_seq":22937,"reported_epoch":444,"state":"active+recovering","last_fresh":"2024-05-15T23:23:05.503322+0000","last_change":"2024-05-15T23:21:09.275620+0000","last_active":"2024-05-15T23:23:05.503322+0000","last_peered":"2024-05-15T23:23:05.503322+0000","last_clean":"2024-05-15T23:21:07.993692+0000","last_became_active":"2024-05-15T23:21:09.221797+0000","last_became_peered":"2024-05-15T23:21:09.221797+0000","last_unstale":"2024-05-15T23:23:05.503322+0000","last_undegraded":"2024-05-15T23:23:05.503322+0000","last_fullsized":"2024-05-15T23:23:05.503322+0000","mapping_epoch":407,"log_start":"386'3900","ondisk_log_start":"386'3900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340000768,"num_objects":5188,"num_object_clones":0,"num_object_copies":10376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4228,"num_objects_unfound":0,"num_objects_dirty":5188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13982,"num_write_kb":613440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4401,"num_bytes_recovered":288423936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":4228},{"shards":"7,10","objects":960}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.1","version":"443'13580","reported_seq":13901,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-05-15T23:23:04.811625+0000","last_change":"2024-05-15T23:22:19.625857+0000","last_active":"2024-05-15T23:23:04.811625+0000","last_peered":"2024-05-15T23:23:04.811625+0000","last_clean":"2024-05-15T23:23:04.811625+0000","last_became_active":"2024-05-15T23:19:47.516865+0000","last_became_peered":"2024-05-15T23:19:47.516865+0000","last_unstale":"2024-05-15T23:23:04.811625+0000","last_undegraded":"2024-05-15T23:23:04.811625+0000","last_fullsized":"2024-05-15T23:23:04.811625+0000","mapping_epoch":374,"log_start":"385'3500","ondisk_log_start":"385'3500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"426'11474","last_scrub_stamp":"2024-05-15T23:22:19.625719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:19.625719+0000","objects_scrubbed":7211,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:23:49.523745+0000","scrub_duration":1229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332791808,"num_objects":5078,"num_object_clones":0,"num_object_copies":10156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13580,"num_write_kb":597056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"444'14050","reported_seq":14200,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:05.486046+0000","last_change":"2024-05-15T23:19:47.305723+0000","last_active":"2024-05-15T23:23:05.486046+0000","last_peered":"2024-05-15T23:23:05.486046+0000","last_clean":"2024-05-15T23:23:05.486046+0000","last_became_active":"2024-05-15T23:19:47.305459+0000","last_became_peered":"2024-05-15T23:19:47.305459+0000","last_unstale":"2024-05-15T23:23:05.486046+0000","last_undegraded":"2024-05-15T23:23:05.486046+0000","last_fullsized":"2024-05-15T23:23:05.486046+0000","mapping_epoch":374,"log_start":"386'4000","ondisk_log_start":"386'4000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342491136,"num_objects":5226,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14050,"num_write_kb":616832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"444'14063","reported_seq":14213,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:05.504955+0000","last_change":"2024-05-15T23:19:47.307499+0000","last_active":"2024-05-15T23:23:05.504955+0000","last_peered":"2024-05-15T23:23:05.504955+0000","last_clean":"2024-05-15T23:23:05.504955+0000","last_became_active":"2024-05-15T23:19:47.307149+0000","last_became_peered":"2024-05-15T23:19:47.307149+0000","last_unstale":"2024-05-15T23:23:05.504955+0000","last_undegraded":"2024-05-15T23:23:05.504955+0000","last_fullsized":"2024-05-15T23:23:05.504955+0000","mapping_epoch":374,"log_start":"386'4000","ondisk_log_start":"386'4000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352780288,"num_objects":5383,"num_object_clones":0,"num_object_copies":10766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14063,"num_write_kb":622272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":927,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:04.892758+0000","last_change":"2024-05-15T23:20:05.905241+0000","last_active":"2024-05-15T23:23:04.892758+0000","last_peered":"2024-05-15T23:23:04.892758+0000","last_clean":"2024-05-15T23:23:04.892758+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:23:04.892758+0000","last_undegraded":"2024-05-15T23:23:04.892758+0000","last_fullsized":"2024-05-15T23:23:04.892758+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:20:05.905105+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:20:05.905105+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":5445386264,"num_objects":83091,"num_object_clones":0,"num_object_copies":166182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12648,"num_objects_unfound":0,"num_objects_dirty":83091,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224025,"num_write_kb":9827649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13241,"num_bytes_recovered":867762176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11777220608,"data_stored":11777212464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161125,"ondisk_log_size":161125,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351208,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1264416,"kb_used_data":727892,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92480736,"statfs":{"total":95995035648,"available":94700273664,"internally_reserved":0,"allocated":745361408,"data_stored":744957034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":549386692},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580141,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1950228,"kb_used_data":1684424,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":91794924,"statfs":{"total":95995035648,"available":93998002176,"internally_reserved":0,"allocated":1724850176,"data_stored":1724453446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":272169408},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1020760,"kb_used_data":664712,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92724392,"statfs":{"total":95995035648,"available":94949777408,"internally_reserved":0,"allocated":680665088,"data_stored":680268358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":364575168},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":828360,"kb_used_data":675004,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92916792,"statfs":{"total":95995035648,"available":95146795008,"internally_reserved":0,"allocated":691204096,"data_stored":690813420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266941,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1080672,"kb_used_data":734352,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92664480,"statfs":{"total":95995035648,"available":94888427520,"internally_reserved":0,"allocated":751976448,"data_stored":751576166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":354613706},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495876,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575133,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1804944,"kb_used_data":1344776,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":91940208,"statfs":{"total":95995035648,"available":94146772992,"internally_reserved":0,"allocated":1377050624,"data_stored":1376653894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":471202246},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016542,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483231,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149971,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":804372,"kb_used_data":385864,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92940780,"statfs":{"total":95995035648,"available":95171358720,"internally_reserved":0,"allocated":395124736,"data_stored":394728006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":428538311},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986444,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1748100,"kb_used_data":1347644,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91997052,"statfs":{"total":95995035648,"available":94204981248,"internally_reserved":0,"allocated":1379987456,"data_stored":1379596780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790217,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2702040,"kb_used_data":2333452,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":91043112,"statfs":{"total":95995035648,"available":93228146688,"internally_reserved":0,"allocated":2389454848,"data_stored":2389054046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377354693},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1365508096,"data_stored":1365508096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":392691712,"data_stored":392691712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1374093312,"data_stored":1374093312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2375553024,"data_stored":2375548952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1696661504,"data_stored":1696661504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":744816640,"data_stored":744816640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":687276032,"data_stored":687276032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":678559744,"data_stored":678559744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1719992320,"data_stored":1719992320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":742068224,"data_stored":742064152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:23:13.383 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:23:13.426 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:13 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:23:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:13 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:23:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:13 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:23:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:13 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:23:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:13 smithi084 ceph-mon[39784]: pgmap v1201: 17 pgs: 3 active+recovering, 14 active+clean; 4.8 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 792 op/s; 11902/158436 objects misplaced (7.512%); 9.2 MiB/s, 146 objects/s recovering 2024-05-15T23:23:13.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:13 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:23:13.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:13 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:23:13.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:13 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:23:13.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:13 smithi090 ceph-mon[44434]: pgmap v1201: 17 pgs: 3 active+recovering, 14 active+clean; 4.8 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 792 op/s; 11902/158436 objects misplaced (7.512%); 9.2 MiB/s, 146 objects/s recovering 2024-05-15T23:23:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:13 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:23:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:13 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:23:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:13 smithi088 ceph-mon[44699]: pgmap v1201: 17 pgs: 3 active+recovering, 14 active+clean; 4.8 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 792 op/s; 11902/158436 objects misplaced (7.512%); 9.2 MiB/s, 146 objects/s recovering 2024-05-15T23:23:13.979 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:14.038 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:23:14.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:14.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:23:14.143 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:23:14.174 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:14.253 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:23:14.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:14.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:23:14.357 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:23:14.398 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:14.565 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:23:14.727 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:14 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2327916733' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:23:14.771 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:23:14.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:14.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:23:14.851 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:23:14.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:14 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2327916733' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:23:14.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:14.948 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:14 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2327916733' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:23:14.948 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:14.982 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:14.985 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:23:15.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:23:15.114 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:15.139 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:23:15.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:15.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:23:15.165 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:23:15.165 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:23:15.166 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.166 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:230993 5.3 5:ce3aa536:::benchmark_data_smithi090_116470_object77433:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:23:15.166 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:15.148696+0000", 2024-05-15T23:23:15.166 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.014590093, 2024-05-15T23:23:15.166 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.014619185999999999, 2024-05-15T23:23:15.166 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:15.166 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:15.166 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:23:15.166 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:15.166 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:23:15.166 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:23:15.166 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 230993 2024-05-15T23:23:15.166 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.167 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:15.167 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.167 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:15.167 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.148696+0000", 2024-05-15T23:23:15.167 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:15.167 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.167 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.167 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:15.167 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.148696+0000", 2024-05-15T23:23:15.167 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:15.167 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.167 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.168 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:15.168 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.148698+0000", 2024-05-15T23:23:15.168 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3120000000000001e-06 2024-05-15T23:23:15.168 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.168 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.168 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:15.168 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.148705+0000", 2024-05-15T23:23:15.168 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.178e-06 2024-05-15T23:23:15.168 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.168 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.168 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:15.168 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.148719+0000", 2024-05-15T23:23:15.168 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4926000000000001e-05 2024-05-15T23:23:15.169 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.169 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.169 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:15.169 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.148728+0000", 2024-05-15T23:23:15.169 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2390000000000002e-06 2024-05-15T23:23:15.169 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.169 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.169 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:15.169 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.151760+0000", 2024-05-15T23:23:15.169 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030322589999999998 2024-05-15T23:23:15.169 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.169 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.169 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:23:15.170 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.151816+0000", 2024-05-15T23:23:15.170 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5829999999999999e-05 2024-05-15T23:23:15.170 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.170 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.170 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:15.170 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.161752+0000", 2024-05-15T23:23:15.170 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0099356779999999999 2024-05-15T23:23:15.170 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.170 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.170 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:15.170 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.161825+0000", 2024-05-15T23:23:15.170 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3342000000000004e-05 2024-05-15T23:23:15.170 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.171 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.171 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:23:15.171 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.161913+0000", 2024-05-15T23:23:15.171 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8253999999999999e-05 2024-05-15T23:23:15.171 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:15.171 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:15.171 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:15.171 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.171 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.171 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPushReply(5.3 445/407 [PushReplyOp(5:ce3aa536:::benchmark_data_smithi090_116470_object77433:head)])", 2024-05-15T23:23:15.171 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:15.153547+0000", 2024-05-15T23:23:15.171 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.0097394999999999999, 2024-05-15T23:23:15.172 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0098608510000000003, 2024-05-15T23:23:15.172 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:15.172 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:15.172 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2024-05-15T23:23:15.172 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:15.172 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.172 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:15.172 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.153547+0000", 2024-05-15T23:23:15.172 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:15.172 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.172 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.172 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:15.172 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.153547+0000", 2024-05-15T23:23:15.172 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:15.173 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.173 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.173 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:15.173 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.153548+0000", 2024-05-15T23:23:15.173 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.7320000000000001e-06 2024-05-15T23:23:15.173 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.173 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.173 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:15.173 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.153556+0000", 2024-05-15T23:23:15.173 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8800000000000008e-06 2024-05-15T23:23:15.173 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.173 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.174 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:15.174 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.153571+0000", 2024-05-15T23:23:15.174 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4399000000000001e-05 2024-05-15T23:23:15.174 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.174 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.174 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:15.174 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.153581+0000", 2024-05-15T23:23:15.174 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0171e-05 2024-05-15T23:23:15.174 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.174 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.174 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:15.174 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.160224+0000", 2024-05-15T23:23:15.174 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0066430100000000004 2024-05-15T23:23:15.174 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:15.175 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:15.175 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:15.175 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.175 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.175 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.25840.0:231000 5.d e445/374 5:bf3d4672:::benchmark_data_smithi090_116470_object77440:head v 445'14666, mlcod=445'14664)", 2024-05-15T23:23:15.175 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:15.162876+0000", 2024-05-15T23:23:15.175 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.00041045200000000001, 2024-05-15T23:23:15.175 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00058712599999999999, 2024-05-15T23:23:15.175 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:15.175 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:15.175 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2024-05-15T23:23:15.175 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:15.175 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.176 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:15.176 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.162876+0000", 2024-05-15T23:23:15.176 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:15.176 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.176 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.176 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:15.176 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.162876+0000", 2024-05-15T23:23:15.176 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:15.176 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.176 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.176 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:15.176 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.162878+0000", 2024-05-15T23:23:15.176 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.136e-06 2024-05-15T23:23:15.177 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.177 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.177 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:15.177 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.162887+0000", 2024-05-15T23:23:15.177 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.1630000000000002e-06 2024-05-15T23:23:15.177 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.177 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.177 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:15.177 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.162898+0000", 2024-05-15T23:23:15.177 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0587e-05 2024-05-15T23:23:15.177 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.177 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.177 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:15.177 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.162906+0000", 2024-05-15T23:23:15.178 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0239999999999997e-06 2024-05-15T23:23:15.178 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.178 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.178 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:15.178 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.162959+0000", 2024-05-15T23:23:15.178 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3847000000000001e-05 2024-05-15T23:23:15.178 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.178 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.178 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:15.178 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.162982+0000", 2024-05-15T23:23:15.179 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2169000000000001e-05 2024-05-15T23:23:15.179 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:15.179 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:15.179 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:15.179 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.179 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.179 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:230968 5.3 5:c834fe18:::benchmark_data_smithi090_116470_object77408:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:23:15.179 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:15.106359+0000", 2024-05-15T23:23:15.179 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.056927138000000002, 2024-05-15T23:23:15.179 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.057163936999999998, 2024-05-15T23:23:15.179 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:15.179 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:15.179 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:23:15.179 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:15.179 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:23:15.180 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:23:15.180 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 230968 2024-05-15T23:23:15.180 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.180 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:15.180 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.180 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:15.180 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.106359+0000", 2024-05-15T23:23:15.180 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:15.180 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.180 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.180 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:15.180 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.106359+0000", 2024-05-15T23:23:15.180 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:15.180 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.180 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.181 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:15.181 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.106362+0000", 2024-05-15T23:23:15.181 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.565e-06 2024-05-15T23:23:15.181 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.181 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.181 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:15.181 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.106370+0000", 2024-05-15T23:23:15.181 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8200000000000003e-06 2024-05-15T23:23:15.181 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.181 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.181 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:15.181 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.106386+0000", 2024-05-15T23:23:15.181 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5526e-05 2024-05-15T23:23:15.181 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.182 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.182 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:15.182 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.106395+0000", 2024-05-15T23:23:15.182 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4870000000000008e-06 2024-05-15T23:23:15.182 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.182 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.182 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:15.182 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.113273+0000", 2024-05-15T23:23:15.182 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068772429999999999 2024-05-15T23:23:15.182 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.182 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.182 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:23:15.182 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.113337+0000", 2024-05-15T23:23:15.183 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4676000000000002e-05 2024-05-15T23:23:15.183 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.183 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.183 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:15.183 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.116702+0000", 2024-05-15T23:23:15.183 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033651269999999999 2024-05-15T23:23:15.183 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.183 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.183 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:15.183 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.116785+0000", 2024-05-15T23:23:15.183 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2962999999999998e-05 2024-05-15T23:23:15.183 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.183 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.183 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:23:15.184 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.116910+0000", 2024-05-15T23:23:15.184 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000124454 2024-05-15T23:23:15.184 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.184 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.184 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:15.184 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.124024+0000", 2024-05-15T23:23:15.184 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.007114357 2024-05-15T23:23:15.184 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:15.184 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:15.184 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:15.184 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.184 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.184 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:230971 5.3 5:c9c30ddb:::benchmark_data_smithi090_116470_object77411:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:23:15.184 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:15.113465+0000", 2024-05-15T23:23:15.185 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.049820845000000002, 2024-05-15T23:23:15.185 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.050148915000000002, 2024-05-15T23:23:15.185 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:15.185 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:15.185 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:23:15.185 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:15.185 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:23:15.185 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:23:15.185 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 230971 2024-05-15T23:23:15.185 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.185 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:15.185 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.185 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:15.186 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.113465+0000", 2024-05-15T23:23:15.186 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:15.186 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.186 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.186 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:15.186 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.113465+0000", 2024-05-15T23:23:15.186 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:15.186 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.186 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.186 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:15.186 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.113467+0000", 2024-05-15T23:23:15.186 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0999999999999998e-06 2024-05-15T23:23:15.186 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.186 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.187 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:15.187 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.113475+0000", 2024-05-15T23:23:15.187 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1849999999999998e-06 2024-05-15T23:23:15.187 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.187 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.187 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:15.187 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.113487+0000", 2024-05-15T23:23:15.187 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1919000000000001e-05 2024-05-15T23:23:15.188 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.188 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.188 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:15.188 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.113496+0000", 2024-05-15T23:23:15.188 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7750000000000004e-06 2024-05-15T23:23:15.188 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.188 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.188 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:15.188 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.123874+0000", 2024-05-15T23:23:15.188 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010378049 2024-05-15T23:23:15.188 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.188 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.188 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:23:15.188 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.123939+0000", 2024-05-15T23:23:15.189 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4488999999999999e-05 2024-05-15T23:23:15.189 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.189 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.189 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:15.189 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.132284+0000", 2024-05-15T23:23:15.189 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0083455319999999993 2024-05-15T23:23:15.189 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.189 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.189 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:15.189 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.132366+0000", 2024-05-15T23:23:15.189 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1233e-05 2024-05-15T23:23:15.189 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.189 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.190 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:23:15.190 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.132508+0000", 2024-05-15T23:23:15.190 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014276800000000001 2024-05-15T23:23:15.190 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.190 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.190 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:15.190 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.144491+0000", 2024-05-15T23:23:15.190 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011982272 2024-05-15T23:23:15.190 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:15.190 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:15.190 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:15.190 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.190 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.190 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.25840.0:230974 5.b e445/374 5:d83cfcfa:::benchmark_data_smithi090_116470_object77414:head v 445'14431, mlcod=445'14426)", 2024-05-15T23:23:15.191 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:15.118160+0000", 2024-05-15T23:23:15.191 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.045126500999999999, 2024-05-15T23:23:15.191 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.045543154000000002, 2024-05-15T23:23:15.191 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:15.191 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:15.191 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2024-05-15T23:23:15.191 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:15.191 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.191 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:15.191 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.118160+0000", 2024-05-15T23:23:15.191 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:15.191 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.191 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:15.192 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.118160+0000", 2024-05-15T23:23:15.192 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:15.192 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.192 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:15.192 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.118161+0000", 2024-05-15T23:23:15.192 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.7099999999999999e-06 2024-05-15T23:23:15.192 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.192 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:15.192 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.118172+0000", 2024-05-15T23:23:15.192 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0502999999999999e-05 2024-05-15T23:23:15.192 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.192 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.193 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:15.193 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.118181+0000", 2024-05-15T23:23:15.193 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.9600000000000006e-06 2024-05-15T23:23:15.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.193 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.193 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:15.193 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.118189+0000", 2024-05-15T23:23:15.193 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7680000000000008e-06 2024-05-15T23:23:15.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.193 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.193 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:15.193 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.119269+0000", 2024-05-15T23:23:15.193 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0010801020000000001 2024-05-15T23:23:15.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.194 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.194 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:15.194 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.119288+0000", 2024-05-15T23:23:15.194 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9738999999999999e-05 2024-05-15T23:23:15.194 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:15.194 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:15.194 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:15.194 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.194 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.194 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:230987 5.3 5:c04b18a2:::benchmark_data_smithi090_116470_object77427:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:23:15.194 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:15.139681+0000", 2024-05-15T23:23:15.194 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.023605633000000001, 2024-05-15T23:23:15.194 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024083430999999999, 2024-05-15T23:23:15.194 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:15.195 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:15.195 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:23:15.195 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:15.195 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:23:15.195 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:23:15.195 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 230987 2024-05-15T23:23:15.195 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.195 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:15.195 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.195 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:15.195 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.139681+0000", 2024-05-15T23:23:15.195 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:15.195 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.195 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.139681+0000", 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.139683+0000", 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9889999999999999e-06 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.139688+0000", 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3299999999999998e-06 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:15.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.139700+0000", 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2641e-05 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.139708+0000", 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6669999999999996e-06 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.144521+0000", 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0048127999999999999 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.144571+0000", 2024-05-15T23:23:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9904999999999997e-05 2024-05-15T23:23:15.198 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.198 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.198 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:15.198 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.147562+0000", 2024-05-15T23:23:15.198 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029913940000000001 2024-05-15T23:23:15.198 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.198 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.198 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:15.198 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.147622+0000", 2024-05-15T23:23:15.198 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9543999999999998e-05 2024-05-15T23:23:15.198 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.198 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.198 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:23:15.198 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.147701+0000", 2024-05-15T23:23:15.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9441000000000001e-05 2024-05-15T23:23:15.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:15.199 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:15.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:15.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:15.160197+0000", 2024-05-15T23:23:15.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.012495648 2024-05-15T23:23:15.199 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:15.199 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:15.199 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:15.199 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:15.199 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2024-05-15T23:23:15.199 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 7 2024-05-15T23:23:15.199 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:23:15.250 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:23:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:15.555 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:15 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:23:15.555 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:15 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2327916733' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:23:15.555 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:15 smithi088 ceph-mon[44699]: osdmap e445: 12 total, 12 up, 9 in 2024-05-15T23:23:15.555 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:15 smithi088 ceph-mon[44699]: pgmap v1203: 17 pgs: 3 active+recovering, 14 active+clean; 4.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 787 op/s; 11902/157958 objects misplaced (7.535%); 9.1 MiB/s, 145 objects/s recovering 2024-05-15T23:23:15.818 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:15 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:23:15.818 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:15 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2327916733' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:23:15.818 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:15 smithi084 ceph-mon[39784]: osdmap e445: 12 total, 12 up, 9 in 2024-05-15T23:23:15.818 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:15 smithi084 ceph-mon[39784]: pgmap v1203: 17 pgs: 3 active+recovering, 14 active+clean; 4.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 787 op/s; 11902/157958 objects misplaced (7.535%); 9.1 MiB/s, 145 objects/s recovering 2024-05-15T23:23:16.064 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:15 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:23:16.064 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:15 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2327916733' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:23:16.064 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:15 smithi090 ceph-mon[44434]: osdmap e445: 12 total, 12 up, 9 in 2024-05-15T23:23:16.064 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:15 smithi090 ceph-mon[44434]: pgmap v1203: 17 pgs: 3 active+recovering, 14 active+clean; 4.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 787 op/s; 11902/157958 objects misplaced (7.535%); 9.1 MiB/s, 145 objects/s recovering 2024-05-15T23:23:16.218 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:16.585 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:23:16.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:16.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:23:16.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:23:16.729 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:16.788 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:23:16.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:16.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:23:16.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:23:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:17.051 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:23:17.053 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:23:17.055 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:23:17.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:17.059 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:23:17.160 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:23:17.189 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:17.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:17 smithi090 ceph-mon[44434]: pgmap v1204: 17 pgs: 3 active+recovering, 14 active+clean; 4.7 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 786 op/s; 11398/153454 objects misplaced (7.428%); 9.3 MiB/s, 148 objects/s recovering 2024-05-15T23:23:17.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:17 smithi084 ceph-mon[39784]: pgmap v1204: 17 pgs: 3 active+recovering, 14 active+clean; 4.7 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 786 op/s; 11398/153454 objects misplaced (7.428%); 9.3 MiB/s, 148 objects/s recovering 2024-05-15T23:23:17.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:17 smithi088 ceph-mon[44699]: pgmap v1204: 17 pgs: 3 active+recovering, 14 active+clean; 4.7 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 786 op/s; 11398/153454 objects misplaced (7.428%); 9.3 MiB/s, 148 objects/s recovering 2024-05-15T23:23:17.923 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1204,"stamp":"2024-05-15T23:23:17.003068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5028774456,"num_objects":76727,"num_object_clones":0,"num_object_copies":153454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11398,"num_objects_unfound":0,"num_objects_dirty":76727,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":230514,"num_write_kb":9829799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14497,"num_bytes_recovered":951268960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":12460916,"kb_used_data":9131612,"kb_used_omap":14,"kb_used_meta":3329201,"kb_avail":831245452,"statfs":{"total":863955320832,"available":851195342848,"internally_reserved":0,"allocated":9350770688,"data_stored":9347154580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14358,"internal_metadata":3409102826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,4,1,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-515375104,"num_objects":-7864,"num_object_clones":0,"num_object_copies":-15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1488,"num_objects_unfound":0,"num_objects_dirty":-7864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7864,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1488,"num_bytes_recovered":97517568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002332"},"pg_stats":[{"pgid":"5.f","version":"444'14145","reported_seq":14293,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:12.499338+0000","last_change":"2024-05-15T23:19:47.305945+0000","last_active":"2024-05-15T23:23:12.499338+0000","last_peered":"2024-05-15T23:23:12.499338+0000","last_clean":"2024-05-15T23:23:12.499338+0000","last_became_active":"2024-05-15T23:19:47.305689+0000","last_became_peered":"2024-05-15T23:19:47.305689+0000","last_unstale":"2024-05-15T23:23:12.499338+0000","last_undegraded":"2024-05-15T23:23:12.499338+0000","last_fullsized":"2024-05-15T23:23:12.499338+0000","mapping_epoch":374,"log_start":"386'4100","ondisk_log_start":"386'4100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322109440,"num_objects":4915,"num_object_clones":0,"num_object_copies":9830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14145,"num_write_kb":609920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"445'14253","reported_seq":14580,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:15.525831+0000","last_change":"2024-05-15T23:22:23.637106+0000","last_active":"2024-05-15T23:23:15.525831+0000","last_peered":"2024-05-15T23:23:15.525831+0000","last_clean":"2024-05-15T23:23:15.525831+0000","last_became_active":"2024-05-15T23:19:47.325414+0000","last_became_peered":"2024-05-15T23:19:47.325414+0000","last_unstale":"2024-05-15T23:23:15.525831+0000","last_undegraded":"2024-05-15T23:23:15.525831+0000","last_fullsized":"2024-05-15T23:23:15.525831+0000","mapping_epoch":374,"log_start":"386'4200","ondisk_log_start":"386'4200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'11848","last_scrub_stamp":"2024-05-15T23:22:23.636778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:23.636778+0000","objects_scrubbed":7095,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:23:39.191980+0000","scrub_duration":997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305987584,"num_objects":4669,"num_object_clones":0,"num_object_copies":9338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14253,"num_write_kb":605504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"444'14506","reported_seq":14655,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:12.083817+0000","last_change":"2024-05-15T23:19:47.306160+0000","last_active":"2024-05-15T23:23:12.083817+0000","last_peered":"2024-05-15T23:23:12.083817+0000","last_clean":"2024-05-15T23:23:12.083817+0000","last_became_active":"2024-05-15T23:19:47.305860+0000","last_became_peered":"2024-05-15T23:19:47.305860+0000","last_unstale":"2024-05-15T23:23:12.083817+0000","last_undegraded":"2024-05-15T23:23:12.083817+0000","last_fullsized":"2024-05-15T23:23:12.083817+0000","mapping_epoch":374,"log_start":"387'4500","ondisk_log_start":"387'4500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323682328,"num_objects":4940,"num_object_clones":0,"num_object_copies":9880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14506,"num_write_kb":622209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"445'14577","reported_seq":14892,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:15.824940+0000","last_change":"2024-05-15T23:22:51.995640+0000","last_active":"2024-05-15T23:23:15.824940+0000","last_peered":"2024-05-15T23:23:15.824940+0000","last_clean":"2024-05-15T23:23:15.824940+0000","last_became_active":"2024-05-15T23:19:47.306168+0000","last_became_peered":"2024-05-15T23:19:47.306168+0000","last_unstale":"2024-05-15T23:23:15.824940+0000","last_undegraded":"2024-05-15T23:23:15.824940+0000","last_fullsized":"2024-05-15T23:23:15.824940+0000","mapping_epoch":374,"log_start":"387'4500","ondisk_log_start":"387'4500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'13335","last_scrub_stamp":"2024-05-15T23:22:51.995473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:51.995473+0000","objects_scrubbed":6018,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:13.168527+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310706176,"num_objects":4741,"num_object_clones":0,"num_object_copies":9482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14577,"num_write_kb":618176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"445'14450","reported_seq":14767,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:15.501591+0000","last_change":"2024-05-15T23:22:48.338084+0000","last_active":"2024-05-15T23:23:15.501591+0000","last_peered":"2024-05-15T23:23:15.501591+0000","last_clean":"2024-05-15T23:23:15.501591+0000","last_became_active":"2024-05-15T23:19:47.307061+0000","last_became_peered":"2024-05-15T23:19:47.307061+0000","last_unstale":"2024-05-15T23:23:15.501591+0000","last_undegraded":"2024-05-15T23:23:15.501591+0000","last_fullsized":"2024-05-15T23:23:15.501591+0000","mapping_epoch":374,"log_start":"387'4400","ondisk_log_start":"387'4400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'13172","last_scrub_stamp":"2024-05-15T23:22:48.337866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:48.337866+0000","objects_scrubbed":6060,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:11.906769+0000","scrub_duration":1844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310771712,"num_objects":4742,"num_object_clones":0,"num_object_copies":9484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14450,"num_write_kb":614144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"445'14492","reported_seq":14644,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:15.474371+0000","last_change":"2024-05-15T23:19:47.325657+0000","last_active":"2024-05-15T23:23:15.474371+0000","last_peered":"2024-05-15T23:23:15.474371+0000","last_clean":"2024-05-15T23:23:15.474371+0000","last_became_active":"2024-05-15T23:19:47.325371+0000","last_became_peered":"2024-05-15T23:19:47.325371+0000","last_unstale":"2024-05-15T23:23:15.474371+0000","last_undegraded":"2024-05-15T23:23:15.474371+0000","last_fullsized":"2024-05-15T23:23:15.474371+0000","mapping_epoch":374,"log_start":"387'4400","ondisk_log_start":"387'4400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309329920,"num_objects":4720,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14492,"num_write_kb":614784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"445'14528","reported_seq":24584,"reported_epoch":445,"state":"active+recovering","last_fresh":"2024-05-15T23:23:15.914303+0000","last_change":"2024-05-15T23:21:09.568311+0000","last_active":"2024-05-15T23:23:15.914303+0000","last_peered":"2024-05-15T23:23:15.914303+0000","last_clean":"2024-05-15T23:21:08.023957+0000","last_became_active":"2024-05-15T23:21:09.454639+0000","last_became_peered":"2024-05-15T23:21:09.454639+0000","last_unstale":"2024-05-15T23:23:15.914303+0000","last_undegraded":"2024-05-15T23:23:15.914303+0000","last_fullsized":"2024-05-15T23:23:15.914303+0000","mapping_epoch":407,"log_start":"387'4500","ondisk_log_start":"387'4500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303169536,"num_objects":4626,"num_object_clones":0,"num_object_copies":9252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3619,"num_objects_unfound":0,"num_objects_dirty":4626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14528,"num_write_kb":612928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4951,"num_bytes_recovered":324468736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":3619},{"shards":"7,11","objects":1007}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"444'14326","reported_seq":14476,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:12.105577+0000","last_change":"2024-05-15T23:19:47.304832+0000","last_active":"2024-05-15T23:23:12.105577+0000","last_peered":"2024-05-15T23:23:12.105577+0000","last_clean":"2024-05-15T23:23:12.105577+0000","last_became_active":"2024-05-15T23:19:47.304547+0000","last_became_peered":"2024-05-15T23:19:47.304547+0000","last_unstale":"2024-05-15T23:23:12.105577+0000","last_undegraded":"2024-05-15T23:23:12.105577+0000","last_fullsized":"2024-05-15T23:23:12.105577+0000","mapping_epoch":374,"log_start":"386'4300","ondisk_log_start":"386'4300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319684608,"num_objects":4878,"num_object_clones":0,"num_object_copies":9756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14326,"num_write_kb":614528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"444'14192","reported_seq":14342,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:12.471788+0000","last_change":"2024-05-15T23:19:47.306634+0000","last_active":"2024-05-15T23:23:12.471788+0000","last_peered":"2024-05-15T23:23:12.471788+0000","last_clean":"2024-05-15T23:23:12.471788+0000","last_became_active":"2024-05-15T23:19:47.306331+0000","last_became_peered":"2024-05-15T23:19:47.306331+0000","last_unstale":"2024-05-15T23:23:12.471788+0000","last_undegraded":"2024-05-15T23:23:12.471788+0000","last_fullsized":"2024-05-15T23:23:12.471788+0000","mapping_epoch":374,"log_start":"386'4100","ondisk_log_start":"386'4100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318767104,"num_objects":4864,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14192,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"444'14407","reported_seq":14557,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-05-15T23:23:12.437158+0000","last_change":"2024-05-15T23:19:47.306611+0000","last_active":"2024-05-15T23:23:12.437158+0000","last_peered":"2024-05-15T23:23:12.437158+0000","last_clean":"2024-05-15T23:23:12.437158+0000","last_became_active":"2024-05-15T23:19:47.306349+0000","last_became_peered":"2024-05-15T23:19:47.306349+0000","last_unstale":"2024-05-15T23:23:12.437158+0000","last_undegraded":"2024-05-15T23:23:12.437158+0000","last_fullsized":"2024-05-15T23:23:12.437158+0000","mapping_epoch":374,"log_start":"387'4400","ondisk_log_start":"387'4400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321323008,"num_objects":4903,"num_object_clones":0,"num_object_copies":9806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14407,"num_write_kb":617920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"445'14657","reported_seq":15039,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:15.824901+0000","last_change":"2024-05-15T23:22:48.067681+0000","last_active":"2024-05-15T23:23:15.824901+0000","last_peered":"2024-05-15T23:23:15.824901+0000","last_clean":"2024-05-15T23:23:15.824901+0000","last_became_active":"2024-05-15T23:19:47.305685+0000","last_became_peered":"2024-05-15T23:19:47.305685+0000","last_unstale":"2024-05-15T23:23:15.824901+0000","last_undegraded":"2024-05-15T23:23:15.824901+0000","last_fullsized":"2024-05-15T23:23:15.824901+0000","mapping_epoch":374,"log_start":"387'4600","ondisk_log_start":"387'4600","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'13277","last_scrub_stamp":"2024-05-15T23:22:48.067507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:48.067507+0000","objects_scrubbed":6142,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:23:49.704509+0000","scrub_duration":929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310575104,"num_objects":4739,"num_object_clones":0,"num_object_copies":9478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14657,"num_write_kb":620672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"444'14307","reported_seq":23784,"reported_epoch":444,"state":"active+recovering","last_fresh":"2024-05-15T23:23:12.498831+0000","last_change":"2024-05-15T23:21:09.470461+0000","last_active":"2024-05-15T23:23:12.498831+0000","last_peered":"2024-05-15T23:23:12.498831+0000","last_clean":"2024-05-15T23:21:07.989871+0000","last_became_active":"2024-05-15T23:21:09.400731+0000","last_became_peered":"2024-05-15T23:21:09.400731+0000","last_unstale":"2024-05-15T23:23:12.498831+0000","last_undegraded":"2024-05-15T23:23:12.498831+0000","last_fullsized":"2024-05-15T23:23:12.498831+0000","mapping_epoch":407,"log_start":"387'4300","ondisk_log_start":"387'4300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326959104,"num_objects":4989,"num_object_clones":0,"num_object_copies":9978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4027,"num_objects_unfound":0,"num_objects_dirty":4989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14307,"num_write_kb":617472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4663,"num_bytes_recovered":305594368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"3,5","objects":962},{"shards":"5,6","objects":4027}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"445'14462","reported_seq":24369,"reported_epoch":445,"state":"active+recovering","last_fresh":"2024-05-15T23:23:15.525414+0000","last_change":"2024-05-15T23:21:09.275620+0000","last_active":"2024-05-15T23:23:15.525414+0000","last_peered":"2024-05-15T23:23:15.525414+0000","last_clean":"2024-05-15T23:21:07.993692+0000","last_became_active":"2024-05-15T23:21:09.221797+0000","last_became_peered":"2024-05-15T23:21:09.221797+0000","last_unstale":"2024-05-15T23:23:15.525414+0000","last_undegraded":"2024-05-15T23:23:15.525414+0000","last_fullsized":"2024-05-15T23:23:15.525414+0000","mapping_epoch":407,"log_start":"387'4400","ondisk_log_start":"387'4400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308543488,"num_objects":4708,"num_object_clones":0,"num_object_copies":9416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3752,"num_objects_unfound":0,"num_objects_dirty":4708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14462,"num_write_kb":613440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4877,"num_bytes_recovered":319619072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":3752},{"shards":"7,10","objects":956}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.1","version":"445'14045","reported_seq":14370,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:14.812550+0000","last_change":"2024-05-15T23:22:19.625857+0000","last_active":"2024-05-15T23:23:14.812550+0000","last_peered":"2024-05-15T23:23:14.812550+0000","last_clean":"2024-05-15T23:23:14.812550+0000","last_became_active":"2024-05-15T23:19:47.516865+0000","last_became_peered":"2024-05-15T23:19:47.516865+0000","last_unstale":"2024-05-15T23:23:14.812550+0000","last_undegraded":"2024-05-15T23:23:14.812550+0000","last_fullsized":"2024-05-15T23:23:14.812550+0000","mapping_epoch":374,"log_start":"386'4000","ondisk_log_start":"386'4000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"426'11474","last_scrub_stamp":"2024-05-15T23:22:19.625719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:19.625719+0000","objects_scrubbed":7211,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:23:49.523745+0000","scrub_duration":1229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302317568,"num_objects":4613,"num_object_clones":0,"num_object_copies":9226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14045,"num_write_kb":597056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"445'14523","reported_seq":14675,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:15.455673+0000","last_change":"2024-05-15T23:19:47.305723+0000","last_active":"2024-05-15T23:23:15.455673+0000","last_peered":"2024-05-15T23:23:15.455673+0000","last_clean":"2024-05-15T23:23:15.455673+0000","last_became_active":"2024-05-15T23:19:47.305459+0000","last_became_peered":"2024-05-15T23:19:47.305459+0000","last_unstale":"2024-05-15T23:23:15.455673+0000","last_undegraded":"2024-05-15T23:23:15.455673+0000","last_fullsized":"2024-05-15T23:23:15.455673+0000","mapping_epoch":374,"log_start":"387'4500","ondisk_log_start":"387'4500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311492608,"num_objects":4753,"num_object_clones":0,"num_object_copies":9506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14523,"num_write_kb":616832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"445'14521","reported_seq":14672,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:15.521114+0000","last_change":"2024-05-15T23:19:47.307499+0000","last_active":"2024-05-15T23:23:15.521114+0000","last_peered":"2024-05-15T23:23:15.521114+0000","last_clean":"2024-05-15T23:23:15.521114+0000","last_became_active":"2024-05-15T23:19:47.307149+0000","last_became_peered":"2024-05-15T23:19:47.307149+0000","last_unstale":"2024-05-15T23:23:15.521114+0000","last_undegraded":"2024-05-15T23:23:15.521114+0000","last_fullsized":"2024-05-15T23:23:15.521114+0000","mapping_epoch":374,"log_start":"387'4500","ondisk_log_start":"387'4500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322764800,"num_objects":4925,"num_object_clones":0,"num_object_copies":9850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14521,"num_write_kb":622272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":929,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:14.568076+0000","last_change":"2024-05-15T23:20:05.905241+0000","last_active":"2024-05-15T23:23:14.568076+0000","last_peered":"2024-05-15T23:23:14.568076+0000","last_clean":"2024-05-15T23:23:14.568076+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:23:14.568076+0000","last_undegraded":"2024-05-15T23:23:14.568076+0000","last_fullsized":"2024-05-15T23:23:14.568076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:20:05.905105+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:20:05.905105+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":5028184088,"num_objects":76725,"num_object_clones":0,"num_object_copies":153450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11398,"num_objects_unfound":0,"num_objects_dirty":76725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":230391,"num_write_kb":9827649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14491,"num_bytes_recovered":949682176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10993803264,"data_stored":10993795120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160691,"ondisk_log_size":160691,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351209,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1232924,"kb_used_data":696404,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92512228,"statfs":{"total":95995035648,"available":94732521472,"internally_reserved":0,"allocated":713117696,"data_stored":712713322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":549386692},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580143,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1803808,"kb_used_data":1529684,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":91941344,"statfs":{"total":95995035648,"available":94147936256,"internally_reserved":0,"allocated":1566396416,"data_stored":1565993632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":280689088},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":982300,"kb_used_data":604052,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92762852,"statfs":{"total":95995035648,"available":94989160448,"internally_reserved":0,"allocated":618549248,"data_stored":618146464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":387316160},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005307,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":788944,"kb_used_data":613384,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92956208,"statfs":{"total":95995035648,"available":95187156992,"internally_reserved":0,"allocated":628105216,"data_stored":627708486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266943,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1005672,"kb_used_data":667676,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92739480,"statfs":{"total":95995035648,"available":94965227520,"internally_reserved":0,"allocated":683700224,"data_stored":683293888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":346094026},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495877,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575135,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1688736,"kb_used_data":1220244,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":92056416,"statfs":{"total":95995035648,"available":94265769984,"internally_reserved":0,"allocated":1249529856,"data_stored":1249127072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":479721926},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016543,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483233,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149973,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":766304,"kb_used_data":356116,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92978848,"statfs":{"total":95995035648,"available":95210340352,"internally_reserved":0,"allocated":364662784,"data_stored":364260000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":420018631},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986446,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1614352,"kb_used_data":1222216,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92130800,"statfs":{"total":95995035648,"available":94341939200,"internally_reserved":0,"allocated":1251549184,"data_stored":1251152454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790218,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2577876,"kb_used_data":2221836,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91167276,"statfs":{"total":95995035648,"available":93355290624,"internally_reserved":0,"allocated":2275160064,"data_stored":2274759262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1240399872,"data_stored":1240399872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":362414080,"data_stored":362414080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1245708288,"data_stored":1245708288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2262831104,"data_stored":2262827032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1696661504,"data_stored":1696661504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":675872768,"data_stored":675872768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":622526464,"data_stored":622526464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":616693760,"data_stored":616693760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1561198592,"data_stored":1561198592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":709496832,"data_stored":709492760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:23:17.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:23:17.945 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:23:17.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:17.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:23:18.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:18.085 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T23:23:18.098 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:18.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:18.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:23:18.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:23:18.208 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:18 smithi084 ceph-mon[39784]: from='client.16197 v1:172.21.15.84:0/743812643' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:18.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:23:18.284 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:18.315 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:23:18.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:18.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:23:18.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:18 smithi088 ceph-mon[44699]: from='client.16197 v1:172.21.15.84:0/743812643' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:18.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:23:18.456 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:18 smithi090 ceph-mon[44434]: from='client.16197 v1:172.21.15.84:0/743812643' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:18.461 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:18.500 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:23:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:18.508 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:23:18.529 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:23:18.609 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:23:18.658 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:18.895 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:19.026 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:23:19.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:19.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:23:19.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:23:19.150 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:19.168 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:19.272 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:23:19.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:19.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:23:19.280 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:19 smithi088 ceph-mon[44699]: pgmap v1205: 17 pgs: 3 active+recovering, 14 active+clean; 4.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 785 op/s; 11118/150478 objects misplaced (7.388%); 9.6 MiB/s, 152 objects/s recovering 2024-05-15T23:23:19.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:19 smithi084 ceph-mon[39784]: pgmap v1205: 17 pgs: 3 active+recovering, 14 active+clean; 4.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 785 op/s; 11118/150478 objects misplaced (7.388%); 9.6 MiB/s, 152 objects/s recovering 2024-05-15T23:23:19.377 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:23:19.460 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:19 smithi090 ceph-mon[44434]: pgmap v1205: 17 pgs: 3 active+recovering, 14 active+clean; 4.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 785 op/s; 11118/150478 objects misplaced (7.388%); 9.6 MiB/s, 152 objects/s recovering 2024-05-15T23:23:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:20 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3346771709' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:20.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:20.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:20 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3346771709' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:20.575 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:20 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3346771709' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:20.575 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:20.575 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:20.680 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:23:20.680 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:23:20.680 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:23:20.680 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:23:20.680 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:23:20.680 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:23:20.680 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:23:20.680 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:23:20.680 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:23:20.680 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:23:20.680 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:23:20.681 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:23:20.681 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:23:20.681 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:23:20.681 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:23:20.681 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:23:20.681 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:23:20.681 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:23:20.681 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:23:20.681 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:23:20.681 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:23:20.681 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:23:20.681 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:23:20.682 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:23:21.059 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:23:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:21.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:23:21.164 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:23:21.216 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:21.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:21 smithi088 ceph-mon[44699]: pgmap v1206: 17 pgs: 3 active+recovering, 14 active+clean; 4.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 790 op/s; 10630/145628 objects misplaced (7.299%); 9.5 MiB/s, 151 objects/s recovering 2024-05-15T23:23:21.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:21 smithi084 ceph-mon[39784]: pgmap v1206: 17 pgs: 3 active+recovering, 14 active+clean; 4.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 790 op/s; 10630/145628 objects misplaced (7.299%); 9.5 MiB/s, 151 objects/s recovering 2024-05-15T23:23:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:21.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:21 smithi090 ceph-mon[44434]: pgmap v1206: 17 pgs: 3 active+recovering, 14 active+clean; 4.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 790 op/s; 10630/145628 objects misplaced (7.299%); 9.5 MiB/s, 151 objects/s recovering 2024-05-15T23:23:22.129 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:23:22.303 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:23:22.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:22.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:23:22.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2990713698' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:23:22.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:22 smithi084 ceph-mon[39784]: from='client.26131 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:23:22.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:23:22.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2990713698' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:23:22.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:22 smithi090 ceph-mon[44434]: from='client.26131 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:23:22.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2990713698' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:23:22.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:22 smithi088 ceph-mon[44699]: from='client.26131 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:23:23.113 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:23.156 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:23:23.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:23.161 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:23:23.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:23:23.295 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:23.371 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:23:23.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:23.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:23:23.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:23:23.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:23 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:23:23.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:23 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:23:23.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:23 smithi090 ceph-mon[44434]: from='client.26131 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:23:23.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:23 smithi090 ceph-mon[44434]: osdmap e446: 12 total, 12 up, 9 in 2024-05-15T23:23:23.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:23:23.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:23:23.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:23 smithi090 ceph-mon[44434]: pgmap v1208: 17 pgs: 3 active+recovering, 14 active+clean; 4.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 939 op/s; 10364/142558 objects misplaced (7.270%); 11 MiB/s, 182 objects/s recovering 2024-05-15T23:23:23.507 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:23.541 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:23:23.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:23.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:23:23.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:23 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:23:23.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:23 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:23:23.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:23 smithi088 ceph-mon[44699]: from='client.26131 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:23:23.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:23 smithi088 ceph-mon[44699]: osdmap e446: 12 total, 12 up, 9 in 2024-05-15T23:23:23.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:23:23.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:23:23.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:23 smithi088 ceph-mon[44699]: pgmap v1208: 17 pgs: 3 active+recovering, 14 active+clean; 4.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 939 op/s; 10364/142558 objects misplaced (7.270%); 11 MiB/s, 182 objects/s recovering 2024-05-15T23:23:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:23 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:23:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:23 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:23:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:23 smithi084 ceph-mon[39784]: from='client.26131 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:23:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:23 smithi084 ceph-mon[39784]: osdmap e446: 12 total, 12 up, 9 in 2024-05-15T23:23:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:23:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:23:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:23 smithi084 ceph-mon[39784]: pgmap v1208: 17 pgs: 3 active+recovering, 14 active+clean; 4.4 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 939 op/s; 10364/142558 objects misplaced (7.270%); 11 MiB/s, 182 objects/s recovering 2024-05-15T23:23:23.649 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:23:23.676 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:23.749 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:23:23.749 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:23:23.825 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:23:23.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:23.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:23:23.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:23:23.962 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:24.161 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:23:24.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:24.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:23:24.265 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:23:24.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:24 smithi084 ceph-mon[39784]: osdmap e447: 12 total, 12 up, 9 in 2024-05-15T23:23:24.392 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:24 smithi090 ceph-mon[44434]: osdmap e447: 12 total, 12 up, 9 in 2024-05-15T23:23:24.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:24 smithi088 ceph-mon[44699]: osdmap e447: 12 total, 12 up, 9 in 2024-05-15T23:23:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:23:24.685 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:23:25.246 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:23:25.268 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:25 smithi088 ceph-mon[44699]: from='client.16212 v1:172.21.15.84:0/630611805' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:25.269 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:25 smithi088 ceph-mon[44699]: pgmap v1210: 17 pgs: 3 active+recovering, 14 active+clean; 4.3 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 712 op/s; 10364/142050 objects misplaced (7.296%); 8.1 MiB/s, 129 objects/s recovering 2024-05-15T23:23:25.269 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:23:25.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:25.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:23:25.358 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T23:23:25.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:25 smithi084 ceph-mon[39784]: from='client.16212 v1:172.21.15.84:0/630611805' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:25.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:25 smithi084 ceph-mon[39784]: pgmap v1210: 17 pgs: 3 active+recovering, 14 active+clean; 4.3 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 712 op/s; 10364/142050 objects misplaced (7.296%); 8.1 MiB/s, 129 objects/s recovering 2024-05-15T23:23:25.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:23:25.412 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:25 smithi090 ceph-mon[44434]: from='client.16212 v1:172.21.15.84:0/630611805' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:25.412 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:25 smithi090 ceph-mon[44434]: pgmap v1210: 17 pgs: 3 active+recovering, 14 active+clean; 4.3 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 712 op/s; 10364/142050 objects misplaced (7.296%); 8.1 MiB/s, 129 objects/s recovering 2024-05-15T23:23:25.414 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:25.454 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:23:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:25.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:23:25.502 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1209,"stamp":"2024-05-15T23:23:23.144367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4671734328,"num_objects":71279,"num_object_clones":0,"num_object_copies":142558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10364,"num_objects_unfound":0,"num_objects_dirty":71279,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":235962,"num_write_kb":9829799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15531,"num_bytes_recovered":1019033184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160633,"ondisk_log_size":160633,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":11882660,"kb_used_data":8527000,"kb_used_omap":14,"kb_used_meta":3355569,"kb_avail":831823708,"statfs":{"total":863955320832,"available":851787476992,"internally_reserved":0,"allocated":8731648000,"data_stored":8728001622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14358,"internal_metadata":3436103658},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-504627200,"num_objects":-7700,"num_object_clones":0,"num_object_copies":-15400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1538,"num_objects_unfound":0,"num_objects_dirty":-7700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7700,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1538,"num_bytes_recovered":100794368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.141691"},"pg_stats":[{"pgid":"5.f","version":"446'14609","reported_seq":14763,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-05-15T23:23:22.460928+0000","last_change":"2024-05-15T23:19:47.305945+0000","last_active":"2024-05-15T23:23:22.460928+0000","last_peered":"2024-05-15T23:23:22.460928+0000","last_clean":"2024-05-15T23:23:22.460928+0000","last_became_active":"2024-05-15T23:19:47.305689+0000","last_became_peered":"2024-05-15T23:19:47.305689+0000","last_unstale":"2024-05-15T23:23:22.460928+0000","last_undegraded":"2024-05-15T23:23:22.460928+0000","last_fullsized":"2024-05-15T23:23:22.460928+0000","mapping_epoch":374,"log_start":"387'4600","ondisk_log_start":"387'4600","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291700736,"num_objects":4451,"num_object_clones":0,"num_object_copies":8902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14609,"num_write_kb":609920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"445'14507","reported_seq":14833,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:20.555013+0000","last_change":"2024-05-15T23:22:23.637106+0000","last_active":"2024-05-15T23:23:20.555013+0000","last_peered":"2024-05-15T23:23:20.555013+0000","last_clean":"2024-05-15T23:23:20.555013+0000","last_became_active":"2024-05-15T23:19:47.325414+0000","last_became_peered":"2024-05-15T23:19:47.325414+0000","last_unstale":"2024-05-15T23:23:20.555013+0000","last_undegraded":"2024-05-15T23:23:20.555013+0000","last_fullsized":"2024-05-15T23:23:20.555013+0000","mapping_epoch":374,"log_start":"387'4500","ondisk_log_start":"387'4500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'11848","last_scrub_stamp":"2024-05-15T23:22:23.636778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:23.636778+0000","objects_scrubbed":7095,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:23:39.191980+0000","scrub_duration":997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289341440,"num_objects":4415,"num_object_clones":0,"num_object_copies":8830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14507,"num_write_kb":605504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"445'15040","reported_seq":15195,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-05-15T23:23:22.123710+0000","last_change":"2024-05-15T23:19:47.306160+0000","last_active":"2024-05-15T23:23:22.123710+0000","last_peered":"2024-05-15T23:23:22.123710+0000","last_clean":"2024-05-15T23:23:22.123710+0000","last_became_active":"2024-05-15T23:19:47.305860+0000","last_became_peered":"2024-05-15T23:19:47.305860+0000","last_unstale":"2024-05-15T23:23:22.123710+0000","last_undegraded":"2024-05-15T23:23:22.123710+0000","last_fullsized":"2024-05-15T23:23:22.123710+0000","mapping_epoch":374,"log_start":"389'5000","ondisk_log_start":"389'5000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288686104,"num_objects":4406,"num_object_clones":0,"num_object_copies":8812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4406,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15040,"num_write_kb":622209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"445'14811","reported_seq":15126,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:20.804998+0000","last_change":"2024-05-15T23:22:51.995640+0000","last_active":"2024-05-15T23:23:20.804998+0000","last_peered":"2024-05-15T23:23:20.804998+0000","last_clean":"2024-05-15T23:23:20.804998+0000","last_became_active":"2024-05-15T23:19:47.306168+0000","last_became_peered":"2024-05-15T23:19:47.306168+0000","last_unstale":"2024-05-15T23:23:20.804998+0000","last_undegraded":"2024-05-15T23:23:20.804998+0000","last_fullsized":"2024-05-15T23:23:20.804998+0000","mapping_epoch":374,"log_start":"387'4800","ondisk_log_start":"387'4800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'13335","last_scrub_stamp":"2024-05-15T23:22:51.995473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:51.995473+0000","objects_scrubbed":6018,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:13.168527+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295370752,"num_objects":4507,"num_object_clones":0,"num_object_copies":9014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14811,"num_write_kb":618176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"445'14703","reported_seq":15020,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:20.567400+0000","last_change":"2024-05-15T23:22:48.338084+0000","last_active":"2024-05-15T23:23:20.567400+0000","last_peered":"2024-05-15T23:23:20.567400+0000","last_clean":"2024-05-15T23:23:20.567400+0000","last_became_active":"2024-05-15T23:19:47.307061+0000","last_became_peered":"2024-05-15T23:19:47.307061+0000","last_unstale":"2024-05-15T23:23:20.567400+0000","last_undegraded":"2024-05-15T23:23:20.567400+0000","last_fullsized":"2024-05-15T23:23:20.567400+0000","mapping_epoch":374,"log_start":"387'4700","ondisk_log_start":"387'4700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'13172","last_scrub_stamp":"2024-05-15T23:22:48.337866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:48.337866+0000","objects_scrubbed":6060,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:11.906769+0000","scrub_duration":1844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294191104,"num_objects":4489,"num_object_clones":0,"num_object_copies":8978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14703,"num_write_kb":614144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"445'14744","reported_seq":14896,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:20.474422+0000","last_change":"2024-05-15T23:19:47.325657+0000","last_active":"2024-05-15T23:23:20.474422+0000","last_peered":"2024-05-15T23:23:20.474422+0000","last_clean":"2024-05-15T23:23:20.474422+0000","last_became_active":"2024-05-15T23:19:47.325371+0000","last_became_peered":"2024-05-15T23:19:47.325371+0000","last_unstale":"2024-05-15T23:23:20.474422+0000","last_undegraded":"2024-05-15T23:23:20.474422+0000","last_fullsized":"2024-05-15T23:23:20.474422+0000","mapping_epoch":374,"log_start":"387'4700","ondisk_log_start":"387'4700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292814848,"num_objects":4468,"num_object_clones":0,"num_object_copies":8936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14744,"num_write_kb":614784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"445'14767","reported_seq":25302,"reported_epoch":445,"state":"active+recovering","last_fresh":"2024-05-15T23:23:20.924961+0000","last_change":"2024-05-15T23:21:09.568311+0000","last_active":"2024-05-15T23:23:20.924961+0000","last_peered":"2024-05-15T23:23:20.924961+0000","last_clean":"2024-05-15T23:21:08.023957+0000","last_became_active":"2024-05-15T23:21:09.454639+0000","last_became_peered":"2024-05-15T23:21:09.454639+0000","last_unstale":"2024-05-15T23:23:20.924961+0000","last_undegraded":"2024-05-15T23:23:20.924961+0000","last_fullsized":"2024-05-15T23:23:20.924961+0000","mapping_epoch":407,"log_start":"387'4700","ondisk_log_start":"387'4700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287506432,"num_objects":4387,"num_object_clones":0,"num_object_copies":8774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3380,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14767,"num_write_kb":612928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5190,"num_bytes_recovered":340131840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":3380},{"shards":"7,11","objects":1007}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"446'14813","reported_seq":14967,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-05-15T23:23:22.123190+0000","last_change":"2024-05-15T23:19:47.304832+0000","last_active":"2024-05-15T23:23:22.123190+0000","last_peered":"2024-05-15T23:23:22.123190+0000","last_clean":"2024-05-15T23:23:22.123190+0000","last_became_active":"2024-05-15T23:19:47.304547+0000","last_became_peered":"2024-05-15T23:19:47.304547+0000","last_unstale":"2024-05-15T23:23:22.123190+0000","last_undegraded":"2024-05-15T23:23:22.123190+0000","last_fullsized":"2024-05-15T23:23:22.123190+0000","mapping_epoch":374,"log_start":"387'4800","ondisk_log_start":"387'4800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287768576,"num_objects":4391,"num_object_clones":0,"num_object_copies":8782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14813,"num_write_kb":614528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"446'14680","reported_seq":14834,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-05-15T23:23:22.509078+0000","last_change":"2024-05-15T23:19:47.306634+0000","last_active":"2024-05-15T23:23:22.509078+0000","last_peered":"2024-05-15T23:23:22.509078+0000","last_clean":"2024-05-15T23:23:22.509078+0000","last_became_active":"2024-05-15T23:19:47.306331+0000","last_became_peered":"2024-05-15T23:19:47.306331+0000","last_unstale":"2024-05-15T23:23:22.509078+0000","last_undegraded":"2024-05-15T23:23:22.509078+0000","last_fullsized":"2024-05-15T23:23:22.509078+0000","mapping_epoch":374,"log_start":"387'4600","ondisk_log_start":"387'4600","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286785536,"num_objects":4376,"num_object_clones":0,"num_object_copies":8752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14680,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"446'14907","reported_seq":15061,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-05-15T23:23:22.502690+0000","last_change":"2024-05-15T23:19:47.306611+0000","last_active":"2024-05-15T23:23:22.502690+0000","last_peered":"2024-05-15T23:23:22.502690+0000","last_clean":"2024-05-15T23:23:22.502690+0000","last_became_active":"2024-05-15T23:19:47.306349+0000","last_became_peered":"2024-05-15T23:19:47.306349+0000","last_unstale":"2024-05-15T23:23:22.502690+0000","last_undegraded":"2024-05-15T23:23:22.502690+0000","last_fullsized":"2024-05-15T23:23:22.502690+0000","mapping_epoch":374,"log_start":"388'4900","ondisk_log_start":"388'4900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288555008,"num_objects":4403,"num_object_clones":0,"num_object_copies":8806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14907,"num_write_kb":617920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"445'14907","reported_seq":15289,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:20.788727+0000","last_change":"2024-05-15T23:22:48.067681+0000","last_active":"2024-05-15T23:23:20.788727+0000","last_peered":"2024-05-15T23:23:20.788727+0000","last_clean":"2024-05-15T23:23:20.788727+0000","last_became_active":"2024-05-15T23:19:47.305685+0000","last_became_peered":"2024-05-15T23:19:47.305685+0000","last_unstale":"2024-05-15T23:23:20.788727+0000","last_undegraded":"2024-05-15T23:23:20.788727+0000","last_fullsized":"2024-05-15T23:23:20.788727+0000","mapping_epoch":374,"log_start":"388'4900","ondisk_log_start":"388'4900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'13277","last_scrub_stamp":"2024-05-15T23:22:48.067507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:48.067507+0000","objects_scrubbed":6142,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:23:49.704509+0000","scrub_duration":929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294191104,"num_objects":4489,"num_object_clones":0,"num_object_copies":8978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14907,"num_write_kb":620672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"446'14857","reported_seq":25432,"reported_epoch":446,"state":"active+recovering","last_fresh":"2024-05-15T23:23:22.518969+0000","last_change":"2024-05-15T23:21:09.470461+0000","last_active":"2024-05-15T23:23:22.518969+0000","last_peered":"2024-05-15T23:23:22.518969+0000","last_clean":"2024-05-15T23:21:07.989871+0000","last_became_active":"2024-05-15T23:21:09.400731+0000","last_became_peered":"2024-05-15T23:21:09.400731+0000","last_unstale":"2024-05-15T23:23:22.518969+0000","last_undegraded":"2024-05-15T23:23:22.518969+0000","last_fullsized":"2024-05-15T23:23:22.518969+0000","mapping_epoch":407,"log_start":"388'4800","ondisk_log_start":"388'4800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290914304,"num_objects":4439,"num_object_clones":0,"num_object_copies":8878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3481,"num_objects_unfound":0,"num_objects_dirty":4439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14857,"num_write_kb":617472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5209,"num_bytes_recovered":341377024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"3,5","objects":958},{"shards":"5,6","objects":3481}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"445'14706","reported_seq":25114,"reported_epoch":445,"state":"active+recovering","last_fresh":"2024-05-15T23:23:20.512075+0000","last_change":"2024-05-15T23:21:09.275620+0000","last_active":"2024-05-15T23:23:20.512075+0000","last_peered":"2024-05-15T23:23:20.512075+0000","last_clean":"2024-05-15T23:21:07.993692+0000","last_became_active":"2024-05-15T23:21:09.221797+0000","last_became_peered":"2024-05-15T23:21:09.221797+0000","last_unstale":"2024-05-15T23:23:20.512075+0000","last_undegraded":"2024-05-15T23:23:20.512075+0000","last_fullsized":"2024-05-15T23:23:20.512075+0000","mapping_epoch":407,"log_start":"387'4700","ondisk_log_start":"387'4700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292552704,"num_objects":4464,"num_object_clones":0,"num_object_copies":8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3503,"num_objects_unfound":0,"num_objects_dirty":4464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14706,"num_write_kb":613440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5126,"num_bytes_recovered":335937536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":3503},{"shards":"7,10","objects":961}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.1","version":"445'14257","reported_seq":14582,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:19.849307+0000","last_change":"2024-05-15T23:22:19.625857+0000","last_active":"2024-05-15T23:23:19.849307+0000","last_peered":"2024-05-15T23:23:19.849307+0000","last_clean":"2024-05-15T23:23:19.849307+0000","last_became_active":"2024-05-15T23:19:47.516865+0000","last_became_peered":"2024-05-15T23:19:47.516865+0000","last_unstale":"2024-05-15T23:23:19.849307+0000","last_undegraded":"2024-05-15T23:23:19.849307+0000","last_fullsized":"2024-05-15T23:23:19.849307+0000","mapping_epoch":374,"log_start":"386'4200","ondisk_log_start":"386'4200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"426'11474","last_scrub_stamp":"2024-05-15T23:22:19.625719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:19.625719+0000","objects_scrubbed":7211,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:23:49.523745+0000","scrub_duration":1229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288423936,"num_objects":4401,"num_object_clones":0,"num_object_copies":8802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14257,"num_write_kb":597056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"445'14759","reported_seq":14911,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:20.527941+0000","last_change":"2024-05-15T23:19:47.305723+0000","last_active":"2024-05-15T23:23:20.527941+0000","last_peered":"2024-05-15T23:23:20.527941+0000","last_clean":"2024-05-15T23:23:20.527941+0000","last_became_active":"2024-05-15T23:19:47.305459+0000","last_became_peered":"2024-05-15T23:19:47.305459+0000","last_unstale":"2024-05-15T23:23:20.527941+0000","last_undegraded":"2024-05-15T23:23:20.527941+0000","last_fullsized":"2024-05-15T23:23:20.527941+0000","mapping_epoch":374,"log_start":"387'4700","ondisk_log_start":"387'4700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296026112,"num_objects":4517,"num_object_clones":0,"num_object_copies":9034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14759,"num_write_kb":616832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"445'14772","reported_seq":14924,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:20.512228+0000","last_change":"2024-05-15T23:19:47.307499+0000","last_active":"2024-05-15T23:23:20.512228+0000","last_peered":"2024-05-15T23:23:20.512228+0000","last_clean":"2024-05-15T23:23:20.512228+0000","last_became_active":"2024-05-15T23:19:47.307149+0000","last_became_peered":"2024-05-15T23:19:47.307149+0000","last_unstale":"2024-05-15T23:23:20.512228+0000","last_undegraded":"2024-05-15T23:23:20.512228+0000","last_fullsized":"2024-05-15T23:23:20.512228+0000","mapping_epoch":374,"log_start":"387'4700","ondisk_log_start":"387'4700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306315264,"num_objects":4674,"num_object_clones":0,"num_object_copies":9348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14772,"num_write_kb":622272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":929,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-05-15T23:23:14.568076+0000","last_change":"2024-05-15T23:20:05.905241+0000","last_active":"2024-05-15T23:23:14.568076+0000","last_peered":"2024-05-15T23:23:14.568076+0000","last_clean":"2024-05-15T23:23:14.568076+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:23:14.568076+0000","last_undegraded":"2024-05-15T23:23:14.568076+0000","last_fullsized":"2024-05-15T23:23:14.568076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:20:05.905105+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:20:05.905105+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4671143960,"num_objects":71277,"num_object_clones":0,"num_object_copies":142554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10364,"num_objects_unfound":0,"num_objects_dirty":71277,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":235839,"num_write_kb":9827649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15525,"num_bytes_recovered":1017446400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10375929856,"data_stored":10375921712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160539,"ondisk_log_size":160539,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351211,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1175528,"kb_used_data":630688,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92569624,"statfs":{"total":95995035648,"available":94791294976,"internally_reserved":0,"allocated":645824512,"data_stored":645414084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":557906372},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580144,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1718364,"kb_used_data":1452564,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92026788,"statfs":{"total":95995035648,"available":94235430912,"internally_reserved":0,"allocated":1487425536,"data_stored":1487022752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":272169408},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776376,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":951580,"kb_used_data":573332,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92793572,"statfs":{"total":95995035648,"available":95020617728,"internally_reserved":0,"allocated":587091968,"data_stored":586689184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":387316160},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005308,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":760284,"kb_used_data":580564,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92984868,"statfs":{"total":95995035648,"available":95216504832,"internally_reserved":0,"allocated":594497536,"data_stored":594094752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266944,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":974116,"kb_used_data":636124,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92771036,"statfs":{"total":95995035648,"available":94997540864,"internally_reserved":0,"allocated":651390976,"data_stored":650984640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":346094026},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495879,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575136,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1630044,"kb_used_data":1157396,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92115108,"statfs":{"total":95995035648,"available":94325870592,"internally_reserved":0,"allocated":1185173504,"data_stored":1184770720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":483981766},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016545,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483234,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149975,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":752540,"kb_used_data":342356,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92992612,"statfs":{"total":95995035648,"available":95224434688,"internally_reserved":0,"allocated":350572544,"data_stored":350169760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":420018631},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1558300,"kb_used_data":1162004,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92186852,"statfs":{"total":95995035648,"available":94399336448,"internally_reserved":0,"allocated":1189892096,"data_stored":1189489312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790220,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2361904,"kb_used_data":1991972,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":91383248,"statfs":{"total":95995035648,"available":93576445952,"internally_reserved":0,"allocated":2039779328,"data_stored":2039366418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176502272,"data_stored":1176502272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":348258304,"data_stored":348258304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1183449088,"data_stored":1183449088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2030768128,"data_stored":2030764056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1696661504,"data_stored":1696661504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":644153344,"data_stored":644153344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":589824000,"data_stored":589824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":583532544,"data_stored":583532544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1480392704,"data_stored":1480392704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":642387968,"data_stored":642383896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:23:25.503 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.9 is not active+clean 2024-05-15T23:23:25.504 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.9', 'version': "445'14767", 'reported_seq': 25302, 'reported_epoch': 445, 'state': 'active+recovering', 'last_fresh': '2024-05-15T23:23:20.924961+0000', 'last_change': '2024-05-15T23:21:09.568311+0000', 'last_active': '2024-05-15T23:23:20.924961+0000', 'last_peered': '2024-05-15T23:23:20.924961+0000', 'last_clean': '2024-05-15T23:21:08.023957+0000', 'last_became_active': '2024-05-15T23:21:09.454639+0000', 'last_became_peered': '2024-05-15T23:21:09.454639+0000', 'last_unstale': '2024-05-15T23:23:20.924961+0000', 'last_undegraded': '2024-05-15T23:23:20.924961+0000', 'last_fullsized': '2024-05-15T23:23:20.924961+0000', 'mapping_epoch': 407, 'log_start': "387'4700", 'ondisk_log_start': "387'4700", 'created': 374, 'last_epoch_clean': 375, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'last_clean_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 287506432, 'num_objects': 4387, 'num_object_clones': 0, 'num_object_copies': 8774, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3380, 'num_objects_unfound': 0, 'num_objects_dirty': 4387, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14767, 'num_write_kb': 612928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5190, 'num_bytes_recovered': 340131840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 11], 'acting': [7, 11], 'avail_no_missing': ['7', '6'], 'object_location_counts': [{'shards': '6,7', 'objects': 3380}, {'shards': '7,11', 'objects': 1007}], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-15T23:23:25.504 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.3 is not active+clean 2024-05-15T23:23:25.504 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.3', 'version': "446'14857", 'reported_seq': 25432, 'reported_epoch': 446, 'state': 'active+recovering', 'last_fresh': '2024-05-15T23:23:22.518969+0000', 'last_change': '2024-05-15T23:21:09.470461+0000', 'last_active': '2024-05-15T23:23:22.518969+0000', 'last_peered': '2024-05-15T23:23:22.518969+0000', 'last_clean': '2024-05-15T23:21:07.989871+0000', 'last_became_active': '2024-05-15T23:21:09.400731+0000', 'last_became_peered': '2024-05-15T23:21:09.400731+0000', 'last_unstale': '2024-05-15T23:23:22.518969+0000', 'last_undegraded': '2024-05-15T23:23:22.518969+0000', 'last_fullsized': '2024-05-15T23:23:22.518969+0000', 'mapping_epoch': 407, 'log_start': "388'4800", 'ondisk_log_start': "388'4800", 'created': 374, 'last_epoch_clean': 375, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'last_clean_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 290914304, 'num_objects': 4439, 'num_object_clones': 0, 'num_object_copies': 8878, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3481, 'num_objects_unfound': 0, 'num_objects_dirty': 4439, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14857, 'num_write_kb': 617472, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5209, 'num_bytes_recovered': 341377024, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': ['5', '6'], 'object_location_counts': [{'shards': '3,5', 'objects': 958}, {'shards': '5,6', 'objects': 3481}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-15T23:23:25.504 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.2 is not active+clean 2024-05-15T23:23:25.504 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.2', 'version': "445'14706", 'reported_seq': 25114, 'reported_epoch': 445, 'state': 'active+recovering', 'last_fresh': '2024-05-15T23:23:20.512075+0000', 'last_change': '2024-05-15T23:21:09.275620+0000', 'last_active': '2024-05-15T23:23:20.512075+0000', 'last_peered': '2024-05-15T23:23:20.512075+0000', 'last_clean': '2024-05-15T23:21:07.993692+0000', 'last_became_active': '2024-05-15T23:21:09.221797+0000', 'last_became_peered': '2024-05-15T23:21:09.221797+0000', 'last_unstale': '2024-05-15T23:23:20.512075+0000', 'last_undegraded': '2024-05-15T23:23:20.512075+0000', 'last_fullsized': '2024-05-15T23:23:20.512075+0000', 'mapping_epoch': 407, 'log_start': "387'4700", 'ondisk_log_start': "387'4700", 'created': 374, 'last_epoch_clean': 375, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'last_clean_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 292552704, 'num_objects': 4464, 'num_object_clones': 0, 'num_object_copies': 8928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3503, 'num_objects_unfound': 0, 'num_objects_dirty': 4464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14706, 'num_write_kb': 613440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5126, 'num_bytes_recovered': 335937536, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [10, 7], 'acting': [10, 7], 'avail_no_missing': ['10', '6'], 'object_location_counts': [{'shards': '6,10', 'objects': 3503}, {'shards': '7,10', 'objects': 961}], 'blocked_by': [], 'up_primary': 10, 'acting_primary': 10, 'purged_snaps': []} 2024-05-15T23:23:25.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph status --format=json 2024-05-15T23:23:25.556 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:25.563 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:23:25.631 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:25.665 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:23:25.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:25.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:23:25.775 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:23:25.839 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:25.853 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:23:25.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:23:25.870 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:23:25.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:25.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:23:25.979 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:23:26.015 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:26.119 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:23:26.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:26.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:23:26.223 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:23:26.253 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:26.453 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:23:26.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:26.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:23:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:26.557 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:23:26.606 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:26.742 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:27.064 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:23:27.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:27.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:23:27.179 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:23:27.232 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:27.285 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:23:27.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:27.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:23:27.293 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:27 smithi088 ceph-mon[44699]: pgmap v1211: 17 pgs: 3 active+recovering, 14 active+clean; 4.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 827 op/s; 9837/137240 objects misplaced (7.168%); 10 MiB/s, 160 objects/s recovering 2024-05-15T23:23:27.293 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:27 smithi088 ceph-mon[44699]: osdmap e448: 12 total, 12 up, 9 in 2024-05-15T23:23:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:27 smithi084 ceph-mon[39784]: pgmap v1211: 17 pgs: 3 active+recovering, 14 active+clean; 4.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 827 op/s; 9837/137240 objects misplaced (7.168%); 10 MiB/s, 160 objects/s recovering 2024-05-15T23:23:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:27 smithi084 ceph-mon[39784]: osdmap e448: 12 total, 12 up, 9 in 2024-05-15T23:23:27.392 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:23:27.468 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:27 smithi090 ceph-mon[44434]: pgmap v1211: 17 pgs: 3 active+recovering, 14 active+clean; 4.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 827 op/s; 9837/137240 objects misplaced (7.168%); 10 MiB/s, 160 objects/s recovering 2024-05-15T23:23:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:27 smithi090 ceph-mon[44434]: osdmap e448: 12 total, 12 up, 9 in 2024-05-15T23:23:28.336 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:23:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:28.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:23:28.446 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:23:28.512 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:29.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:29 smithi084 ceph-mon[39784]: pgmap v1213: 17 pgs: 3 active+recovering, 14 active+clean; 4.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 831 op/s; 9586/134180 objects misplaced (7.144%); 9.5 MiB/s, 151 objects/s recovering 2024-05-15T23:23:29.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:29 smithi090 ceph-mon[44434]: pgmap v1213: 17 pgs: 3 active+recovering, 14 active+clean; 4.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 831 op/s; 9586/134180 objects misplaced (7.144%); 9.5 MiB/s, 151 objects/s recovering 2024-05-15T23:23:29.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:29 smithi088 ceph-mon[44699]: pgmap v1213: 17 pgs: 3 active+recovering, 14 active+clean; 4.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 831 op/s; 9586/134180 objects misplaced (7.144%); 9.5 MiB/s, 151 objects/s recovering 2024-05-15T23:23:29.583 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:23:29.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:29.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:23:29.693 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:23:29.723 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:29.745 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:23:29.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:29.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:23:29.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:23:29.876 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:30.005 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:23:30.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:30.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:23:30.118 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:23:30.149 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:30.417 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:23:30.417 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:23:30.489 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:23:30.512 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:23:30.534 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:23:30.557 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:23:30.585 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:23:30.611 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:23:30.638 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:23:30.662 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:23:30.685 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:23:30.702 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:23:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:30.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:23:30.708 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:23:30.729 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:23:30.755 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:23:30.757 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:23:30.757 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:23:30.757 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:23:30.757 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:23:30.757 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:23:30.757 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:23:30.757 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:23:30.757 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:23:30.757 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:23:30.757 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:23:30.757 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:23:30.757 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:23:30.780 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:30 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2882412734' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:30.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:30 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2882412734' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:30.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:23:30.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:30 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2882412734' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:30.845 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:31.534 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:23:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/775253476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:23:31.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:31 smithi084 ceph-mon[39784]: pgmap v1214: 17 pgs: 3 active+recovering, 14 active+clean; 3.9 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 835 op/s; 9082/129182 objects misplaced (7.030%); 10 MiB/s, 160 objects/s recovering 2024-05-15T23:23:31.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/775253476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:23:31.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:31 smithi090 ceph-mon[44434]: pgmap v1214: 17 pgs: 3 active+recovering, 14 active+clean; 3.9 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 835 op/s; 9082/129182 objects misplaced (7.030%); 10 MiB/s, 160 objects/s recovering 2024-05-15T23:23:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/775253476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:23:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:31 smithi088 ceph-mon[44699]: pgmap v1214: 17 pgs: 3 active+recovering, 14 active+clean; 3.9 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 835 op/s; 9082/129182 objects misplaced (7.030%); 10 MiB/s, 160 objects/s recovering 2024-05-15T23:23:32.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:32 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:23:32.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/775253476' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:23:32.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:32 smithi088 ceph-mon[44699]: osdmap e449: 12 total, 12 up, 9 in 2024-05-15T23:23:32.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:32 smithi088 ceph-mon[44699]: osdmap e450: 12 total, 12 up, 9 in 2024-05-15T23:23:32.704 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:23:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:32.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:23:32.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:32 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:23:32.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/775253476' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:23:32.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:32 smithi090 ceph-mon[44434]: osdmap e449: 12 total, 12 up, 9 in 2024-05-15T23:23:32.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:32 smithi090 ceph-mon[44434]: osdmap e450: 12 total, 12 up, 9 in 2024-05-15T23:23:32.807 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:23:32.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:32 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:23:32.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/775253476' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:23:32.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:32 smithi084 ceph-mon[39784]: osdmap e449: 12 total, 12 up, 9 in 2024-05-15T23:23:32.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:32 smithi084 ceph-mon[39784]: osdmap e450: 12 total, 12 up, 9 in 2024-05-15T23:23:32.835 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:32.860 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:23:32.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:32.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:23:32.964 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:23:32.994 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:33.020 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:23:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:33.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:23:33.128 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:23:33.186 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:23:33.187 INFO:teuthology.orchestra.run.smithi084.stdout:{"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1492,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":450,"num_osds":12,"num_up_osds":12,"osd_up_since":1715815247,"num_in_osds":9,"osd_in_since":1715815268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovering","count":3}],"num_pgs":17,"num_pools":2,"num_objects":64591,"data_bytes":4233429560,"bytes_used":11375624192,"bytes_avail":852579696640,"bytes_total":863955320832,"misplaced_objects":9082,"misplaced_total":129182,"misplaced_ratio":0.070303912309764516,"recovering_objects_per_sec":160,"recovering_bytes_per_sec":10500330,"recovering_keys_per_sec":0,"num_objects_recovered":1282,"num_bytes_recovered":84017152,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":835},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.84:8443/"}},"servicemap":{"epoch":34,"modified":"2024-05-15T23:23:09.001385+0000","services":{}},"progress_events":{}} 2024-05-15T23:23:33.207 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:33.299 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:23:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:33.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:23:33.362 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T23:23:33.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:23:33.462 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:33.486 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:23:33.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:33.495 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:23:33.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:23:33.623 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:33.722 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:23:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:33.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:23:33.795 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:23:33.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:33 smithi090 ceph-mon[44434]: pgmap v1217: 17 pgs: 3 active+recovering, 14 active+clean; 3.8 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 927 op/s; 8798/126102 objects misplaced (6.977%); 11 MiB/s, 173 objects/s recovering 2024-05-15T23:23:33.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:33 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1776969473' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T23:23:33.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:33 smithi088 ceph-mon[44699]: pgmap v1217: 17 pgs: 3 active+recovering, 14 active+clean; 3.8 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 927 op/s; 8798/126102 objects misplaced (6.977%); 11 MiB/s, 173 objects/s recovering 2024-05-15T23:23:33.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:33 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1776969473' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T23:23:33.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:23:33.839 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:23:33.855 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:33.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:33 smithi084 ceph-mon[39784]: pgmap v1217: 17 pgs: 3 active+recovering, 14 active+clean; 3.8 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 927 op/s; 8798/126102 objects misplaced (6.977%); 11 MiB/s, 173 objects/s recovering 2024-05-15T23:23:33.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:33 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1776969473' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T23:23:33.882 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:23:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:33.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:23:33.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:34.019 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:34.043 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:34.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:34.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:23:34.150 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:23:34.179 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:34.328 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:23:34.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:34.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:23:34.342 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-05-15T23:23:34.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:23:34.435 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:23:34.482 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:34.486 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:34.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:23:35.356 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:23:35.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:35.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:23:35.367 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:35 smithi088 ceph-mon[44699]: pgmap v1218: 17 pgs: 3 active+recovering, 14 active+clean; 3.8 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 732 op/s; 8798/125592 objects misplaced (7.005%); 8.2 MiB/s, 130 objects/s recovering 2024-05-15T23:23:35.367 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:35 smithi084 ceph-mon[39784]: pgmap v1218: 17 pgs: 3 active+recovering, 14 active+clean; 3.8 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 732 op/s; 8798/125592 objects misplaced (7.005%); 8.2 MiB/s, 130 objects/s recovering 2024-05-15T23:23:35.370 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:35.466 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:23:35.518 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:35.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:35 smithi090 ceph-mon[44434]: pgmap v1218: 17 pgs: 3 active+recovering, 14 active+clean; 3.8 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 732 op/s; 8798/125592 objects misplaced (7.005%); 8.2 MiB/s, 130 objects/s recovering 2024-05-15T23:23:36.511 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:23:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:36.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:23:36.616 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:23:36.688 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:37.350 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:23:37.350 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:37 smithi088 ceph-mon[44699]: pgmap v1219: 17 pgs: 3 active+recovering, 14 active+clean; 3.7 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 833 op/s; 8236/120842 objects misplaced (6.816%); 11 MiB/s, 168 objects/s recovering 2024-05-15T23:23:37.353 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:37 smithi088 ceph-mon[44699]: osdmap e451: 12 total, 12 up, 9 in 2024-05-15T23:23:37.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:37.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:23:37.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:37 smithi084 ceph-mon[39784]: pgmap v1219: 17 pgs: 3 active+recovering, 14 active+clean; 3.7 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 833 op/s; 8236/120842 objects misplaced (6.816%); 11 MiB/s, 168 objects/s recovering 2024-05-15T23:23:37.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:37 smithi084 ceph-mon[39784]: osdmap e451: 12 total, 12 up, 9 in 2024-05-15T23:23:37.455 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:23:37.513 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:37.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:37 smithi090 ceph-mon[44434]: pgmap v1219: 17 pgs: 3 active+recovering, 14 active+clean; 3.7 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 833 op/s; 8236/120842 objects misplaced (6.816%); 11 MiB/s, 168 objects/s recovering 2024-05-15T23:23:37.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:37 smithi090 ceph-mon[44434]: osdmap e451: 12 total, 12 up, 9 in 2024-05-15T23:23:37.986 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:23:37.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:37.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:23:38.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:38.138 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:38.166 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:38.170 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:23:38.270 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:23:38.303 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:38.503 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:23:38.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:38.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:23:38.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:23:38.649 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:38.673 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:23:38.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:38.679 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:23:38.782 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:23:38.813 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:39.033 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:23:39.034 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:23:39.034 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2024-05-15T23:23:39.034 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:23:39.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:39 smithi084 ceph-mon[39784]: pgmap v1221: 17 pgs: 3 active+recovering, 14 active+clean; 3.6 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 776 op/s; 7967/117572 objects misplaced (6.776%); 9.3 MiB/s, 149 objects/s recovering 2024-05-15T23:23:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:39 smithi090 ceph-mon[44434]: pgmap v1221: 17 pgs: 3 active+recovering, 14 active+clean; 3.6 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 776 op/s; 7967/117572 objects misplaced (6.776%); 9.3 MiB/s, 149 objects/s recovering 2024-05-15T23:23:39.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:39 smithi088 ceph-mon[44699]: pgmap v1221: 17 pgs: 3 active+recovering, 14 active+clean; 3.6 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 776 op/s; 7967/117572 objects misplaced (6.776%); 9.3 MiB/s, 149 objects/s recovering 2024-05-15T23:23:39.846 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:23:39.846 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:23:40.065 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:23:40.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:40.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:23:40.169 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:23:40.242 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:40 smithi088 ceph-mon[44699]: from='client.16236 v1:172.21.15.84:0/2209958654' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:40.249 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:40.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:40 smithi084 ceph-mon[39784]: from='client.16236 v1:172.21.15.84:0/2209958654' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:40.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:40 smithi090 ceph-mon[44434]: from='client.16236 v1:172.21.15.84:0/2209958654' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:40.606 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:23:40.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:40.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:23:40.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:23:40.741 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T23:23:40.753 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:40.776 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:23:40.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:40.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:23:40.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:23:40.930 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:40.952 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:23:40.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:40.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:23:41.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:23:41.098 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:41.117 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1221,"stamp":"2024-05-15T23:23:39.006975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3852993080,"num_objects":58786,"num_object_clones":0,"num_object_copies":117572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7967,"num_objects_unfound":0,"num_objects_dirty":58786,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":248455,"num_write_kb":9829799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17928,"num_bytes_recovered":1176122976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160726,"ondisk_log_size":160726,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":10462220,"kb_used_data":7088372,"kb_used_omap":14,"kb_used_meta":3373617,"kb_avail":833244148,"statfs":{"total":863955320832,"available":853242007552,"internally_reserved":0,"allocated":7258492928,"data_stored":7254574570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14358,"internal_metadata":3454584810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,6,5,4,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-380436480,"num_objects":-5805,"num_object_clones":0,"num_object_copies":-11610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1115,"num_objects_unfound":0,"num_objects_dirty":-5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5805,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1115,"num_bytes_recovered":73072640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.477355"},"pg_stats":[{"pgid":"5.f","version":"451'15378","reported_seq":15542,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-05-15T23:23:37.600196+0000","last_change":"2024-05-15T23:19:47.305945+0000","last_active":"2024-05-15T23:23:37.600196+0000","last_peered":"2024-05-15T23:23:37.600196+0000","last_clean":"2024-05-15T23:23:37.600196+0000","last_became_active":"2024-05-15T23:19:47.305689+0000","last_became_peered":"2024-05-15T23:19:47.305689+0000","last_unstale":"2024-05-15T23:23:37.600196+0000","last_undegraded":"2024-05-15T23:23:37.600196+0000","last_fullsized":"2024-05-15T23:23:37.600196+0000","mapping_epoch":374,"log_start":"390'5300","ondisk_log_start":"390'5300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241303552,"num_objects":3682,"num_object_clones":0,"num_object_copies":7364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15378,"num_write_kb":609920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"450'15280","reported_seq":15617,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-05-15T23:23:35.586485+0000","last_change":"2024-05-15T23:22:23.637106+0000","last_active":"2024-05-15T23:23:35.586485+0000","last_peered":"2024-05-15T23:23:35.586485+0000","last_clean":"2024-05-15T23:23:35.586485+0000","last_became_active":"2024-05-15T23:19:47.325414+0000","last_became_peered":"2024-05-15T23:19:47.325414+0000","last_unstale":"2024-05-15T23:23:35.586485+0000","last_undegraded":"2024-05-15T23:23:35.586485+0000","last_fullsized":"2024-05-15T23:23:35.586485+0000","mapping_epoch":374,"log_start":"390'5200","ondisk_log_start":"390'5200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'11848","last_scrub_stamp":"2024-05-15T23:22:23.636778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:23.636778+0000","objects_scrubbed":7095,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:23:39.191980+0000","scrub_duration":997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238682112,"num_objects":3642,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15280,"num_write_kb":605504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"451'15850","reported_seq":16015,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-05-15T23:23:37.125628+0000","last_change":"2024-05-15T23:19:47.306160+0000","last_active":"2024-05-15T23:23:37.125628+0000","last_peered":"2024-05-15T23:23:37.125628+0000","last_clean":"2024-05-15T23:23:37.125628+0000","last_became_active":"2024-05-15T23:19:47.305860+0000","last_became_peered":"2024-05-15T23:19:47.305860+0000","last_unstale":"2024-05-15T23:23:37.125628+0000","last_undegraded":"2024-05-15T23:23:37.125628+0000","last_fullsized":"2024-05-15T23:23:37.125628+0000","mapping_epoch":374,"log_start":"392'5800","ondisk_log_start":"392'5800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235601944,"num_objects":3596,"num_object_clones":0,"num_object_copies":7192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3596,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15850,"num_write_kb":622209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"450'15608","reported_seq":15933,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-05-15T23:23:35.765526+0000","last_change":"2024-05-15T23:22:51.995640+0000","last_active":"2024-05-15T23:23:35.765526+0000","last_peered":"2024-05-15T23:23:35.765526+0000","last_clean":"2024-05-15T23:23:35.765526+0000","last_became_active":"2024-05-15T23:19:47.306168+0000","last_became_peered":"2024-05-15T23:19:47.306168+0000","last_unstale":"2024-05-15T23:23:35.765526+0000","last_undegraded":"2024-05-15T23:23:35.765526+0000","last_fullsized":"2024-05-15T23:23:35.765526+0000","mapping_epoch":374,"log_start":"392'5600","ondisk_log_start":"392'5600","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'13335","last_scrub_stamp":"2024-05-15T23:22:51.995473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:51.995473+0000","objects_scrubbed":6018,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:13.168527+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243138560,"num_objects":3710,"num_object_clones":0,"num_object_copies":7420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15608,"num_write_kb":618176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"450'15460","reported_seq":15788,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-05-15T23:23:35.575547+0000","last_change":"2024-05-15T23:22:48.338084+0000","last_active":"2024-05-15T23:23:35.575547+0000","last_peered":"2024-05-15T23:23:35.575547+0000","last_clean":"2024-05-15T23:23:35.575547+0000","last_became_active":"2024-05-15T23:19:47.307061+0000","last_became_peered":"2024-05-15T23:19:47.307061+0000","last_unstale":"2024-05-15T23:23:35.575547+0000","last_undegraded":"2024-05-15T23:23:35.575547+0000","last_fullsized":"2024-05-15T23:23:35.575547+0000","mapping_epoch":374,"log_start":"390'5400","ondisk_log_start":"390'5400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'13172","last_scrub_stamp":"2024-05-15T23:22:48.337866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:48.337866+0000","objects_scrubbed":6060,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:11.906769+0000","scrub_duration":1844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244580352,"num_objects":3732,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15460,"num_write_kb":614144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"450'15532","reported_seq":15694,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-05-15T23:23:35.560902+0000","last_change":"2024-05-15T23:19:47.325657+0000","last_active":"2024-05-15T23:23:35.560902+0000","last_peered":"2024-05-15T23:23:35.560902+0000","last_clean":"2024-05-15T23:23:35.560902+0000","last_became_active":"2024-05-15T23:19:47.325371+0000","last_became_peered":"2024-05-15T23:19:47.325371+0000","last_unstale":"2024-05-15T23:23:35.560902+0000","last_undegraded":"2024-05-15T23:23:35.560902+0000","last_fullsized":"2024-05-15T23:23:35.560902+0000","mapping_epoch":374,"log_start":"391'5500","ondisk_log_start":"391'5500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241172480,"num_objects":3680,"num_object_clones":0,"num_object_copies":7360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15532,"num_write_kb":614784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"450'15511","reported_seq":27570,"reported_epoch":450,"state":"active+recovering","last_fresh":"2024-05-15T23:23:36.020583+0000","last_change":"2024-05-15T23:21:09.568311+0000","last_active":"2024-05-15T23:23:36.020583+0000","last_peered":"2024-05-15T23:23:36.020583+0000","last_clean":"2024-05-15T23:21:08.023957+0000","last_became_active":"2024-05-15T23:21:09.454639+0000","last_became_peered":"2024-05-15T23:21:09.454639+0000","last_unstale":"2024-05-15T23:23:36.020583+0000","last_undegraded":"2024-05-15T23:23:36.020583+0000","last_fullsized":"2024-05-15T23:23:36.020583+0000","mapping_epoch":407,"log_start":"391'5500","ondisk_log_start":"391'5500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238747648,"num_objects":3643,"num_object_clones":0,"num_object_copies":7286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2623,"num_objects_unfound":0,"num_objects_dirty":3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15511,"num_write_kb":612928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5947,"num_bytes_recovered":389742592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":2623},{"shards":"7,11","objects":1020}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"451'15543","reported_seq":15707,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-05-15T23:23:37.076730+0000","last_change":"2024-05-15T23:19:47.304832+0000","last_active":"2024-05-15T23:23:37.076730+0000","last_peered":"2024-05-15T23:23:37.076730+0000","last_clean":"2024-05-15T23:23:37.076730+0000","last_became_active":"2024-05-15T23:19:47.304547+0000","last_became_peered":"2024-05-15T23:19:47.304547+0000","last_unstale":"2024-05-15T23:23:37.076730+0000","last_undegraded":"2024-05-15T23:23:37.076730+0000","last_fullsized":"2024-05-15T23:23:37.076730+0000","mapping_epoch":374,"log_start":"391'5500","ondisk_log_start":"391'5500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239927296,"num_objects":3661,"num_object_clones":0,"num_object_copies":7322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15543,"num_write_kb":614528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"451'15451","reported_seq":15615,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-05-15T23:23:37.568353+0000","last_change":"2024-05-15T23:19:47.306634+0000","last_active":"2024-05-15T23:23:37.568353+0000","last_peered":"2024-05-15T23:23:37.568353+0000","last_clean":"2024-05-15T23:23:37.568353+0000","last_became_active":"2024-05-15T23:19:47.306331+0000","last_became_peered":"2024-05-15T23:19:47.306331+0000","last_unstale":"2024-05-15T23:23:37.568353+0000","last_undegraded":"2024-05-15T23:23:37.568353+0000","last_fullsized":"2024-05-15T23:23:37.568353+0000","mapping_epoch":374,"log_start":"391'5400","ondisk_log_start":"391'5400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236257280,"num_objects":3605,"num_object_clones":0,"num_object_copies":7210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15451,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"451'15730","reported_seq":15894,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-05-15T23:23:37.545599+0000","last_change":"2024-05-15T23:19:47.306611+0000","last_active":"2024-05-15T23:23:37.545599+0000","last_peered":"2024-05-15T23:23:37.545599+0000","last_clean":"2024-05-15T23:23:37.545599+0000","last_became_active":"2024-05-15T23:19:47.306349+0000","last_became_peered":"2024-05-15T23:19:47.306349+0000","last_unstale":"2024-05-15T23:23:37.545599+0000","last_undegraded":"2024-05-15T23:23:37.545599+0000","last_fullsized":"2024-05-15T23:23:37.545599+0000","mapping_epoch":374,"log_start":"392'5700","ondisk_log_start":"392'5700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234618880,"num_objects":3580,"num_object_clones":0,"num_object_copies":7160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15730,"num_write_kb":617920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"450'15656","reported_seq":16048,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-05-15T23:23:35.802513+0000","last_change":"2024-05-15T23:22:48.067681+0000","last_active":"2024-05-15T23:23:35.802513+0000","last_peered":"2024-05-15T23:23:35.802513+0000","last_clean":"2024-05-15T23:23:35.802513+0000","last_became_active":"2024-05-15T23:19:47.305685+0000","last_became_peered":"2024-05-15T23:19:47.305685+0000","last_unstale":"2024-05-15T23:23:35.802513+0000","last_undegraded":"2024-05-15T23:23:35.802513+0000","last_fullsized":"2024-05-15T23:23:35.802513+0000","mapping_epoch":374,"log_start":"391'5600","ondisk_log_start":"391'5600","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'13277","last_scrub_stamp":"2024-05-15T23:22:48.067507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:48.067507+0000","objects_scrubbed":6142,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:23:49.704509+0000","scrub_duration":929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245104640,"num_objects":3740,"num_object_clones":0,"num_object_copies":7480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15656,"num_write_kb":620672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"451'15659","reported_seq":27851,"reported_epoch":451,"state":"active+recovering","last_fresh":"2024-05-15T23:23:37.603105+0000","last_change":"2024-05-15T23:21:09.470461+0000","last_active":"2024-05-15T23:23:37.603105+0000","last_peered":"2024-05-15T23:23:37.603105+0000","last_clean":"2024-05-15T23:21:07.989871+0000","last_became_active":"2024-05-15T23:21:09.400731+0000","last_became_peered":"2024-05-15T23:21:09.400731+0000","last_unstale":"2024-05-15T23:23:37.603105+0000","last_undegraded":"2024-05-15T23:23:37.603105+0000","last_fullsized":"2024-05-15T23:23:37.603105+0000","mapping_epoch":407,"log_start":"392'5600","ondisk_log_start":"392'5600","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238354432,"num_objects":3637,"num_object_clones":0,"num_object_copies":7274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2677,"num_objects_unfound":0,"num_objects_dirty":3637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15659,"num_write_kb":617472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6013,"num_bytes_recovered":394067968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"3,5","objects":960},{"shards":"5,6","objects":2677}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"450'15546","reported_seq":27636,"reported_epoch":450,"state":"active+recovering","last_fresh":"2024-05-15T23:23:35.562041+0000","last_change":"2024-05-15T23:21:09.275620+0000","last_active":"2024-05-15T23:23:35.562041+0000","last_peered":"2024-05-15T23:23:35.562041+0000","last_clean":"2024-05-15T23:21:07.993692+0000","last_became_active":"2024-05-15T23:21:09.221797+0000","last_became_peered":"2024-05-15T23:21:09.221797+0000","last_unstale":"2024-05-15T23:23:35.562041+0000","last_undegraded":"2024-05-15T23:23:35.562041+0000","last_fullsized":"2024-05-15T23:23:35.562041+0000","mapping_epoch":407,"log_start":"391'5500","ondisk_log_start":"391'5500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237502464,"num_objects":3624,"num_object_clones":0,"num_object_copies":7248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2667,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15546,"num_write_kb":613440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5962,"num_bytes_recovered":390725632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":2667},{"shards":"7,10","objects":957}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.1","version":"450'15002","reported_seq":15337,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-05-15T23:23:34.895838+0000","last_change":"2024-05-15T23:22:19.625857+0000","last_active":"2024-05-15T23:23:34.895838+0000","last_peered":"2024-05-15T23:23:34.895838+0000","last_clean":"2024-05-15T23:23:34.895838+0000","last_became_active":"2024-05-15T23:19:47.516865+0000","last_became_peered":"2024-05-15T23:19:47.516865+0000","last_unstale":"2024-05-15T23:23:34.895838+0000","last_undegraded":"2024-05-15T23:23:34.895838+0000","last_fullsized":"2024-05-15T23:23:34.895838+0000","mapping_epoch":374,"log_start":"390'5000","ondisk_log_start":"390'5000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"426'11474","last_scrub_stamp":"2024-05-15T23:22:19.625719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:19.625719+0000","objects_scrubbed":7211,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:23:49.523745+0000","scrub_duration":1229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239599616,"num_objects":3656,"num_object_clones":0,"num_object_copies":7312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15002,"num_write_kb":597056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"450'15520","reported_seq":15682,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-05-15T23:23:35.480280+0000","last_change":"2024-05-15T23:19:47.305723+0000","last_active":"2024-05-15T23:23:35.480280+0000","last_peered":"2024-05-15T23:23:35.480280+0000","last_clean":"2024-05-15T23:23:35.480280+0000","last_became_active":"2024-05-15T23:19:47.305459+0000","last_became_peered":"2024-05-15T23:19:47.305459+0000","last_unstale":"2024-05-15T23:23:35.480280+0000","last_undegraded":"2024-05-15T23:23:35.480280+0000","last_fullsized":"2024-05-15T23:23:35.480280+0000","mapping_epoch":374,"log_start":"391'5500","ondisk_log_start":"391'5500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246153216,"num_objects":3756,"num_object_clones":0,"num_object_copies":7512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15520,"num_write_kb":616832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"450'15606","reported_seq":15768,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-05-15T23:23:35.541369+0000","last_change":"2024-05-15T23:19:47.307499+0000","last_active":"2024-05-15T23:23:35.541369+0000","last_peered":"2024-05-15T23:23:35.541369+0000","last_clean":"2024-05-15T23:23:35.541369+0000","last_became_active":"2024-05-15T23:19:47.307149+0000","last_became_peered":"2024-05-15T23:19:47.307149+0000","last_unstale":"2024-05-15T23:23:35.541369+0000","last_undegraded":"2024-05-15T23:23:35.541369+0000","last_fullsized":"2024-05-15T23:23:35.541369+0000","mapping_epoch":374,"log_start":"392'5600","ondisk_log_start":"392'5600","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251658240,"num_objects":3840,"num_object_clones":0,"num_object_copies":7680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15606,"num_write_kb":622272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":939,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-05-15T23:23:32.061218+0000","last_change":"2024-05-15T23:20:05.905241+0000","last_active":"2024-05-15T23:23:32.061218+0000","last_peered":"2024-05-15T23:23:32.061218+0000","last_clean":"2024-05-15T23:23:32.061218+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:23:32.061218+0000","last_undegraded":"2024-05-15T23:23:32.061218+0000","last_fullsized":"2024-05-15T23:23:32.061218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:20:05.905105+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:20:05.905105+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3852402712,"num_objects":58784,"num_object_clones":0,"num_object_copies":117568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7967,"num_objects_unfound":0,"num_objects_dirty":58784,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":248332,"num_write_kb":9827649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17922,"num_bytes_recovered":1174536192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8899534848,"data_stored":8899526704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160632,"ondisk_log_size":160632,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351214,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1072868,"kb_used_data":532188,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92672284,"statfs":{"total":95995035648,"available":94896418816,"internally_reserved":0,"allocated":544960512,"data_stored":544519864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":553646532},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580147,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1483672,"kb_used_data":1203984,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":92261480,"statfs":{"total":95995035648,"available":94475755520,"internally_reserved":0,"allocated":1232879616,"data_stored":1232446612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":286390720},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776379,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":860248,"kb_used_data":473680,"kb_used_omap":1,"kb_used_meta":386558,"kb_avail":92884904,"statfs":{"total":95995035648,"available":95114141696,"internally_reserved":0,"allocated":485048320,"data_stored":484615316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":395835840},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005311,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663840,"kb_used_data":479952,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93081312,"statfs":{"total":95995035648,"available":95315263488,"internally_reserved":0,"allocated":491470848,"data_stored":491037844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266947,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":870368,"kb_used_data":532376,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92874784,"statfs":{"total":95995035648,"available":95103778816,"internally_reserved":0,"allocated":545153024,"data_stored":544716468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":346094026},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495882,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575139,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1429464,"kb_used_data":960976,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":92315688,"statfs":{"total":95995035648,"available":94531264512,"internally_reserved":0,"allocated":984039424,"data_stored":983606420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":479721926},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016548,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483237,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149978,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":714264,"kb_used_data":295760,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93030888,"statfs":{"total":95995035648,"available":95263629312,"internally_reserved":0,"allocated":302858240,"data_stored":302425236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":428538311},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986450,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1360028,"kb_used_data":971920,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":92385124,"statfs":{"total":95995035648,"available":94602366976,"internally_reserved":0,"allocated":995246080,"data_stored":994813076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790223,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2007468,"kb_used_data":1637536,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":91737684,"statfs":{"total":95995035648,"available":93939388416,"internally_reserved":0,"allocated":1676836864,"data_stored":1676393734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":974192640,"data_stored":974192640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":299827200,"data_stored":299827200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":985989120,"data_stored":985989120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1667043328,"data_stored":1667039256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1696661504,"data_stored":1696661504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":542834688,"data_stored":542834688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":486604800,"data_stored":486604800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":483196928,"data_stored":483196928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1221591040,"data_stored":1221591040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":541593600,"data_stored":541589528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:23:41.119 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:23:41.238 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:23:41.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:23:41.239 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:23:41.239 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:23:41.239 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:23:41.239 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:23:41.239 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:23:41.239 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:23:41.239 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:23:41.239 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:23:41.239 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:23:41.239 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:23:41.239 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:23:41.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:23:41.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:23:41.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:23:41.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:23:41.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:23:41.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:23:41.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:23:41.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:23:41.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:23:41.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:23:41.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:23:41.241 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:23:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:23:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:23:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: osdmap e452: 12 total, 12 up, 9 in 2024-05-15T23:23:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=3756826548,config=3313370793,health=3340005642,kv=1034917633}) 2024-05-15T23:23:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=3808566674,logm=594696456}) 2024-05-15T23:23:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1859166793}) 2024-05-15T23:23:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1272622389}) 2024-05-15T23:23:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2828952471}) 2024-05-15T23:23:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=827751385}) 2024-05-15T23:23:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1949763737}) 2024-05-15T23:23:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1901508842}) 2024-05-15T23:23:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3187387302}) 2024-05-15T23:23:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=482389370}) 2024-05-15T23:23:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3428545426}) 2024-05-15T23:23:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764703124}) 2024-05-15T23:23:41.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=167838868}) 2024-05-15T23:23:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4288689889}) 2024-05-15T23:23:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2008805134}) 2024-05-15T23:23:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2276994830}) 2024-05-15T23:23:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999945854}) 2024-05-15T23:23:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3592033875}) 2024-05-15T23:23:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500388241}) 2024-05-15T23:23:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2286287071}) 2024-05-15T23:23:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1706753728}) 2024-05-15T23:23:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4247784644}) 2024-05-15T23:23:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3479424494}) 2024-05-15T23:23:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3414559639}) 2024-05-15T23:23:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=793594312}) 2024-05-15T23:23:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2239707059}) 2024-05-15T23:23:41.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=266193507}) 2024-05-15T23:23:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2247473252}) 2024-05-15T23:23:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=209261101}) 2024-05-15T23:23:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=249773966}) 2024-05-15T23:23:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3206428814}) 2024-05-15T23:23:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073588038}) 2024-05-15T23:23:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1773355237}) 2024-05-15T23:23:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1092208074}) 2024-05-15T23:23:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2166996993}) 2024-05-15T23:23:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3469367850}) 2024-05-15T23:23:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3997813687}) 2024-05-15T23:23:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3263599681}) 2024-05-15T23:23:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2189251660}) 2024-05-15T23:23:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=159173}) 2024-05-15T23:23:41.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4254659318}) 2024-05-15T23:23:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3244364947}) 2024-05-15T23:23:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=217009549}) 2024-05-15T23:23:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3654563546}) 2024-05-15T23:23:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3848120438}) 2024-05-15T23:23:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2561178273}) 2024-05-15T23:23:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2526869702}) 2024-05-15T23:23:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2738004208}) 2024-05-15T23:23:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514958691}) 2024-05-15T23:23:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1248160596}) 2024-05-15T23:23:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2524318879}) 2024-05-15T23:23:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3663269057}) 2024-05-15T23:23:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3300158717}) 2024-05-15T23:23:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4104640742}) 2024-05-15T23:23:41.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3469977157}) 2024-05-15T23:23:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2017219131}) 2024-05-15T23:23:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1261291036}) 2024-05-15T23:23:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=61,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=15} crc {logm=610257476,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2334445753}) 2024-05-15T23:23:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=2,mon_config_key=98} crc {mgrstat=4263447627,mon_config_key=2329063236}) 2024-05-15T23:23:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=819631405}) 2024-05-15T23:23:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=86,monmap=5,osd_metadata=9} crc {mon_config_key=1658623506,monmap=1478146745,osd_metadata=1476378719}) 2024-05-15T23:23:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osdmap=96} crc {osd_metadata=1494755061,osd_pg_creating=859180981,osdmap=3932947652}) 2024-05-15T23:23:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=41} crc {osdmap=183594719}) 2024-05-15T23:23:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3221329884' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:23:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2594320617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:41.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:41 smithi084 ceph-mon[39784]: pgmap v1223: 17 pgs: 3 active+recovering, 14 active+clean; 3.4 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 841 op/s; 7686/112642 objects misplaced (6.823%); 8.7 MiB/s, 138 objects/s recovering 2024-05-15T23:23:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:23:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:23:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: osdmap e452: 12 total, 12 up, 9 in 2024-05-15T23:23:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=3756826548,config=3313370793,health=3340005642,kv=1034917633}) 2024-05-15T23:23:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=3808566674,logm=594696456}) 2024-05-15T23:23:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1859166793}) 2024-05-15T23:23:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1272622389}) 2024-05-15T23:23:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2828952471}) 2024-05-15T23:23:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=827751385}) 2024-05-15T23:23:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1949763737}) 2024-05-15T23:23:41.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1901508842}) 2024-05-15T23:23:41.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3187387302}) 2024-05-15T23:23:41.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=482389370}) 2024-05-15T23:23:41.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3428545426}) 2024-05-15T23:23:41.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764703124}) 2024-05-15T23:23:41.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=167838868}) 2024-05-15T23:23:41.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4288689889}) 2024-05-15T23:23:41.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2008805134}) 2024-05-15T23:23:41.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2276994830}) 2024-05-15T23:23:41.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999945854}) 2024-05-15T23:23:41.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3592033875}) 2024-05-15T23:23:41.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500388241}) 2024-05-15T23:23:41.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2286287071}) 2024-05-15T23:23:41.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1706753728}) 2024-05-15T23:23:41.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4247784644}) 2024-05-15T23:23:41.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3479424494}) 2024-05-15T23:23:41.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3414559639}) 2024-05-15T23:23:41.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=793594312}) 2024-05-15T23:23:41.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2239707059}) 2024-05-15T23:23:41.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=266193507}) 2024-05-15T23:23:41.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2247473252}) 2024-05-15T23:23:41.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=209261101}) 2024-05-15T23:23:41.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=249773966}) 2024-05-15T23:23:41.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3206428814}) 2024-05-15T23:23:41.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073588038}) 2024-05-15T23:23:41.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1773355237}) 2024-05-15T23:23:41.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1092208074}) 2024-05-15T23:23:41.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2166996993}) 2024-05-15T23:23:41.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3469367850}) 2024-05-15T23:23:41.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3997813687}) 2024-05-15T23:23:41.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3263599681}) 2024-05-15T23:23:41.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2189251660}) 2024-05-15T23:23:41.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=159173}) 2024-05-15T23:23:41.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4254659318}) 2024-05-15T23:23:41.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3244364947}) 2024-05-15T23:23:41.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=217009549}) 2024-05-15T23:23:41.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3654563546}) 2024-05-15T23:23:41.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3848120438}) 2024-05-15T23:23:41.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2561178273}) 2024-05-15T23:23:41.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2526869702}) 2024-05-15T23:23:41.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2738004208}) 2024-05-15T23:23:41.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514958691}) 2024-05-15T23:23:41.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1248160596}) 2024-05-15T23:23:41.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2524318879}) 2024-05-15T23:23:41.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3663269057}) 2024-05-15T23:23:41.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3300158717}) 2024-05-15T23:23:41.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4104640742}) 2024-05-15T23:23:41.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3469977157}) 2024-05-15T23:23:41.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2017219131}) 2024-05-15T23:23:41.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1261291036}) 2024-05-15T23:23:41.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=61,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=15} crc {logm=610257476,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2334445753}) 2024-05-15T23:23:41.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=2,mon_config_key=98} crc {mgrstat=4263447627,mon_config_key=2329063236}) 2024-05-15T23:23:41.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=819631405}) 2024-05-15T23:23:41.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=86,monmap=5,osd_metadata=9} crc {mon_config_key=1658623506,monmap=1478146745,osd_metadata=1476378719}) 2024-05-15T23:23:41.559 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osdmap=96} crc {osd_metadata=1494755061,osd_pg_creating=859180981,osdmap=3932947652}) 2024-05-15T23:23:41.559 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=41} crc {osdmap=183594719}) 2024-05-15T23:23:41.559 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3221329884' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:23:41.559 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2594320617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:41.559 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:41 smithi090 ceph-mon[44434]: pgmap v1223: 17 pgs: 3 active+recovering, 14 active+clean; 3.4 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 841 op/s; 7686/112642 objects misplaced (6.823%); 8.7 MiB/s, 138 objects/s recovering 2024-05-15T23:23:41.559 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:23:41.559 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:23:41.560 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: osdmap e452: 12 total, 12 up, 9 in 2024-05-15T23:23:41.560 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=3756826548,config=3313370793,health=3340005642,kv=1034917633}) 2024-05-15T23:23:41.560 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=3808566674,logm=594696456}) 2024-05-15T23:23:41.560 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1859166793}) 2024-05-15T23:23:41.560 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1272622389}) 2024-05-15T23:23:41.560 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2828952471}) 2024-05-15T23:23:41.560 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=827751385}) 2024-05-15T23:23:41.560 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1949763737}) 2024-05-15T23:23:41.560 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1901508842}) 2024-05-15T23:23:41.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3187387302}) 2024-05-15T23:23:41.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=482389370}) 2024-05-15T23:23:41.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3428545426}) 2024-05-15T23:23:41.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764703124}) 2024-05-15T23:23:41.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=167838868}) 2024-05-15T23:23:41.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4288689889}) 2024-05-15T23:23:41.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2008805134}) 2024-05-15T23:23:41.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2276994830}) 2024-05-15T23:23:41.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999945854}) 2024-05-15T23:23:41.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3592033875}) 2024-05-15T23:23:41.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500388241}) 2024-05-15T23:23:41.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2286287071}) 2024-05-15T23:23:41.561 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1706753728}) 2024-05-15T23:23:41.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4247784644}) 2024-05-15T23:23:41.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3479424494}) 2024-05-15T23:23:41.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3414559639}) 2024-05-15T23:23:41.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=793594312}) 2024-05-15T23:23:41.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2239707059}) 2024-05-15T23:23:41.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=266193507}) 2024-05-15T23:23:41.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2247473252}) 2024-05-15T23:23:41.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=209261101}) 2024-05-15T23:23:41.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=249773966}) 2024-05-15T23:23:41.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3206428814}) 2024-05-15T23:23:41.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073588038}) 2024-05-15T23:23:41.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1773355237}) 2024-05-15T23:23:41.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1092208074}) 2024-05-15T23:23:41.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2166996993}) 2024-05-15T23:23:41.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3469367850}) 2024-05-15T23:23:41.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3997813687}) 2024-05-15T23:23:41.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3263599681}) 2024-05-15T23:23:41.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2189251660}) 2024-05-15T23:23:41.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=159173}) 2024-05-15T23:23:41.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4254659318}) 2024-05-15T23:23:41.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3244364947}) 2024-05-15T23:23:41.563 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=217009549}) 2024-05-15T23:23:41.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3654563546}) 2024-05-15T23:23:41.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3848120438}) 2024-05-15T23:23:41.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2561178273}) 2024-05-15T23:23:41.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2526869702}) 2024-05-15T23:23:41.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2738004208}) 2024-05-15T23:23:41.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514958691}) 2024-05-15T23:23:41.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1248160596}) 2024-05-15T23:23:41.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2524318879}) 2024-05-15T23:23:41.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3663269057}) 2024-05-15T23:23:41.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3300158717}) 2024-05-15T23:23:41.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4104640742}) 2024-05-15T23:23:41.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3469977157}) 2024-05-15T23:23:41.564 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2017219131}) 2024-05-15T23:23:41.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1261291036}) 2024-05-15T23:23:41.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=61,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=15} crc {logm=610257476,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2334445753}) 2024-05-15T23:23:41.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=2,mon_config_key=98} crc {mgrstat=4263447627,mon_config_key=2329063236}) 2024-05-15T23:23:41.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=819631405}) 2024-05-15T23:23:41.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=86,monmap=5,osd_metadata=9} crc {mon_config_key=1658623506,monmap=1478146745,osd_metadata=1476378719}) 2024-05-15T23:23:41.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osdmap=96} crc {osd_metadata=1494755061,osd_pg_creating=859180981,osdmap=3932947652}) 2024-05-15T23:23:41.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=41} crc {osdmap=183594719}) 2024-05-15T23:23:41.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3221329884' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:23:41.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2594320617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:41.565 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:41 smithi088 ceph-mon[44699]: pgmap v1223: 17 pgs: 3 active+recovering, 14 active+clean; 3.4 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 841 op/s; 7686/112642 objects misplaced (6.823%); 8.7 MiB/s, 138 objects/s recovering 2024-05-15T23:23:42.109 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:23:42.109 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:23:42.109 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.109 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.110 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.110 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.110 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.110 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.110 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.110 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:23:42.110 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3221329884' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:23:42.110 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: osdmap e453: 12 total, 12 up, 9 in 2024-05-15T23:23:42.110 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.110 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.110 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.110 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.110 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:23:42.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:23:42.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:23:42.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3221329884' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:23:42.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: osdmap e453: 12 total, 12 up, 9 in 2024-05-15T23:23:42.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:23:42.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:23:42.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:23:42.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3221329884' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:23:42.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: osdmap e453: 12 total, 12 up, 9 in 2024-05-15T23:23:42.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:42.658 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:23:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:42.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:23:42.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:42.808 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:42.841 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:42.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:42.845 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:23:42.946 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:23:42.974 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:42.995 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:23:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:42.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:23:43.099 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:23:43.166 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:43.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:43 smithi084 ceph-mon[39784]: pgmap v1225: 17 pgs: 3 active+recovering, 14 active+clean; 3.3 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 7134/108746 objects misplaced (6.560%); 11 MiB/s, 183 objects/s recovering 2024-05-15T23:23:43.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:43 smithi090 ceph-mon[44434]: pgmap v1225: 17 pgs: 3 active+recovering, 14 active+clean; 3.3 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 7134/108746 objects misplaced (6.560%); 11 MiB/s, 183 objects/s recovering 2024-05-15T23:23:43.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:43 smithi088 ceph-mon[44699]: pgmap v1225: 17 pgs: 3 active+recovering, 14 active+clean; 3.3 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 7134/108746 objects misplaced (6.560%); 11 MiB/s, 183 objects/s recovering 2024-05-15T23:23:43.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:23:44.119 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:23:44.220 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:44.451 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:23:44.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:44.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:23:44.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:23:44.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:23:44.659 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:44.688 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:23:44.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:44.694 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:23:44.795 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:23:44.885 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:44.909 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:23:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:44.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:23:45.018 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:23:45.048 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:45.072 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:23:45.074 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:45.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:45.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:23:45.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:45.254 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:45.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:23:45.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:23:45.397 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:45.421 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:23:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:45.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:23:45.529 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:23:45.583 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:45.647 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:23:45.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:45.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:23:45.719 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:23:45.719 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:23:45.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:23:45.784 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:45.911 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:23:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:45.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:23:46.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:23:46.041 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:45 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:23:46.041 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:45 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:23:46.042 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:45 smithi084 ceph-mon[39784]: osdmap e454: 12 total, 12 up, 9 in 2024-05-15T23:23:46.042 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:45 smithi084 ceph-mon[39784]: pgmap v1227: 17 pgs: 3 active+recovering, 14 active+clean; 3.3 GiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 780 op/s; 7134/108198 objects misplaced (6.593%); 8.7 MiB/s, 138 objects/s recovering 2024-05-15T23:23:46.043 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:46.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:45 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:23:46.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:45 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:23:46.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:45 smithi090 ceph-mon[44434]: osdmap e454: 12 total, 12 up, 9 in 2024-05-15T23:23:46.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:45 smithi090 ceph-mon[44434]: pgmap v1227: 17 pgs: 3 active+recovering, 14 active+clean; 3.3 GiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 780 op/s; 7134/108198 objects misplaced (6.593%); 8.7 MiB/s, 138 objects/s recovering 2024-05-15T23:23:46.070 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:23:46.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:46.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:23:46.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:45 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:23:46.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:45 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:23:46.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:45 smithi088 ceph-mon[44699]: osdmap e454: 12 total, 12 up, 9 in 2024-05-15T23:23:46.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:45 smithi088 ceph-mon[44699]: pgmap v1227: 17 pgs: 3 active+recovering, 14 active+clean; 3.3 GiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 780 op/s; 7134/108198 objects misplaced (6.593%); 8.7 MiB/s, 138 objects/s recovering 2024-05-15T23:23:46.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:23:46.208 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:46.246 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:23:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:46.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:23:46.353 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:23:46.389 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:46.731 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:46 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:23:46.731 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:46 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:23:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:46 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:23:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:46 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:23:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:46 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:23:47.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:46 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:23:47.793 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:23:47.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:47.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:23:47.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:23:47.911 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T23:23:47.942 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:47.974 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:23:47.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:47.983 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:23:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:47 smithi090 ceph-mon[44434]: osdmap e455: 12 total, 12 up, 9 in 2024-05-15T23:23:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:47 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:23:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:47 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:23:48.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:47 smithi090 ceph-mon[44434]: pgmap v1229: 17 pgs: 3 active+recovering, 14 active+clean; 3.1 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 787 op/s; 6570/103196 objects misplaced (6.367%); 12 MiB/s, 185 objects/s recovering 2024-05-15T23:23:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:47 smithi088 ceph-mon[44699]: osdmap e455: 12 total, 12 up, 9 in 2024-05-15T23:23:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:47 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:23:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:47 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:23:48.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:47 smithi088 ceph-mon[44699]: pgmap v1229: 17 pgs: 3 active+recovering, 14 active+clean; 3.1 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 787 op/s; 6570/103196 objects misplaced (6.367%); 12 MiB/s, 185 objects/s recovering 2024-05-15T23:23:48.084 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:23:48.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:47 smithi084 ceph-mon[39784]: osdmap e455: 12 total, 12 up, 9 in 2024-05-15T23:23:48.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:47 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:23:48.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:47 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:23:48.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:47 smithi084 ceph-mon[39784]: pgmap v1229: 17 pgs: 3 active+recovering, 14 active+clean; 3.1 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 787 op/s; 6570/103196 objects misplaced (6.367%); 12 MiB/s, 185 objects/s recovering 2024-05-15T23:23:48.149 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:48.206 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:23:48.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:48.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:23:48.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:23:48.338 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:23:48.345 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:48.596 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:23:48.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:48.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:23:48.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:48.739 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:48.769 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:23:48.776 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:48.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:48.781 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:23:48.835 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:23:48.837 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:23:48.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:48 smithi084 ceph-mon[39784]: osdmap e456: 12 total, 12 up, 9 in 2024-05-15T23:23:48.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:48 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:23:48.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:48 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:23:48.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2898638731' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:23:48.881 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:23:48.911 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:48.998 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:48 smithi088 ceph-mon[44699]: osdmap e456: 12 total, 12 up, 9 in 2024-05-15T23:23:48.998 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:48 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:23:48.998 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:48 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:23:48.998 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2898638731' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:23:49.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:48 smithi090 ceph-mon[44434]: osdmap e456: 12 total, 12 up, 9 in 2024-05-15T23:23:49.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:48 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:23:49.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:48 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:23:49.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2898638731' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:23:49.962 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:23:49.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:49.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:23:50.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:49 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:23:50.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:49 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2898638731' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:23:50.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:49 smithi090 ceph-mon[44434]: osdmap e457: 12 total, 12 up, 9 in 2024-05-15T23:23:50.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:49 smithi090 ceph-mon[44434]: from='client.16257 v1:172.21.15.84:0/228139881' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:50.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:49 smithi090 ceph-mon[44434]: pgmap v1232: 17 pgs: 3 active+recovering, 14 active+clean; 3.0 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s; 6273/99874 objects misplaced (6.281%); 13 MiB/s, 200 objects/s recovering 2024-05-15T23:23:50.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:23:50.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:49 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:23:50.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:49 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2898638731' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:23:50.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:49 smithi088 ceph-mon[44699]: osdmap e457: 12 total, 12 up, 9 in 2024-05-15T23:23:50.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:49 smithi088 ceph-mon[44699]: from='client.16257 v1:172.21.15.84:0/228139881' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:50.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:49 smithi088 ceph-mon[44699]: pgmap v1232: 17 pgs: 3 active+recovering, 14 active+clean; 3.0 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s; 6273/99874 objects misplaced (6.281%); 13 MiB/s, 200 objects/s recovering 2024-05-15T23:23:50.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:49 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:23:50.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:49 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2898638731' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:23:50.080 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:49 smithi084 ceph-mon[39784]: osdmap e457: 12 total, 12 up, 9 in 2024-05-15T23:23:50.080 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:49 smithi084 ceph-mon[39784]: from='client.16257 v1:172.21.15.84:0/228139881' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:50.080 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:49 smithi084 ceph-mon[39784]: pgmap v1232: 17 pgs: 3 active+recovering, 14 active+clean; 3.0 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s; 6273/99874 objects misplaced (6.281%); 13 MiB/s, 200 objects/s recovering 2024-05-15T23:23:50.104 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:23:50.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:50.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:23:50.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:23:50.271 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:50.303 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:23:50.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:50.306 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:23:50.344 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:23:50.370 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:23:50.396 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:23:50.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:50.418 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:23:50.448 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:50.450 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:23:50.471 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:50.474 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:23:50.476 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:23:50.505 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:23:50.536 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:23:50.558 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:23:50.574 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:23:50.587 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:23:50.609 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:50.615 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:23:50.641 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:23:50.642 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:23:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:23:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:23:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:23:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:23:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:23:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:23:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:23:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:23:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:23:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:23:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:23:50.781 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:50 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:23:50.781 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:50 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:23:50.781 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:50 smithi090 ceph-mon[44434]: 5.5 scrub starts 2024-05-15T23:23:50.781 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:50 smithi090 ceph-mon[44434]: 5.1 scrub starts 2024-05-15T23:23:50.781 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2469282975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:50.781 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:50.781 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:50.801 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1231,"stamp":"2024-05-15T23:23:48.764381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3273065016,"num_objects":49937,"num_object_clones":0,"num_object_copies":99874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6273,"num_objects_unfound":0,"num_objects_dirty":49937,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":257304,"num_write_kb":9829799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19622,"num_bytes_recovered":1287140960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160775,"ondisk_log_size":160775,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":9463984,"kb_used_data":6062360,"kb_used_omap":14,"kb_used_meta":3401393,"kb_avail":834242384,"statfs":{"total":863955320832,"available":854264201216,"internally_reserved":0,"allocated":6207856640,"data_stored":6203727742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14358,"internal_metadata":3483027434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-290717696,"num_objects":-4436,"num_object_clones":0,"num_object_copies":-8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-861,"num_objects_unfound":0,"num_objects_dirty":-4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4436,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":861,"num_bytes_recovered":56426496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.756500"},"pg_stats":[{"pgid":"5.f","version":"455'15916","reported_seq":16087,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-05-15T23:23:47.620350+0000","last_change":"2024-05-15T23:19:47.305945+0000","last_active":"2024-05-15T23:23:47.620350+0000","last_peered":"2024-05-15T23:23:47.620350+0000","last_clean":"2024-05-15T23:23:47.620350+0000","last_became_active":"2024-05-15T23:19:47.305689+0000","last_became_peered":"2024-05-15T23:19:47.305689+0000","last_unstale":"2024-05-15T23:23:47.620350+0000","last_undegraded":"2024-05-15T23:23:47.620350+0000","last_fullsized":"2024-05-15T23:23:47.620350+0000","mapping_epoch":374,"log_start":"393'5900","ondisk_log_start":"393'5900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206045184,"num_objects":3144,"num_object_clones":0,"num_object_copies":6288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15916,"num_write_kb":609920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"454'15814","reported_seq":16159,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-05-15T23:23:45.543817+0000","last_change":"2024-05-15T23:22:23.637106+0000","last_active":"2024-05-15T23:23:45.543817+0000","last_peered":"2024-05-15T23:23:45.543817+0000","last_clean":"2024-05-15T23:23:45.543817+0000","last_became_active":"2024-05-15T23:19:47.325414+0000","last_became_peered":"2024-05-15T23:19:47.325414+0000","last_unstale":"2024-05-15T23:23:45.543817+0000","last_undegraded":"2024-05-15T23:23:45.543817+0000","last_fullsized":"2024-05-15T23:23:45.543817+0000","mapping_epoch":374,"log_start":"392'5800","ondisk_log_start":"392'5800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'11848","last_scrub_stamp":"2024-05-15T23:22:23.636778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:23.636778+0000","objects_scrubbed":7095,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203685888,"num_objects":3108,"num_object_clones":0,"num_object_copies":6216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15814,"num_write_kb":605504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"455'16391","reported_seq":16564,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-05-15T23:23:47.141881+0000","last_change":"2024-05-15T23:19:47.306160+0000","last_active":"2024-05-15T23:23:47.141881+0000","last_peered":"2024-05-15T23:23:47.141881+0000","last_clean":"2024-05-15T23:23:47.141881+0000","last_became_active":"2024-05-15T23:19:47.305860+0000","last_became_peered":"2024-05-15T23:19:47.305860+0000","last_unstale":"2024-05-15T23:23:47.141881+0000","last_undegraded":"2024-05-15T23:23:47.141881+0000","last_fullsized":"2024-05-15T23:23:47.141881+0000","mapping_epoch":374,"log_start":"394'6300","ondisk_log_start":"394'6300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200146968,"num_objects":3055,"num_object_clones":0,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16391,"num_write_kb":622209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"454'16156","reported_seq":16489,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-05-15T23:23:45.820323+0000","last_change":"2024-05-15T23:22:51.995640+0000","last_active":"2024-05-15T23:23:45.820323+0000","last_peered":"2024-05-15T23:23:45.820323+0000","last_clean":"2024-05-15T23:23:45.820323+0000","last_became_active":"2024-05-15T23:19:47.306168+0000","last_became_peered":"2024-05-15T23:19:47.306168+0000","last_unstale":"2024-05-15T23:23:45.820323+0000","last_undegraded":"2024-05-15T23:23:45.820323+0000","last_fullsized":"2024-05-15T23:23:45.820323+0000","mapping_epoch":374,"log_start":"393'6100","ondisk_log_start":"393'6100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'13335","last_scrub_stamp":"2024-05-15T23:22:51.995473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:51.995473+0000","objects_scrubbed":6018,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:13.168527+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207224832,"num_objects":3162,"num_object_clones":0,"num_object_copies":6324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16156,"num_write_kb":618176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"454'16011","reported_seq":16347,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-05-15T23:23:45.587688+0000","last_change":"2024-05-15T23:22:48.338084+0000","last_active":"2024-05-15T23:23:45.587688+0000","last_peered":"2024-05-15T23:23:45.587688+0000","last_clean":"2024-05-15T23:23:45.587688+0000","last_became_active":"2024-05-15T23:19:47.307061+0000","last_became_peered":"2024-05-15T23:19:47.307061+0000","last_unstale":"2024-05-15T23:23:45.587688+0000","last_undegraded":"2024-05-15T23:23:45.587688+0000","last_fullsized":"2024-05-15T23:23:45.587688+0000","mapping_epoch":374,"log_start":"393'6000","ondisk_log_start":"393'6000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'13172","last_scrub_stamp":"2024-05-15T23:22:48.337866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:48.337866+0000","objects_scrubbed":6060,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:11.906769+0000","scrub_duration":1844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208470016,"num_objects":3181,"num_object_clones":0,"num_object_copies":6362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16011,"num_write_kb":614144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"454'16087","reported_seq":16257,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-05-15T23:23:45.563417+0000","last_change":"2024-05-15T23:19:47.325657+0000","last_active":"2024-05-15T23:23:45.563417+0000","last_peered":"2024-05-15T23:23:45.563417+0000","last_clean":"2024-05-15T23:23:45.563417+0000","last_became_active":"2024-05-15T23:19:47.325371+0000","last_became_peered":"2024-05-15T23:19:47.325371+0000","last_unstale":"2024-05-15T23:23:45.563417+0000","last_undegraded":"2024-05-15T23:23:45.563417+0000","last_fullsized":"2024-05-15T23:23:45.563417+0000","mapping_epoch":374,"log_start":"393'6000","ondisk_log_start":"393'6000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204800000,"num_objects":3125,"num_object_clones":0,"num_object_copies":6250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16087,"num_write_kb":614784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"454'16079","reported_seq":29263,"reported_epoch":454,"state":"active+recovering","last_fresh":"2024-05-15T23:23:46.028054+0000","last_change":"2024-05-15T23:21:09.568311+0000","last_active":"2024-05-15T23:23:46.028054+0000","last_peered":"2024-05-15T23:23:46.028054+0000","last_clean":"2024-05-15T23:21:08.023957+0000","last_became_active":"2024-05-15T23:21:09.454639+0000","last_became_peered":"2024-05-15T23:21:09.454639+0000","last_unstale":"2024-05-15T23:23:46.028054+0000","last_undegraded":"2024-05-15T23:23:46.028054+0000","last_fullsized":"2024-05-15T23:23:46.028054+0000","mapping_epoch":407,"log_start":"393'6000","ondisk_log_start":"393'6000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201523200,"num_objects":3075,"num_object_clones":0,"num_object_copies":6150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2064,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16079,"num_write_kb":612928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6506,"num_bytes_recovered":426377216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":2064},{"shards":"7,11","objects":1011}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"455'16124","reported_seq":16296,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-05-15T23:23:47.105651+0000","last_change":"2024-05-15T23:19:47.304832+0000","last_active":"2024-05-15T23:23:47.105651+0000","last_peered":"2024-05-15T23:23:47.105651+0000","last_clean":"2024-05-15T23:23:47.105651+0000","last_became_active":"2024-05-15T23:19:47.304547+0000","last_became_peered":"2024-05-15T23:19:47.304547+0000","last_unstale":"2024-05-15T23:23:47.105651+0000","last_undegraded":"2024-05-15T23:23:47.105651+0000","last_fullsized":"2024-05-15T23:23:47.105651+0000","mapping_epoch":374,"log_start":"393'6100","ondisk_log_start":"393'6100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201850880,"num_objects":3080,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16124,"num_write_kb":614528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"455'16021","reported_seq":16193,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-05-15T23:23:47.620426+0000","last_change":"2024-05-15T23:19:47.306634+0000","last_active":"2024-05-15T23:23:47.620426+0000","last_peered":"2024-05-15T23:23:47.620426+0000","last_clean":"2024-05-15T23:23:47.620426+0000","last_became_active":"2024-05-15T23:19:47.306331+0000","last_became_peered":"2024-05-15T23:19:47.306331+0000","last_unstale":"2024-05-15T23:23:47.620426+0000","last_undegraded":"2024-05-15T23:23:47.620426+0000","last_fullsized":"2024-05-15T23:23:47.620426+0000","mapping_epoch":374,"log_start":"393'6000","ondisk_log_start":"393'6000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198901760,"num_objects":3035,"num_object_clones":0,"num_object_copies":6070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16021,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"455'16287","reported_seq":16458,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-05-15T23:23:47.624497+0000","last_change":"2024-05-15T23:19:47.306611+0000","last_active":"2024-05-15T23:23:47.624497+0000","last_peered":"2024-05-15T23:23:47.624497+0000","last_clean":"2024-05-15T23:23:47.624497+0000","last_became_active":"2024-05-15T23:19:47.306349+0000","last_became_peered":"2024-05-15T23:19:47.306349+0000","last_unstale":"2024-05-15T23:23:47.624497+0000","last_undegraded":"2024-05-15T23:23:47.624497+0000","last_fullsized":"2024-05-15T23:23:47.624497+0000","mapping_epoch":374,"log_start":"393'6200","ondisk_log_start":"393'6200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198115328,"num_objects":3023,"num_object_clones":0,"num_object_copies":6046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16287,"num_write_kb":617920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"454'16207","reported_seq":16607,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-05-15T23:23:45.792358+0000","last_change":"2024-05-15T23:22:48.067681+0000","last_active":"2024-05-15T23:23:45.792358+0000","last_peered":"2024-05-15T23:23:45.792358+0000","last_clean":"2024-05-15T23:23:45.792358+0000","last_became_active":"2024-05-15T23:19:47.305685+0000","last_became_peered":"2024-05-15T23:19:47.305685+0000","last_unstale":"2024-05-15T23:23:45.792358+0000","last_undegraded":"2024-05-15T23:23:45.792358+0000","last_fullsized":"2024-05-15T23:23:45.792358+0000","mapping_epoch":374,"log_start":"394'6200","ondisk_log_start":"394'6200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'13277","last_scrub_stamp":"2024-05-15T23:22:48.067507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:48.067507+0000","objects_scrubbed":6142,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:23:49.704509+0000","scrub_duration":929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208994304,"num_objects":3189,"num_object_clones":0,"num_object_copies":6378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16207,"num_write_kb":620672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"455'16205","reported_seq":29569,"reported_epoch":455,"state":"active+recovering","last_fresh":"2024-05-15T23:23:47.627306+0000","last_change":"2024-05-15T23:21:09.470461+0000","last_active":"2024-05-15T23:23:47.627306+0000","last_peered":"2024-05-15T23:23:47.627306+0000","last_clean":"2024-05-15T23:21:07.989871+0000","last_became_active":"2024-05-15T23:21:09.400731+0000","last_became_peered":"2024-05-15T23:21:09.400731+0000","last_unstale":"2024-05-15T23:23:47.627306+0000","last_undegraded":"2024-05-15T23:23:47.627306+0000","last_fullsized":"2024-05-15T23:23:47.627306+0000","mapping_epoch":407,"log_start":"394'6200","ondisk_log_start":"394'6200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202571776,"num_objects":3091,"num_object_clones":0,"num_object_copies":6182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2095,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16205,"num_write_kb":617472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6595,"num_bytes_recovered":432209920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"3,5","objects":996},{"shards":"5,6","objects":2095}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"454'16099","reported_seq":29302,"reported_epoch":454,"state":"active+recovering","last_fresh":"2024-05-15T23:23:45.593932+0000","last_change":"2024-05-15T23:21:09.275620+0000","last_active":"2024-05-15T23:23:45.593932+0000","last_peered":"2024-05-15T23:23:45.593932+0000","last_clean":"2024-05-15T23:21:07.993692+0000","last_became_active":"2024-05-15T23:21:09.221797+0000","last_became_peered":"2024-05-15T23:21:09.221797+0000","last_unstale":"2024-05-15T23:23:45.593932+0000","last_undegraded":"2024-05-15T23:23:45.593932+0000","last_fullsized":"2024-05-15T23:23:45.593932+0000","mapping_epoch":407,"log_start":"392'6000","ondisk_log_start":"392'6000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201261056,"num_objects":3071,"num_object_clones":0,"num_object_copies":6142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2114,"num_objects_unfound":0,"num_objects_dirty":3071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16099,"num_write_kb":613440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6515,"num_bytes_recovered":426967040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":2114},{"shards":"7,10","objects":957}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.1","version":"454'15537","reported_seq":15880,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-05-15T23:23:44.826147+0000","last_change":"2024-05-15T23:22:19.625857+0000","last_active":"2024-05-15T23:23:44.826147+0000","last_peered":"2024-05-15T23:23:44.826147+0000","last_clean":"2024-05-15T23:23:44.826147+0000","last_became_active":"2024-05-15T23:19:47.516865+0000","last_became_peered":"2024-05-15T23:19:47.516865+0000","last_unstale":"2024-05-15T23:23:44.826147+0000","last_undegraded":"2024-05-15T23:23:44.826147+0000","last_fullsized":"2024-05-15T23:23:44.826147+0000","mapping_epoch":374,"log_start":"392'5500","ondisk_log_start":"392'5500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"426'11474","last_scrub_stamp":"2024-05-15T23:22:19.625719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:19.625719+0000","objects_scrubbed":7211,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:23:49.523745+0000","scrub_duration":1229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204537856,"num_objects":3121,"num_object_clones":0,"num_object_copies":6242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15537,"num_write_kb":597056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"454'16109","reported_seq":16279,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-05-15T23:23:45.563478+0000","last_change":"2024-05-15T23:19:47.305723+0000","last_active":"2024-05-15T23:23:45.563478+0000","last_peered":"2024-05-15T23:23:45.563478+0000","last_clean":"2024-05-15T23:23:45.563478+0000","last_became_active":"2024-05-15T23:19:47.305459+0000","last_became_peered":"2024-05-15T23:19:47.305459+0000","last_unstale":"2024-05-15T23:23:45.563478+0000","last_undegraded":"2024-05-15T23:23:45.563478+0000","last_fullsized":"2024-05-15T23:23:45.563478+0000","mapping_epoch":374,"log_start":"393'6100","ondisk_log_start":"393'6100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207552512,"num_objects":3167,"num_object_clones":0,"num_object_copies":6334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16109,"num_write_kb":616832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"454'16138","reported_seq":16308,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-05-15T23:23:45.542695+0000","last_change":"2024-05-15T23:19:47.307499+0000","last_active":"2024-05-15T23:23:45.542695+0000","last_peered":"2024-05-15T23:23:45.542695+0000","last_clean":"2024-05-15T23:23:45.542695+0000","last_became_active":"2024-05-15T23:19:47.307149+0000","last_became_peered":"2024-05-15T23:19:47.307149+0000","last_unstale":"2024-05-15T23:23:45.542695+0000","last_undegraded":"2024-05-15T23:23:45.542695+0000","last_fullsized":"2024-05-15T23:23:45.542695+0000","mapping_epoch":374,"log_start":"393'6100","ondisk_log_start":"393'6100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216793088,"num_objects":3308,"num_object_clones":0,"num_object_copies":6616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16138,"num_write_kb":622272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":947,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-05-15T23:23:44.708016+0000","last_change":"2024-05-15T23:20:05.905241+0000","last_active":"2024-05-15T23:23:44.708016+0000","last_peered":"2024-05-15T23:23:44.708016+0000","last_clean":"2024-05-15T23:23:44.708016+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:23:44.708016+0000","last_undegraded":"2024-05-15T23:23:44.708016+0000","last_fullsized":"2024-05-15T23:23:44.708016+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:20:05.905105+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:20:05.905105+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3272474648,"num_objects":49935,"num_object_clones":0,"num_object_copies":99870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6273,"num_objects_unfound":0,"num_objects_dirty":49935,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":257181,"num_write_kb":9827649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19616,"num_bytes_recovered":1285554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7846043648,"data_stored":7846035504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160681,"ondisk_log_size":160681,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351216,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":996900,"kb_used_data":460376,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92748252,"statfs":{"total":95995035648,"available":94974210048,"internally_reserved":0,"allocated":471425024,"data_stored":470954306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":549386692},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580149,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1317068,"kb_used_data":1024896,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92428084,"statfs":{"total":95995035648,"available":94646358016,"internally_reserved":0,"allocated":1049493504,"data_stored":1049036434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":299170240},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776381,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":779272,"kb_used_data":405184,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92965880,"statfs":{"total":95995035648,"available":95197061120,"internally_reserved":0,"allocated":414908416,"data_stored":414451346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":383056320},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005313,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":583228,"kb_used_data":411828,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93161924,"statfs":{"total":95995035648,"available":95397810176,"internally_reserved":0,"allocated":421711872,"data_stored":421260806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266949,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":801936,"kb_used_data":459784,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92943216,"statfs":{"total":95995035648,"available":95173853184,"internally_reserved":0,"allocated":470818816,"data_stored":470358194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":350353866},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495884,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575141,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1314380,"kb_used_data":818112,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92430772,"statfs":{"total":95995035648,"available":94649110528,"internally_reserved":0,"allocated":837746688,"data_stored":837289618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":508164550},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016550,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483239,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149980,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":671948,"kb_used_data":261760,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93073204,"statfs":{"total":95995035648,"available":95306960896,"internally_reserved":0,"allocated":268042240,"data_stored":267585170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":420018631},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986452,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1227068,"kb_used_data":830772,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92518084,"statfs":{"total":95995035648,"available":94738518016,"internally_reserved":0,"allocated":850710528,"data_stored":850259462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790225,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1772184,"kb_used_data":1389648,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":91972968,"statfs":{"total":95995035648,"available":94180319232,"internally_reserved":0,"allocated":1422999552,"data_stored":1422532406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":828702720,"data_stored":828702720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265027584,"data_stored":265027584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":840105984,"data_stored":840105984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1413353472,"data_stored":1413349400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1696661504,"data_stored":1696661504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":468582400,"data_stored":468582400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":415694848,"data_stored":415694848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":412024832,"data_stored":412024832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1037828096,"data_stored":1037828096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":468062208,"data_stored":468058136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:23:50.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:23:50.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:50 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:23:50.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:50 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:23:50.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:50 smithi084 ceph-mon[39784]: 5.5 scrub starts 2024-05-15T23:23:50.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:50 smithi084 ceph-mon[39784]: 5.1 scrub starts 2024-05-15T23:23:50.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2469282975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:50.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:50.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:50.964 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:23:50.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:50.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:23:51.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:23:51.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:50 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:23:51.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:50 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:23:51.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:50 smithi088 ceph-mon[44699]: 5.5 scrub starts 2024-05-15T23:23:51.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:50 smithi088 ceph-mon[44699]: 5.1 scrub starts 2024-05-15T23:23:51.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2469282975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:51.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:51.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:51.118 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:51.151 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:23:51.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:51.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:23:51.257 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:23:51.317 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:51.370 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:23:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:51.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:23:51.477 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:51.530 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:51.562 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:51.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:51.568 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:23:51.670 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:23:51.718 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:23:51.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:51.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:23:51.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:23:51.965 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:51.972 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:51.972 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:51.972 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:51.972 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:51.972 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:51.972 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:51.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:51.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:51.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:51.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:51.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:51 smithi088 ceph-mon[44699]: osdmap e458: 12 total, 12 up, 9 in 2024-05-15T23:23:51.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:51 smithi088 ceph-mon[44699]: 5.1 scrub ok 2024-05-15T23:23:51.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:51 smithi088 ceph-mon[44699]: 5.5 scrub ok 2024-05-15T23:23:51.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:51 smithi088 ceph-mon[44699]: pgmap v1234: 17 pgs: 3 active+recovering, 14 active+clean; 2.9 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 976 op/s; 5964/94840 objects misplaced (6.288%); 8.9 MiB/s, 141 objects/s recovering 2024-05-15T23:23:51.986 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:23:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:51.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:23:52.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:51 smithi090 ceph-mon[44434]: osdmap e458: 12 total, 12 up, 9 in 2024-05-15T23:23:52.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:51 smithi090 ceph-mon[44434]: 5.1 scrub ok 2024-05-15T23:23:52.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:51 smithi090 ceph-mon[44434]: 5.5 scrub ok 2024-05-15T23:23:52.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:51 smithi090 ceph-mon[44434]: pgmap v1234: 17 pgs: 3 active+recovering, 14 active+clean; 2.9 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 976 op/s; 5964/94840 objects misplaced (6.288%); 8.9 MiB/s, 141 objects/s recovering 2024-05-15T23:23:52.088 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:52.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:23:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:51 smithi084 ceph-mon[39784]: osdmap e458: 12 total, 12 up, 9 in 2024-05-15T23:23:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:51 smithi084 ceph-mon[39784]: 5.1 scrub ok 2024-05-15T23:23:52.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:51 smithi084 ceph-mon[39784]: 5.5 scrub ok 2024-05-15T23:23:52.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:51 smithi084 ceph-mon[39784]: pgmap v1234: 17 pgs: 3 active+recovering, 14 active+clean; 2.9 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 976 op/s; 5964/94840 objects misplaced (6.288%); 8.9 MiB/s, 141 objects/s recovering 2024-05-15T23:23:52.134 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:52.170 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:23:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:52.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:23:52.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:23:52.338 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:52.533 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:23:52.533 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:23:52.533 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:23:52.534 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:23:52.534 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.534 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:135069 4.a 4:54e24872:::benchmark_data_smithi090_104448_object135068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e306)", 2024-05-15T23:23:52.534 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:16:04.604696+0000", 2024-05-15T23:23:52.534 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 467.924063655, 2024-05-15T23:23:52.534 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.215105308, 2024-05-15T23:23:52.534 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.534 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.534 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.534 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.534 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.534 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.534 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 135069 2024-05-15T23:23:52.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.535 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.535 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.535 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604696+0000", 2024-05-15T23:23:52.535 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.535 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.535 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604696+0000", 2024-05-15T23:23:52.535 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.535 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604699+0000", 2024-05-15T23:23:52.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.514e-06 2024-05-15T23:23:52.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.536 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604753+0000", 2024-05-15T23:23:52.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4558999999999997e-05 2024-05-15T23:23:52.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.536 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604765+0000", 2024-05-15T23:23:52.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2041999999999999e-05 2024-05-15T23:23:52.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.537 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.537 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604773+0000", 2024-05-15T23:23:52.537 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4259999999999997e-06 2024-05-15T23:23:52.537 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.537 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.537 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.537 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604817+0000", 2024-05-15T23:23:52.537 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4700999999999998e-05 2024-05-15T23:23:52.537 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.537 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.537 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.537 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.604918+0000", 2024-05-15T23:23:52.537 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00010052000000000001 2024-05-15T23:23:52.537 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.537 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.605016+0000", 2024-05-15T23:23:52.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8197999999999997e-05 2024-05-15T23:23:52.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.609170+0000", 2024-05-15T23:23:52.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0041537689999999999 2024-05-15T23:23:52.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.819738+0000", 2024-05-15T23:23:52.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21056836700000001 2024-05-15T23:23:52.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.819773+0000", 2024-05-15T23:23:52.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5132999999999997e-05 2024-05-15T23:23:52.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.819801+0000", 2024-05-15T23:23:52.540 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8079e-05 2024-05-15T23:23:52.540 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.540 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.540 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.540 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:135097 4.a 4:5c1029ec:::benchmark_data_smithi090_104448_object135096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e306)", 2024-05-15T23:23:52.540 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:16:04.626796+0000", 2024-05-15T23:23:52.540 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 467.90196357600001, 2024-05-15T23:23:52.540 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19805182700000001, 2024-05-15T23:23:52.540 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.540 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.540 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.540 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 135097 2024-05-15T23:23:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.626796+0000", 2024-05-15T23:23:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.626796+0000", 2024-05-15T23:23:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.626800+0000", 2024-05-15T23:23:52.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7179999999999998e-06 2024-05-15T23:23:52.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.626842+0000", 2024-05-15T23:23:52.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2415e-05 2024-05-15T23:23:52.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.626858+0000", 2024-05-15T23:23:52.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6186000000000001e-05 2024-05-15T23:23:52.543 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.626866+0000", 2024-05-15T23:23:52.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0150000000000003e-06 2024-05-15T23:23:52.543 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.626921+0000", 2024-05-15T23:23:52.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4092000000000003e-05 2024-05-15T23:23:52.543 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.627016+0000", 2024-05-15T23:23:52.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5414000000000005e-05 2024-05-15T23:23:52.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.544 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.627110+0000", 2024-05-15T23:23:52.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4067000000000006e-05 2024-05-15T23:23:52.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.544 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.631312+0000", 2024-05-15T23:23:52.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042020879999999997 2024-05-15T23:23:52.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.544 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.544 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.824777+0000", 2024-05-15T23:23:52.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19346450400000001 2024-05-15T23:23:52.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.824811+0000", 2024-05-15T23:23:52.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4106000000000002e-05 2024-05-15T23:23:52.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.824848+0000", 2024-05-15T23:23:52.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7221999999999997e-05 2024-05-15T23:23:52.545 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.545 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.545 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.546 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:135103 4.a 4:59f15f3d:::benchmark_data_smithi090_104448_object135102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e306)", 2024-05-15T23:23:52.546 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:16:04.632612+0000", 2024-05-15T23:23:52.546 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 467.89614772099998, 2024-05-15T23:23:52.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.192344814, 2024-05-15T23:23:52.546 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.546 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.546 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.546 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.546 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.546 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.546 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 135103 2024-05-15T23:23:52.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.546 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.547 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632612+0000", 2024-05-15T23:23:52.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.547 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632612+0000", 2024-05-15T23:23:52.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.547 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632614+0000", 2024-05-15T23:23:52.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2050000000000001e-06 2024-05-15T23:23:52.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632653+0000", 2024-05-15T23:23:52.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8399000000000003e-05 2024-05-15T23:23:52.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632668+0000", 2024-05-15T23:23:52.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5554999999999999e-05 2024-05-15T23:23:52.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632679+0000", 2024-05-15T23:23:52.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0550999999999999e-05 2024-05-15T23:23:52.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.549 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632737+0000", 2024-05-15T23:23:52.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8569000000000001e-05 2024-05-15T23:23:52.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.549 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.632860+0000", 2024-05-15T23:23:52.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012253599999999999 2024-05-15T23:23:52.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.550 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.550 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.633022+0000", 2024-05-15T23:23:52.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016184899999999999 2024-05-15T23:23:52.550 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.550 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.550 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.637320+0000", 2024-05-15T23:23:52.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042986190000000001 2024-05-15T23:23:52.550 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.824896+0000", 2024-05-15T23:23:52.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18757555200000001 2024-05-15T23:23:52.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.824930+0000", 2024-05-15T23:23:52.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4551999999999997e-05 2024-05-15T23:23:52.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.551 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.551 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.824957+0000", 2024-05-15T23:23:52.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6427e-05 2024-05-15T23:23:52.551 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.552 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.552 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.552 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:135124 4.a 4:5c4f84a3:::benchmark_data_smithi090_104448_object135123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e306)", 2024-05-15T23:23:52.552 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:16:04.645139+0000", 2024-05-15T23:23:52.552 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 467.88362068399999, 2024-05-15T23:23:52.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17994866500000001, 2024-05-15T23:23:52.552 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.552 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.552 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.552 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.552 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.552 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.553 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 135124 2024-05-15T23:23:52.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.553 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645139+0000", 2024-05-15T23:23:52.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645139+0000", 2024-05-15T23:23:52.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645140+0000", 2024-05-15T23:23:52.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.1999999999999998e-07 2024-05-15T23:23:52.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645233+0000", 2024-05-15T23:23:52.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2946000000000001e-05 2024-05-15T23:23:52.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645240+0000", 2024-05-15T23:23:52.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2200000000000003e-06 2024-05-15T23:23:52.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645246+0000", 2024-05-15T23:23:52.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0270000000000001e-06 2024-05-15T23:23:52.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645295+0000", 2024-05-15T23:23:52.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9301000000000001e-05 2024-05-15T23:23:52.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645412+0000", 2024-05-15T23:23:52.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011633699999999999 2024-05-15T23:23:52.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.645553+0000", 2024-05-15T23:23:52.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014087799999999999 2024-05-15T23:23:52.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.649870+0000", 2024-05-15T23:23:52.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0043172430000000001 2024-05-15T23:23:52.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.824997+0000", 2024-05-15T23:23:52.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17512725000000001 2024-05-15T23:23:52.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.825049+0000", 2024-05-15T23:23:52.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2145999999999998e-05 2024-05-15T23:23:52.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:16:04.825088+0000", 2024-05-15T23:23:52.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8396999999999998e-05 2024-05-15T23:23:52.557 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.557 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.557 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.558 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304280 4.a 4:5b65c4c5:::benchmark_data_smithi090_104448_object131042:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:23:52.558 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.815719+0000", 2024-05-15T23:23:52.558 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 326.71304069500002, 2024-05-15T23:23:52.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.213512642, 2024-05-15T23:23:52.558 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.558 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.558 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.558 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.558 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.558 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.558 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304280 2024-05-15T23:23:52.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.558 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815719+0000", 2024-05-15T23:23:52.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815719+0000", 2024-05-15T23:23:52.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815721+0000", 2024-05-15T23:23:52.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0279999999999999e-06 2024-05-15T23:23:52.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815728+0000", 2024-05-15T23:23:52.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7270000000000001e-06 2024-05-15T23:23:52.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815737+0000", 2024-05-15T23:23:52.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6150000000000003e-06 2024-05-15T23:23:52.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815744+0000", 2024-05-15T23:23:52.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2589999999999997e-06 2024-05-15T23:23:52.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815805+0000", 2024-05-15T23:23:52.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1169999999999999e-05 2024-05-15T23:23:52.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815916+0000", 2024-05-15T23:23:52.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011129300000000001 2024-05-15T23:23:52.562 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.562 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.562 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816037+0000", 2024-05-15T23:23:52.562 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000120598 2024-05-15T23:23:52.562 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.562 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.562 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.824204+0000", 2024-05-15T23:23:52.562 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008167545 2024-05-15T23:23:52.562 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.562 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.562 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.029121+0000", 2024-05-15T23:23:52.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20491626399999999 2024-05-15T23:23:52.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.563 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.029181+0000", 2024-05-15T23:23:52.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0053999999999998e-05 2024-05-15T23:23:52.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.563 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.029232+0000", 2024-05-15T23:23:52.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1088999999999998e-05 2024-05-15T23:23:52.563 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.563 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.563 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.564 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304283 4.a 4:58e8bfc1:::benchmark_data_smithi090_104448_object131045:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:23:52.564 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.816913+0000", 2024-05-15T23:23:52.564 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 326.71184714700001, 2024-05-15T23:23:52.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22379715, 2024-05-15T23:23:52.564 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.564 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.564 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.564 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.564 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.564 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.564 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304283 2024-05-15T23:23:52.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816913+0000", 2024-05-15T23:23:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816913+0000", 2024-05-15T23:23:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816914+0000", 2024-05-15T23:23:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.519e-06 2024-05-15T23:23:52.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816919+0000", 2024-05-15T23:23:52.566 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.977e-06 2024-05-15T23:23:52.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816928+0000", 2024-05-15T23:23:52.566 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0310000000000004e-06 2024-05-15T23:23:52.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816935+0000", 2024-05-15T23:23:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3610000000000003e-06 2024-05-15T23:23:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.819984+0000", 2024-05-15T23:23:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030482909999999998 2024-05-15T23:23:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.820096+0000", 2024-05-15T23:23:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000112496 2024-05-15T23:23:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.820173+0000", 2024-05-15T23:23:52.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7179000000000001e-05 2024-05-15T23:23:52.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.568 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.568 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.825431+0000", 2024-05-15T23:23:52.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0052575199999999999 2024-05-15T23:23:52.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.568 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.568 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.040648+0000", 2024-05-15T23:23:52.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21521739400000001 2024-05-15T23:23:52.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.568 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.040681+0000", 2024-05-15T23:23:52.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2372000000000001e-05 2024-05-15T23:23:52.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.040710+0000", 2024-05-15T23:23:52.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9009999999999998e-05 2024-05-15T23:23:52.569 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.569 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.569 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.569 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304296 4.a 4:59aa04f1:::benchmark_data_smithi090_104448_object131058:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:23:52.569 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.828982+0000", 2024-05-15T23:23:52.569 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 326.699777617, 2024-05-15T23:23:52.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22203131700000001, 2024-05-15T23:23:52.570 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.570 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.570 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.570 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.570 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.570 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.570 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304296 2024-05-15T23:23:52.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.570 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.828982+0000", 2024-05-15T23:23:52.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.828982+0000", 2024-05-15T23:23:52.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.828984+0000", 2024-05-15T23:23:52.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.243e-06 2024-05-15T23:23:52.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.571 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.571 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.828992+0000", 2024-05-15T23:23:52.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4479999999999997e-06 2024-05-15T23:23:52.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.829004+0000", 2024-05-15T23:23:52.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1747e-05 2024-05-15T23:23:52.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.829012+0000", 2024-05-15T23:23:52.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.9590000000000001e-06 2024-05-15T23:23:52.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.829077+0000", 2024-05-15T23:23:52.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4916e-05 2024-05-15T23:23:52.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.829221+0000", 2024-05-15T23:23:52.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014339600000000001 2024-05-15T23:23:52.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.829352+0000", 2024-05-15T23:23:52.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000130996 2024-05-15T23:23:52.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.834093+0000", 2024-05-15T23:23:52.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0047415189999999996 2024-05-15T23:23:52.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.574 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.574 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.050919+0000", 2024-05-15T23:23:52.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21682558900000001 2024-05-15T23:23:52.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.574 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.574 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.050970+0000", 2024-05-15T23:23:52.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0618000000000002e-05 2024-05-15T23:23:52.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.574 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051013+0000", 2024-05-15T23:23:52.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3886e-05 2024-05-15T23:23:52.575 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.575 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.575 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.575 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.575 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304319 4.a 4:5860cfdb:::benchmark_data_smithi090_104448_object131081:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:23:52.575 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.849831+0000", 2024-05-15T23:23:52.575 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 326.678928574, 2024-05-15T23:23:52.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20133589099999999, 2024-05-15T23:23:52.575 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.575 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.575 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.576 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.576 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.576 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.576 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304319 2024-05-15T23:23:52.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.576 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849831+0000", 2024-05-15T23:23:52.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849831+0000", 2024-05-15T23:23:52.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849834+0000", 2024-05-15T23:23:52.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.875e-06 2024-05-15T23:23:52.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849842+0000", 2024-05-15T23:23:52.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.4659999999999999e-06 2024-05-15T23:23:52.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849858+0000", 2024-05-15T23:23:52.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5536000000000002e-05 2024-05-15T23:23:52.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.578 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.578 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849868+0000", 2024-05-15T23:23:52.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7610000000000002e-06 2024-05-15T23:23:52.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.578 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.578 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849918+0000", 2024-05-15T23:23:52.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0219000000000001e-05 2024-05-15T23:23:52.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.578 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.579 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.850026+0000", 2024-05-15T23:23:52.579 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000108256 2024-05-15T23:23:52.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.579 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.850126+0000", 2024-05-15T23:23:52.579 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9479000000000001e-05 2024-05-15T23:23:52.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.579 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.854202+0000", 2024-05-15T23:23:52.579 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004076774 2024-05-15T23:23:52.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051073+0000", 2024-05-15T23:23:52.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19687078499999999 2024-05-15T23:23:52.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051124+0000", 2024-05-15T23:23:52.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0436999999999998e-05 2024-05-15T23:23:52.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051167+0000", 2024-05-15T23:23:52.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3303000000000002e-05 2024-05-15T23:23:52.580 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.581 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.581 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.581 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304341 4.a 4:53ff7ba8:::benchmark_data_smithi090_104448_object131103:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:23:52.581 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.865333+0000", 2024-05-15T23:23:52.581 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 326.66342642500001, 2024-05-15T23:23:52.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18599020299999999, 2024-05-15T23:23:52.581 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.581 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.581 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.581 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.581 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.581 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.582 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304341 2024-05-15T23:23:52.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.582 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865333+0000", 2024-05-15T23:23:52.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865333+0000", 2024-05-15T23:23:52.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865334+0000", 2024-05-15T23:23:52.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0999999999999998e-07 2024-05-15T23:23:52.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865337+0000", 2024-05-15T23:23:52.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7470000000000001e-06 2024-05-15T23:23:52.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865341+0000", 2024-05-15T23:23:52.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4789999999999998e-06 2024-05-15T23:23:52.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.584 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.584 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865344+0000", 2024-05-15T23:23:52.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1030000000000002e-06 2024-05-15T23:23:52.584 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.584 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.584 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.584 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865395+0000", 2024-05-15T23:23:52.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0340999999999998e-05 2024-05-15T23:23:52.584 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.584 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.584 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.584 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865511+0000", 2024-05-15T23:23:52.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000116851 2024-05-15T23:23:52.584 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.584 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.585 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.585 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865618+0000", 2024-05-15T23:23:52.585 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000106659 2024-05-15T23:23:52.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.585 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.585 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.585 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.870311+0000", 2024-05-15T23:23:52.585 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0046930080000000002 2024-05-15T23:23:52.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.585 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.585 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.585 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051226+0000", 2024-05-15T23:23:52.585 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.180914669 2024-05-15T23:23:52.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.585 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051279+0000", 2024-05-15T23:23:52.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3637999999999997e-05 2024-05-15T23:23:52.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051323+0000", 2024-05-15T23:23:52.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4097999999999997e-05 2024-05-15T23:23:52.586 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.586 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.586 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.586 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304347 4.a 4:5813fd51:::benchmark_data_smithi090_104448_object131109:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:23:52.587 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.873270+0000", 2024-05-15T23:23:52.587 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 326.655490133, 2024-05-15T23:23:52.587 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17820270799999999, 2024-05-15T23:23:52.587 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.587 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.587 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.587 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.587 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.587 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.587 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304347 2024-05-15T23:23:52.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.587 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.587 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.587 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873270+0000", 2024-05-15T23:23:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873270+0000", 2024-05-15T23:23:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873270+0000", 2024-05-15T23:23:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6700000000000003e-07 2024-05-15T23:23:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.589 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.589 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873274+0000", 2024-05-15T23:23:52.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5300000000000001e-06 2024-05-15T23:23:52.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.589 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.589 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873281+0000", 2024-05-15T23:23:52.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3300000000000001e-06 2024-05-15T23:23:52.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.589 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.589 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873288+0000", 2024-05-15T23:23:52.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8569999999999997e-06 2024-05-15T23:23:52.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.590 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873335+0000", 2024-05-15T23:23:52.590 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7434e-05 2024-05-15T23:23:52.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.590 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.590 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873420+0000", 2024-05-15T23:23:52.590 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5000000000000006e-05 2024-05-15T23:23:52.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.590 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.590 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873494+0000", 2024-05-15T23:23:52.590 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3901999999999998e-05 2024-05-15T23:23:52.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882687+0000", 2024-05-15T23:23:52.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091921250000000006 2024-05-15T23:23:52.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051380+0000", 2024-05-15T23:23:52.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168693808 2024-05-15T23:23:52.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051431+0000", 2024-05-15T23:23:52.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1150000000000003e-05 2024-05-15T23:23:52.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.592 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.592 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051472+0000", 2024-05-15T23:23:52.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0805e-05 2024-05-15T23:23:52.592 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.592 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.592 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.592 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.592 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304351 4.a 4:582c10c7:::benchmark_data_smithi090_104448_object131113:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:23:52.592 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.874105+0000", 2024-05-15T23:23:52.592 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 326.65465496299998, 2024-05-15T23:23:52.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18030819300000001, 2024-05-15T23:23:52.593 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.593 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.593 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.593 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.593 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.593 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.593 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304351 2024-05-15T23:23:52.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.593 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.593 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874105+0000", 2024-05-15T23:23:52.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874105+0000", 2024-05-15T23:23:52.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874106+0000", 2024-05-15T23:23:52.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2950000000000001e-06 2024-05-15T23:23:52.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874111+0000", 2024-05-15T23:23:52.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8080000000000004e-06 2024-05-15T23:23:52.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.595 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874126+0000", 2024-05-15T23:23:52.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4915e-05 2024-05-15T23:23:52.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.595 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874133+0000", 2024-05-15T23:23:52.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4760000000000001e-06 2024-05-15T23:23:52.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.595 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.877862+0000", 2024-05-15T23:23:52.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037288489999999998 2024-05-15T23:23:52.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.877973+0000", 2024-05-15T23:23:52.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011109599999999999 2024-05-15T23:23:52.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.878075+0000", 2024-05-15T23:23:52.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000102221 2024-05-15T23:23:52.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882814+0000", 2024-05-15T23:23:52.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004738669 2024-05-15T23:23:52.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054322+0000", 2024-05-15T23:23:52.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171508251 2024-05-15T23:23:52.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054373+0000", 2024-05-15T23:23:52.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0822999999999998e-05 2024-05-15T23:23:52.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054413+0000", 2024-05-15T23:23:52.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9789999999999997e-05 2024-05-15T23:23:52.598 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.598 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.598 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.598 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304363 4.a 4:571d2066:::benchmark_data_smithi090_104448_object131125:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:23:52.598 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.882815+0000", 2024-05-15T23:23:52.598 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 326.645944321, 2024-05-15T23:23:52.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17174726100000001, 2024-05-15T23:23:52.598 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.598 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.599 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.599 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.599 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.599 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.599 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304363 2024-05-15T23:23:52.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.599 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.599 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.599 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882815+0000", 2024-05-15T23:23:52.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.599 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882815+0000", 2024-05-15T23:23:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882817+0000", 2024-05-15T23:23:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8950000000000001e-06 2024-05-15T23:23:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882823+0000", 2024-05-15T23:23:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3410000000000002e-06 2024-05-15T23:23:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882835+0000", 2024-05-15T23:23:52.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2853999999999999e-05 2024-05-15T23:23:52.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882848+0000", 2024-05-15T23:23:52.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2695e-05 2024-05-15T23:23:52.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882894+0000", 2024-05-15T23:23:52.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5383999999999998e-05 2024-05-15T23:23:52.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883006+0000", 2024-05-15T23:23:52.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011286299999999999 2024-05-15T23:23:52.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883119+0000", 2024-05-15T23:23:52.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011275199999999999 2024-05-15T23:23:52.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.891222+0000", 2024-05-15T23:23:52.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0081027779999999997 2024-05-15T23:23:52.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054470+0000", 2024-05-15T23:23:52.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.163247948 2024-05-15T23:23:52.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054521+0000", 2024-05-15T23:23:52.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0760999999999997e-05 2024-05-15T23:23:52.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054563+0000", 2024-05-15T23:23:52.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1990000000000003e-05 2024-05-15T23:23:52.603 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.603 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.604 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.604 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304365 4.a 4:5a74db35:::benchmark_data_smithi090_104448_object131127:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:23:52.604 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.883969+0000", 2024-05-15T23:23:52.604 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 326.644790554, 2024-05-15T23:23:52.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17074013900000001, 2024-05-15T23:23:52.604 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.604 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.604 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.604 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.604 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.604 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.604 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304365 2024-05-15T23:23:52.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.605 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.605 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.605 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883969+0000", 2024-05-15T23:23:52.605 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.605 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.605 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.605 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883969+0000", 2024-05-15T23:23:52.605 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.605 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.605 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.606 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883971+0000", 2024-05-15T23:23:52.606 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.567e-06 2024-05-15T23:23:52.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.606 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.606 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.606 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883976+0000", 2024-05-15T23:23:52.606 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0209999999999999e-06 2024-05-15T23:23:52.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.606 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.606 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.606 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883988+0000", 2024-05-15T23:23:52.606 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.256e-05 2024-05-15T23:23:52.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.607 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.607 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883996+0000", 2024-05-15T23:23:52.607 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8779999999999998e-06 2024-05-15T23:23:52.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.607 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.607 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.887685+0000", 2024-05-15T23:23:52.607 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036892100000000001 2024-05-15T23:23:52.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.887768+0000", 2024-05-15T23:23:52.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2952000000000001e-05 2024-05-15T23:23:52.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.887841+0000", 2024-05-15T23:23:52.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2880999999999996e-05 2024-05-15T23:23:52.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.891325+0000", 2024-05-15T23:23:52.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034839340000000002 2024-05-15T23:23:52.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054613+0000", 2024-05-15T23:23:52.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.163287922 2024-05-15T23:23:52.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054668+0000", 2024-05-15T23:23:52.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4787999999999998e-05 2024-05-15T23:23:52.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054709+0000", 2024-05-15T23:23:52.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1425999999999999e-05 2024-05-15T23:23:52.610 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.610 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.610 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.610 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304372 4.a 4:519d11bc:::benchmark_data_smithi090_104448_object131134:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:23:52.610 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.892294+0000", 2024-05-15T23:23:52.610 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 326.63646525299998, 2024-05-15T23:23:52.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17457294600000001, 2024-05-15T23:23:52.610 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.610 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.610 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.610 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.611 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.611 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.611 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304372 2024-05-15T23:23:52.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.611 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.611 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.611 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892294+0000", 2024-05-15T23:23:52.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.612 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.612 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892294+0000", 2024-05-15T23:23:52.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.612 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.612 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892296+0000", 2024-05-15T23:23:52.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8479999999999999e-06 2024-05-15T23:23:52.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892303+0000", 2024-05-15T23:23:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3899999999999998e-06 2024-05-15T23:23:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892313+0000", 2024-05-15T23:23:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9599999999999995e-06 2024-05-15T23:23:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892320+0000", 2024-05-15T23:23:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0419999999999997e-06 2024-05-15T23:23:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892359+0000", 2024-05-15T23:23:52.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9001999999999997e-05 2024-05-15T23:23:52.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892450+0000", 2024-05-15T23:23:52.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0978999999999997e-05 2024-05-15T23:23:52.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892524+0000", 2024-05-15T23:23:52.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3943999999999995e-05 2024-05-15T23:23:52.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.896062+0000", 2024-05-15T23:23:52.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0035380670000000002 2024-05-15T23:23:52.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.066773+0000", 2024-05-15T23:23:52.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.170711062 2024-05-15T23:23:52.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.066825+0000", 2024-05-15T23:23:52.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1969999999999999e-05 2024-05-15T23:23:52.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.066867+0000", 2024-05-15T23:23:52.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2682000000000002e-05 2024-05-15T23:23:52.616 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.616 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.616 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.616 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313500 4.a 4:54957ed1:::benchmark_data_smithi090_104448_object140262:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:23:52.616 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.433300+0000", 2024-05-15T23:23:52.616 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 319.095459506, 2024-05-15T23:23:52.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.232290565, 2024-05-15T23:23:52.617 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.617 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.617 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.617 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.617 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.617 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.617 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313500 2024-05-15T23:23:52.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.617 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.617 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.617 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433300+0000", 2024-05-15T23:23:52.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433300+0000", 2024-05-15T23:23:52.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433301+0000", 2024-05-15T23:23:52.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7600000000000006e-07 2024-05-15T23:23:52.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433305+0000", 2024-05-15T23:23:52.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2880000000000003e-06 2024-05-15T23:23:52.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433311+0000", 2024-05-15T23:23:52.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1279999999999999e-06 2024-05-15T23:23:52.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433314+0000", 2024-05-15T23:23:52.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8130000000000002e-06 2024-05-15T23:23:52.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.619 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433357+0000", 2024-05-15T23:23:52.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.214e-05 2024-05-15T23:23:52.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433465+0000", 2024-05-15T23:23:52.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000108762 2024-05-15T23:23:52.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433582+0000", 2024-05-15T23:23:52.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011637 2024-05-15T23:23:52.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.438171+0000", 2024-05-15T23:23:52.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045897919999999997 2024-05-15T23:23:52.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665486+0000", 2024-05-15T23:23:52.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22731428400000001 2024-05-15T23:23:52.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665546+0000", 2024-05-15T23:23:52.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9988999999999998e-05 2024-05-15T23:23:52.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665591+0000", 2024-05-15T23:23:52.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5022999999999999e-05 2024-05-15T23:23:52.622 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.622 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.622 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.622 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313526 4.a 4:537fd6b5:::benchmark_data_smithi090_104448_object140288:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:23:52.622 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.459071+0000", 2024-05-15T23:23:52.622 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 319.06968835399999, 2024-05-15T23:23:52.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20662114200000001, 2024-05-15T23:23:52.622 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.622 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.622 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.623 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.623 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.623 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.623 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313526 2024-05-15T23:23:52.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.623 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459071+0000", 2024-05-15T23:23:52.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459071+0000", 2024-05-15T23:23:52.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459073+0000", 2024-05-15T23:23:52.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8980000000000001e-06 2024-05-15T23:23:52.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459079+0000", 2024-05-15T23:23:52.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2730000000000002e-06 2024-05-15T23:23:52.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459090+0000", 2024-05-15T23:23:52.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1619e-05 2024-05-15T23:23:52.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459097+0000", 2024-05-15T23:23:52.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4189999999999999e-06 2024-05-15T23:23:52.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459148+0000", 2024-05-15T23:23:52.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1921e-05 2024-05-15T23:23:52.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459248+0000", 2024-05-15T23:23:52.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9388000000000005e-05 2024-05-15T23:23:52.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.626 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459325+0000", 2024-05-15T23:23:52.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7024e-05 2024-05-15T23:23:52.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.626 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.462998+0000", 2024-05-15T23:23:52.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036728120000000001 2024-05-15T23:23:52.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.626 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665627+0000", 2024-05-15T23:23:52.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.202629165 2024-05-15T23:23:52.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665657+0000", 2024-05-15T23:23:52.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9984999999999998e-05 2024-05-15T23:23:52.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665692+0000", 2024-05-15T23:23:52.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5638e-05 2024-05-15T23:23:52.627 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.627 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.627 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.628 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313534 4.a 4:53d3c98e:::benchmark_data_smithi090_104448_object140296:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:23:52.628 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.467942+0000", 2024-05-15T23:23:52.628 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 319.06081725400003, 2024-05-15T23:23:52.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20189164800000001, 2024-05-15T23:23:52.628 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.628 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.628 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.628 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.628 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.628 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.628 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313534 2024-05-15T23:23:52.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.628 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467942+0000", 2024-05-15T23:23:52.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467942+0000", 2024-05-15T23:23:52.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467944+0000", 2024-05-15T23:23:52.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1459999999999999e-06 2024-05-15T23:23:52.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467948+0000", 2024-05-15T23:23:52.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8389999999999997e-06 2024-05-15T23:23:52.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467957+0000", 2024-05-15T23:23:52.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0790000000000001e-06 2024-05-15T23:23:52.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467962+0000", 2024-05-15T23:23:52.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5180000000000004e-06 2024-05-15T23:23:52.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.468001+0000", 2024-05-15T23:23:52.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9119999999999998e-05 2024-05-15T23:23:52.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.468074+0000", 2024-05-15T23:23:52.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2698000000000001e-05 2024-05-15T23:23:52.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.468139+0000", 2024-05-15T23:23:52.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.512e-05 2024-05-15T23:23:52.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.471787+0000", 2024-05-15T23:23:52.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036482730000000001 2024-05-15T23:23:52.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669769+0000", 2024-05-15T23:23:52.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19798183499999999 2024-05-15T23:23:52.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669803+0000", 2024-05-15T23:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3670999999999997e-05 2024-05-15T23:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669834+0000", 2024-05-15T23:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1349000000000001e-05 2024-05-15T23:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.633 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313558 4.a 4:526840ac:::benchmark_data_smithi090_104448_object140320:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:23:52.634 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.489279+0000", 2024-05-15T23:23:52.634 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 319.03948076199998, 2024-05-15T23:23:52.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18070008900000001, 2024-05-15T23:23:52.634 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.634 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.634 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.634 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.634 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.634 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.634 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313558 2024-05-15T23:23:52.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.634 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489279+0000", 2024-05-15T23:23:52.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489279+0000", 2024-05-15T23:23:52.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489281+0000", 2024-05-15T23:23:52.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.784e-06 2024-05-15T23:23:52.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489286+0000", 2024-05-15T23:23:52.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1699999999999996e-06 2024-05-15T23:23:52.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489296+0000", 2024-05-15T23:23:52.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.005e-05 2024-05-15T23:23:52.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489302+0000", 2024-05-15T23:23:52.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5309999999999998e-06 2024-05-15T23:23:52.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489348+0000", 2024-05-15T23:23:52.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5028000000000003e-05 2024-05-15T23:23:52.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489429+0000", 2024-05-15T23:23:52.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1233e-05 2024-05-15T23:23:52.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489499+0000", 2024-05-15T23:23:52.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0450000000000005e-05 2024-05-15T23:23:52.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.493430+0000", 2024-05-15T23:23:52.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0039305049999999999 2024-05-15T23:23:52.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669881+0000", 2024-05-15T23:23:52.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17645111499999999 2024-05-15T23:23:52.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669936+0000", 2024-05-15T23:23:52.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5328999999999998e-05 2024-05-15T23:23:52.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669979+0000", 2024-05-15T23:23:52.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2894e-05 2024-05-15T23:23:52.639 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.639 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.639 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.639 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313566 4.a 4:56b72cc5:::benchmark_data_smithi090_104448_object140328:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:23:52.639 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.498107+0000", 2024-05-15T23:23:52.639 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 319.03065246900002, 2024-05-15T23:23:52.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18568280400000001, 2024-05-15T23:23:52.639 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.640 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.640 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.640 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.640 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.640 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.640 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313566 2024-05-15T23:23:52.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.640 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.640 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498107+0000", 2024-05-15T23:23:52.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498107+0000", 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498110+0000", 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3140000000000002e-06 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498117+0000", 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1690000000000003e-06 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498128+0000", 2024-05-15T23:23:52.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1362000000000001e-05 2024-05-15T23:23:52.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498136+0000", 2024-05-15T23:23:52.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5769999999999996e-06 2024-05-15T23:23:52.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498184+0000", 2024-05-15T23:23:52.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8631000000000002e-05 2024-05-15T23:23:52.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498280+0000", 2024-05-15T23:23:52.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5958999999999994e-05 2024-05-15T23:23:52.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498356+0000", 2024-05-15T23:23:52.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5574000000000005e-05 2024-05-15T23:23:52.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.503160+0000", 2024-05-15T23:23:52.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0048046499999999997 2024-05-15T23:23:52.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683729+0000", 2024-05-15T23:23:52.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.180568066 2024-05-15T23:23:52.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683760+0000", 2024-05-15T23:23:52.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1798000000000003e-05 2024-05-15T23:23:52.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683790+0000", 2024-05-15T23:23:52.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9703999999999999e-05 2024-05-15T23:23:52.645 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.645 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.645 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.646 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313575 4.a 4:5e037716:::benchmark_data_smithi090_104448_object140337:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:23:52.646 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.507823+0000", 2024-05-15T23:23:52.646 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 319.02093689999998, 2024-05-15T23:23:52.646 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17607811600000001, 2024-05-15T23:23:52.646 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:23:52.646 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:23:52.646 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:23:52.646 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:23:52.646 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:23:52.646 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:23:52.646 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313575 2024-05-15T23:23:52.646 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.646 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:23:52.646 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:23:52.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507823+0000", 2024-05-15T23:23:52.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:23:52.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507823+0000", 2024-05-15T23:23:52.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:23:52.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.647 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:23:52.647 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507825+0000", 2024-05-15T23:23:52.647 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3099999999999999e-06 2024-05-15T23:23:52.647 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.647 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:23:52.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507831+0000", 2024-05-15T23:23:52.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0120000000000002e-06 2024-05-15T23:23:52.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:23:52.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507843+0000", 2024-05-15T23:23:52.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1572e-05 2024-05-15T23:23:52.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.648 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.648 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:23:52.648 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507851+0000", 2024-05-15T23:23:52.648 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3159999999999997e-06 2024-05-15T23:23:52.648 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:23:52.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507900+0000", 2024-05-15T23:23:52.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9444999999999998e-05 2024-05-15T23:23:52.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:23:52.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507981+0000", 2024-05-15T23:23:52.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0547e-05 2024-05-15T23:23:52.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.649 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.649 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:23:52.649 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.508068+0000", 2024-05-15T23:23:52.649 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6699000000000005e-05 2024-05-15T23:23:52.649 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:23:52.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.513050+0000", 2024-05-15T23:23:52.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049819829999999997 2024-05-15T23:23:52.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:23:52.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683820+0000", 2024-05-15T23:23:52.650 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17076991899999999 2024-05-15T23:23:52.650 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.650 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.650 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:23:52.650 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683858+0000", 2024-05-15T23:23:52.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7917999999999998e-05 2024-05-15T23:23:52.651 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:23:52.651 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:23:52.651 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:23:52.651 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683901+0000", 2024-05-15T23:23:52.651 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3395e-05 2024-05-15T23:23:52.651 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.651 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.651 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.651 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:23:52.651 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:23:52.651 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:23:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:52.652 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:23:52.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:23:52.753 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:23:52.795 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:52.822 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:23:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:52.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:23:52.926 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:23:52.955 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:53.054 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:23:53.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:53.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:23:53.162 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:23:53.227 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:53.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:53 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:23:53.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:53 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:23:53.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:53 smithi088 ceph-mon[44699]: osdmap e459: 12 total, 12 up, 9 in 2024-05-15T23:23:53.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:53 smithi088 ceph-mon[44699]: pgmap v1236: 17 pgs: 3 active+recovering, 14 active+clean; 2.8 GiB data, 8.6 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 855 op/s; 5386/90870 objects misplaced (5.927%); 11 MiB/s, 168 objects/s recovering 2024-05-15T23:23:53.369 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:23:53.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:53 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:23:53.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:53 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:23:53.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:53 smithi084 ceph-mon[39784]: osdmap e459: 12 total, 12 up, 9 in 2024-05-15T23:23:53.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:53 smithi084 ceph-mon[39784]: pgmap v1236: 17 pgs: 3 active+recovering, 14 active+clean; 2.8 GiB data, 8.6 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 855 op/s; 5386/90870 objects misplaced (5.927%); 11 MiB/s, 168 objects/s recovering 2024-05-15T23:23:53.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:53.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:23:53.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:53.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:23:53.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:53 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:23:53.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:53 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:23:53.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:53 smithi090 ceph-mon[44434]: osdmap e459: 12 total, 12 up, 9 in 2024-05-15T23:23:53.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:53 smithi090 ceph-mon[44434]: pgmap v1236: 17 pgs: 3 active+recovering, 14 active+clean; 2.8 GiB data, 8.6 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 855 op/s; 5386/90870 objects misplaced (5.927%); 11 MiB/s, 168 objects/s recovering 2024-05-15T23:23:53.528 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:53.558 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:53.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:53.561 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:23:53.662 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:23:53.696 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:53.798 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:23:53.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:53.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:23:53.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:23:53.950 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:53.986 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:23:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:53.990 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:23:54.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:23:54.083 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:54 smithi084 ceph-mon[39784]: 5.e scrub starts 2024-05-15T23:23:54.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:23:54.124 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:54.128 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:54.143 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:23:54.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:54.147 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:23:54.248 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:23:54.277 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:54.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:54 smithi090 ceph-mon[44434]: 5.e scrub starts 2024-05-15T23:23:54.372 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T23:23:54.373 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:54 smithi088 ceph-mon[44699]: 5.e scrub starts 2024-05-15T23:23:54.381 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:23:54.383 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:23:54.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:54.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:23:54.390 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:23:54.488 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:23:54.568 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:54.617 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:23:54.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:54.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:23:54.731 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:23:54.761 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:54.841 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:23:55.209 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:23:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:55.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:23:55.315 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:23:55.358 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:55 smithi088 ceph-mon[44699]: 5.e scrub ok 2024-05-15T23:23:55.358 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:55 smithi088 ceph-mon[44699]: from='client.36152 v1:172.21.15.84:0/4027111509' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:55.358 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:55 smithi088 ceph-mon[44699]: pgmap v1237: 17 pgs: 3 active+recovering, 14 active+clean; 2.8 GiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 765 op/s; 5386/90310 objects misplaced (5.964%); 8.9 MiB/s, 142 objects/s recovering 2024-05-15T23:23:55.361 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:55.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:55 smithi084 ceph-mon[39784]: 5.e scrub ok 2024-05-15T23:23:55.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:55 smithi084 ceph-mon[39784]: from='client.36152 v1:172.21.15.84:0/4027111509' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:55.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:55 smithi084 ceph-mon[39784]: pgmap v1237: 17 pgs: 3 active+recovering, 14 active+clean; 2.8 GiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 765 op/s; 5386/90310 objects misplaced (5.964%); 8.9 MiB/s, 142 objects/s recovering 2024-05-15T23:23:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:55 smithi090 ceph-mon[44434]: 5.e scrub ok 2024-05-15T23:23:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:55 smithi090 ceph-mon[44434]: from='client.36152 v1:172.21.15.84:0/4027111509' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:23:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:55 smithi090 ceph-mon[44434]: pgmap v1237: 17 pgs: 3 active+recovering, 14 active+clean; 2.8 GiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 765 op/s; 5386/90310 objects misplaced (5.964%); 8.9 MiB/s, 142 objects/s recovering 2024-05-15T23:23:55.705 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:55.732 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:23:55.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:55.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:23:55.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:23:55.871 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:55.902 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:23:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:55.911 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:23:56.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:23:56.040 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:56.140 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:23:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:56.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:23:56.154 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1236,"stamp":"2024-05-15T23:23:53.009648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2978021944,"num_objects":45435,"num_object_clones":0,"num_object_copies":90870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5386,"num_objects_unfound":0,"num_objects_dirty":45435,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":261806,"num_write_kb":9829799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20509,"num_bytes_recovered":1345271392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160877,"ondisk_log_size":160877,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":8988536,"kb_used_data":5546560,"kb_used_omap":14,"kb_used_meta":3441649,"kb_avail":834717832,"statfs":{"total":863955320832,"available":854751059968,"internally_reserved":0,"allocated":5679677440,"data_stored":5675368422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14358,"internal_metadata":3524249578},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-295043072,"num_objects":-4502,"num_object_clones":0,"num_object_copies":-9004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-887,"num_objects_unfound":0,"num_objects_dirty":-4502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4502,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":887,"num_bytes_recovered":58130432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.262226"},"pg_stats":[{"pgid":"5.f","version":"459'16214","reported_seq":16394,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-05-15T23:23:52.600226+0000","last_change":"2024-05-15T23:19:47.305945+0000","last_active":"2024-05-15T23:23:52.600226+0000","last_peered":"2024-05-15T23:23:52.600226+0000","last_clean":"2024-05-15T23:23:52.600226+0000","last_became_active":"2024-05-15T23:19:47.305689+0000","last_became_peered":"2024-05-15T23:19:47.305689+0000","last_unstale":"2024-05-15T23:23:52.600226+0000","last_undegraded":"2024-05-15T23:23:52.600226+0000","last_fullsized":"2024-05-15T23:23:52.600226+0000","mapping_epoch":374,"log_start":"394'6200","ondisk_log_start":"394'6200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186515456,"num_objects":2846,"num_object_clones":0,"num_object_copies":5692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16214,"num_write_kb":609920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"457'16075","reported_seq":16426,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-05-15T23:23:50.549676+0000","last_change":"2024-05-15T23:22:23.637106+0000","last_active":"2024-05-15T23:23:50.549676+0000","last_peered":"2024-05-15T23:23:50.549676+0000","last_clean":"2024-05-15T23:23:50.549676+0000","last_became_active":"2024-05-15T23:19:47.325414+0000","last_became_peered":"2024-05-15T23:19:47.325414+0000","last_unstale":"2024-05-15T23:23:50.549676+0000","last_undegraded":"2024-05-15T23:23:50.549676+0000","last_fullsized":"2024-05-15T23:23:50.549676+0000","mapping_epoch":374,"log_start":"393'6000","ondisk_log_start":"393'6000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'11848","last_scrub_stamp":"2024-05-15T23:22:23.636778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:23.636778+0000","objects_scrubbed":7095,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186580992,"num_objects":2847,"num_object_clones":0,"num_object_copies":5694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16075,"num_write_kb":605504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"459'16660","reported_seq":16841,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-05-15T23:23:52.092050+0000","last_change":"2024-05-15T23:19:47.306160+0000","last_active":"2024-05-15T23:23:52.092050+0000","last_peered":"2024-05-15T23:23:52.092050+0000","last_clean":"2024-05-15T23:23:52.092050+0000","last_became_active":"2024-05-15T23:19:47.305860+0000","last_became_peered":"2024-05-15T23:19:47.305860+0000","last_unstale":"2024-05-15T23:23:52.092050+0000","last_undegraded":"2024-05-15T23:23:52.092050+0000","last_fullsized":"2024-05-15T23:23:52.092050+0000","mapping_epoch":374,"log_start":"394'6600","ondisk_log_start":"394'6600","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182517784,"num_objects":2786,"num_object_clones":0,"num_object_copies":5572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16660,"num_write_kb":622209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"458'16423","reported_seq":16764,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-05-15T23:23:50.808320+0000","last_change":"2024-05-15T23:22:51.995640+0000","last_active":"2024-05-15T23:23:50.808320+0000","last_peered":"2024-05-15T23:23:50.808320+0000","last_clean":"2024-05-15T23:23:50.808320+0000","last_became_active":"2024-05-15T23:19:47.306168+0000","last_became_peered":"2024-05-15T23:19:47.306168+0000","last_unstale":"2024-05-15T23:23:50.808320+0000","last_undegraded":"2024-05-15T23:23:50.808320+0000","last_fullsized":"2024-05-15T23:23:50.808320+0000","mapping_epoch":374,"log_start":"394'6400","ondisk_log_start":"394'6400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'13335","last_scrub_stamp":"2024-05-15T23:22:51.995473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:51.995473+0000","objects_scrubbed":6018,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:13.168527+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189726720,"num_objects":2895,"num_object_clones":0,"num_object_copies":5790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16423,"num_write_kb":618176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"457'16290","reported_seq":16632,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-05-15T23:23:50.556439+0000","last_change":"2024-05-15T23:22:48.338084+0000","last_active":"2024-05-15T23:23:50.556439+0000","last_peered":"2024-05-15T23:23:50.556439+0000","last_clean":"2024-05-15T23:23:50.556439+0000","last_became_active":"2024-05-15T23:19:47.307061+0000","last_became_peered":"2024-05-15T23:19:47.307061+0000","last_unstale":"2024-05-15T23:23:50.556439+0000","last_undegraded":"2024-05-15T23:23:50.556439+0000","last_fullsized":"2024-05-15T23:23:50.556439+0000","mapping_epoch":374,"log_start":"394'6200","ondisk_log_start":"394'6200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'13172","last_scrub_stamp":"2024-05-15T23:22:48.337866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:48.337866+0000","objects_scrubbed":6060,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:11.906769+0000","scrub_duration":1844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190185472,"num_objects":2902,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16290,"num_write_kb":614144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"457'16365","reported_seq":16541,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-05-15T23:23:50.586170+0000","last_change":"2024-05-15T23:19:47.325657+0000","last_active":"2024-05-15T23:23:50.586170+0000","last_peered":"2024-05-15T23:23:50.586170+0000","last_clean":"2024-05-15T23:23:50.586170+0000","last_became_active":"2024-05-15T23:19:47.325371+0000","last_became_peered":"2024-05-15T23:19:47.325371+0000","last_unstale":"2024-05-15T23:23:50.586170+0000","last_undegraded":"2024-05-15T23:23:50.586170+0000","last_fullsized":"2024-05-15T23:23:50.586170+0000","mapping_epoch":374,"log_start":"394'6300","ondisk_log_start":"394'6300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186580992,"num_objects":2847,"num_object_clones":0,"num_object_copies":5694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16365,"num_write_kb":614784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"458'16375","reported_seq":30168,"reported_epoch":458,"state":"active+recovering","last_fresh":"2024-05-15T23:23:51.044237+0000","last_change":"2024-05-15T23:21:09.568311+0000","last_active":"2024-05-15T23:23:51.044237+0000","last_peered":"2024-05-15T23:23:51.044237+0000","last_clean":"2024-05-15T23:21:08.023957+0000","last_became_active":"2024-05-15T23:21:09.454639+0000","last_became_peered":"2024-05-15T23:21:09.454639+0000","last_unstale":"2024-05-15T23:23:51.044237+0000","last_undegraded":"2024-05-15T23:23:51.044237+0000","last_fullsized":"2024-05-15T23:23:51.044237+0000","mapping_epoch":407,"log_start":"394'6300","ondisk_log_start":"394'6300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182124544,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1764,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16375,"num_write_kb":612928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6806,"num_bytes_recovered":446038016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1764},{"shards":"7,11","objects":1015}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"459'16410","reported_seq":16590,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-05-15T23:23:52.126175+0000","last_change":"2024-05-15T23:19:47.304832+0000","last_active":"2024-05-15T23:23:52.126175+0000","last_peered":"2024-05-15T23:23:52.126175+0000","last_clean":"2024-05-15T23:23:52.126175+0000","last_became_active":"2024-05-15T23:19:47.304547+0000","last_became_peered":"2024-05-15T23:19:47.304547+0000","last_unstale":"2024-05-15T23:23:52.126175+0000","last_undegraded":"2024-05-15T23:23:52.126175+0000","last_fullsized":"2024-05-15T23:23:52.126175+0000","mapping_epoch":374,"log_start":"394'6400","ondisk_log_start":"394'6400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183107584,"num_objects":2794,"num_object_clones":0,"num_object_copies":5588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16410,"num_write_kb":614528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"459'16304","reported_seq":16484,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-05-15T23:23:52.615580+0000","last_change":"2024-05-15T23:19:47.306634+0000","last_active":"2024-05-15T23:23:52.615580+0000","last_peered":"2024-05-15T23:23:52.615580+0000","last_clean":"2024-05-15T23:23:52.615580+0000","last_became_active":"2024-05-15T23:19:47.306331+0000","last_became_peered":"2024-05-15T23:19:47.306331+0000","last_unstale":"2024-05-15T23:23:52.615580+0000","last_undegraded":"2024-05-15T23:23:52.615580+0000","last_fullsized":"2024-05-15T23:23:52.615580+0000","mapping_epoch":374,"log_start":"394'6300","ondisk_log_start":"394'6300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16304,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"459'16561","reported_seq":16741,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-05-15T23:23:52.577384+0000","last_change":"2024-05-15T23:19:47.306611+0000","last_active":"2024-05-15T23:23:52.577384+0000","last_peered":"2024-05-15T23:23:52.577384+0000","last_clean":"2024-05-15T23:23:52.577384+0000","last_became_active":"2024-05-15T23:19:47.306349+0000","last_became_peered":"2024-05-15T23:19:47.306349+0000","last_unstale":"2024-05-15T23:23:52.577384+0000","last_undegraded":"2024-05-15T23:23:52.577384+0000","last_fullsized":"2024-05-15T23:23:52.577384+0000","mapping_epoch":374,"log_start":"394'6500","ondisk_log_start":"394'6500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180158464,"num_objects":2749,"num_object_clones":0,"num_object_copies":5498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16561,"num_write_kb":617920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"458'16481","reported_seq":16926,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-05-15T23:23:50.826126+0000","last_change":"2024-05-15T23:23:50.810750+0000","last_active":"2024-05-15T23:23:50.826126+0000","last_peered":"2024-05-15T23:23:50.826126+0000","last_clean":"2024-05-15T23:23:50.826126+0000","last_became_active":"2024-05-15T23:19:47.305685+0000","last_became_peered":"2024-05-15T23:19:47.305685+0000","last_unstale":"2024-05-15T23:23:50.826126+0000","last_undegraded":"2024-05-15T23:23:50.826126+0000","last_fullsized":"2024-05-15T23:23:50.826126+0000","mapping_epoch":374,"log_start":"394'6400","ondisk_log_start":"394'6400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'16478","last_scrub_stamp":"2024-05-15T23:23:50.810636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:50.810636+0000","objects_scrubbed":2939,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:20.026294+0000","scrub_duration":608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191037440,"num_objects":2915,"num_object_clones":0,"num_object_copies":5830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16481,"num_write_kb":620672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"459'16484","reported_seq":30413,"reported_epoch":459,"state":"active+recovering","last_fresh":"2024-05-15T23:23:52.643137+0000","last_change":"2024-05-15T23:21:09.470461+0000","last_active":"2024-05-15T23:23:52.643137+0000","last_peered":"2024-05-15T23:23:52.643137+0000","last_clean":"2024-05-15T23:21:07.989871+0000","last_became_active":"2024-05-15T23:21:09.400731+0000","last_became_peered":"2024-05-15T23:21:09.400731+0000","last_unstale":"2024-05-15T23:23:52.643137+0000","last_undegraded":"2024-05-15T23:23:52.643137+0000","last_fullsized":"2024-05-15T23:23:52.643137+0000","mapping_epoch":407,"log_start":"394'6400","ondisk_log_start":"394'6400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184287232,"num_objects":2812,"num_object_clones":0,"num_object_copies":5624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1817,"num_objects_unfound":0,"num_objects_dirty":2812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16484,"num_write_kb":617472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6873,"num_bytes_recovered":450428928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"3,5","objects":995},{"shards":"5,6","objects":1817}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"457'16404","reported_seq":30232,"reported_epoch":457,"state":"active+recovering","last_fresh":"2024-05-15T23:23:50.580589+0000","last_change":"2024-05-15T23:21:09.275620+0000","last_active":"2024-05-15T23:23:50.580589+0000","last_peered":"2024-05-15T23:23:50.580589+0000","last_clean":"2024-05-15T23:21:07.993692+0000","last_became_active":"2024-05-15T23:21:09.221797+0000","last_became_peered":"2024-05-15T23:21:09.221797+0000","last_unstale":"2024-05-15T23:23:50.580589+0000","last_undegraded":"2024-05-15T23:23:50.580589+0000","last_fullsized":"2024-05-15T23:23:50.580589+0000","mapping_epoch":407,"log_start":"394'6400","ondisk_log_start":"394'6400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181272576,"num_objects":2766,"num_object_clones":0,"num_object_copies":5532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1805,"num_objects_unfound":0,"num_objects_dirty":2766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16404,"num_write_kb":613440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6824,"num_bytes_recovered":447217664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":1805},{"shards":"7,10","objects":961}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.1","version":"457'15819","reported_seq":16168,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-05-15T23:23:49.907042+0000","last_change":"2024-05-15T23:22:19.625857+0000","last_active":"2024-05-15T23:23:49.907042+0000","last_peered":"2024-05-15T23:23:49.907042+0000","last_clean":"2024-05-15T23:23:49.907042+0000","last_became_active":"2024-05-15T23:19:47.516865+0000","last_became_peered":"2024-05-15T23:19:47.516865+0000","last_unstale":"2024-05-15T23:23:49.907042+0000","last_undegraded":"2024-05-15T23:23:49.907042+0000","last_fullsized":"2024-05-15T23:23:49.907042+0000","mapping_epoch":374,"log_start":"393'5800","ondisk_log_start":"393'5800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"426'11474","last_scrub_stamp":"2024-05-15T23:22:19.625719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:19.625719+0000","objects_scrubbed":7211,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186056704,"num_objects":2839,"num_object_clones":0,"num_object_copies":5678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15819,"num_write_kb":597056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"457'16385","reported_seq":16562,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-05-15T23:23:50.575749+0000","last_change":"2024-05-15T23:19:47.305723+0000","last_active":"2024-05-15T23:23:50.575749+0000","last_peered":"2024-05-15T23:23:50.575749+0000","last_clean":"2024-05-15T23:23:50.575749+0000","last_became_active":"2024-05-15T23:19:47.305459+0000","last_became_peered":"2024-05-15T23:19:47.305459+0000","last_unstale":"2024-05-15T23:23:50.575749+0000","last_undegraded":"2024-05-15T23:23:50.575749+0000","last_fullsized":"2024-05-15T23:23:50.575749+0000","mapping_epoch":374,"log_start":"394'6300","ondisk_log_start":"394'6300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189464576,"num_objects":2891,"num_object_clones":0,"num_object_copies":5782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16385,"num_write_kb":616832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"457'16433","reported_seq":16609,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-05-15T23:23:50.588319+0000","last_change":"2024-05-15T23:19:47.307499+0000","last_active":"2024-05-15T23:23:50.588319+0000","last_peered":"2024-05-15T23:23:50.588319+0000","last_clean":"2024-05-15T23:23:50.588319+0000","last_became_active":"2024-05-15T23:19:47.307149+0000","last_became_peered":"2024-05-15T23:19:47.307149+0000","last_unstale":"2024-05-15T23:23:50.588319+0000","last_undegraded":"2024-05-15T23:23:50.588319+0000","last_fullsized":"2024-05-15T23:23:50.588319+0000","mapping_epoch":374,"log_start":"394'6400","ondisk_log_start":"394'6400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197459968,"num_objects":3013,"num_object_clones":0,"num_object_copies":6026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16433,"num_write_kb":622272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":955,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-05-15T23:23:50.796537+0000","last_change":"2024-05-15T23:20:05.905241+0000","last_active":"2024-05-15T23:23:50.796537+0000","last_peered":"2024-05-15T23:23:50.796537+0000","last_clean":"2024-05-15T23:23:50.796537+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:23:50.796537+0000","last_undegraded":"2024-05-15T23:23:50.796537+0000","last_fullsized":"2024-05-15T23:23:50.796537+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:20:05.905105+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:20:05.905105+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2977431576,"num_objects":45433,"num_object_clones":0,"num_object_copies":90866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5386,"num_objects_unfound":0,"num_objects_dirty":45433,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":261683,"num_write_kb":9827649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20503,"num_bytes_recovered":1343684608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7318937600,"data_stored":7318929456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160783,"ondisk_log_size":160783,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351217,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965256,"kb_used_data":424572,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92779896,"statfs":{"total":95995035648,"available":95006613504,"internally_reserved":0,"allocated":434761728,"data_stored":434272998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":553646532},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580150,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1211700,"kb_used_data":932004,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":92533452,"statfs":{"total":95995035648,"available":94754254848,"internally_reserved":0,"allocated":954372096,"data_stored":953897014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":286390720},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776382,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744304,"kb_used_data":370212,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93000848,"statfs":{"total":95995035648,"available":95232868352,"internally_reserved":0,"allocated":379097088,"data_stored":378622006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":383056320},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005314,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":553960,"kb_used_data":378392,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93191192,"statfs":{"total":95995035648,"available":95427780608,"internally_reserved":0,"allocated":387473408,"data_stored":387004330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266950,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":806984,"kb_used_data":424568,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92938168,"statfs":{"total":95995035648,"available":95168684032,"internally_reserved":0,"allocated":434757632,"data_stored":434272994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":391576010},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495885,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575142,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1248620,"kb_used_data":748196,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92496532,"statfs":{"total":95995035648,"available":94716448768,"internally_reserved":0,"allocated":766152704,"data_stored":765677622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":512424390},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016551,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483240,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149981,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":657452,"kb_used_data":243108,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93087700,"statfs":{"total":95995035648,"available":95321804800,"internally_reserved":0,"allocated":248942592,"data_stored":248467510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":424278471},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986453,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1164148,"kb_used_data":763556,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92581004,"statfs":{"total":95995035648,"available":94802948096,"internally_reserved":0,"allocated":781881344,"data_stored":781406262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790226,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1636112,"kb_used_data":1261952,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92109040,"statfs":{"total":95995035648,"available":94319656960,"internally_reserved":0,"allocated":1292238848,"data_stored":1291747686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":756350976,"data_stored":756350976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":246743040,"data_stored":246743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":771096576,"data_stored":771096576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1284247552,"data_stored":1284243480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1696661504,"data_stored":1696661504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":431685632,"data_stored":431685632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":380633088,"data_stored":380633088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":376766464,"data_stored":376766464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":943128576,"data_stored":943128576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":431624192,"data_stored":431620120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:23:56.155 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.9 is not active+clean 2024-05-15T23:23:56.155 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.9', 'version': "458'16375", 'reported_seq': 30168, 'reported_epoch': 458, 'state': 'active+recovering', 'last_fresh': '2024-05-15T23:23:51.044237+0000', 'last_change': '2024-05-15T23:21:09.568311+0000', 'last_active': '2024-05-15T23:23:51.044237+0000', 'last_peered': '2024-05-15T23:23:51.044237+0000', 'last_clean': '2024-05-15T23:21:08.023957+0000', 'last_became_active': '2024-05-15T23:21:09.454639+0000', 'last_became_peered': '2024-05-15T23:21:09.454639+0000', 'last_unstale': '2024-05-15T23:23:51.044237+0000', 'last_undegraded': '2024-05-15T23:23:51.044237+0000', 'last_fullsized': '2024-05-15T23:23:51.044237+0000', 'mapping_epoch': 407, 'log_start': "394'6300", 'ondisk_log_start': "394'6300", 'created': 374, 'last_epoch_clean': 375, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'last_clean_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 182124544, 'num_objects': 2779, 'num_object_clones': 0, 'num_object_copies': 5558, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1764, 'num_objects_unfound': 0, 'num_objects_dirty': 2779, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16375, 'num_write_kb': 612928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6806, 'num_bytes_recovered': 446038016, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 11], 'acting': [7, 11], 'avail_no_missing': ['7', '6'], 'object_location_counts': [{'shards': '6,7', 'objects': 1764}, {'shards': '7,11', 'objects': 1015}], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-15T23:23:56.155 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.3 is not active+clean 2024-05-15T23:23:56.155 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.3', 'version': "459'16484", 'reported_seq': 30413, 'reported_epoch': 459, 'state': 'active+recovering', 'last_fresh': '2024-05-15T23:23:52.643137+0000', 'last_change': '2024-05-15T23:21:09.470461+0000', 'last_active': '2024-05-15T23:23:52.643137+0000', 'last_peered': '2024-05-15T23:23:52.643137+0000', 'last_clean': '2024-05-15T23:21:07.989871+0000', 'last_became_active': '2024-05-15T23:21:09.400731+0000', 'last_became_peered': '2024-05-15T23:21:09.400731+0000', 'last_unstale': '2024-05-15T23:23:52.643137+0000', 'last_undegraded': '2024-05-15T23:23:52.643137+0000', 'last_fullsized': '2024-05-15T23:23:52.643137+0000', 'mapping_epoch': 407, 'log_start': "394'6400", 'ondisk_log_start': "394'6400", 'created': 374, 'last_epoch_clean': 375, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'last_clean_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 184287232, 'num_objects': 2812, 'num_object_clones': 0, 'num_object_copies': 5624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1817, 'num_objects_unfound': 0, 'num_objects_dirty': 2812, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16484, 'num_write_kb': 617472, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6873, 'num_bytes_recovered': 450428928, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': ['5', '6'], 'object_location_counts': [{'shards': '3,5', 'objects': 995}, {'shards': '5,6', 'objects': 1817}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-15T23:23:56.155 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.2 is not active+clean 2024-05-15T23:23:56.156 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.2', 'version': "457'16404", 'reported_seq': 30232, 'reported_epoch': 457, 'state': 'active+recovering', 'last_fresh': '2024-05-15T23:23:50.580589+0000', 'last_change': '2024-05-15T23:21:09.275620+0000', 'last_active': '2024-05-15T23:23:50.580589+0000', 'last_peered': '2024-05-15T23:23:50.580589+0000', 'last_clean': '2024-05-15T23:21:07.993692+0000', 'last_became_active': '2024-05-15T23:21:09.221797+0000', 'last_became_peered': '2024-05-15T23:21:09.221797+0000', 'last_unstale': '2024-05-15T23:23:50.580589+0000', 'last_undegraded': '2024-05-15T23:23:50.580589+0000', 'last_fullsized': '2024-05-15T23:23:50.580589+0000', 'mapping_epoch': 407, 'log_start': "394'6400", 'ondisk_log_start': "394'6400", 'created': 374, 'last_epoch_clean': 375, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'last_clean_scrub_stamp': '2024-05-15T23:19:46.285287+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 181272576, 'num_objects': 2766, 'num_object_clones': 0, 'num_object_copies': 5532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1805, 'num_objects_unfound': 0, 'num_objects_dirty': 2766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16404, 'num_write_kb': 613440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6824, 'num_bytes_recovered': 447217664, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [10, 7], 'acting': [10, 7], 'avail_no_missing': ['10', '6'], 'object_location_counts': [{'shards': '6,10', 'objects': 1805}, {'shards': '7,10', 'objects': 961}], 'blocked_by': [], 'up_primary': 10, 'acting_primary': 10, 'purged_snaps': []} 2024-05-15T23:23:56.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph status --format=json 2024-05-15T23:23:56.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:23:56.308 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:56.340 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:23:56.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:56.344 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:23:56.445 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:23:56.477 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:56.744 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/827758763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:56.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:23:56.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:23:56.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:23:56.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:23:56.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:23:56.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:23:56.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:23:56.997 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:23:56.997 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:23:56.997 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:23:56.997 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:23:56.997 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:23:56.997 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:23:56.998 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:23:56.998 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:23:56.998 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:23:56.998 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:23:56.998 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:23:56.998 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:23:56.999 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:23:56.999 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:23:56.999 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:23:56.999 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:23:56.999 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:23:57.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/827758763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:57.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/827758763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:23:57.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:57.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:57.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:57.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:57 smithi084 ceph-mon[39784]: pgmap v1238: 17 pgs: 3 active+recovering, 14 active+clean; 2.6 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 922 op/s; 4801/85110 objects misplaced (5.641%); 11 MiB/s, 183 objects/s recovering 2024-05-15T23:23:58.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:57 smithi090 ceph-mon[44434]: pgmap v1238: 17 pgs: 3 active+recovering, 14 active+clean; 2.6 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 922 op/s; 4801/85110 objects misplaced (5.641%); 11 MiB/s, 183 objects/s recovering 2024-05-15T23:23:58.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:57 smithi088 ceph-mon[44699]: pgmap v1238: 17 pgs: 3 active+recovering, 14 active+clean; 2.6 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 922 op/s; 4801/85110 objects misplaced (5.641%); 11 MiB/s, 183 objects/s recovering 2024-05-15T23:23:58.174 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:23:58.434 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:23:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:58.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:23:58.494 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:23:58.495 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:23:58.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:23:58.583 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:58.606 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:23:58.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:58.614 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:23:58.714 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:23:58.769 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.769 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:58.770 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:59.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:59.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:23:59.319 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T23:23:59.342 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:23:59.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:59.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:23:59.448 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:23:59.512 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:23:59.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:59.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:23:59.674 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:23:59.725 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:59.768 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:23:59.786 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:23:59.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:59.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:23:59.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:23:59 smithi088 ceph-mon[44699]: pgmap v1239: 17 pgs: 3 active+recovering, 14 active+clean; 2.5 GiB data, 8.1 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 856 op/s; 4525/81840 objects misplaced (5.529%); 11 MiB/s, 174 objects/s recovering 2024-05-15T23:23:59.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:23:59.944 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:23:59.973 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:23:59.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:23:59.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:24:00.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:23:59 smithi090 ceph-mon[44434]: pgmap v1239: 17 pgs: 3 active+recovering, 14 active+clean; 2.5 GiB data, 8.1 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 856 op/s; 4525/81840 objects misplaced (5.529%); 11 MiB/s, 174 objects/s recovering 2024-05-15T23:24:00.077 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:24:00.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:23:59 smithi084 ceph-mon[39784]: pgmap v1239: 17 pgs: 3 active+recovering, 14 active+clean; 2.5 GiB data, 8.1 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 856 op/s; 4525/81840 objects misplaced (5.529%); 11 MiB/s, 174 objects/s recovering 2024-05-15T23:24:00.136 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:24:00.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:00.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:24:00.269 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:24:00.302 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:00.650 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:24:00.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:00.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:24:00.755 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:24:00.828 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:00.963 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:24:00.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:00.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:24:01.073 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:01.105 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:01.230 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:01.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:01.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:01.334 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:24:01.334 INFO:teuthology.orchestra.run.smithi084.stdout:{"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1521,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":459,"num_osds":12,"num_up_osds":12,"osd_up_since":1715815247,"num_in_osds":9,"osd_in_since":1715815268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovering","count":3}],"num_pgs":17,"num_pools":2,"num_objects":38307,"data_bytes":2510881336,"bytes_used":8398630912,"bytes_avail":855556689920,"bytes_total":863955320832,"misplaced_objects":4213,"misplaced_total":76614,"misplaced_ratio":0.054989949617563372,"recovering_objects_per_sec":175,"recovering_bytes_per_sec":11473341,"recovering_keys_per_sec":0,"num_objects_recovered":1751,"num_bytes_recovered":114753536,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":911},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.84:8443/"}},"servicemap":{"epoch":34,"modified":"2024-05-15T23:23:09.001385+0000","services":{}},"progress_events":{}} 2024-05-15T23:24:01.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:24:01.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:01 smithi084 ceph-mon[39784]: pgmap v1240: 17 pgs: 3 active+recovering, 14 active+clean; 2.3 GiB data, 7.8 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 911 op/s; 4213/76614 objects misplaced (5.499%); 11 MiB/s, 175 objects/s recovering 2024-05-15T23:24:01.384 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:01 smithi090 ceph-mon[44434]: pgmap v1240: 17 pgs: 3 active+recovering, 14 active+clean; 2.3 GiB data, 7.8 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 911 op/s; 4213/76614 objects misplaced (5.499%); 11 MiB/s, 175 objects/s recovering 2024-05-15T23:24:01.386 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:01.418 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:24:01.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:01.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:24:01.424 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:24:01.522 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:24:01.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:01 smithi088 ceph-mon[44699]: pgmap v1240: 17 pgs: 3 active+recovering, 14 active+clean; 2.3 GiB data, 7.8 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 911 op/s; 4213/76614 objects misplaced (5.499%); 11 MiB/s, 175 objects/s recovering 2024-05-15T23:24:01.584 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:01.713 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:24:01.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:01.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:24:01.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:24:01.847 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:02.077 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:24:02.077 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:24:02.116 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:24:02.202 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-05-15T23:24:02.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:24:02.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:02 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:24:02.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:02 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:24:02.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2336513737' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T23:24:02.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/567938147' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:24:02.398 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:24:02.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:02.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:24:02.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:24:02.533 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:02.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:02 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:24:02.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:02 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:24:02.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2336513737' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T23:24:02.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/567938147' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:24:02.556 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:24:02.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:02 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:24:02.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:02 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:24:02.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2336513737' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T23:24:02.562 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/567938147' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:24:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:02.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:24:02.665 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:02.691 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:02.893 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:02.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:02.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:02.976 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T23:24:02.977 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:03.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:03.015 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:03.037 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:03.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:03 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:24:03.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:03 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:24:03.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/567938147' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:24:03.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:03 smithi084 ceph-mon[39784]: osdmap e460: 12 total, 12 up, 9 in 2024-05-15T23:24:03.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:03 smithi084 ceph-mon[39784]: pgmap v1242: 17 pgs: 3 active+recovering, 14 active+clean; 2.2 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 924 op/s; 3601/72386 objects misplaced (4.975%); 11 MiB/s, 178 objects/s recovering 2024-05-15T23:24:03.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:03 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:24:03.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:03 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:24:03.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/567938147' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:24:03.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:03 smithi088 ceph-mon[44699]: osdmap e460: 12 total, 12 up, 9 in 2024-05-15T23:24:03.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:03 smithi088 ceph-mon[44699]: pgmap v1242: 17 pgs: 3 active+recovering, 14 active+clean; 2.2 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 924 op/s; 3601/72386 objects misplaced (4.975%); 11 MiB/s, 178 objects/s recovering 2024-05-15T23:24:03.415 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:24:03.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:03 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:24:03.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:03 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:24:03.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/567938147' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:24:03.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:03 smithi090 ceph-mon[44434]: osdmap e460: 12 total, 12 up, 9 in 2024-05-15T23:24:03.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:03 smithi090 ceph-mon[44434]: pgmap v1242: 17 pgs: 3 active+recovering, 14 active+clean; 2.2 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 924 op/s; 3601/72386 objects misplaced (4.975%); 11 MiB/s, 178 objects/s recovering 2024-05-15T23:24:04.421 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:04.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:04.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:24:04.555 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:04.579 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:24:04.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:04.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:24:04.684 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:24:04.726 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:04.761 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:24:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:04.764 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:24:04.865 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:24:04.932 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:24:04.941 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:04.979 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:24:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:04.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:24:05.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:24:05.125 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:05.152 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:24:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:05.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:24:05.258 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:24:05.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:05 smithi084 ceph-mon[39784]: pgmap v1243: 17 pgs: 3 active+recovering, 14 active+clean; 2.2 GiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 923 op/s; 3601/71832 objects misplaced (5.013%); 11 MiB/s, 178 objects/s recovering 2024-05-15T23:24:05.287 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:05.306 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:05 smithi088 ceph-mon[44699]: pgmap v1243: 17 pgs: 3 active+recovering, 14 active+clean; 2.2 GiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 923 op/s; 3601/71832 objects misplaced (5.013%); 11 MiB/s, 178 objects/s recovering 2024-05-15T23:24:05.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:05 smithi090 ceph-mon[44434]: pgmap v1243: 17 pgs: 3 active+recovering, 14 active+clean; 2.2 GiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 923 op/s; 3601/71832 objects misplaced (5.013%); 11 MiB/s, 178 objects/s recovering 2024-05-15T23:24:05.639 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:24:05.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:05.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:24:05.716 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:24:05.716 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:24:05.742 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:24:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:05.818 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:24:05.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:05.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:24:05.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:05.972 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:06.003 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:06.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:06.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:06.113 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:24:06.153 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:06.186 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:24:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:06.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:24:06.292 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:06.325 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:06.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T23:24:06.703 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:06.741 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:24:06.766 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:24:06.768 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:24:06.775 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:24:06.791 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:24:06.813 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:24:06.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:06 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2773575450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:06.839 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:24:06.867 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:24:06.890 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:24:06.915 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:24:06.939 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:24:06.961 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:24:06.986 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:24:07.012 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:24:07.013 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:24:07.013 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:24:07.013 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:24:07.013 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:24:07.013 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:24:07.014 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:24:07.014 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:24:07.014 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:24:07.014 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:24:07.014 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:24:07.014 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:24:07.014 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:24:07.034 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:06 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2773575450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:07.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:06 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2773575450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:07.223 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:24:07.480 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:07.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:07.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:07.584 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:24:07.637 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:07.669 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:24:07.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:07.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:24:07.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:24:07.823 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:07.823 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:07 smithi090 ceph-mon[44434]: from='client.16293 v1:172.21.15.84:0/3199392601' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:24:07.823 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:07.823 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:07.824 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:07.824 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:07.824 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:07.824 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:07.824 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:07.824 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:07.824 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:07 smithi090 ceph-mon[44434]: pgmap v1244: 17 pgs: 3 active+recovering, 14 active+clean; 2.0 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 924 op/s; 2970/66612 objects misplaced (4.459%); 11 MiB/s, 183 objects/s recovering 2024-05-15T23:24:07.824 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:07 smithi090 ceph-mon[44434]: osdmap e461: 12 total, 12 up, 9 in 2024-05-15T23:24:07.825 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:07.931 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:24:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:07.935 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:24:08.023 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1243,"stamp":"2024-05-15T23:24:05.011863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2354184760,"num_objects":35916,"num_object_clones":0,"num_object_copies":71832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3601,"num_objects_unfound":0,"num_objects_dirty":35916,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":271325,"num_write_kb":9829799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22294,"num_bytes_recovered":1462253152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160796,"ondisk_log_size":160796,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":7862880,"kb_used_data":4363992,"kb_used_omap":14,"kb_used_meta":3498545,"kb_avail":835843488,"statfs":{"total":863955320832,"available":855903731712,"internally_reserved":0,"allocated":4468727808,"data_stored":4464310718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14358,"internal_metadata":3582511082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-605487104,"num_objects":-9239,"num_object_clones":0,"num_object_copies":-18478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1785,"num_objects_unfound":0,"num_objects_dirty":-9239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9239,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1785,"num_bytes_recovered":116981760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001859"},"pg_stats":[{"pgid":"5.f","version":"460'16830","reported_seq":17012,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-05-15T23:24:02.657732+0000","last_change":"2024-05-15T23:19:47.305945+0000","last_active":"2024-05-15T23:24:02.657732+0000","last_peered":"2024-05-15T23:24:02.657732+0000","last_clean":"2024-05-15T23:24:02.657732+0000","last_became_active":"2024-05-15T23:19:47.305689+0000","last_became_peered":"2024-05-15T23:19:47.305689+0000","last_unstale":"2024-05-15T23:24:02.657732+0000","last_undegraded":"2024-05-15T23:24:02.657732+0000","last_fullsized":"2024-05-15T23:24:02.657732+0000","mapping_epoch":374,"log_start":"398'6800","ondisk_log_start":"398'6800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146145280,"num_objects":2230,"num_object_clones":0,"num_object_copies":4460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16830,"num_write_kb":609920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"459'16629","reported_seq":17018,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-05-15T23:24:00.597785+0000","last_change":"2024-05-15T23:23:53.872535+0000","last_active":"2024-05-15T23:24:00.597785+0000","last_peered":"2024-05-15T23:24:00.597785+0000","last_clean":"2024-05-15T23:24:00.597785+0000","last_became_active":"2024-05-15T23:19:47.325414+0000","last_became_peered":"2024-05-15T23:19:47.325414+0000","last_unstale":"2024-05-15T23:24:00.597785+0000","last_undegraded":"2024-05-15T23:24:00.597785+0000","last_fullsized":"2024-05-15T23:24:00.597785+0000","mapping_epoch":374,"log_start":"395'6600","ondisk_log_start":"395'6600","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'16258","last_scrub_stamp":"2024-05-15T23:23:53.872309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:53.872309+0000","objects_scrubbed":2678,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:15.839794+0000","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150274048,"num_objects":2293,"num_object_clones":0,"num_object_copies":4586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16629,"num_write_kb":605504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"460'17207","reported_seq":17390,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-05-15T23:24:02.134169+0000","last_change":"2024-05-15T23:19:47.306160+0000","last_active":"2024-05-15T23:24:02.134169+0000","last_peered":"2024-05-15T23:24:02.134169+0000","last_clean":"2024-05-15T23:24:02.134169+0000","last_became_active":"2024-05-15T23:19:47.305860+0000","last_became_peered":"2024-05-15T23:19:47.305860+0000","last_unstale":"2024-05-15T23:24:02.134169+0000","last_undegraded":"2024-05-15T23:24:02.134169+0000","last_fullsized":"2024-05-15T23:24:02.134169+0000","mapping_epoch":374,"log_start":"398'7200","ondisk_log_start":"398'7200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146669592,"num_objects":2239,"num_object_clones":0,"num_object_copies":4478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17207,"num_write_kb":622209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"459'17034","reported_seq":17377,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-05-15T23:24:00.837504+0000","last_change":"2024-05-15T23:22:51.995640+0000","last_active":"2024-05-15T23:24:00.837504+0000","last_peered":"2024-05-15T23:24:00.837504+0000","last_clean":"2024-05-15T23:24:00.837504+0000","last_became_active":"2024-05-15T23:19:47.306168+0000","last_became_peered":"2024-05-15T23:19:47.306168+0000","last_unstale":"2024-05-15T23:24:00.837504+0000","last_undegraded":"2024-05-15T23:24:00.837504+0000","last_fullsized":"2024-05-15T23:24:00.837504+0000","mapping_epoch":374,"log_start":"398'7000","ondisk_log_start":"398'7000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'13335","last_scrub_stamp":"2024-05-15T23:22:51.995473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:51.995473+0000","objects_scrubbed":6018,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:13.168527+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149684224,"num_objects":2284,"num_object_clones":0,"num_object_copies":4568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17034,"num_write_kb":618176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"459'16864","reported_seq":17210,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-05-15T23:24:00.567753+0000","last_change":"2024-05-15T23:22:48.338084+0000","last_active":"2024-05-15T23:24:00.567753+0000","last_peered":"2024-05-15T23:24:00.567753+0000","last_clean":"2024-05-15T23:24:00.567753+0000","last_became_active":"2024-05-15T23:19:47.307061+0000","last_became_peered":"2024-05-15T23:19:47.307061+0000","last_unstale":"2024-05-15T23:24:00.567753+0000","last_undegraded":"2024-05-15T23:24:00.567753+0000","last_fullsized":"2024-05-15T23:24:00.567753+0000","mapping_epoch":374,"log_start":"396'6800","ondisk_log_start":"396'6800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'13172","last_scrub_stamp":"2024-05-15T23:22:48.337866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:48.337866+0000","objects_scrubbed":6060,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:11.906769+0000","scrub_duration":1844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152567808,"num_objects":2328,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16864,"num_write_kb":614144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"459'16936","reported_seq":17116,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-05-15T23:24:00.584315+0000","last_change":"2024-05-15T23:19:47.325657+0000","last_active":"2024-05-15T23:24:00.584315+0000","last_peered":"2024-05-15T23:24:00.584315+0000","last_clean":"2024-05-15T23:24:00.584315+0000","last_became_active":"2024-05-15T23:19:47.325371+0000","last_became_peered":"2024-05-15T23:19:47.325371+0000","last_unstale":"2024-05-15T23:24:00.584315+0000","last_undegraded":"2024-05-15T23:24:00.584315+0000","last_fullsized":"2024-05-15T23:24:00.584315+0000","mapping_epoch":374,"log_start":"396'6900","ondisk_log_start":"396'6900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149159936,"num_objects":2276,"num_object_clones":0,"num_object_copies":4552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16936,"num_write_kb":614784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"459'16938","reported_seq":31954,"reported_epoch":459,"state":"active+recovering","last_fresh":"2024-05-15T23:24:01.049891+0000","last_change":"2024-05-15T23:21:09.568311+0000","last_active":"2024-05-15T23:24:01.049891+0000","last_peered":"2024-05-15T23:24:01.049891+0000","last_clean":"2024-05-15T23:21:08.023957+0000","last_became_active":"2024-05-15T23:21:09.454639+0000","last_became_peered":"2024-05-15T23:21:09.454639+0000","last_unstale":"2024-05-15T23:24:01.049891+0000","last_undegraded":"2024-05-15T23:24:01.049891+0000","last_fullsized":"2024-05-15T23:24:01.049891+0000","mapping_epoch":407,"log_start":"396'6900","ondisk_log_start":"396'6900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145227776,"num_objects":2216,"num_object_clones":0,"num_object_copies":4432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1153,"num_objects_unfound":0,"num_objects_dirty":2216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16938,"num_write_kb":612928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7417,"num_bytes_recovered":486080512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1153},{"shards":"7,11","objects":1063}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"460'17018","reported_seq":17200,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-05-15T23:24:02.162377+0000","last_change":"2024-05-15T23:19:47.304832+0000","last_active":"2024-05-15T23:24:02.162377+0000","last_peered":"2024-05-15T23:24:02.162377+0000","last_clean":"2024-05-15T23:24:02.162377+0000","last_became_active":"2024-05-15T23:19:47.304547+0000","last_became_peered":"2024-05-15T23:19:47.304547+0000","last_unstale":"2024-05-15T23:24:02.162377+0000","last_undegraded":"2024-05-15T23:24:02.162377+0000","last_fullsized":"2024-05-15T23:24:02.162377+0000","mapping_epoch":374,"log_start":"398'7000","ondisk_log_start":"398'7000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143261696,"num_objects":2186,"num_object_clones":0,"num_object_copies":4372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17018,"num_write_kb":614528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"460'16853","reported_seq":17035,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-05-15T23:24:02.652463+0000","last_change":"2024-05-15T23:19:47.306634+0000","last_active":"2024-05-15T23:24:02.652463+0000","last_peered":"2024-05-15T23:24:02.652463+0000","last_clean":"2024-05-15T23:24:02.652463+0000","last_became_active":"2024-05-15T23:19:47.306331+0000","last_became_peered":"2024-05-15T23:19:47.306331+0000","last_unstale":"2024-05-15T23:24:02.652463+0000","last_undegraded":"2024-05-15T23:24:02.652463+0000","last_fullsized":"2024-05-15T23:24:02.652463+0000","mapping_epoch":374,"log_start":"396'6800","ondisk_log_start":"396'6800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144375808,"num_objects":2203,"num_object_clones":0,"num_object_copies":4406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16853,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"460'17110","reported_seq":17292,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-05-15T23:24:02.666838+0000","last_change":"2024-05-15T23:19:47.306611+0000","last_active":"2024-05-15T23:24:02.666838+0000","last_peered":"2024-05-15T23:24:02.666838+0000","last_clean":"2024-05-15T23:24:02.666838+0000","last_became_active":"2024-05-15T23:19:47.306349+0000","last_became_peered":"2024-05-15T23:19:47.306349+0000","last_unstale":"2024-05-15T23:24:02.666838+0000","last_undegraded":"2024-05-15T23:24:02.666838+0000","last_fullsized":"2024-05-15T23:24:02.666838+0000","mapping_epoch":374,"log_start":"398'7100","ondisk_log_start":"398'7100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144179200,"num_objects":2200,"num_object_clones":0,"num_object_copies":4400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17110,"num_write_kb":617920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"459'17066","reported_seq":17512,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-05-15T23:24:00.796028+0000","last_change":"2024-05-15T23:23:50.810750+0000","last_active":"2024-05-15T23:24:00.796028+0000","last_peered":"2024-05-15T23:24:00.796028+0000","last_clean":"2024-05-15T23:24:00.796028+0000","last_became_active":"2024-05-15T23:19:47.305685+0000","last_became_peered":"2024-05-15T23:19:47.305685+0000","last_unstale":"2024-05-15T23:24:00.796028+0000","last_undegraded":"2024-05-15T23:24:00.796028+0000","last_fullsized":"2024-05-15T23:24:00.796028+0000","mapping_epoch":374,"log_start":"398'7000","ondisk_log_start":"398'7000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'16478","last_scrub_stamp":"2024-05-15T23:23:50.810636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:50.810636+0000","objects_scrubbed":2939,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:20.026294+0000","scrub_duration":608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152698880,"num_objects":2330,"num_object_clones":0,"num_object_copies":4660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17066,"num_write_kb":620672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"460'17099","reported_seq":32185,"reported_epoch":460,"state":"active+recovering","last_fresh":"2024-05-15T23:24:02.657023+0000","last_change":"2024-05-15T23:21:09.470461+0000","last_active":"2024-05-15T23:24:02.657023+0000","last_peered":"2024-05-15T23:24:02.657023+0000","last_clean":"2024-05-15T23:21:07.989871+0000","last_became_active":"2024-05-15T23:21:09.400731+0000","last_became_peered":"2024-05-15T23:21:09.400731+0000","last_unstale":"2024-05-15T23:24:02.657023+0000","last_undegraded":"2024-05-15T23:24:02.657023+0000","last_fullsized":"2024-05-15T23:24:02.657023+0000","mapping_epoch":407,"log_start":"398'7000","ondisk_log_start":"398'7000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143982592,"num_objects":2197,"num_object_clones":0,"num_object_copies":4394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1239,"num_objects_unfound":0,"num_objects_dirty":2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17099,"num_write_kb":617472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7451,"num_bytes_recovered":488308736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"3,5","objects":958},{"shards":"5,6","objects":1239}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"459'16955","reported_seq":31977,"reported_epoch":459,"state":"active+recovering","last_fresh":"2024-05-15T23:24:00.613983+0000","last_change":"2024-05-15T23:21:09.275620+0000","last_active":"2024-05-15T23:24:00.613983+0000","last_peered":"2024-05-15T23:24:00.613983+0000","last_clean":"2024-05-15T23:21:07.993692+0000","last_became_active":"2024-05-15T23:21:09.221797+0000","last_became_peered":"2024-05-15T23:21:09.221797+0000","last_unstale":"2024-05-15T23:24:00.613983+0000","last_undegraded":"2024-05-15T23:24:00.613983+0000","last_fullsized":"2024-05-15T23:24:00.613983+0000","mapping_epoch":407,"log_start":"396'6900","ondisk_log_start":"396'6900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145162240,"num_objects":2215,"num_object_clones":0,"num_object_copies":4430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1209,"num_objects_unfound":0,"num_objects_dirty":2215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16955,"num_write_kb":613440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7420,"num_bytes_recovered":486277120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":1209},{"shards":"7,10","objects":1006}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.1","version":"460'16657","reported_seq":17049,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-05-15T23:24:04.892905+0000","last_change":"2024-05-15T23:23:50.798735+0000","last_active":"2024-05-15T23:24:04.892905+0000","last_peered":"2024-05-15T23:24:04.892905+0000","last_clean":"2024-05-15T23:24:04.892905+0000","last_became_active":"2024-05-15T23:19:47.516865+0000","last_became_peered":"2024-05-15T23:19:47.516865+0000","last_unstale":"2024-05-15T23:24:04.892905+0000","last_undegraded":"2024-05-15T23:24:04.892905+0000","last_fullsized":"2024-05-15T23:24:04.892905+0000","mapping_epoch":374,"log_start":"396'6600","ondisk_log_start":"396'6600","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'15864","last_scrub_stamp":"2024-05-15T23:23:50.798612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:50.798612+0000","objects_scrubbed":2807,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:56.167477+0000","scrub_duration":584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131137536,"num_objects":2001,"num_object_clones":0,"num_object_copies":4002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16657,"num_write_kb":597056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"459'16941","reported_seq":17122,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-05-15T23:24:00.592441+0000","last_change":"2024-05-15T23:19:47.305723+0000","last_active":"2024-05-15T23:24:00.592441+0000","last_peered":"2024-05-15T23:24:00.592441+0000","last_clean":"2024-05-15T23:24:00.592441+0000","last_became_active":"2024-05-15T23:19:47.305459+0000","last_became_peered":"2024-05-15T23:19:47.305459+0000","last_unstale":"2024-05-15T23:24:00.592441+0000","last_undegraded":"2024-05-15T23:24:00.592441+0000","last_fullsized":"2024-05-15T23:24:00.592441+0000","mapping_epoch":374,"log_start":"397'6900","ondisk_log_start":"397'6900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153026560,"num_objects":2335,"num_object_clones":0,"num_object_copies":4670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16941,"num_write_kb":616832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"459'17065","reported_seq":17246,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-05-15T23:24:00.593786+0000","last_change":"2024-05-15T23:19:47.307499+0000","last_active":"2024-05-15T23:24:00.593786+0000","last_peered":"2024-05-15T23:24:00.593786+0000","last_clean":"2024-05-15T23:24:00.593786+0000","last_became_active":"2024-05-15T23:19:47.307149+0000","last_became_peered":"2024-05-15T23:19:47.307149+0000","last_unstale":"2024-05-15T23:24:00.593786+0000","last_undegraded":"2024-05-15T23:24:00.593786+0000","last_fullsized":"2024-05-15T23:24:00.593786+0000","mapping_epoch":374,"log_start":"398'7000","ondisk_log_start":"398'7000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156041216,"num_objects":2381,"num_object_clones":0,"num_object_copies":4762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17065,"num_write_kb":622272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":957,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-05-15T23:23:52.070867+0000","last_change":"2024-05-15T23:20:05.905241+0000","last_active":"2024-05-15T23:23:52.070867+0000","last_peered":"2024-05-15T23:23:52.070867+0000","last_clean":"2024-05-15T23:23:52.070867+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:23:52.070867+0000","last_undegraded":"2024-05-15T23:23:52.070867+0000","last_fullsized":"2024-05-15T23:23:52.070867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:20:05.905105+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:20:05.905105+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2353594392,"num_objects":35914,"num_object_clones":0,"num_object_copies":71828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3601,"num_objects_unfound":0,"num_objects_dirty":35914,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":271202,"num_write_kb":9827649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22288,"num_bytes_recovered":1460666368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6115368960,"data_stored":6115360816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351219,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":886928,"kb_used_data":354568,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92858224,"statfs":{"total":95995035648,"available":95086821376,"internally_reserved":0,"allocated":363077632,"data_stored":362582898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":545126852},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580152,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1039364,"kb_used_data":747132,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92705788,"statfs":{"total":95995035648,"available":94930726912,"internally_reserved":0,"allocated":765063168,"data_stored":764576078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":299170240},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776384,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":676036,"kb_used_data":297724,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93069116,"statfs":{"total":95995035648,"available":95302774784,"internally_reserved":0,"allocated":304869376,"data_stored":304382286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":387316160},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005317,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":449040,"kb_used_data":265096,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93296112,"statfs":{"total":95995035648,"available":95535218688,"internally_reserved":0,"allocated":271458304,"data_stored":270965210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266952,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":733836,"kb_used_data":355588,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93011316,"statfs":{"total":95995035648,"available":95243587584,"internally_reserved":0,"allocated":364122112,"data_stored":363631470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":387316170},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495887,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575144,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1113284,"kb_used_data":604540,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92631868,"statfs":{"total":95995035648,"available":94855032832,"internally_reserved":0,"allocated":619048960,"data_stored":618561870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":520944070},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016553,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483243,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149983,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":665284,"kb_used_data":206524,"kb_used_omap":1,"kb_used_meta":458750,"kb_avail":93079868,"statfs":{"total":95995035648,"available":95313784832,"internally_reserved":0,"allocated":211480576,"data_stored":210993486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":469760455},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986456,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":931792,"kb_used_data":535432,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92813360,"statfs":{"total":95995035648,"available":95040880640,"internally_reserved":0,"allocated":548282368,"data_stored":547789274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790228,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1367316,"kb_used_data":997388,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92377836,"statfs":{"total":95995035648,"available":94594904064,"internally_reserved":0,"allocated":1021325312,"data_stored":1020828146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":606208000,"data_stored":606208000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209387520,"data_stored":209387520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":538509312,"data_stored":538509312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1016074240,"data_stored":1016070168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1696661504,"data_stored":1696661504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":362020864,"data_stored":362020864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":270401536,"data_stored":270401536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":302710784,"data_stored":302710784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":753074176,"data_stored":753074176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":360321024,"data_stored":360316952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:24:08.035 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:08.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:07 smithi084 ceph-mon[39784]: from='client.16293 v1:172.21.15.84:0/3199392601' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:24:08.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:07 smithi084 ceph-mon[39784]: pgmap v1244: 17 pgs: 3 active+recovering, 14 active+clean; 2.0 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 924 op/s; 2970/66612 objects misplaced (4.459%); 11 MiB/s, 183 objects/s recovering 2024-05-15T23:24:08.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:07 smithi084 ceph-mon[39784]: osdmap e461: 12 total, 12 up, 9 in 2024-05-15T23:24:08.067 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:08.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:07 smithi088 ceph-mon[44699]: from='client.16293 v1:172.21.15.84:0/3199392601' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:24:08.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:07 smithi088 ceph-mon[44699]: pgmap v1244: 17 pgs: 3 active+recovering, 14 active+clean; 2.0 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 924 op/s; 2970/66612 objects misplaced (4.459%); 11 MiB/s, 183 objects/s recovering 2024-05-15T23:24:08.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:07 smithi088 ceph-mon[44699]: osdmap e461: 12 total, 12 up, 9 in 2024-05-15T23:24:08.219 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:08.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:08.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:08.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:24:08.368 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:08.800 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.800 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.800 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:08.924 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:24:08.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:08.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:24:09.029 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:24:09.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:09.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:09.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:09.057 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:09.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:09.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:09.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:09.510 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:24:09.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:09.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:24:09.589 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:24:09.589 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:24:09.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.589 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25840.0:278903 5.e e461/374 5:7369ca37:::benchmark_data_smithi090_116470_object125343:head v 461'17175, mlcod=461'17172)", 2024-05-15T23:24:09.589 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:09.573335+0000", 2024-05-15T23:24:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0093766279999999997, 2024-05-15T23:24:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094070649999999992, 2024-05-15T23:24:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:24:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.573335+0000", 2024-05-15T23:24:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.573335+0000", 2024-05-15T23:24:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.573336+0000", 2024-05-15T23:24:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.175e-06 2024-05-15T23:24:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.573342+0000", 2024-05-15T23:24:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7559999999999996e-06 2024-05-15T23:24:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.573351+0000", 2024-05-15T23:24:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6969999999999999e-06 2024-05-15T23:24:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.573357+0000", 2024-05-15T23:24:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5980000000000002e-06 2024-05-15T23:24:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.577737+0000", 2024-05-15T23:24:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043802750000000003 2024-05-15T23:24:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.577761+0000", 2024-05-15T23:24:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3238e-05 2024-05-15T23:24:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25840.0:278906 5.e e461/374 5:7e93e508:::benchmark_data_smithi090_116470_object125346:head v 461'17176, mlcod=461'17172)", 2024-05-15T23:24:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:09.578816+0000", 2024-05-15T23:24:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.003895262, 2024-05-15T23:24:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039924740000000002, 2024-05-15T23:24:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:24:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.578816+0000", 2024-05-15T23:24:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.578816+0000", 2024-05-15T23:24:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.578818+0000", 2024-05-15T23:24:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3909999999999999e-06 2024-05-15T23:24:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.578825+0000", 2024-05-15T23:24:09.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7710000000000006e-06 2024-05-15T23:24:09.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.578834+0000", 2024-05-15T23:24:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2024-05-15T23:24:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.578840+0000", 2024-05-15T23:24:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2024-05-15T23:24:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:09.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.582227+0000", 2024-05-15T23:24:09.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033868029999999999 2024-05-15T23:24:09.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:09.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.582244+0000", 2024-05-15T23:24:09.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6628000000000001e-05 2024-05-15T23:24:09.599 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:09.599 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:09.599 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:09.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.599 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25840.0:278896 5.a e461/374 5:52c80713:::benchmark_data_smithi090_116470_object125336:head v 461'17451, mlcod=461'17448)", 2024-05-15T23:24:09.599 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:09.581220+0000", 2024-05-15T23:24:09.600 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.001491107, 2024-05-15T23:24:09.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00164846, 2024-05-15T23:24:09.600 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:09.600 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:09.600 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:24:09.600 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:09.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:09.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.581220+0000", 2024-05-15T23:24:09.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:09.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:09.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.581220+0000", 2024-05-15T23:24:09.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:09.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:09.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.581222+0000", 2024-05-15T23:24:09.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.683e-06 2024-05-15T23:24:09.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:09.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.581231+0000", 2024-05-15T23:24:09.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3389999999999999e-06 2024-05-15T23:24:09.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:09.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.581247+0000", 2024-05-15T23:24:09.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.521e-05 2024-05-15T23:24:09.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:09.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.581255+0000", 2024-05-15T23:24:09.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8779999999999998e-06 2024-05-15T23:24:09.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:09.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.582457+0000", 2024-05-15T23:24:09.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001202657 2024-05-15T23:24:09.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:09.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.582477+0000", 2024-05-15T23:24:09.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9592000000000001e-05 2024-05-15T23:24:09.603 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:09.603 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:09.603 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:09.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.603 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25840.0:278900 5.a e461/374)", 2024-05-15T23:24:09.603 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:09.581273+0000", 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0014382469999999999, 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00165044, 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.581273+0000", 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.581273+0000", 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:09.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:09.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.581274+0000", 2024-05-15T23:24:09.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8100000000000003e-07 2024-05-15T23:24:09.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:09.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.581278+0000", 2024-05-15T23:24:09.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5650000000000003e-06 2024-05-15T23:24:09.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:09.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.581283+0000", 2024-05-15T23:24:09.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6460000000000001e-06 2024-05-15T23:24:09.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:09.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.581287+0000", 2024-05-15T23:24:09.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.16e-06 2024-05-15T23:24:09.606 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:09.606 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:09.606 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:09.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.606 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.25840.0:278913 5.a e461/374)", 2024-05-15T23:24:09.606 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:09.582746+0000", 2024-05-15T23:24:09.606 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4294967295.9999657, 2024-05-15T23:24:09.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000225238, 2024-05-15T23:24:09.607 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:09.607 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:09.607 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:24:09.607 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:09.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:09.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.582746+0000", 2024-05-15T23:24:09.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:09.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:09.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.582746+0000", 2024-05-15T23:24:09.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:09.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:09.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.582747+0000", 2024-05-15T23:24:09.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.341e-06 2024-05-15T23:24:09.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:09.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.582755+0000", 2024-05-15T23:24:09.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2024-05-15T23:24:09.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:09.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.582764+0000", 2024-05-15T23:24:09.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1649999999999995e-06 2024-05-15T23:24:09.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:09.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:09.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:09.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:09.582776+0000", 2024-05-15T23:24:09.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2082000000000001e-05 2024-05-15T23:24:09.609 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:09.609 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:09.609 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:09.609 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:09.609 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-05-15T23:24:09.609 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 5 2024-05-15T23:24:09.610 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:24:09.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:24:09.657 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:09.828 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:24:09.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:09.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:24:09.934 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:24:09.949 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:09 smithi090 ceph-mon[44434]: pgmap v1246: 17 pgs: 3 active+recovering, 14 active+clean; 1.9 GiB data, 7.1 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 857 op/s; 2616/62884 objects misplaced (4.160%); 12 MiB/s, 199 objects/s recovering 2024-05-15T23:24:09.990 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:10.033 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:24:10.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:10.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:24:10.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:09 smithi088 ceph-mon[44699]: pgmap v1246: 17 pgs: 3 active+recovering, 14 active+clean; 1.9 GiB data, 7.1 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 857 op/s; 2616/62884 objects misplaced (4.160%); 12 MiB/s, 199 objects/s recovering 2024-05-15T23:24:10.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:09 smithi084 ceph-mon[39784]: pgmap v1246: 17 pgs: 3 active+recovering, 14 active+clean; 1.9 GiB data, 7.1 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 857 op/s; 2616/62884 objects misplaced (4.160%); 12 MiB/s, 199 objects/s recovering 2024-05-15T23:24:10.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:10.184 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:10.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:24:10.833 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:24:10.852 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:10.856 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:10 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2128270598' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:24:10.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:10.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:10.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T23:24:10.886 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:10.960 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:24:10.989 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:11.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:24:11.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:10 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2128270598' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:24:11.189 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:24:11.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:11.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:24:11.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:24:11.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:10 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2128270598' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:24:11.331 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:11.381 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:24:11.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:11.394 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:24:11.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:24:11.543 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:11.578 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:24:11.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:11.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:24:11.612 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:24:11.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:11.740 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:11.817 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:11 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:24:11.817 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:11 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2128270598' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:24:11.817 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:11 smithi088 ceph-mon[44699]: osdmap e462: 12 total, 12 up, 9 in 2024-05-15T23:24:11.817 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:11 smithi088 ceph-mon[44699]: pgmap v1248: 17 pgs: 3 active+recovering, 14 active+clean; 1.7 GiB data, 6.8 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 943 op/s; 2314/57288 objects misplaced (4.039%); 10 MiB/s, 160 objects/s recovering 2024-05-15T23:24:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:11 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:24:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:11 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2128270598' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:24:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:11 smithi090 ceph-mon[44434]: osdmap e462: 12 total, 12 up, 9 in 2024-05-15T23:24:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:11 smithi090 ceph-mon[44434]: pgmap v1248: 17 pgs: 3 active+recovering, 14 active+clean; 1.7 GiB data, 6.8 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 943 op/s; 2314/57288 objects misplaced (4.039%); 10 MiB/s, 160 objects/s recovering 2024-05-15T23:24:12.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:11 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:24:12.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:11 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2128270598' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:24:12.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:11 smithi084 ceph-mon[39784]: osdmap e462: 12 total, 12 up, 9 in 2024-05-15T23:24:12.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:11 smithi084 ceph-mon[39784]: pgmap v1248: 17 pgs: 3 active+recovering, 14 active+clean; 1.7 GiB data, 6.8 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 943 op/s; 2314/57288 objects misplaced (4.039%); 10 MiB/s, 160 objects/s recovering 2024-05-15T23:24:13.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:13 smithi084 ceph-mon[39784]: osdmap e463: 12 total, 12 up, 9 in 2024-05-15T23:24:13.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:13 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:24:13.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:13 smithi084 ceph-mon[39784]: pgmap v1250: 17 pgs: 3 active+recovering, 14 active+clean; 1.6 GiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s; 1550/52888 objects misplaced (2.931%); 15 MiB/s, 236 objects/s recovering 2024-05-15T23:24:13.426 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:13.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:13 smithi090 ceph-mon[44434]: osdmap e463: 12 total, 12 up, 9 in 2024-05-15T23:24:13.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:13 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:24:13.559 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:13 smithi090 ceph-mon[44434]: pgmap v1250: 17 pgs: 3 active+recovering, 14 active+clean; 1.6 GiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s; 1550/52888 objects misplaced (2.931%); 15 MiB/s, 236 objects/s recovering 2024-05-15T23:24:13.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:13 smithi088 ceph-mon[44699]: osdmap e463: 12 total, 12 up, 9 in 2024-05-15T23:24:13.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:13 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:24:13.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:13 smithi088 ceph-mon[44699]: pgmap v1250: 17 pgs: 3 active+recovering, 14 active+clean; 1.6 GiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s; 1550/52888 objects misplaced (2.931%); 15 MiB/s, 236 objects/s recovering 2024-05-15T23:24:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:13.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:13.939 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:14.040 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:24:14.078 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:14.109 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:24:14.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:14.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:24:14.213 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:24:14.275 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:14.306 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:24:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:14.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:24:14.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:24:14.444 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:14.897 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:24:14.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:14.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:24:14.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:24:15.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:24:15.038 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:15.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:15 smithi084 ceph-mon[39784]: pgmap v1251: 17 pgs: 3 active+recovering, 14 active+clean; 1.6 GiB data, 6.5 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 900 op/s; 1550/52284 objects misplaced (2.965%); 11 MiB/s, 178 objects/s recovering 2024-05-15T23:24:15.478 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:15 smithi090 ceph-mon[44434]: pgmap v1251: 17 pgs: 3 active+recovering, 14 active+clean; 1.6 GiB data, 6.5 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 900 op/s; 1550/52284 objects misplaced (2.965%); 11 MiB/s, 178 objects/s recovering 2024-05-15T23:24:15.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:15 smithi088 ceph-mon[44699]: pgmap v1251: 17 pgs: 3 active+recovering, 14 active+clean; 1.6 GiB data, 6.5 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 900 op/s; 1550/52284 objects misplaced (2.965%); 11 MiB/s, 178 objects/s recovering 2024-05-15T23:24:15.668 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:24:15.668 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:24:15.668 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:24:15.668 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:24:15.668 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:24:15.668 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:24:15.669 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:24:15.669 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:24:15.669 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:24:15.669 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:24:15.669 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:24:15.669 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:24:15.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:24:15.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:24:15.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:24:15.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:24:15.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:24:15.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:24:15.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:24:15.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:24:15.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:24:15.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:24:15.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:24:15.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:24:15.893 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:24:15.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:15.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:24:15.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:24:16.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:16 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3571735971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:16.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:16.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:16.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:16.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:16.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:16.407 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:16 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3571735971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:16.407 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:16.407 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:16.407 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:16.407 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:16.407 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:16.410 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:16.446 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:24:16.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:16.453 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:24:16.554 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:24:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:16 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3571735971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:16.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:16.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:16.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:16.582 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:16.711 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:24:16.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:16.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:24:16.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:24:16.878 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:17.020 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:17.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.101 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:17 smithi084 ceph-mon[39784]: pgmap v1252: 17 pgs: 3 active+recovering, 14 active+clean; 1.4 GiB data, 6.2 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 829/46618 objects misplaced (1.778%); 14 MiB/s, 223 objects/s recovering 2024-05-15T23:24:17.254 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:24:17.259 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:24:17.275 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:24:17.276 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:24:17.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:17 smithi090 ceph-mon[44434]: pgmap v1252: 17 pgs: 3 active+recovering, 14 active+clean; 1.4 GiB data, 6.2 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 829/46618 objects misplaced (1.778%); 14 MiB/s, 223 objects/s recovering 2024-05-15T23:24:17.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:17.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:17 smithi088 ceph-mon[44699]: pgmap v1252: 17 pgs: 3 active+recovering, 14 active+clean; 1.4 GiB data, 6.2 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 829/46618 objects misplaced (1.778%); 14 MiB/s, 223 objects/s recovering 2024-05-15T23:24:17.861 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:24:17.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:17.867 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:24:17.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:24:18.000 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:18.021 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:24:18.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:18.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:24:18.127 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:18.155 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:18.195 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:18.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:18.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:18.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:18 smithi090 ceph-mon[44434]: from='client.26233 v1:172.21.15.84:0/3313282890' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:24:18.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:24:18.352 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:18.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:18 smithi084 ceph-mon[39784]: from='client.26233 v1:172.21.15.84:0/3313282890' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:24:18.388 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:24:18.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:18.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:24:18.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:24:18.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T23:24:18.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:18.548 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:18 smithi088 ceph-mon[44699]: from='client.26233 v1:172.21.15.84:0/3313282890' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:24:18.605 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:18.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:18.714 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:24:18.746 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:18.794 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:24:18.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:18.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:24:18.866 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:24:18.907 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:24:18.942 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:18.972 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:24:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:18.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:24:19.080 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:24:19.112 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:19.152 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1252,"stamp":"2024-05-15T23:24:17.013995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1527972408,"num_objects":23309,"num_object_clones":0,"num_object_copies":46618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":829,"num_objects_unfound":0,"num_objects_dirty":23309,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":283932,"num_write_kb":9829799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25066,"num_bytes_recovered":1643918944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160903,"ondisk_log_size":160903,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":6485932,"kb_used_data":2967832,"kb_used_omap":14,"kb_used_meta":3518001,"kb_avail":837220436,"statfs":{"total":863955320832,"available":857313726464,"internally_reserved":0,"allocated":3039059968,"data_stored":3034449450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14358,"internal_metadata":3602434026},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-533004288,"num_objects":-8133,"num_object_clones":0,"num_object_copies":-16266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1787,"num_objects_unfound":0,"num_objects_dirty":-8133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1787,"num_bytes_recovered":117112832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001467"},"pg_stats":[{"pgid":"5.f","version":"463'17490","reported_seq":17678,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T23:24:12.654754+0000","last_change":"2024-05-15T23:19:47.305945+0000","last_active":"2024-05-15T23:24:12.654754+0000","last_peered":"2024-05-15T23:24:12.654754+0000","last_clean":"2024-05-15T23:24:12.654754+0000","last_became_active":"2024-05-15T23:19:47.305689+0000","last_became_peered":"2024-05-15T23:19:47.305689+0000","last_unstale":"2024-05-15T23:24:12.654754+0000","last_undegraded":"2024-05-15T23:24:12.654754+0000","last_fullsized":"2024-05-15T23:24:12.654754+0000","mapping_epoch":374,"log_start":"399'7400","ondisk_log_start":"399'7400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102891520,"num_objects":1570,"num_object_clones":0,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17490,"num_write_kb":609920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"463'17546","reported_seq":17941,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T23:24:15.622718+0000","last_change":"2024-05-15T23:23:53.872535+0000","last_active":"2024-05-15T23:24:15.622718+0000","last_peered":"2024-05-15T23:24:15.622718+0000","last_clean":"2024-05-15T23:24:15.622718+0000","last_became_active":"2024-05-15T23:19:47.325414+0000","last_became_peered":"2024-05-15T23:19:47.325414+0000","last_unstale":"2024-05-15T23:24:15.622718+0000","last_undegraded":"2024-05-15T23:24:15.622718+0000","last_fullsized":"2024-05-15T23:24:15.622718+0000","mapping_epoch":374,"log_start":"399'7500","ondisk_log_start":"399'7500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'16258","last_scrub_stamp":"2024-05-15T23:23:53.872309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:53.872309+0000","objects_scrubbed":2678,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:15.839794+0000","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90177536,"num_objects":1376,"num_object_clones":0,"num_object_copies":2752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17546,"num_write_kb":605504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"463'17828","reported_seq":18017,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T23:24:12.181688+0000","last_change":"2024-05-15T23:19:47.306160+0000","last_active":"2024-05-15T23:24:12.181688+0000","last_peered":"2024-05-15T23:24:12.181688+0000","last_clean":"2024-05-15T23:24:12.181688+0000","last_became_active":"2024-05-15T23:19:47.305860+0000","last_became_peered":"2024-05-15T23:19:47.305860+0000","last_unstale":"2024-05-15T23:24:12.181688+0000","last_undegraded":"2024-05-15T23:24:12.181688+0000","last_fullsized":"2024-05-15T23:24:12.181688+0000","mapping_epoch":374,"log_start":"399'7800","ondisk_log_start":"399'7800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105971736,"num_objects":1618,"num_object_clones":0,"num_object_copies":3236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1618,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17828,"num_write_kb":622209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"463'17941","reported_seq":18292,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T23:24:15.828458+0000","last_change":"2024-05-15T23:22:51.995640+0000","last_active":"2024-05-15T23:24:15.828458+0000","last_peered":"2024-05-15T23:24:15.828458+0000","last_clean":"2024-05-15T23:24:15.828458+0000","last_became_active":"2024-05-15T23:19:47.306168+0000","last_became_peered":"2024-05-15T23:19:47.306168+0000","last_unstale":"2024-05-15T23:24:15.828458+0000","last_undegraded":"2024-05-15T23:24:15.828458+0000","last_fullsized":"2024-05-15T23:24:15.828458+0000","mapping_epoch":374,"log_start":"400'7900","ondisk_log_start":"400'7900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'13335","last_scrub_stamp":"2024-05-15T23:22:51.995473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:51.995473+0000","objects_scrubbed":6018,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90243072,"num_objects":1377,"num_object_clones":0,"num_object_copies":2754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17941,"num_write_kb":618176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"463'17769","reported_seq":18123,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T23:24:15.588221+0000","last_change":"2024-05-15T23:22:48.338084+0000","last_active":"2024-05-15T23:24:15.588221+0000","last_peered":"2024-05-15T23:24:15.588221+0000","last_clean":"2024-05-15T23:24:15.588221+0000","last_became_active":"2024-05-15T23:19:47.307061+0000","last_became_peered":"2024-05-15T23:19:47.307061+0000","last_unstale":"2024-05-15T23:24:15.588221+0000","last_undegraded":"2024-05-15T23:24:15.588221+0000","last_fullsized":"2024-05-15T23:24:15.588221+0000","mapping_epoch":374,"log_start":"400'7700","ondisk_log_start":"400'7700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'13172","last_scrub_stamp":"2024-05-15T23:22:48.337866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:48.337866+0000","objects_scrubbed":6060,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:17.550238+0000","scrub_duration":1844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93257728,"num_objects":1423,"num_object_clones":0,"num_object_copies":2846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17769,"num_write_kb":614144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"463'17832","reported_seq":18021,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T23:24:15.604607+0000","last_change":"2024-05-15T23:19:47.325657+0000","last_active":"2024-05-15T23:24:15.604607+0000","last_peered":"2024-05-15T23:24:15.604607+0000","last_clean":"2024-05-15T23:24:15.604607+0000","last_became_active":"2024-05-15T23:19:47.325371+0000","last_became_peered":"2024-05-15T23:19:47.325371+0000","last_unstale":"2024-05-15T23:24:15.604607+0000","last_undegraded":"2024-05-15T23:24:15.604607+0000","last_fullsized":"2024-05-15T23:24:15.604607+0000","mapping_epoch":374,"log_start":"400'7800","ondisk_log_start":"400'7800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90439680,"num_objects":1380,"num_object_clones":0,"num_object_copies":2760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17832,"num_write_kb":614784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"463'17827","reported_seq":35011,"reported_epoch":463,"state":"active+recovering","last_fresh":"2024-05-15T23:24:16.071122+0000","last_change":"2024-05-15T23:21:09.568311+0000","last_active":"2024-05-15T23:24:16.071122+0000","last_peered":"2024-05-15T23:24:16.071122+0000","last_clean":"2024-05-15T23:21:08.023957+0000","last_became_active":"2024-05-15T23:21:09.454639+0000","last_became_peered":"2024-05-15T23:21:09.454639+0000","last_unstale":"2024-05-15T23:24:16.071122+0000","last_undegraded":"2024-05-15T23:24:16.071122+0000","last_fullsized":"2024-05-15T23:24:16.071122+0000","mapping_epoch":407,"log_start":"400'7800","ondisk_log_start":"400'7800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86966272,"num_objects":1327,"num_object_clones":0,"num_object_copies":2654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":74,"num_objects_unfound":0,"num_objects_dirty":1327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17827,"num_write_kb":612928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8496,"num_bytes_recovered":556793856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":74},{"shards":"7,11","objects":1253}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"463'17623","reported_seq":17812,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T23:24:12.196798+0000","last_change":"2024-05-15T23:19:47.304832+0000","last_active":"2024-05-15T23:24:12.196798+0000","last_peered":"2024-05-15T23:24:12.196798+0000","last_clean":"2024-05-15T23:24:12.196798+0000","last_became_active":"2024-05-15T23:19:47.304547+0000","last_became_peered":"2024-05-15T23:19:47.304547+0000","last_unstale":"2024-05-15T23:24:12.196798+0000","last_undegraded":"2024-05-15T23:24:12.196798+0000","last_fullsized":"2024-05-15T23:24:12.196798+0000","mapping_epoch":374,"log_start":"399'7600","ondisk_log_start":"399'7600","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103612416,"num_objects":1581,"num_object_clones":0,"num_object_copies":3162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17623,"num_write_kb":614528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"463'17471","reported_seq":17660,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T23:24:12.647691+0000","last_change":"2024-05-15T23:19:47.306634+0000","last_active":"2024-05-15T23:24:12.647691+0000","last_peered":"2024-05-15T23:24:12.647691+0000","last_clean":"2024-05-15T23:24:12.647691+0000","last_became_active":"2024-05-15T23:19:47.306331+0000","last_became_peered":"2024-05-15T23:19:47.306331+0000","last_unstale":"2024-05-15T23:24:12.647691+0000","last_undegraded":"2024-05-15T23:24:12.647691+0000","last_fullsized":"2024-05-15T23:24:12.647691+0000","mapping_epoch":374,"log_start":"398'7400","ondisk_log_start":"398'7400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103874560,"num_objects":1585,"num_object_clones":0,"num_object_copies":3170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17471,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"463'17725","reported_seq":17914,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T23:24:12.665839+0000","last_change":"2024-05-15T23:19:47.306611+0000","last_active":"2024-05-15T23:24:12.665839+0000","last_peered":"2024-05-15T23:24:12.665839+0000","last_clean":"2024-05-15T23:24:12.665839+0000","last_became_active":"2024-05-15T23:19:47.306349+0000","last_became_peered":"2024-05-15T23:19:47.306349+0000","last_unstale":"2024-05-15T23:24:12.665839+0000","last_undegraded":"2024-05-15T23:24:12.665839+0000","last_fullsized":"2024-05-15T23:24:12.665839+0000","mapping_epoch":374,"log_start":"399'7700","ondisk_log_start":"399'7700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103874560,"num_objects":1585,"num_object_clones":0,"num_object_copies":3170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17725,"num_write_kb":617920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"463'17975","reported_seq":18431,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T23:24:15.846867+0000","last_change":"2024-05-15T23:23:50.810750+0000","last_active":"2024-05-15T23:24:15.846867+0000","last_peered":"2024-05-15T23:24:15.846867+0000","last_clean":"2024-05-15T23:24:15.846867+0000","last_became_active":"2024-05-15T23:19:47.305685+0000","last_became_peered":"2024-05-15T23:19:47.305685+0000","last_unstale":"2024-05-15T23:24:15.846867+0000","last_undegraded":"2024-05-15T23:24:15.846867+0000","last_fullsized":"2024-05-15T23:24:15.846867+0000","mapping_epoch":374,"log_start":"400'7900","ondisk_log_start":"400'7900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'16478","last_scrub_stamp":"2024-05-15T23:23:50.810636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:50.810636+0000","objects_scrubbed":2939,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:20.026294+0000","scrub_duration":608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93126656,"num_objects":1421,"num_object_clones":0,"num_object_copies":2842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17975,"num_write_kb":620672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"463'17739","reported_seq":34368,"reported_epoch":463,"state":"active+recovering","last_fresh":"2024-05-15T23:24:12.707807+0000","last_change":"2024-05-15T23:21:09.470461+0000","last_active":"2024-05-15T23:24:12.707807+0000","last_peered":"2024-05-15T23:24:12.707807+0000","last_clean":"2024-05-15T23:21:07.989871+0000","last_became_active":"2024-05-15T23:21:09.400731+0000","last_became_peered":"2024-05-15T23:21:09.400731+0000","last_unstale":"2024-05-15T23:24:12.707807+0000","last_undegraded":"2024-05-15T23:24:12.707807+0000","last_fullsized":"2024-05-15T23:24:12.707807+0000","mapping_epoch":407,"log_start":"399'7700","ondisk_log_start":"399'7700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102039552,"num_objects":1557,"num_object_clones":0,"num_object_copies":3114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":471,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17739,"num_write_kb":617472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8219,"num_bytes_recovered":538640384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"3,5","objects":1086},{"shards":"5,6","objects":471}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"463'17872","reported_seq":34754,"reported_epoch":463,"state":"active+recovering","last_fresh":"2024-05-15T23:24:15.634130+0000","last_change":"2024-05-15T23:21:09.275620+0000","last_active":"2024-05-15T23:24:15.634130+0000","last_peered":"2024-05-15T23:24:15.634130+0000","last_clean":"2024-05-15T23:21:07.993692+0000","last_became_active":"2024-05-15T23:21:09.221797+0000","last_became_peered":"2024-05-15T23:21:09.221797+0000","last_unstale":"2024-05-15T23:24:15.634130+0000","last_undegraded":"2024-05-15T23:24:15.634130+0000","last_fullsized":"2024-05-15T23:24:15.634130+0000","mapping_epoch":407,"log_start":"400'7800","ondisk_log_start":"400'7800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85065728,"num_objects":1298,"num_object_clones":0,"num_object_copies":2596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":284,"num_objects_unfound":0,"num_objects_dirty":1298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17872,"num_write_kb":613440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8345,"num_bytes_recovered":546897920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":284},{"shards":"7,10","objects":1014}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.1","version":"463'17253","reported_seq":17651,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T23:24:14.902934+0000","last_change":"2024-05-15T23:23:50.798735+0000","last_active":"2024-05-15T23:24:14.902934+0000","last_peered":"2024-05-15T23:24:14.902934+0000","last_clean":"2024-05-15T23:24:14.902934+0000","last_became_active":"2024-05-15T23:19:47.516865+0000","last_became_peered":"2024-05-15T23:19:47.516865+0000","last_unstale":"2024-05-15T23:24:14.902934+0000","last_undegraded":"2024-05-15T23:24:14.902934+0000","last_fullsized":"2024-05-15T23:24:14.902934+0000","mapping_epoch":374,"log_start":"398'7200","ondisk_log_start":"398'7200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'15864","last_scrub_stamp":"2024-05-15T23:23:50.798612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:50.798612+0000","objects_scrubbed":2807,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:56.167477+0000","scrub_duration":584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92078080,"num_objects":1405,"num_object_clones":0,"num_object_copies":2810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17253,"num_write_kb":597056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"463'17899","reported_seq":18088,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T23:24:15.613587+0000","last_change":"2024-05-15T23:19:47.305723+0000","last_active":"2024-05-15T23:24:15.613587+0000","last_peered":"2024-05-15T23:24:15.613587+0000","last_clean":"2024-05-15T23:24:15.613587+0000","last_became_active":"2024-05-15T23:19:47.305459+0000","last_became_peered":"2024-05-15T23:19:47.305459+0000","last_unstale":"2024-05-15T23:24:15.613587+0000","last_undegraded":"2024-05-15T23:24:15.613587+0000","last_fullsized":"2024-05-15T23:24:15.613587+0000","mapping_epoch":374,"log_start":"400'7800","ondisk_log_start":"400'7800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90243072,"num_objects":1377,"num_object_clones":0,"num_object_copies":2754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17899,"num_write_kb":616832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"463'18019","reported_seq":18208,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T23:24:15.595915+0000","last_change":"2024-05-15T23:19:47.307499+0000","last_active":"2024-05-15T23:24:15.595915+0000","last_peered":"2024-05-15T23:24:15.595915+0000","last_clean":"2024-05-15T23:24:15.595915+0000","last_became_active":"2024-05-15T23:19:47.307149+0000","last_became_peered":"2024-05-15T23:19:47.307149+0000","last_unstale":"2024-05-15T23:24:15.595915+0000","last_undegraded":"2024-05-15T23:24:15.595915+0000","last_fullsized":"2024-05-15T23:24:15.595915+0000","mapping_epoch":374,"log_start":"401'8000","ondisk_log_start":"401'8000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93519872,"num_objects":1427,"num_object_clones":0,"num_object_copies":2854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18019,"num_write_kb":622272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":965,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-05-15T23:24:12.061151+0000","last_change":"2024-05-15T23:20:05.905241+0000","last_active":"2024-05-15T23:24:12.061151+0000","last_peered":"2024-05-15T23:24:12.061151+0000","last_clean":"2024-05-15T23:24:12.061151+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:24:12.061151+0000","last_undegraded":"2024-05-15T23:24:12.061151+0000","last_fullsized":"2024-05-15T23:24:12.061151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:20:05.905105+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:20:05.905105+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1527382040,"num_objects":23307,"num_object_clones":0,"num_object_copies":46614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":829,"num_objects_unfound":0,"num_objects_dirty":23307,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":283809,"num_write_kb":9827649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25060,"num_bytes_recovered":1642332160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4681244672,"data_stored":4681236528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160809,"ondisk_log_size":160809,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351221,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":859000,"kb_used_data":282220,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92886152,"statfs":{"total":95995035648,"available":95115419648,"internally_reserved":0,"allocated":288993280,"data_stored":288480434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":590608836},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580155,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":742900,"kb_used_data":450732,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93002252,"statfs":{"total":95995035648,"available":95234306048,"internally_reserved":0,"allocated":461549568,"data_stored":461038312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":299170240},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776387,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":550772,"kb_used_data":180844,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93194380,"statfs":{"total":95995035648,"available":95431045120,"internally_reserved":0,"allocated":185184256,"data_stored":184673000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":378796480},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005319,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":358580,"kb_used_data":187180,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93386572,"statfs":{"total":95995035648,"available":95627849728,"internally_reserved":0,"allocated":191672320,"data_stored":191161064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266955,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":615676,"kb_used_data":241588,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93129476,"statfs":{"total":95995035648,"available":95364583424,"internally_reserved":0,"allocated":247386112,"data_stored":246871304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":383056330},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495889,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575147,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":874932,"kb_used_data":366188,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92870220,"statfs":{"total":95995035648,"available":95099105280,"internally_reserved":0,"allocated":374976512,"data_stored":374465256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":520944070},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016555,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483245,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149986,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":618036,"kb_used_data":163436,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93127116,"statfs":{"total":95995035648,"available":95362166784,"internally_reserved":0,"allocated":167358464,"data_stored":166847208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":465500615},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986458,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":768760,"kb_used_data":380780,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":92976392,"statfs":{"total":95995035648,"available":95207825408,"internally_reserved":0,"allocated":389918720,"data_stored":389407464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790230,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1097276,"kb_used_data":714864,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92647876,"statfs":{"total":95995035648,"available":94871425024,"internally_reserved":0,"allocated":732020736,"data_stored":731505408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":364052480,"data_stored":364052480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165412864,"data_stored":165412864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":377880576,"data_stored":377880576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":726405120,"data_stored":726401048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1696661504,"data_stored":1696661504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":243597312,"data_stored":243597312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":189792256,"data_stored":189792256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":183435264,"data_stored":183435264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":447807488,"data_stored":447807488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":286199808,"data_stored":286195736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:24:19.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:24:19.163 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:24:19.163 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:19.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:19.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:24:19.274 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:24:19.305 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:19.344 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:24:19.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:19.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:24:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:19 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:24:19.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:19 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:24:19.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:24:19.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:24:19.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:19 smithi084 ceph-mon[39784]: pgmap v1253: 17 pgs: 2 active+recovering, 15 active+clean; 1.3 GiB data, 6.0 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 961 op/s; 358/42758 objects misplaced (0.837%); 15 MiB/s, 238 objects/s recovering 2024-05-15T23:24:19.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:24:19.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:19 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:24:19.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:19 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:24:19.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:19 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:24:19.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:19 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:24:19.493 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:19 smithi090 ceph-mon[44434]: pgmap v1253: 17 pgs: 2 active+recovering, 15 active+clean; 1.3 GiB data, 6.0 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 961 op/s; 358/42758 objects misplaced (0.837%); 15 MiB/s, 238 objects/s recovering 2024-05-15T23:24:19.496 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:19 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:24:19.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:19 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:24:19.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:19 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:24:19.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:19 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:24:19.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:19 smithi088 ceph-mon[44699]: pgmap v1253: 17 pgs: 2 active+recovering, 15 active+clean; 1.3 GiB data, 6.0 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 961 op/s; 358/42758 objects misplaced (0.837%); 15 MiB/s, 238 objects/s recovering 2024-05-15T23:24:19.858 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:24:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:19.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:24:19.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:24:20.010 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:20.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:20 smithi084 ceph-mon[39784]: osdmap e464: 12 total, 12 up, 9 in 2024-05-15T23:24:20.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:20 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:24:20.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:20 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:24:20.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:20 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:24:20.389 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:20 smithi090 ceph-mon[44434]: osdmap e464: 12 total, 12 up, 9 in 2024-05-15T23:24:20.389 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:20 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:24:20.389 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:20 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:24:20.389 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:20 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:24:20.466 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:24:20.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:20.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:24:20.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:20 smithi088 ceph-mon[44699]: osdmap e464: 12 total, 12 up, 9 in 2024-05-15T23:24:20.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:20 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:24:20.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:20 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:24:20.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:20 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:24:20.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:24:20.616 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:20.751 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:24:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:20.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:24:20.857 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:24:20.898 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:20.926 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:24:20.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:20.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:24:21.033 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:21.086 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:24:21.086 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:24:21.087 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:21.114 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:21.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:21.131 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:21.140 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:24:21.220 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:24:21.261 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:21 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1590677412' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:24:21.261 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:21 smithi088 ceph-mon[44699]: from='client.26239 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:24:21.261 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:21 smithi088 ceph-mon[44699]: pgmap v1255: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 5.7 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s; 74/37046 objects misplaced (0.200%); 13 MiB/s, 211 objects/s recovering 2024-05-15T23:24:21.266 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:21.297 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:24:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:21.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:24:21.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:21 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1590677412' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:24:21.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:21 smithi084 ceph-mon[39784]: from='client.26239 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:24:21.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:21 smithi084 ceph-mon[39784]: pgmap v1255: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 5.7 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s; 74/37046 objects misplaced (0.200%); 13 MiB/s, 211 objects/s recovering 2024-05-15T23:24:21.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:24:21.440 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:21.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:21 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1590677412' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:24:21.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:21 smithi090 ceph-mon[44434]: from='client.26239 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:24:21.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:21 smithi090 ceph-mon[44434]: pgmap v1255: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 5.7 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s; 74/37046 objects misplaced (0.200%); 13 MiB/s, 211 objects/s recovering 2024-05-15T23:24:21.569 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:24:21.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:21.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:24:21.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:24:21.704 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:21.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:24:21.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:24:21.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:24:21.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:24:21.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:24:21.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:24:21.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:24:21.765 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:24:21.765 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:24:21.765 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:24:21.765 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:24:21.765 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:24:21.765 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:24:21.765 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:24:21.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:24:21.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:24:21.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:24:21.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:24:21.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:24:21.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:24:21.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:24:21.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:24:21.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:24:21.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:24:21.782 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:24:21.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:21.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:24:21.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T23:24:21.885 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:21.912 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:22.211 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:24:22.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:22 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:24:22.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:22 smithi084 ceph-mon[39784]: from='client.26239 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:24:22.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:22 smithi084 ceph-mon[39784]: osdmap e465: 12 total, 12 up, 9 in 2024-05-15T23:24:22.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4216266271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:22.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:22.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:22.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:22.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:22.464 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:22.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:22.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:22.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:22 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:24:22.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:22 smithi090 ceph-mon[44434]: from='client.26239 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:24:22.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:22 smithi090 ceph-mon[44434]: osdmap e465: 12 total, 12 up, 9 in 2024-05-15T23:24:22.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4216266271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:22.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:22.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:22.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:22.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:22.569 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:24:22.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:22 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:24:22.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:22 smithi088 ceph-mon[44699]: from='client.26239 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:24:22.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:22 smithi088 ceph-mon[44699]: osdmap e465: 12 total, 12 up, 9 in 2024-05-15T23:24:22.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4216266271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:22.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:22.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:22.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:22.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:22.599 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:22.747 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:24:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:22.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:24:22.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:24:22.881 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:23.138 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:24:23.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:24:23.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:23 smithi090 ceph-mon[44434]: osdmap e466: 12 total, 12 up, 9 in 2024-05-15T23:24:23.139 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:23 smithi090 ceph-mon[44434]: pgmap v1258: 17 pgs: 17 active+clean; 1010 MiB data, 5.4 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.19k op/s; 8.6 MiB/s, 138 objects/s recovering 2024-05-15T23:24:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:24:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:24:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:23 smithi088 ceph-mon[44699]: osdmap e466: 12 total, 12 up, 9 in 2024-05-15T23:24:23.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:23 smithi088 ceph-mon[44699]: pgmap v1258: 17 pgs: 17 active+clean; 1010 MiB data, 5.4 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.19k op/s; 8.6 MiB/s, 138 objects/s recovering 2024-05-15T23:24:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:24:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:24:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:23 smithi084 ceph-mon[39784]: osdmap e466: 12 total, 12 up, 9 in 2024-05-15T23:24:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:23 smithi084 ceph-mon[39784]: pgmap v1258: 17 pgs: 17 active+clean; 1010 MiB data, 5.4 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.19k op/s; 8.6 MiB/s, 138 objects/s recovering 2024-05-15T23:24:23.730 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:24:23.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:23.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:24:23.834 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:24:23.876 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:23.918 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:24:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:23.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:24:24.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:24:24.075 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:24.148 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:24 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:24:24.148 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:24 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:24:24.247 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:24:24.303 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:24:24.303 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:24:24.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.303 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:294257 5.b 5:da8794bb:::benchmark_data_smithi090_116470_object140697:head [delete] snapc 0=[] ondisk+write+known_if_redirected e467)", 2024-05-15T23:24:24.303 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:24.288765+0000", 2024-05-15T23:24:24.303 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0086202780000000003, 2024-05-15T23:24:24.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0086430469999999992, 2024-05-15T23:24:24.304 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:24.304 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:24.304 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:24:24.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:24.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:24.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:24.304 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 294257 2024-05-15T23:24:24.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.304 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:24.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:24.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.288765+0000", 2024-05-15T23:24:24.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:24.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:24.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.288765+0000", 2024-05-15T23:24:24.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:24.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:24.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.288767+0000", 2024-05-15T23:24:24.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1830000000000001e-06 2024-05-15T23:24:24.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:24.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.288773+0000", 2024-05-15T23:24:24.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8889999999999999e-06 2024-05-15T23:24:24.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:24.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.288785+0000", 2024-05-15T23:24:24.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1328999999999999e-05 2024-05-15T23:24:24.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:24.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.288793+0000", 2024-05-15T23:24:24.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-05-15T23:24:24.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:24.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.295388+0000", 2024-05-15T23:24:24.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0065953959999999999 2024-05-15T23:24:24.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:24.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.295490+0000", 2024-05-15T23:24:24.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102243 2024-05-15T23:24:24.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:24:24.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.295593+0000", 2024-05-15T23:24:24.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103082 2024-05-15T23:24:24.307 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:24.307 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:294258 5.b 5.5abafb1b (undecoded) ondisk+write+known_if_redirected e467)", 2024-05-15T23:24:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:24.295225+0000", 2024-05-15T23:24:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0021610520000000001, 2024-05-15T23:24:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022317819999999999, 2024-05-15T23:24:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:24:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 294258 2024-05-15T23:24:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.295225+0000", 2024-05-15T23:24:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.295225+0000", 2024-05-15T23:24:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.295226+0000", 2024-05-15T23:24:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8729999999999999e-06 2024-05-15T23:24:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.295234+0000", 2024-05-15T23:24:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-05-15T23:24:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.295248+0000", 2024-05-15T23:24:24.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3550000000000001e-05 2024-05-15T23:24:24.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:24.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.295256+0000", 2024-05-15T23:24:24.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1960000000000008e-06 2024-05-15T23:24:24.311 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:24.311 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:24.311 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:24.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.311 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:294270 5.e 5:71ef95ed:::benchmark_data_smithi090_116470_object140710:head [delete] snapc 0=[] ondisk+write+known_if_redirected e467)", 2024-05-15T23:24:24.311 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:24.296610+0000", 2024-05-15T23:24:24.311 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.00077551900000000001, 2024-05-15T23:24:24.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087947000000000003, 2024-05-15T23:24:24.312 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:24.312 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:24.312 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:24:24.312 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:24.312 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:24.312 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:24.312 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 294270 2024-05-15T23:24:24.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.312 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:24.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:24.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.296610+0000", 2024-05-15T23:24:24.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:24.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:24.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.296610+0000", 2024-05-15T23:24:24.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:24.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:24.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.296611+0000", 2024-05-15T23:24:24.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.344e-06 2024-05-15T23:24:24.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:24.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.296616+0000", 2024-05-15T23:24:24.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7339999999999999e-06 2024-05-15T23:24:24.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:24.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.296627+0000", 2024-05-15T23:24:24.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0681e-05 2024-05-15T23:24:24.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:24.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.296634+0000", 2024-05-15T23:24:24.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9829999999999999e-06 2024-05-15T23:24:24.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:24.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.296688+0000", 2024-05-15T23:24:24.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4459000000000002e-05 2024-05-15T23:24:24.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:24.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.296797+0000", 2024-05-15T23:24:24.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108989 2024-05-15T23:24:24.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:24.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:24.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 8", 2024-05-15T23:24:24.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:24.296902+0000", 2024-05-15T23:24:24.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010453 2024-05-15T23:24:24.315 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:24.315 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:24.315 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:24.315 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:24.315 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-05-15T23:24:24.316 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 3 2024-05-15T23:24:24.316 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:24:24.316 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:24 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:24:24.316 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:24 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:24:24.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:24 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:24:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:24 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:24:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:24:24.582 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:24:25.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T23:24:25.126 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:24:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:25.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:24:25.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:25.262 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:25.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:25 smithi090 ceph-mon[44434]: osdmap e467: 12 total, 12 up, 9 in 2024-05-15T23:24:25.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:25 smithi090 ceph-mon[44434]: from='client.16332 v1:172.21.15.84:0/1170727483' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:24:25.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:25 smithi090 ceph-mon[44434]: pgmap v1260: 17 pgs: 17 active+clean; 987 MiB data, 5.3 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 947 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-05-15T23:24:25.461 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:24:25.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:25 smithi084 ceph-mon[39784]: osdmap e467: 12 total, 12 up, 9 in 2024-05-15T23:24:25.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:25 smithi084 ceph-mon[39784]: from='client.16332 v1:172.21.15.84:0/1170727483' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:24:25.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:25 smithi084 ceph-mon[39784]: pgmap v1260: 17 pgs: 17 active+clean; 987 MiB data, 5.3 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 947 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-05-15T23:24:25.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:25 smithi088 ceph-mon[44699]: osdmap e467: 12 total, 12 up, 9 in 2024-05-15T23:24:25.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:25 smithi088 ceph-mon[44699]: from='client.16332 v1:172.21.15.84:0/1170727483' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:24:25.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:25 smithi088 ceph-mon[44699]: pgmap v1260: 17 pgs: 17 active+clean; 987 MiB data, 5.3 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 947 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-05-15T23:24:25.605 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1259,"stamp":"2024-05-15T23:24:24.138569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1058996792,"num_objects":16153,"num_object_clones":0,"num_object_copies":32306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16153,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":291088,"num_write_kb":9829799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25895,"num_bytes_recovered":1698248288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160959,"ondisk_log_size":160959,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":5699660,"kb_used_data":2165928,"kb_used_omap":14,"kb_used_meta":3533297,"kb_avail":838006708,"statfs":{"total":863955320832,"available":858118868992,"internally_reserved":0,"allocated":2217910272,"data_stored":2213227556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14358,"internal_metadata":3618097130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-342491136,"num_objects":-5226,"num_object_clones":0,"num_object_copies":-10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-358,"num_objects_unfound":0,"num_objects_dirty":-5226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5226,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":358,"num_bytes_recovered":23461888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.124157"},"pg_stats":[{"pgid":"5.f","version":"466'18166","reported_seq":18361,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-05-15T23:24:22.716539+0000","last_change":"2024-05-15T23:19:47.305945+0000","last_active":"2024-05-15T23:24:22.716539+0000","last_peered":"2024-05-15T23:24:22.716539+0000","last_clean":"2024-05-15T23:24:22.716539+0000","last_became_active":"2024-05-15T23:19:47.305689+0000","last_became_peered":"2024-05-15T23:19:47.305689+0000","last_unstale":"2024-05-15T23:24:22.716539+0000","last_undegraded":"2024-05-15T23:24:22.716539+0000","last_fullsized":"2024-05-15T23:24:22.716539+0000","mapping_epoch":374,"log_start":"404'8100","ondisk_log_start":"404'8100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:25.513191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58589184,"num_objects":894,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18166,"num_write_kb":609920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"464'17846","reported_seq":18246,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-05-15T23:24:20.590316+0000","last_change":"2024-05-15T23:23:53.872535+0000","last_active":"2024-05-15T23:24:20.590316+0000","last_peered":"2024-05-15T23:24:20.590316+0000","last_clean":"2024-05-15T23:24:20.590316+0000","last_became_active":"2024-05-15T23:19:47.325414+0000","last_became_peered":"2024-05-15T23:19:47.325414+0000","last_unstale":"2024-05-15T23:24:20.590316+0000","last_undegraded":"2024-05-15T23:24:20.590316+0000","last_fullsized":"2024-05-15T23:24:20.590316+0000","mapping_epoch":374,"log_start":"401'7800","ondisk_log_start":"401'7800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'16258","last_scrub_stamp":"2024-05-15T23:23:53.872309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:53.872309+0000","objects_scrubbed":2678,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:15.839794+0000","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70516736,"num_objects":1076,"num_object_clones":0,"num_object_copies":2152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17846,"num_write_kb":605504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"466'18473","reported_seq":18669,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-05-15T23:24:22.178620+0000","last_change":"2024-05-15T23:19:47.306160+0000","last_active":"2024-05-15T23:24:22.178620+0000","last_peered":"2024-05-15T23:24:22.178620+0000","last_clean":"2024-05-15T23:24:22.178620+0000","last_became_active":"2024-05-15T23:19:47.305860+0000","last_became_peered":"2024-05-15T23:19:47.305860+0000","last_unstale":"2024-05-15T23:24:22.178620+0000","last_undegraded":"2024-05-15T23:24:22.178620+0000","last_fullsized":"2024-05-15T23:24:22.178620+0000","mapping_epoch":374,"log_start":"404'8400","ondisk_log_start":"404'8400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:24.866783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63701016,"num_objects":973,"num_object_clones":0,"num_object_copies":1946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":973,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18473,"num_write_kb":622209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"464'18234","reported_seq":18587,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-05-15T23:24:20.878235+0000","last_change":"2024-05-15T23:22:51.995640+0000","last_active":"2024-05-15T23:24:20.878235+0000","last_peered":"2024-05-15T23:24:20.878235+0000","last_clean":"2024-05-15T23:24:20.878235+0000","last_became_active":"2024-05-15T23:19:47.306168+0000","last_became_peered":"2024-05-15T23:19:47.306168+0000","last_unstale":"2024-05-15T23:24:20.878235+0000","last_undegraded":"2024-05-15T23:24:20.878235+0000","last_fullsized":"2024-05-15T23:24:20.878235+0000","mapping_epoch":374,"log_start":"404'8200","ondisk_log_start":"404'8200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'13335","last_scrub_stamp":"2024-05-15T23:22:51.995473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:51.995473+0000","objects_scrubbed":6018,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:21.353486+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041024,"num_objects":1084,"num_object_clones":0,"num_object_copies":2168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18234,"num_write_kb":618176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"464'18136","reported_seq":18492,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-05-15T23:24:20.600263+0000","last_change":"2024-05-15T23:22:48.338084+0000","last_active":"2024-05-15T23:24:20.600263+0000","last_peered":"2024-05-15T23:24:20.600263+0000","last_clean":"2024-05-15T23:24:20.600263+0000","last_became_active":"2024-05-15T23:19:47.307061+0000","last_became_peered":"2024-05-15T23:19:47.307061+0000","last_unstale":"2024-05-15T23:24:20.600263+0000","last_undegraded":"2024-05-15T23:24:20.600263+0000","last_fullsized":"2024-05-15T23:24:20.600263+0000","mapping_epoch":374,"log_start":"404'8100","ondisk_log_start":"404'8100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'13172","last_scrub_stamp":"2024-05-15T23:22:48.337866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:48.337866+0000","objects_scrubbed":6060,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:24.501754+0000","scrub_duration":1844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69206016,"num_objects":1056,"num_object_clones":0,"num_object_copies":2112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18136,"num_write_kb":614144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"464'18151","reported_seq":18342,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-05-15T23:24:20.613164+0000","last_change":"2024-05-15T23:19:47.325657+0000","last_active":"2024-05-15T23:24:20.613164+0000","last_peered":"2024-05-15T23:24:20.613164+0000","last_clean":"2024-05-15T23:24:20.613164+0000","last_became_active":"2024-05-15T23:19:47.325371+0000","last_became_peered":"2024-05-15T23:19:47.325371+0000","last_unstale":"2024-05-15T23:24:20.613164+0000","last_undegraded":"2024-05-15T23:24:20.613164+0000","last_fullsized":"2024-05-15T23:24:20.613164+0000","mapping_epoch":374,"log_start":"402'8100","ondisk_log_start":"402'8100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69533696,"num_objects":1061,"num_object_clones":0,"num_object_copies":2122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18151,"num_write_kb":614784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"464'18141","reported_seq":35478,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-05-15T23:24:21.061401+0000","last_change":"2024-05-15T23:24:17.033232+0000","last_active":"2024-05-15T23:24:21.061401+0000","last_peered":"2024-05-15T23:24:21.061401+0000","last_clean":"2024-05-15T23:24:21.061401+0000","last_became_active":"2024-05-15T23:21:09.454639+0000","last_became_peered":"2024-05-15T23:21:09.454639+0000","last_unstale":"2024-05-15T23:24:21.061401+0000","last_undegraded":"2024-05-15T23:24:21.061401+0000","last_fullsized":"2024-05-15T23:24:21.061401+0000","mapping_epoch":407,"log_start":"402'8100","ondisk_log_start":"402'8100","created":374,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66387968,"num_objects":1013,"num_object_clones":0,"num_object_copies":2026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18141,"num_write_kb":612928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8570,"num_bytes_recovered":561643520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"466'18284","reported_seq":18479,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-05-15T23:24:22.203746+0000","last_change":"2024-05-15T23:19:47.304832+0000","last_active":"2024-05-15T23:24:22.203746+0000","last_peered":"2024-05-15T23:24:22.203746+0000","last_clean":"2024-05-15T23:24:22.203746+0000","last_became_active":"2024-05-15T23:19:47.304547+0000","last_became_peered":"2024-05-15T23:19:47.304547+0000","last_unstale":"2024-05-15T23:24:22.203746+0000","last_undegraded":"2024-05-15T23:24:22.203746+0000","last_fullsized":"2024-05-15T23:24:22.203746+0000","mapping_epoch":374,"log_start":"404'8200","ondisk_log_start":"404'8200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:24.866819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60293120,"num_objects":920,"num_object_clones":0,"num_object_copies":1840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18284,"num_write_kb":614528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"466'18111","reported_seq":18304,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-05-15T23:24:22.708334+0000","last_change":"2024-05-15T23:19:47.306634+0000","last_active":"2024-05-15T23:24:22.708334+0000","last_peered":"2024-05-15T23:24:22.708334+0000","last_clean":"2024-05-15T23:24:22.708334+0000","last_became_active":"2024-05-15T23:19:47.306331+0000","last_became_peered":"2024-05-15T23:19:47.306331+0000","last_unstale":"2024-05-15T23:24:22.708334+0000","last_undegraded":"2024-05-15T23:24:22.708334+0000","last_fullsized":"2024-05-15T23:24:22.708334+0000","mapping_epoch":374,"log_start":"404'8100","ondisk_log_start":"404'8100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:25.531081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61931520,"num_objects":945,"num_object_clones":0,"num_object_copies":1890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18111,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"466'18398","reported_seq":18593,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-05-15T23:24:22.721368+0000","last_change":"2024-05-15T23:19:47.306611+0000","last_active":"2024-05-15T23:24:22.721368+0000","last_peered":"2024-05-15T23:24:22.721368+0000","last_clean":"2024-05-15T23:24:22.721368+0000","last_became_active":"2024-05-15T23:19:47.306349+0000","last_became_peered":"2024-05-15T23:19:47.306349+0000","last_unstale":"2024-05-15T23:24:22.721368+0000","last_undegraded":"2024-05-15T23:24:22.721368+0000","last_fullsized":"2024-05-15T23:24:22.721368+0000","mapping_epoch":374,"log_start":"404'8300","ondisk_log_start":"404'8300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:25.531105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59768832,"num_objects":912,"num_object_clones":0,"num_object_copies":1824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18398,"num_write_kb":617920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"464'18300","reported_seq":18758,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-05-15T23:24:20.842432+0000","last_change":"2024-05-15T23:23:50.810750+0000","last_active":"2024-05-15T23:24:20.842432+0000","last_peered":"2024-05-15T23:24:20.842432+0000","last_clean":"2024-05-15T23:24:20.842432+0000","last_became_active":"2024-05-15T23:19:47.305685+0000","last_became_peered":"2024-05-15T23:19:47.305685+0000","last_unstale":"2024-05-15T23:24:20.842432+0000","last_undegraded":"2024-05-15T23:24:20.842432+0000","last_fullsized":"2024-05-15T23:24:20.842432+0000","mapping_epoch":374,"log_start":"404'8200","ondisk_log_start":"404'8200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'16478","last_scrub_stamp":"2024-05-15T23:23:50.810636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:50.810636+0000","objects_scrubbed":2939,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:20.026294+0000","scrub_duration":608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71827456,"num_objects":1096,"num_object_clones":0,"num_object_copies":2192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18300,"num_write_kb":620672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"466'18430","reported_seq":36009,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-05-15T23:24:22.739088+0000","last_change":"2024-05-15T23:24:17.572412+0000","last_active":"2024-05-15T23:24:22.739088+0000","last_peered":"2024-05-15T23:24:22.739088+0000","last_clean":"2024-05-15T23:24:22.739088+0000","last_became_active":"2024-05-15T23:21:09.400731+0000","last_became_peered":"2024-05-15T23:21:09.400731+0000","last_unstale":"2024-05-15T23:24:22.739088+0000","last_undegraded":"2024-05-15T23:24:22.739088+0000","last_fullsized":"2024-05-15T23:24:22.739088+0000","mapping_epoch":407,"log_start":"404'8400","ondisk_log_start":"404'8400","created":374,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:25.531911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56754176,"num_objects":866,"num_object_clones":0,"num_object_copies":1732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18430,"num_write_kb":617472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8690,"num_bytes_recovered":569507840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"464'18181","reported_seq":35636,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-05-15T23:24:20.590066+0000","last_change":"2024-05-15T23:24:19.774443+0000","last_active":"2024-05-15T23:24:20.590066+0000","last_peered":"2024-05-15T23:24:20.590066+0000","last_clean":"2024-05-15T23:24:20.590066+0000","last_became_active":"2024-05-15T23:21:09.221797+0000","last_became_peered":"2024-05-15T23:21:09.221797+0000","last_unstale":"2024-05-15T23:24:20.590066+0000","last_undegraded":"2024-05-15T23:24:20.590066+0000","last_fullsized":"2024-05-15T23:24:20.590066+0000","mapping_epoch":407,"log_start":"402'8100","ondisk_log_start":"402'8100","created":374,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64815104,"num_objects":989,"num_object_clones":0,"num_object_copies":1978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18181,"num_write_kb":613440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8629,"num_bytes_recovered":565510144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.1","version":"464'17556","reported_seq":17956,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-05-15T23:24:19.897922+0000","last_change":"2024-05-15T23:23:50.798735+0000","last_active":"2024-05-15T23:24:19.897922+0000","last_peered":"2024-05-15T23:24:19.897922+0000","last_clean":"2024-05-15T23:24:19.897922+0000","last_became_active":"2024-05-15T23:19:47.516865+0000","last_became_peered":"2024-05-15T23:19:47.516865+0000","last_unstale":"2024-05-15T23:24:19.897922+0000","last_undegraded":"2024-05-15T23:24:19.897922+0000","last_fullsized":"2024-05-15T23:24:19.897922+0000","mapping_epoch":374,"log_start":"399'7500","ondisk_log_start":"399'7500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'15864","last_scrub_stamp":"2024-05-15T23:23:50.798612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:50.798612+0000","objects_scrubbed":2807,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:56.167477+0000","scrub_duration":584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72220672,"num_objects":1102,"num_object_clones":0,"num_object_copies":2204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17556,"num_write_kb":597056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"464'18220","reported_seq":18411,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-05-15T23:24:20.599295+0000","last_change":"2024-05-15T23:19:47.305723+0000","last_active":"2024-05-15T23:24:20.599295+0000","last_peered":"2024-05-15T23:24:20.599295+0000","last_clean":"2024-05-15T23:24:20.599295+0000","last_became_active":"2024-05-15T23:19:47.305459+0000","last_became_peered":"2024-05-15T23:19:47.305459+0000","last_unstale":"2024-05-15T23:24:20.599295+0000","last_undegraded":"2024-05-15T23:24:20.599295+0000","last_fullsized":"2024-05-15T23:24:20.599295+0000","mapping_epoch":374,"log_start":"404'8200","ondisk_log_start":"404'8200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69206016,"num_objects":1056,"num_object_clones":0,"num_object_copies":2112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18220,"num_write_kb":616832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"464'18338","reported_seq":18529,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-05-15T23:24:20.620758+0000","last_change":"2024-05-15T23:19:47.307499+0000","last_active":"2024-05-15T23:24:20.620758+0000","last_peered":"2024-05-15T23:24:20.620758+0000","last_clean":"2024-05-15T23:24:20.620758+0000","last_became_active":"2024-05-15T23:19:47.307149+0000","last_became_peered":"2024-05-15T23:19:47.307149+0000","last_unstale":"2024-05-15T23:24:20.620758+0000","last_undegraded":"2024-05-15T23:24:20.620758+0000","last_fullsized":"2024-05-15T23:24:20.620758+0000","mapping_epoch":374,"log_start":"404'8300","ondisk_log_start":"404'8300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72613888,"num_objects":1108,"num_object_clones":0,"num_object_copies":2216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18338,"num_write_kb":622272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":968,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-05-15T23:24:19.117984+0000","last_change":"2024-05-15T23:20:05.905241+0000","last_active":"2024-05-15T23:24:19.117984+0000","last_peered":"2024-05-15T23:24:19.117984+0000","last_clean":"2024-05-15T23:24:19.117984+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:24:19.117984+0000","last_undegraded":"2024-05-15T23:24:19.117984+0000","last_fullsized":"2024-05-15T23:24:19.117984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:20:05.905105+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:20:05.905105+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1058406424,"num_objects":16151,"num_object_clones":0,"num_object_copies":32302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16151,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":290965,"num_write_kb":9827649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25889,"num_bytes_recovered":1696661504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3814531072,"data_stored":3814522928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160865,"ondisk_log_size":160865,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351223,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":755740,"kb_used_data":187280,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92989412,"statfs":{"total":95995035648,"available":95221157888,"internally_reserved":0,"allocated":191774720,"data_stored":191243762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":582089156},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580156,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":630916,"kb_used_data":351224,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":93114236,"statfs":{"total":95995035648,"available":95348977664,"internally_reserved":0,"allocated":359653376,"data_stored":359136116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":286390720},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776388,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":530948,"kb_used_data":138808,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93214204,"statfs":{"total":95995035648,"available":95451344896,"internally_reserved":0,"allocated":142139392,"data_stored":141622132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":401537472},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325108,"kb_used_data":153644,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93420044,"statfs":{"total":95995035648,"available":95662125056,"internally_reserved":0,"allocated":157331456,"data_stored":156820200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266956,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":575628,"kb_used_data":197376,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93169524,"statfs":{"total":95995035648,"available":95405592576,"internally_reserved":0,"allocated":202113024,"data_stored":201592212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":387316170},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495891,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575148,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":782784,"kb_used_data":286520,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92962368,"statfs":{"total":95995035648,"available":95193464832,"internally_reserved":0,"allocated":293396480,"data_stored":292879220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":508164550},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016557,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483247,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149987,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":588672,"kb_used_data":134072,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93156480,"statfs":{"total":95995035648,"available":95392235520,"internally_reserved":0,"allocated":137289728,"data_stored":136772468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":465500615},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986459,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":693316,"kb_used_data":300984,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93051836,"statfs":{"total":95995035648,"available":95285080064,"internally_reserved":0,"allocated":308207616,"data_stored":307690356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790232,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":816548,"kb_used_data":416020,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92928604,"statfs":{"total":95995035648,"available":95158890496,"internally_reserved":0,"allocated":426004480,"data_stored":425471090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":281542656,"data_stored":281542656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130678784,"data_stored":130678784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":297533440,"data_stored":297533440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":419565568,"data_stored":419561496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1669332992,"data_stored":1669332992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":196739072,"data_stored":196739072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":149618688,"data_stored":149618688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":139657216,"data_stored":139657216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":343998464,"data_stored":343998464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":185864192,"data_stored":185860120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:24:25.606 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph status --format=json 2024-05-15T23:24:25.618 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:25.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:25.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:25.624 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:25.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:24:25.752 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:25.810 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:24:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:25.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:24:25.916 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:24:25.992 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:26.026 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:24:26.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:26.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:24:26.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:24:26.182 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:26.479 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:26.903 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:24:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:27 smithi084 ceph-mon[39784]: pgmap v1261: 17 pgs: 17 active+clean; 790 MiB data, 5.0 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 982 op/s; 789 KiB/s, 12 objects/s recovering 2024-05-15T23:24:27.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:27 smithi090 ceph-mon[44434]: pgmap v1261: 17 pgs: 17 active+clean; 790 MiB data, 5.0 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 982 op/s; 789 KiB/s, 12 objects/s recovering 2024-05-15T23:24:27.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:27 smithi088 ceph-mon[44699]: pgmap v1261: 17 pgs: 17 active+clean; 790 MiB data, 5.0 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 982 op/s; 789 KiB/s, 12 objects/s recovering 2024-05-15T23:24:28.509 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:24:28.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3250692157' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:24:28.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:28 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:24:28.747 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:24:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:28.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:24:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3250692157' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:24:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:28 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:24:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3250692157' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:24:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:28 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:24:28.853 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:24:28.898 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:28.928 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:24:28.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:28.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:24:29.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:29.085 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:29.442 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:29.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:29.447 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:29.519 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:29 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3250692157' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:24:29.519 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:29 smithi084 ceph-mon[39784]: osdmap e468: 12 total, 12 up, 9 in 2024-05-15T23:24:29.519 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:29 smithi084 ceph-mon[39784]: pgmap v1263: 17 pgs: 17 active+clean; 659 MiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:24:29.547 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:24:29.601 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:29.640 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:24:29.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:29.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:24:29.744 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:24:29.758 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:29 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3250692157' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:24:29.758 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:29 smithi090 ceph-mon[44434]: osdmap e468: 12 total, 12 up, 9 in 2024-05-15T23:24:29.758 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:29 smithi090 ceph-mon[44434]: pgmap v1263: 17 pgs: 17 active+clean; 659 MiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:24:29.806 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:29 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3250692157' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:24:29.806 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:29 smithi088 ceph-mon[44699]: osdmap e468: 12 total, 12 up, 9 in 2024-05-15T23:24:29.806 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:29 smithi088 ceph-mon[44699]: pgmap v1263: 17 pgs: 17 active+clean; 659 MiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:24:29.808 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:29.815 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:29.846 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:24:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:29.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:24:29.962 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:24:30.011 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:30.044 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:24:30.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:30.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:24:30.117 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:24:30.118 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:24:30.118 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:24:30.118 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:24:30.118 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:24:30.151 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:24:30.180 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:30.199 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:24:30.199 INFO:teuthology.orchestra.run.smithi084.stdout:{"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1549,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":468,"num_osds":12,"num_up_osds":12,"osd_up_since":1715815247,"num_in_osds":9,"osd_in_since":1715815268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17}],"num_pgs":17,"num_pools":2,"num_objects":10537,"data_bytes":690946616,"bytes_used":5145526272,"bytes_avail":858809794560,"bytes_total":863955320832,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1113},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.84:8443/"}},"servicemap":{"epoch":35,"modified":"2024-05-15T23:24:21.014866+0000","services":{"osd":{"daemons":{"summary":"","6":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-05-15T23:24:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:24:30.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:30.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:24:30.333 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:24:30.371 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:30.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:30 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3741646394' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T23:24:30.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:30 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3741646394' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T23:24:30.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:30 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3741646394' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T23:24:30.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T23:24:31.240 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:24:31.306 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:24:31.370 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-05-15T23:24:31.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:24:31.425 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:24:31.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:31.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:24:31.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:24:31.584 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:31.617 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:24:31.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:31.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:24:31.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:31.767 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:31.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:31.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:31 smithi090 ceph-mon[44434]: pgmap v1264: 17 pgs: 17 active+clean; 467 MiB data, 4.4 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s 2024-05-15T23:24:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:31 smithi088 ceph-mon[44699]: pgmap v1264: 17 pgs: 17 active+clean; 467 MiB data, 4.4 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s 2024-05-15T23:24:31.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:31 smithi084 ceph-mon[39784]: pgmap v1264: 17 pgs: 17 active+clean; 467 MiB data, 4.4 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s 2024-05-15T23:24:31.901 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:24:31.928 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:31.987 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:24:31.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:31.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:24:32.046 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:32.100 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:24:32.161 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:32.192 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:24:32.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:32.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:24:32.305 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:24:32.350 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:32.506 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:24:32.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:32.515 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:24:32.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:32 smithi088 ceph-mon[44699]: 5.d scrub starts 2024-05-15T23:24:32.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:32 smithi088 ceph-mon[44699]: 5.d scrub ok 2024-05-15T23:24:32.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:32 smithi088 ceph-mon[44699]: 5.b scrub starts 2024-05-15T23:24:32.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:32 smithi088 ceph-mon[44699]: 5.b scrub ok 2024-05-15T23:24:32.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:32 smithi088 ceph-mon[44699]: 5.9 scrub starts 2024-05-15T23:24:32.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:32 smithi088 ceph-mon[44699]: 5.9 scrub ok 2024-05-15T23:24:32.598 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:32 smithi084 ceph-mon[39784]: 5.d scrub starts 2024-05-15T23:24:32.598 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:32 smithi084 ceph-mon[39784]: 5.d scrub ok 2024-05-15T23:24:32.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:32 smithi084 ceph-mon[39784]: 5.b scrub starts 2024-05-15T23:24:32.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:32 smithi084 ceph-mon[39784]: 5.b scrub ok 2024-05-15T23:24:32.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:32 smithi084 ceph-mon[39784]: 5.9 scrub starts 2024-05-15T23:24:32.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:32 smithi084 ceph-mon[39784]: 5.9 scrub ok 2024-05-15T23:24:32.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:32 smithi084 ceph-mon[39784]: osdmap e469: 12 total, 12 up, 9 in 2024-05-15T23:24:32.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:24:32.653 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:32 smithi090 ceph-mon[44434]: 5.d scrub starts 2024-05-15T23:24:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:32 smithi090 ceph-mon[44434]: 5.d scrub ok 2024-05-15T23:24:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:32 smithi090 ceph-mon[44434]: 5.b scrub starts 2024-05-15T23:24:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:32 smithi090 ceph-mon[44434]: 5.b scrub ok 2024-05-15T23:24:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:32 smithi090 ceph-mon[44434]: 5.9 scrub starts 2024-05-15T23:24:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:32 smithi090 ceph-mon[44434]: 5.9 scrub ok 2024-05-15T23:24:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:32 smithi090 ceph-mon[44434]: osdmap e469: 12 total, 12 up, 9 in 2024-05-15T23:24:32.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:32 smithi088 ceph-mon[44699]: osdmap e469: 12 total, 12 up, 9 in 2024-05-15T23:24:33.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:33 smithi084 ceph-mon[39784]: 5.8 scrub starts 2024-05-15T23:24:33.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:33 smithi084 ceph-mon[39784]: 5.8 scrub ok 2024-05-15T23:24:33.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:33 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:24:33.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:33 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:24:33.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:33 smithi084 ceph-mon[39784]: pgmap v1266: 17 pgs: 17 active+clean; 321 MiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.33k op/s 2024-05-15T23:24:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:33 smithi090 ceph-mon[44434]: 5.8 scrub starts 2024-05-15T23:24:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:33 smithi090 ceph-mon[44434]: 5.8 scrub ok 2024-05-15T23:24:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:33 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:24:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:33 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:24:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:33 smithi090 ceph-mon[44434]: pgmap v1266: 17 pgs: 17 active+clean; 321 MiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.33k op/s 2024-05-15T23:24:33.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:33 smithi088 ceph-mon[44699]: 5.8 scrub starts 2024-05-15T23:24:33.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:33 smithi088 ceph-mon[44699]: 5.8 scrub ok 2024-05-15T23:24:33.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:33 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:24:33.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:33 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:24:33.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:33 smithi088 ceph-mon[44699]: pgmap v1266: 17 pgs: 17 active+clean; 321 MiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.33k op/s 2024-05-15T23:24:34.054 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:24:34.057 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:34.081 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:24:34.107 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:24:34.121 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:24:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:34.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:24:34.127 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:24:34.151 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:24:34.175 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:24:34.200 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:24:34.223 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:24:34.227 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:24:34.244 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:24:34.266 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:24:34.288 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:34.294 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:24:34.316 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:24:34.318 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:24:34.318 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:24:34.318 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:24:34.319 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:24:34.319 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:24:34.319 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:24:34.319 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:24:34.319 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:24:34.319 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:24:34.319 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:24:34.319 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:24:34.319 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:24:34.322 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:24:34.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:34.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:24:34.427 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:34.465 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:34.538 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:34.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:34.546 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:34 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2156120019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:34.546 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.546 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.546 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.546 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.546 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.546 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.546 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.546 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:34 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:34.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:34 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2156120019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:34.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.703 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:34 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.703 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:34.785 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:34.793 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:34.842 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:24:34.843 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:24:34.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:34 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2156120019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:34.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:34 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:34.895 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:24:34.924 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:34.967 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:24:34.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:34.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:24:35.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:24:35.117 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:35.547 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:24:35.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:35.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:35.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:35.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:35.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:35 smithi084 ceph-mon[39784]: pgmap v1267: 17 pgs: 17 active+clean; 301 MiB data, 4.0 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 977 op/s 2024-05-15T23:24:35.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:35 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/70572921' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:24:35.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:35.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:35.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:35.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:35.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:35 smithi090 ceph-mon[44434]: pgmap v1267: 17 pgs: 17 active+clean; 301 MiB data, 4.0 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 977 op/s 2024-05-15T23:24:35.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:35 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/70572921' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:24:35.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:35.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:35.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:35.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:35.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:35 smithi088 ceph-mon[44699]: pgmap v1267: 17 pgs: 17 active+clean; 301 MiB data, 4.0 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 977 op/s 2024-05-15T23:24:35.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:35 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/70572921' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:24:36.033 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Removed 153557 objects 2024-05-15T23:24:36.034 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Clean up completed and total clean up time :185.964 2024-05-15T23:24:36.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T23:24:36.177 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:24:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:36.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:24:36.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:24:36.309 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:36.328 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:24:36.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:36.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:24:36.431 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:24:36.460 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:36.484 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:24:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:36 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:24:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:36 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:24:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/70572921' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:24:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:36 smithi088 ceph-mon[44699]: osdmap e470: 12 total, 12 up, 9 in 2024-05-15T23:24:36.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:36 smithi088 ceph-mon[44699]: 5.c scrub starts 2024-05-15T23:24:36.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:36 smithi088 ceph-mon[44699]: 5.c scrub ok 2024-05-15T23:24:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:36 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:24:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:36 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:24:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/70572921' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:24:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:36 smithi084 ceph-mon[39784]: osdmap e470: 12 total, 12 up, 9 in 2024-05-15T23:24:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:36 smithi084 ceph-mon[39784]: 5.c scrub starts 2024-05-15T23:24:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:36 smithi084 ceph-mon[39784]: 5.c scrub ok 2024-05-15T23:24:36.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:36 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:24:36.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:36 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:24:36.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/70572921' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:24:36.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:36 smithi090 ceph-mon[44434]: osdmap e470: 12 total, 12 up, 9 in 2024-05-15T23:24:36.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:36 smithi090 ceph-mon[44434]: 5.c scrub starts 2024-05-15T23:24:36.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:36 smithi090 ceph-mon[44434]: 5.c scrub ok 2024-05-15T23:24:36.808 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:24:36.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:36.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:24:36.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:24:36.941 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:36.960 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:24:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:36.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:24:37.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:24:37.095 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:37.525 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:24:37.543 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:24:37.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:37.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:24:37.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:24:37.671 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:24:37.672 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:24:37.674 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:37 smithi090 ceph-mon[44434]: pgmap v1269: 17 pgs: 17 active+clean; 106 MiB data, 3.7 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-05-15T23:24:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:37 smithi090 ceph-mon[44434]: osdmap e471: 12 total, 12 up, 9 in 2024-05-15T23:24:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:37 smithi088 ceph-mon[44699]: pgmap v1269: 17 pgs: 17 active+clean; 106 MiB data, 3.7 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-05-15T23:24:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:37 smithi088 ceph-mon[44699]: osdmap e471: 12 total, 12 up, 9 in 2024-05-15T23:24:37.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:37 smithi084 ceph-mon[39784]: pgmap v1269: 17 pgs: 17 active+clean; 106 MiB data, 3.7 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-05-15T23:24:37.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:37 smithi084 ceph-mon[39784]: osdmap e471: 12 total, 12 up, 9 in 2024-05-15T23:24:37.871 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:24:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:37.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:24:37.976 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:24:38.003 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:38.151 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:24:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:38.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:24:38.246 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:38.247 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1270,"stamp":"2024-05-15T23:24:37.064422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":111215160,"num_objects":1691,"num_object_clones":0,"num_object_copies":3382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1691,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":305550,"num_write_kb":9829799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25895,"num_bytes_recovered":1698248288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":3865744,"kb_used_data":279224,"kb_used_omap":14,"kb_used_meta":3586033,"kb_avail":839840624,"statfs":{"total":863955320832,"available":859996798976,"internally_reserved":0,"allocated":285925376,"data_stored":280977184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14358,"internal_metadata":3672098794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,5,8],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-378077184,"num_objects":-5769,"num_object_clones":0,"num_object_copies":-11538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5769,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.047888"},"pg_stats":[{"pgid":"5.f","version":"469'18831","reported_seq":19032,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-05-15T23:24:32.688478+0000","last_change":"2024-05-15T23:19:47.305945+0000","last_active":"2024-05-15T23:24:32.688478+0000","last_peered":"2024-05-15T23:24:32.688478+0000","last_clean":"2024-05-15T23:24:32.688478+0000","last_became_active":"2024-05-15T23:19:47.305689+0000","last_became_peered":"2024-05-15T23:19:47.305689+0000","last_unstale":"2024-05-15T23:24:32.688478+0000","last_undegraded":"2024-05-15T23:24:32.688478+0000","last_fullsized":"2024-05-15T23:24:32.688478+0000","mapping_epoch":374,"log_start":"410'8800","ondisk_log_start":"410'8800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15007744,"num_objects":229,"num_object_clones":0,"num_object_copies":458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18831,"num_write_kb":609920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"470'18889","reported_seq":19301,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-05-15T23:24:35.606376+0000","last_change":"2024-05-15T23:23:53.872535+0000","last_active":"2024-05-15T23:24:35.606376+0000","last_peered":"2024-05-15T23:24:35.606376+0000","last_clean":"2024-05-15T23:24:35.606376+0000","last_became_active":"2024-05-15T23:19:47.325414+0000","last_became_peered":"2024-05-15T23:19:47.325414+0000","last_unstale":"2024-05-15T23:24:35.606376+0000","last_undegraded":"2024-05-15T23:24:35.606376+0000","last_fullsized":"2024-05-15T23:24:35.606376+0000","mapping_epoch":374,"log_start":"410'8800","ondisk_log_start":"410'8800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'16258","last_scrub_stamp":"2024-05-15T23:23:53.872309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:53.872309+0000","objects_scrubbed":2678,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:15.839794+0000","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2162688,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18889,"num_write_kb":605504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"469'19175","reported_seq":19389,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-05-15T23:24:32.205336+0000","last_change":"2024-05-15T23:24:31.086260+0000","last_active":"2024-05-15T23:24:32.205336+0000","last_peered":"2024-05-15T23:24:32.205336+0000","last_clean":"2024-05-15T23:24:32.205336+0000","last_became_active":"2024-05-15T23:19:47.305860+0000","last_became_peered":"2024-05-15T23:19:47.305860+0000","last_unstale":"2024-05-15T23:24:32.205336+0000","last_undegraded":"2024-05-15T23:24:32.205336+0000","last_fullsized":"2024-05-15T23:24:32.205336+0000","mapping_epoch":374,"log_start":"410'9100","ondisk_log_start":"410'9100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'19105","last_scrub_stamp":"2024-05-15T23:24:31.086098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:24:31.086098+0000","objects_scrubbed":342,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:59.030006+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17694744,"num_objects":271,"num_object_clones":0,"num_object_copies":542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":271,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":19175,"num_write_kb":622209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"470'19311","reported_seq":19677,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-05-15T23:24:35.865900+0000","last_change":"2024-05-15T23:22:51.995640+0000","last_active":"2024-05-15T23:24:35.865900+0000","last_peered":"2024-05-15T23:24:35.865900+0000","last_clean":"2024-05-15T23:24:35.865900+0000","last_became_active":"2024-05-15T23:19:47.306168+0000","last_became_peered":"2024-05-15T23:19:47.306168+0000","last_unstale":"2024-05-15T23:24:35.865900+0000","last_undegraded":"2024-05-15T23:24:35.865900+0000","last_fullsized":"2024-05-15T23:24:35.865900+0000","mapping_epoch":374,"log_start":"411'9300","ondisk_log_start":"411'9300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'13335","last_scrub_stamp":"2024-05-15T23:22:51.995473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:22:51.995473+0000","objects_scrubbed":6018,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19311,"num_write_kb":618176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"470'19157","reported_seq":19537,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-05-15T23:24:35.627431+0000","last_change":"2024-05-15T23:24:31.742721+0000","last_active":"2024-05-15T23:24:35.627431+0000","last_peered":"2024-05-15T23:24:35.627431+0000","last_clean":"2024-05-15T23:24:35.627431+0000","last_became_active":"2024-05-15T23:19:47.307061+0000","last_became_peered":"2024-05-15T23:19:47.307061+0000","last_unstale":"2024-05-15T23:24:35.627431+0000","last_undegraded":"2024-05-15T23:24:35.627431+0000","last_fullsized":"2024-05-15T23:24:35.627431+0000","mapping_epoch":374,"log_start":"410'9100","ondisk_log_start":"410'9100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'18883","last_scrub_stamp":"2024-05-15T23:24:31.742550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:24:31.742550+0000","objects_scrubbed":314,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:42.153206+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2293760,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19157,"num_write_kb":614144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"470'19180","reported_seq":19382,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-05-15T23:24:35.641401+0000","last_change":"2024-05-15T23:19:47.325657+0000","last_active":"2024-05-15T23:24:35.641401+0000","last_peered":"2024-05-15T23:24:35.641401+0000","last_clean":"2024-05-15T23:24:35.641401+0000","last_became_active":"2024-05-15T23:19:47.325371+0000","last_became_peered":"2024-05-15T23:19:47.325371+0000","last_unstale":"2024-05-15T23:24:35.641401+0000","last_undegraded":"2024-05-15T23:24:35.641401+0000","last_fullsized":"2024-05-15T23:24:35.641401+0000","mapping_epoch":374,"log_start":"410'9100","ondisk_log_start":"410'9100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097152,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19180,"num_write_kb":614784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"470'19154","reported_seq":36513,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-05-15T23:24:36.016608+0000","last_change":"2024-05-15T23:24:31.917656+0000","last_active":"2024-05-15T23:24:36.016608+0000","last_peered":"2024-05-15T23:24:36.016608+0000","last_clean":"2024-05-15T23:24:36.016608+0000","last_became_active":"2024-05-15T23:21:09.454639+0000","last_became_peered":"2024-05-15T23:21:09.454639+0000","last_unstale":"2024-05-15T23:24:36.016608+0000","last_undegraded":"2024-05-15T23:24:36.016608+0000","last_fullsized":"2024-05-15T23:24:36.016608+0000","mapping_epoch":407,"log_start":"410'9100","ondisk_log_start":"410'9100","created":374,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'18881","last_scrub_stamp":"2024-05-15T23:24:31.917402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:24:31.917402+0000","objects_scrubbed":273,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:59.276172+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19154,"num_write_kb":612928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8570,"num_bytes_recovered":561643520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"469'18928","reported_seq":19139,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-05-15T23:24:32.188083+0000","last_change":"2024-05-15T23:24:32.036285+0000","last_active":"2024-05-15T23:24:32.188083+0000","last_peered":"2024-05-15T23:24:32.188083+0000","last_clean":"2024-05-15T23:24:32.188083+0000","last_became_active":"2024-05-15T23:19:47.304547+0000","last_became_peered":"2024-05-15T23:19:47.304547+0000","last_unstale":"2024-05-15T23:24:32.188083+0000","last_undegraded":"2024-05-15T23:24:32.188083+0000","last_fullsized":"2024-05-15T23:24:32.188083+0000","mapping_epoch":374,"log_start":"410'8900","ondisk_log_start":"410'8900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'18917","last_scrub_stamp":"2024-05-15T23:24:32.035988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:24:32.035988+0000","objects_scrubbed":288,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:51.721919+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18087936,"num_objects":276,"num_object_clones":0,"num_object_copies":552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18928,"num_write_kb":614528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"469'18834","reported_seq":19035,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-05-15T23:24:32.744359+0000","last_change":"2024-05-15T23:19:47.306634+0000","last_active":"2024-05-15T23:24:32.744359+0000","last_peered":"2024-05-15T23:24:32.744359+0000","last_clean":"2024-05-15T23:24:32.744359+0000","last_became_active":"2024-05-15T23:19:47.306331+0000","last_became_peered":"2024-05-15T23:19:47.306331+0000","last_unstale":"2024-05-15T23:24:32.744359+0000","last_undegraded":"2024-05-15T23:24:32.744359+0000","last_fullsized":"2024-05-15T23:24:32.744359+0000","mapping_epoch":374,"log_start":"410'8800","ondisk_log_start":"410'8800","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14548992,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18834,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"469'19103","reported_seq":19304,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-05-15T23:24:32.756191+0000","last_change":"2024-05-15T23:19:47.306611+0000","last_active":"2024-05-15T23:24:32.756191+0000","last_peered":"2024-05-15T23:24:32.756191+0000","last_clean":"2024-05-15T23:24:32.756191+0000","last_became_active":"2024-05-15T23:19:47.306349+0000","last_became_peered":"2024-05-15T23:19:47.306349+0000","last_unstale":"2024-05-15T23:24:32.756191+0000","last_undegraded":"2024-05-15T23:24:32.756191+0000","last_fullsized":"2024-05-15T23:24:32.756191+0000","mapping_epoch":374,"log_start":"410'9100","ondisk_log_start":"410'9100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13565952,"num_objects":207,"num_object_clones":0,"num_object_copies":414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19103,"num_write_kb":617920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"470'19380","reported_seq":19850,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-05-15T23:24:35.860961+0000","last_change":"2024-05-15T23:23:50.810750+0000","last_active":"2024-05-15T23:24:35.860961+0000","last_peered":"2024-05-15T23:24:35.860961+0000","last_clean":"2024-05-15T23:24:35.860961+0000","last_became_active":"2024-05-15T23:19:47.305685+0000","last_became_peered":"2024-05-15T23:19:47.305685+0000","last_unstale":"2024-05-15T23:24:35.860961+0000","last_undegraded":"2024-05-15T23:24:35.860961+0000","last_fullsized":"2024-05-15T23:24:35.860961+0000","mapping_epoch":374,"log_start":"410'9300","ondisk_log_start":"410'9300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'16478","last_scrub_stamp":"2024-05-15T23:23:50.810636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:50.810636+0000","objects_scrubbed":2939,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:20.026294+0000","scrub_duration":608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19380,"num_write_kb":620672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"469'19075","reported_seq":36661,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-05-15T23:24:32.738310+0000","last_change":"2024-05-15T23:24:17.572412+0000","last_active":"2024-05-15T23:24:32.738310+0000","last_peered":"2024-05-15T23:24:32.738310+0000","last_clean":"2024-05-15T23:24:32.738310+0000","last_became_active":"2024-05-15T23:21:09.400731+0000","last_became_peered":"2024-05-15T23:21:09.400731+0000","last_unstale":"2024-05-15T23:24:32.738310+0000","last_undegraded":"2024-05-15T23:24:32.738310+0000","last_fullsized":"2024-05-15T23:24:32.738310+0000","mapping_epoch":407,"log_start":"410'9000","ondisk_log_start":"410'9000","created":374,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14483456,"num_objects":221,"num_object_clones":0,"num_object_copies":442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19075,"num_write_kb":617472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8690,"num_bytes_recovered":569507840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"470'19149","reported_seq":36616,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-05-15T23:24:35.594827+0000","last_change":"2024-05-15T23:24:19.774443+0000","last_active":"2024-05-15T23:24:35.594827+0000","last_peered":"2024-05-15T23:24:35.594827+0000","last_clean":"2024-05-15T23:24:35.594827+0000","last_became_active":"2024-05-15T23:21:09.221797+0000","last_became_peered":"2024-05-15T23:21:09.221797+0000","last_unstale":"2024-05-15T23:24:35.594827+0000","last_undegraded":"2024-05-15T23:24:35.594827+0000","last_fullsized":"2024-05-15T23:24:35.594827+0000","mapping_epoch":407,"log_start":"410'9100","ondisk_log_start":"410'9100","created":374,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19149,"num_write_kb":613440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8629,"num_bytes_recovered":565510144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.1","version":"469'18591","reported_seq":19001,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-05-15T23:24:34.909970+0000","last_change":"2024-05-15T23:23:50.798735+0000","last_active":"2024-05-15T23:24:34.909970+0000","last_peered":"2024-05-15T23:24:34.909970+0000","last_clean":"2024-05-15T23:24:34.909970+0000","last_became_active":"2024-05-15T23:19:47.516865+0000","last_became_peered":"2024-05-15T23:19:47.516865+0000","last_unstale":"2024-05-15T23:24:34.909970+0000","last_undegraded":"2024-05-15T23:24:34.909970+0000","last_fullsized":"2024-05-15T23:24:34.909970+0000","mapping_epoch":374,"log_start":"410'8500","ondisk_log_start":"410'8500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'15864","last_scrub_stamp":"2024-05-15T23:23:50.798612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:50.798612+0000","objects_scrubbed":2807,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:56.167477+0000","scrub_duration":584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4390912,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18591,"num_write_kb":597056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"470'19246","reported_seq":19448,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-05-15T23:24:35.645046+0000","last_change":"2024-05-15T23:19:47.305723+0000","last_active":"2024-05-15T23:24:35.645046+0000","last_peered":"2024-05-15T23:24:35.645046+0000","last_clean":"2024-05-15T23:24:35.645046+0000","last_became_active":"2024-05-15T23:19:47.305459+0000","last_became_peered":"2024-05-15T23:19:47.305459+0000","last_unstale":"2024-05-15T23:24:35.645046+0000","last_undegraded":"2024-05-15T23:24:35.645046+0000","last_fullsized":"2024-05-15T23:24:35.645046+0000","mapping_epoch":374,"log_start":"410'9200","ondisk_log_start":"410'9200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1966080,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19246,"num_write_kb":616832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"470'19424","reported_seq":19627,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-05-15T23:24:35.628926+0000","last_change":"2024-05-15T23:19:47.307499+0000","last_active":"2024-05-15T23:24:35.628926+0000","last_peered":"2024-05-15T23:24:35.628926+0000","last_clean":"2024-05-15T23:24:35.628926+0000","last_became_active":"2024-05-15T23:19:47.307149+0000","last_became_peered":"2024-05-15T23:19:47.307149+0000","last_unstale":"2024-05-15T23:24:35.628926+0000","last_undegraded":"2024-05-15T23:24:35.628926+0000","last_fullsized":"2024-05-15T23:24:35.628926+0000","mapping_epoch":374,"log_start":"411'9400","ondisk_log_start":"411'9400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19424,"num_write_kb":622272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":988,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-05-15T23:24:35.576699+0000","last_change":"2024-05-15T23:24:32.827864+0000","last_active":"2024-05-15T23:24:35.576699+0000","last_peered":"2024-05-15T23:24:35.576699+0000","last_clean":"2024-05-15T23:24:35.576699+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:24:35.576699+0000","last_undegraded":"2024-05-15T23:24:35.576699+0000","last_fullsized":"2024-05-15T23:24:35.576699+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:24:32.827775+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:24:32.827775+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:42.135474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":110624792,"num_objects":1689,"num_object_clones":0,"num_object_copies":3378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1689,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":305427,"num_write_kb":9827649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25889,"num_bytes_recovered":1696661504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1813782528,"data_stored":1813774384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351225,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":632700,"kb_used_data":55860,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93112452,"statfs":{"total":95995035648,"available":95347150848,"internally_reserved":0,"allocated":57200640,"data_stored":56651620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":590608836},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580159,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308796,"kb_used_data":24884,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93436356,"statfs":{"total":95995035648,"available":95678828544,"internally_reserved":0,"allocated":25481216,"data_stored":24933786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":290650560},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776391,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":406412,"kb_used_data":5952,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93338740,"statfs":{"total":95995035648,"available":95578869760,"internally_reserved":0,"allocated":6094848,"data_stored":5541364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":410057152},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005323,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":219712,"kb_used_data":21940,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93525440,"statfs":{"total":95995035648,"available":95770050560,"internally_reserved":0,"allocated":22466560,"data_stored":21919130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266959,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":378964,"kb_used_data":4744,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93366188,"statfs":{"total":95995035648,"available":95606976512,"internally_reserved":0,"allocated":4857856,"data_stored":4300820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":383056330},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495893,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":514492,"kb_used_data":18164,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93230660,"statfs":{"total":95995035648,"available":95468195840,"internally_reserved":0,"allocated":18599936,"data_stored":18052506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":508164550},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016561,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483249,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149990,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":452604,"kb_used_data":6324,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93292548,"statfs":{"total":95995035648,"available":95531569152,"internally_reserved":0,"allocated":6475776,"data_stored":5928346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":456980935},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986462,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":441788,"kb_used_data":31540,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93303364,"statfs":{"total":95995035648,"available":95542644736,"internally_reserved":0,"allocated":32296960,"data_stored":31749530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790234,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":510276,"kb_used_data":109816,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93234876,"statfs":{"total":95995035648,"available":95472513024,"internally_reserved":0,"allocated":112451584,"data_stored":111900082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2883584,"data_stored":2883584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":524288,"data_stored":524288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18415616,"data_stored":18415616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":102240256,"data_stored":102236184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1610350592,"data_stored":1610350592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":13697024,"data_stored":13697024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4390912,"data_stored":4390912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":9043968,"data_stored":9043968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":52236288,"data_stored":52232216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:24:38.255 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:38.282 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:38.405 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:24:38.405 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:24:38.530 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:38.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:38.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:38.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:24:38.661 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:38.775 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:38 smithi090 ceph-mon[44434]: from='client.16362 v1:172.21.15.84:0/3326839950' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:24:38.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:38 smithi088 ceph-mon[44699]: from='client.16362 v1:172.21.15.84:0/3326839950' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:24:38.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:38 smithi084 ceph-mon[39784]: from='client.16362 v1:172.21.15.84:0/3326839950' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:24:39.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T23:24:39.361 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:24:39.612 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:24:39.615 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-05-15T23:24:39.615 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-05-15T23:24:39.810 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:24:39.812 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:39 smithi084 ceph-mon[39784]: 5.f deep-scrub starts 2024-05-15T23:24:39.812 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:39 smithi084 ceph-mon[39784]: 5.f deep-scrub ok 2024-05-15T23:24:39.812 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:39 smithi084 ceph-mon[39784]: 5.4 scrub starts 2024-05-15T23:24:39.813 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:39 smithi084 ceph-mon[39784]: 5.4 scrub ok 2024-05-15T23:24:39.813 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:39 smithi084 ceph-mon[39784]: pgmap v1271: 17 pgs: 17 active+clean; 17 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 986 op/s 2024-05-15T23:24:39.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:39.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:24:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:39 smithi088 ceph-mon[44699]: 5.f deep-scrub starts 2024-05-15T23:24:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:39 smithi088 ceph-mon[44699]: 5.f deep-scrub ok 2024-05-15T23:24:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:39 smithi088 ceph-mon[44699]: 5.4 scrub starts 2024-05-15T23:24:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:39 smithi088 ceph-mon[44699]: 5.4 scrub ok 2024-05-15T23:24:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:39 smithi088 ceph-mon[44699]: pgmap v1271: 17 pgs: 17 active+clean; 17 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 986 op/s 2024-05-15T23:24:39.848 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:39 smithi090 ceph-mon[44434]: 5.f deep-scrub starts 2024-05-15T23:24:39.848 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:39 smithi090 ceph-mon[44434]: 5.f deep-scrub ok 2024-05-15T23:24:39.848 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:39 smithi090 ceph-mon[44434]: 5.4 scrub starts 2024-05-15T23:24:39.848 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:39 smithi090 ceph-mon[44434]: 5.4 scrub ok 2024-05-15T23:24:39.848 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:39 smithi090 ceph-mon[44434]: pgmap v1271: 17 pgs: 17 active+clean; 17 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 986 op/s 2024-05-15T23:24:39.914 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:24:39.948 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:39.970 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:24:39.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:39.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:24:40.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:24:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:40.577 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:40.783 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:24:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:40.786 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:24:40.887 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:40.917 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:40.938 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:40.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:40.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:41.042 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:24:41.061 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:24:41.062 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:24:41.071 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:41.090 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:24:41.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:41.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:24:41.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:41 smithi084 ceph-mon[39784]: pgmap v1272: 17 pgs: 17 active+clean; 577 KiB data, 3.4 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 640 op/s 2024-05-15T23:24:41.195 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:24:41.222 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:41.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:24:41.451 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:41 smithi090 ceph-mon[44434]: pgmap v1272: 17 pgs: 17 active+clean; 577 KiB data, 3.4 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 640 op/s 2024-05-15T23:24:41.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:41 smithi088 ceph-mon[44699]: pgmap v1272: 17 pgs: 17 active+clean; 577 KiB data, 3.4 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 640 op/s 2024-05-15T23:24:41.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T23:24:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:24:41.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:41.773 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:24:41.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:24:41.901 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:41.919 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:24:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:41.923 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:24:42.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:42.038 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:24:42.051 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:42.072 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:42.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:42.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:42.176 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:42.203 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:42.217 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:24:42.218 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:24:42.218 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:24:42.218 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:24:42.218 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:24:42.218 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:24:42.219 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:24:42.219 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:24:42.219 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:24:42.219 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:24:42.219 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:24:42.219 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:24:42.220 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:24:42.220 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:24:42.220 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:24:42.220 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:24:42.220 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:24:42.220 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:24:42.220 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:24:42.221 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:24:42.221 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:24:42.221 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:24:42.221 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:24:42.221 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:24:42.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:42 smithi090 ceph-mon[44434]: 5.a deep-scrub starts 2024-05-15T23:24:42.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:42 smithi090 ceph-mon[44434]: 5.a deep-scrub ok 2024-05-15T23:24:42.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:42 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2927262628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:42.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:42 smithi084 ceph-mon[39784]: 5.a deep-scrub starts 2024-05-15T23:24:42.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:42 smithi084 ceph-mon[39784]: 5.a deep-scrub ok 2024-05-15T23:24:42.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:42 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2927262628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:42.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:42 smithi088 ceph-mon[44699]: 5.a deep-scrub starts 2024-05-15T23:24:42.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:42 smithi088 ceph-mon[44699]: 5.a deep-scrub ok 2024-05-15T23:24:42.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:42 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2927262628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:42.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:42.862 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:42.876 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:43.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:43 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:43.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:43 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:43.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:43 smithi084 ceph-mon[39784]: 5.2 scrub starts 2024-05-15T23:24:43.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:43 smithi084 ceph-mon[39784]: 5.2 scrub ok 2024-05-15T23:24:43.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:43 smithi084 ceph-mon[39784]: pgmap v1273: 17 pgs: 17 active+clean; 577 KiB data, 3.4 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 600 op/s 2024-05-15T23:24:43.201 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:43 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:43.201 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:43 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:43.201 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:43 smithi090 ceph-mon[44434]: 5.2 scrub starts 2024-05-15T23:24:43.201 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:43 smithi090 ceph-mon[44434]: 5.2 scrub ok 2024-05-15T23:24:43.201 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:43 smithi090 ceph-mon[44434]: pgmap v1273: 17 pgs: 17 active+clean; 577 KiB data, 3.4 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 600 op/s 2024-05-15T23:24:43.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:43 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:43.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:43 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:43.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:43 smithi088 ceph-mon[44699]: 5.2 scrub starts 2024-05-15T23:24:43.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:43 smithi088 ceph-mon[44699]: 5.2 scrub ok 2024-05-15T23:24:43.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:43 smithi088 ceph-mon[44699]: pgmap v1273: 17 pgs: 17 active+clean; 577 KiB data, 3.4 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 600 op/s 2024-05-15T23:24:43.752 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:43.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:43.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:137212 5.2 5:464b809f:::benchmark_data_smithi090_116470_object137211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e407)", 2024-05-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:21:09.055224+0000", 2024-05-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 214.70108926899999, 2024-05-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19662302300000001, 2024-05-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 137212 2024-05-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055224+0000", 2024-05-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055224+0000", 2024-05-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055229+0000", 2024-05-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2650000000000004e-06 2024-05-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055290+0000", 2024-05-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0659000000000003e-05 2024-05-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055315+0000", 2024-05-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5089999999999999e-05 2024-05-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055336+0000", 2024-05-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1308e-05 2024-05-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.056970+0000", 2024-05-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016330590000000001 2024-05-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.056983+0000", 2024-05-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3324e-05 2024-05-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.230289+0000", 2024-05-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17330585000000001 2024-05-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.232525+0000", 2024-05-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002236371 2024-05-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.232646+0000", 2024-05-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012105899999999999 2024-05-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.243624+0000", 2024-05-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010977519 2024-05-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.249752+0000", 2024-05-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061286020000000004 2024-05-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.249809+0000", 2024-05-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7015999999999998e-05 2024-05-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.251847+0000", 2024-05-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020379009999999999 2024-05-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:137224 5.2 5:4a2fdf83:::benchmark_data_smithi090_116470_object137223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e407)", 2024-05-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:21:09.055374+0000", 2024-05-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 214.700939226, 2024-05-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.222436774, 2024-05-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 137224 2024-05-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055374+0000", 2024-05-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055374+0000", 2024-05-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055376+0000", 2024-05-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.344e-06 2024-05-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055479+0000", 2024-05-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103572 2024-05-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055492+0000", 2024-05-15T23:24:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2476e-05 2024-05-15T23:24:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055501+0000", 2024-05-15T23:24:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8020000000000003e-06 2024-05-15T23:24:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.057010+0000", 2024-05-15T23:24:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001508398 2024-05-15T23:24:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:24:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.057017+0000", 2024-05-15T23:24:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7819999999999996e-06 2024-05-15T23:24:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.238054+0000", 2024-05-15T23:24:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.181037584 2024-05-15T23:24:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.238164+0000", 2024-05-15T23:24:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109669 2024-05-15T23:24:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.238285+0000", 2024-05-15T23:24:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121564 2024-05-15T23:24:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.251856+0000", 2024-05-15T23:24:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013570334 2024-05-15T23:24:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.275543+0000", 2024-05-15T23:24:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023687666 2024-05-15T23:24:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.275618+0000", 2024-05-15T23:24:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4657e-05 2024-05-15T23:24:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.277811+0000", 2024-05-15T23:24:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002192926 2024-05-15T23:24:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:137225 5.2 5:4b6dbee4:::benchmark_data_smithi090_116470_object137224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e407)", 2024-05-15T23:24:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:21:09.055512+0000", 2024-05-15T23:24:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 214.70080134, 2024-05-15T23:24:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22482041799999999, 2024-05-15T23:24:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 137225 2024-05-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055512+0000", 2024-05-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055512+0000", 2024-05-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055513+0000", 2024-05-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9299999999999997e-07 2024-05-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055543+0000", 2024-05-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0385000000000001e-05 2024-05-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055551+0000", 2024-05-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.272e-06 2024-05-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055555+0000", 2024-05-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3080000000000001e-06 2024-05-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.057041+0000", 2024-05-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001486069 2024-05-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.057047+0000", 2024-05-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2439999999999998e-06 2024-05-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.241359+0000", 2024-05-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18431156200000001 2024-05-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.241440+0000", 2024-05-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1303000000000004e-05 2024-05-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.241538+0000", 2024-05-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7639999999999994e-05 2024-05-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.251859+0000", 2024-05-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010321545 2024-05-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.277889+0000", 2024-05-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026029904 2024-05-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.277950+0000", 2024-05-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1321000000000005e-05 2024-05-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.280332+0000", 2024-05-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023820719999999998 2024-05-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:137241 5.2 5:489254f4:::benchmark_data_smithi090_116470_object137240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e407)", 2024-05-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:21:09.055562+0000", 2024-05-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 214.70075115899999, 2024-05-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22730246500000001, 2024-05-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 137241 2024-05-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055562+0000", 2024-05-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055562+0000", 2024-05-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055563+0000", 2024-05-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5599999999999995e-07 2024-05-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055604+0000", 2024-05-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1254000000000002e-05 2024-05-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055610+0000", 2024-05-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2439999999999998e-06 2024-05-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055614+0000", 2024-05-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.72e-06 2024-05-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.057072+0000", 2024-05-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001458095 2024-05-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.057077+0000", 2024-05-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1209999999999998e-06 2024-05-15T23:24:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.243678+0000", 2024-05-15T23:24:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18660078599999999 2024-05-15T23:24:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.243767+0000", 2024-05-15T23:24:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8941000000000001e-05 2024-05-15T23:24:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.243881+0000", 2024-05-15T23:24:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114404 2024-05-15T23:24:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.251866+0000", 2024-05-15T23:24:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079848430000000001 2024-05-15T23:24:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.280382+0000", 2024-05-15T23:24:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028516258999999999 2024-05-15T23:24:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.280419+0000", 2024-05-15T23:24:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6256e-05 2024-05-15T23:24:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.282865+0000", 2024-05-15T23:24:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024459859999999998 2024-05-15T23:24:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:137274 5.2 5:46696d63:::benchmark_data_smithi090_116470_object137273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e407)", 2024-05-15T23:24:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:21:09.055619+0000", 2024-05-15T23:24:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 214.70069435900001, 2024-05-15T23:24:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22972378399999999, 2024-05-15T23:24:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 137274 2024-05-15T23:24:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055619+0000", 2024-05-15T23:24:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055619+0000", 2024-05-15T23:24:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055620+0000", 2024-05-15T23:24:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9299999999999998e-07 2024-05-15T23:24:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055661+0000", 2024-05-15T23:24:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1100999999999998e-05 2024-05-15T23:24:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055667+0000", 2024-05-15T23:24:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3500000000000002e-06 2024-05-15T23:24:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055671+0000", 2024-05-15T23:24:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0289999999999999e-06 2024-05-15T23:24:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.057101+0000", 2024-05-15T23:24:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001429804 2024-05-15T23:24:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:24:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.057105+0000", 2024-05-15T23:24:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8840000000000004e-06 2024-05-15T23:24:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.245759+0000", 2024-05-15T23:24:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18865458700000001 2024-05-15T23:24:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.245843+0000", 2024-05-15T23:24:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4169e-05 2024-05-15T23:24:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.245943+0000", 2024-05-15T23:24:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9207000000000001e-05 2024-05-15T23:24:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.251870+0000", 2024-05-15T23:24:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0059269099999999996 2024-05-15T23:24:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.283457+0000", 2024-05-15T23:24:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031587478000000002 2024-05-15T23:24:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.283516+0000", 2024-05-15T23:24:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8899e-05 2024-05-15T23:24:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.285343+0000", 2024-05-15T23:24:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018268729999999999 2024-05-15T23:24:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:137275 5.2 5:47612e21:::benchmark_data_smithi090_116470_object137274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e407)", 2024-05-15T23:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:21:09.055676+0000", 2024-05-15T23:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 214.70063703599999, 2024-05-15T23:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.231978464, 2024-05-15T23:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 137275 2024-05-15T23:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055676+0000", 2024-05-15T23:24:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055676+0000", 2024-05-15T23:24:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055677+0000", 2024-05-15T23:24:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3000000000000001e-07 2024-05-15T23:24:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055718+0000", 2024-05-15T23:24:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1638999999999999e-05 2024-05-15T23:24:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055725+0000", 2024-05-15T23:24:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0759999999999999e-06 2024-05-15T23:24:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055728+0000", 2024-05-15T23:24:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6449999999999998e-06 2024-05-15T23:24:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.057125+0000", 2024-05-15T23:24:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001396792 2024-05-15T23:24:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:24:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.057129+0000", 2024-05-15T23:24:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8010000000000001e-06 2024-05-15T23:24:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.247847+0000", 2024-05-15T23:24:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19071782100000001 2024-05-15T23:24:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.247962+0000", 2024-05-15T23:24:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011519 2024-05-15T23:24:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.248074+0000", 2024-05-15T23:24:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111953 2024-05-15T23:24:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.251873+0000", 2024-05-15T23:24:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037992970000000001 2024-05-15T23:24:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.285414+0000", 2024-05-15T23:24:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033540984000000003 2024-05-15T23:24:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.285472+0000", 2024-05-15T23:24:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7701000000000002e-05 2024-05-15T23:24:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.287655+0000", 2024-05-15T23:24:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021831350000000001 2024-05-15T23:24:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:244173 5.2 5:4c09fe4f:::benchmark_data_smithi090_116470_object90613:head [delete] snapc 0=[] ondisk+write+known_if_redirected e448)", 2024-05-15T23:24:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:23:31.340639+0000", 2024-05-15T23:24:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 72.415674339999995, 2024-05-15T23:24:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.155972844, 2024-05-15T23:24:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 244173 2024-05-15T23:24:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.340639+0000", 2024-05-15T23:24:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.340639+0000", 2024-05-15T23:24:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.340640+0000", 2024-05-15T23:24:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4100000000000001e-06 2024-05-15T23:24:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.340645+0000", 2024-05-15T23:24:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9599999999999999e-06 2024-05-15T23:24:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.340653+0000", 2024-05-15T23:24:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9340000000000006e-06 2024-05-15T23:24:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.340661+0000", 2024-05-15T23:24:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.622e-06 2024-05-15T23:24:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.340817+0000", 2024-05-15T23:24:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015564800000000001 2024-05-15T23:24:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:24:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.340878+0000", 2024-05-15T23:24:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1686000000000005e-05 2024-05-15T23:24:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.347913+0000", 2024-05-15T23:24:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0070347980000000001 2024-05-15T23:24:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.348005+0000", 2024-05-15T23:24:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2145000000000006e-05 2024-05-15T23:24:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.348139+0000", 2024-05-15T23:24:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133788 2024-05-15T23:24:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.357037+0000", 2024-05-15T23:24:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0088982450000000008 2024-05-15T23:24:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.495013+0000", 2024-05-15T23:24:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13797609499999999 2024-05-15T23:24:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.495078+0000", 2024-05-15T23:24:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4327999999999998e-05 2024-05-15T23:24:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.496612+0000", 2024-05-15T23:24:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001534185 2024-05-15T23:24:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:244190 5.2 5:439501f6:::benchmark_data_smithi090_116470_object90630:head [delete] snapc 0=[] ondisk+write+known_if_redirected e448)", 2024-05-15T23:24:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:23:31.356317+0000", 2024-05-15T23:24:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 72.399996181000006, 2024-05-15T23:24:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16502819199999999, 2024-05-15T23:24:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-05-15T23:24:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.868 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.868 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.868 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 244190 2024-05-15T23:24:43.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.868 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.356317+0000", 2024-05-15T23:24:43.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.356317+0000", 2024-05-15T23:24:43.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.356319+0000", 2024-05-15T23:24:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5999999999999999e-06 2024-05-15T23:24:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.356324+0000", 2024-05-15T23:24:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1050000000000003e-06 2024-05-15T23:24:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.356334+0000", 2024-05-15T23:24:43.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0434e-05 2024-05-15T23:24:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.356342+0000", 2024-05-15T23:24:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.481e-06 2024-05-15T23:24:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.356982+0000", 2024-05-15T23:24:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000640217 2024-05-15T23:24:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:24:43.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.357028+0000", 2024-05-15T23:24:43.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5637999999999999e-05 2024-05-15T23:24:43.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.521330+0000", 2024-05-15T23:24:43.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.164302807 2024-05-15T23:24:43.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:31.521345+0000", 2024-05-15T23:24:43.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.491e-05 2024-05-15T23:24:43.871 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.871 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.871 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.872 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:244726 5.2 5:4b091763:::benchmark_data_smithi090_116470_object91166:head [delete] snapc 0=[] ondisk+write+known_if_redirected e449)", 2024-05-15T23:24:43.872 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:23:32.017999+0000", 2024-05-15T23:24:43.872 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.738314122000006, 2024-05-15T23:24:43.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14906883900000001, 2024-05-15T23:24:43.872 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.872 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.872 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.872 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 244726 2024-05-15T23:24:43.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.872 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.017999+0000", 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.017999+0000", 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.018001+0000", 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.018007+0000", 2024-05-15T23:24:43.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8440000000000003e-06 2024-05-15T23:24:43.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.018020+0000", 2024-05-15T23:24:43.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2969000000000001e-05 2024-05-15T23:24:43.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.018029+0000", 2024-05-15T23:24:43.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2024-05-15T23:24:43.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.018746+0000", 2024-05-15T23:24:43.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071723999999999996 2024-05-15T23:24:43.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:24:43.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.018806+0000", 2024-05-15T23:24:43.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.995e-05 2024-05-15T23:24:43.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.113230+0000", 2024-05-15T23:24:43.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094424354000000002 2024-05-15T23:24:43.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.113326+0000", 2024-05-15T23:24:43.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5538000000000006e-05 2024-05-15T23:24:43.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.113464+0000", 2024-05-15T23:24:43.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013851400000000001 2024-05-15T23:24:43.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.124076+0000", 2024-05-15T23:24:43.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010612228 2024-05-15T23:24:43.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.165946+0000", 2024-05-15T23:24:43.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041869854999999997 2024-05-15T23:24:43.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.165979+0000", 2024-05-15T23:24:43.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2459000000000002e-05 2024-05-15T23:24:43.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.167068+0000", 2024-05-15T23:24:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010894209999999999 2024-05-15T23:24:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:244896 5.2 5:4f31b821:::benchmark_data_smithi090_116470_object91336:head [delete] snapc 0=[] ondisk+write+known_if_redirected e450)", 2024-05-15T23:24:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:23:32.223553+0000", 2024-05-15T23:24:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.532760819999993, 2024-05-15T23:24:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15303488100000001, 2024-05-15T23:24:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.879 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.879 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.879 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.879 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.879 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 244896 2024-05-15T23:24:43.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.879 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.223553+0000", 2024-05-15T23:24:43.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.223553+0000", 2024-05-15T23:24:43.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.223554+0000", 2024-05-15T23:24:43.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5039999999999999e-06 2024-05-15T23:24:43.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.223560+0000", 2024-05-15T23:24:43.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.913e-06 2024-05-15T23:24:43.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.223573+0000", 2024-05-15T23:24:43.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2801e-05 2024-05-15T23:24:43.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.223582+0000", 2024-05-15T23:24:43.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8759999999999994e-06 2024-05-15T23:24:43.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.228757+0000", 2024-05-15T23:24:43.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00517527 2024-05-15T23:24:43.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:24:43.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.229360+0000", 2024-05-15T23:24:43.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060303699999999995 2024-05-15T23:24:43.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.337287+0000", 2024-05-15T23:24:43.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10792689 2024-05-15T23:24:43.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.337366+0000", 2024-05-15T23:24:43.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9345999999999996e-05 2024-05-15T23:24:43.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.337454+0000", 2024-05-15T23:24:43.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8041000000000006e-05 2024-05-15T23:24:43.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.347552+0000", 2024-05-15T23:24:43.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010097971000000001 2024-05-15T23:24:43.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.375354+0000", 2024-05-15T23:24:43.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027801968 2024-05-15T23:24:43.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.375403+0000", 2024-05-15T23:24:43.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9153000000000002e-05 2024-05-15T23:24:43.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.376587+0000", 2024-05-15T23:24:43.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011841110000000001 2024-05-15T23:24:43.886 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.886 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.886 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.886 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:245008 5.2 5:4bd4b5a3:::benchmark_data_smithi090_116470_object91448:head [delete] snapc 0=[] ondisk+write+known_if_redirected e450)", 2024-05-15T23:24:43.887 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:23:32.391517+0000", 2024-05-15T23:24:43.887 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.364796138000003, 2024-05-15T23:24:43.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15896173699999999, 2024-05-15T23:24:43.887 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.887 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.887 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.887 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.887 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.887 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.887 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 245008 2024-05-15T23:24:43.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.887 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.391517+0000", 2024-05-15T23:24:43.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.391517+0000", 2024-05-15T23:24:43.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.391519+0000", 2024-05-15T23:24:43.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.097e-06 2024-05-15T23:24:43.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.391527+0000", 2024-05-15T23:24:43.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5170000000000002e-06 2024-05-15T23:24:43.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.391541+0000", 2024-05-15T23:24:43.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3987e-05 2024-05-15T23:24:43.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.391550+0000", 2024-05-15T23:24:43.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3330000000000003e-06 2024-05-15T23:24:43.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.402429+0000", 2024-05-15T23:24:43.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010878909000000001 2024-05-15T23:24:43.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:24:43.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.403858+0000", 2024-05-15T23:24:43.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014286310000000001 2024-05-15T23:24:43.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.430671+0000", 2024-05-15T23:24:43.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026813605000000001 2024-05-15T23:24:43.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.430768+0000", 2024-05-15T23:24:43.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7046000000000006e-05 2024-05-15T23:24:43.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.430912+0000", 2024-05-15T23:24:43.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014318600000000001 2024-05-15T23:24:43.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.440135+0000", 2024-05-15T23:24:43.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092235440000000002 2024-05-15T23:24:43.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.549593+0000", 2024-05-15T23:24:43.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10945772400000001 2024-05-15T23:24:43.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.549629+0000", 2024-05-15T23:24:43.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.659e-05 2024-05-15T23:24:43.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.550479+0000", 2024-05-15T23:24:43.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084956799999999998 2024-05-15T23:24:43.893 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.893 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.893 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.893 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:245016 5.2 5:4ea62511:::benchmark_data_smithi090_116470_object91456:head [delete] snapc 0=[] ondisk+write+known_if_redirected e450)", 2024-05-15T23:24:43.893 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:23:32.401190+0000", 2024-05-15T23:24:43.893 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.355123172999996, 2024-05-15T23:24:43.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.149936657, 2024-05-15T23:24:43.893 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.894 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.894 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.894 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.894 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.894 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.894 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 245016 2024-05-15T23:24:43.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.894 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.401190+0000", 2024-05-15T23:24:43.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.401190+0000", 2024-05-15T23:24:43.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.401192+0000", 2024-05-15T23:24:43.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.232e-06 2024-05-15T23:24:43.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.401209+0000", 2024-05-15T23:24:43.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6232e-05 2024-05-15T23:24:43.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.401219+0000", 2024-05-15T23:24:43.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0332999999999999e-05 2024-05-15T23:24:43.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.401225+0000", 2024-05-15T23:24:43.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2299999999999996e-06 2024-05-15T23:24:43.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.403946+0000", 2024-05-15T23:24:43.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027203779999999999 2024-05-15T23:24:43.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:24:43.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.404513+0000", 2024-05-15T23:24:43.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056743400000000002 2024-05-15T23:24:43.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.444114+0000", 2024-05-15T23:24:43.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039601198999999997 2024-05-15T23:24:43.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.444179+0000", 2024-05-15T23:24:43.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4999000000000006e-05 2024-05-15T23:24:43.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.444285+0000", 2024-05-15T23:24:43.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000105618 2024-05-15T23:24:43.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.455307+0000", 2024-05-15T23:24:43.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011022323000000001 2024-05-15T23:24:43.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.550516+0000", 2024-05-15T23:24:43.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095209243999999998 2024-05-15T23:24:43.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.550549+0000", 2024-05-15T23:24:43.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2660999999999998e-05 2024-05-15T23:24:43.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.551127+0000", 2024-05-15T23:24:43.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057777399999999997 2024-05-15T23:24:43.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.899 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.900 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:245019 5.2 5:4dba47af:::benchmark_data_smithi090_116470_object91459:head [delete] snapc 0=[] ondisk+write+known_if_redirected e450)", 2024-05-15T23:24:43.900 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:23:32.401527+0000", 2024-05-15T23:24:43.900 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.354786813999993, 2024-05-15T23:24:43.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15046275200000001, 2024-05-15T23:24:43.900 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.900 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.900 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.900 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.900 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.901 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.901 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 245019 2024-05-15T23:24:43.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.901 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.401527+0000", 2024-05-15T23:24:43.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.401527+0000", 2024-05-15T23:24:43.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.401528+0000", 2024-05-15T23:24:43.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6190000000000001e-06 2024-05-15T23:24:43.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.401534+0000", 2024-05-15T23:24:43.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5470000000000004e-06 2024-05-15T23:24:43.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.401544+0000", 2024-05-15T23:24:43.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0644e-05 2024-05-15T23:24:43.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.401550+0000", 2024-05-15T23:24:43.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8939999999999998e-06 2024-05-15T23:24:43.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.404545+0000", 2024-05-15T23:24:43.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002994366 2024-05-15T23:24:43.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:24:43.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.405143+0000", 2024-05-15T23:24:43.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059813900000000003 2024-05-15T23:24:43.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.455350+0000", 2024-05-15T23:24:43.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050207556 2024-05-15T23:24:43.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.455445+0000", 2024-05-15T23:24:43.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4401000000000006e-05 2024-05-15T23:24:43.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.455593+0000", 2024-05-15T23:24:43.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148042 2024-05-15T23:24:43.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.469969+0000", 2024-05-15T23:24:43.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014375847000000001 2024-05-15T23:24:43.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.551182+0000", 2024-05-15T23:24:43.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.081213140000000003 2024-05-15T23:24:43.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.551228+0000", 2024-05-15T23:24:43.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6072000000000002e-05 2024-05-15T23:24:43.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.551989+0000", 2024-05-15T23:24:43.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000761485 2024-05-15T23:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:245023 5.2 5:4cbb571a:::benchmark_data_smithi090_116470_object91463:head [delete] snapc 0=[] ondisk+write+known_if_redirected e450)", 2024-05-15T23:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:23:32.407909+0000", 2024-05-15T23:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.348404478999996, 2024-05-15T23:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14964609800000001, 2024-05-15T23:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 245023 2024-05-15T23:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.407909+0000", 2024-05-15T23:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.407909+0000", 2024-05-15T23:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.407910+0000", 2024-05-15T23:24:43.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.358e-06 2024-05-15T23:24:43.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.407915+0000", 2024-05-15T23:24:43.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7829999999999998e-06 2024-05-15T23:24:43.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.407924+0000", 2024-05-15T23:24:43.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2639999999999997e-06 2024-05-15T23:24:43.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.407930+0000", 2024-05-15T23:24:43.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7470000000000002e-06 2024-05-15T23:24:43.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.412112+0000", 2024-05-15T23:24:43.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041819500000000003 2024-05-15T23:24:43.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:24:43.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.412158+0000", 2024-05-15T23:24:43.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5627999999999997e-05 2024-05-15T23:24:43.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.506854+0000", 2024-05-15T23:24:43.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094696187000000001 2024-05-15T23:24:43.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.506912+0000", 2024-05-15T23:24:43.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7739999999999999e-05 2024-05-15T23:24:43.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.506987+0000", 2024-05-15T23:24:43.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5556000000000006e-05 2024-05-15T23:24:43.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.520319+0000", 2024-05-15T23:24:43.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013331793 2024-05-15T23:24:43.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.556159+0000", 2024-05-15T23:24:43.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035839844000000003 2024-05-15T23:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.556217+0000", 2024-05-15T23:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8125999999999999e-05 2024-05-15T23:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.557555+0000", 2024-05-15T23:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001338122 2024-05-15T23:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.914 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:245032 5.2 5:40c9bdc8:::benchmark_data_smithi090_116470_object91472:head [delete] snapc 0=[] ondisk+write+known_if_redirected e450)", 2024-05-15T23:24:43.914 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:23:32.415697+0000", 2024-05-15T23:24:43.914 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.340616643999994, 2024-05-15T23:24:43.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.150150541, 2024-05-15T23:24:43.914 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.914 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.914 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.914 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.914 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.915 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.915 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 245032 2024-05-15T23:24:43.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.915 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.415697+0000", 2024-05-15T23:24:43.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.415697+0000", 2024-05-15T23:24:43.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.415698+0000", 2024-05-15T23:24:43.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6300000000000001e-06 2024-05-15T23:24:43.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.415704+0000", 2024-05-15T23:24:43.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1880000000000001e-06 2024-05-15T23:24:43.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.415713+0000", 2024-05-15T23:24:43.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2900000000000008e-06 2024-05-15T23:24:43.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.415720+0000", 2024-05-15T23:24:43.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2024-05-15T23:24:43.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.416495+0000", 2024-05-15T23:24:43.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077515899999999998 2024-05-15T23:24:43.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:24:43.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.416984+0000", 2024-05-15T23:24:43.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048837499999999999 2024-05-15T23:24:43.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.516354+0000", 2024-05-15T23:24:43.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099370017000000005 2024-05-15T23:24:43.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.516418+0000", 2024-05-15T23:24:43.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4918000000000004e-05 2024-05-15T23:24:43.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.516523+0000", 2024-05-15T23:24:43.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010491 2024-05-15T23:24:43.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.526587+0000", 2024-05-15T23:24:43.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010063466 2024-05-15T23:24:43.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.564680+0000", 2024-05-15T23:24:43.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038092979999999999 2024-05-15T23:24:43.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.564740+0000", 2024-05-15T23:24:43.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0647999999999999e-05 2024-05-15T23:24:43.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.565847+0000", 2024-05-15T23:24:43.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011067970000000001 2024-05-15T23:24:43.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.920 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.921 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.921 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:245034 5.2 5:4b13c27e:::benchmark_data_smithi090_116470_object91474:head [delete] snapc 0=[] ondisk+write+known_if_redirected e450)", 2024-05-15T23:24:43.921 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:23:32.417347+0000", 2024-05-15T23:24:43.921 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.338966384000003, 2024-05-15T23:24:43.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14919851100000001, 2024-05-15T23:24:43.921 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.921 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.921 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.921 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 245034 2024-05-15T23:24:43.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.922 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.417347+0000", 2024-05-15T23:24:43.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.417347+0000", 2024-05-15T23:24:43.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.417348+0000", 2024-05-15T23:24:43.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3659999999999999e-06 2024-05-15T23:24:43.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.417353+0000", 2024-05-15T23:24:43.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8899999999999998e-06 2024-05-15T23:24:43.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.417362+0000", 2024-05-15T23:24:43.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2430000000000005e-06 2024-05-15T23:24:43.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.417369+0000", 2024-05-15T23:24:43.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3309999999999998e-06 2024-05-15T23:24:43.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.417419+0000", 2024-05-15T23:24:43.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.041e-05 2024-05-15T23:24:43.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:24:43.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.417865+0000", 2024-05-15T23:24:43.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044530200000000001 2024-05-15T23:24:43.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.522031+0000", 2024-05-15T23:24:43.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10416671800000001 2024-05-15T23:24:43.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.522098+0000", 2024-05-15T23:24:43.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6902000000000004e-05 2024-05-15T23:24:43.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.522185+0000", 2024-05-15T23:24:43.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6805000000000007e-05 2024-05-15T23:24:43.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.531904+0000", 2024-05-15T23:24:43.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0097186939999999999 2024-05-15T23:24:43.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.565894+0000", 2024-05-15T23:24:43.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033990549000000002 2024-05-15T23:24:43.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.565928+0000", 2024-05-15T23:24:43.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4276000000000002e-05 2024-05-15T23:24:43.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.566546+0000", 2024-05-15T23:24:43.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000617025 2024-05-15T23:24:43.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.927 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.928 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:245040 5.2 5:420b5397:::benchmark_data_smithi090_116470_object91480:head [delete] snapc 0=[] ondisk+write+known_if_redirected e450)", 2024-05-15T23:24:43.928 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:23:32.423317+0000", 2024-05-15T23:24:43.928 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.332995943, 2024-05-15T23:24:43.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14383687000000001, 2024-05-15T23:24:43.928 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.928 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.928 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.928 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 245040 2024-05-15T23:24:43.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.928 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.423317+0000", 2024-05-15T23:24:43.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.423317+0000", 2024-05-15T23:24:43.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.423319+0000", 2024-05-15T23:24:43.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1069999999999999e-06 2024-05-15T23:24:43.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.423323+0000", 2024-05-15T23:24:43.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7509999999999999e-06 2024-05-15T23:24:43.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.423333+0000", 2024-05-15T23:24:43.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2939999999999994e-06 2024-05-15T23:24:43.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.423338+0000", 2024-05-15T23:24:43.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6640000000000003e-06 2024-05-15T23:24:43.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.423378+0000", 2024-05-15T23:24:43.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0219999999999998e-05 2024-05-15T23:24:43.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:24:43.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.423833+0000", 2024-05-15T23:24:43.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045479299999999999 2024-05-15T23:24:43.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.533809+0000", 2024-05-15T23:24:43.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.109975455 2024-05-15T23:24:43.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.533879+0000", 2024-05-15T23:24:43.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9919000000000006e-05 2024-05-15T23:24:43.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.533975+0000", 2024-05-15T23:24:43.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5916999999999997e-05 2024-05-15T23:24:43.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.543319+0000", 2024-05-15T23:24:43.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093444340000000004 2024-05-15T23:24:43.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.566595+0000", 2024-05-15T23:24:43.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023275985999999999 2024-05-15T23:24:43.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.566624+0000", 2024-05-15T23:24:43.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9183000000000002e-05 2024-05-15T23:24:43.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.567154+0000", 2024-05-15T23:24:43.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053014700000000002 2024-05-15T23:24:43.934 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.934 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.934 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.934 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:245042 5.2 5:45005af1:::benchmark_data_smithi090_116470_object91482:head [delete] snapc 0=[] ondisk+write+known_if_redirected e450)", 2024-05-15T23:24:43.934 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:23:32.426424+0000", 2024-05-15T23:24:43.934 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.329889722000004, 2024-05-15T23:24:43.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14687866999999999, 2024-05-15T23:24:43.934 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.935 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.935 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.935 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.935 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.935 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.935 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 245042 2024-05-15T23:24:43.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.935 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.426424+0000", 2024-05-15T23:24:43.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.426424+0000", 2024-05-15T23:24:43.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.426425+0000", 2024-05-15T23:24:43.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.451e-06 2024-05-15T23:24:43.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.426429+0000", 2024-05-15T23:24:43.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0040000000000001e-06 2024-05-15T23:24:43.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.426438+0000", 2024-05-15T23:24:43.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2024-05-15T23:24:43.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.426444+0000", 2024-05-15T23:24:43.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7599999999999999e-06 2024-05-15T23:24:43.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.426484+0000", 2024-05-15T23:24:43.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9852999999999999e-05 2024-05-15T23:24:43.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:24:43.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.426962+0000", 2024-05-15T23:24:43.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000478371 2024-05-15T23:24:43.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.545621+0000", 2024-05-15T23:24:43.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11865893 2024-05-15T23:24:43.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.545685+0000", 2024-05-15T23:24:43.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4298e-05 2024-05-15T23:24:43.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.545774+0000", 2024-05-15T23:24:43.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8869000000000006e-05 2024-05-15T23:24:43.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.551135+0000", 2024-05-15T23:24:43.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053608390000000001 2024-05-15T23:24:43.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.572364+0000", 2024-05-15T23:24:43.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021228648999999999 2024-05-15T23:24:43.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.572400+0000", 2024-05-15T23:24:43.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5988000000000002e-05 2024-05-15T23:24:43.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:32.573302+0000", 2024-05-15T23:24:43.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090261900000000001 2024-05-15T23:24:43.940 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.941 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.941 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.941 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:265898 5.2 5:4d5aa5f3:::benchmark_data_smithi090_116470_object112338:head [delete] snapc 0=[] ondisk+write+known_if_redirected e459)", 2024-05-15T23:24:43.941 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:23:55.954018+0000", 2024-05-15T23:24:43.941 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.802295233999999, 2024-05-15T23:24:43.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14134997799999999, 2024-05-15T23:24:43.941 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.941 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.941 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.941 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.941 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.941 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.942 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 265898 2024-05-15T23:24:43.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.942 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:55.954018+0000", 2024-05-15T23:24:43.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:55.954018+0000", 2024-05-15T23:24:43.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:55.954019+0000", 2024-05-15T23:24:43.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6400000000000002e-07 2024-05-15T23:24:43.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:55.954022+0000", 2024-05-15T23:24:43.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0089999999999999e-06 2024-05-15T23:24:43.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:55.954027+0000", 2024-05-15T23:24:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1719999999999998e-06 2024-05-15T23:24:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:55.954032+0000", 2024-05-15T23:24:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2639999999999999e-06 2024-05-15T23:24:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:55.958573+0000", 2024-05-15T23:24:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045405810000000001 2024-05-15T23:24:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:55.958657+0000", 2024-05-15T23:24:43.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3747999999999998e-05 2024-05-15T23:24:43.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:24:43.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:55.958730+0000", 2024-05-15T23:24:43.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3343999999999994e-05 2024-05-15T23:24:43.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:55.964244+0000", 2024-05-15T23:24:43.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055137179999999999 2024-05-15T23:24:43.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:56.094329+0000", 2024-05-15T23:24:43.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13008523399999999 2024-05-15T23:24:43.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:56.094391+0000", 2024-05-15T23:24:43.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2483000000000005e-05 2024-05-15T23:24:43.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:56.095368+0000", 2024-05-15T23:24:43.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00097676100000000004 2024-05-15T23:24:43.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.946 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.947 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:298431 5.0 5:0dfdbfcd:::benchmark_data_smithi090_116470_object144871:head [delete] snapc 0=[] ondisk+write+known_if_redirected e467)", 2024-05-15T23:24:43.947 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:28.006395+0000", 2024-05-15T23:24:43.947 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.749918073, 2024-05-15T23:24:43.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14882483199999999, 2024-05-15T23:24:43.947 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:43.947 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:43.947 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:43.947 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:43.947 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:43.947 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:43.947 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 298431 2024-05-15T23:24:43.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.947 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:43.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:43.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.006395+0000", 2024-05-15T23:24:43.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:43.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.006395+0000", 2024-05-15T23:24:43.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:43.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:43.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.006397+0000", 2024-05-15T23:24:43.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8050000000000001e-06 2024-05-15T23:24:43.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:43.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.006402+0000", 2024-05-15T23:24:43.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0810000000000001e-06 2024-05-15T23:24:43.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:43.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.006412+0000", 2024-05-15T23:24:43.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5659999999999996e-06 2024-05-15T23:24:43.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:43.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.006419+0000", 2024-05-15T23:24:43.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8789999999999997e-06 2024-05-15T23:24:43.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:43.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.007217+0000", 2024-05-15T23:24:43.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079883299999999999 2024-05-15T23:24:43.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:43.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.007323+0000", 2024-05-15T23:24:43.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000105414 2024-05-15T23:24:43.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:24:43.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.007429+0000", 2024-05-15T23:24:43.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010586000000000001 2024-05-15T23:24:43.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:43.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.016983+0000", 2024-05-15T23:24:43.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095540959999999998 2024-05-15T23:24:43.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:43.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155115+0000", 2024-05-15T23:24:43.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13813219500000001 2024-05-15T23:24:43.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:43.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155172+0000", 2024-05-15T23:24:43.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6678000000000002e-05 2024-05-15T23:24:43.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:43.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:43.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:43.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155220+0000", 2024-05-15T23:24:43.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8424999999999998e-05 2024-05-15T23:24:43.952 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.952 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.952 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.952 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:43.952 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:43.952 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:24:43.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:24:43.981 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:44.001 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:24:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:44.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:24:44.105 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:44.134 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:44.257 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:44.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:44.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:24:44.388 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:44.460 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:24:44.462 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:44 smithi090 ceph-mon[44434]: 5.7 deep-scrub starts 2024-05-15T23:24:44.462 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:44 smithi090 ceph-mon[44434]: 5.7 deep-scrub ok 2024-05-15T23:24:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:44.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:24:44.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T23:24:44.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:44 smithi084 ceph-mon[39784]: 5.7 deep-scrub starts 2024-05-15T23:24:44.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:44 smithi084 ceph-mon[39784]: 5.7 deep-scrub ok 2024-05-15T23:24:44.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:44.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:44 smithi088 ceph-mon[44699]: 5.7 deep-scrub starts 2024-05-15T23:24:44.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:44 smithi088 ceph-mon[44699]: 5.7 deep-scrub ok 2024-05-15T23:24:44.595 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:44.614 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:44.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:44.718 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:24:44.752 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:44.773 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:24:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:44.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:24:44.789 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:24:44.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:24:44.904 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:44.905 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:24:45.181 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:24:45.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:45 smithi090 ceph-mon[44434]: 5.0 scrub starts 2024-05-15T23:24:45.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:45 smithi090 ceph-mon[44434]: 5.0 scrub ok 2024-05-15T23:24:45.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/313964252' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:24:45.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:45 smithi090 ceph-mon[44434]: 5.6 deep-scrub starts 2024-05-15T23:24:45.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:45 smithi090 ceph-mon[44434]: 5.6 deep-scrub ok 2024-05-15T23:24:45.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:45 smithi090 ceph-mon[44434]: pgmap v1274: 17 pgs: 17 active+clean; 577 KiB data, 3.4 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 507 op/s 2024-05-15T23:24:45.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:45 smithi084 ceph-mon[39784]: 5.0 scrub starts 2024-05-15T23:24:45.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:45 smithi084 ceph-mon[39784]: 5.0 scrub ok 2024-05-15T23:24:45.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/313964252' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:24:45.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:45 smithi084 ceph-mon[39784]: 5.6 deep-scrub starts 2024-05-15T23:24:45.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:45 smithi084 ceph-mon[39784]: 5.6 deep-scrub ok 2024-05-15T23:24:45.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:45 smithi084 ceph-mon[39784]: pgmap v1274: 17 pgs: 17 active+clean; 577 KiB data, 3.4 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 507 op/s 2024-05-15T23:24:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:45 smithi088 ceph-mon[44699]: 5.0 scrub starts 2024-05-15T23:24:45.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:45 smithi088 ceph-mon[44699]: 5.0 scrub ok 2024-05-15T23:24:45.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/313964252' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:24:45.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:45 smithi088 ceph-mon[44699]: 5.6 deep-scrub starts 2024-05-15T23:24:45.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:45 smithi088 ceph-mon[44699]: 5.6 deep-scrub ok 2024-05-15T23:24:45.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:45 smithi088 ceph-mon[44699]: pgmap v1274: 17 pgs: 17 active+clean; 577 KiB data, 3.4 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 507 op/s 2024-05-15T23:24:46.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:46 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:24:46.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:46 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/313964252' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:24:46.286 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:46 smithi090 ceph-mon[44434]: osdmap e472: 12 total, 12 up, 9 in 2024-05-15T23:24:46.557 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:24:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:24:46.735 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-05-15T23:24:46.735 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:24:46.737 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:46 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:24:46.737 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:46 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/313964252' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:24:46.737 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:46 smithi084 ceph-mon[39784]: osdmap e472: 12 total, 12 up, 9 in 2024-05-15T23:24:46.737 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:46 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:24:46.737 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:46 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/313964252' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:24:46.737 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:46 smithi088 ceph-mon[44699]: osdmap e472: 12 total, 12 up, 9 in 2024-05-15T23:24:46.755 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:24:46.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:46.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:24:46.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:46.890 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:47.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T23:24:47.255 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:47 smithi084 ceph-mon[39784]: pgmap v1276: 17 pgs: 17 active+clean; 577 KiB data, 3.4 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 169 op/s 2024-05-15T23:24:47.256 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:47 smithi084 ceph-mon[39784]: osdmap e473: 12 total, 12 up, 9 in 2024-05-15T23:24:47.318 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:47.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:47.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:47.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:47.451 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:47.496 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:24:47.549 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:47.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:47.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:47 smithi090 ceph-mon[44434]: pgmap v1276: 17 pgs: 17 active+clean; 577 KiB data, 3.4 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 169 op/s 2024-05-15T23:24:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:47 smithi090 ceph-mon[44434]: osdmap e473: 12 total, 12 up, 9 in 2024-05-15T23:24:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:47 smithi088 ceph-mon[44699]: pgmap v1276: 17 pgs: 17 active+clean; 577 KiB data, 3.4 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 169 op/s 2024-05-15T23:24:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:47 smithi088 ceph-mon[44699]: osdmap e473: 12 total, 12 up, 9 in 2024-05-15T23:24:47.638 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:24:47.640 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:24:47.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:47.680 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:47.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:47.856 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:24:47.890 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:24:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:47.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:24:48.016 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:24:48.045 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:48.066 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:24:48.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:48.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:24:48.170 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:48.189 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_3' removed 2024-05-15T23:24:48.199 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:48.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:48 smithi090 ceph-mon[44434]: from='client.16380 v1:172.21.15.84:0/4118055762' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:24:48.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3891671306' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:24:48.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:48 smithi084 ceph-mon[39784]: from='client.16380 v1:172.21.15.84:0/4118055762' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:24:48.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3891671306' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:24:48.454 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1277,"stamp":"2024-05-15T23:24:47.068385+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":307239,"num_write_kb":9829799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25895,"num_bytes_recovered":1698248288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160910,"ondisk_log_size":160910,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":3609596,"kb_used_data":10964,"kb_used_omap":14,"kb_used_meta":3598513,"kb_avail":840096772,"statfs":{"total":863955320832,"available":860259094528,"internally_reserved":0,"allocated":11227136,"data_stored":6172062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14358,"internal_metadata":3684878314},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17235968,"num_objects":-263,"num_object_clones":0,"num_object_copies":-526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.050369"},"pg_stats":[{"pgid":"5.f","version":"470'19060","reported_seq":19273,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-05-15T23:24:38.453344+0000","last_change":"2024-05-15T23:24:38.453375+0000","last_active":"2024-05-15T23:24:38.453344+0000","last_peered":"2024-05-15T23:24:38.453344+0000","last_clean":"2024-05-15T23:24:38.453344+0000","last_became_active":"2024-05-15T23:19:47.305689+0000","last_became_peered":"2024-05-15T23:19:47.305689+0000","last_unstale":"2024-05-15T23:24:38.453344+0000","last_undegraded":"2024-05-15T23:24:38.453344+0000","last_fullsized":"2024-05-15T23:24:38.453344+0000","mapping_epoch":374,"log_start":"410'9000","ondisk_log_start":"410'9000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'19060","last_scrub_stamp":"2024-05-15T23:24:38.453247+0000","last_deep_scrub":"470'19060","last_deep_scrub_stamp":"2024-05-15T23:24:38.453247+0000","last_clean_scrub_stamp":"2024-05-15T23:24:38.453247+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:52.397366+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19060,"num_write_kb":609920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"470'18922","reported_seq":19338,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T23:24:45.292234+0000","last_change":"2024-05-15T23:23:53.872535+0000","last_active":"2024-05-15T23:24:45.292234+0000","last_peered":"2024-05-15T23:24:45.292234+0000","last_clean":"2024-05-15T23:24:45.292234+0000","last_became_active":"2024-05-15T23:19:47.325414+0000","last_became_peered":"2024-05-15T23:19:47.325414+0000","last_unstale":"2024-05-15T23:24:45.292234+0000","last_undegraded":"2024-05-15T23:24:45.292234+0000","last_fullsized":"2024-05-15T23:24:45.292234+0000","mapping_epoch":374,"log_start":"410'8900","ondisk_log_start":"410'8900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'16258","last_scrub_stamp":"2024-05-15T23:23:53.872309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:53.872309+0000","objects_scrubbed":2678,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:15.839794+0000","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18922,"num_write_kb":605504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"470'19446","reported_seq":19664,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-05-15T23:24:37.068663+0000","last_change":"2024-05-15T23:24:31.086260+0000","last_active":"2024-05-15T23:24:37.068663+0000","last_peered":"2024-05-15T23:24:37.068663+0000","last_clean":"2024-05-15T23:24:37.068663+0000","last_became_active":"2024-05-15T23:19:47.305860+0000","last_became_peered":"2024-05-15T23:19:47.305860+0000","last_unstale":"2024-05-15T23:24:37.068663+0000","last_undegraded":"2024-05-15T23:24:37.068663+0000","last_fullsized":"2024-05-15T23:24:37.068663+0000","mapping_epoch":374,"log_start":"411'9400","ondisk_log_start":"411'9400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'19105","last_scrub_stamp":"2024-05-15T23:24:31.086098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:24:31.086098+0000","objects_scrubbed":342,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:59.030006+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":19446,"num_write_kb":622209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"470'19318","reported_seq":19696,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T23:24:45.291689+0000","last_change":"2024-05-15T23:24:36.086224+0000","last_active":"2024-05-15T23:24:45.291689+0000","last_peered":"2024-05-15T23:24:45.291689+0000","last_clean":"2024-05-15T23:24:45.291689+0000","last_became_active":"2024-05-15T23:19:47.306168+0000","last_became_peered":"2024-05-15T23:19:47.306168+0000","last_unstale":"2024-05-15T23:24:45.291689+0000","last_undegraded":"2024-05-15T23:24:45.291689+0000","last_fullsized":"2024-05-15T23:24:45.291689+0000","mapping_epoch":374,"log_start":"411'9300","ondisk_log_start":"411'9300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'19318","last_scrub_stamp":"2024-05-15T23:24:36.086144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:24:36.086144+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:59.237549+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19318,"num_write_kb":618176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"470'19192","reported_seq":19576,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T23:24:45.291981+0000","last_change":"2024-05-15T23:24:31.742721+0000","last_active":"2024-05-15T23:24:45.291981+0000","last_peered":"2024-05-15T23:24:45.291981+0000","last_clean":"2024-05-15T23:24:45.291981+0000","last_became_active":"2024-05-15T23:19:47.307061+0000","last_became_peered":"2024-05-15T23:19:47.307061+0000","last_unstale":"2024-05-15T23:24:45.291981+0000","last_undegraded":"2024-05-15T23:24:45.291981+0000","last_fullsized":"2024-05-15T23:24:45.291981+0000","mapping_epoch":374,"log_start":"410'9100","ondisk_log_start":"410'9100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'18883","last_scrub_stamp":"2024-05-15T23:24:31.742550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:24:31.742550+0000","objects_scrubbed":314,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:42.153206+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19192,"num_write_kb":614144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"470'19212","reported_seq":19427,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T23:24:45.292182+0000","last_change":"2024-05-15T23:24:40.995860+0000","last_active":"2024-05-15T23:24:45.292182+0000","last_peered":"2024-05-15T23:24:45.292182+0000","last_clean":"2024-05-15T23:24:45.292182+0000","last_became_active":"2024-05-15T23:19:47.325371+0000","last_became_peered":"2024-05-15T23:19:47.325371+0000","last_unstale":"2024-05-15T23:24:45.292182+0000","last_undegraded":"2024-05-15T23:24:45.292182+0000","last_fullsized":"2024-05-15T23:24:45.292182+0000","mapping_epoch":374,"log_start":"410'9200","ondisk_log_start":"410'9200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'19212","last_scrub_stamp":"2024-05-15T23:24:40.995759+0000","last_deep_scrub":"470'19212","last_deep_scrub_stamp":"2024-05-15T23:24:40.995759+0000","last_clean_scrub_stamp":"2024-05-15T23:24:40.995759+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:43.589907+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19212,"num_write_kb":614784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"470'19154","reported_seq":36517,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T23:24:45.178807+0000","last_change":"2024-05-15T23:24:31.917656+0000","last_active":"2024-05-15T23:24:45.178807+0000","last_peered":"2024-05-15T23:24:45.178807+0000","last_clean":"2024-05-15T23:24:45.178807+0000","last_became_active":"2024-05-15T23:21:09.454639+0000","last_became_peered":"2024-05-15T23:21:09.454639+0000","last_unstale":"2024-05-15T23:24:45.178807+0000","last_undegraded":"2024-05-15T23:24:45.178807+0000","last_fullsized":"2024-05-15T23:24:45.178807+0000","mapping_epoch":407,"log_start":"410'9100","ondisk_log_start":"410'9100","created":374,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'18881","last_scrub_stamp":"2024-05-15T23:24:31.917402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:24:31.917402+0000","objects_scrubbed":273,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:59.276172+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19154,"num_write_kb":612928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8570,"num_bytes_recovered":561643520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"470'19204","reported_seq":19419,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-05-15T23:24:37.068582+0000","last_change":"2024-05-15T23:24:32.036285+0000","last_active":"2024-05-15T23:24:37.068582+0000","last_peered":"2024-05-15T23:24:37.068582+0000","last_clean":"2024-05-15T23:24:37.068582+0000","last_became_active":"2024-05-15T23:19:47.304547+0000","last_became_peered":"2024-05-15T23:19:47.304547+0000","last_unstale":"2024-05-15T23:24:37.068582+0000","last_undegraded":"2024-05-15T23:24:37.068582+0000","last_fullsized":"2024-05-15T23:24:37.068582+0000","mapping_epoch":374,"log_start":"410'9200","ondisk_log_start":"410'9200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'18917","last_scrub_stamp":"2024-05-15T23:24:32.035988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:24:32.035988+0000","objects_scrubbed":288,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:51.721919+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19204,"num_write_kb":614528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"470'19056","reported_seq":19261,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-05-15T23:24:37.565411+0000","last_change":"2024-05-15T23:19:47.306634+0000","last_active":"2024-05-15T23:24:37.565411+0000","last_peered":"2024-05-15T23:24:37.565411+0000","last_clean":"2024-05-15T23:24:37.565411+0000","last_became_active":"2024-05-15T23:19:47.306331+0000","last_became_peered":"2024-05-15T23:19:47.306331+0000","last_unstale":"2024-05-15T23:24:37.565411+0000","last_undegraded":"2024-05-15T23:24:37.565411+0000","last_fullsized":"2024-05-15T23:24:37.565411+0000","mapping_epoch":374,"log_start":"410'9000","ondisk_log_start":"410'9000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19056,"num_write_kb":609792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"470'19310","reported_seq":19515,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-05-15T23:24:37.565367+0000","last_change":"2024-05-15T23:19:47.306611+0000","last_active":"2024-05-15T23:24:37.565367+0000","last_peered":"2024-05-15T23:24:37.565367+0000","last_clean":"2024-05-15T23:24:37.565367+0000","last_became_active":"2024-05-15T23:19:47.306349+0000","last_became_peered":"2024-05-15T23:19:47.306349+0000","last_unstale":"2024-05-15T23:24:37.565367+0000","last_undegraded":"2024-05-15T23:24:37.565367+0000","last_fullsized":"2024-05-15T23:24:37.565367+0000","mapping_epoch":374,"log_start":"410'9300","ondisk_log_start":"410'9300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19310,"num_write_kb":617920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"470'19396","reported_seq":19870,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T23:24:45.291702+0000","last_change":"2024-05-15T23:23:50.810750+0000","last_active":"2024-05-15T23:24:45.291702+0000","last_peered":"2024-05-15T23:24:45.291702+0000","last_clean":"2024-05-15T23:24:45.291702+0000","last_became_active":"2024-05-15T23:19:47.305685+0000","last_became_peered":"2024-05-15T23:19:47.305685+0000","last_unstale":"2024-05-15T23:24:45.291702+0000","last_undegraded":"2024-05-15T23:24:45.291702+0000","last_fullsized":"2024-05-15T23:24:45.291702+0000","mapping_epoch":374,"log_start":"410'9300","ondisk_log_start":"410'9300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'16478","last_scrub_stamp":"2024-05-15T23:23:50.810636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:50.810636+0000","objects_scrubbed":2939,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:20.026294+0000","scrub_duration":608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19396,"num_write_kb":620672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"470'19296","reported_seq":36886,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-05-15T23:24:37.565361+0000","last_change":"2024-05-15T23:24:17.572412+0000","last_active":"2024-05-15T23:24:37.565361+0000","last_peered":"2024-05-15T23:24:37.565361+0000","last_clean":"2024-05-15T23:24:37.565361+0000","last_became_active":"2024-05-15T23:21:09.400731+0000","last_became_peered":"2024-05-15T23:21:09.400731+0000","last_unstale":"2024-05-15T23:24:37.565361+0000","last_undegraded":"2024-05-15T23:24:37.565361+0000","last_fullsized":"2024-05-15T23:24:37.565361+0000","mapping_epoch":407,"log_start":"410'9200","ondisk_log_start":"410'9200","created":374,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:19:46.285287+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19296,"num_write_kb":617472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8690,"num_bytes_recovered":569507840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"470'19170","reported_seq":36649,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T23:24:45.292065+0000","last_change":"2024-05-15T23:24:42.021390+0000","last_active":"2024-05-15T23:24:45.292065+0000","last_peered":"2024-05-15T23:24:45.292065+0000","last_clean":"2024-05-15T23:24:45.292065+0000","last_became_active":"2024-05-15T23:21:09.221797+0000","last_became_peered":"2024-05-15T23:21:09.221797+0000","last_unstale":"2024-05-15T23:24:45.292065+0000","last_undegraded":"2024-05-15T23:24:45.292065+0000","last_fullsized":"2024-05-15T23:24:45.292065+0000","mapping_epoch":407,"log_start":"410'9100","ondisk_log_start":"410'9100","created":374,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'19170","last_scrub_stamp":"2024-05-15T23:24:42.021232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:24:42.021232+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:45.631337+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19170,"num_write_kb":613440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8629,"num_bytes_recovered":565510144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.1","version":"470'18658","reported_seq":19072,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-05-15T23:24:38.441867+0000","last_change":"2024-05-15T23:23:50.798735+0000","last_active":"2024-05-15T23:24:38.441867+0000","last_peered":"2024-05-15T23:24:38.441867+0000","last_clean":"2024-05-15T23:24:38.441867+0000","last_became_active":"2024-05-15T23:19:47.516865+0000","last_became_peered":"2024-05-15T23:19:47.516865+0000","last_unstale":"2024-05-15T23:24:38.441867+0000","last_undegraded":"2024-05-15T23:24:38.441867+0000","last_fullsized":"2024-05-15T23:24:38.441867+0000","mapping_epoch":374,"log_start":"410'8600","ondisk_log_start":"410'8600","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'15864","last_scrub_stamp":"2024-05-15T23:23:50.798612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:23:50.798612+0000","objects_scrubbed":2807,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:24:56.167477+0000","scrub_duration":584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18658,"num_write_kb":597056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"470'19276","reported_seq":19491,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T23:24:45.292020+0000","last_change":"2024-05-15T23:24:44.040581+0000","last_active":"2024-05-15T23:24:45.292020+0000","last_peered":"2024-05-15T23:24:45.292020+0000","last_clean":"2024-05-15T23:24:45.292020+0000","last_became_active":"2024-05-15T23:19:47.305459+0000","last_became_peered":"2024-05-15T23:19:47.305459+0000","last_unstale":"2024-05-15T23:24:45.292020+0000","last_undegraded":"2024-05-15T23:24:45.292020+0000","last_fullsized":"2024-05-15T23:24:45.292020+0000","mapping_epoch":374,"log_start":"410'9200","ondisk_log_start":"410'9200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'19276","last_scrub_stamp":"2024-05-15T23:24:44.040479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:24:44.040479+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:58.645456+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19276,"num_write_kb":616832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"470'19446","reported_seq":19661,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T23:24:45.291980+0000","last_change":"2024-05-15T23:24:39.006023+0000","last_active":"2024-05-15T23:24:45.291980+0000","last_peered":"2024-05-15T23:24:45.291980+0000","last_clean":"2024-05-15T23:24:45.291980+0000","last_became_active":"2024-05-15T23:19:47.307149+0000","last_became_peered":"2024-05-15T23:19:47.307149+0000","last_unstale":"2024-05-15T23:24:45.291980+0000","last_undegraded":"2024-05-15T23:24:45.291980+0000","last_fullsized":"2024-05-15T23:24:45.291980+0000","mapping_epoch":374,"log_start":"411'9400","ondisk_log_start":"411'9400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'19446","last_scrub_stamp":"2024-05-15T23:24:39.005930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:19:46.285287+0000","last_clean_scrub_stamp":"2024-05-15T23:24:39.005930+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:52.555777+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19446,"num_write_kb":622272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":992,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-05-15T23:24:45.178804+0000","last_change":"2024-05-15T23:24:32.827864+0000","last_active":"2024-05-15T23:24:45.178804+0000","last_peered":"2024-05-15T23:24:45.178804+0000","last_clean":"2024-05-15T23:24:45.178804+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:24:45.178804+0000","last_undegraded":"2024-05-15T23:24:45.178804+0000","last_fullsized":"2024-05-15T23:24:45.178804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:24:32.827775+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:24:32.827775+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:25:42.135474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":307116,"num_write_kb":9827649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25889,"num_bytes_recovered":1696661504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1551368192,"data_stored":1551368192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160816,"ondisk_log_size":160816,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351227,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582564,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93162588,"statfs":{"total":95995035648,"available":95398490112,"internally_reserved":0,"allocated":1658880,"data_stored":1095770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":594868676},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580161,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":284948,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93460204,"statfs":{"total":95995035648,"available":95703248896,"internally_reserved":0,"allocated":1126400,"data_stored":566862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":290650560},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776393,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":401560,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93343592,"statfs":{"total":95995035648,"available":95583838208,"internally_reserved":0,"allocated":1126400,"data_stored":566862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":410057152},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":203032,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93542120,"statfs":{"total":95995035648,"available":95787130880,"internally_reserved":0,"allocated":1126400,"data_stored":566862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266961,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":375724,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93369428,"statfs":{"total":95995035648,"available":95610294272,"internally_reserved":0,"allocated":1671168,"data_stored":1102024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":383056330},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495895,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575153,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":501524,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93243628,"statfs":{"total":95995035648,"available":95481475072,"internally_reserved":0,"allocated":1126400,"data_stored":566862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":512424390},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016563,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483251,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149992,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":451556,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93293596,"statfs":{"total":95995035648,"available":95532642304,"internally_reserved":0,"allocated":1138688,"data_stored":573096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":461240775},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986464,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415448,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93329704,"statfs":{"total":95995035648,"available":95569616896,"internally_reserved":0,"allocated":1126400,"data_stored":566862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790236,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":393240,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93351912,"statfs":{"total":95995035648,"available":95592357888,"internally_reserved":0,"allocated":1126400,"data_stored":566862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1551368192,"data_stored":1551368192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:24:48.455 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-15T23:24:48.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:48 smithi088 ceph-mon[44699]: from='client.16380 v1:172.21.15.84:0/4118055762' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:24:48.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3891671306' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:24:48.647 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:48.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:48.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:48.752 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:48.778 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:48.984 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:48.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:49.089 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:24:49.115 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:49.134 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:24:49.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:49.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:24:49.243 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:24:49.256 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:24:49.256 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:24:49.256 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:24:49.256 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:24:49.256 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:24:49.273 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:49.277 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:49 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:24:49.278 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:49 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:24:49.278 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:49 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3891671306' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:24:49.278 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:49 smithi090 ceph-mon[44434]: osdmap e474: 12 total, 12 up, 9 in 2024-05-15T23:24:49.278 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:49 smithi090 ceph-mon[44434]: pgmap v1279: 1 pgs: 1 active+clean; 577 KiB data, 3.4 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:24:49.294 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:24:49.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:49.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:24:49.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:49 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:24:49.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:49 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:24:49.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:49 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3891671306' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:24:49.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:49 smithi084 ceph-mon[39784]: osdmap e474: 12 total, 12 up, 9 in 2024-05-15T23:24:49.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:49 smithi084 ceph-mon[39784]: pgmap v1279: 1 pgs: 1 active+clean; 577 KiB data, 3.4 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:24:49.398 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:24:49.426 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:49.429 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:49.447 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:24:49.449 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:49 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:24:49.449 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:49 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:24:49.449 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:49 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3891671306' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:24:49.449 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:49 smithi088 ceph-mon[44699]: osdmap e474: 12 total, 12 up, 9 in 2024-05-15T23:24:49.449 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:49 smithi088 ceph-mon[44699]: pgmap v1279: 1 pgs: 1 active+clean; 577 KiB data, 3.4 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:24:49.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:49.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:24:49.464 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T23:24:49.464 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T23:24:49.465 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-05-15T23:24:49.465 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-05-15T23:24:49.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:49.581 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:49.930 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:49.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:49.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:49.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T23:24:50.033 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:50.060 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:50.237 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:24:50.702 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:50.708 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:50.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:50.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:50.813 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:24:50.840 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:50.965 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:24:50.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:50.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:24:51.016 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:24:51.069 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:24:51.097 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:51.237 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:51 smithi084 ceph-mon[39784]: pgmap v1280: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:24:51.247 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:24:51.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:51.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:24:51.351 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:24:51.382 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:51 smithi088 ceph-mon[44699]: pgmap v1280: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:24:51.385 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:51.407 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:24:51.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:51.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:24:51.477 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:51 smithi090 ceph-mon[44434]: pgmap v1280: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:24:51.511 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:24:51.540 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:51.939 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:24:51.958 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:24:51.977 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:24:51.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:24:52.016 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:24:52.020 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:24:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:24:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:24:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:24:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:90775 5.d 5:b3f85306:::benchmark_data_smithi090_116470_object90774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e392)", 2024-05-15T23:24:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:34.279052+0000", 2024-05-15T23:24:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 257.737260655, 2024-05-15T23:24:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13313823499999999, 2024-05-15T23:24:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90775 2024-05-15T23:24:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279052+0000", 2024-05-15T23:24:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279052+0000", 2024-05-15T23:24:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279053+0000", 2024-05-15T23:24:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4499999999999996e-07 2024-05-15T23:24:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279073+0000", 2024-05-15T23:24:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0598000000000002e-05 2024-05-15T23:24:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279080+0000", 2024-05-15T23:24:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0679999999999999e-06 2024-05-15T23:24:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.279087+0000", 2024-05-15T23:24:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7309999999999996e-06 2024-05-15T23:24:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.282248+0000", 2024-05-15T23:24:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031613000000000001 2024-05-15T23:24:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.282376+0000", 2024-05-15T23:24:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799 2024-05-15T23:24:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:24:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.282511+0000", 2024-05-15T23:24:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134544 2024-05-15T23:24:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412011+0000", 2024-05-15T23:24:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12950052200000001 2024-05-15T23:24:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412093+0000", 2024-05-15T23:24:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1329999999999996e-05 2024-05-15T23:24:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412141+0000", 2024-05-15T23:24:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8327e-05 2024-05-15T23:24:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412190+0000", 2024-05-15T23:24:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8980000000000002e-05 2024-05-15T23:24:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:90784 5.d 5:bc54d435:::benchmark_data_smithi090_116470_object90783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e392)", 2024-05-15T23:24:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:34.288478+0000", 2024-05-15T23:24:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 257.72783449899998, 2024-05-15T23:24:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12653699900000001, 2024-05-15T23:24:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 90784 2024-05-15T23:24:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288478+0000", 2024-05-15T23:24:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288478+0000", 2024-05-15T23:24:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288481+0000", 2024-05-15T23:24:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5459999999999998e-06 2024-05-15T23:24:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288504+0000", 2024-05-15T23:24:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3923000000000001e-05 2024-05-15T23:24:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288516+0000", 2024-05-15T23:24:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1364e-05 2024-05-15T23:24:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288524+0000", 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0090000000000007e-06 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288581+0000", 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.749e-05 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288709+0000", 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012748100000000001 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:24:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288841+0000", 2024-05-15T23:24:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131958 2024-05-15T23:24:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412079+0000", 2024-05-15T23:24:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.123238218 2024-05-15T23:24:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.414956+0000", 2024-05-15T23:24:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028775430000000002 2024-05-15T23:24:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.414987+0000", 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0614999999999997e-05 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.415015+0000", 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7852e-05 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.25840.0:90775 5.d e392/374)", 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:34.288603+0000", 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 257.72770924000002, 2024-05-15T23:24:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12341743199999999, 2024-05-15T23:24:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:24:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288603+0000", 2024-05-15T23:24:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288603+0000", 2024-05-15T23:24:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288604+0000", 2024-05-15T23:24:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2240000000000001e-06 2024-05-15T23:24:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288609+0000", 2024-05-15T23:24:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5920000000000002e-06 2024-05-15T23:24:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288617+0000", 2024-05-15T23:24:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6839999999999996e-06 2024-05-15T23:24:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.288623+0000", 2024-05-15T23:24:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3110000000000002e-06 2024-05-15T23:24:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.411996+0000", 2024-05-15T23:24:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12337332199999999 2024-05-15T23:24:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412005+0000", 2024-05-15T23:24:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3389999999999993e-06 2024-05-15T23:24:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412021+0000", 2024-05-15T23:24:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.596e-05 2024-05-15T23:24:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.25840.0:90784 5.d e392/374 ondisk, result = 0)", 2024-05-15T23:24:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:34.291602+0000", 2024-05-15T23:24:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 257.72471019599999, 2024-05-15T23:24:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12060193600000001, 2024-05-15T23:24:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.036 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.036 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:24:52.036 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.291602+0000", 2024-05-15T23:24:52.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.291602+0000", 2024-05-15T23:24:52.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.291604+0000", 2024-05-15T23:24:52.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.731e-06 2024-05-15T23:24:52.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.291612+0000", 2024-05-15T23:24:52.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-05-15T23:24:52.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.408963+0000", 2024-05-15T23:24:52.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11735071 2024-05-15T23:24:52.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.408978+0000", 2024-05-15T23:24:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4948e-05 2024-05-15T23:24:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412063+0000", 2024-05-15T23:24:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003084669 2024-05-15T23:24:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412070+0000", 2024-05-15T23:24:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5449999999999998e-06 2024-05-15T23:24:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:34.412204+0000", 2024-05-15T23:24:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013396799999999999 2024-05-15T23:24:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:109154 5.8 5:121e6b3a:::benchmark_data_smithi090_116470_object109153:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-05-15T23:24:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:46.486383+0000", 2024-05-15T23:24:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 245.529929345, 2024-05-15T23:24:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.127318981, 2024-05-15T23:24:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109154 2024-05-15T23:24:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.486383+0000", 2024-05-15T23:24:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.486383+0000", 2024-05-15T23:24:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.486385+0000", 2024-05-15T23:24:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6899999999999999e-06 2024-05-15T23:24:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.486408+0000", 2024-05-15T23:24:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2920000000000001e-05 2024-05-15T23:24:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.486416+0000", 2024-05-15T23:24:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8979999999999996e-06 2024-05-15T23:24:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.486422+0000", 2024-05-15T23:24:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9429999999999997e-06 2024-05-15T23:24:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.489130+0000", 2024-05-15T23:24:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027081520000000001 2024-05-15T23:24:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.489252+0000", 2024-05-15T23:24:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012243200000000001 2024-05-15T23:24:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:24:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.489332+0000", 2024-05-15T23:24:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0045999999999999e-05 2024-05-15T23:24:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492263+0000", 2024-05-15T23:24:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029307510000000001 2024-05-15T23:24:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.613598+0000", 2024-05-15T23:24:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.121335048 2024-05-15T23:24:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.613652+0000", 2024-05-15T23:24:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4079000000000002e-05 2024-05-15T23:24:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.613702+0000", 2024-05-15T23:24:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0022000000000003e-05 2024-05-15T23:24:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:109165 5.8 5:197248c9:::benchmark_data_smithi090_116470_object109164:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-05-15T23:24:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:46.492898+0000", 2024-05-15T23:24:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 245.523413979, 2024-05-15T23:24:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12576668099999999, 2024-05-15T23:24:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109165 2024-05-15T23:24:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492898+0000", 2024-05-15T23:24:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492898+0000", 2024-05-15T23:24:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492900+0000", 2024-05-15T23:24:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9180000000000001e-06 2024-05-15T23:24:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492922+0000", 2024-05-15T23:24:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.122e-05 2024-05-15T23:24:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492931+0000", 2024-05-15T23:24:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5910000000000002e-06 2024-05-15T23:24:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492938+0000", 2024-05-15T23:24:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9990000000000002e-06 2024-05-15T23:24:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.492979+0000", 2024-05-15T23:24:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0571999999999997e-05 2024-05-15T23:24:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.609709+0000", 2024-05-15T23:24:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11673001 2024-05-15T23:24:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:24:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.609876+0000", 2024-05-15T23:24:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016714599999999999 2024-05-15T23:24:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.617471+0000", 2024-05-15T23:24:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075946499999999997 2024-05-15T23:24:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.618564+0000", 2024-05-15T23:24:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010938440000000001 2024-05-15T23:24:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.618622+0000", 2024-05-15T23:24:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7992999999999999e-05 2024-05-15T23:24:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.618665+0000", 2024-05-15T23:24:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2738000000000003e-05 2024-05-15T23:24:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.25840.0:109154 5.8 e396/374 ondisk, result = 0)", 2024-05-15T23:24:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:46.493242+0000", 2024-05-15T23:24:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 245.52307087299999, 2024-05-15T23:24:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120466937, 2024-05-15T23:24:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:24:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.493242+0000", 2024-05-15T23:24:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.493242+0000", 2024-05-15T23:24:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.493242+0000", 2024-05-15T23:24:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9699999999999995e-07 2024-05-15T23:24:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.493246+0000", 2024-05-15T23:24:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0180000000000003e-06 2024-05-15T23:24:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.609486+0000", 2024-05-15T23:24:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11623934299999999 2024-05-15T23:24:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.609511+0000", 2024-05-15T23:24:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5476000000000001e-05 2024-05-15T23:24:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.613584+0000", 2024-05-15T23:24:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040730100000000002 2024-05-15T23:24:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.613592+0000", 2024-05-15T23:24:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1270000000000003e-06 2024-05-15T23:24:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.613709+0000", 2024-05-15T23:24:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116166 2024-05-15T23:24:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:109219 5.d 5:b88eda13:::benchmark_data_smithi090_116470_object109218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-05-15T23:24:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:46.637687+0000", 2024-05-15T23:24:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 245.378625954, 2024-05-15T23:24:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11331556499999999, 2024-05-15T23:24:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109219 2024-05-15T23:24:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637687+0000", 2024-05-15T23:24:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637687+0000", 2024-05-15T23:24:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637688+0000", 2024-05-15T23:24:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.855e-06 2024-05-15T23:24:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637712+0000", 2024-05-15T23:24:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3506000000000002e-05 2024-05-15T23:24:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637722+0000", 2024-05-15T23:24:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0464e-05 2024-05-15T23:24:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637731+0000", 2024-05-15T23:24:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3310000000000004e-06 2024-05-15T23:24:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637787+0000", 2024-05-15T23:24:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5942000000000001e-05 2024-05-15T23:24:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.637900+0000", 2024-05-15T23:24:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113751 2024-05-15T23:24:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:24:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.638026+0000", 2024-05-15T23:24:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125532 2024-05-15T23:24:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.641729+0000", 2024-05-15T23:24:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037033399999999998 2024-05-15T23:24:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.750899+0000", 2024-05-15T23:24:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.109169718 2024-05-15T23:24:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.750940+0000", 2024-05-15T23:24:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1186999999999997e-05 2024-05-15T23:24:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.751002+0000", 2024-05-15T23:24:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1939000000000004e-05 2024-05-15T23:24:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:109223 5.d 5:b95a4800:::benchmark_data_smithi090_116470_object109222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-05-15T23:24:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:46.642357+0000", 2024-05-15T23:24:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 245.37395591399999, 2024-05-15T23:24:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11458536699999999, 2024-05-15T23:24:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109223 2024-05-15T23:24:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642357+0000", 2024-05-15T23:24:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642357+0000", 2024-05-15T23:24:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642359+0000", 2024-05-15T23:24:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1950000000000002e-06 2024-05-15T23:24:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642382+0000", 2024-05-15T23:24:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3337999999999999e-05 2024-05-15T23:24:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642394+0000", 2024-05-15T23:24:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2024-05-15T23:24:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642402+0000", 2024-05-15T23:24:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9659999999999996e-06 2024-05-15T23:24:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642454+0000", 2024-05-15T23:24:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2698999999999998e-05 2024-05-15T23:24:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642540+0000", 2024-05-15T23:24:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5768999999999998e-05 2024-05-15T23:24:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:24:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.642625+0000", 2024-05-15T23:24:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4457999999999997e-05 2024-05-15T23:24:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.751046+0000", 2024-05-15T23:24:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.108421383 2024-05-15T23:24:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.756850+0000", 2024-05-15T23:24:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0058037239999999997 2024-05-15T23:24:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.756898+0000", 2024-05-15T23:24:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8031999999999997e-05 2024-05-15T23:24:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.756942+0000", 2024-05-15T23:24:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4057999999999997e-05 2024-05-15T23:24:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:109227 5.d 5:bc936f0d:::benchmark_data_smithi090_116470_object109226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-05-15T23:24:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:46.644710+0000", 2024-05-15T23:24:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 245.37160223500001, 2024-05-15T23:24:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11603258299999999, 2024-05-15T23:24:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 109227 2024-05-15T23:24:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.644710+0000", 2024-05-15T23:24:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.644710+0000", 2024-05-15T23:24:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.644711+0000", 2024-05-15T23:24:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.054e-06 2024-05-15T23:24:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.644772+0000", 2024-05-15T23:24:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129000000000004e-05 2024-05-15T23:24:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.644781+0000", 2024-05-15T23:24:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5259999999999993e-06 2024-05-15T23:24:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.644787+0000", 2024-05-15T23:24:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5080000000000002e-06 2024-05-15T23:24:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.747813+0000", 2024-05-15T23:24:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10302538999999999 2024-05-15T23:24:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.747905+0000", 2024-05-15T23:24:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1855999999999996e-05 2024-05-15T23:24:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:24:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.747993+0000", 2024-05-15T23:24:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8801000000000006e-05 2024-05-15T23:24:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.751088+0000", 2024-05-15T23:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030946799999999998 2024-05-15T23:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.760647+0000", 2024-05-15T23:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095584699999999995 2024-05-15T23:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.760698+0000", 2024-05-15T23:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0893999999999998e-05 2024-05-15T23:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:46.760743+0000", 2024-05-15T23:24:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5275000000000003e-05 2024-05-15T23:24:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:112241 5.8 5:180485a9:::benchmark_data_smithi090_116470_object112240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e398)", 2024-05-15T23:24:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:48.753350+0000", 2024-05-15T23:24:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 243.26296229799999, 2024-05-15T23:24:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.112608941, 2024-05-15T23:24:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 112241 2024-05-15T23:24:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.753350+0000", 2024-05-15T23:24:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.753350+0000", 2024-05-15T23:24:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.753351+0000", 2024-05-15T23:24:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3300000000000001e-07 2024-05-15T23:24:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.753368+0000", 2024-05-15T23:24:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6728e-05 2024-05-15T23:24:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.753374+0000", 2024-05-15T23:24:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1e-06 2024-05-15T23:24:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.753378+0000", 2024-05-15T23:24:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3749999999999996e-06 2024-05-15T23:24:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.758202+0000", 2024-05-15T23:24:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004824226 2024-05-15T23:24:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.758286+0000", 2024-05-15T23:24:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3924000000000005e-05 2024-05-15T23:24:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:24:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.758422+0000", 2024-05-15T23:24:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578 2024-05-15T23:24:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.857856+0000", 2024-05-15T23:24:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099434211999999994 2024-05-15T23:24:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.865858+0000", 2024-05-15T23:24:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0080013389999999997 2024-05-15T23:24:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.865914+0000", 2024-05-15T23:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6514000000000001e-05 2024-05-15T23:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.865959+0000", 2024-05-15T23:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4910000000000002e-05 2024-05-15T23:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:112264 5.8 5:16d6920f:::benchmark_data_smithi090_116470_object112263:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e398)", 2024-05-15T23:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:20:48.767258+0000", 2024-05-15T23:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 243.24905427900001, 2024-05-15T23:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11474749300000001, 2024-05-15T23:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 112264 2024-05-15T23:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.767258+0000", 2024-05-15T23:24:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.767258+0000", 2024-05-15T23:24:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.767260+0000", 2024-05-15T23:24:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8169999999999999e-06 2024-05-15T23:24:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.767280+0000", 2024-05-15T23:24:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0117999999999999e-05 2024-05-15T23:24:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.856514+0000", 2024-05-15T23:24:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089234340999999995 2024-05-15T23:24:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.856525+0000", 2024-05-15T23:24:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0795000000000001e-05 2024-05-15T23:24:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.857904+0000", 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001379187 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.858039+0000", 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013446899999999999 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.858168+0000", 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128944 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.865971+0000", 2024-05-15T23:24:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0078029570000000001 2024-05-15T23:24:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.881910+0000", 2024-05-15T23:24:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015938933999999998 2024-05-15T23:24:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.881962+0000", 2024-05-15T23:24:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1924e-05 2024-05-15T23:24:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:20:48.882006+0000", 2024-05-15T23:24:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4007000000000001e-05 2024-05-15T23:24:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(5.9 426/407 [PushOp(5:915457ca:::benchmark_data_smithi090_116470_object33671:head, version: 382'2136, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:915457ca:::benchmark_data_smithi090_116470_object33671:head@382'2136, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-05-15T23:24:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:22:17.125971+0000", 2024-05-15T23:24:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 154.890341132, 2024-05-15T23:24:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12290139899999999, 2024-05-15T23:24:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:24:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.125971+0000", 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.125971+0000", 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.125972+0000", 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.16e-07 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.126038+0000", 2024-05-15T23:24:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5753999999999994e-05 2024-05-15T23:24:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.126050+0000", 2024-05-15T23:24:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2345e-05 2024-05-15T23:24:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.126063+0000", 2024-05-15T23:24:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2952000000000001e-05 2024-05-15T23:24:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.225456+0000", 2024-05-15T23:24:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099392709999999995 2024-05-15T23:24:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.225466+0000", 2024-05-15T23:24:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0152e-05 2024-05-15T23:24:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.248873+0000", 2024-05-15T23:24:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023406670000000001 2024-05-15T23:24:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(5.9 426/407 [PushOp(5:94b945b6:::benchmark_data_smithi090_116470_object33680:head, version: 382'2138, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:94b945b6:::benchmark_data_smithi090_116470_object33680:head@382'2138, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-05-15T23:24:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:22:17.127143+0000", 2024-05-15T23:24:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 154.889169611, 2024-05-15T23:24:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.110198669, 2024-05-15T23:24:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:24:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.127143+0000", 2024-05-15T23:24:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.127143+0000", 2024-05-15T23:24:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.127144+0000", 2024-05-15T23:24:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.85e-07 2024-05-15T23:24:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.127224+0000", 2024-05-15T23:24:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0759000000000004e-05 2024-05-15T23:24:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.211584+0000", 2024-05-15T23:24:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.084359782999999994 2024-05-15T23:24:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.211619+0000", 2024-05-15T23:24:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4660999999999999e-05 2024-05-15T23:24:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.232681+0000", 2024-05-15T23:24:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021062506000000002 2024-05-15T23:24:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.232695+0000", 2024-05-15T23:24:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3346999999999999e-05 2024-05-15T23:24:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.237342+0000", 2024-05-15T23:24:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046467280000000001 2024-05-15T23:24:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:231953 5.d 5:b01a298e:::benchmark_data_smithi090_116470_object78393:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:24:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:23:16.306799+0000", 2024-05-15T23:24:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.709513127999998, 2024-05-15T23:24:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11780294600000001, 2024-05-15T23:24:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 231953 2024-05-15T23:24:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306799+0000", 2024-05-15T23:24:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306799+0000", 2024-05-15T23:24:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306802+0000", 2024-05-15T23:24:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.157e-06 2024-05-15T23:24:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306811+0000", 2024-05-15T23:24:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1100000000000003e-06 2024-05-15T23:24:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306825+0000", 2024-05-15T23:24:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4642e-05 2024-05-15T23:24:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306835+0000", 2024-05-15T23:24:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0241000000000001e-05 2024-05-15T23:24:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306899+0000", 2024-05-15T23:24:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3317000000000004e-05 2024-05-15T23:24:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306983+0000", 2024-05-15T23:24:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4265999999999996e-05 2024-05-15T23:24:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:24:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.307055+0000", 2024-05-15T23:24:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2002000000000006e-05 2024-05-15T23:24:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.311076+0000", 2024-05-15T23:24:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040205149999999997 2024-05-15T23:24:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.424519+0000", 2024-05-15T23:24:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.113443878 2024-05-15T23:24:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.424556+0000", 2024-05-15T23:24:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6987999999999999e-05 2024-05-15T23:24:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.424602+0000", 2024-05-15T23:24:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.583e-05 2024-05-15T23:24:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:296795 5.8 5:1fb73cfc:::benchmark_data_smithi090_116470_object143235:head [delete] snapc 0=[] ondisk+write+known_if_redirected e467)", 2024-05-15T23:24:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:26.526457+0000", 2024-05-15T23:24:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.489855775999999, 2024-05-15T23:24:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10681265400000001, 2024-05-15T23:24:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 296795 2024-05-15T23:24:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:26.526457+0000", 2024-05-15T23:24:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:26.526457+0000", 2024-05-15T23:24:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:26.526459+0000", 2024-05-15T23:24:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1569999999999998e-06 2024-05-15T23:24:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:26.526464+0000", 2024-05-15T23:24:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9629999999999997e-06 2024-05-15T23:24:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:26.526474+0000", 2024-05-15T23:24:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0502999999999999e-05 2024-05-15T23:24:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:26.526482+0000", 2024-05-15T23:24:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0090000000000007e-06 2024-05-15T23:24:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:26.526543+0000", 2024-05-15T23:24:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1116000000000002e-05 2024-05-15T23:24:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:26.526665+0000", 2024-05-15T23:24:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121805 2024-05-15T23:24:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:24:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:26.526779+0000", 2024-05-15T23:24:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011345000000000001 2024-05-15T23:24:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:26.530687+0000", 2024-05-15T23:24:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039079659999999997 2024-05-15T23:24:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:26.633151+0000", 2024-05-15T23:24:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102464574 2024-05-15T23:24:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:26.633220+0000", 2024-05-15T23:24:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8418999999999997e-05 2024-05-15T23:24:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:26.633269+0000", 2024-05-15T23:24:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9691999999999998e-05 2024-05-15T23:24:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:298436 5.8 5:193e65af:::benchmark_data_smithi090_116470_object144876:head [delete] snapc 0=[] ondisk+write+known_if_redirected e467)", 2024-05-15T23:24:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:28.011449+0000", 2024-05-15T23:24:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.004863933999999, 2024-05-15T23:24:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14426002900000001, 2024-05-15T23:24:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 298436 2024-05-15T23:24:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011449+0000", 2024-05-15T23:24:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011449+0000", 2024-05-15T23:24:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011451+0000", 2024-05-15T23:24:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.007e-06 2024-05-15T23:24:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011456+0000", 2024-05-15T23:24:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5890000000000002e-06 2024-05-15T23:24:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011466+0000", 2024-05-15T23:24:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8519999999999999e-06 2024-05-15T23:24:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011474+0000", 2024-05-15T23:24:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6109999999999996e-06 2024-05-15T23:24:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011530+0000", 2024-05-15T23:24:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.643e-05 2024-05-15T23:24:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011628+0000", 2024-05-15T23:24:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7677999999999996e-05 2024-05-15T23:24:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:24:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011701+0000", 2024-05-15T23:24:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3036999999999999e-05 2024-05-15T23:24:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.021745+0000", 2024-05-15T23:24:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010044628 2024-05-15T23:24:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155616+0000", 2024-05-15T23:24:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13387112800000001 2024-05-15T23:24:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155668+0000", 2024-05-15T23:24:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1533999999999998e-05 2024-05-15T23:24:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155709+0000", 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0534999999999997e-05 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:298437 5.8 5:16c84348:::benchmark_data_smithi090_116470_object144877:head [delete] snapc 0=[] ondisk+write+known_if_redirected e467)", 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:28.011503+0000", 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.004809543, 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14441126500000001, 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 298437 2024-05-15T23:24:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011503+0000", 2024-05-15T23:24:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011503+0000", 2024-05-15T23:24:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011504+0000", 2024-05-15T23:24:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7100000000000001e-07 2024-05-15T23:24:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011507+0000", 2024-05-15T23:24:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2969999999999999e-06 2024-05-15T23:24:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011513+0000", 2024-05-15T23:24:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2929999999999997e-06 2024-05-15T23:24:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011518+0000", 2024-05-15T23:24:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0170000000000004e-06 2024-05-15T23:24:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.016632+0000", 2024-05-15T23:24:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005113345 2024-05-15T23:24:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.016718+0000", 2024-05-15T23:24:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5926000000000004e-05 2024-05-15T23:24:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:24:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.016829+0000", 2024-05-15T23:24:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011133000000000001 2024-05-15T23:24:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.031495+0000", 2024-05-15T23:24:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014666026 2024-05-15T23:24:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155838+0000", 2024-05-15T23:24:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.124343411 2024-05-15T23:24:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155881+0000", 2024-05-15T23:24:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2661999999999999e-05 2024-05-15T23:24:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155914+0000", 2024-05-15T23:24:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3186999999999999e-05 2024-05-15T23:24:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:298440 5.8 5:19a667cf:::benchmark_data_smithi090_116470_object144880:head [delete] snapc 0=[] ondisk+write+known_if_redirected e467)", 2024-05-15T23:24:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:28.017029+0000", 2024-05-15T23:24:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.999283248000001, 2024-05-15T23:24:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.138994741, 2024-05-15T23:24:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 298440 2024-05-15T23:24:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.017029+0000", 2024-05-15T23:24:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.017029+0000", 2024-05-15T23:24:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.017031+0000", 2024-05-15T23:24:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1169999999999998e-06 2024-05-15T23:24:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.017037+0000", 2024-05-15T23:24:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3020000000000002e-06 2024-05-15T23:24:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.017048+0000", 2024-05-15T23:24:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1418e-05 2024-05-15T23:24:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.017056+0000", 2024-05-15T23:24:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2390000000000002e-06 2024-05-15T23:24:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.021778+0000", 2024-05-15T23:24:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047214989999999997 2024-05-15T23:24:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.021903+0000", 2024-05-15T23:24:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012554299999999999 2024-05-15T23:24:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:24:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.022001+0000", 2024-05-15T23:24:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7408e-05 2024-05-15T23:24:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.031500+0000", 2024-05-15T23:24:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009499347 2024-05-15T23:24:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155953+0000", 2024-05-15T23:24:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.124453067 2024-05-15T23:24:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155993+0000", 2024-05-15T23:24:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0021999999999997e-05 2024-05-15T23:24:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.156024+0000", 2024-05-15T23:24:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0778999999999998e-05 2024-05-15T23:24:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:298453 5.8 5:19488f07:::benchmark_data_smithi090_116470_object144893:head [delete] snapc 0=[] ondisk+write+known_if_redirected e467)", 2024-05-15T23:24:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:28.023509+0000", 2024-05-15T23:24:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.992803871, 2024-05-15T23:24:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.132728863, 2024-05-15T23:24:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:24:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:24:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:24:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:24:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:24:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:24:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 298453 2024-05-15T23:24:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:24:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:24:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.023509+0000", 2024-05-15T23:24:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:24:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.023509+0000", 2024-05-15T23:24:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:24:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:24:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.023511+0000", 2024-05-15T23:24:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3599999999999999e-06 2024-05-15T23:24:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:24:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.023517+0000", 2024-05-15T23:24:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7579999999999998e-06 2024-05-15T23:24:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:24:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.023527+0000", 2024-05-15T23:24:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0665999999999999e-05 2024-05-15T23:24:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:24:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.023535+0000", 2024-05-15T23:24:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5639999999999999e-06 2024-05-15T23:24:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:24:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.026406+0000", 2024-05-15T23:24:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002871401 2024-05-15T23:24:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.026516+0000", 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109431 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.026619+0000", 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010360400000000001 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.031659+0000", 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050393149999999999 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:24:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.156152+0000", 2024-05-15T23:24:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.124493797 2024-05-15T23:24:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:24:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.156202+0000", 2024-05-15T23:24:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9336000000000003e-05 2024-05-15T23:24:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:24:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:24:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:24:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.156237+0000", 2024-05-15T23:24:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5630999999999998e-05 2024-05-15T23:24:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:24:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:24:52.126 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:24:52.127 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:24:52.127 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:24:52.127 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:24:52.127 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:24:52.127 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:24:52.148 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:24:52.163 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:24:52.164 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:24:52.164 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:24:52.165 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:24:52.165 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:24:52.165 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:24:52.165 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:24:52.165 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:24:52.165 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:24:52.166 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:24:52.166 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:24:52.166 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:24:52.166 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:24:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:52 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1725517512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:52.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:52 smithi090 ceph-mon[44434]: osdmap e475: 12 total, 12 up, 9 in 2024-05-15T23:24:52.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:52 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1725517512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:52.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:52 smithi084 ceph-mon[39784]: osdmap e475: 12 total, 12 up, 9 in 2024-05-15T23:24:52.489 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:24:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:52.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:24:52.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:52 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1725517512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:24:52.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:52 smithi088 ceph-mon[44699]: osdmap e475: 12 total, 12 up, 9 in 2024-05-15T23:24:52.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:24:52.626 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:52.679 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:24:52.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:52.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:24:52.704 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T23:24:52.784 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:52.813 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:53.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:53 smithi084 ceph-mon[39784]: pgmap v1282: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:24:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:53 smithi090 ceph-mon[44434]: pgmap v1282: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:24:53.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:24:53.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:53 smithi088 ceph-mon[44699]: pgmap v1282: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:24:53.766 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:53.767 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:53.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:53.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:53.873 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:24:53.906 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:53.928 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:24:53.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:53.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:24:54.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:24:54.068 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:54.090 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:24:54.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:54.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:24:54.110 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_4' created 2024-05-15T23:24:54.194 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:24:54.223 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:54 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:24:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:54 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:24:54.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:54 smithi088 ceph-mon[44699]: osdmap e476: 12 total, 12 up, 9 in 2024-05-15T23:24:54.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2594990578' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-05-15T23:24:54.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:54 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:24:54.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:54 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:24:54.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:54 smithi084 ceph-mon[39784]: osdmap e476: 12 total, 12 up, 9 in 2024-05-15T23:24:54.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2594990578' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-05-15T23:24:54.370 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:24:54.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:54.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:24:54.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:54.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:54 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:24:54.506 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:54 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:24:54.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:54 smithi090 ceph-mon[44434]: osdmap e476: 12 total, 12 up, 9 in 2024-05-15T23:24:54.507 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2594990578' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-05-15T23:24:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:54.532 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:54.633 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:24:54.666 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:54.688 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:24:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:54.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:24:54.793 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:24:54.820 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:54.849 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:24:54.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:54.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:24:54.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-05-15T23:24:54.953 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:24:54.986 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:55.008 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:24:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:55.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:24:55.113 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:24:55.143 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:55.164 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:24:55.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:55.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:24:55.268 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:24:55.294 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:55 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2594990578' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-05-15T23:24:55.294 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:55 smithi088 ceph-mon[44699]: osdmap e477: 12 total, 12 up, 9 in 2024-05-15T23:24:55.294 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:55 smithi088 ceph-mon[44699]: pgmap v1285: 17 pgs: 2 creating+peering, 14 unknown, 1 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:24:55.295 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:55.313 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:55.315 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:24:55.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:55.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:24:55.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:55 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2594990578' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-05-15T23:24:55.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:55 smithi084 ceph-mon[39784]: osdmap e477: 12 total, 12 up, 9 in 2024-05-15T23:24:55.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:55 smithi084 ceph-mon[39784]: pgmap v1285: 17 pgs: 2 creating+peering, 14 unknown, 1 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:24:55.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:24:55.446 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:55 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2594990578' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-05-15T23:24:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:55 smithi090 ceph-mon[44434]: osdmap e477: 12 total, 12 up, 9 in 2024-05-15T23:24:55.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:55 smithi090 ceph-mon[44434]: pgmap v1285: 17 pgs: 2 creating+peering, 14 unknown, 1 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:24:55.761 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:24:56.194 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:24:56.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:56.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:24:56.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:24:56.331 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:56.353 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:24:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:56.356 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:24:56.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:56 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:24:56.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:56 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:24:56.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:56 smithi084 ceph-mon[39784]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:24:56.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:56 smithi084 ceph-mon[39784]: osdmap e478: 12 total, 12 up, 9 in 2024-05-15T23:24:56.456 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:24:56.487 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:56 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:24:56.487 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:56 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:24:56.487 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:56 smithi088 ceph-mon[44699]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:24:56.488 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:56 smithi088 ceph-mon[44699]: osdmap e478: 12 total, 12 up, 9 in 2024-05-15T23:24:56.489 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:56.511 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:24:56.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:56.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:24:56.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:56 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:24:56.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:56 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:24:56.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:56 smithi090 ceph-mon[44434]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:24:56.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:56 smithi090 ceph-mon[44434]: osdmap e478: 12 total, 12 up, 9 in 2024-05-15T23:24:56.616 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:56.643 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:56.991 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:56.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:56.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:57.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:57.128 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:57.150 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:57.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:57.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:57.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:57 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:24:57.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:57 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:24:57.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:57 smithi084 ceph-mon[39784]: osdmap e479: 12 total, 12 up, 9 in 2024-05-15T23:24:57.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:24:57.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:24:57.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:24:57.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:24:57.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:24:57.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:24:57.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:57 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:24:57.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:57.281 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:57.300 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:57.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:57.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:24:57.434 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:57.470 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:24:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:57.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:24:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:57 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:24:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:57 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:24:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:57 smithi090 ceph-mon[44434]: osdmap e479: 12 total, 12 up, 9 in 2024-05-15T23:24:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:24:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:24:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:24:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:24:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:24:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:24:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:57 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:24:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:57 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:24:57.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:57 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:24:57.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:57 smithi088 ceph-mon[44699]: osdmap e479: 12 total, 12 up, 9 in 2024-05-15T23:24:57.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:24:57.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:24:57.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:24:57.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:24:57.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:24:57.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:24:57.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:57 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:24:57.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:57.605 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:57.623 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:57.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:57.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:57.728 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:24:57.761 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:57.783 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:24:57.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:57.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:24:57.830 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:24:57.830 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:24:57.887 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:24:57.916 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:58.091 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:58.132 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:24:58.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:58 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3552729384' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:24:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:58 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3552729384' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:24:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:58 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3552729384' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:24:58.865 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:24:58.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:58.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:24:58.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:24:59.002 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:59.024 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:24:59.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:59.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:24:59.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:24:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:24:59.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:59.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:24:59.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:24:59.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:59 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:24:59.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:59 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:24:59.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:59 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:24:59.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:59 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3552729384' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:24:59.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:59 smithi090 ceph-mon[44434]: osdmap e480: 12 total, 12 up, 9 in 2024-05-15T23:24:59.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:24:59 smithi090 ceph-mon[44434]: pgmap v1290: 17 pgs: 15 active+clean, 2 creating+peering; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:24:59.308 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:59.328 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:24:59.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:59.331 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:24:59.373 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T23:24:59.374 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:24:59.400 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:59 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:24:59.401 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:59 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:24:59.401 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:59 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:24:59.401 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:59 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3552729384' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:24:59.401 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:59 smithi084 ceph-mon[39784]: osdmap e480: 12 total, 12 up, 9 in 2024-05-15T23:24:59.401 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:24:59 smithi084 ceph-mon[39784]: pgmap v1290: 17 pgs: 15 active+clean, 2 creating+peering; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:24:59.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:24:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:24:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:59.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:24:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:59 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:24:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:59 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:24:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:59 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:24:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:59 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3552729384' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:24:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:59 smithi088 ceph-mon[44699]: osdmap e480: 12 total, 12 up, 9 in 2024-05-15T23:24:59.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:24:59 smithi088 ceph-mon[44699]: pgmap v1290: 17 pgs: 15 active+clean, 2 creating+peering; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:24:59.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:24:59.607 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:59.660 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:24:59.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:59.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:24:59.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:24:59.792 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:59.811 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:24:59.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:59.813 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:24:59.914 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:24:59.944 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:24:59.964 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:24:59.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:24:59.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:00.068 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:25:00.101 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:00.123 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:25:00.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:00.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:25:00.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:25:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:00.278 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:25:00.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:00.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:25:00.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:25:00.410 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:00.430 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:25:00.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:00.433 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:25:00.534 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:00.566 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:00.591 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:00.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:00.695 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:25:00.728 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:00.750 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:25:00.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:00.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:25:00.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:25:00.883 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:00.891 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:25:00.901 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:25:00.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:00.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:25:01.006 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:25:01.032 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:01.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:01 smithi084 ceph-mon[39784]: pgmap v1291: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:25:01.189 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:25:01.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:01.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:25:01.294 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:25:01.327 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:01.349 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:25:01.350 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:01 smithi088 ceph-mon[44699]: pgmap v1291: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:25:01.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:01.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:25:01.453 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:25:01.483 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:01.504 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:25:01.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:01.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:25:01.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:01 smithi090 ceph-mon[44434]: pgmap v1291: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:25:01.607 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:01.636 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:01.657 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:01.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:01.666 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:01.700 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:01.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:25:01.795 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:01.817 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:25:01.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:01.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:25:01.922 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:01.951 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:02.299 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:02.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:02.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:02.403 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:25:02.419 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:25:02.430 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:02.439 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:25:02.460 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:25:02.480 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:25:02.500 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:25:02.519 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:25:02.539 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:25:02.558 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:25:02.580 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:25:02.599 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:25:02.618 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:25:02.638 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:25:02.639 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:25:02.639 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:25:02.640 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:25:02.640 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:25:02.640 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:25:02.640 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:25:02.640 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:25:02.640 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:25:02.640 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:25:02.640 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:25:02.640 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:25:02.640 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:25:02.858 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:25:02.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:02.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:25:02.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:25:02.995 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:03.016 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:25:03.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:03.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:25:03.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:03 smithi084 ceph-mon[39784]: osdmap e481: 12 total, 12 up, 9 in 2024-05-15T23:25:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3552409276' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:25:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:03.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:03 smithi084 ceph-mon[39784]: pgmap v1293: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:25:03.120 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:03.149 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:03.176 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:03.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:03.280 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:03 smithi090 ceph-mon[44434]: osdmap e481: 12 total, 12 up, 9 in 2024-05-15T23:25:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3552409276' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:25:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:03 smithi090 ceph-mon[44434]: pgmap v1293: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:25:03.313 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:03.335 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:03.337 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:03 smithi088 ceph-mon[44699]: osdmap e481: 12 total, 12 up, 9 in 2024-05-15T23:25:03.338 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3552409276' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:25:03.338 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:03.338 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:03.338 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:03.338 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:03.338 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:03.338 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:03 smithi088 ceph-mon[44699]: pgmap v1293: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:25:03.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:03.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:03.439 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:25:03.468 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:03.489 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:25:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:03.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:25:03.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:03.622 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:04.070 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:04.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:04.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:04.095 INFO:teuthology.orchestra.run.smithi084.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-05-15T23:25:04.174 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:25:04.201 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:04.221 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:25:04.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:04.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:25:04.329 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:25:04.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/836737058' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:25:04.359 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.359 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.359 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.359 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.360 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.360 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.360 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.360 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/836737058' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:25:04.361 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:25:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:04.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:25:04.487 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:04.516 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:04.537 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:04.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:04.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/836737058' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:25:04.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:25:04.669 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:05.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:05 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/836737058' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:25:05.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:05 smithi084 ceph-mon[39784]: osdmap e482: 12 total, 12 up, 9 in 2024-05-15T23:25:05.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:05 smithi084 ceph-mon[39784]: pgmap v1295: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:25:05.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:05 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/836737058' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:25:05.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:05 smithi090 ceph-mon[44434]: osdmap e482: 12 total, 12 up, 9 in 2024-05-15T23:25:05.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:05 smithi090 ceph-mon[44434]: pgmap v1295: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:25:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:05 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/836737058' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:25:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:05 smithi088 ceph-mon[44699]: osdmap e482: 12 total, 12 up, 9 in 2024-05-15T23:25:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:05 smithi088 ceph-mon[44699]: pgmap v1295: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:25:05.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:25:06.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:06 smithi084 ceph-mon[39784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:25:06.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:06 smithi090 ceph-mon[44434]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:25:06.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:06 smithi088 ceph-mon[44699]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:25:06.948 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:25:06.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:06.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:25:07.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:07.078 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:25:07.087 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:07.111 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:07.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:07.114 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:07.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:25:07.245 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:07.265 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:25:07.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:07.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:25:07.313 DEBUG:teuthology.orchestra.run.smithi090:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-05-15T23:25:07.315 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T23:25:07.315 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T23:25:07.369 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:07 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2208657761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:25:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:07 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:25:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:07 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:25:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:07 smithi084 ceph-mon[39784]: pgmap v1296: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:25:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:07 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:25:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:07 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2208657761' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:25:07.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:07 smithi084 ceph-mon[39784]: osdmap e483: 12 total, 12 up, 9 in 2024-05-15T23:25:07.401 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:07 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2208657761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:25:07.401 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:07 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:25:07.401 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:07 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:25:07.401 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:07 smithi088 ceph-mon[44699]: pgmap v1296: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:25:07.401 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:07 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:25:07.401 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:07 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2208657761' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:25:07.401 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:07 smithi088 ceph-mon[44699]: osdmap e483: 12 total, 12 up, 9 in 2024-05-15T23:25:07.403 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:07.425 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:07.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:07.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:07.447 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:hints = 1 2024-05-15T23:25:07.447 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T23:25:07.447 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object prefix: benchmark_data_smithi090_131278 2024-05-15T23:25:07.448 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:25:07.448 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 0 0 0 0 0 0 - 0 2024-05-15T23:25:07.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:25:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:07 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2208657761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:25:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:07 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:25:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:07 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:25:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:07 smithi090 ceph-mon[44434]: pgmap v1296: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:25:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:07 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:25:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:07 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2208657761' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:25:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:07 smithi090 ceph-mon[44434]: osdmap e483: 12 total, 12 up, 9 in 2024-05-15T23:25:07.569 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:07.597 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:25:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:07.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:25:07.702 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:07.750 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:07.802 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:25:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:25:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:25:07.802 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:25:07.802 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:25:08.311 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:08.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:08.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:08 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:25:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:08 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:25:08.419 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:25:08.448 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 1 16 6135 6119 382.4 382.438 0.00167797 0.00260867 2024-05-15T23:25:08.456 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-05-15T23:25:08.456 INFO:tasks.thrashosds.thrasher:Killing osd 8, live_osds are [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2, 0] 2024-05-15T23:25:08.456 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-15T23:25:08.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.8 2024-05-15T23:25:08.459 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:08 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:25:08.460 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:08 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:25:08.462 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:08.489 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:25:08.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:08 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:25:08.490 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:08 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:25:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:08.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:25:08.594 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:25:08.633 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:08.663 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:25:08.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:08.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:25:08.770 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:08.803 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:25:08 smithi090 systemd[1]: Stopping Ceph osd.8 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:25:08.813 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:08.846 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:08.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:08.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:08.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:25:08.993 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:09.019 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:25:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:09.022 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:25:09.119 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T23:25:09.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:09.145 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:09.180 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:09.303 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:25:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[50278]: 2024-05-15T23:25:08.870+0000 7f0ac4e4b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:25:09.303 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:25:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[50278]: 2024-05-15T23:25:08.870+0000 7f0ac4e4b700 -1 osd.8 484 *** Got signal Terminated *** 2024-05-15T23:25:09.303 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:25:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[50278]: 2024-05-15T23:25:08.870+0000 7f0ac4e4b700 -1 osd.8 484 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:25:09.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:09 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:25:09.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:09 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:25:09.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:09 smithi090 ceph-mon[44434]: osdmap e484: 12 total, 12 up, 9 in 2024-05-15T23:25:09.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:09 smithi090 ceph-mon[44434]: osd.8 marked itself down and dead 2024-05-15T23:25:09.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:09 smithi090 ceph-mon[44434]: pgmap v1299: 17 pgs: 17 active+clean; 24 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 3.8 MiB/s wr, 61 op/s 2024-05-15T23:25:09.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:09 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:25:09.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:09 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:25:09.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:09 smithi084 ceph-mon[39784]: osdmap e484: 12 total, 12 up, 9 in 2024-05-15T23:25:09.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:09 smithi084 ceph-mon[39784]: osd.8 marked itself down and dead 2024-05-15T23:25:09.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:09 smithi084 ceph-mon[39784]: pgmap v1299: 17 pgs: 17 active+clean; 24 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 3.8 MiB/s wr, 61 op/s 2024-05-15T23:25:09.428 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:09.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:09.448 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 2 16 9813 9797 306.122 229.875 0.00215005 0.00277684 2024-05-15T23:25:09.532 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:25:09.559 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:09 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:25:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:09 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:25:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:09 smithi088 ceph-mon[44699]: osdmap e484: 12 total, 12 up, 9 in 2024-05-15T23:25:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:09 smithi088 ceph-mon[44699]: osd.8 marked itself down and dead 2024-05-15T23:25:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:09 smithi088 ceph-mon[44699]: pgmap v1299: 17 pgs: 17 active+clean; 24 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 3.8 MiB/s wr, 61 op/s 2024-05-15T23:25:09.632 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:25:09.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:09.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:25:09.736 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:09.764 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:10.002 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:10.013 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:10.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:10.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:10.118 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:10.145 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:10.267 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:10.269 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:10 smithi084 ceph-mon[39784]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-15T23:25:10.269 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:10 smithi084 ceph-mon[39784]: osdmap e485: 12 total, 11 up, 9 in 2024-05-15T23:25:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:10.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:10.303 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:25:10 smithi090 bash[131405]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8 2024-05-15T23:25:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:10 smithi090 ceph-mon[44434]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-15T23:25:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:10 smithi090 ceph-mon[44434]: osdmap e485: 12 total, 11 up, 9 in 2024-05-15T23:25:10.351 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:25:10.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:10.402 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:10.422 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:10.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:10.448 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 3 16 9813 9797 204.06 0 - 0.00277684 2024-05-15T23:25:10.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:10.554 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:10.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:10 smithi088 ceph-mon[44699]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-15T23:25:10.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:10 smithi088 ceph-mon[44699]: osdmap e485: 12 total, 11 up, 9 in 2024-05-15T23:25:10.805 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:10.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:10.809 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:10.910 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:11.138 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:11.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:11.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:11.155 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:11.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:11 smithi084 ceph-mon[39784]: osdmap e486: 12 total, 11 up, 9 in 2024-05-15T23:25:11.236 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:11 smithi084 ceph-mon[39784]: pgmap v1302: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 401 MiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 100 MiB/s wr, 1.60k op/s; 1214/12822 objects degraded (9.468%) 2024-05-15T23:25:11.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:11.275 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:11.449 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 4 16 9813 9797 153.047 0 - 0.00277684 2024-05-15T23:25:11.475 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:11.476 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:11 smithi090 ceph-mon[44434]: osdmap e486: 12 total, 11 up, 9 in 2024-05-15T23:25:11.476 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:11 smithi090 ceph-mon[44434]: pgmap v1302: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 401 MiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 100 MiB/s wr, 1.60k op/s; 1214/12822 objects degraded (9.468%) 2024-05-15T23:25:11.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:11.478 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:11 smithi088 ceph-mon[44699]: osdmap e486: 12 total, 11 up, 9 in 2024-05-15T23:25:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:11 smithi088 ceph-mon[44699]: pgmap v1302: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 401 MiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 100 MiB/s wr, 1.60k op/s; 1214/12822 objects degraded (9.468%) 2024-05-15T23:25:11.579 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:25:11.608 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:11.808 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:25:11.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:11.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:25:11.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:25:11.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.8.service' 2024-05-15T23:25:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:11.960 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:25:11.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:11.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:25:11.988 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:25:11.988 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-15T23:25:11.988 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd down 8 2024-05-15T23:25:12.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:12.092 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:12 smithi090 ceph-mon[44434]: Health check failed: Degraded data redundancy: 1214/12822 objects degraded (9.468%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:25:12.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:12 smithi090 ceph-mon[44434]: osdmap e487: 12 total, 11 up, 9 in 2024-05-15T23:25:12.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:12 smithi084 ceph-mon[39784]: Health check failed: Degraded data redundancy: 1214/12822 objects degraded (9.468%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:25:12.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:12 smithi084 ceph-mon[39784]: osdmap e487: 12 total, 11 up, 9 in 2024-05-15T23:25:12.449 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 5 16 9813 9797 122.438 0 - 0.00277684 2024-05-15T23:25:12.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:12 smithi088 ceph-mon[44699]: Health check failed: Degraded data redundancy: 1214/12822 objects degraded (9.468%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:25:12.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:12 smithi088 ceph-mon[44699]: osdmap e487: 12 total, 11 up, 9 in 2024-05-15T23:25:13.142 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:13.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:13.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:13.246 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:13.279 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:13.301 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:13.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:13.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:13.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/155977458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:25:13.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:13 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:13.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:13 smithi084 ceph-mon[39784]: pgmap v1304: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 120 MiB/s wr, 1.92k op/s; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:13.405 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:25:13.435 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:13.450 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 6 16 9813 9797 102.033 0 - 0.00277684 2024-05-15T23:25:13.533 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:25:13.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/155977458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:25:13.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:13 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:13.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:13 smithi090 ceph-mon[44434]: pgmap v1304: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 120 MiB/s wr, 1.92k op/s; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:13.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:25:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/155977458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:25:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:13 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:13 smithi088 ceph-mon[44699]: pgmap v1304: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 120 MiB/s wr, 1.92k op/s; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:13.658 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:13.690 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:13.712 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:13.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:13.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:13.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:25:13.850 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:13.871 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:25:13.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:13.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:25:13.976 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:25:14.005 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:14.025 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:25:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:14.028 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:25:14.129 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:14.162 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:14.183 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:14.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:14.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:14.288 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:25:14.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.316 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:14.320 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:25:14.320 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:25:14.335 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:25:14.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:14.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:25:14.439 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:14.449 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 7 16 9813 9797 87.4575 0 - 0.00277684 2024-05-15T23:25:14.470 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.470 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.470 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.470 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.470 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.470 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.470 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.470 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.470 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.470 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.472 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:14.494 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:14.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:14.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:14.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:14.598 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:14.648 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:14.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:14.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:14.753 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:25:14.782 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:14.802 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:25:14.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:14.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:25:14.908 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:14.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:25:14.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:25:14.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:25:14.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:25:14.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:25:14.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:25:14.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:25:14.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:25:14.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:25:14.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:25:14.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:25:14.914 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:25:14.914 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:25:14.914 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:25:14.914 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:25:14.914 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:25:14.914 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:25:14.914 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:25:14.914 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:25:14.914 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-05-15T23:25:14.914 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:25:14.914 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:25:14.915 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:25:14.937 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:14.956 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:14.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:15.061 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:15.092 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:15.166 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:15.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:15.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:15.271 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:15.319 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:15.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:15.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:15.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:15 smithi084 ceph-mon[39784]: pgmap v1305: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:15.423 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:25:15.449 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 8 16 9813 9797 76.5255 0 - 0.00277684 2024-05-15T23:25:15.450 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:15.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:15 smithi090 ceph-mon[44434]: pgmap v1305: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:15.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:15 smithi088 ceph-mon[44699]: pgmap v1305: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:25:15.922 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:25:15.945 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:15.951 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:25:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:15.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:25:16.056 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:16.085 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:16.160 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:25:16.410 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:16 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/646125974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:25:16.433 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:16.435 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T23:25:16.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:16.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:16.449 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 9 16 9813 9797 68.0226 0 - 0.00277684 2024-05-15T23:25:16.537 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:25:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:16 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/646125974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:25:16.563 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:16 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/646125974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:25:17.110 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:25:17.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:17.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:25:17.215 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:17.249 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:17.270 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:17.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:17.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:17 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:25:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:17 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/646125974' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:25:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:17 smithi084 ceph-mon[39784]: osdmap e488: 12 total, 11 up, 9 in 2024-05-15T23:25:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:17 smithi084 ceph-mon[39784]: pgmap v1307: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 74 MiB/s wr, 1.19k op/s; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:17.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:17 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 1214/19598 objects degraded (6.195%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:25:17.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:17 smithi084 ceph-mon[39784]: osdmap e489: 12 total, 11 up, 9 in 2024-05-15T23:25:17.374 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:25:17.401 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:17.420 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:25:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:17.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:25:17.450 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 10 16 9813 9797 61.2206 0 - 0.00277684 2024-05-15T23:25:17.524 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:17.551 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:17.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:17 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:25:17.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:17 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/646125974' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:25:17.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:17 smithi090 ceph-mon[44434]: osdmap e488: 12 total, 11 up, 9 in 2024-05-15T23:25:17.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:17 smithi090 ceph-mon[44434]: pgmap v1307: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 74 MiB/s wr, 1.19k op/s; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:17.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:17 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 1214/19598 objects degraded (6.195%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:25:17.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:17 smithi090 ceph-mon[44434]: osdmap e489: 12 total, 11 up, 9 in 2024-05-15T23:25:17.570 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:17.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:17 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:25:17.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:17 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/646125974' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:25:17.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:17 smithi088 ceph-mon[44699]: osdmap e488: 12 total, 11 up, 9 in 2024-05-15T23:25:17.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:17 smithi088 ceph-mon[44699]: pgmap v1307: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 74 MiB/s wr, 1.19k op/s; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:17.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:17 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 1214/19598 objects degraded (6.195%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:25:17.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:17 smithi088 ceph-mon[44699]: osdmap e489: 12 total, 11 up, 9 in 2024-05-15T23:25:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:17.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:17.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:17.701 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:17.922 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:25:18.135 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:18.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:18.155 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:18.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:18.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:25:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:25:18.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:18.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:25:18.416 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:18.446 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:18.449 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 11 16 9813 9797 55.6551 0 - 0.00277684 2024-05-15T23:25:18.467 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:18.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:18.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:18.572 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:25:18.607 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:18.609 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:18.630 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:25:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:18.634 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:25:18.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:25:18.764 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:18.783 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:25:18.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:18.786 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:25:18.887 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:18.920 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:18.942 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:18.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:18.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:19.047 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:19.075 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:19.146 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:25:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:19 smithi088 ceph-mon[44699]: pgmap v1309: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 14 MiB/s wr, 218 op/s; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:19 smithi084 ceph-mon[39784]: pgmap v1309: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 14 MiB/s wr, 218 op/s; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:19.450 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 12 16 9813 9797 51.0174 0 - 0.00277684 2024-05-15T23:25:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:19 smithi090 ceph-mon[44434]: pgmap v1309: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 14 MiB/s wr, 218 op/s; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:20.003 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:20.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:20.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:20.108 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:20.141 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:20.163 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:20.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:20.268 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:20.297 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:20.317 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:20.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:20.422 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:25:20.449 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:20.450 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 13 16 9813 9797 47.093 0 - 0.00277684 2024-05-15T23:25:20.469 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:25:20.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:20.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:25:20.573 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:25:20.603 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:20.630 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:25:20.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:20.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:25:20.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:20.761 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:21.321 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:25:21.322 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:25:21.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:21 smithi084 ceph-mon[39784]: pgmap v1310: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:21.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:21 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:25:21.108+0000 7fa0938b7700 -1 mon.a@0(leader).osd e489 definitely_dead 0 2024-05-15T23:25:21.431 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8 is already down. 2024-05-15T23:25:21.450 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 14 16 9813 9797 43.7293 0 - 0.00277684 2024-05-15T23:25:21.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:21 smithi090 ceph-mon[44434]: pgmap v1310: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:21.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:21 smithi088 ceph-mon[44699]: pgmap v1310: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:22.025 INFO:tasks.thrashosds.thrasher:Removing osd 8, in_osds are: [1, 3, 4, 5, 7, 8, 11, 9, 10] 2024-05-15T23:25:22.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd out 8 2024-05-15T23:25:22.240 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:22.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:22.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:22.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/117321101' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["8"]}]: dispatch 2024-05-15T23:25:22.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:25:22.372 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:22.450 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 15 16 9813 9797 40.814 0 - 0.00277684 2024-05-15T23:25:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/117321101' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["8"]}]: dispatch 2024-05-15T23:25:22.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/117321101' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["8"]}]: dispatch 2024-05-15T23:25:22.676 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T23:25:23.063 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:23.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:25:23.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:25:23.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:23 smithi084 ceph-mon[39784]: pgmap v1311: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:23.450 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 16 16 9813 9797 38.2632 0 - 0.00277684 2024-05-15T23:25:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:25:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:25:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:23 smithi090 ceph-mon[44434]: pgmap v1311: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:25:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:25:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:23 smithi088 ceph-mon[44699]: pgmap v1311: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:23.969 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:24 smithi084 ceph-mon[39784]: osdmap e490: 12 total, 11 up, 9 in 2024-05-15T23:25:24.450 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 17 16 9813 9797 36.0124 0 - 0.00277684 2024-05-15T23:25:24.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:24 smithi090 ceph-mon[44434]: osdmap e490: 12 total, 11 up, 9 in 2024-05-15T23:25:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:24 smithi088 ceph-mon[44699]: osdmap e490: 12 total, 11 up, 9 in 2024-05-15T23:25:24.794 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:25:24.813 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:25:24.832 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:25:24.851 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:25:24.874 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:25:24.894 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:25:24.914 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:25:24.934 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:25:24.954 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:25:24.973 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:25:24.994 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:25:24.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:25:24.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:25:24.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:25:24.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:25:24.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:25:24.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:25:24.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:25:24.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:25:24.995 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-05-15T23:25:24.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:25:24.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:25:24.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:25:25.022 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:25:25.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:25.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:25:25.126 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:25:25.156 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:25.194 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:25:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:25.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:25:25.298 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:25.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:25 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3432056377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:25:25.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:25.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:25 smithi084 ceph-mon[39784]: pgmap v1313: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:25.324 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:25.344 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:25.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:25.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:25:25.451 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 18 16 9813 9797 34.0118 0 - 0.00277684 2024-05-15T23:25:25.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:25 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3432056377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:25:25.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:25.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:25 smithi090 ceph-mon[44434]: pgmap v1313: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:25.482 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:25.504 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:25:25.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:25.507 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:25:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:25 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3432056377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:25:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:25 smithi088 ceph-mon[44699]: pgmap v1313: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:25.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:25.636 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:25.664 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:25.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:25.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:25.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:25:25.802 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:25.823 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:25:25.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:25.827 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:25:25.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:25.958 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:25.978 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:25.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:25.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:26.082 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:26.115 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:26.129 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:25:26.139 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:26.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:26.244 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:26.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:26 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:26 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4252913504' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:25:26.272 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:26.395 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:26.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:26.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:26.451 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 19 16 9813 9797 32.2217 0 - 0.00277684 2024-05-15T23:25:26.498 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:25:26.525 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:26 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:26 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4252913504' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:25:26.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:26 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:26.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:26 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4252913504' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:25:26.873 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:25:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:26.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:25:26.925 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:26.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:27.010 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:27.031 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:27.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:27.034 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:27.135 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:27.167 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:27.188 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:27.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:27.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:27.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:27.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:25:27.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:25:27.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:27 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:25:27.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:27 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4252913504' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:25:27.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:27 smithi090 ceph-mon[44434]: osdmap e491: 12 total, 11 up, 9 in 2024-05-15T23:25:27.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:27 smithi090 ceph-mon[44434]: pgmap v1315: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:27.322 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:27.342 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:27.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:27.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:27.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:25:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:25:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:27 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:25:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:27 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4252913504' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:25:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:27 smithi084 ceph-mon[39784]: osdmap e491: 12 total, 11 up, 9 in 2024-05-15T23:25:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:27 smithi084 ceph-mon[39784]: pgmap v1315: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:27.446 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:25:27.451 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:25:27.450426 min lat: 0.0010209 max lat: 0.0101484 avg lat: 0.00277684 2024-05-15T23:25:27.451 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:25:27.451 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 20 16 9813 9797 30.6107 0 - 0.00277684 2024-05-15T23:25:27.476 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:25:27.476 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:25:27.476 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:27 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:25:27.476 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:27 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4252913504' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:25:27.477 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:27 smithi088 ceph-mon[44699]: osdmap e491: 12 total, 11 up, 9 in 2024-05-15T23:25:27.477 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:27 smithi088 ceph-mon[44699]: pgmap v1315: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:27.479 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:27.501 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:25:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:27.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:25:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:25:27.559 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:25:27.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:25:27.637 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:27.658 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:25:27.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:27.661 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:25:27.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:25:27.792 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:27.846 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:25:27.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:27.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:25:27.963 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:25:27.996 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:28.018 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:25:28.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:28.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:25:28.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:28.149 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:28.221 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:28.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:28.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:28.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:25:28.356 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:25:28.357 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:25:28.357 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:25:28.357 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:25:28.357 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:25:28.358 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:28.380 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:25:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:28.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:25:28.451 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 21 16 9813 9797 29.1531 0 - 0.00277684 2024-05-15T23:25:28.484 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:28.517 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:28.539 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:28.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:28.643 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:25:28.673 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:28.693 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:25:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:28.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:25:28.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:25:28.828 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:28.849 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:25:28.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:28.853 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:25:28.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:25:28.982 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:29.129 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:25:29.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:29.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:25:29.153 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T23:25:29.233 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:29.261 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:29 smithi084 ceph-mon[39784]: pgmap v1316: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:29.263 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:29.451 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 22 16 9813 9797 27.828 0 - 0.00277684 2024-05-15T23:25:29.512 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:29.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:29.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:29.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:29 smithi090 ceph-mon[44434]: pgmap v1316: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:29.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:25:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:29 smithi088 ceph-mon[44699]: pgmap v1316: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:29.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:25:29.648 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:29.670 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:25:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:29.674 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:25:29.775 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:25:29.805 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:29.825 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:25:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:29.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:25:29.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:29.958 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:29.978 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:29.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:29.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:30.067 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:30.081 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:25:30.114 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:30.136 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:25:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:30.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:25:30.144 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:25:30.241 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:25:30.270 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:30.291 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:25:30.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:30.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:25:30.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:30.428 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:30.449 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:30.451 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 23 16 9813 9797 26.6182 0 - 0.00277684 2024-05-15T23:25:30.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:30.452 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:30.553 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:25:30.583 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:30.602 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:25:30.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:30.605 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:25:30.706 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:25:30.734 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:25:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:30.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:25:30.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:25:30.884 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:30.903 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:25:30.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:30.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:25:31.006 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:25:31.039 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:31.060 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:25:31.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:31.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:25:31.165 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:31.192 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:31.211 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:31.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:31.314 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:25:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:31 smithi088 ceph-mon[44699]: pgmap v1317: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:31.344 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:31.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:31 smithi084 ceph-mon[39784]: pgmap v1317: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:31.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:31 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:25:31.092+0000 7fa0938b7700 -1 mon.a@0(leader).osd e491 definitely_dead 0 2024-05-15T23:25:31.451 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 24 16 9813 9797 25.5091 0 - 0.00277684 2024-05-15T23:25:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:31 smithi090 ceph-mon[44434]: pgmap v1317: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:31.692 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:25:31.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:31.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:25:31.796 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:31.828 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:31.849 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:31.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:31.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:31.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:31.984 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:32.006 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:32.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:32.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:32.090 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.8. 2024-05-15T23:25:32.110 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:32.143 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:32.164 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:32.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:32.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:32.253 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:25:32.253 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-05-15T23:25:32.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-05-15T23:25:32.254 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:25:32.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.254 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329853 4.b 4.c5da395b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.254 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027062+0000", 2024-05-15T23:25:32.254 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.222778441, 2024-05-15T23:25:32.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29250239, 2024-05-15T23:25:32.254 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.254 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.254 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.254 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.254 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.254 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.255 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329853 2024-05-15T23:25:32.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.255 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027062+0000", 2024-05-15T23:25:32.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027062+0000", 2024-05-15T23:25:32.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027064+0000", 2024-05-15T23:25:32.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8899999999999999e-06 2024-05-15T23:25:32.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027069+0000", 2024-05-15T23:25:32.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5729999999999998e-06 2024-05-15T23:25:32.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027081+0000", 2024-05-15T23:25:32.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1493e-05 2024-05-15T23:25:32.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027090+0000", 2024-05-15T23:25:32.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9800000000000004e-06 2024-05-15T23:25:32.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027175+0000", 2024-05-15T23:25:32.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4954e-05 2024-05-15T23:25:32.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027185+0000", 2024-05-15T23:25:32.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0351e-05 2024-05-15T23:25:32.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319531+0000", 2024-05-15T23:25:32.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29234572399999997 2024-05-15T23:25:32.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319564+0000", 2024-05-15T23:25:32.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3424999999999999e-05 2024-05-15T23:25:32.259 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.259 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.259 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.259 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329870 4.b 4.4b410cb (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.259 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027397+0000", 2024-05-15T23:25:32.259 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.22244378099998, 2024-05-15T23:25:32.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29232418599999999, 2024-05-15T23:25:32.260 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.260 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.260 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.260 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.260 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.260 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.260 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329870 2024-05-15T23:25:32.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.260 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027397+0000", 2024-05-15T23:25:32.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027397+0000", 2024-05-15T23:25:32.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027397+0000", 2024-05-15T23:25:32.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5600000000000005e-07 2024-05-15T23:25:32.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027400+0000", 2024-05-15T23:25:32.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5210000000000001e-06 2024-05-15T23:25:32.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027409+0000", 2024-05-15T23:25:32.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3039999999999994e-06 2024-05-15T23:25:32.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027417+0000", 2024-05-15T23:25:32.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9039999999999992e-06 2024-05-15T23:25:32.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027462+0000", 2024-05-15T23:25:32.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4540999999999999e-05 2024-05-15T23:25:32.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027469+0000", 2024-05-15T23:25:32.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.841e-06 2024-05-15T23:25:32.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319715+0000", 2024-05-15T23:25:32.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29224517700000002 2024-05-15T23:25:32.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319721+0000", 2024-05-15T23:25:32.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1419999999999998e-06 2024-05-15T23:25:32.264 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.264 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.264 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.264 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329880 4.b 4.cc30174b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.264 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027430+0000", 2024-05-15T23:25:32.264 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.222410701, 2024-05-15T23:25:32.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29233345300000002, 2024-05-15T23:25:32.264 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.265 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.265 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.265 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.265 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.265 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.265 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329880 2024-05-15T23:25:32.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.265 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027430+0000", 2024-05-15T23:25:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027430+0000", 2024-05-15T23:25:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027430+0000", 2024-05-15T23:25:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7100000000000001e-07 2024-05-15T23:25:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027433+0000", 2024-05-15T23:25:32.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7360000000000001e-06 2024-05-15T23:25:32.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027439+0000", 2024-05-15T23:25:32.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6729999999999997e-06 2024-05-15T23:25:32.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.268 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.268 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027445+0000", 2024-05-15T23:25:32.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7699999999999998e-06 2024-05-15T23:25:32.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.268 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.268 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027508+0000", 2024-05-15T23:25:32.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3566999999999996e-05 2024-05-15T23:25:32.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.268 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.268 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027517+0000", 2024-05-15T23:25:32.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3919999999999992e-06 2024-05-15T23:25:32.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319755+0000", 2024-05-15T23:25:32.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29223720600000003 2024-05-15T23:25:32.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319763+0000", 2024-05-15T23:25:32.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4379999999999995e-06 2024-05-15T23:25:32.269 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.269 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.269 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.270 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329890 4.b 4.4872c0cb (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.270 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027451+0000", 2024-05-15T23:25:32.270 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.222389475, 2024-05-15T23:25:32.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.292331216, 2024-05-15T23:25:32.270 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.270 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.270 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.270 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.270 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.270 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.270 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329890 2024-05-15T23:25:32.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.271 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027451+0000", 2024-05-15T23:25:32.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027451+0000", 2024-05-15T23:25:32.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027452+0000", 2024-05-15T23:25:32.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1799999999999995e-07 2024-05-15T23:25:32.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027454+0000", 2024-05-15T23:25:32.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1629999999999999e-06 2024-05-15T23:25:32.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027458+0000", 2024-05-15T23:25:32.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7600000000000002e-06 2024-05-15T23:25:32.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027463+0000", 2024-05-15T23:25:32.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0509999999999996e-06 2024-05-15T23:25:32.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027551+0000", 2024-05-15T23:25:32.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7164999999999996e-05 2024-05-15T23:25:32.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027558+0000", 2024-05-15T23:25:32.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4619999999999999e-06 2024-05-15T23:25:32.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319778+0000", 2024-05-15T23:25:32.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29221955900000002 2024-05-15T23:25:32.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319782+0000", 2024-05-15T23:25:32.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4379999999999997e-06 2024-05-15T23:25:32.275 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.275 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.275 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.275 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329893 4.b 4.9d86cb5b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.275 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027472+0000", 2024-05-15T23:25:32.275 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.22236812099999, 2024-05-15T23:25:32.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29242339299999998, 2024-05-15T23:25:32.275 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.275 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.275 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.276 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.276 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.276 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.276 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329893 2024-05-15T23:25:32.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.276 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.276 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027472+0000", 2024-05-15T23:25:32.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.276 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027472+0000", 2024-05-15T23:25:32.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027473+0000", 2024-05-15T23:25:32.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9699999999999996e-07 2024-05-15T23:25:32.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027475+0000", 2024-05-15T23:25:32.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2680000000000001e-06 2024-05-15T23:25:32.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027480+0000", 2024-05-15T23:25:32.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1819999999999997e-06 2024-05-15T23:25:32.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027485+0000", 2024-05-15T23:25:32.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0300000000000001e-06 2024-05-15T23:25:32.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027583+0000", 2024-05-15T23:25:32.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7405999999999995e-05 2024-05-15T23:25:32.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027586+0000", 2024-05-15T23:25:32.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8510000000000001e-06 2024-05-15T23:25:32.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319890+0000", 2024-05-15T23:25:32.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29230399000000001 2024-05-15T23:25:32.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319896+0000", 2024-05-15T23:25:32.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.169e-06 2024-05-15T23:25:32.279 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.279 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.280 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.280 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329904 4.b 4.45b9298b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.280 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027497+0000", 2024-05-15T23:25:32.280 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.222343303, 2024-05-15T23:25:32.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29242595799999999, 2024-05-15T23:25:32.280 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.280 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.280 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.280 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.280 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.280 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.280 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329904 2024-05-15T23:25:32.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.281 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027497+0000", 2024-05-15T23:25:32.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027497+0000", 2024-05-15T23:25:32.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027497+0000", 2024-05-15T23:25:32.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9499999999999998e-07 2024-05-15T23:25:32.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027499+0000", 2024-05-15T23:25:32.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7799999999999999e-06 2024-05-15T23:25:32.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027503+0000", 2024-05-15T23:25:32.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6100000000000002e-06 2024-05-15T23:25:32.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027506+0000", 2024-05-15T23:25:32.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9510000000000002e-06 2024-05-15T23:25:32.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027616+0000", 2024-05-15T23:25:32.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010980999999999999 2024-05-15T23:25:32.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027623+0000", 2024-05-15T23:25:32.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0890000000000002e-06 2024-05-15T23:25:32.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319918+0000", 2024-05-15T23:25:32.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29229501099999999 2024-05-15T23:25:32.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319923+0000", 2024-05-15T23:25:32.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3120000000000001e-06 2024-05-15T23:25:32.284 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.284 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.284 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.284 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329913 4.b 4.89604feb (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.284 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027509+0000", 2024-05-15T23:25:32.284 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.22233139299999, 2024-05-15T23:25:32.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29243947100000001, 2024-05-15T23:25:32.285 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.285 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.285 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.285 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.285 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.285 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.285 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329913 2024-05-15T23:25:32.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.285 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027509+0000", 2024-05-15T23:25:32.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027509+0000", 2024-05-15T23:25:32.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027509+0000", 2024-05-15T23:25:32.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0600000000000001e-07 2024-05-15T23:25:32.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027511+0000", 2024-05-15T23:25:32.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.345e-06 2024-05-15T23:25:32.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027513+0000", 2024-05-15T23:25:32.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7159999999999999e-06 2024-05-15T23:25:32.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027516+0000", 2024-05-15T23:25:32.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7e-06 2024-05-15T23:25:32.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027657+0000", 2024-05-15T23:25:32.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000140922 2024-05-15T23:25:32.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027666+0000", 2024-05-15T23:25:32.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9339999999999995e-06 2024-05-15T23:25:32.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319944+0000", 2024-05-15T23:25:32.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.292278223 2024-05-15T23:25:32.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319948+0000", 2024-05-15T23:25:32.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3250000000000001e-06 2024-05-15T23:25:32.289 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.289 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.289 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.289 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329964 4.b 4.e00aa51b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.289 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027519+0000", 2024-05-15T23:25:32.289 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.222321712, 2024-05-15T23:25:32.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29245270200000001, 2024-05-15T23:25:32.290 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.290 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.290 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.290 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.290 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.290 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.290 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329964 2024-05-15T23:25:32.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.290 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027519+0000", 2024-05-15T23:25:32.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027519+0000", 2024-05-15T23:25:32.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027519+0000", 2024-05-15T23:25:32.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1899999999999998e-07 2024-05-15T23:25:32.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027520+0000", 2024-05-15T23:25:32.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.328e-06 2024-05-15T23:25:32.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027523+0000", 2024-05-15T23:25:32.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.593e-06 2024-05-15T23:25:32.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027526+0000", 2024-05-15T23:25:32.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8380000000000002e-06 2024-05-15T23:25:32.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027702+0000", 2024-05-15T23:25:32.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017643300000000001 2024-05-15T23:25:32.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027708+0000", 2024-05-15T23:25:32.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2330000000000003e-06 2024-05-15T23:25:32.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319967+0000", 2024-05-15T23:25:32.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29225864299999998 2024-05-15T23:25:32.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319971+0000", 2024-05-15T23:25:32.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3150000000000002e-06 2024-05-15T23:25:32.294 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.294 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.294 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.294 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329975 4.b 4.564567eb (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.294 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027528+0000", 2024-05-15T23:25:32.294 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.22231206499998, 2024-05-15T23:25:32.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.292466526, 2024-05-15T23:25:32.294 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.294 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.294 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.295 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.295 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.295 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.295 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329975 2024-05-15T23:25:32.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.295 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027528+0000", 2024-05-15T23:25:32.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027528+0000", 2024-05-15T23:25:32.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027529+0000", 2024-05-15T23:25:32.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1899999999999998e-07 2024-05-15T23:25:32.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027530+0000", 2024-05-15T23:25:32.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.257e-06 2024-05-15T23:25:32.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027533+0000", 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9639999999999999e-06 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027537+0000", 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.743e-06 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027733+0000", 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019618200000000001 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027737+0000", 2024-05-15T23:25:32.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3939999999999998e-06 2024-05-15T23:25:32.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319991+0000", 2024-05-15T23:25:32.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29225351999999999 2024-05-15T23:25:32.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319995+0000", 2024-05-15T23:25:32.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1470000000000003e-06 2024-05-15T23:25:32.298 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.298 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.298 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.299 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330018 4.b 4.f747267b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.299 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027539+0000", 2024-05-15T23:25:32.299 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.22230100600001, 2024-05-15T23:25:32.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29248020699999999, 2024-05-15T23:25:32.299 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.299 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.299 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.299 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.299 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.299 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.299 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330018 2024-05-15T23:25:32.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.300 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027539+0000", 2024-05-15T23:25:32.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027539+0000", 2024-05-15T23:25:32.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027540+0000", 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.22e-07 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027541+0000", 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2309999999999999e-06 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027544+0000", 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.638e-06 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027546+0000", 2024-05-15T23:25:32.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7180000000000001e-06 2024-05-15T23:25:32.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027758+0000", 2024-05-15T23:25:32.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021151000000000001 2024-05-15T23:25:32.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027761+0000", 2024-05-15T23:25:32.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.698e-06 2024-05-15T23:25:32.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320016+0000", 2024-05-15T23:25:32.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29225404100000002 2024-05-15T23:25:32.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320020+0000", 2024-05-15T23:25:32.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0489999999999997e-06 2024-05-15T23:25:32.303 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.303 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.303 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.303 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330033 4.b 4.4f74750b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.303 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027549+0000", 2024-05-15T23:25:32.303 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.22229148299999, 2024-05-15T23:25:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29250598100000003, 2024-05-15T23:25:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330033 2024-05-15T23:25:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027549+0000", 2024-05-15T23:25:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027549+0000", 2024-05-15T23:25:32.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027549+0000", 2024-05-15T23:25:32.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0699999999999998e-07 2024-05-15T23:25:32.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027551+0000", 2024-05-15T23:25:32.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3710000000000001e-06 2024-05-15T23:25:32.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027553+0000", 2024-05-15T23:25:32.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.65e-06 2024-05-15T23:25:32.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027556+0000", 2024-05-15T23:25:32.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.892e-06 2024-05-15T23:25:32.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027782+0000", 2024-05-15T23:25:32.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00022592599999999999 2024-05-15T23:25:32.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027786+0000", 2024-05-15T23:25:32.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.501e-06 2024-05-15T23:25:32.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320048+0000", 2024-05-15T23:25:32.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29226260300000001 2024-05-15T23:25:32.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320055+0000", 2024-05-15T23:25:32.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7309999999999996e-06 2024-05-15T23:25:32.308 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.308 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.308 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.308 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330041 4.b 4.b7bfdfdb (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.308 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027559+0000", 2024-05-15T23:25:32.308 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.222281732, 2024-05-15T23:25:32.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.292511574, 2024-05-15T23:25:32.308 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.308 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.308 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.308 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.309 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.309 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.309 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330041 2024-05-15T23:25:32.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.309 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027559+0000", 2024-05-15T23:25:32.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027559+0000", 2024-05-15T23:25:32.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027559+0000", 2024-05-15T23:25:32.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4400000000000001e-07 2024-05-15T23:25:32.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027560+0000", 2024-05-15T23:25:32.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3030000000000001e-06 2024-05-15T23:25:32.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027563+0000", 2024-05-15T23:25:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7989999999999998e-06 2024-05-15T23:25:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027566+0000", 2024-05-15T23:25:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3900000000000002e-06 2024-05-15T23:25:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027804+0000", 2024-05-15T23:25:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00023764500000000001 2024-05-15T23:25:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027811+0000", 2024-05-15T23:25:32.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4350000000000002e-06 2024-05-15T23:25:32.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320066+0000", 2024-05-15T23:25:32.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29225564599999998 2024-05-15T23:25:32.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320070+0000", 2024-05-15T23:25:32.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0119999999999999e-06 2024-05-15T23:25:32.312 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.312 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.312 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.313 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330051 4.b 4.b4ae7c1b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.313 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027569+0000", 2024-05-15T23:25:32.313 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.22227122599998, 2024-05-15T23:25:32.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29253093299999999, 2024-05-15T23:25:32.313 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.313 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.313 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.313 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.313 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.313 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.313 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330051 2024-05-15T23:25:32.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.313 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027569+0000", 2024-05-15T23:25:32.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027569+0000", 2024-05-15T23:25:32.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027569+0000", 2024-05-15T23:25:32.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8200000000000001e-07 2024-05-15T23:25:32.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027571+0000", 2024-05-15T23:25:32.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7120000000000001e-06 2024-05-15T23:25:32.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027574+0000", 2024-05-15T23:25:32.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1389999999999999e-06 2024-05-15T23:25:32.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027577+0000", 2024-05-15T23:25:32.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6240000000000002e-06 2024-05-15T23:25:32.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027833+0000", 2024-05-15T23:25:32.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025610100000000001 2024-05-15T23:25:32.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027836+0000", 2024-05-15T23:25:32.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4070000000000002e-06 2024-05-15T23:25:32.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320095+0000", 2024-05-15T23:25:32.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29225901999999998 2024-05-15T23:25:32.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320100+0000", 2024-05-15T23:25:32.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6480000000000002e-06 2024-05-15T23:25:32.317 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.317 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.317 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.317 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330066 4.b 4.4e710d8b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.317 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027579+0000", 2024-05-15T23:25:32.317 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.22226086699999, 2024-05-15T23:25:32.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.292543579, 2024-05-15T23:25:32.318 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.318 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.318 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.318 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.318 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.318 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.318 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330066 2024-05-15T23:25:32.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.318 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027579+0000", 2024-05-15T23:25:32.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027579+0000", 2024-05-15T23:25:32.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027580+0000", 2024-05-15T23:25:32.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0499999999999999e-07 2024-05-15T23:25:32.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027584+0000", 2024-05-15T23:25:32.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.245e-06 2024-05-15T23:25:32.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027587+0000", 2024-05-15T23:25:32.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9579999999999999e-06 2024-05-15T23:25:32.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027590+0000", 2024-05-15T23:25:32.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.196e-06 2024-05-15T23:25:32.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027856+0000", 2024-05-15T23:25:32.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026541500000000002 2024-05-15T23:25:32.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027859+0000", 2024-05-15T23:25:32.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.659e-06 2024-05-15T23:25:32.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320119+0000", 2024-05-15T23:25:32.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29225978499999999 2024-05-15T23:25:32.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320123+0000", 2024-05-15T23:25:32.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0160000000000002e-06 2024-05-15T23:25:32.322 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.322 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.322 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.322 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330077 4.b 4.59c2b1db (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.322 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027594+0000", 2024-05-15T23:25:32.322 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.22224639699999, 2024-05-15T23:25:32.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29255331299999998, 2024-05-15T23:25:32.322 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.323 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.323 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.323 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.323 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.323 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.323 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330077 2024-05-15T23:25:32.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.323 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027594+0000", 2024-05-15T23:25:32.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027594+0000", 2024-05-15T23:25:32.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027594+0000", 2024-05-15T23:25:32.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7e-07 2024-05-15T23:25:32.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027596+0000", 2024-05-15T23:25:32.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9810000000000002e-06 2024-05-15T23:25:32.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027601+0000", 2024-05-15T23:25:32.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3370000000000001e-06 2024-05-15T23:25:32.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027606+0000", 2024-05-15T23:25:32.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8690000000000003e-06 2024-05-15T23:25:32.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027883+0000", 2024-05-15T23:25:32.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027723899999999998 2024-05-15T23:25:32.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027886+0000", 2024-05-15T23:25:32.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3019999999999999e-06 2024-05-15T23:25:32.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320143+0000", 2024-05-15T23:25:32.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.292256976 2024-05-15T23:25:32.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320147+0000", 2024-05-15T23:25:32.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1389999999999997e-06 2024-05-15T23:25:32.327 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.327 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.327 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.327 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330169 4.b 4.f7b3457b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:25:32.327 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027611+0000", 2024-05-15T23:25:32.327 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 385.22222959499999, 2024-05-15T23:25:32.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29255992400000003, 2024-05-15T23:25:32.327 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.327 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.327 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:25:32.327 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.328 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.328 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.328 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330169 2024-05-15T23:25:32.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.328 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027611+0000", 2024-05-15T23:25:32.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027611+0000", 2024-05-15T23:25:32.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027611+0000", 2024-05-15T23:25:32.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0200000000000002e-07 2024-05-15T23:25:32.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027614+0000", 2024-05-15T23:25:32.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4279999999999999e-06 2024-05-15T23:25:32.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027619+0000", 2024-05-15T23:25:32.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2009999999999998e-06 2024-05-15T23:25:32.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027624+0000", 2024-05-15T23:25:32.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2789999999999998e-06 2024-05-15T23:25:32.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027899+0000", 2024-05-15T23:25:32.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027479700000000002 2024-05-15T23:25:32.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:25:32.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027903+0000", 2024-05-15T23:25:32.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4809999999999998e-06 2024-05-15T23:25:32.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320167+0000", 2024-05-15T23:25:32.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29226418500000001 2024-05-15T23:25:32.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320171+0000", 2024-05-15T23:25:32.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9509999999999999e-06 2024-05-15T23:25:32.331 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.331 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.331 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.332 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330051 4.b 4:d83e752d:::benchmark_data_smithi090_104448_object156813:head [delete] snapc 0=[] RETRY=5 ondisk+retry+write+known_if_redirected e365)", 2024-05-15T23:25:32.332 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:10.016947+0000", 2024-05-15T23:25:32.332 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 382.232892886, 2024-05-15T23:25:32.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.124735232, 2024-05-15T23:25:32.332 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.332 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.332 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:32.332 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.332 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.332 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.332 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330051 2024-05-15T23:25:32.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.332 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016947+0000", 2024-05-15T23:25:32.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016947+0000", 2024-05-15T23:25:32.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016948+0000", 2024-05-15T23:25:32.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7800000000000002e-07 2024-05-15T23:25:32.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016954+0000", 2024-05-15T23:25:32.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3999999999999997e-06 2024-05-15T23:25:32.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016960+0000", 2024-05-15T23:25:32.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0340000000000002e-06 2024-05-15T23:25:32.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016965+0000", 2024-05-15T23:25:32.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8919999999999999e-06 2024-05-15T23:25:32.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.111838+0000", 2024-05-15T23:25:32.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.094872234 2024-05-15T23:25:32.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:32.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.111919+0000", 2024-05-15T23:25:32.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1933000000000003e-05 2024-05-15T23:25:32.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:25:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.111985+0000", 2024-05-15T23:25:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5178000000000005e-05 2024-05-15T23:25:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.116888+0000", 2024-05-15T23:25:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049034289999999999 2024-05-15T23:25:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141593+0000", 2024-05-15T23:25:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024704525000000001 2024-05-15T23:25:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141639+0000", 2024-05-15T23:25:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6659000000000002e-05 2024-05-15T23:25:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141683+0000", 2024-05-15T23:25:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3470000000000002e-05 2024-05-15T23:25:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.338 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330066 4.b 4:d1b08e72:::benchmark_data_smithi090_104448_object156828:head [delete] snapc 0=[] RETRY=5 ondisk+retry+write+known_if_redirected e365)", 2024-05-15T23:25:32.338 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:10.016971+0000", 2024-05-15T23:25:32.338 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 382.23286964200003, 2024-05-15T23:25:32.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12479114700000001, 2024-05-15T23:25:32.338 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.338 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.338 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:32.338 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.338 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.338 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.338 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330066 2024-05-15T23:25:32.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.338 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016971+0000", 2024-05-15T23:25:32.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016971+0000", 2024-05-15T23:25:32.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016971+0000", 2024-05-15T23:25:32.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2900000000000004e-07 2024-05-15T23:25:32.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016974+0000", 2024-05-15T23:25:32.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4210000000000002e-06 2024-05-15T23:25:32.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016985+0000", 2024-05-15T23:25:32.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1239e-05 2024-05-15T23:25:32.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016990+0000", 2024-05-15T23:25:32.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5690000000000004e-06 2024-05-15T23:25:32.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.116915+0000", 2024-05-15T23:25:32.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.099924916000000003 2024-05-15T23:25:32.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:32.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.117029+0000", 2024-05-15T23:25:32.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011380800000000001 2024-05-15T23:25:32.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:25:32.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.117138+0000", 2024-05-15T23:25:32.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000108933 2024-05-15T23:25:32.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:32.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.131447+0000", 2024-05-15T23:25:32.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014309240000000001 2024-05-15T23:25:32.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:32.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141691+0000", 2024-05-15T23:25:32.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010243835999999999 2024-05-15T23:25:32.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:32.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141730+0000", 2024-05-15T23:25:32.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8302e-05 2024-05-15T23:25:32.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141762+0000", 2024-05-15T23:25:32.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2354000000000002e-05 2024-05-15T23:25:32.343 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.343 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.343 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.343 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330077 4.b 4:db8d439a:::benchmark_data_smithi090_104448_object156839:head [delete] snapc 0=[] RETRY=5 ondisk+retry+write+known_if_redirected e365)", 2024-05-15T23:25:32.343 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:10.016997+0000", 2024-05-15T23:25:32.343 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 382.23284329299997, 2024-05-15T23:25:32.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.124849182, 2024-05-15T23:25:32.344 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.344 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.344 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:32.344 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.344 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.344 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.344 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330077 2024-05-15T23:25:32.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.344 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016997+0000", 2024-05-15T23:25:32.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016997+0000", 2024-05-15T23:25:32.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016998+0000", 2024-05-15T23:25:32.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9599999999999999e-07 2024-05-15T23:25:32.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017000+0000", 2024-05-15T23:25:32.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2749999999999998e-06 2024-05-15T23:25:32.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017005+0000", 2024-05-15T23:25:32.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7779999999999998e-06 2024-05-15T23:25:32.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017009+0000", 2024-05-15T23:25:32.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.357e-06 2024-05-15T23:25:32.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.120900+0000", 2024-05-15T23:25:32.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.103890867 2024-05-15T23:25:32.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:32.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.121033+0000", 2024-05-15T23:25:32.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000132874 2024-05-15T23:25:32.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:25:32.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.121166+0000", 2024-05-15T23:25:32.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013360100000000001 2024-05-15T23:25:32.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.131491+0000", 2024-05-15T23:25:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010325175000000001 2024-05-15T23:25:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141770+0000", 2024-05-15T23:25:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010278123 2024-05-15T23:25:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141807+0000", 2024-05-15T23:25:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7818000000000003e-05 2024-05-15T23:25:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141846+0000", 2024-05-15T23:25:32.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8818e-05 2024-05-15T23:25:32.349 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.349 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.349 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.349 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330169 4.b 4:dea2cdef:::benchmark_data_smithi090_104448_object156931:head [delete] snapc 0=[] RETRY=5 ondisk+retry+write+known_if_redirected e365)", 2024-05-15T23:25:32.349 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:10.017013+0000", 2024-05-15T23:25:32.349 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 382.23282722900001, 2024-05-15T23:25:32.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.125039171, 2024-05-15T23:25:32.349 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:32.349 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:32.349 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:32.350 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:32.350 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:25:32.350 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:25:32.350 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330169 2024-05-15T23:25:32.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.350 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:32.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:32.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017013+0000", 2024-05-15T23:25:32.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:32.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017013+0000", 2024-05-15T23:25:32.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:32.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:32.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017014+0000", 2024-05-15T23:25:32.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9299999999999999e-07 2024-05-15T23:25:32.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:32.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017015+0000", 2024-05-15T23:25:32.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8309999999999999e-06 2024-05-15T23:25:32.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:32.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017023+0000", 2024-05-15T23:25:32.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8169999999999999e-06 2024-05-15T23:25:32.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:32.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017035+0000", 2024-05-15T23:25:32.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1515000000000001e-05 2024-05-15T23:25:32.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:32.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.126480+0000", 2024-05-15T23:25:32.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.10944483200000001 2024-05-15T23:25:32.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:32.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.126563+0000", 2024-05-15T23:25:32.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3803999999999999e-05 2024-05-15T23:25:32.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:25:32.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.126662+0000", 2024-05-15T23:25:32.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9072000000000003e-05 2024-05-15T23:25:32.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:32.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141907+0000", 2024-05-15T23:25:32.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015245015000000001 2024-05-15T23:25:32.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:32.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141974+0000", 2024-05-15T23:25:32.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7088999999999994e-05 2024-05-15T23:25:32.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:32.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.142017+0000", 2024-05-15T23:25:32.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2011000000000001e-05 2024-05-15T23:25:32.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:32.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:32.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:32.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.142052+0000", 2024-05-15T23:25:32.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5791999999999999e-05 2024-05-15T23:25:32.355 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.355 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.355 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.355 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:32.355 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:32.355 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:25:32.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:32.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2335974705' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-05-15T23:25:32.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:32 smithi084 ceph-mon[39784]: Client client.admin marked osd.8 out, after it was down for 21 seconds 2024-05-15T23:25:32.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:32 smithi084 ceph-mon[39784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-15T23:25:32.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2335974705' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-05-15T23:25:32.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:32 smithi084 ceph-mon[39784]: osdmap e492: 12 total, 11 up, 8 in 2024-05-15T23:25:32.383 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2335974705' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-05-15T23:25:32.384 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:32 smithi090 ceph-mon[44434]: Client client.admin marked osd.8 out, after it was down for 21 seconds 2024-05-15T23:25:32.384 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:32 smithi090 ceph-mon[44434]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-15T23:25:32.384 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2335974705' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-05-15T23:25:32.384 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:32 smithi090 ceph-mon[44434]: osdmap e492: 12 total, 11 up, 8 in 2024-05-15T23:25:32.385 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:32.406 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:32.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:32.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:32.451 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 25 16 9813 9797 24.4887 0 - 0.00277684 2024-05-15T23:25:32.509 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:32.540 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:32.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2335974705' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-05-15T23:25:32.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:32 smithi088 ceph-mon[44699]: Client client.admin marked osd.8 out, after it was down for 21 seconds 2024-05-15T23:25:32.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:32 smithi088 ceph-mon[44699]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-15T23:25:32.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2335974705' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-05-15T23:25:32.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:32 smithi088 ceph-mon[44699]: osdmap e492: 12 total, 11 up, 8 in 2024-05-15T23:25:32.841 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:33.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:33 smithi084 ceph-mon[39784]: pgmap v1319: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:33.452 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 26 16 9813 9797 23.5469 0 - 0.00277684 2024-05-15T23:25:33.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:33 smithi090 ceph-mon[44434]: pgmap v1319: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:33.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:33 smithi088 ceph-mon[44699]: pgmap v1319: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 613 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1214/19598 objects degraded (6.195%) 2024-05-15T23:25:33.598 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T23:25:33.889 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:33.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:33.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:33.929 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.8 2024-05-15T23:25:33.929 INFO:tasks.thrashosds.thrasher:remote for osd 8 is ubuntu@smithi090.front.sepia.ceph.com 2024-05-15T23:25:33.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -n osd.8 --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-8 --journal-path /var/lib/ceph/osd/ceph-8/journal --op list-pgs 2024-05-15T23:25:33.993 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:25:34.019 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:34.223 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/osd.8/config 2024-05-15T23:25:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:34 smithi090 ceph-mon[44434]: osdmap e493: 12 total, 11 up, 8 in 2024-05-15T23:25:34.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:34 smithi084 ceph-mon[39784]: osdmap e493: 12 total, 11 up, 8 in 2024-05-15T23:25:34.452 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 27 15 11635 11620 26.8941 4.5575 0.00442767 0.0371792 2024-05-15T23:25:34.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:34 smithi088 ceph-mon[44699]: osdmap e493: 12 total, 11 up, 8 in 2024-05-15T23:25:34.767 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:25:34.770 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:34.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:34.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:25:34.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:25:34.916 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:35.128 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:25:35.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:35.133 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:25:35.234 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:35.271 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:35.348 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:35.352 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:35 smithi084 ceph-mon[39784]: pgmap v1321: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 643 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 3.8 MiB/s wr, 60 op/s; 1214/20570 objects degraded (5.902%) 2024-05-15T23:25:35.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:35.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:35.452 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 28 15 15880 15865 35.4076 265.312 0.00488447 0.0282404 2024-05-15T23:25:35.453 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:35.490 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:35.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:35 smithi090 ceph-mon[44434]: pgmap v1321: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 643 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 3.8 MiB/s wr, 60 op/s; 1214/20570 objects degraded (5.902%) 2024-05-15T23:25:35.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:35 smithi088 ceph-mon[44699]: pgmap v1321: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 643 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 3.8 MiB/s wr, 60 op/s; 1214/20570 objects degraded (5.902%) 2024-05-15T23:25:35.765 INFO:teuthology.orchestra.run.smithi090.stderr:Creating an OSD daemon form without an OSD FSID value 2024-05-15T23:25:35.933 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:35.945 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:35.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:35.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:36.054 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:36.090 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:36.119 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:36.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:36.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:36.126 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:25:36.224 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:36.276 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:36.359 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:36.363 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:36 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 1214/20570 objects degraded (5.902%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:25:36.363 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/817953916' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:25:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:36.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:36.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:36 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 1214/20570 objects degraded (5.902%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:25:36.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/817953916' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:25:36.452 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 29 15 19748 19733 42.5218 241.75 0.00467715 0.0235138 2024-05-15T23:25:36.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:36.504 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:36 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 1214/20570 objects degraded (5.902%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:25:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/817953916' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:25:37.060 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:37.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:37.167 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:37.207 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:37.234 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:37.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:37.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:37 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:25:37.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:37 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/817953916' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:25:37.304 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:37 smithi084 ceph-mon[39784]: osdmap e494: 12 total, 11 up, 8 in 2024-05-15T23:25:37.304 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:37 smithi084 ceph-mon[39784]: pgmap v1323: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 979 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 61 MiB/s wr, 975 op/s; 773/31308 objects degraded (2.469%); 4.6 MiB/s, 73 objects/s recovering 2024-05-15T23:25:37.304 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:37 smithi084 ceph-mon[39784]: osdmap e495: 12 total, 11 up, 8 in 2024-05-15T23:25:37.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:37.382 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:37.452 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 30 16 23490 23474 48.8967 233.812 0.00427648 0.0204495 2024-05-15T23:25:37.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:37 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:25:37.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:37 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/817953916' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:25:37.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:37 smithi090 ceph-mon[44434]: osdmap e494: 12 total, 11 up, 8 in 2024-05-15T23:25:37.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:37 smithi090 ceph-mon[44434]: pgmap v1323: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 979 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 61 MiB/s wr, 975 op/s; 773/31308 objects degraded (2.469%); 4.6 MiB/s, 73 objects/s recovering 2024-05-15T23:25:37.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:37 smithi090 ceph-mon[44434]: osdmap e495: 12 total, 11 up, 8 in 2024-05-15T23:25:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:37 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:25:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:37 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/817953916' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:25:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:37 smithi088 ceph-mon[44699]: osdmap e494: 12 total, 11 up, 8 in 2024-05-15T23:25:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:37 smithi088 ceph-mon[44699]: pgmap v1323: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 979 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 61 MiB/s wr, 975 op/s; 773/31308 objects degraded (2.469%); 4.6 MiB/s, 73 objects/s recovering 2024-05-15T23:25:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:37 smithi088 ceph-mon[44699]: osdmap e495: 12 total, 11 up, 8 in 2024-05-15T23:25:37.711 INFO:teuthology.orchestra.run.smithi090.stdout:6.1 2024-05-15T23:25:37.711 INFO:teuthology.orchestra.run.smithi090.stdout:6.9 2024-05-15T23:25:37.838 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:25:37.840 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:25:37.869 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:25:37.896 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:25:37.925 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:25:37.957 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:25:37.984 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:25:38.008 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:25:38.034 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:25:38.062 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:25:38.090 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:25:38.090 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:25:38.090 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:25:38.090 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:25:38.091 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:25:38.091 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:25:38.091 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:25:38.091 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:25:38.091 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:25:38.091 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-05-15T23:25:38.091 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:25:38.091 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:25:38.091 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:25:38.273 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:38.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:38 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4019315746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:25:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:38 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4019315746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:25:38.380 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:25:38.419 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:38 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4019315746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:25:38.422 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:38.452 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:25:38.452 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 31 15 27314 27299 55.03 239.062 0.00473907 0.0181692 2024-05-15T23:25:38.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:38.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:25:38.558 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:38.596 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:38.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -n osd.8 --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-8 --journal-path /var/lib/ceph/osd/ceph-8/journal --op export --pgid 6.9 --file /var/log/ceph/exp.6.9.8 2024-05-15T23:25:39.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.228 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/osd.8/config 2024-05-15T23:25:39.231 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:25:39.232 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:25:39.452 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 32 16 31102 31086 60.7058 236.688 0.00341738 0.0164713 2024-05-15T23:25:39.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.486 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.487 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.487 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.487 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.487 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.487 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:39 smithi084 ceph-mon[39784]: pgmap v1325: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 1.1 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s; 773/36886 objects degraded (2.096%); 4.6 MiB/s, 73 objects/s recovering 2024-05-15T23:25:39.531 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.531 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.531 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.531 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.531 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.531 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.531 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.531 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.531 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:39 smithi090 ceph-mon[44434]: pgmap v1325: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 1.1 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s; 773/36886 objects degraded (2.096%); 4.6 MiB/s, 73 objects/s recovering 2024-05-15T23:25:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:39.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:39 smithi088 ceph-mon[44699]: pgmap v1325: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 1.1 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s; 773/36886 objects degraded (2.096%); 4.6 MiB/s, 73 objects/s recovering 2024-05-15T23:25:39.953 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:39.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:40.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:40.090 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:40.295 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:40.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:40.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:40.314 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:25:40.400 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:40.438 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:40.452 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 33 16 34677 34661 65.6362 223.438 0.00499287 0.0152333 2024-05-15T23:25:40.471 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:40.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:40.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:40.577 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:40.611 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:25:40.612 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:25:40.626 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:40.650 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:40.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:40.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:40.713 INFO:teuthology.orchestra.run.smithi090.stderr:Creating an OSD daemon form without an OSD FSID value 2024-05-15T23:25:40.758 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:25:40.800 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:40.833 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:25:40.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:40.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:25:40.939 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:40.989 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:41.014 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:41.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:41.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:41.118 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:41.155 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:41.233 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:25:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:41 smithi090 ceph-mon[44434]: pgmap v1326: 17 pgs: 17 active+clean; 1.8 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 160 MiB/s wr, 2.56k op/s; 9.6 MiB/s, 153 objects/s recovering 2024-05-15T23:25:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:41 smithi090 ceph-mon[44434]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 773/36886 objects degraded (2.096%), 2 pgs degraded) 2024-05-15T23:25:41.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:41 smithi084 ceph-mon[39784]: pgmap v1326: 17 pgs: 17 active+clean; 1.8 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 160 MiB/s wr, 2.56k op/s; 9.6 MiB/s, 153 objects/s recovering 2024-05-15T23:25:41.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:41 smithi084 ceph-mon[39784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 773/36886 objects degraded (2.096%), 2 pgs degraded) 2024-05-15T23:25:41.453 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 34 15 37899 37884 69.6296 201.438 0.00357296 0.0143598 2024-05-15T23:25:41.479 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:41.511 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:41.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:41.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:41.530 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T23:25:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:41 smithi088 ceph-mon[44699]: pgmap v1326: 17 pgs: 17 active+clean; 1.8 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 160 MiB/s wr, 2.56k op/s; 9.6 MiB/s, 153 objects/s recovering 2024-05-15T23:25:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:41 smithi088 ceph-mon[44699]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 773/36886 objects degraded (2.096%), 2 pgs degraded) 2024-05-15T23:25:41.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:41.676 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:41.741 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:41.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:41.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:41.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:25:41.894 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:41.925 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:25:41.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:41.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:25:42.030 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:25:42.066 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:42.096 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:25:42.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:42.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:25:42.201 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:25:42.239 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:42.344 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:25:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:42.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:25:42.450 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:42.453 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 35 16 40933 40917 73.0557 189.562 0.00497897 0.013687 2024-05-15T23:25:42.489 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:42.494 INFO:teuthology.orchestra.run.smithi090.stderr:Exporting 6.9 info 6.9( v 484'602 (0'0,484'602] local-lis/les=477/478 n=602 ec=477/477 lis/c=477/477 les/c/f=478/478/0 sis=477) 2024-05-15T23:25:42.494 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:900a8860:::benchmark_data_smithi090_131278_object1821:head# 2024-05-15T23:25:42.497 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:900cf9ee:::benchmark_data_smithi090_131278_object7434:head# 2024-05-15T23:25:42.497 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:900fa84b:::benchmark_data_smithi090_131278_object3679:head# 2024-05-15T23:25:42.498 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:90266830:::benchmark_data_smithi090_131278_object947:head# 2024-05-15T23:25:42.499 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:902d356d:::benchmark_data_smithi090_131278_object979:head# 2024-05-15T23:25:42.499 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9044e792:::benchmark_data_smithi090_131278_object9667:head# 2024-05-15T23:25:42.499 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9055633e:::benchmark_data_smithi090_131278_object4029:head# 2024-05-15T23:25:42.500 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:905c2c2d:::benchmark_data_smithi090_131278_object5916:head# 2024-05-15T23:25:42.502 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:905fd676:::benchmark_data_smithi090_131278_object130:head# 2024-05-15T23:25:42.502 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:906f8484:::benchmark_data_smithi090_131278_object5186:head# 2024-05-15T23:25:42.502 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9074d1f2:::benchmark_data_smithi090_131278_object4971:head# 2024-05-15T23:25:42.502 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:90776185:::benchmark_data_smithi090_131278_object5453:head# 2024-05-15T23:25:42.503 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:907e00b3:::benchmark_data_smithi090_131278_object575:head# 2024-05-15T23:25:42.503 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9080cd7a:::benchmark_data_smithi090_131278_object7506:head# 2024-05-15T23:25:42.504 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9081e45c:::benchmark_data_smithi090_131278_object7685:head# 2024-05-15T23:25:42.504 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9086d21d:::benchmark_data_smithi090_131278_object957:head# 2024-05-15T23:25:42.504 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:908b457c:::benchmark_data_smithi090_131278_object8916:head# 2024-05-15T23:25:42.505 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:908e9390:::benchmark_data_smithi090_131278_object5727:head# 2024-05-15T23:25:42.506 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:90997555:::benchmark_data_smithi090_131278_object3013:head# 2024-05-15T23:25:42.506 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:909a6a12:::benchmark_data_smithi090_131278_object4786:head# 2024-05-15T23:25:42.507 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:90a61f86:::benchmark_data_smithi090_131278_object3349:head# 2024-05-15T23:25:42.508 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:90a9a546:::benchmark_data_smithi090_131278_object4415:head# 2024-05-15T23:25:42.508 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:90b130ec:::benchmark_data_smithi090_131278_object4552:head# 2024-05-15T23:25:42.508 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:90c23108:::benchmark_data_smithi090_131278_object1429:head# 2024-05-15T23:25:42.511 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:90d6fb50:::benchmark_data_smithi090_131278_object4051:head# 2024-05-15T23:25:42.511 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:90e046b4:::benchmark_data_smithi090_131278_object5414:head# 2024-05-15T23:25:42.511 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:90e6a12b:::benchmark_data_smithi090_131278_object1636:head# 2024-05-15T23:25:42.511 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:90ec37fd:::benchmark_data_smithi090_131278_object5131:head# 2024-05-15T23:25:42.511 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:90ee0c58:::benchmark_data_smithi090_131278_object224:head# 2024-05-15T23:25:42.512 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:90f7881d:::benchmark_data_smithi090_131278_object8233:head# 2024-05-15T23:25:42.512 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:910001ca:::benchmark_data_smithi090_131278_object4537:head# 2024-05-15T23:25:42.514 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91152a18:::benchmark_data_smithi090_131278_object4850:head# 2024-05-15T23:25:42.515 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9118abda:::benchmark_data_smithi090_131278_object1310:head# 2024-05-15T23:25:42.515 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9128cb6b:::benchmark_data_smithi090_131278_object7382:head# 2024-05-15T23:25:42.515 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:42.515 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:912cdb75:::benchmark_data_smithi090_131278_object2311:head# 2024-05-15T23:25:42.517 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:913af216:::benchmark_data_smithi090_131278_object3819:head# 2024-05-15T23:25:42.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:42.519 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:42.519 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:913cc6fa:::benchmark_data_smithi090_131278_object2789:head# 2024-05-15T23:25:42.520 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9140c596:::benchmark_data_smithi090_131278_object746:head# 2024-05-15T23:25:42.520 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9144cf37:::benchmark_data_smithi090_131278_object8394:head# 2024-05-15T23:25:42.520 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:915243fc:::benchmark_data_smithi090_131278_object2946:head# 2024-05-15T23:25:42.520 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91656e07:::benchmark_data_smithi090_131278_object6503:head# 2024-05-15T23:25:42.521 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9167c6f1:::benchmark_data_smithi090_131278_object4014:head# 2024-05-15T23:25:42.522 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:916a9cb3:::benchmark_data_smithi090_131278_object6531:head# 2024-05-15T23:25:42.522 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9174043b:::benchmark_data_smithi090_131278_object7054:head# 2024-05-15T23:25:42.522 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:917a6a16:::benchmark_data_smithi090_131278_object5628:head# 2024-05-15T23:25:42.523 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:917aa1ee:::benchmark_data_smithi090_131278_object3003:head# 2024-05-15T23:25:42.525 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9181e19b:::benchmark_data_smithi090_131278_object8384:head# 2024-05-15T23:25:42.525 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9183ccc5:::benchmark_data_smithi090_131278_object9606:head# 2024-05-15T23:25:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9194bccf:::benchmark_data_smithi090_131278_object3184:head# 2024-05-15T23:25:42.527 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91aafd64:::benchmark_data_smithi090_131278_object5202:head# 2024-05-15T23:25:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91b22b33:::benchmark_data_smithi090_131278_object91:head# 2024-05-15T23:25:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91b5188e:::benchmark_data_smithi090_131278_object5173:head# 2024-05-15T23:25:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91bea12d:::benchmark_data_smithi090_131278_object7957:head# 2024-05-15T23:25:42.533 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91c10862:::benchmark_data_smithi090_131278_object2214:head# 2024-05-15T23:25:42.533 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91c437bc:::benchmark_data_smithi090_131278_object2093:head# 2024-05-15T23:25:42.533 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91cb2873:::benchmark_data_smithi090_131278_object659:head# 2024-05-15T23:25:42.535 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91ce2faa:::benchmark_data_smithi090_131278_object8431:head# 2024-05-15T23:25:42.535 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91ce345c:::benchmark_data_smithi090_131278_object5340:head# 2024-05-15T23:25:42.536 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91d81e23:::benchmark_data_smithi090_131278_object4136:head# 2024-05-15T23:25:42.536 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91de77b3:::benchmark_data_smithi090_131278_object779:head# 2024-05-15T23:25:42.537 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91e2113f:::benchmark_data_smithi090_131278_object264:head# 2024-05-15T23:25:42.537 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91e80464:::benchmark_data_smithi090_131278_object8227:head# 2024-05-15T23:25:42.537 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91ed3a68:::benchmark_data_smithi090_131278_object1520:head# 2024-05-15T23:25:42.538 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91f18ccb:::benchmark_data_smithi090_131278_object5347:head# 2024-05-15T23:25:42.540 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91fd3e1f:::benchmark_data_smithi090_131278_object9680:head# 2024-05-15T23:25:42.540 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:91ff00cb:::benchmark_data_smithi090_131278_object4514:head# 2024-05-15T23:25:42.540 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:920aafa3:::benchmark_data_smithi090_131278_object6651:head# 2024-05-15T23:25:42.540 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:920d03d9:::benchmark_data_smithi090_131278_object6911:head# 2024-05-15T23:25:42.543 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:921005e8:::benchmark_data_smithi090_131278_object8783:head# 2024-05-15T23:25:42.543 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9215a8a9:::benchmark_data_smithi090_131278_object1757:head# 2024-05-15T23:25:42.547 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9225e3d6:::benchmark_data_smithi090_131278_object9300:head# 2024-05-15T23:25:42.547 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:923e52fe:::benchmark_data_smithi090_131278_object5615:head# 2024-05-15T23:25:42.547 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92412d69:::benchmark_data_smithi090_131278_object2882:head# 2024-05-15T23:25:42.547 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92495281:::benchmark_data_smithi090_131278_object4113:head# 2024-05-15T23:25:42.548 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9249f851:::benchmark_data_smithi090_131278_object1269:head# 2024-05-15T23:25:42.549 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:924d6a0f:::benchmark_data_smithi090_131278_object8563:head# 2024-05-15T23:25:42.550 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:924e8b3d:::benchmark_data_smithi090_131278_object8871:head# 2024-05-15T23:25:42.552 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:925102b5:::benchmark_data_smithi090_131278_object9349:head# 2024-05-15T23:25:42.554 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92539e92:::benchmark_data_smithi090_131278_object6778:head# 2024-05-15T23:25:42.555 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9260f3ac:::benchmark_data_smithi090_131278_object3548:head# 2024-05-15T23:25:42.555 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:926573d2:::benchmark_data_smithi090_131278_object1817:head# 2024-05-15T23:25:42.556 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:926c4fdc:::benchmark_data_smithi090_131278_object9066:head# 2024-05-15T23:25:42.556 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:926f5d34:::benchmark_data_smithi090_131278_object3095:head# 2024-05-15T23:25:42.557 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9274824d:::benchmark_data_smithi090_131278_object4690:head# 2024-05-15T23:25:42.557 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92767b63:::benchmark_data_smithi090_131278_object3831:head# 2024-05-15T23:25:42.558 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:927ce598:::benchmark_data_smithi090_131278_object3015:head# 2024-05-15T23:25:42.559 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92843974:::benchmark_data_smithi090_131278_object9548:head# 2024-05-15T23:25:42.560 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:928ceb53:::benchmark_data_smithi090_131278_object6015:head# 2024-05-15T23:25:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9299c45a:::benchmark_data_smithi090_131278_object2888:head# 2024-05-15T23:25:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:929a3323:::benchmark_data_smithi090_131278_object499:head# 2024-05-15T23:25:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:929c54ec:::benchmark_data_smithi090_131278_object9329:head# 2024-05-15T23:25:42.568 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92a0122d:::benchmark_data_smithi090_131278_object1379:head# 2024-05-15T23:25:42.568 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92a2db3a:::benchmark_data_smithi090_131278_object6034:head# 2024-05-15T23:25:42.568 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92a37c7d:::benchmark_data_smithi090_131278_object4030:head# 2024-05-15T23:25:42.569 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92a6b734:::benchmark_data_smithi090_131278_object1707:head# 2024-05-15T23:25:42.569 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92ab8d4d:::benchmark_data_smithi090_131278_object3566:head# 2024-05-15T23:25:42.570 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92abac80:::benchmark_data_smithi090_131278_object5443:head# 2024-05-15T23:25:42.570 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92aea64d:::benchmark_data_smithi090_131278_object2061:head# 2024-05-15T23:25:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92b6ec25:::benchmark_data_smithi090_131278_object3581:head# 2024-05-15T23:25:42.571 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92bb1250:::benchmark_data_smithi090_131278_object7190:head# 2024-05-15T23:25:42.572 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92c0bf0b:::benchmark_data_smithi090_131278_object5639:head# 2024-05-15T23:25:42.573 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92c3bf64:::benchmark_data_smithi090_131278_object2666:head# 2024-05-15T23:25:42.574 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92c5174e:::benchmark_data_smithi090_131278_object8110:head# 2024-05-15T23:25:42.575 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92c941e8:::benchmark_data_smithi090_131278_object5546:head# 2024-05-15T23:25:42.575 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92cc3736:::benchmark_data_smithi090_131278_object7304:head# 2024-05-15T23:25:42.575 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92d2ba4d:::benchmark_data_smithi090_131278_object8420:head# 2024-05-15T23:25:42.577 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92d7442d:::benchmark_data_smithi090_131278_object2692:head# 2024-05-15T23:25:42.578 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92d85552:::benchmark_data_smithi090_131278_object5284:head# 2024-05-15T23:25:42.578 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92dd538e:::benchmark_data_smithi090_131278_object5480:head# 2024-05-15T23:25:42.579 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92de3097:::benchmark_data_smithi090_131278_object8388:head# 2024-05-15T23:25:42.579 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92e02dcb:::benchmark_data_smithi090_131278_object7260:head# 2024-05-15T23:25:42.579 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92e20416:::benchmark_data_smithi090_131278_object7367:head# 2024-05-15T23:25:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92f10b99:::benchmark_data_smithi090_131278_object4660:head# 2024-05-15T23:25:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92f5d7a7:::benchmark_data_smithi090_131278_object6357:head# 2024-05-15T23:25:42.585 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:92f9f3ab:::benchmark_data_smithi090_131278_object9610:head# 2024-05-15T23:25:42.585 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:930ff82f:::benchmark_data_smithi090_131278_object459:head# 2024-05-15T23:25:42.587 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9310d8e1:::benchmark_data_smithi090_131278_object3393:head# 2024-05-15T23:25:42.587 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:931470cf:::benchmark_data_smithi090_131278_object5043:head# 2024-05-15T23:25:42.588 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:931f1d9b:::benchmark_data_smithi090_131278_object8527:head# 2024-05-15T23:25:42.588 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:932912b6:::benchmark_data_smithi090_131278_object5991:head# 2024-05-15T23:25:42.589 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93373f4c:::benchmark_data_smithi090_131278_object2872:head# 2024-05-15T23:25:42.591 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:934a88f5:::benchmark_data_smithi090_131278_object6986:head# 2024-05-15T23:25:42.592 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:934c53cd:::benchmark_data_smithi090_131278_object3638:head# 2024-05-15T23:25:42.592 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93562d91:::benchmark_data_smithi090_131278_object9123:head# 2024-05-15T23:25:42.594 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9357fb1c:::benchmark_data_smithi090_131278_object6599:head# 2024-05-15T23:25:42.595 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:935a081f:::benchmark_data_smithi090_131278_object5572:head# 2024-05-15T23:25:42.595 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93617c94:::benchmark_data_smithi090_131278_object2908:head# 2024-05-15T23:25:42.597 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93705e0b:::benchmark_data_smithi090_131278_object1888:head# 2024-05-15T23:25:42.598 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:937995a2:::benchmark_data_smithi090_131278_object2597:head# 2024-05-15T23:25:42.598 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:937afca6:::benchmark_data_smithi090_131278_object7002:head# 2024-05-15T23:25:42.599 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9385fa7d:::benchmark_data_smithi090_131278_object5562:head# 2024-05-15T23:25:42.599 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:938c342a:::benchmark_data_smithi090_131278_object7172:head# 2024-05-15T23:25:42.600 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93a114de:::benchmark_data_smithi090_131278_object5112:head# 2024-05-15T23:25:42.603 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93b10e8c:::benchmark_data_smithi090_131278_object2400:head# 2024-05-15T23:25:42.603 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93b39d95:::benchmark_data_smithi090_131278_object2283:head# 2024-05-15T23:25:42.604 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93b91e2b:::benchmark_data_smithi090_131278_object6069:head# 2024-05-15T23:25:42.604 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93bde6c3:::benchmark_data_smithi090_131278_object3287:head# 2024-05-15T23:25:42.605 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93be36c0:::benchmark_data_smithi090_131278_object8960:head# 2024-05-15T23:25:42.605 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93c75023:::benchmark_data_smithi090_131278_object216:head# 2024-05-15T23:25:42.606 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93d48def:::benchmark_data_smithi090_131278_object1365:head# 2024-05-15T23:25:42.608 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93d5ae4b:::benchmark_data_smithi090_131278_object8463:head# 2024-05-15T23:25:42.608 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93dc08d5:::benchmark_data_smithi090_131278_object1141:head# 2024-05-15T23:25:42.610 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93e9402b:::benchmark_data_smithi090_131278_object1846:head# 2024-05-15T23:25:42.612 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93ea9ef7:::benchmark_data_smithi090_131278_object9510:head# 2024-05-15T23:25:42.613 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93f00e7a:::benchmark_data_smithi090_131278_object198:head# 2024-05-15T23:25:42.614 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:93fb60c0:::benchmark_data_smithi090_131278_object7118:head# 2024-05-15T23:25:42.617 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:940bce32:::benchmark_data_smithi090_131278_object4932:head# 2024-05-15T23:25:42.617 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9411779d:::benchmark_data_smithi090_131278_object2355:head# 2024-05-15T23:25:42.617 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94178e55:::benchmark_data_smithi090_131278_object2674:head# 2024-05-15T23:25:42.618 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:941c4580:::benchmark_data_smithi090_131278_object5027:head# 2024-05-15T23:25:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:941dbf10:::benchmark_data_smithi090_131278_object2748:head# 2024-05-15T23:25:42.620 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9420d63c:::benchmark_data_smithi090_131278_object1172:head# 2024-05-15T23:25:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94216fd7:::benchmark_data_smithi090_131278_object9294:head# 2024-05-15T23:25:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9436f45a:::benchmark_data_smithi090_131278_object7935:head# 2024-05-15T23:25:42.622 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94380a75:::benchmark_data_smithi090_131278_object7349:head# 2024-05-15T23:25:42.622 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9448bb75:::benchmark_data_smithi090_131278_object5214:head# 2024-05-15T23:25:42.622 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:944a240b:::benchmark_data_smithi090_131278_object5486:head# 2024-05-15T23:25:42.624 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94595d1e:::benchmark_data_smithi090_131278_object3463:head# 2024-05-15T23:25:42.624 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:945eb9a9:::benchmark_data_smithi090_131278_object808:head# 2024-05-15T23:25:42.627 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:946404cd:::benchmark_data_smithi090_131278_object2905:head# 2024-05-15T23:25:42.627 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:946a9386:::benchmark_data_smithi090_131278_object4509:head# 2024-05-15T23:25:42.628 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94728d39:::benchmark_data_smithi090_131278_object427:head# 2024-05-15T23:25:42.628 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94729876:::benchmark_data_smithi090_131278_object567:head# 2024-05-15T23:25:42.628 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:947bf817:::benchmark_data_smithi090_131278_object4599:head# 2024-05-15T23:25:42.629 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:947e2348:::benchmark_data_smithi090_131278_object8078:head# 2024-05-15T23:25:42.630 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94830482:::benchmark_data_smithi090_131278_object7618:head# 2024-05-15T23:25:42.631 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:948bcd5f:::benchmark_data_smithi090_131278_object610:head# 2024-05-15T23:25:42.631 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:948d5068:::benchmark_data_smithi090_131278_object4408:head# 2024-05-15T23:25:42.633 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:949e9b9b:::benchmark_data_smithi090_131278_object6709:head# 2024-05-15T23:25:42.633 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:949ede60:::benchmark_data_smithi090_131278_object3771:head# 2024-05-15T23:25:42.634 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94a06147:::benchmark_data_smithi090_131278_object8727:head# 2024-05-15T23:25:42.634 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94a0dfba:::benchmark_data_smithi090_131278_object4727:head# 2024-05-15T23:25:42.635 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94a37c66:::benchmark_data_smithi090_131278_object7683:head# 2024-05-15T23:25:42.635 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94a71d2d:::benchmark_data_smithi090_131278_object7867:head# 2024-05-15T23:25:42.636 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94a7dc86:::benchmark_data_smithi090_131278_object1033:head# 2024-05-15T23:25:42.636 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94a90cac:::benchmark_data_smithi090_131278_object7894:head# 2024-05-15T23:25:42.637 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94b36084:::benchmark_data_smithi090_131278_object2639:head# 2024-05-15T23:25:42.640 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94cab279:::benchmark_data_smithi090_131278_object4201:head# 2024-05-15T23:25:42.642 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94caf73e:::benchmark_data_smithi090_131278_object6520:head# 2024-05-15T23:25:42.644 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94d25761:::benchmark_data_smithi090_131278_object6692:head# 2024-05-15T23:25:42.644 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94d2bf2c:::benchmark_data_smithi090_131278_object8547:head# 2024-05-15T23:25:42.645 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94d87b74:::benchmark_data_smithi090_131278_object6346:head# 2024-05-15T23:25:42.645 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94dd8525:::benchmark_data_smithi090_131278_object4905:head# 2024-05-15T23:25:42.645 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:94def3b7:::benchmark_data_smithi090_131278_object7982:head# 2024-05-15T23:25:42.646 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95100197:::benchmark_data_smithi090_131278_object4002:head# 2024-05-15T23:25:42.646 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9512b18a:::benchmark_data_smithi090_131278_object9243:head# 2024-05-15T23:25:42.649 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95133a9f:::benchmark_data_smithi090_131278_object398:head# 2024-05-15T23:25:42.651 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9524aa99:::benchmark_data_smithi090_131278_object8042:head# 2024-05-15T23:25:42.651 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9535c58b:::benchmark_data_smithi090_131278_object3460:head# 2024-05-15T23:25:42.653 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95485594:::benchmark_data_smithi090_131278_object7356:head# 2024-05-15T23:25:42.654 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:954e08ab:::benchmark_data_smithi090_131278_object6963:head# 2024-05-15T23:25:42.654 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:955b60dd:::benchmark_data_smithi090_131278_object5739:head# 2024-05-15T23:25:42.655 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95657cff:::benchmark_data_smithi090_131278_object6750:head# 2024-05-15T23:25:42.656 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95699807:::benchmark_data_smithi090_131278_object6783:head# 2024-05-15T23:25:42.656 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:957bae68:::benchmark_data_smithi090_131278_object565:head# 2024-05-15T23:25:42.659 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9580cf5a:::benchmark_data_smithi090_131278_object4076:head# 2024-05-15T23:25:42.659 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95912ce2:::benchmark_data_smithi090_131278_object5864:head# 2024-05-15T23:25:42.660 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:959e77bb:::benchmark_data_smithi090_131278_object6533:head# 2024-05-15T23:25:42.660 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95b66e8d:::benchmark_data_smithi090_131278_object5715:head# 2024-05-15T23:25:42.660 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95b681bb:::benchmark_data_smithi090_131278_object6609:head# 2024-05-15T23:25:42.660 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95bb2622:::benchmark_data_smithi090_131278_object1913:head# 2024-05-15T23:25:42.663 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95bdcf61:::benchmark_data_smithi090_131278_object9144:head# 2024-05-15T23:25:42.663 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95be5218:::benchmark_data_smithi090_131278_object4890:head# 2024-05-15T23:25:42.663 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95c89615:::benchmark_data_smithi090_131278_object9325:head# 2024-05-15T23:25:42.666 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95cf6e84:::benchmark_data_smithi090_131278_object2138:head# 2024-05-15T23:25:42.666 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95d223be:::benchmark_data_smithi090_131278_object1538:head# 2024-05-15T23:25:42.667 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95d343fe:::benchmark_data_smithi090_131278_object6171:head# 2024-05-15T23:25:42.667 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:42.668 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95d53b90:::benchmark_data_smithi090_131278_object8855:head# 2024-05-15T23:25:42.668 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95d6d24d:::benchmark_data_smithi090_131278_object4650:head# 2024-05-15T23:25:42.671 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95d98694:::benchmark_data_smithi090_131278_object614:head# 2024-05-15T23:25:42.672 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95e78d34:::benchmark_data_smithi090_131278_object8841:head# 2024-05-15T23:25:42.672 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95e9dd8d:::benchmark_data_smithi090_131278_object3576:head# 2024-05-15T23:25:42.673 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95ea9beb:::benchmark_data_smithi090_131278_object8363:head# 2024-05-15T23:25:42.673 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95eafad2:::benchmark_data_smithi090_131278_object7812:head# 2024-05-15T23:25:42.676 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95f08883:::benchmark_data_smithi090_131278_object9172:head# 2024-05-15T23:25:42.676 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95f51205:::benchmark_data_smithi090_131278_object80:head# 2024-05-15T23:25:42.677 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95f92a9c:::benchmark_data_smithi090_131278_object8335:head# 2024-05-15T23:25:42.678 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95fccb1a:::benchmark_data_smithi090_131278_object2487:head# 2024-05-15T23:25:42.680 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:95ff2008:::benchmark_data_smithi090_131278_object4603:head# 2024-05-15T23:25:42.681 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9605858a:::benchmark_data_smithi090_131278_object1009:head# 2024-05-15T23:25:42.681 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96144309:::benchmark_data_smithi090_131278_object8644:head# 2024-05-15T23:25:42.682 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:961ae428:::benchmark_data_smithi090_131278_object932:head# 2024-05-15T23:25:42.682 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:961c1b01:::benchmark_data_smithi090_131278_object237:head# 2024-05-15T23:25:42.683 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9629e108:::benchmark_data_smithi090_131278_object2212:head# 2024-05-15T23:25:42.685 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:962e67cc:::benchmark_data_smithi090_131278_object4405:head# 2024-05-15T23:25:42.685 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:962f561a:::benchmark_data_smithi090_131278_object9408:head# 2024-05-15T23:25:42.686 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:963af2f9:::benchmark_data_smithi090_131278_object5084:head# 2024-05-15T23:25:42.686 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96413d1b:::benchmark_data_smithi090_131278_object4038:head# 2024-05-15T23:25:42.687 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9648a8a8:::benchmark_data_smithi090_131278_object7616:head# 2024-05-15T23:25:42.687 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:964a98c7:::benchmark_data_smithi090_131278_object5099:head# 2024-05-15T23:25:42.688 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:964c19b2:::benchmark_data_smithi090_131278_object451:head# 2024-05-15T23:25:42.689 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:964fc41e:::benchmark_data_smithi090_131278_object5748:head# 2024-05-15T23:25:42.689 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9652b361:::benchmark_data_smithi090_131278_object6771:head# 2024-05-15T23:25:42.690 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:965dab2e:::benchmark_data_smithi090_131278_object2711:head# 2024-05-15T23:25:42.690 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96610a16:::benchmark_data_smithi090_131278_object5364:head# 2024-05-15T23:25:42.691 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9661fb48:::benchmark_data_smithi090_131278_object7814:head# 2024-05-15T23:25:42.691 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:966c1829:::benchmark_data_smithi090_131278_object3777:head# 2024-05-15T23:25:42.692 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9670a470:::benchmark_data_smithi090_131278_object6195:head# 2024-05-15T23:25:42.692 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9673a44b:::benchmark_data_smithi090_131278_object6868:head# 2024-05-15T23:25:42.693 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:967bb389:::benchmark_data_smithi090_131278_object8214:head# 2024-05-15T23:25:42.693 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96882dd2:::benchmark_data_smithi090_131278_object5853:head# 2024-05-15T23:25:42.693 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:968a4470:::benchmark_data_smithi090_131278_object2347:head# 2024-05-15T23:25:42.694 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9692ff1e:::benchmark_data_smithi090_131278_object2828:head# 2024-05-15T23:25:42.695 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96a313a1:::benchmark_data_smithi090_131278_object4170:head# 2024-05-15T23:25:42.696 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96a554c2:::benchmark_data_smithi090_131278_object2603:head# 2024-05-15T23:25:42.696 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96a780fa:::benchmark_data_smithi090_131278_object5735:head# 2024-05-15T23:25:42.697 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96af4ab0:::benchmark_data_smithi090_131278_object1836:head# 2024-05-15T23:25:42.697 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96b1020a:::benchmark_data_smithi090_131278_object8770:head# 2024-05-15T23:25:42.697 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96bd8879:::benchmark_data_smithi090_131278_object6768:head# 2024-05-15T23:25:42.697 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96c139fb:::benchmark_data_smithi090_131278_object8640:head# 2024-05-15T23:25:42.698 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96c5f185:::benchmark_data_smithi090_131278_object6965:head# 2024-05-15T23:25:42.699 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96dc7f40:::benchmark_data_smithi090_131278_object2921:head# 2024-05-15T23:25:42.699 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:42.702 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96de17c8:::benchmark_data_smithi090_131278_object4055:head# 2024-05-15T23:25:42.702 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96e17973:::benchmark_data_smithi090_131278_object7498:head# 2024-05-15T23:25:42.703 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96e639b4:::benchmark_data_smithi090_131278_object2930:head# 2024-05-15T23:25:42.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:42.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:42.706 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96e668b5:::benchmark_data_smithi090_131278_object6826:head# 2024-05-15T23:25:42.706 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96eaeaff:::benchmark_data_smithi090_131278_object4332:head# 2024-05-15T23:25:42.707 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96f3fcfa:::benchmark_data_smithi090_131278_object2739:head# 2024-05-15T23:25:42.708 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96f5df00:::benchmark_data_smithi090_131278_object2665:head# 2024-05-15T23:25:42.708 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96f70686:::benchmark_data_smithi090_131278_object7771:head# 2024-05-15T23:25:42.709 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:96fe1032:::benchmark_data_smithi090_131278_object3477:head# 2024-05-15T23:25:42.712 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9709d4a1:::benchmark_data_smithi090_131278_object1868:head# 2024-05-15T23:25:42.713 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9710f278:::benchmark_data_smithi090_131278_object4747:head# 2024-05-15T23:25:42.714 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9711f604:::benchmark_data_smithi090_131278_object9377:head# 2024-05-15T23:25:42.715 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97182232:::benchmark_data_smithi090_131278_object9401:head# 2024-05-15T23:25:42.717 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9731637d:::benchmark_data_smithi090_131278_object1547:head# 2024-05-15T23:25:42.718 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9731ca9c:::benchmark_data_smithi090_131278_object6260:head# 2024-05-15T23:25:42.718 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97326c9b:::benchmark_data_smithi090_131278_object6918:head# 2024-05-15T23:25:42.719 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:973b9c04:::benchmark_data_smithi090_131278_object1491:head# 2024-05-15T23:25:42.719 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97430b30:::benchmark_data_smithi090_131278_object9612:head# 2024-05-15T23:25:42.719 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97507a79:::benchmark_data_smithi090_131278_object5769:head# 2024-05-15T23:25:42.720 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:975e87b5:::benchmark_data_smithi090_131278_object7488:head# 2024-05-15T23:25:42.721 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9763a040:::benchmark_data_smithi090_131278_object47:head# 2024-05-15T23:25:42.721 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97789906:::benchmark_data_smithi090_131278_object5571:head# 2024-05-15T23:25:42.721 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9780d85b:::benchmark_data_smithi090_131278_object998:head# 2024-05-15T23:25:42.722 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9785e778:::benchmark_data_smithi090_131278_object4809:head# 2024-05-15T23:25:42.725 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:979505ab:::benchmark_data_smithi090_131278_object5481:head# 2024-05-15T23:25:42.726 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:979ef2af:::benchmark_data_smithi090_131278_object4275:head# 2024-05-15T23:25:42.727 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97a3796e:::benchmark_data_smithi090_131278_object7548:head# 2024-05-15T23:25:42.728 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97b57061:::benchmark_data_smithi090_131278_object7269:head# 2024-05-15T23:25:42.729 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97bd84e7:::benchmark_data_smithi090_131278_object2565:head# 2024-05-15T23:25:42.730 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97c6d3d2:::benchmark_data_smithi090_131278_object6869:head# 2024-05-15T23:25:42.730 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97c8ed08:::benchmark_data_smithi090_131278_object3574:head# 2024-05-15T23:25:42.731 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97c9937f:::benchmark_data_smithi090_131278_object4214:head# 2024-05-15T23:25:42.731 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97d29d33:::benchmark_data_smithi090_131278_object3850:head# 2024-05-15T23:25:42.731 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97d458b4:::benchmark_data_smithi090_131278_object3369:head# 2024-05-15T23:25:42.732 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97db4286:::benchmark_data_smithi090_131278_object4226:head# 2024-05-15T23:25:42.732 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97df1b60:::benchmark_data_smithi090_131278_object4287:head# 2024-05-15T23:25:42.733 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97ec5df4:::benchmark_data_smithi090_131278_object2757:head# 2024-05-15T23:25:42.733 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97eccac4:::benchmark_data_smithi090_131278_object9673:head# 2024-05-15T23:25:42.734 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97ef60fd:::benchmark_data_smithi090_131278_object4967:head# 2024-05-15T23:25:42.734 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97f048cc:::benchmark_data_smithi090_131278_object1440:head# 2024-05-15T23:25:42.735 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97f0d4e1:::benchmark_data_smithi090_131278_object1249:head# 2024-05-15T23:25:42.735 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97f36121:::benchmark_data_smithi090_131278_object5802:head# 2024-05-15T23:25:42.736 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:97fc8009:::benchmark_data_smithi090_131278_object6485:head# 2024-05-15T23:25:42.736 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:980632c6:::benchmark_data_smithi090_131278_object6004:head# 2024-05-15T23:25:42.736 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:980a69b8:::benchmark_data_smithi090_131278_object1865:head# 2024-05-15T23:25:42.739 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98182a1a:::benchmark_data_smithi090_131278_object9392:head# 2024-05-15T23:25:42.739 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:981a65a5:::benchmark_data_smithi090_131278_object1639:head# 2024-05-15T23:25:42.739 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:981ec54e:::benchmark_data_smithi090_131278_object3290:head# 2024-05-15T23:25:42.739 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:982cc58f:::benchmark_data_smithi090_131278_object1097:head# 2024-05-15T23:25:42.740 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98409a2d:::benchmark_data_smithi090_131278_object5424:head# 2024-05-15T23:25:42.740 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9846f5d9:::benchmark_data_smithi090_131278_object5967:head# 2024-05-15T23:25:42.741 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98484084:::benchmark_data_smithi090_131278_object5393:head# 2024-05-15T23:25:42.741 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98492c2a:::benchmark_data_smithi090_131278_object741:head# 2024-05-15T23:25:42.744 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9853cffc:::benchmark_data_smithi090_131278_object5437:head# 2024-05-15T23:25:42.745 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9855c320:::benchmark_data_smithi090_131278_object4305:head# 2024-05-15T23:25:42.747 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:985ada70:::benchmark_data_smithi090_131278_object7271:head# 2024-05-15T23:25:42.748 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9861e8ff:::benchmark_data_smithi090_131278_object4653:head# 2024-05-15T23:25:42.748 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9862f134:::benchmark_data_smithi090_131278_object6852:head# 2024-05-15T23:25:42.751 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98653afd:::benchmark_data_smithi090_131278_object7686:head# 2024-05-15T23:25:42.754 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:987c8433:::benchmark_data_smithi090_131278_object762:head# 2024-05-15T23:25:42.754 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:987d659e:::benchmark_data_smithi090_131278_object1973:head# 2024-05-15T23:25:42.757 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9885e6ed:::benchmark_data_smithi090_131278_object2396:head# 2024-05-15T23:25:42.757 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:988a332b:::benchmark_data_smithi090_131278_object8522:head# 2024-05-15T23:25:42.757 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:988aa60f:::benchmark_data_smithi090_131278_object1058:head# 2024-05-15T23:25:42.761 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98904da5:::benchmark_data_smithi090_131278_object5846:head# 2024-05-15T23:25:42.761 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9896b9ff:::benchmark_data_smithi090_131278_object902:head# 2024-05-15T23:25:42.761 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9896db18:::benchmark_data_smithi090_131278_object9504:head# 2024-05-15T23:25:42.762 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98a9ee26:::benchmark_data_smithi090_131278_object6310:head# 2024-05-15T23:25:42.762 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98b2b02f:::benchmark_data_smithi090_131278_object2217:head# 2024-05-15T23:25:42.762 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98b4c635:::benchmark_data_smithi090_131278_object3873:head# 2024-05-15T23:25:42.762 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98b64984:::benchmark_data_smithi090_131278_object1652:head# 2024-05-15T23:25:42.763 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98ba2d77:::benchmark_data_smithi090_131278_object4327:head# 2024-05-15T23:25:42.764 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98d3b6ed:::benchmark_data_smithi090_131278_object8643:head# 2024-05-15T23:25:42.764 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98d7caae:::benchmark_data_smithi090_131278_object16:head# 2024-05-15T23:25:42.764 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98e4851f:::benchmark_data_smithi090_131278_object7226:head# 2024-05-15T23:25:42.766 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98e5082d:::benchmark_data_smithi090_131278_object6461:head# 2024-05-15T23:25:42.768 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98eac5bb:::benchmark_data_smithi090_131278_object6323:head# 2024-05-15T23:25:42.769 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98ecfcbe:::benchmark_data_smithi090_131278_object7594:head# 2024-05-15T23:25:42.772 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:98f71035:::benchmark_data_smithi090_131278_object910:head# 2024-05-15T23:25:42.773 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:990e50f8:::benchmark_data_smithi090_131278_object7714:head# 2024-05-15T23:25:42.773 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:990e574f:::benchmark_data_smithi090_131278_object7739:head# 2024-05-15T23:25:42.773 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9914d66d:::benchmark_data_smithi090_131278_object1132:head# 2024-05-15T23:25:42.774 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9926917f:::benchmark_data_smithi090_131278_object1189:head# 2024-05-15T23:25:42.774 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99360b88:::benchmark_data_smithi090_131278_object1696:head# 2024-05-15T23:25:42.775 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9939b578:::benchmark_data_smithi090_131278_object1077:head# 2024-05-15T23:25:42.775 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9951f709:::benchmark_data_smithi090_131278_object4525:head# 2024-05-15T23:25:42.776 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:995ad316:::benchmark_data_smithi090_131278_object6664:head# 2024-05-15T23:25:42.776 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:995fe52e:::benchmark_data_smithi090_131278_object2969:head# 2024-05-15T23:25:42.777 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99609ade:::benchmark_data_smithi090_131278_object9098:head# 2024-05-15T23:25:42.777 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9970e88a:::benchmark_data_smithi090_131278_object7737:head# 2024-05-15T23:25:42.778 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:997a17b2:::benchmark_data_smithi090_131278_object9014:head# 2024-05-15T23:25:42.778 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:997feeb5:::benchmark_data_smithi090_131278_object5536:head# 2024-05-15T23:25:42.779 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9982a1b6:::benchmark_data_smithi090_131278_object2721:head# 2024-05-15T23:25:42.779 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9994411d:::benchmark_data_smithi090_131278_object1421:head# 2024-05-15T23:25:42.780 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:999b5d18:::benchmark_data_smithi090_131278_object8612:head# 2024-05-15T23:25:42.782 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:999c2b33:::benchmark_data_smithi090_131278_object3810:head# 2024-05-15T23:25:42.782 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99a67de9:::benchmark_data_smithi090_131278_object5981:head# 2024-05-15T23:25:42.783 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99a68734:::benchmark_data_smithi090_131278_object8792:head# 2024-05-15T23:25:42.783 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99a78ecc:::benchmark_data_smithi090_131278_object7075:head# 2024-05-15T23:25:42.784 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99ac3645:::benchmark_data_smithi090_131278_object8782:head# 2024-05-15T23:25:42.784 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99adbb5d:::benchmark_data_smithi090_131278_object4452:head# 2024-05-15T23:25:42.787 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99afde95:::benchmark_data_smithi090_131278_object8322:head# 2024-05-15T23:25:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99b65fed:::benchmark_data_smithi090_131278_object375:head# 2024-05-15T23:25:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99c083c9:::benchmark_data_smithi090_131278_object2734:head# 2024-05-15T23:25:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99c5eff9:::benchmark_data_smithi090_131278_object7426:head# 2024-05-15T23:25:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99c9a10b:::benchmark_data_smithi090_131278_object532:head# 2024-05-15T23:25:42.792 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99d379b2:::benchmark_data_smithi090_131278_object8897:head# 2024-05-15T23:25:42.793 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99d3cbbd:::benchmark_data_smithi090_131278_object1441:head# 2024-05-15T23:25:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99d70b97:::benchmark_data_smithi090_131278_object4495:head# 2024-05-15T23:25:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99d94960:::benchmark_data_smithi090_131278_object7051:head# 2024-05-15T23:25:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99e0a2da:::benchmark_data_smithi090_131278_object6375:head# 2024-05-15T23:25:42.797 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99e20fdb:::benchmark_data_smithi090_131278_object8156:head# 2024-05-15T23:25:42.797 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:42.797 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:42.798 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99e6b100:::benchmark_data_smithi090_131278_object4475:head# 2024-05-15T23:25:42.798 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99ee8f0e:::benchmark_data_smithi090_131278_object3285:head# 2024-05-15T23:25:42.798 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99f0ad40:::benchmark_data_smithi090_131278_object6245:head# 2024-05-15T23:25:42.799 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99f655b6:::benchmark_data_smithi090_131278_object8661:head# 2024-05-15T23:25:42.799 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:99fee220:::benchmark_data_smithi090_131278_object1954:head# 2024-05-15T23:25:42.799 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a069c8f:::benchmark_data_smithi090_131278_object6462:head# 2024-05-15T23:25:42.802 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a06ea3c:::benchmark_data_smithi090_131278_object7111:head# 2024-05-15T23:25:42.802 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a080fb0:::benchmark_data_smithi090_131278_object6248:head# 2024-05-15T23:25:42.802 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a0945d9:::benchmark_data_smithi090_131278_object5657:head# 2024-05-15T23:25:42.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:42.806 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a0fe2e8:::benchmark_data_smithi090_131278_object6716:head# 2024-05-15T23:25:42.806 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a10925d:::benchmark_data_smithi090_131278_object7285:head# 2024-05-15T23:25:42.806 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a1d4d5e:::benchmark_data_smithi090_131278_object4862:head# 2024-05-15T23:25:42.807 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a226cb9:::benchmark_data_smithi090_131278_object353:head# 2024-05-15T23:25:42.807 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a35b31e:::benchmark_data_smithi090_131278_object6550:head# 2024-05-15T23:25:42.807 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a47b639:::benchmark_data_smithi090_131278_object8683:head# 2024-05-15T23:25:42.808 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a49abee:::benchmark_data_smithi090_131278_object9469:head# 2024-05-15T23:25:42.808 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a4d3c51:::benchmark_data_smithi090_131278_object3264:head# 2024-05-15T23:25:42.809 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a53f566:::benchmark_data_smithi090_131278_object3598:head# 2024-05-15T23:25:42.810 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a5c7670:::benchmark_data_smithi090_131278_object5081:head# 2024-05-15T23:25:42.812 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a5d62d8:::benchmark_data_smithi090_131278_object4870:head# 2024-05-15T23:25:42.812 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a653bbd:::benchmark_data_smithi090_131278_object7069:head# 2024-05-15T23:25:42.813 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a66f376:::benchmark_data_smithi090_131278_object6685:head# 2024-05-15T23:25:42.813 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a6c882c:::benchmark_data_smithi090_131278_object771:head# 2024-05-15T23:25:42.816 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a6f8001:::benchmark_data_smithi090_131278_object1381:head# 2024-05-15T23:25:42.816 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a7049ea:::benchmark_data_smithi090_131278_object6751:head# 2024-05-15T23:25:42.816 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a755e85:::benchmark_data_smithi090_131278_object8120:head# 2024-05-15T23:25:42.817 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a795b3a:::benchmark_data_smithi090_131278_object4186:head# 2024-05-15T23:25:42.817 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a7ef7c6:::benchmark_data_smithi090_131278_object6154:head# 2024-05-15T23:25:42.818 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a8058d8:::benchmark_data_smithi090_131278_object8915:head# 2024-05-15T23:25:42.819 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a8452a8:::benchmark_data_smithi090_131278_object4754:head# 2024-05-15T23:25:42.819 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a87f556:::benchmark_data_smithi090_131278_object387:head# 2024-05-15T23:25:42.819 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9a92c87c:::benchmark_data_smithi090_131278_object7141:head# 2024-05-15T23:25:42.820 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9aa6d52c:::benchmark_data_smithi090_131278_object7327:head# 2024-05-15T23:25:42.821 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9aa71a6e:::benchmark_data_smithi090_131278_object3419:head# 2024-05-15T23:25:42.821 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9aac9854:::benchmark_data_smithi090_131278_object857:head# 2024-05-15T23:25:42.821 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ab71579:::benchmark_data_smithi090_131278_object881:head# 2024-05-15T23:25:42.822 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9abbfd44:::benchmark_data_smithi090_131278_object4528:head# 2024-05-15T23:25:42.822 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9abcc2dd:::benchmark_data_smithi090_131278_object1535:head# 2024-05-15T23:25:42.823 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9abf4bad:::benchmark_data_smithi090_131278_object7579:head# 2024-05-15T23:25:42.826 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ac2b86d:::benchmark_data_smithi090_131278_object7079:head# 2024-05-15T23:25:42.826 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ac7c235:::benchmark_data_smithi090_131278_object1811:head# 2024-05-15T23:25:42.827 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ac7e4c1:::benchmark_data_smithi090_131278_object8124:head# 2024-05-15T23:25:42.827 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ace30f1:::benchmark_data_smithi090_131278_object5527:head# 2024-05-15T23:25:42.828 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ad9c434:::benchmark_data_smithi090_131278_object8516:head# 2024-05-15T23:25:42.828 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ae487d7:::benchmark_data_smithi090_131278_object3857:head# 2024-05-15T23:25:42.828 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9aeb4312:::benchmark_data_smithi090_131278_object8141:head# 2024-05-15T23:25:42.829 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9af269c9:::benchmark_data_smithi090_131278_object4042:head# 2024-05-15T23:25:42.830 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9af3fa0a:::benchmark_data_smithi090_131278_object2519:head# 2024-05-15T23:25:42.830 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9af6aaaa:::benchmark_data_smithi090_131278_object2371:head# 2024-05-15T23:25:42.830 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b066dcc:::benchmark_data_smithi090_131278_object1754:head# 2024-05-15T23:25:42.831 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b0b1faa:::benchmark_data_smithi090_131278_object7747:head# 2024-05-15T23:25:42.832 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b0ce0e1:::benchmark_data_smithi090_131278_object903:head# 2024-05-15T23:25:42.832 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b1576db:::benchmark_data_smithi090_131278_object9560:head# 2024-05-15T23:25:42.833 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b22a059:::benchmark_data_smithi090_131278_object8713:head# 2024-05-15T23:25:42.833 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b25236f:::benchmark_data_smithi090_131278_object2107:head# 2024-05-15T23:25:42.834 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b335d17:::benchmark_data_smithi090_131278_object4110:head# 2024-05-15T23:25:42.834 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b34622b:::benchmark_data_smithi090_131278_object8057:head# 2024-05-15T23:25:42.835 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b354a80:::benchmark_data_smithi090_131278_object7129:head# 2024-05-15T23:25:42.835 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b39771e:::benchmark_data_smithi090_131278_object1683:head# 2024-05-15T23:25:42.836 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b3ae3c3:::benchmark_data_smithi090_131278_object4584:head# 2024-05-15T23:25:42.836 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b3dc58c:::benchmark_data_smithi090_131278_object7060:head# 2024-05-15T23:25:42.836 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b3f59cf:::benchmark_data_smithi090_131278_object7496:head# 2024-05-15T23:25:42.837 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b5e7ce8:::benchmark_data_smithi090_131278_object8406:head# 2024-05-15T23:25:42.840 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b65a6ff:::benchmark_data_smithi090_131278_object6309:head# 2024-05-15T23:25:42.840 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b6ce2b5:::benchmark_data_smithi090_131278_object6342:head# 2024-05-15T23:25:42.841 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b6db155:::benchmark_data_smithi090_131278_object3494:head# 2024-05-15T23:25:42.841 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b7f9a72:::benchmark_data_smithi090_131278_object6889:head# 2024-05-15T23:25:42.841 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b88417d:::benchmark_data_smithi090_131278_object1776:head# 2024-05-15T23:25:42.842 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b8a8520:::benchmark_data_smithi090_131278_object6110:head# 2024-05-15T23:25:42.842 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b93acfe:::benchmark_data_smithi090_131278_object9044:head# 2024-05-15T23:25:42.842 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b95535b:::benchmark_data_smithi090_131278_object1064:head# 2024-05-15T23:25:42.843 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9b969468:::benchmark_data_smithi090_131278_object3593:head# 2024-05-15T23:25:42.846 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ba10268:::benchmark_data_smithi090_131278_object1743:head# 2024-05-15T23:25:42.846 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ba20efd:::benchmark_data_smithi090_131278_object8533:head# 2024-05-15T23:25:42.848 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ba3f132:::benchmark_data_smithi090_131278_object5978:head# 2024-05-15T23:25:42.849 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9baaf4f5:::benchmark_data_smithi090_131278_object2046:head# 2024-05-15T23:25:42.851 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9badb61b:::benchmark_data_smithi090_131278_object1621:head# 2024-05-15T23:25:42.853 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9bb14c1a:::benchmark_data_smithi090_131278_object4526:head# 2024-05-15T23:25:42.854 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9bba2eab:::benchmark_data_smithi090_131278_object7254:head# 2024-05-15T23:25:42.854 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9bded8f9:::benchmark_data_smithi090_131278_object181:head# 2024-05-15T23:25:42.855 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9be087b9:::benchmark_data_smithi090_131278_object853:head# 2024-05-15T23:25:42.855 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9be65b33:::benchmark_data_smithi090_131278_object7563:head# 2024-05-15T23:25:42.857 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9bfbd328:::benchmark_data_smithi090_131278_object1709:head# 2024-05-15T23:25:42.858 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c033f65:::benchmark_data_smithi090_131278_object9537:head# 2024-05-15T23:25:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c12383d:::benchmark_data_smithi090_131278_object8286:head# 2024-05-15T23:25:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c198c0e:::benchmark_data_smithi090_131278_object5539:head# 2024-05-15T23:25:42.862 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c1b617a:::benchmark_data_smithi090_131278_object5560:head# 2024-05-15T23:25:42.862 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c229728:::benchmark_data_smithi090_131278_object3838:head# 2024-05-15T23:25:42.864 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:42.865 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c233228:::benchmark_data_smithi090_131278_object5474:head# 2024-05-15T23:25:42.865 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c2634c4:::benchmark_data_smithi090_131278_object6276:head# 2024-05-15T23:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c270aad:::benchmark_data_smithi090_131278_object7837:head# 2024-05-15T23:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c2f620f:::benchmark_data_smithi090_131278_object6604:head# 2024-05-15T23:25:42.867 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c364d3b:::benchmark_data_smithi090_131278_object8833:head# 2024-05-15T23:25:42.867 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c4921ca:::benchmark_data_smithi090_131278_object5482:head# 2024-05-15T23:25:42.868 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c54027b:::benchmark_data_smithi090_131278_object3882:head# 2024-05-15T23:25:42.869 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c605c2d:::benchmark_data_smithi090_131278_object3860:head# 2024-05-15T23:25:42.869 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c7ea4f8:::benchmark_data_smithi090_131278_object1211:head# 2024-05-15T23:25:42.869 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c7fec58:::benchmark_data_smithi090_131278_object5385:head# 2024-05-15T23:25:42.870 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c87246c:::benchmark_data_smithi090_131278_object7405:head# 2024-05-15T23:25:42.870 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c8f3c46:::benchmark_data_smithi090_131278_object1704:head# 2024-05-15T23:25:42.871 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c9094ce:::benchmark_data_smithi090_131278_object5626:head# 2024-05-15T23:25:42.873 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c94417b:::benchmark_data_smithi090_131278_object831:head# 2024-05-15T23:25:42.874 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9c9d9d5a:::benchmark_data_smithi090_131278_object2382:head# 2024-05-15T23:25:42.874 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9cad4358:::benchmark_data_smithi090_131278_object8210:head# 2024-05-15T23:25:42.874 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9cc01491:::benchmark_data_smithi090_131278_object1255:head# 2024-05-15T23:25:42.874 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9cd26e9f:::benchmark_data_smithi090_131278_object6804:head# 2024-05-15T23:25:42.877 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9cd53de7:::benchmark_data_smithi090_131278_object4596:head# 2024-05-15T23:25:42.879 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9cdeebe5:::benchmark_data_smithi090_131278_object3068:head# 2024-05-15T23:25:42.880 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ce5f6a2:::benchmark_data_smithi090_131278_object9330:head# 2024-05-15T23:25:42.883 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9cf15f57:::benchmark_data_smithi090_131278_object7645:head# 2024-05-15T23:25:42.883 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9cf1e8bb:::benchmark_data_smithi090_131278_object2966:head# 2024-05-15T23:25:42.883 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9cf3287a:::benchmark_data_smithi090_131278_object6525:head# 2024-05-15T23:25:42.883 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9cf761ed:::benchmark_data_smithi090_131278_object9440:head# 2024-05-15T23:25:42.884 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d05eb0c:::benchmark_data_smithi090_131278_object1638:head# 2024-05-15T23:25:42.884 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d0ed73f:::benchmark_data_smithi090_131278_object1187:head# 2024-05-15T23:25:42.884 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d13c342:::benchmark_data_smithi090_131278_object3222:head# 2024-05-15T23:25:42.885 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d1c3dc6:::benchmark_data_smithi090_131278_object1435:head# 2024-05-15T23:25:42.886 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d24ab0a:::benchmark_data_smithi090_131278_object1814:head# 2024-05-15T23:25:42.886 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d2902ac:::benchmark_data_smithi090_131278_object307:head# 2024-05-15T23:25:42.886 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d2db021:::benchmark_data_smithi090_131278_object2076:head# 2024-05-15T23:25:42.887 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d2e511c:::benchmark_data_smithi090_131278_object368:head# 2024-05-15T23:25:42.887 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d35fa6d:::benchmark_data_smithi090_131278_object6514:head# 2024-05-15T23:25:42.888 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d363a9f:::benchmark_data_smithi090_131278_object3379:head# 2024-05-15T23:25:42.888 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d399153:::benchmark_data_smithi090_131278_object2662:head# 2024-05-15T23:25:42.890 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d3eb19c:::benchmark_data_smithi090_131278_object2916:head# 2024-05-15T23:25:42.891 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d3ff959:::benchmark_data_smithi090_131278_object1095:head# 2024-05-15T23:25:42.893 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d404fec:::benchmark_data_smithi090_131278_object3317:head# 2024-05-15T23:25:42.893 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d493b23:::benchmark_data_smithi090_131278_object474:head# 2024-05-15T23:25:42.893 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d53d3de:::benchmark_data_smithi090_131278_object721:head# 2024-05-15T23:25:42.894 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d6d0d8d:::benchmark_data_smithi090_131278_object8642:head# 2024-05-15T23:25:42.896 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d7215b8:::benchmark_data_smithi090_131278_object2786:head# 2024-05-15T23:25:42.896 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d76ba88:::benchmark_data_smithi090_131278_object1274:head# 2024-05-15T23:25:42.898 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d775aa8:::benchmark_data_smithi090_131278_object8447:head# 2024-05-15T23:25:42.898 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d78807f:::benchmark_data_smithi090_131278_object4898:head# 2024-05-15T23:25:42.900 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d7a46bf:::benchmark_data_smithi090_131278_object8093:head# 2024-05-15T23:25:42.900 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d7d6c50:::benchmark_data_smithi090_131278_object5568:head# 2024-05-15T23:25:42.901 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d861ffa:::benchmark_data_smithi090_131278_object4677:head# 2024-05-15T23:25:42.901 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d8a5089:::benchmark_data_smithi090_131278_object4377:head# 2024-05-15T23:25:42.901 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d914513:::benchmark_data_smithi090_131278_object7205:head# 2024-05-15T23:25:42.901 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d930859:::benchmark_data_smithi090_131278_object6136:head# 2024-05-15T23:25:42.902 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9d9f1239:::benchmark_data_smithi090_131278_object1948:head# 2024-05-15T23:25:42.902 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9da640f0:::benchmark_data_smithi090_131278_object4243:head# 2024-05-15T23:25:42.903 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9dac740d:::benchmark_data_smithi090_131278_object2457:head# 2024-05-15T23:25:42.903 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9db02ac7:::benchmark_data_smithi090_131278_object6924:head# 2024-05-15T23:25:42.904 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9dbf7d54:::benchmark_data_smithi090_131278_object8825:head# 2024-05-15T23:25:42.905 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ddb030e:::benchmark_data_smithi090_131278_object5358:head# 2024-05-15T23:25:42.905 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9de2ac8d:::benchmark_data_smithi090_131278_object7475:head# 2024-05-15T23:25:42.911 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9de3fbc5:::benchmark_data_smithi090_131278_object9340:head# 2024-05-15T23:25:42.911 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9deb845a:::benchmark_data_smithi090_131278_object9608:head# 2024-05-15T23:25:42.911 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9dec4f3f:::benchmark_data_smithi090_131278_object4354:head# 2024-05-15T23:25:42.911 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e086b1b:::benchmark_data_smithi090_131278_object5370:head# 2024-05-15T23:25:42.911 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e0d81b3:::benchmark_data_smithi090_131278_object4866:head# 2024-05-15T23:25:42.912 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e1f5c08:::benchmark_data_smithi090_131278_object9454:head# 2024-05-15T23:25:42.912 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e2411e8:::benchmark_data_smithi090_131278_object7013:head# 2024-05-15T23:25:42.913 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e290cec:::benchmark_data_smithi090_131278_object6125:head# 2024-05-15T23:25:42.913 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e3692dd:::benchmark_data_smithi090_131278_object4585:head# 2024-05-15T23:25:42.914 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e424afd:::benchmark_data_smithi090_131278_object8500:head# 2024-05-15T23:25:42.915 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e4a098e:::benchmark_data_smithi090_131278_object3531:head# 2024-05-15T23:25:42.916 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e4b76da:::benchmark_data_smithi090_131278_object2469:head# 2024-05-15T23:25:42.920 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e504142:::benchmark_data_smithi090_131278_object2595:head# 2024-05-15T23:25:42.920 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e57314f:::benchmark_data_smithi090_131278_object6976:head# 2024-05-15T23:25:42.920 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e629404:::benchmark_data_smithi090_131278_object6947:head# 2024-05-15T23:25:42.921 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e67b675:::benchmark_data_smithi090_131278_object849:head# 2024-05-15T23:25:42.921 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e68c340:::benchmark_data_smithi090_131278_object4689:head# 2024-05-15T23:25:42.922 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e698bf6:::benchmark_data_smithi090_131278_object5260:head# 2024-05-15T23:25:42.922 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e6d0702:::benchmark_data_smithi090_131278_object2956:head# 2024-05-15T23:25:42.923 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e6f8539:::benchmark_data_smithi090_131278_object9426:head# 2024-05-15T23:25:42.924 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e759594:::benchmark_data_smithi090_131278_object9480:head# 2024-05-15T23:25:42.925 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e77c5e1:::benchmark_data_smithi090_131278_object5750:head# 2024-05-15T23:25:42.925 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e8bf076:::benchmark_data_smithi090_131278_object3537:head# 2024-05-15T23:25:42.926 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e8ff872:::benchmark_data_smithi090_131278_object5949:head# 2024-05-15T23:25:42.926 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e9024c8:::benchmark_data_smithi090_131278_object8567:head# 2024-05-15T23:25:42.928 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e91597e:::benchmark_data_smithi090_131278_object3965:head# 2024-05-15T23:25:42.928 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e953a21:::benchmark_data_smithi090_131278_object1023:head# 2024-05-15T23:25:42.929 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e97951e:::benchmark_data_smithi090_131278_object9166:head# 2024-05-15T23:25:42.931 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9e987fbb:::benchmark_data_smithi090_131278_object2346:head# 2024-05-15T23:25:42.932 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ea28cb5:::benchmark_data_smithi090_131278_object4364:head# 2024-05-15T23:25:42.933 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ea4d97b:::benchmark_data_smithi090_131278_object9546:head# 2024-05-15T23:25:42.933 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ea5efbd:::benchmark_data_smithi090_131278_object8270:head# 2024-05-15T23:25:42.933 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ea76e82:::benchmark_data_smithi090_131278_object6897:head# 2024-05-15T23:25:42.935 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9eb28238:::benchmark_data_smithi090_131278_object4399:head# 2024-05-15T23:25:42.935 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ebc5b20:::benchmark_data_smithi090_131278_object8126:head# 2024-05-15T23:25:42.935 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ebc9534:::benchmark_data_smithi090_131278_object3916:head# 2024-05-15T23:25:42.935 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ec5dff4:::benchmark_data_smithi090_131278_object5363:head# 2024-05-15T23:25:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ecc038a:::benchmark_data_smithi090_131278_object5427:head# 2024-05-15T23:25:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ed6099d:::benchmark_data_smithi090_131278_object5432:head# 2024-05-15T23:25:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ee0c233:::benchmark_data_smithi090_131278_object4117:head# 2024-05-15T23:25:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ee4219c:::benchmark_data_smithi090_131278_object3975:head# 2024-05-15T23:25:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ee5a36e:::benchmark_data_smithi090_131278_object1883:head# 2024-05-15T23:25:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ef9dd1b:::benchmark_data_smithi090_131278_object3268:head# 2024-05-15T23:25:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f041bda:::benchmark_data_smithi090_131278_object7296:head# 2024-05-15T23:25:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f05093f:::benchmark_data_smithi090_131278_object5554:head# 2024-05-15T23:25:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f07d819:::benchmark_data_smithi090_131278_object3470:head# 2024-05-15T23:25:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f0d5197:::benchmark_data_smithi090_131278_object6766:head# 2024-05-15T23:25:42.944 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f1e3d01:::benchmark_data_smithi090_131278_object9276:head# 2024-05-15T23:25:42.945 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f3a85a0:::benchmark_data_smithi090_131278_object8339:head# 2024-05-15T23:25:42.945 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f3ba423:::benchmark_data_smithi090_131278_object8317:head# 2024-05-15T23:25:42.946 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f3d7ecd:::benchmark_data_smithi090_131278_object3799:head# 2024-05-15T23:25:42.946 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f40333d:::benchmark_data_smithi090_131278_object3513:head# 2024-05-15T23:25:42.947 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f4b8b94:::benchmark_data_smithi090_131278_object1961:head# 2024-05-15T23:25:42.947 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f4f29a0:::benchmark_data_smithi090_131278_object1982:head# 2024-05-15T23:25:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f532c22:::benchmark_data_smithi090_131278_object5706:head# 2024-05-15T23:25:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f541f69:::benchmark_data_smithi090_131278_object3022:head# 2024-05-15T23:25:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f5c5602:::benchmark_data_smithi090_131278_object2287:head# 2024-05-15T23:25:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f6028d4:::benchmark_data_smithi090_131278_object9000:head# 2024-05-15T23:25:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f61bde5:::benchmark_data_smithi090_131278_object1385:head# 2024-05-15T23:25:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f63abda:::benchmark_data_smithi090_131278_object5148:head# 2024-05-15T23:25:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f64899a:::benchmark_data_smithi090_131278_object8437:head# 2024-05-15T23:25:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f69d06a:::benchmark_data_smithi090_131278_object2375:head# 2024-05-15T23:25:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f6a49f2:::benchmark_data_smithi090_131278_object9250:head# 2024-05-15T23:25:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f70c94a:::benchmark_data_smithi090_131278_object73:head# 2024-05-15T23:25:42.954 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f76f4e1:::benchmark_data_smithi090_131278_object1694:head# 2024-05-15T23:25:42.955 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f78091b:::benchmark_data_smithi090_131278_object2114:head# 2024-05-15T23:25:42.955 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f7bd4dd:::benchmark_data_smithi090_131278_object6218:head# 2024-05-15T23:25:42.957 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f7ce14d:::benchmark_data_smithi090_131278_object9616:head# 2024-05-15T23:25:42.957 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f82de16:::benchmark_data_smithi090_131278_object3098:head# 2024-05-15T23:25:42.958 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f83b75d:::benchmark_data_smithi090_131278_object8146:head# 2024-05-15T23:25:42.959 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f84b8b6:::benchmark_data_smithi090_131278_object9624:head# 2024-05-15T23:25:42.959 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f86552d:::benchmark_data_smithi090_131278_object6011:head# 2024-05-15T23:25:42.962 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f8aa762:::benchmark_data_smithi090_131278_object4925:head# 2024-05-15T23:25:42.966 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f914907:::benchmark_data_smithi090_131278_object5889:head# 2024-05-15T23:25:42.966 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f9686d3:::benchmark_data_smithi090_131278_object8734:head# 2024-05-15T23:25:42.966 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9f998076:::benchmark_data_smithi090_131278_object4940:head# 2024-05-15T23:25:42.966 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9fb50385:::benchmark_data_smithi090_131278_object6802:head# 2024-05-15T23:25:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9fb9cc48:::benchmark_data_smithi090_131278_object4103:head# 2024-05-15T23:25:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9fbd50f2:::benchmark_data_smithi090_131278_object2094:head# 2024-05-15T23:25:42.970 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9fca588a:::benchmark_data_smithi090_131278_object8397:head# 2024-05-15T23:25:42.972 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9fccdcec:::benchmark_data_smithi090_131278_object5179:head# 2024-05-15T23:25:42.973 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9fce1e00:::benchmark_data_smithi090_131278_object7745:head# 2024-05-15T23:25:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9fd39d23:::benchmark_data_smithi090_131278_object8454:head# 2024-05-15T23:25:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9fdbf0ad:::benchmark_data_smithi090_131278_object1014:head# 2024-05-15T23:25:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9fe1c7cf:::benchmark_data_smithi090_131278_object5391:head# 2024-05-15T23:25:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9fe91d49:::benchmark_data_smithi090_131278_object4062:head# 2024-05-15T23:25:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9feca161:::benchmark_data_smithi090_131278_object5777:head# 2024-05-15T23:25:42.976 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9fee0593:::benchmark_data_smithi090_131278_object6191:head# 2024-05-15T23:25:42.977 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ff1d1ae:::benchmark_data_smithi090_131278_object6519:head# 2024-05-15T23:25:42.977 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ff22a19:::benchmark_data_smithi090_131278_object2502:head# 2024-05-15T23:25:42.979 INFO:teuthology.orchestra.run.smithi090.stderr:Read #6:9ff39dbd:::benchmark_data_smithi090_131278_object7610:head# 2024-05-15T23:25:42.983 INFO:teuthology.orchestra.run.smithi090.stderr:Export successful 2024-05-15T23:25:43.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:43 smithi084 ceph-mon[39784]: pgmap v1327: 17 pgs: 17 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 207 MiB/s wr, 3.31k op/s; 9.5 MiB/s, 151 objects/s recovering 2024-05-15T23:25:43.119 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:43.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:43.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:43.228 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:43.261 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:43.389 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:43.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:43.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:43.453 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 36 15 43942 43927 76.2514 188.125 0.00563044 0.0131125 2024-05-15T23:25:43.492 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:43.532 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:43 smithi088 ceph-mon[44699]: pgmap v1327: 17 pgs: 17 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 207 MiB/s wr, 3.31k op/s; 9.5 MiB/s, 151 objects/s recovering 2024-05-15T23:25:43.535 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:43.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:43 smithi090 ceph-mon[44434]: pgmap v1327: 17 pgs: 17 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 207 MiB/s wr, 3.31k op/s; 9.5 MiB/s, 151 objects/s recovering 2024-05-15T23:25:43.561 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:43.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:43.665 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:25:43.705 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:44.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -n osd.8 --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-8 --journal-path /var/lib/ceph/osd/ceph-8/journal --force --op remove --pgid 6.9 2024-05-15T23:25:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:25:44.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:44.063 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:25:44.164 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:44.206 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:44.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:44.246 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3766907234' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:25:44.246 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:44 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:25:44.338 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/osd.8/config 2024-05-15T23:25:44.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:44.381 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:44.453 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 37 15 46677 46662 78.81 170.938 0.00429534 0.0126859 2024-05-15T23:25:44.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3766907234' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:25:44.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:44 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:25:44.584 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:25:44.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3766907234' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:25:44.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:44 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:25:45.453 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 38 16 49443 49427 81.2832 172.812 0.00493752 0.0123015 2024-05-15T23:25:45.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3766907234' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:25:45.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:45 smithi090 ceph-mon[44434]: osdmap e496: 12 total, 11 up, 8 in 2024-05-15T23:25:45.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:45 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:25:45.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:45 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:25:45.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:45 smithi090 ceph-mon[44434]: pgmap v1329: 17 pgs: 17 active+clean; 2.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 179 MiB/s wr, 2.87k op/s; 6.0 MiB/s, 96 objects/s recovering 2024-05-15T23:25:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3766907234' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:25:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:45 smithi088 ceph-mon[44699]: osdmap e496: 12 total, 11 up, 8 in 2024-05-15T23:25:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:45 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:25:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:45 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:25:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:45 smithi088 ceph-mon[44699]: pgmap v1329: 17 pgs: 17 active+clean; 2.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 179 MiB/s wr, 2.87k op/s; 6.0 MiB/s, 96 objects/s recovering 2024-05-15T23:25:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3766907234' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:25:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:45 smithi084 ceph-mon[39784]: osdmap e496: 12 total, 11 up, 8 in 2024-05-15T23:25:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:45 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:25:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:45 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:25:45.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:45 smithi084 ceph-mon[39784]: pgmap v1329: 17 pgs: 17 active+clean; 2.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 179 MiB/s wr, 2.87k op/s; 6.0 MiB/s, 96 objects/s recovering 2024-05-15T23:25:45.798 INFO:teuthology.orchestra.run.smithi090.stderr:Creating an OSD daemon form without an OSD FSID value 2024-05-15T23:25:46.456 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 39 16 51553 51537 82.5769 131.875 0.0112162 0.0121086 2024-05-15T23:25:46.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:46 smithi090 ceph-mon[44434]: osdmap e497: 12 total, 11 up, 8 in 2024-05-15T23:25:46.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:46 smithi088 ceph-mon[44699]: osdmap e497: 12 total, 11 up, 8 in 2024-05-15T23:25:46.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:46 smithi084 ceph-mon[39784]: osdmap e497: 12 total, 11 up, 8 in 2024-05-15T23:25:47.455 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:25:47.454145 min lat: 0.0010209 max lat: 24.9077 avg lat: 0.0118224 2024-05-15T23:25:47.455 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:25:47.455 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 40 16 54151 54135 84.5713 162.375 0.00417499 0.0118224 2024-05-15T23:25:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:47 smithi090 ceph-mon[44434]: pgmap v1331: 17 pgs: 17 active+clean; 3.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 235 MiB/s wr, 3.76k op/s; 6.0 MiB/s, 96 objects/s recovering 2024-05-15T23:25:47.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:47 smithi088 ceph-mon[44699]: pgmap v1331: 17 pgs: 17 active+clean; 3.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 235 MiB/s wr, 3.76k op/s; 6.0 MiB/s, 96 objects/s recovering 2024-05-15T23:25:47.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:47 smithi084 ceph-mon[39784]: pgmap v1331: 17 pgs: 17 active+clean; 3.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 235 MiB/s wr, 3.76k op/s; 6.0 MiB/s, 96 objects/s recovering 2024-05-15T23:25:48.312 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:48.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:48.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:48.420 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:25:48.454 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 41 16 56754 56738 86.4761 162.688 0.00589692 0.0115619 2024-05-15T23:25:48.462 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:48.492 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:25:48.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:48.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:25:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:48 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:25:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:48 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:25:48.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:48 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:25:48.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:48 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:25:48.597 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:48.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:48 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:25:48.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:48 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:25:48.643 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:48.672 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:48.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:48.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:48.777 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: marking collection for removal 2024-05-15T23:25:48.789 INFO:teuthology.orchestra.run.smithi090.stdout:setting '_remove' omap key 2024-05-15T23:25:48.795 INFO:teuthology.orchestra.run.smithi090.stdout:finish_remove_pgs 6.9_head removing 6.9 2024-05-15T23:25:48.808 INFO:teuthology.orchestra.run.smithi090.stdout:Remove successful 2024-05-15T23:25:48.816 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:48.855 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:48.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:48.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:48.961 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:25:49.001 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:49.030 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:25:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:49.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:25:49.138 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:49.174 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:49.282 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:49.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:49.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:49.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:49 smithi084 ceph-mon[39784]: pgmap v1332: 17 pgs: 17 active+clean; 3.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-05-15T23:25:49.389 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:25:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:49.440 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:25:49.440 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:25:49.455 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 42 15 59170 59155 88.0135 151.062 0.00519811 0.0113607 2024-05-15T23:25:49.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:49 smithi090 ceph-mon[44434]: pgmap v1332: 17 pgs: 17 active+clean; 3.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-05-15T23:25:49.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:49 smithi088 ceph-mon[44699]: pgmap v1332: 17 pgs: 17 active+clean; 3.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-05-15T23:25:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:25:49.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:49.593 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:25:49.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:25:49.729 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:50.081 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:25:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:50.084 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:25:50.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -n osd.8 --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-8 --journal-path /var/lib/ceph/osd/ceph-8/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.6.9.8 2024-05-15T23:25:50.180 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:25:50.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:50.202 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:25:50.223 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:50.227 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:25:50.251 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:25:50.257 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:50.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:50.265 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:50.280 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:25:50.312 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:25:50.341 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:25:50.366 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:25:50.368 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:25:50.395 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:25:50.417 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:50.423 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:25:50.432 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/osd.8/config 2024-05-15T23:25:50.442 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:25:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:50.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:25:50.447 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:25:50.447 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:25:50.448 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:25:50.448 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:25:50.448 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:25:50.448 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:25:50.448 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:25:50.448 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:25:50.448 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:25:50.448 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-05-15T23:25:50.448 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:25:50.448 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:25:50.448 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:25:50.459 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 43 16 61568 61552 89.4422 149.812 0.00664546 0.0111784 2024-05-15T23:25:50.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:50.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3054977680' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:25:50.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:50.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3054977680' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:25:50.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:50.583 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:50.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3054977680' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:25:50.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:50.714 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:50.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:50.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:50.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T23:25:50.792 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:25:50.824 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:25:50.885 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:50.941 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:25:50.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:50.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:25:51.048 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:51.088 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:51.120 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:51.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:51.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:51.228 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:25:51.272 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:51.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:51 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:25:51.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:51 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:25:51.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:51 smithi090 ceph-mon[44434]: osdmap e498: 12 total, 11 up, 8 in 2024-05-15T23:25:51.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.308 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:25:51.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:51.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:25:51.413 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:25:51.456 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:25:51.458 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 44 16 63721 63705 90.4673 134.562 0.0026129 0.0110511 2024-05-15T23:25:51.470 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:51.502 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:51.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:51.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:51.521 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:51.534 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:51.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:51 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:51 smithi090 ceph-mon[44434]: pgmap v1334: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 201 MiB/s wr, 3.21k op/s 2024-05-15T23:25:51.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:51 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:25:51.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:51 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:25:51.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:51 smithi088 ceph-mon[44699]: osdmap e498: 12 total, 11 up, 8 in 2024-05-15T23:25:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:51 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:51 smithi088 ceph-mon[44699]: pgmap v1334: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 201 MiB/s wr, 3.21k op/s 2024-05-15T23:25:51.609 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:51 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:25:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:51 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:25:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:51 smithi084 ceph-mon[39784]: osdmap e498: 12 total, 11 up, 8 in 2024-05-15T23:25:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:51 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:25:51.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:51 smithi084 ceph-mon[39784]: pgmap v1334: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 201 MiB/s wr, 3.21k op/s 2024-05-15T23:25:51.648 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:51.676 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:51.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:51.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:51.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:51.826 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:51.862 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:51.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:51.869 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:51.925 INFO:teuthology.orchestra.run.smithi090.stderr:Creating an OSD daemon form without an OSD FSID value 2024-05-15T23:25:51.970 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:52.000 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:52.459 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 45 16 65727 65711 91.2427 125.375 0.00335384 0.0109582 2024-05-15T23:25:52.650 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:52.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:52.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:52.663 INFO:teuthology.orchestra.run.smithi090.stdout:get_pg_num_history pg_num_history pg_num_history(e484 pg_nums {1={14=1},5={374=16},6={477=16}} deleted_pools 474,5) 2024-05-15T23:25:52.663 INFO:teuthology.orchestra.run.smithi090.stdout:Importing pgid 6.9 2024-05-15T23:25:52.664 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:900a8860:::benchmark_data_smithi090_131278_object1821:head# 2024-05-15T23:25:52.664 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.665 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:900cf9ee:::benchmark_data_smithi090_131278_object7434:head# 2024-05-15T23:25:52.665 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.665 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:900fa84b:::benchmark_data_smithi090_131278_object3679:head# 2024-05-15T23:25:52.665 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.665 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:90266830:::benchmark_data_smithi090_131278_object947:head# 2024-05-15T23:25:52.665 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.665 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:902d356d:::benchmark_data_smithi090_131278_object979:head# 2024-05-15T23:25:52.665 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.665 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9044e792:::benchmark_data_smithi090_131278_object9667:head# 2024-05-15T23:25:52.666 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.667 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9055633e:::benchmark_data_smithi090_131278_object4029:head# 2024-05-15T23:25:52.667 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.667 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:905c2c2d:::benchmark_data_smithi090_131278_object5916:head# 2024-05-15T23:25:52.667 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.667 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:905fd676:::benchmark_data_smithi090_131278_object130:head# 2024-05-15T23:25:52.667 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.667 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:906f8484:::benchmark_data_smithi090_131278_object5186:head# 2024-05-15T23:25:52.667 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.667 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9074d1f2:::benchmark_data_smithi090_131278_object4971:head# 2024-05-15T23:25:52.667 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.667 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:90776185:::benchmark_data_smithi090_131278_object5453:head# 2024-05-15T23:25:52.668 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.668 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:907e00b3:::benchmark_data_smithi090_131278_object575:head# 2024-05-15T23:25:52.668 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.669 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9080cd7a:::benchmark_data_smithi090_131278_object7506:head# 2024-05-15T23:25:52.669 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.669 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9081e45c:::benchmark_data_smithi090_131278_object7685:head# 2024-05-15T23:25:52.669 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.669 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9086d21d:::benchmark_data_smithi090_131278_object957:head# 2024-05-15T23:25:52.669 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.669 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:908b457c:::benchmark_data_smithi090_131278_object8916:head# 2024-05-15T23:25:52.669 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.669 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:908e9390:::benchmark_data_smithi090_131278_object5727:head# 2024-05-15T23:25:52.669 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.670 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:90997555:::benchmark_data_smithi090_131278_object3013:head# 2024-05-15T23:25:52.670 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.671 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:909a6a12:::benchmark_data_smithi090_131278_object4786:head# 2024-05-15T23:25:52.671 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.671 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:90a61f86:::benchmark_data_smithi090_131278_object3349:head# 2024-05-15T23:25:52.671 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.672 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:90a9a546:::benchmark_data_smithi090_131278_object4415:head# 2024-05-15T23:25:52.672 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.672 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:90b130ec:::benchmark_data_smithi090_131278_object4552:head# 2024-05-15T23:25:52.672 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.672 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:90c23108:::benchmark_data_smithi090_131278_object1429:head# 2024-05-15T23:25:52.672 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.672 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:90d6fb50:::benchmark_data_smithi090_131278_object4051:head# 2024-05-15T23:25:52.672 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.672 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:90e046b4:::benchmark_data_smithi090_131278_object5414:head# 2024-05-15T23:25:52.672 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.672 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:90e6a12b:::benchmark_data_smithi090_131278_object1636:head# 2024-05-15T23:25:52.672 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.673 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:90ec37fd:::benchmark_data_smithi090_131278_object5131:head# 2024-05-15T23:25:52.674 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.674 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:90ee0c58:::benchmark_data_smithi090_131278_object224:head# 2024-05-15T23:25:52.674 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.675 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:90f7881d:::benchmark_data_smithi090_131278_object8233:head# 2024-05-15T23:25:52.676 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.678 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:910001ca:::benchmark_data_smithi090_131278_object4537:head# 2024-05-15T23:25:52.678 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.678 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91152a18:::benchmark_data_smithi090_131278_object4850:head# 2024-05-15T23:25:52.678 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.679 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9118abda:::benchmark_data_smithi090_131278_object1310:head# 2024-05-15T23:25:52.679 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.679 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9128cb6b:::benchmark_data_smithi090_131278_object7382:head# 2024-05-15T23:25:52.679 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.680 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:912cdb75:::benchmark_data_smithi090_131278_object2311:head# 2024-05-15T23:25:52.680 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.680 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:913af216:::benchmark_data_smithi090_131278_object3819:head# 2024-05-15T23:25:52.680 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.680 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:913cc6fa:::benchmark_data_smithi090_131278_object2789:head# 2024-05-15T23:25:52.680 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.681 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9140c596:::benchmark_data_smithi090_131278_object746:head# 2024-05-15T23:25:52.681 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.681 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9144cf37:::benchmark_data_smithi090_131278_object8394:head# 2024-05-15T23:25:52.681 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.681 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:915243fc:::benchmark_data_smithi090_131278_object2946:head# 2024-05-15T23:25:52.681 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.681 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91656e07:::benchmark_data_smithi090_131278_object6503:head# 2024-05-15T23:25:52.682 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.682 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9167c6f1:::benchmark_data_smithi090_131278_object4014:head# 2024-05-15T23:25:52.682 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.683 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:916a9cb3:::benchmark_data_smithi090_131278_object6531:head# 2024-05-15T23:25:52.683 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.683 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9174043b:::benchmark_data_smithi090_131278_object7054:head# 2024-05-15T23:25:52.683 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.683 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:917a6a16:::benchmark_data_smithi090_131278_object5628:head# 2024-05-15T23:25:52.683 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.683 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:917aa1ee:::benchmark_data_smithi090_131278_object3003:head# 2024-05-15T23:25:52.683 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.684 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9181e19b:::benchmark_data_smithi090_131278_object8384:head# 2024-05-15T23:25:52.684 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.684 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9183ccc5:::benchmark_data_smithi090_131278_object9606:head# 2024-05-15T23:25:52.684 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.685 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9194bccf:::benchmark_data_smithi090_131278_object3184:head# 2024-05-15T23:25:52.685 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.685 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91aafd64:::benchmark_data_smithi090_131278_object5202:head# 2024-05-15T23:25:52.685 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.685 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91b22b33:::benchmark_data_smithi090_131278_object91:head# 2024-05-15T23:25:52.685 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.685 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91b5188e:::benchmark_data_smithi090_131278_object5173:head# 2024-05-15T23:25:52.685 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.686 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91bea12d:::benchmark_data_smithi090_131278_object7957:head# 2024-05-15T23:25:52.686 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.686 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91c10862:::benchmark_data_smithi090_131278_object2214:head# 2024-05-15T23:25:52.686 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.686 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91c437bc:::benchmark_data_smithi090_131278_object2093:head# 2024-05-15T23:25:52.686 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.687 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91cb2873:::benchmark_data_smithi090_131278_object659:head# 2024-05-15T23:25:52.687 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.687 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91ce2faa:::benchmark_data_smithi090_131278_object8431:head# 2024-05-15T23:25:52.687 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.688 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91ce345c:::benchmark_data_smithi090_131278_object5340:head# 2024-05-15T23:25:52.688 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.688 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91d81e23:::benchmark_data_smithi090_131278_object4136:head# 2024-05-15T23:25:52.688 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.690 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91de77b3:::benchmark_data_smithi090_131278_object779:head# 2024-05-15T23:25:52.690 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.692 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91e2113f:::benchmark_data_smithi090_131278_object264:head# 2024-05-15T23:25:52.692 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.692 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91e80464:::benchmark_data_smithi090_131278_object8227:head# 2024-05-15T23:25:52.692 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.692 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91ed3a68:::benchmark_data_smithi090_131278_object1520:head# 2024-05-15T23:25:52.693 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.693 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91f18ccb:::benchmark_data_smithi090_131278_object5347:head# 2024-05-15T23:25:52.693 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.693 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91fd3e1f:::benchmark_data_smithi090_131278_object9680:head# 2024-05-15T23:25:52.693 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.696 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:91ff00cb:::benchmark_data_smithi090_131278_object4514:head# 2024-05-15T23:25:52.697 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.697 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:920aafa3:::benchmark_data_smithi090_131278_object6651:head# 2024-05-15T23:25:52.697 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.697 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:920d03d9:::benchmark_data_smithi090_131278_object6911:head# 2024-05-15T23:25:52.698 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.698 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:921005e8:::benchmark_data_smithi090_131278_object8783:head# 2024-05-15T23:25:52.698 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.698 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9215a8a9:::benchmark_data_smithi090_131278_object1757:head# 2024-05-15T23:25:52.698 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.699 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9225e3d6:::benchmark_data_smithi090_131278_object9300:head# 2024-05-15T23:25:52.700 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.700 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:923e52fe:::benchmark_data_smithi090_131278_object5615:head# 2024-05-15T23:25:52.700 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.700 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92412d69:::benchmark_data_smithi090_131278_object2882:head# 2024-05-15T23:25:52.700 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.700 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92495281:::benchmark_data_smithi090_131278_object4113:head# 2024-05-15T23:25:52.700 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.700 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9249f851:::benchmark_data_smithi090_131278_object1269:head# 2024-05-15T23:25:52.700 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.700 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:924d6a0f:::benchmark_data_smithi090_131278_object8563:head# 2024-05-15T23:25:52.700 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.701 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:924e8b3d:::benchmark_data_smithi090_131278_object8871:head# 2024-05-15T23:25:52.701 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.704 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:925102b5:::benchmark_data_smithi090_131278_object9349:head# 2024-05-15T23:25:52.704 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.704 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92539e92:::benchmark_data_smithi090_131278_object6778:head# 2024-05-15T23:25:52.704 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.704 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9260f3ac:::benchmark_data_smithi090_131278_object3548:head# 2024-05-15T23:25:52.704 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.704 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:926573d2:::benchmark_data_smithi090_131278_object1817:head# 2024-05-15T23:25:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:926c4fdc:::benchmark_data_smithi090_131278_object9066:head# 2024-05-15T23:25:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:926f5d34:::benchmark_data_smithi090_131278_object3095:head# 2024-05-15T23:25:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9274824d:::benchmark_data_smithi090_131278_object4690:head# 2024-05-15T23:25:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92767b63:::benchmark_data_smithi090_131278_object3831:head# 2024-05-15T23:25:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:927ce598:::benchmark_data_smithi090_131278_object3015:head# 2024-05-15T23:25:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92843974:::benchmark_data_smithi090_131278_object9548:head# 2024-05-15T23:25:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:928ceb53:::benchmark_data_smithi090_131278_object6015:head# 2024-05-15T23:25:52.705 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.706 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9299c45a:::benchmark_data_smithi090_131278_object2888:head# 2024-05-15T23:25:52.706 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.706 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:929a3323:::benchmark_data_smithi090_131278_object499:head# 2024-05-15T23:25:52.706 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.706 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:929c54ec:::benchmark_data_smithi090_131278_object9329:head# 2024-05-15T23:25:52.707 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.707 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92a0122d:::benchmark_data_smithi090_131278_object1379:head# 2024-05-15T23:25:52.707 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.707 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92a2db3a:::benchmark_data_smithi090_131278_object6034:head# 2024-05-15T23:25:52.707 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.708 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92a37c7d:::benchmark_data_smithi090_131278_object4030:head# 2024-05-15T23:25:52.708 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.709 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92a6b734:::benchmark_data_smithi090_131278_object1707:head# 2024-05-15T23:25:52.709 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.709 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92ab8d4d:::benchmark_data_smithi090_131278_object3566:head# 2024-05-15T23:25:52.709 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.709 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92abac80:::benchmark_data_smithi090_131278_object5443:head# 2024-05-15T23:25:52.709 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.711 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92aea64d:::benchmark_data_smithi090_131278_object2061:head# 2024-05-15T23:25:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92b6ec25:::benchmark_data_smithi090_131278_object3581:head# 2024-05-15T23:25:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92bb1250:::benchmark_data_smithi090_131278_object7190:head# 2024-05-15T23:25:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.712 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92c0bf0b:::benchmark_data_smithi090_131278_object5639:head# 2024-05-15T23:25:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92c3bf64:::benchmark_data_smithi090_131278_object2666:head# 2024-05-15T23:25:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92c5174e:::benchmark_data_smithi090_131278_object8110:head# 2024-05-15T23:25:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92c941e8:::benchmark_data_smithi090_131278_object5546:head# 2024-05-15T23:25:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.713 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92cc3736:::benchmark_data_smithi090_131278_object7304:head# 2024-05-15T23:25:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92d2ba4d:::benchmark_data_smithi090_131278_object8420:head# 2024-05-15T23:25:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92d7442d:::benchmark_data_smithi090_131278_object2692:head# 2024-05-15T23:25:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92d85552:::benchmark_data_smithi090_131278_object5284:head# 2024-05-15T23:25:52.714 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.717 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92dd538e:::benchmark_data_smithi090_131278_object5480:head# 2024-05-15T23:25:52.717 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.717 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92de3097:::benchmark_data_smithi090_131278_object8388:head# 2024-05-15T23:25:52.717 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.717 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92e02dcb:::benchmark_data_smithi090_131278_object7260:head# 2024-05-15T23:25:52.718 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.718 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92e20416:::benchmark_data_smithi090_131278_object7367:head# 2024-05-15T23:25:52.718 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.718 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92f10b99:::benchmark_data_smithi090_131278_object4660:head# 2024-05-15T23:25:52.718 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.718 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92f5d7a7:::benchmark_data_smithi090_131278_object6357:head# 2024-05-15T23:25:52.718 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.718 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:92f9f3ab:::benchmark_data_smithi090_131278_object9610:head# 2024-05-15T23:25:52.718 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.718 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:930ff82f:::benchmark_data_smithi090_131278_object459:head# 2024-05-15T23:25:52.718 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.718 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9310d8e1:::benchmark_data_smithi090_131278_object3393:head# 2024-05-15T23:25:52.718 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.719 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:931470cf:::benchmark_data_smithi090_131278_object5043:head# 2024-05-15T23:25:52.719 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.719 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:931f1d9b:::benchmark_data_smithi090_131278_object8527:head# 2024-05-15T23:25:52.719 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.719 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:932912b6:::benchmark_data_smithi090_131278_object5991:head# 2024-05-15T23:25:52.719 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.720 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93373f4c:::benchmark_data_smithi090_131278_object2872:head# 2024-05-15T23:25:52.720 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.720 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:934a88f5:::benchmark_data_smithi090_131278_object6986:head# 2024-05-15T23:25:52.720 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.721 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:934c53cd:::benchmark_data_smithi090_131278_object3638:head# 2024-05-15T23:25:52.721 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.721 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93562d91:::benchmark_data_smithi090_131278_object9123:head# 2024-05-15T23:25:52.721 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.721 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9357fb1c:::benchmark_data_smithi090_131278_object6599:head# 2024-05-15T23:25:52.721 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.721 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:935a081f:::benchmark_data_smithi090_131278_object5572:head# 2024-05-15T23:25:52.721 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.721 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93617c94:::benchmark_data_smithi090_131278_object2908:head# 2024-05-15T23:25:52.721 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.721 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93705e0b:::benchmark_data_smithi090_131278_object1888:head# 2024-05-15T23:25:52.722 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.722 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:937995a2:::benchmark_data_smithi090_131278_object2597:head# 2024-05-15T23:25:52.723 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.723 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:937afca6:::benchmark_data_smithi090_131278_object7002:head# 2024-05-15T23:25:52.723 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.723 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9385fa7d:::benchmark_data_smithi090_131278_object5562:head# 2024-05-15T23:25:52.723 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.726 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:938c342a:::benchmark_data_smithi090_131278_object7172:head# 2024-05-15T23:25:52.727 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.727 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93a114de:::benchmark_data_smithi090_131278_object5112:head# 2024-05-15T23:25:52.727 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.727 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93b10e8c:::benchmark_data_smithi090_131278_object2400:head# 2024-05-15T23:25:52.728 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.728 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93b39d95:::benchmark_data_smithi090_131278_object2283:head# 2024-05-15T23:25:52.728 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.728 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93b91e2b:::benchmark_data_smithi090_131278_object6069:head# 2024-05-15T23:25:52.728 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.729 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93bde6c3:::benchmark_data_smithi090_131278_object3287:head# 2024-05-15T23:25:52.729 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.729 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93be36c0:::benchmark_data_smithi090_131278_object8960:head# 2024-05-15T23:25:52.729 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.733 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93c75023:::benchmark_data_smithi090_131278_object216:head# 2024-05-15T23:25:52.733 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.733 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93d48def:::benchmark_data_smithi090_131278_object1365:head# 2024-05-15T23:25:52.733 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.733 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93d5ae4b:::benchmark_data_smithi090_131278_object8463:head# 2024-05-15T23:25:52.733 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.733 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93dc08d5:::benchmark_data_smithi090_131278_object1141:head# 2024-05-15T23:25:52.733 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.733 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93e9402b:::benchmark_data_smithi090_131278_object1846:head# 2024-05-15T23:25:52.733 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.733 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93ea9ef7:::benchmark_data_smithi090_131278_object9510:head# 2024-05-15T23:25:52.733 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.734 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93f00e7a:::benchmark_data_smithi090_131278_object198:head# 2024-05-15T23:25:52.734 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.734 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:93fb60c0:::benchmark_data_smithi090_131278_object7118:head# 2024-05-15T23:25:52.734 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.734 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:940bce32:::benchmark_data_smithi090_131278_object4932:head# 2024-05-15T23:25:52.734 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.734 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9411779d:::benchmark_data_smithi090_131278_object2355:head# 2024-05-15T23:25:52.734 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.734 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94178e55:::benchmark_data_smithi090_131278_object2674:head# 2024-05-15T23:25:52.734 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.734 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:941c4580:::benchmark_data_smithi090_131278_object5027:head# 2024-05-15T23:25:52.734 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.734 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:941dbf10:::benchmark_data_smithi090_131278_object2748:head# 2024-05-15T23:25:52.735 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.735 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9420d63c:::benchmark_data_smithi090_131278_object1172:head# 2024-05-15T23:25:52.735 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.735 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94216fd7:::benchmark_data_smithi090_131278_object9294:head# 2024-05-15T23:25:52.735 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.736 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9436f45a:::benchmark_data_smithi090_131278_object7935:head# 2024-05-15T23:25:52.736 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.736 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94380a75:::benchmark_data_smithi090_131278_object7349:head# 2024-05-15T23:25:52.736 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.736 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9448bb75:::benchmark_data_smithi090_131278_object5214:head# 2024-05-15T23:25:52.736 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.736 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:944a240b:::benchmark_data_smithi090_131278_object5486:head# 2024-05-15T23:25:52.736 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.739 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94595d1e:::benchmark_data_smithi090_131278_object3463:head# 2024-05-15T23:25:52.739 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.739 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:945eb9a9:::benchmark_data_smithi090_131278_object808:head# 2024-05-15T23:25:52.739 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.739 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:946404cd:::benchmark_data_smithi090_131278_object2905:head# 2024-05-15T23:25:52.739 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.739 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:946a9386:::benchmark_data_smithi090_131278_object4509:head# 2024-05-15T23:25:52.739 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.740 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94728d39:::benchmark_data_smithi090_131278_object427:head# 2024-05-15T23:25:52.740 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.740 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94729876:::benchmark_data_smithi090_131278_object567:head# 2024-05-15T23:25:52.740 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.744 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:947bf817:::benchmark_data_smithi090_131278_object4599:head# 2024-05-15T23:25:52.744 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.747 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:947e2348:::benchmark_data_smithi090_131278_object8078:head# 2024-05-15T23:25:52.747 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.747 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94830482:::benchmark_data_smithi090_131278_object7618:head# 2024-05-15T23:25:52.747 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.747 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:948bcd5f:::benchmark_data_smithi090_131278_object610:head# 2024-05-15T23:25:52.747 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.747 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:948d5068:::benchmark_data_smithi090_131278_object4408:head# 2024-05-15T23:25:52.747 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.747 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:949e9b9b:::benchmark_data_smithi090_131278_object6709:head# 2024-05-15T23:25:52.747 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.747 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:949ede60:::benchmark_data_smithi090_131278_object3771:head# 2024-05-15T23:25:52.748 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.748 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94a06147:::benchmark_data_smithi090_131278_object8727:head# 2024-05-15T23:25:52.748 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.748 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94a0dfba:::benchmark_data_smithi090_131278_object4727:head# 2024-05-15T23:25:52.748 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.749 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94a37c66:::benchmark_data_smithi090_131278_object7683:head# 2024-05-15T23:25:52.749 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.749 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94a71d2d:::benchmark_data_smithi090_131278_object7867:head# 2024-05-15T23:25:52.749 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.749 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94a7dc86:::benchmark_data_smithi090_131278_object1033:head# 2024-05-15T23:25:52.749 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.749 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94a90cac:::benchmark_data_smithi090_131278_object7894:head# 2024-05-15T23:25:52.749 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.749 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94b36084:::benchmark_data_smithi090_131278_object2639:head# 2024-05-15T23:25:52.749 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.750 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94cab279:::benchmark_data_smithi090_131278_object4201:head# 2024-05-15T23:25:52.750 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.750 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94caf73e:::benchmark_data_smithi090_131278_object6520:head# 2024-05-15T23:25:52.750 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.751 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94d25761:::benchmark_data_smithi090_131278_object6692:head# 2024-05-15T23:25:52.751 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.751 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94d2bf2c:::benchmark_data_smithi090_131278_object8547:head# 2024-05-15T23:25:52.751 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.751 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94d87b74:::benchmark_data_smithi090_131278_object6346:head# 2024-05-15T23:25:52.751 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.751 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94dd8525:::benchmark_data_smithi090_131278_object4905:head# 2024-05-15T23:25:52.752 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.752 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:94def3b7:::benchmark_data_smithi090_131278_object7982:head# 2024-05-15T23:25:52.752 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.752 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95100197:::benchmark_data_smithi090_131278_object4002:head# 2024-05-15T23:25:52.752 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.752 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9512b18a:::benchmark_data_smithi090_131278_object9243:head# 2024-05-15T23:25:52.752 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.753 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95133a9f:::benchmark_data_smithi090_131278_object398:head# 2024-05-15T23:25:52.753 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.753 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9524aa99:::benchmark_data_smithi090_131278_object8042:head# 2024-05-15T23:25:52.753 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.753 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9535c58b:::benchmark_data_smithi090_131278_object3460:head# 2024-05-15T23:25:52.753 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.753 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95485594:::benchmark_data_smithi090_131278_object7356:head# 2024-05-15T23:25:52.753 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.753 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:954e08ab:::benchmark_data_smithi090_131278_object6963:head# 2024-05-15T23:25:52.753 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.753 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:955b60dd:::benchmark_data_smithi090_131278_object5739:head# 2024-05-15T23:25:52.754 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.754 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95657cff:::benchmark_data_smithi090_131278_object6750:head# 2024-05-15T23:25:52.754 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.754 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95699807:::benchmark_data_smithi090_131278_object6783:head# 2024-05-15T23:25:52.755 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.755 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:957bae68:::benchmark_data_smithi090_131278_object565:head# 2024-05-15T23:25:52.755 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.755 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9580cf5a:::benchmark_data_smithi090_131278_object4076:head# 2024-05-15T23:25:52.755 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.755 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95912ce2:::benchmark_data_smithi090_131278_object5864:head# 2024-05-15T23:25:52.755 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.755 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:959e77bb:::benchmark_data_smithi090_131278_object6533:head# 2024-05-15T23:25:52.755 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.756 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:52.757 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95b66e8d:::benchmark_data_smithi090_131278_object5715:head# 2024-05-15T23:25:52.757 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.757 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95b681bb:::benchmark_data_smithi090_131278_object6609:head# 2024-05-15T23:25:52.758 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.758 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95bb2622:::benchmark_data_smithi090_131278_object1913:head# 2024-05-15T23:25:52.758 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.758 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95bdcf61:::benchmark_data_smithi090_131278_object9144:head# 2024-05-15T23:25:52.758 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.758 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95be5218:::benchmark_data_smithi090_131278_object4890:head# 2024-05-15T23:25:52.758 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.759 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95c89615:::benchmark_data_smithi090_131278_object9325:head# 2024-05-15T23:25:52.759 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.762 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95cf6e84:::benchmark_data_smithi090_131278_object2138:head# 2024-05-15T23:25:52.762 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.762 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95d223be:::benchmark_data_smithi090_131278_object1538:head# 2024-05-15T23:25:52.762 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.762 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95d343fe:::benchmark_data_smithi090_131278_object6171:head# 2024-05-15T23:25:52.762 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.762 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95d53b90:::benchmark_data_smithi090_131278_object8855:head# 2024-05-15T23:25:52.763 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.763 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95d6d24d:::benchmark_data_smithi090_131278_object4650:head# 2024-05-15T23:25:52.763 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.764 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95d98694:::benchmark_data_smithi090_131278_object614:head# 2024-05-15T23:25:52.764 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.764 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95e78d34:::benchmark_data_smithi090_131278_object8841:head# 2024-05-15T23:25:52.764 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.764 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95e9dd8d:::benchmark_data_smithi090_131278_object3576:head# 2024-05-15T23:25:52.764 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.766 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95ea9beb:::benchmark_data_smithi090_131278_object8363:head# 2024-05-15T23:25:52.766 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.767 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95eafad2:::benchmark_data_smithi090_131278_object7812:head# 2024-05-15T23:25:52.767 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.767 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95f08883:::benchmark_data_smithi090_131278_object9172:head# 2024-05-15T23:25:52.767 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.771 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95f51205:::benchmark_data_smithi090_131278_object80:head# 2024-05-15T23:25:52.771 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.771 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95f92a9c:::benchmark_data_smithi090_131278_object8335:head# 2024-05-15T23:25:52.771 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.771 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95fccb1a:::benchmark_data_smithi090_131278_object2487:head# 2024-05-15T23:25:52.771 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.771 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:95ff2008:::benchmark_data_smithi090_131278_object4603:head# 2024-05-15T23:25:52.771 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.772 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9605858a:::benchmark_data_smithi090_131278_object1009:head# 2024-05-15T23:25:52.772 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.772 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96144309:::benchmark_data_smithi090_131278_object8644:head# 2024-05-15T23:25:52.772 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.773 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:961ae428:::benchmark_data_smithi090_131278_object932:head# 2024-05-15T23:25:52.773 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.773 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:961c1b01:::benchmark_data_smithi090_131278_object237:head# 2024-05-15T23:25:52.773 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.774 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9629e108:::benchmark_data_smithi090_131278_object2212:head# 2024-05-15T23:25:52.774 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.775 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:962e67cc:::benchmark_data_smithi090_131278_object4405:head# 2024-05-15T23:25:52.775 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.775 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:962f561a:::benchmark_data_smithi090_131278_object9408:head# 2024-05-15T23:25:52.775 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.775 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:963af2f9:::benchmark_data_smithi090_131278_object5084:head# 2024-05-15T23:25:52.775 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.776 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96413d1b:::benchmark_data_smithi090_131278_object4038:head# 2024-05-15T23:25:52.776 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.778 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9648a8a8:::benchmark_data_smithi090_131278_object7616:head# 2024-05-15T23:25:52.778 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.778 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:964a98c7:::benchmark_data_smithi090_131278_object5099:head# 2024-05-15T23:25:52.778 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.778 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:964c19b2:::benchmark_data_smithi090_131278_object451:head# 2024-05-15T23:25:52.778 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.778 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:964fc41e:::benchmark_data_smithi090_131278_object5748:head# 2024-05-15T23:25:52.779 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.779 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9652b361:::benchmark_data_smithi090_131278_object6771:head# 2024-05-15T23:25:52.779 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.779 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:965dab2e:::benchmark_data_smithi090_131278_object2711:head# 2024-05-15T23:25:52.779 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.779 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96610a16:::benchmark_data_smithi090_131278_object5364:head# 2024-05-15T23:25:52.779 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.780 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9661fb48:::benchmark_data_smithi090_131278_object7814:head# 2024-05-15T23:25:52.780 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.781 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:966c1829:::benchmark_data_smithi090_131278_object3777:head# 2024-05-15T23:25:52.781 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.781 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9670a470:::benchmark_data_smithi090_131278_object6195:head# 2024-05-15T23:25:52.781 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.781 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9673a44b:::benchmark_data_smithi090_131278_object6868:head# 2024-05-15T23:25:52.781 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.781 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:967bb389:::benchmark_data_smithi090_131278_object8214:head# 2024-05-15T23:25:52.781 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.781 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96882dd2:::benchmark_data_smithi090_131278_object5853:head# 2024-05-15T23:25:52.781 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.781 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:968a4470:::benchmark_data_smithi090_131278_object2347:head# 2024-05-15T23:25:52.781 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.782 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9692ff1e:::benchmark_data_smithi090_131278_object2828:head# 2024-05-15T23:25:52.782 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.782 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96a313a1:::benchmark_data_smithi090_131278_object4170:head# 2024-05-15T23:25:52.782 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.782 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96a554c2:::benchmark_data_smithi090_131278_object2603:head# 2024-05-15T23:25:52.782 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.783 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96a780fa:::benchmark_data_smithi090_131278_object5735:head# 2024-05-15T23:25:52.783 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.784 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96af4ab0:::benchmark_data_smithi090_131278_object1836:head# 2024-05-15T23:25:52.784 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.784 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96b1020a:::benchmark_data_smithi090_131278_object8770:head# 2024-05-15T23:25:52.784 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.787 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96bd8879:::benchmark_data_smithi090_131278_object6768:head# 2024-05-15T23:25:52.787 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.787 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96c139fb:::benchmark_data_smithi090_131278_object8640:head# 2024-05-15T23:25:52.787 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.788 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96c5f185:::benchmark_data_smithi090_131278_object6965:head# 2024-05-15T23:25:52.788 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.789 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96dc7f40:::benchmark_data_smithi090_131278_object2921:head# 2024-05-15T23:25:52.789 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.789 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96de17c8:::benchmark_data_smithi090_131278_object4055:head# 2024-05-15T23:25:52.789 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.789 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96e17973:::benchmark_data_smithi090_131278_object7498:head# 2024-05-15T23:25:52.789 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.789 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96e639b4:::benchmark_data_smithi090_131278_object2930:head# 2024-05-15T23:25:52.789 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.790 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96e668b5:::benchmark_data_smithi090_131278_object6826:head# 2024-05-15T23:25:52.790 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.790 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96eaeaff:::benchmark_data_smithi090_131278_object4332:head# 2024-05-15T23:25:52.790 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.790 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96f3fcfa:::benchmark_data_smithi090_131278_object2739:head# 2024-05-15T23:25:52.790 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.794 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96f5df00:::benchmark_data_smithi090_131278_object2665:head# 2024-05-15T23:25:52.794 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.794 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96f70686:::benchmark_data_smithi090_131278_object7771:head# 2024-05-15T23:25:52.794 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.794 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:96fe1032:::benchmark_data_smithi090_131278_object3477:head# 2024-05-15T23:25:52.794 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.794 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9709d4a1:::benchmark_data_smithi090_131278_object1868:head# 2024-05-15T23:25:52.794 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.794 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9710f278:::benchmark_data_smithi090_131278_object4747:head# 2024-05-15T23:25:52.795 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.795 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9711f604:::benchmark_data_smithi090_131278_object9377:head# 2024-05-15T23:25:52.795 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.795 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97182232:::benchmark_data_smithi090_131278_object9401:head# 2024-05-15T23:25:52.795 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.795 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9731637d:::benchmark_data_smithi090_131278_object1547:head# 2024-05-15T23:25:52.795 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.795 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9731ca9c:::benchmark_data_smithi090_131278_object6260:head# 2024-05-15T23:25:52.795 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.800 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97326c9b:::benchmark_data_smithi090_131278_object6918:head# 2024-05-15T23:25:52.800 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.801 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:973b9c04:::benchmark_data_smithi090_131278_object1491:head# 2024-05-15T23:25:52.801 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.801 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97430b30:::benchmark_data_smithi090_131278_object9612:head# 2024-05-15T23:25:52.802 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.802 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97507a79:::benchmark_data_smithi090_131278_object5769:head# 2024-05-15T23:25:52.802 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.803 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:975e87b5:::benchmark_data_smithi090_131278_object7488:head# 2024-05-15T23:25:52.803 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.803 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9763a040:::benchmark_data_smithi090_131278_object47:head# 2024-05-15T23:25:52.803 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.803 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:52.805 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97789906:::benchmark_data_smithi090_131278_object5571:head# 2024-05-15T23:25:52.805 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.805 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9780d85b:::benchmark_data_smithi090_131278_object998:head# 2024-05-15T23:25:52.805 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.806 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9785e778:::benchmark_data_smithi090_131278_object4809:head# 2024-05-15T23:25:52.806 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.806 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:979505ab:::benchmark_data_smithi090_131278_object5481:head# 2024-05-15T23:25:52.806 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.810 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:979ef2af:::benchmark_data_smithi090_131278_object4275:head# 2024-05-15T23:25:52.810 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.810 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97a3796e:::benchmark_data_smithi090_131278_object7548:head# 2024-05-15T23:25:52.810 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.810 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97b57061:::benchmark_data_smithi090_131278_object7269:head# 2024-05-15T23:25:52.810 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.810 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97bd84e7:::benchmark_data_smithi090_131278_object2565:head# 2024-05-15T23:25:52.810 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.810 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97c6d3d2:::benchmark_data_smithi090_131278_object6869:head# 2024-05-15T23:25:52.810 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.811 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97c8ed08:::benchmark_data_smithi090_131278_object3574:head# 2024-05-15T23:25:52.811 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.811 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97c9937f:::benchmark_data_smithi090_131278_object4214:head# 2024-05-15T23:25:52.812 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.812 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97d29d33:::benchmark_data_smithi090_131278_object3850:head# 2024-05-15T23:25:52.812 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.812 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97d458b4:::benchmark_data_smithi090_131278_object3369:head# 2024-05-15T23:25:52.812 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.812 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97db4286:::benchmark_data_smithi090_131278_object4226:head# 2024-05-15T23:25:52.812 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.812 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97df1b60:::benchmark_data_smithi090_131278_object4287:head# 2024-05-15T23:25:52.812 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.813 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97ec5df4:::benchmark_data_smithi090_131278_object2757:head# 2024-05-15T23:25:52.814 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.816 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97eccac4:::benchmark_data_smithi090_131278_object9673:head# 2024-05-15T23:25:52.816 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.816 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97ef60fd:::benchmark_data_smithi090_131278_object4967:head# 2024-05-15T23:25:52.816 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.816 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97f048cc:::benchmark_data_smithi090_131278_object1440:head# 2024-05-15T23:25:52.816 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.816 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97f0d4e1:::benchmark_data_smithi090_131278_object1249:head# 2024-05-15T23:25:52.817 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.817 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97f36121:::benchmark_data_smithi090_131278_object5802:head# 2024-05-15T23:25:52.817 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.817 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:97fc8009:::benchmark_data_smithi090_131278_object6485:head# 2024-05-15T23:25:52.817 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.817 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:980632c6:::benchmark_data_smithi090_131278_object6004:head# 2024-05-15T23:25:52.817 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.817 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:980a69b8:::benchmark_data_smithi090_131278_object1865:head# 2024-05-15T23:25:52.817 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.819 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98182a1a:::benchmark_data_smithi090_131278_object9392:head# 2024-05-15T23:25:52.819 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.820 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:981a65a5:::benchmark_data_smithi090_131278_object1639:head# 2024-05-15T23:25:52.820 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.820 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:981ec54e:::benchmark_data_smithi090_131278_object3290:head# 2024-05-15T23:25:52.820 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.821 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:982cc58f:::benchmark_data_smithi090_131278_object1097:head# 2024-05-15T23:25:52.821 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.824 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98409a2d:::benchmark_data_smithi090_131278_object5424:head# 2024-05-15T23:25:52.824 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.825 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9846f5d9:::benchmark_data_smithi090_131278_object5967:head# 2024-05-15T23:25:52.825 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.825 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98484084:::benchmark_data_smithi090_131278_object5393:head# 2024-05-15T23:25:52.825 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.825 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98492c2a:::benchmark_data_smithi090_131278_object741:head# 2024-05-15T23:25:52.825 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.826 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9853cffc:::benchmark_data_smithi090_131278_object5437:head# 2024-05-15T23:25:52.826 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.826 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9855c320:::benchmark_data_smithi090_131278_object4305:head# 2024-05-15T23:25:52.826 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.828 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:985ada70:::benchmark_data_smithi090_131278_object7271:head# 2024-05-15T23:25:52.828 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.828 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9861e8ff:::benchmark_data_smithi090_131278_object4653:head# 2024-05-15T23:25:52.829 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.829 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9862f134:::benchmark_data_smithi090_131278_object6852:head# 2024-05-15T23:25:52.829 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.829 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98653afd:::benchmark_data_smithi090_131278_object7686:head# 2024-05-15T23:25:52.829 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.829 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:987c8433:::benchmark_data_smithi090_131278_object762:head# 2024-05-15T23:25:52.829 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.830 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:987d659e:::benchmark_data_smithi090_131278_object1973:head# 2024-05-15T23:25:52.830 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.830 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9885e6ed:::benchmark_data_smithi090_131278_object2396:head# 2024-05-15T23:25:52.831 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.831 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:52.831 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:988a332b:::benchmark_data_smithi090_131278_object8522:head# 2024-05-15T23:25:52.831 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.832 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:988aa60f:::benchmark_data_smithi090_131278_object1058:head# 2024-05-15T23:25:52.832 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:52.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:52.834 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98904da5:::benchmark_data_smithi090_131278_object5846:head# 2024-05-15T23:25:52.834 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.834 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9896b9ff:::benchmark_data_smithi090_131278_object902:head# 2024-05-15T23:25:52.834 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.835 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9896db18:::benchmark_data_smithi090_131278_object9504:head# 2024-05-15T23:25:52.835 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.835 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98a9ee26:::benchmark_data_smithi090_131278_object6310:head# 2024-05-15T23:25:52.835 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.835 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98b2b02f:::benchmark_data_smithi090_131278_object2217:head# 2024-05-15T23:25:52.835 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.836 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98b4c635:::benchmark_data_smithi090_131278_object3873:head# 2024-05-15T23:25:52.836 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.837 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98b64984:::benchmark_data_smithi090_131278_object1652:head# 2024-05-15T23:25:52.837 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.837 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98ba2d77:::benchmark_data_smithi090_131278_object4327:head# 2024-05-15T23:25:52.837 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.837 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98d3b6ed:::benchmark_data_smithi090_131278_object8643:head# 2024-05-15T23:25:52.837 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.838 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98d7caae:::benchmark_data_smithi090_131278_object16:head# 2024-05-15T23:25:52.838 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.838 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98e4851f:::benchmark_data_smithi090_131278_object7226:head# 2024-05-15T23:25:52.838 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.838 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98e5082d:::benchmark_data_smithi090_131278_object6461:head# 2024-05-15T23:25:52.838 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.840 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98eac5bb:::benchmark_data_smithi090_131278_object6323:head# 2024-05-15T23:25:52.840 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.840 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98ecfcbe:::benchmark_data_smithi090_131278_object7594:head# 2024-05-15T23:25:52.840 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.840 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:98f71035:::benchmark_data_smithi090_131278_object910:head# 2024-05-15T23:25:52.840 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.841 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:990e50f8:::benchmark_data_smithi090_131278_object7714:head# 2024-05-15T23:25:52.841 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.841 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:990e574f:::benchmark_data_smithi090_131278_object7739:head# 2024-05-15T23:25:52.841 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.841 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9914d66d:::benchmark_data_smithi090_131278_object1132:head# 2024-05-15T23:25:52.841 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.841 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9926917f:::benchmark_data_smithi090_131278_object1189:head# 2024-05-15T23:25:52.841 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.843 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99360b88:::benchmark_data_smithi090_131278_object1696:head# 2024-05-15T23:25:52.843 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.843 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9939b578:::benchmark_data_smithi090_131278_object1077:head# 2024-05-15T23:25:52.844 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.844 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9951f709:::benchmark_data_smithi090_131278_object4525:head# 2024-05-15T23:25:52.844 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.844 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:995ad316:::benchmark_data_smithi090_131278_object6664:head# 2024-05-15T23:25:52.844 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.844 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:995fe52e:::benchmark_data_smithi090_131278_object2969:head# 2024-05-15T23:25:52.844 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.844 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99609ade:::benchmark_data_smithi090_131278_object9098:head# 2024-05-15T23:25:52.844 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.844 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9970e88a:::benchmark_data_smithi090_131278_object7737:head# 2024-05-15T23:25:52.844 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.844 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:997a17b2:::benchmark_data_smithi090_131278_object9014:head# 2024-05-15T23:25:52.844 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.845 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:997feeb5:::benchmark_data_smithi090_131278_object5536:head# 2024-05-15T23:25:52.845 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.845 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9982a1b6:::benchmark_data_smithi090_131278_object2721:head# 2024-05-15T23:25:52.845 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.845 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9994411d:::benchmark_data_smithi090_131278_object1421:head# 2024-05-15T23:25:52.845 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.845 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:999b5d18:::benchmark_data_smithi090_131278_object8612:head# 2024-05-15T23:25:52.845 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.845 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:999c2b33:::benchmark_data_smithi090_131278_object3810:head# 2024-05-15T23:25:52.845 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.846 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99a67de9:::benchmark_data_smithi090_131278_object5981:head# 2024-05-15T23:25:52.846 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.846 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99a68734:::benchmark_data_smithi090_131278_object8792:head# 2024-05-15T23:25:52.847 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.847 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99a78ecc:::benchmark_data_smithi090_131278_object7075:head# 2024-05-15T23:25:52.847 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.847 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99ac3645:::benchmark_data_smithi090_131278_object8782:head# 2024-05-15T23:25:52.847 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.847 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99adbb5d:::benchmark_data_smithi090_131278_object4452:head# 2024-05-15T23:25:52.847 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.847 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99afde95:::benchmark_data_smithi090_131278_object8322:head# 2024-05-15T23:25:52.847 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.847 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99b65fed:::benchmark_data_smithi090_131278_object375:head# 2024-05-15T23:25:52.847 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.847 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99c083c9:::benchmark_data_smithi090_131278_object2734:head# 2024-05-15T23:25:52.847 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.850 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99c5eff9:::benchmark_data_smithi090_131278_object7426:head# 2024-05-15T23:25:52.851 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.851 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99c9a10b:::benchmark_data_smithi090_131278_object532:head# 2024-05-15T23:25:52.851 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.851 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99d379b2:::benchmark_data_smithi090_131278_object8897:head# 2024-05-15T23:25:52.851 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.851 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99d3cbbd:::benchmark_data_smithi090_131278_object1441:head# 2024-05-15T23:25:52.851 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.851 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99d70b97:::benchmark_data_smithi090_131278_object4495:head# 2024-05-15T23:25:52.851 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.851 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99d94960:::benchmark_data_smithi090_131278_object7051:head# 2024-05-15T23:25:52.851 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.851 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99e0a2da:::benchmark_data_smithi090_131278_object6375:head# 2024-05-15T23:25:52.851 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.852 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99e20fdb:::benchmark_data_smithi090_131278_object8156:head# 2024-05-15T23:25:52.852 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.852 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99e6b100:::benchmark_data_smithi090_131278_object4475:head# 2024-05-15T23:25:52.852 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.853 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99ee8f0e:::benchmark_data_smithi090_131278_object3285:head# 2024-05-15T23:25:52.853 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.853 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99f0ad40:::benchmark_data_smithi090_131278_object6245:head# 2024-05-15T23:25:52.853 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.853 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99f655b6:::benchmark_data_smithi090_131278_object8661:head# 2024-05-15T23:25:52.853 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.853 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:99fee220:::benchmark_data_smithi090_131278_object1954:head# 2024-05-15T23:25:52.853 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.854 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a069c8f:::benchmark_data_smithi090_131278_object6462:head# 2024-05-15T23:25:52.854 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.854 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a06ea3c:::benchmark_data_smithi090_131278_object7111:head# 2024-05-15T23:25:52.854 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.854 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a080fb0:::benchmark_data_smithi090_131278_object6248:head# 2024-05-15T23:25:52.854 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.858 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a0945d9:::benchmark_data_smithi090_131278_object5657:head# 2024-05-15T23:25:52.858 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.858 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a0fe2e8:::benchmark_data_smithi090_131278_object6716:head# 2024-05-15T23:25:52.858 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.858 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a10925d:::benchmark_data_smithi090_131278_object7285:head# 2024-05-15T23:25:52.858 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.858 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a1d4d5e:::benchmark_data_smithi090_131278_object4862:head# 2024-05-15T23:25:52.858 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.858 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a226cb9:::benchmark_data_smithi090_131278_object353:head# 2024-05-15T23:25:52.859 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.859 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a35b31e:::benchmark_data_smithi090_131278_object6550:head# 2024-05-15T23:25:52.859 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.859 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a47b639:::benchmark_data_smithi090_131278_object8683:head# 2024-05-15T23:25:52.859 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.859 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a49abee:::benchmark_data_smithi090_131278_object9469:head# 2024-05-15T23:25:52.859 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.859 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a4d3c51:::benchmark_data_smithi090_131278_object3264:head# 2024-05-15T23:25:52.859 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.859 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a53f566:::benchmark_data_smithi090_131278_object3598:head# 2024-05-15T23:25:52.859 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.861 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a5c7670:::benchmark_data_smithi090_131278_object5081:head# 2024-05-15T23:25:52.861 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.861 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a5d62d8:::benchmark_data_smithi090_131278_object4870:head# 2024-05-15T23:25:52.861 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.861 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a653bbd:::benchmark_data_smithi090_131278_object7069:head# 2024-05-15T23:25:52.861 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.861 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a66f376:::benchmark_data_smithi090_131278_object6685:head# 2024-05-15T23:25:52.861 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.861 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a6c882c:::benchmark_data_smithi090_131278_object771:head# 2024-05-15T23:25:52.861 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.861 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a6f8001:::benchmark_data_smithi090_131278_object1381:head# 2024-05-15T23:25:52.862 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.862 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a7049ea:::benchmark_data_smithi090_131278_object6751:head# 2024-05-15T23:25:52.862 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.862 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a755e85:::benchmark_data_smithi090_131278_object8120:head# 2024-05-15T23:25:52.862 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.862 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a795b3a:::benchmark_data_smithi090_131278_object4186:head# 2024-05-15T23:25:52.862 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.862 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a7ef7c6:::benchmark_data_smithi090_131278_object6154:head# 2024-05-15T23:25:52.862 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.862 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a8058d8:::benchmark_data_smithi090_131278_object8915:head# 2024-05-15T23:25:52.862 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.863 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a8452a8:::benchmark_data_smithi090_131278_object4754:head# 2024-05-15T23:25:52.863 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.863 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a87f556:::benchmark_data_smithi090_131278_object387:head# 2024-05-15T23:25:52.864 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.864 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9a92c87c:::benchmark_data_smithi090_131278_object7141:head# 2024-05-15T23:25:52.864 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.864 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9aa6d52c:::benchmark_data_smithi090_131278_object7327:head# 2024-05-15T23:25:52.864 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.864 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9aa71a6e:::benchmark_data_smithi090_131278_object3419:head# 2024-05-15T23:25:52.864 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.864 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9aac9854:::benchmark_data_smithi090_131278_object857:head# 2024-05-15T23:25:52.864 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.864 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ab71579:::benchmark_data_smithi090_131278_object881:head# 2024-05-15T23:25:52.865 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.865 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9abbfd44:::benchmark_data_smithi090_131278_object4528:head# 2024-05-15T23:25:52.865 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.865 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9abcc2dd:::benchmark_data_smithi090_131278_object1535:head# 2024-05-15T23:25:52.865 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.865 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9abf4bad:::benchmark_data_smithi090_131278_object7579:head# 2024-05-15T23:25:52.866 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.866 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ac2b86d:::benchmark_data_smithi090_131278_object7079:head# 2024-05-15T23:25:52.866 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.866 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ac7c235:::benchmark_data_smithi090_131278_object1811:head# 2024-05-15T23:25:52.866 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.866 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ac7e4c1:::benchmark_data_smithi090_131278_object8124:head# 2024-05-15T23:25:52.866 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.866 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ace30f1:::benchmark_data_smithi090_131278_object5527:head# 2024-05-15T23:25:52.866 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.866 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ad9c434:::benchmark_data_smithi090_131278_object8516:head# 2024-05-15T23:25:52.866 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.866 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ae487d7:::benchmark_data_smithi090_131278_object3857:head# 2024-05-15T23:25:52.866 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.867 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9aeb4312:::benchmark_data_smithi090_131278_object8141:head# 2024-05-15T23:25:52.867 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.867 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9af269c9:::benchmark_data_smithi090_131278_object4042:head# 2024-05-15T23:25:52.867 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.869 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9af3fa0a:::benchmark_data_smithi090_131278_object2519:head# 2024-05-15T23:25:52.869 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.869 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9af6aaaa:::benchmark_data_smithi090_131278_object2371:head# 2024-05-15T23:25:52.869 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.869 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b066dcc:::benchmark_data_smithi090_131278_object1754:head# 2024-05-15T23:25:52.870 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.871 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b0b1faa:::benchmark_data_smithi090_131278_object7747:head# 2024-05-15T23:25:52.871 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.871 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b0ce0e1:::benchmark_data_smithi090_131278_object903:head# 2024-05-15T23:25:52.871 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.871 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b1576db:::benchmark_data_smithi090_131278_object9560:head# 2024-05-15T23:25:52.871 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.871 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b22a059:::benchmark_data_smithi090_131278_object8713:head# 2024-05-15T23:25:52.872 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.872 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b25236f:::benchmark_data_smithi090_131278_object2107:head# 2024-05-15T23:25:52.872 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.872 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b335d17:::benchmark_data_smithi090_131278_object4110:head# 2024-05-15T23:25:52.872 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.872 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b34622b:::benchmark_data_smithi090_131278_object8057:head# 2024-05-15T23:25:52.872 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.872 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b354a80:::benchmark_data_smithi090_131278_object7129:head# 2024-05-15T23:25:52.873 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.873 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b39771e:::benchmark_data_smithi090_131278_object1683:head# 2024-05-15T23:25:52.873 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.874 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b3ae3c3:::benchmark_data_smithi090_131278_object4584:head# 2024-05-15T23:25:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b3dc58c:::benchmark_data_smithi090_131278_object7060:head# 2024-05-15T23:25:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b3f59cf:::benchmark_data_smithi090_131278_object7496:head# 2024-05-15T23:25:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b5e7ce8:::benchmark_data_smithi090_131278_object8406:head# 2024-05-15T23:25:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b65a6ff:::benchmark_data_smithi090_131278_object6309:head# 2024-05-15T23:25:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b6ce2b5:::benchmark_data_smithi090_131278_object6342:head# 2024-05-15T23:25:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b6db155:::benchmark_data_smithi090_131278_object3494:head# 2024-05-15T23:25:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.876 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b7f9a72:::benchmark_data_smithi090_131278_object6889:head# 2024-05-15T23:25:52.876 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.876 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b88417d:::benchmark_data_smithi090_131278_object1776:head# 2024-05-15T23:25:52.876 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.876 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b8a8520:::benchmark_data_smithi090_131278_object6110:head# 2024-05-15T23:25:52.876 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.876 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b93acfe:::benchmark_data_smithi090_131278_object9044:head# 2024-05-15T23:25:52.876 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.877 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b95535b:::benchmark_data_smithi090_131278_object1064:head# 2024-05-15T23:25:52.877 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.878 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9b969468:::benchmark_data_smithi090_131278_object3593:head# 2024-05-15T23:25:52.878 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.878 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ba10268:::benchmark_data_smithi090_131278_object1743:head# 2024-05-15T23:25:52.878 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.878 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ba20efd:::benchmark_data_smithi090_131278_object8533:head# 2024-05-15T23:25:52.878 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.878 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ba3f132:::benchmark_data_smithi090_131278_object5978:head# 2024-05-15T23:25:52.878 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.878 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9baaf4f5:::benchmark_data_smithi090_131278_object2046:head# 2024-05-15T23:25:52.878 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.878 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9badb61b:::benchmark_data_smithi090_131278_object1621:head# 2024-05-15T23:25:52.878 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.878 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9bb14c1a:::benchmark_data_smithi090_131278_object4526:head# 2024-05-15T23:25:52.878 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.879 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9bba2eab:::benchmark_data_smithi090_131278_object7254:head# 2024-05-15T23:25:52.879 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.880 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9bded8f9:::benchmark_data_smithi090_131278_object181:head# 2024-05-15T23:25:52.881 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.881 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9be087b9:::benchmark_data_smithi090_131278_object853:head# 2024-05-15T23:25:52.881 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.881 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9be65b33:::benchmark_data_smithi090_131278_object7563:head# 2024-05-15T23:25:52.881 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.881 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9bfbd328:::benchmark_data_smithi090_131278_object1709:head# 2024-05-15T23:25:52.881 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.881 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c033f65:::benchmark_data_smithi090_131278_object9537:head# 2024-05-15T23:25:52.881 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.881 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c12383d:::benchmark_data_smithi090_131278_object8286:head# 2024-05-15T23:25:52.881 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.881 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c198c0e:::benchmark_data_smithi090_131278_object5539:head# 2024-05-15T23:25:52.881 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.881 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c1b617a:::benchmark_data_smithi090_131278_object5560:head# 2024-05-15T23:25:52.882 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.882 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c229728:::benchmark_data_smithi090_131278_object3838:head# 2024-05-15T23:25:52.882 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.882 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c233228:::benchmark_data_smithi090_131278_object5474:head# 2024-05-15T23:25:52.882 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.882 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c2634c4:::benchmark_data_smithi090_131278_object6276:head# 2024-05-15T23:25:52.882 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.882 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c270aad:::benchmark_data_smithi090_131278_object7837:head# 2024-05-15T23:25:52.882 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.884 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c2f620f:::benchmark_data_smithi090_131278_object6604:head# 2024-05-15T23:25:52.885 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.885 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c364d3b:::benchmark_data_smithi090_131278_object8833:head# 2024-05-15T23:25:52.885 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.885 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c4921ca:::benchmark_data_smithi090_131278_object5482:head# 2024-05-15T23:25:52.885 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.885 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c54027b:::benchmark_data_smithi090_131278_object3882:head# 2024-05-15T23:25:52.885 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.885 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c605c2d:::benchmark_data_smithi090_131278_object3860:head# 2024-05-15T23:25:52.885 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.885 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c7ea4f8:::benchmark_data_smithi090_131278_object1211:head# 2024-05-15T23:25:52.885 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.885 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c7fec58:::benchmark_data_smithi090_131278_object5385:head# 2024-05-15T23:25:52.885 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.885 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c87246c:::benchmark_data_smithi090_131278_object7405:head# 2024-05-15T23:25:52.885 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.886 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c8f3c46:::benchmark_data_smithi090_131278_object1704:head# 2024-05-15T23:25:52.886 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.886 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c9094ce:::benchmark_data_smithi090_131278_object5626:head# 2024-05-15T23:25:52.886 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.886 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c94417b:::benchmark_data_smithi090_131278_object831:head# 2024-05-15T23:25:52.886 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.886 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9c9d9d5a:::benchmark_data_smithi090_131278_object2382:head# 2024-05-15T23:25:52.886 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.886 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9cad4358:::benchmark_data_smithi090_131278_object8210:head# 2024-05-15T23:25:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9cc01491:::benchmark_data_smithi090_131278_object1255:head# 2024-05-15T23:25:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9cd26e9f:::benchmark_data_smithi090_131278_object6804:head# 2024-05-15T23:25:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9cd53de7:::benchmark_data_smithi090_131278_object4596:head# 2024-05-15T23:25:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9cdeebe5:::benchmark_data_smithi090_131278_object3068:head# 2024-05-15T23:25:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ce5f6a2:::benchmark_data_smithi090_131278_object9330:head# 2024-05-15T23:25:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9cf15f57:::benchmark_data_smithi090_131278_object7645:head# 2024-05-15T23:25:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9cf1e8bb:::benchmark_data_smithi090_131278_object2966:head# 2024-05-15T23:25:52.887 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.889 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9cf3287a:::benchmark_data_smithi090_131278_object6525:head# 2024-05-15T23:25:52.889 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.889 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9cf761ed:::benchmark_data_smithi090_131278_object9440:head# 2024-05-15T23:25:52.889 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.889 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d05eb0c:::benchmark_data_smithi090_131278_object1638:head# 2024-05-15T23:25:52.890 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.890 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d0ed73f:::benchmark_data_smithi090_131278_object1187:head# 2024-05-15T23:25:52.890 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.890 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d13c342:::benchmark_data_smithi090_131278_object3222:head# 2024-05-15T23:25:52.890 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.890 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d1c3dc6:::benchmark_data_smithi090_131278_object1435:head# 2024-05-15T23:25:52.890 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.890 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d24ab0a:::benchmark_data_smithi090_131278_object1814:head# 2024-05-15T23:25:52.890 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.890 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d2902ac:::benchmark_data_smithi090_131278_object307:head# 2024-05-15T23:25:52.891 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.891 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d2db021:::benchmark_data_smithi090_131278_object2076:head# 2024-05-15T23:25:52.891 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.891 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d2e511c:::benchmark_data_smithi090_131278_object368:head# 2024-05-15T23:25:52.891 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.892 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d35fa6d:::benchmark_data_smithi090_131278_object6514:head# 2024-05-15T23:25:52.892 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.892 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d363a9f:::benchmark_data_smithi090_131278_object3379:head# 2024-05-15T23:25:52.892 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.893 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d399153:::benchmark_data_smithi090_131278_object2662:head# 2024-05-15T23:25:52.893 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.893 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d3eb19c:::benchmark_data_smithi090_131278_object2916:head# 2024-05-15T23:25:52.893 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.893 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d3ff959:::benchmark_data_smithi090_131278_object1095:head# 2024-05-15T23:25:52.893 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.893 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d404fec:::benchmark_data_smithi090_131278_object3317:head# 2024-05-15T23:25:52.893 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.893 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d493b23:::benchmark_data_smithi090_131278_object474:head# 2024-05-15T23:25:52.893 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.893 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d53d3de:::benchmark_data_smithi090_131278_object721:head# 2024-05-15T23:25:52.893 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.893 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d6d0d8d:::benchmark_data_smithi090_131278_object8642:head# 2024-05-15T23:25:52.893 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.893 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d7215b8:::benchmark_data_smithi090_131278_object2786:head# 2024-05-15T23:25:52.894 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.894 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d76ba88:::benchmark_data_smithi090_131278_object1274:head# 2024-05-15T23:25:52.894 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.896 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d775aa8:::benchmark_data_smithi090_131278_object8447:head# 2024-05-15T23:25:52.896 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.896 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d78807f:::benchmark_data_smithi090_131278_object4898:head# 2024-05-15T23:25:52.896 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.896 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d7a46bf:::benchmark_data_smithi090_131278_object8093:head# 2024-05-15T23:25:52.896 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.896 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d7d6c50:::benchmark_data_smithi090_131278_object5568:head# 2024-05-15T23:25:52.896 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.896 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d861ffa:::benchmark_data_smithi090_131278_object4677:head# 2024-05-15T23:25:52.896 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d8a5089:::benchmark_data_smithi090_131278_object4377:head# 2024-05-15T23:25:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d914513:::benchmark_data_smithi090_131278_object7205:head# 2024-05-15T23:25:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d930859:::benchmark_data_smithi090_131278_object6136:head# 2024-05-15T23:25:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9d9f1239:::benchmark_data_smithi090_131278_object1948:head# 2024-05-15T23:25:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9da640f0:::benchmark_data_smithi090_131278_object4243:head# 2024-05-15T23:25:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9dac740d:::benchmark_data_smithi090_131278_object2457:head# 2024-05-15T23:25:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9db02ac7:::benchmark_data_smithi090_131278_object6924:head# 2024-05-15T23:25:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.898 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9dbf7d54:::benchmark_data_smithi090_131278_object8825:head# 2024-05-15T23:25:52.898 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.898 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ddb030e:::benchmark_data_smithi090_131278_object5358:head# 2024-05-15T23:25:52.898 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.903 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9de2ac8d:::benchmark_data_smithi090_131278_object7475:head# 2024-05-15T23:25:52.904 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.904 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9de3fbc5:::benchmark_data_smithi090_131278_object9340:head# 2024-05-15T23:25:52.904 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.904 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9deb845a:::benchmark_data_smithi090_131278_object9608:head# 2024-05-15T23:25:52.904 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.904 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9dec4f3f:::benchmark_data_smithi090_131278_object4354:head# 2024-05-15T23:25:52.904 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.904 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e086b1b:::benchmark_data_smithi090_131278_object5370:head# 2024-05-15T23:25:52.904 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.904 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e0d81b3:::benchmark_data_smithi090_131278_object4866:head# 2024-05-15T23:25:52.904 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.904 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e1f5c08:::benchmark_data_smithi090_131278_object9454:head# 2024-05-15T23:25:52.904 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.905 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e2411e8:::benchmark_data_smithi090_131278_object7013:head# 2024-05-15T23:25:52.905 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.905 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e290cec:::benchmark_data_smithi090_131278_object6125:head# 2024-05-15T23:25:52.905 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.905 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e3692dd:::benchmark_data_smithi090_131278_object4585:head# 2024-05-15T23:25:52.905 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.905 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e424afd:::benchmark_data_smithi090_131278_object8500:head# 2024-05-15T23:25:52.905 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.905 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e4a098e:::benchmark_data_smithi090_131278_object3531:head# 2024-05-15T23:25:52.905 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.905 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e4b76da:::benchmark_data_smithi090_131278_object2469:head# 2024-05-15T23:25:52.905 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.905 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e504142:::benchmark_data_smithi090_131278_object2595:head# 2024-05-15T23:25:52.906 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.906 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e57314f:::benchmark_data_smithi090_131278_object6976:head# 2024-05-15T23:25:52.906 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.906 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e629404:::benchmark_data_smithi090_131278_object6947:head# 2024-05-15T23:25:52.906 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.906 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e67b675:::benchmark_data_smithi090_131278_object849:head# 2024-05-15T23:25:52.906 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.906 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e68c340:::benchmark_data_smithi090_131278_object4689:head# 2024-05-15T23:25:52.906 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.906 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e698bf6:::benchmark_data_smithi090_131278_object5260:head# 2024-05-15T23:25:52.906 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.906 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e6d0702:::benchmark_data_smithi090_131278_object2956:head# 2024-05-15T23:25:52.906 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.907 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e6f8539:::benchmark_data_smithi090_131278_object9426:head# 2024-05-15T23:25:52.907 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.907 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e759594:::benchmark_data_smithi090_131278_object9480:head# 2024-05-15T23:25:52.907 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.907 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e77c5e1:::benchmark_data_smithi090_131278_object5750:head# 2024-05-15T23:25:52.907 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.907 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e8bf076:::benchmark_data_smithi090_131278_object3537:head# 2024-05-15T23:25:52.907 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.907 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e8ff872:::benchmark_data_smithi090_131278_object5949:head# 2024-05-15T23:25:52.907 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.907 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e9024c8:::benchmark_data_smithi090_131278_object8567:head# 2024-05-15T23:25:52.907 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.907 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e91597e:::benchmark_data_smithi090_131278_object3965:head# 2024-05-15T23:25:52.907 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.907 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e953a21:::benchmark_data_smithi090_131278_object1023:head# 2024-05-15T23:25:52.908 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.908 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e97951e:::benchmark_data_smithi090_131278_object9166:head# 2024-05-15T23:25:52.908 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.908 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9e987fbb:::benchmark_data_smithi090_131278_object2346:head# 2024-05-15T23:25:52.908 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.908 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ea28cb5:::benchmark_data_smithi090_131278_object4364:head# 2024-05-15T23:25:52.908 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.908 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ea4d97b:::benchmark_data_smithi090_131278_object9546:head# 2024-05-15T23:25:52.908 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.917 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ea5efbd:::benchmark_data_smithi090_131278_object8270:head# 2024-05-15T23:25:52.917 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.917 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ea76e82:::benchmark_data_smithi090_131278_object6897:head# 2024-05-15T23:25:52.917 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.917 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9eb28238:::benchmark_data_smithi090_131278_object4399:head# 2024-05-15T23:25:52.917 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.917 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ebc5b20:::benchmark_data_smithi090_131278_object8126:head# 2024-05-15T23:25:52.917 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.917 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ebc9534:::benchmark_data_smithi090_131278_object3916:head# 2024-05-15T23:25:52.917 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.918 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ec5dff4:::benchmark_data_smithi090_131278_object5363:head# 2024-05-15T23:25:52.918 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.918 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ecc038a:::benchmark_data_smithi090_131278_object5427:head# 2024-05-15T23:25:52.918 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.918 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ed6099d:::benchmark_data_smithi090_131278_object5432:head# 2024-05-15T23:25:52.918 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.918 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ee0c233:::benchmark_data_smithi090_131278_object4117:head# 2024-05-15T23:25:52.918 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.918 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ee4219c:::benchmark_data_smithi090_131278_object3975:head# 2024-05-15T23:25:52.919 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.919 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ee5a36e:::benchmark_data_smithi090_131278_object1883:head# 2024-05-15T23:25:52.919 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.919 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ef9dd1b:::benchmark_data_smithi090_131278_object3268:head# 2024-05-15T23:25:52.919 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.919 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f041bda:::benchmark_data_smithi090_131278_object7296:head# 2024-05-15T23:25:52.920 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.920 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f05093f:::benchmark_data_smithi090_131278_object5554:head# 2024-05-15T23:25:52.920 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.920 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f07d819:::benchmark_data_smithi090_131278_object3470:head# 2024-05-15T23:25:52.920 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.921 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f0d5197:::benchmark_data_smithi090_131278_object6766:head# 2024-05-15T23:25:52.921 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.921 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f1e3d01:::benchmark_data_smithi090_131278_object9276:head# 2024-05-15T23:25:52.921 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.921 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f3a85a0:::benchmark_data_smithi090_131278_object8339:head# 2024-05-15T23:25:52.921 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.922 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f3ba423:::benchmark_data_smithi090_131278_object8317:head# 2024-05-15T23:25:52.922 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.922 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f3d7ecd:::benchmark_data_smithi090_131278_object3799:head# 2024-05-15T23:25:52.922 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.922 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f40333d:::benchmark_data_smithi090_131278_object3513:head# 2024-05-15T23:25:52.923 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.923 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f4b8b94:::benchmark_data_smithi090_131278_object1961:head# 2024-05-15T23:25:52.923 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.923 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f4f29a0:::benchmark_data_smithi090_131278_object1982:head# 2024-05-15T23:25:52.923 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.923 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f532c22:::benchmark_data_smithi090_131278_object5706:head# 2024-05-15T23:25:52.924 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.924 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f541f69:::benchmark_data_smithi090_131278_object3022:head# 2024-05-15T23:25:52.924 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.924 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f5c5602:::benchmark_data_smithi090_131278_object2287:head# 2024-05-15T23:25:52.924 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.924 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f6028d4:::benchmark_data_smithi090_131278_object9000:head# 2024-05-15T23:25:52.925 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.925 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f61bde5:::benchmark_data_smithi090_131278_object1385:head# 2024-05-15T23:25:52.925 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.925 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f63abda:::benchmark_data_smithi090_131278_object5148:head# 2024-05-15T23:25:52.925 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.925 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f64899a:::benchmark_data_smithi090_131278_object8437:head# 2024-05-15T23:25:52.926 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.926 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f69d06a:::benchmark_data_smithi090_131278_object2375:head# 2024-05-15T23:25:52.926 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.926 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f6a49f2:::benchmark_data_smithi090_131278_object9250:head# 2024-05-15T23:25:52.926 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.926 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f70c94a:::benchmark_data_smithi090_131278_object73:head# 2024-05-15T23:25:52.927 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.927 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f76f4e1:::benchmark_data_smithi090_131278_object1694:head# 2024-05-15T23:25:52.927 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.927 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f78091b:::benchmark_data_smithi090_131278_object2114:head# 2024-05-15T23:25:52.927 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.927 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f7bd4dd:::benchmark_data_smithi090_131278_object6218:head# 2024-05-15T23:25:52.927 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.928 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f7ce14d:::benchmark_data_smithi090_131278_object9616:head# 2024-05-15T23:25:52.928 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.928 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f82de16:::benchmark_data_smithi090_131278_object3098:head# 2024-05-15T23:25:52.928 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.928 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f83b75d:::benchmark_data_smithi090_131278_object8146:head# 2024-05-15T23:25:52.928 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.929 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f84b8b6:::benchmark_data_smithi090_131278_object9624:head# 2024-05-15T23:25:52.929 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.929 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f86552d:::benchmark_data_smithi090_131278_object6011:head# 2024-05-15T23:25:52.929 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.929 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f8aa762:::benchmark_data_smithi090_131278_object4925:head# 2024-05-15T23:25:52.929 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.929 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f914907:::benchmark_data_smithi090_131278_object5889:head# 2024-05-15T23:25:52.929 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.930 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f9686d3:::benchmark_data_smithi090_131278_object8734:head# 2024-05-15T23:25:52.930 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.930 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9f998076:::benchmark_data_smithi090_131278_object4940:head# 2024-05-15T23:25:52.930 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.930 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9fb50385:::benchmark_data_smithi090_131278_object6802:head# 2024-05-15T23:25:52.930 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.930 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9fb9cc48:::benchmark_data_smithi090_131278_object4103:head# 2024-05-15T23:25:52.931 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.931 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9fbd50f2:::benchmark_data_smithi090_131278_object2094:head# 2024-05-15T23:25:52.931 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.931 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9fca588a:::benchmark_data_smithi090_131278_object8397:head# 2024-05-15T23:25:52.931 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.931 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9fccdcec:::benchmark_data_smithi090_131278_object5179:head# 2024-05-15T23:25:52.931 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.932 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9fce1e00:::benchmark_data_smithi090_131278_object7745:head# 2024-05-15T23:25:52.932 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.932 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9fd39d23:::benchmark_data_smithi090_131278_object8454:head# 2024-05-15T23:25:52.932 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.932 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9fdbf0ad:::benchmark_data_smithi090_131278_object1014:head# 2024-05-15T23:25:52.932 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.935 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9fe1c7cf:::benchmark_data_smithi090_131278_object5391:head# 2024-05-15T23:25:52.936 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.936 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9fe91d49:::benchmark_data_smithi090_131278_object4062:head# 2024-05-15T23:25:52.936 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.936 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9feca161:::benchmark_data_smithi090_131278_object5777:head# 2024-05-15T23:25:52.936 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.936 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9fee0593:::benchmark_data_smithi090_131278_object6191:head# 2024-05-15T23:25:52.936 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.936 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ff1d1ae:::benchmark_data_smithi090_131278_object6519:head# 2024-05-15T23:25:52.937 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.937 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ff22a19:::benchmark_data_smithi090_131278_object2502:head# 2024-05-15T23:25:52.937 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.937 INFO:teuthology.orchestra.run.smithi090.stdout:Write #6:9ff39dbd:::benchmark_data_smithi090_131278_object7610:head# 2024-05-15T23:25:52.937 INFO:teuthology.orchestra.run.smithi090.stdout:snapset 0=[]:{} 2024-05-15T23:25:52.937 INFO:teuthology.orchestra.run.smithi090.stdout:write_pg epoch 484 info 6.9( v 484'602 (0'0,484'602] local-lis/les=477/478 n=602 ec=477/477 lis/c=477/477 les/c/f=478/478/0 sis=477) 2024-05-15T23:25:52.937 INFO:teuthology.orchestra.run.smithi090.stdout:Import successful 2024-05-15T23:25:52.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:25:52.974 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:53.086 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:25:53.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:53.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:25:53.192 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:25:53.210 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:53 smithi084 ceph-mon[39784]: osdmap e499: 12 total, 11 up, 8 in 2024-05-15T23:25:53.210 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:53 smithi084 ceph-mon[39784]: pgmap v1336: 17 pgs: 17 active+clean; 3.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 196 MiB/s wr, 3.13k op/s 2024-05-15T23:25:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:53.269 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:25:53.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:53.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:25:53.376 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:25:53.409 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:53.453 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:53 smithi090 ceph-mon[44434]: osdmap e499: 12 total, 11 up, 8 in 2024-05-15T23:25:53.453 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:53 smithi090 ceph-mon[44434]: pgmap v1336: 17 pgs: 17 active+clean; 3.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 196 MiB/s wr, 3.13k op/s 2024-05-15T23:25:53.456 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:25:53.459 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 46 16 67929 67913 92.2506 137.625 0.00580771 0.0108388 2024-05-15T23:25:53.564 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:25:53.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:53.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:25:53.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:53 smithi088 ceph-mon[44699]: osdmap e499: 12 total, 11 up, 8 in 2024-05-15T23:25:53.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:53 smithi088 ceph-mon[44699]: pgmap v1336: 17 pgs: 17 active+clean; 3.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 196 MiB/s wr, 3.13k op/s 2024-05-15T23:25:53.667 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:25:53.703 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:54.058 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:25:54.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:54.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:25:54.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:25:54.212 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:54.459 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 47 16 70177 70161 93.2768 140.5 0.00621543 0.0107196 2024-05-15T23:25:54.989 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:55.247 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:55 smithi084 ceph-mon[39784]: pgmap v1337: 17 pgs: 17 active+clean; 3.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-05-15T23:25:55.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/exp.6.9.8 2024-05-15T23:25:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:25:55.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:55.277 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:25:55.323 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-05-15T23:25:55.323 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-15T23:25:55.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:25:55.378 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:25:55.413 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:55 smithi090 ceph-mon[44434]: pgmap v1337: 17 pgs: 17 active+clean; 3.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-05-15T23:25:55.415 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:55.447 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:25:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:55.451 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:25:55.459 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 48 16 72277 72261 94.0675 131.25 0.00381053 0.0106295 2024-05-15T23:25:55.552 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:55 smithi088 ceph-mon[44699]: pgmap v1337: 17 pgs: 17 active+clean; 3.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-05-15T23:25:55.604 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:55.960 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:55.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:55.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:56.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:25:56.096 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:56.123 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:25:56.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:56.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:25:56.129 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:25:56.229 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:25:56.256 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/574916709' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:25:56.256 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:56 smithi084 ceph-mon[39784]: from='client.26389 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:25:56.272 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:56.459 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 49 16 74446 74430 94.914 135.562 0.00675789 0.0105345 2024-05-15T23:25:56.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/574916709' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:25:56.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:56 smithi090 ceph-mon[44434]: from='client.26389 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:25:56.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/574916709' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:25:56.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:56 smithi088 ceph-mon[44699]: from='client.26389 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:25:57.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:57 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:25:57.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:57 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:25:57.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:57 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:25:57.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:57 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:25:57.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:57 smithi084 ceph-mon[39784]: from='client.26389 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:25:57.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:57 smithi084 ceph-mon[39784]: osdmap e500: 12 total, 11 up, 8 in 2024-05-15T23:25:57.304 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:57 smithi084 ceph-mon[39784]: pgmap v1339: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 177 MiB/s wr, 2.83k op/s 2024-05-15T23:25:57.304 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:57 smithi084 ceph-mon[39784]: osdmap e501: 12 total, 11 up, 8 in 2024-05-15T23:25:57.460 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 50 16 76528 76512 95.6167 130.125 0.0118166 0.0104564 2024-05-15T23:25:57.526 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:25:57.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:57.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:25:57.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:57 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:25:57.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:57 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:25:57.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:57 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:25:57.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:57 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:25:57.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:57 smithi090 ceph-mon[44434]: from='client.26389 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:25:57.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:57 smithi090 ceph-mon[44434]: osdmap e500: 12 total, 11 up, 8 in 2024-05-15T23:25:57.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:57 smithi090 ceph-mon[44434]: pgmap v1339: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 177 MiB/s wr, 2.83k op/s 2024-05-15T23:25:57.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:57 smithi090 ceph-mon[44434]: osdmap e501: 12 total, 11 up, 8 in 2024-05-15T23:25:57.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:57 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:25:57.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:57 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:25:57.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:57 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:25:57.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:57 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:25:57.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:57 smithi088 ceph-mon[44699]: from='client.26389 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:25:57.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:57 smithi088 ceph-mon[44699]: osdmap e500: 12 total, 11 up, 8 in 2024-05-15T23:25:57.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:57 smithi088 ceph-mon[44699]: pgmap v1339: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 177 MiB/s wr, 2.83k op/s 2024-05-15T23:25:57.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:57 smithi088 ceph-mon[44699]: osdmap e501: 12 total, 11 up, 8 in 2024-05-15T23:25:57.613 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:25:57.613 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-05-15T23:25:57.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-05-15T23:25:57.613 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:25:57.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.613 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:63047 6.b 6:d4728f2e:::benchmark_data_smithi090_131278_object63046:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e498)", 2024-05-15T23:25:57.613 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:51.122185+0000", 2024-05-15T23:25:57.613 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 6.4860130739999997, 2024-05-15T23:25:57.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052459549000000001, 2024-05-15T23:25:57.614 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.614 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.614 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.614 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:57.614 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:25:57.614 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:25:57.614 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 63047 2024-05-15T23:25:57.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.614 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.122185+0000", 2024-05-15T23:25:57.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.122185+0000", 2024-05-15T23:25:57.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.122187+0000", 2024-05-15T23:25:57.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.124e-06 2024-05-15T23:25:57.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.122261+0000", 2024-05-15T23:25:57.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4424000000000004e-05 2024-05-15T23:25:57.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.122270+0000", 2024-05-15T23:25:57.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0280000000000006e-06 2024-05-15T23:25:57.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.122275+0000", 2024-05-15T23:25:57.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.147e-06 2024-05-15T23:25:57.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.122319+0000", 2024-05-15T23:25:57.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3764000000000003e-05 2024-05-15T23:25:57.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.122425+0000", 2024-05-15T23:25:57.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000106104 2024-05-15T23:25:57.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:25:57.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.122514+0000", 2024-05-15T23:25:57.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8840999999999999e-05 2024-05-15T23:25:57.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:57.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.124547+0000", 2024-05-15T23:25:57.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020334519999999998 2024-05-15T23:25:57.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:57.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.174556+0000", 2024-05-15T23:25:57.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.050008885000000003 2024-05-15T23:25:57.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.174606+0000", 2024-05-15T23:25:57.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9434999999999997e-05 2024-05-15T23:25:57.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.174645+0000", 2024-05-15T23:25:57.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9344999999999997e-05 2024-05-15T23:25:57.619 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.619 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.619 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.619 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:64879 6.6 6:63059de2:::benchmark_data_smithi090_131278_object64878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e498)", 2024-05-15T23:25:57.619 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:51.973806+0000", 2024-05-15T23:25:57.619 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 5.6343922400000004, 2024-05-15T23:25:57.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089020967000000006, 2024-05-15T23:25:57.619 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.619 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.620 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.620 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:57.620 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:25:57.620 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:25:57.620 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 64879 2024-05-15T23:25:57.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.620 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.973806+0000", 2024-05-15T23:25:57.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.973806+0000", 2024-05-15T23:25:57.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.973807+0000", 2024-05-15T23:25:57.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.89e-07 2024-05-15T23:25:57.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.973911+0000", 2024-05-15T23:25:57.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0001047 2024-05-15T23:25:57.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.973916+0000", 2024-05-15T23:25:57.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3640000000000001e-06 2024-05-15T23:25:57.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.973919+0000", 2024-05-15T23:25:57.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2830000000000002e-06 2024-05-15T23:25:57.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.975926+0000", 2024-05-15T23:25:57.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020069879999999999 2024-05-15T23:25:57.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.976049+0000", 2024-05-15T23:25:57.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012327200000000001 2024-05-15T23:25:57.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:25:57.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.976151+0000", 2024-05-15T23:25:57.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101857 2024-05-15T23:25:57.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:57.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.977864+0000", 2024-05-15T23:25:57.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017128130000000001 2024-05-15T23:25:57.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:57.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062567+0000", 2024-05-15T23:25:57.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.084703422 2024-05-15T23:25:57.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062727+0000", 2024-05-15T23:25:57.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000159891 2024-05-15T23:25:57.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062827+0000", 2024-05-15T23:25:57.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9888000000000003e-05 2024-05-15T23:25:57.624 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.624 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.625 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.625 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:64888 6.b 6:d471ed81:::benchmark_data_smithi090_131278_object64887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e498)", 2024-05-15T23:25:57.625 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:51.976852+0000", 2024-05-15T23:25:57.625 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 5.6313460879999999, 2024-05-15T23:25:57.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.088423591999999995, 2024-05-15T23:25:57.625 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.625 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.625 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.625 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:57.625 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:25:57.625 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:25:57.625 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 64888 2024-05-15T23:25:57.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.626 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.976852+0000", 2024-05-15T23:25:57.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.976852+0000", 2024-05-15T23:25:57.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.976853+0000", 2024-05-15T23:25:57.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9900000000000009e-07 2024-05-15T23:25:57.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.976936+0000", 2024-05-15T23:25:57.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2479999999999996e-05 2024-05-15T23:25:57.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.976942+0000", 2024-05-15T23:25:57.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9390000000000002e-06 2024-05-15T23:25:57.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.976946+0000", 2024-05-15T23:25:57.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2939999999999999e-06 2024-05-15T23:25:57.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.978998+0000", 2024-05-15T23:25:57.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002052028 2024-05-15T23:25:57.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979143+0000", 2024-05-15T23:25:57.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014504599999999999 2024-05-15T23:25:57.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:25:57.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979283+0000", 2024-05-15T23:25:57.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013940600000000001 2024-05-15T23:25:57.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:57.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981380+0000", 2024-05-15T23:25:57.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002097486 2024-05-15T23:25:57.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:57.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.065189+0000", 2024-05-15T23:25:57.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.083808636000000006 2024-05-15T23:25:57.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.065236+0000", 2024-05-15T23:25:57.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6918e-05 2024-05-15T23:25:57.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.065276+0000", 2024-05-15T23:25:57.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.036e-05 2024-05-15T23:25:57.630 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.630 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.630 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.631 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:64889 6.c 6:34eaf55e:::benchmark_data_smithi090_131278_object64888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e498)", 2024-05-15T23:25:57.631 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:51.976955+0000", 2024-05-15T23:25:57.631 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 5.6312438369999995, 2024-05-15T23:25:57.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.085874566999999999, 2024-05-15T23:25:57.631 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.631 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.631 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.631 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:57.631 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:25:57.631 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:25:57.631 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 64889 2024-05-15T23:25:57.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.631 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.976955+0000", 2024-05-15T23:25:57.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.976955+0000", 2024-05-15T23:25:57.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.976955+0000", 2024-05-15T23:25:57.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6899999999999998e-07 2024-05-15T23:25:57.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.976994+0000", 2024-05-15T23:25:57.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8751999999999998e-05 2024-05-15T23:25:57.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.976999+0000", 2024-05-15T23:25:57.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4070000000000001e-06 2024-05-15T23:25:57.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.977002+0000", 2024-05-15T23:25:57.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1669999999999999e-06 2024-05-15T23:25:57.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.977042+0000", 2024-05-15T23:25:57.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0024000000000002e-05 2024-05-15T23:25:57.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.977146+0000", 2024-05-15T23:25:57.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000103779 2024-05-15T23:25:57.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2024-05-15T23:25:57.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.977260+0000", 2024-05-15T23:25:57.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114015 2024-05-15T23:25:57.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:57.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981450+0000", 2024-05-15T23:25:57.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041893829999999996 2024-05-15T23:25:57.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:57.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.983831+0000", 2024-05-15T23:25:57.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023809899999999999 2024-05-15T23:25:57.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062707+0000", 2024-05-15T23:25:57.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.078876166999999997 2024-05-15T23:25:57.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062829+0000", 2024-05-15T23:25:57.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000122414 2024-05-15T23:25:57.636 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.636 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.636 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.636 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:64890 6.c 6:3178a152:::benchmark_data_smithi090_131278_object64889:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e498)", 2024-05-15T23:25:57.636 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:51.979073+0000", 2024-05-15T23:25:57.636 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 5.6291259179999997, 2024-05-15T23:25:57.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.083935341999999996, 2024-05-15T23:25:57.637 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.637 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.637 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.637 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:57.637 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:25:57.637 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:25:57.637 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 64890 2024-05-15T23:25:57.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.637 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979073+0000", 2024-05-15T23:25:57.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979073+0000", 2024-05-15T23:25:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979074+0000", 2024-05-15T23:25:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9099999999999999e-06 2024-05-15T23:25:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979128+0000", 2024-05-15T23:25:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3139000000000001e-05 2024-05-15T23:25:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979140+0000", 2024-05-15T23:25:57.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2879e-05 2024-05-15T23:25:57.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979148+0000", 2024-05-15T23:25:57.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0199999999999994e-06 2024-05-15T23:25:57.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979269+0000", 2024-05-15T23:25:57.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012016899999999999 2024-05-15T23:25:57.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979384+0000", 2024-05-15T23:25:57.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011555 2024-05-15T23:25:57.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2024-05-15T23:25:57.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979522+0000", 2024-05-15T23:25:57.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013781199999999999 2024-05-15T23:25:57.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:57.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062875+0000", 2024-05-15T23:25:57.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.083352972999999997 2024-05-15T23:25:57.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:57.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062920+0000", 2024-05-15T23:25:57.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4804e-05 2024-05-15T23:25:57.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062969+0000", 2024-05-15T23:25:57.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8999999999999998e-05 2024-05-15T23:25:57.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.063008+0000", 2024-05-15T23:25:57.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9085999999999998e-05 2024-05-15T23:25:57.642 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.642 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.642 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.642 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.26326.0:64886 6.7 e498/477 6:e39331a8:::benchmark_data_smithi090_131278_object64885:head v 498'4101, mlcod=498'4098)", 2024-05-15T23:25:57.642 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:51.979194+0000", 2024-05-15T23:25:57.642 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 5.6290044610000001, 2024-05-15T23:25:57.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.083504487000000002, 2024-05-15T23:25:57.642 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.642 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.642 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.642 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979194+0000", 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979194+0000", 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979195+0000", 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4900000000000004e-07 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979250+0000", 2024-05-15T23:25:57.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5096999999999998e-05 2024-05-15T23:25:57.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979257+0000", 2024-05-15T23:25:57.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7039999999999997e-06 2024-05-15T23:25:57.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979263+0000", 2024-05-15T23:25:57.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1e-06 2024-05-15T23:25:57.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979304+0000", 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0982000000000002e-05 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.979317+0000", 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3557000000000001e-05 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.984619+0000", 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053014919999999997 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062698+0000", 2024-05-15T23:25:57.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.078079605999999996 2024-05-15T23:25:57.646 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.646 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.646 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.646 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:64891 6.c 6:39a0672a:::benchmark_data_smithi090_131278_object64890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e498)", 2024-05-15T23:25:57.646 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:51.981169+0000", 2024-05-15T23:25:57.646 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 5.6270296599999998, 2024-05-15T23:25:57.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082130670000000003, 2024-05-15T23:25:57.646 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.646 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 64891 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981169+0000", 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981169+0000", 2024-05-15T23:25:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981170+0000", 2024-05-15T23:25:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.367e-06 2024-05-15T23:25:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981225+0000", 2024-05-15T23:25:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4731000000000002e-05 2024-05-15T23:25:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981233+0000", 2024-05-15T23:25:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8949999999999998e-06 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981238+0000", 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2739999999999998e-06 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981473+0000", 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00023493900000000001 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981597+0000", 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000124191 2024-05-15T23:25:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2024-05-15T23:25:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981734+0000", 2024-05-15T23:25:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013637100000000001 2024-05-15T23:25:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062882+0000", 2024-05-15T23:25:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.081148227000000003 2024-05-15T23:25:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.063216+0000", 2024-05-15T23:25:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033458000000000001 2024-05-15T23:25:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.063263+0000", 2024-05-15T23:25:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6468999999999998e-05 2024-05-15T23:25:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.063299+0000", 2024-05-15T23:25:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6625999999999998e-05 2024-05-15T23:25:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:64892 6.0 6:0ea32096:::benchmark_data_smithi090_131278_object64891:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e498)", 2024-05-15T23:25:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:51.981251+0000", 2024-05-15T23:25:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 5.6269472589999996, 2024-05-15T23:25:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.082228356000000002, 2024-05-15T23:25:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:25:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:25:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 64892 2024-05-15T23:25:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981251+0000", 2024-05-15T23:25:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981251+0000", 2024-05-15T23:25:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981252+0000", 2024-05-15T23:25:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0399999999999995e-07 2024-05-15T23:25:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981283+0000", 2024-05-15T23:25:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1572000000000002e-05 2024-05-15T23:25:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981288+0000", 2024-05-15T23:25:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5229999999999997e-06 2024-05-15T23:25:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981292+0000", 2024-05-15T23:25:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.507e-06 2024-05-15T23:25:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981334+0000", 2024-05-15T23:25:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2247999999999999e-05 2024-05-15T23:25:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981474+0000", 2024-05-15T23:25:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014007900000000001 2024-05-15T23:25:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2024-05-15T23:25:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981604+0000", 2024-05-15T23:25:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000129691 2024-05-15T23:25:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062722+0000", 2024-05-15T23:25:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.081118414 2024-05-15T23:25:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.063373+0000", 2024-05-15T23:25:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00065099000000000005 2024-05-15T23:25:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.063440+0000", 2024-05-15T23:25:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6810000000000006e-05 2024-05-15T23:25:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.063480+0000", 2024-05-15T23:25:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9817999999999997e-05 2024-05-15T23:25:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(client.26326.0:64879 6.6 e498/477 ondisk, result = 0)", 2024-05-15T23:25:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:51.981397+0000", 2024-05-15T23:25:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 5.6268016159999998, 2024-05-15T23:25:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.081447080000000005, 2024-05-15T23:25:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-05-15T23:25:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981397+0000", 2024-05-15T23:25:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981397+0000", 2024-05-15T23:25:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981397+0000", 2024-05-15T23:25:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0399999999999996e-07 2024-05-15T23:25:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981400+0000", 2024-05-15T23:25:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6000000000000001e-06 2024-05-15T23:25:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981405+0000", 2024-05-15T23:25:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6399999999999996e-06 2024-05-15T23:25:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981409+0000", 2024-05-15T23:25:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1690000000000002e-06 2024-05-15T23:25:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.983825+0000", 2024-05-15T23:25:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002416652 2024-05-15T23:25:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.983834+0000", 2024-05-15T23:25:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1459999999999996e-06 2024-05-15T23:25:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062844+0000", 2024-05-15T23:25:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079010368999999997 2024-05-15T23:25:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:64898 6.b 6:d69fa49d:::benchmark_data_smithi090_131278_object64897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e498)", 2024-05-15T23:25:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:51.981499+0000", 2024-05-15T23:25:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 5.6266990569999997, 2024-05-15T23:25:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.087243869000000002, 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 64898 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981499+0000", 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981499+0000", 2024-05-15T23:25:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981501+0000", 2024-05-15T23:25:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.44e-06 2024-05-15T23:25:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.984007+0000", 2024-05-15T23:25:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025062259999999999 2024-05-15T23:25:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062593+0000", 2024-05-15T23:25:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.078586271999999999 2024-05-15T23:25:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062625+0000", 2024-05-15T23:25:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1622000000000003e-05 2024-05-15T23:25:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062899+0000", 2024-05-15T23:25:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027390599999999998 2024-05-15T23:25:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.063064+0000", 2024-05-15T23:25:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016539500000000001 2024-05-15T23:25:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:25:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.063228+0000", 2024-05-15T23:25:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000163499 2024-05-15T23:25:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.065402+0000", 2024-05-15T23:25:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021746280000000001 2024-05-15T23:25:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.068665+0000", 2024-05-15T23:25:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003262797 2024-05-15T23:25:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.068708+0000", 2024-05-15T23:25:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2432000000000003e-05 2024-05-15T23:25:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.068743+0000", 2024-05-15T23:25:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5651999999999997e-05 2024-05-15T23:25:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(client.26326.0:64889 6.c e498/477 ondisk, result = 0)", 2024-05-15T23:25:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:51.981735+0000", 2024-05-15T23:25:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 5.62646304, 2024-05-15T23:25:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.081115950000000006, 2024-05-15T23:25:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-05-15T23:25:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981735+0000", 2024-05-15T23:25:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981735+0000", 2024-05-15T23:25:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981736+0000", 2024-05-15T23:25:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.95e-07 2024-05-15T23:25:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981740+0000", 2024-05-15T23:25:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9349999999999996e-06 2024-05-15T23:25:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981748+0000", 2024-05-15T23:25:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6920000000000002e-06 2024-05-15T23:25:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981753+0000", 2024-05-15T23:25:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9259999999999999e-06 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.983821+0000", 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00206763 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.983827+0000", 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9719999999999997e-06 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062851+0000", 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079024800000000006 2024-05-15T23:25:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(client.26326.0:64890 6.c e498/477 ondisk, result = 0)", 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:51.981949+0000", 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 5.6262499540000004, 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.081068157000000002, 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981949+0000", 2024-05-15T23:25:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981949+0000", 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981950+0000", 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.032e-06 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981958+0000", 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1729999999999995e-06 2024-05-15T23:25:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981967+0000", 2024-05-15T23:25:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2949999999999999e-06 2024-05-15T23:25:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.981974+0000", 2024-05-15T23:25:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5950000000000004e-06 2024-05-15T23:25:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062906+0000", 2024-05-15T23:25:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.080932755999999995 2024-05-15T23:25:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062915+0000", 2024-05-15T23:25:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3310000000000004e-06 2024-05-15T23:25:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.063017+0000", 2024-05-15T23:25:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101975 2024-05-15T23:25:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.26326.0:64897 6.2 e498/477 6:4c3ceb30:::benchmark_data_smithi090_131278_object64896:head v 498'3936, mlcod=498'3934)", 2024-05-15T23:25:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:51.982008+0000", 2024-05-15T23:25:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 5.6261904310000004, 2024-05-15T23:25:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.080711599999999994, 2024-05-15T23:25:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.982008+0000", 2024-05-15T23:25:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.982008+0000", 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.982009+0000", 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6199999999999997e-07 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.982084+0000", 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4986000000000003e-05 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.982092+0000", 2024-05-15T23:25:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1599999999999998e-06 2024-05-15T23:25:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.982098+0000", 2024-05-15T23:25:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9689999999999999e-06 2024-05-15T23:25:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.982144+0000", 2024-05-15T23:25:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5617e-05 2024-05-15T23:25:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.982158+0000", 2024-05-15T23:25:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.398e-05 2024-05-15T23:25:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062597+0000", 2024-05-15T23:25:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.080439571000000001 2024-05-15T23:25:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062720+0000", 2024-05-15T23:25:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012255499999999999 2024-05-15T23:25:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(client.26326.0:64891 6.c e498/477 ondisk, result = 0)", 2024-05-15T23:25:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:51.984203+0000", 2024-05-15T23:25:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 5.6239955950000002, 2024-05-15T23:25:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.079102706999999994, 2024-05-15T23:25:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-05-15T23:25:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.984203+0000", 2024-05-15T23:25:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.984203+0000", 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.984204+0000", 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4130000000000001e-06 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.984209+0000", 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2599999999999999e-06 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062881+0000", 2024-05-15T23:25:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.078672830999999999 2024-05-15T23:25:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062889+0000", 2024-05-15T23:25:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.255e-06 2024-05-15T23:25:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.063204+0000", 2024-05-15T23:25:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031491800000000002 2024-05-15T23:25:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.063211+0000", 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8019999999999992e-06 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.063306+0000", 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4228000000000007e-05 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:65217 6.c 6:3555de86:::benchmark_data_smithi090_131278_object65216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e499)", 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:52.190271+0000", 2024-05-15T23:25:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 5.4179274040000003, 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.050302881000000001, 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 65217 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.190271+0000", 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.190271+0000", 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.190272+0000", 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2099999999999996e-07 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.190317+0000", 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5377999999999998e-05 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.190323+0000", 2024-05-15T23:25:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.942e-06 2024-05-15T23:25:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.190327+0000", 2024-05-15T23:25:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2869999999999998e-06 2024-05-15T23:25:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.190360+0000", 2024-05-15T23:25:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2700999999999997e-05 2024-05-15T23:25:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.236135+0000", 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.045774416999999998 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.236285+0000", 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015030299999999999 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.240440+0000", 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041550270000000004 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.240474+0000", 2024-05-15T23:25:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4267999999999998e-05 2024-05-15T23:25:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.240534+0000", 2024-05-15T23:25:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9787000000000002e-05 2024-05-15T23:25:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.240574+0000", 2024-05-15T23:25:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0049999999999998e-05 2024-05-15T23:25:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:65218 6.b 6:dc491366:::benchmark_data_smithi090_131278_object65217:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e499)", 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:52.190529+0000", 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 5.4176697660000004, 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.055219609000000003, 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 65218 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.190529+0000", 2024-05-15T23:25:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.190529+0000", 2024-05-15T23:25:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.190530+0000", 2024-05-15T23:25:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0699999999999996e-07 2024-05-15T23:25:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.190636+0000", 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000106072 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.236138+0000", 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.045501931000000002 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.236149+0000", 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1948e-05 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.236213+0000", 2024-05-15T23:25:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3684999999999998e-05 2024-05-15T23:25:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.236348+0000", 2024-05-15T23:25:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013436700000000001 2024-05-15T23:25:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:25:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.236496+0000", 2024-05-15T23:25:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014863800000000001 2024-05-15T23:25:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.243802+0000", 2024-05-15T23:25:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073054790000000001 2024-05-15T23:25:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.245641+0000", 2024-05-15T23:25:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018397680000000001 2024-05-15T23:25:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.245701+0000", 2024-05-15T23:25:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9632000000000001e-05 2024-05-15T23:25:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.245748+0000", 2024-05-15T23:25:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7281999999999999e-05 2024-05-15T23:25:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:66606 6.6 6:6bc9dc2c:::benchmark_data_smithi090_131278_object66605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e499)", 2024-05-15T23:25:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:52.858721+0000", 2024-05-15T23:25:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 4.7494775520000001, 2024-05-15T23:25:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052810528000000002, 2024-05-15T23:25:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:25:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:25:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 66606 2024-05-15T23:25:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.858721+0000", 2024-05-15T23:25:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.858721+0000", 2024-05-15T23:25:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.858721+0000", 2024-05-15T23:25:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7899999999999998e-07 2024-05-15T23:25:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.858743+0000", 2024-05-15T23:25:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1821e-05 2024-05-15T23:25:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.858748+0000", 2024-05-15T23:25:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4580000000000003e-06 2024-05-15T23:25:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.858751+0000", 2024-05-15T23:25:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2459999999999999e-06 2024-05-15T23:25:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.858790+0000", 2024-05-15T23:25:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9436e-05 2024-05-15T23:25:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.858914+0000", 2024-05-15T23:25:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012342399999999999 2024-05-15T23:25:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:25:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.859048+0000", 2024-05-15T23:25:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133657 2024-05-15T23:25:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.862136+0000", 2024-05-15T23:25:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030888780000000002 2024-05-15T23:25:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.911456+0000", 2024-05-15T23:25:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.049319776000000003 2024-05-15T23:25:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.911492+0000", 2024-05-15T23:25:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5422000000000001e-05 2024-05-15T23:25:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.911531+0000", 2024-05-15T23:25:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9830999999999999e-05 2024-05-15T23:25:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:66611 6.6 6:618b0e62:::benchmark_data_smithi090_131278_object66610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e499)", 2024-05-15T23:25:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:52.859185+0000", 2024-05-15T23:25:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 4.7490134670000002, 2024-05-15T23:25:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.052485679, 2024-05-15T23:25:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:25:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:25:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 66611 2024-05-15T23:25:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.859185+0000", 2024-05-15T23:25:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.859185+0000", 2024-05-15T23:25:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.859186+0000", 2024-05-15T23:25:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1489999999999999e-06 2024-05-15T23:25:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.859251+0000", 2024-05-15T23:25:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5047000000000002e-05 2024-05-15T23:25:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.859262+0000", 2024-05-15T23:25:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0342e-05 2024-05-15T23:25:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.859269+0000", 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4080000000000001e-06 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.860709+0000", 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001440321 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.860798+0000", 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8479000000000004e-05 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.860891+0000", 2024-05-15T23:25:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3336000000000002e-05 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.862249+0000", 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0013579729999999999 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.911578+0000", 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.049329016000000003 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.911632+0000", 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4086999999999999e-05 2024-05-15T23:25:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.911671+0000", 2024-05-15T23:25:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8520999999999999e-05 2024-05-15T23:25:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:70534 6.c 6:3979d2e2:::benchmark_data_smithi090_131278_object70533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e499)", 2024-05-15T23:25:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:54.630904+0000", 2024-05-15T23:25:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.9772941820000001, 2024-05-15T23:25:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.053162876999999997, 2024-05-15T23:25:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:25:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:25:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 70534 2024-05-15T23:25:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.630904+0000", 2024-05-15T23:25:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.630904+0000", 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.630905+0000", 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5199999999999995e-07 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.630994+0000", 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9207000000000002e-05 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.631002+0000", 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3730000000000004e-06 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.631009+0000", 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7700000000000004e-06 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.634783+0000", 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003774543 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.634913+0000", 2024-05-15T23:25:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000129479 2024-05-15T23:25:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2024-05-15T23:25:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.635049+0000", 2024-05-15T23:25:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013652900000000001 2024-05-15T23:25:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.638215+0000", 2024-05-15T23:25:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003165887 2024-05-15T23:25:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.683965+0000", 2024-05-15T23:25:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.045749816999999998 2024-05-15T23:25:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.684020+0000", 2024-05-15T23:25:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5105999999999997e-05 2024-05-15T23:25:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.684067+0000", 2024-05-15T23:25:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7213999999999999e-05 2024-05-15T23:25:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:70544 6.0 6:0444858f:::benchmark_data_smithi090_131278_object70543:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e499)", 2024-05-15T23:25:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:54.634750+0000", 2024-05-15T23:25:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.9734480240000001, 2024-05-15T23:25:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.049459397000000002, 2024-05-15T23:25:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:25:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:25:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:25:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:25:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:25:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:25:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 70544 2024-05-15T23:25:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:25:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:25:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.634750+0000", 2024-05-15T23:25:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:25:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.634750+0000", 2024-05-15T23:25:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:25:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:25:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.634752+0000", 2024-05-15T23:25:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5460000000000001e-06 2024-05-15T23:25:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:25:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.634824+0000", 2024-05-15T23:25:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2405999999999998e-05 2024-05-15T23:25:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:25:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.634834+0000", 2024-05-15T23:25:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.166e-06 2024-05-15T23:25:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:25:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.634841+0000", 2024-05-15T23:25:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4039999999999998e-06 2024-05-15T23:25:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:25:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.634891+0000", 2024-05-15T23:25:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0339e-05 2024-05-15T23:25:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:25:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.635006+0000", 2024-05-15T23:25:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114258 2024-05-15T23:25:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2024-05-15T23:25:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.635129+0000", 2024-05-15T23:25:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012376299999999999 2024-05-15T23:25:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:25:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.637176+0000", 2024-05-15T23:25:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002046605 2024-05-15T23:25:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:25:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.684113+0000", 2024-05-15T23:25:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046936735 2024-05-15T23:25:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:25:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.684166+0000", 2024-05-15T23:25:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3803e-05 2024-05-15T23:25:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:25:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:25:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:25:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:54.684210+0000", 2024-05-15T23:25:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3371999999999997e-05 2024-05-15T23:25:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:25:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:25:57.716 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:25:57.716 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:25:57.759 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:58.405 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:58 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:25:58.406 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:58 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:25:58.406 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:58 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:25:58.406 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:58 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:25:58.460 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 51 16 78488 78472 96.1435 122.5 0.0103249 0.0103997 2024-05-15T23:25:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:58 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:25:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:58 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:25:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:58 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:25:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:58 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:25:58.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:58 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:25:58.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:58 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:25:58.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:58 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:25:58.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:58 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:25:58.723 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:25:58.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:58.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:25:58.839 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:25:58.884 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:58.917 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:25:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:58.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:25:59.021 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:59.086 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T23:25:59.101 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:59.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:59.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:25:59.205 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:25:59.262 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:59.291 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:25:59.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:59.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:25:59.312 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:25:59.368 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:25:59.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:59 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:25:59.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:59 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:25:59.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:59 smithi084 ceph-mon[39784]: osdmap e502: 12 total, 11 up, 8 in 2024-05-15T23:25:59.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:25:59 smithi084 ceph-mon[39784]: pgmap v1342: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-05-15T23:25:59.395 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:25:59.399 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:25:59.423 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:25:59.443 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:59.451 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:25:59.460 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 52 16 80439 80423 96.6393 121.938 0.0118786 0.0103461 2024-05-15T23:25:59.489 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:25:59.520 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:25:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:59 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:25:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:59 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:25:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:59 smithi090 ceph-mon[44434]: osdmap e502: 12 total, 11 up, 8 in 2024-05-15T23:25:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:25:59 smithi090 ceph-mon[44434]: pgmap v1342: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-05-15T23:25:59.555 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:25:59.555 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:25:59.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:59 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:25:59.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:59 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:25:59.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:59 smithi088 ceph-mon[44699]: osdmap e502: 12 total, 11 up, 8 in 2024-05-15T23:25:59.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:25:59 smithi088 ceph-mon[44699]: pgmap v1342: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-05-15T23:25:59.575 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:25:59.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:59.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:25:59.589 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:25:59.619 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:25:59.645 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:25:59.677 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:25:59.677 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:25:59.677 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:25:59.677 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:25:59.677 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:25:59.677 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:25:59.677 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:25:59.677 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:25:59.678 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:25:59.678 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-05-15T23:25:59.678 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:25:59.678 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:25:59.678 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:25:59.678 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:25:59.718 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:59.806 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:25:59.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:59.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:25:59.910 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:25:59.962 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:25:59.989 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:25:59.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:25:59.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:26:00.092 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:00.128 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:00.138 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:00 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/523026524' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:00.138 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:00.138 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:00.138 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:00.138 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:00.138 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:00.154 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:00.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:00.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:00.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:26:00.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:26:00.289 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:00.312 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:26:00.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:00.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:26:00.417 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:26:00.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:00 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/523026524' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:00.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:00.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:00.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:00.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:00.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:00.460 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 53 16 82336 82320 97.0527 118.562 0.0129235 0.0103016 2024-05-15T23:26:00.461 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:00.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:00 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/523026524' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:00.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:00.919 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:26:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:00.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:26:00.931 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:26:00.931 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:26:01.024 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:26:01.080 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:01.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.243 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:26:01.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:01.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:26:01.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:01 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:01 smithi088 ceph-mon[44699]: pgmap v1343: 17 pgs: 17 active+clean; 4.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 162 MiB/s wr, 2.60k op/s 2024-05-15T23:26:01.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:26:01.382 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:01.460 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 54 16 84304 84288 97.533 123 0.00687301 0.0102511 2024-05-15T23:26:01.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:01 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:01 smithi084 ceph-mon[39784]: pgmap v1343: 17 pgs: 17 active+clean; 4.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 162 MiB/s wr, 2.60k op/s 2024-05-15T23:26:01.482 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:26:01.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:01.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:26:01.526 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:01.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:01 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:01.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:01 smithi090 ceph-mon[44434]: pgmap v1343: 17 pgs: 17 active+clean; 4.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 162 MiB/s wr, 2.60k op/s 2024-05-15T23:26:01.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:01.622 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:01.974 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:26:01.974 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:26:02.279 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:02.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:02.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:02.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:02 smithi084 ceph-mon[39784]: 6.6 deep-scrub starts 2024-05-15T23:26:02.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:02 smithi084 ceph-mon[39784]: osdmap e503: 12 total, 11 up, 8 in 2024-05-15T23:26:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:02 smithi088 ceph-mon[44699]: 6.6 deep-scrub starts 2024-05-15T23:26:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:02 smithi088 ceph-mon[44699]: osdmap e503: 12 total, 11 up, 8 in 2024-05-15T23:26:02.385 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:26:02.430 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:02.460 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 55 15 86036 86021 97.7288 108.312 0.00307605 0.0102304 2024-05-15T23:26:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:02 smithi090 ceph-mon[44434]: 6.6 deep-scrub starts 2024-05-15T23:26:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:02 smithi090 ceph-mon[44434]: osdmap e503: 12 total, 11 up, 8 in 2024-05-15T23:26:02.618 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:26:02.619 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:26:02.787 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:26:02.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:02.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:26:02.807 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T23:26:02.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:26:02.932 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:02.932 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:26:02.957 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:26:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:02.963 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:26:03.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:03.107 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:03.240 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:26:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:03 smithi090 ceph-mon[44434]: from='client.36350 v1:172.21.15.84:0/1963408251' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:26:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:03 smithi090 ceph-mon[44434]: pgmap v1345: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-05-15T23:26:03.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:03 smithi084 ceph-mon[39784]: from='client.36350 v1:172.21.15.84:0/1963408251' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:26:03.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:03 smithi084 ceph-mon[39784]: pgmap v1345: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-05-15T23:26:03.460 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 56 15 87608 87593 97.7379 98.25 0.00922143 0.0102278 2024-05-15T23:26:03.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:03 smithi088 ceph-mon[44699]: from='client.36350 v1:172.21.15.84:0/1963408251' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:26:03.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:03 smithi088 ceph-mon[44699]: pgmap v1345: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-05-15T23:26:03.971 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:03.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:03.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:03.994 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1344,"stamp":"2024-05-15T23:26:02.094694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5365563936,"num_objects":81865,"num_object_clones":0,"num_object_copies":163730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81865,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":81986,"num_write_kb":5241382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1220,"num_bytes_recovered":81147488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81957,"ondisk_log_size":81957,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14025828,"kb_used_data":10313996,"kb_used_omap":12,"kb_used_meta":3711731,"kb_avail":735935388,"statfs":{"total":767960285184,"available":753597837312,"internally_reserved":0,"allocated":10561531904,"data_stored":10555541834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12763,"internal_metadata":3800813093},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,6,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":668663808,"num_objects":10203,"num_object_clones":0,"num_object_copies":20406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10203,"num_write_kb":652992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.968473"},"pg_stats":[{"pgid":"6.f","version":"502'5113","reported_seq":5172,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-05-15T23:26:00.631595+0000","last_change":"2024-05-15T23:24:55.118094+0000","last_active":"2024-05-15T23:26:00.631595+0000","last_peered":"2024-05-15T23:26:00.631595+0000","last_clean":"2024-05-15T23:26:00.631595+0000","last_became_active":"2024-05-15T23:24:55.117866+0000","last_became_peered":"2024-05-15T23:24:55.117866+0000","last_unstale":"2024-05-15T23:26:00.631595+0000","last_undegraded":"2024-05-15T23:26:00.631595+0000","last_fullsized":"2024-05-15T23:26:00.631595+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:10.508552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335085568,"num_objects":5113,"num_object_clones":0,"num_object_copies":10226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5113,"num_write_kb":327232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.e","version":"502'5191","reported_seq":5248,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-05-15T23:25:59.942143+0000","last_change":"2024-05-15T23:24:55.123099+0000","last_active":"2024-05-15T23:25:59.942143+0000","last_peered":"2024-05-15T23:25:59.942143+0000","last_clean":"2024-05-15T23:25:59.942143+0000","last_became_active":"2024-05-15T23:24:55.122879+0000","last_became_peered":"2024-05-15T23:24:55.122879+0000","last_unstale":"2024-05-15T23:25:59.942143+0000","last_undegraded":"2024-05-15T23:25:59.942143+0000","last_fullsized":"2024-05-15T23:25:59.942143+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5191,"log_dups_size":0,"ondisk_log_size":5191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:06.327799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340197376,"num_objects":5191,"num_object_clones":0,"num_object_copies":10382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5191,"num_write_kb":332224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"501'4820","reported_seq":4877,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-05-15T23:25:57.770993+0000","last_change":"2024-05-15T23:24:55.123856+0000","last_active":"2024-05-15T23:25:57.770993+0000","last_peered":"2024-05-15T23:25:57.770993+0000","last_clean":"2024-05-15T23:25:57.770993+0000","last_became_active":"2024-05-15T23:24:55.123573+0000","last_became_peered":"2024-05-15T23:24:55.123573+0000","last_unstale":"2024-05-15T23:25:57.770993+0000","last_undegraded":"2024-05-15T23:25:57.770993+0000","last_fullsized":"2024-05-15T23:25:57.770993+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:07.444200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315883520,"num_objects":4820,"num_object_clones":0,"num_object_copies":9640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4820,"num_write_kb":308480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"502'5222","reported_seq":5281,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-05-15T23:26:00.948704+0000","last_change":"2024-05-15T23:24:55.123823+0000","last_active":"2024-05-15T23:26:00.948704+0000","last_peered":"2024-05-15T23:26:00.948704+0000","last_clean":"2024-05-15T23:26:00.948704+0000","last_became_active":"2024-05-15T23:24:55.123532+0000","last_became_peered":"2024-05-15T23:24:55.123532+0000","last_unstale":"2024-05-15T23:26:00.948704+0000","last_undegraded":"2024-05-15T23:26:00.948704+0000","last_fullsized":"2024-05-15T23:26:00.948704+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5222,"log_dups_size":0,"ondisk_log_size":5222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:17.518088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342228992,"num_objects":5222,"num_object_clones":0,"num_object_copies":10444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":334208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"502'5104","reported_seq":5163,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-05-15T23:26:00.943899+0000","last_change":"2024-05-15T23:24:55.123743+0000","last_active":"2024-05-15T23:26:00.943899+0000","last_peered":"2024-05-15T23:26:00.943899+0000","last_clean":"2024-05-15T23:26:00.943899+0000","last_became_active":"2024-05-15T23:24:55.123344+0000","last_became_peered":"2024-05-15T23:24:55.123344+0000","last_unstale":"2024-05-15T23:26:00.943899+0000","last_undegraded":"2024-05-15T23:26:00.943899+0000","last_fullsized":"2024-05-15T23:26:00.943899+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5104,"log_dups_size":0,"ondisk_log_size":5104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:19.886620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334495744,"num_objects":5104,"num_object_clones":0,"num_object_copies":10208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5104,"num_write_kb":326656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"501'4896","reported_seq":4953,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-05-15T23:25:57.791048+0000","last_change":"2024-05-15T23:24:55.122463+0000","last_active":"2024-05-15T23:25:57.791048+0000","last_peered":"2024-05-15T23:25:57.791048+0000","last_clean":"2024-05-15T23:25:57.791048+0000","last_became_active":"2024-05-15T23:24:55.122242+0000","last_became_peered":"2024-05-15T23:24:55.122242+0000","last_unstale":"2024-05-15T23:25:57.791048+0000","last_undegraded":"2024-05-15T23:25:57.791048+0000","last_fullsized":"2024-05-15T23:25:57.791048+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":4896,"log_dups_size":0,"ondisk_log_size":4896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:14.694288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320864256,"num_objects":4896,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4896,"num_write_kb":313344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"502'5096","reported_seq":6368,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-05-15T23:26:00.627139+0000","last_change":"2024-05-15T23:25:38.025027+0000","last_active":"2024-05-15T23:26:00.627139+0000","last_peered":"2024-05-15T23:26:00.627139+0000","last_clean":"2024-05-15T23:26:00.627139+0000","last_became_active":"2024-05-15T23:25:34.034608+0000","last_became_peered":"2024-05-15T23:25:34.034608+0000","last_unstale":"2024-05-15T23:26:00.627139+0000","last_undegraded":"2024-05-15T23:26:00.627139+0000","last_fullsized":"2024-05-15T23:26:00.627139+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":495,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5096,"log_dups_size":0,"ondisk_log_size":5096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333971456,"num_objects":5096,"num_object_clones":0,"num_object_copies":10192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5096,"num_write_kb":326144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":602,"num_bytes_recovered":39452672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.8","version":"502'5288","reported_seq":5347,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-05-15T23:26:01.102634+0000","last_change":"2024-05-15T23:24:55.414355+0000","last_active":"2024-05-15T23:26:01.102634+0000","last_peered":"2024-05-15T23:26:01.102634+0000","last_clean":"2024-05-15T23:26:01.102634+0000","last_became_active":"2024-05-15T23:24:55.414161+0000","last_became_peered":"2024-05-15T23:24:55.414161+0000","last_unstale":"2024-05-15T23:26:01.102634+0000","last_undegraded":"2024-05-15T23:26:01.102634+0000","last_fullsized":"2024-05-15T23:26:01.102634+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5288,"log_dups_size":0,"ondisk_log_size":5288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:13.277018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346554368,"num_objects":5288,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5288,"num_write_kb":338432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"502'5266","reported_seq":5325,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-05-15T23:26:00.947327+0000","last_change":"2024-05-15T23:24:55.124182+0000","last_active":"2024-05-15T23:26:00.947327+0000","last_peered":"2024-05-15T23:26:00.947327+0000","last_clean":"2024-05-15T23:26:00.947327+0000","last_became_active":"2024-05-15T23:24:55.123938+0000","last_became_peered":"2024-05-15T23:24:55.123938+0000","last_unstale":"2024-05-15T23:26:00.947327+0000","last_undegraded":"2024-05-15T23:26:00.947327+0000","last_fullsized":"2024-05-15T23:26:00.947327+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5266,"log_dups_size":0,"ondisk_log_size":5266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345112576,"num_objects":5266,"num_object_clones":0,"num_object_copies":10532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5266,"num_write_kb":337024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"502'5188","reported_seq":5247,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-05-15T23:26:01.094883+0000","last_change":"2024-05-15T23:24:55.122847+0000","last_active":"2024-05-15T23:26:01.094883+0000","last_peered":"2024-05-15T23:26:01.094883+0000","last_clean":"2024-05-15T23:26:01.094883+0000","last_became_active":"2024-05-15T23:24:55.122611+0000","last_became_peered":"2024-05-15T23:24:55.122611+0000","last_unstale":"2024-05-15T23:26:01.094883+0000","last_undegraded":"2024-05-15T23:26:01.094883+0000","last_fullsized":"2024-05-15T23:26:01.094883+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5188,"log_dups_size":0,"ondisk_log_size":5188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:19.174298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340000768,"num_objects":5188,"num_object_clones":0,"num_object_copies":10376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5188,"num_write_kb":332032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"502'5204","reported_seq":5263,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-05-15T23:26:00.885578+0000","last_change":"2024-05-15T23:24:55.123928+0000","last_active":"2024-05-15T23:26:00.885578+0000","last_peered":"2024-05-15T23:26:00.885578+0000","last_clean":"2024-05-15T23:26:00.885578+0000","last_became_active":"2024-05-15T23:24:55.123559+0000","last_became_peered":"2024-05-15T23:24:55.123559+0000","last_unstale":"2024-05-15T23:26:00.885578+0000","last_undegraded":"2024-05-15T23:26:00.885578+0000","last_fullsized":"2024-05-15T23:26:00.885578+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5204,"log_dups_size":0,"ondisk_log_size":5204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:09.823435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341049344,"num_objects":5204,"num_object_clones":0,"num_object_copies":10408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5204,"num_write_kb":333056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"502'5297","reported_seq":5356,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-05-15T23:26:01.106908+0000","last_change":"2024-05-15T23:24:55.414492+0000","last_active":"2024-05-15T23:26:01.106908+0000","last_peered":"2024-05-15T23:26:01.106908+0000","last_clean":"2024-05-15T23:26:01.106908+0000","last_became_active":"2024-05-15T23:24:55.414245+0000","last_became_peered":"2024-05-15T23:24:55.414245+0000","last_unstale":"2024-05-15T23:26:01.106908+0000","last_undegraded":"2024-05-15T23:26:01.106908+0000","last_fullsized":"2024-05-15T23:26:01.106908+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5297,"log_dups_size":0,"ondisk_log_size":5297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:06.322903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347144192,"num_objects":5297,"num_object_clones":0,"num_object_copies":10594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5297,"num_write_kb":339008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"502'4952","reported_seq":5011,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-05-15T23:25:59.944464+0000","last_change":"2024-05-15T23:24:55.122439+0000","last_active":"2024-05-15T23:25:59.944464+0000","last_peered":"2024-05-15T23:25:59.944464+0000","last_clean":"2024-05-15T23:25:59.944464+0000","last_became_active":"2024-05-15T23:24:55.122178+0000","last_became_peered":"2024-05-15T23:24:55.122178+0000","last_unstale":"2024-05-15T23:25:59.944464+0000","last_undegraded":"2024-05-15T23:25:59.944464+0000","last_fullsized":"2024-05-15T23:25:59.944464+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":4952,"log_dups_size":0,"ondisk_log_size":4952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:02.033448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324534272,"num_objects":4952,"num_object_clones":0,"num_object_copies":9904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4952,"num_write_kb":316928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"502'5160","reported_seq":6452,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-05-15T23:26:00.627231+0000","last_change":"2024-05-15T23:25:38.082859+0000","last_active":"2024-05-15T23:26:00.627231+0000","last_peered":"2024-05-15T23:26:00.627231+0000","last_clean":"2024-05-15T23:26:00.627231+0000","last_became_active":"2024-05-15T23:25:34.037025+0000","last_became_peered":"2024-05-15T23:25:34.037025+0000","last_unstale":"2024-05-15T23:26:00.627231+0000","last_undegraded":"2024-05-15T23:26:00.627231+0000","last_fullsized":"2024-05-15T23:26:00.627231+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":495,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5160,"log_dups_size":0,"ondisk_log_size":5160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:12.162550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338165760,"num_objects":5160,"num_object_clones":0,"num_object_copies":10320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5160,"num_write_kb":330240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":612,"num_bytes_recovered":40108032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.0","version":"502'5197","reported_seq":5256,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-05-15T23:26:00.940431+0000","last_change":"2024-05-15T23:24:55.124437+0000","last_active":"2024-05-15T23:26:00.940431+0000","last_peered":"2024-05-15T23:26:00.940431+0000","last_clean":"2024-05-15T23:26:00.940431+0000","last_became_active":"2024-05-15T23:24:55.123587+0000","last_became_peered":"2024-05-15T23:24:55.123587+0000","last_unstale":"2024-05-15T23:26:00.940431+0000","last_undegraded":"2024-05-15T23:26:00.940431+0000","last_fullsized":"2024-05-15T23:26:00.940431+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5197,"log_dups_size":0,"ondisk_log_size":5197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:15.246805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340590592,"num_objects":5197,"num_object_clones":0,"num_object_copies":10394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5197,"num_write_kb":332608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"501'4869","reported_seq":4924,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-05-15T23:25:57.774773+0000","last_change":"2024-05-15T23:24:55.124421+0000","last_active":"2024-05-15T23:25:57.774773+0000","last_peered":"2024-05-15T23:25:57.774773+0000","last_clean":"2024-05-15T23:25:57.774773+0000","last_became_active":"2024-05-15T23:24:55.124153+0000","last_became_peered":"2024-05-15T23:24:55.124153+0000","last_unstale":"2024-05-15T23:25:57.774773+0000","last_undegraded":"2024-05-15T23:25:57.774773+0000","last_fullsized":"2024-05-15T23:25:57.774773+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":4869,"log_dups_size":0,"ondisk_log_size":4869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:01.986642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319094784,"num_objects":4869,"num_object_clones":0,"num_object_copies":9738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4869,"num_write_kb":311616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1059,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-05-15T23:25:58.142307+0000","last_change":"2024-05-15T23:25:47.899431+0000","last_active":"2024-05-15T23:25:58.142307+0000","last_peered":"2024-05-15T23:25:58.142307+0000","last_clean":"2024-05-15T23:25:58.142307+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:25:58.142307+0000","last_undegraded":"2024-05-15T23:25:58.142307+0000","last_fullsized":"2024-05-15T23:25:58.142307+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:25:47.899301+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:25:47.899301+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:27:14.442348+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":5364973568,"num_objects":81863,"num_object_clones":0,"num_object_copies":163726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81863,"num_write_kb":5239232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1214,"num_bytes_recovered":79560704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10670112768,"data_stored":10670112768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81863,"ondisk_log_size":81863,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351242,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1176576,"kb_used_data":608116,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92568576,"statfs":{"total":95995035648,"available":94790221824,"internally_reserved":0,"allocated":622710784,"data_stored":621969144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":582089156},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580176,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1280972,"kb_used_data":970756,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92464180,"statfs":{"total":95995035648,"available":94683320320,"internally_reserved":0,"allocated":994054144,"data_stored":993304018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":317651392},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776408,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2048272,"kb_used_data":1620036,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":91696880,"statfs":{"total":95995035648,"available":93897605120,"internally_reserved":0,"allocated":1658916864,"data_stored":1658166738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":438499776},"hb_peers":[0,1,2,3,4,5,6,7,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266976,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1743896,"kb_used_data":1329548,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92001256,"statfs":{"total":95995035648,"available":94209286144,"internally_reserved":0,"allocated":1361457152,"data_stored":1360703474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":424278474},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495910,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575168,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1549068,"kb_used_data":1004228,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92196084,"statfs":{"total":95995035648,"available":94408790016,"internally_reserved":0,"allocated":1028329472,"data_stored":1027579346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016578,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483266,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150007,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2487436,"kb_used_data":1968964,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91257716,"statfs":{"total":95995035648,"available":93447901184,"internally_reserved":0,"allocated":2016219136,"data_stored":2015469010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":530905543},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986479,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1404948,"kb_used_data":968388,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92340204,"statfs":{"total":95995035648,"available":94556368896,"internally_reserved":0,"allocated":991629312,"data_stored":990879186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790251,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2334660,"kb_used_data":1843960,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":91410492,"statfs":{"total":95995035648,"available":93604343808,"internally_reserved":0,"allocated":1888215040,"data_stored":1887470918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1028653056,"data_stored":1028653056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2040463360,"data_stored":2040463360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1002110976,"data_stored":1002110976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1904672768,"data_stored":1904672768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1369047040,"data_stored":1369047040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1687355392,"data_stored":1687355392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1007419392,"data_stored":1007419392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":630390784,"data_stored":630390784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:26:03.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:26:03.998 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:04.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:26:04.118 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:04.146 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:26:04.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:04.150 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:26:04.161 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:04 smithi088 ceph-mon[44699]: 6.9 scrub starts 2024-05-15T23:26:04.161 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:04 smithi088 ceph-mon[44699]: 6.7 scrub starts 2024-05-15T23:26:04.251 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:04.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:04 smithi084 ceph-mon[39784]: 6.9 scrub starts 2024-05-15T23:26:04.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:04 smithi084 ceph-mon[39784]: 6.7 scrub starts 2024-05-15T23:26:04.306 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:04.460 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 57 16 88924 88908 97.4651 82.1875 0.0026397 0.010257 2024-05-15T23:26:04.510 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:04.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:04.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:04 smithi090 ceph-mon[44434]: 6.9 scrub starts 2024-05-15T23:26:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:04 smithi090 ceph-mon[44434]: 6.7 scrub starts 2024-05-15T23:26:04.615 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:26:04.665 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:05.460 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 58 16 90271 90255 97.2361 84.1875 0.0047798 0.0102825 2024-05-15T23:26:05.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:05 smithi090 ceph-mon[44434]: 6.2 scrub starts 2024-05-15T23:26:05.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:05 smithi090 ceph-mon[44434]: 6.9 scrub ok 2024-05-15T23:26:05.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:05 smithi090 ceph-mon[44434]: 6.7 scrub ok 2024-05-15T23:26:05.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:05 smithi090 ceph-mon[44434]: pgmap v1346: 17 pgs: 17 active+clean; 5.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-05-15T23:26:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:05 smithi088 ceph-mon[44699]: 6.2 scrub starts 2024-05-15T23:26:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:05 smithi088 ceph-mon[44699]: 6.9 scrub ok 2024-05-15T23:26:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:05 smithi088 ceph-mon[44699]: 6.7 scrub ok 2024-05-15T23:26:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:05 smithi088 ceph-mon[44699]: pgmap v1346: 17 pgs: 17 active+clean; 5.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-05-15T23:26:05.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:05 smithi084 ceph-mon[39784]: 6.2 scrub starts 2024-05-15T23:26:05.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:05 smithi084 ceph-mon[39784]: 6.9 scrub ok 2024-05-15T23:26:05.595 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:05 smithi084 ceph-mon[39784]: 6.7 scrub ok 2024-05-15T23:26:05.595 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:05 smithi084 ceph-mon[39784]: pgmap v1346: 17 pgs: 17 active+clean; 5.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-05-15T23:26:06.177 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:26:06.221 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:26:06.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:06.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:26:06.314 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:26:06.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:26:06.341 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:26:06.359 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:06.368 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:26:06.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:06 smithi084 ceph-mon[39784]: 6.2 scrub ok 2024-05-15T23:26:06.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:06 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2159786999' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:26:06.387 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:26:06.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:06.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:26:06.396 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:26:06.425 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:26:06.453 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:26:06.460 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 59 16 91932 91916 97.3474 103.812 0.00380319 0.0102698 2024-05-15T23:26:06.478 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:26:06.493 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:26:06.502 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:26:06.529 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:26:06.530 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:06 smithi088 ceph-mon[44699]: 6.2 scrub ok 2024-05-15T23:26:06.530 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:06 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2159786999' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:26:06.536 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:06.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:06 smithi090 ceph-mon[44434]: 6.2 scrub ok 2024-05-15T23:26:06.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:06 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2159786999' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:26:06.553 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:26:06.581 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:26:06.581 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:26:06.581 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:26:06.581 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:26:06.581 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:26:06.581 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:26:06.581 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:26:06.581 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:26:06.581 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:26:06.582 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-05-15T23:26:06.582 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:26:06.582 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:26:06.582 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:26:07.456 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:07 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:07.456 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:07 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2159786999' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:26:07.456 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:07 smithi088 ceph-mon[44699]: osdmap e504: 12 total, 11 up, 8 in 2024-05-15T23:26:07.456 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:07 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3779726235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:07.456 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:07.457 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:07.457 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:07.457 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:07.457 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:07.457 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:07 smithi088 ceph-mon[44699]: pgmap v1348: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-05-15T23:26:07.457 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:07 smithi088 ceph-mon[44699]: osdmap e505: 12 total, 11 up, 8 in 2024-05-15T23:26:07.460 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:26:07.460201 min lat: 0.0010209 max lat: 24.9077 avg lat: 0.0102686 2024-05-15T23:26:07.460 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:26:07.460 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 60 16 93503 93487 97.3612 98.1875 0.020695 0.0102686 2024-05-15T23:26:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:07 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:07 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2159786999' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:26:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:07 smithi090 ceph-mon[44434]: osdmap e504: 12 total, 11 up, 8 in 2024-05-15T23:26:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:07 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3779726235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:07 smithi090 ceph-mon[44434]: pgmap v1348: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-05-15T23:26:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:07 smithi090 ceph-mon[44434]: osdmap e505: 12 total, 11 up, 8 in 2024-05-15T23:26:07.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:07 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:07.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:07 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2159786999' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:26:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:07 smithi084 ceph-mon[39784]: osdmap e504: 12 total, 11 up, 8 in 2024-05-15T23:26:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:07 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3779726235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:07 smithi084 ceph-mon[39784]: pgmap v1348: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-05-15T23:26:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:07 smithi084 ceph-mon[39784]: osdmap e505: 12 total, 11 up, 8 in 2024-05-15T23:26:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:26:07.716 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:26:07.781 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.462 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 61 16 94961 94945 97.2589 91.125 0.00679909 0.0102806 2024-05-15T23:26:08.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:08.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:26:09.006 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:26:09.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:09.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:26:09.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:26:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:09.180 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:26:09.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:09.187 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:26:09.288 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:26:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:09 smithi088 ceph-mon[44699]: pgmap v1350: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-05-15T23:26:09.329 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:09.330 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:09.361 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:26:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:09.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:26:09.461 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 62 16 96402 96386 97.1427 90.0625 0.0321825 0.0102915 2024-05-15T23:26:09.466 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:26:09.507 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:09.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:09 smithi084 ceph-mon[39784]: pgmap v1350: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-05-15T23:26:09.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:09 smithi090 ceph-mon[44434]: pgmap v1350: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-05-15T23:26:09.567 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:26:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:09.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:26:09.671 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:26:09.714 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:09.716 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:26:09.747 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:26:09.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:09.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:26:09.756 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:26:09.756 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:26:09.756 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:26:09.756 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:26:09.756 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:26:09.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:26:09.888 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:26:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:09.924 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:26:10.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:10.072 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:10.217 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:26:10.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:10.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:26:10.321 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:26:10.364 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:10.460 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:26:10.460 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 63 16 97807 97791 96.9945 87.8125 0.00557664 0.0103082 2024-05-15T23:26:10.463 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:26:10.627 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:26:10.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:10.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:26:10.642 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T23:26:10.732 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:26:10.764 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:10.842 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:26:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:10.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:26:10.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:26:10.984 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:11.006 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:26:11.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:11.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:26:11.086 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:26:11.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:26:11.144 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:11.273 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:26:11.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:11.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:26:11.307 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1350,"stamp":"2024-05-15T23:26:09.037785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5991170592,"num_objects":91411,"num_object_clones":0,"num_object_copies":182822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91411,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":91532,"num_write_kb":5852326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1220,"num_bytes_recovered":81147488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91835,"ondisk_log_size":91835,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15353408,"kb_used_data":11640180,"kb_used_omap":12,"kb_used_meta":3713139,"kb_avail":734607808,"statfs":{"total":767960285184,"available":752238395392,"internally_reserved":0,"allocated":11919544320,"data_stored":11913469594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12763,"internal_metadata":3802254885},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":625606656,"num_objects":9546,"num_object_clones":0,"num_object_copies":19092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9546,"num_write_kb":610944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.943091"},"pg_stats":[{"pgid":"6.f","version":"503'5616","reported_seq":5677,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-05-15T23:26:05.627367+0000","last_change":"2024-05-15T23:24:55.118094+0000","last_active":"2024-05-15T23:26:05.627367+0000","last_peered":"2024-05-15T23:26:05.627367+0000","last_clean":"2024-05-15T23:26:05.627367+0000","last_became_active":"2024-05-15T23:24:55.117866+0000","last_became_peered":"2024-05-15T23:24:55.117866+0000","last_unstale":"2024-05-15T23:26:05.627367+0000","last_undegraded":"2024-05-15T23:26:05.627367+0000","last_fullsized":"2024-05-15T23:26:05.627367+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5616,"log_dups_size":0,"ondisk_log_size":5616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:10.508552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368050176,"num_objects":5616,"num_object_clones":0,"num_object_copies":11232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5616,"num_write_kb":359424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.e","version":"503'5718","reported_seq":5779,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-05-15T23:26:04.971918+0000","last_change":"2024-05-15T23:24:55.123099+0000","last_active":"2024-05-15T23:26:04.971918+0000","last_peered":"2024-05-15T23:26:04.971918+0000","last_clean":"2024-05-15T23:26:04.971918+0000","last_became_active":"2024-05-15T23:24:55.122879+0000","last_became_peered":"2024-05-15T23:24:55.122879+0000","last_unstale":"2024-05-15T23:26:04.971918+0000","last_undegraded":"2024-05-15T23:26:04.971918+0000","last_fullsized":"2024-05-15T23:26:04.971918+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5718,"log_dups_size":0,"ondisk_log_size":5718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:06.327799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374734848,"num_objects":5718,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5718,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"505'5851","reported_seq":5916,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-05-15T23:26:07.793880+0000","last_change":"2024-05-15T23:24:55.123856+0000","last_active":"2024-05-15T23:26:07.793880+0000","last_peered":"2024-05-15T23:26:07.793880+0000","last_clean":"2024-05-15T23:26:07.793880+0000","last_became_active":"2024-05-15T23:24:55.123573+0000","last_became_peered":"2024-05-15T23:24:55.123573+0000","last_unstale":"2024-05-15T23:26:07.793880+0000","last_undegraded":"2024-05-15T23:26:07.793880+0000","last_fullsized":"2024-05-15T23:26:07.793880+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5851,"log_dups_size":0,"ondisk_log_size":5851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383451136,"num_objects":5851,"num_object_clones":0,"num_object_copies":11702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5851,"num_write_kb":374464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"503'5720","reported_seq":5780,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-05-15T23:26:05.956816+0000","last_change":"2024-05-15T23:24:55.123823+0000","last_active":"2024-05-15T23:26:05.956816+0000","last_peered":"2024-05-15T23:26:05.956816+0000","last_clean":"2024-05-15T23:26:05.956816+0000","last_became_active":"2024-05-15T23:24:55.123532+0000","last_became_peered":"2024-05-15T23:24:55.123532+0000","last_unstale":"2024-05-15T23:26:05.956816+0000","last_undegraded":"2024-05-15T23:26:05.956816+0000","last_fullsized":"2024-05-15T23:26:05.956816+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5720,"log_dups_size":0,"ondisk_log_size":5720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:17.518088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374865920,"num_objects":5720,"num_object_clones":0,"num_object_copies":11440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5720,"num_write_kb":366080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"503'5578","reported_seq":5639,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-05-15T23:26:05.948254+0000","last_change":"2024-05-15T23:24:55.123743+0000","last_active":"2024-05-15T23:26:05.948254+0000","last_peered":"2024-05-15T23:26:05.948254+0000","last_clean":"2024-05-15T23:26:05.948254+0000","last_became_active":"2024-05-15T23:24:55.123344+0000","last_became_peered":"2024-05-15T23:24:55.123344+0000","last_unstale":"2024-05-15T23:26:05.948254+0000","last_undegraded":"2024-05-15T23:26:05.948254+0000","last_fullsized":"2024-05-15T23:26:05.948254+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5578,"log_dups_size":0,"ondisk_log_size":5578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:19.886620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365559808,"num_objects":5578,"num_object_clones":0,"num_object_copies":11156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5578,"num_write_kb":356992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"505'5955","reported_seq":6020,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-05-15T23:26:07.775442+0000","last_change":"2024-05-15T23:24:55.122463+0000","last_active":"2024-05-15T23:26:07.775442+0000","last_peered":"2024-05-15T23:26:07.775442+0000","last_clean":"2024-05-15T23:26:07.775442+0000","last_became_active":"2024-05-15T23:24:55.122242+0000","last_became_peered":"2024-05-15T23:24:55.122242+0000","last_unstale":"2024-05-15T23:26:07.775442+0000","last_undegraded":"2024-05-15T23:26:07.775442+0000","last_fullsized":"2024-05-15T23:26:07.775442+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5955,"log_dups_size":0,"ondisk_log_size":5955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:14.694288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390266880,"num_objects":5955,"num_object_clones":0,"num_object_copies":11910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5955,"num_write_kb":381120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"503'5601","reported_seq":6936,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-05-15T23:26:05.624519+0000","last_change":"2024-05-15T23:26:03.865823+0000","last_active":"2024-05-15T23:26:05.624519+0000","last_peered":"2024-05-15T23:26:05.624519+0000","last_clean":"2024-05-15T23:26:05.624519+0000","last_became_active":"2024-05-15T23:25:34.034608+0000","last_became_peered":"2024-05-15T23:25:34.034608+0000","last_unstale":"2024-05-15T23:26:05.624519+0000","last_undegraded":"2024-05-15T23:26:05.624519+0000","last_fullsized":"2024-05-15T23:26:05.624519+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":495,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'5444","last_scrub_stamp":"2024-05-15T23:26:03.865526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:26:03.865526+0000","objects_scrubbed":5396,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:27:08.150176+0000","scrub_duration":989,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367067136,"num_objects":5601,"num_object_clones":0,"num_object_copies":11202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5601,"num_write_kb":358464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":602,"num_bytes_recovered":39452672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.8","version":"503'5785","reported_seq":5846,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-05-15T23:26:06.104819+0000","last_change":"2024-05-15T23:24:55.414355+0000","last_active":"2024-05-15T23:26:06.104819+0000","last_peered":"2024-05-15T23:26:06.104819+0000","last_clean":"2024-05-15T23:26:06.104819+0000","last_became_active":"2024-05-15T23:24:55.414161+0000","last_became_peered":"2024-05-15T23:24:55.414161+0000","last_unstale":"2024-05-15T23:26:06.104819+0000","last_undegraded":"2024-05-15T23:26:06.104819+0000","last_fullsized":"2024-05-15T23:26:06.104819+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5785,"log_dups_size":0,"ondisk_log_size":5785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:13.277018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379125760,"num_objects":5785,"num_object_clones":0,"num_object_copies":11570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5785,"num_write_kb":370240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"503'6068","reported_seq":6237,"reported_epoch":503,"state":"active+clean+scrubbing+deep","last_fresh":"2024-05-15T23:26:05.977071+0000","last_change":"2024-05-15T23:26:01.438866+0000","last_active":"2024-05-15T23:26:05.977071+0000","last_peered":"2024-05-15T23:26:05.977071+0000","last_clean":"2024-05-15T23:26:05.977071+0000","last_became_active":"2024-05-15T23:24:55.123938+0000","last_became_peered":"2024-05-15T23:24:55.123938+0000","last_unstale":"2024-05-15T23:26:05.977071+0000","last_undegraded":"2024-05-15T23:26:05.977071+0000","last_fullsized":"2024-05-15T23:26:05.977071+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":2623,"log_size":6068,"log_dups_size":0,"ondisk_log_size":6068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 4s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375914496,"num_objects":5736,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5736,"num_write_kb":367104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"503'5662","reported_seq":5723,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-05-15T23:26:06.087371+0000","last_change":"2024-05-15T23:24:55.122847+0000","last_active":"2024-05-15T23:26:06.087371+0000","last_peered":"2024-05-15T23:26:06.087371+0000","last_clean":"2024-05-15T23:26:06.087371+0000","last_became_active":"2024-05-15T23:24:55.122611+0000","last_became_peered":"2024-05-15T23:24:55.122611+0000","last_unstale":"2024-05-15T23:26:06.087371+0000","last_undegraded":"2024-05-15T23:26:06.087371+0000","last_fullsized":"2024-05-15T23:26:06.087371+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5662,"log_dups_size":0,"ondisk_log_size":5662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:19.174298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371064832,"num_objects":5662,"num_object_clones":0,"num_object_copies":11324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5662,"num_write_kb":362368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"503'5704","reported_seq":5765,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-05-15T23:26:05.895994+0000","last_change":"2024-05-15T23:24:55.123928+0000","last_active":"2024-05-15T23:26:05.895994+0000","last_peered":"2024-05-15T23:26:05.895994+0000","last_clean":"2024-05-15T23:26:05.895994+0000","last_became_active":"2024-05-15T23:24:55.123559+0000","last_became_peered":"2024-05-15T23:24:55.123559+0000","last_unstale":"2024-05-15T23:26:05.895994+0000","last_undegraded":"2024-05-15T23:26:05.895994+0000","last_fullsized":"2024-05-15T23:26:05.895994+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5704,"log_dups_size":0,"ondisk_log_size":5704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:09.823435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373817344,"num_objects":5704,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5704,"num_write_kb":365056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"503'5771","reported_seq":5832,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-05-15T23:26:06.095623+0000","last_change":"2024-05-15T23:24:55.414492+0000","last_active":"2024-05-15T23:26:06.095623+0000","last_peered":"2024-05-15T23:26:06.095623+0000","last_clean":"2024-05-15T23:26:06.095623+0000","last_became_active":"2024-05-15T23:24:55.414245+0000","last_became_peered":"2024-05-15T23:24:55.414245+0000","last_unstale":"2024-05-15T23:26:06.095623+0000","last_undegraded":"2024-05-15T23:26:06.095623+0000","last_fullsized":"2024-05-15T23:26:06.095623+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5771,"log_dups_size":0,"ondisk_log_size":5771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:06.322903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378208256,"num_objects":5771,"num_object_clones":0,"num_object_copies":11542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5771,"num_write_kb":369344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"503'5461","reported_seq":5584,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-05-15T23:26:04.971539+0000","last_change":"2024-05-15T23:26:04.382948+0000","last_active":"2024-05-15T23:26:04.971539+0000","last_peered":"2024-05-15T23:26:04.971539+0000","last_clean":"2024-05-15T23:26:04.971539+0000","last_became_active":"2024-05-15T23:24:55.122178+0000","last_became_peered":"2024-05-15T23:24:55.122178+0000","last_unstale":"2024-05-15T23:26:04.971539+0000","last_undegraded":"2024-05-15T23:26:04.971539+0000","last_fullsized":"2024-05-15T23:26:04.971539+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'5408","last_scrub_stamp":"2024-05-15T23:26:04.382757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:26:04.382757+0000","objects_scrubbed":5363,"log_size":5461,"log_dups_size":0,"ondisk_log_size":5461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:27:10.162109+0000","scrub_duration":932,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357892096,"num_objects":5461,"num_object_clones":0,"num_object_copies":10922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5461,"num_write_kb":349504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"503'5647","reported_seq":6941,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-05-15T23:26:05.616134+0000","last_change":"2024-05-15T23:25:38.082859+0000","last_active":"2024-05-15T23:26:05.616134+0000","last_peered":"2024-05-15T23:26:05.616134+0000","last_clean":"2024-05-15T23:26:05.616134+0000","last_became_active":"2024-05-15T23:25:34.037025+0000","last_became_peered":"2024-05-15T23:25:34.037025+0000","last_unstale":"2024-05-15T23:26:05.616134+0000","last_undegraded":"2024-05-15T23:26:05.616134+0000","last_fullsized":"2024-05-15T23:26:05.616134+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":495,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:12.162550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370081792,"num_objects":5647,"num_object_clones":0,"num_object_copies":11294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5647,"num_write_kb":361408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":612,"num_bytes_recovered":40108032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.0","version":"503'5702","reported_seq":5763,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-05-15T23:26:05.956606+0000","last_change":"2024-05-15T23:24:55.124437+0000","last_active":"2024-05-15T23:26:05.956606+0000","last_peered":"2024-05-15T23:26:05.956606+0000","last_clean":"2024-05-15T23:26:05.956606+0000","last_became_active":"2024-05-15T23:24:55.123587+0000","last_became_peered":"2024-05-15T23:24:55.123587+0000","last_unstale":"2024-05-15T23:26:05.956606+0000","last_undegraded":"2024-05-15T23:26:05.956606+0000","last_fullsized":"2024-05-15T23:26:05.956606+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:24:54.096267+0000","objects_scrubbed":0,"log_size":5702,"log_dups_size":0,"ondisk_log_size":5702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:26:15.246805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373686272,"num_objects":5702,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5702,"num_write_kb":364928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"505'5902","reported_seq":6030,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-05-15T23:26:07.772312+0000","last_change":"2024-05-15T23:26:04.365440+0000","last_active":"2024-05-15T23:26:07.772312+0000","last_peered":"2024-05-15T23:26:07.772312+0000","last_clean":"2024-05-15T23:26:07.772312+0000","last_became_active":"2024-05-15T23:24:55.124153+0000","last_became_peered":"2024-05-15T23:24:55.124153+0000","last_unstale":"2024-05-15T23:26:07.772312+0000","last_undegraded":"2024-05-15T23:26:07.772312+0000","last_fullsized":"2024-05-15T23:26:07.772312+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'5601","last_scrub_stamp":"2024-05-15T23:26:04.365222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:24:54.096267+0000","last_clean_scrub_stamp":"2024-05-15T23:26:04.365222+0000","objects_scrubbed":5560,"log_size":5902,"log_dups_size":0,"ondisk_log_size":5902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:27:17.576562+0000","scrub_duration":929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386793472,"num_objects":5902,"num_object_clones":0,"num_object_copies":11804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5902,"num_write_kb":377728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1061,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-05-15T23:26:02.101782+0000","last_change":"2024-05-15T23:25:47.899431+0000","last_active":"2024-05-15T23:26:02.101782+0000","last_peered":"2024-05-15T23:26:02.101782+0000","last_clean":"2024-05-15T23:26:02.101782+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:26:02.101782+0000","last_undegraded":"2024-05-15T23:26:02.101782+0000","last_fullsized":"2024-05-15T23:26:02.101782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:25:47.899301+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:25:47.899301+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:27:14.442348+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":5990580224,"num_objects":91409,"num_object_clones":0,"num_object_copies":182818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91409,"num_write_kb":5850176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1214,"num_bytes_recovered":79560704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11998461952,"data_stored":11998461952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91741,"ondisk_log_size":91741,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351244,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1325744,"kb_used_data":748964,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92419408,"statfs":{"total":95995035648,"available":94637473792,"internally_reserved":0,"allocated":766939136,"data_stored":766173330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":590608836},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580177,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1392024,"kb_used_data":1072080,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92353128,"statfs":{"total":95995035648,"available":94569603072,"internally_reserved":0,"allocated":1097809920,"data_stored":1097053740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":327612864},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776409,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2223384,"kb_used_data":1786832,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":91521768,"statfs":{"total":95995035648,"available":93718290432,"internally_reserved":0,"allocated":1829715968,"data_stored":1828959788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":447019456},"hb_peers":[0,1,2,3,4,5,6,7,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266977,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1858976,"kb_used_data":1452952,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":91886176,"statfs":{"total":95995035648,"available":94091444224,"internally_reserved":0,"allocated":1487822848,"data_stored":1487063116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":415758794},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495912,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575169,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1635096,"kb_used_data":1098576,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92110056,"statfs":{"total":95995035648,"available":94320697344,"internally_reserved":0,"allocated":1124941824,"data_stored":1124185644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":549386694},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016579,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1249835483267,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150008,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2689816,"kb_used_data":2158864,"kb_used_omap":1,"kb_used_meta":530942,"kb_avail":91055336,"statfs":{"total":95995035648,"available":93240664064,"internally_reserved":0,"allocated":2210676736,"data_stored":2209920556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":543685063},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986480,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1493528,"kb_used_data":1069456,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92251624,"statfs":{"total":95995035648,"available":94465662976,"internally_reserved":0,"allocated":1095122944,"data_stored":1094366764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790253,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2734840,"kb_used_data":2252456,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":91010312,"statfs":{"total":95995035648,"available":93194559488,"internally_reserved":0,"allocated":2306514944,"data_stored":2305746656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1125318656,"data_stored":1125318656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2229862400,"data_stored":2229862400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1103101952,"data_stored":1103101952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2318336000,"data_stored":2318336000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1494810624,"data_stored":1494810624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1850277888,"data_stored":1850277888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1105264640,"data_stored":1105264640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":771489792,"data_stored":771489792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:26:11.309 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-15T23:26:11.309 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-05-15T23:26:11.378 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:26:11.410 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:11.461 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 64 16 99168 99152 96.808 85.0625 0.0494278 0.0103282 2024-05-15T23:26:11.865 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:26:11.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:11.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:26:11.871 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:11.970 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:11.999 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:11 smithi084 ceph-mon[39784]: from='client.36362 v1:172.21.15.84:0/3419406325' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:26:11.999 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:11 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:26:11.999 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:11 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:26:11.999 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:11 smithi084 ceph-mon[39784]: osdmap e506: 12 total, 11 up, 8 in 2024-05-15T23:26:12.000 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:11 smithi084 ceph-mon[39784]: pgmap v1352: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-05-15T23:26:12.002 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:12.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:11 smithi090 ceph-mon[44434]: from='client.36362 v1:172.21.15.84:0/3419406325' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:26:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:11 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:26:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:11 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:26:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:11 smithi090 ceph-mon[44434]: osdmap e506: 12 total, 11 up, 8 in 2024-05-15T23:26:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:11 smithi090 ceph-mon[44434]: pgmap v1352: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-05-15T23:26:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:11 smithi088 ceph-mon[44699]: from='client.36362 v1:172.21.15.84:0/3419406325' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:26:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:11 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:26:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:11 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:26:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:11 smithi088 ceph-mon[44699]: osdmap e506: 12 total, 11 up, 8 in 2024-05-15T23:26:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:11 smithi088 ceph-mon[44699]: pgmap v1352: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-05-15T23:26:12.139 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:12.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:12.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:12.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:26:12.279 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:12.313 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:26:12.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:12.317 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:26:12.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:26:12.460 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:12.461 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 65 16 100635 100619 96.7291 91.6875 0.00472953 0.0103362 2024-05-15T23:26:13.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:13 smithi088 ceph-mon[44699]: osdmap e507: 12 total, 11 up, 8 in 2024-05-15T23:26:13.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:13 smithi088 ceph-mon[44699]: pgmap v1354: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 6.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-05-15T23:26:13.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3606806166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:13.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:13 smithi088 ceph-mon[44699]: from='client.36368 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:13.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:13 smithi084 ceph-mon[39784]: osdmap e507: 12 total, 11 up, 8 in 2024-05-15T23:26:13.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:13 smithi084 ceph-mon[39784]: pgmap v1354: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 6.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-05-15T23:26:13.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3606806166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:13.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:13 smithi084 ceph-mon[39784]: from='client.36368 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:13.372 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:26:13.372 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:26:13.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:26:13.373 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:26:13.373 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.373 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278178 4.d 4:b4a56e92:::benchmark_data_smithi090_104448_object104940:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:26:13.373 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.204118+0000", 2024-05-15T23:26:13.373 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.16034457000001, 2024-05-15T23:26:13.373 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22432159900000001, 2024-05-15T23:26:13.373 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.373 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.373 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.373 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.373 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:13.373 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:13.373 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278178 2024-05-15T23:26:13.373 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.374 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.374 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.374 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.374 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.204118+0000", 2024-05-15T23:26:13.374 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.374 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.374 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.374 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.375 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.204118+0000", 2024-05-15T23:26:13.375 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.375 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.375 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.375 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.375 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.204118+0000", 2024-05-15T23:26:13.375 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.13e-07 2024-05-15T23:26:13.375 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.375 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.375 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.375 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.204121+0000", 2024-05-15T23:26:13.375 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.367e-06 2024-05-15T23:26:13.375 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.375 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.204125+0000", 2024-05-15T23:26:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5000000000000001e-06 2024-05-15T23:26:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.204130+0000", 2024-05-15T23:26:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.848e-06 2024-05-15T23:26:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.208629+0000", 2024-05-15T23:26:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004498747 2024-05-15T23:26:13.376 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.377 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.377 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.208753+0000", 2024-05-15T23:26:13.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000124733 2024-05-15T23:26:13.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.377 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:26:13.377 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.208844+0000", 2024-05-15T23:26:13.377 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0866999999999996e-05 2024-05-15T23:26:13.377 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.377 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.377 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.217585+0000", 2024-05-15T23:26:13.378 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008741034 2024-05-15T23:26:13.378 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.428366+0000", 2024-05-15T23:26:13.378 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21078121999999999 2024-05-15T23:26:13.378 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.378 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.378 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.378 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.428399+0000", 2024-05-15T23:26:13.378 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2113999999999998e-05 2024-05-15T23:26:13.378 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.428439+0000", 2024-05-15T23:26:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0556000000000003e-05 2024-05-15T23:26:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278185 4.d 4:bd70842c:::benchmark_data_smithi090_104448_object104947:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:26:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.209728+0000", 2024-05-15T23:26:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.15473439499999, 2024-05-15T23:26:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21882157399999999, 2024-05-15T23:26:13.379 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278185 2024-05-15T23:26:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.209728+0000", 2024-05-15T23:26:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.380 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.209728+0000", 2024-05-15T23:26:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.209730+0000", 2024-05-15T23:26:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.86e-06 2024-05-15T23:26:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.209735+0000", 2024-05-15T23:26:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4380000000000003e-06 2024-05-15T23:26:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.381 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.209748+0000", 2024-05-15T23:26:13.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3152999999999999e-05 2024-05-15T23:26:13.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.382 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.382 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.209756+0000", 2024-05-15T23:26:13.382 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6559999999999992e-06 2024-05-15T23:26:13.382 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.382 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.212660+0000", 2024-05-15T23:26:13.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029041710000000001 2024-05-15T23:26:13.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.383 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.212788+0000", 2024-05-15T23:26:13.383 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000128169 2024-05-15T23:26:13.383 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.383 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.383 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:26:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.212901+0000", 2024-05-15T23:26:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000113251 2024-05-15T23:26:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.217621+0000", 2024-05-15T23:26:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0047191280000000004 2024-05-15T23:26:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.428471+0000", 2024-05-15T23:26:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21085094200000001 2024-05-15T23:26:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.384 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.428509+0000", 2024-05-15T23:26:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7642999999999999e-05 2024-05-15T23:26:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.428549+0000", 2024-05-15T23:26:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0163000000000002e-05 2024-05-15T23:26:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278196 4.d 4:b76d4705:::benchmark_data_smithi090_104448_object104958:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:26:13.385 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.226357+0000", 2024-05-15T23:26:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.13810478900001, 2024-05-15T23:26:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208817265, 2024-05-15T23:26:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278196 2024-05-15T23:26:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.386 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226357+0000", 2024-05-15T23:26:13.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226357+0000", 2024-05-15T23:26:13.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226360+0000", 2024-05-15T23:26:13.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3420000000000002e-06 2024-05-15T23:26:13.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.388 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226368+0000", 2024-05-15T23:26:13.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2900000000000002e-06 2024-05-15T23:26:13.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.388 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.388 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226382+0000", 2024-05-15T23:26:13.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3608000000000001e-05 2024-05-15T23:26:13.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.388 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.388 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226390+0000", 2024-05-15T23:26:13.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1149999999999994e-06 2024-05-15T23:26:13.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226440+0000", 2024-05-15T23:26:13.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0476000000000003e-05 2024-05-15T23:26:13.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226533+0000", 2024-05-15T23:26:13.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3282999999999994e-05 2024-05-15T23:26:13.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:26:13.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226648+0000", 2024-05-15T23:26:13.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000114387 2024-05-15T23:26:13.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.236296+0000", 2024-05-15T23:26:13.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096484080000000007 2024-05-15T23:26:13.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.435080+0000", 2024-05-15T23:26:13.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19878395500000001 2024-05-15T23:26:13.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.435121+0000", 2024-05-15T23:26:13.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0774e-05 2024-05-15T23:26:13.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.435175+0000", 2024-05-15T23:26:13.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3627e-05 2024-05-15T23:26:13.391 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.391 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.391 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.391 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278201 4.d 4:b6101031:::benchmark_data_smithi090_104448_object104963:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:26:13.391 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.226528+0000", 2024-05-15T23:26:13.391 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.137934202, 2024-05-15T23:26:13.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.208752454, 2024-05-15T23:26:13.392 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.392 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.392 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.392 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.392 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:13.392 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:13.392 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278201 2024-05-15T23:26:13.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.392 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226528+0000", 2024-05-15T23:26:13.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226528+0000", 2024-05-15T23:26:13.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226529+0000", 2024-05-15T23:26:13.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8199999999999999e-07 2024-05-15T23:26:13.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226532+0000", 2024-05-15T23:26:13.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7519999999999998e-06 2024-05-15T23:26:13.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226539+0000", 2024-05-15T23:26:13.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4169999999999997e-06 2024-05-15T23:26:13.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.226544+0000", 2024-05-15T23:26:13.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6289999999999998e-06 2024-05-15T23:26:13.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.231428+0000", 2024-05-15T23:26:13.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0048830419999999998 2024-05-15T23:26:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.231522+0000", 2024-05-15T23:26:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4277999999999994e-05 2024-05-15T23:26:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:26:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.231615+0000", 2024-05-15T23:26:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3298e-05 2024-05-15T23:26:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.237393+0000", 2024-05-15T23:26:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0057775719999999999 2024-05-15T23:26:13.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.435216+0000", 2024-05-15T23:26:13.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.197823205 2024-05-15T23:26:13.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.435244+0000", 2024-05-15T23:26:13.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8017e-05 2024-05-15T23:26:13.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.435280+0000", 2024-05-15T23:26:13.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6461999999999998e-05 2024-05-15T23:26:13.397 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.397 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.397 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.397 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278218 4.d 4:b6a9e0eb:::benchmark_data_smithi090_104448_object104980:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:26:13.397 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.238248+0000", 2024-05-15T23:26:13.397 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.12621360899999, 2024-05-15T23:26:13.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.201633439, 2024-05-15T23:26:13.397 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.397 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.397 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.397 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.397 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:13.397 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:13.398 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278218 2024-05-15T23:26:13.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.398 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238248+0000", 2024-05-15T23:26:13.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238248+0000", 2024-05-15T23:26:13.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238250+0000", 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3990000000000001e-06 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238254+0000", 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4139999999999996e-06 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238263+0000", 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0880000000000001e-06 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.399 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238269+0000", 2024-05-15T23:26:13.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7309999999999998e-06 2024-05-15T23:26:13.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238323+0000", 2024-05-15T23:26:13.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4277999999999998e-05 2024-05-15T23:26:13.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238456+0000", 2024-05-15T23:26:13.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013282999999999999 2024-05-15T23:26:13.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:26:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.238550+0000", 2024-05-15T23:26:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3916e-05 2024-05-15T23:26:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.243359+0000", 2024-05-15T23:26:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0048091829999999999 2024-05-15T23:26:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.439791+0000", 2024-05-15T23:26:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19643179 2024-05-15T23:26:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.439835+0000", 2024-05-15T23:26:13.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3438e-05 2024-05-15T23:26:13.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.439882+0000", 2024-05-15T23:26:13.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7372e-05 2024-05-15T23:26:13.402 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.402 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.402 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.402 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278247 4.d 4:bf4844ca:::benchmark_data_smithi090_104448_object105009:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:26:13.403 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.269137+0000", 2024-05-15T23:26:13.403 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.095325237, 2024-05-15T23:26:13.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17087540700000001, 2024-05-15T23:26:13.403 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.403 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.403 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.403 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.403 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:13.403 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:13.403 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278247 2024-05-15T23:26:13.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.403 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269137+0000", 2024-05-15T23:26:13.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269137+0000", 2024-05-15T23:26:13.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269139+0000", 2024-05-15T23:26:13.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.1390000000000002e-06 2024-05-15T23:26:13.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269145+0000", 2024-05-15T23:26:13.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.711e-06 2024-05-15T23:26:13.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.405 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269158+0000", 2024-05-15T23:26:13.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3174e-05 2024-05-15T23:26:13.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.405 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269167+0000", 2024-05-15T23:26:13.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6249999999999996e-06 2024-05-15T23:26:13.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.405 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269226+0000", 2024-05-15T23:26:13.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9342999999999997e-05 2024-05-15T23:26:13.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269313+0000", 2024-05-15T23:26:13.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7183999999999997e-05 2024-05-15T23:26:13.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:26:13.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.269432+0000", 2024-05-15T23:26:13.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000118987 2024-05-15T23:26:13.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.274514+0000", 2024-05-15T23:26:13.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005082128 2024-05-15T23:26:13.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.439926+0000", 2024-05-15T23:26:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16541231200000001 2024-05-15T23:26:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.439978+0000", 2024-05-15T23:26:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1944999999999998e-05 2024-05-15T23:26:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.440012+0000", 2024-05-15T23:26:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3859000000000002e-05 2024-05-15T23:26:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.409 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278301 4.d 4:b78c14fc:::benchmark_data_smithi090_104448_object105063:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:26:13.409 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.324748+0000", 2024-05-15T23:26:13.409 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.03971438000002, 2024-05-15T23:26:13.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12313596, 2024-05-15T23:26:13.409 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.409 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.409 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.409 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.409 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:13.409 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:13.409 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278301 2024-05-15T23:26:13.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.409 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324748+0000", 2024-05-15T23:26:13.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324748+0000", 2024-05-15T23:26:13.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324749+0000", 2024-05-15T23:26:13.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4419999999999999e-06 2024-05-15T23:26:13.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324754+0000", 2024-05-15T23:26:13.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6369999999999998e-06 2024-05-15T23:26:13.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324764+0000", 2024-05-15T23:26:13.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0465999999999999e-05 2024-05-15T23:26:13.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324772+0000", 2024-05-15T23:26:13.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4649999999999997e-06 2024-05-15T23:26:13.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324833+0000", 2024-05-15T23:26:13.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0776999999999998e-05 2024-05-15T23:26:13.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.412 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.412 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.324973+0000", 2024-05-15T23:26:13.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000140378 2024-05-15T23:26:13.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:26:13.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.325060+0000", 2024-05-15T23:26:13.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6712000000000006e-05 2024-05-15T23:26:13.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.328886+0000", 2024-05-15T23:26:13.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0038261649999999999 2024-05-15T23:26:13.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.447814+0000", 2024-05-15T23:26:13.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11892828699999999 2024-05-15T23:26:13.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.447845+0000", 2024-05-15T23:26:13.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0970999999999999e-05 2024-05-15T23:26:13.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.447884+0000", 2024-05-15T23:26:13.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8659999999999999e-05 2024-05-15T23:26:13.414 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.414 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.414 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.414 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278311 4.d 4:b37c2762:::benchmark_data_smithi090_104448_object105073:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:26:13.415 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.338775+0000", 2024-05-15T23:26:13.415 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.025687344, 2024-05-15T23:26:13.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11404146900000001, 2024-05-15T23:26:13.415 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.415 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.415 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.415 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.415 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:13.415 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:13.415 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278311 2024-05-15T23:26:13.415 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.415 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338775+0000", 2024-05-15T23:26:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338775+0000", 2024-05-15T23:26:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338777+0000", 2024-05-15T23:26:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8729999999999999e-06 2024-05-15T23:26:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.416 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338782+0000", 2024-05-15T23:26:13.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4389999999999999e-06 2024-05-15T23:26:13.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338793+0000", 2024-05-15T23:26:13.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1182000000000001e-05 2024-05-15T23:26:13.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338800+0000", 2024-05-15T23:26:13.417 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4459999999999998e-06 2024-05-15T23:26:13.417 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.417 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.417 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.417 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338887+0000", 2024-05-15T23:26:13.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7468999999999999e-05 2024-05-15T23:26:13.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.338969+0000", 2024-05-15T23:26:13.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1698000000000003e-05 2024-05-15T23:26:13.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:26:13.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.339040+0000", 2024-05-15T23:26:13.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1340000000000005e-05 2024-05-15T23:26:13.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.347909+0000", 2024-05-15T23:26:13.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0088689029999999992 2024-05-15T23:26:13.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.452710+0000", 2024-05-15T23:26:13.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.104800421 2024-05-15T23:26:13.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.452763+0000", 2024-05-15T23:26:13.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3227999999999999e-05 2024-05-15T23:26:13.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.452816+0000", 2024-05-15T23:26:13.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3470000000000001e-05 2024-05-15T23:26:13.420 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.420 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.420 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.420 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278317 4.d 4:b931cefd:::benchmark_data_smithi090_104448_object105079:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:26:13.420 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.340282+0000", 2024-05-15T23:26:13.420 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.02418044799998, 2024-05-15T23:26:13.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.112684298, 2024-05-15T23:26:13.420 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.420 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.420 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.420 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.421 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:13.421 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:13.421 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278317 2024-05-15T23:26:13.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.421 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.340282+0000", 2024-05-15T23:26:13.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.340282+0000", 2024-05-15T23:26:13.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.340283+0000", 2024-05-15T23:26:13.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5689999999999999e-06 2024-05-15T23:26:13.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.340289+0000", 2024-05-15T23:26:13.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8039999999999998e-06 2024-05-15T23:26:13.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.422 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.422 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.340301+0000", 2024-05-15T23:26:13.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1783999999999999e-05 2024-05-15T23:26:13.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.340308+0000", 2024-05-15T23:26:13.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5750000000000003e-06 2024-05-15T23:26:13.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.343824+0000", 2024-05-15T23:26:13.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0035159119999999999 2024-05-15T23:26:13.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.343942+0000", 2024-05-15T23:26:13.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000118126 2024-05-15T23:26:13.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:26:13.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.344062+0000", 2024-05-15T23:26:13.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000119957 2024-05-15T23:26:13.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.348054+0000", 2024-05-15T23:26:13.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003991199 2024-05-15T23:26:13.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.452864+0000", 2024-05-15T23:26:13.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10481017300000001 2024-05-15T23:26:13.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.452916+0000", 2024-05-15T23:26:13.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1983e-05 2024-05-15T23:26:13.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.452966+0000", 2024-05-15T23:26:13.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0216000000000002e-05 2024-05-15T23:26:13.426 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.426 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.426 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.426 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278324 4.d 4:b32aacc2:::benchmark_data_smithi090_104448_object105086:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:26:13.426 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.349495+0000", 2024-05-15T23:26:13.426 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.01496707699999, 2024-05-15T23:26:13.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11114384200000001, 2024-05-15T23:26:13.427 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.427 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.427 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.427 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.427 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:13.427 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:13.427 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278324 2024-05-15T23:26:13.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.427 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349495+0000", 2024-05-15T23:26:13.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349495+0000", 2024-05-15T23:26:13.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.428 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.428 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349496+0000", 2024-05-15T23:26:13.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5700000000000002e-07 2024-05-15T23:26:13.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349499+0000", 2024-05-15T23:26:13.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8779999999999998e-06 2024-05-15T23:26:13.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349504+0000", 2024-05-15T23:26:13.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.519e-06 2024-05-15T23:26:13.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349509+0000", 2024-05-15T23:26:13.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6609999999999999e-06 2024-05-15T23:26:13.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349570+0000", 2024-05-15T23:26:13.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1458999999999996e-05 2024-05-15T23:26:13.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349647+0000", 2024-05-15T23:26:13.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6907000000000001e-05 2024-05-15T23:26:13.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:26:13.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.349717+0000", 2024-05-15T23:26:13.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9311000000000001e-05 2024-05-15T23:26:13.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.353196+0000", 2024-05-15T23:26:13.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034794919999999998 2024-05-15T23:26:13.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.460561+0000", 2024-05-15T23:26:13.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107364949 2024-05-15T23:26:13.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.460591+0000", 2024-05-15T23:26:13.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9745999999999999e-05 2024-05-15T23:26:13.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.460639+0000", 2024-05-15T23:26:13.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8162999999999999e-05 2024-05-15T23:26:13.433 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.433 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.433 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.433 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278331 4.d 4:be6484c2:::benchmark_data_smithi090_104448_object105093:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:26:13.433 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.359850+0000", 2024-05-15T23:26:13.433 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.00461206900002, 2024-05-15T23:26:13.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11042561400000001, 2024-05-15T23:26:13.433 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.433 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.433 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.433 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.434 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:13.434 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:13.434 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278331 2024-05-15T23:26:13.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.434 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359850+0000", 2024-05-15T23:26:13.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359850+0000", 2024-05-15T23:26:13.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359851+0000", 2024-05-15T23:26:13.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.344e-06 2024-05-15T23:26:13.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359856+0000", 2024-05-15T23:26:13.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2629999999999997e-06 2024-05-15T23:26:13.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.435 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.435 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359864+0000", 2024-05-15T23:26:13.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1249999999999993e-06 2024-05-15T23:26:13.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359869+0000", 2024-05-15T23:26:13.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2650000000000004e-06 2024-05-15T23:26:13.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359908+0000", 2024-05-15T23:26:13.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8668000000000003e-05 2024-05-15T23:26:13.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.359986+0000", 2024-05-15T23:26:13.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7886000000000006e-05 2024-05-15T23:26:13.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:26:13.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.360051+0000", 2024-05-15T23:26:13.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5531999999999995e-05 2024-05-15T23:26:13.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.364149+0000", 2024-05-15T23:26:13.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0040982019999999996 2024-05-15T23:26:13.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.470176+0000", 2024-05-15T23:26:13.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.106026286 2024-05-15T23:26:13.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.470229+0000", 2024-05-15T23:26:13.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3693000000000002e-05 2024-05-15T23:26:13.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.470276+0000", 2024-05-15T23:26:13.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6350000000000002e-05 2024-05-15T23:26:13.438 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.438 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.439 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.439 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278342 4.d 4:b6201b5e:::benchmark_data_smithi090_104448_object105104:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-05-15T23:26:13.439 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.371607+0000", 2024-05-15T23:26:13.439 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 488.99285478600001, 2024-05-15T23:26:13.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.11209290099999999, 2024-05-15T23:26:13.439 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.439 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.439 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.439 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.439 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:13.439 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:13.439 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278342 2024-05-15T23:26:13.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.440 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371607+0000", 2024-05-15T23:26:13.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371607+0000", 2024-05-15T23:26:13.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371609+0000", 2024-05-15T23:26:13.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3430000000000001e-06 2024-05-15T23:26:13.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371613+0000", 2024-05-15T23:26:13.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6469999999999997e-06 2024-05-15T23:26:13.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371623+0000", 2024-05-15T23:26:13.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0131e-05 2024-05-15T23:26:13.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371631+0000", 2024-05-15T23:26:13.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9789999999999993e-06 2024-05-15T23:26:13.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371694+0000", 2024-05-15T23:26:13.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2917999999999996e-05 2024-05-15T23:26:13.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371811+0000", 2024-05-15T23:26:13.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000116381 2024-05-15T23:26:13.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:26:13.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.371904+0000", 2024-05-15T23:26:13.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3083000000000003e-05 2024-05-15T23:26:13.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.375745+0000", 2024-05-15T23:26:13.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00384074 2024-05-15T23:26:13.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.483632+0000", 2024-05-15T23:26:13.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107886998 2024-05-15T23:26:13.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.483664+0000", 2024-05-15T23:26:13.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2793000000000003e-05 2024-05-15T23:26:13.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.483700+0000", 2024-05-15T23:26:13.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5887999999999999e-05 2024-05-15T23:26:13.444 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.444 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.444 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.445 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278342 4.d 4:b6201b5e:::benchmark_data_smithi090_104448_object105104:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e345)", 2024-05-15T23:26:13.445 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.385798+0000", 2024-05-15T23:26:13.445 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 488.978664299, 2024-05-15T23:26:13.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.097873819000000001, 2024-05-15T23:26:13.445 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.445 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.445 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "waiting for ondisk", 2024-05-15T23:26:13.445 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.445 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:13.445 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:13.445 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278342 2024-05-15T23:26:13.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.445 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385798+0000", 2024-05-15T23:26:13.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385798+0000", 2024-05-15T23:26:13.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385798+0000", 2024-05-15T23:26:13.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0999999999999999e-07 2024-05-15T23:26:13.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385800+0000", 2024-05-15T23:26:13.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.1880000000000001e-06 2024-05-15T23:26:13.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385807+0000", 2024-05-15T23:26:13.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3169999999999998e-06 2024-05-15T23:26:13.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385815+0000", 2024-05-15T23:26:13.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1839999999999999e-06 2024-05-15T23:26:13.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.386530+0000", 2024-05-15T23:26:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00071482399999999997 2024-05-15T23:26:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for ondisk", 2024-05-15T23:26:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.386557+0000", 2024-05-15T23:26:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6783000000000001e-05 2024-05-15T23:26:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.483672+0000", 2024-05-15T23:26:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.097115013 2024-05-15T23:26:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.449 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.449 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.449 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:278350 4.d 4:b848d388:::benchmark_data_smithi090_104448_object105112:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e345)", 2024-05-15T23:26:13.449 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:04.385820+0000", 2024-05-15T23:26:13.449 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 488.97864194200002, 2024-05-15T23:26:13.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.097980265999999996, 2024-05-15T23:26:13.449 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.449 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.449 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.449 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.449 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:13.449 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:13.450 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 278350 2024-05-15T23:26:13.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.450 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385820+0000", 2024-05-15T23:26:13.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385820+0000", 2024-05-15T23:26:13.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385821+0000", 2024-05-15T23:26:13.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6400000000000003e-07 2024-05-15T23:26:13.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385823+0000", 2024-05-15T23:26:13.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.255e-06 2024-05-15T23:26:13.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385826+0000", 2024-05-15T23:26:13.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5109999999999999e-06 2024-05-15T23:26:13.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.385829+0000", 2024-05-15T23:26:13.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8040000000000002e-06 2024-05-15T23:26:13.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.386580+0000", 2024-05-15T23:26:13.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00075106100000000002 2024-05-15T23:26:13.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.386697+0000", 2024-05-15T23:26:13.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011701099999999999 2024-05-15T23:26:13.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:26:13.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.386827+0000", 2024-05-15T23:26:13.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000130004 2024-05-15T23:26:13.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.391672+0000", 2024-05-15T23:26:13.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0048445520000000002 2024-05-15T23:26:13.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.483736+0000", 2024-05-15T23:26:13.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092064423000000006 2024-05-15T23:26:13.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.483771+0000", 2024-05-15T23:26:13.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4900999999999997e-05 2024-05-15T23:26:13.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:04.483800+0000", 2024-05-15T23:26:13.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9280000000000001e-05 2024-05-15T23:26:13.454 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.454 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.454 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.455 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:94654 6.e 6:74360dd9:::benchmark_data_smithi090_131278_object94653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e505)", 2024-05-15T23:26:13.455 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:26:08.245467+0000", 2024-05-15T23:26:13.455 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 5.1189954889999996, 2024-05-15T23:26:13.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.117853851, 2024-05-15T23:26:13.455 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.455 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.455 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.455 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.455 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:26:13.455 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:26:13.455 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 94654 2024-05-15T23:26:13.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.456 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.245467+0000", 2024-05-15T23:26:13.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.245467+0000", 2024-05-15T23:26:13.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.245469+0000", 2024-05-15T23:26:13.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.984e-06 2024-05-15T23:26:13.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.245496+0000", 2024-05-15T23:26:13.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7506999999999999e-05 2024-05-15T23:26:13.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.245505+0000", 2024-05-15T23:26:13.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3030000000000005e-06 2024-05-15T23:26:13.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.245512+0000", 2024-05-15T23:26:13.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4590000000000003e-06 2024-05-15T23:26:13.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.284715+0000", 2024-05-15T23:26:13.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.039203100999999997 2024-05-15T23:26:13.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.284841+0000", 2024-05-15T23:26:13.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012596800000000001 2024-05-15T23:26:13.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:13.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.284960+0000", 2024-05-15T23:26:13.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011882100000000001 2024-05-15T23:26:13.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.360738+0000", 2024-05-15T23:26:13.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075777757000000001 2024-05-15T23:26:13.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.363236+0000", 2024-05-15T23:26:13.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024987030000000001 2024-05-15T23:26:13.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.460 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.363283+0000", 2024-05-15T23:26:13.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6560000000000001e-05 2024-05-15T23:26:13.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.460 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.363320+0000", 2024-05-15T23:26:13.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7688000000000002e-05 2024-05-15T23:26:13.460 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.460 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.460 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.460 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:94707 6.e 6:7f82f31e:::benchmark_data_smithi090_131278_object94706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e505)", 2024-05-15T23:26:13.461 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:26:08.275948+0000", 2024-05-15T23:26:13.461 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 5.0885137480000004, 2024-05-15T23:26:13.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.097771747000000006, 2024-05-15T23:26:13.461 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.461 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.461 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.461 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.461 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:26:13.461 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:26:13.461 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 94707 2024-05-15T23:26:13.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.461 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.275948+0000", 2024-05-15T23:26:13.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.275948+0000", 2024-05-15T23:26:13.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.275954+0000", 2024-05-15T23:26:13.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3020000000000002e-06 2024-05-15T23:26:13.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.276004+0000", 2024-05-15T23:26:13.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0299000000000001e-05 2024-05-15T23:26:13.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.276014+0000", 2024-05-15T23:26:13.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6630000000000005e-06 2024-05-15T23:26:13.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.276019+0000", 2024-05-15T23:26:13.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.631e-06 2024-05-15T23:26:13.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.287887+0000", 2024-05-15T23:26:13.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011867857000000001 2024-05-15T23:26:13.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.288003+0000", 2024-05-15T23:26:13.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000115964 2024-05-15T23:26:13.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.464 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:13.464 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.357552+0000", 2024-05-15T23:26:13.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.069549269999999996 2024-05-15T23:26:13.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.365919+0000", 2024-05-15T23:26:13.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0083670350000000001 2024-05-15T23:26:13.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.373610+0000", 2024-05-15T23:26:13.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0076904340000000003 2024-05-15T23:26:13.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.373663+0000", 2024-05-15T23:26:13.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3018e-05 2024-05-15T23:26:13.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.373720+0000", 2024-05-15T23:26:13.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7274000000000001e-05 2024-05-15T23:26:13.466 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.466 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.466 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.466 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:96693 6.e 6:7e8532f2:::benchmark_data_smithi090_131278_object96692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e505)", 2024-05-15T23:26:13.466 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:26:09.669971+0000", 2024-05-15T23:26:13.466 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 3.6944907979999999, 2024-05-15T23:26:13.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.103696179, 2024-05-15T23:26:13.467 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.467 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.467 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.467 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.467 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:26:13.467 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:26:13.467 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 96693 2024-05-15T23:26:13.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.467 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.669971+0000", 2024-05-15T23:26:13.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.669971+0000", 2024-05-15T23:26:13.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.669973+0000", 2024-05-15T23:26:13.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.886e-06 2024-05-15T23:26:13.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.670026+0000", 2024-05-15T23:26:13.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2735000000000002e-05 2024-05-15T23:26:13.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.670036+0000", 2024-05-15T23:26:13.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0361e-05 2024-05-15T23:26:13.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.670044+0000", 2024-05-15T23:26:13.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1419999999999993e-06 2024-05-15T23:26:13.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.763501+0000", 2024-05-15T23:26:13.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.093457042000000004 2024-05-15T23:26:13.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.763635+0000", 2024-05-15T23:26:13.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013397999999999999 2024-05-15T23:26:13.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:13.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.763785+0000", 2024-05-15T23:26:13.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014941800000000001 2024-05-15T23:26:13.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.767009+0000", 2024-05-15T23:26:13.471 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032239080000000002 2024-05-15T23:26:13.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.471 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.773589+0000", 2024-05-15T23:26:13.471 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0065803850000000002 2024-05-15T23:26:13.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.471 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.773626+0000", 2024-05-15T23:26:13.471 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6761999999999998e-05 2024-05-15T23:26:13.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.773667+0000", 2024-05-15T23:26:13.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1560000000000002e-05 2024-05-15T23:26:13.472 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.472 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.472 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.472 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:96710 6.e 6:71303ed7:::benchmark_data_smithi090_131278_object96709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e505)", 2024-05-15T23:26:13.472 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:26:09.684225+0000", 2024-05-15T23:26:13.472 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 3.680237091, 2024-05-15T23:26:13.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.096060770000000004, 2024-05-15T23:26:13.472 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.472 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.472 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.473 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.473 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:26:13.473 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:26:13.473 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 96710 2024-05-15T23:26:13.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.473 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.684225+0000", 2024-05-15T23:26:13.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.684225+0000", 2024-05-15T23:26:13.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.684227+0000", 2024-05-15T23:26:13.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2280000000000001e-06 2024-05-15T23:26:13.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.684271+0000", 2024-05-15T23:26:13.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4103000000000001e-05 2024-05-15T23:26:13.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.684282+0000", 2024-05-15T23:26:13.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0283e-05 2024-05-15T23:26:13.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.684290+0000", 2024-05-15T23:26:13.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0839999999999992e-06 2024-05-15T23:26:13.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.767129+0000", 2024-05-15T23:26:13.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082839641000000006 2024-05-15T23:26:13.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.767285+0000", 2024-05-15T23:26:13.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015611600000000001 2024-05-15T23:26:13.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:13.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.767431+0000", 2024-05-15T23:26:13.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014538999999999999 2024-05-15T23:26:13.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.773717+0000", 2024-05-15T23:26:13.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0062857049999999999 2024-05-15T23:26:13.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.780198+0000", 2024-05-15T23:26:13.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0064813029999999999 2024-05-15T23:26:13.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.780255+0000", 2024-05-15T23:26:13.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7021000000000002e-05 2024-05-15T23:26:13.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.477 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.477 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.780286+0000", 2024-05-15T23:26:13.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0895999999999997e-05 2024-05-15T23:26:13.478 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.478 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.478 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.478 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:98301 6.e 6:7ed317cd:::benchmark_data_smithi090_131278_object98300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e506)", 2024-05-15T23:26:13.478 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:26:10.822000+0000", 2024-05-15T23:26:13.478 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 2.5424623550000001, 2024-05-15T23:26:13.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.105313092, 2024-05-15T23:26:13.478 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.478 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.478 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.478 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.478 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:26:13.479 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:26:13.479 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 98301 2024-05-15T23:26:13.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.479 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.822000+0000", 2024-05-15T23:26:13.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.822000+0000", 2024-05-15T23:26:13.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.822001+0000", 2024-05-15T23:26:13.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3799999999999996e-07 2024-05-15T23:26:13.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.822064+0000", 2024-05-15T23:26:13.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2910000000000006e-05 2024-05-15T23:26:13.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.822072+0000", 2024-05-15T23:26:13.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8869999999999998e-06 2024-05-15T23:26:13.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.822078+0000", 2024-05-15T23:26:13.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9070000000000004e-06 2024-05-15T23:26:13.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.882889+0000", 2024-05-15T23:26:13.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.06081023 2024-05-15T23:26:13.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.883017+0000", 2024-05-15T23:26:13.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000128498 2024-05-15T23:26:13.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:13.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.883154+0000", 2024-05-15T23:26:13.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013733199999999999 2024-05-15T23:26:13.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.892756+0000", 2024-05-15T23:26:13.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.009601666 2024-05-15T23:26:13.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.927240+0000", 2024-05-15T23:26:13.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.034483595999999998 2024-05-15T23:26:13.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.927272+0000", 2024-05-15T23:26:13.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2407000000000003e-05 2024-05-15T23:26:13.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.927313+0000", 2024-05-15T23:26:13.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0720999999999999e-05 2024-05-15T23:26:13.483 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.483 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.483 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.484 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:100886 6.e 6:78d51e73:::benchmark_data_smithi090_131278_object100885:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e507)", 2024-05-15T23:26:13.484 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:26:12.625320+0000", 2024-05-15T23:26:13.484 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.73914184999999999, 2024-05-15T23:26:13.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.09467333, 2024-05-15T23:26:13.484 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:13.484 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:13.484 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:13.484 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:13.484 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:26:13.484 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:26:13.484 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 100886 2024-05-15T23:26:13.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.484 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:13.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.485 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:13.485 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:12.625320+0000", 2024-05-15T23:26:13.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.485 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:13.485 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:12.625320+0000", 2024-05-15T23:26:13.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:13.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.485 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:13.485 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:12.625321+0000", 2024-05-15T23:26:13.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1800000000000005e-07 2024-05-15T23:26:13.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:13.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:12.625354+0000", 2024-05-15T23:26:13.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2653000000000001e-05 2024-05-15T23:26:13.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:13.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:12.625361+0000", 2024-05-15T23:26:13.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1459999999999999e-06 2024-05-15T23:26:13.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:13.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:12.625367+0000", 2024-05-15T23:26:13.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2979999999999997e-06 2024-05-15T23:26:13.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:13.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:12.707236+0000", 2024-05-15T23:26:13.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.081868877000000007 2024-05-15T23:26:13.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:13.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:12.707322+0000", 2024-05-15T23:26:13.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5878999999999995e-05 2024-05-15T23:26:13.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:13.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:12.707417+0000", 2024-05-15T23:26:13.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5005000000000003e-05 2024-05-15T23:26:13.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:13.489 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:12.713119+0000", 2024-05-15T23:26:13.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0057017279999999997 2024-05-15T23:26:13.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.490 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.490 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:13.490 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:12.719927+0000", 2024-05-15T23:26:13.490 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.006807998 2024-05-15T23:26:13.490 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.490 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.490 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:13.490 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:12.719960+0000", 2024-05-15T23:26:13.490 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3600999999999999e-05 2024-05-15T23:26:13.490 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:13.490 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:13.490 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:13.490 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:12.719994+0000", 2024-05-15T23:26:13.490 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3327000000000001e-05 2024-05-15T23:26:13.491 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.491 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.491 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.491 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:13.491 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:13.491 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:26:13.492 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 66 16 102055 102039 96.6081 88.75 0.0109422 0.0103496 2024-05-15T23:26:13.492 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:26:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:13 smithi090 ceph-mon[44434]: osdmap e507: 12 total, 11 up, 8 in 2024-05-15T23:26:13.633 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:13 smithi090 ceph-mon[44434]: pgmap v1354: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 6.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-05-15T23:26:13.633 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3606806166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:13.633 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:13 smithi090 ceph-mon[44434]: from='client.36368 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:13.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:13.635 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:26:13.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:26:13.736 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:26:13.794 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:14.164 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:26:14.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:14.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:26:14.201 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T23:26:14.276 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:26:14.329 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:14.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:14 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:14.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:14 smithi084 ceph-mon[39784]: from='client.36368 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:26:14.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:14 smithi084 ceph-mon[39784]: osdmap e508: 12 total, 11 up, 8 in 2024-05-15T23:26:14.378 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:26:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:14.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:26:14.387 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:14 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:14.387 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:14 smithi088 ceph-mon[44699]: from='client.36368 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:26:14.387 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:14 smithi088 ceph-mon[44699]: osdmap e508: 12 total, 11 up, 8 in 2024-05-15T23:26:14.461 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 67 16 103657 103641 96.6605 100.125 0.00757238 0.0103438 2024-05-15T23:26:14.485 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:14.525 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:14 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:14 smithi090 ceph-mon[44434]: from='client.36368 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:26:14.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:14 smithi090 ceph-mon[44434]: osdmap e508: 12 total, 11 up, 8 in 2024-05-15T23:26:14.601 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:14.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:14.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:14.668 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:26:14.721 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:14.761 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:15.246 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:26:15.246 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:26:15.246 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:26:15.246 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:26:15.246 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:26:15.246 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:26:15.247 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:26:15.247 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:26:15.247 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:26:15.247 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:26:15.247 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:26:15.247 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:26:15.247 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:26:15.247 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:26:15.247 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:26:15.247 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:26:15.247 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:26:15.247 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:26:15.248 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:26:15.248 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-05-15T23:26:15.248 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:26:15.248 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:26:15.248 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:26:15.255 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:15 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3059425202' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:15.255 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:15 smithi088 ceph-mon[44699]: pgmap v1356: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 6.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-05-15T23:26:15.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:15 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3059425202' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:15.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:15 smithi084 ceph-mon[39784]: pgmap v1356: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 6.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-05-15T23:26:15.423 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:26:15.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:15.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:26:15.461 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 68 15 105216 105201 96.6727 97.5 0.0116335 0.0103429 2024-05-15T23:26:15.527 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:26:15.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:15 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3059425202' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:15.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:15 smithi090 ceph-mon[44434]: pgmap v1356: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 6.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-05-15T23:26:15.559 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:15.583 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:26:15.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:15.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:26:15.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:26:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:15.762 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:26:15.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:15.765 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:26:15.866 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:26:15.900 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:16.037 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:26:16.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:16.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:26:16.070 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:26:16.071 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:26:16.142 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:26:16.178 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:16.204 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:26:16.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:16.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:26:16.312 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:26:16.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.344 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:16.368 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:26:16.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:16.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:26:16.461 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 69 16 106807 106791 96.7117 99.375 0.00720793 0.0103386 2024-05-15T23:26:16.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:26:16.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.501 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:16.529 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:26:16.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:16.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:26:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:16.634 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:26:16.665 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:16.696 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:26:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:16.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:26:16.803 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:26:16.870 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:16.903 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:26:16.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:16.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:26:16.920 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:26:16.920 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:26:17.009 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:26:17.048 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:17.073 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:26:17.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:17.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:26:17.138 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:17 smithi088 ceph-mon[44699]: pgmap v1357: 17 pgs: 17 active+clean; 6.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-05-15T23:26:17.138 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:17 smithi088 ceph-mon[44699]: osdmap e509: 12 total, 11 up, 8 in 2024-05-15T23:26:17.180 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:26:17.191 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:26:17.222 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:17.250 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:26:17.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:17.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:26:17.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:26:17.386 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:17 smithi090 ceph-mon[44434]: pgmap v1357: 17 pgs: 17 active+clean; 6.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-05-15T23:26:17.386 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:17 smithi090 ceph-mon[44434]: osdmap e509: 12 total, 11 up, 8 in 2024-05-15T23:26:17.388 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:17.413 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:26:17.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:17.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:26:17.461 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 70 16 108278 108262 96.6434 91.9375 0.00301096 0.0103425 2024-05-15T23:26:17.516 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:17.518 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:26:17.562 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:17.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:17 smithi084 ceph-mon[39784]: pgmap v1357: 17 pgs: 17 active+clean; 6.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-05-15T23:26:17.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:17 smithi084 ceph-mon[39784]: osdmap e509: 12 total, 11 up, 8 in 2024-05-15T23:26:18.071 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:26:18.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:18 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:26:18.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:18 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:26:18.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:18 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:26:18.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:18 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:26:18.461 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 71 16 109803 109787 96.6245 95.3125 0.00646998 0.0103479 2024-05-15T23:26:18.516 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:18.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:18 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:26:18.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:18 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:26:18.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:18 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:26:18.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:18 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:26:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:18 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:26:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:18 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:26:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:18 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:26:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:18 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:26:19.037 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:26:19.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:19.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:26:19.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:26:19.173 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:19.203 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:26:19.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:19.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:26:19.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:26:19.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:19 smithi088 ceph-mon[44699]: osdmap e510: 12 total, 11 up, 8 in 2024-05-15T23:26:19.326 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:19 smithi088 ceph-mon[44699]: pgmap v1360: 17 pgs: 17 active+clean; 6.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-05-15T23:26:19.343 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:19.461 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 72 16 111397 111381 96.6661 99.625 0.00515906 0.0103431 2024-05-15T23:26:19.466 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:19 smithi084 ceph-mon[39784]: osdmap e510: 12 total, 11 up, 8 in 2024-05-15T23:26:19.466 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:19 smithi084 ceph-mon[39784]: pgmap v1360: 17 pgs: 17 active+clean; 6.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-05-15T23:26:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:19 smithi090 ceph-mon[44434]: osdmap e510: 12 total, 11 up, 8 in 2024-05-15T23:26:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:19 smithi090 ceph-mon[44434]: pgmap v1360: 17 pgs: 17 active+clean; 6.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-05-15T23:26:19.603 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T23:26:20.030 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:26:20.462 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 73 16 112841 112825 96.5781 90.25 0.0146385 0.0103531 2024-05-15T23:26:21.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:21 smithi084 ceph-mon[39784]: pgmap v1361: 17 pgs: 17 active+clean; 6.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-05-15T23:26:21.462 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 74 16 114365 114349 96.5601 95.25 0.00670918 0.0103548 2024-05-15T23:26:21.473 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:21 smithi088 ceph-mon[44699]: pgmap v1361: 17 pgs: 17 active+clean; 6.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-05-15T23:26:21.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:21 smithi090 ceph-mon[44434]: pgmap v1361: 17 pgs: 17 active+clean; 6.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-05-15T23:26:22.195 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:26:22.195 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:26:22.462 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 75 16 115820 115804 96.485 90.9375 0.00701511 0.0103625 2024-05-15T23:26:23.072 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T23:26:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:26:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:26:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:23 smithi088 ceph-mon[44699]: osdmap e511: 12 total, 11 up, 8 in 2024-05-15T23:26:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:23 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:26:23.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:23 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:26:23.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:23 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:26:23.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:23 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:26:23.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:23 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:26:23.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:23 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:26:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:26:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:26:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:23 smithi084 ceph-mon[39784]: osdmap e511: 12 total, 11 up, 8 in 2024-05-15T23:26:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:23 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:26:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:23 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:26:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:23 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:26:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:23 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:26:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:23 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:26:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:23 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:26:23.449 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:26:23.462 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 76 16 117114 117098 96.2796 80.875 0.0275156 0.0103846 2024-05-15T23:26:23.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:26:23.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:26:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:23 smithi090 ceph-mon[44434]: osdmap e511: 12 total, 11 up, 8 in 2024-05-15T23:26:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:23 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:26:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:23 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:26:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:23 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:26:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:23 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:26:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:23 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:26:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:23 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:26:23.778 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:26:23.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:23.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:26:23.884 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:26:23.938 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:23.974 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:26:23.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:23.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:26:24.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:26:24.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:26:24.119 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:24.148 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:26:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:24.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:26:24.256 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:24.294 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:24.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:26:24.462 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 77 16 118602 118586 96.2369 93 0.00664671 0.0103901 2024-05-15T23:26:24.549 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:24.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:26:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:24.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:24.654 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:24.694 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:24.749 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:26:24.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:24.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:26:24.855 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:26:24.906 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:25.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:25 smithi088 ceph-mon[44699]: pgmap v1364: 17 pgs: 17 active+clean; 7.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-05-15T23:26:25.462 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 78 16 120061 120045 96.1721 91.1875 0.00571514 0.0103965 2024-05-15T23:26:25.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:25 smithi090 ceph-mon[44434]: pgmap v1364: 17 pgs: 17 active+clean; 7.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-05-15T23:26:25.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:25 smithi084 ceph-mon[39784]: pgmap v1364: 17 pgs: 17 active+clean; 7.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-05-15T23:26:25.982 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:26:25.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:25.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:26:26.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:26:26.119 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:26.140 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:26:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:26:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:26.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:26:26.244 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:26 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3273918867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:26:26.244 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:26 smithi088 ceph-mon[44699]: from='client.26428 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:26:26.255 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:26.295 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:26.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:26 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3273918867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:26:26.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:26 smithi084 ceph-mon[39784]: from='client.26428 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:26:26.462 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 79 16 121503 121487 96.0954 90.125 0.0131283 0.010405 2024-05-15T23:26:26.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:26 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3273918867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:26:26.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:26 smithi090 ceph-mon[44434]: from='client.26428 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:26:27.054 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:27.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:27.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:27.162 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:27.194 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:27.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:26:27.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:26:27.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:26:27.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:26:27.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:26:27.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:26:27.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:26:27.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:26:27.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:26:27.198 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:26:27.198 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:26:27.198 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:26:27.198 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:26:27.198 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:26:27.198 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:26:27.198 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:26:27.198 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:26:27.198 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:26:27.198 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:26:27.198 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-05-15T23:26:27.199 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:26:27.199 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:26:27.199 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:26:27.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:26:27.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:26:27.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:27 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:27.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:27 smithi088 ceph-mon[44699]: from='client.26428 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:26:27.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:27 smithi088 ceph-mon[44699]: osdmap e512: 12 total, 11 up, 8 in 2024-05-15T23:26:27.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:27 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2009488456' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:27.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:27.326 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:27.326 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:27.326 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:27.326 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:27 smithi088 ceph-mon[44699]: pgmap v1366: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T23:26:27.326 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:27 smithi088 ceph-mon[44699]: osdmap e513: 12 total, 11 up, 8 in 2024-05-15T23:26:27.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:26:27.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:26:27.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:27 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:27 smithi084 ceph-mon[39784]: from='client.26428 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:26:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:27 smithi084 ceph-mon[39784]: osdmap e512: 12 total, 11 up, 8 in 2024-05-15T23:26:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:27 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2009488456' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:27 smithi084 ceph-mon[39784]: pgmap v1366: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T23:26:27.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:27 smithi084 ceph-mon[39784]: osdmap e513: 12 total, 11 up, 8 in 2024-05-15T23:26:27.385 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:26:27.385 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:26:27.385 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:26:27.385 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:26:27.385 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.386 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:137229 5.3 5:cb7b8dd7:::benchmark_data_smithi090_116470_object137228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e408)", 2024-05-15T23:26:27.386 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:09.074219+0000", 2024-05-15T23:26:27.386 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 318.30556179000001, 2024-05-15T23:26:27.386 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.36923392399999999, 2024-05-15T23:26:27.386 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.386 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.386 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.386 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.386 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.386 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.386 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 137229 2024-05-15T23:26:27.386 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.386 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.386 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074219+0000", 2024-05-15T23:26:27.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074219+0000", 2024-05-15T23:26:27.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.387 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.387 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.387 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074222+0000", 2024-05-15T23:26:27.387 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4740000000000001e-06 2024-05-15T23:26:27.387 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.388 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.388 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074340+0000", 2024-05-15T23:26:27.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000117756 2024-05-15T23:26:27.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.388 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.388 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074357+0000", 2024-05-15T23:26:27.388 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.7181e-05 2024-05-15T23:26:27.388 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.388 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074374+0000", 2024-05-15T23:26:27.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.7135000000000001e-05 2024-05-15T23:26:27.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.076214+0000", 2024-05-15T23:26:27.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0018394749999999999 2024-05-15T23:26:27.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.389 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.389 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:26:27.389 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.076226+0000", 2024-05-15T23:26:27.389 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2014000000000001e-05 2024-05-15T23:26:27.389 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.409250+0000", 2024-05-15T23:26:27.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.33302412799999997 2024-05-15T23:26:27.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.390 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.411419+0000", 2024-05-15T23:26:27.390 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0021693509999999999 2024-05-15T23:26:27.390 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.390 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.390 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.411521+0000", 2024-05-15T23:26:27.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00010147800000000001 2024-05-15T23:26:27.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.427415+0000", 2024-05-15T23:26:27.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015894612999999998 2024-05-15T23:26:27.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.391 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.441398+0000", 2024-05-15T23:26:27.391 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013983293000000001 2024-05-15T23:26:27.391 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.391 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.391 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.441432+0000", 2024-05-15T23:26:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3983999999999999e-05 2024-05-15T23:26:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.443452+0000", 2024-05-15T23:26:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020200420000000001 2024-05-15T23:26:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:137272 5.3 5:c61533c8:::benchmark_data_smithi090_116470_object137271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e408)", 2024-05-15T23:26:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:09.074402+0000", 2024-05-15T23:26:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 318.30537801000003, 2024-05-15T23:26:27.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.37109481, 2024-05-15T23:26:27.393 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.393 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.393 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.393 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.393 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.393 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.393 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 137272 2024-05-15T23:26:27.393 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.393 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.393 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.393 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.393 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074402+0000", 2024-05-15T23:26:27.393 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074402+0000", 2024-05-15T23:26:27.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.394 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074403+0000", 2024-05-15T23:26:27.394 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5099999999999998e-07 2024-05-15T23:26:27.394 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.394 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.394 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074443+0000", 2024-05-15T23:26:27.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9894999999999997e-05 2024-05-15T23:26:27.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074453+0000", 2024-05-15T23:26:27.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7480000000000005e-06 2024-05-15T23:26:27.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.395 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074461+0000", 2024-05-15T23:26:27.395 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5259999999999993e-06 2024-05-15T23:26:27.395 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.395 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.395 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.076252+0000", 2024-05-15T23:26:27.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0017901530000000001 2024-05-15T23:26:27.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:26:27.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.076259+0000", 2024-05-15T23:26:27.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9029999999999998e-06 2024-05-15T23:26:27.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.396 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.396 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.416388+0000", 2024-05-15T23:26:27.396 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.34012952499999999 2024-05-15T23:26:27.396 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.396 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.416484+0000", 2024-05-15T23:26:27.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5592000000000002e-05 2024-05-15T23:26:27.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.416583+0000", 2024-05-15T23:26:27.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8877999999999998e-05 2024-05-15T23:26:27.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.397 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.397 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.427422+0000", 2024-05-15T23:26:27.397 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010839577 2024-05-15T23:26:27.397 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.397 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.443513+0000", 2024-05-15T23:26:27.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.016090444999999998 2024-05-15T23:26:27.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.443571+0000", 2024-05-15T23:26:27.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8838000000000001e-05 2024-05-15T23:26:27.398 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.398 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.398 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.398 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.445497+0000", 2024-05-15T23:26:27.398 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0019257790000000001 2024-05-15T23:26:27.398 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.399 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.399 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.399 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.399 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.399 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:137273 5.3 5:c9608721:::benchmark_data_smithi090_116470_object137272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e408)", 2024-05-15T23:26:27.399 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:09.074473+0000", 2024-05-15T23:26:27.399 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 318.30530723099997, 2024-05-15T23:26:27.399 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.373155032, 2024-05-15T23:26:27.399 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.399 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.399 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.399 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.399 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.399 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.400 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 137273 2024-05-15T23:26:27.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.400 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074473+0000", 2024-05-15T23:26:27.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.400 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074473+0000", 2024-05-15T23:26:27.400 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.400 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.400 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.400 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074474+0000", 2024-05-15T23:26:27.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3500000000000005e-07 2024-05-15T23:26:27.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074523+0000", 2024-05-15T23:26:27.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8662000000000003e-05 2024-05-15T23:26:27.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.401 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074531+0000", 2024-05-15T23:26:27.401 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8550000000000008e-06 2024-05-15T23:26:27.401 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.401 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.401 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.074539+0000", 2024-05-15T23:26:27.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1409999999999999e-06 2024-05-15T23:26:27.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.076362+0000", 2024-05-15T23:26:27.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0018231390000000001 2024-05-15T23:26:27.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.402 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.402 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:26:27.402 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.076373+0000", 2024-05-15T23:26:27.402 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1211e-05 2024-05-15T23:26:27.402 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.419704+0000", 2024-05-15T23:26:27.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.34333087200000001 2024-05-15T23:26:27.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.419789+0000", 2024-05-15T23:26:27.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5165000000000002e-05 2024-05-15T23:26:27.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.403 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.403 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.403 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.419887+0000", 2024-05-15T23:26:27.403 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8108000000000003e-05 2024-05-15T23:26:27.403 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.427428+0000", 2024-05-15T23:26:27.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0075411109999999996 2024-05-15T23:26:27.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.445552+0000", 2024-05-15T23:26:27.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018123862000000001 2024-05-15T23:26:27.404 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.404 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.404 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.404 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.445607+0000", 2024-05-15T23:26:27.404 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5111999999999997e-05 2024-05-15T23:26:27.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.405 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.405 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.447628+0000", 2024-05-15T23:26:27.405 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020209590000000001 2024-05-15T23:26:27.405 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.405 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.405 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.405 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.405 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.405 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:204828 5.3 5:cf8dfcc5:::benchmark_data_smithi090_116470_object51268:head [delete] snapc 0=[] ondisk+write+known_if_redirected e435)", 2024-05-15T23:26:27.405 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:22:41.378103+0000", 2024-05-15T23:26:27.405 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 226.00167717599999, 2024-05-15T23:26:27.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15704944600000001, 2024-05-15T23:26:27.406 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.406 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.406 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.406 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.406 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.406 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.406 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 204828 2024-05-15T23:26:27.406 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.406 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.406 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.406 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.406 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.378103+0000", 2024-05-15T23:26:27.406 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.378103+0000", 2024-05-15T23:26:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.378105+0000", 2024-05-15T23:26:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8789999999999999e-06 2024-05-15T23:26:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.378110+0000", 2024-05-15T23:26:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8219999999999997e-06 2024-05-15T23:26:27.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.378119+0000", 2024-05-15T23:26:27.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.833e-06 2024-05-15T23:26:27.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.378125+0000", 2024-05-15T23:26:27.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0480000000000004e-06 2024-05-15T23:26:27.408 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.408 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.408 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.408 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.403407+0000", 2024-05-15T23:26:27.408 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.025281834999999999 2024-05-15T23:26:27.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:27.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.403879+0000", 2024-05-15T23:26:27.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00047247599999999999 2024-05-15T23:26:27.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.496462+0000", 2024-05-15T23:26:27.409 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092582806000000004 2024-05-15T23:26:27.409 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.409 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.409 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.409 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.496575+0000", 2024-05-15T23:26:27.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000113409 2024-05-15T23:26:27.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.496720+0000", 2024-05-15T23:26:27.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014483899999999999 2024-05-15T23:26:27.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.410 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.511709+0000", 2024-05-15T23:26:27.410 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.014989339000000001 2024-05-15T23:26:27.410 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.410 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.410 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.533732+0000", 2024-05-15T23:26:27.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022022672 2024-05-15T23:26:27.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.533784+0000", 2024-05-15T23:26:27.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1786999999999997e-05 2024-05-15T23:26:27.411 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.411 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.411 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.411 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.535153+0000", 2024-05-15T23:26:27.411 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013687009999999999 2024-05-15T23:26:27.411 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.412 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.412 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.412 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.412 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.412 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:204832 5.3 5:c0ed1591:::benchmark_data_smithi090_116470_object51272:head [delete] snapc 0=[] ondisk+write+known_if_redirected e435)", 2024-05-15T23:26:27.412 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:22:41.378901+0000", 2024-05-15T23:26:27.412 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 226.000879682, 2024-05-15T23:26:27.412 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15760426599999999, 2024-05-15T23:26:27.412 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.412 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.412 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.412 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.412 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 204832 2024-05-15T23:26:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.378901+0000", 2024-05-15T23:26:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.378901+0000", 2024-05-15T23:26:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.378901+0000", 2024-05-15T23:26:27.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7199999999999999e-07 2024-05-15T23:26:27.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.378903+0000", 2024-05-15T23:26:27.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2019999999999998e-06 2024-05-15T23:26:27.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.414 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.414 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.414 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.378910+0000", 2024-05-15T23:26:27.414 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0759999999999999e-06 2024-05-15T23:26:27.414 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.415 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.415 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.415 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.378915+0000", 2024-05-15T23:26:27.415 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5350000000000004e-06 2024-05-15T23:26:27.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.418 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.403962+0000", 2024-05-15T23:26:27.418 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.025047059999999999 2024-05-15T23:26:27.418 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.418 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.418 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:27.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.404484+0000", 2024-05-15T23:26:27.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00052161999999999998 2024-05-15T23:26:27.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.506490+0000", 2024-05-15T23:26:27.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.102006045 2024-05-15T23:26:27.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.419 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.506575+0000", 2024-05-15T23:26:27.419 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5542000000000002e-05 2024-05-15T23:26:27.419 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.419 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.419 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.506700+0000", 2024-05-15T23:26:27.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012489900000000001 2024-05-15T23:26:27.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.522003+0000", 2024-05-15T23:26:27.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015302971 2024-05-15T23:26:27.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.420 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.420 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.535223+0000", 2024-05-15T23:26:27.420 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013219574 2024-05-15T23:26:27.420 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.420 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.535280+0000", 2024-05-15T23:26:27.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7021999999999997e-05 2024-05-15T23:26:27.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.421 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.421 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.536505+0000", 2024-05-15T23:26:27.421 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0012251479999999999 2024-05-15T23:26:27.421 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.421 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.421 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.421 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.421 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.421 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:204834 5.3 5:cec543f4:::benchmark_data_smithi090_116470_object51274:head [delete] snapc 0=[] ondisk+write+known_if_redirected e435)", 2024-05-15T23:26:27.422 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:22:41.384607+0000", 2024-05-15T23:26:27.422 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 225.995173045, 2024-05-15T23:26:27.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15346394499999999, 2024-05-15T23:26:27.422 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.422 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.422 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.422 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.422 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.422 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.422 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 204834 2024-05-15T23:26:27.422 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.422 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.422 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.384607+0000", 2024-05-15T23:26:27.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.384607+0000", 2024-05-15T23:26:27.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.423 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.423 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.384609+0000", 2024-05-15T23:26:27.423 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8080000000000001e-06 2024-05-15T23:26:27.423 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.423 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.384614+0000", 2024-05-15T23:26:27.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4889999999999997e-06 2024-05-15T23:26:27.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.384624+0000", 2024-05-15T23:26:27.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0281e-05 2024-05-15T23:26:27.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.424 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.424 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.384632+0000", 2024-05-15T23:26:27.424 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9559999999999997e-06 2024-05-15T23:26:27.424 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.424 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.404513+0000", 2024-05-15T23:26:27.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019880732000000002 2024-05-15T23:26:27.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:27.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.404989+0000", 2024-05-15T23:26:27.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00047627200000000002 2024-05-15T23:26:27.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.425 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.425 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.425 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.516064+0000", 2024-05-15T23:26:27.425 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.111075272 2024-05-15T23:26:27.425 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.516161+0000", 2024-05-15T23:26:27.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7256000000000005e-05 2024-05-15T23:26:27.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.516312+0000", 2024-05-15T23:26:27.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150252 2024-05-15T23:26:27.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.426 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.426 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.426 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.533687+0000", 2024-05-15T23:26:27.426 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.017375699000000001 2024-05-15T23:26:27.426 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.536559+0000", 2024-05-15T23:26:27.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028719309999999999 2024-05-15T23:26:27.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.536618+0000", 2024-05-15T23:26:27.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8724000000000002e-05 2024-05-15T23:26:27.427 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.427 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.427 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.427 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.538071+0000", 2024-05-15T23:26:27.427 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0014532729999999999 2024-05-15T23:26:27.428 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.428 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.428 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.428 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.428 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.428 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:204845 5.3 5:c8a5f984:::benchmark_data_smithi090_116470_object51285:head [delete] snapc 0=[] ondisk+write+known_if_redirected e435)", 2024-05-15T23:26:27.428 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:22:41.398750+0000", 2024-05-15T23:26:27.428 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 225.98103037600001, 2024-05-15T23:26:27.428 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.157622544, 2024-05-15T23:26:27.428 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.428 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.428 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.428 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.428 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.429 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.429 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 204845 2024-05-15T23:26:27.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.429 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.398750+0000", 2024-05-15T23:26:27.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.429 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.429 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.429 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.429 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.398750+0000", 2024-05-15T23:26:27.429 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.398752+0000", 2024-05-15T23:26:27.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.1509999999999998e-06 2024-05-15T23:26:27.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.398758+0000", 2024-05-15T23:26:27.430 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7690000000000002e-06 2024-05-15T23:26:27.430 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.430 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.430 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.398770+0000", 2024-05-15T23:26:27.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1705999999999999e-05 2024-05-15T23:26:27.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.398778+0000", 2024-05-15T23:26:27.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.4139999999999994e-06 2024-05-15T23:26:27.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.431 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.405025+0000", 2024-05-15T23:26:27.431 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0062468360000000004 2024-05-15T23:26:27.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.431 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:27.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.405600+0000", 2024-05-15T23:26:27.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00057553499999999996 2024-05-15T23:26:27.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.538111+0000", 2024-05-15T23:26:27.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.132510504 2024-05-15T23:26:27.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.432 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.432 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.538196+0000", 2024-05-15T23:26:27.432 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.4651e-05 2024-05-15T23:26:27.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.538323+0000", 2024-05-15T23:26:27.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012779800000000001 2024-05-15T23:26:27.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.547920+0000", 2024-05-15T23:26:27.433 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0095964009999999992 2024-05-15T23:26:27.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.433 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.433 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.554041+0000", 2024-05-15T23:26:27.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0061207750000000002 2024-05-15T23:26:27.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.554111+0000", 2024-05-15T23:26:27.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0825e-05 2024-05-15T23:26:27.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.434 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.434 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:41.556373+0000", 2024-05-15T23:26:27.434 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0022611789999999999 2024-05-15T23:26:27.434 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.434 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.434 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.435 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:229441 5.3 5:cbd17028:::benchmark_data_smithi090_116470_object75881:head [delete] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-05-15T23:26:27.435 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:13.147788+0000", 2024-05-15T23:26:27.435 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 194.23199240899999, 2024-05-15T23:26:27.435 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15506007999999999, 2024-05-15T23:26:27.435 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.435 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.435 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.435 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.435 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.435 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.435 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 229441 2024-05-15T23:26:27.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.436 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:13.147788+0000", 2024-05-15T23:26:27.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:13.147788+0000", 2024-05-15T23:26:27.436 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.436 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.436 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:13.147789+0000", 2024-05-15T23:26:27.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3379999999999999e-06 2024-05-15T23:26:27.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:13.147794+0000", 2024-05-15T23:26:27.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8810000000000004e-06 2024-05-15T23:26:27.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:13.195088+0000", 2024-05-15T23:26:27.437 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047293986000000003 2024-05-15T23:26:27.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.437 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.437 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:13.195101+0000", 2024-05-15T23:26:27.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2605999999999999e-05 2024-05-15T23:26:27.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:13.217957+0000", 2024-05-15T23:26:27.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022855935000000001 2024-05-15T23:26:27.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.438 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:27.438 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:13.218433+0000", 2024-05-15T23:26:27.438 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00047650600000000002 2024-05-15T23:26:27.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.439 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.439 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:13.259069+0000", 2024-05-15T23:26:27.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.040636085000000002 2024-05-15T23:26:27.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.439 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.439 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:13.259170+0000", 2024-05-15T23:26:27.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00010105700000000001 2024-05-15T23:26:27.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.439 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.439 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:13.259326+0000", 2024-05-15T23:26:27.439 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015570800000000001 2024-05-15T23:26:27.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:13.275118+0000", 2024-05-15T23:26:27.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015791544000000001 2024-05-15T23:26:27.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:13.301050+0000", 2024-05-15T23:26:27.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.025932598000000001 2024-05-15T23:26:27.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.440 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.440 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:13.301103+0000", 2024-05-15T23:26:27.440 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2602000000000002e-05 2024-05-15T23:26:27.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.441 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.441 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:13.302848+0000", 2024-05-15T23:26:27.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001745234 2024-05-15T23:26:27.441 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.441 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.441 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.441 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:231868 5.3 5:c5ef3481:::benchmark_data_smithi090_116470_object78308:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:26:27.441 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:16.209511+0000", 2024-05-15T23:26:27.441 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 191.170269625, 2024-05-15T23:26:27.441 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.27284666099999999, 2024-05-15T23:26:27.441 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.441 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.442 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.442 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.442 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.442 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.442 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 231868 2024-05-15T23:26:27.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.442 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.442 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.209511+0000", 2024-05-15T23:26:27.442 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.442 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.209511+0000", 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.209514+0000", 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8720000000000002e-06 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.209522+0000", 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6249999999999996e-06 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.443 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.209541+0000", 2024-05-15T23:26:27.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8403e-05 2024-05-15T23:26:27.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.209550+0000", 2024-05-15T23:26:27.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5510000000000005e-06 2024-05-15T23:26:27.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.213024+0000", 2024-05-15T23:26:27.444 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034739599999999999 2024-05-15T23:26:27.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.444 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:27.444 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.213081+0000", 2024-05-15T23:26:27.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7213999999999998e-05 2024-05-15T23:26:27.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.445084+0000", 2024-05-15T23:26:27.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.23200284099999999 2024-05-15T23:26:27.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.445 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.445175+0000", 2024-05-15T23:26:27.445 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.1021999999999997e-05 2024-05-15T23:26:27.445 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.445 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.445 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.445319+0000", 2024-05-15T23:26:27.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014392499999999999 2024-05-15T23:26:27.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.473131+0000", 2024-05-15T23:26:27.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.027811501999999998 2024-05-15T23:26:27.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.446 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.446 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.480535+0000", 2024-05-15T23:26:27.446 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074045100000000004 2024-05-15T23:26:27.446 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.446 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.480582+0000", 2024-05-15T23:26:27.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7267e-05 2024-05-15T23:26:27.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.447 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.447 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.482357+0000", 2024-05-15T23:26:27.447 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0017749689999999999 2024-05-15T23:26:27.447 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.447 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.447 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.447 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.447 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.448 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPushReply(5.3 445/407 [PushReplyOp(5:c5ef3481:::benchmark_data_smithi090_116470_object78308:head)])", 2024-05-15T23:26:27.448 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:16.217314+0000", 2024-05-15T23:26:27.448 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 191.162466314, 2024-05-15T23:26:27.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22778457899999999, 2024-05-15T23:26:27.448 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.448 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.448 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2024-05-15T23:26:27.448 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.448 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.217314+0000", 2024-05-15T23:26:27.448 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.448 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.448 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.448 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.217314+0000", 2024-05-15T23:26:27.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.217315+0000", 2024-05-15T23:26:27.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.274e-06 2024-05-15T23:26:27.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.449 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.449 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.217321+0000", 2024-05-15T23:26:27.449 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6289999999999998e-06 2024-05-15T23:26:27.449 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.449 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.235123+0000", 2024-05-15T23:26:27.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.017801625000000001 2024-05-15T23:26:27.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.235144+0000", 2024-05-15T23:26:27.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.126e-05 2024-05-15T23:26:27.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.450 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.450 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.450 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.440100+0000", 2024-05-15T23:26:27.450 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20495602199999999 2024-05-15T23:26:27.450 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.451 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.451 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.445099+0000", 2024-05-15T23:26:27.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049987690000000001 2024-05-15T23:26:27.451 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.451 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.451 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.451 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.451 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.451 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:231908 5.3 5:ce15ba5e:::benchmark_data_smithi090_116470_object78348:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:26:27.451 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:16.262585+0000", 2024-05-15T23:26:27.451 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 191.117194915, 2024-05-15T23:26:27.451 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16722419499999999, 2024-05-15T23:26:27.451 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.452 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.452 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.452 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.452 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.452 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.452 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 231908 2024-05-15T23:26:27.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.452 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.452 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.262585+0000", 2024-05-15T23:26:27.452 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.452 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.452 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.452 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.262585+0000", 2024-05-15T23:26:27.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.262588+0000", 2024-05-15T23:26:27.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4849999999999999e-06 2024-05-15T23:26:27.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.453 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.262595+0000", 2024-05-15T23:26:27.453 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9469999999999997e-06 2024-05-15T23:26:27.453 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.453 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.453 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.262606+0000", 2024-05-15T23:26:27.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0954999999999999e-05 2024-05-15T23:26:27.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.262614+0000", 2024-05-15T23:26:27.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7729999999999999e-06 2024-05-15T23:26:27.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.454 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.277872+0000", 2024-05-15T23:26:27.454 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015258272999999999 2024-05-15T23:26:27.454 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.454 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.454 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:27.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.278378+0000", 2024-05-15T23:26:27.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00050590300000000004 2024-05-15T23:26:27.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.296679+0000", 2024-05-15T23:26:27.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018301007000000001 2024-05-15T23:26:27.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.455 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.296762+0000", 2024-05-15T23:26:27.455 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3055999999999999e-05 2024-05-15T23:26:27.455 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.455 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.455 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.296873+0000", 2024-05-15T23:26:27.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011111 2024-05-15T23:26:27.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.310470+0000", 2024-05-15T23:26:27.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013597514 2024-05-15T23:26:27.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.456 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.456 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.427908+0000", 2024-05-15T23:26:27.456 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.117437177 2024-05-15T23:26:27.456 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.456 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.427958+0000", 2024-05-15T23:26:27.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9889999999999998e-05 2024-05-15T23:26:27.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.457 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.457 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.429810+0000", 2024-05-15T23:26:27.457 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0018521049999999999 2024-05-15T23:26:27.457 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.457 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.457 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.457 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.457 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.457 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:231918 5.3 5:c1a8710c:::benchmark_data_smithi090_116470_object78358:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:26:27.457 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:16.272388+0000", 2024-05-15T23:26:27.458 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 191.10739214099999, 2024-05-15T23:26:27.458 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17114908600000001, 2024-05-15T23:26:27.458 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.458 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.458 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.458 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.458 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.458 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.458 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 231918 2024-05-15T23:26:27.458 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.458 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.458 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.458 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.458 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.272388+0000", 2024-05-15T23:26:27.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.272388+0000", 2024-05-15T23:26:27.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.459 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.272390+0000", 2024-05-15T23:26:27.459 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.731e-06 2024-05-15T23:26:27.459 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.459 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.459 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.272395+0000", 2024-05-15T23:26:27.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3380000000000004e-06 2024-05-15T23:26:27.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.460 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.272405+0000", 2024-05-15T23:26:27.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0006e-05 2024-05-15T23:26:27.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.460 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.460 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.272413+0000", 2024-05-15T23:26:27.460 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3189999999999997e-06 2024-05-15T23:26:27.460 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.460 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.460 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.283794+0000", 2024-05-15T23:26:27.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011381677 2024-05-15T23:26:27.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:27.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.284463+0000", 2024-05-15T23:26:27.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00066903100000000005 2024-05-15T23:26:27.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.461 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.461 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.310509+0000", 2024-05-15T23:26:27.461 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.026045926 2024-05-15T23:26:27.461 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.461 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.310603+0000", 2024-05-15T23:26:27.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3505999999999995e-05 2024-05-15T23:26:27.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.310739+0000", 2024-05-15T23:26:27.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013646400000000001 2024-05-15T23:26:27.462 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.462 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.462 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.462 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.432360+0000", 2024-05-15T23:26:27.462 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.121620451 2024-05-15T23:26:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.442638+0000", 2024-05-15T23:26:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010278252999999999 2024-05-15T23:26:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.442679+0000", 2024-05-15T23:26:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0695000000000003e-05 2024-05-15T23:26:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.443537+0000", 2024-05-15T23:26:27.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00085868900000000005 2024-05-15T23:26:27.464 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.464 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.464 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.464 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.464 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.464 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:231939 5.3 5:c53f00b6:::benchmark_data_smithi090_116470_object78379:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:26:27.464 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:16.295747+0000", 2024-05-15T23:26:27.464 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 191.084033463, 2024-05-15T23:26:27.464 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22807998199999999, 2024-05-15T23:26:27.464 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.464 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.464 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.464 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.464 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.465 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.465 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 231939 2024-05-15T23:26:27.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.465 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.295747+0000", 2024-05-15T23:26:27.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.465 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.465 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.295747+0000", 2024-05-15T23:26:27.465 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.465 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.465 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.295748+0000", 2024-05-15T23:26:27.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.499e-06 2024-05-15T23:26:27.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.295752+0000", 2024-05-15T23:26:27.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7359999999999999e-06 2024-05-15T23:26:27.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.466 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.466 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.466 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.295759+0000", 2024-05-15T23:26:27.466 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7900000000000002e-06 2024-05-15T23:26:27.466 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.295764+0000", 2024-05-15T23:26:27.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.899e-06 2024-05-15T23:26:27.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.429854+0000", 2024-05-15T23:26:27.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.13409058800000001 2024-05-15T23:26:27.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.467 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.467 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:27.467 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.430583+0000", 2024-05-15T23:26:27.467 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00072818299999999998 2024-05-15T23:26:27.467 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.500866+0000", 2024-05-15T23:26:27.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070282931000000007 2024-05-15T23:26:27.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.500958+0000", 2024-05-15T23:26:27.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2608999999999994e-05 2024-05-15T23:26:27.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.468 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.468 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.468 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.501101+0000", 2024-05-15T23:26:27.468 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014264400000000001 2024-05-15T23:26:27.468 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.522643+0000", 2024-05-15T23:26:27.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021542696 2024-05-15T23:26:27.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.522661+0000", 2024-05-15T23:26:27.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.7730000000000001e-05 2024-05-15T23:26:27.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.469 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.469 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.469 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.522716+0000", 2024-05-15T23:26:27.469 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4490999999999997e-05 2024-05-15T23:26:27.469 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.470 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.470 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.523827+0000", 2024-05-15T23:26:27.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0011111859999999999 2024-05-15T23:26:27.470 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.470 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.470 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.470 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.470 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.470 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:231952 5.3 5:c981d936:::benchmark_data_smithi090_116470_object78392:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:26:27.470 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:16.306693+0000", 2024-05-15T23:26:27.470 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 191.07308772799999, 2024-05-15T23:26:27.470 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18248579700000001, 2024-05-15T23:26:27.470 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.470 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.471 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.471 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.471 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.471 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.471 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 231952 2024-05-15T23:26:27.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.471 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.471 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.306693+0000", 2024-05-15T23:26:27.471 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.471 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.471 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.471 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.471 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.306693+0000", 2024-05-15T23:26:27.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.306694+0000", 2024-05-15T23:26:27.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.248e-06 2024-05-15T23:26:27.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.306699+0000", 2024-05-15T23:26:27.472 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6709999999999998e-06 2024-05-15T23:26:27.472 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.472 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.472 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.472 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.306708+0000", 2024-05-15T23:26:27.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0650000000000005e-06 2024-05-15T23:26:27.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.306715+0000", 2024-05-15T23:26:27.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2110000000000001e-06 2024-05-15T23:26:27.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.473 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.473 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.430613+0000", 2024-05-15T23:26:27.473 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.123898018 2024-05-15T23:26:27.473 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.473 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:27.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.431389+0000", 2024-05-15T23:26:27.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00077635099999999999 2024-05-15T23:26:27.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.464999+0000", 2024-05-15T23:26:27.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.033609828000000001 2024-05-15T23:26:27.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.474 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.474 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.474 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.465093+0000", 2024-05-15T23:26:27.474 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4006999999999996e-05 2024-05-15T23:26:27.474 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.465233+0000", 2024-05-15T23:26:27.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014000199999999999 2024-05-15T23:26:27.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.487265+0000", 2024-05-15T23:26:27.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022031575000000001 2024-05-15T23:26:27.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.475 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.475 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.475 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.487365+0000", 2024-05-15T23:26:27.475 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9912000000000002e-05 2024-05-15T23:26:27.475 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.487418+0000", 2024-05-15T23:26:27.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.376e-05 2024-05-15T23:26:27.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.476 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.476 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.476 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:16.489178+0000", 2024-05-15T23:26:27.476 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0017601489999999999 2024-05-15T23:26:27.476 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.476 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.476 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.476 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.477 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.477 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:232631 5.3 5:cb1e7787:::benchmark_data_smithi090_116470_object79071:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:26:27.477 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:17.238952+0000", 2024-05-15T23:26:27.477 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 190.14082880300001, 2024-05-15T23:26:27.477 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.178666766, 2024-05-15T23:26:27.477 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.477 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.477 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.477 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.477 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.477 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.477 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 232631 2024-05-15T23:26:27.477 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.477 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.238952+0000", 2024-05-15T23:26:27.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.238952+0000", 2024-05-15T23:26:27.478 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.478 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.478 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.478 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.478 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.238953+0000", 2024-05-15T23:26:27.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.342e-06 2024-05-15T23:26:27.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.238957+0000", 2024-05-15T23:26:27.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1579999999999998e-06 2024-05-15T23:26:27.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.479 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.238964+0000", 2024-05-15T23:26:27.479 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9639999999999997e-06 2024-05-15T23:26:27.479 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.479 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.479 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.238969+0000", 2024-05-15T23:26:27.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1560000000000003e-06 2024-05-15T23:26:27.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.249878+0000", 2024-05-15T23:26:27.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010909284999999999 2024-05-15T23:26:27.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.480 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.480 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.249969+0000", 2024-05-15T23:26:27.480 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0427999999999995e-05 2024-05-15T23:26:27.480 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.480 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.250119+0000", 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000150612 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.313223+0000", 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063103048999999994 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.328412+0000", 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015189876 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.481 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.328439+0000", 2024-05-15T23:26:27.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6911999999999999e-05 2024-05-15T23:26:27.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.482 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.482 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.417618+0000", 2024-05-15T23:26:27.482 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.089178984000000003 2024-05-15T23:26:27.482 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.482 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.482 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.482 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.482 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.482 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:232636 5.3 5:cd5d0ef6:::benchmark_data_smithi090_116470_object79076:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:17.248394+0000", 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 190.13138619599999, 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17106091400000001, 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 232636 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.248394+0000", 2024-05-15T23:26:27.483 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.248394+0000", 2024-05-15T23:26:27.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.248404+0000", 2024-05-15T23:26:27.484 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3670000000000003e-06 2024-05-15T23:26:27.484 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.484 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.484 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.484 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.248410+0000", 2024-05-15T23:26:27.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1569999999999997e-06 2024-05-15T23:26:27.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.485 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.485 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.248420+0000", 2024-05-15T23:26:27.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0553e-05 2024-05-15T23:26:27.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.485 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.485 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.248428+0000", 2024-05-15T23:26:27.485 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5939999999999996e-06 2024-05-15T23:26:27.485 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.485 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.485 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.256806+0000", 2024-05-15T23:26:27.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0083786590000000001 2024-05-15T23:26:27.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.256897+0000", 2024-05-15T23:26:27.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0871000000000005e-05 2024-05-15T23:26:27.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.486 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.486 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.257021+0000", 2024-05-15T23:26:27.486 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000123619 2024-05-15T23:26:27.486 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.486 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.313258+0000", 2024-05-15T23:26:27.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.056237202999999999 2024-05-15T23:26:27.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.417633+0000", 2024-05-15T23:26:27.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.104374339 2024-05-15T23:26:27.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.487 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.487 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.487 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.417676+0000", 2024-05-15T23:26:27.487 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3899000000000001e-05 2024-05-15T23:26:27.487 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.488 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.488 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.419455+0000", 2024-05-15T23:26:27.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001778653 2024-05-15T23:26:27.488 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.488 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.488 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.488 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.488 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.488 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:232649 5.3 5:c9a04aed:::benchmark_data_smithi090_116470_object79089:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:26:27.488 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:17.255691+0000", 2024-05-15T23:26:27.488 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 190.12408937399999, 2024-05-15T23:26:27.488 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16521760499999999, 2024-05-15T23:26:27.488 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.489 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.489 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.489 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.489 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.489 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.489 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 232649 2024-05-15T23:26:27.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.489 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.489 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.255691+0000", 2024-05-15T23:26:27.489 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.489 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.489 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.489 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.490 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.255691+0000", 2024-05-15T23:26:27.490 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.490 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.490 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.490 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.490 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.255692+0000", 2024-05-15T23:26:27.490 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.44e-06 2024-05-15T23:26:27.490 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.490 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.490 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.490 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.255698+0000", 2024-05-15T23:26:27.490 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0209999999999996e-06 2024-05-15T23:26:27.490 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.491 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.491 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.491 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.255707+0000", 2024-05-15T23:26:27.491 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7909999999999993e-06 2024-05-15T23:26:27.491 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.491 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.491 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.491 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.255714+0000", 2024-05-15T23:26:27.491 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0589999999999997e-06 2024-05-15T23:26:27.491 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.491 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.491 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.491 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.308130+0000", 2024-05-15T23:26:27.491 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052415612 2024-05-15T23:26:27.491 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.492 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.492 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.492 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.308247+0000", 2024-05-15T23:26:27.492 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000116971 2024-05-15T23:26:27.492 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.492 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.492 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.492 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.308385+0000", 2024-05-15T23:26:27.492 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000138128 2024-05-15T23:26:27.492 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.492 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.492 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.492 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.318221+0000", 2024-05-15T23:26:27.492 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0098358009999999999 2024-05-15T23:26:27.493 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.493 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.493 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.493 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.419473+0000", 2024-05-15T23:26:27.493 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.101251902 2024-05-15T23:26:27.493 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.493 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.493 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.493 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.419523+0000", 2024-05-15T23:26:27.493 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0365000000000003e-05 2024-05-15T23:26:27.493 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.493 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.493 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.493 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.420909+0000", 2024-05-15T23:26:27.494 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0013855149999999999 2024-05-15T23:26:27.494 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.494 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.494 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.494 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.494 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.494 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:232655 5.3 5:c7065349:::benchmark_data_smithi090_116470_object79095:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:26:27.494 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:17.264003+0000", 2024-05-15T23:26:27.494 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 190.115777673, 2024-05-15T23:26:27.494 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.173507151, 2024-05-15T23:26:27.494 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.494 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.494 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.494 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.495 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.495 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.495 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 232655 2024-05-15T23:26:27.495 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.495 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.495 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.495 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.495 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.264003+0000", 2024-05-15T23:26:27.495 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.495 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.495 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.495 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.495 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.264003+0000", 2024-05-15T23:26:27.496 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.496 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.496 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.496 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.496 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.264005+0000", 2024-05-15T23:26:27.496 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4310000000000001e-06 2024-05-15T23:26:27.496 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.496 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.496 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.496 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.264013+0000", 2024-05-15T23:26:27.496 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5499999999999997e-06 2024-05-15T23:26:27.496 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.496 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.496 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.497 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.306012+0000", 2024-05-15T23:26:27.497 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.04199961 2024-05-15T23:26:27.497 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.497 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.497 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.497 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.306024+0000", 2024-05-15T23:26:27.497 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1796e-05 2024-05-15T23:26:27.497 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.497 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.497 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.497 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.318254+0000", 2024-05-15T23:26:27.497 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.012230151 2024-05-15T23:26:27.497 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.497 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.498 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:27.498 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.318312+0000", 2024-05-15T23:26:27.498 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8072999999999998e-05 2024-05-15T23:26:27.498 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.498 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.498 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.498 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.420974+0000", 2024-05-15T23:26:27.498 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.102661499 2024-05-15T23:26:27.498 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.498 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.498 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.498 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.421079+0000", 2024-05-15T23:26:27.498 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000104924 2024-05-15T23:26:27.498 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.498 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.499 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.421232+0000", 2024-05-15T23:26:27.499 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015320399999999999 2024-05-15T23:26:27.499 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.499 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.499 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.434768+0000", 2024-05-15T23:26:27.499 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013536317000000001 2024-05-15T23:26:27.499 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.499 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.499 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.499 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.435886+0000", 2024-05-15T23:26:27.499 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0011175689999999999 2024-05-15T23:26:27.499 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.499 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.500 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.500 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.435925+0000", 2024-05-15T23:26:27.500 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8782999999999998e-05 2024-05-15T23:26:27.500 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.500 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.500 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.500 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.437510+0000", 2024-05-15T23:26:27.500 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001585244 2024-05-15T23:26:27.500 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.500 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.500 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.500 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.500 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.501 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:232679 5.3 5:c3cf0280:::benchmark_data_smithi090_116470_object79119:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:26:27.501 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:17.320232+0000", 2024-05-15T23:26:27.501 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 190.05954869300001, 2024-05-15T23:26:27.501 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16885504400000001, 2024-05-15T23:26:27.501 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.501 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.501 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.501 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.501 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.501 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.501 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 232679 2024-05-15T23:26:27.501 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.501 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.501 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.501 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.502 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.320232+0000", 2024-05-15T23:26:27.502 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.502 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.502 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.502 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.502 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.320232+0000", 2024-05-15T23:26:27.502 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.502 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.502 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.502 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.502 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.320234+0000", 2024-05-15T23:26:27.502 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0600000000000002e-06 2024-05-15T23:26:27.502 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.502 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.502 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.503 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.320241+0000", 2024-05-15T23:26:27.503 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8260000000000004e-06 2024-05-15T23:26:27.503 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.503 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.503 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.503 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.320249+0000", 2024-05-15T23:26:27.503 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3939999999999996e-06 2024-05-15T23:26:27.503 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.503 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.503 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.503 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.320257+0000", 2024-05-15T23:26:27.503 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7849999999999992e-06 2024-05-15T23:26:27.503 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.503 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.503 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.324912+0000", 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0046556970000000003 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.324969+0000", 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.622e-05 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.447530+0000", 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.12256175900000001 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.504 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.447636+0000", 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000105962 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.447786+0000", 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000149946 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.484525+0000", 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036738732000000003 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.487282+0000", 2024-05-15T23:26:27.505 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027568789999999998 2024-05-15T23:26:27.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.506 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.506 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.487336+0000", 2024-05-15T23:26:27.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3856000000000001e-05 2024-05-15T23:26:27.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.506 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.506 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.489087+0000", 2024-05-15T23:26:27.506 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0017509279999999999 2024-05-15T23:26:27.506 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.506 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.506 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.506 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.506 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.507 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:232688 5.3 5:c1cca45c:::benchmark_data_smithi090_116470_object79128:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:26:27.507 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:23:17.328136+0000", 2024-05-15T23:26:27.507 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 190.05164450800001, 2024-05-15T23:26:27.507 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16200962499999999, 2024-05-15T23:26:27.507 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:27.507 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:27.507 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:27.507 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:27.507 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:27.507 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:27.507 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 232688 2024-05-15T23:26:27.507 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.507 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.328136+0000", 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.328136+0000", 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.328137+0000", 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5099999999999999e-06 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.508 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.509 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:27.509 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.328142+0000", 2024-05-15T23:26:27.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3510000000000004e-06 2024-05-15T23:26:27.509 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.509 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.509 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:27.509 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.328149+0000", 2024-05-15T23:26:27.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0060000000000003e-06 2024-05-15T23:26:27.509 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.509 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.509 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:27.509 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.328155+0000", 2024-05-15T23:26:27.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8340000000000004e-06 2024-05-15T23:26:27.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.510 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.510 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.510 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.434812+0000", 2024-05-15T23:26:27.510 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.106657565 2024-05-15T23:26:27.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.510 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.510 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:27.510 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.435328+0000", 2024-05-15T23:26:27.510 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00051578699999999997 2024-05-15T23:26:27.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.510 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.510 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:27.510 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.457708+0000", 2024-05-15T23:26:27.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022379973000000001 2024-05-15T23:26:27.511 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.511 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.511 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:27.511 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.457805+0000", 2024-05-15T23:26:27.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6868999999999997e-05 2024-05-15T23:26:27.511 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.511 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.511 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:26:27.511 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.457940+0000", 2024-05-15T23:26:27.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000135029 2024-05-15T23:26:27.511 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.511 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.511 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:27.512 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.484569+0000", 2024-05-15T23:26:27.512 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.026628880000000001 2024-05-15T23:26:27.512 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.512 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.512 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:27.512 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.489098+0000", 2024-05-15T23:26:27.512 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045295600000000002 2024-05-15T23:26:27.512 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.512 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.512 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:27.512 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.489150+0000", 2024-05-15T23:26:27.512 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2014e-05 2024-05-15T23:26:27.512 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:27.512 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:27.512 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:27.513 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:23:17.490145+0000", 2024-05-15T23:26:27.513 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0009952470000000001 2024-05-15T23:26:27.513 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.513 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.513 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.513 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:27.513 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:27.513 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:26:27.514 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:26:27.461864 min lat: 0.0010209 max lat: 24.9077 avg lat: 0.0104233 2024-05-15T23:26:27.514 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:26:27.514 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 80 16 122815 122799 95.9192 82 0.00470072 0.0104233 2024-05-15T23:26:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:26:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:26:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:27 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:27 smithi090 ceph-mon[44434]: from='client.26428 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:26:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:27 smithi090 ceph-mon[44434]: osdmap e512: 12 total, 11 up, 8 in 2024-05-15T23:26:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:27 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2009488456' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:27 smithi090 ceph-mon[44434]: pgmap v1366: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T23:26:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:27 smithi090 ceph-mon[44434]: osdmap e513: 12 total, 11 up, 8 in 2024-05-15T23:26:27.915 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:26:27.916 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:26:27.948 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:26:27.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:27.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:26:28.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:26:28.093 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:28.118 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:26:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:28.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:26:28.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.130 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:28 smithi088 ceph-mon[44699]: 6.5 scrub starts 2024-05-15T23:26:28.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.131 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:28 smithi088 ceph-mon[44699]: 6.0 scrub starts 2024-05-15T23:26:28.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:28 smithi084 ceph-mon[39784]: 6.5 scrub starts 2024-05-15T23:26:28.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:28 smithi084 ceph-mon[39784]: 6.0 scrub starts 2024-05-15T23:26:28.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:28.264 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:28.299 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:26:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:28.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:26:28.403 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:26:28.453 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:28.462 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 81 16 124085 124069 95.7148 79.375 0.0109052 0.0104463 2024-05-15T23:26:28.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:28 smithi090 ceph-mon[44434]: 6.5 scrub starts 2024-05-15T23:26:28.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:28.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:28 smithi090 ceph-mon[44434]: 6.0 scrub starts 2024-05-15T23:26:29.089 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:26:29.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:29 smithi084 ceph-mon[39784]: 6.4 scrub starts 2024-05-15T23:26:29.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:29 smithi084 ceph-mon[39784]: 6.1 scrub starts 2024-05-15T23:26:29.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:29 smithi084 ceph-mon[39784]: 6.5 scrub ok 2024-05-15T23:26:29.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:29 smithi084 ceph-mon[39784]: pgmap v1368: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-05-15T23:26:29.458 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:29.462 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 82 16 125335 125319 95.5003 78.125 0.0131389 0.0104703 2024-05-15T23:26:29.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:29 smithi090 ceph-mon[44434]: 6.4 scrub starts 2024-05-15T23:26:29.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:29 smithi090 ceph-mon[44434]: 6.1 scrub starts 2024-05-15T23:26:29.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:29 smithi090 ceph-mon[44434]: 6.5 scrub ok 2024-05-15T23:26:29.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:29 smithi090 ceph-mon[44434]: pgmap v1368: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-05-15T23:26:29.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:29 smithi088 ceph-mon[44699]: 6.4 scrub starts 2024-05-15T23:26:29.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:29 smithi088 ceph-mon[44699]: 6.1 scrub starts 2024-05-15T23:26:29.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:29 smithi088 ceph-mon[44699]: 6.5 scrub ok 2024-05-15T23:26:29.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:29 smithi088 ceph-mon[44699]: pgmap v1368: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-05-15T23:26:29.726 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:26:29.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:29.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:26:29.770 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T23:26:29.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:29.866 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:29.890 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:26:29.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:29.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:26:29.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:26:29.994 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:30.026 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:30.244 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:26:30.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:30 smithi084 ceph-mon[39784]: 6.d scrub starts 2024-05-15T23:26:30.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:30 smithi084 ceph-mon[39784]: 6.0 scrub ok 2024-05-15T23:26:30.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:30 smithi084 ceph-mon[39784]: 6.1 scrub ok 2024-05-15T23:26:30.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:30 smithi084 ceph-mon[39784]: 6.4 scrub ok 2024-05-15T23:26:30.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:30 smithi084 ceph-mon[39784]: 6.b scrub starts 2024-05-15T23:26:30.464 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 83 16 126489 126473 95.2171 72.125 0.0128114 0.0105006 2024-05-15T23:26:30.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:30 smithi090 ceph-mon[44434]: 6.d scrub starts 2024-05-15T23:26:30.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:30 smithi090 ceph-mon[44434]: 6.0 scrub ok 2024-05-15T23:26:30.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:30 smithi090 ceph-mon[44434]: 6.1 scrub ok 2024-05-15T23:26:30.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:30 smithi090 ceph-mon[44434]: 6.4 scrub ok 2024-05-15T23:26:30.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:30 smithi090 ceph-mon[44434]: 6.b scrub starts 2024-05-15T23:26:30.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:30 smithi088 ceph-mon[44699]: 6.d scrub starts 2024-05-15T23:26:30.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:30 smithi088 ceph-mon[44699]: 6.0 scrub ok 2024-05-15T23:26:30.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:30 smithi088 ceph-mon[44699]: 6.1 scrub ok 2024-05-15T23:26:30.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:30 smithi088 ceph-mon[44699]: 6.4 scrub ok 2024-05-15T23:26:30.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:30 smithi088 ceph-mon[44699]: 6.b scrub starts 2024-05-15T23:26:30.923 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:30.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:30.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:31.027 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:26:31.069 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:31.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:31 smithi088 ceph-mon[44699]: 6.e scrub starts 2024-05-15T23:26:31.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:31 smithi088 ceph-mon[44699]: 6.d scrub ok 2024-05-15T23:26:31.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:31 smithi088 ceph-mon[44699]: 6.a scrub starts 2024-05-15T23:26:31.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:31 smithi088 ceph-mon[44699]: pgmap v1369: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-05-15T23:26:31.464 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 84 15 127638 127623 94.9392 71.875 0.0152835 0.0105312 2024-05-15T23:26:31.536 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:31 smithi090 ceph-mon[44434]: 6.e scrub starts 2024-05-15T23:26:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:31 smithi090 ceph-mon[44434]: 6.d scrub ok 2024-05-15T23:26:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:31 smithi090 ceph-mon[44434]: 6.a scrub starts 2024-05-15T23:26:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:31 smithi090 ceph-mon[44434]: pgmap v1369: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-05-15T23:26:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:31 smithi084 ceph-mon[39784]: 6.e scrub starts 2024-05-15T23:26:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:31 smithi084 ceph-mon[39784]: 6.d scrub ok 2024-05-15T23:26:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:31 smithi084 ceph-mon[39784]: 6.a scrub starts 2024-05-15T23:26:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:31 smithi084 ceph-mon[39784]: pgmap v1369: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-05-15T23:26:32.178 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:26:32.447 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:26:32.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:32.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:26:32.453 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:32 smithi088 ceph-mon[44699]: 6.b scrub ok 2024-05-15T23:26:32.453 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/398159716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:32.453 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:32 smithi088 ceph-mon[44699]: from='client.36386 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:32.464 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 85 16 128932 128916 94.7729 80.8125 0.006697 0.0105499 2024-05-15T23:26:32.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:32 smithi090 ceph-mon[44434]: 6.b scrub ok 2024-05-15T23:26:32.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/398159716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:32.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:32 smithi090 ceph-mon[44434]: from='client.36386 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:32.553 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:32.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:32 smithi084 ceph-mon[39784]: 6.b scrub ok 2024-05-15T23:26:32.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/398159716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:32.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:32 smithi084 ceph-mon[39784]: from='client.36386 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:32.587 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:32.628 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:26:32.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:32.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:26:32.735 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:26:32.766 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:33.464 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 86 16 130243 130227 94.6236 81.9375 0.0174741 0.0105671 2024-05-15T23:26:33.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:33 smithi088 ceph-mon[44699]: 6.e scrub ok 2024-05-15T23:26:33.507 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:33 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:26:33.507 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:33 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:26:33.507 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:33 smithi088 ceph-mon[44699]: from='client.36386 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:26:33.507 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:33 smithi088 ceph-mon[44699]: osdmap e514: 12 total, 11 up, 8 in 2024-05-15T23:26:33.507 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:33 smithi088 ceph-mon[44699]: 6.a scrub ok 2024-05-15T23:26:33.507 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:33 smithi088 ceph-mon[44699]: pgmap v1371: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-05-15T23:26:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:26:33.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:33.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:26:33.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:33 smithi090 ceph-mon[44434]: 6.e scrub ok 2024-05-15T23:26:33.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:33 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:26:33.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:33 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:26:33.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:33 smithi090 ceph-mon[44434]: from='client.36386 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:26:33.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:33 smithi090 ceph-mon[44434]: osdmap e514: 12 total, 11 up, 8 in 2024-05-15T23:26:33.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:33 smithi090 ceph-mon[44434]: 6.a scrub ok 2024-05-15T23:26:33.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:33 smithi090 ceph-mon[44434]: pgmap v1371: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-05-15T23:26:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:33 smithi084 ceph-mon[39784]: 6.e scrub ok 2024-05-15T23:26:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:33 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:26:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:33 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:26:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:33 smithi084 ceph-mon[39784]: from='client.36386 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:26:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:33 smithi084 ceph-mon[39784]: osdmap e514: 12 total, 11 up, 8 in 2024-05-15T23:26:33.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:33 smithi084 ceph-mon[39784]: 6.a scrub ok 2024-05-15T23:26:33.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:33 smithi084 ceph-mon[39784]: pgmap v1371: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-05-15T23:26:33.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:26:33.673 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:33.703 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:26:33.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:33.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:26:33.811 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:26:33.845 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:33.875 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:26:33.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:33.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:26:33.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:26:34.017 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:34.181 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:34 smithi088 ceph-mon[44699]: 6.6 scrub starts 2024-05-15T23:26:34.224 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:26:34.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:34.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:26:34.309 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:26:34.309 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:26:34.309 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2024-05-15T23:26:34.309 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:26:34.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:26:34.369 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:34.451 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:26:34.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:34 smithi084 ceph-mon[39784]: 6.6 scrub starts 2024-05-15T23:26:34.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:34.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:26:34.464 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 87 16 131525 131509 94.4569 80.125 0.0173854 0.0105852 2024-05-15T23:26:34.512 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:26:34.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:34 smithi090 ceph-mon[44434]: 6.6 scrub starts 2024-05-15T23:26:34.555 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:26:34.559 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:34.594 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:26:34.604 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:34.631 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:26:34.636 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:34.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:34.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:34.660 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:26:34.698 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:26:34.729 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:26:34.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:26:34.766 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:26:34.780 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:34.807 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:26:34.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:34.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:26:34.816 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:26:34.847 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:26:34.885 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:26:34.885 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:26:34.885 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:26:34.885 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:26:34.886 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:26:34.886 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:26:34.886 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:26:34.886 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:26:34.886 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:26:34.886 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-05-15T23:26:34.886 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:26:34.886 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:26:34.886 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:26:34.912 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:34.964 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:35.126 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:35.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:35.200 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:35 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:26:35.200 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:35 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:26:35.200 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:35 smithi084 ceph-mon[39784]: 6.c scrub starts 2024-05-15T23:26:35.200 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:35 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3524825592' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:35.200 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:35.200 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:35.200 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:35 smithi084 ceph-mon[39784]: pgmap v1372: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-05-15T23:26:35.236 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:26:35.281 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:35.288 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:26:35.464 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 88 16 132683 132667 94.2059 72.375 0.00568253 0.0106123 2024-05-15T23:26:35.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:35 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:26:35.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:35 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:26:35.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:35 smithi090 ceph-mon[44434]: 6.c scrub starts 2024-05-15T23:26:35.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:35 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3524825592' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:35.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:35.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:35.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:35 smithi090 ceph-mon[44434]: pgmap v1372: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-05-15T23:26:35.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:35 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:26:35.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:35 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:26:35.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:35 smithi088 ceph-mon[44699]: 6.c scrub starts 2024-05-15T23:26:35.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:35 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3524825592' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:35.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:35.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:35.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:35 smithi088 ceph-mon[44699]: pgmap v1372: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-05-15T23:26:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:26:35.730 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:26:35.787 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:35.855 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T23:26:35.886 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:26:35.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:35.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:26:35.991 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:26:36.026 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:36.067 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:26:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:36.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:26:36.173 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:26:36.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:36 smithi088 ceph-mon[44699]: 6.6 scrub ok 2024-05-15T23:26:36.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:36 smithi088 ceph-mon[44699]: osdmap e515: 12 total, 11 up, 8 in 2024-05-15T23:26:36.243 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:36.264 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:26:36.294 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:26:36.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:36.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:26:36.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:36 smithi084 ceph-mon[39784]: 6.6 scrub ok 2024-05-15T23:26:36.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:36 smithi084 ceph-mon[39784]: osdmap e515: 12 total, 11 up, 8 in 2024-05-15T23:26:36.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:26:36.438 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:36.464 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 89 16 133940 133924 94.0301 78.5625 0.00736648 0.0106329 2024-05-15T23:26:36.465 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:26:36.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:36.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:26:36.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:36.472 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:36 smithi090 ceph-mon[44434]: 6.6 scrub ok 2024-05-15T23:26:36.473 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:36 smithi090 ceph-mon[44434]: osdmap e515: 12 total, 11 up, 8 in 2024-05-15T23:26:36.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:26:36.606 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:36.633 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:26:36.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:36.637 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:26:36.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:26:36.769 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:36.794 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:26:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:36.799 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:26:36.900 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:36.935 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:37.290 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:26:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:37.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:26:37.398 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:26:37.429 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:37.464 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total time run: 90.0121 2024-05-15T23:26:37.465 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total writes made: 135191 2024-05-15T23:26:37.465 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Write size: 65536 2024-05-15T23:26:37.465 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object size: 65536 2024-05-15T23:26:37.465 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Bandwidth (MB/sec): 93.87 2024-05-15T23:26:37.465 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Bandwidth: 77.414 2024-05-15T23:26:37.465 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max bandwidth (MB/sec): 382.438 2024-05-15T23:26:37.465 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min bandwidth (MB/sec): 0 2024-05-15T23:26:37.465 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average IOPS: 1501 2024-05-15T23:26:37.465 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev IOPS: 1238.64 2024-05-15T23:26:37.465 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max IOPS: 6119 2024-05-15T23:26:37.465 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min IOPS: 0 2024-05-15T23:26:37.465 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average Latency(s): 0.0106522 2024-05-15T23:26:37.465 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Latency(s): 0.270868 2024-05-15T23:26:37.465 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max latency(s): 24.9077 2024-05-15T23:26:37.466 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min latency(s): 0.0010209 2024-05-15T23:26:37.470 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T23:26:37.500 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:37 smithi084 ceph-mon[39784]: 6.c scrub ok 2024-05-15T23:26:37.500 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:37 smithi084 ceph-mon[39784]: pgmap v1374: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-05-15T23:26:37.504 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:26:37.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:37.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:26:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:37 smithi090 ceph-mon[44434]: 6.c scrub ok 2024-05-15T23:26:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:37 smithi090 ceph-mon[44434]: pgmap v1374: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-05-15T23:26:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:37 smithi088 ceph-mon[44699]: 6.c scrub ok 2024-05-15T23:26:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:37 smithi088 ceph-mon[44699]: pgmap v1374: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-05-15T23:26:37.609 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:26:37.648 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:37.677 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:26:37.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:37.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:26:37.731 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:26:37.781 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:26:37.825 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:26:38.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:38.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:26:38.100 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:38.147 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:38.178 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:38.235 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:26:38.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:38 smithi088 ceph-mon[44699]: 6.f deep-scrub starts 2024-05-15T23:26:38.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:38 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2759165870' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:26:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:26:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:26:38.433 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:38 smithi084 ceph-mon[39784]: 6.f deep-scrub starts 2024-05-15T23:26:38.433 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:38 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2759165870' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:26:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:38 smithi090 ceph-mon[44434]: 6.f deep-scrub starts 2024-05-15T23:26:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:38 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2759165870' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:26:39.202 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T23:26:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:39 smithi090 ceph-mon[44434]: 6.3 scrub starts 2024-05-15T23:26:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:39 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:26:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:39 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:26:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:39 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2759165870' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:26:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:39 smithi090 ceph-mon[44434]: osdmap e516: 12 total, 11 up, 8 in 2024-05-15T23:26:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:39 smithi090 ceph-mon[44434]: pgmap v1376: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 148 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-05-15T23:26:39.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:39 smithi084 ceph-mon[39784]: 6.3 scrub starts 2024-05-15T23:26:39.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:39 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:26:39.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:39 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:26:39.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:39 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:39.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2759165870' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:26:39.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:39 smithi084 ceph-mon[39784]: osdmap e516: 12 total, 11 up, 8 in 2024-05-15T23:26:39.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:39 smithi084 ceph-mon[39784]: pgmap v1376: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 148 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-05-15T23:26:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:39 smithi088 ceph-mon[44699]: 6.3 scrub starts 2024-05-15T23:26:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:39 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:26:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:39 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:26:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:39 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2759165870' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:26:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:39 smithi088 ceph-mon[44699]: osdmap e516: 12 total, 11 up, 8 in 2024-05-15T23:26:39.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:39 smithi088 ceph-mon[44699]: pgmap v1376: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 148 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-05-15T23:26:39.584 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:26:40.111 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:40.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:40.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:40.216 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:40.247 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:40.297 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:26:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:40.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:26:40.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:40.453 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:40.702 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:26:40.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:40.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:26:40.723 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:26:40.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:26:40.773 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:26:40.797 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:26:40.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:26:40.824 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:26:40.840 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:40.850 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:26:40.862 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:26:40.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:40.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:26:40.875 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:26:40.904 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:26:40.935 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:26:40.960 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:26:40.966 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:26:40.985 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:26:40.985 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:26:40.985 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:26:40.985 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:26:40.985 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:26:40.985 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:26:40.986 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:26:40.986 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:26:40.986 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:26:40.986 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-05-15T23:26:40.986 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:26:40.986 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:26:40.986 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:26:41.000 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:40 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1812605140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:41.003 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:40 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1812605140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:41.006 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:41.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:40 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1812605140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:41.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:26:41.800 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:26:41.801 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:26:41.878 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:26:41.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:41.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:26:41.937 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:41.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:42.012 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.012 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.012 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.013 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.013 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.013 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.013 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.013 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.013 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.013 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.013 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:41 smithi088 ceph-mon[44699]: pgmap v1377: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 127 B/s rd, 59 MiB/s wr, 1.24k op/s 2024-05-15T23:26:42.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:41 smithi084 ceph-mon[39784]: pgmap v1377: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 127 B/s rd, 59 MiB/s wr, 1.24k op/s 2024-05-15T23:26:42.015 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:41 smithi090 ceph-mon[44434]: pgmap v1377: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 127 B/s rd, 59 MiB/s wr, 1.24k op/s 2024-05-15T23:26:42.216 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:26:42.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:42.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:26:42.320 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:26:42.358 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:26:42.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:42.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:26:42.498 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:26:42.539 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:42.745 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:26:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:42.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:26:42.850 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:26:42.890 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:42.986 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:26:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:42.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:26:43.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:43.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:43.096 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:26:43.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:43.151 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:43.166 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:26:43.166 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:26:43.166 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:26:43.166 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:26:43.166 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.166 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304280 4.a 4:5b65c4c5:::benchmark_data_smithi090_104448_object131042:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:26:43.166 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.815719+0000", 2024-05-15T23:26:43.166 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 497.34233246100001, 2024-05-15T23:26:43.167 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.213512642, 2024-05-15T23:26:43.167 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.167 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.167 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.167 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.167 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.167 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.167 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304280 2024-05-15T23:26:43.167 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.167 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.168 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.168 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.168 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815719+0000", 2024-05-15T23:26:43.168 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.168 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.168 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.168 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.168 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815719+0000", 2024-05-15T23:26:43.168 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.168 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.168 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.168 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.168 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815721+0000", 2024-05-15T23:26:43.169 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0279999999999999e-06 2024-05-15T23:26:43.169 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.169 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.169 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.169 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815728+0000", 2024-05-15T23:26:43.169 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7270000000000001e-06 2024-05-15T23:26:43.169 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.169 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.169 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.169 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815737+0000", 2024-05-15T23:26:43.169 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6150000000000003e-06 2024-05-15T23:26:43.169 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.169 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.170 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.170 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815744+0000", 2024-05-15T23:26:43.170 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2589999999999997e-06 2024-05-15T23:26:43.170 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.170 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.170 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.170 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815805+0000", 2024-05-15T23:26:43.170 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1169999999999999e-05 2024-05-15T23:26:43.170 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.170 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.170 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.170 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.815916+0000", 2024-05-15T23:26:43.170 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011129300000000001 2024-05-15T23:26:43.170 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.171 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.171 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.171 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816037+0000", 2024-05-15T23:26:43.171 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000120598 2024-05-15T23:26:43.171 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.171 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.171 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.171 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.824204+0000", 2024-05-15T23:26:43.171 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.008167545 2024-05-15T23:26:43.171 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.171 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.171 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.171 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.029121+0000", 2024-05-15T23:26:43.171 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20491626399999999 2024-05-15T23:26:43.172 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.172 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.172 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.172 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.029181+0000", 2024-05-15T23:26:43.172 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0053999999999998e-05 2024-05-15T23:26:43.172 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.172 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.172 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.172 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.029232+0000", 2024-05-15T23:26:43.172 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1088999999999998e-05 2024-05-15T23:26:43.172 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.172 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.172 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.172 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.172 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.173 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304283 4.a 4:58e8bfc1:::benchmark_data_smithi090_104448_object131045:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:26:43.173 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.816913+0000", 2024-05-15T23:26:43.173 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 497.34113891300001, 2024-05-15T23:26:43.173 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22379715, 2024-05-15T23:26:43.173 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.173 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.173 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.173 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.173 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.173 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.173 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304283 2024-05-15T23:26:43.173 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.173 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.173 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.174 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.174 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816913+0000", 2024-05-15T23:26:43.174 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.174 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.174 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.174 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.174 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816913+0000", 2024-05-15T23:26:43.174 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.174 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.174 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.174 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.174 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816914+0000", 2024-05-15T23:26:43.174 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.519e-06 2024-05-15T23:26:43.174 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.175 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.175 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.175 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816919+0000", 2024-05-15T23:26:43.175 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.977e-06 2024-05-15T23:26:43.175 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.175 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.175 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.175 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816928+0000", 2024-05-15T23:26:43.175 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0310000000000004e-06 2024-05-15T23:26:43.175 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.175 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.175 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.175 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.816935+0000", 2024-05-15T23:26:43.175 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3610000000000003e-06 2024-05-15T23:26:43.175 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.176 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.176 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.176 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.819984+0000", 2024-05-15T23:26:43.176 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030482909999999998 2024-05-15T23:26:43.176 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.176 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.176 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.176 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.820096+0000", 2024-05-15T23:26:43.176 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000112496 2024-05-15T23:26:43.177 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.177 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.177 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.177 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.820173+0000", 2024-05-15T23:26:43.177 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7179000000000001e-05 2024-05-15T23:26:43.177 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.177 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.177 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.177 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.825431+0000", 2024-05-15T23:26:43.178 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0052575199999999999 2024-05-15T23:26:43.178 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.178 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.178 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.178 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.040648+0000", 2024-05-15T23:26:43.178 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21521739400000001 2024-05-15T23:26:43.178 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.178 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.178 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.178 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.040681+0000", 2024-05-15T23:26:43.178 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2372000000000001e-05 2024-05-15T23:26:43.178 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.178 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.179 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.179 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.040710+0000", 2024-05-15T23:26:43.179 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9009999999999998e-05 2024-05-15T23:26:43.179 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.179 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.179 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.179 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.179 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.179 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304296 4.a 4:59aa04f1:::benchmark_data_smithi090_104448_object131058:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:26:43.179 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.828982+0000", 2024-05-15T23:26:43.179 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 497.32906938299999, 2024-05-15T23:26:43.179 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22203131700000001, 2024-05-15T23:26:43.179 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.179 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.179 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.180 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.180 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.180 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.180 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304296 2024-05-15T23:26:43.180 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.180 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.180 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.180 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.180 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.828982+0000", 2024-05-15T23:26:43.180 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.180 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.180 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.180 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.180 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.828982+0000", 2024-05-15T23:26:43.180 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.828984+0000", 2024-05-15T23:26:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.243e-06 2024-05-15T23:26:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.828992+0000", 2024-05-15T23:26:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4479999999999997e-06 2024-05-15T23:26:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.829004+0000", 2024-05-15T23:26:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1747e-05 2024-05-15T23:26:43.182 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.182 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.182 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.182 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.829012+0000", 2024-05-15T23:26:43.182 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.9590000000000001e-06 2024-05-15T23:26:43.182 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.182 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.182 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.182 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.829077+0000", 2024-05-15T23:26:43.182 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4916e-05 2024-05-15T23:26:43.182 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.182 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.182 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.183 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.829221+0000", 2024-05-15T23:26:43.183 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014339600000000001 2024-05-15T23:26:43.183 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.183 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.183 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.183 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.829352+0000", 2024-05-15T23:26:43.183 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000130996 2024-05-15T23:26:43.183 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.183 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.184 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.184 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.834093+0000", 2024-05-15T23:26:43.184 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0047415189999999996 2024-05-15T23:26:43.184 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.184 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.184 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.184 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.050919+0000", 2024-05-15T23:26:43.184 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21682558900000001 2024-05-15T23:26:43.185 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.185 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.185 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.185 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.050970+0000", 2024-05-15T23:26:43.185 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0618000000000002e-05 2024-05-15T23:26:43.185 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.185 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.185 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.186 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051013+0000", 2024-05-15T23:26:43.186 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3886e-05 2024-05-15T23:26:43.186 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.186 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.186 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.186 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.186 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.187 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304319 4.a 4:5860cfdb:::benchmark_data_smithi090_104448_object131081:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:26:43.187 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.849831+0000", 2024-05-15T23:26:43.187 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 497.30822033999999, 2024-05-15T23:26:43.187 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20133589099999999, 2024-05-15T23:26:43.187 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.187 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.187 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.188 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.188 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.188 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.188 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304319 2024-05-15T23:26:43.188 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.188 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.188 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.189 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.189 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849831+0000", 2024-05-15T23:26:43.189 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.189 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.189 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.189 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.189 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849831+0000", 2024-05-15T23:26:43.190 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.190 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.190 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.190 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.190 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849834+0000", 2024-05-15T23:26:43.190 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.875e-06 2024-05-15T23:26:43.191 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.191 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.191 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.191 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849842+0000", 2024-05-15T23:26:43.191 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.4659999999999999e-06 2024-05-15T23:26:43.191 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.191 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.192 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849858+0000", 2024-05-15T23:26:43.192 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5536000000000002e-05 2024-05-15T23:26:43.192 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.192 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.192 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.193 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849868+0000", 2024-05-15T23:26:43.193 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7610000000000002e-06 2024-05-15T23:26:43.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.193 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.193 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.193 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.849918+0000", 2024-05-15T23:26:43.193 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0219000000000001e-05 2024-05-15T23:26:43.193 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.194 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.194 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.194 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.850026+0000", 2024-05-15T23:26:43.194 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000108256 2024-05-15T23:26:43.194 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.194 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.194 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.195 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.850126+0000", 2024-05-15T23:26:43.195 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9479000000000001e-05 2024-05-15T23:26:43.195 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.195 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.195 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.195 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.854202+0000", 2024-05-15T23:26:43.195 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004076774 2024-05-15T23:26:43.195 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.196 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051073+0000", 2024-05-15T23:26:43.196 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19687078499999999 2024-05-15T23:26:43.196 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.196 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051124+0000", 2024-05-15T23:26:43.197 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0436999999999998e-05 2024-05-15T23:26:43.197 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.197 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.197 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.197 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051167+0000", 2024-05-15T23:26:43.197 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3303000000000002e-05 2024-05-15T23:26:43.197 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.197 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.198 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.198 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.198 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.198 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304341 4.a 4:53ff7ba8:::benchmark_data_smithi090_104448_object131103:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:26:43.198 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.865333+0000", 2024-05-15T23:26:43.198 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 497.29271819100001, 2024-05-15T23:26:43.198 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18599020299999999, 2024-05-15T23:26:43.199 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.199 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.199 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.199 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.199 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.199 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.199 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304341 2024-05-15T23:26:43.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.200 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865333+0000", 2024-05-15T23:26:43.200 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865333+0000", 2024-05-15T23:26:43.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.201 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.201 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.201 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865334+0000", 2024-05-15T23:26:43.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0999999999999998e-07 2024-05-15T23:26:43.201 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.202 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.202 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.202 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865337+0000", 2024-05-15T23:26:43.202 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7470000000000001e-06 2024-05-15T23:26:43.202 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.202 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.202 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.202 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865341+0000", 2024-05-15T23:26:43.202 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4789999999999998e-06 2024-05-15T23:26:43.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.203 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.203 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865344+0000", 2024-05-15T23:26:43.203 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1030000000000002e-06 2024-05-15T23:26:43.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.203 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.204 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865395+0000", 2024-05-15T23:26:43.204 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0340999999999998e-05 2024-05-15T23:26:43.204 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.204 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.204 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.204 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865511+0000", 2024-05-15T23:26:43.204 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000116851 2024-05-15T23:26:43.204 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.205 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.205 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.205 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.865618+0000", 2024-05-15T23:26:43.205 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000106659 2024-05-15T23:26:43.205 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.205 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.205 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.205 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.870311+0000", 2024-05-15T23:26:43.205 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0046930080000000002 2024-05-15T23:26:43.205 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.205 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.205 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.206 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051226+0000", 2024-05-15T23:26:43.206 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.180914669 2024-05-15T23:26:43.206 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.206 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.206 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.206 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051279+0000", 2024-05-15T23:26:43.206 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3637999999999997e-05 2024-05-15T23:26:43.206 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.206 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.206 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.206 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051323+0000", 2024-05-15T23:26:43.206 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4097999999999997e-05 2024-05-15T23:26:43.206 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.207 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.207 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.207 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.207 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.207 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304347 4.a 4:5813fd51:::benchmark_data_smithi090_104448_object131109:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:26:43.207 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.873270+0000", 2024-05-15T23:26:43.207 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 497.284781899, 2024-05-15T23:26:43.207 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17820270799999999, 2024-05-15T23:26:43.207 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.207 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.207 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.207 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.208 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.208 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.208 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304347 2024-05-15T23:26:43.208 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.208 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.208 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.208 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.208 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873270+0000", 2024-05-15T23:26:43.208 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.208 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.208 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.208 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.208 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873270+0000", 2024-05-15T23:26:43.208 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.208 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.209 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.209 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.209 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873270+0000", 2024-05-15T23:26:43.209 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6700000000000003e-07 2024-05-15T23:26:43.209 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.209 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.209 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.209 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873274+0000", 2024-05-15T23:26:43.209 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5300000000000001e-06 2024-05-15T23:26:43.209 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.209 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.209 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.209 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873281+0000", 2024-05-15T23:26:43.209 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3300000000000001e-06 2024-05-15T23:26:43.209 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.210 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.210 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.210 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873288+0000", 2024-05-15T23:26:43.210 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8569999999999997e-06 2024-05-15T23:26:43.210 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.210 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.210 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.210 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873335+0000", 2024-05-15T23:26:43.210 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7434e-05 2024-05-15T23:26:43.210 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.210 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.210 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.210 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873420+0000", 2024-05-15T23:26:43.210 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5000000000000006e-05 2024-05-15T23:26:43.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.873494+0000", 2024-05-15T23:26:43.211 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3901999999999998e-05 2024-05-15T23:26:43.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882687+0000", 2024-05-15T23:26:43.211 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0091921250000000006 2024-05-15T23:26:43.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051380+0000", 2024-05-15T23:26:43.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.168693808 2024-05-15T23:26:43.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051431+0000", 2024-05-15T23:26:43.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1150000000000003e-05 2024-05-15T23:26:43.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.051472+0000", 2024-05-15T23:26:43.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0805e-05 2024-05-15T23:26:43.212 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.212 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.213 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.213 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.213 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.213 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304351 4.a 4:582c10c7:::benchmark_data_smithi090_104448_object131113:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:26:43.214 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.874105+0000", 2024-05-15T23:26:43.214 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 497.28394672899998, 2024-05-15T23:26:43.214 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18030819300000001, 2024-05-15T23:26:43.214 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.214 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.214 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.215 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.215 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.215 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.215 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304351 2024-05-15T23:26:43.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.215 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.215 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874105+0000", 2024-05-15T23:26:43.215 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874105+0000", 2024-05-15T23:26:43.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874106+0000", 2024-05-15T23:26:43.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2950000000000001e-06 2024-05-15T23:26:43.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874111+0000", 2024-05-15T23:26:43.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8080000000000004e-06 2024-05-15T23:26:43.217 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.217 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.217 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.217 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874126+0000", 2024-05-15T23:26:43.217 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4915e-05 2024-05-15T23:26:43.217 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.217 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.217 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.217 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.874133+0000", 2024-05-15T23:26:43.218 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4760000000000001e-06 2024-05-15T23:26:43.218 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.218 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.218 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.218 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.877862+0000", 2024-05-15T23:26:43.218 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037288489999999998 2024-05-15T23:26:43.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.219 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.219 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.219 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.877973+0000", 2024-05-15T23:26:43.219 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011109599999999999 2024-05-15T23:26:43.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.220 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.878075+0000", 2024-05-15T23:26:43.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000102221 2024-05-15T23:26:43.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.221 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.221 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.221 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882814+0000", 2024-05-15T23:26:43.221 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004738669 2024-05-15T23:26:43.221 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.221 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054322+0000", 2024-05-15T23:26:43.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.171508251 2024-05-15T23:26:43.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054373+0000", 2024-05-15T23:26:43.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0822999999999998e-05 2024-05-15T23:26:43.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054413+0000", 2024-05-15T23:26:43.224 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9789999999999997e-05 2024-05-15T23:26:43.224 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.224 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.224 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.224 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.224 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304363 4.a 4:571d2066:::benchmark_data_smithi090_104448_object131125:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:26:43.225 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.882815+0000", 2024-05-15T23:26:43.225 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 497.275236087, 2024-05-15T23:26:43.225 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17174726100000001, 2024-05-15T23:26:43.225 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.225 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.225 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.226 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.226 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.226 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.226 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304363 2024-05-15T23:26:43.226 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.226 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.227 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.227 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.227 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882815+0000", 2024-05-15T23:26:43.227 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.227 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.227 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882815+0000", 2024-05-15T23:26:43.228 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.228 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882817+0000", 2024-05-15T23:26:43.229 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8950000000000001e-06 2024-05-15T23:26:43.229 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.229 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.229 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.229 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882823+0000", 2024-05-15T23:26:43.229 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3410000000000002e-06 2024-05-15T23:26:43.229 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.229 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.229 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.229 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882835+0000", 2024-05-15T23:26:43.229 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2853999999999999e-05 2024-05-15T23:26:43.230 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.230 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.230 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.230 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882848+0000", 2024-05-15T23:26:43.230 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2695e-05 2024-05-15T23:26:43.230 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.230 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.230 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.230 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.882894+0000", 2024-05-15T23:26:43.231 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5383999999999998e-05 2024-05-15T23:26:43.231 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.231 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.231 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.231 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883006+0000", 2024-05-15T23:26:43.231 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011286299999999999 2024-05-15T23:26:43.231 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.232 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.232 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.232 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883119+0000", 2024-05-15T23:26:43.232 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011275199999999999 2024-05-15T23:26:43.232 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.232 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.232 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.891222+0000", 2024-05-15T23:26:43.233 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0081027779999999997 2024-05-15T23:26:43.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.233 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.233 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054470+0000", 2024-05-15T23:26:43.233 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.163247948 2024-05-15T23:26:43.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.234 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.234 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054521+0000", 2024-05-15T23:26:43.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0760999999999997e-05 2024-05-15T23:26:43.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.235 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.235 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054563+0000", 2024-05-15T23:26:43.235 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1990000000000003e-05 2024-05-15T23:26:43.235 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.235 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.235 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.235 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.235 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.236 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304365 4.a 4:5a74db35:::benchmark_data_smithi090_104448_object131127:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:26:43.236 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.883969+0000", 2024-05-15T23:26:43.236 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 497.27408231999999, 2024-05-15T23:26:43.236 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17074013900000001, 2024-05-15T23:26:43.236 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.236 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.236 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.237 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.237 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.237 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.237 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304365 2024-05-15T23:26:43.237 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.237 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.237 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883969+0000", 2024-05-15T23:26:43.238 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.238 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883969+0000", 2024-05-15T23:26:43.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.239 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.239 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883971+0000", 2024-05-15T23:26:43.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.567e-06 2024-05-15T23:26:43.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883976+0000", 2024-05-15T23:26:43.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0209999999999999e-06 2024-05-15T23:26:43.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.241 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.241 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.241 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883988+0000", 2024-05-15T23:26:43.241 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.256e-05 2024-05-15T23:26:43.241 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.241 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.242 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.242 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.883996+0000", 2024-05-15T23:26:43.242 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8779999999999998e-06 2024-05-15T23:26:43.242 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.242 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.242 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.242 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.887685+0000", 2024-05-15T23:26:43.242 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036892100000000001 2024-05-15T23:26:43.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.243 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.887768+0000", 2024-05-15T23:26:43.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2952000000000001e-05 2024-05-15T23:26:43.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.244 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.887841+0000", 2024-05-15T23:26:43.244 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2880999999999996e-05 2024-05-15T23:26:43.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.245 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.891325+0000", 2024-05-15T23:26:43.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0034839340000000002 2024-05-15T23:26:43.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054613+0000", 2024-05-15T23:26:43.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.163287922 2024-05-15T23:26:43.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.247 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.247 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054668+0000", 2024-05-15T23:26:43.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4787999999999998e-05 2024-05-15T23:26:43.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.247 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.247 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.054709+0000", 2024-05-15T23:26:43.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1425999999999999e-05 2024-05-15T23:26:43.247 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.247 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.247 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.247 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:304372 4.a 4:519d11bc:::benchmark_data_smithi090_104448_object131134:head [delete] snapc 0=[] ondisk+write+known_if_redirected e348)", 2024-05-15T23:26:43.247 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:25.892294+0000", 2024-05-15T23:26:43.248 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 497.26575701899998, 2024-05-15T23:26:43.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17457294600000001, 2024-05-15T23:26:43.248 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.248 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.248 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.248 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.248 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.248 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.248 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 304372 2024-05-15T23:26:43.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.248 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.248 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892294+0000", 2024-05-15T23:26:43.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892294+0000", 2024-05-15T23:26:43.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892296+0000", 2024-05-15T23:26:43.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8479999999999999e-06 2024-05-15T23:26:43.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892303+0000", 2024-05-15T23:26:43.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3899999999999998e-06 2024-05-15T23:26:43.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892313+0000", 2024-05-15T23:26:43.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9599999999999995e-06 2024-05-15T23:26:43.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892320+0000", 2024-05-15T23:26:43.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0419999999999997e-06 2024-05-15T23:26:43.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892359+0000", 2024-05-15T23:26:43.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9001999999999997e-05 2024-05-15T23:26:43.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892450+0000", 2024-05-15T23:26:43.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0978999999999997e-05 2024-05-15T23:26:43.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.892524+0000", 2024-05-15T23:26:43.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3943999999999995e-05 2024-05-15T23:26:43.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:25.896062+0000", 2024-05-15T23:26:43.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0035380670000000002 2024-05-15T23:26:43.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.066773+0000", 2024-05-15T23:26:43.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.170711062 2024-05-15T23:26:43.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.066825+0000", 2024-05-15T23:26:43.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1969999999999999e-05 2024-05-15T23:26:43.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:26.066867+0000", 2024-05-15T23:26:43.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2682000000000002e-05 2024-05-15T23:26:43.253 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.253 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.253 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.253 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313500 4.a 4:54957ed1:::benchmark_data_smithi090_104448_object140262:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:26:43.253 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.433300+0000", 2024-05-15T23:26:43.253 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.72475127199999, 2024-05-15T23:26:43.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.232290565, 2024-05-15T23:26:43.253 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.253 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313500 2024-05-15T23:26:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433300+0000", 2024-05-15T23:26:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433300+0000", 2024-05-15T23:26:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433301+0000", 2024-05-15T23:26:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7600000000000006e-07 2024-05-15T23:26:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433305+0000", 2024-05-15T23:26:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2880000000000003e-06 2024-05-15T23:26:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433311+0000", 2024-05-15T23:26:43.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1279999999999999e-06 2024-05-15T23:26:43.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433314+0000", 2024-05-15T23:26:43.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8130000000000002e-06 2024-05-15T23:26:43.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433357+0000", 2024-05-15T23:26:43.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.214e-05 2024-05-15T23:26:43.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433465+0000", 2024-05-15T23:26:43.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000108762 2024-05-15T23:26:43.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.433582+0000", 2024-05-15T23:26:43.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011637 2024-05-15T23:26:43.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.438171+0000", 2024-05-15T23:26:43.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045897919999999997 2024-05-15T23:26:43.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665486+0000", 2024-05-15T23:26:43.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.22731428400000001 2024-05-15T23:26:43.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665546+0000", 2024-05-15T23:26:43.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9988999999999998e-05 2024-05-15T23:26:43.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665591+0000", 2024-05-15T23:26:43.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5022999999999999e-05 2024-05-15T23:26:43.258 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.259 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.259 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.259 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313526 4.a 4:537fd6b5:::benchmark_data_smithi090_104448_object140288:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:26:43.259 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.459071+0000", 2024-05-15T23:26:43.259 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.69898011999999, 2024-05-15T23:26:43.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20662114200000001, 2024-05-15T23:26:43.259 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.259 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.259 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.259 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.259 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.259 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.259 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313526 2024-05-15T23:26:43.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.260 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459071+0000", 2024-05-15T23:26:43.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459071+0000", 2024-05-15T23:26:43.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459073+0000", 2024-05-15T23:26:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8980000000000001e-06 2024-05-15T23:26:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459079+0000", 2024-05-15T23:26:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2730000000000002e-06 2024-05-15T23:26:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459090+0000", 2024-05-15T23:26:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1619e-05 2024-05-15T23:26:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459097+0000", 2024-05-15T23:26:43.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4189999999999999e-06 2024-05-15T23:26:43.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459148+0000", 2024-05-15T23:26:43.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1921e-05 2024-05-15T23:26:43.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459248+0000", 2024-05-15T23:26:43.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9388000000000005e-05 2024-05-15T23:26:43.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.459325+0000", 2024-05-15T23:26:43.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7024e-05 2024-05-15T23:26:43.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.462998+0000", 2024-05-15T23:26:43.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036728120000000001 2024-05-15T23:26:43.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665627+0000", 2024-05-15T23:26:43.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.202629165 2024-05-15T23:26:43.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665657+0000", 2024-05-15T23:26:43.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9984999999999998e-05 2024-05-15T23:26:43.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.665692+0000", 2024-05-15T23:26:43.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5638e-05 2024-05-15T23:26:43.264 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.264 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.264 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.264 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313534 4.a 4:53d3c98e:::benchmark_data_smithi090_104448_object140296:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:26:43.264 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.467942+0000", 2024-05-15T23:26:43.265 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.69010902000002, 2024-05-15T23:26:43.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.20189164800000001, 2024-05-15T23:26:43.265 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.265 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.265 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.265 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.265 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.265 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.265 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313534 2024-05-15T23:26:43.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.265 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467942+0000", 2024-05-15T23:26:43.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467942+0000", 2024-05-15T23:26:43.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467944+0000", 2024-05-15T23:26:43.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1459999999999999e-06 2024-05-15T23:26:43.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467948+0000", 2024-05-15T23:26:43.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8389999999999997e-06 2024-05-15T23:26:43.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467957+0000", 2024-05-15T23:26:43.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0790000000000001e-06 2024-05-15T23:26:43.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.467962+0000", 2024-05-15T23:26:43.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5180000000000004e-06 2024-05-15T23:26:43.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.468001+0000", 2024-05-15T23:26:43.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9119999999999998e-05 2024-05-15T23:26:43.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.468074+0000", 2024-05-15T23:26:43.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2698000000000001e-05 2024-05-15T23:26:43.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.468139+0000", 2024-05-15T23:26:43.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.512e-05 2024-05-15T23:26:43.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.471787+0000", 2024-05-15T23:26:43.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036482730000000001 2024-05-15T23:26:43.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669769+0000", 2024-05-15T23:26:43.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.19798183499999999 2024-05-15T23:26:43.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669803+0000", 2024-05-15T23:26:43.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3670999999999997e-05 2024-05-15T23:26:43.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669834+0000", 2024-05-15T23:26:43.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1349000000000001e-05 2024-05-15T23:26:43.270 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.270 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.270 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.270 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313558 4.a 4:526840ac:::benchmark_data_smithi090_104448_object140320:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:26:43.271 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.489279+0000", 2024-05-15T23:26:43.271 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.66877252799998, 2024-05-15T23:26:43.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18070008900000001, 2024-05-15T23:26:43.271 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.271 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.271 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.271 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.271 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.271 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.271 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313558 2024-05-15T23:26:43.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.271 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489279+0000", 2024-05-15T23:26:43.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489279+0000", 2024-05-15T23:26:43.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489281+0000", 2024-05-15T23:26:43.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.784e-06 2024-05-15T23:26:43.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489286+0000", 2024-05-15T23:26:43.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1699999999999996e-06 2024-05-15T23:26:43.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489296+0000", 2024-05-15T23:26:43.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.005e-05 2024-05-15T23:26:43.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489302+0000", 2024-05-15T23:26:43.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5309999999999998e-06 2024-05-15T23:26:43.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489348+0000", 2024-05-15T23:26:43.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5028000000000003e-05 2024-05-15T23:26:43.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489429+0000", 2024-05-15T23:26:43.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1233e-05 2024-05-15T23:26:43.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.489499+0000", 2024-05-15T23:26:43.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0450000000000005e-05 2024-05-15T23:26:43.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.493430+0000", 2024-05-15T23:26:43.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0039305049999999999 2024-05-15T23:26:43.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669881+0000", 2024-05-15T23:26:43.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17645111499999999 2024-05-15T23:26:43.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669936+0000", 2024-05-15T23:26:43.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5328999999999998e-05 2024-05-15T23:26:43.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.669979+0000", 2024-05-15T23:26:43.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2894e-05 2024-05-15T23:26:43.276 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.276 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.276 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.276 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313566 4.a 4:56b72cc5:::benchmark_data_smithi090_104448_object140328:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:26:43.276 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.498107+0000", 2024-05-15T23:26:43.276 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.65994423500001, 2024-05-15T23:26:43.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.18568280400000001, 2024-05-15T23:26:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313566 2024-05-15T23:26:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498107+0000", 2024-05-15T23:26:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498107+0000", 2024-05-15T23:26:43.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498110+0000", 2024-05-15T23:26:43.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3140000000000002e-06 2024-05-15T23:26:43.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498117+0000", 2024-05-15T23:26:43.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1690000000000003e-06 2024-05-15T23:26:43.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498128+0000", 2024-05-15T23:26:43.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1362000000000001e-05 2024-05-15T23:26:43.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498136+0000", 2024-05-15T23:26:43.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5769999999999996e-06 2024-05-15T23:26:43.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498184+0000", 2024-05-15T23:26:43.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8631000000000002e-05 2024-05-15T23:26:43.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498280+0000", 2024-05-15T23:26:43.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5958999999999994e-05 2024-05-15T23:26:43.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.498356+0000", 2024-05-15T23:26:43.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5574000000000005e-05 2024-05-15T23:26:43.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.503160+0000", 2024-05-15T23:26:43.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0048046499999999997 2024-05-15T23:26:43.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683729+0000", 2024-05-15T23:26:43.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.180568066 2024-05-15T23:26:43.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683760+0000", 2024-05-15T23:26:43.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1798000000000003e-05 2024-05-15T23:26:43.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683790+0000", 2024-05-15T23:26:43.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9703999999999999e-05 2024-05-15T23:26:43.282 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.282 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.282 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.282 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25441.0:313575 4.a 4:5e037716:::benchmark_data_smithi090_104448_object140337:head [delete] snapc 0=[] ondisk+write+known_if_redirected e350)", 2024-05-15T23:26:43.282 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:18:33.507823+0000", 2024-05-15T23:26:43.282 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 489.65022866599998, 2024-05-15T23:26:43.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17607811600000001, 2024-05-15T23:26:43.282 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:43.282 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:43.282 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:43.282 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:43.283 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25441", 2024-05-15T23:26:43.283 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:26:43.283 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 313575 2024-05-15T23:26:43.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.283 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:43.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:43.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507823+0000", 2024-05-15T23:26:43.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:43.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507823+0000", 2024-05-15T23:26:43.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:43.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:43.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507825+0000", 2024-05-15T23:26:43.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3099999999999999e-06 2024-05-15T23:26:43.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:43.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507831+0000", 2024-05-15T23:26:43.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0120000000000002e-06 2024-05-15T23:26:43.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:43.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507843+0000", 2024-05-15T23:26:43.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1572e-05 2024-05-15T23:26:43.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:43.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507851+0000", 2024-05-15T23:26:43.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3159999999999997e-06 2024-05-15T23:26:43.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:43.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507900+0000", 2024-05-15T23:26:43.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9444999999999998e-05 2024-05-15T23:26:43.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:43.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.507981+0000", 2024-05-15T23:26:43.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0547e-05 2024-05-15T23:26:43.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:26:43.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.508068+0000", 2024-05-15T23:26:43.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6699000000000005e-05 2024-05-15T23:26:43.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:43.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.513050+0000", 2024-05-15T23:26:43.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049819829999999997 2024-05-15T23:26:43.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:43.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683820+0000", 2024-05-15T23:26:43.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.17076991899999999 2024-05-15T23:26:43.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:43.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683858+0000", 2024-05-15T23:26:43.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7917999999999998e-05 2024-05-15T23:26:43.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:43.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:43.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:43.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:18:33.683901+0000", 2024-05-15T23:26:43.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3395e-05 2024-05-15T23:26:43.287 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.287 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.287 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.287 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:43.288 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:43.288 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:26:43.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:26:43.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:26:43.390 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:43.417 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:43.616 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:43.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:43.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:26:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:26:43.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:43.785 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:26:43.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:26:43.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:43 smithi084 ceph-mon[39784]: pgmap v1378: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 127 B/s rd, 55 MiB/s wr, 1.43k op/s 2024-05-15T23:26:43.886 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:43.921 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:43.955 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:43.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:43.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:44.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:43 smithi090 ceph-mon[44434]: pgmap v1378: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 127 B/s rd, 55 MiB/s wr, 1.43k op/s 2024-05-15T23:26:44.063 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:26:44.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:43 smithi088 ceph-mon[44699]: pgmap v1378: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 127 B/s rd, 55 MiB/s wr, 1.43k op/s 2024-05-15T23:26:44.103 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:44.153 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:26:44.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:44.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:26:44.171 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:44.258 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:26:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:44.507 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:26:44.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:44.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:26:44.559 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T23:26:44.660 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:26:44.714 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:44.738 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:26:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:44.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:26:44.771 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:26:44.772 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3836820961' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:44.772 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:44 smithi088 ceph-mon[44699]: from='client.26461 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:44.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:26:44.878 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:45.000 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:26:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3836820961' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:44 smithi090 ceph-mon[44434]: from='client.26461 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:45.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3836820961' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:45.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:44 smithi084 ceph-mon[39784]: from='client.26461 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:26:45.733 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:26:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:45.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:26:45.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:45 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:45.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:45 smithi088 ceph-mon[44699]: from='client.26461 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:26:45.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:45 smithi088 ceph-mon[44699]: osdmap e517: 12 total, 11 up, 8 in 2024-05-15T23:26:45.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:45 smithi088 ceph-mon[44699]: pgmap v1380: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 127 B/s rd, 22 MiB/s wr, 889 op/s 2024-05-15T23:26:45.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:26:45.881 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:45.906 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:26:45.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:45.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:26:46.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:26:46.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:45 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:46.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:45 smithi084 ceph-mon[39784]: from='client.26461 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:26:46.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:45 smithi084 ceph-mon[39784]: osdmap e517: 12 total, 11 up, 8 in 2024-05-15T23:26:46.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:45 smithi084 ceph-mon[39784]: pgmap v1380: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 127 B/s rd, 22 MiB/s wr, 889 op/s 2024-05-15T23:26:46.048 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:45 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:45 smithi090 ceph-mon[44434]: from='client.26461 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:26:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:45 smithi090 ceph-mon[44434]: osdmap e517: 12 total, 11 up, 8 in 2024-05-15T23:26:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:45 smithi090 ceph-mon[44434]: pgmap v1380: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 127 B/s rd, 22 MiB/s wr, 889 op/s 2024-05-15T23:26:46.151 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:26:46.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:46.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:26:46.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:46.299 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:46.533 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:46.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:46.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:46.591 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:26:46.615 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:26:46.640 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:26:46.644 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:26:46.672 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:26:46.673 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:46.698 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:26:46.700 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:26:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:46.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:26:46.720 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:26:46.750 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:26:46.778 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:26:46.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:46.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:26:46.832 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:26:46.841 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:46.863 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:26:46.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:26:46.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:26:46.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:26:46.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:26:46.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:26:46.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:26:46.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:26:46.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:26:46.864 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-05-15T23:26:46.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:26:46.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:26:46.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:26:46.966 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:46 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/891010719' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:47.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:46 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/891010719' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:47.056 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:46 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/891010719' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:47.196 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:26:47.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:47.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:26:47.300 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:26:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:47.371 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:26:47.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:47.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:26:47.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:26:47.514 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:47.791 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.791 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.791 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.791 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.791 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.791 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.791 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.791 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.792 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.792 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.792 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:47 smithi090 ceph-mon[44434]: pgmap v1381: 17 pgs: 17 active+clean; 7.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 12 MiB/s wr, 1.20k op/s 2024-05-15T23:26:47.792 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.792 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.792 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.792 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.792 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.792 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.792 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.793 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.793 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.793 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:47.793 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:47 smithi088 ceph-mon[44699]: pgmap v1381: 17 pgs: 17 active+clean; 7.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 12 MiB/s wr, 1.20k op/s 2024-05-15T23:26:47.822 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:26:47.823 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:26:47.846 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:26:47.846 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:26:47.864 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:26:47.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:47.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:26:47.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:26:47.998 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:48.021 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:26:48.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:48.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:26:48.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:48.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:48.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:48.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:48.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:48.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:48.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:48.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:48.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:48.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:48.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:47 smithi084 ceph-mon[39784]: pgmap v1381: 17 pgs: 17 active+clean; 7.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 12 MiB/s wr, 1.20k op/s 2024-05-15T23:26:48.125 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:26:48.185 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:48.259 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:26:48.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:48.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:26:48.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:48.401 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:48.423 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:26:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:48.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:26:48.528 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:48.563 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:48.586 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:26:48.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:48.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:26:48.624 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T23:26:48.693 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:26:48.744 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:48.781 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:26:48.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:48.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:48.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:26:48.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:26:48.925 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:48.950 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:26:48.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:48.954 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:26:48.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:26:49.023 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:26:49.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:49.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:26:49.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:49.086 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:49.110 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:26:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:49.113 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:26:49.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:26:49.253 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:49.287 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:26:49.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:49.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:26:49.354 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:49.400 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:26:49.458 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:49.504 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:26:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:49.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:26:49.612 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:26:49.655 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:49.824 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:26:49.860 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:49 smithi084 ceph-mon[39784]: pgmap v1382: 17 pgs: 17 active+clean; 7.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 11 MiB/s wr, 1.17k op/s 2024-05-15T23:26:50.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:49 smithi090 ceph-mon[44434]: pgmap v1382: 17 pgs: 17 active+clean; 7.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 11 MiB/s wr, 1.17k op/s 2024-05-15T23:26:50.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:49 smithi088 ceph-mon[44699]: pgmap v1382: 17 pgs: 17 active+clean; 7.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 11 MiB/s wr, 1.17k op/s 2024-05-15T23:26:50.921 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:26:50.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:50.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:26:51.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:26:51.059 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:51.084 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:26:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:51.087 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:26:51.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:26:51.221 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:51 smithi084 ceph-mon[39784]: pgmap v1383: 17 pgs: 17 active+clean; 7.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.9 MiB/s wr, 1.04k op/s 2024-05-15T23:26:51.224 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:51.230 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:51 smithi088 ceph-mon[44699]: pgmap v1383: 17 pgs: 17 active+clean; 7.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.9 MiB/s wr, 1.04k op/s 2024-05-15T23:26:51.255 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:26:51.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:51.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:26:51.309 INFO:tasks.thrashosds.thrasher:Reviving osd 8 2024-05-15T23:26:51.310 INFO:tasks.cephadm.osd.8:Restarting osd.8 (starting--it wasn't running)... 2024-05-15T23:26:51.310 DEBUG:teuthology.orchestra.run.smithi090:osd.8> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.8.service 2024-05-15T23:26:51.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl start ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.8 2024-05-15T23:26:51.345 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:51 smithi090 ceph-mon[44434]: pgmap v1383: 17 pgs: 17 active+clean; 7.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.9 MiB/s wr, 1.04k op/s 2024-05-15T23:26:51.352 INFO:journalctl@ceph.osd.8.smithi090.stdout:-- Logs begin at Wed 2024-05-15 22:43:42 UTC. -- 2024-05-15T23:26:51.365 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:26:51.413 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:51.571 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:51.625 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:26:51.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:51.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:26:51.732 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:51.774 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:51.803 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:26:51 smithi090 systemd[1]: Starting Ceph osd.8 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:26:51.851 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:26:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:51.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:26:51.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:26:51.997 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:52.113 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:26:52.166 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:26:52.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:52.177 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:26:52.178 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:52 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:26:52.178 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:52 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:26:52.178 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:52 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3828236311' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:26:52.278 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:26:52.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:52 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:26:52.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:52 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:26:52.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:52 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3828236311' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:26:52.336 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:52.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:52 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:26:52.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:52 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:26:52.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:52 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3828236311' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:26:52.370 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:26:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:52.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:26:52.378 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:26:52.378 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:26:52.479 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:26:52.533 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:52.658 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:26:52.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:52.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:26:52.766 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:26:52.802 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:53.052 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:26:52 smithi090 bash[136082]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-05-15T23:26:53.052 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:26:52 smithi090 bash[136082]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-05-15T23:26:53.052 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:26:52 smithi090 bash[136082]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-8 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-05-15T23:26:53.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:26:52 smithi090 bash[136082]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-05-15T23:26:53.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:26:52 smithi090 bash[136082]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-05-15T23:26:53.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:26:52 smithi090 bash[136082]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-8/block 2024-05-15T23:26:53.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:26:52 smithi090 bash[136082]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-05-15T23:26:53.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:26:52 smithi090 bash[136082]: --> ceph-volume raw activate successful for osd ID: 8 2024-05-15T23:26:53.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:53 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:53.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:53 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3828236311' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:26:53.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:53 smithi084 ceph-mon[39784]: osdmap e518: 12 total, 11 up, 8 in 2024-05-15T23:26:53.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:53 smithi084 ceph-mon[39784]: pgmap v1385: 17 pgs: 17 active+clean; 7.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.46k op/s 2024-05-15T23:26:53.431 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:53 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:53.431 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:53 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3828236311' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:26:53.431 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:53 smithi088 ceph-mon[44699]: osdmap e518: 12 total, 11 up, 8 in 2024-05-15T23:26:53.432 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:53 smithi088 ceph-mon[44699]: pgmap v1385: 17 pgs: 17 active+clean; 7.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.46k op/s 2024-05-15T23:26:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:53 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:26:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:53 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3828236311' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:26:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:53 smithi090 ceph-mon[44434]: osdmap e518: 12 total, 11 up, 8 in 2024-05-15T23:26:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:53 smithi090 ceph-mon[44434]: pgmap v1385: 17 pgs: 17 active+clean; 7.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.46k op/s 2024-05-15T23:26:53.858 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T23:26:53.878 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:26:53.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:53.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:26:53.986 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:26:54.025 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:54.328 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:26:54.376 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:26:54.403 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:26:54.428 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:26:54.457 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:26:54.484 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:26:54.517 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:26:54.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T23:26:54.545 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:26:54.576 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:26:54.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:54.580 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:26:54.581 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:26:54.614 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:26:54.645 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:26:54.669 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:26:54.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:26:54.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:26:54.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:26:54.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:26:54.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:26:54.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:26:54.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:26:54.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:26:54.670 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-05-15T23:26:54.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:26:54.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:26:54.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:26:54.679 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2158454351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:54.680 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:26:54.687 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2158454351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:54.695 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2158454351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:26:54.695 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:26:54 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:26:54.515+0000 7f4052e837c0 -1 Falling back to public interface 2024-05-15T23:26:54.695 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:26:54 smithi090 bash[136351]: c967d2b2fc3e5c41e4d6e3f2d31960c5894063a8a3b432662d37891fb11b9a5d 2024-05-15T23:26:54.696 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:26:54 smithi090 systemd[1]: Started Ceph osd.8 for 3999fb7e-130e-11ef-bc99-c7b262605968. 2024-05-15T23:26:54.732 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:54.768 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:26:54.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:54.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:26:54.873 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:26:54.943 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:55.239 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:26:55.286 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:26:55.700 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.700 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.700 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.700 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.700 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:55 smithi088 ceph-mon[44699]: 6.3 scrub starts 2024-05-15T23:26:55.700 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.700 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.700 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.700 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.701 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:55 smithi088 ceph-mon[44699]: pgmap v1386: 17 pgs: 17 active+clean; 7.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:26:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:55 smithi090 ceph-mon[44434]: 6.3 scrub starts 2024-05-15T23:26:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:55 smithi090 ceph-mon[44434]: pgmap v1386: 17 pgs: 17 active+clean; 7.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:26:55.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:55 smithi084 ceph-mon[39784]: 6.3 scrub starts 2024-05-15T23:26:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:55.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:55 smithi084 ceph-mon[39784]: pgmap v1386: 17 pgs: 17 active+clean; 7.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:26:56.405 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:26:56.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:56.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:26:56.428 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:26:56.429 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:26:56.496 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:56.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:26:56.551 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:56 smithi090 ceph-mon[44434]: 6.8 scrub starts 2024-05-15T23:26:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:56.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:56 smithi088 ceph-mon[44699]: 6.8 scrub starts 2024-05-15T23:26:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:26:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:56 smithi084 ceph-mon[39784]: 6.8 scrub starts 2024-05-15T23:26:57.023 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:26:57.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:57.030 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:26:57.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:26:57.170 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:57.197 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:26:57.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:57.202 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:26:57.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:26:57 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:26:57.080+0000 7f4052e837c0 -1 osd.8 484 log_to_monitors true 2024-05-15T23:26:57.302 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:26:57.343 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:57.424 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:26:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:57.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:26:57.533 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:26:57.566 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:57.594 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:26:57.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:57.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:26:57.602 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:57 smithi088 ceph-mon[44699]: 6.3 scrub ok 2024-05-15T23:26:57.602 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:57 smithi088 ceph-mon[44699]: pgmap v1387: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:26:57.602 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:57 smithi088 ceph-mon[44699]: from='osd.8 v1:172.21.15.90:6800/703598286' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T23:26:57.602 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:57 smithi088 ceph-mon[44699]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T23:26:57.700 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:26:57.731 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:57 smithi084 ceph-mon[39784]: 6.3 scrub ok 2024-05-15T23:26:57.731 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:57 smithi084 ceph-mon[39784]: pgmap v1387: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:26:57.731 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:57 smithi084 ceph-mon[39784]: from='osd.8 v1:172.21.15.90:6800/703598286' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T23:26:57.731 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:57 smithi084 ceph-mon[39784]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T23:26:57.732 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:57.769 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:57 smithi090 ceph-mon[44434]: 6.3 scrub ok 2024-05-15T23:26:57.769 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:57 smithi090 ceph-mon[44434]: pgmap v1387: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:26:57.769 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:57 smithi090 ceph-mon[44434]: from='osd.8 v1:172.21.15.90:6800/703598286' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T23:26:57.770 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:57 smithi090 ceph-mon[44434]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-15T23:26:57.770 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:26:57 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:26:57.448+0000 7f40442da700 -1 osd.8 519 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T23:26:58.194 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:26:58.195 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:26:58.195 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:26:58.195 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:26:58.195 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.195 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:137208 5.9 5:9eefd750:::benchmark_data_smithi090_116470_object137207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e408)", 2024-05-15T23:26:58.195 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:09.079769+0000", 2024-05-15T23:26:58.195 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 349.11003370700001, 2024-05-15T23:26:58.195 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.41414984700000002, 2024-05-15T23:26:58.195 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.195 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.195 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.195 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.196 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.196 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.196 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 137208 2024-05-15T23:26:58.196 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.196 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.196 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.079769+0000", 2024-05-15T23:26:58.196 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.196 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.196 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.196 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.196 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.079769+0000", 2024-05-15T23:26:58.197 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.197 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.197 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.197 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.197 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.079788+0000", 2024-05-15T23:26:58.197 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8924e-05 2024-05-15T23:26:58.197 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.197 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.197 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.197 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.079891+0000", 2024-05-15T23:26:58.197 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000102837 2024-05-15T23:26:58.197 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.197 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.197 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.197 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.079914+0000", 2024-05-15T23:26:58.198 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.296e-05 2024-05-15T23:26:58.198 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.198 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.198 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.198 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.079940+0000", 2024-05-15T23:26:58.198 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6800000000000001e-05 2024-05-15T23:26:58.198 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.198 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.198 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.198 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.084420+0000", 2024-05-15T23:26:58.198 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0044796150000000002 2024-05-15T23:26:58.198 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.198 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:26:58.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.084434+0000", 2024-05-15T23:26:58.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3546e-05 2024-05-15T23:26:58.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.199 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.471085+0000", 2024-05-15T23:26:58.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.38665144200000001 2024-05-15T23:26:58.199 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.199 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.199 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.199 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.471238+0000", 2024-05-15T23:26:58.199 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015261699999999999 2024-05-15T23:26:58.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.471407+0000", 2024-05-15T23:26:58.200 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016973499999999999 2024-05-15T23:26:58.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.488865+0000", 2024-05-15T23:26:58.200 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.017457724000000001 2024-05-15T23:26:58.200 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.200 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.200 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.200 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.488875+0000", 2024-05-15T23:26:58.200 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9799999999999993e-06 2024-05-15T23:26:58.201 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.201 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.201 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.488911+0000", 2024-05-15T23:26:58.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5540999999999997e-05 2024-05-15T23:26:58.201 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.201 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.201 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.201 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.493919+0000", 2024-05-15T23:26:58.201 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050081259999999999 2024-05-15T23:26:58.201 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.202 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.202 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.202 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.202 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.202 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:137211 5.9 5:94202bff:::benchmark_data_smithi090_116470_object137210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e408)", 2024-05-15T23:26:58.202 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:09.079966+0000", 2024-05-15T23:26:58.202 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 349.10983668300003, 2024-05-15T23:26:58.202 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.430775405, 2024-05-15T23:26:58.202 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.202 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.202 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.202 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.202 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.202 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.203 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 137211 2024-05-15T23:26:58.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.203 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.203 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.203 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.079966+0000", 2024-05-15T23:26:58.203 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.203 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.203 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.079966+0000", 2024-05-15T23:26:58.203 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.203 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.203 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.203 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.079967+0000", 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9500000000000002e-07 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.079992+0000", 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5517999999999999e-05 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.079998+0000", 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.643e-06 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.204 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080002+0000", 2024-05-15T23:26:58.205 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.087e-06 2024-05-15T23:26:58.205 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.205 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.205 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.205 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.084534+0000", 2024-05-15T23:26:58.205 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0045321650000000003 2024-05-15T23:26:58.205 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.205 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.205 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:26:58.205 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.084542+0000", 2024-05-15T23:26:58.205 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3979999999999999e-06 2024-05-15T23:26:58.205 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.205 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.205 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.205 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.480895+0000", 2024-05-15T23:26:58.206 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.39635293599999999 2024-05-15T23:26:58.206 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.206 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.206 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.206 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.481008+0000", 2024-05-15T23:26:58.206 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000113093 2024-05-15T23:26:58.206 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.206 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.206 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.206 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.481122+0000", 2024-05-15T23:26:58.206 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000113637 2024-05-15T23:26:58.206 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.206 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.206 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.206 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.494013+0000", 2024-05-15T23:26:58.207 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.012890766999999999 2024-05-15T23:26:58.207 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.207 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.207 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.207 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.505346+0000", 2024-05-15T23:26:58.207 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011332946 2024-05-15T23:26:58.207 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.207 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.207 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.207 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.505412+0000", 2024-05-15T23:26:58.207 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6659e-05 2024-05-15T23:26:58.207 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.207 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.207 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.207 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.510741+0000", 2024-05-15T23:26:58.208 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0053288609999999998 2024-05-15T23:26:58.208 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.208 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.208 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.208 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.208 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.208 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:137254 5.9 5:924cbd99:::benchmark_data_smithi090_116470_object137253:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e408)", 2024-05-15T23:26:58.208 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:09.080007+0000", 2024-05-15T23:26:58.208 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 349.10979517200002, 2024-05-15T23:26:58.208 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.43689284699999997, 2024-05-15T23:26:58.208 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.208 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.208 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.208 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.208 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.209 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.209 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 137254 2024-05-15T23:26:58.209 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.209 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.209 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.209 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.209 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080007+0000", 2024-05-15T23:26:58.209 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.209 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.209 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.209 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.209 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080007+0000", 2024-05-15T23:26:58.209 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.209 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.209 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.210 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.210 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080008+0000", 2024-05-15T23:26:58.210 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2800000000000002e-07 2024-05-15T23:26:58.210 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.210 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.210 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.210 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080067+0000", 2024-05-15T23:26:58.210 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9240000000000002e-05 2024-05-15T23:26:58.210 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.210 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.210 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.210 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080073+0000", 2024-05-15T23:26:58.210 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.463e-06 2024-05-15T23:26:58.210 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080089+0000", 2024-05-15T23:26:58.211 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6412000000000001e-05 2024-05-15T23:26:58.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.084582+0000", 2024-05-15T23:26:58.211 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0044932189999999997 2024-05-15T23:26:58.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.211 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.211 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:26:58.211 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.084590+0000", 2024-05-15T23:26:58.211 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0809999999999994e-06 2024-05-15T23:26:58.211 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.485251+0000", 2024-05-15T23:26:58.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.400660619 2024-05-15T23:26:58.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.485368+0000", 2024-05-15T23:26:58.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000117649 2024-05-15T23:26:58.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.212 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.212 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.212 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.485541+0000", 2024-05-15T23:26:58.212 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000172363 2024-05-15T23:26:58.212 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.213 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.213 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.213 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.494018+0000", 2024-05-15T23:26:58.213 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0084768280000000005 2024-05-15T23:26:58.213 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.213 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.213 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.213 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.510815+0000", 2024-05-15T23:26:58.213 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.016797757999999999 2024-05-15T23:26:58.213 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.213 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.213 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.213 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.510874+0000", 2024-05-15T23:26:58.213 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8143000000000002e-05 2024-05-15T23:26:58.213 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.214 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.214 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.214 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.516900+0000", 2024-05-15T23:26:58.214 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0060266440000000003 2024-05-15T23:26:58.214 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.214 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.214 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.214 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.214 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.214 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:137255 5.9 5:960ee34f:::benchmark_data_smithi090_116470_object137254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e408)", 2024-05-15T23:26:58.214 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:09.080095+0000", 2024-05-15T23:26:58.214 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 349.10970760200001, 2024-05-15T23:26:58.214 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.48287862999999998, 2024-05-15T23:26:58.214 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.215 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.215 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.215 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.215 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.215 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.215 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 137255 2024-05-15T23:26:58.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.215 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.215 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080095+0000", 2024-05-15T23:26:58.215 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.215 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.215 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.215 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080095+0000", 2024-05-15T23:26:58.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080095+0000", 2024-05-15T23:26:58.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4799999999999999e-07 2024-05-15T23:26:58.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.216 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080128+0000", 2024-05-15T23:26:58.216 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2293999999999999e-05 2024-05-15T23:26:58.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.216 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.216 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.217 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080132+0000", 2024-05-15T23:26:58.217 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3970000000000004e-06 2024-05-15T23:26:58.217 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.217 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.217 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.217 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080137+0000", 2024-05-15T23:26:58.217 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6210000000000003e-06 2024-05-15T23:26:58.217 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.217 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.217 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.217 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.084625+0000", 2024-05-15T23:26:58.217 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0044880969999999999 2024-05-15T23:26:58.217 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.217 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.217 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:26:58.218 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.084633+0000", 2024-05-15T23:26:58.218 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0469999999999994e-06 2024-05-15T23:26:58.218 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.218 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.218 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.218 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.494055+0000", 2024-05-15T23:26:58.218 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.40942179099999998 2024-05-15T23:26:58.218 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.218 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.218 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.218 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.494183+0000", 2024-05-15T23:26:58.218 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012852800000000001 2024-05-15T23:26:58.218 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.218 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.218 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.219 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.494310+0000", 2024-05-15T23:26:58.219 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000126632 2024-05-15T23:26:58.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.219 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.219 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.219 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.510761+0000", 2024-05-15T23:26:58.219 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.016451633 2024-05-15T23:26:58.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.219 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.219 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.219 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.557735+0000", 2024-05-15T23:26:58.219 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046973266 2024-05-15T23:26:58.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.219 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.219 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.557771+0000", 2024-05-15T23:26:58.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6452999999999998e-05 2024-05-15T23:26:58.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.220 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.220 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.562974+0000", 2024-05-15T23:26:58.220 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0052024230000000003 2024-05-15T23:26:58.220 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.220 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.220 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.220 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.220 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.220 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:137264 5.9 5:95e92f12:::benchmark_data_smithi090_116470_object137263:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e408)", 2024-05-15T23:26:58.220 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:09.080143+0000", 2024-05-15T23:26:58.220 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 349.10965907399998, 2024-05-15T23:26:58.221 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.49348138699999999, 2024-05-15T23:26:58.221 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.221 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.221 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.221 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.221 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.221 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.221 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 137264 2024-05-15T23:26:58.221 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.221 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.221 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.221 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.221 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080143+0000", 2024-05-15T23:26:58.221 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.221 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080143+0000", 2024-05-15T23:26:58.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080144+0000", 2024-05-15T23:26:58.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2199999999999999e-07 2024-05-15T23:26:58.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.222 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.222 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.222 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080176+0000", 2024-05-15T23:26:58.222 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1742000000000002e-05 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080185+0000", 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.268e-06 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080199+0000", 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3611000000000001e-05 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.084658+0000", 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0044592720000000002 2024-05-15T23:26:58.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.224 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.224 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:26:58.224 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.084662+0000", 2024-05-15T23:26:58.224 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7029999999999997e-06 2024-05-15T23:26:58.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.224 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.224 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.224 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.497487+0000", 2024-05-15T23:26:58.224 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.41282451999999997 2024-05-15T23:26:58.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.224 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.224 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.224 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.497644+0000", 2024-05-15T23:26:58.224 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015707199999999999 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.497821+0000", 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017702899999999999 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.524978+0000", 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.027156823 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.568271+0000", 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043293167 2024-05-15T23:26:58.225 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.226 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.226 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.226 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.568310+0000", 2024-05-15T23:26:58.226 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8791999999999997e-05 2024-05-15T23:26:58.226 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.226 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.226 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.226 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.573625+0000", 2024-05-15T23:26:58.226 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005314966 2024-05-15T23:26:58.226 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.226 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.226 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.226 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.226 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.226 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:137280 5.9 5:95f3fdc2:::benchmark_data_smithi090_116470_object137279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e408)", 2024-05-15T23:26:58.227 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:09.080203+0000", 2024-05-15T23:26:58.227 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 349.10959916399997, 2024-05-15T23:26:58.227 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.49823662299999999, 2024-05-15T23:26:58.227 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.227 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.227 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.227 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.227 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.227 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.227 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 137280 2024-05-15T23:26:58.227 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.227 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.227 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.227 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080203+0000", 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080203+0000", 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080204+0000", 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4200000000000001e-07 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.228 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080247+0000", 2024-05-15T23:26:58.229 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3417000000000001e-05 2024-05-15T23:26:58.229 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.229 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.229 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.229 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080252+0000", 2024-05-15T23:26:58.229 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.459e-06 2024-05-15T23:26:58.229 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.229 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.229 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.229 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080256+0000", 2024-05-15T23:26:58.229 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3019999999999996e-06 2024-05-15T23:26:58.229 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.229 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.229 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.229 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.084692+0000", 2024-05-15T23:26:58.230 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0044355360000000003 2024-05-15T23:26:58.230 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.230 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.230 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:26:58.230 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.084703+0000", 2024-05-15T23:26:58.230 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1603000000000001e-05 2024-05-15T23:26:58.230 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.230 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.230 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.230 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.501877+0000", 2024-05-15T23:26:58.230 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.41717406600000001 2024-05-15T23:26:58.230 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.230 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.230 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.230 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.502002+0000", 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012518800000000001 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.502102+0000", 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9872999999999997e-05 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.524987+0000", 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022884366 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.573679+0000", 2024-05-15T23:26:58.231 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.048692105999999999 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.573725+0000", 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5766000000000002e-05 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.578440+0000", 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0047154989999999997 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.232 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:137290 5.9 5:950a3e61:::benchmark_data_smithi090_116470_object137289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e408)", 2024-05-15T23:26:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:09.080261+0000", 2024-05-15T23:26:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 349.10954123599998, 2024-05-15T23:26:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.50411139900000002, 2024-05-15T23:26:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 137290 2024-05-15T23:26:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080261+0000", 2024-05-15T23:26:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080261+0000", 2024-05-15T23:26:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080262+0000", 2024-05-15T23:26:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9900000000000001e-07 2024-05-15T23:26:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080296+0000", 2024-05-15T23:26:58.235 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3927999999999998e-05 2024-05-15T23:26:58.235 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.235 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.235 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.235 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080300+0000", 2024-05-15T23:26:58.235 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1489999999999996e-06 2024-05-15T23:26:58.235 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.235 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.235 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.235 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.080303+0000", 2024-05-15T23:26:58.235 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6660000000000001e-06 2024-05-15T23:26:58.235 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.235 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.235 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.235 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.084741+0000", 2024-05-15T23:26:58.236 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0044377660000000001 2024-05-15T23:26:58.236 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.236 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.236 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for peered", 2024-05-15T23:26:58.236 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.084745+0000", 2024-05-15T23:26:58.236 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6820000000000001e-06 2024-05-15T23:26:58.236 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.236 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.236 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.236 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.516957+0000", 2024-05-15T23:26:58.236 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.43221231399999999 2024-05-15T23:26:58.236 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.236 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.236 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.237 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.517135+0000", 2024-05-15T23:26:58.237 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00017776599999999999 2024-05-15T23:26:58.237 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.237 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.237 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.237 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.517305+0000", 2024-05-15T23:26:58.237 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016992799999999999 2024-05-15T23:26:58.237 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.237 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.237 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.237 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.533486+0000", 2024-05-15T23:26:58.237 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.016180949 2024-05-15T23:26:58.237 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.237 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.578517+0000", 2024-05-15T23:26:58.238 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.04503132 2024-05-15T23:26:58.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.238 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.578573+0000", 2024-05-15T23:26:58.238 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5648e-05 2024-05-15T23:26:58.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.238 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.238 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.238 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:09.584373+0000", 2024-05-15T23:26:58.238 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0057998839999999999 2024-05-15T23:26:58.238 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.238 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.239 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.239 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.239 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:169125 5.9 5:921b54a0:::benchmark_data_smithi090_116470_object15565:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-05-15T23:26:58.239 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:51.254840+0000", 2024-05-15T23:26:58.239 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 306.93496265499999, 2024-05-15T23:26:58.239 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.404451589, 2024-05-15T23:26:58.239 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.239 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.239 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.239 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.239 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.239 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.239 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 169125 2024-05-15T23:26:58.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.240 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.254840+0000", 2024-05-15T23:26:58.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.254840+0000", 2024-05-15T23:26:58.240 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.240 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.240 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.240 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.254843+0000", 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1080000000000001e-06 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.254851+0000", 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9899999999999997e-06 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.254863+0000", 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2451e-05 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.254874+0000", 2024-05-15T23:26:58.241 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0237999999999999e-05 2024-05-15T23:26:58.242 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.242 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.242 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.242 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.292327+0000", 2024-05-15T23:26:58.242 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.037453688999999998 2024-05-15T23:26:58.242 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.242 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.242 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:58.242 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.292399+0000", 2024-05-15T23:26:58.242 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1957000000000002e-05 2024-05-15T23:26:58.242 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.242 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.242 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.242 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.611404+0000", 2024-05-15T23:26:58.242 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.31900474200000001 2024-05-15T23:26:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.611503+0000", 2024-05-15T23:26:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8999999999999994e-05 2024-05-15T23:26:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.611641+0000", 2024-05-15T23:26:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013842300000000001 2024-05-15T23:26:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.622768+0000", 2024-05-15T23:26:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011126717 2024-05-15T23:26:58.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.244 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.654162+0000", 2024-05-15T23:26:58.244 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031393388000000001 2024-05-15T23:26:58.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.244 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.654230+0000", 2024-05-15T23:26:58.244 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8167e-05 2024-05-15T23:26:58.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.244 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.244 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.244 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.659291+0000", 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050617190000000001 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:169276 5.9 5:9faaf749:::benchmark_data_smithi090_116470_object15716:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:51.474420+0000", 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 306.71538236200001, 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.400756047, 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.245 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.246 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 169276 2024-05-15T23:26:58.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.246 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.474420+0000", 2024-05-15T23:26:58.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.246 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.246 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.474420+0000", 2024-05-15T23:26:58.246 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.246 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.474422+0000", 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9769999999999999e-06 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.474429+0000", 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5339999999999996e-06 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.474439+0000", 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0353000000000001e-05 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.247 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.248 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.474448+0000", 2024-05-15T23:26:58.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.867e-06 2024-05-15T23:26:58.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.248 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.501887+0000", 2024-05-15T23:26:58.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.027439568000000001 2024-05-15T23:26:58.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.248 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:58.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.501929+0000", 2024-05-15T23:26:58.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1668999999999997e-05 2024-05-15T23:26:58.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.849980+0000", 2024-05-15T23:26:58.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.34805133300000002 2024-05-15T23:26:58.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.249 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.850072+0000", 2024-05-15T23:26:58.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.1549e-05 2024-05-15T23:26:58.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.850173+0000", 2024-05-15T23:26:58.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00010064 2024-05-15T23:26:58.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.869377+0000", 2024-05-15T23:26:58.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019204705999999998 2024-05-15T23:26:58.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.869971+0000", 2024-05-15T23:26:58.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00059348300000000001 2024-05-15T23:26:58.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.870018+0000", 2024-05-15T23:26:58.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7580000000000002e-05 2024-05-15T23:26:58.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.875176+0000", 2024-05-15T23:26:58.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.005157788 2024-05-15T23:26:58.251 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.251 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.251 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.251 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:169304 5.9 5:993b3720:::benchmark_data_smithi090_116470_object15744:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-05-15T23:26:58.252 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:51.503432+0000", 2024-05-15T23:26:58.252 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 306.68637056300003, 2024-05-15T23:26:58.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.45313241300000001, 2024-05-15T23:26:58.252 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.252 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.252 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.252 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.252 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.252 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.252 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 169304 2024-05-15T23:26:58.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.252 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.503432+0000", 2024-05-15T23:26:58.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.503432+0000", 2024-05-15T23:26:58.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.503434+0000", 2024-05-15T23:26:58.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.43e-06 2024-05-15T23:26:58.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.503441+0000", 2024-05-15T23:26:58.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6769999999999998e-06 2024-05-15T23:26:58.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.503453+0000", 2024-05-15T23:26:58.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1605e-05 2024-05-15T23:26:58.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.503461+0000", 2024-05-15T23:26:58.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0139999999999998e-06 2024-05-15T23:26:58.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.528370+0000", 2024-05-15T23:26:58.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024909153999999999 2024-05-15T23:26:58.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:58.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.528424+0000", 2024-05-15T23:26:58.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4286999999999997e-05 2024-05-15T23:26:58.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.255 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.898127+0000", 2024-05-15T23:26:58.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.36970264400000002 2024-05-15T23:26:58.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.255 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.898216+0000", 2024-05-15T23:26:58.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8975999999999996e-05 2024-05-15T23:26:58.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.898335+0000", 2024-05-15T23:26:58.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000119548 2024-05-15T23:26:58.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.931445+0000", 2024-05-15T23:26:58.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.033109185999999999 2024-05-15T23:26:58.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.951302+0000", 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019857297999999999 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.951357+0000", 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4839000000000001e-05 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.956564+0000", 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0052077549999999997 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.258 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPushReply(5.9 419/407 [PushReplyOp(5:993b3720:::benchmark_data_smithi090_116470_object15744:head)])", 2024-05-15T23:26:58.258 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:51.532180+0000", 2024-05-15T23:26:58.258 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 306.65762279299997, 2024-05-15T23:26:58.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.36589978400000001, 2024-05-15T23:26:58.258 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.258 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.258 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2024-05-15T23:26:58.258 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.532180+0000", 2024-05-15T23:26:58.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.532180+0000", 2024-05-15T23:26:58.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.532183+0000", 2024-05-15T23:26:58.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4209999999999999e-06 2024-05-15T23:26:58.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.532195+0000", 2024-05-15T23:26:58.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1429e-05 2024-05-15T23:26:58.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.532220+0000", 2024-05-15T23:26:58.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5245999999999999e-05 2024-05-15T23:26:58.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.532232+0000", 2024-05-15T23:26:58.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2479e-05 2024-05-15T23:26:58.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.887803+0000", 2024-05-15T23:26:58.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.35557028200000002 2024-05-15T23:26:58.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.898080+0000", 2024-05-15T23:26:58.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010276927 2024-05-15T23:26:58.261 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.261 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.261 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.261 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:169358 5.9 5:9f1c8b4b:::benchmark_data_smithi090_116470_object15798:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-05-15T23:26:58.261 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:51.583397+0000", 2024-05-15T23:26:58.261 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 306.606405091, 2024-05-15T23:26:58.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.42508884000000002, 2024-05-15T23:26:58.261 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.261 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.262 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.262 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.262 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 169358 2024-05-15T23:26:58.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.262 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.583397+0000", 2024-05-15T23:26:58.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.583397+0000", 2024-05-15T23:26:58.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.583400+0000", 2024-05-15T23:26:58.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3080000000000002e-06 2024-05-15T23:26:58.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.583407+0000", 2024-05-15T23:26:58.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7850000000000003e-06 2024-05-15T23:26:58.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.583420+0000", 2024-05-15T23:26:58.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3495999999999999e-05 2024-05-15T23:26:58.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.583430+0000", 2024-05-15T23:26:58.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0037000000000001e-05 2024-05-15T23:26:58.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.600578+0000", 2024-05-15T23:26:58.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.017148010000000002 2024-05-15T23:26:58.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:58.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.600641+0000", 2024-05-15T23:26:58.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3145999999999995e-05 2024-05-15T23:26:58.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.982480+0000", 2024-05-15T23:26:58.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.38183840299999999 2024-05-15T23:26:58.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.982582+0000", 2024-05-15T23:26:58.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000101956 2024-05-15T23:26:58.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.982717+0000", 2024-05-15T23:26:58.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013549999999999999 2024-05-15T23:26:58.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.998421+0000", 2024-05-15T23:26:58.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015704083000000001 2024-05-15T23:26:58.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:52.003391+0000", 2024-05-15T23:26:58.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049695379999999999 2024-05-15T23:26:58.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:52.003444+0000", 2024-05-15T23:26:58.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3723e-05 2024-05-15T23:26:58.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:52.008486+0000", 2024-05-15T23:26:58.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0050418549999999996 2024-05-15T23:26:58.267 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.268 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.268 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.268 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPushReply(5.9 419/407 [PushReplyOp(5:9f1c8b4b:::benchmark_data_smithi090_116470_object15798:head)])", 2024-05-15T23:26:58.268 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:51.587333+0000", 2024-05-15T23:26:58.268 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 306.60246980800002, 2024-05-15T23:26:58.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.39510278999999998, 2024-05-15T23:26:58.268 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.268 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.268 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2024-05-15T23:26:58.268 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.587333+0000", 2024-05-15T23:26:58.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.587333+0000", 2024-05-15T23:26:58.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.587335+0000", 2024-05-15T23:26:58.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.762e-06 2024-05-15T23:26:58.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.587341+0000", 2024-05-15T23:26:58.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.878e-06 2024-05-15T23:26:58.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.587354+0000", 2024-05-15T23:26:58.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2662e-05 2024-05-15T23:26:58.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.587364+0000", 2024-05-15T23:26:58.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0135999999999999e-05 2024-05-15T23:26:58.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.972757+0000", 2024-05-15T23:26:58.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.385392658 2024-05-15T23:26:58.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.982436+0000", 2024-05-15T23:26:58.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0096786939999999998 2024-05-15T23:26:58.271 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.271 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.271 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.272 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:169393 5.9 5:98ccf96c:::benchmark_data_smithi090_116470_object15833:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-05-15T23:26:58.272 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:51.630635+0000", 2024-05-15T23:26:58.272 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 306.559167694, 2024-05-15T23:26:58.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.39571926099999999, 2024-05-15T23:26:58.272 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.272 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.272 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.272 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.272 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.272 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.272 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 169393 2024-05-15T23:26:58.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.272 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.630635+0000", 2024-05-15T23:26:58.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.630635+0000", 2024-05-15T23:26:58.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.630637+0000", 2024-05-15T23:26:58.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4389999999999999e-06 2024-05-15T23:26:58.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.630646+0000", 2024-05-15T23:26:58.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2330000000000006e-06 2024-05-15T23:26:58.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.630660+0000", 2024-05-15T23:26:58.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4789e-05 2024-05-15T23:26:58.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.630669+0000", 2024-05-15T23:26:58.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8619999999999992e-06 2024-05-15T23:26:58.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.695889+0000", 2024-05-15T23:26:58.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065219422999999999 2024-05-15T23:26:58.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:58.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.695939+0000", 2024-05-15T23:26:58.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0574000000000001e-05 2024-05-15T23:26:58.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.998449+0000", 2024-05-15T23:26:58.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.3025098 2024-05-15T23:26:58.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.998528+0000", 2024-05-15T23:26:58.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.928e-05 2024-05-15T23:26:58.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.998643+0000", 2024-05-15T23:26:58.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000115005 2024-05-15T23:26:58.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.276 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:52.021153+0000", 2024-05-15T23:26:58.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022509708999999999 2024-05-15T23:26:58.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:52.021170+0000", 2024-05-15T23:26:58.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.7169000000000001e-05 2024-05-15T23:26:58.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:52.021234+0000", 2024-05-15T23:26:58.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4074999999999999e-05 2024-05-15T23:26:58.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:52.026354+0000", 2024-05-15T23:26:58.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0051199030000000003 2024-05-15T23:26:58.277 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.278 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.278 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.278 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "MOSDPGPushReply(5.9 419/407 [PushReplyOp(5:98ccf96c:::benchmark_data_smithi090_116470_object15833:head)])", 2024-05-15T23:26:58.278 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:51.643157+0000", 2024-05-15T23:26:58.278 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 306.546645367, 2024-05-15T23:26:58.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.35526593299999998, 2024-05-15T23:26:58.278 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.278 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.278 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "reached pg", 2024-05-15T23:26:58.278 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.643157+0000", 2024-05-15T23:26:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.643157+0000", 2024-05-15T23:26:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.643160+0000", 2024-05-15T23:26:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.43e-06 2024-05-15T23:26:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.643167+0000", 2024-05-15T23:26:58.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8699999999999992e-06 2024-05-15T23:26:58.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.643181+0000", 2024-05-15T23:26:58.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3539e-05 2024-05-15T23:26:58.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.643189+0000", 2024-05-15T23:26:58.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1089999999999998e-06 2024-05-15T23:26:58.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.988258+0000", 2024-05-15T23:26:58.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.345068924 2024-05-15T23:26:58.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:51.998423+0000", 2024-05-15T23:26:58.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.010165061 2024-05-15T23:26:58.281 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.281 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.281 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.281 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:172362 5.9 5:9507c5af:::benchmark_data_smithi090_116470_object18802:head [delete] snapc 0=[] ondisk+write+known_if_redirected e422)", 2024-05-15T23:26:58.281 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:55.732967+0000", 2024-05-15T23:26:58.281 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 302.45683576800002, 2024-05-15T23:26:58.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.37309847000000002, 2024-05-15T23:26:58.282 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.282 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.282 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.282 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.282 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.282 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.282 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 172362 2024-05-15T23:26:58.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.282 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.282 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.282 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.732967+0000", 2024-05-15T23:26:58.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.282 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.282 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.732967+0000", 2024-05-15T23:26:58.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.732970+0000", 2024-05-15T23:26:58.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.236e-06 2024-05-15T23:26:58.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.732981+0000", 2024-05-15T23:26:58.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0662000000000001e-05 2024-05-15T23:26:58.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.733000+0000", 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9358999999999999e-05 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.733012+0000", 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1837e-05 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.855306+0000", 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.122293864 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:58.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.855360+0000", 2024-05-15T23:26:58.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4112e-05 2024-05-15T23:26:58.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:56.014471+0000", 2024-05-15T23:26:58.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15911151200000001 2024-05-15T23:26:58.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:56.014565+0000", 2024-05-15T23:26:58.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3888999999999995e-05 2024-05-15T23:26:58.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:56.014701+0000", 2024-05-15T23:26:58.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013536399999999999 2024-05-15T23:26:58.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:56.046588+0000", 2024-05-15T23:26:58.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031887226999999997 2024-05-15T23:26:58.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:56.101592+0000", 2024-05-15T23:26:58.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055004482 2024-05-15T23:26:58.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:56.101664+0000", 2024-05-15T23:26:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1606999999999994e-05 2024-05-15T23:26:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:56.106065+0000", 2024-05-15T23:26:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0044013189999999999 2024-05-15T23:26:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:172466 5.9 5:99f58345:::benchmark_data_smithi090_116470_object18906:head [delete] snapc 0=[] ondisk+write+known_if_redirected e422)", 2024-05-15T23:26:58.288 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:55.878429+0000", 2024-05-15T23:26:58.288 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 302.311373683, 2024-05-15T23:26:58.288 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.370006047, 2024-05-15T23:26:58.288 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.288 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.288 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.288 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.288 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.288 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.288 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 172466 2024-05-15T23:26:58.288 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.288 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.288 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.288 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.288 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.878429+0000", 2024-05-15T23:26:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.878429+0000", 2024-05-15T23:26:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.878431+0000", 2024-05-15T23:26:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4370000000000001e-06 2024-05-15T23:26:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.878438+0000", 2024-05-15T23:26:58.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5819999999999998e-06 2024-05-15T23:26:58.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.878452+0000", 2024-05-15T23:26:58.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3732e-05 2024-05-15T23:26:58.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.878460+0000", 2024-05-15T23:26:58.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.7819999999999999e-06 2024-05-15T23:26:58.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.905345+0000", 2024-05-15T23:26:58.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0268848 2024-05-15T23:26:58.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:58.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:55.905416+0000", 2024-05-15T23:26:58.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0368000000000001e-05 2024-05-15T23:26:58.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:56.143121+0000", 2024-05-15T23:26:58.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.23770575799999999 2024-05-15T23:26:58.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:56.143222+0000", 2024-05-15T23:26:58.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00010110100000000001 2024-05-15T23:26:58.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:56.143346+0000", 2024-05-15T23:26:58.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012316000000000001 2024-05-15T23:26:58.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:56.174286+0000", 2024-05-15T23:26:58.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.030940156 2024-05-15T23:26:58.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:56.243413+0000", 2024-05-15T23:26:58.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.069127599999999997 2024-05-15T23:26:58.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.294 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:56.243477+0000", 2024-05-15T23:26:58.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3769000000000006e-05 2024-05-15T23:26:58.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.294 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:56.248435+0000", 2024-05-15T23:26:58.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049578019999999999 2024-05-15T23:26:58.295 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.295 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.295 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.295 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:174299 5.9 5:9e179cc4:::benchmark_data_smithi090_116470_object20739:head [delete] snapc 0=[] ondisk+write+known_if_redirected e423)", 2024-05-15T23:26:58.295 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:21:58.618142+0000", 2024-05-15T23:26:58.295 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 299.57166032399999, 2024-05-15T23:26:58.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.44698792100000001, 2024-05-15T23:26:58.295 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.295 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.295 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.295 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 174299 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:58.618142+0000", 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:58.618142+0000", 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:58.618145+0000", 2024-05-15T23:26:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2699999999999999e-06 2024-05-15T23:26:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:58.618150+0000", 2024-05-15T23:26:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9440000000000002e-06 2024-05-15T23:26:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:58.618165+0000", 2024-05-15T23:26:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4178e-05 2024-05-15T23:26:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:58.618172+0000", 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0090000000000001e-06 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:58.773160+0000", 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.15498873799999999 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:58.773240+0000", 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.9249e-05 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:58.933713+0000", 2024-05-15T23:26:58.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.16047315300000001 2024-05-15T23:26:58.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:58.933828+0000", 2024-05-15T23:26:58.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011561300000000001 2024-05-15T23:26:58.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.299 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:58.933992+0000", 2024-05-15T23:26:58.299 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016359600000000001 2024-05-15T23:26:58.299 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.299 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.299 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:58.952382+0000", 2024-05-15T23:26:58.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.018390185999999999 2024-05-15T23:26:58.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:59.060379+0000", 2024-05-15T23:26:58.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.107996861 2024-05-15T23:26:58.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.300 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:59.060418+0000", 2024-05-15T23:26:58.300 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8890999999999997e-05 2024-05-15T23:26:58.300 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.300 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.300 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:21:59.065130+0000", 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0047122329999999997 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:175665 5.9 5:9b33993c:::benchmark_data_smithi090_116470_object22105:head [delete] snapc 0=[] ondisk+write+known_if_redirected e423)", 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:22:00.535115+0000", 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 297.65468777900003, 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.41693332700000002, 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.301 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.302 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.302 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 175665 2024-05-15T23:26:58.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.302 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:00.535115+0000", 2024-05-15T23:26:58.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:00.535115+0000", 2024-05-15T23:26:58.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:00.535116+0000", 2024-05-15T23:26:58.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2900000000000002e-07 2024-05-15T23:26:58.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:00.535119+0000", 2024-05-15T23:26:58.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.478e-06 2024-05-15T23:26:58.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.303 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.303 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:00.535129+0000", 2024-05-15T23:26:58.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4220000000000006e-06 2024-05-15T23:26:58.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:00.535134+0000", 2024-05-15T23:26:58.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7899999999999996e-06 2024-05-15T23:26:58.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:00.628111+0000", 2024-05-15T23:26:58.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092977044999999994 2024-05-15T23:26:58.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:58.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:00.628189+0000", 2024-05-15T23:26:58.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7595999999999994e-05 2024-05-15T23:26:58.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:00.840922+0000", 2024-05-15T23:26:58.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.21273294200000001 2024-05-15T23:26:58.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:00.840991+0000", 2024-05-15T23:26:58.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8886000000000004e-05 2024-05-15T23:26:58.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:00.841067+0000", 2024-05-15T23:26:58.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5889999999999993e-05 2024-05-15T23:26:58.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:00.869372+0000", 2024-05-15T23:26:58.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028304815000000001 2024-05-15T23:26:58.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:00.946814+0000", 2024-05-15T23:26:58.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.077442192000000007 2024-05-15T23:26:58.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:00.946860+0000", 2024-05-15T23:26:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6533000000000003e-05 2024-05-15T23:26:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:00.952048+0000", 2024-05-15T23:26:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0051878089999999998 2024-05-15T23:26:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.25840.0:189291 5.9 5:9591a413:::benchmark_data_smithi090_116470_object35731:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-05-15T23:26:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:22:20.256541+0000", 2024-05-15T23:26:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 277.93326115399998, 2024-05-15T23:26:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.37637016600000001, 2024-05-15T23:26:58.308 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:26:58.308 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:26:58.308 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:26:58.308 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:26:58.308 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.25840", 2024-05-15T23:26:58.308 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:26:58.308 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 189291 2024-05-15T23:26:58.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.308 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:26:58.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:26:58.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:20.256541+0000", 2024-05-15T23:26:58.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:26:58.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:20.256541+0000", 2024-05-15T23:26:58.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:26:58.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:26:58.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:20.256544+0000", 2024-05-15T23:26:58.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.429e-06 2024-05-15T23:26:58.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:26:58.309 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:20.256550+0000", 2024-05-15T23:26:58.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5950000000000004e-06 2024-05-15T23:26:58.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:26:58.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:20.256564+0000", 2024-05-15T23:26:58.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3910000000000001e-05 2024-05-15T23:26:58.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:26:58.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:20.256573+0000", 2024-05-15T23:26:58.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.4230000000000005e-06 2024-05-15T23:26:58.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:20.265244+0000", 2024-05-15T23:26:58.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0086713150000000006 2024-05-15T23:26:58.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for degraded object", 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:20.265316+0000", 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2333e-05 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:20.565926+0000", 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.30060984499999999 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:20.565995+0000", 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.868e-05 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 11", 2024-05-15T23:26:58.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:20.566091+0000", 2024-05-15T23:26:58.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6037000000000002e-05 2024-05-15T23:26:58.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:26:58.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:20.588024+0000", 2024-05-15T23:26:58.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.021933517 2024-05-15T23:26:58.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:26:58.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:20.628005+0000", 2024-05-15T23:26:58.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.039980538000000003 2024-05-15T23:26:58.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:26:58.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:20.628075+0000", 2024-05-15T23:26:58.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0424000000000002e-05 2024-05-15T23:26:58.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:26:58.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:26:58.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:26:58.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:22:20.632912+0000", 2024-05-15T23:26:58.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0048361200000000002 2024-05-15T23:26:58.313 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.313 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.313 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.313 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:26:58.313 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:26:58.313 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:26:58.430 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:26:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:58 smithi088 ceph-mon[44699]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-15T23:26:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:58 smithi088 ceph-mon[44699]: from='osd.8 v1:172.21.15.90:6800/703598286' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:26:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:58 smithi088 ceph-mon[44699]: osdmap e519: 12 total, 11 up, 8 in 2024-05-15T23:26:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:58 smithi088 ceph-mon[44699]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:26:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:58 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:26:58.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:58 smithi088 ceph-mon[44699]: 6.8 scrub ok 2024-05-15T23:26:58.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:58 smithi090 ceph-mon[44434]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-15T23:26:58.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:58 smithi090 ceph-mon[44434]: from='osd.8 v1:172.21.15.90:6800/703598286' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:26:58.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:58 smithi090 ceph-mon[44434]: osdmap e519: 12 total, 11 up, 8 in 2024-05-15T23:26:58.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:58 smithi090 ceph-mon[44434]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:26:58.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:58 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:26:58.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:58 smithi090 ceph-mon[44434]: 6.8 scrub ok 2024-05-15T23:26:58.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:58 smithi084 ceph-mon[39784]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-15T23:26:58.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:58 smithi084 ceph-mon[39784]: from='osd.8 v1:172.21.15.90:6800/703598286' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:26:58.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:58 smithi084 ceph-mon[39784]: osdmap e519: 12 total, 11 up, 8 in 2024-05-15T23:26:58.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:58 smithi084 ceph-mon[39784]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-05-15T23:26:58.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:58 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:26:58.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:58 smithi084 ceph-mon[39784]: 6.8 scrub ok 2024-05-15T23:26:59.091 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:26:59.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:59.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:26:59.202 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:26:59.205 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:26:59.205 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:26:59.205 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-05-15T23:26:59.205 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:26:59.242 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:59.306 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:26:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:59.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:26:59.415 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:26:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:26:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:59 smithi090 ceph-mon[44434]: osd.8 v1:172.21.15.90:6800/703598286 boot 2024-05-15T23:26:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:59 smithi090 ceph-mon[44434]: osdmap e520: 12 total, 12 up, 8 in 2024-05-15T23:26:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:59 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:26:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:26:59 smithi090 ceph-mon[44434]: pgmap v1390: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 997 op/s 2024-05-15T23:26:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:59 smithi088 ceph-mon[44699]: osd.8 v1:172.21.15.90:6800/703598286 boot 2024-05-15T23:26:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:59 smithi088 ceph-mon[44699]: osdmap e520: 12 total, 12 up, 8 in 2024-05-15T23:26:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:59 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:26:59.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:26:59 smithi088 ceph-mon[44699]: pgmap v1390: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 997 op/s 2024-05-15T23:26:59.594 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:26:59.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:26:59.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:26:59.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:59 smithi084 ceph-mon[39784]: osd.8 v1:172.21.15.90:6800/703598286 boot 2024-05-15T23:26:59.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:59 smithi084 ceph-mon[39784]: osdmap e520: 12 total, 12 up, 8 in 2024-05-15T23:26:59.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:59 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-15T23:26:59.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:26:59 smithi084 ceph-mon[39784]: pgmap v1390: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 997 op/s 2024-05-15T23:26:59.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T23:26:59.703 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:26:59.704 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:26:59.749 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:00.002 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:27:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:00.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:27:00.109 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:00.141 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:00.473 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:27:00.473 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-05-15T23:27:00.474 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-05-15T23:27:00.533 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:27:00.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:00.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:00 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:27:00.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:00 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:27:00.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:00 smithi084 ceph-mon[39784]: osdmap e521: 12 total, 12 up, 8 in 2024-05-15T23:27:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:00 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:27:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:00 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:27:00.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:00 smithi090 ceph-mon[44434]: osdmap e521: 12 total, 12 up, 8 in 2024-05-15T23:27:00.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:00.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:00 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:27:00.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:00 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:27:00.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:00 smithi088 ceph-mon[44699]: osdmap e521: 12 total, 12 up, 8 in 2024-05-15T23:27:01.198 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:01.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:01.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:01.302 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:01.342 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:01.460 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:01 smithi090 ceph-mon[44434]: 6.f scrub starts 2024-05-15T23:27:01.460 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:01 smithi090 ceph-mon[44434]: pgmap v1392: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-05-15T23:27:01.481 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:01.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:01.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:01.518 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:01.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:27:01.639 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:01.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:01 smithi088 ceph-mon[44699]: 6.f scrub starts 2024-05-15T23:27:01.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:01 smithi088 ceph-mon[44699]: pgmap v1392: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-05-15T23:27:01.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:01 smithi084 ceph-mon[39784]: 6.f scrub starts 2024-05-15T23:27:01.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:01 smithi084 ceph-mon[39784]: pgmap v1392: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-05-15T23:27:02.003 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:27:02.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:02.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:27:02.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:27:02.139 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:02.304 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:27:02.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:02.310 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:27:02.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:27:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:02.477 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:27:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:02.481 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:27:02.483 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:27:02.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:02 smithi090 ceph-mon[44434]: 6.f scrub ok 2024-05-15T23:27:02.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3527222310' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:27:02.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:02 smithi090 ceph-mon[44434]: from='client.26491 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:27:02.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:02.624 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:02.649 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:27:02.649 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:27:02.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.649 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:164735 6.9 6:91c5183d:::benchmark_data_smithi090_131278_object29541:head [delete] snapc 0=[] ondisk+write+known_if_redirected e522)", 2024-05-15T23:27:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:27:02.625801+0000", 2024-05-15T23:27:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.022287703999999998, 2024-05-15T23:27:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022311284000000001, 2024-05-15T23:27:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:27:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:27:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:27:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:27:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:27:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:27:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 164735 2024-05-15T23:27:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:27:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:27:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.625801+0000", 2024-05-15T23:27:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:27:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:27:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.625801+0000", 2024-05-15T23:27:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:27:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:27:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.625803+0000", 2024-05-15T23:27:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.423e-06 2024-05-15T23:27:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:27:02.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.625807+0000", 2024-05-15T23:27:02.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9119999999999997e-06 2024-05-15T23:27:02.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:27:02.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.625818+0000", 2024-05-15T23:27:02.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0325e-05 2024-05-15T23:27:02.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:27:02.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.625825+0000", 2024-05-15T23:27:02.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6729999999999992e-06 2024-05-15T23:27:02.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:27:02.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.636583+0000", 2024-05-15T23:27:02.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010757182000000001 2024-05-15T23:27:02.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:27:02.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.636688+0000", 2024-05-15T23:27:02.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000105593 2024-05-15T23:27:02.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:27:02.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.636776+0000", 2024-05-15T23:27:02.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7974999999999997e-05 2024-05-15T23:27:02.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:27:02.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.642502+0000", 2024-05-15T23:27:02.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0057257109999999996 2024-05-15T23:27:02.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:27:02.654 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:27:02.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:27:02.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.654 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:164747 6.1 6:8fb7037a:::benchmark_data_smithi090_131278_object29553:head [delete] snapc 0=[] ondisk+write+known_if_redirected e522)", 2024-05-15T23:27:02.654 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:27:02.636436+0000", 2024-05-15T23:27:02.654 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.011652582999999999, 2024-05-15T23:27:02.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011730756, 2024-05-15T23:27:02.655 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:27:02.655 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:27:02.655 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:27:02.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:27:02.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:27:02.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:27:02.655 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 164747 2024-05-15T23:27:02.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.655 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:27:02.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:27:02.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.636436+0000", 2024-05-15T23:27:02.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:27:02.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:27:02.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.636436+0000", 2024-05-15T23:27:02.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:27:02.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:27:02.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.636439+0000", 2024-05-15T23:27:02.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0539999999999999e-06 2024-05-15T23:27:02.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:27:02.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.636447+0000", 2024-05-15T23:27:02.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2269999999999993e-06 2024-05-15T23:27:02.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:27:02.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.636463+0000", 2024-05-15T23:27:02.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5688e-05 2024-05-15T23:27:02.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:27:02.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.636472+0000", 2024-05-15T23:27:02.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1719999999999996e-06 2024-05-15T23:27:02.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:27:02.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.636593+0000", 2024-05-15T23:27:02.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120308 2024-05-15T23:27:02.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:27:02.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.636685+0000", 2024-05-15T23:27:02.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2e-05 2024-05-15T23:27:02.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:27:02.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.636786+0000", 2024-05-15T23:27:02.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010107000000000001 2024-05-15T23:27:02.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:27:02.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.642491+0000", 2024-05-15T23:27:02.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0057048480000000002 2024-05-15T23:27:02.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:27:02.659 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:27:02.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:27:02.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.659 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:164755 6.9 6:9e1f1d67:::benchmark_data_smithi090_131278_object29561:head [delete] snapc 0=[] ondisk+write+known_if_redirected e522)", 2024-05-15T23:27:02.659 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:27:02.642877+0000", 2024-05-15T23:27:02.659 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0052118119999999997, 2024-05-15T23:27:02.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053566559999999996, 2024-05-15T23:27:02.659 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:27:02.659 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:27:02.660 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:27:02.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:27:02.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:27:02.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:27:02.660 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 164755 2024-05-15T23:27:02.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.660 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:27:02.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:27:02.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.642877+0000", 2024-05-15T23:27:02.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:27:02.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:27:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.642877+0000", 2024-05-15T23:27:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:27:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:27:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.642879+0000", 2024-05-15T23:27:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9350000000000001e-06 2024-05-15T23:27:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:27:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.642887+0000", 2024-05-15T23:27:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2199999999999992e-06 2024-05-15T23:27:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:27:02.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.642899+0000", 2024-05-15T23:27:02.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1503e-05 2024-05-15T23:27:02.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:27:02.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.642917+0000", 2024-05-15T23:27:02.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8340000000000001e-05 2024-05-15T23:27:02.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:27:02.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.642971+0000", 2024-05-15T23:27:02.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4234000000000003e-05 2024-05-15T23:27:02.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:27:02.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.643090+0000", 2024-05-15T23:27:02.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118986 2024-05-15T23:27:02.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:27:02.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.643226+0000", 2024-05-15T23:27:02.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136002 2024-05-15T23:27:02.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:27:02.663 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:27:02.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:27:02.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.663 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.26326.0:164756 6.5 e522/477 6:a880dac6:::benchmark_data_smithi090_131278_object29562:head v 522'10161, mlcod=522'10159)", 2024-05-15T23:27:02.663 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:27:02.643326+0000", 2024-05-15T23:27:02.664 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0047631569999999996, 2024-05-15T23:27:02.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049515840000000002, 2024-05-15T23:27:02.664 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:27:02.664 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:27:02.664 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:27:02.664 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:27:02.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:27:02.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.643326+0000", 2024-05-15T23:27:02.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:27:02.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:27:02.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.643326+0000", 2024-05-15T23:27:02.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:27:02.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:27:02.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.643327+0000", 2024-05-15T23:27:02.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.22e-06 2024-05-15T23:27:02.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:27:02.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.643334+0000", 2024-05-15T23:27:02.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.447e-06 2024-05-15T23:27:02.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:27:02.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.643343+0000", 2024-05-15T23:27:02.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4710000000000007e-06 2024-05-15T23:27:02.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:27:02.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.643349+0000", 2024-05-15T23:27:02.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6159999999999998e-06 2024-05-15T23:27:02.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:27:02.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.643413+0000", 2024-05-15T23:27:02.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3627999999999995e-05 2024-05-15T23:27:02.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:02.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:02.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:27:02.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:02.643433+0000", 2024-05-15T23:27:02.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0262e-05 2024-05-15T23:27:02.667 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:27:02.667 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:27:02.667 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:27:02.667 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:27:02.667 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-05-15T23:27:02.667 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 4 2024-05-15T23:27:02.667 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:27:02.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:02.668 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:02.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:02.731 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:02 smithi084 ceph-mon[39784]: 6.f scrub ok 2024-05-15T23:27:02.731 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3527222310' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:27:02.731 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:02 smithi084 ceph-mon[39784]: from='client.26491 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:27:02.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:27:02.803 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:02.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:02 smithi088 ceph-mon[44699]: 6.f scrub ok 2024-05-15T23:27:02.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3527222310' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:27:02.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:02 smithi088 ceph-mon[44699]: from='client.26491 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:27:03.174 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:27:03.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:27:03.223 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:27:03.248 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:27:03.275 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:27:03.310 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:27:03.340 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:27:03.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:27:03.391 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:27:03.419 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:27:03.446 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:27:03.468 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:27:03.470 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:27:03.470 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:27:03.470 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:27:03.470 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:27:03.470 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:27:03.470 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:27:03.470 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:27:03.470 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:27:03.470 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:27:03.470 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:27:03.470 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:27:03.470 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:27:03.659 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:27:03.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:03.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:27:03.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:27:03.796 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:03 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:27:03.796 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:03 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:27:03.797 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:03 smithi090 ceph-mon[44434]: from='client.26491 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:27:03.797 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:03 smithi090 ceph-mon[44434]: osdmap e522: 12 total, 12 up, 8 in 2024-05-15T23:27:03.797 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:03 smithi090 ceph-mon[44434]: pgmap v1394: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:27:03.797 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1943234618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:27:03.797 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:03.797 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:03.797 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:03.797 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:03.798 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:03.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:03 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:27:03.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:03 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:27:03.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:03 smithi088 ceph-mon[44699]: from='client.26491 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:27:03.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:03 smithi088 ceph-mon[44699]: osdmap e522: 12 total, 12 up, 8 in 2024-05-15T23:27:03.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:03 smithi088 ceph-mon[44699]: pgmap v1394: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:27:03.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1943234618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:27:03.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:03.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:03.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:03.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:03 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:27:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:03 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:27:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:03 smithi084 ceph-mon[39784]: from='client.26491 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:27:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:03 smithi084 ceph-mon[39784]: osdmap e522: 12 total, 12 up, 8 in 2024-05-15T23:27:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:03 smithi084 ceph-mon[39784]: pgmap v1394: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:27:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1943234618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:27:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:03.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:03.934 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:27:03.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:03.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:27:04.045 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:27:04.079 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:04.156 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:27:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:04.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:27:04.260 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:27:04.309 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:04.352 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:27:04.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:04.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:27:04.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:27:04.484 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.485 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.485 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.485 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.485 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.485 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.485 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.485 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.488 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.572 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:27:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:04.582 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:27:04.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T23:27:04.683 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:27:04.726 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:04.751 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:27:04.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:04.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:27:04.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:27:04.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:04.905 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:04.933 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:27:04.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:04.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:27:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:27:05.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:27:05.075 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:05.182 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:27:05.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:05.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:27:05.288 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:05.322 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:05.422 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:05.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:05.530 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:27:05.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:05 smithi090 ceph-mon[44434]: pgmap v1395: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 865 op/s 2024-05-15T23:27:05.563 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:05.603 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:27:05.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:05.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:27:05.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:27:05.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:27:05.732 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:05 smithi084 ceph-mon[39784]: pgmap v1395: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 865 op/s 2024-05-15T23:27:05.765 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-05-15T23:27:05.765 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-05-15T23:27:05.765 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-05-15T23:27:05.765 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-05-15T23:27:05.765 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-05-15T23:27:05.765 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-05-15T23:27:05.765 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-05-15T23:27:05.765 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-05-15T23:27:05.766 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-05-15T23:27:05.766 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-05-15T23:27:05.766 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-05-15T23:27:05.766 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-05-15T23:27:05.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:05 smithi088 ceph-mon[44699]: pgmap v1395: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 865 op/s 2024-05-15T23:27:05.831 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:06.501 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:27:06.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:06.507 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:27:06.598 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T23:27:06.599 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T23:27:06.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-05-15T23:27:06.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:27:06.617 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:06.636 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:07.089 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:27:07.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:07.093 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:27:07.193 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:07.200 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:07 smithi084 ceph-mon[39784]: pgmap v1396: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.33k op/s 2024-05-15T23:27:07.216 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:27:07.235 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:07.286 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:07.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:07.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:07.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:07 smithi090 ceph-mon[44434]: pgmap v1396: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.33k op/s 2024-05-15T23:27:07.394 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:27:07.432 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:07.492 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:27:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:07.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:27:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:07 smithi088 ceph-mon[44699]: pgmap v1396: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.33k op/s 2024-05-15T23:27:07.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:07.636 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:07.680 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:07.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:07.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:07.744 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:27:07.744 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:27:07.784 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:27:07.815 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:07.974 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:27:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:07.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:27:08.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:27:08.114 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:08.319 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:08.435 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:08 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:27:08.436 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:08 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:27:08.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T23:27:08.471 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:27:08.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:08.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:27:08.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:08 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:27:08.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:08 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:27:08.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:27:08.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:08 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:27:08.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:08 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:27:08.624 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:08.833 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:27:08.965 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:09.155 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:09 smithi084 ceph-mon[39784]: osdmap e523: 12 total, 12 up, 8 in 2024-05-15T23:27:09.155 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:09 smithi084 ceph-mon[39784]: pgmap v1398: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:27:09.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:09 smithi090 ceph-mon[44434]: osdmap e523: 12 total, 12 up, 8 in 2024-05-15T23:27:09.393 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:09 smithi090 ceph-mon[44434]: pgmap v1398: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:27:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:09 smithi088 ceph-mon[44699]: osdmap e523: 12 total, 12 up, 8 in 2024-05-15T23:27:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:09 smithi088 ceph-mon[44699]: pgmap v1398: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:27:09.786 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:27:09.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:09.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:27:09.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:27:09.927 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:10.037 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:27:10.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:10.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:27:10.149 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:10.170 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:27:10.194 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:10.216 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:10.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:10.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:10.305 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:10 smithi084 ceph-mon[39784]: 6.9 scrub starts 2024-05-15T23:27:10.306 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:10 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1930614949' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:27:10.324 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:10.363 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:10.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:10 smithi090 ceph-mon[44434]: 6.9 scrub starts 2024-05-15T23:27:10.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:10 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1930614949' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:27:10.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:10 smithi088 ceph-mon[44699]: 6.9 scrub starts 2024-05-15T23:27:10.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:10 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1930614949' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:27:10.919 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:10.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:10.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:11.023 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:11.061 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:11.106 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:11.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:11.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:27:11.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:11 smithi090 ceph-mon[44434]: 6.9 scrub ok 2024-05-15T23:27:11.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:11 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:27:11.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:11 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1930614949' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:27:11.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:11 smithi090 ceph-mon[44434]: osdmap e524: 12 total, 12 up, 8 in 2024-05-15T23:27:11.249 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:11 smithi090 ceph-mon[44434]: pgmap v1400: 17 pgs: 17 active+clean; 6.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-05-15T23:27:11.250 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:11.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:11 smithi088 ceph-mon[44699]: 6.9 scrub ok 2024-05-15T23:27:11.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:11 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:27:11.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:11 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1930614949' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:27:11.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:11 smithi088 ceph-mon[44699]: osdmap e524: 12 total, 12 up, 8 in 2024-05-15T23:27:11.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:11 smithi088 ceph-mon[44699]: pgmap v1400: 17 pgs: 17 active+clean; 6.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-05-15T23:27:11.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:11 smithi084 ceph-mon[39784]: 6.9 scrub ok 2024-05-15T23:27:11.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:11 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:27:11.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:11 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1930614949' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:27:11.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:11 smithi084 ceph-mon[39784]: osdmap e524: 12 total, 12 up, 8 in 2024-05-15T23:27:11.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:11 smithi084 ceph-mon[39784]: pgmap v1400: 17 pgs: 17 active+clean; 6.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-05-15T23:27:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:27:11.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:11.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:27:11.627 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:11.665 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:11.772 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:11.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:11.851 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-05-15T23:27:11.852 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-05-15T23:27:11.852 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-05-15T23:27:11.852 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-05-15T23:27:11.852 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-05-15T23:27:11.852 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-05-15T23:27:11.852 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-05-15T23:27:11.852 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-05-15T23:27:11.852 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-05-15T23:27:11.852 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-05-15T23:27:11.852 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-05-15T23:27:11.852 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-05-15T23:27:11.877 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:11.931 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:12.125 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:12.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:12.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:12.176 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:27:12.177 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:27:12.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:12.265 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:12.473 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:12.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:12.478 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:12.578 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:27:12.619 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:12.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T23:27:12.981 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:27:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:12.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:27:13.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:27:13.117 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:27:13.137 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:27:13.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:13 smithi090 ceph-mon[44434]: pgmap v1401: 17 pgs: 17 active+clean; 5.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.51k op/s 2024-05-15T23:27:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:13.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:27:13.267 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:27:13.312 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:13.337 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:27:13.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:13.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:27:13.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:13 smithi084 ceph-mon[39784]: pgmap v1401: 17 pgs: 17 active+clean; 5.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.51k op/s 2024-05-15T23:27:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:27:13.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:27:13.466 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T23:27:13.466 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T23:27:13.467 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-05-15T23:27:13.491 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:13.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:13 smithi088 ceph-mon[44699]: pgmap v1401: 17 pgs: 17 active+clean; 5.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.51k op/s 2024-05-15T23:27:13.704 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:27:13.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:13.708 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:27:13.736 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:27:13.755 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:13.776 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:27:13.802 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:27:13.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:27:13.836 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:27:13.850 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:13.874 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:27:13.908 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:27:13.934 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:27:13.952 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:27:13.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:13.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:27:13.964 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:27:13.987 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:27:14.009 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:27:14.038 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:27:14.059 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:27:14.065 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:27:14.067 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:27:14.067 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:27:14.067 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:27:14.067 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:27:14.067 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:27:14.067 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:27:14.067 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:27:14.067 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:27:14.067 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:27:14.067 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:27:14.067 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:27:14.067 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:27:14.092 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:14.126 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:27:14.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:14.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:27:14.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:27:14.263 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:14.305 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:27:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:14.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:27:14.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:14 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:27:14.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:14 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:27:14.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:14 smithi090 ceph-mon[44434]: osdmap e525: 12 total, 12 up, 8 in 2024-05-15T23:27:14.397 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:14 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3837181159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:27:14.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:14.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:14.413 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:27:14.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:14 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:27:14.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:14 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:27:14.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:14 smithi084 ceph-mon[39784]: osdmap e525: 12 total, 12 up, 8 in 2024-05-15T23:27:14.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:14 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3837181159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:27:14.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:14.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:14.454 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:14 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:27:14.455 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:14 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:27:14.455 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:14 smithi088 ceph-mon[44699]: osdmap e525: 12 total, 12 up, 8 in 2024-05-15T23:27:14.455 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:14 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3837181159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:27:14.455 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:14.455 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:14.462 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:14.486 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:27:14.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:14.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:27:14.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:14.625 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:15.283 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:15.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:15.394 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:27:15.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:15 smithi084 ceph-mon[39784]: pgmap v1403: 17 pgs: 17 active+clean; 5.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 872 op/s 2024-05-15T23:27:15.428 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:15.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:15 smithi090 ceph-mon[44434]: pgmap v1403: 17 pgs: 17 active+clean; 5.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 872 op/s 2024-05-15T23:27:15.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:15.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:15 smithi088 ceph-mon[44699]: pgmap v1403: 17 pgs: 17 active+clean; 5.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 872 op/s 2024-05-15T23:27:15.688 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:27:15.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:15.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:27:15.789 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:15.795 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:15.833 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:16.283 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:16.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:16.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:16.317 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:27:16.317 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:27:16.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.317 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.26326.0:181241 6.9 e526/492 6:984022b7:::benchmark_data_smithi090_131278_object46047:head v 526'11306, mlcod=526'11303)", 2024-05-15T23:27:16.317 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:27:16.308792+0000", 2024-05-15T23:27:16.318 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0064949990000000004, 2024-05-15T23:27:16.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.006523877, 2024-05-15T23:27:16.318 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:27:16.318 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:27:16.321 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:27:16.321 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:27:16.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:27:16.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.308792+0000", 2024-05-15T23:27:16.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:27:16.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:16.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:27:16.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.308792+0000", 2024-05-15T23:27:16.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:27:16.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:16.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:27:16.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.308794+0000", 2024-05-15T23:27:16.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9580000000000001e-06 2024-05-15T23:27:16.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:16.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:27:16.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.308803+0000", 2024-05-15T23:27:16.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9600000000000006e-06 2024-05-15T23:27:16.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:16.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:27:16.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.308818+0000", 2024-05-15T23:27:16.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5401999999999999e-05 2024-05-15T23:27:16.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:16.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:27:16.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.308827+0000", 2024-05-15T23:27:16.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7830000000000004e-06 2024-05-15T23:27:16.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:16.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:27:16.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.310015+0000", 2024-05-15T23:27:16.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001188592 2024-05-15T23:27:16.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:16.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:27:16.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.310041+0000", 2024-05-15T23:27:16.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5879000000000001e-05 2024-05-15T23:27:16.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:27:16.324 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:27:16.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:27:16.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:16.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.324 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.26326.0:181243 6.9 e526/492 6:9a6bef12:::benchmark_data_smithi090_131278_object46049:head v 526'11307, mlcod=526'11303)", 2024-05-15T23:27:16.324 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:27:16.313398+0000", 2024-05-15T23:27:16.324 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.001888549, 2024-05-15T23:27:16.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019987479999999998, 2024-05-15T23:27:16.324 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:27:16.324 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:27:16.325 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:27:16.325 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:27:16.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:27:16.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.313398+0000", 2024-05-15T23:27:16.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:27:16.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:16.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:27:16.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.313398+0000", 2024-05-15T23:27:16.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:27:16.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:16.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:27:16.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.313400+0000", 2024-05-15T23:27:16.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.765e-06 2024-05-15T23:27:16.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:16.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:27:16.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.313408+0000", 2024-05-15T23:27:16.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7780000000000007e-06 2024-05-15T23:27:16.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:16.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:27:16.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.313420+0000", 2024-05-15T23:27:16.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2351000000000001e-05 2024-05-15T23:27:16.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:16.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:27:16.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.313428+0000", 2024-05-15T23:27:16.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2190000000000004e-06 2024-05-15T23:27:16.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:16.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:27:16.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.315122+0000", 2024-05-15T23:27:16.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001693197 2024-05-15T23:27:16.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:27:16.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:27:16.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:27:16.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:27:16.315142+0000", 2024-05-15T23:27:16.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0386000000000001e-05 2024-05-15T23:27:16.329 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:27:16.329 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:27:16.329 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:27:16.329 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:27:16.329 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-05-15T23:27:16.329 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 2 2024-05-15T23:27:16.329 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:27:16.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:16 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:27:16.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:16 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:27:16.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:16 smithi084 ceph-mon[39784]: osdmap e526: 12 total, 12 up, 8 in 2024-05-15T23:27:16.389 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:27:16.433 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:16 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:27:16.433 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:16 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:27:16.433 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:16 smithi088 ceph-mon[44699]: osdmap e526: 12 total, 12 up, 8 in 2024-05-15T23:27:16.443 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:27:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:16.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:27:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:16 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:27:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:16 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:27:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:16 smithi090 ceph-mon[44434]: osdmap e526: 12 total, 12 up, 8 in 2024-05-15T23:27:16.580 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:16.623 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:17.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T23:27:17.174 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:27:17.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:17 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1068502683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:27:17.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:17 smithi090 ceph-mon[44434]: from='client.26509 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:27:17.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:17 smithi090 ceph-mon[44434]: pgmap v1405: 17 pgs: 17 active+clean; 5.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.47k op/s 2024-05-15T23:27:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:17 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1068502683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:27:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:17 smithi084 ceph-mon[39784]: from='client.26509 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:27:17.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:17 smithi084 ceph-mon[39784]: pgmap v1405: 17 pgs: 17 active+clean; 5.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.47k op/s 2024-05-15T23:27:17.426 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:27:17.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:17 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1068502683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:27:17.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:17 smithi088 ceph-mon[44699]: from='client.26509 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:27:17.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:17 smithi088 ceph-mon[44699]: pgmap v1405: 17 pgs: 17 active+clean; 5.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.47k op/s 2024-05-15T23:27:17.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:27:17.785 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:17.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:17.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:17.889 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:27:17.945 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:18.199 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:18 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:27:18.199 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:18 smithi084 ceph-mon[39784]: from='client.26509 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:27:18.199 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:18 smithi084 ceph-mon[39784]: osdmap e527: 12 total, 12 up, 8 in 2024-05-15T23:27:18.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:18 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:27:18.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:18 smithi090 ceph-mon[44434]: from='client.26509 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:27:18.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:18 smithi090 ceph-mon[44434]: osdmap e527: 12 total, 12 up, 8 in 2024-05-15T23:27:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:18 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:27:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:18 smithi088 ceph-mon[44699]: from='client.26509 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:27:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:18 smithi088 ceph-mon[44699]: osdmap e527: 12 total, 12 up, 8 in 2024-05-15T23:27:18.917 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:27:18.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:18.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:27:18.998 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-05-15T23:27:18.998 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-05-15T23:27:18.998 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-05-15T23:27:18.998 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-05-15T23:27:18.998 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-05-15T23:27:18.998 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-05-15T23:27:18.999 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-05-15T23:27:18.999 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-05-15T23:27:18.999 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-05-15T23:27:18.999 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-05-15T23:27:18.999 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-05-15T23:27:18.999 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-05-15T23:27:19.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:27:19.037 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:19.055 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:19.078 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:27:19.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:19.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:27:19.189 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:27:19.228 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:19.259 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:27:19.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:19.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:27:19.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:19 smithi090 ceph-mon[44434]: pgmap v1407: 17 pgs: 17 active+clean; 5.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T23:27:19.364 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:27:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:19 smithi084 ceph-mon[39784]: pgmap v1407: 17 pgs: 17 active+clean; 5.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T23:27:19.405 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:19.426 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:27:19.426 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:27:19.427 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:27:19.427 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:27:19.427 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:27:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:19 smithi088 ceph-mon[44699]: pgmap v1407: 17 pgs: 17 active+clean; 5.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T23:27:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T23:27:19.740 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T23:27:19.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-05-15T23:27:19.966 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:27:19.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:19.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:27:20.071 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:20.115 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:20.149 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:20.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:20.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:20.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:27:20.308 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:20.339 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:27:20.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:20.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:27:20.453 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:27:20.491 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:20.520 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:27:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:20.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:27:20.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:27:20.672 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:20.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:27:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:27:20.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:20.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:27:20.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T23:27:20.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:21.026 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:21.036 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:21.065 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:21.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:21.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:21.108 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:21.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:27:21.228 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:21 smithi090 ceph-mon[44434]: pgmap v1408: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-05-15T23:27:21.235 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:21.262 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:27:21.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:21.273 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:27:21.317 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:27:21.348 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:21 smithi084 ceph-mon[39784]: pgmap v1408: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-05-15T23:27:21.374 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:21.406 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:21.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:21 smithi088 ceph-mon[44699]: pgmap v1408: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-05-15T23:27:21.610 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:21.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:21.614 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:21.715 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:27:21.745 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:21.772 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:27:21.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:21.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:27:21.878 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:27:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:21.958 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:27:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:21.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:27:22.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:27:22.097 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:22.304 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:27:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:22.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:27:22.409 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:27:22.443 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:22.551 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:27:22.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:22.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:27:22.655 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:27:22.702 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:22.727 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:27:22.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:22.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:27:22.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:27:22.854 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:27:22.872 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:22.876 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:27:22.904 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:27:22.932 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:27:22.962 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:27:22.988 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:27:23.012 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:27:23.039 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:27:23.075 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:27:23.109 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:27:23.141 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:27:23.163 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:27:23.166 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:27:23.166 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:27:23.166 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:27:23.166 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:27:23.167 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:27:23.167 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:27:23.167 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:27:23.167 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:27:23.167 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:27:23.167 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:27:23.167 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:27:23.167 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:27:23.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:23 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4125394945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:27:23.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:23 smithi084 ceph-mon[39784]: pgmap v1409: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.51k op/s 2024-05-15T23:27:23.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:23.488 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:23 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4125394945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:27:23.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:23 smithi090 ceph-mon[44434]: pgmap v1409: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.51k op/s 2024-05-15T23:27:23.489 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:23.526 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:27:23.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:23.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:27:23.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:23 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4125394945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:27:23.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:23 smithi088 ceph-mon[44699]: pgmap v1409: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.51k op/s 2024-05-15T23:27:23.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:23.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:23.667 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:23.876 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:23.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:23.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:23.936 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-05-15T23:27:23.937 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-05-15T23:27:23.937 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-05-15T23:27:23.937 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-05-15T23:27:23.937 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-05-15T23:27:23.937 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-05-15T23:27:23.937 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-05-15T23:27:23.937 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-05-15T23:27:23.937 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-05-15T23:27:23.937 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-05-15T23:27:23.937 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-05-15T23:27:23.937 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-05-15T23:27:23.981 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:27:24.031 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:24.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:27:24.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:27:24.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:27:24.135 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:24 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:27:24.312 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:27:24.312 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:27:24.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:27:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:27:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:27:24.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:24 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:27:24.577 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.577 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:27:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:27:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:27:24.578 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:24 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:27:25.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T23:27:25.264 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T23:27:25.265 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T23:27:25.265 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-05-15T23:27:25.286 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:27:25.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:25.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:27:25.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:25 smithi090 ceph-mon[44434]: pgmap v1410: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:27:25.392 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:27:25.403 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:25 smithi084 ceph-mon[39784]: pgmap v1410: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:27:25.423 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:25.436 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:27:25.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:25 smithi088 ceph-mon[44699]: pgmap v1410: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:27:26.284 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:27:26.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:26.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:27:26.392 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:27:26.422 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:26.560 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:27:26.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:26.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:27:26.589 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:26.662 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:26.715 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:26.749 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:26.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:26.855 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:27:26.891 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:26.923 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:27:26.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:26.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:27:27.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:27:27.077 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:27.104 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:27:27.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:27.111 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:27:27.212 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:27:27.252 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:27.280 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:27:27.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:27.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:27:27.385 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:27:27.402 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:27:27.402 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:27:27.423 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:27.463 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:27:27.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:27.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:27:27.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:27.607 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:27.767 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:27.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:27.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:27.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:27:27.907 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:27.946 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:27:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:27.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:27:28.051 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:27:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:27:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:27 smithi090 ceph-mon[44434]: osdmap e528: 12 total, 12 up, 8 in 2024-05-15T23:27:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:27 smithi090 ceph-mon[44434]: pgmap v1412: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:27:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:27 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4152983027' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:27:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:27 smithi090 ceph-mon[44434]: from='client.26533 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:27:28.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:27:28.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:27:28.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:27 smithi084 ceph-mon[39784]: osdmap e528: 12 total, 12 up, 8 in 2024-05-15T23:27:28.060 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:27 smithi084 ceph-mon[39784]: pgmap v1412: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:27:28.060 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:27 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4152983027' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:27:28.060 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:27 smithi084 ceph-mon[39784]: from='client.26533 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:27:28.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:27:28.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:27:28.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:27 smithi088 ceph-mon[44699]: osdmap e528: 12 total, 12 up, 8 in 2024-05-15T23:27:28.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:27 smithi088 ceph-mon[44699]: pgmap v1412: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:27:28.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:27 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4152983027' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:27:28.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:27 smithi088 ceph-mon[44699]: from='client.26533 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:27:28.090 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:28.124 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:28.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:28.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:28.147 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:27:28.230 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:28.262 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:28.298 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:27:28.415 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:28.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:28.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:28.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:29.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:28 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:27:29.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:28 smithi084 ceph-mon[39784]: from='client.26533 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:27:29.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:28 smithi084 ceph-mon[39784]: osdmap e529: 12 total, 12 up, 8 in 2024-05-15T23:27:29.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:28 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:27:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:28 smithi088 ceph-mon[44699]: from='client.26533 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:27:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:28 smithi088 ceph-mon[44699]: osdmap e529: 12 total, 12 up, 8 in 2024-05-15T23:27:29.103 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:28 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:27:29.103 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:28 smithi090 ceph-mon[44434]: from='client.26533 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:27:29.103 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:28 smithi090 ceph-mon[44434]: osdmap e529: 12 total, 12 up, 8 in 2024-05-15T23:27:29.116 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:29.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:29.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:29.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T23:27:29.222 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:27:29.255 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:29.282 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:27:29.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:29.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:27:29.394 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:29.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:27:29.408 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-05-15T23:27:29.408 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-05-15T23:27:29.408 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-05-15T23:27:29.408 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-05-15T23:27:29.408 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-05-15T23:27:29.409 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-05-15T23:27:29.409 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-05-15T23:27:29.409 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-05-15T23:27:29.409 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-05-15T23:27:29.409 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-05-15T23:27:29.409 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-05-15T23:27:29.409 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-05-15T23:27:29.450 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:29.488 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:27:29.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:29.492 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:27:29.563 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:27:29.593 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:27:29.652 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:29.687 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:27:29.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:29.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:27:29.801 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:27:29.834 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:29.853 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:27:29.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:29.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:27:29.957 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:27:29.965 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:29 smithi084 ceph-mon[39784]: pgmap v1414: 17 pgs: 17 active+clean; 4.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:27:29.988 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:30.055 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:27:30.055 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:27:30.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:29 smithi088 ceph-mon[44699]: pgmap v1414: 17 pgs: 17 active+clean; 4.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:27:30.084 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:29 smithi090 ceph-mon[44434]: pgmap v1414: 17 pgs: 17 active+clean; 4.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:27:30.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:30.184 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:30.385 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:30.388 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:30.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:27:30.522 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:31.095 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:27:31.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:30 smithi084 ceph-mon[39784]: 6.2 scrub starts 2024-05-15T23:27:31.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:30 smithi090 ceph-mon[44434]: 6.2 scrub starts 2024-05-15T23:27:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:30 smithi088 ceph-mon[44699]: 6.2 scrub starts 2024-05-15T23:27:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:27:31.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:31.385 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:27:31.486 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:27:31.537 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:31.569 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:27:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:31.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:27:31.674 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:27:31.726 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:31.756 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:27:31.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:31.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:27:31.866 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:31.899 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:31.925 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:31.927 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:31 smithi090 ceph-mon[44434]: 6.2 scrub ok 2024-05-15T23:27:31.927 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:31 smithi090 ceph-mon[44434]: pgmap v1415: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T23:27:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:31.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:32.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:32.069 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:32.085 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:27:32.085 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:27:32.099 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:32.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:32.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:31 smithi084 ceph-mon[39784]: 6.2 scrub ok 2024-05-15T23:27:32.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:31 smithi084 ceph-mon[39784]: pgmap v1415: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T23:27:32.209 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:27:32.276 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:31 smithi088 ceph-mon[44699]: 6.2 scrub ok 2024-05-15T23:27:32.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:31 smithi088 ceph-mon[44699]: pgmap v1415: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T23:27:32.282 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:32.314 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:27:32.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:32.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:27:32.420 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:32.470 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:32.503 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:32.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:32.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:32.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:27:32.639 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:32.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T23:27:33.170 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:27:33.433 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:33 smithi090 ceph-mon[44434]: 6.7 scrub starts 2024-05-15T23:27:33.434 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:33 smithi090 ceph-mon[44434]: pgmap v1416: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.47k op/s 2024-05-15T23:27:33.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:33 smithi088 ceph-mon[44699]: 6.7 scrub starts 2024-05-15T23:27:33.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:33 smithi088 ceph-mon[44699]: pgmap v1416: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.47k op/s 2024-05-15T23:27:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:33 smithi084 ceph-mon[39784]: 6.7 scrub starts 2024-05-15T23:27:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:33 smithi084 ceph-mon[39784]: pgmap v1416: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.47k op/s 2024-05-15T23:27:34.471 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:34 smithi090 ceph-mon[44434]: 6.7 scrub ok 2024-05-15T23:27:34.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:34 smithi088 ceph-mon[44699]: 6.7 scrub ok 2024-05-15T23:27:34.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:34 smithi084 ceph-mon[39784]: 6.7 scrub ok 2024-05-15T23:27:35.133 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:27:35.133 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:27:35.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:35 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:27:35.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:35 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:27:35.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:35 smithi090 ceph-mon[44434]: pgmap v1417: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 837 op/s 2024-05-15T23:27:35.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:35 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:27:35.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:35 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:27:35.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:35 smithi088 ceph-mon[44699]: pgmap v1417: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 837 op/s 2024-05-15T23:27:35.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:35 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:27:35.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:35 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:27:35.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:35 smithi084 ceph-mon[39784]: pgmap v1417: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 837 op/s 2024-05-15T23:27:36.422 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:36 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:27:36.422 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:36 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:27:36.422 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:36 smithi084 ceph-mon[39784]: osdmap e530: 12 total, 12 up, 8 in 2024-05-15T23:27:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T23:27:36.465 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T23:27:36.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-05-15T23:27:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:36 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:27:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:36 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:27:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:36 smithi090 ceph-mon[44434]: osdmap e530: 12 total, 12 up, 8 in 2024-05-15T23:27:36.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:36 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:27:36.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:36 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:27:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:36 smithi088 ceph-mon[44699]: osdmap e530: 12 total, 12 up, 8 in 2024-05-15T23:27:36.613 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T23:27:36.704 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:27:36.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:36.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:27:36.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:27:36.852 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:36.894 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:27:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:36.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:27:36.999 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:37.034 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:37.050 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:37.078 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:37.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:37.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:37.113 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:27:37.137 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:27:37.162 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:37 smithi088 ceph-mon[44699]: pgmap v1419: 17 pgs: 17 active+clean; 4.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s 2024-05-15T23:27:37.162 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:37 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4265752309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:27:37.166 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:27:37.184 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:27:37.194 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:27:37.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:27:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:37.244 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:27:37.252 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:27:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:37.260 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:27:37.265 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:27:37.293 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:27:37.326 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:27:37.342 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:37.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:27:37.369 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:27:37.390 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:37.412 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:27:37.415 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:37 smithi090 ceph-mon[44434]: pgmap v1419: 17 pgs: 17 active+clean; 4.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s 2024-05-15T23:27:37.415 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:37 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4265752309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:27:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:37.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:27:37.419 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:27:37.455 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:27:37.465 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:27:37.465 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:27:37.465 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:27:37.465 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:27:37.465 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:27:37.465 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:27:37.465 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:27:37.465 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:27:37.465 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:27:37.466 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:27:37.466 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:27:37.466 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:27:37.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:37 smithi084 ceph-mon[39784]: pgmap v1419: 17 pgs: 17 active+clean; 4.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s 2024-05-15T23:27:37.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:37 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4265752309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:27:37.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:37.554 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:37.579 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:37.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:37.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:37.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:37.723 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:37.752 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:37.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:37.755 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:37.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:27:37.894 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:27:37.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:37.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:27:38.030 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:27:38.075 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:38.100 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:27:38.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:38.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:27:38.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:38.247 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:38.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.326 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:38.328 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:38.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:38.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:38.437 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:27:38.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.466 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:38.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:39.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:39.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:39.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:39 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:27:39.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:39 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:27:39.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:39 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:27:39.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:39 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:27:39.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:39 smithi084 ceph-mon[39784]: pgmap v1420: 17 pgs: 17 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:27:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:39 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:27:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:39 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:27:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:39 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:27:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:39 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:27:39.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:39 smithi090 ceph-mon[44434]: pgmap v1420: 17 pgs: 17 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:27:39.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:39.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:39.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:39 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:27:39.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:39 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:27:39.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:39 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:27:39.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:39 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:27:39.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:39 smithi088 ceph-mon[44699]: pgmap v1420: 17 pgs: 17 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:27:39.955 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:27:39.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:39.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:27:40.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:27:40.090 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:40.120 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:27:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:40.124 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:27:40.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:40 smithi084 ceph-mon[39784]: 6.5 scrub starts 2024-05-15T23:27:40.217 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:40 smithi084 ceph-mon[39784]: osdmap e531: 12 total, 12 up, 8 in 2024-05-15T23:27:40.225 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:27:40.268 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:40.433 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:27:40.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:40.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:27:40.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:40 smithi090 ceph-mon[44434]: 6.5 scrub starts 2024-05-15T23:27:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:40 smithi090 ceph-mon[44434]: osdmap e531: 12 total, 12 up, 8 in 2024-05-15T23:27:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:40 smithi088 ceph-mon[44699]: 6.5 scrub starts 2024-05-15T23:27:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:40 smithi088 ceph-mon[44699]: osdmap e531: 12 total, 12 up, 8 in 2024-05-15T23:27:40.575 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:40.597 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:40.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:40.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:27:40.747 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:40.852 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:27:40.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:40.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:27:40.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:27:40.994 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:41.134 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:27:41.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:41.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:27:41.236 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:41.295 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:41.333 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:41.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:41.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:41.440 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:41.493 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:41 smithi088 ceph-mon[44699]: 6.5 scrub ok 2024-05-15T23:27:41.493 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:41 smithi088 ceph-mon[44699]: pgmap v1422: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T23:27:41.495 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:41.536 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:41.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:41.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:41 smithi090 ceph-mon[44434]: 6.5 scrub ok 2024-05-15T23:27:41.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:41 smithi090 ceph-mon[44434]: pgmap v1422: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T23:27:41.582 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:27:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:27:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-05-15T23:27:41.583 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:27:41.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:41 smithi084 ceph-mon[39784]: 6.5 scrub ok 2024-05-15T23:27:41.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:41 smithi084 ceph-mon[39784]: pgmap v1422: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T23:27:41.645 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:42.054 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:42.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:42.163 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:42.195 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:42.228 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:42.229 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:27:42.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:42.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:42.334 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:27:42.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:42 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2715707695' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:27:42.380 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:42.493 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:27:42.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:42.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:27:42.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:42 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2715707695' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:27:42.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:42 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2715707695' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:27:42.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:27:42.630 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:42.844 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T23:27:43.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:43 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:27:43.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:43 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:27:43.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:43 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:27:43.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:43 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:27:43.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:43 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2715707695' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:27:43.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:43 smithi084 ceph-mon[39784]: osdmap e532: 12 total, 12 up, 8 in 2024-05-15T23:27:43.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:43 smithi084 ceph-mon[39784]: pgmap v1424: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.40k op/s 2024-05-15T23:27:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:43 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:27:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:43 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:27:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:43 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:27:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:43 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:27:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:43 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2715707695' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:27:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:43 smithi090 ceph-mon[44434]: osdmap e532: 12 total, 12 up, 8 in 2024-05-15T23:27:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:43 smithi090 ceph-mon[44434]: pgmap v1424: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.40k op/s 2024-05-15T23:27:43.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:43 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:27:43.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:43 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:27:43.575 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:43 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:27:43.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:43 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:27:43.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:43 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2715707695' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:27:43.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:43 smithi088 ceph-mon[44699]: osdmap e532: 12 total, 12 up, 8 in 2024-05-15T23:27:43.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:43 smithi088 ceph-mon[44699]: pgmap v1424: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.40k op/s 2024-05-15T23:27:44.093 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:27:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:44.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:27:44.200 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:27:44.247 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:44.280 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:27:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:44.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:27:44.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:27:44.445 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:44.480 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:27:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:44.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:27:44.545 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:44.563 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-05-15T23:27:44.564 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-05-15T23:27:44.564 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-05-15T23:27:44.564 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-05-15T23:27:44.564 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-05-15T23:27:44.564 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-05-15T23:27:44.564 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-05-15T23:27:44.565 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-05-15T23:27:44.565 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-05-15T23:27:44.565 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-05-15T23:27:44.565 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-05-15T23:27:44.565 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-05-15T23:27:44.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:27:44.633 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:45.544 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:27:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:45 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:27:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:45 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:27:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:45 smithi090 ceph-mon[44434]: 6.0 scrub starts 2024-05-15T23:27:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:45 smithi090 ceph-mon[44434]: osdmap e533: 12 total, 12 up, 8 in 2024-05-15T23:27:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:45 smithi090 ceph-mon[44434]: pgmap v1426: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T23:27:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:45 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:27:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:45 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:27:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:45 smithi088 ceph-mon[44699]: 6.0 scrub starts 2024-05-15T23:27:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:45 smithi088 ceph-mon[44699]: osdmap e533: 12 total, 12 up, 8 in 2024-05-15T23:27:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:45 smithi088 ceph-mon[44699]: pgmap v1426: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T23:27:45.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:45 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:27:45.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:45 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:27:45.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:45 smithi084 ceph-mon[39784]: 6.0 scrub starts 2024-05-15T23:27:45.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:45 smithi084 ceph-mon[39784]: osdmap e533: 12 total, 12 up, 8 in 2024-05-15T23:27:45.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:45 smithi084 ceph-mon[39784]: pgmap v1426: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-05-15T23:27:45.623 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:27:45.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:45.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:27:45.646 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T23:27:45.646 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T23:27:45.646 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-05-15T23:27:45.729 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:27:45.770 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:45.802 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:27:45.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:45.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:27:45.907 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:45.940 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:45.988 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:27:46.150 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:46.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:46.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:46.253 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:46 smithi084 ceph-mon[39784]: 6.0 scrub ok 2024-05-15T23:27:46.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:46.293 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:46.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:46 smithi090 ceph-mon[44434]: 6.0 scrub ok 2024-05-15T23:27:46.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:46 smithi088 ceph-mon[44699]: 6.0 scrub ok 2024-05-15T23:27:46.768 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:46.825 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:47.048 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:47.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:47.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:47.155 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:27:47.210 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:47.239 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:27:47.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:47.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:27:47.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:47 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:27:47.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:47 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:27:47.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:47 smithi084 ceph-mon[39784]: osdmap e534: 12 total, 12 up, 8 in 2024-05-15T23:27:47.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:47 smithi084 ceph-mon[39784]: pgmap v1428: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.46k op/s 2024-05-15T23:27:47.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:27:47.394 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:47.523 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:27:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:47.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:27:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:47 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:27:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:47 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:27:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:47 smithi090 ceph-mon[44434]: osdmap e534: 12 total, 12 up, 8 in 2024-05-15T23:27:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:47 smithi090 ceph-mon[44434]: pgmap v1428: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.46k op/s 2024-05-15T23:27:47.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:47 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:27:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:47 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:27:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:47 smithi088 ceph-mon[44699]: osdmap e534: 12 total, 12 up, 8 in 2024-05-15T23:27:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:47 smithi088 ceph-mon[44699]: pgmap v1428: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.46k op/s 2024-05-15T23:27:47.627 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:27:47.677 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:47.693 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:27:47.693 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:27:47.693 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:27:47.694 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:27:47.694 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:27:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:48 smithi090 ceph-mon[44434]: 6.6 scrub starts 2024-05-15T23:27:48.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:48 smithi088 ceph-mon[44699]: 6.6 scrub starts 2024-05-15T23:27:48.596 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:48 smithi084 ceph-mon[39784]: 6.6 scrub starts 2024-05-15T23:27:49.043 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:27:49.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:49.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:27:49.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:49.184 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:49.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T23:27:49.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:49 smithi090 ceph-mon[44434]: 6.6 scrub ok 2024-05-15T23:27:49.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:49 smithi090 ceph-mon[44434]: 6.4 scrub starts 2024-05-15T23:27:49.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:49 smithi090 ceph-mon[44434]: pgmap v1429: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-05-15T23:27:49.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:49 smithi088 ceph-mon[44699]: 6.6 scrub ok 2024-05-15T23:27:49.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:49 smithi088 ceph-mon[44699]: 6.4 scrub starts 2024-05-15T23:27:49.576 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:49 smithi088 ceph-mon[44699]: pgmap v1429: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-05-15T23:27:49.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:49 smithi084 ceph-mon[39784]: 6.6 scrub ok 2024-05-15T23:27:49.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:49 smithi084 ceph-mon[39784]: 6.4 scrub starts 2024-05-15T23:27:49.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:49 smithi084 ceph-mon[39784]: pgmap v1429: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-05-15T23:27:49.938 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:49.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:50.047 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:27:50.084 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:50.187 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:27:50.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:50.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:27:50.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:50.326 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:50.361 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:50.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:50.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:50.466 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:27:50.506 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:50.517 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-05-15T23:27:50.518 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-05-15T23:27:50.518 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-05-15T23:27:50.518 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-05-15T23:27:50.518 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-05-15T23:27:50.518 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-05-15T23:27:50.518 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-05-15T23:27:50.518 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-05-15T23:27:50.518 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-05-15T23:27:50.518 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-05-15T23:27:50.518 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-05-15T23:27:50.518 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-05-15T23:27:50.531 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:50 smithi084 ceph-mon[39784]: 6.4 scrub ok 2024-05-15T23:27:50.542 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:27:50.544 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:50 smithi088 ceph-mon[44699]: 6.4 scrub ok 2024-05-15T23:27:50.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:50.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:27:50.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:50 smithi090 ceph-mon[44434]: 6.4 scrub ok 2024-05-15T23:27:50.647 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:50.692 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:50.733 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:50.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:50.840 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:27:50.876 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:50.910 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:27:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:50.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:27:51.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:27:51.054 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:51.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:51.089 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:27:51.090 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:27:51.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:27:51.224 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:51.258 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:27:51.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:51.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:27:51.295 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:51.301 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:51 smithi088 ceph-mon[44699]: pgmap v1430: 17 pgs: 17 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:27:51.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:51 smithi090 ceph-mon[44434]: pgmap v1430: 17 pgs: 17 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:27:51.324 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:51.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:27:51.403 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:51.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:51 smithi084 ceph-mon[39784]: pgmap v1430: 17 pgs: 17 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:27:51.557 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:27:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:51.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:27:51.664 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:51.705 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:51.731 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:51.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:51.734 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:51.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:27:51.872 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:51.903 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:27:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:51.910 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:27:52.011 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:52.045 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:52.077 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:52.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:52.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:27:52.239 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:52.278 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:27:52.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:52.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:27:52.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:27:52.428 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:52.455 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:27:52.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:52.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:27:52.561 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:27:52.608 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:27:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:52.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:27:52.752 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:27:52.802 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:52.833 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:27:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:52.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:27:52.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:27:52.969 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:53.010 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:27:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:53.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:27:53.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:27:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:53.182 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:27:53.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:53.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:27:53.287 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:53.351 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:53.378 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:53.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:53.437 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:53 smithi084 ceph-mon[39784]: pgmap v1431: 17 pgs: 17 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.40k op/s 2024-05-15T23:27:53.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:27:53.535 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:53 smithi090 ceph-mon[44434]: pgmap v1431: 17 pgs: 17 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.40k op/s 2024-05-15T23:27:53.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:53 smithi088 ceph-mon[44699]: pgmap v1431: 17 pgs: 17 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.40k op/s 2024-05-15T23:27:53.928 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:27:53.952 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:27:53.980 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:27:53.998 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:27:54.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:54.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:27:54.006 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:27:54.032 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:27:54.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:27:54.082 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:27:54.102 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:27:54.111 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:27:54.141 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:54.142 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:27:54.164 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:27:54.175 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:27:54.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:54.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:27:54.193 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:27:54.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:27:54.226 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:27:54.227 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:27:54.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:27:54.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:27:54.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:27:54.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:27:54.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:27:54.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:27:54.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:27:54.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:27:54.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:27:54.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:27:54.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:27:54.254 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2903555724' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:27:54.254 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:54 smithi084 ceph-mon[39784]: from='client.26563 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:27:54.254 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1486633145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:27:54.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:27:54.316 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2903555724' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:27:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:54 smithi090 ceph-mon[44434]: from='client.26563 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:27:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1486633145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:27:54.567 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:27:54.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2903555724' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:27:54.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:54 smithi088 ceph-mon[44699]: from='client.26563 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:27:54.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1486633145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:27:54.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:54.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:27:54.604 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T23:27:54.605 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T23:27:54.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-05-15T23:27:54.681 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:54.720 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:54.800 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:54.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:54.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:27:54.944 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:54.974 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:27:54.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:54.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:27:55.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:55.117 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:55.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: 6.e scrub starts 2024-05-15T23:27:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:27:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: from='client.26563 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:27:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: osdmap e535: 12 total, 12 up, 8 in 2024-05-15T23:27:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:55 smithi090 ceph-mon[44434]: pgmap v1433: 17 pgs: 17 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 964 op/s 2024-05-15T23:27:55.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: 6.e scrub starts 2024-05-15T23:27:55.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:27:55.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: from='client.26563 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:27:55.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: osdmap e535: 12 total, 12 up, 8 in 2024-05-15T23:27:55.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:55 smithi084 ceph-mon[39784]: pgmap v1433: 17 pgs: 17 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 964 op/s 2024-05-15T23:27:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: 6.e scrub starts 2024-05-15T23:27:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:27:55.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: from='client.26563 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:27:55.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: osdmap e535: 12 total, 12 up, 8 in 2024-05-15T23:27:55.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:27:55.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:55 smithi088 ceph-mon[44699]: pgmap v1433: 17 pgs: 17 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 964 op/s 2024-05-15T23:27:56.205 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:56.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:56.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:56.309 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:27:56.369 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:56.436 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:27:56.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:56.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:27:56.542 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:56.574 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:56.706 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:56.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:56.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:56.810 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:27:56.851 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:56.887 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:27:56.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:56.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:27:56.991 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:57.038 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:57.069 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:57.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:57.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:57.082 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:57.179 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:27:57.208 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:57.241 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:27:57.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:57.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:27:57.351 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:27:57.392 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:57 smithi084 ceph-mon[39784]: pgmap v1434: 17 pgs: 17 active+clean; 2.6 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:27:57.394 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:57 smithi090 ceph-mon[44434]: pgmap v1434: 17 pgs: 17 active+clean; 2.6 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:27:57.396 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:57.423 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:27:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:57.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:27:57.529 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:57.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:57 smithi088 ceph-mon[44699]: pgmap v1434: 17 pgs: 17 active+clean; 2.6 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:27:57.575 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:57.608 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:57.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:57.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:57.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:57.753 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:57.956 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:57.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:57.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:58.055 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:27:58.055 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:27:58.061 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:27:58.092 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:58.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:27:58.296 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:27:58.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:58.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:27:58.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:27:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:58.458 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:27:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:58.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:27:58.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:27:58.590 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:58.611 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:27:58.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:58.619 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:27:58.720 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:27:58.765 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:58.797 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:27:58.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:58.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:27:58.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:27:58.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T23:27:58.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:27:58.934 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:58.958 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:27:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:58.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:27:59.065 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:27:59.100 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:59.138 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:27:59.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:59.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:27:59.245 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:27:59.284 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:59.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:27:59 smithi090 ceph-mon[44434]: pgmap v1435: 17 pgs: 17 active+clean; 2.5 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:27:59.314 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:27:59.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:59.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:27:59.421 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:27:59.453 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:27:59 smithi088 ceph-mon[44699]: pgmap v1435: 17 pgs: 17 active+clean; 2.5 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:27:59.453 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:59.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:27:59 smithi084 ceph-mon[39784]: pgmap v1435: 17 pgs: 17 active+clean; 2.5 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:27:59.486 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:27:59.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:59.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:27:59.512 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:27:59.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:27:59.630 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:27:59.789 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:27:59.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:27:59.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:27:59.903 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:27:59.940 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:00.046 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:28:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:00.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:28:00.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:28:00.186 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:00.206 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:28:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:00.210 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:28:00.311 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:28:00.348 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:00.374 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:28:00.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:00.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:28:00.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:28:00.497 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-05-15T23:28:00.497 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-05-15T23:28:00.497 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-05-15T23:28:00.497 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-05-15T23:28:00.498 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-05-15T23:28:00.498 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-05-15T23:28:00.498 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-05-15T23:28:00.498 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-05-15T23:28:00.498 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-05-15T23:28:00.498 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-05-15T23:28:00.498 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-05-15T23:28:00.498 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-05-15T23:28:00.513 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:00.537 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:28:00.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:00.543 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:28:00.645 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:28:00.699 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:00.723 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:28:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:00.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:28:00.827 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:00.871 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:00.901 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:00.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:01.000 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:01.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:28:01.052 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:01.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:01 smithi084 ceph-mon[39784]: pgmap v1436: 17 pgs: 17 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:28:01.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:01 smithi090 ceph-mon[44434]: pgmap v1436: 17 pgs: 17 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:28:01.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:01 smithi088 ceph-mon[44699]: pgmap v1436: 17 pgs: 17 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:28:01.706 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:28:01.709 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T23:28:01.709 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-15T23:28:01.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:01.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:28:01.811 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:01.856 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:01.885 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:01.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:01.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:01.990 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:02.037 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:02.071 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:02.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:02.115 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:02.175 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:28:02.214 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:02.767 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:28:02.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:02.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:28:02.875 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:28:02.908 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:02.932 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:28:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:02.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:28:03.039 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:28:03.103 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:03.133 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:28:03.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:03.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:28:03.240 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:28:03.270 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:03 smithi090 ceph-mon[44434]: pgmap v1437: 17 pgs: 17 active+clean; 2.1 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:28:03.272 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:03.297 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:28:03.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:03.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:28:03.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:03 smithi084 ceph-mon[39784]: pgmap v1437: 17 pgs: 17 active+clean; 2.1 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:28:03.408 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:03.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:03 smithi088 ceph-mon[44699]: pgmap v1437: 17 pgs: 17 active+clean; 2.1 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:28:03.465 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:03.499 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:03.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:03.609 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:28:03.665 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:03.706 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:28:03.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:03.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:28:03.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:03.849 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:03.886 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:03.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:03.997 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:28:04.050 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:04.099 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:28:04.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:04.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:28:04.210 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:04.247 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:04.280 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:04.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:04.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:04.384 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:28:04.425 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:05.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:05 smithi084 ceph-mon[39784]: pgmap v1438: 17 pgs: 17 active+clean; 2.1 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:28:05.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:05 smithi090 ceph-mon[44434]: pgmap v1438: 17 pgs: 17 active+clean; 2.1 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:28:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:05 smithi088 ceph-mon[44699]: pgmap v1438: 17 pgs: 17 active+clean; 2.1 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:28:05.580 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:28:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:05.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:28:05.607 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:28:05.640 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:28:05.666 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:28:05.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:28:05.708 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:28:05.724 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:05.733 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:28:05.763 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:28:05.766 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:28:05.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:05.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:28:05.804 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:28:05.835 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:28:05.867 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:28:05.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:28:05.894 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:28:05.914 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:05.933 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:28:05.945 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:28:05.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:05.951 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:28:05.968 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:28:05.969 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:28:05.970 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:28:05.970 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:28:05.970 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:28:05.970 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:28:05.970 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:28:05.970 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:28:05.970 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:28:05.970 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:28:05.970 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:28:05.970 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:28:05.970 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:28:06.051 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:06.109 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:06.140 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:28:06.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:06 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2024154382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:28:06.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:06 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1430391293' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:28:06.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:06 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:06.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:06 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:06.147 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:06.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:06.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:06.218 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:28:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:28:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-05-15T23:28:06.218 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:28:06.256 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:28:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:06 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2024154382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:28:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:06 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1430391293' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:28:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:06 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:06.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:06 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:06.304 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:06.386 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:28:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:06.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:28:06.489 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:06.524 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:06 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2024154382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:28:06.524 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:06 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1430391293' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:28:06.524 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:06 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:06.524 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:06 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:06.526 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:06.562 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:06.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:06.667 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:28:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:06.710 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10] out_osds: [2, 0, 6, 8] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:28:06.711 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:28:06.711 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-05-15T23:28:06.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd in 2 2024-05-15T23:28:06.725 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:28:06.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:06.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:28:06.830 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:28:06.871 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:07.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:07 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:28:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:07 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1430391293' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:28:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:07 smithi090 ceph-mon[44434]: osdmap e536: 12 total, 12 up, 8 in 2024-05-15T23:28:07.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:07 smithi090 ceph-mon[44434]: pgmap v1440: 17 pgs: 17 active+clean; 1.8 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:28:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:07 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:28:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:07 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1430391293' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:28:07.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:07 smithi088 ceph-mon[44699]: osdmap e536: 12 total, 12 up, 8 in 2024-05-15T23:28:07.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:07 smithi088 ceph-mon[44699]: pgmap v1440: 17 pgs: 17 active+clean; 1.8 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:28:07.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:07 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:28:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:07 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1430391293' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:28:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:07 smithi084 ceph-mon[39784]: osdmap e536: 12 total, 12 up, 8 in 2024-05-15T23:28:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:07 smithi084 ceph-mon[39784]: pgmap v1440: 17 pgs: 17 active+clean; 1.8 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-05-15T23:28:07.737 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:28:07.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:07.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:28:07.843 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:28:07.880 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:07.905 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:28:07.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:07.911 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:28:08.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:08.047 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:08.067 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:08.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:08.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:08.172 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:28:08.203 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:09.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:09 smithi090 ceph-mon[44434]: pgmap v1441: 17 pgs: 17 active+clean; 1.8 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:28:09.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:09 smithi088 ceph-mon[44699]: pgmap v1441: 17 pgs: 17 active+clean; 1.8 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:28:09.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:09 smithi084 ceph-mon[39784]: pgmap v1441: 17 pgs: 17 active+clean; 1.8 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:28:09.761 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:28:11.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:11 smithi084 ceph-mon[39784]: pgmap v1442: 17 pgs: 17 active+clean; 1.6 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:28:11.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:11 smithi090 ceph-mon[44434]: pgmap v1442: 17 pgs: 17 active+clean; 1.6 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:28:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:11 smithi088 ceph-mon[44699]: pgmap v1442: 17 pgs: 17 active+clean; 1.6 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-05-15T23:28:11.762 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:28:11.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:11.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:28:11.870 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:28:11.902 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:12.108 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:28:12.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:12.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:28:12.123 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T23:28:12.147 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:12.172 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:12.216 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:12.247 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:12.274 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:12.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:12.378 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:12.412 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:12.444 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:12.453 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:12.554 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:12.600 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:12.637 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:12.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:12.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:12.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:12.804 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:12.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:12.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:12.910 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:28:12.972 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:13.009 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:28:13.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:13.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:28:13.116 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:28:13.156 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:13.310 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:28:13.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:13 smithi084 ceph-mon[39784]: pgmap v1443: 17 pgs: 17 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:28:13.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:13.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:28:13.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:13 smithi088 ceph-mon[44699]: pgmap v1443: 17 pgs: 17 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:28:13.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:13.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:13 smithi090 ceph-mon[44434]: pgmap v1443: 17 pgs: 17 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:28:13.445 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:13.473 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:13.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:13.478 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:13.579 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:28:13.611 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:13.641 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:28:13.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:13.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:28:13.749 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:13.760 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:28:13.786 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:13.814 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:13.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:13.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:13.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:28:13.954 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:13.977 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:28:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:13.982 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:28:14.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:28:14.137 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:14.654 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:14.696 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:28:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:14.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:28:14.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:28:14.837 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:14.867 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:28:14.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:14.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:28:14.976 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:15.040 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:15.065 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:15.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:15.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:15.136 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:15 smithi090 ceph-mon[44434]: pgmap v1444: 17 pgs: 17 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:28:15.177 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:28:15.221 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:15.255 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:28:15.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:15.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:28:15.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:15 smithi088 ceph-mon[44699]: pgmap v1444: 17 pgs: 17 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:28:15.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:15 smithi084 ceph-mon[39784]: pgmap v1444: 17 pgs: 17 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:28:15.362 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:28:15.398 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:15.424 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:28:15.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:15.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:28:15.435 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:15.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:28:15.590 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:15.637 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:28:15.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:15.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:28:15.750 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:28:15.791 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:15.877 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:28:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:15.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:28:15.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:28:16.011 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:16.035 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:28:16.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:16.038 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:28:16.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:16 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:28:16.047+0000 7fa0938b7700 -1 mon.a@0(leader).osd e536 definitely_dead 0 2024-05-15T23:28:16.139 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:28:16.162 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:28:16.169 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.2. 2024-05-15T23:28:16.190 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:16.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:16 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2629803073' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:28:16.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:16 smithi090 ceph-mon[44434]: from='client.26593 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:28:16.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:16 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3613634753' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-05-15T23:28:16.392 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:16 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2629803073' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:28:16.393 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:16 smithi084 ceph-mon[39784]: from='client.26593 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:28:16.393 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:16 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3613634753' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-05-15T23:28:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:16 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2629803073' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:28:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:16 smithi088 ceph-mon[44699]: from='client.26593 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:28:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:16 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3613634753' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-05-15T23:28:16.842 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:28:16.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:16.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:28:16.946 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:28:17.016 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:17.037 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:28:17.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:17.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:28:17.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:28:17.174 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:17.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:17 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:28:17.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:17 smithi088 ceph-mon[44699]: from='client.26593 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:28:17.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:17 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3613634753' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-05-15T23:28:17.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:17 smithi088 ceph-mon[44699]: osdmap e537: 12 total, 12 up, 9 in 2024-05-15T23:28:17.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:17 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:28:17.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:17 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:28:17.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:17 smithi088 ceph-mon[44699]: pgmap v1446: 17 pgs: 1 peering, 16 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:28:17.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:17 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:28:17.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:17 smithi084 ceph-mon[39784]: from='client.26593 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:28:17.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:17 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3613634753' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-05-15T23:28:17.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:17 smithi084 ceph-mon[39784]: osdmap e537: 12 total, 12 up, 9 in 2024-05-15T23:28:17.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:17 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:28:17.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:17 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:28:17.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:17 smithi084 ceph-mon[39784]: pgmap v1446: 17 pgs: 1 peering, 16 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:28:17.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:17 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:28:17.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:17 smithi090 ceph-mon[44434]: from='client.26593 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:28:17.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:17 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3613634753' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-05-15T23:28:17.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:17 smithi090 ceph-mon[44434]: osdmap e537: 12 total, 12 up, 9 in 2024-05-15T23:28:17.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:17 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:28:17.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:17 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:28:17.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:17 smithi090 ceph-mon[44434]: pgmap v1446: 17 pgs: 1 peering, 16 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-05-15T23:28:17.925 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:28:17.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:17.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:28:18.029 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:28:18.059 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:18.084 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:28:18.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:18.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:28:18.188 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:28:18.216 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:18.236 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:28:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:18.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:28:18.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:18 smithi088 ceph-mon[44699]: osdmap e538: 12 total, 12 up, 9 in 2024-05-15T23:28:18.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:28:18.373 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:18.396 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:28:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:18.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:28:18.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:18 smithi084 ceph-mon[39784]: osdmap e538: 12 total, 12 up, 9 in 2024-05-15T23:28:18.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:28:18.527 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:18 smithi090 ceph-mon[44434]: osdmap e538: 12 total, 12 up, 9 in 2024-05-15T23:28:18.529 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:18.549 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:28:18.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:18.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:28:18.653 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:28:18.686 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:18.821 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:28:18.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:18.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:28:18.864 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-05-15T23:28:18.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:28:18.966 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:19.150 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:28:19.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:19.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:28:19.262 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:19.294 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:19.327 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:19.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:19.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:19.410 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:19 smithi084 ceph-mon[39784]: 6.1 scrub starts 2024-05-15T23:28:19.410 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:19 smithi084 ceph-mon[39784]: 6.1 scrub ok 2024-05-15T23:28:19.410 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:19 smithi084 ceph-mon[39784]: 6.e scrub starts 2024-05-15T23:28:19.410 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:19 smithi084 ceph-mon[39784]: 6.e scrub ok 2024-05-15T23:28:19.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:19 smithi084 ceph-mon[39784]: osdmap e539: 12 total, 12 up, 9 in 2024-05-15T23:28:19.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:19 smithi084 ceph-mon[39784]: 6.f starting backfill to osd.2 from (0'0,0'0] MAX to 536'16936 2024-05-15T23:28:19.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:19 smithi084 ceph-mon[39784]: 6.9 starting backfill to osd.2 from (0'0,0'0] MAX to 536'15754 2024-05-15T23:28:19.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:19 smithi084 ceph-mon[39784]: 6.0 starting backfill to osd.2 from (0'0,0'0] MAX to 536'15785 2024-05-15T23:28:19.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:19 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:28:19.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:19 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:28:19.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:19 smithi084 ceph-mon[39784]: pgmap v1449: 17 pgs: 1 peering, 16 active+clean; 1.1 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 942 op/s 2024-05-15T23:28:19.436 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:28:19.499 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:19.518 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:28:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:28:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:28:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:28:19.519 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:28:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:19 smithi090 ceph-mon[44434]: 6.1 scrub starts 2024-05-15T23:28:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:19 smithi090 ceph-mon[44434]: 6.1 scrub ok 2024-05-15T23:28:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:19 smithi090 ceph-mon[44434]: 6.e scrub starts 2024-05-15T23:28:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:19 smithi090 ceph-mon[44434]: 6.e scrub ok 2024-05-15T23:28:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:19 smithi090 ceph-mon[44434]: osdmap e539: 12 total, 12 up, 9 in 2024-05-15T23:28:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:19 smithi090 ceph-mon[44434]: 6.f starting backfill to osd.2 from (0'0,0'0] MAX to 536'16936 2024-05-15T23:28:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:19 smithi090 ceph-mon[44434]: 6.9 starting backfill to osd.2 from (0'0,0'0] MAX to 536'15754 2024-05-15T23:28:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:19 smithi090 ceph-mon[44434]: 6.0 starting backfill to osd.2 from (0'0,0'0] MAX to 536'15785 2024-05-15T23:28:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:19 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:28:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:19 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:28:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:19 smithi090 ceph-mon[44434]: pgmap v1449: 17 pgs: 1 peering, 16 active+clean; 1.1 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 942 op/s 2024-05-15T23:28:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:19 smithi088 ceph-mon[44699]: 6.1 scrub starts 2024-05-15T23:28:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:19 smithi088 ceph-mon[44699]: 6.1 scrub ok 2024-05-15T23:28:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:19 smithi088 ceph-mon[44699]: 6.e scrub starts 2024-05-15T23:28:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:19 smithi088 ceph-mon[44699]: 6.e scrub ok 2024-05-15T23:28:19.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:19 smithi088 ceph-mon[44699]: osdmap e539: 12 total, 12 up, 9 in 2024-05-15T23:28:19.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:19 smithi088 ceph-mon[44699]: 6.f starting backfill to osd.2 from (0'0,0'0] MAX to 536'16936 2024-05-15T23:28:19.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:19 smithi088 ceph-mon[44699]: 6.9 starting backfill to osd.2 from (0'0,0'0] MAX to 536'15754 2024-05-15T23:28:19.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:19 smithi088 ceph-mon[44699]: 6.0 starting backfill to osd.2 from (0'0,0'0] MAX to 536'15785 2024-05-15T23:28:19.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:19 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:28:19.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:19 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:28:19.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:19 smithi088 ceph-mon[44699]: pgmap v1449: 17 pgs: 1 peering, 16 active+clean; 1.1 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 942 op/s 2024-05-15T23:28:19.713 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:28:19.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:19.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:28:19.820 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:28:19.852 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:19.883 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:28:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:19.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:28:19.995 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:28:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:20.063 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:28:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:20.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:28:20.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:20.208 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:20.245 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:20.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:20.300 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T23:28:20.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:28:20.390 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:20.422 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:28:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:20.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:28:20.527 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:28:20.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:20 smithi090 ceph-mon[44434]: osdmap e540: 12 total, 12 up, 9 in 2024-05-15T23:28:20.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:20 smithi090 ceph-mon[44434]: 6.a scrub starts 2024-05-15T23:28:20.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:20 smithi090 ceph-mon[44434]: 6.a scrub ok 2024-05-15T23:28:20.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:20 smithi088 ceph-mon[44699]: osdmap e540: 12 total, 12 up, 9 in 2024-05-15T23:28:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:20 smithi088 ceph-mon[44699]: 6.a scrub starts 2024-05-15T23:28:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:20 smithi088 ceph-mon[44699]: 6.a scrub ok 2024-05-15T23:28:20.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:20 smithi084 ceph-mon[39784]: osdmap e540: 12 total, 12 up, 9 in 2024-05-15T23:28:20.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:20 smithi084 ceph-mon[39784]: 6.a scrub starts 2024-05-15T23:28:20.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:20 smithi084 ceph-mon[39784]: 6.a scrub ok 2024-05-15T23:28:20.584 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:20.750 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:28:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:20.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:28:20.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:28:20.900 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:21.020 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:28:21.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:21.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:28:21.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:28:21.187 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:21.231 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:28:21.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:21.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:28:21.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:21.398 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:21.400 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:21.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:21.402 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:21.429 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:28:21.481 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:28:21.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:28:21.517 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:28:21.547 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:21 smithi084 ceph-mon[39784]: 6.8 scrub starts 2024-05-15T23:28:21.547 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:21 smithi084 ceph-mon[39784]: 6.8 scrub ok 2024-05-15T23:28:21.547 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:21 smithi084 ceph-mon[39784]: pgmap v1451: 17 pgs: 2 active+remapped+backfilling, 1 peering, 14 active+clean; 1010 MiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 915 op/s; 1252/32304 objects misplaced (3.876%); 8.5 MiB/s, 135 objects/s recovering 2024-05-15T23:28:21.550 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:21.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:21 smithi090 ceph-mon[44434]: 6.8 scrub starts 2024-05-15T23:28:21.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:21 smithi090 ceph-mon[44434]: 6.8 scrub ok 2024-05-15T23:28:21.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:21 smithi090 ceph-mon[44434]: pgmap v1451: 17 pgs: 2 active+remapped+backfilling, 1 peering, 14 active+clean; 1010 MiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 915 op/s; 1252/32304 objects misplaced (3.876%); 8.5 MiB/s, 135 objects/s recovering 2024-05-15T23:28:21.552 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:28:21.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:21 smithi088 ceph-mon[44699]: 6.8 scrub starts 2024-05-15T23:28:21.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:21 smithi088 ceph-mon[44699]: 6.8 scrub ok 2024-05-15T23:28:21.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:21 smithi088 ceph-mon[44699]: pgmap v1451: 17 pgs: 2 active+remapped+backfilling, 1 peering, 14 active+clean; 1010 MiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 915 op/s; 1252/32304 objects misplaced (3.876%); 8.5 MiB/s, 135 objects/s recovering 2024-05-15T23:28:21.596 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:28:21.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:28:21.664 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:28:21.683 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:28:21.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:21.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:28:21.700 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:28:21.734 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:28:21.770 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:28:21.789 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:21.798 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:28:21.831 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:28:21.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:28:21.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:28:21.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:28:21.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:28:21.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:28:21.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:28:21.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:28:21.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:28:21.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:28:21.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:28:21.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:28:21.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:28:21.840 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:21.873 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:21.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:21.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:28:22.011 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:22.049 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:28:22.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:22.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:28:22.158 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:28:22.208 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:22.218 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3144956991' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:28:22.218 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:22 smithi088 ceph-mon[44699]: 6.d scrub starts 2024-05-15T23:28:22.218 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.218 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.219 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:22 smithi088 ceph-mon[44699]: 6.d scrub ok 2024-05-15T23:28:22.219 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.219 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.219 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.219 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3144956991' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:28:22.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:22 smithi090 ceph-mon[44434]: 6.d scrub starts 2024-05-15T23:28:22.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:22 smithi090 ceph-mon[44434]: 6.d scrub ok 2024-05-15T23:28:22.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.569 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:28:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:22.576 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3144956991' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:28:22.576 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:22 smithi084 ceph-mon[39784]: 6.d scrub starts 2024-05-15T23:28:22.577 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.577 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.577 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:22 smithi084 ceph-mon[39784]: 6.d scrub ok 2024-05-15T23:28:22.577 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.577 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.577 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.577 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:22.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:28:22.678 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:28:22.718 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:22.747 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:28:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:22.752 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:28:22.790 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:28:22.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:22.889 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:22.913 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:22.918 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:23.019 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:28:23.086 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:23.115 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:28:23.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:23.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:28:23.221 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:23.282 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:23.305 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:23.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:23.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:23 smithi084 ceph-mon[39784]: pgmap v1452: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 875 MiB data, 5.7 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 670 op/s; 1796/27986 objects misplaced (6.417%); 11 MiB/s, 173 objects/s recovering 2024-05-15T23:28:23.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:23.451 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:23.481 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:23.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:23.486 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.487 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.487 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.487 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.487 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.487 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.487 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:23 smithi090 ceph-mon[44434]: pgmap v1452: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 875 MiB data, 5.7 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 670 op/s; 1796/27986 objects misplaced (6.417%); 11 MiB/s, 173 objects/s recovering 2024-05-15T23:28:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:23 smithi088 ceph-mon[44699]: pgmap v1452: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 875 MiB data, 5.7 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 670 op/s; 1796/27986 objects misplaced (6.417%); 11 MiB/s, 173 objects/s recovering 2024-05-15T23:28:23.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:23.630 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:23.664 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:23.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:23.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:23.756 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:23.770 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:28:23.802 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:23.865 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10, 2] out_osds: [0, 6, 8] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:28:23.866 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:28:23.866 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-05-15T23:28:23.866 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump -f json-pretty 2024-05-15T23:28:24.025 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:28:24.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:24.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:28:24.133 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:28:24.195 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:24.253 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:28:24.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:24.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:28:24.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:28:24.420 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:24.540 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:28:24.540 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:28:24.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:24 smithi090 ceph-mon[44434]: 6.3 scrub starts 2024-05-15T23:28:24.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:24 smithi090 ceph-mon[44434]: 6.3 scrub ok 2024-05-15T23:28:24.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:24 smithi084 ceph-mon[39784]: 6.3 scrub starts 2024-05-15T23:28:24.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:24 smithi084 ceph-mon[39784]: 6.3 scrub ok 2024-05-15T23:28:24.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:24 smithi088 ceph-mon[44699]: 6.3 scrub starts 2024-05-15T23:28:24.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:24 smithi088 ceph-mon[44699]: 6.3 scrub ok 2024-05-15T23:28:25.150 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:28:25.179 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:28:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:25.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:28:25.283 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:25.310 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:25.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:25.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:25.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:28:25.463 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:25.483 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:28:25.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:25.486 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:28:25.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:25 smithi090 ceph-mon[44434]: osdmap e541: 12 total, 12 up, 9 in 2024-05-15T23:28:25.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:25 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:28:25.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:25 smithi090 ceph-mon[44434]: pgmap v1454: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 875 MiB data, 5.8 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 471 op/s; 1796/27986 objects misplaced (6.417%); 9.4 MiB/s, 150 objects/s recovering 2024-05-15T23:28:25.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:25 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:25.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:25 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:25.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:25 smithi088 ceph-mon[44699]: osdmap e541: 12 total, 12 up, 9 in 2024-05-15T23:28:25.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:25 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:28:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:25 smithi088 ceph-mon[44699]: pgmap v1454: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 875 MiB data, 5.8 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 471 op/s; 1796/27986 objects misplaced (6.417%); 9.4 MiB/s, 150 objects/s recovering 2024-05-15T23:28:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:25 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:25.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:25 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:25.587 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:25.613 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:25.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:25 smithi084 ceph-mon[39784]: osdmap e541: 12 total, 12 up, 9 in 2024-05-15T23:28:25.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:25 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:28:25.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:25 smithi084 ceph-mon[39784]: pgmap v1454: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 875 MiB data, 5.8 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 471 op/s; 1796/27986 objects misplaced (6.417%); 9.4 MiB/s, 150 objects/s recovering 2024-05-15T23:28:25.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:25 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:25.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:25 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:25.632 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:25.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:25.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:25.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:28:25.768 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:25.786 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:28:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:25.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:28:25.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T23:28:25.889 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:28:25.915 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:25.934 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:28:25.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:25.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:28:26.038 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:26.065 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:28:26.084 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:26.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:26.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:26.188 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:26.214 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:26.233 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:26.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:26.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:26.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:26.368 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:26.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:26 smithi090 ceph-mon[44434]: osdmap e542: 12 total, 12 up, 9 in 2024-05-15T23:28:26.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:26 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:26.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:26 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:26.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:26 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:26.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:26 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:26.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:26 smithi090 ceph-mon[44434]: osdmap e543: 12 total, 12 up, 9 in 2024-05-15T23:28:26.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:26 smithi084 ceph-mon[39784]: osdmap e542: 12 total, 12 up, 9 in 2024-05-15T23:28:26.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:26 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:26.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:26 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:26.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:26 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:26.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:26 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:26.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:26 smithi084 ceph-mon[39784]: osdmap e543: 12 total, 12 up, 9 in 2024-05-15T23:28:26.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:26 smithi088 ceph-mon[44699]: osdmap e542: 12 total, 12 up, 9 in 2024-05-15T23:28:26.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:26 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:26.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:26 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:26.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:26 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:26.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:26 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:26.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:26 smithi088 ceph-mon[44699]: osdmap e543: 12 total, 12 up, 9 in 2024-05-15T23:28:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:26.889 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:27.331 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:27.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:27.441 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:28:27.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:28:27.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:28:27.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:27 smithi084 ceph-mon[39784]: 6.b scrub starts 2024-05-15T23:28:27.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:27 smithi084 ceph-mon[39784]: 6.b scrub ok 2024-05-15T23:28:27.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:27.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:27.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:28:27.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:28:27.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:27.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:27 smithi084 ceph-mon[39784]: pgmap v1457: 17 pgs: 2 peering, 1 active+remapped+backfilling, 14 active+clean; 688 MiB data, 5.4 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 858 op/s; 580/21990 objects misplaced (2.638%); 15 MiB/s, 234 objects/s recovering 2024-05-15T23:28:27.501 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:28:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:28:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:27 smithi090 ceph-mon[44434]: 6.b scrub starts 2024-05-15T23:28:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:27 smithi090 ceph-mon[44434]: 6.b scrub ok 2024-05-15T23:28:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:28:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:28:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:27.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:27 smithi090 ceph-mon[44434]: pgmap v1457: 17 pgs: 2 peering, 1 active+remapped+backfilling, 14 active+clean; 688 MiB data, 5.4 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 858 op/s; 580/21990 objects misplaced (2.638%); 15 MiB/s, 234 objects/s recovering 2024-05-15T23:28:27.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:28:27.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:28:27.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:27 smithi088 ceph-mon[44699]: 6.b scrub starts 2024-05-15T23:28:27.573 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:27 smithi088 ceph-mon[44699]: 6.b scrub ok 2024-05-15T23:28:27.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:27.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:27.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:28:27.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:28:27.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:28:27.574 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:27 smithi088 ceph-mon[44699]: pgmap v1457: 17 pgs: 2 peering, 1 active+remapped+backfilling, 14 active+clean; 688 MiB data, 5.4 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 858 op/s; 580/21990 objects misplaced (2.638%); 15 MiB/s, 234 objects/s recovering 2024-05-15T23:28:28.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:28.067 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:28:28.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:28:28.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:28.201 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:28.280 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:28:28.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:28 smithi090 ceph-mon[44434]: 6.9 scrub starts 2024-05-15T23:28:28.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:28 smithi090 ceph-mon[44434]: 6.9 scrub ok 2024-05-15T23:28:28.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3278110444' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:28:28.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:28 smithi084 ceph-mon[39784]: 6.9 scrub starts 2024-05-15T23:28:28.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:28 smithi084 ceph-mon[39784]: 6.9 scrub ok 2024-05-15T23:28:28.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3278110444' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:28:28.401 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:28.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:28.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:28.507 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:28.544 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:28 smithi088 ceph-mon[44699]: 6.9 scrub starts 2024-05-15T23:28:28.544 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:28 smithi088 ceph-mon[44699]: 6.9 scrub ok 2024-05-15T23:28:28.544 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3278110444' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:28:28.546 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:28.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:28.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:28.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:28:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:28:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:28.759 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:28:28.859 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:28.903 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:28.950 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:28.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:29.042 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:28:29.042 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:28:29.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:28:29.089 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:29.250 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:29 smithi088 ceph-mon[44699]: 6.c scrub starts 2024-05-15T23:28:29.250 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:29 smithi088 ceph-mon[44699]: 6.c scrub ok 2024-05-15T23:28:29.251 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:29 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:28:29.251 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:29 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:28:29.251 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:29 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3278110444' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:28:29.251 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:29 smithi088 ceph-mon[44699]: osdmap e544: 12 total, 12 up, 9 in 2024-05-15T23:28:29.251 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:29 smithi088 ceph-mon[44699]: pgmap v1459: 17 pgs: 2 peering, 1 active+remapped+backfilling, 14 active+clean; 646 MiB data, 5.3 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 757 op/s; 580/20646 objects misplaced (2.809%); 13 MiB/s, 213 objects/s recovering 2024-05-15T23:28:29.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:29 smithi084 ceph-mon[39784]: 6.c scrub starts 2024-05-15T23:28:29.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:29 smithi084 ceph-mon[39784]: 6.c scrub ok 2024-05-15T23:28:29.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:29 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:28:29.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:29 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:28:29.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:29 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3278110444' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:28:29.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:29 smithi084 ceph-mon[39784]: osdmap e544: 12 total, 12 up, 9 in 2024-05-15T23:28:29.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:29 smithi084 ceph-mon[39784]: pgmap v1459: 17 pgs: 2 peering, 1 active+remapped+backfilling, 14 active+clean; 646 MiB data, 5.3 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 757 op/s; 580/20646 objects misplaced (2.809%); 13 MiB/s, 213 objects/s recovering 2024-05-15T23:28:29.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:29 smithi090 ceph-mon[44434]: 6.c scrub starts 2024-05-15T23:28:29.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:29 smithi090 ceph-mon[44434]: 6.c scrub ok 2024-05-15T23:28:29.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:29 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:28:29.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:29 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:28:29.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:29 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3278110444' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:28:29.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:29 smithi090 ceph-mon[44434]: osdmap e544: 12 total, 12 up, 9 in 2024-05-15T23:28:29.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:29 smithi090 ceph-mon[44434]: pgmap v1459: 17 pgs: 2 peering, 1 active+remapped+backfilling, 14 active+clean; 646 MiB data, 5.3 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 757 op/s; 580/20646 objects misplaced (2.809%); 13 MiB/s, 213 objects/s recovering 2024-05-15T23:28:29.848 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:28:29.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:29.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:28:29.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:28:29.983 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:30.381 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T23:28:30.445 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:28:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:30.449 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:28:30.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:30.582 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:30.607 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:30.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:30.614 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:30 smithi090 ceph-mon[44434]: 6.f scrub starts 2024-05-15T23:28:30.614 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:30 smithi090 ceph-mon[44434]: 6.f scrub ok 2024-05-15T23:28:30.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:30.716 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:28:30.751 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:30 smithi088 ceph-mon[44699]: 6.f scrub starts 2024-05-15T23:28:30.751 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:30 smithi088 ceph-mon[44699]: 6.f scrub ok 2024-05-15T23:28:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:28:30.753 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:30.787 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:28:30.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:30.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:28:30.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:30 smithi084 ceph-mon[39784]: 6.f scrub starts 2024-05-15T23:28:30.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:30 smithi084 ceph-mon[39784]: 6.f scrub ok 2024-05-15T23:28:30.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:28:30.925 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:30.976 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:28:30.976 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:28:30.976 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:28:30.976 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:28:30.989 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:28:31.010 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:28:31.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:31.017 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:28:31.021 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:28:31.062 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:28:31.100 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:28:31.117 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:28:31.140 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:28:31.176 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:28:31.177 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:31.216 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:28:31.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:31.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:28:31.229 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:28:31.251 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:28:31.252 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:28:31.252 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 544, 2024-05-15T23:28:31.252 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "3999fb7e-130e-11ef-bc99-c7b262605968", 2024-05-15T23:28:31.252 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-05-15T22:56:41.212493+0000", 2024-05-15T23:28:31.252 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-05-15T23:28:28.246435+0000", 2024-05-15T23:28:31.252 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-05-15T23:26:58.424794+0000", 2024-05-15T23:28:31.252 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-05-15T23:28:16.134275+0000", 2024-05-15T23:28:31.252 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-15T23:28:31.252 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5799936, 2024-05-15T23:28:31.252 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-05-15T23:28:31.252 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-05-15T23:28:31.252 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-05-15T23:28:31.253 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-05-15T23:28:31.253 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-05-15T23:28:31.253 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:28:31.253 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 30, 2024-05-15T23:28:31.253 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-05-15T23:28:31.253 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-15T23:28:31.253 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-15T23:28:31.253 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-05-15T23:28:31.253 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 6, 2024-05-15T23:28:31.253 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 12, 2024-05-15T23:28:31.253 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-05-15T23:28:31.253 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "jewel", 2024-05-15T23:28:31.253 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-05-15T23:28:31.253 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-05-15T23:28:31.254 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-05-15T23:28:31.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.254 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-05-15T23:28:31.254 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-05-15T23:28:31.254 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-05-15T22:59:38.603575+0000", 2024-05-15T23:28:31.254 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-05-15T23:28:31.254 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-05-15T23:28:31.254 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-05-15T23:28:31.254 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2024-05-15T23:28:31.254 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-05-15T23:28:31.254 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-05-15T23:28:31.254 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-05-15T23:28:31.254 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-05-15T23:28:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T23:28:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T23:28:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-05-15T23:28:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-05-15T23:28:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 1, 2024-05-15T23:28:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 1, 2024-05-15T23:28:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 1, 2024-05-15T23:28:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 1, 2024-05-15T23:28:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 1, 2024-05-15T23:28:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-05-15T23:28:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-05-15T23:28:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-05-15T23:28:31.255 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-05-15T23:28:31.256 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-05-15T23:28:31.256 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-05-15T23:28:31.256 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-05-15T23:28:31.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.256 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "16", 2024-05-15T23:28:31.256 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-05-15T23:28:31.256 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T23:28:31.256 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T23:28:31.256 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-05-15T23:28:31.256 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-05-15T23:28:31.256 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-05-15T23:28:31.256 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-05-15T23:28:31.257 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-05-15T23:28:31.257 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-05-15T23:28:31.257 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-05-15T23:28:31.257 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-05-15T23:28:31.257 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-05-15T23:28:31.257 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-05-15T23:28:31.257 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-05-15T23:28:31.257 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-05-15T23:28:31.257 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-05-15T23:28:31.257 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-05-15T23:28:31.257 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-05-15T23:28:31.257 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T23:28:31.257 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T23:28:31.257 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T23:28:31.258 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-05-15T23:28:31.258 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-05-15T23:28:31.258 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-05-15T23:28:31.258 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-05-15T23:28:31.258 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-05-15T23:28:31.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.258 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-05-15T23:28:31.258 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-05-15T23:28:31.258 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-05-15T23:28:31.258 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-05-15T23:28:31.258 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-05-15T23:28:31.258 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T23:28:31.259 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-05-15T23:28:31.259 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-05-15T23:28:31.259 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-05-15T23:28:31.259 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-05-15T23:28:31.259 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-05-15T23:28:31.259 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-05-15T23:28:31.259 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-05-15T23:28:31.259 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-05-15T23:28:31.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.259 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-05-15T23:28:31.259 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-05-15T23:28:31.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.259 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-05-15T23:28:31.259 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-05-15T23:28:31.260 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 11.760000228881836, 2024-05-15T23:28:31.260 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 11.760000228881836, 2024-05-15T23:28:31.260 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.17000000178813934, 2024-05-15T23:28:31.261 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-05-15T23:28:31.261 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-05-15T23:28:31.261 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.96207165718078613, 2024-05-15T23:28:31.261 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 0.95999997854232788, 2024-05-15T23:28:31.261 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.95999997854232788 2024-05-15T23:28:31.261 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.261 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 6, 2024-05-15T23:28:31.261 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_4", 2024-05-15T23:28:31.261 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-05-15T23:24:53.409510+0000", 2024-05-15T23:28:31.261 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-05-15T23:28:31.262 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-05-15T23:28:31.262 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-05-15T23:28:31.262 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2024-05-15T23:28:31.262 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-05-15T23:28:31.262 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-05-15T23:28:31.262 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-05-15T23:28:31.262 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-05-15T23:28:31.262 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T23:28:31.262 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T23:28:31.262 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-05-15T23:28:31.262 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-05-15T23:28:31.262 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 16, 2024-05-15T23:28:31.263 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 16, 2024-05-15T23:28:31.263 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 16, 2024-05-15T23:28:31.263 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 16, 2024-05-15T23:28:31.263 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 16, 2024-05-15T23:28:31.263 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-05-15T23:28:31.263 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-05-15T23:28:31.263 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-05-15T23:28:31.263 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-05-15T23:28:31.263 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-05-15T23:28:31.263 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-05-15T23:28:31.263 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-05-15T23:28:31.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.263 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "482", 2024-05-15T23:28:31.263 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-05-15T23:28:31.264 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T23:28:31.264 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T23:28:31.264 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-05-15T23:28:31.264 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-05-15T23:28:31.264 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-05-15T23:28:31.264 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-05-15T23:28:31.264 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-05-15T23:28:31.264 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-05-15T23:28:31.264 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-05-15T23:28:31.264 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-05-15T23:28:31.264 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-05-15T23:28:31.264 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-05-15T23:28:31.264 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-05-15T23:28:31.264 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-05-15T23:28:31.264 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-05-15T23:28:31.265 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-05-15T23:28:31.265 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-05-15T23:28:31.265 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T23:28:31.265 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T23:28:31.265 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T23:28:31.265 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-05-15T23:28:31.265 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-05-15T23:28:31.265 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-05-15T23:28:31.265 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-05-15T23:28:31.265 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-05-15T23:28:31.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.265 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-05-15T23:28:31.265 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-05-15T23:28:31.265 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-05-15T23:28:31.266 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-05-15T23:28:31.266 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-05-15T23:28:31.266 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T23:28:31.266 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-05-15T23:28:31.266 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-05-15T23:28:31.266 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-05-15T23:28:31.266 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-05-15T23:28:31.266 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-05-15T23:28:31.266 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-05-15T23:28:31.266 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-05-15T23:28:31.266 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-05-15T23:28:31.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.266 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-05-15T23:28:31.266 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-05-15T23:28:31.267 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 3, 2024-05-15T23:28:31.267 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 3, 2024-05-15T23:28:31.267 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.75, 2024-05-15T23:28:31.267 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.25, 2024-05-15T23:28:31.267 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.25, 2024-05-15T23:28:31.267 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.96207165718078613, 2024-05-15T23:28:31.267 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 0.95999997854232788, 2024-05-15T23:28:31.267 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.95999997854232788 2024-05-15T23:28:31.267 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.267 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.267 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:28:31.267 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-05-15T23:28:31.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.267 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-05-15T23:28:31.268 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "80159cc8-1b6c-4aa6-a114-966786d7be1e", 2024-05-15T23:28:31.268 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:28:31.268 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-05-15T23:28:31.268 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-05-15T23:28:31.268 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:28:31.268 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 8, 2024-05-15T23:28:31.268 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 355, 2024-05-15T23:28:31.268 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 397, 2024-05-15T23:28:31.269 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 283, 2024-05-15T23:28:31.269 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 356, 2024-05-15T23:28:31.269 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:28:31.269 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:28:31.269 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.269 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.269 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-05-15T23:28:31.269 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:28:31.269 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.269 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.269 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:28:31.269 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.270 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.270 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-05-15T23:28:31.270 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:28:31.270 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.270 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.270 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:28:31.270 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.270 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.270 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-05-15T23:28:31.271 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:28:31.271 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.271 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.271 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:28:31.271 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.271 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.271 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-05-15T23:28:31.271 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:28:31.271 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.271 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.272 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/3077815271", 2024-05-15T23:28:31.272 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/3077815271", 2024-05-15T23:28:31.272 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/3077815271", 2024-05-15T23:28:31.272 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/3077815271", 2024-05-15T23:28:31.272 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:28:31.272 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:28:31.272 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:28:31.272 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.272 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-05-15T23:28:31.272 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "9c2c76c9-ce0f-4c2d-9d96-c87a10187607", 2024-05-15T23:28:31.272 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:28:31.272 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:28:31.273 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:28:31.273 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:28:31.273 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:28:31.273 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:28:31.273 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-05-15T23:28:31.273 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 477, 2024-05-15T23:28:31.273 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:28:31.273 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:28:31.273 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:28:31.273 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.273 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.273 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-05-15T23:28:31.273 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:28:31.274 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.274 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.274 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:28:31.274 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.274 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.274 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-05-15T23:28:31.274 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:28:31.274 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.274 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:28:31.275 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.276 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.276 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/2140830229", 2024-05-15T23:28:31.276 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/2140830229", 2024-05-15T23:28:31.276 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/2140830229", 2024-05-15T23:28:31.276 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/2140830229", 2024-05-15T23:28:31.276 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:28:31.276 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:28:31.276 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:28:31.276 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.276 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-05-15T23:28:31.277 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "e8514daa-b5f9-4f2f-a432-53f4a4eb86bc", 2024-05-15T23:28:31.277 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:28:31.277 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:28:31.277 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:28:31.277 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:28:31.277 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 20, 2024-05-15T23:28:31.277 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 261, 2024-05-15T23:28:31.277 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 291, 2024-05-15T23:28:31.277 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 541, 2024-05-15T23:28:31.277 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 262, 2024-05-15T23:28:31.277 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:28:31.277 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:28:31.277 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.278 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.278 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-05-15T23:28:31.278 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:28:31.278 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.278 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.278 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:28:31.278 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.278 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.278 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-05-15T23:28:31.278 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:28:31.279 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.279 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.279 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:28:31.279 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.279 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.282 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-05-15T23:28:31.282 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:28:31.282 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.282 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.283 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:28:31.283 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.283 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.283 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-05-15T23:28:31.283 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:28:31.283 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.283 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.283 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/941507493", 2024-05-15T23:28:31.283 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/941507493", 2024-05-15T23:28:31.283 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/941507493", 2024-05-15T23:28:31.284 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/941507493", 2024-05-15T23:28:31.284 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:28:31.284 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:28:31.284 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:28:31.284 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.284 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-05-15T23:28:31.284 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "5d9965f9-c0b2-48ce-869c-582990985623", 2024-05-15T23:28:31.284 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:28:31.284 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:28:31.284 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:28:31.284 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:28:31.285 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:28:31.285 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:28:31.285 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 26, 2024-05-15T23:28:31.285 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 542, 2024-05-15T23:28:31.285 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:28:31.285 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:28:31.285 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:28:31.285 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.285 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.285 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-05-15T23:28:31.285 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:28:31.285 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.286 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.286 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:28:31.286 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.286 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.286 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-05-15T23:28:31.286 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:28:31.286 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.286 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.286 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:28:31.286 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.287 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.287 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-05-15T23:28:31.287 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:28:31.287 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.287 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.287 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:28:31.287 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.287 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.287 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-05-15T23:28:31.287 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:28:31.288 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.288 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.288 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2204039828", 2024-05-15T23:28:31.288 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2204039828", 2024-05-15T23:28:31.288 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2204039828", 2024-05-15T23:28:31.288 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2204039828", 2024-05-15T23:28:31.288 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:28:31.288 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:28:31.288 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:28:31.288 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.289 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-05-15T23:28:31.289 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4bc50523-8480-44d2-8d76-31479e839c85", 2024-05-15T23:28:31.289 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:28:31.289 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:28:31.289 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:28:31.289 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:28:31.289 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:28:31.289 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:28:31.289 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-05-15T23:28:31.289 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 477, 2024-05-15T23:28:31.289 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:28:31.289 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:28:31.289 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:28:31.289 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.290 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.290 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6800", 2024-05-15T23:28:31.290 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:28:31.290 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.290 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.290 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:28:31.290 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.290 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.290 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6801", 2024-05-15T23:28:31.290 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:28:31.291 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.291 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.291 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:28:31.291 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.291 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.291 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6803", 2024-05-15T23:28:31.291 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:28:31.291 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.291 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.291 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:28:31.291 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.292 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.292 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6802", 2024-05-15T23:28:31.292 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:28:31.292 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.292 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.292 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6800/2267955292", 2024-05-15T23:28:31.292 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6801/2267955292", 2024-05-15T23:28:31.292 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6803/2267955292", 2024-05-15T23:28:31.292 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6802/2267955292", 2024-05-15T23:28:31.292 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:28:31.292 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:28:31.293 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:28:31.293 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.293 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-05-15T23:28:31.293 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20", 2024-05-15T23:28:31.293 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:28:31.293 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:28:31.293 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:28:31.293 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:28:31.293 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:28:31.293 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:28:31.293 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 37, 2024-05-15T23:28:31.294 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 477, 2024-05-15T23:28:31.294 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:28:31.294 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:28:31.294 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:28:31.294 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.294 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.294 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6804", 2024-05-15T23:28:31.294 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:28:31.294 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.294 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.295 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:28:31.295 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.295 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.295 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6805", 2024-05-15T23:28:31.295 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:28:31.295 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.295 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.295 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:28:31.295 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.295 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.296 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6807", 2024-05-15T23:28:31.296 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:28:31.296 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.296 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.296 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:28:31.296 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.296 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.296 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6806", 2024-05-15T23:28:31.296 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:28:31.296 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.297 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.297 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6804/661041538", 2024-05-15T23:28:31.297 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6805/661041538", 2024-05-15T23:28:31.297 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6807/661041538", 2024-05-15T23:28:31.297 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6806/661041538", 2024-05-15T23:28:31.297 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:28:31.297 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:28:31.297 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:28:31.297 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.297 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-05-15T23:28:31.297 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f", 2024-05-15T23:28:31.298 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:28:31.298 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-05-15T23:28:31.298 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-05-15T23:28:31.298 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:28:31.298 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:28:31.298 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:28:31.298 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 46, 2024-05-15T23:28:31.298 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 374, 2024-05-15T23:28:31.298 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:28:31.298 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:28:31.298 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:28:31.298 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.299 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.299 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6808", 2024-05-15T23:28:31.299 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:28:31.299 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.299 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.299 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:28:31.299 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.299 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.299 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6809", 2024-05-15T23:28:31.299 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:28:31.299 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:28:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6811", 2024-05-15T23:28:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:28:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:28:31.300 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.301 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.301 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6810", 2024-05-15T23:28:31.301 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:28:31.301 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.301 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.301 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6808/380856869", 2024-05-15T23:28:31.301 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6809/380856869", 2024-05-15T23:28:31.301 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6811/380856869", 2024-05-15T23:28:31.301 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6810/380856869", 2024-05-15T23:28:31.301 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:28:31.301 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:28:31.302 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:28:31.302 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.302 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-05-15T23:28:31.302 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34", 2024-05-15T23:28:31.302 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:28:31.302 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:28:31.302 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:28:31.302 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:28:31.302 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:28:31.302 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:28:31.302 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 53, 2024-05-15T23:28:31.303 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 477, 2024-05-15T23:28:31.303 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:28:31.303 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:28:31.303 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:28:31.303 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.303 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.303 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6812", 2024-05-15T23:28:31.303 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:28:31.303 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.303 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.303 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:28:31.304 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.304 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.304 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6813", 2024-05-15T23:28:31.304 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:28:31.304 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.304 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.304 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:28:31.304 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.305 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.305 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6815", 2024-05-15T23:28:31.305 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:28:31.305 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.305 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.305 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:28:31.305 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.305 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.305 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6814", 2024-05-15T23:28:31.305 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:28:31.305 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.305 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.306 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6812/1044709063", 2024-05-15T23:28:31.306 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6813/1044709063", 2024-05-15T23:28:31.306 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6815/1044709063", 2024-05-15T23:28:31.306 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6814/1044709063", 2024-05-15T23:28:31.306 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:28:31.306 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:28:31.306 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:28:31.306 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.306 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-05-15T23:28:31.306 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "02cab286-d487-4a66-982c-3cf28f55e851", 2024-05-15T23:28:31.306 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:28:31.307 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-05-15T23:28:31.307 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-05-15T23:28:31.307 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 0.54486083984375, 2024-05-15T23:28:31.307 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 61, 2024-05-15T23:28:31.307 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 484, 2024-05-15T23:28:31.307 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 520, 2024-05-15T23:28:31.307 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 202, 2024-05-15T23:28:31.307 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 485, 2024-05-15T23:28:31.307 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:28:31.308 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:28:31.308 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.308 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.308 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6800", 2024-05-15T23:28:31.308 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:28:31.308 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.308 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.308 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:28:31.308 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.308 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.309 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6801", 2024-05-15T23:28:31.309 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:28:31.309 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.309 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.309 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:28:31.309 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.309 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.309 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6803", 2024-05-15T23:28:31.309 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:28:31.309 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.309 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.310 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:28:31.310 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.310 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.310 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6802", 2024-05-15T23:28:31.310 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:28:31.310 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.310 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.310 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6800/703598286", 2024-05-15T23:28:31.310 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6801/703598286", 2024-05-15T23:28:31.310 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6803/703598286", 2024-05-15T23:28:31.311 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6802/703598286", 2024-05-15T23:28:31.311 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:28:31.311 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:28:31.311 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:28:31.311 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-05-15T23:28:31.311 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613", 2024-05-15T23:28:31.311 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:28:31.311 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:28:31.311 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:28:31.311 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:28:31.312 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:28:31.312 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:28:31.312 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 68, 2024-05-15T23:28:31.312 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 542, 2024-05-15T23:28:31.312 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:28:31.312 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:28:31.312 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:28:31.312 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.312 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.312 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6804", 2024-05-15T23:28:31.312 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:28:31.313 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.313 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.313 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:28:31.313 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.313 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.313 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6805", 2024-05-15T23:28:31.313 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:28:31.313 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.313 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.313 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:28:31.313 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.314 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.314 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6807", 2024-05-15T23:28:31.314 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:28:31.314 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.314 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.314 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:28:31.314 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.314 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.314 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6806", 2024-05-15T23:28:31.314 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:28:31.315 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.315 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.315 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6804/2794571133", 2024-05-15T23:28:31.315 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6805/2794571133", 2024-05-15T23:28:31.315 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6807/2794571133", 2024-05-15T23:28:31.315 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6806/2794571133", 2024-05-15T23:28:31.315 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:28:31.315 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:28:31.315 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:28:31.315 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-05-15T23:28:31.316 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473", 2024-05-15T23:28:31.316 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:28:31.316 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:28:31.316 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:28:31.316 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:28:31.316 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:28:31.316 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:28:31.316 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 74, 2024-05-15T23:28:31.316 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 477, 2024-05-15T23:28:31.316 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:28:31.316 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:28:31.316 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:28:31.317 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.317 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.317 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6808", 2024-05-15T23:28:31.317 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:28:31.317 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.317 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.317 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:28:31.317 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.317 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.317 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6809", 2024-05-15T23:28:31.317 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:28:31.318 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.318 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.318 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:28:31.318 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.318 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.318 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6811", 2024-05-15T23:28:31.318 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:28:31.318 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.318 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.319 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:28:31.319 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.319 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.319 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6810", 2024-05-15T23:28:31.319 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:28:31.319 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.319 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.319 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6808/3735380744", 2024-05-15T23:28:31.319 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6809/3735380744", 2024-05-15T23:28:31.319 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6811/3735380744", 2024-05-15T23:28:31.320 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6810/3735380744", 2024-05-15T23:28:31.320 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:28:31.320 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:28:31.320 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:28:31.320 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.320 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-05-15T23:28:31.320 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5", 2024-05-15T23:28:31.320 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:28:31.320 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:28:31.320 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:28:31.320 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:28:31.320 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 81, 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 374, 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6812", 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:28:31.321 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.322 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.322 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6813", 2024-05-15T23:28:31.322 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:28:31.322 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.322 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.322 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:28:31.322 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.322 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.322 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6815", 2024-05-15T23:28:31.322 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:28:31.322 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6814", 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6812/2265615270", 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6813/2265615270", 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6815/2265615270", 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6814/2265615270", 2024-05-15T23:28:31.323 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:28:31.324 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:28:31.324 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:28:31.324 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:28:31.324 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.324 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:28:31.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-05-15T23:28:31.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-05-15T23:28:31.324 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:28:31.324 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:28:31.324 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:28:31.324 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:28:31.324 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-05-15T23:28:31.324 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:27:41.714540+0000", 2024-05-15T23:28:31.325 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 355 2024-05-15T23:28:31.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-05-15T23:28:31.325 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:28:31.325 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:28:31.325 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:28:31.325 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:28:31.325 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:28:31.325 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:27:44.031726+0000", 2024-05-15T23:28:31.325 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:28:31.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-05-15T23:28:31.326 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:28:31.326 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:28:31.326 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:28:31.326 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:28:31.326 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:28:31.326 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:27:14.985300+0000", 2024-05-15T23:28:31.326 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 261 2024-05-15T23:28:31.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-05-15T23:28:31.326 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:28:31.326 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:28:31.326 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:28:31.327 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:28:31.327 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:28:31.327 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:27:38.279689+0000", 2024-05-15T23:28:31.327 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:28:31.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-05-15T23:28:31.327 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:28:31.327 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:28:31.327 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:28:31.327 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:28:31.327 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:28:31.327 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:27:12.758793+0000", 2024-05-15T23:28:31.327 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:28:31.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-05-15T23:28:31.328 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:28:31.328 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:28:31.328 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:28:31.328 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:28:31.328 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:28:31.328 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:27:38.986834+0000", 2024-05-15T23:28:31.328 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:28:31.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-05-15T23:28:31.328 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:28:31.328 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:28:31.329 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:28:31.329 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:28:31.329 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-05-15T23:28:31.329 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:28:16.648144+0000", 2024-05-15T23:28:31.329 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:28:31.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.329 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-05-15T23:28:31.329 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:28:31.329 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:28:31.329 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:28:31.329 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:28:31.329 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:28:31.329 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:27:45.869077+0000", 2024-05-15T23:28:31.330 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:28:31.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.330 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-05-15T23:28:31.330 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:28:31.330 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:28:31.330 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:28:31.330 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:28:31.330 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-05-15T23:28:31.330 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:28:18.487447+0000", 2024-05-15T23:28:31.330 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 484 2024-05-15T23:28:31.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.330 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-05-15T23:28:31.331 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:28:31.331 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:28:31.331 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:28:31.331 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:28:31.331 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:28:31.331 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:28:25.532815+0000", 2024-05-15T23:28:31.331 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:28:31.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.331 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-05-15T23:28:31.331 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:28:31.331 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:28:31.331 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:28:31.331 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:28:31.332 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:28:31.332 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:27:26.746134+0000", 2024-05-15T23:28:31.332 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:28:31.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:28:31.332 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-05-15T23:28:31.332 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:28:31.332 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:28:31.332 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:28:31.332 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:28:31.332 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:28:31.332 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:27:35.121116+0000", 2024-05-15T23:28:31.332 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:28:31.332 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.333 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:28:31.333 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [], 2024-05-15T23:28:31.333 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [], 2024-05-15T23:28:31.333 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-05-15T23:28:31.333 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-05-15T23:28:31.333 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-05-15T23:28:31.333 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-05-15T23:28:31.333 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2581408231": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:28:31.333 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/1136984393": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:28:31.333 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2443201864": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:28:31.333 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1252671888": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:28:31.333 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1451001371": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:28:31.333 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4084814866": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:28:31.333 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4076409285": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:28:31.333 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2988977694": "2024-05-16T22:57:38.505263+0000" 2024-05-15T23:28:31.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.334 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-05-15T23:28:31.334 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-05-15T23:28:31.334 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-05-15T23:28:31.334 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-05-15T23:28:31.334 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-05-15T23:28:31.334 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-05-15T23:28:31.334 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-05-15T23:28:31.334 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-05-15T23:28:31.334 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:28:31.334 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [], 2024-05-15T23:28:31.334 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2024-05-15T23:28:31.334 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-05-15T23:28:31.335 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-05-15T23:28:31.335 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-05-15T23:28:31.335 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-05-15T23:28:31.335 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-05-15T23:28:31.335 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-05-15T23:28:31.335 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-05-15T23:28:31.335 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-05-15T23:28:31.335 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-05-15T23:28:31.335 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:28:31.335 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:28:31.335 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:28:31.337 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:28:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:28:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:28:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:28:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:28:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:28:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:28:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:28:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:28:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:28:31.338 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:28:31.338 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:28:31.338 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:28:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2093433814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:28:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:31 smithi090 ceph-mon[44434]: pgmap v1460: 17 pgs: 1 peering, 16 active+clean; 508 MiB data, 5.0 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 978 op/s; 16 MiB/s, 256 objects/s recovering 2024-05-15T23:28:31.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:31.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4149865782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:28:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2093433814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:28:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:31 smithi084 ceph-mon[39784]: pgmap v1460: 17 pgs: 1 peering, 16 active+clean; 508 MiB data, 5.0 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 978 op/s; 16 MiB/s, 256 objects/s recovering 2024-05-15T23:28:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4149865782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:28:31.629 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:28:31.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:31.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:28:31.733 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:31.756 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:28:31.778 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:31.780 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2093433814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:28:31.780 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:31.780 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:31.780 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:31.780 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:31 smithi088 ceph-mon[44699]: pgmap v1460: 17 pgs: 1 peering, 16 active+clean; 508 MiB data, 5.0 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 978 op/s; 16 MiB/s, 256 objects/s recovering 2024-05-15T23:28:31.780 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:31.780 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4149865782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:28:31.814 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:31.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:31.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:31.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:28:31.948 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:32.206 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:28:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:32.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:28:32.317 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:28:32.348 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:32.478 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:28:32.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:32 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:28:32.480 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:32 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:28:32.480 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.480 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.480 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.480 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.480 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:32 smithi090 ceph-mon[44434]: osdmap e545: 12 total, 12 up, 9 in 2024-05-15T23:28:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:32.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:28:32.583 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:28:32.657 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:32.676 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:28:32.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:32.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:28:32.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:32 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:28:32.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:32 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:28:32.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.683 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:32 smithi088 ceph-mon[44699]: osdmap e545: 12 total, 12 up, 9 in 2024-05-15T23:28:32.778 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:28:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:28:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:28:32.778 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:28:32.779 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:28:32.783 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:28:32.815 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.815 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.815 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.815 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.815 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:32 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:28:32.815 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:32 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:28:32.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:32.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:32 smithi084 ceph-mon[39784]: osdmap e545: 12 total, 12 up, 9 in 2024-05-15T23:28:32.817 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:33.035 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:28:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:33.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:28:33.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:33.442 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:33 smithi090 ceph-mon[44434]: pgmap v1462: 17 pgs: 17 active+clean; 275 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 4.6 MiB/s, 73 objects/s recovering 2024-05-15T23:28:33.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T23:28:33.531 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:33.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:33.532 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:33.633 INFO:tasks.thrashosds.thrasher:j is {'epoch': 544, 'fsid': '3999fb7e-130e-11ef-bc99-c7b262605968', 'created': '2024-05-15T22:56:41.212493+0000', 'modified': '2024-05-15T23:28:28.246435+0000', 'last_up_change': '2024-05-15T23:26:58.424794+0000', 'last_in_change': '2024-05-15T23:28:16.134275+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 30, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T22:59:38.603575+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9620716571807861, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.9599999785423279}}, {'pool': 6, 'pool_name': 'unique_pool_4', 'create_time': '2024-05-15T23:24:53.409510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '482', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9620716571807861, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.9599999785423279}}], 'osds': [{'osd': 0, 'uuid': '80159cc8-1b6c-4aa6-a114-966786d7be1e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 355, 'up_from': 397, 'up_thru': 283, 'down_at': 356, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6801', 'nonce': 3077815271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6802', 'nonce': 3077815271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6804', 'nonce': 3077815271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6803', 'nonce': 3077815271}]}, 'public_addr': '172.21.15.84:6801/3077815271', 'cluster_addr': '172.21.15.84:6802/3077815271', 'heartbeat_back_addr': '172.21.15.84:6804/3077815271', 'heartbeat_front_addr': '172.21.15.84:6803/3077815271', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9c2c76c9-ce0f-4c2d-9d96-c87a10187607', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 477, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6805', 'nonce': 2140830229}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6806', 'nonce': 2140830229}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6808', 'nonce': 2140830229}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6807', 'nonce': 2140830229}]}, 'public_addr': '172.21.15.84:6805/2140830229', 'cluster_addr': '172.21.15.84:6806/2140830229', 'heartbeat_back_addr': '172.21.15.84:6808/2140830229', 'heartbeat_front_addr': '172.21.15.84:6807/2140830229', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e8514daa-b5f9-4f2f-a432-53f4a4eb86bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 261, 'up_from': 291, 'up_thru': 541, 'down_at': 262, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6809', 'nonce': 941507493}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6810', 'nonce': 941507493}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6812', 'nonce': 941507493}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6811', 'nonce': 941507493}]}, 'public_addr': '172.21.15.84:6809/941507493', 'cluster_addr': '172.21.15.84:6810/941507493', 'heartbeat_back_addr': '172.21.15.84:6812/941507493', 'heartbeat_front_addr': '172.21.15.84:6811/941507493', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5d9965f9-c0b2-48ce-869c-582990985623', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 542, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6813', 'nonce': 2204039828}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6814', 'nonce': 2204039828}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6816', 'nonce': 2204039828}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6815', 'nonce': 2204039828}]}, 'public_addr': '172.21.15.84:6813/2204039828', 'cluster_addr': '172.21.15.84:6814/2204039828', 'heartbeat_back_addr': '172.21.15.84:6816/2204039828', 'heartbeat_front_addr': '172.21.15.84:6815/2204039828', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4bc50523-8480-44d2-8d76-31479e839c85', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 477, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6800', 'nonce': 2267955292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6801', 'nonce': 2267955292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6803', 'nonce': 2267955292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6802', 'nonce': 2267955292}]}, 'public_addr': '172.21.15.88:6800/2267955292', 'cluster_addr': '172.21.15.88:6801/2267955292', 'heartbeat_back_addr': '172.21.15.88:6803/2267955292', 'heartbeat_front_addr': '172.21.15.88:6802/2267955292', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'd4db8d7f-afc7-43dc-81fe-f70c290d8b20', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 477, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6804', 'nonce': 661041538}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6805', 'nonce': 661041538}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6807', 'nonce': 661041538}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6806', 'nonce': 661041538}]}, 'public_addr': '172.21.15.88:6804/661041538', 'cluster_addr': '172.21.15.88:6805/661041538', 'heartbeat_back_addr': '172.21.15.88:6807/661041538', 'heartbeat_front_addr': '172.21.15.88:6806/661041538', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'a2773454-2318-4a5b-8f31-0952b1b07f2f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6808', 'nonce': 380856869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6809', 'nonce': 380856869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6811', 'nonce': 380856869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6810', 'nonce': 380856869}]}, 'public_addr': '172.21.15.88:6808/380856869', 'cluster_addr': '172.21.15.88:6809/380856869', 'heartbeat_back_addr': '172.21.15.88:6811/380856869', 'heartbeat_front_addr': '172.21.15.88:6810/380856869', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '028ca7b6-3e00-4466-a33f-b726cd60cb34', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 477, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6812', 'nonce': 1044709063}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6813', 'nonce': 1044709063}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6815', 'nonce': 1044709063}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6814', 'nonce': 1044709063}]}, 'public_addr': '172.21.15.88:6812/1044709063', 'cluster_addr': '172.21.15.88:6813/1044709063', 'heartbeat_back_addr': '172.21.15.88:6815/1044709063', 'heartbeat_front_addr': '172.21.15.88:6814/1044709063', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '02cab286-d487-4a66-982c-3cf28f55e851', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.54486083984375, 'last_clean_begin': 61, 'last_clean_end': 484, 'up_from': 520, 'up_thru': 202, 'down_at': 485, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6800', 'nonce': 703598286}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 703598286}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 703598286}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6802', 'nonce': 703598286}]}, 'public_addr': '172.21.15.90:6800/703598286', 'cluster_addr': '172.21.15.90:6801/703598286', 'heartbeat_back_addr': '172.21.15.90:6803/703598286', 'heartbeat_front_addr': '172.21.15.90:6802/703598286', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '47d38bc7-39ba-4b35-af24-e282354dd613', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 68, 'up_thru': 542, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6804', 'nonce': 2794571133}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 2794571133}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 2794571133}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6806', 'nonce': 2794571133}]}, 'public_addr': '172.21.15.90:6804/2794571133', 'cluster_addr': '172.21.15.90:6805/2794571133', 'heartbeat_back_addr': '172.21.15.90:6807/2794571133', 'heartbeat_front_addr': '172.21.15.90:6806/2794571133', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'd7b2aeae-d012-41eb-9f91-e4c05c649473', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 477, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6808', 'nonce': 3735380744}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 3735380744}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 3735380744}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 3735380744}]}, 'public_addr': '172.21.15.90:6808/3735380744', 'cluster_addr': '172.21.15.90:6809/3735380744', 'heartbeat_back_addr': '172.21.15.90:6811/3735380744', 'heartbeat_front_addr': '172.21.15.90:6810/3735380744', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6812', 'nonce': 2265615270}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 2265615270}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 2265615270}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 2265615270}]}, 'public_addr': '172.21.15.90:6812/2265615270', 'cluster_addr': '172.21.15.90:6813/2265615270', 'heartbeat_back_addr': '172.21.15.90:6815/2265615270', 'heartbeat_front_addr': '172.21.15.90:6814/2265615270', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T23:27:41.714540+0000', 'dead_epoch': 355}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:27:44.031726+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:27:14.985300+0000', 'dead_epoch': 261}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:27:38.279689+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:27:12.758793+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:27:38.986834+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T23:28:16.648144+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:27:45.869077+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T23:28:18.487447+0000', 'dead_epoch': 484}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:28:25.532815+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:27:26.746134+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:27:35.121116+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/2581408231': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:6800/1136984393': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:6800/2443201864': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/1252671888': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/1451001371': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/4084814866': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:0/4076409285': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/2988977694': '2024-05-16T22:57:38.505263+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-15T23:28:33.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:28:33.635 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:33.674 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:33.703 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:33.705 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:33 smithi088 ceph-mon[44699]: pgmap v1462: 17 pgs: 17 active+clean; 275 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 4.6 MiB/s, 73 objects/s recovering 2024-05-15T23:28:33.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:33.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:33.780 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:33.807 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:33.837 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:33.855 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:28:33.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:33 smithi084 ceph-mon[39784]: pgmap v1462: 17 pgs: 17 active+clean; 275 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 4.6 MiB/s, 73 objects/s recovering 2024-05-15T23:28:33.881 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:33.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:33.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:33.988 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:34.070 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:34.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:34.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:34.175 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:34.219 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:34.245 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:34.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:34.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:34.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:28:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:34.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:34 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:28:34.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:34 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:28:34.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:34 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:28:34.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:34 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:28:34.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:34 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:28:34.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:34 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:28:34.859 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Removed 135191 objects 2024-05-15T23:28:34.860 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Clean up completed and total clean up time :117.389 2024-05-15T23:28:34.869 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-05-15T23:28:34.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-05-15T23:28:35.046 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:28:35.146 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:35.181 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:35.445 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:28:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:35.449 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:28:35.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:28:35.577 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:35.630 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:28:35.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:35.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:28:35.653 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:35 smithi084 ceph-mon[39784]: osdmap e546: 12 total, 12 up, 9 in 2024-05-15T23:28:35.653 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:35 smithi084 ceph-mon[39784]: pgmap v1464: 17 pgs: 17 active+clean; 251 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 925 op/s; 4.6 MiB/s, 74 objects/s recovering 2024-05-15T23:28:35.734 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:28:35.763 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:35 smithi090 ceph-mon[44434]: osdmap e546: 12 total, 12 up, 9 in 2024-05-15T23:28:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:35 smithi090 ceph-mon[44434]: pgmap v1464: 17 pgs: 17 active+clean; 251 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 925 op/s; 4.6 MiB/s, 74 objects/s recovering 2024-05-15T23:28:35.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:35 smithi088 ceph-mon[44699]: osdmap e546: 12 total, 12 up, 9 in 2024-05-15T23:28:35.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:35 smithi088 ceph-mon[44699]: pgmap v1464: 17 pgs: 17 active+clean; 251 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 925 op/s; 4.6 MiB/s, 74 objects/s recovering 2024-05-15T23:28:36.311 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:28:36.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:36.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:28:36.416 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:28:36.449 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:36.471 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:28:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:36.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:28:36.539 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:28:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:28:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:28:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [] 2024-05-15T23:28:36.540 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:28:36.574 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:36.603 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:36.623 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:36.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:36.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:28:36.756 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:36.776 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:28:36.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:36.779 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:28:36.880 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:28:36.913 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:36.935 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:28:36.936 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:28:36.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:36.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:28:37.039 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:28:37.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/347913995' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:28:37.068 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:37.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/347913995' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:28:37.090 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:28:37.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:37.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:28:37.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/347913995' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:28:37.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T23:28:37.194 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:28:37.256 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:37.293 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:28:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:37.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:28:37.398 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:37.431 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:37.453 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:37.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:37.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:37.558 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:37.587 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:37.608 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:37.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:37.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:37.712 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:37.740 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:37.759 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:37.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:37.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:37.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:28:37.890 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:38.031 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:37 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:28:38.031 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:37 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/347913995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:28:38.031 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:37 smithi084 ceph-mon[39784]: osdmap e547: 12 total, 12 up, 9 in 2024-05-15T23:28:38.031 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:37 smithi084 ceph-mon[39784]: pgmap v1466: 17 pgs: 17 active+clean; 29 MiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.28k op/s 2024-05-15T23:28:38.039 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:28:38.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:38.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:28:38.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:37 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:28:38.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:37 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/347913995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:28:38.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:37 smithi088 ceph-mon[44699]: osdmap e547: 12 total, 12 up, 9 in 2024-05-15T23:28:38.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:37 smithi088 ceph-mon[44699]: pgmap v1466: 17 pgs: 17 active+clean; 29 MiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.28k op/s 2024-05-15T23:28:38.143 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:38.176 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:38.197 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:38.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:38.302 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:38.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:37 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:28:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:37 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/347913995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:28:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:37 smithi090 ceph-mon[44434]: osdmap e547: 12 total, 12 up, 9 in 2024-05-15T23:28:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:37 smithi090 ceph-mon[44434]: pgmap v1466: 17 pgs: 17 active+clean; 29 MiB data, 4.2 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.28k op/s 2024-05-15T23:28:38.332 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:38.353 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:38.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:38.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:38.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:28:38.487 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:38.491 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:38.493 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:28:38.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:38.516 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:28:38.617 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:28:38.647 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:38.827 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_4' removed 2024-05-15T23:28:39.023 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:38 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2678260286' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:28:39.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:38 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2678260286' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:28:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:38 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2678260286' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:28:40.042 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:28:40.045 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:28:40.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:28:40.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2678260286' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:28:40.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:39 smithi084 ceph-mon[39784]: osdmap e548: 12 total, 12 up, 9 in 2024-05-15T23:28:40.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:39 smithi084 ceph-mon[39784]: pgmap v1468: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:40.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2678260286' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:28:40.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:39 smithi090 ceph-mon[44434]: osdmap e548: 12 total, 12 up, 9 in 2024-05-15T23:28:40.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:39 smithi090 ceph-mon[44434]: pgmap v1468: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:40.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2678260286' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:28:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:39 smithi088 ceph-mon[44699]: osdmap e548: 12 total, 12 up, 9 in 2024-05-15T23:28:40.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:39 smithi088 ceph-mon[44699]: pgmap v1468: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:40.524 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T23:28:40.525 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T23:28:40.525 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-05-15T23:28:40.525 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-05-15T23:28:41.033 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:28:41.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:41.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:28:41.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: from='client.16713 v1:172.21.15.84:0/1660496446' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:28:41.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=315147769,config=3313370793,health=3340005642,kv=3647639065}) 2024-05-15T23:28:41.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {kv=10,logm=90} crc {kv=427831151,logm=533623030}) 2024-05-15T23:28:41.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=889529583}) 2024-05-15T23:28:41.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3512469978}) 2024-05-15T23:28:41.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3660023132}) 2024-05-15T23:28:41.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2595841472}) 2024-05-15T23:28:41.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673764393}) 2024-05-15T23:28:41.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3535254861}) 2024-05-15T23:28:41.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3842787137}) 2024-05-15T23:28:41.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=817617871}) 2024-05-15T23:28:41.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777379797}) 2024-05-15T23:28:41.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3753238043}) 2024-05-15T23:28:41.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2539400953}) 2024-05-15T23:28:41.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3609286067}) 2024-05-15T23:28:41.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2909515059}) 2024-05-15T23:28:41.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3426881303}) 2024-05-15T23:28:41.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2958464652}) 2024-05-15T23:28:41.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=750760903}) 2024-05-15T23:28:41.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=536469103}) 2024-05-15T23:28:41.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2317426316}) 2024-05-15T23:28:41.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2233921957}) 2024-05-15T23:28:41.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=917061187}) 2024-05-15T23:28:41.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1706592754}) 2024-05-15T23:28:41.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3372687037}) 2024-05-15T23:28:41.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2543482136}) 2024-05-15T23:28:41.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2335095849}) 2024-05-15T23:28:41.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3080593912}) 2024-05-15T23:28:41.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2900338183}) 2024-05-15T23:28:41.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3727823391}) 2024-05-15T23:28:41.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1878645070}) 2024-05-15T23:28:41.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1166748679}) 2024-05-15T23:28:41.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2127505432}) 2024-05-15T23:28:41.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=344203213}) 2024-05-15T23:28:41.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=150796470}) 2024-05-15T23:28:41.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2495454089}) 2024-05-15T23:28:41.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=52827065}) 2024-05-15T23:28:41.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2062640336}) 2024-05-15T23:28:41.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1441305767}) 2024-05-15T23:28:41.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=655739197}) 2024-05-15T23:28:41.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=346025466}) 2024-05-15T23:28:41.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212957214}) 2024-05-15T23:28:41.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=17488141}) 2024-05-15T23:28:41.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4036361209}) 2024-05-15T23:28:41.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3045516840}) 2024-05-15T23:28:41.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=310254995}) 2024-05-15T23:28:41.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=802295082}) 2024-05-15T23:28:41.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1363228243}) 2024-05-15T23:28:41.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3696918253}) 2024-05-15T23:28:41.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3727219742}) 2024-05-15T23:28:41.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3480560491}) 2024-05-15T23:28:41.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2546106356}) 2024-05-15T23:28:41.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=490377144}) 2024-05-15T23:28:41.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=415213718}) 2024-05-15T23:28:41.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2128496394}) 2024-05-15T23:28:41.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1177667039}) 2024-05-15T23:28:41.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3614034691}) 2024-05-15T23:28:41.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=868171783}) 2024-05-15T23:28:41.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1366021760}) 2024-05-15T23:28:41.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=235321618}) 2024-05-15T23:28:41.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4196861290}) 2024-05-15T23:28:41.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1331781451}) 2024-05-15T23:28:41.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1670623067}) 2024-05-15T23:28:41.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=85948862}) 2024-05-15T23:28:41.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2443810468}) 2024-05-15T23:28:41.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2128143605}) 2024-05-15T23:28:41.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2107221544}) 2024-05-15T23:28:41.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933512214}) 2024-05-15T23:28:41.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3198934658}) 2024-05-15T23:28:41.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=61,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=3} crc {logm=2506641169,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2828803848,mon_config_key=2284734774}) 2024-05-15T23:28:41.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=441413611}) 2024-05-15T23:28:41.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=276819327}) 2024-05-15T23:28:41.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=81,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=1} crc {mon_config_key=1011324632,monmap=1478146745,osd_metadata=2611665884,osd_pg_creating=592995163,osdmap=3770923679}) 2024-05-15T23:28:41.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1010391248}) 2024-05-15T23:28:41.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=10} crc {osdmap=3346820953}) 2024-05-15T23:28:41.138 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:28:41.169 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: from='client.16713 v1:172.21.15.84:0/1660496446' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:28:41.169 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=315147769,config=3313370793,health=3340005642,kv=3647639065}) 2024-05-15T23:28:41.169 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {kv=10,logm=90} crc {kv=427831151,logm=533623030}) 2024-05-15T23:28:41.169 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=889529583}) 2024-05-15T23:28:41.169 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3512469978}) 2024-05-15T23:28:41.169 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3660023132}) 2024-05-15T23:28:41.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2595841472}) 2024-05-15T23:28:41.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673764393}) 2024-05-15T23:28:41.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3535254861}) 2024-05-15T23:28:41.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3842787137}) 2024-05-15T23:28:41.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=817617871}) 2024-05-15T23:28:41.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777379797}) 2024-05-15T23:28:41.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3753238043}) 2024-05-15T23:28:41.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2539400953}) 2024-05-15T23:28:41.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3609286067}) 2024-05-15T23:28:41.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2909515059}) 2024-05-15T23:28:41.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3426881303}) 2024-05-15T23:28:41.170 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2958464652}) 2024-05-15T23:28:41.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=750760903}) 2024-05-15T23:28:41.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=536469103}) 2024-05-15T23:28:41.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2317426316}) 2024-05-15T23:28:41.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2233921957}) 2024-05-15T23:28:41.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=917061187}) 2024-05-15T23:28:41.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1706592754}) 2024-05-15T23:28:41.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3372687037}) 2024-05-15T23:28:41.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2543482136}) 2024-05-15T23:28:41.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2335095849}) 2024-05-15T23:28:41.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3080593912}) 2024-05-15T23:28:41.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2900338183}) 2024-05-15T23:28:41.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3727823391}) 2024-05-15T23:28:41.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1878645070}) 2024-05-15T23:28:41.171 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1166748679}) 2024-05-15T23:28:41.172 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2127505432}) 2024-05-15T23:28:41.172 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=344203213}) 2024-05-15T23:28:41.172 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=150796470}) 2024-05-15T23:28:41.172 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2495454089}) 2024-05-15T23:28:41.172 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=52827065}) 2024-05-15T23:28:41.172 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2062640336}) 2024-05-15T23:28:41.172 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1441305767}) 2024-05-15T23:28:41.172 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=655739197}) 2024-05-15T23:28:41.172 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=346025466}) 2024-05-15T23:28:41.172 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212957214}) 2024-05-15T23:28:41.172 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=17488141}) 2024-05-15T23:28:41.172 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4036361209}) 2024-05-15T23:28:41.172 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3045516840}) 2024-05-15T23:28:41.172 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=310254995}) 2024-05-15T23:28:41.172 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=802295082}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1363228243}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3696918253}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3727219742}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3480560491}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2546106356}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=490377144}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=415213718}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2128496394}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1177667039}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3614034691}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=868171783}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1366021760}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=235321618}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4196861290}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1331781451}) 2024-05-15T23:28:41.173 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1670623067}) 2024-05-15T23:28:41.174 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=85948862}) 2024-05-15T23:28:41.174 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2443810468}) 2024-05-15T23:28:41.174 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2128143605}) 2024-05-15T23:28:41.174 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2107221544}) 2024-05-15T23:28:41.174 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933512214}) 2024-05-15T23:28:41.174 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3198934658}) 2024-05-15T23:28:41.174 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=61,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=3} crc {logm=2506641169,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2828803848,mon_config_key=2284734774}) 2024-05-15T23:28:41.174 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=441413611}) 2024-05-15T23:28:41.174 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=276819327}) 2024-05-15T23:28:41.174 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=81,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=1} crc {mon_config_key=1011324632,monmap=1478146745,osd_metadata=2611665884,osd_pg_creating=592995163,osdmap=3770923679}) 2024-05-15T23:28:41.174 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1010391248}) 2024-05-15T23:28:41.174 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=10} crc {osdmap=3346820953}) 2024-05-15T23:28:41.175 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:41.194 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:28:41.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:41.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:28:41.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:28:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: from='client.16713 v1:172.21.15.84:0/1660496446' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:28:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=315147769,config=3313370793,health=3340005642,kv=3647639065}) 2024-05-15T23:28:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {kv=10,logm=90} crc {kv=427831151,logm=533623030}) 2024-05-15T23:28:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=889529583}) 2024-05-15T23:28:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3512469978}) 2024-05-15T23:28:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3660023132}) 2024-05-15T23:28:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2595841472}) 2024-05-15T23:28:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673764393}) 2024-05-15T23:28:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3535254861}) 2024-05-15T23:28:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3842787137}) 2024-05-15T23:28:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=817617871}) 2024-05-15T23:28:41.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777379797}) 2024-05-15T23:28:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3753238043}) 2024-05-15T23:28:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2539400953}) 2024-05-15T23:28:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3609286067}) 2024-05-15T23:28:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2909515059}) 2024-05-15T23:28:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3426881303}) 2024-05-15T23:28:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2958464652}) 2024-05-15T23:28:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=750760903}) 2024-05-15T23:28:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=536469103}) 2024-05-15T23:28:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2317426316}) 2024-05-15T23:28:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2233921957}) 2024-05-15T23:28:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=917061187}) 2024-05-15T23:28:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1706592754}) 2024-05-15T23:28:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3372687037}) 2024-05-15T23:28:41.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2543482136}) 2024-05-15T23:28:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2335095849}) 2024-05-15T23:28:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3080593912}) 2024-05-15T23:28:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2900338183}) 2024-05-15T23:28:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3727823391}) 2024-05-15T23:28:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1878645070}) 2024-05-15T23:28:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1166748679}) 2024-05-15T23:28:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2127505432}) 2024-05-15T23:28:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=344203213}) 2024-05-15T23:28:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=150796470}) 2024-05-15T23:28:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2495454089}) 2024-05-15T23:28:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=52827065}) 2024-05-15T23:28:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2062640336}) 2024-05-15T23:28:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1441305767}) 2024-05-15T23:28:41.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=655739197}) 2024-05-15T23:28:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=346025466}) 2024-05-15T23:28:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212957214}) 2024-05-15T23:28:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=17488141}) 2024-05-15T23:28:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4036361209}) 2024-05-15T23:28:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3045516840}) 2024-05-15T23:28:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=310254995}) 2024-05-15T23:28:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=802295082}) 2024-05-15T23:28:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1363228243}) 2024-05-15T23:28:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3696918253}) 2024-05-15T23:28:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3727219742}) 2024-05-15T23:28:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3480560491}) 2024-05-15T23:28:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2546106356}) 2024-05-15T23:28:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=490377144}) 2024-05-15T23:28:41.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=415213718}) 2024-05-15T23:28:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2128496394}) 2024-05-15T23:28:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1177667039}) 2024-05-15T23:28:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3614034691}) 2024-05-15T23:28:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=868171783}) 2024-05-15T23:28:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1366021760}) 2024-05-15T23:28:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=235321618}) 2024-05-15T23:28:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4196861290}) 2024-05-15T23:28:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1331781451}) 2024-05-15T23:28:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1670623067}) 2024-05-15T23:28:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=85948862}) 2024-05-15T23:28:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2443810468}) 2024-05-15T23:28:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2128143605}) 2024-05-15T23:28:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2107221544}) 2024-05-15T23:28:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933512214}) 2024-05-15T23:28:41.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3198934658}) 2024-05-15T23:28:41.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=61,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=3} crc {logm=2506641169,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2828803848,mon_config_key=2284734774}) 2024-05-15T23:28:41.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=441413611}) 2024-05-15T23:28:41.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=276819327}) 2024-05-15T23:28:41.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=81,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=1} crc {mon_config_key=1011324632,monmap=1478146745,osd_metadata=2611665884,osd_pg_creating=592995163,osdmap=3770923679}) 2024-05-15T23:28:41.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1010391248}) 2024-05-15T23:28:41.308 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=10} crc {osdmap=3346820953}) 2024-05-15T23:28:41.330 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:41.352 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:28:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:41.355 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:28:41.456 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:28:41.484 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:42.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:41 smithi084 ceph-mon[39784]: pgmap v1469: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:42.276 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1468,"stamp":"2024-05-15T23:28:39.076284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":4297036,"kb_used_data":77264,"kb_used_omap":16,"kb_used_meta":4218863,"kb_avail":839409332,"statfs":{"total":863955320832,"available":859555155968,"internally_reserved":0,"allocated":79118336,"data_stored":76049262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17176,"internal_metadata":4320115944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,2,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-287834136,"num_objects":-4393,"num_object_clones":0,"num_object_copies":-8786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4393,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-265991,"num_write_kb":-8652225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3791,"num_bytes_recovered":-248446976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001069"},"pg_stats":[{"pgid":"1.0","version":"57'94","reported_seq":1157,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-05-15T23:28:34.459415+0000","last_change":"2024-05-15T23:27:33.965697+0000","last_active":"2024-05-15T23:28:34.459415+0000","last_peered":"2024-05-15T23:28:34.459415+0000","last_clean":"2024-05-15T23:28:34.459415+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:28:34.459415+0000","last_undegraded":"2024-05-15T23:28:34.459415+0000","last_fullsized":"2024-05-15T23:28:34.459415+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:27:33.965566+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:27:33.965566+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:28:41.264357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351274,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622496,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":621182,"kb_avail":93122656,"statfs":{"total":95995035648,"available":95357599744,"internally_reserved":0,"allocated":1196032,"data_stored":858934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":636090820},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580207,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365720,"kb_used_data":27592,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93379432,"statfs":{"total":95995035648,"available":95620538368,"internally_reserved":0,"allocated":28254208,"data_stored":27920682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":346094016},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776439,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":505496,"kb_used_data":22984,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93239656,"statfs":{"total":95995035648,"available":95477407744,"internally_reserved":0,"allocated":23535616,"data_stored":23202090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":493943232},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":2233382993943,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267007,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":473952,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93271200,"statfs":{"total":95995035648,"available":95509708800,"internally_reserved":0,"allocated":1196032,"data_stored":858954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":483981770},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495942,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575200,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":573400,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93171752,"statfs":{"total":95995035648,"available":95407874048,"internally_reserved":0,"allocated":663552,"data_stored":330026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":586348998},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016609,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483297,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94348,"kb_used_data":21760,"kb_used_omap":4,"kb_used_meta":72571,"kb_avail":93650804,"statfs":{"total":95995035648,"available":95898423296,"internally_reserved":0,"allocated":22282240,"data_stored":21888498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":74313411},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150038,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595608,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93149544,"statfs":{"total":95995035648,"available":95385133056,"internally_reserved":0,"allocated":663552,"data_stored":330026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":609089991},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986510,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":496912,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93248240,"statfs":{"total":95995035648,"available":95486197760,"internally_reserved":0,"allocated":663552,"data_stored":330026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790283,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":569104,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93176048,"statfs":{"total":95995035648,"available":95412273152,"internally_reserved":0,"allocated":663552,"data_stored":330026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:28:42.277 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [6, 3] 2024-05-15T23:28:42.277 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.0', '6', '3'] 2024-05-15T23:28:42.277 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pg-upmap 1.0 6 3 2024-05-15T23:28:42.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:41 smithi090 ceph-mon[44434]: pgmap v1469: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:42.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:41 smithi088 ceph-mon[44699]: pgmap v1469: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:43.262 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:28:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:43.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:28:43.367 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:43.400 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:43.424 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:43 smithi088 ceph-mon[44699]: pgmap v1470: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:43.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:43.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:43.503 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:43 smithi084 ceph-mon[39784]: pgmap v1470: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:43.526 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:28:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:43 smithi090 ceph-mon[44434]: pgmap v1470: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:43.555 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:43.691 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:43.692 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:43.697 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:28:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:43.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:28:43.721 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:28:43.721 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:28:43.722 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-05-15T23:28:43.722 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:28:43.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:28:43.829 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:44.331 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:45.284 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T23:28:45.444 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:45 smithi084 ceph-mon[39784]: pgmap v1471: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:45.507 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:28:45.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:45.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:28:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:45 smithi090 ceph-mon[44434]: pgmap v1471: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:45.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:45 smithi088 ceph-mon[44699]: pgmap v1471: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:45.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:28:45.645 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:45.666 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:28:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:45.669 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:28:45.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:28:45.800 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:45.821 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:28:45.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:45.824 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:28:45.925 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:45.959 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:45.980 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:45.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:45.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:46.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:28:46.114 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:46.134 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:28:46.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:46.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:28:46.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:28:46.267 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:46.286 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:28:46.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:46.289 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:28:46.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:46 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4273675434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:28:46.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:46.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:46 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:46.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:46 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4273675434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:28:46.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:46.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:46 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:46.391 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:28:46.419 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:46.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:46 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4273675434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:28:46.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:46.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:46 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:47 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:47 smithi084 ceph-mon[39784]: pgmap v1472: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:47 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:47 smithi090 ceph-mon[44434]: pgmap v1472: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:47 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:47.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:47 smithi088 ceph-mon[44699]: pgmap v1472: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:48.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:28:48.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:28:48.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:28:48.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:28:48.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:28:48.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:28:48.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:28:48.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:28:48.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:28:48.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:28:48.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:28:48.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:28:48.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:28:48.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:28:48.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:28:48.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:28:48.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:28:48.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:28:48.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:28:48.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:28:48.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:28:48.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:28:48.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:28:48.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:28:49.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:49 smithi084 ceph-mon[39784]: pgmap v1473: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:49.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:49 smithi090 ceph-mon[44434]: pgmap v1473: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:49.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:49 smithi088 ceph-mon[44699]: pgmap v1473: 1 pgs: 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:50.302 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:28:50.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:50.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:28:50.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:28:50.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3509299830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-05-15T23:28:50.439 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:50.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3509299830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-05-15T23:28:50.617 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_5' created 2024-05-15T23:28:50.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3509299830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-05-15T23:28:51.288 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-05-15T23:28:51.296 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:28:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:51.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:28:51.309 INFO:teuthology.orchestra.run.smithi084.stderr:set 1.0 pg_upmap mapping to [6,3] 2024-05-15T23:28:51.400 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:28:51.432 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:51.440 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:28:51.454 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:28:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:51.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:28:51.559 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:51.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3509299830' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-05-15T23:28:51.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:51 smithi088 ceph-mon[44699]: osdmap e549: 12 total, 12 up, 9 in 2024-05-15T23:28:51.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3907588980' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [6, 3]}]: dispatch 2024-05-15T23:28:51.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:51 smithi088 ceph-mon[44699]: pgmap v1475: 17 pgs: 4 creating+peering, 12 unknown, 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:51.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3907588980' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [6, 3]}]': finished 2024-05-15T23:28:51.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:51 smithi088 ceph-mon[44699]: osdmap e550: 12 total, 12 up, 9 in 2024-05-15T23:28:51.588 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:51.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3509299830' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-05-15T23:28:51.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:51 smithi084 ceph-mon[39784]: osdmap e549: 12 total, 12 up, 9 in 2024-05-15T23:28:51.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3907588980' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [6, 3]}]: dispatch 2024-05-15T23:28:51.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:51 smithi084 ceph-mon[39784]: pgmap v1475: 17 pgs: 4 creating+peering, 12 unknown, 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:51.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3907588980' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [6, 3]}]': finished 2024-05-15T23:28:51.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:51 smithi084 ceph-mon[39784]: osdmap e550: 12 total, 12 up, 9 in 2024-05-15T23:28:51.609 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:51.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:51.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:51.713 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:28:51.741 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:51.760 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:28:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:51.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:28:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3509299830' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-05-15T23:28:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:51 smithi090 ceph-mon[44434]: osdmap e549: 12 total, 12 up, 9 in 2024-05-15T23:28:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3907588980' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [6, 3]}]: dispatch 2024-05-15T23:28:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:51 smithi090 ceph-mon[44434]: pgmap v1475: 17 pgs: 4 creating+peering, 12 unknown, 1 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3907588980' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [6, 3]}]': finished 2024-05-15T23:28:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:51 smithi090 ceph-mon[44434]: osdmap e550: 12 total, 12 up, 9 in 2024-05-15T23:28:51.864 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:28:51.894 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:51.953 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:28:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:51.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:28:51.960 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:52.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:28:52.091 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:52.113 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:28:52.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:52.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:28:52.218 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:28:52.247 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:28:52.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:52 smithi084 ceph-mon[39784]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:28:52.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:52 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3726301303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:28:52.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:52 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:28:52.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:52 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3726301303' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:28:52.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:52 smithi084 ceph-mon[39784]: osdmap e551: 12 total, 12 up, 9 in 2024-05-15T23:28:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:52 smithi090 ceph-mon[44434]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:28:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:52 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3726301303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:28:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:52 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:28:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:52 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3726301303' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:28:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:52 smithi090 ceph-mon[44434]: osdmap e551: 12 total, 12 up, 9 in 2024-05-15T23:28:52.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:52 smithi088 ceph-mon[44699]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:28:52.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:52 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3726301303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:28:52.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:52 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:28:52.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:52 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3726301303' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:28:52.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:52 smithi088 ceph-mon[44699]: osdmap e551: 12 total, 12 up, 9 in 2024-05-15T23:28:52.897 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:28:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:52.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:28:53.001 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:28:53.028 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:53.180 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:28:53.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:53.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:28:53.284 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:28:53.311 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:53.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:53 smithi088 ceph-mon[44699]: pgmap v1478: 17 pgs: 5 active+clean, 8 creating+peering, 4 unknown; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:53 smithi084 ceph-mon[39784]: pgmap v1478: 17 pgs: 5 active+clean, 8 creating+peering, 4 unknown; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:53 smithi090 ceph-mon[44434]: pgmap v1478: 17 pgs: 5 active+clean, 8 creating+peering, 4 unknown; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:53.862 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:28:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:53.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:28:53.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:28:54.000 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:54.021 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:28:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:54.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:28:54.125 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:28:54.155 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:54.364 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:28:54.364 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:28:54.723 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:54.936 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:55.427 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:55 smithi084 ceph-mon[39784]: pgmap v1479: 17 pgs: 7 active+clean, 8 creating+peering, 2 unknown; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:55.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:55 smithi090 ceph-mon[44434]: pgmap v1479: 17 pgs: 7 active+clean, 8 creating+peering, 2 unknown; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:55.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:55 smithi088 ceph-mon[44699]: pgmap v1479: 17 pgs: 7 active+clean, 8 creating+peering, 2 unknown; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:55.704 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:28:55.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:55.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:28:55.810 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:28:55.842 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:55.864 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:28:55.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:55.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:28:55.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:28:56.004 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:56.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T23:28:56.234 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:56 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:28:56.234 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:56 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:28:56.253 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:28:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:56.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:28:56.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:28:56.392 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:56.413 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:28:56.415 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:56 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:28:56.415 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:56 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:28:56.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:56.417 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:28:56.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:56.548 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:56.569 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:56.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:56 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:28:56.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:56 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:28:56.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:56.572 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:56.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:28:56.701 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:56.760 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:28:56.823 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:28:56.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:56.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:28:56.926 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:28:56.953 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:57.152 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:28:57.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:57.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:28:57.192 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:57.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:28:57.290 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:57.311 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:28:57.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:57.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:28:57.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:57 smithi084 ceph-mon[39784]: osdmap e552: 12 total, 12 up, 9 in 2024-05-15T23:28:57.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:57 smithi084 ceph-mon[39784]: pgmap v1481: 17 pgs: 17 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:57.416 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:28:57.446 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:57 smithi090 ceph-mon[44434]: osdmap e552: 12 total, 12 up, 9 in 2024-05-15T23:28:57.447 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:57 smithi090 ceph-mon[44434]: pgmap v1481: 17 pgs: 17 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:57.449 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:57.470 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:28:57.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:57.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:28:57.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:57 smithi088 ceph-mon[44699]: osdmap e552: 12 total, 12 up, 9 in 2024-05-15T23:28:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:57 smithi088 ceph-mon[44699]: pgmap v1481: 17 pgs: 17 active+clean; 577 KiB data, 4.1 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:57.574 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:28:57.603 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:57.851 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:28:57.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:57.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:28:57.907 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10, 2] out_osds: [0, 6, 8] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:28:57.908 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:28:57.908 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-05-15T23:28:57.908 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump -f json-pretty 2024-05-15T23:28:57.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:28:57.973 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:28:57.981 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:57.993 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:28:58.016 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:28:58.036 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:28:58.056 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:28:58.075 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:28:58.095 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:28:58.116 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:28:58.137 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:28:58.156 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:28:58.178 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:28:58.198 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:28:58.199 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:28:58.199 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:28:58.199 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:28:58.199 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:28:58.199 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:28:58.199 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:28:58.200 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:28:58.200 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:28:58.200 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:28:58.200 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:28:58.200 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:28:58.200 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:28:58.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:58 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3318771888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:28:58.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:58 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3318771888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:28:58.606 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:58 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3318771888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:28:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:28:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:28:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: osdmap e553: 12 total, 12 up, 9 in 2024-05-15T23:28:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:28:59 smithi090 ceph-mon[44434]: pgmap v1483: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:28:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:28:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: osdmap e553: 12 total, 12 up, 9 in 2024-05-15T23:28:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:28:59 smithi088 ceph-mon[44699]: pgmap v1483: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:59.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:28:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:28:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: osdmap e553: 12 total, 12 up, 9 in 2024-05-15T23:28:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:28:59.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:28:59 smithi084 ceph-mon[39784]: pgmap v1483: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:28:59.715 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:59.733 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:28:59.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:59.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:28:59.755 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:28:59.838 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:28:59.871 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:28:59.893 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:28:59.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:28:59.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:28:59.997 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:29:00.027 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:00.047 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:29:00.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:00.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:29:00.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:29:00.179 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:00.806 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:29:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:00.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:29:00.923 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:29:00.952 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:01.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:00 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/313883038' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:29:01.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:29:01.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:00 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/313883038' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:29:01.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:00 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/313883038' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:29:01.901 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:01 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/313883038' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:29:01.901 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:01 smithi084 ceph-mon[39784]: osdmap e554: 12 total, 12 up, 9 in 2024-05-15T23:29:01.901 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:01 smithi084 ceph-mon[39784]: pgmap v1485: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:29:01.902 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:01 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:29:01.902 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:01 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:29:02.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:01 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/313883038' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:29:02.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:01 smithi090 ceph-mon[44434]: osdmap e554: 12 total, 12 up, 9 in 2024-05-15T23:29:02.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:01 smithi090 ceph-mon[44434]: pgmap v1485: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:29:02.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:01 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:29:02.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:01 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:29:02.101 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:29:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:02.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:29:02.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:02.240 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:02.262 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:02.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:02.266 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:01 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/313883038' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:29:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:01 smithi088 ceph-mon[44699]: osdmap e554: 12 total, 12 up, 9 in 2024-05-15T23:29:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:01 smithi088 ceph-mon[44699]: pgmap v1485: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:29:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:01 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:29:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:01 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:29:02.366 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:29:02.395 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:02.423 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:29:02.423 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-05-15T23:29:02.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-05-15T23:29:02.424 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:29:02.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.424 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329853 4.b 4.c5da395b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.424 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027062+0000", 2024-05-15T23:29:02.424 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.340472159, 2024-05-15T23:29:02.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29250239, 2024-05-15T23:29:02.424 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.424 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.424 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.424 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.424 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.424 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.424 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329853 2024-05-15T23:29:02.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.425 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027062+0000", 2024-05-15T23:29:02.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027062+0000", 2024-05-15T23:29:02.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027064+0000", 2024-05-15T23:29:02.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8899999999999999e-06 2024-05-15T23:29:02.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027069+0000", 2024-05-15T23:29:02.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5729999999999998e-06 2024-05-15T23:29:02.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027081+0000", 2024-05-15T23:29:02.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1493e-05 2024-05-15T23:29:02.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027090+0000", 2024-05-15T23:29:02.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9800000000000004e-06 2024-05-15T23:29:02.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027175+0000", 2024-05-15T23:29:02.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4954e-05 2024-05-15T23:29:02.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027185+0000", 2024-05-15T23:29:02.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0351e-05 2024-05-15T23:29:02.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319531+0000", 2024-05-15T23:29:02.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29234572399999997 2024-05-15T23:29:02.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319564+0000", 2024-05-15T23:29:02.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3424999999999999e-05 2024-05-15T23:29:02.429 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.429 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.429 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.429 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329870 4.b 4.4b410cb (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.429 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027397+0000", 2024-05-15T23:29:02.430 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.34013749899998, 2024-05-15T23:29:02.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29232418599999999, 2024-05-15T23:29:02.430 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.430 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.430 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.430 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.430 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.430 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.430 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329870 2024-05-15T23:29:02.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.430 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027397+0000", 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027397+0000", 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027397+0000", 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5600000000000005e-07 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027400+0000", 2024-05-15T23:29:02.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5210000000000001e-06 2024-05-15T23:29:02.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027409+0000", 2024-05-15T23:29:02.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3039999999999994e-06 2024-05-15T23:29:02.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027417+0000", 2024-05-15T23:29:02.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9039999999999992e-06 2024-05-15T23:29:02.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027462+0000", 2024-05-15T23:29:02.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4540999999999999e-05 2024-05-15T23:29:02.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027469+0000", 2024-05-15T23:29:02.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.841e-06 2024-05-15T23:29:02.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319715+0000", 2024-05-15T23:29:02.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29224517700000002 2024-05-15T23:29:02.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319721+0000", 2024-05-15T23:29:02.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1419999999999998e-06 2024-05-15T23:29:02.434 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.434 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.434 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.434 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329880 4.b 4.cc30174b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.434 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027430+0000", 2024-05-15T23:29:02.434 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.340104419, 2024-05-15T23:29:02.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29233345300000002, 2024-05-15T23:29:02.434 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.434 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.434 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.434 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329880 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027430+0000", 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027430+0000", 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027430+0000", 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7100000000000001e-07 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027433+0000", 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7360000000000001e-06 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027439+0000", 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6729999999999997e-06 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027445+0000", 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7699999999999998e-06 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027508+0000", 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3566999999999996e-05 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027517+0000", 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3919999999999992e-06 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319755+0000", 2024-05-15T23:29:02.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29223720600000003 2024-05-15T23:29:02.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319763+0000", 2024-05-15T23:29:02.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4379999999999995e-06 2024-05-15T23:29:02.439 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.439 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.439 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.441 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329890 4.b 4.4872c0cb (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.441 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027451+0000", 2024-05-15T23:29:02.441 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.34008319300005, 2024-05-15T23:29:02.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.292331216, 2024-05-15T23:29:02.441 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.441 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.441 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.441 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.441 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.441 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.441 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329890 2024-05-15T23:29:02.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.442 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027451+0000", 2024-05-15T23:29:02.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027451+0000", 2024-05-15T23:29:02.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027452+0000", 2024-05-15T23:29:02.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1799999999999995e-07 2024-05-15T23:29:02.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027454+0000", 2024-05-15T23:29:02.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1629999999999999e-06 2024-05-15T23:29:02.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027458+0000", 2024-05-15T23:29:02.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7600000000000002e-06 2024-05-15T23:29:02.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027463+0000", 2024-05-15T23:29:02.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0509999999999996e-06 2024-05-15T23:29:02.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027551+0000", 2024-05-15T23:29:02.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7164999999999996e-05 2024-05-15T23:29:02.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027558+0000", 2024-05-15T23:29:02.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4619999999999999e-06 2024-05-15T23:29:02.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319778+0000", 2024-05-15T23:29:02.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29221955900000002 2024-05-15T23:29:02.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319782+0000", 2024-05-15T23:29:02.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4379999999999997e-06 2024-05-15T23:29:02.445 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.445 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.445 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.446 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329893 4.b 4.9d86cb5b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.446 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027472+0000", 2024-05-15T23:29:02.446 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.34006183899999, 2024-05-15T23:29:02.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29242339299999998, 2024-05-15T23:29:02.446 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.446 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.446 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.446 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.446 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.446 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.446 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329893 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027472+0000", 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027472+0000", 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027473+0000", 2024-05-15T23:29:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9699999999999996e-07 2024-05-15T23:29:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027475+0000", 2024-05-15T23:29:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2680000000000001e-06 2024-05-15T23:29:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027480+0000", 2024-05-15T23:29:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1819999999999997e-06 2024-05-15T23:29:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027485+0000", 2024-05-15T23:29:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0300000000000001e-06 2024-05-15T23:29:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027583+0000", 2024-05-15T23:29:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7405999999999995e-05 2024-05-15T23:29:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027586+0000", 2024-05-15T23:29:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8510000000000001e-06 2024-05-15T23:29:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319890+0000", 2024-05-15T23:29:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29230399000000001 2024-05-15T23:29:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319896+0000", 2024-05-15T23:29:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.169e-06 2024-05-15T23:29:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329904 4.b 4.45b9298b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027497+0000", 2024-05-15T23:29:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.34003702099994, 2024-05-15T23:29:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29242595799999999, 2024-05-15T23:29:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329904 2024-05-15T23:29:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027497+0000", 2024-05-15T23:29:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027497+0000", 2024-05-15T23:29:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027497+0000", 2024-05-15T23:29:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9499999999999998e-07 2024-05-15T23:29:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027499+0000", 2024-05-15T23:29:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7799999999999999e-06 2024-05-15T23:29:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027503+0000", 2024-05-15T23:29:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6100000000000002e-06 2024-05-15T23:29:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027506+0000", 2024-05-15T23:29:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9510000000000002e-06 2024-05-15T23:29:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027616+0000", 2024-05-15T23:29:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010980999999999999 2024-05-15T23:29:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027623+0000", 2024-05-15T23:29:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0890000000000002e-06 2024-05-15T23:29:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319918+0000", 2024-05-15T23:29:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29229501099999999 2024-05-15T23:29:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319923+0000", 2024-05-15T23:29:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3120000000000001e-06 2024-05-15T23:29:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329913 4.b 4.89604feb (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027509+0000", 2024-05-15T23:29:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.34002511100005, 2024-05-15T23:29:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29243947100000001, 2024-05-15T23:29:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329913 2024-05-15T23:29:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027509+0000", 2024-05-15T23:29:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027509+0000", 2024-05-15T23:29:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027509+0000", 2024-05-15T23:29:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0600000000000001e-07 2024-05-15T23:29:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027511+0000", 2024-05-15T23:29:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.345e-06 2024-05-15T23:29:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027513+0000", 2024-05-15T23:29:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7159999999999999e-06 2024-05-15T23:29:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027516+0000", 2024-05-15T23:29:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7e-06 2024-05-15T23:29:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027657+0000", 2024-05-15T23:29:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000140922 2024-05-15T23:29:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027666+0000", 2024-05-15T23:29:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9339999999999995e-06 2024-05-15T23:29:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319944+0000", 2024-05-15T23:29:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.292278223 2024-05-15T23:29:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319948+0000", 2024-05-15T23:29:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3250000000000001e-06 2024-05-15T23:29:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329964 4.b 4.e00aa51b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027519+0000", 2024-05-15T23:29:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.34001542999999, 2024-05-15T23:29:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29245270200000001, 2024-05-15T23:29:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329964 2024-05-15T23:29:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027519+0000", 2024-05-15T23:29:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027519+0000", 2024-05-15T23:29:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027519+0000", 2024-05-15T23:29:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1899999999999998e-07 2024-05-15T23:29:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027520+0000", 2024-05-15T23:29:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.328e-06 2024-05-15T23:29:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027523+0000", 2024-05-15T23:29:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.593e-06 2024-05-15T23:29:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027526+0000", 2024-05-15T23:29:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8380000000000002e-06 2024-05-15T23:29:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027702+0000", 2024-05-15T23:29:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017643300000000001 2024-05-15T23:29:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027708+0000", 2024-05-15T23:29:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2330000000000003e-06 2024-05-15T23:29:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319967+0000", 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29225864299999998 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319971+0000", 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3150000000000002e-06 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:329975 4.b 4.564567eb (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027528+0000", 2024-05-15T23:29:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.34000578300004, 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.292466526, 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 329975 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027528+0000", 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027528+0000", 2024-05-15T23:29:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027529+0000", 2024-05-15T23:29:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1899999999999998e-07 2024-05-15T23:29:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027530+0000", 2024-05-15T23:29:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.257e-06 2024-05-15T23:29:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027533+0000", 2024-05-15T23:29:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9639999999999999e-06 2024-05-15T23:29:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027537+0000", 2024-05-15T23:29:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.743e-06 2024-05-15T23:29:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027733+0000", 2024-05-15T23:29:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019618200000000001 2024-05-15T23:29:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027737+0000", 2024-05-15T23:29:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3939999999999998e-06 2024-05-15T23:29:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319991+0000", 2024-05-15T23:29:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29225351999999999 2024-05-15T23:29:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.319995+0000", 2024-05-15T23:29:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1470000000000003e-06 2024-05-15T23:29:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330018 4.b 4.f747267b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027539+0000", 2024-05-15T23:29:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.33999472400001, 2024-05-15T23:29:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29248020699999999, 2024-05-15T23:29:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330018 2024-05-15T23:29:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027539+0000", 2024-05-15T23:29:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027539+0000", 2024-05-15T23:29:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027540+0000", 2024-05-15T23:29:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.22e-07 2024-05-15T23:29:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027541+0000", 2024-05-15T23:29:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2309999999999999e-06 2024-05-15T23:29:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027544+0000", 2024-05-15T23:29:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.638e-06 2024-05-15T23:29:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027546+0000", 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7180000000000001e-06 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027758+0000", 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021151000000000001 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027761+0000", 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.698e-06 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320016+0000", 2024-05-15T23:29:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29225404100000002 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320020+0000", 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0489999999999997e-06 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330033 4.b 4.4f74750b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027549+0000", 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.33998520099999, 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29250598100000003, 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330033 2024-05-15T23:29:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027549+0000", 2024-05-15T23:29:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027549+0000", 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027549+0000", 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0699999999999998e-07 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027551+0000", 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3710000000000001e-06 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027553+0000", 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.65e-06 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027556+0000", 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.892e-06 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027782+0000", 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00022592599999999999 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027786+0000", 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.501e-06 2024-05-15T23:29:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320048+0000", 2024-05-15T23:29:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29226260300000001 2024-05-15T23:29:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320055+0000", 2024-05-15T23:29:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7309999999999996e-06 2024-05-15T23:29:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330041 4.b 4.b7bfdfdb (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027559+0000", 2024-05-15T23:29:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.33997545, 2024-05-15T23:29:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.292511574, 2024-05-15T23:29:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330041 2024-05-15T23:29:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027559+0000", 2024-05-15T23:29:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027559+0000", 2024-05-15T23:29:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027559+0000", 2024-05-15T23:29:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4400000000000001e-07 2024-05-15T23:29:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027560+0000", 2024-05-15T23:29:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3030000000000001e-06 2024-05-15T23:29:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027563+0000", 2024-05-15T23:29:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7989999999999998e-06 2024-05-15T23:29:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027566+0000", 2024-05-15T23:29:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3900000000000002e-06 2024-05-15T23:29:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027804+0000", 2024-05-15T23:29:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00023764500000000001 2024-05-15T23:29:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027811+0000", 2024-05-15T23:29:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4350000000000002e-06 2024-05-15T23:29:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320066+0000", 2024-05-15T23:29:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29225564599999998 2024-05-15T23:29:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320070+0000", 2024-05-15T23:29:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0119999999999999e-06 2024-05-15T23:29:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330051 4.b 4.b4ae7c1b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027569+0000", 2024-05-15T23:29:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.33996494400003, 2024-05-15T23:29:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29253093299999999, 2024-05-15T23:29:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330051 2024-05-15T23:29:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027569+0000", 2024-05-15T23:29:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027569+0000", 2024-05-15T23:29:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027569+0000", 2024-05-15T23:29:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8200000000000001e-07 2024-05-15T23:29:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027571+0000", 2024-05-15T23:29:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7120000000000001e-06 2024-05-15T23:29:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027574+0000", 2024-05-15T23:29:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1389999999999999e-06 2024-05-15T23:29:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027577+0000", 2024-05-15T23:29:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6240000000000002e-06 2024-05-15T23:29:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027833+0000", 2024-05-15T23:29:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025610100000000001 2024-05-15T23:29:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027836+0000", 2024-05-15T23:29:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4070000000000002e-06 2024-05-15T23:29:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320095+0000", 2024-05-15T23:29:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29225901999999998 2024-05-15T23:29:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320100+0000", 2024-05-15T23:29:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6480000000000002e-06 2024-05-15T23:29:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330066 4.b 4.4e710d8b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027579+0000", 2024-05-15T23:29:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.33995458499999, 2024-05-15T23:29:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.292543579, 2024-05-15T23:29:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330066 2024-05-15T23:29:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027579+0000", 2024-05-15T23:29:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027579+0000", 2024-05-15T23:29:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027580+0000", 2024-05-15T23:29:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0499999999999999e-07 2024-05-15T23:29:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027584+0000", 2024-05-15T23:29:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.245e-06 2024-05-15T23:29:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027587+0000", 2024-05-15T23:29:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9579999999999999e-06 2024-05-15T23:29:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027590+0000", 2024-05-15T23:29:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.196e-06 2024-05-15T23:29:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027856+0000", 2024-05-15T23:29:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026541500000000002 2024-05-15T23:29:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027859+0000", 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.659e-06 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320119+0000", 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29225978499999999 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320123+0000", 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0160000000000002e-06 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330077 4.b 4.59c2b1db (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027594+0000", 2024-05-15T23:29:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.33994011499999, 2024-05-15T23:29:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29255331299999998, 2024-05-15T23:29:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330077 2024-05-15T23:29:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027594+0000", 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027594+0000", 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027594+0000", 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7e-07 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027596+0000", 2024-05-15T23:29:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9810000000000002e-06 2024-05-15T23:29:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027601+0000", 2024-05-15T23:29:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3370000000000001e-06 2024-05-15T23:29:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027606+0000", 2024-05-15T23:29:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8690000000000003e-06 2024-05-15T23:29:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027883+0000", 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027723899999999998 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027886+0000", 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3019999999999999e-06 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320143+0000", 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.292256976 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320147+0000", 2024-05-15T23:29:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1389999999999997e-06 2024-05-15T23:29:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330169 4.b 4.f7b3457b (undecoded) ondisk+retry+write+known_if_redirected e363)", 2024-05-15T23:29:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:07.027611+0000", 2024-05-15T23:29:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 595.33992331299999, 2024-05-15T23:29:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29255992400000003, 2024-05-15T23:29:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-05-15T23:29:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330169 2024-05-15T23:29:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027611+0000", 2024-05-15T23:29:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027611+0000", 2024-05-15T23:29:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027611+0000", 2024-05-15T23:29:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0200000000000002e-07 2024-05-15T23:29:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027614+0000", 2024-05-15T23:29:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4279999999999999e-06 2024-05-15T23:29:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027619+0000", 2024-05-15T23:29:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2009999999999998e-06 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027624+0000", 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2789999999999998e-06 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027899+0000", 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027479700000000002 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.027903+0000", 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4809999999999998e-06 2024-05-15T23:29:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320167+0000", 2024-05-15T23:29:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29226418500000001 2024-05-15T23:29:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:07.320171+0000", 2024-05-15T23:29:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9509999999999999e-06 2024-05-15T23:29:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330066 4.b 4:d1b08e72:::benchmark_data_smithi090_104448_object156828:head [delete] snapc 0=[] RETRY=5 ondisk+retry+write+known_if_redirected e365)", 2024-05-15T23:29:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:10.016971+0000", 2024-05-15T23:29:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 592.35056336000002, 2024-05-15T23:29:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12479114700000001, 2024-05-15T23:29:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330066 2024-05-15T23:29:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016971+0000", 2024-05-15T23:29:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016971+0000", 2024-05-15T23:29:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016971+0000", 2024-05-15T23:29:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2900000000000004e-07 2024-05-15T23:29:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016974+0000", 2024-05-15T23:29:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4210000000000002e-06 2024-05-15T23:29:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016985+0000", 2024-05-15T23:29:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1239e-05 2024-05-15T23:29:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016990+0000", 2024-05-15T23:29:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5690000000000004e-06 2024-05-15T23:29:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.116915+0000", 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.099924916000000003 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.117029+0000", 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011380800000000001 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.117138+0000", 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000108933 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.131447+0000", 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014309240000000001 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141691+0000", 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010243835999999999 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141730+0000", 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8302e-05 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141762+0000", 2024-05-15T23:29:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2354000000000002e-05 2024-05-15T23:29:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330077 4.b 4:db8d439a:::benchmark_data_smithi090_104448_object156839:head [delete] snapc 0=[] RETRY=5 ondisk+retry+write+known_if_redirected e365)", 2024-05-15T23:29:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:10.016997+0000", 2024-05-15T23:29:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 592.35053701100003, 2024-05-15T23:29:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.124849182, 2024-05-15T23:29:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330077 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016997+0000", 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016997+0000", 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.016998+0000", 2024-05-15T23:29:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9599999999999999e-07 2024-05-15T23:29:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017000+0000", 2024-05-15T23:29:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2749999999999998e-06 2024-05-15T23:29:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017005+0000", 2024-05-15T23:29:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7779999999999998e-06 2024-05-15T23:29:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017009+0000", 2024-05-15T23:29:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.357e-06 2024-05-15T23:29:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.120900+0000", 2024-05-15T23:29:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.103890867 2024-05-15T23:29:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.121033+0000", 2024-05-15T23:29:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000132874 2024-05-15T23:29:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:29:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.121166+0000", 2024-05-15T23:29:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013360100000000001 2024-05-15T23:29:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.131491+0000", 2024-05-15T23:29:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010325175000000001 2024-05-15T23:29:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141770+0000", 2024-05-15T23:29:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010278123 2024-05-15T23:29:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141807+0000", 2024-05-15T23:29:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7818000000000003e-05 2024-05-15T23:29:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141846+0000", 2024-05-15T23:29:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8818e-05 2024-05-15T23:29:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.25441.0:330169 4.b 4:dea2cdef:::benchmark_data_smithi090_104448_object156931:head [delete] snapc 0=[] RETRY=5 ondisk+retry+write+known_if_redirected e365)", 2024-05-15T23:29:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:19:10.017013+0000", 2024-05-15T23:29:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 592.35052094699995, 2024-05-15T23:29:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.125039171, 2024-05-15T23:29:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.25441", 2024-05-15T23:29:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/154552803", 2024-05-15T23:29:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 330169 2024-05-15T23:29:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017013+0000", 2024-05-15T23:29:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017013+0000", 2024-05-15T23:29:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017014+0000", 2024-05-15T23:29:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9299999999999999e-07 2024-05-15T23:29:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017015+0000", 2024-05-15T23:29:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8309999999999999e-06 2024-05-15T23:29:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017023+0000", 2024-05-15T23:29:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8169999999999999e-06 2024-05-15T23:29:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.017035+0000", 2024-05-15T23:29:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1515000000000001e-05 2024-05-15T23:29:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.126480+0000", 2024-05-15T23:29:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.10944483200000001 2024-05-15T23:29:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.126563+0000", 2024-05-15T23:29:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3803999999999999e-05 2024-05-15T23:29:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:29:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.126662+0000", 2024-05-15T23:29:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9072000000000003e-05 2024-05-15T23:29:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141907+0000", 2024-05-15T23:29:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015245015000000001 2024-05-15T23:29:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.141974+0000", 2024-05-15T23:29:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7088999999999994e-05 2024-05-15T23:29:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.142017+0000", 2024-05-15T23:29:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2011000000000001e-05 2024-05-15T23:29:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:19:10.142052+0000", 2024-05-15T23:29:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5791999999999999e-05 2024-05-15T23:29:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:141933 6.4 6:293b50f6:::benchmark_data_smithi090_131278_object6739:head [delete] snapc 0=[] ondisk+write+known_if_redirected e516)", 2024-05-15T23:29:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:26:43.228423+0000", 2024-05-15T23:29:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 139.13911135800001, 2024-05-15T23:29:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.232009509, 2024-05-15T23:29:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 141933 2024-05-15T23:29:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.228423+0000", 2024-05-15T23:29:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.228423+0000", 2024-05-15T23:29:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.228425+0000", 2024-05-15T23:29:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.046e-06 2024-05-15T23:29:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.228430+0000", 2024-05-15T23:29:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5119999999999999e-06 2024-05-15T23:29:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.228442+0000", 2024-05-15T23:29:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2047e-05 2024-05-15T23:29:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.228449+0000", 2024-05-15T23:29:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6189999999999996e-06 2024-05-15T23:29:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.228503+0000", 2024-05-15T23:29:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3820999999999999e-05 2024-05-15T23:29:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.228613+0000", 2024-05-15T23:29:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011066899999999999 2024-05-15T23:29:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2024-05-15T23:29:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.228715+0000", 2024-05-15T23:29:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101679 2024-05-15T23:29:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.232182+0000", 2024-05-15T23:29:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034672449999999999 2024-05-15T23:29:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.460352+0000", 2024-05-15T23:29:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.228169918 2024-05-15T23:29:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.460395+0000", 2024-05-15T23:29:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2599999999999999e-05 2024-05-15T23:29:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.460432+0000", 2024-05-15T23:29:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7353e-05 2024-05-15T23:29:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:02.523 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:29:02.647 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:29:02.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:02.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:29:02.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:29:02.780 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:02.846 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:29:02.858 INFO:teuthology.orchestra.run.smithi084.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-05-15T23:29:03.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:02 smithi084 ceph-mon[39784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:29:03.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:02 smithi084 ceph-mon[39784]: osdmap e555: 12 total, 12 up, 9 in 2024-05-15T23:29:03.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:02 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:29:03.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:02 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:29:03.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1512695977' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:29:03.128 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:29:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:03.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:29:03.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:29:03.264 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:02 smithi090 ceph-mon[44434]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:29:03.264 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:02 smithi090 ceph-mon[44434]: osdmap e555: 12 total, 12 up, 9 in 2024-05-15T23:29:03.264 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:02 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:29:03.264 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:02 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:29:03.265 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1512695977' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:29:03.267 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:03.288 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:29:03.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:03.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:29:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:02 smithi088 ceph-mon[44699]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:29:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:02 smithi088 ceph-mon[44699]: osdmap e555: 12 total, 12 up, 9 in 2024-05-15T23:29:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:02 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:29:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:02 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:29:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1512695977' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:29:03.392 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:29:03.421 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:03 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:29:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1512695977' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:29:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:03 smithi084 ceph-mon[39784]: osdmap e556: 12 total, 12 up, 9 in 2024-05-15T23:29:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:03 smithi084 ceph-mon[39784]: pgmap v1488: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:29:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:03 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:29:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1512695977' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:29:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:03 smithi090 ceph-mon[44434]: osdmap e556: 12 total, 12 up, 9 in 2024-05-15T23:29:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:03 smithi090 ceph-mon[44434]: pgmap v1488: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:29:04.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:03 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:29:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1512695977' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:29:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:03 smithi088 ceph-mon[44699]: osdmap e556: 12 total, 12 up, 9 in 2024-05-15T23:29:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:03 smithi088 ceph-mon[44699]: pgmap v1488: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:29:04.551 DEBUG:teuthology.orchestra.run.smithi090:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-05-15T23:29:04.554 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T23:29:04.554 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T23:29:04.689 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:hints = 1 2024-05-15T23:29:04.689 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T23:29:04.689 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object prefix: benchmark_data_smithi090_143373 2024-05-15T23:29:04.690 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:29:04.690 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 0 0 0 0 0 0 - 0 2024-05-15T23:29:04.930 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T23:29:05.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:04 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:29:05.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:04 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:29:05.200 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:29:05.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:05.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:29:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:04 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:29:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:04 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:29:05.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:04 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:29:05.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:04 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:29:05.349 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:05.373 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:05.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:05.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:05.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:29:05.517 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:05.689 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 1 16 5419 5403 337.67 337.688 0.00426475 0.00295594 2024-05-15T23:29:05.714 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:29:05.721 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:29:05.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:05.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:29:05.828 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:05.868 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:05.901 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:05.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:05.905 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:05.985 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:05 smithi084 ceph-mon[39784]: osdmap e557: 12 total, 12 up, 9 in 2024-05-15T23:29:05.985 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:05 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:29:05.985 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:05 smithi084 ceph-mon[39784]: pgmap v1490: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:29:06.005 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:29:06.036 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:06.060 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:29:06.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:06.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:29:06.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:29:06.199 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:05 smithi090 ceph-mon[44434]: osdmap e557: 12 total, 12 up, 9 in 2024-05-15T23:29:06.199 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:05 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:29:06.199 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:05 smithi090 ceph-mon[44434]: pgmap v1490: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:29:06.202 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:06.233 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:29:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:06.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:29:06.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:05 smithi088 ceph-mon[44699]: osdmap e557: 12 total, 12 up, 9 in 2024-05-15T23:29:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:05 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:29:06.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:05 smithi088 ceph-mon[44699]: pgmap v1490: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 801 GiB / 805 GiB avail 2024-05-15T23:29:06.339 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:29:06.381 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:06.537 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:29:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:06.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:29:06.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:29:06.643 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:29:06.674 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:06.689 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 2 16 10397 10381 324.386 311.125 0.00142909 0.00306472 2024-05-15T23:29:06.696 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:29:06.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:06.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:29:06.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:06.841 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:06.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:06 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:29:06.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:06 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:29:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:06.936 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:29:07.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:07.022 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:07.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:06 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:29:07.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:06 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:29:07.123 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:29:07.153 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:07.178 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:29:07.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:07.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:29:07.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:29:07.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:06 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:29:07.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:06 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:29:07.324 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:07.351 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:29:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:07.355 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:29:07.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:29:07.493 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:07.513 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:29:07.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:07.516 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:29:07.617 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:29:07.644 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:07.690 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 3 15 13953 13938 290.354 222.312 0.0056208 0.00344208 2024-05-15T23:29:08.021 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:07 smithi084 ceph-mon[39784]: osdmap e558: 12 total, 12 up, 9 in 2024-05-15T23:29:08.021 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:07 smithi084 ceph-mon[39784]: pgmap v1492: 17 pgs: 17 active+clean; 277 MiB data, 4.5 GiB used, 800 GiB / 805 GiB avail; 53 MiB/s wr, 844 op/s 2024-05-15T23:29:08.232 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:29:08.237 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:29:08.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:08.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:29:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:07 smithi090 ceph-mon[44434]: osdmap e558: 12 total, 12 up, 9 in 2024-05-15T23:29:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:07 smithi090 ceph-mon[44434]: pgmap v1492: 17 pgs: 17 active+clean; 277 MiB data, 4.5 GiB used, 800 GiB / 805 GiB avail; 53 MiB/s wr, 844 op/s 2024-05-15T23:29:08.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:07 smithi088 ceph-mon[44699]: osdmap e558: 12 total, 12 up, 9 in 2024-05-15T23:29:08.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:07 smithi088 ceph-mon[44699]: pgmap v1492: 17 pgs: 17 active+clean; 277 MiB data, 4.5 GiB used, 800 GiB / 805 GiB avail; 53 MiB/s wr, 844 op/s 2024-05-15T23:29:08.339 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:29:08.368 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:08.390 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:29:08.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:08.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:29:08.440 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:29:08.473 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:29:08.494 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:29:08.507 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:29:08.526 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:08.551 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:29:08.552 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:29:08.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:08.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:29:08.585 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:29:08.616 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:29:08.649 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:29:08.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:08.684 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:29:08.690 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 4 16 18131 18115 283.025 261.062 0.00227963 0.00352944 2024-05-15T23:29:08.695 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:08.717 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:29:08.729 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:08.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:08.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:08.754 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:29:08.791 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:29:08.827 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:29:08.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:29:08.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:29:08.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:29:08.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:29:08.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:29:08.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:29:08.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:29:08.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:29:08.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:29:08.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:29:08.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:29:08.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:29:08.841 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:29:08.879 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:08.954 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:29:08.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:08.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:29:09.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3596045326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:29:09.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:09.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:09.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:09.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:29:09.089 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:09.108 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:29:09.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:09.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:29:09.212 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:29:09.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3596045326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:29:09.232 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:09.232 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:09.232 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:09.238 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3596045326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:29:09.238 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:09.238 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:09.239 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:09.241 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:09.266 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:29:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:09.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:29:09.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:29:09.417 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:09.690 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 5 16 21497 21481 268.484 210.375 0.00416044 0.00372249 2024-05-15T23:29:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:09 smithi090 ceph-mon[44434]: pgmap v1493: 17 pgs: 17 active+clean; 442 MiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-05-15T23:29:10.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:09 smithi088 ceph-mon[44699]: pgmap v1493: 17 pgs: 17 active+clean; 442 MiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-05-15T23:29:10.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:10.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:09 smithi084 ceph-mon[39784]: pgmap v1493: 17 pgs: 17 active+clean; 442 MiB data, 4.8 GiB used, 800 GiB / 805 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-05-15T23:29:10.690 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 6 16 25457 25441 264.981 247.5 0.00423921 0.00377134 2024-05-15T23:29:11.501 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:11 smithi084 ceph-mon[39784]: pgmap v1494: 17 pgs: 17 active+clean; 1.0 GiB data, 6.2 GiB used, 798 GiB / 805 GiB avail; 133 MiB/s wr, 2.12k op/s 2024-05-15T23:29:11.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:11 smithi090 ceph-mon[44434]: pgmap v1494: 17 pgs: 17 active+clean; 1.0 GiB data, 6.2 GiB used, 798 GiB / 805 GiB avail; 133 MiB/s wr, 2.12k op/s 2024-05-15T23:29:11.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:11 smithi088 ceph-mon[44699]: pgmap v1494: 17 pgs: 17 active+clean; 1.0 GiB data, 6.2 GiB used, 798 GiB / 805 GiB avail; 133 MiB/s wr, 2.12k op/s 2024-05-15T23:29:11.586 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:29:11.586 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:29:11.691 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 7 16 29076 29060 259.409 226.188 0.00326663 0.00385279 2024-05-15T23:29:11.920 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:29:11.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:11.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:29:12.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:12.065 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:12.091 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:12.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:12.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:12.198 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:29:12.228 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:12.250 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:29:12.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:12.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:29:12.258 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:29:12.258 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:29:12.258 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 558, 2024-05-15T23:29:12.258 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "3999fb7e-130e-11ef-bc99-c7b262605968", 2024-05-15T23:29:12.259 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-05-15T22:56:41.212493+0000", 2024-05-15T23:29:12.259 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-05-15T23:29:06.866194+0000", 2024-05-15T23:29:12.259 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-05-15T23:26:58.424794+0000", 2024-05-15T23:29:12.259 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-05-15T23:28:16.134275+0000", 2024-05-15T23:29:12.259 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-15T23:29:12.259 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5804032, 2024-05-15T23:29:12.259 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-05-15T23:29:12.259 INFO:teuthology.orchestra.run.smithi084.stdout: "nodeep-scrub", 2024-05-15T23:29:12.259 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-05-15T23:29:12.260 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-05-15T23:29:12.260 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-05-15T23:29:12.260 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-05-15T23:29:12.260 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:29:12.260 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 30, 2024-05-15T23:29:12.260 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-05-15T23:29:12.260 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-15T23:29:12.260 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-15T23:29:12.260 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-05-15T23:29:12.261 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 7, 2024-05-15T23:29:12.261 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 12, 2024-05-15T23:29:12.261 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-05-15T23:29:12.261 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-05-15T23:29:12.261 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-05-15T23:29:12.261 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-05-15T23:29:12.261 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-05-15T23:29:12.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.261 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-05-15T23:29:12.261 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-05-15T23:29:12.261 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-05-15T22:59:38.603575+0000", 2024-05-15T23:29:12.261 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-05-15T23:29:12.261 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-05-15T23:29:12.262 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-05-15T23:29:12.262 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2024-05-15T23:29:12.262 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-05-15T23:29:12.262 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-05-15T23:29:12.262 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-05-15T23:29:12.262 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-05-15T23:29:12.262 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T23:29:12.262 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T23:29:12.262 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-05-15T23:29:12.262 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-05-15T23:29:12.262 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 1, 2024-05-15T23:29:12.262 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 1, 2024-05-15T23:29:12.263 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 1, 2024-05-15T23:29:12.263 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 1, 2024-05-15T23:29:12.263 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 1, 2024-05-15T23:29:12.263 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-05-15T23:29:12.263 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-05-15T23:29:12.263 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-05-15T23:29:12.263 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-05-15T23:29:12.263 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-05-15T23:29:12.263 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-05-15T23:29:12.263 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-05-15T23:29:12.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.263 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "16", 2024-05-15T23:29:12.263 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-05-15T23:29:12.263 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T23:29:12.264 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T23:29:12.264 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-05-15T23:29:12.264 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-05-15T23:29:12.264 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-05-15T23:29:12.264 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-05-15T23:29:12.264 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-05-15T23:29:12.264 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-05-15T23:29:12.264 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-05-15T23:29:12.264 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-05-15T23:29:12.264 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-05-15T23:29:12.264 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-05-15T23:29:12.264 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-05-15T23:29:12.265 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-05-15T23:29:12.265 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-05-15T23:29:12.265 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-05-15T23:29:12.265 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-05-15T23:29:12.265 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T23:29:12.265 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T23:29:12.265 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T23:29:12.265 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-05-15T23:29:12.265 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-05-15T23:29:12.265 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-05-15T23:29:12.265 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-05-15T23:29:12.266 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-05-15T23:29:12.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.266 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-05-15T23:29:12.266 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-05-15T23:29:12.266 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-05-15T23:29:12.266 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-05-15T23:29:12.266 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-05-15T23:29:12.266 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T23:29:12.266 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-05-15T23:29:12.266 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-05-15T23:29:12.267 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-05-15T23:29:12.267 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-05-15T23:29:12.267 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-05-15T23:29:12.267 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-05-15T23:29:12.267 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-05-15T23:29:12.267 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-05-15T23:29:12.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.267 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-05-15T23:29:12.267 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-05-15T23:29:12.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.267 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-05-15T23:29:12.267 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-05-15T23:29:12.267 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 11.760000228881836, 2024-05-15T23:29:12.268 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 11.760000228881836, 2024-05-15T23:29:12.268 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.17000000178813934, 2024-05-15T23:29:12.268 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-05-15T23:29:12.268 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-05-15T23:29:12.268 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.96207165718078613, 2024-05-15T23:29:12.268 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 0.95999997854232788, 2024-05-15T23:29:12.268 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.95999997854232788 2024-05-15T23:29:12.268 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.268 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-05-15T23:29:12.268 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_5", 2024-05-15T23:29:12.268 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-05-15T23:28:50.243463+0000", 2024-05-15T23:29:12.269 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-05-15T23:29:12.269 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-05-15T23:29:12.269 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-05-15T23:29:12.269 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2024-05-15T23:29:12.269 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-05-15T23:29:12.269 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-05-15T23:29:12.269 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-05-15T23:29:12.269 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-05-15T23:29:12.269 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T23:29:12.269 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T23:29:12.269 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-05-15T23:29:12.269 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-05-15T23:29:12.269 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 16, 2024-05-15T23:29:12.269 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 16, 2024-05-15T23:29:12.270 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 16, 2024-05-15T23:29:12.270 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 16, 2024-05-15T23:29:12.270 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 16, 2024-05-15T23:29:12.270 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-05-15T23:29:12.270 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-05-15T23:29:12.270 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-05-15T23:29:12.270 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-05-15T23:29:12.270 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-05-15T23:29:12.270 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-05-15T23:29:12.270 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-05-15T23:29:12.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.270 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "554", 2024-05-15T23:29:12.271 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-05-15T23:29:12.271 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T23:29:12.271 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T23:29:12.271 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-05-15T23:29:12.271 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-05-15T23:29:12.271 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-05-15T23:29:12.271 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-05-15T23:29:12.271 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-05-15T23:29:12.271 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-05-15T23:29:12.271 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-05-15T23:29:12.271 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-05-15T23:29:12.271 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-05-15T23:29:12.271 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-05-15T23:29:12.272 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-05-15T23:29:12.272 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-05-15T23:29:12.272 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-05-15T23:29:12.272 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-05-15T23:29:12.272 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-05-15T23:29:12.272 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T23:29:12.272 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T23:29:12.272 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T23:29:12.272 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-05-15T23:29:12.272 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-05-15T23:29:12.272 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-05-15T23:29:12.272 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-05-15T23:29:12.272 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-05-15T23:29:12.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.272 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-05-15T23:29:12.273 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-05-15T23:29:12.273 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-05-15T23:29:12.273 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-05-15T23:29:12.273 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-05-15T23:29:12.273 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T23:29:12.273 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-05-15T23:29:12.273 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-05-15T23:29:12.273 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-05-15T23:29:12.273 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-05-15T23:29:12.273 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-05-15T23:29:12.273 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-05-15T23:29:12.273 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-05-15T23:29:12.273 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-05-15T23:29:12.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.274 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-05-15T23:29:12.274 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-05-15T23:29:12.274 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 3, 2024-05-15T23:29:12.274 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 3, 2024-05-15T23:29:12.274 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.75, 2024-05-15T23:29:12.274 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.25, 2024-05-15T23:29:12.274 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.25, 2024-05-15T23:29:12.274 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.96207165718078613, 2024-05-15T23:29:12.274 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 0.95999997854232788, 2024-05-15T23:29:12.274 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.95999997854232788 2024-05-15T23:29:12.274 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.274 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:29:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-05-15T23:29:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-05-15T23:29:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "80159cc8-1b6c-4aa6-a114-966786d7be1e", 2024-05-15T23:29:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:29:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-05-15T23:29:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-05-15T23:29:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:29:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 8, 2024-05-15T23:29:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 355, 2024-05-15T23:29:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 397, 2024-05-15T23:29:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 283, 2024-05-15T23:29:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 356, 2024-05-15T23:29:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:29:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:29:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-05-15T23:29:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:29:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:29:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-05-15T23:29:12.277 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:29:12.277 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.277 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.277 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:29:12.277 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.277 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.277 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-05-15T23:29:12.277 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:29:12.277 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.277 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.277 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:29:12.277 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.278 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.278 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-05-15T23:29:12.278 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:29:12.278 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.278 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.278 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/3077815271", 2024-05-15T23:29:12.278 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/3077815271", 2024-05-15T23:29:12.278 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/3077815271", 2024-05-15T23:29:12.278 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/3077815271", 2024-05-15T23:29:12.278 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:29:12.278 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:29:12.278 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "9c2c76c9-ce0f-4c2d-9d96-c87a10187607", 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:29:12.279 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:29:12.280 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.280 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.280 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-05-15T23:29:12.280 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:29:12.280 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.280 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.280 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:29:12.280 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.280 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.280 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-05-15T23:29:12.280 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:29:12.281 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.281 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.281 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:29:12.281 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.281 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.281 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-05-15T23:29:12.281 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:29:12.281 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.281 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.281 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:29:12.281 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.282 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.282 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-05-15T23:29:12.282 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:29:12.282 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.282 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.282 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/2140830229", 2024-05-15T23:29:12.282 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/2140830229", 2024-05-15T23:29:12.282 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/2140830229", 2024-05-15T23:29:12.282 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/2140830229", 2024-05-15T23:29:12.282 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:29:12.282 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:29:12.282 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "e8514daa-b5f9-4f2f-a432-53f4a4eb86bc", 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 20, 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 261, 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 291, 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 262, 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:29:12.283 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:29:12.284 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.284 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.284 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-05-15T23:29:12.284 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:29:12.284 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.284 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.284 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:29:12.284 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.284 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.284 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-05-15T23:29:12.284 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:29:12.285 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.285 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.285 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:29:12.285 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.285 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.285 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-05-15T23:29:12.285 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:29:12.285 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.285 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.285 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:29:12.286 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.286 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.286 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-05-15T23:29:12.286 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:29:12.286 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.286 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.286 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/941507493", 2024-05-15T23:29:12.286 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/941507493", 2024-05-15T23:29:12.286 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/941507493", 2024-05-15T23:29:12.286 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/941507493", 2024-05-15T23:29:12.286 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:29:12.287 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:29:12.287 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:29:12.287 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.287 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-05-15T23:29:12.287 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "5d9965f9-c0b2-48ce-869c-582990985623", 2024-05-15T23:29:12.287 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:29:12.287 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:29:12.287 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:29:12.287 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:29:12.287 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:29:12.287 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:29:12.287 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 26, 2024-05-15T23:29:12.288 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:29:12.288 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:29:12.288 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:29:12.288 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:29:12.288 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.288 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.288 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-05-15T23:29:12.288 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:29:12.288 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.288 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.288 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:29:12.288 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.289 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.289 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-05-15T23:29:12.289 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:29:12.289 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.289 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.289 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:29:12.289 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.289 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.289 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-05-15T23:29:12.289 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:29:12.289 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.290 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.290 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:29:12.290 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.290 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.290 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-05-15T23:29:12.290 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:29:12.290 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.290 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.290 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2204039828", 2024-05-15T23:29:12.290 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2204039828", 2024-05-15T23:29:12.290 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2204039828", 2024-05-15T23:29:12.291 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2204039828", 2024-05-15T23:29:12.291 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:29:12.291 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:29:12.291 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:29:12.291 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.291 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-05-15T23:29:12.291 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4bc50523-8480-44d2-8d76-31479e839c85", 2024-05-15T23:29:12.291 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:29:12.291 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:29:12.291 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:29:12.291 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:29:12.291 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:29:12.292 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:29:12.292 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-05-15T23:29:12.292 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:29:12.292 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:29:12.292 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:29:12.292 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:29:12.292 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.292 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.292 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6800", 2024-05-15T23:29:12.292 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:29:12.292 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.292 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.292 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6801", 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6803", 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.293 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.294 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:29:12.294 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.294 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.294 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6802", 2024-05-15T23:29:12.294 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:29:12.294 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.294 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.294 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6800/2267955292", 2024-05-15T23:29:12.294 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6801/2267955292", 2024-05-15T23:29:12.294 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6803/2267955292", 2024-05-15T23:29:12.295 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6802/2267955292", 2024-05-15T23:29:12.295 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:29:12.295 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:29:12.295 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:29:12.295 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.295 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-05-15T23:29:12.295 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20", 2024-05-15T23:29:12.295 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:29:12.295 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:29:12.295 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:29:12.295 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:29:12.295 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:29:12.295 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:29:12.296 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 37, 2024-05-15T23:29:12.296 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:29:12.296 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:29:12.296 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:29:12.296 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:29:12.296 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.296 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.296 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6804", 2024-05-15T23:29:12.296 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:29:12.296 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.296 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.296 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:29:12.297 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.297 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.297 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6805", 2024-05-15T23:29:12.297 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:29:12.297 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.297 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.297 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:29:12.297 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.297 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.297 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6807", 2024-05-15T23:29:12.297 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:29:12.298 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.298 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.298 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:29:12.298 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.298 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.298 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6806", 2024-05-15T23:29:12.298 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:29:12.298 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.298 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.298 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6804/661041538", 2024-05-15T23:29:12.299 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6805/661041538", 2024-05-15T23:29:12.299 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6807/661041538", 2024-05-15T23:29:12.299 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6806/661041538", 2024-05-15T23:29:12.299 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:29:12.299 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:29:12.299 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:29:12.299 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.299 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-05-15T23:29:12.299 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f", 2024-05-15T23:29:12.299 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:29:12.299 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-05-15T23:29:12.300 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-05-15T23:29:12.300 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:29:12.300 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:29:12.300 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:29:12.300 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 46, 2024-05-15T23:29:12.300 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 374, 2024-05-15T23:29:12.300 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:29:12.300 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:29:12.300 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:29:12.300 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.300 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.300 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6808", 2024-05-15T23:29:12.300 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:29:12.300 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.301 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.301 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:29:12.301 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.301 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.301 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6809", 2024-05-15T23:29:12.301 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:29:12.301 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.301 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.301 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:29:12.301 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.302 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.302 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6811", 2024-05-15T23:29:12.302 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:29:12.302 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.302 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.302 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:29:12.302 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.302 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.302 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6810", 2024-05-15T23:29:12.302 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:29:12.302 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.302 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.303 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6808/380856869", 2024-05-15T23:29:12.303 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6809/380856869", 2024-05-15T23:29:12.303 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6811/380856869", 2024-05-15T23:29:12.303 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6810/380856869", 2024-05-15T23:29:12.303 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:29:12.303 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:29:12.303 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:29:12.303 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.303 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-05-15T23:29:12.303 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34", 2024-05-15T23:29:12.303 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:29:12.304 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:29:12.304 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:29:12.304 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:29:12.304 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:29:12.304 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:29:12.304 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 53, 2024-05-15T23:29:12.304 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:29:12.304 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:29:12.304 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:29:12.304 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:29:12.304 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.304 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.304 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6812", 2024-05-15T23:29:12.305 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:29:12.305 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.305 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.305 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:29:12.305 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.305 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.305 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6813", 2024-05-15T23:29:12.305 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:29:12.305 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.305 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.306 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:29:12.306 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.306 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.306 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6815", 2024-05-15T23:29:12.306 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:29:12.306 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.306 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.306 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:29:12.306 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.306 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.306 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6814", 2024-05-15T23:29:12.307 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:29:12.307 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.307 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.307 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6812/1044709063", 2024-05-15T23:29:12.307 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6813/1044709063", 2024-05-15T23:29:12.307 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6815/1044709063", 2024-05-15T23:29:12.307 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6814/1044709063", 2024-05-15T23:29:12.307 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:29:12.307 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:29:12.307 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:29:12.307 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-05-15T23:29:12.308 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "02cab286-d487-4a66-982c-3cf28f55e851", 2024-05-15T23:29:12.308 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:29:12.308 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-05-15T23:29:12.308 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-05-15T23:29:12.308 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 0.54486083984375, 2024-05-15T23:29:12.308 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 61, 2024-05-15T23:29:12.308 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 484, 2024-05-15T23:29:12.308 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 520, 2024-05-15T23:29:12.308 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 202, 2024-05-15T23:29:12.308 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 485, 2024-05-15T23:29:12.308 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:29:12.309 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:29:12.309 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.309 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.309 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6800", 2024-05-15T23:29:12.309 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:29:12.309 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.309 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.309 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:29:12.309 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.309 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.309 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6801", 2024-05-15T23:29:12.310 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:29:12.310 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.310 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.310 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:29:12.310 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.310 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.310 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6803", 2024-05-15T23:29:12.310 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:29:12.310 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.310 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.310 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:29:12.311 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.311 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.311 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6802", 2024-05-15T23:29:12.311 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:29:12.311 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.311 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.311 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6800/703598286", 2024-05-15T23:29:12.311 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6801/703598286", 2024-05-15T23:29:12.311 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6803/703598286", 2024-05-15T23:29:12.311 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6802/703598286", 2024-05-15T23:29:12.311 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:29:12.311 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:29:12.311 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:29:12.312 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-05-15T23:29:12.312 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613", 2024-05-15T23:29:12.312 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:29:12.312 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:29:12.312 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:29:12.312 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:29:12.312 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:29:12.312 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:29:12.312 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 68, 2024-05-15T23:29:12.312 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:29:12.313 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:29:12.313 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:29:12.313 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:29:12.313 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.313 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.313 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6804", 2024-05-15T23:29:12.313 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:29:12.313 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.313 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.313 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:29:12.313 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.314 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.314 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6805", 2024-05-15T23:29:12.314 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:29:12.314 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.314 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.314 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:29:12.314 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.314 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.314 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6807", 2024-05-15T23:29:12.314 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:29:12.315 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.315 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.315 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:29:12.315 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.315 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.315 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6806", 2024-05-15T23:29:12.315 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:29:12.315 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.315 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.315 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6804/2794571133", 2024-05-15T23:29:12.316 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6805/2794571133", 2024-05-15T23:29:12.316 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6807/2794571133", 2024-05-15T23:29:12.316 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6806/2794571133", 2024-05-15T23:29:12.316 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:29:12.316 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:29:12.316 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:29:12.316 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-05-15T23:29:12.316 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473", 2024-05-15T23:29:12.316 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:29:12.316 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:29:12.316 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:29:12.317 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:29:12.317 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:29:12.317 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:29:12.317 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 74, 2024-05-15T23:29:12.317 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 477, 2024-05-15T23:29:12.317 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:29:12.317 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:29:12.317 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:29:12.317 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.317 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.317 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6808", 2024-05-15T23:29:12.317 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:29:12.318 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.318 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.318 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:29:12.318 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.318 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.318 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6809", 2024-05-15T23:29:12.318 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:29:12.318 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.318 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.318 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:29:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6811", 2024-05-15T23:29:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:29:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:29:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6810", 2024-05-15T23:29:12.319 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:29:12.320 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.320 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.320 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6808/3735380744", 2024-05-15T23:29:12.320 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6809/3735380744", 2024-05-15T23:29:12.320 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6811/3735380744", 2024-05-15T23:29:12.320 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6810/3735380744", 2024-05-15T23:29:12.320 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:29:12.320 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:29:12.320 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:29:12.320 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.320 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-05-15T23:29:12.320 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5", 2024-05-15T23:29:12.321 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:29:12.321 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:29:12.321 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:29:12.321 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:29:12.321 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:29:12.321 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:29:12.321 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 81, 2024-05-15T23:29:12.321 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:29:12.321 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:29:12.321 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:29:12.321 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:29:12.321 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.322 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.322 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6812", 2024-05-15T23:29:12.322 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:29:12.322 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.322 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.322 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:29:12.322 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.322 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.322 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6813", 2024-05-15T23:29:12.322 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:29:12.322 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.323 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.323 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:29:12.323 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.323 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.323 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6815", 2024-05-15T23:29:12.323 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:29:12.323 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.323 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.323 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:29:12.323 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:29:12.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.323 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:29:12.324 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6814", 2024-05-15T23:29:12.324 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:29:12.324 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.324 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.324 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6812/2265615270", 2024-05-15T23:29:12.324 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6813/2265615270", 2024-05-15T23:29:12.324 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6815/2265615270", 2024-05-15T23:29:12.324 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6814/2265615270", 2024-05-15T23:29:12.324 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:29:12.324 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:29:12.324 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:29:12.324 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.324 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.324 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:29:12.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-05-15T23:29:12.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-05-15T23:29:12.325 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:29:12.325 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:29:12.325 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:29:12.325 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:29:12.325 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-05-15T23:29:12.325 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:29:04.095119+0000", 2024-05-15T23:29:12.325 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 355 2024-05-15T23:29:12.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-05-15T23:29:12.325 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:29:12.326 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:29:12.326 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:29:12.326 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:29:12.326 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:29:12.326 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:29:02.134643+0000", 2024-05-15T23:29:12.326 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:29:12.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-05-15T23:29:12.326 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:29:12.326 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:29:12.326 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:29:12.326 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:29:12.326 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:29:12.327 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:28:33.820424+0000", 2024-05-15T23:29:12.327 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 261 2024-05-15T23:29:12.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-05-15T23:29:12.327 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:29:12.327 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:29:12.327 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:29:12.327 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:29:12.327 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:29:12.327 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:29:01.338231+0000", 2024-05-15T23:29:12.327 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:29:12.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-05-15T23:29:12.328 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:29:12.328 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:29:12.328 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:29:12.328 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:29:12.328 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:29:12.328 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:28:31.063832+0000", 2024-05-15T23:29:12.328 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:29:12.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-05-15T23:29:12.328 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:29:12.328 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:29:12.328 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:29:12.328 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:29:12.328 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:29:12.329 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:29:06.012891+0000", 2024-05-15T23:29:12.329 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:29:12.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.329 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-05-15T23:29:12.329 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:29:12.329 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:29:12.329 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:29:12.329 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:29:12.329 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-05-15T23:29:12.329 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:28:16.648144+0000", 2024-05-15T23:29:12.329 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:29:12.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.330 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-05-15T23:29:12.330 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:29:12.330 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:29:12.330 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:29:12.330 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:29:12.330 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:29:12.330 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:27:45.869077+0000", 2024-05-15T23:29:12.330 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:29:12.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.330 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-05-15T23:29:12.330 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:29:12.330 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:29:12.330 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:29:12.330 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:29:12.331 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-05-15T23:29:12.331 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:28:18.487447+0000", 2024-05-15T23:29:12.331 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 484 2024-05-15T23:29:12.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.331 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-05-15T23:29:12.331 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:29:12.331 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:29:12.331 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:29:12.331 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:29:12.331 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:29:12.331 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:28:25.532815+0000", 2024-05-15T23:29:12.331 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:29:12.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.332 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-05-15T23:29:12.332 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:29:12.332 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:29:12.332 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:29:12.332 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:29:12.332 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:29:12.332 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:28:55.443111+0000", 2024-05-15T23:29:12.332 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:29:12.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.332 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-05-15T23:29:12.332 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:29:12.332 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:28:58.075463+0000", 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "1.0", 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: 6, 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: 3 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.333 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:29:12.334 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [], 2024-05-15T23:29:12.334 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-05-15T23:29:12.334 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-05-15T23:29:12.334 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-05-15T23:29:12.334 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-05-15T23:29:12.334 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2581408231": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:29:12.334 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/1136984393": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:29:12.334 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2443201864": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:29:12.334 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1252671888": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:29:12.334 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1451001371": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:29:12.334 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4084814866": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:29:12.334 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4076409285": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:29:12.334 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2988977694": "2024-05-16T22:57:38.505263+0000" 2024-05-15T23:29:12.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.335 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-05-15T23:29:12.335 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-05-15T23:29:12.335 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-05-15T23:29:12.335 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-05-15T23:29:12.335 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-05-15T23:29:12.335 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-05-15T23:29:12.335 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-05-15T23:29:12.335 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-05-15T23:29:12.335 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:12.335 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [], 2024-05-15T23:29:12.335 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2024-05-15T23:29:12.335 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-05-15T23:29:12.335 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-05-15T23:29:12.336 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-05-15T23:29:12.336 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-05-15T23:29:12.336 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-05-15T23:29:12.336 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-05-15T23:29:12.336 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-05-15T23:29:12.336 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-05-15T23:29:12.336 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-05-15T23:29:12.336 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:12.336 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:29:12.356 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:29:12.406 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:12.692 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 8 16 32329 32313 252.395 203.312 0.00554765 0.00396027 2024-05-15T23:29:13.539 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3036391500' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:29:13.539 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:13 smithi084 ceph-mon[39784]: pgmap v1495: 17 pgs: 17 active+clean; 1.6 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail; 201 MiB/s wr, 3.22k op/s 2024-05-15T23:29:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3036391500' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:29:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:13 smithi090 ceph-mon[44434]: pgmap v1495: 17 pgs: 17 active+clean; 1.6 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail; 201 MiB/s wr, 3.22k op/s 2024-05-15T23:29:13.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3036391500' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:29:13.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:13 smithi088 ceph-mon[44699]: pgmap v1495: 17 pgs: 17 active+clean; 1.6 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail; 201 MiB/s wr, 3.22k op/s 2024-05-15T23:29:13.585 INFO:tasks.thrashosds.thrasher:j is {'epoch': 558, 'fsid': '3999fb7e-130e-11ef-bc99-c7b262605968', 'created': '2024-05-15T22:56:41.212493+0000', 'modified': '2024-05-15T23:29:06.866194+0000', 'last_up_change': '2024-05-15T23:26:58.424794+0000', 'last_in_change': '2024-05-15T23:28:16.134275+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 30, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T22:59:38.603575+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9620716571807861, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.9599999785423279}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-05-15T23:28:50.243463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '554', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9620716571807861, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.9599999785423279}}], 'osds': [{'osd': 0, 'uuid': '80159cc8-1b6c-4aa6-a114-966786d7be1e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 355, 'up_from': 397, 'up_thru': 283, 'down_at': 356, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6801', 'nonce': 3077815271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6802', 'nonce': 3077815271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6804', 'nonce': 3077815271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6803', 'nonce': 3077815271}]}, 'public_addr': '172.21.15.84:6801/3077815271', 'cluster_addr': '172.21.15.84:6802/3077815271', 'heartbeat_back_addr': '172.21.15.84:6804/3077815271', 'heartbeat_front_addr': '172.21.15.84:6803/3077815271', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9c2c76c9-ce0f-4c2d-9d96-c87a10187607', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6805', 'nonce': 2140830229}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6806', 'nonce': 2140830229}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6808', 'nonce': 2140830229}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6807', 'nonce': 2140830229}]}, 'public_addr': '172.21.15.84:6805/2140830229', 'cluster_addr': '172.21.15.84:6806/2140830229', 'heartbeat_back_addr': '172.21.15.84:6808/2140830229', 'heartbeat_front_addr': '172.21.15.84:6807/2140830229', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e8514daa-b5f9-4f2f-a432-53f4a4eb86bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 261, 'up_from': 291, 'up_thru': 549, 'down_at': 262, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6809', 'nonce': 941507493}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6810', 'nonce': 941507493}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6812', 'nonce': 941507493}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6811', 'nonce': 941507493}]}, 'public_addr': '172.21.15.84:6809/941507493', 'cluster_addr': '172.21.15.84:6810/941507493', 'heartbeat_back_addr': '172.21.15.84:6812/941507493', 'heartbeat_front_addr': '172.21.15.84:6811/941507493', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5d9965f9-c0b2-48ce-869c-582990985623', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6813', 'nonce': 2204039828}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6814', 'nonce': 2204039828}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6816', 'nonce': 2204039828}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6815', 'nonce': 2204039828}]}, 'public_addr': '172.21.15.84:6813/2204039828', 'cluster_addr': '172.21.15.84:6814/2204039828', 'heartbeat_back_addr': '172.21.15.84:6816/2204039828', 'heartbeat_front_addr': '172.21.15.84:6815/2204039828', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4bc50523-8480-44d2-8d76-31479e839c85', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6800', 'nonce': 2267955292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6801', 'nonce': 2267955292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6803', 'nonce': 2267955292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6802', 'nonce': 2267955292}]}, 'public_addr': '172.21.15.88:6800/2267955292', 'cluster_addr': '172.21.15.88:6801/2267955292', 'heartbeat_back_addr': '172.21.15.88:6803/2267955292', 'heartbeat_front_addr': '172.21.15.88:6802/2267955292', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'd4db8d7f-afc7-43dc-81fe-f70c290d8b20', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6804', 'nonce': 661041538}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6805', 'nonce': 661041538}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6807', 'nonce': 661041538}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6806', 'nonce': 661041538}]}, 'public_addr': '172.21.15.88:6804/661041538', 'cluster_addr': '172.21.15.88:6805/661041538', 'heartbeat_back_addr': '172.21.15.88:6807/661041538', 'heartbeat_front_addr': '172.21.15.88:6806/661041538', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'a2773454-2318-4a5b-8f31-0952b1b07f2f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6808', 'nonce': 380856869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6809', 'nonce': 380856869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6811', 'nonce': 380856869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6810', 'nonce': 380856869}]}, 'public_addr': '172.21.15.88:6808/380856869', 'cluster_addr': '172.21.15.88:6809/380856869', 'heartbeat_back_addr': '172.21.15.88:6811/380856869', 'heartbeat_front_addr': '172.21.15.88:6810/380856869', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '028ca7b6-3e00-4466-a33f-b726cd60cb34', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6812', 'nonce': 1044709063}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6813', 'nonce': 1044709063}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6815', 'nonce': 1044709063}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6814', 'nonce': 1044709063}]}, 'public_addr': '172.21.15.88:6812/1044709063', 'cluster_addr': '172.21.15.88:6813/1044709063', 'heartbeat_back_addr': '172.21.15.88:6815/1044709063', 'heartbeat_front_addr': '172.21.15.88:6814/1044709063', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '02cab286-d487-4a66-982c-3cf28f55e851', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.54486083984375, 'last_clean_begin': 61, 'last_clean_end': 484, 'up_from': 520, 'up_thru': 202, 'down_at': 485, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6800', 'nonce': 703598286}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 703598286}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 703598286}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6802', 'nonce': 703598286}]}, 'public_addr': '172.21.15.90:6800/703598286', 'cluster_addr': '172.21.15.90:6801/703598286', 'heartbeat_back_addr': '172.21.15.90:6803/703598286', 'heartbeat_front_addr': '172.21.15.90:6802/703598286', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '47d38bc7-39ba-4b35-af24-e282354dd613', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 68, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6804', 'nonce': 2794571133}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 2794571133}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 2794571133}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6806', 'nonce': 2794571133}]}, 'public_addr': '172.21.15.90:6804/2794571133', 'cluster_addr': '172.21.15.90:6805/2794571133', 'heartbeat_back_addr': '172.21.15.90:6807/2794571133', 'heartbeat_front_addr': '172.21.15.90:6806/2794571133', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'd7b2aeae-d012-41eb-9f91-e4c05c649473', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 477, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6808', 'nonce': 3735380744}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 3735380744}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 3735380744}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 3735380744}]}, 'public_addr': '172.21.15.90:6808/3735380744', 'cluster_addr': '172.21.15.90:6809/3735380744', 'heartbeat_back_addr': '172.21.15.90:6811/3735380744', 'heartbeat_front_addr': '172.21.15.90:6810/3735380744', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6812', 'nonce': 2265615270}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 2265615270}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 2265615270}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 2265615270}]}, 'public_addr': '172.21.15.90:6812/2265615270', 'cluster_addr': '172.21.15.90:6813/2265615270', 'heartbeat_back_addr': '172.21.15.90:6815/2265615270', 'heartbeat_front_addr': '172.21.15.90:6814/2265615270', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T23:29:04.095119+0000', 'dead_epoch': 355}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:29:02.134643+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:28:33.820424+0000', 'dead_epoch': 261}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:29:01.338231+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:28:31.063832+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:29:06.012891+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T23:28:16.648144+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:27:45.869077+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T23:28:18.487447+0000', 'dead_epoch': 484}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:28:25.532815+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:28:55.443111+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:28:58.075463+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [6, 3]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/2581408231': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:6800/1136984393': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:6800/2443201864': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/1252671888': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/1451001371': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/4084814866': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:0/4076409285': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/2988977694': '2024-05-16T22:57:38.505263+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-15T23:29:13.585 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:29:13.691 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 9 16 35617 35601 247.185 205.5 0.00346068 0.00404385 2024-05-15T23:29:13.902 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:29:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:13.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:29:13.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T23:29:14.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:29:14.053 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:14.074 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:29:14.080 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:29:14.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:14.084 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:29:14.185 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:29:14.212 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:14.232 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:29:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:14.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:29:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:14 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:29:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:14 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:29:14.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:14 smithi090 ceph-mon[44434]: osdmap e559: 12 total, 12 up, 9 in 2024-05-15T23:29:14.335 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:29:14.377 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:14.404 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:29:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:14.406 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:29:14.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:29:14.538 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:14.545 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:29:14.560 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:29:14.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:14.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:29:14.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:14 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:29:14.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:14 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:29:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:14 smithi088 ceph-mon[44699]: osdmap e559: 12 total, 12 up, 9 in 2024-05-15T23:29:14.583 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:29:14.601 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:29:14.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:14 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:29:14.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:14 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:29:14.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:14 smithi084 ceph-mon[39784]: osdmap e559: 12 total, 12 up, 9 in 2024-05-15T23:29:14.664 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:29:14.691 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 10 15 38022 38007 237.503 150.375 0.00565974 0.00420824 2024-05-15T23:29:14.692 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:14.714 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:29:14.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:14.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:29:14.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:29:14.856 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:29:14.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:14.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:29:14.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:15.030 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:15.055 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:15.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:15.059 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:15.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:29:15.195 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:15.203 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:29:15.221 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:29:15.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:15.225 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:29:15.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:29:15.352 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:15.372 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:29:15.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:15.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:29:15.476 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:29:15.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:15 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1973792026' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:29:15.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:15 smithi088 ceph-mon[44699]: from='client.36623 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:29:15.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:15 smithi088 ceph-mon[44699]: pgmap v1497: 17 pgs: 17 active+clean; 1.8 GiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 185 MiB/s wr, 2.96k op/s 2024-05-15T23:29:15.508 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:15.530 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:29:15.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:15.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:29:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:15 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1973792026' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:29:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:15 smithi090 ceph-mon[44434]: from='client.36623 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:29:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:15 smithi090 ceph-mon[44434]: pgmap v1497: 17 pgs: 17 active+clean; 1.8 GiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 185 MiB/s wr, 2.96k op/s 2024-05-15T23:29:15.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:15 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1973792026' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:29:15.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:15 smithi084 ceph-mon[39784]: from='client.36623 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:29:15.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:15 smithi084 ceph-mon[39784]: pgmap v1497: 17 pgs: 17 active+clean; 1.8 GiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 185 MiB/s wr, 2.96k op/s 2024-05-15T23:29:15.635 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:29:15.665 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:29:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:15.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:29:15.691 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 11 16 40708 40692 231.167 167.812 0.00481966 0.00432382 2024-05-15T23:29:15.791 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:29:15.845 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:15.930 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:29:15.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:15.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:29:16.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:16.073 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:16.099 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:16.102 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:16.204 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:29:16.248 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:16.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:16 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:29:16.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:16 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:29:16.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:16 smithi084 ceph-mon[39784]: from='client.36623 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:29:16.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:16 smithi084 ceph-mon[39784]: osdmap e560: 12 total, 12 up, 9 in 2024-05-15T23:29:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:16 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:29:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:16 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:29:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:16 smithi090 ceph-mon[44434]: from='client.36623 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:29:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:16 smithi090 ceph-mon[44434]: osdmap e560: 12 total, 12 up, 9 in 2024-05-15T23:29:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:16 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:29:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:16 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:29:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:16 smithi088 ceph-mon[44699]: from='client.36623 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:29:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:16 smithi088 ceph-mon[44699]: osdmap e560: 12 total, 12 up, 9 in 2024-05-15T23:29:16.610 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:29:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:16.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:29:16.692 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 12 16 43560 43544 226.756 178.25 0.00276619 0.00440812 2024-05-15T23:29:16.716 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:29:16.757 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:17.012 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:29:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:17.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:29:17.117 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:29:17.147 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:17.172 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:29:17.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:17.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:29:17.276 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:29:17.316 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:17 smithi084 ceph-mon[39784]: pgmap v1499: 17 pgs: 17 active+clean; 2.4 GiB data, 8.8 GiB used, 796 GiB / 805 GiB avail; 249 MiB/s wr, 3.99k op/s 2024-05-15T23:29:17.316 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:17 smithi084 ceph-mon[39784]: osdmap e561: 12 total, 12 up, 9 in 2024-05-15T23:29:17.320 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:17.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:17 smithi090 ceph-mon[44434]: pgmap v1499: 17 pgs: 17 active+clean; 2.4 GiB data, 8.8 GiB used, 796 GiB / 805 GiB avail; 249 MiB/s wr, 3.99k op/s 2024-05-15T23:29:17.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:17 smithi090 ceph-mon[44434]: osdmap e561: 12 total, 12 up, 9 in 2024-05-15T23:29:17.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:17 smithi088 ceph-mon[44699]: pgmap v1499: 17 pgs: 17 active+clean; 2.4 GiB data, 8.8 GiB used, 796 GiB / 805 GiB avail; 249 MiB/s wr, 3.99k op/s 2024-05-15T23:29:17.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:17 smithi088 ceph-mon[44699]: osdmap e561: 12 total, 12 up, 9 in 2024-05-15T23:29:17.692 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 13 16 46361 46345 222.779 175.062 0.0144227 0.00448759 2024-05-15T23:29:18.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:29:18.590 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:29:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:18.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:29:18.691 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 14 16 49043 49027 218.839 167.625 0.00239225 0.00456763 2024-05-15T23:29:18.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:18.733 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:18.756 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:18.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:18.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:18.862 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:29:18.903 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:18.974 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:29:18.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:18.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:29:19.079 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:29:19.108 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:19.131 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:29:19.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:19.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:29:19.235 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:29:19.264 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:19.285 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:29:19.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:19.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:29:19.389 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:29:19.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:19 smithi090 ceph-mon[44434]: pgmap v1501: 17 pgs: 17 active+clean; 2.5 GiB data, 9.1 GiB used, 796 GiB / 805 GiB avail; 157 MiB/s wr, 2.51k op/s 2024-05-15T23:29:19.422 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:19.446 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:29:19.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:19.454 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:29:19.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:29:19.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:19 smithi088 ceph-mon[44699]: pgmap v1501: 17 pgs: 17 active+clean; 2.5 GiB data, 9.1 GiB used, 796 GiB / 805 GiB avail; 157 MiB/s wr, 2.51k op/s 2024-05-15T23:29:19.578 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:29:19.595 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:19.608 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:29:19.620 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:29:19.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:19 smithi084 ceph-mon[39784]: pgmap v1501: 17 pgs: 17 active+clean; 2.5 GiB data, 9.1 GiB used, 796 GiB / 805 GiB avail; 157 MiB/s wr, 2.51k op/s 2024-05-15T23:29:19.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:19.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:29:19.644 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:29:19.682 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:29:19.691 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 15 16 51440 51424 214.237 149.812 0.00320778 0.00466587 2024-05-15T23:29:19.713 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:29:19.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:29:19.756 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:29:19.779 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:19.798 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:29:19.832 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:29:19.867 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:29:19.900 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:29:19.929 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:29:19.961 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:29:19.967 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:29:19.968 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:29:19.968 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:29:19.968 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:29:19.968 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:29:19.968 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:29:19.968 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:29:19.968 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:29:19.968 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:29:19.968 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:29:19.968 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:29:19.968 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:29:19.989 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:29:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:19.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:29:20.099 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:29:20.152 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:20.423 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:29:20.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:20.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:29:20.428 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:20 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1276578900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:29:20.428 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:20.428 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:20.523 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:29:20.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:20.529 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:29:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:20 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1276578900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:29:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:20.566 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:20.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:20 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1276578900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:29:20.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:20.593 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:20.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:20.598 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:20.692 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 16 16 53661 53645 209.522 138.812 0.00297375 0.00477053 2024-05-15T23:29:20.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:29:20.738 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:20.766 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:29:20.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:20.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:29:20.873 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:29:20.902 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:20.925 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:29:20.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:20.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:29:21.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:29:21.069 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:21.093 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:29:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:21.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:29:21.199 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:29:21.228 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:21.251 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:29:21.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:21.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:29:21.357 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:29:21.388 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:21.414 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:29:21.416 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.416 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.416 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:21 smithi088 ceph-mon[44699]: from='client.36635 v1:172.21.15.84:0/4118256320' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:29:21.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:21 smithi088 ceph-mon[44699]: pgmap v1502: 17 pgs: 17 active+clean; 2.9 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 172 MiB/s wr, 2.75k op/s 2024-05-15T23:29:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:21.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:29:21.519 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:29:21.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:21 smithi090 ceph-mon[44434]: from='client.36635 v1:172.21.15.84:0/4118256320' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:29:21.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:21 smithi090 ceph-mon[44434]: pgmap v1502: 17 pgs: 17 active+clean; 2.9 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 172 MiB/s wr, 2.75k op/s 2024-05-15T23:29:21.565 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:21.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:21.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:21 smithi084 ceph-mon[39784]: from='client.36635 v1:172.21.15.84:0/4118256320' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:29:21.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:21 smithi084 ceph-mon[39784]: pgmap v1502: 17 pgs: 17 active+clean; 2.9 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 172 MiB/s wr, 2.75k op/s 2024-05-15T23:29:21.683 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:29:21.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:21.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:29:21.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 17 16 56084 56068 206.105 151.438 0.00430417 0.00485021 2024-05-15T23:29:21.787 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:29:21.842 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:21.906 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:29:21.907 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:29:21.907 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:29:21.907 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:29:21.907 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:29:21.915 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:29:21.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:21.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:29:22.020 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:29:22.052 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:22.079 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:29:22.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:22.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:29:22.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:22.221 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:22.251 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:22.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:22.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:22.356 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:29:22.400 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:22.695 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 18 15 58423 58408 202.779 146.25 0.0055011 0.00493033 2024-05-15T23:29:22.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T23:29:23.135 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1501,"stamp":"2024-05-15T23:29:19.088004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2726232608,"num_objects":41592,"num_object_clones":0,"num_object_copies":83184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41592,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":41713,"num_write_kb":2663910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41684,"ondisk_log_size":41684,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":9552504,"kb_used_data":5197400,"kb_used_omap":16,"kb_used_meta":4354927,"kb_avail":834153864,"statfs":{"total":863955320832,"available":854173556736,"internally_reserved":0,"allocated":5322137600,"data_stored":5318350795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17176,"internal_metadata":4459445480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":989003776,"num_objects":15091,"num_object_clones":0,"num_object_copies":30182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15091,"num_write_kb":965824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003162"},"pg_stats":[{"pgid":"7.f","version":"560'2501","reported_seq":2532,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-05-15T23:29:15.797108+0000","last_change":"2024-05-15T23:28:51.305473+0000","last_active":"2024-05-15T23:29:15.797108+0000","last_peered":"2024-05-15T23:29:15.797108+0000","last_clean":"2024-05-15T23:29:15.797108+0000","last_became_active":"2024-05-15T23:28:51.305173+0000","last_became_peered":"2024-05-15T23:28:51.305173+0000","last_unstale":"2024-05-15T23:29:15.797108+0000","last_undegraded":"2024-05-15T23:29:15.797108+0000","last_fullsized":"2024-05-15T23:29:15.797108+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2501,"log_dups_size":0,"ondisk_log_size":2501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:30:18.729877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163905536,"num_objects":2501,"num_object_clones":0,"num_object_copies":5002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2501,"num_write_kb":160064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"559'2296","reported_seq":2325,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-05-15T23:29:14.110451+0000","last_change":"2024-05-15T23:28:51.519687+0000","last_active":"2024-05-15T23:29:14.110451+0000","last_peered":"2024-05-15T23:29:14.110451+0000","last_clean":"2024-05-15T23:29:14.110451+0000","last_became_active":"2024-05-15T23:28:51.519387+0000","last_became_peered":"2024-05-15T23:28:51.519387+0000","last_unstale":"2024-05-15T23:29:14.110451+0000","last_undegraded":"2024-05-15T23:29:14.110451+0000","last_fullsized":"2024-05-15T23:29:14.110451+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2296,"log_dups_size":0,"ondisk_log_size":2296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:30:00.551145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150470656,"num_objects":2296,"num_object_clones":0,"num_object_copies":4592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2296,"num_write_kb":146944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"560'2506","reported_seq":2537,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-05-15T23:29:15.998904+0000","last_change":"2024-05-15T23:28:51.306905+0000","last_active":"2024-05-15T23:29:15.998904+0000","last_peered":"2024-05-15T23:29:15.998904+0000","last_clean":"2024-05-15T23:29:15.998904+0000","last_became_active":"2024-05-15T23:28:51.306664+0000","last_became_peered":"2024-05-15T23:28:51.306664+0000","last_unstale":"2024-05-15T23:29:15.998904+0000","last_undegraded":"2024-05-15T23:29:15.998904+0000","last_fullsized":"2024-05-15T23:29:15.998904+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2506,"log_dups_size":0,"ondisk_log_size":2506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:30:17.340746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164233216,"num_objects":2506,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2506,"num_write_kb":160384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"560'2526","reported_seq":2557,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-05-15T23:29:16.201735+0000","last_change":"2024-05-15T23:28:51.305984+0000","last_active":"2024-05-15T23:29:16.201735+0000","last_peered":"2024-05-15T23:29:16.201735+0000","last_clean":"2024-05-15T23:29:16.201735+0000","last_became_active":"2024-05-15T23:28:51.305728+0000","last_became_peered":"2024-05-15T23:28:51.305728+0000","last_unstale":"2024-05-15T23:29:16.201735+0000","last_undegraded":"2024-05-15T23:29:16.201735+0000","last_fullsized":"2024-05-15T23:29:16.201735+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2526,"log_dups_size":0,"ondisk_log_size":2526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:29:52.162780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165543936,"num_objects":2526,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2526,"num_write_kb":161664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"560'2560","reported_seq":2591,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-05-15T23:29:16.012568+0000","last_change":"2024-05-15T23:28:51.307005+0000","last_active":"2024-05-15T23:29:16.012568+0000","last_peered":"2024-05-15T23:29:16.012568+0000","last_clean":"2024-05-15T23:29:16.012568+0000","last_became_active":"2024-05-15T23:28:51.306794+0000","last_became_peered":"2024-05-15T23:28:51.306794+0000","last_unstale":"2024-05-15T23:29:16.012568+0000","last_undegraded":"2024-05-15T23:29:16.012568+0000","last_fullsized":"2024-05-15T23:29:16.012568+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2560,"log_dups_size":0,"ondisk_log_size":2560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:30:06.356201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":2560,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2560,"num_write_kb":163840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"560'2574","reported_seq":2605,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-05-15T23:29:15.785034+0000","last_change":"2024-05-15T23:28:51.308505+0000","last_active":"2024-05-15T23:29:15.785034+0000","last_peered":"2024-05-15T23:29:15.785034+0000","last_clean":"2024-05-15T23:29:15.785034+0000","last_became_active":"2024-05-15T23:28:51.308216+0000","last_became_peered":"2024-05-15T23:28:51.308216+0000","last_unstale":"2024-05-15T23:29:15.785034+0000","last_undegraded":"2024-05-15T23:29:15.785034+0000","last_fullsized":"2024-05-15T23:29:15.785034+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2574,"log_dups_size":0,"ondisk_log_size":2574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:30:14.411782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168689664,"num_objects":2574,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":164736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"561'2988","reported_seq":3020,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-05-15T23:29:17.308075+0000","last_change":"2024-05-15T23:28:51.520069+0000","last_active":"2024-05-15T23:29:17.308075+0000","last_peered":"2024-05-15T23:29:17.308075+0000","last_clean":"2024-05-15T23:29:17.308075+0000","last_became_active":"2024-05-15T23:28:51.519752+0000","last_became_peered":"2024-05-15T23:28:51.519752+0000","last_unstale":"2024-05-15T23:29:17.308075+0000","last_undegraded":"2024-05-15T23:29:17.308075+0000","last_fullsized":"2024-05-15T23:29:17.308075+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2988,"log_dups_size":0,"ondisk_log_size":2988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:29:56.342628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195821568,"num_objects":2988,"num_object_clones":0,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2988,"num_write_kb":191232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"559'2347","reported_seq":2376,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-05-15T23:29:14.113307+0000","last_change":"2024-05-15T23:28:51.305199+0000","last_active":"2024-05-15T23:29:14.113307+0000","last_peered":"2024-05-15T23:29:14.113307+0000","last_clean":"2024-05-15T23:29:14.113307+0000","last_became_active":"2024-05-15T23:28:51.304930+0000","last_became_peered":"2024-05-15T23:28:51.304930+0000","last_unstale":"2024-05-15T23:29:14.113307+0000","last_undegraded":"2024-05-15T23:29:14.113307+0000","last_fullsized":"2024-05-15T23:29:14.113307+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2347,"log_dups_size":0,"ondisk_log_size":2347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:30:10.701744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153812992,"num_objects":2347,"num_object_clones":0,"num_object_copies":4694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2347,"num_write_kb":150208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"560'2627","reported_seq":2657,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-05-15T23:29:16.228090+0000","last_change":"2024-05-15T23:28:51.307151+0000","last_active":"2024-05-15T23:29:16.228090+0000","last_peered":"2024-05-15T23:29:16.228090+0000","last_clean":"2024-05-15T23:29:16.228090+0000","last_became_active":"2024-05-15T23:28:51.306983+0000","last_became_peered":"2024-05-15T23:28:51.306983+0000","last_unstale":"2024-05-15T23:29:16.228090+0000","last_undegraded":"2024-05-15T23:29:16.228090+0000","last_fullsized":"2024-05-15T23:29:16.228090+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2627,"log_dups_size":0,"ondisk_log_size":2627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:30:15.449725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172163072,"num_objects":2627,"num_object_clones":0,"num_object_copies":5254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2627,"num_write_kb":168128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.5","version":"561'2859","reported_seq":2892,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-05-15T23:29:17.299364+0000","last_change":"2024-05-15T23:28:51.303904+0000","last_active":"2024-05-15T23:29:17.299364+0000","last_peered":"2024-05-15T23:29:17.299364+0000","last_clean":"2024-05-15T23:29:17.299364+0000","last_became_active":"2024-05-15T23:28:51.303581+0000","last_became_peered":"2024-05-15T23:28:51.303581+0000","last_unstale":"2024-05-15T23:29:17.299364+0000","last_undegraded":"2024-05-15T23:29:17.299364+0000","last_fullsized":"2024-05-15T23:29:17.299364+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2859,"log_dups_size":0,"ondisk_log_size":2859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:29:51.230804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187367424,"num_objects":2859,"num_object_clones":0,"num_object_copies":5718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":182976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"559'2513","reported_seq":2542,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-05-15T23:29:15.119057+0000","last_change":"2024-05-15T23:28:51.304646+0000","last_active":"2024-05-15T23:29:15.119057+0000","last_peered":"2024-05-15T23:29:15.119057+0000","last_clean":"2024-05-15T23:29:15.119057+0000","last_became_active":"2024-05-15T23:28:51.304327+0000","last_became_peered":"2024-05-15T23:28:51.304327+0000","last_unstale":"2024-05-15T23:29:15.119057+0000","last_undegraded":"2024-05-15T23:29:15.119057+0000","last_fullsized":"2024-05-15T23:29:15.119057+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2513,"log_dups_size":0,"ondisk_log_size":2513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:30:01.744441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164691968,"num_objects":2513,"num_object_clones":0,"num_object_copies":5026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2513,"num_write_kb":160832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"560'2696","reported_seq":2726,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-05-15T23:29:16.218609+0000","last_change":"2024-05-15T23:28:51.305982+0000","last_active":"2024-05-15T23:29:16.218609+0000","last_peered":"2024-05-15T23:29:16.218609+0000","last_clean":"2024-05-15T23:29:16.218609+0000","last_became_active":"2024-05-15T23:28:51.305691+0000","last_became_peered":"2024-05-15T23:28:51.305691+0000","last_unstale":"2024-05-15T23:29:16.218609+0000","last_undegraded":"2024-05-15T23:29:16.218609+0000","last_fullsized":"2024-05-15T23:29:16.218609+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2696,"log_dups_size":0,"ondisk_log_size":2696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:29:52.896594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176685056,"num_objects":2696,"num_object_clones":0,"num_object_copies":5392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2696,"num_write_kb":172544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"560'2716","reported_seq":2747,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-05-15T23:29:16.226510+0000","last_change":"2024-05-15T23:28:51.519631+0000","last_active":"2024-05-15T23:29:16.226510+0000","last_peered":"2024-05-15T23:29:16.226510+0000","last_clean":"2024-05-15T23:29:16.226510+0000","last_became_active":"2024-05-15T23:28:51.519343+0000","last_became_peered":"2024-05-15T23:28:51.519343+0000","last_unstale":"2024-05-15T23:29:16.226510+0000","last_undegraded":"2024-05-15T23:29:16.226510+0000","last_fullsized":"2024-05-15T23:29:16.226510+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2716,"log_dups_size":0,"ondisk_log_size":2716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:29:53.052140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177995776,"num_objects":2716,"num_object_clones":0,"num_object_copies":5432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2716,"num_write_kb":173824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"559'2368","reported_seq":2397,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-05-15T23:29:15.115413+0000","last_change":"2024-05-15T23:28:51.519869+0000","last_active":"2024-05-15T23:29:15.115413+0000","last_peered":"2024-05-15T23:29:15.115413+0000","last_clean":"2024-05-15T23:29:15.115413+0000","last_became_active":"2024-05-15T23:28:51.519585+0000","last_became_peered":"2024-05-15T23:28:51.519585+0000","last_unstale":"2024-05-15T23:29:15.115413+0000","last_undegraded":"2024-05-15T23:29:15.115413+0000","last_fullsized":"2024-05-15T23:29:15.115413+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2368,"log_dups_size":0,"ondisk_log_size":2368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:30:02.123476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":2368,"num_object_clones":0,"num_object_copies":4736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2368,"num_write_kb":151552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"561'2910","reported_seq":2943,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-05-15T23:29:18.001266+0000","last_change":"2024-05-15T23:28:51.303933+0000","last_active":"2024-05-15T23:29:18.001266+0000","last_peered":"2024-05-15T23:29:18.001266+0000","last_clean":"2024-05-15T23:29:18.001266+0000","last_became_active":"2024-05-15T23:28:51.303621+0000","last_became_peered":"2024-05-15T23:28:51.303621+0000","last_unstale":"2024-05-15T23:29:18.001266+0000","last_undegraded":"2024-05-15T23:29:18.001266+0000","last_fullsized":"2024-05-15T23:29:18.001266+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2910,"log_dups_size":0,"ondisk_log_size":2910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:30:11.619906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190709760,"num_objects":2910,"num_object_clones":0,"num_object_copies":5820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2910,"num_write_kb":186240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"560'2603","reported_seq":2634,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-05-15T23:29:16.327063+0000","last_change":"2024-05-15T23:28:51.304581+0000","last_active":"2024-05-15T23:29:16.327063+0000","last_peered":"2024-05-15T23:29:16.327063+0000","last_clean":"2024-05-15T23:29:16.327063+0000","last_became_active":"2024-05-15T23:28:51.304317+0000","last_became_peered":"2024-05-15T23:28:51.304317+0000","last_unstale":"2024-05-15T23:29:16.327063+0000","last_undegraded":"2024-05-15T23:29:16.327063+0000","last_fullsized":"2024-05-15T23:29:16.327063+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":2603,"log_dups_size":0,"ondisk_log_size":2603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:30:06.465518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170590208,"num_objects":2603,"num_object_clones":0,"num_object_copies":5206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2603,"num_write_kb":166592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1193,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-05-15T23:29:15.190523+0000","last_change":"2024-05-15T23:29:05.074454+0000","last_active":"2024-05-15T23:29:15.190523+0000","last_peered":"2024-05-15T23:29:15.190523+0000","last_clean":"2024-05-15T23:29:15.190523+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:29:15.190523+0000","last_undegraded":"2024-05-15T23:29:15.190523+0000","last_fullsized":"2024-05-15T23:29:15.190523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:29:05.074359+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:29:05.074359+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:30:28.599422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2725642240,"num_objects":41590,"num_object_clones":0,"num_object_copies":83180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41590,"num_write_kb":2661760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5461704704,"data_stored":5461704704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41590,"ondisk_log_size":41590,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351282,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1310856,"kb_used_data":684092,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92434296,"statfs":{"total":95995035648,"available":94652719104,"internally_reserved":0,"allocated":700510208,"data_stored":700086002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":641792452},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580215,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019580,"kb_used_data":655220,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92725572,"statfs":{"total":95995035648,"available":94950985728,"internally_reserved":0,"allocated":670945280,"data_stored":670524646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":373094848},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1180864,"kb_used_data":640180,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92564288,"statfs":{"total":95995035648,"available":94785830912,"internally_reserved":0,"allocated":655544320,"data_stored":655123686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":553646528},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":2233382993951,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267015,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":792712,"kb_used_data":332540,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92952440,"statfs":{"total":95995035648,"available":95183298560,"internally_reserved":0,"allocated":340520960,"data_stored":340096774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":471202250},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495950,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575208,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1247932,"kb_used_data":657268,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92497220,"statfs":{"total":95995035648,"available":94717153280,"internally_reserved":0,"allocated":673042432,"data_stored":672621798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":604830150},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016617,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483305,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":630708,"kb_used_data":562216,"kb_used_omap":4,"kb_used_meta":68411,"kb_avail":93114444,"statfs":{"total":95995035648,"available":95349190656,"internally_reserved":0,"allocated":575709184,"data_stored":575294575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":70053571},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150047,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1615104,"kb_used_data":1002228,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92130048,"statfs":{"total":95995035648,"available":94341169152,"internally_reserved":0,"allocated":1026281472,"data_stored":1025860838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":627571143},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":815024,"kb_used_data":296552,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92930128,"statfs":{"total":95995035648,"available":95160451072,"internally_reserved":0,"allocated":303669248,"data_stored":303254639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790291,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":939724,"kb_used_data":367104,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92805428,"statfs":{"total":95995035648,"available":95032758272,"internally_reserved":0,"allocated":375914496,"data_stored":375487837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":676462592,"data_stored":676462592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":610992128,"data_stored":610992128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1028849664,"data_stored":1028849664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":319946752,"data_stored":319946752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":381091840,"data_stored":381091840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":349700096,"data_stored":349700096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":670236672,"data_stored":670236672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":680919040,"data_stored":680919040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":743505920,"data_stored":743505920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:29:23.136 INFO:tasks.thrashosds.thrasher:Setting 7.4 to [7, 0, 4, 9] 2024-05-15T23:29:23.136 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '7.4', '7', '0', '4', '9'] 2024-05-15T23:29:23.136 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pg-upmap-items 7.4 7 0 4 9 2024-05-15T23:29:23.264 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:29:23.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:23.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:29:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:23 smithi084 ceph-mon[39784]: osdmap e562: 12 total, 12 up, 9 in 2024-05-15T23:29:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:23 smithi084 ceph-mon[39784]: pgmap v1504: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-05-15T23:29:23.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:23.410 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:23.438 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:23.443 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:23 smithi090 ceph-mon[44434]: osdmap e562: 12 total, 12 up, 9 in 2024-05-15T23:29:23.444 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:23 smithi090 ceph-mon[44434]: pgmap v1504: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-05-15T23:29:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:23.444 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:23.545 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:29:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:23 smithi088 ceph-mon[44699]: osdmap e562: 12 total, 12 up, 9 in 2024-05-15T23:29:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:23 smithi088 ceph-mon[44699]: pgmap v1504: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-05-15T23:29:23.601 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:23.603 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:29:23.645 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:29:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:23.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:29:23.692 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 19 16 60524 60508 199.014 131.25 0.00401039 0.00502295 2024-05-15T23:29:23.761 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:29:23.821 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:23.864 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:29:23.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:23.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:29:23.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:29:24.004 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:24.034 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:29:24.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:24.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:29:24.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:24.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:24.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:24.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:29:24.366 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:24.499 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:29:24.502 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:29:24.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:24.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:29:24.504 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:29:24.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:29:24.658 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:24.692 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:29:24.692117 min lat: 0.000985917 max lat: 0.20623 avg lat: 0.00510615 2024-05-15T23:29:24.692 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:29:24.692 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 20 16 62675 62659 195.785 134.438 0.00990481 0.00510615 2024-05-15T23:29:24.774 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:29:24.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:24.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:29:24.881 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:29:24.909 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:24.935 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:29:24.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:24.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:29:25.040 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:29:25.082 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:25.291 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:25 smithi084 ceph-mon[39784]: pgmap v1505: 17 pgs: 17 active+clean; 3.4 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 132 MiB/s wr, 2.10k op/s 2024-05-15T23:29:25.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:25 smithi090 ceph-mon[44434]: pgmap v1505: 17 pgs: 17 active+clean; 3.4 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 132 MiB/s wr, 2.10k op/s 2024-05-15T23:29:25.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:25 smithi088 ceph-mon[44699]: pgmap v1505: 17 pgs: 17 active+clean; 3.4 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 132 MiB/s wr, 2.10k op/s 2024-05-15T23:29:25.659 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:29:25.692 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 21 16 64862 64846 192.97 136.688 0.00419781 0.00518052 2024-05-15T23:29:26.692 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 22 15 67029 67014 190.357 135.5 0.00562006 0.00525196 2024-05-15T23:29:27.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:26 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:29:27.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:26 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:29:27.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:26 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:29:27.692 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 23 16 68885 68869 187.121 115.938 0.00566533 0.00534286 2024-05-15T23:29:27.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:27 smithi084 ceph-mon[39784]: pgmap v1506: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-05-15T23:29:27.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:29:27.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:29:27.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:27 smithi084 ceph-mon[39784]: osdmap e563: 12 total, 12 up, 9 in 2024-05-15T23:29:27.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:29:27.839 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:29:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:27 smithi090 ceph-mon[44434]: pgmap v1506: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-05-15T23:29:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:29:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:29:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:27 smithi090 ceph-mon[44434]: osdmap e563: 12 total, 12 up, 9 in 2024-05-15T23:29:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:29:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:29:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:27 smithi088 ceph-mon[44699]: pgmap v1506: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-05-15T23:29:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:29:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:29:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:27 smithi088 ceph-mon[44699]: osdmap e563: 12 total, 12 up, 9 in 2024-05-15T23:29:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:29:28.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:29:28.692 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 24 16 70838 70822 184.41 122.062 0.00457171 0.00542115 2024-05-15T23:29:28.782 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:29:28.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:28.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:29:28.792 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:29:28.891 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:29:28.937 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:28 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:29:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:28 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:29:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:29:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:29:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:29:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/81070494' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:29:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:28 smithi090 ceph-mon[44434]: from='client.36638 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:29:29.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:28 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:29:29.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:28 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:29:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:29:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:29:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:29:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/81070494' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:29:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:28 smithi088 ceph-mon[44699]: from='client.36638 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:29:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:28 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:29:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:28 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:29:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:29:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:29:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:29:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/81070494' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:29:29.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:28 smithi084 ceph-mon[39784]: from='client.36638 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:29:29.296 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:29:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:29.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:29:29.403 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:29:29.434 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:29.455 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:29:29.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:29.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:29:29.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:29:29.606 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:29.692 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 25 16 72782 72766 181.894 121.5 0.00352782 0.0054964 2024-05-15T23:29:29.735 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:29:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:29.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:29:29.748 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:29 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:29:29.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:29:29.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:29 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:29:29.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:29 smithi084 ceph-mon[39784]: from='client.36638 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:29:29.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:29 smithi084 ceph-mon[39784]: osdmap e564: 12 total, 12 up, 9 in 2024-05-15T23:29:29.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:29 smithi084 ceph-mon[39784]: pgmap v1509: 17 pgs: 17 active+clean; 4.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-05-15T23:29:29.941 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:29.969 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:29:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:29.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:29:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:29 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:29:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:29 smithi090 ceph-mon[44434]: from='client.36638 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:29:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:29 smithi090 ceph-mon[44434]: osdmap e564: 12 total, 12 up, 9 in 2024-05-15T23:29:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:29 smithi090 ceph-mon[44434]: pgmap v1509: 17 pgs: 17 active+clean; 4.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-05-15T23:29:30.060 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:29:30.061 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-05-15T23:29:30.061 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-05-15T23:29:30.061 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:29:30.061 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.061 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:64879 6.6 6:63059de2:::benchmark_data_smithi090_131278_object64878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e498)", 2024-05-15T23:29:30.061 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:25:51.973806+0000", 2024-05-15T23:29:30.061 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 218.080500349, 2024-05-15T23:29:30.061 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089020967000000006, 2024-05-15T23:29:30.061 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.061 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.061 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.061 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.062 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.062 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.062 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 64879 2024-05-15T23:29:30.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.062 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.062 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.062 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.062 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.973806+0000", 2024-05-15T23:29:30.062 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.062 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.062 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.062 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.973806+0000", 2024-05-15T23:29:30.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.063 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.063 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.063 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.063 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.973807+0000", 2024-05-15T23:29:30.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.89e-07 2024-05-15T23:29:30.063 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.063 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.063 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.063 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.973911+0000", 2024-05-15T23:29:30.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0001047 2024-05-15T23:29:30.063 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.063 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.063 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.973916+0000", 2024-05-15T23:29:30.064 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3640000000000001e-06 2024-05-15T23:29:30.064 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.064 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.973919+0000", 2024-05-15T23:29:30.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2830000000000002e-06 2024-05-15T23:29:30.065 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.065 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.065 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.065 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.975926+0000", 2024-05-15T23:29:30.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020069879999999999 2024-05-15T23:29:30.065 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.065 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.066 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.066 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.976049+0000", 2024-05-15T23:29:30.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012327200000000001 2024-05-15T23:29:30.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.066 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.066 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:29:30.066 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.976151+0000", 2024-05-15T23:29:30.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101857 2024-05-15T23:29:30.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.066 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.066 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.067 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:51.977864+0000", 2024-05-15T23:29:30.067 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017128130000000001 2024-05-15T23:29:30.067 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.067 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.067 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.067 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062567+0000", 2024-05-15T23:29:30.067 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.084703422 2024-05-15T23:29:30.067 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.067 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.067 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.067 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062727+0000", 2024-05-15T23:29:30.068 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000159891 2024-05-15T23:29:30.068 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.068 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.068 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.068 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:25:52.062827+0000", 2024-05-15T23:29:30.068 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9888000000000003e-05 2024-05-15T23:29:30.068 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.068 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.068 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.068 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.068 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.069 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:129217 6.6 6:62de5787:::benchmark_data_smithi090_131278_object129216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e514)", 2024-05-15T23:29:30.069 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:26:32.666117+0000", 2024-05-15T23:29:30.069 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 177.38818922799999, 2024-05-15T23:29:30.069 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.098574879000000004, 2024-05-15T23:29:30.069 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.069 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.069 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.069 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.069 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.069 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.069 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 129217 2024-05-15T23:29:30.069 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.070 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.666117+0000", 2024-05-15T23:29:30.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.070 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.666117+0000", 2024-05-15T23:29:30.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.070 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.071 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.071 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.071 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.666118+0000", 2024-05-15T23:29:30.071 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8599999999999998e-07 2024-05-15T23:29:30.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.071 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.071 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.071 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.666165+0000", 2024-05-15T23:29:30.071 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6594999999999997e-05 2024-05-15T23:29:30.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.071 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.071 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.072 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.666170+0000", 2024-05-15T23:29:30.072 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2179999999999998e-06 2024-05-15T23:29:30.072 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.072 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.072 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.666173+0000", 2024-05-15T23:29:30.072 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3459999999999998e-06 2024-05-15T23:29:30.072 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.072 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.072 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.666213+0000", 2024-05-15T23:29:30.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9876999999999998e-05 2024-05-15T23:29:30.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.073 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.666303+0000", 2024-05-15T23:29:30.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0321000000000005e-05 2024-05-15T23:29:30.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.073 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:29:30.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.666422+0000", 2024-05-15T23:29:30.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000118363 2024-05-15T23:29:30.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.673985+0000", 2024-05-15T23:29:30.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0075634370000000001 2024-05-15T23:29:30.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.764596+0000", 2024-05-15T23:29:30.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090610649000000001 2024-05-15T23:29:30.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.075 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.764651+0000", 2024-05-15T23:29:30.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4758e-05 2024-05-15T23:29:30.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.075 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.075 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.075 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.764692+0000", 2024-05-15T23:29:30.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1628999999999997e-05 2024-05-15T23:29:30.075 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.075 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.077 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.077 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.077 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.077 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:129218 6.6 6:6d1de178:::benchmark_data_smithi090_131278_object129217:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e514)", 2024-05-15T23:29:30.077 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:26:32.666340+0000", 2024-05-15T23:29:30.077 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 177.387966963, 2024-05-15T23:29:30.078 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.098508457999999993, 2024-05-15T23:29:30.078 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.078 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.078 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.078 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.078 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.078 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.078 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 129218 2024-05-15T23:29:30.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.078 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.078 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.666340+0000", 2024-05-15T23:29:30.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.666340+0000", 2024-05-15T23:29:30.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.666340+0000", 2024-05-15T23:29:30.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5599999999999995e-07 2024-05-15T23:29:30.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.666406+0000", 2024-05-15T23:29:30.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5567000000000004e-05 2024-05-15T23:29:30.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.666411+0000", 2024-05-15T23:29:30.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8779999999999997e-06 2024-05-15T23:29:30.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.666414+0000", 2024-05-15T23:29:30.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7629999999999998e-06 2024-05-15T23:29:30.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.081 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.081 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.081 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.670040+0000", 2024-05-15T23:29:30.081 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003625666 2024-05-15T23:29:30.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.081 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.081 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.081 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.670183+0000", 2024-05-15T23:29:30.081 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014340800000000001 2024-05-15T23:29:30.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.081 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.081 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:29:30.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.670334+0000", 2024-05-15T23:29:30.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015024999999999999 2024-05-15T23:29:30.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.082 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.674081+0000", 2024-05-15T23:29:30.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037469140000000001 2024-05-15T23:29:30.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.082 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.764752+0000", 2024-05-15T23:29:30.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090671703000000006 2024-05-15T23:29:30.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.764807+0000", 2024-05-15T23:29:30.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4323000000000001e-05 2024-05-15T23:29:30.083 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.764848+0000", 2024-05-15T23:29:30.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1430000000000001e-05 2024-05-15T23:29:30.083 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.083 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.083 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.083 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.084 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:129230 6.6 6:68ae7d65:::benchmark_data_smithi090_131278_object129229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e514)", 2024-05-15T23:29:30.084 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:26:32.675773+0000", 2024-05-15T23:29:30.084 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 177.37853316799999, 2024-05-15T23:29:30.084 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.089216541999999996, 2024-05-15T23:29:30.084 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.084 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.084 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.084 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.084 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.084 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.084 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 129230 2024-05-15T23:29:30.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.084 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.675773+0000", 2024-05-15T23:29:30.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.675773+0000", 2024-05-15T23:29:30.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.675775+0000", 2024-05-15T23:29:30.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.714e-06 2024-05-15T23:29:30.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.086 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.086 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.086 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.675824+0000", 2024-05-15T23:29:30.086 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9187999999999997e-05 2024-05-15T23:29:30.086 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.086 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.086 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.086 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.675833+0000", 2024-05-15T23:29:30.086 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3450000000000006e-06 2024-05-15T23:29:30.086 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.087 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.087 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.087 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.675838+0000", 2024-05-15T23:29:30.087 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6439999999999997e-06 2024-05-15T23:29:30.087 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.087 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.087 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.087 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.675887+0000", 2024-05-15T23:29:30.087 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8717999999999997e-05 2024-05-15T23:29:30.087 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.087 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.087 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.087 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.676057+0000", 2024-05-15T23:29:30.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017034 2024-05-15T23:29:30.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.088 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.088 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:29:30.088 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.676205+0000", 2024-05-15T23:29:30.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014731400000000001 2024-05-15T23:29:30.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.088 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.088 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.088 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.692067+0000", 2024-05-15T23:29:30.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015862475000000001 2024-05-15T23:29:30.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.088 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.088 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.088 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.764906+0000", 2024-05-15T23:29:30.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.072838753000000006 2024-05-15T23:29:30.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.089 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.764955+0000", 2024-05-15T23:29:30.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9262999999999999e-05 2024-05-15T23:29:30.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.089 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.764990+0000", 2024-05-15T23:29:30.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4788e-05 2024-05-15T23:29:30.089 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.089 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.089 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.090 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:129233 6.6 6:6c1e7d01:::benchmark_data_smithi090_131278_object129232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e514)", 2024-05-15T23:29:30.090 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:26:32.676632+0000", 2024-05-15T23:29:30.090 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 177.37767421000001, 2024-05-15T23:29:30.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.088491466000000005, 2024-05-15T23:29:30.090 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.090 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.090 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.090 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.090 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.090 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.090 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 129233 2024-05-15T23:29:30.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.091 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.676632+0000", 2024-05-15T23:29:30.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.676632+0000", 2024-05-15T23:29:30.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.676633+0000", 2024-05-15T23:29:30.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1179999999999999e-06 2024-05-15T23:29:30.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.092 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.676687+0000", 2024-05-15T23:29:30.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3300000000000001e-05 2024-05-15T23:29:30.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.092 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.676697+0000", 2024-05-15T23:29:30.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.91e-06 2024-05-15T23:29:30.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.092 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.676703+0000", 2024-05-15T23:29:30.093 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4389999999999997e-06 2024-05-15T23:29:30.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.093 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.093 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.679808+0000", 2024-05-15T23:29:30.093 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031051899999999999 2024-05-15T23:29:30.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.093 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.093 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.680008+0000", 2024-05-15T23:29:30.093 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000199416 2024-05-15T23:29:30.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.094 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:29:30.094 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.680160+0000", 2024-05-15T23:29:30.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015185599999999999 2024-05-15T23:29:30.094 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.094 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.094 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.094 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.692078+0000", 2024-05-15T23:29:30.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011917966 2024-05-15T23:29:30.094 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.094 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.094 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.094 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.765028+0000", 2024-05-15T23:29:30.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.072950608 2024-05-15T23:29:30.094 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.095 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.095 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.765085+0000", 2024-05-15T23:29:30.095 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7330999999999997e-05 2024-05-15T23:29:30.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.095 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.095 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.765124+0000", 2024-05-15T23:29:30.095 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8331999999999998e-05 2024-05-15T23:29:30.095 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.095 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.095 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.096 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:129234 6.6 6:6e541807:::benchmark_data_smithi090_131278_object129233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e514)", 2024-05-15T23:29:30.096 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:26:32.676750+0000", 2024-05-15T23:29:30.096 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 177.37755653299999, 2024-05-15T23:29:30.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.095859584999999997, 2024-05-15T23:29:30.096 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.096 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.096 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.096 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.096 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.096 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.096 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 129234 2024-05-15T23:29:30.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.096 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.676750+0000", 2024-05-15T23:29:30.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.676750+0000", 2024-05-15T23:29:30.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.676751+0000", 2024-05-15T23:29:30.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.23e-07 2024-05-15T23:29:30.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.676832+0000", 2024-05-15T23:29:30.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1353999999999994e-05 2024-05-15T23:29:30.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.676840+0000", 2024-05-15T23:29:30.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1550000000000007e-06 2024-05-15T23:29:30.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.676847+0000", 2024-05-15T23:29:30.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3450000000000002e-06 2024-05-15T23:29:30.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.099 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.683553+0000", 2024-05-15T23:29:30.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067059679999999997 2024-05-15T23:29:30.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.099 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.683642+0000", 2024-05-15T23:29:30.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8974999999999994e-05 2024-05-15T23:29:30.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.099 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:29:30.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.683787+0000", 2024-05-15T23:29:30.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144986 2024-05-15T23:29:30.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.692082+0000", 2024-05-15T23:29:30.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.008295574 2024-05-15T23:29:30.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.772517+0000", 2024-05-15T23:29:30.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.080435039999999999 2024-05-15T23:29:30.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.772567+0000", 2024-05-15T23:29:30.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9410999999999998e-05 2024-05-15T23:29:30.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.772610+0000", 2024-05-15T23:29:30.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2954000000000002e-05 2024-05-15T23:29:30.101 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.101 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.101 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.101 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:129239 6.6 6:61a11443:::benchmark_data_smithi090_131278_object129238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e514)", 2024-05-15T23:29:30.101 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:26:32.680283+0000", 2024-05-15T23:29:30.101 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 177.37402403199999, 2024-05-15T23:29:30.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.095697988999999997, 2024-05-15T23:29:30.102 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.102 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.102 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.102 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.102 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.102 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.102 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 129239 2024-05-15T23:29:30.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.102 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.102 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.102 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.680283+0000", 2024-05-15T23:29:30.102 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.680283+0000", 2024-05-15T23:29:30.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.680283+0000", 2024-05-15T23:29:30.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6599999999999995e-07 2024-05-15T23:29:30.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.680334+0000", 2024-05-15T23:29:30.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0711999999999998e-05 2024-05-15T23:29:30.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.680339+0000", 2024-05-15T23:29:30.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.322e-06 2024-05-15T23:29:30.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.680343+0000", 2024-05-15T23:29:30.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6909999999999999e-06 2024-05-15T23:29:30.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.688140+0000", 2024-05-15T23:29:30.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0077968960000000002 2024-05-15T23:29:30.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.688278+0000", 2024-05-15T23:29:30.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000138102 2024-05-15T23:29:30.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4", 2024-05-15T23:29:30.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.688426+0000", 2024-05-15T23:29:30.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014836100000000001 2024-05-15T23:29:30.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.692269+0000", 2024-05-15T23:29:30.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0038422360000000002 2024-05-15T23:29:30.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.775908+0000", 2024-05-15T23:29:30.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.083639721 2024-05-15T23:29:30.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.775954+0000", 2024-05-15T23:29:30.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6128999999999998e-05 2024-05-15T23:29:30.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:32.775981+0000", 2024-05-15T23:29:30.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6052999999999999e-05 2024-05-15T23:29:30.107 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.107 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.107 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.107 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:141917 6.c 6:3f51eedb:::benchmark_data_smithi090_131278_object6723:head [delete] snapc 0=[] ondisk+write+known_if_redirected e516)", 2024-05-15T23:29:30.107 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:26:43.216073+0000", 2024-05-15T23:29:30.107 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 166.83823349400001, 2024-05-15T23:29:30.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23638284000000001, 2024-05-15T23:29:30.107 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.107 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.107 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.108 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.108 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.108 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.108 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 141917 2024-05-15T23:29:30.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.108 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.108 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.108 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.216073+0000", 2024-05-15T23:29:30.108 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.108 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.108 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.216073+0000", 2024-05-15T23:29:30.108 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.216075+0000", 2024-05-15T23:29:30.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.578e-06 2024-05-15T23:29:30.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.216082+0000", 2024-05-15T23:29:30.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6420000000000007e-06 2024-05-15T23:29:30.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.216092+0000", 2024-05-15T23:29:30.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8919999999999995e-06 2024-05-15T23:29:30.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.216098+0000", 2024-05-15T23:29:30.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5400000000000003e-06 2024-05-15T23:29:30.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.216150+0000", 2024-05-15T23:29:30.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1950999999999998e-05 2024-05-15T23:29:30.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.216284+0000", 2024-05-15T23:29:30.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013398500000000001 2024-05-15T23:29:30.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2024-05-15T23:29:30.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.216387+0000", 2024-05-15T23:29:30.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000103736 2024-05-15T23:29:30.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.222881+0000", 2024-05-15T23:29:30.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0064933769999999998 2024-05-15T23:29:30.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.452335+0000", 2024-05-15T23:29:30.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22945468399999999 2024-05-15T23:29:30.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.452399+0000", 2024-05-15T23:29:30.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3869999999999997e-05 2024-05-15T23:29:30.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.452456+0000", 2024-05-15T23:29:30.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6585000000000001e-05 2024-05-15T23:29:30.112 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.112 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.113 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.113 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:141919 6.c 6:3888dcf6:::benchmark_data_smithi090_131278_object6725:head [delete] snapc 0=[] ondisk+write+known_if_redirected e516)", 2024-05-15T23:29:30.113 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:26:43.216111+0000", 2024-05-15T23:29:30.113 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 166.83819596999999, 2024-05-15T23:29:30.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.24109823399999999, 2024-05-15T23:29:30.113 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.113 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.113 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.113 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.113 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.113 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.113 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 141919 2024-05-15T23:29:30.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.114 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.216111+0000", 2024-05-15T23:29:30.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.216111+0000", 2024-05-15T23:29:30.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.216111+0000", 2024-05-15T23:29:30.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0900000000000002e-07 2024-05-15T23:29:30.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.216113+0000", 2024-05-15T23:29:30.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.88e-06 2024-05-15T23:29:30.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.216117+0000", 2024-05-15T23:29:30.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3409999999999996e-06 2024-05-15T23:29:30.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.216122+0000", 2024-05-15T23:29:30.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5800000000000002e-06 2024-05-15T23:29:30.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.219595+0000", 2024-05-15T23:29:30.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034729090000000002 2024-05-15T23:29:30.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.219675+0000", 2024-05-15T23:29:30.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0327000000000005e-05 2024-05-15T23:29:30.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2024-05-15T23:29:30.117 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.219775+0000", 2024-05-15T23:29:30.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010011 2024-05-15T23:29:30.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.117 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.117 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.223093+0000", 2024-05-15T23:29:30.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033174160000000001 2024-05-15T23:29:30.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.117 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.117 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.457139+0000", 2024-05-15T23:29:30.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23404625300000001 2024-05-15T23:29:30.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.457175+0000", 2024-05-15T23:29:30.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.612e-05 2024-05-15T23:29:30.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.457209+0000", 2024-05-15T23:29:30.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3788999999999998e-05 2024-05-15T23:29:30.118 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.118 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.118 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.118 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:141932 6.0 6:0197855f:::benchmark_data_smithi090_131278_object6738:head [delete] snapc 0=[] ondisk+write+known_if_redirected e516)", 2024-05-15T23:29:30.119 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:26:43.225687+0000", 2024-05-15T23:29:30.119 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 166.828619242, 2024-05-15T23:29:30.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23566463500000001, 2024-05-15T23:29:30.119 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.119 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.119 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.119 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.119 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.119 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.119 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 141932 2024-05-15T23:29:30.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.119 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.119 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.225687+0000", 2024-05-15T23:29:30.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.225687+0000", 2024-05-15T23:29:30.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.225689+0000", 2024-05-15T23:29:30.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.209e-06 2024-05-15T23:29:30.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.225696+0000", 2024-05-15T23:29:30.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2609999999999996e-06 2024-05-15T23:29:30.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.225707+0000", 2024-05-15T23:29:30.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1358e-05 2024-05-15T23:29:30.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.225715+0000", 2024-05-15T23:29:30.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9729999999999997e-06 2024-05-15T23:29:30.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.225768+0000", 2024-05-15T23:29:30.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2877999999999998e-05 2024-05-15T23:29:30.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.225876+0000", 2024-05-15T23:29:30.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107495 2024-05-15T23:29:30.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2024-05-15T23:29:30.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.225985+0000", 2024-05-15T23:29:30.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000109169 2024-05-15T23:29:30.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.229740+0000", 2024-05-15T23:29:30.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037550169999999998 2024-05-15T23:29:30.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.461154+0000", 2024-05-15T23:29:30.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.23141403799999999 2024-05-15T23:29:30.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.461236+0000", 2024-05-15T23:29:30.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2015000000000007e-05 2024-05-15T23:29:30.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.124 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.461352+0000", 2024-05-15T23:29:30.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000116222 2024-05-15T23:29:30.124 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.124 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.124 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.124 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:141940 6.c 6:3bdf0518:::benchmark_data_smithi090_131278_object6746:head [delete] snapc 0=[] ondisk+write+known_if_redirected e516)", 2024-05-15T23:29:30.124 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:26:43.233215+0000", 2024-05-15T23:29:30.125 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 166.82109108200001, 2024-05-15T23:29:30.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.224711144, 2024-05-15T23:29:30.125 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.125 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.125 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.125 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.125 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.125 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.125 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 141940 2024-05-15T23:29:30.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.125 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.233215+0000", 2024-05-15T23:29:30.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.233215+0000", 2024-05-15T23:29:30.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.233216+0000", 2024-05-15T23:29:30.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9400000000000005e-07 2024-05-15T23:29:30.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.233218+0000", 2024-05-15T23:29:30.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.407e-06 2024-05-15T23:29:30.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.233224+0000", 2024-05-15T23:29:30.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9869999999999996e-06 2024-05-15T23:29:30.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.233233+0000", 2024-05-15T23:29:30.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0700000000000007e-06 2024-05-15T23:29:30.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.233266+0000", 2024-05-15T23:29:30.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3302000000000001e-05 2024-05-15T23:29:30.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.233344+0000", 2024-05-15T23:29:30.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8603000000000006e-05 2024-05-15T23:29:30.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2024-05-15T23:29:30.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.233425+0000", 2024-05-15T23:29:30.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0399000000000001e-05 2024-05-15T23:29:30.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.236709+0000", 2024-05-15T23:29:30.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0032841150000000002 2024-05-15T23:29:30.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.457835+0000", 2024-05-15T23:29:30.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22112565200000001 2024-05-15T23:29:30.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.457886+0000", 2024-05-15T23:29:30.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1477999999999997e-05 2024-05-15T23:29:30.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.130 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.130 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.457927+0000", 2024-05-15T23:29:30.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0537000000000002e-05 2024-05-15T23:29:30.130 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.130 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.130 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.130 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:141947 6.0 6:0855be48:::benchmark_data_smithi090_131278_object6753:head [delete] snapc 0=[] ondisk+write+known_if_redirected e516)", 2024-05-15T23:29:30.130 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:26:43.253152+0000", 2024-05-15T23:29:30.130 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 166.80115477300001, 2024-05-15T23:29:30.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.208336626, 2024-05-15T23:29:30.130 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.130 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.131 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.131 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.131 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.131 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.131 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 141947 2024-05-15T23:29:30.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.131 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.253152+0000", 2024-05-15T23:29:30.131 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.253152+0000", 2024-05-15T23:29:30.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.253154+0000", 2024-05-15T23:29:30.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.804e-06 2024-05-15T23:29:30.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.253161+0000", 2024-05-15T23:29:30.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5730000000000002e-06 2024-05-15T23:29:30.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.253178+0000", 2024-05-15T23:29:30.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6793e-05 2024-05-15T23:29:30.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.253194+0000", 2024-05-15T23:29:30.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5753e-05 2024-05-15T23:29:30.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.253324+0000", 2024-05-15T23:29:30.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000130221 2024-05-15T23:29:30.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.253471+0000", 2024-05-15T23:29:30.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014714400000000001 2024-05-15T23:29:30.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5", 2024-05-15T23:29:30.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.253588+0000", 2024-05-15T23:29:30.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011710900000000001 2024-05-15T23:29:30.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.257789+0000", 2024-05-15T23:29:30.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042012250000000003 2024-05-15T23:29:30.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.461407+0000", 2024-05-15T23:29:30.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.20361794499999999 2024-05-15T23:29:30.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.461454+0000", 2024-05-15T23:29:30.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6999999999999997e-05 2024-05-15T23:29:30.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:43.461488+0000", 2024-05-15T23:29:30.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4059e-05 2024-05-15T23:29:30.136 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.136 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.136 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.136 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:156622 6.b 6:dc1215ad:::benchmark_data_smithi090_131278_object21428:head [delete] snapc 0=[] ondisk+write+known_if_redirected e518)", 2024-05-15T23:29:30.136 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:26:55.445005+0000", 2024-05-15T23:29:30.136 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 154.60930163800001, 2024-05-15T23:29:30.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.148119841, 2024-05-15T23:29:30.136 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.136 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.136 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.136 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.136 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.137 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.137 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 156622 2024-05-15T23:29:30.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.137 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.445005+0000", 2024-05-15T23:29:30.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.445005+0000", 2024-05-15T23:29:30.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.445006+0000", 2024-05-15T23:29:30.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.412e-06 2024-05-15T23:29:30.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.445011+0000", 2024-05-15T23:29:30.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6129999999999997e-06 2024-05-15T23:29:30.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.445020+0000", 2024-05-15T23:29:30.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0769999999999997e-06 2024-05-15T23:29:30.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.445027+0000", 2024-05-15T23:29:30.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7120000000000003e-06 2024-05-15T23:29:30.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.445083+0000", 2024-05-15T23:29:30.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6354000000000003e-05 2024-05-15T23:29:30.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.445197+0000", 2024-05-15T23:29:30.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011417 2024-05-15T23:29:30.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:29:30.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.445304+0000", 2024-05-15T23:29:30.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000106774 2024-05-15T23:29:30.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.448963+0000", 2024-05-15T23:29:30.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0036589029999999998 2024-05-15T23:29:30.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.593034+0000", 2024-05-15T23:29:30.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14407140700000001 2024-05-15T23:29:30.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.593084+0000", 2024-05-15T23:29:30.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0108999999999997e-05 2024-05-15T23:29:30.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.593125+0000", 2024-05-15T23:29:30.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0309999999999999e-05 2024-05-15T23:29:30.141 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.141 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.141 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.142 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:156638 6.b 6:d1c7c7d0:::benchmark_data_smithi090_131278_object21444:head [delete] snapc 0=[] ondisk+write+known_if_redirected e518)", 2024-05-15T23:29:30.142 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:26:55.456546+0000", 2024-05-15T23:29:30.142 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 154.59776047400001, 2024-05-15T23:29:30.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14592569799999999, 2024-05-15T23:29:30.142 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.142 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.142 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.142 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.142 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.142 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.142 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 156638 2024-05-15T23:29:30.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.142 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.456546+0000", 2024-05-15T23:29:30.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.456546+0000", 2024-05-15T23:29:30.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.456547+0000", 2024-05-15T23:29:30.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.068e-06 2024-05-15T23:29:30.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.456551+0000", 2024-05-15T23:29:30.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6399999999999999e-06 2024-05-15T23:29:30.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.456559+0000", 2024-05-15T23:29:30.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6179999999999995e-06 2024-05-15T23:29:30.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.456567+0000", 2024-05-15T23:29:30.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5299999999999999e-06 2024-05-15T23:29:30.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.456617+0000", 2024-05-15T23:29:30.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9880999999999999e-05 2024-05-15T23:29:30.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.456702+0000", 2024-05-15T23:29:30.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4677000000000003e-05 2024-05-15T23:29:30.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:29:30.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.456764+0000", 2024-05-15T23:29:30.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2398999999999997e-05 2024-05-15T23:29:30.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.461426+0000", 2024-05-15T23:29:30.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046621639999999999 2024-05-15T23:29:30.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.602385+0000", 2024-05-15T23:29:30.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14095892199999999 2024-05-15T23:29:30.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.602431+0000", 2024-05-15T23:29:30.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5734e-05 2024-05-15T23:29:30.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.602472+0000", 2024-05-15T23:29:30.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1065000000000001e-05 2024-05-15T23:29:30.147 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.147 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.147 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.147 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:156656 6.b 6:dfb5b17e:::benchmark_data_smithi090_131278_object21462:head [delete] snapc 0=[] ondisk+write+known_if_redirected e518)", 2024-05-15T23:29:30.147 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:26:55.477373+0000", 2024-05-15T23:29:30.148 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 154.57693396799999, 2024-05-15T23:29:30.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12969223699999999, 2024-05-15T23:29:30.148 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.148 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.148 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.148 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.148 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.148 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.148 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 156656 2024-05-15T23:29:30.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.148 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.477373+0000", 2024-05-15T23:29:30.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.477373+0000", 2024-05-15T23:29:30.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.477375+0000", 2024-05-15T23:29:30.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2709999999999999e-06 2024-05-15T23:29:30.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.477383+0000", 2024-05-15T23:29:30.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3399999999999998e-06 2024-05-15T23:29:30.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.477404+0000", 2024-05-15T23:29:30.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1044999999999999e-05 2024-05-15T23:29:30.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.477413+0000", 2024-05-15T23:29:30.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4819999999999994e-06 2024-05-15T23:29:30.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.477475+0000", 2024-05-15T23:29:30.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2725000000000007e-05 2024-05-15T23:29:30.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.477614+0000", 2024-05-15T23:29:30.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013830799999999999 2024-05-15T23:29:30.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:29:30.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.477688+0000", 2024-05-15T23:29:30.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4695000000000002e-05 2024-05-15T23:29:30.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.481449+0000", 2024-05-15T23:29:30.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037604230000000002 2024-05-15T23:29:30.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.606978+0000", 2024-05-15T23:29:30.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.125529536 2024-05-15T23:29:30.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.607016+0000", 2024-05-15T23:29:30.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7397000000000001e-05 2024-05-15T23:29:30.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:26:55.607065+0000", 2024-05-15T23:29:30.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9014999999999997e-05 2024-05-15T23:29:30.153 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.153 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.153 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.153 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:253367 6.0 6:0e86c3ca:::benchmark_data_smithi090_131278_object118173:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e538)", 2024-05-15T23:29:30.153 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:28:18.173094+0000", 2024-05-15T23:29:30.153 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 71.881212234000003, 2024-05-15T23:29:30.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.49972758, 2024-05-15T23:29:30.153 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.153 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.154 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.154 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.154 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.154 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.154 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 253367 2024-05-15T23:29:30.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.154 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173094+0000", 2024-05-15T23:29:30.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173094+0000", 2024-05-15T23:29:30.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173096+0000", 2024-05-15T23:29:30.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.68e-06 2024-05-15T23:29:30.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173102+0000", 2024-05-15T23:29:30.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4790000000000001e-06 2024-05-15T23:29:30.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173115+0000", 2024-05-15T23:29:30.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2822e-05 2024-05-15T23:29:30.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173126+0000", 2024-05-15T23:29:30.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0772e-05 2024-05-15T23:29:30.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173196+0000", 2024-05-15T23:29:30.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0185999999999995e-05 2024-05-15T23:29:30.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:30.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173208+0000", 2024-05-15T23:29:30.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1864e-05 2024-05-15T23:29:30.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.658018+0000", 2024-05-15T23:29:30.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48480959400000001 2024-05-15T23:29:30.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.658110+0000", 2024-05-15T23:29:30.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1831999999999998e-05 2024-05-15T23:29:30.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-05-15T23:29:30.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.658222+0000", 2024-05-15T23:29:30.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112246 2024-05-15T23:29:30.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.662314+0000", 2024-05-15T23:29:30.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040926060000000004 2024-05-15T23:29:30.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.672508+0000", 2024-05-15T23:29:30.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010193631 2024-05-15T23:29:30.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.672670+0000", 2024-05-15T23:29:30.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016161600000000001 2024-05-15T23:29:30.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.672712+0000", 2024-05-15T23:29:30.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2165e-05 2024-05-15T23:29:30.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.672822+0000", 2024-05-15T23:29:30.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110087 2024-05-15T23:29:30.159 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:253378 6.0 6:059058dc:::benchmark_data_smithi090_131278_object118184:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e538)", 2024-05-15T23:29:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:28:18.173145+0000", 2024-05-15T23:29:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 71.881161603999999, 2024-05-15T23:29:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.499898065, 2024-05-15T23:29:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.160 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.161 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 253378 2024-05-15T23:29:30.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.161 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173145+0000", 2024-05-15T23:29:30.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173145+0000", 2024-05-15T23:29:30.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173146+0000", 2024-05-15T23:29:30.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0299999999999999e-07 2024-05-15T23:29:30.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173148+0000", 2024-05-15T23:29:30.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1959999999999998e-06 2024-05-15T23:29:30.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173153+0000", 2024-05-15T23:29:30.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7620000000000003e-06 2024-05-15T23:29:30.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173156+0000", 2024-05-15T23:29:30.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8600000000000003e-06 2024-05-15T23:29:30.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173259+0000", 2024-05-15T23:29:30.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000103077 2024-05-15T23:29:30.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:30.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173266+0000", 2024-05-15T23:29:30.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3319999999999997e-06 2024-05-15T23:29:30.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.662369+0000", 2024-05-15T23:29:30.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.48910341200000002 2024-05-15T23:29:30.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.662469+0000", 2024-05-15T23:29:30.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9755999999999998e-05 2024-05-15T23:29:30.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-05-15T23:29:30.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.662557+0000", 2024-05-15T23:29:30.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8094000000000001e-05 2024-05-15T23:29:30.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.672837+0000", 2024-05-15T23:29:30.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010280025999999999 2024-05-15T23:29:30.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.672894+0000", 2024-05-15T23:29:30.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7071999999999998e-05 2024-05-15T23:29:30.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.672953+0000", 2024-05-15T23:29:30.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8616999999999998e-05 2024-05-15T23:29:30.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.672987+0000", 2024-05-15T23:29:30.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4449000000000002e-05 2024-05-15T23:29:30.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.673043+0000", 2024-05-15T23:29:30.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5809e-05 2024-05-15T23:29:30.167 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.167 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.167 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.167 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:253388 6.0 6:0afc1266:::benchmark_data_smithi090_131278_object118194:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e538)", 2024-05-15T23:29:30.167 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:28:18.173171+0000", 2024-05-15T23:29:30.167 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 71.881135880000002, 2024-05-15T23:29:30.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.50028118300000002, 2024-05-15T23:29:30.167 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.167 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.167 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.167 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.167 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.168 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.168 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 253388 2024-05-15T23:29:30.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.168 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173171+0000", 2024-05-15T23:29:30.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173171+0000", 2024-05-15T23:29:30.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173171+0000", 2024-05-15T23:29:30.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.65e-07 2024-05-15T23:29:30.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173173+0000", 2024-05-15T23:29:30.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.829e-06 2024-05-15T23:29:30.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173176+0000", 2024-05-15T23:29:30.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.275e-06 2024-05-15T23:29:30.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173179+0000", 2024-05-15T23:29:30.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9799999999999998e-06 2024-05-15T23:29:30.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173301+0000", 2024-05-15T23:29:30.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012235800000000001 2024-05-15T23:29:30.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:30.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173309+0000", 2024-05-15T23:29:30.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6009999999999997e-06 2024-05-15T23:29:30.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.668240+0000", 2024-05-15T23:29:30.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.494930971 2024-05-15T23:29:30.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.668360+0000", 2024-05-15T23:29:30.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011972 2024-05-15T23:29:30.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-05-15T23:29:30.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.668491+0000", 2024-05-15T23:29:30.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013135999999999999 2024-05-15T23:29:30.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.673051+0000", 2024-05-15T23:29:30.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045594199999999998 2024-05-15T23:29:30.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.673100+0000", 2024-05-15T23:29:30.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9341000000000001e-05 2024-05-15T23:29:30.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.673293+0000", 2024-05-15T23:29:30.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000193489 2024-05-15T23:29:30.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.673349+0000", 2024-05-15T23:29:30.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5757000000000002e-05 2024-05-15T23:29:30.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.673452+0000", 2024-05-15T23:29:30.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102717 2024-05-15T23:29:30.173 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.173 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.173 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.174 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:253389 6.0 6:011536c0:::benchmark_data_smithi090_131278_object118195:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e538)", 2024-05-15T23:29:30.174 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:28:18.173190+0000", 2024-05-15T23:29:30.174 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 71.881117044000007, 2024-05-15T23:29:30.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.51899835699999997, 2024-05-15T23:29:30.174 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.174 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.174 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.174 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.174 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.174 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.174 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 253389 2024-05-15T23:29:30.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.175 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173190+0000", 2024-05-15T23:29:30.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173190+0000", 2024-05-15T23:29:30.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173190+0000", 2024-05-15T23:29:30.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2800000000000003e-07 2024-05-15T23:29:30.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173191+0000", 2024-05-15T23:29:30.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6029999999999999e-06 2024-05-15T23:29:30.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173195+0000", 2024-05-15T23:29:30.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0790000000000001e-06 2024-05-15T23:29:30.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173197+0000", 2024-05-15T23:29:30.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.847e-06 2024-05-15T23:29:30.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173336+0000", 2024-05-15T23:29:30.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013902200000000001 2024-05-15T23:29:30.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:30.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173340+0000", 2024-05-15T23:29:30.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9859999999999996e-06 2024-05-15T23:29:30.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.673490+0000", 2024-05-15T23:29:30.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.500149336 2024-05-15T23:29:30.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.673563+0000", 2024-05-15T23:29:30.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3654999999999999e-05 2024-05-15T23:29:30.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-05-15T23:29:30.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.673633+0000", 2024-05-15T23:29:30.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9370999999999997e-05 2024-05-15T23:29:30.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.683734+0000", 2024-05-15T23:29:30.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010101222999999999 2024-05-15T23:29:30.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.683774+0000", 2024-05-15T23:29:30.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9604000000000002e-05 2024-05-15T23:29:30.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.692032+0000", 2024-05-15T23:29:30.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082585520000000006 2024-05-15T23:29:30.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.692095+0000", 2024-05-15T23:29:30.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3241e-05 2024-05-15T23:29:30.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.692188+0000", 2024-05-15T23:29:30.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2509999999999993e-05 2024-05-15T23:29:30.180 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.180 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.180 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.26326.0:253392 6.0 6:0f936e1a:::benchmark_data_smithi090_131278_object118198:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e538)", 2024-05-15T23:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:28:18.173201+0000", 2024-05-15T23:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 71.881105493999996, 2024-05-15T23:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.51909059499999999, 2024-05-15T23:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.26326", 2024-05-15T23:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 253392 2024-05-15T23:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.182 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:29:30.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:29:30.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173201+0000", 2024-05-15T23:29:30.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:29:30.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173201+0000", 2024-05-15T23:29:30.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:29:30.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:29:30.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173201+0000", 2024-05-15T23:29:30.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1100000000000002e-07 2024-05-15T23:29:30.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:29:30.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173203+0000", 2024-05-15T23:29:30.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4580000000000001e-06 2024-05-15T23:29:30.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:29:30.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173206+0000", 2024-05-15T23:29:30.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7410000000000001e-06 2024-05-15T23:29:30.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:29:30.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173208+0000", 2024-05-15T23:29:30.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7010000000000001e-06 2024-05-15T23:29:30.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173379+0000", 2024-05-15T23:29:30.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017037000000000001 2024-05-15T23:29:30.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:29:30.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.173386+0000", 2024-05-15T23:29:30.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2960000000000001e-06 2024-05-15T23:29:30.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:29:30.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.678415+0000", 2024-05-15T23:29:30.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.50502940100000004 2024-05-15T23:29:30.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:29:30.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.678513+0000", 2024-05-15T23:29:30.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7157000000000005e-05 2024-05-15T23:29:30.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-05-15T23:29:30.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.678586+0000", 2024-05-15T23:29:30.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3779e-05 2024-05-15T23:29:30.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.683834+0000", 2024-05-15T23:29:30.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0052476889999999998 2024-05-15T23:29:30.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:29:30.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.683893+0000", 2024-05-15T23:29:30.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8693999999999997e-05 2024-05-15T23:29:30.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:29:30.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.692197+0000", 2024-05-15T23:29:30.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083043100000000005 2024-05-15T23:29:30.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:29:30.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.692233+0000", 2024-05-15T23:29:30.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5945000000000002e-05 2024-05-15T23:29:30.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:29:30.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:29:30.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:29:30.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:28:18.692292+0000", 2024-05-15T23:29:30.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8743000000000003e-05 2024-05-15T23:29:30.187 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.187 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.187 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.187 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:29:30.187 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:29:30.187 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:29:30.188 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:29:30.189 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:29 smithi088 ceph-mon[44699]: from='client.36638 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:29:30.189 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:29 smithi088 ceph-mon[44699]: osdmap e564: 12 total, 12 up, 9 in 2024-05-15T23:29:30.189 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:29 smithi088 ceph-mon[44699]: pgmap v1509: 17 pgs: 17 active+clean; 4.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-05-15T23:29:30.218 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:30.242 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:29:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:30.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:29:30.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:30.388 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:30.413 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:30.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:30.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:30.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:29:30.567 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:29:30.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:30.596 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:29:30.692 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 26 16 74619 74603 179.313 114.812 0.00501348 0.0055749 2024-05-15T23:29:30.697 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:29:30.736 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:30.762 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:29:30.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:30.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:29:30.791 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:30 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1817805156' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 0, 4, 9]}]: dispatch 2024-05-15T23:29:30.792 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:30 smithi090 ceph-mon[44434]: from='client.36641 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 0, 4, 9]}]: dispatch 2024-05-15T23:29:30.799 INFO:teuthology.orchestra.run.smithi084.stderr:set 7.4 pg_upmap_items mapping to [7->0,4->9] 2024-05-15T23:29:30.819 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:29:30.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:30 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1817805156' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 0, 4, 9]}]: dispatch 2024-05-15T23:29:30.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:30 smithi084 ceph-mon[39784]: from='client.36641 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 0, 4, 9]}]: dispatch 2024-05-15T23:29:30.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:29:30.902 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:30.924 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:29:30.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:30.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:29:31.028 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:29:31.052 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:30 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1817805156' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 0, 4, 9]}]: dispatch 2024-05-15T23:29:31.052 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:30 smithi088 ceph-mon[44699]: from='client.36641 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 0, 4, 9]}]: dispatch 2024-05-15T23:29:31.054 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:31.073 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:29:31.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:31.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:29:31.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:29:31.204 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:31.222 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:29:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:31.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:29:31.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:29:31.352 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:31.370 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:29:31.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:31.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:29:31.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:29:31.501 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:31.519 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:29:31.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:31.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:29:31.623 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:29:31.652 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:31.673 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:29:31.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:31.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:29:31.687 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T23:29:31.693 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 27 16 74996 74980 173.545 23.5625 0.00321493 0.00558471 2024-05-15T23:29:31.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:29:31.807 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:31.826 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:29:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:31.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:29:31.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:29:31.957 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:31.976 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:29:31.977 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:29:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:31.979 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:29:32.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:31 smithi090 ceph-mon[44434]: from='client.36641 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 0, 4, 9]}]': finished 2024-05-15T23:29:32.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:31 smithi090 ceph-mon[44434]: osdmap e565: 12 total, 12 up, 9 in 2024-05-15T23:29:32.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:31 smithi090 ceph-mon[44434]: pgmap v1511: 17 pgs: 1 unknown, 16 active+clean; 4.0 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-05-15T23:29:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:31 smithi088 ceph-mon[44699]: from='client.36641 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 0, 4, 9]}]': finished 2024-05-15T23:29:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:31 smithi088 ceph-mon[44699]: osdmap e565: 12 total, 12 up, 9 in 2024-05-15T23:29:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:31 smithi088 ceph-mon[44699]: pgmap v1511: 17 pgs: 1 unknown, 16 active+clean; 4.0 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-05-15T23:29:32.080 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:29:32.106 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:32.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:31 smithi084 ceph-mon[39784]: from='client.36641 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 0, 4, 9]}]': finished 2024-05-15T23:29:32.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:31 smithi084 ceph-mon[39784]: osdmap e565: 12 total, 12 up, 9 in 2024-05-15T23:29:32.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:31 smithi084 ceph-mon[39784]: pgmap v1511: 17 pgs: 1 unknown, 16 active+clean; 4.0 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-05-15T23:29:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:29:32.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:32.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:29:32.409 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:29:32.443 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:32.612 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:29:32.693 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 28 16 74996 74980 167.347 0 - 0.00558471 2024-05-15T23:29:33.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:32 smithi090 ceph-mon[44434]: osdmap e566: 12 total, 12 up, 9 in 2024-05-15T23:29:33.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:32 smithi088 ceph-mon[44699]: osdmap e566: 12 total, 12 up, 9 in 2024-05-15T23:29:33.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:32 smithi084 ceph-mon[39784]: osdmap e566: 12 total, 12 up, 9 in 2024-05-15T23:29:33.693 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 29 16 75470 75454 162.598 14.8125 0.00867465 0.00614788 2024-05-15T23:29:33.808 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:33 smithi088 ceph-mon[44699]: osdmap e567: 12 total, 12 up, 9 in 2024-05-15T23:29:33.808 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:33 smithi088 ceph-mon[44699]: pgmap v1514: 17 pgs: 1 unknown, 16 active+clean; 4.2 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 60 MiB/s wr, 953 op/s 2024-05-15T23:29:34.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:33 smithi084 ceph-mon[39784]: osdmap e567: 12 total, 12 up, 9 in 2024-05-15T23:29:34.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:33 smithi084 ceph-mon[39784]: pgmap v1514: 17 pgs: 1 unknown, 16 active+clean; 4.2 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 60 MiB/s wr, 953 op/s 2024-05-15T23:29:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:33 smithi090 ceph-mon[44434]: osdmap e567: 12 total, 12 up, 9 in 2024-05-15T23:29:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:33 smithi090 ceph-mon[44434]: pgmap v1514: 17 pgs: 1 unknown, 16 active+clean; 4.2 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 60 MiB/s wr, 953 op/s 2024-05-15T23:29:34.368 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:29:34.368 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:29:34.368 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:29:34.368 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:76550 7.1 7:8e936aef:::benchmark_data_smithi090_143373_object76549:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e567)", 2024-05-15T23:29:34.368 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:29:34.363812+0000", 2024-05-15T23:29:34.368 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.0032291490000000002, 2024-05-15T23:29:34.368 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032623259999999999, 2024-05-15T23:29:34.369 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:29:34.369 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:29:34.369 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:29:34.369 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:29:34.369 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:29:34.369 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:29:34.369 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 76550 2024-05-15T23:29:34.369 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:29:34.369 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:29:34.369 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:29:34.369 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:29:34.369 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:29:34.363812+0000", 2024-05-15T23:29:34.369 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:29:34.369 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:29:34.369 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:29:34.370 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:29:34.370 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:29:34.363812+0000", 2024-05-15T23:29:34.370 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:29:34.370 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:29:34.370 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:29:34.370 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:29:34.370 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:29:34.363814+0000", 2024-05-15T23:29:34.370 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.7570000000000001e-06 2024-05-15T23:29:34.370 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:29:34.370 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:29:34.370 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:29:34.370 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:29:34.363867+0000", 2024-05-15T23:29:34.370 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3186999999999997e-05 2024-05-15T23:29:34.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:29:34.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:29:34.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:29:34.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:29:34.363878+0000", 2024-05-15T23:29:34.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1734e-05 2024-05-15T23:29:34.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:29:34.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:29:34.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:29:34.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:29:34.363885+0000", 2024-05-15T23:29:34.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1260000000000003e-06 2024-05-15T23:29:34.371 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:29:34.371 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:29:34.371 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:29:34.371 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:29:34.363926+0000", 2024-05-15T23:29:34.371 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1570999999999999e-05 2024-05-15T23:29:34.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:29:34.372 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:29:34.372 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:29:34.372 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:29:34.364035+0000", 2024-05-15T23:29:34.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000108847 2024-05-15T23:29:34.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:29:34.372 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:29:34.372 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 10", 2024-05-15T23:29:34.372 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:29:34.364175+0000", 2024-05-15T23:29:34.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013954000000000001 2024-05-15T23:29:34.372 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:29:34.372 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:29:34.372 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:29:34.372 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:29:34.366067+0000", 2024-05-15T23:29:34.372 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001892814 2024-05-15T23:29:34.373 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:29:34.373 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:29:34.373 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:29:34.373 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:29:34.373 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2024-05-15T23:29:34.373 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 1 2024-05-15T23:29:34.373 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:29:34.693 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 30 16 77054 77038 160.477 99 0.0118227 0.00622853 2024-05-15T23:29:35.142 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T23:29:35.444 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:29:35.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:35 smithi090 ceph-mon[44434]: pgmap v1515: 17 pgs: 1 unknown, 16 active+clean; 4.3 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 49 MiB/s wr, 781 op/s 2024-05-15T23:29:35.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:35 smithi088 ceph-mon[44699]: pgmap v1515: 17 pgs: 1 unknown, 16 active+clean; 4.3 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 49 MiB/s wr, 781 op/s 2024-05-15T23:29:35.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:35 smithi084 ceph-mon[39784]: pgmap v1515: 17 pgs: 1 unknown, 16 active+clean; 4.3 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 49 MiB/s wr, 781 op/s 2024-05-15T23:29:35.693 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 31 16 78598 78582 158.413 96.5 0.00564354 0.00630961 2024-05-15T23:29:36.693 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 32 15 80164 80149 156.523 97.9375 0.0078115 0.00638616 2024-05-15T23:29:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:37 smithi090 ceph-mon[44434]: pgmap v1516: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 100 MiB/s wr, 1.60k op/s; 4786/156086 objects misplaced (3.066%); 770 KiB/s, 12 objects/s recovering 2024-05-15T23:29:37.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:37 smithi088 ceph-mon[44699]: pgmap v1516: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 100 MiB/s wr, 1.60k op/s; 4786/156086 objects misplaced (3.066%); 770 KiB/s, 12 objects/s recovering 2024-05-15T23:29:37.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:37 smithi084 ceph-mon[39784]: pgmap v1516: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 100 MiB/s wr, 1.60k op/s; 4786/156086 objects misplaced (3.066%); 770 KiB/s, 12 objects/s recovering 2024-05-15T23:29:37.693 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 33 16 81815 81799 154.905 103.125 0.0173203 0.00645368 2024-05-15T23:29:38.338 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:29:38.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:38.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:29:38.448 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:29:38.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:38 smithi084 ceph-mon[39784]: osdmap e568: 12 total, 12 up, 9 in 2024-05-15T23:29:38.499 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:38 smithi090 ceph-mon[44434]: osdmap e568: 12 total, 12 up, 9 in 2024-05-15T23:29:38.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:38 smithi088 ceph-mon[44699]: osdmap e568: 12 total, 12 up, 9 in 2024-05-15T23:29:38.693 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 34 16 83378 83362 153.222 97.6875 0.00819205 0.00652503 2024-05-15T23:29:38.759 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:29:38.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:38.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:29:38.865 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:29:38.897 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:39 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:29:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:39 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:29:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:39 smithi090 ceph-mon[44434]: osdmap e569: 12 total, 12 up, 9 in 2024-05-15T23:29:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:39 smithi090 ceph-mon[44434]: pgmap v1519: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s; 4786/158424 objects misplaced (3.021%); 771 KiB/s, 12 objects/s recovering 2024-05-15T23:29:39.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:39 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:29:39.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:39 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:29:39.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:39 smithi088 ceph-mon[44699]: osdmap e569: 12 total, 12 up, 9 in 2024-05-15T23:29:39.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:39 smithi088 ceph-mon[44699]: pgmap v1519: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s; 4786/158424 objects misplaced (3.021%); 771 KiB/s, 12 objects/s recovering 2024-05-15T23:29:39.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:39 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:29:39.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:39 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:29:39.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:39 smithi084 ceph-mon[39784]: osdmap e569: 12 total, 12 up, 9 in 2024-05-15T23:29:39.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:39 smithi084 ceph-mon[39784]: pgmap v1519: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s; 4786/158424 objects misplaced (3.021%); 771 KiB/s, 12 objects/s recovering 2024-05-15T23:29:39.683 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:29:39.693 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 35 16 84906 84890 151.573 95.5 0.00318805 0.00659501 2024-05-15T23:29:40.448 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:29:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:40.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:29:40.553 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:29:40.582 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:40.693 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 36 16 86383 86367 149.926 92.3125 0.0109633 0.00666785 2024-05-15T23:29:40.778 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:40 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4017634259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:29:40.781 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:29:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:40.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:29:40.870 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:29:40.870 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:29:40.870 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:29:40.870 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:29:40.870 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:29:40.871 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:29:40.871 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:29:40.876 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:29:40.886 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:29:40.915 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:29:40.948 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:40 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4017634259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:29:40.951 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:40.961 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:29:41.006 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:29:41.018 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:29:41.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:41.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:29:41.044 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:29:41.046 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:29:41.046 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:29:41.046 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:29:41.046 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:29:41.046 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:29:41.046 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:29:41.046 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:29:41.047 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:29:41.047 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:29:41.047 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:29:41.047 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:29:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:40 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4017634259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:29:41.101 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:29:41.101 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:29:41.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:29:41.193 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:41.222 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:29:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:41.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:29:41.336 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:29:41.367 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:41.392 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:29:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:41.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:29:41.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:29:41.617 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:41.693 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 37 16 87826 87810 148.312 90.1875 0.0202041 0.00674124 2024-05-15T23:29:41.752 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:29:41.759 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:41 smithi084 ceph-mon[39784]: pgmap v1520: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.1 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 105 MiB/s wr, 1.68k op/s; 4620/166196 objects misplaced (2.780%); 1.9 MiB/s, 30 objects/s recovering 2024-05-15T23:29:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:41.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:29:41.862 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:29:41.890 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:41.916 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:29:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:41.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:29:41.943 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.944 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.944 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.944 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.944 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.944 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.944 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.944 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.944 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.944 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.944 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:41.944 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:41 smithi088 ceph-mon[44699]: pgmap v1520: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.1 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 105 MiB/s wr, 1.68k op/s; 4620/166196 objects misplaced (2.780%); 1.9 MiB/s, 30 objects/s recovering 2024-05-15T23:29:41.945 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T23:29:42.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:29:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:41 smithi090 ceph-mon[44434]: pgmap v1520: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.1 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 105 MiB/s wr, 1.68k op/s; 4620/166196 objects misplaced (2.780%); 1.9 MiB/s, 30 objects/s recovering 2024-05-15T23:29:42.070 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:42.096 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:29:42.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:42.103 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:29:42.204 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:29:42.293 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:29:42.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 38 16 89382 89366 146.968 97.25 0.0109007 0.00680219 2024-05-15T23:29:43.006 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:43.007 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:42 smithi088 ceph-mon[44699]: osdmap e570: 12 total, 12 up, 9 in 2024-05-15T23:29:43.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:43.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:42 smithi090 ceph-mon[44434]: osdmap e570: 12 total, 12 up, 9 in 2024-05-15T23:29:43.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:29:43.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:42 smithi084 ceph-mon[39784]: osdmap e570: 12 total, 12 up, 9 in 2024-05-15T23:29:43.504 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:43.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 39 16 90881 90865 145.601 93.6875 0.00307161 0.00686401 2024-05-15T23:29:43.730 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10, 2] out_osds: [0, 6, 8] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:29:43.730 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:29:43.730 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-05-15T23:29:43.730 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T23:29:43.730 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T23:29:43.731 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-05-15T23:29:43.968 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:43 smithi084 ceph-mon[39784]: pgmap v1522: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 93 MiB/s wr, 1.48k op/s; 4620/173900 objects misplaced (2.657%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T23:29:44.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:43 smithi090 ceph-mon[44434]: pgmap v1522: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 93 MiB/s wr, 1.48k op/s; 4620/173900 objects misplaced (2.657%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T23:29:44.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:43 smithi088 ceph-mon[44699]: pgmap v1522: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 93 MiB/s wr, 1.48k op/s; 4620/173900 objects misplaced (2.657%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T23:29:44.453 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:29:44.509 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:29:44.510 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:29:44.592 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:29:44.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:44.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:29:44.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:29:44.693848 min lat: 0.000985917 max lat: 2.57351 avg lat: 0.00693453 2024-05-15T23:29:44.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:29:44.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 40 16 92289 92273 144.161 88 0.00932088 0.00693453 2024-05-15T23:29:44.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:29:44.732 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:44.767 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:29:44.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:44.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:29:44.787 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:44 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:29:44.787 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:44 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:29:44.805 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:44 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:29:44.805 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:44 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:29:44.839 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:29:44.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:44.907 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:44.933 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:44.940 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:45.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:44 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:29:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:44 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:29:45.069 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:45.097 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:45.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:45.103 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:45.204 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:29:45.235 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:45.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 41 16 93598 93582 142.64 81.8125 0.00912572 0.00700607 2024-05-15T23:29:45.988 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:29:46.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:45 smithi090 ceph-mon[44434]: osdmap e571: 12 total, 12 up, 9 in 2024-05-15T23:29:46.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:45 smithi090 ceph-mon[44434]: pgmap v1524: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 79 MiB/s wr, 1.27k op/s; 4620/175894 objects misplaced (2.627%); 1.5 MiB/s, 24 objects/s recovering 2024-05-15T23:29:46.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2144638594' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:29:46.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:45 smithi090 ceph-mon[44434]: from='client.36647 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:29:46.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:45 smithi088 ceph-mon[44699]: osdmap e571: 12 total, 12 up, 9 in 2024-05-15T23:29:46.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:45 smithi088 ceph-mon[44699]: pgmap v1524: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 79 MiB/s wr, 1.27k op/s; 4620/175894 objects misplaced (2.627%); 1.5 MiB/s, 24 objects/s recovering 2024-05-15T23:29:46.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2144638594' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:29:46.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:45 smithi088 ceph-mon[44699]: from='client.36647 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:29:46.210 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:45 smithi084 ceph-mon[39784]: osdmap e571: 12 total, 12 up, 9 in 2024-05-15T23:29:46.210 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:45 smithi084 ceph-mon[39784]: pgmap v1524: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 79 MiB/s wr, 1.27k op/s; 4620/175894 objects misplaced (2.627%); 1.5 MiB/s, 24 objects/s recovering 2024-05-15T23:29:46.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2144638594' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:29:46.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:45 smithi084 ceph-mon[39784]: from='client.36647 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:29:46.579 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T23:29:46.586 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:29:46.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:46.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:29:46.690 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:29:46.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 42 16 95001 94985 141.332 87.6875 0.00883627 0.00707421 2024-05-15T23:29:46.717 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:46.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:46 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:29:46.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:46 smithi088 ceph-mon[44699]: from='client.36647 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:29:46.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:46 smithi088 ceph-mon[44699]: osdmap e572: 12 total, 12 up, 9 in 2024-05-15T23:29:47.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:46 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:29:47.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:46 smithi090 ceph-mon[44434]: from='client.36647 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:29:47.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:46 smithi090 ceph-mon[44434]: osdmap e572: 12 total, 12 up, 9 in 2024-05-15T23:29:47.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:46 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:29:47.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:46 smithi084 ceph-mon[39784]: from='client.36647 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:29:47.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:46 smithi084 ceph-mon[39784]: osdmap e572: 12 total, 12 up, 9 in 2024-05-15T23:29:47.527 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:29:47.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 43 16 96438 96422 140.134 89.8125 0.00417318 0.00713419 2024-05-15T23:29:47.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:47 smithi088 ceph-mon[44699]: pgmap v1526: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 102 MiB/s wr, 1.63k op/s; 4455/185770 objects misplaced (2.398%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T23:29:47.900 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:29:47.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:47.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:29:48.004 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:48.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:47 smithi090 ceph-mon[44434]: pgmap v1526: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 102 MiB/s wr, 1.63k op/s; 4455/185770 objects misplaced (2.398%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T23:29:48.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:47 smithi084 ceph-mon[39784]: pgmap v1526: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 102 MiB/s wr, 1.63k op/s; 4455/185770 objects misplaced (2.398%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T23:29:48.134 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:29:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:48.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:29:48.239 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:29:48.266 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:48.322 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:29:48.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:48.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:29:48.428 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:29:48.456 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:48.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 44 16 97862 97846 138.971 89 0.00350875 0.00719177 2024-05-15T23:29:49.306 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:29:49.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:49.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:29:49.411 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:29:49.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:49 smithi084 ceph-mon[39784]: pgmap v1527: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 78 MiB/s wr, 1.25k op/s; 4455/188312 objects misplaced (2.366%); 1.5 MiB/s, 23 objects/s recovering 2024-05-15T23:29:49.538 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:29:49.538 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-15T23:29:49.538 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:29:49.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:49 smithi090 ceph-mon[44434]: pgmap v1527: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 78 MiB/s wr, 1.25k op/s; 4455/188312 objects misplaced (2.366%); 1.5 MiB/s, 23 objects/s recovering 2024-05-15T23:29:49.557 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:49 smithi088 ceph-mon[44699]: pgmap v1527: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 78 MiB/s wr, 1.25k op/s; 4455/188312 objects misplaced (2.366%); 1.5 MiB/s, 23 objects/s recovering 2024-05-15T23:29:49.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 45 15 99231 99216 137.786 85.625 0.00730248 0.0072559 2024-05-15T23:29:50.228 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:50.252 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:29:50.252 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:29:50.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 46 16 100664 100648 136.736 89.5 0.0106459 0.00731197 2024-05-15T23:29:51.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:51 smithi088 ceph-mon[44699]: pgmap v1528: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.0 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 84 MiB/s wr, 1.34k op/s; 4288/195390 objects misplaced (2.195%); 2.6 MiB/s, 41 objects/s recovering 2024-05-15T23:29:51.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:51 smithi090 ceph-mon[44434]: pgmap v1528: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.0 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 84 MiB/s wr, 1.34k op/s; 4288/195390 objects misplaced (2.195%); 2.6 MiB/s, 41 objects/s recovering 2024-05-15T23:29:51.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:51 smithi084 ceph-mon[39784]: pgmap v1528: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.0 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 84 MiB/s wr, 1.34k op/s; 4288/195390 objects misplaced (2.195%); 2.6 MiB/s, 41 objects/s recovering 2024-05-15T23:29:51.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 47 16 102055 102039 135.676 86.9375 0.0185787 0.00736878 2024-05-15T23:29:52.138 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T23:29:52.431 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:29:52.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 48 16 103402 103386 134.603 84.1875 0.00381439 0.00742738 2024-05-15T23:29:53.449 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:53 smithi088 ceph-mon[44699]: pgmap v1529: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.2 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s; 4288/202356 objects misplaced (2.119%); 2.5 MiB/s, 39 objects/s recovering 2024-05-15T23:29:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:53 smithi090 ceph-mon[44434]: pgmap v1529: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.2 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s; 4288/202356 objects misplaced (2.119%); 2.5 MiB/s, 39 objects/s recovering 2024-05-15T23:29:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:53 smithi084 ceph-mon[39784]: pgmap v1529: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.2 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s; 4288/202356 objects misplaced (2.119%); 2.5 MiB/s, 39 objects/s recovering 2024-05-15T23:29:53.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 49 16 104682 104666 133.489 80 0.00483895 0.00748867 2024-05-15T23:29:53.912 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:29:53.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:53.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:29:53.962 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:29:54.019 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:29:54.059 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:54.527 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:29:54.527 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:29:54.695 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 50 16 105983 105967 132.445 81.3125 0.016915 0.00754839 2024-05-15T23:29:55.281 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:55 smithi088 ceph-mon[44699]: pgmap v1530: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.2 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 88 MiB/s wr, 1.41k op/s; 4288/204038 objects misplaced (2.102%); 2.1 MiB/s, 33 objects/s recovering 2024-05-15T23:29:55.315 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T23:29:55.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:55 smithi090 ceph-mon[44434]: pgmap v1530: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.2 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 88 MiB/s wr, 1.41k op/s; 4288/204038 objects misplaced (2.102%); 2.1 MiB/s, 33 objects/s recovering 2024-05-15T23:29:55.615 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:29:55.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:55 smithi084 ceph-mon[39784]: pgmap v1530: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.2 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 88 MiB/s wr, 1.41k op/s; 4288/204038 objects misplaced (2.102%); 2.1 MiB/s, 33 objects/s recovering 2024-05-15T23:29:55.694 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 51 16 107295 107279 131.456 82 0.0172516 0.00760498 2024-05-15T23:29:55.733 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:29:55.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:55.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:29:55.840 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:29:55.899 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:55.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:29:56.364 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:29:56.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:56.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:29:56.383 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10, 2] out_osds: [0, 6, 8] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:29:56.383 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:29:56.383 INFO:tasks.thrashosds.thrasher:Reweighting osd 4 to 0.3968249104293252 2024-05-15T23:29:56.383 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 4 0.3968249104293252 2024-05-15T23:29:56.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:29:56.515 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:56.544 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:29:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:56.548 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:29:56.649 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:29:56.695 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 52 16 108600 108584 130.496 81.5625 0.00826541 0.00766098 2024-05-15T23:29:56.707 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:57 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:29:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:57 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:29:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:57 smithi088 ceph-mon[44699]: osdmap e573: 12 total, 12 up, 9 in 2024-05-15T23:29:57.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:57 smithi088 ceph-mon[44699]: pgmap v1532: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.5 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 86 MiB/s wr, 1.37k op/s; 4126/213166 objects misplaced (1.936%); 2.1 MiB/s, 32 objects/s recovering 2024-05-15T23:29:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:57 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:29:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:57 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:29:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:57 smithi090 ceph-mon[44434]: osdmap e573: 12 total, 12 up, 9 in 2024-05-15T23:29:57.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:57 smithi090 ceph-mon[44434]: pgmap v1532: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.5 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 86 MiB/s wr, 1.37k op/s; 4126/213166 objects misplaced (1.936%); 2.1 MiB/s, 32 objects/s recovering 2024-05-15T23:29:57.653 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:29:57.654 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:29:57.661 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:57 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:29:57.661 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:57 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:29:57.661 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:57 smithi084 ceph-mon[39784]: osdmap e573: 12 total, 12 up, 9 in 2024-05-15T23:29:57.661 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:57 smithi084 ceph-mon[39784]: pgmap v1532: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.5 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 86 MiB/s wr, 1.37k op/s; 4126/213166 objects misplaced (1.936%); 2.1 MiB/s, 32 objects/s recovering 2024-05-15T23:29:57.695 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 53 16 109901 109885 129.568 81.3125 0.0220293 0.00771628 2024-05-15T23:29:58.398 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:58 smithi088 ceph-mon[44699]: osdmap e574: 12 total, 12 up, 9 in 2024-05-15T23:29:58.431 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T23:29:58.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:58 smithi090 ceph-mon[44434]: osdmap e574: 12 total, 12 up, 9 in 2024-05-15T23:29:58.577 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:29:58.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:58.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:29:58.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:58 smithi084 ceph-mon[39784]: osdmap e574: 12 total, 12 up, 9 in 2024-05-15T23:29:58.681 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:29:58.695 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 54 16 111160 111144 128.626 78.6875 0.00981595 0.00777209 2024-05-15T23:29:58.714 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:58.741 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:29:58.741 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:29:58.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:58.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:29:58.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:29:58.894 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:59.261 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:29:59.267 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:29:59.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:59.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:29:59.280 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:29:59.291 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:59 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:29:59.291 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:59 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:29:59.291 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:29:59 smithi088 ceph-mon[44699]: pgmap v1534: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.6 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 79 MiB/s wr, 1.26k op/s; 4126/215512 objects misplaced (1.915%); 1.3 MiB/s, 20 objects/s recovering 2024-05-15T23:29:59.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:29:59.422 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:59.446 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:29:59.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:29:59.449 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:29:59.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:29:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:59 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:29:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:59 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:29:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:29:59 smithi090 ceph-mon[44434]: pgmap v1534: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.6 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 79 MiB/s wr, 1.26k op/s; 4126/215512 objects misplaced (1.915%); 1.3 MiB/s, 20 objects/s recovering 2024-05-15T23:29:59.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:59 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:29:59.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:59 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:29:59.557 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:29:59 smithi084 ceph-mon[39784]: pgmap v1534: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.6 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 79 MiB/s wr, 1.26k op/s; 4126/215512 objects misplaced (1.915%); 1.3 MiB/s, 20 objects/s recovering 2024-05-15T23:29:59.608 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:29:59.695 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 55 16 112411 112395 127.709 78.1875 0.00475108 0.0078281 2024-05-15T23:30:00.516 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:00 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:30:00.516 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:00 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:30:00.516 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:00 smithi088 ceph-mon[44699]: osdmap e575: 12 total, 12 up, 9 in 2024-05-15T23:30:00.516 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:00 smithi088 ceph-mon[44699]: overall HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-05-15T23:30:00.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:00 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:30:00.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:00 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:30:00.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:00 smithi090 ceph-mon[44434]: osdmap e575: 12 total, 12 up, 9 in 2024-05-15T23:30:00.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:00 smithi090 ceph-mon[44434]: overall HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-05-15T23:30:00.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:00 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:30:00.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:00 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:30:00.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:00 smithi084 ceph-mon[39784]: osdmap e575: 12 total, 12 up, 9 in 2024-05-15T23:30:00.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:00 smithi084 ceph-mon[39784]: overall HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-05-15T23:30:00.696 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 56 16 113690 113674 126.853 79.9375 0.0122501 0.00788189 2024-05-15T23:30:01.016 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:30:01.016 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:30:01.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:01 smithi088 ceph-mon[44699]: pgmap v1536: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.7 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 81 MiB/s wr, 1.30k op/s; 4126/219650 objects misplaced (1.878%); 1.7 MiB/s, 26 objects/s recovering 2024-05-15T23:30:01.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:01 smithi084 ceph-mon[39784]: pgmap v1536: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.7 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 81 MiB/s wr, 1.30k op/s; 4126/219650 objects misplaced (1.878%); 1.7 MiB/s, 26 objects/s recovering 2024-05-15T23:30:01.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:01 smithi090 ceph-mon[44434]: pgmap v1536: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.7 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 81 MiB/s wr, 1.30k op/s; 4126/219650 objects misplaced (1.878%); 1.7 MiB/s, 26 objects/s recovering 2024-05-15T23:30:01.696 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 57 16 114892 114876 125.945 75.125 0.00581159 0.00793805 2024-05-15T23:30:01.818 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T23:30:02.140 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:30:02.696 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 58 16 116155 116139 125.135 78.9375 0.00903429 0.00798975 2024-05-15T23:30:03.203 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:03 smithi084 ceph-mon[39784]: osdmap e576: 12 total, 12 up, 9 in 2024-05-15T23:30:03.203 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/154604068' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:30:03.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:03 smithi088 ceph-mon[44699]: osdmap e576: 12 total, 12 up, 9 in 2024-05-15T23:30:03.531 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/154604068' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:30:03.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:03 smithi090 ceph-mon[44434]: osdmap e576: 12 total, 12 up, 9 in 2024-05-15T23:30:03.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/154604068' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:30:03.699 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 59 16 117386 117370 124.312 76.9375 0.019059 0.00804207 2024-05-15T23:30:04.179 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:30:04.179 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:30:04.179 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:30:04.179 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:30:04.179 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:30:04.524 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:30:04.524 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:30:04.524 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:30:04.524 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:30:04.524 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:30:04.524 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:30:04.524 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:30:04.524 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:30:04.524 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:30:04.524 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:30:04.524 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:30:04.525 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:30:04.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:30:04.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:30:04.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:30:04.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:30:04.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:30:04.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:30:04.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:30:04.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:30:04.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:30:04.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:30:04.526 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:30:04.526 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:30:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:04 smithi090 ceph-mon[44434]: pgmap v1538: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.0 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 79 MiB/s wr, 1.27k op/s; 3929/228446 objects misplaced (1.720%); 2.0 MiB/s, 32 objects/s recovering 2024-05-15T23:30:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:04 smithi088 ceph-mon[44699]: pgmap v1538: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.0 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 79 MiB/s wr, 1.27k op/s; 3929/228446 objects misplaced (1.720%); 2.0 MiB/s, 32 objects/s recovering 2024-05-15T23:30:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:04 smithi084 ceph-mon[39784]: pgmap v1538: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.0 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 79 MiB/s wr, 1.27k op/s; 3929/228446 objects misplaced (1.720%); 2.0 MiB/s, 32 objects/s recovering 2024-05-15T23:30:04.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:04.699 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:30:04.699296 min lat: 0.000985917 max lat: 2.57351 avg lat: 0.00808753 2024-05-15T23:30:04.699 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:30:04.699 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 60 16 118718 118702 123.628 83.25 0.0289708 0.00808753 2024-05-15T23:30:05.212 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:30:05.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:05.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:30:05.324 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:30:05.356 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:05.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:05.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:05.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:05.699 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 61 16 119931 119915 122.844 75.8125 0.0117582 0.00813678 2024-05-15T23:30:05.806 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:30:05.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:05.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:30:05.911 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:30:05.939 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:06.019 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:30:06.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:06.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:30:06.124 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:30:06.153 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:06.401 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:06 smithi088 ceph-mon[44699]: pgmap v1539: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.0 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 66 MiB/s wr, 1.06k op/s; 3929/229968 objects misplaced (1.708%); 1.6 MiB/s, 24 objects/s recovering 2024-05-15T23:30:06.402 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T23:30:06.404 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:30:06.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:06.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:30:06.508 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:30:06.536 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:06.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:06 smithi090 ceph-mon[44434]: pgmap v1539: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.0 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 66 MiB/s wr, 1.06k op/s; 3929/229968 objects misplaced (1.708%); 1.6 MiB/s, 24 objects/s recovering 2024-05-15T23:30:06.563 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:30:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:06.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:30:06.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:06 smithi084 ceph-mon[39784]: pgmap v1539: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.0 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 66 MiB/s wr, 1.06k op/s; 3929/229968 objects misplaced (1.708%); 1.6 MiB/s, 24 objects/s recovering 2024-05-15T23:30:06.668 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:30:06.697 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:06.699 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 62 16 121137 121121 122.078 75.375 0.0144696 0.00818929 2024-05-15T23:30:06.714 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:30:06.719 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:30:06.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:06.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:30:06.826 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:30:06.854 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:06.980 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:30:06.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:06.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:30:07.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:30:07.123 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:07.155 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:30:07.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:07.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:30:07.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:30:07.288 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:07.362 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:30:07.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:07.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:30:07.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:30:07.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:30:07.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:30:07.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:30:07.499 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:30:07.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:30:07.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:30:07.500 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:07 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:30:07.502 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:07.536 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:30:07.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:07.542 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:30:07.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:30:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:30:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:30:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:30:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:30:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:30:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:07 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:30:07.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:30:07.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:30:07.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:30:07.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:30:07.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:30:07.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:30:07.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:07 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:30:07.643 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:30:07.699 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 63 16 122307 122291 121.301 73.125 0.0291736 0.00824215 2024-05-15T23:30:07.719 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:08.313 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:30:08.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:08.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:30:08.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:30:08.449 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:08 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:30:08.450 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:08 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:30:08.450 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/807666269' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:30:08.487 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:08 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:30:08.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:08 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:30:08.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/807666269' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:30:08.488 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:08.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:08 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:30:08.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:08 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:30:08.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/807666269' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:30:08.700 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 64 16 123416 123400 120.489 69.3125 0.00670165 0.0082982 2024-05-15T23:30:08.986 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:30:08.986 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:30:08.986 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:30:08.986 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2024-05-15T23:30:08.986 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:30:09.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:30:09.167 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:30:09.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:09 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:30:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:09 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/807666269' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:30:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:09 smithi088 ceph-mon[44699]: osdmap e577: 12 total, 12 up, 9 in 2024-05-15T23:30:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:09 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2054584683' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.3968249104293252}]: dispatch 2024-05-15T23:30:09.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:09 smithi088 ceph-mon[44699]: from='client.36662 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.3968249104293252}]: dispatch 2024-05-15T23:30:09.345 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 0.396825 (6596) 2024-05-15T23:30:09.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:09 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:30:09.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:09 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/807666269' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:30:09.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:09 smithi084 ceph-mon[39784]: osdmap e577: 12 total, 12 up, 9 in 2024-05-15T23:30:09.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:09 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2054584683' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.3968249104293252}]: dispatch 2024-05-15T23:30:09.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:09 smithi084 ceph-mon[39784]: from='client.36662 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.3968249104293252}]: dispatch 2024-05-15T23:30:09.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:09 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:30:09.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:09 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/807666269' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:30:09.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:09 smithi090 ceph-mon[44434]: osdmap e577: 12 total, 12 up, 9 in 2024-05-15T23:30:09.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:09 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2054584683' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.3968249104293252}]: dispatch 2024-05-15T23:30:09.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:09 smithi090 ceph-mon[44434]: from='client.36662 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.3968249104293252}]: dispatch 2024-05-15T23:30:09.700 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 65 16 124199 124183 119.388 48.9375 0.00396742 0.00832261 2024-05-15T23:30:10.135 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T23:30:10.431 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:30:10.471 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:30:10.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:10.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:30:10.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:10 smithi090 ceph-mon[44434]: pgmap v1542: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.3 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 83 MiB/s wr, 1.32k op/s; 3734/240790 objects misplaced (1.551%); 3.1 MiB/s, 48 objects/s recovering 2024-05-15T23:30:10.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:10 smithi090 ceph-mon[44434]: from='client.36662 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 0.3968249104293252}]': finished 2024-05-15T23:30:10.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:10 smithi090 ceph-mon[44434]: osdmap e578: 12 total, 12 up, 9 in 2024-05-15T23:30:10.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:10 smithi090 ceph-mon[44434]: 7.c scrub starts 2024-05-15T23:30:10.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:10 smithi090 ceph-mon[44434]: osdmap e579: 12 total, 12 up, 9 in 2024-05-15T23:30:10.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:10 smithi088 ceph-mon[44699]: pgmap v1542: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.3 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 83 MiB/s wr, 1.32k op/s; 3734/240790 objects misplaced (1.551%); 3.1 MiB/s, 48 objects/s recovering 2024-05-15T23:30:10.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:10 smithi088 ceph-mon[44699]: from='client.36662 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 0.3968249104293252}]': finished 2024-05-15T23:30:10.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:10 smithi088 ceph-mon[44699]: osdmap e578: 12 total, 12 up, 9 in 2024-05-15T23:30:10.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:10 smithi088 ceph-mon[44699]: 7.c scrub starts 2024-05-15T23:30:10.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:10 smithi088 ceph-mon[44699]: osdmap e579: 12 total, 12 up, 9 in 2024-05-15T23:30:10.582 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:30:10.609 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:10.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:10 smithi084 ceph-mon[39784]: pgmap v1542: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.3 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 83 MiB/s wr, 1.32k op/s; 3734/240790 objects misplaced (1.551%); 3.1 MiB/s, 48 objects/s recovering 2024-05-15T23:30:10.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:10 smithi084 ceph-mon[39784]: from='client.36662 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 0.3968249104293252}]': finished 2024-05-15T23:30:10.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:10 smithi084 ceph-mon[39784]: osdmap e578: 12 total, 12 up, 9 in 2024-05-15T23:30:10.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:10 smithi084 ceph-mon[39784]: 7.c scrub starts 2024-05-15T23:30:10.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:10 smithi084 ceph-mon[39784]: osdmap e579: 12 total, 12 up, 9 in 2024-05-15T23:30:10.700 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 66 16 124341 124325 117.714 8.875 0.37257 0.00848061 2024-05-15T23:30:10.858 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:30:10.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:10.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:30:10.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:30:10.992 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:11.019 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:30:11.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:11.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:30:11.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:30:11.164 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:11.200 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:30:11.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:11.208 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:30:11.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:11 smithi090 ceph-mon[44434]: 7.6 scrub starts 2024-05-15T23:30:11.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:11 smithi090 ceph-mon[44434]: 7.c scrub ok 2024-05-15T23:30:11.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:11 smithi090 ceph-mon[44434]: 7.b scrub starts 2024-05-15T23:30:11.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:11 smithi090 ceph-mon[44434]: 7.3 scrub starts 2024-05-15T23:30:11.259 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:11 smithi090 ceph-mon[44434]: pgmap v1545: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 7.4 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 74 MiB/s wr, 1.18k op/s; 11390/244100 objects misplaced (4.666%); 2.0 MiB/s, 32 objects/s recovering 2024-05-15T23:30:11.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:30:11.339 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:11.363 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:30:11.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:11.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:30:11.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:30:11.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:11 smithi088 ceph-mon[44699]: 7.6 scrub starts 2024-05-15T23:30:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:11 smithi088 ceph-mon[44699]: 7.c scrub ok 2024-05-15T23:30:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:11 smithi088 ceph-mon[44699]: 7.b scrub starts 2024-05-15T23:30:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:11 smithi088 ceph-mon[44699]: 7.3 scrub starts 2024-05-15T23:30:11.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:11 smithi088 ceph-mon[44699]: pgmap v1545: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 7.4 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 74 MiB/s wr, 1.18k op/s; 11390/244100 objects misplaced (4.666%); 2.0 MiB/s, 32 objects/s recovering 2024-05-15T23:30:11.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:11 smithi084 ceph-mon[39784]: 7.6 scrub starts 2024-05-15T23:30:11.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:11 smithi084 ceph-mon[39784]: 7.c scrub ok 2024-05-15T23:30:11.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:11 smithi084 ceph-mon[39784]: 7.b scrub starts 2024-05-15T23:30:11.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:11 smithi084 ceph-mon[39784]: 7.3 scrub starts 2024-05-15T23:30:11.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:11 smithi084 ceph-mon[39784]: pgmap v1545: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 7.4 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 74 MiB/s wr, 1.18k op/s; 11390/244100 objects misplaced (4.666%); 2.0 MiB/s, 32 objects/s recovering 2024-05-15T23:30:11.626 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:11.700 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 67 16 125217 125201 116.774 54.75 0.0116633 0.00856139 2024-05-15T23:30:11.902 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:30:11.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:11.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:30:12.006 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:30:12.035 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:12.260 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:12 smithi084 ceph-mon[39784]: 7.6 scrub ok 2024-05-15T23:30:12.260 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:12 smithi084 ceph-mon[39784]: 7.b scrub ok 2024-05-15T23:30:12.260 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:12 smithi084 ceph-mon[39784]: osdmap e580: 12 total, 12 up, 9 in 2024-05-15T23:30:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:30:12.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:12.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:30:12.390 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:30:12.418 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:12.441 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:30:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:12.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:30:12.549 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:30:12.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:12 smithi090 ceph-mon[44434]: 7.6 scrub ok 2024-05-15T23:30:12.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:12 smithi090 ceph-mon[44434]: 7.b scrub ok 2024-05-15T23:30:12.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:12 smithi090 ceph-mon[44434]: osdmap e580: 12 total, 12 up, 9 in 2024-05-15T23:30:12.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:12 smithi088 ceph-mon[44699]: 7.6 scrub ok 2024-05-15T23:30:12.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:12 smithi088 ceph-mon[44699]: 7.b scrub ok 2024-05-15T23:30:12.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:12 smithi088 ceph-mon[44699]: osdmap e580: 12 total, 12 up, 9 in 2024-05-15T23:30:12.588 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:12.631 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:30:12.631 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:30:12.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.631 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:126130 7.6 7:6879cb9d:::benchmark_data_smithi090_143373_object126129:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e580)", 2024-05-15T23:30:12.631 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:30:12.624652+0000", 2024-05-15T23:30:12.631 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.005774609, 2024-05-15T23:30:12.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0058055299999999997, 2024-05-15T23:30:12.632 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:30:12.632 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:30:12.632 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:30:12.632 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:30:12.632 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:30:12.632 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:30:12.632 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 126130 2024-05-15T23:30:12.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.632 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:30:12.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:30:12.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.624652+0000", 2024-05-15T23:30:12.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:30:12.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:30:12.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.624652+0000", 2024-05-15T23:30:12.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:30:12.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:30:12.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.624654+0000", 2024-05-15T23:30:12.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0140000000000001e-06 2024-05-15T23:30:12.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:30:12.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.624680+0000", 2024-05-15T23:30:12.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6605e-05 2024-05-15T23:30:12.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:30:12.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.624690+0000", 2024-05-15T23:30:12.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0176999999999999e-05 2024-05-15T23:30:12.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:30:12.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.624697+0000", 2024-05-15T23:30:12.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4799999999999998e-06 2024-05-15T23:30:12.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:30:12.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.624738+0000", 2024-05-15T23:30:12.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0707999999999997e-05 2024-05-15T23:30:12.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:30:12.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.624854+0000", 2024-05-15T23:30:12.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000116476 2024-05-15T23:30:12.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:30:12.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.624957+0000", 2024-05-15T23:30:12.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000102828 2024-05-15T23:30:12.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.627494+0000", 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0025376589999999998 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.630394+0000", 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0028995840000000002 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.630453+0000", 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9116000000000001e-05 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:30:12.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.637 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.16764.0:126133 7.8 e580/549 7:11cf43f0:::benchmark_data_smithi090_143373_object126132:head v 580'8017, mlcod=580'8014)", 2024-05-15T23:30:12.637 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:30:12.628328+0000", 2024-05-15T23:30:12.637 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 0.0020977930000000001, 2024-05-15T23:30:12.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0022399109999999998, 2024-05-15T23:30:12.637 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:30:12.637 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:30:12.637 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "started", 2024-05-15T23:30:12.637 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:30:12.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:30:12.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.628328+0000", 2024-05-15T23:30:12.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:30:12.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:30:12.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.628328+0000", 2024-05-15T23:30:12.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:30:12.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:30:12.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.628329+0000", 2024-05-15T23:30:12.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0130000000000001e-06 2024-05-15T23:30:12.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:30:12.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.628392+0000", 2024-05-15T23:30:12.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2948999999999996e-05 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.628403+0000", 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0998e-05 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.628411+0000", 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.011e-06 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.628448+0000", 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7079999999999997e-05 2024-05-15T23:30:12.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:30:12.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:30:12.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:30:12.640 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:30:12.628462+0000", 2024-05-15T23:30:12.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3247e-05 2024-05-15T23:30:12.640 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:30:12.640 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:30:12.640 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:30:12.640 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:30:12.640 INFO:teuthology.orchestra.run.smithi088.stdout: ], 2024-05-15T23:30:12.640 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 2 2024-05-15T23:30:12.640 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:30:12.700 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 68 16 126192 126176 115.953 60.9375 0.0302745 0.00862131 2024-05-15T23:30:13.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:13 smithi088 ceph-mon[44699]: 7.3 scrub ok 2024-05-15T23:30:13.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:13 smithi088 ceph-mon[44699]: 7.5 scrub starts 2024-05-15T23:30:13.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:13 smithi088 ceph-mon[44699]: pgmap v1547: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 7.6 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 57 MiB/s wr, 916 op/s; 11237/249766 objects misplaced (4.499%); 2.0 MiB/s, 31 objects/s recovering 2024-05-15T23:30:13.419 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T23:30:13.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:13 smithi084 ceph-mon[39784]: 7.3 scrub ok 2024-05-15T23:30:13.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:13 smithi084 ceph-mon[39784]: 7.5 scrub starts 2024-05-15T23:30:13.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:13 smithi084 ceph-mon[39784]: pgmap v1547: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 7.6 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 57 MiB/s wr, 916 op/s; 11237/249766 objects misplaced (4.499%); 2.0 MiB/s, 31 objects/s recovering 2024-05-15T23:30:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:13 smithi090 ceph-mon[44434]: 7.3 scrub ok 2024-05-15T23:30:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:13 smithi090 ceph-mon[44434]: 7.5 scrub starts 2024-05-15T23:30:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:13 smithi090 ceph-mon[44434]: pgmap v1547: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 7.6 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 57 MiB/s wr, 916 op/s; 11237/249766 objects misplaced (4.499%); 2.0 MiB/s, 31 objects/s recovering 2024-05-15T23:30:13.700 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 69 16 127134 127118 115.126 58.875 0.0167782 0.00868313 2024-05-15T23:30:13.702 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:30:13.947 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:30:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:13.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:30:13.969 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:30:14.055 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:30:14.084 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:14.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:14 smithi090 ceph-mon[44434]: 7.5 scrub ok 2024-05-15T23:30:14.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:14 smithi088 ceph-mon[44699]: 7.5 scrub ok 2024-05-15T23:30:14.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:14 smithi084 ceph-mon[39784]: 7.5 scrub ok 2024-05-15T23:30:14.700 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 70 16 128122 128106 114.363 61.75 0.0211717 0.0087417 2024-05-15T23:30:15.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:30:15.565 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:30:15.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:15.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:30:15.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:30:15.700 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 71 15 129161 129146 113.668 65 0.012813 0.008795 2024-05-15T23:30:15.700 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:30:15.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:15.736 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:30:15.837 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:30:15.868 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:15.942 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:30:15.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:15.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:30:16.047 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:30:16.076 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:16.150 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:30:16.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:16.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:30:16.248 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:30:16.248 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:30:16.248 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:30:16.248 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:30:16.248 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:30:16.254 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:30:16.281 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:16 smithi090 ceph-mon[44434]: pgmap v1548: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 7.6 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 51 MiB/s wr, 815 op/s; 11237/250578 objects misplaced (4.484%); 1.6 MiB/s, 25 objects/s recovering 2024-05-15T23:30:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:16 smithi090 ceph-mon[44434]: 7.9 scrub starts 2024-05-15T23:30:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:16 smithi088 ceph-mon[44699]: pgmap v1548: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 7.6 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 51 MiB/s wr, 815 op/s; 11237/250578 objects misplaced (4.484%); 1.6 MiB/s, 25 objects/s recovering 2024-05-15T23:30:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:16 smithi088 ceph-mon[44699]: 7.9 scrub starts 2024-05-15T23:30:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:16 smithi084 ceph-mon[39784]: pgmap v1548: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 7.6 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 51 MiB/s wr, 815 op/s; 11237/250578 objects misplaced (4.484%); 1.6 MiB/s, 25 objects/s recovering 2024-05-15T23:30:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:16 smithi084 ceph-mon[39784]: 7.9 scrub starts 2024-05-15T23:30:16.700 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 72 16 130055 130039 112.864 55.8125 0.0197457 0.00885858 2024-05-15T23:30:16.910 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:30:17.030 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:30:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:17.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:30:17.048 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T23:30:17.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:30:17.184 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:17.225 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:30:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:17.232 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:30:17.333 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:30:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:17.389 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:30:17.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:17.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:30:17.401 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:30:17.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:30:17.541 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:17.573 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:30:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:17.580 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:30:17.682 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:30:17.700 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 73 16 130949 130933 112.083 55.875 0.0103255 0.00891923 2024-05-15T23:30:17.709 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:17.732 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:30:17.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:17.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:30:17.765 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:30:17.765 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:30:17.782 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:30:17.818 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:30:17.836 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:30:17.853 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:30:17.864 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:17.872 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:30:17.889 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:30:17.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:17.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:30:17.906 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:30:17.932 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:30:17.966 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:30:17.994 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:30:18.000 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:30:18.030 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:30:18.048 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:18.062 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:30:18.068 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:30:18.068 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:30:18.068 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:30:18.069 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:30:18.069 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:30:18.069 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:30:18.069 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:30:18.069 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:30:18.069 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:30:18.069 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:30:18.069 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:30:18.069 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:30:18.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:18 smithi084 ceph-mon[39784]: 7.0 scrub starts 2024-05-15T23:30:18.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:18 smithi084 ceph-mon[39784]: pgmap v1549: 17 pgs: 2 active+recovering, 15 active+clean; 7.9 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 60 MiB/s wr, 956 op/s; 10936/257386 objects misplaced (4.249%); 3.6 MiB/s, 57 objects/s recovering 2024-05-15T23:30:18.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:18 smithi084 ceph-mon[39784]: 7.9 scrub ok 2024-05-15T23:30:18.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:18 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2480276108' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:30:18.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:18.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:18 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:18.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:18 smithi090 ceph-mon[44434]: 7.0 scrub starts 2024-05-15T23:30:18.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:18 smithi090 ceph-mon[44434]: pgmap v1549: 17 pgs: 2 active+recovering, 15 active+clean; 7.9 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 60 MiB/s wr, 956 op/s; 10936/257386 objects misplaced (4.249%); 3.6 MiB/s, 57 objects/s recovering 2024-05-15T23:30:18.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:18 smithi090 ceph-mon[44434]: 7.9 scrub ok 2024-05-15T23:30:18.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:18 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2480276108' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:30:18.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:18.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:18 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:18.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:18 smithi088 ceph-mon[44699]: 7.0 scrub starts 2024-05-15T23:30:18.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:18 smithi088 ceph-mon[44699]: pgmap v1549: 17 pgs: 2 active+recovering, 15 active+clean; 7.9 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 60 MiB/s wr, 956 op/s; 10936/257386 objects misplaced (4.249%); 3.6 MiB/s, 57 objects/s recovering 2024-05-15T23:30:18.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:18 smithi088 ceph-mon[44699]: 7.9 scrub ok 2024-05-15T23:30:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:18 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2480276108' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:30:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:18.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:18 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:18.701 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 74 16 131827 131811 111.31 54.875 0.00686164 0.00898182 2024-05-15T23:30:18.727 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:30:18.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:18.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:30:18.837 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:30:18.866 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:18.957 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10, 2] out_osds: [0, 6, 8] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:30:18.958 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:30:18.958 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-05-15T23:30:18.958 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T23:30:18.958 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T23:30:18.958 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-05-15T23:30:19.065 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:30:19.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:19.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:30:19.172 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:30:19.200 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.200 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.200 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.201 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.201 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.201 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.201 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.201 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.201 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:19 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:30:19.201 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:19 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:30:19.201 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:19 smithi084 ceph-mon[39784]: 7.0 scrub ok 2024-05-15T23:30:19.201 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.201 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.201 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:19 smithi084 ceph-mon[39784]: osdmap e581: 12 total, 12 up, 9 in 2024-05-15T23:30:19.201 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:19 smithi084 ceph-mon[39784]: 7.e scrub starts 2024-05-15T23:30:19.202 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:19.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:19 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:30:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:19 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:30:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:19 smithi088 ceph-mon[44699]: 7.0 scrub ok 2024-05-15T23:30:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:19 smithi088 ceph-mon[44699]: osdmap e581: 12 total, 12 up, 9 in 2024-05-15T23:30:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:19 smithi088 ceph-mon[44699]: 7.e scrub starts 2024-05-15T23:30:19.404 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:30:19.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:19.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:30:19.509 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:30:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:19 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:30:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:19 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:30:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:19 smithi090 ceph-mon[44434]: 7.0 scrub ok 2024-05-15T23:30:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:19 smithi090 ceph-mon[44434]: osdmap e581: 12 total, 12 up, 9 in 2024-05-15T23:30:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:19 smithi090 ceph-mon[44434]: 7.e scrub starts 2024-05-15T23:30:19.566 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:19.671 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:30:19.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:19.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:30:19.704 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 75 16 132723 132707 110.568 56 0.00504267 0.00904195 2024-05-15T23:30:19.781 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:30:19.823 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:19.856 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:30:20.093 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:30:20.093 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:30:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:20 smithi088 ceph-mon[44699]: pgmap v1551: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 7.9 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 63 MiB/s wr, 1.01k op/s; 10936/260258 objects misplaced (4.202%); 3.5 MiB/s, 56 objects/s recovering 2024-05-15T23:30:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:20 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:30:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:20 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:30:20.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:20 smithi084 ceph-mon[39784]: pgmap v1551: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 7.9 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 63 MiB/s wr, 1.01k op/s; 10936/260258 objects misplaced (4.202%); 3.5 MiB/s, 56 objects/s recovering 2024-05-15T23:30:20.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:20 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:30:20.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:20 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:30:20.704 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 76 16 133660 133644 109.883 58.5625 0.011198 0.00909897 2024-05-15T23:30:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:20 smithi090 ceph-mon[44434]: pgmap v1551: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 7.9 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 63 MiB/s wr, 1.01k op/s; 10936/260258 objects misplaced (4.202%); 3.5 MiB/s, 56 objects/s recovering 2024-05-15T23:30:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:20 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:30:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:20 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:30:21.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T23:30:21.353 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:30:21.471 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:30:21.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:21 smithi090 ceph-mon[44434]: 7.e scrub ok 2024-05-15T23:30:21.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:21 smithi090 ceph-mon[44434]: osdmap e582: 12 total, 12 up, 9 in 2024-05-15T23:30:21.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:21 smithi090 ceph-mon[44434]: 7.8 scrub starts 2024-05-15T23:30:21.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:21 smithi090 ceph-mon[44434]: pgmap v1553: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 8.0 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 48 MiB/s wr, 766 op/s; 10836/262032 objects misplaced (4.135%); 3.1 MiB/s, 50 objects/s recovering 2024-05-15T23:30:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:21 smithi084 ceph-mon[39784]: 7.e scrub ok 2024-05-15T23:30:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:21 smithi084 ceph-mon[39784]: osdmap e582: 12 total, 12 up, 9 in 2024-05-15T23:30:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:21 smithi084 ceph-mon[39784]: 7.8 scrub starts 2024-05-15T23:30:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:21 smithi084 ceph-mon[39784]: pgmap v1553: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 8.0 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 48 MiB/s wr, 766 op/s; 10836/262032 objects misplaced (4.135%); 3.1 MiB/s, 50 objects/s recovering 2024-05-15T23:30:21.704 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 77 16 134523 134507 109.157 53.9375 0.0244183 0.00915905 2024-05-15T23:30:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:21 smithi088 ceph-mon[44699]: 7.e scrub ok 2024-05-15T23:30:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:21 smithi088 ceph-mon[44699]: osdmap e582: 12 total, 12 up, 9 in 2024-05-15T23:30:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:21 smithi088 ceph-mon[44699]: 7.8 scrub starts 2024-05-15T23:30:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:21 smithi088 ceph-mon[44699]: pgmap v1553: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 8.0 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 48 MiB/s wr, 766 op/s; 10836/262032 objects misplaced (4.135%); 3.1 MiB/s, 50 objects/s recovering 2024-05-15T23:30:22.358 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:30:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:22.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:30:22.462 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:30:22.469 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:30:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:22.571 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:30:22.571 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "" 2024-05-15T23:30:22.571 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:30:22.704 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 78 16 135432 135416 108.486 56.8125 0.00775607 0.0092156 2024-05-15T23:30:23.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:23 smithi084 ceph-mon[39784]: 7.8 scrub ok 2024-05-15T23:30:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:30:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:30:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:23 smithi084 ceph-mon[39784]: osdmap e583: 12 total, 12 up, 9 in 2024-05-15T23:30:23.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:23 smithi084 ceph-mon[39784]: pgmap v1555: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 8.2 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 56 MiB/s wr, 895 op/s; 10625/268130 objects misplaced (3.963%); 3.2 MiB/s, 51 objects/s recovering 2024-05-15T23:30:23.583 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:30:23.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:23.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:30:23.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:30:23.704 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 79 16 136340 136324 107.831 56.75 0.00690697 0.00927154 2024-05-15T23:30:23.727 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:23 smithi090 ceph-mon[44434]: 7.8 scrub ok 2024-05-15T23:30:23.727 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:30:23.727 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:30:23.727 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:23 smithi090 ceph-mon[44434]: osdmap e583: 12 total, 12 up, 9 in 2024-05-15T23:30:23.727 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:23 smithi090 ceph-mon[44434]: pgmap v1555: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 8.2 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 56 MiB/s wr, 895 op/s; 10625/268130 objects misplaced (3.963%); 3.2 MiB/s, 51 objects/s recovering 2024-05-15T23:30:23.728 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:23.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:23 smithi088 ceph-mon[44699]: 7.8 scrub ok 2024-05-15T23:30:23.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:30:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:30:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:23 smithi088 ceph-mon[44699]: osdmap e583: 12 total, 12 up, 9 in 2024-05-15T23:30:23.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:23 smithi088 ceph-mon[44699]: pgmap v1555: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 8.2 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 56 MiB/s wr, 895 op/s; 10625/268130 objects misplaced (3.963%); 3.2 MiB/s, 51 objects/s recovering 2024-05-15T23:30:23.886 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:30:23.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:23.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:30:23.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:30:24.026 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:24.184 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:30:24.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:24.192 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:30:24.245 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:30:24.245 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:30:24.293 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:30:24.323 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:24.345 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:30:24.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:24.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:30:24.390 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:30:24.449 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:30:24.482 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:24.521 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:30:24.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:24.525 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:30:24.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:24 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1485912198' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:30:24.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:24 smithi088 ceph-mon[44699]: from='client.36671 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:30:24.626 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:30:24.653 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:24.673 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:30:24.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:24.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:30:24.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:24 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1485912198' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:30:24.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:24 smithi090 ceph-mon[44434]: from='client.36671 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:30:24.705 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:30:24.704511 min lat: 0.000985917 max lat: 2.57351 avg lat: 0.00932842 2024-05-15T23:30:24.705 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:30:24.705 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 80 16 137217 137201 107.168 54.8125 0.0143278 0.00932842 2024-05-15T23:30:24.777 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:30:24.811 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:24 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1485912198' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:30:24.812 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:24 smithi084 ceph-mon[39784]: from='client.36671 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:30:24.816 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:25.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T23:30:25.499 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:30:25.708 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 81 16 138140 138124 106.553 57.6875 0.0135735 0.00938406 2024-05-15T23:30:25.785 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:25 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:30:25.785 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:25 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:30:25.785 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:25 smithi090 ceph-mon[44434]: from='client.36671 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:30:25.785 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:25 smithi090 ceph-mon[44434]: osdmap e584: 12 total, 12 up, 9 in 2024-05-15T23:30:25.785 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:25 smithi090 ceph-mon[44434]: pgmap v1557: 17 pgs: 2 active+recovering, 15 active+clean; 8.2 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 47 MiB/s wr, 750 op/s; 10625/269266 objects misplaced (3.946%); 3.2 MiB/s, 51 objects/s recovering 2024-05-15T23:30:25.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:25 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:30:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:25 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:30:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:25 smithi088 ceph-mon[44699]: from='client.36671 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:30:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:25 smithi088 ceph-mon[44699]: osdmap e584: 12 total, 12 up, 9 in 2024-05-15T23:30:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:25 smithi088 ceph-mon[44699]: pgmap v1557: 17 pgs: 2 active+recovering, 15 active+clean; 8.2 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 47 MiB/s wr, 750 op/s; 10625/269266 objects misplaced (3.946%); 3.2 MiB/s, 51 objects/s recovering 2024-05-15T23:30:25.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:25 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:30:25.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:25 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:30:25.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:25 smithi084 ceph-mon[39784]: from='client.36671 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:30:25.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:25 smithi084 ceph-mon[39784]: osdmap e584: 12 total, 12 up, 9 in 2024-05-15T23:30:25.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:25 smithi084 ceph-mon[39784]: pgmap v1557: 17 pgs: 2 active+recovering, 15 active+clean; 8.2 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 47 MiB/s wr, 750 op/s; 10625/269266 objects misplaced (3.946%); 3.2 MiB/s, 51 objects/s recovering 2024-05-15T23:30:26.486 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:30:26.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:26.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:30:26.593 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:30:26.648 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:26.708 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 82 16 139067 139051 105.96 57.9375 0.00955257 0.00943481 2024-05-15T23:30:26.807 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:30:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:26.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:30:26.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:30:26.992 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:27.110 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:30:27.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:27.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:30:27.179 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:30:27.216 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:30:27.223 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:30:27.246 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:27.253 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:30:27.281 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:30:27.304 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:30:27.324 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:30:27.350 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:30:27.385 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:30:27.418 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:30:27.451 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:30:27.488 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:30:27.528 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:30:27.545 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:30:27.545 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:30:27.545 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:30:27.546 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:30:27.546 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:30:27.546 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:30:27.546 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:30:27.546 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:30:27.546 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:30:27.546 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:30:27.546 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:30:27.546 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:30:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:30:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:30:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:27 smithi090 ceph-mon[44434]: osdmap e585: 12 total, 12 up, 9 in 2024-05-15T23:30:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:27 smithi090 ceph-mon[44434]: pgmap v1559: 17 pgs: 2 active+recovering, 15 active+clean; 8.4 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 70 MiB/s wr, 1.12k op/s; 10306/275460 objects misplaced (3.741%); 5.5 MiB/s, 88 objects/s recovering 2024-05-15T23:30:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:27 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3697376466' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:30:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:27.708 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 83 16 139910 139894 105.318 52.6875 0.0214061 0.00949255 2024-05-15T23:30:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:30:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:30:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:27 smithi088 ceph-mon[44699]: osdmap e585: 12 total, 12 up, 9 in 2024-05-15T23:30:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:27 smithi088 ceph-mon[44699]: pgmap v1559: 17 pgs: 2 active+recovering, 15 active+clean; 8.4 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 70 MiB/s wr, 1.12k op/s; 10306/275460 objects misplaced (3.741%); 5.5 MiB/s, 88 objects/s recovering 2024-05-15T23:30:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:27 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3697376466' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:30:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:27.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:30:27.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:30:27.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:27 smithi084 ceph-mon[39784]: osdmap e585: 12 total, 12 up, 9 in 2024-05-15T23:30:27.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:27 smithi084 ceph-mon[39784]: pgmap v1559: 17 pgs: 2 active+recovering, 15 active+clean; 8.4 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 70 MiB/s wr, 1.12k op/s; 10306/275460 objects misplaced (3.741%); 5.5 MiB/s, 88 objects/s recovering 2024-05-15T23:30:27.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:27 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3697376466' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:30:27.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:27.961 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:30:28.421 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10, 2] out_osds: [0, 6, 8] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:30:28.421 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:30:28.421 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-05-15T23:30:28.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump -f json-pretty 2024-05-15T23:30:28.708 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 84 16 140845 140829 104.76 58.4375 0.00726204 0.00954402 2024-05-15T23:30:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:28.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:29.672 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:30:29.680 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:30:29.708 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 85 16 141756 141740 104.197 56.9375 0.0350283 0.00959515 2024-05-15T23:30:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:29 smithi090 ceph-mon[44434]: pgmap v1560: 17 pgs: 2 active+recovering, 15 active+clean; 8.5 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 49 MiB/s wr, 791 op/s; 10306/278278 objects misplaced (3.703%); 2.9 MiB/s, 47 objects/s recovering 2024-05-15T23:30:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:29 smithi088 ceph-mon[44699]: pgmap v1560: 17 pgs: 2 active+recovering, 15 active+clean; 8.5 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 49 MiB/s wr, 791 op/s; 10306/278278 objects misplaced (3.703%); 2.9 MiB/s, 47 objects/s recovering 2024-05-15T23:30:29.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:29 smithi084 ceph-mon[39784]: pgmap v1560: 17 pgs: 2 active+recovering, 15 active+clean; 8.5 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 49 MiB/s wr, 791 op/s; 10306/278278 objects misplaced (3.703%); 2.9 MiB/s, 47 objects/s recovering 2024-05-15T23:30:30.358 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:30:30.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:30.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:30:30.465 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:30:30.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:30 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:30:30.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:30 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:30:30.506 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:30.709 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 86 16 142674 142658 103.652 57.375 0.0101643 0.00964569 2024-05-15T23:30:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:30 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:30:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:30 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:30:30.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:30 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:30:30.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:30 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:30:31.529 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:30:31.711 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 87 16 143588 143572 103.115 57.125 0.0181529 0.00969557 2024-05-15T23:30:31.718 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:31 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:30:31.719 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:31 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:30:31.719 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:31 smithi090 ceph-mon[44434]: pgmap v1561: 17 pgs: 2 active+recovering, 15 active+clean; 8.5 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 46 MiB/s wr, 740 op/s; 10198/279976 objects misplaced (3.642%); 3.3 MiB/s, 53 objects/s recovering 2024-05-15T23:30:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:31 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:30:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:31 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:30:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:31 smithi088 ceph-mon[44699]: pgmap v1561: 17 pgs: 2 active+recovering, 15 active+clean; 8.5 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 46 MiB/s wr, 740 op/s; 10198/279976 objects misplaced (3.642%); 3.3 MiB/s, 53 objects/s recovering 2024-05-15T23:30:31.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:31 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:30:31.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:31 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:30:31.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:31 smithi084 ceph-mon[39784]: pgmap v1561: 17 pgs: 2 active+recovering, 15 active+clean; 8.5 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 46 MiB/s wr, 740 op/s; 10198/279976 objects misplaced (3.642%); 3.3 MiB/s, 53 objects/s recovering 2024-05-15T23:30:32.273 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:30:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:32.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:30:32.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:30:32.422 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:32.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:32 smithi084 ceph-mon[39784]: osdmap e586: 12 total, 12 up, 9 in 2024-05-15T23:30:32.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:32 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:30:32.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:32 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:30:32.711 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 88 16 144350 144334 102.485 47.625 0.0135804 0.0097549 2024-05-15T23:30:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:32 smithi090 ceph-mon[44434]: osdmap e586: 12 total, 12 up, 9 in 2024-05-15T23:30:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:32 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:30:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:32 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:30:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:32 smithi088 ceph-mon[44699]: osdmap e586: 12 total, 12 up, 9 in 2024-05-15T23:30:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:32 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:30:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:32 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:30:33.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:33 smithi084 ceph-mon[39784]: osdmap e587: 12 total, 12 up, 9 in 2024-05-15T23:30:33.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:33 smithi084 ceph-mon[39784]: pgmap v1564: 17 pgs: 2 active+recovering, 15 active+clean; 8.7 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 49 MiB/s wr, 790 op/s; 9971/286118 objects misplaced (3.485%); 3.1 MiB/s, 49 objects/s recovering 2024-05-15T23:30:33.711 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 89 16 145234 145218 101.954 55.25 0.0173442 0.00980705 2024-05-15T23:30:33.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:33 smithi090 ceph-mon[44434]: osdmap e587: 12 total, 12 up, 9 in 2024-05-15T23:30:33.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:33 smithi090 ceph-mon[44434]: pgmap v1564: 17 pgs: 2 active+recovering, 15 active+clean; 8.7 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 49 MiB/s wr, 790 op/s; 9971/286118 objects misplaced (3.485%); 3.1 MiB/s, 49 objects/s recovering 2024-05-15T23:30:33.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:33 smithi088 ceph-mon[44699]: osdmap e587: 12 total, 12 up, 9 in 2024-05-15T23:30:33.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:33 smithi088 ceph-mon[44699]: pgmap v1564: 17 pgs: 2 active+recovering, 15 active+clean; 8.7 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 49 MiB/s wr, 790 op/s; 9971/286118 objects misplaced (3.485%); 3.1 MiB/s, 49 objects/s recovering 2024-05-15T23:30:34.021 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:30:34.711 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total time run: 90.0147 2024-05-15T23:30:34.711 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total writes made: 146168 2024-05-15T23:30:34.711 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Write size: 65536 2024-05-15T23:30:34.711 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object size: 65536 2024-05-15T23:30:34.711 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Bandwidth (MB/sec): 101.489 2024-05-15T23:30:34.712 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Bandwidth: 62.0371 2024-05-15T23:30:34.712 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max bandwidth (MB/sec): 337.688 2024-05-15T23:30:34.712 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min bandwidth (MB/sec): 0 2024-05-15T23:30:34.712 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average IOPS: 1623 2024-05-15T23:30:34.712 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev IOPS: 992.593 2024-05-15T23:30:34.712 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max IOPS: 5403 2024-05-15T23:30:34.712 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min IOPS: 0 2024-05-15T23:30:34.712 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average Latency(s): 0.00985269 2024-05-15T23:30:34.712 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Latency(s): 0.0305186 2024-05-15T23:30:34.712 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max latency(s): 2.57351 2024-05-15T23:30:34.712 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min latency(s): 0.000985917 2024-05-15T23:30:34.718 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T23:30:35.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:35 smithi090 ceph-mon[44434]: pgmap v1565: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 46 MiB/s wr, 732 op/s; 9971/287190 objects misplaced (3.472%); 2.6 MiB/s, 41 objects/s recovering 2024-05-15T23:30:35.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:35 smithi088 ceph-mon[44699]: pgmap v1565: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 46 MiB/s wr, 732 op/s; 9971/287190 objects misplaced (3.472%); 2.6 MiB/s, 41 objects/s recovering 2024-05-15T23:30:35.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:35 smithi084 ceph-mon[39784]: pgmap v1565: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 46 MiB/s wr, 732 op/s; 9971/287190 objects misplaced (3.472%); 2.6 MiB/s, 41 objects/s recovering 2024-05-15T23:30:36.080 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:30:36.080 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:30:36.080 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 587, 2024-05-15T23:30:36.080 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "3999fb7e-130e-11ef-bc99-c7b262605968", 2024-05-15T23:30:36.080 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-05-15T22:56:41.212493+0000", 2024-05-15T23:30:36.081 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-05-15T23:30:32.440220+0000", 2024-05-15T23:30:36.081 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-05-15T23:26:58.424794+0000", 2024-05-15T23:30:36.081 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-05-15T23:28:16.134275+0000", 2024-05-15T23:30:36.081 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-15T23:30:36.081 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5799936, 2024-05-15T23:30:36.081 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-05-15T23:30:36.081 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-05-15T23:30:36.081 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-05-15T23:30:36.081 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-05-15T23:30:36.081 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-05-15T23:30:36.081 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:30:36.081 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 30, 2024-05-15T23:30:36.082 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-05-15T23:30:36.082 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-15T23:30:36.082 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-15T23:30:36.082 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-05-15T23:30:36.082 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 7, 2024-05-15T23:30:36.082 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 12, 2024-05-15T23:30:36.082 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-05-15T23:30:36.082 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-05-15T23:30:36.082 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-05-15T23:30:36.082 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-05-15T23:30:36.082 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-05-15T23:30:36.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.082 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-05-15T23:30:36.083 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-05-15T23:30:36.083 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-05-15T22:59:38.603575+0000", 2024-05-15T23:30:36.083 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-05-15T23:30:36.083 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-05-15T23:30:36.083 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-05-15T23:30:36.083 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2024-05-15T23:30:36.083 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-05-15T23:30:36.083 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-05-15T23:30:36.083 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-05-15T23:30:36.083 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-05-15T23:30:36.083 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T23:30:36.083 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T23:30:36.083 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-05-15T23:30:36.084 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-05-15T23:30:36.084 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 1, 2024-05-15T23:30:36.084 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 1, 2024-05-15T23:30:36.084 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 1, 2024-05-15T23:30:36.084 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 1, 2024-05-15T23:30:36.084 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 1, 2024-05-15T23:30:36.084 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-05-15T23:30:36.084 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-05-15T23:30:36.084 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-05-15T23:30:36.084 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-05-15T23:30:36.084 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-05-15T23:30:36.084 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-05-15T23:30:36.085 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-05-15T23:30:36.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.085 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "16", 2024-05-15T23:30:36.085 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-05-15T23:30:36.085 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T23:30:36.085 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T23:30:36.085 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-05-15T23:30:36.085 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-05-15T23:30:36.085 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-05-15T23:30:36.085 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-05-15T23:30:36.085 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-05-15T23:30:36.085 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-05-15T23:30:36.085 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-05-15T23:30:36.085 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-05-15T23:30:36.086 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-05-15T23:30:36.086 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-05-15T23:30:36.086 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-05-15T23:30:36.086 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-05-15T23:30:36.086 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-05-15T23:30:36.086 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-05-15T23:30:36.086 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-05-15T23:30:36.086 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T23:30:36.086 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T23:30:36.086 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T23:30:36.086 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-05-15T23:30:36.086 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-05-15T23:30:36.086 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-05-15T23:30:36.087 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-05-15T23:30:36.087 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-05-15T23:30:36.087 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.087 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-05-15T23:30:36.087 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-05-15T23:30:36.087 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-05-15T23:30:36.087 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-05-15T23:30:36.087 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-05-15T23:30:36.087 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T23:30:36.087 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-05-15T23:30:36.087 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-05-15T23:30:36.087 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-05-15T23:30:36.088 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-05-15T23:30:36.088 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-05-15T23:30:36.088 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-05-15T23:30:36.088 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-05-15T23:30:36.088 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-05-15T23:30:36.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.088 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-05-15T23:30:36.088 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-05-15T23:30:36.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.088 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-05-15T23:30:36.088 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-05-15T23:30:36.088 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 11.760000228881836, 2024-05-15T23:30:36.088 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 11.760000228881836, 2024-05-15T23:30:36.088 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.17000000178813934, 2024-05-15T23:30:36.089 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-05-15T23:30:36.089 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-05-15T23:30:36.089 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.96207165718078613, 2024-05-15T23:30:36.089 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 0.95999997854232788, 2024-05-15T23:30:36.089 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.95999997854232788 2024-05-15T23:30:36.089 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.089 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-05-15T23:30:36.089 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_5", 2024-05-15T23:30:36.089 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-05-15T23:28:50.243463+0000", 2024-05-15T23:30:36.089 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-05-15T23:30:36.089 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-05-15T23:30:36.089 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-05-15T23:30:36.090 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2024-05-15T23:30:36.090 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-05-15T23:30:36.090 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-05-15T23:30:36.090 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-05-15T23:30:36.090 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-05-15T23:30:36.090 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T23:30:36.090 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T23:30:36.090 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-05-15T23:30:36.090 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-05-15T23:30:36.090 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 16, 2024-05-15T23:30:36.090 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 16, 2024-05-15T23:30:36.090 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 16, 2024-05-15T23:30:36.090 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 16, 2024-05-15T23:30:36.091 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 16, 2024-05-15T23:30:36.091 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-05-15T23:30:36.091 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-05-15T23:30:36.091 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-05-15T23:30:36.091 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-05-15T23:30:36.091 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-05-15T23:30:36.091 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-05-15T23:30:36.091 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-05-15T23:30:36.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.091 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "554", 2024-05-15T23:30:36.091 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-05-15T23:30:36.091 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T23:30:36.091 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T23:30:36.091 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-05-15T23:30:36.092 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-05-15T23:30:36.092 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-05-15T23:30:36.092 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-05-15T23:30:36.092 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-05-15T23:30:36.092 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-05-15T23:30:36.092 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-05-15T23:30:36.092 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-05-15T23:30:36.092 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-05-15T23:30:36.092 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-05-15T23:30:36.092 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-05-15T23:30:36.092 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-05-15T23:30:36.092 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-05-15T23:30:36.093 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-05-15T23:30:36.093 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-05-15T23:30:36.093 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T23:30:36.093 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T23:30:36.093 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T23:30:36.093 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-05-15T23:30:36.093 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-05-15T23:30:36.093 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-05-15T23:30:36.093 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-05-15T23:30:36.093 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-05-15T23:30:36.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.093 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-05-15T23:30:36.093 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-05-15T23:30:36.093 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-05-15T23:30:36.093 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-05-15T23:30:36.094 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-05-15T23:30:36.094 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T23:30:36.094 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-05-15T23:30:36.094 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-05-15T23:30:36.094 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-05-15T23:30:36.094 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-05-15T23:30:36.094 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-05-15T23:30:36.094 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-05-15T23:30:36.094 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-05-15T23:30:36.094 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-05-15T23:30:36.094 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.094 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-05-15T23:30:36.094 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-05-15T23:30:36.095 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.9900000095367432, 2024-05-15T23:30:36.095 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.9900000095367432, 2024-05-15T23:30:36.095 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.67000001668930054, 2024-05-15T23:30:36.095 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-05-15T23:30:36.095 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-05-15T23:30:36.095 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.96207165718078613, 2024-05-15T23:30:36.095 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 0.95999997854232788, 2024-05-15T23:30:36.095 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.95999997854232788 2024-05-15T23:30:36.095 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.095 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.095 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:30:36.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-05-15T23:30:36.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-05-15T23:30:36.096 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "80159cc8-1b6c-4aa6-a114-966786d7be1e", 2024-05-15T23:30:36.096 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:30:36.096 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-05-15T23:30:36.096 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-05-15T23:30:36.096 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:30:36.096 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 8, 2024-05-15T23:30:36.096 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 355, 2024-05-15T23:30:36.096 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 397, 2024-05-15T23:30:36.096 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 283, 2024-05-15T23:30:36.096 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 356, 2024-05-15T23:30:36.096 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:30:36.096 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:30:36.097 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.097 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.097 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-05-15T23:30:36.097 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:30:36.097 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.097 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.097 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:30:36.097 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.097 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.097 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-05-15T23:30:36.097 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:30:36.097 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.098 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.098 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:30:36.098 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.098 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.098 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-05-15T23:30:36.098 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:30:36.098 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.098 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.099 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:30:36.099 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.099 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.099 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-05-15T23:30:36.099 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:30:36.099 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.099 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.099 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/3077815271", 2024-05-15T23:30:36.099 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/3077815271", 2024-05-15T23:30:36.099 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/3077815271", 2024-05-15T23:30:36.099 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/3077815271", 2024-05-15T23:30:36.099 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:30:36.100 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:30:36.100 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:30:36.100 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.100 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-05-15T23:30:36.100 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "9c2c76c9-ce0f-4c2d-9d96-c87a10187607", 2024-05-15T23:30:36.100 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:30:36.100 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:30:36.100 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:30:36.100 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:30:36.100 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:30:36.100 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:30:36.100 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-05-15T23:30:36.100 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 578, 2024-05-15T23:30:36.101 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:30:36.101 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:30:36.101 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:30:36.101 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.101 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.101 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-05-15T23:30:36.101 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:30:36.101 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.101 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.101 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:30:36.101 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.102 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.102 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-05-15T23:30:36.102 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:30:36.102 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.102 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.102 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:30:36.102 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.102 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.102 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-05-15T23:30:36.102 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:30:36.103 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.103 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.103 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:30:36.103 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.103 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.103 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-05-15T23:30:36.103 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:30:36.103 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.103 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.103 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/2140830229", 2024-05-15T23:30:36.103 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/2140830229", 2024-05-15T23:30:36.104 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/2140830229", 2024-05-15T23:30:36.104 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/2140830229", 2024-05-15T23:30:36.104 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:30:36.104 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:30:36.104 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:30:36.104 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.104 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-05-15T23:30:36.104 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "e8514daa-b5f9-4f2f-a432-53f4a4eb86bc", 2024-05-15T23:30:36.104 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:30:36.104 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:30:36.104 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:30:36.104 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:30:36.105 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 20, 2024-05-15T23:30:36.105 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 261, 2024-05-15T23:30:36.105 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 291, 2024-05-15T23:30:36.105 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:30:36.105 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 262, 2024-05-15T23:30:36.105 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:30:36.105 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:30:36.105 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.105 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.105 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-05-15T23:30:36.105 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:30:36.105 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.105 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.106 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:30:36.106 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.106 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.106 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-05-15T23:30:36.106 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:30:36.106 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.106 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.106 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:30:36.106 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.106 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.107 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-05-15T23:30:36.107 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:30:36.107 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.107 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.107 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:30:36.107 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.107 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.107 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-05-15T23:30:36.107 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:30:36.107 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.107 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.108 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/941507493", 2024-05-15T23:30:36.108 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/941507493", 2024-05-15T23:30:36.108 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/941507493", 2024-05-15T23:30:36.108 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/941507493", 2024-05-15T23:30:36.108 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:30:36.108 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:30:36.108 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:30:36.108 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.108 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-05-15T23:30:36.108 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "5d9965f9-c0b2-48ce-869c-582990985623", 2024-05-15T23:30:36.108 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:30:36.108 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:30:36.108 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:30:36.109 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:30:36.109 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:30:36.109 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:30:36.109 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 26, 2024-05-15T23:30:36.109 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:30:36.109 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:30:36.109 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:30:36.109 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:30:36.109 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.109 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.109 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-05-15T23:30:36.109 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:30:36.110 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.110 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.110 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:30:36.110 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.110 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.110 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-05-15T23:30:36.110 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:30:36.110 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.110 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.110 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:30:36.111 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.111 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.111 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-05-15T23:30:36.111 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:30:36.111 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.111 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.111 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:30:36.111 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.111 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.111 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-05-15T23:30:36.111 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:30:36.111 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.112 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.112 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2204039828", 2024-05-15T23:30:36.112 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2204039828", 2024-05-15T23:30:36.112 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2204039828", 2024-05-15T23:30:36.112 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2204039828", 2024-05-15T23:30:36.112 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:30:36.112 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:30:36.112 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:30:36.112 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.112 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-05-15T23:30:36.112 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4bc50523-8480-44d2-8d76-31479e839c85", 2024-05-15T23:30:36.113 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:30:36.113 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:30:36.113 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0.396820068359375, 2024-05-15T23:30:36.113 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:30:36.113 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:30:36.113 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:30:36.113 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-05-15T23:30:36.113 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:30:36.113 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:30:36.113 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:30:36.113 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:30:36.113 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.114 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.114 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6800", 2024-05-15T23:30:36.114 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:30:36.114 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.114 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.114 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:30:36.114 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.114 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.114 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6801", 2024-05-15T23:30:36.114 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:30:36.114 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.115 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.115 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:30:36.115 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.115 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.115 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6803", 2024-05-15T23:30:36.115 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:30:36.115 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.115 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.115 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:30:36.115 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.116 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.116 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6802", 2024-05-15T23:30:36.116 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:30:36.116 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.116 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.116 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6800/2267955292", 2024-05-15T23:30:36.116 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6801/2267955292", 2024-05-15T23:30:36.116 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6803/2267955292", 2024-05-15T23:30:36.116 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6802/2267955292", 2024-05-15T23:30:36.116 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:30:36.116 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:30:36.116 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:30:36.117 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.117 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-05-15T23:30:36.117 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20", 2024-05-15T23:30:36.117 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:30:36.117 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:30:36.117 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:30:36.117 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:30:36.117 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:30:36.117 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:30:36.117 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 37, 2024-05-15T23:30:36.117 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:30:36.118 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:30:36.118 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:30:36.118 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:30:36.118 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.118 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.118 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6804", 2024-05-15T23:30:36.118 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:30:36.118 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.118 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.118 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:30:36.118 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.119 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.119 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6805", 2024-05-15T23:30:36.119 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:30:36.119 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.119 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.119 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:30:36.119 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.119 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.119 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6807", 2024-05-15T23:30:36.119 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:30:36.119 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.119 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.120 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:30:36.120 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.120 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.120 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6806", 2024-05-15T23:30:36.120 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:30:36.120 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.120 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.120 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6804/661041538", 2024-05-15T23:30:36.120 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6805/661041538", 2024-05-15T23:30:36.120 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6807/661041538", 2024-05-15T23:30:36.120 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6806/661041538", 2024-05-15T23:30:36.121 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:30:36.121 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:30:36.121 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:30:36.121 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.121 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-05-15T23:30:36.121 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f", 2024-05-15T23:30:36.121 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:30:36.121 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-05-15T23:30:36.121 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-05-15T23:30:36.121 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:30:36.121 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:30:36.121 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:30:36.122 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 46, 2024-05-15T23:30:36.122 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 374, 2024-05-15T23:30:36.122 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:30:36.122 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:30:36.122 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:30:36.122 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.122 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.122 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6808", 2024-05-15T23:30:36.122 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:30:36.122 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.122 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.123 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:30:36.123 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.123 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.123 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6809", 2024-05-15T23:30:36.123 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:30:36.123 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.123 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.123 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:30:36.123 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.123 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.124 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6811", 2024-05-15T23:30:36.124 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:30:36.124 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.124 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.124 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:30:36.124 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.124 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.124 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6810", 2024-05-15T23:30:36.124 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:30:36.124 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.124 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.125 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6808/380856869", 2024-05-15T23:30:36.125 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6809/380856869", 2024-05-15T23:30:36.125 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6811/380856869", 2024-05-15T23:30:36.125 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6810/380856869", 2024-05-15T23:30:36.125 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:30:36.125 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:30:36.125 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:30:36.125 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.125 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-05-15T23:30:36.125 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34", 2024-05-15T23:30:36.125 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:30:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:30:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:30:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:30:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:30:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:30:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 53, 2024-05-15T23:30:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:30:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:30:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:30:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:30:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6812", 2024-05-15T23:30:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:30:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:30:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6813", 2024-05-15T23:30:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:30:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:30:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6815", 2024-05-15T23:30:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:30:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:30:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6814", 2024-05-15T23:30:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:30:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6812/1044709063", 2024-05-15T23:30:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6813/1044709063", 2024-05-15T23:30:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6815/1044709063", 2024-05-15T23:30:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6814/1044709063", 2024-05-15T23:30:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:30:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:30:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:30:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-05-15T23:30:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "02cab286-d487-4a66-982c-3cf28f55e851", 2024-05-15T23:30:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:30:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-05-15T23:30:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-05-15T23:30:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 0.54486083984375, 2024-05-15T23:30:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 61, 2024-05-15T23:30:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 484, 2024-05-15T23:30:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 520, 2024-05-15T23:30:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 202, 2024-05-15T23:30:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 485, 2024-05-15T23:30:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:30:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:30:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6800", 2024-05-15T23:30:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:30:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:30:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6801", 2024-05-15T23:30:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:30:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:30:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6803", 2024-05-15T23:30:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:30:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:30:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6802", 2024-05-15T23:30:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:30:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6800/703598286", 2024-05-15T23:30:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6801/703598286", 2024-05-15T23:30:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6803/703598286", 2024-05-15T23:30:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6802/703598286", 2024-05-15T23:30:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:30:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:30:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:30:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-05-15T23:30:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613", 2024-05-15T23:30:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:30:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:30:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:30:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:30:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:30:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:30:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 68, 2024-05-15T23:30:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 565, 2024-05-15T23:30:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:30:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:30:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:30:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6804", 2024-05-15T23:30:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:30:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:30:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6805", 2024-05-15T23:30:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:30:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:30:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6807", 2024-05-15T23:30:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:30:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:30:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6806", 2024-05-15T23:30:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:30:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6804/2794571133", 2024-05-15T23:30:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6805/2794571133", 2024-05-15T23:30:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6807/2794571133", 2024-05-15T23:30:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6806/2794571133", 2024-05-15T23:30:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:30:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:30:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:30:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-05-15T23:30:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473", 2024-05-15T23:30:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:30:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:30:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:30:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:30:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:30:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:30:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 74, 2024-05-15T23:30:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 578, 2024-05-15T23:30:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:30:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:30:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:30:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6808", 2024-05-15T23:30:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:30:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:30:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6809", 2024-05-15T23:30:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:30:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:30:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6811", 2024-05-15T23:30:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:30:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:30:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6810", 2024-05-15T23:30:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:30:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6808/3735380744", 2024-05-15T23:30:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6809/3735380744", 2024-05-15T23:30:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6811/3735380744", 2024-05-15T23:30:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6810/3735380744", 2024-05-15T23:30:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:30:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:30:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:30:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-05-15T23:30:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5", 2024-05-15T23:30:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:30:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:30:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:30:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:30:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:30:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:30:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 81, 2024-05-15T23:30:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:30:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:30:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:30:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:30:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6812", 2024-05-15T23:30:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:30:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:30:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6813", 2024-05-15T23:30:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:30:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:30:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6815", 2024-05-15T23:30:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:30:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:30:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:30:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:30:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6814", 2024-05-15T23:30:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:30:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6812/2265615270", 2024-05-15T23:30:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6813/2265615270", 2024-05-15T23:30:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6815/2265615270", 2024-05-15T23:30:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6814/2265615270", 2024-05-15T23:30:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:30:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:30:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:30:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:30:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-05-15T23:30:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-05-15T23:30:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:30:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:30:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:30:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:30:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-05-15T23:30:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:30:21.891882+0000", 2024-05-15T23:30:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 355 2024-05-15T23:30:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-05-15T23:30:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:30:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:30:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:30:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:30:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:30:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:30:07.645290+0000", 2024-05-15T23:30:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:30:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-05-15T23:30:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:30:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:30:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:30:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:30:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:30:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:29:58.580698+0000", 2024-05-15T23:30:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 261 2024-05-15T23:30:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-05-15T23:30:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:30:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:30:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:30:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:30:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:30:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:30:19.343484+0000", 2024-05-15T23:30:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:30:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-05-15T23:30:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:30:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:30:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:30:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:30:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:30:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:29:55.861477+0000", 2024-05-15T23:30:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:30:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-05-15T23:30:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:30:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:30:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:30:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:30:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:30:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:30:18.004887+0000", 2024-05-15T23:30:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:30:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-05-15T23:30:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:30:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:30:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:30:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:30:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-05-15T23:30:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:30:30.662979+0000", 2024-05-15T23:30:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:30:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-05-15T23:30:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:30:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:30:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:30:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:30:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:30:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:30:31.780285+0000", 2024-05-15T23:30:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:30:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-05-15T23:30:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:30:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:30:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:30:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:30:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-05-15T23:30:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:29:43.787716+0000", 2024-05-15T23:30:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 484 2024-05-15T23:30:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-05-15T23:30:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:30:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:30:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:30:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:30:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:30:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:29:37.888583+0000", 2024-05-15T23:30:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:30:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-05-15T23:30:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:30:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:30:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:30:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:30:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:30:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:29:58.358309+0000", 2024-05-15T23:30:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:30:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-05-15T23:30:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:30:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:30:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:30:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:30:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:30:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:30:26.167473+0000", 2024-05-15T23:30:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:30:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:30:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2024-05-15T23:30:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:30:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "1.0", 2024-05-15T23:30:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-05-15T23:30:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: 6, 2024-05-15T23:30:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: 3 2024-05-15T23:30:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:30:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:30:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [], 2024-05-15T23:30:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-05-15T23:30:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-05-15T23:30:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-05-15T23:30:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-05-15T23:30:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2581408231": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:30:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/1136984393": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:30:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2443201864": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:30:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1252671888": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:30:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1451001371": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:30:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4084814866": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:30:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4076409285": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:30:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2988977694": "2024-05-16T22:57:38.505263+0000" 2024-05-15T23:30:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-05-15T23:30:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-05-15T23:30:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-05-15T23:30:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-05-15T23:30:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-05-15T23:30:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-05-15T23:30:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-05-15T23:30:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-05-15T23:30:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:30:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [], 2024-05-15T23:30:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2024-05-15T23:30:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-05-15T23:30:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-05-15T23:30:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-05-15T23:30:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-05-15T23:30:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-05-15T23:30:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-05-15T23:30:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-05-15T23:30:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-05-15T23:30:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-05-15T23:30:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:30:36.159 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:30:36.173 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:30:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:36.178 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:30:36.200 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:30:36.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:30:36.336 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:36.421 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/465723626' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:30:36.421 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:36 smithi084 ceph-mon[39784]: from='client.26746 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:30:36.421 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4154662229' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:30:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/465723626' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:30:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:36 smithi090 ceph-mon[44434]: from='client.26746 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:30:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4154662229' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:30:36.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/465723626' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:30:36.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:36 smithi088 ceph-mon[44699]: from='client.26746 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:30:36.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4154662229' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:30:36.594 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:30:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:36.598 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:30:36.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:30:36.741 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:37.206 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:30:37.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:37.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:30:37.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:30:37.359 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:37.493 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:30:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:37.501 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:30:37.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:37 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:30:37.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:37 smithi090 ceph-mon[44434]: from='client.26746 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:30:37.501 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:37 smithi090 ceph-mon[44434]: osdmap e588: 12 total, 12 up, 9 in 2024-05-15T23:30:37.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:37 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:30:37.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:37 smithi088 ceph-mon[44699]: from='client.26746 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:30:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:37 smithi088 ceph-mon[44699]: osdmap e588: 12 total, 12 up, 9 in 2024-05-15T23:30:37.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:30:37.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:37 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:30:37.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:37 smithi084 ceph-mon[39784]: from='client.26746 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:30:37.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:37 smithi084 ceph-mon[39784]: osdmap e588: 12 total, 12 up, 9 in 2024-05-15T23:30:37.637 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:38.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:38 smithi084 ceph-mon[39784]: pgmap v1567: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 170 B/s rd, 59 MiB/s wr, 1.10k op/s; 9641/289634 objects misplaced (3.329%); 5.8 MiB/s, 92 objects/s recovering 2024-05-15T23:30:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:38 smithi090 ceph-mon[44434]: pgmap v1567: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 170 B/s rd, 59 MiB/s wr, 1.10k op/s; 9641/289634 objects misplaced (3.329%); 5.8 MiB/s, 92 objects/s recovering 2024-05-15T23:30:38.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:38 smithi088 ceph-mon[44699]: pgmap v1567: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 170 B/s rd, 59 MiB/s wr, 1.10k op/s; 9641/289634 objects misplaced (3.329%); 5.8 MiB/s, 92 objects/s recovering 2024-05-15T23:30:38.670 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:30:38.670 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:30:38.670 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:30:38.670 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:30:38.671 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:30:38.894 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:30:38.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:38.902 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:30:39.003 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:30:39.029 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:39.049 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:30:39.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:39.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:30:39.153 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:30:39.200 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:39.559 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:30:39.563 INFO:tasks.thrashosds.thrasher:j is {'epoch': 587, 'fsid': '3999fb7e-130e-11ef-bc99-c7b262605968', 'created': '2024-05-15T22:56:41.212493+0000', 'modified': '2024-05-15T23:30:32.440220+0000', 'last_up_change': '2024-05-15T23:26:58.424794+0000', 'last_in_change': '2024-05-15T23:28:16.134275+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 30, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T22:59:38.603575+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9620716571807861, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.9599999785423279}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-05-15T23:28:50.243463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '554', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.6700000166893005, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9620716571807861, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.9599999785423279}}], 'osds': [{'osd': 0, 'uuid': '80159cc8-1b6c-4aa6-a114-966786d7be1e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 355, 'up_from': 397, 'up_thru': 283, 'down_at': 356, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6801', 'nonce': 3077815271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6802', 'nonce': 3077815271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6804', 'nonce': 3077815271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6803', 'nonce': 3077815271}]}, 'public_addr': '172.21.15.84:6801/3077815271', 'cluster_addr': '172.21.15.84:6802/3077815271', 'heartbeat_back_addr': '172.21.15.84:6804/3077815271', 'heartbeat_front_addr': '172.21.15.84:6803/3077815271', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9c2c76c9-ce0f-4c2d-9d96-c87a10187607', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 578, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6805', 'nonce': 2140830229}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6806', 'nonce': 2140830229}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6808', 'nonce': 2140830229}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6807', 'nonce': 2140830229}]}, 'public_addr': '172.21.15.84:6805/2140830229', 'cluster_addr': '172.21.15.84:6806/2140830229', 'heartbeat_back_addr': '172.21.15.84:6808/2140830229', 'heartbeat_front_addr': '172.21.15.84:6807/2140830229', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e8514daa-b5f9-4f2f-a432-53f4a4eb86bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 261, 'up_from': 291, 'up_thru': 549, 'down_at': 262, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6809', 'nonce': 941507493}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6810', 'nonce': 941507493}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6812', 'nonce': 941507493}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6811', 'nonce': 941507493}]}, 'public_addr': '172.21.15.84:6809/941507493', 'cluster_addr': '172.21.15.84:6810/941507493', 'heartbeat_back_addr': '172.21.15.84:6812/941507493', 'heartbeat_front_addr': '172.21.15.84:6811/941507493', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5d9965f9-c0b2-48ce-869c-582990985623', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6813', 'nonce': 2204039828}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6814', 'nonce': 2204039828}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6816', 'nonce': 2204039828}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6815', 'nonce': 2204039828}]}, 'public_addr': '172.21.15.84:6813/2204039828', 'cluster_addr': '172.21.15.84:6814/2204039828', 'heartbeat_back_addr': '172.21.15.84:6816/2204039828', 'heartbeat_front_addr': '172.21.15.84:6815/2204039828', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4bc50523-8480-44d2-8d76-31479e839c85', 'up': 1, 'in': 1, 'weight': 0.396820068359375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6800', 'nonce': 2267955292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6801', 'nonce': 2267955292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6803', 'nonce': 2267955292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6802', 'nonce': 2267955292}]}, 'public_addr': '172.21.15.88:6800/2267955292', 'cluster_addr': '172.21.15.88:6801/2267955292', 'heartbeat_back_addr': '172.21.15.88:6803/2267955292', 'heartbeat_front_addr': '172.21.15.88:6802/2267955292', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'd4db8d7f-afc7-43dc-81fe-f70c290d8b20', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6804', 'nonce': 661041538}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6805', 'nonce': 661041538}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6807', 'nonce': 661041538}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6806', 'nonce': 661041538}]}, 'public_addr': '172.21.15.88:6804/661041538', 'cluster_addr': '172.21.15.88:6805/661041538', 'heartbeat_back_addr': '172.21.15.88:6807/661041538', 'heartbeat_front_addr': '172.21.15.88:6806/661041538', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'a2773454-2318-4a5b-8f31-0952b1b07f2f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6808', 'nonce': 380856869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6809', 'nonce': 380856869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6811', 'nonce': 380856869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6810', 'nonce': 380856869}]}, 'public_addr': '172.21.15.88:6808/380856869', 'cluster_addr': '172.21.15.88:6809/380856869', 'heartbeat_back_addr': '172.21.15.88:6811/380856869', 'heartbeat_front_addr': '172.21.15.88:6810/380856869', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '028ca7b6-3e00-4466-a33f-b726cd60cb34', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6812', 'nonce': 1044709063}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6813', 'nonce': 1044709063}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6815', 'nonce': 1044709063}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6814', 'nonce': 1044709063}]}, 'public_addr': '172.21.15.88:6812/1044709063', 'cluster_addr': '172.21.15.88:6813/1044709063', 'heartbeat_back_addr': '172.21.15.88:6815/1044709063', 'heartbeat_front_addr': '172.21.15.88:6814/1044709063', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '02cab286-d487-4a66-982c-3cf28f55e851', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.54486083984375, 'last_clean_begin': 61, 'last_clean_end': 484, 'up_from': 520, 'up_thru': 202, 'down_at': 485, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6800', 'nonce': 703598286}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 703598286}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 703598286}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6802', 'nonce': 703598286}]}, 'public_addr': '172.21.15.90:6800/703598286', 'cluster_addr': '172.21.15.90:6801/703598286', 'heartbeat_back_addr': '172.21.15.90:6803/703598286', 'heartbeat_front_addr': '172.21.15.90:6802/703598286', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '47d38bc7-39ba-4b35-af24-e282354dd613', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 68, 'up_thru': 565, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6804', 'nonce': 2794571133}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 2794571133}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 2794571133}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6806', 'nonce': 2794571133}]}, 'public_addr': '172.21.15.90:6804/2794571133', 'cluster_addr': '172.21.15.90:6805/2794571133', 'heartbeat_back_addr': '172.21.15.90:6807/2794571133', 'heartbeat_front_addr': '172.21.15.90:6806/2794571133', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'd7b2aeae-d012-41eb-9f91-e4c05c649473', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 578, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6808', 'nonce': 3735380744}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 3735380744}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 3735380744}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 3735380744}]}, 'public_addr': '172.21.15.90:6808/3735380744', 'cluster_addr': '172.21.15.90:6809/3735380744', 'heartbeat_back_addr': '172.21.15.90:6811/3735380744', 'heartbeat_front_addr': '172.21.15.90:6810/3735380744', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6812', 'nonce': 2265615270}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 2265615270}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 2265615270}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 2265615270}]}, 'public_addr': '172.21.15.90:6812/2265615270', 'cluster_addr': '172.21.15.90:6813/2265615270', 'heartbeat_back_addr': '172.21.15.90:6815/2265615270', 'heartbeat_front_addr': '172.21.15.90:6814/2265615270', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T23:30:21.891882+0000', 'dead_epoch': 355}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:30:07.645290+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:29:58.580698+0000', 'dead_epoch': 261}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:30:19.343484+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:29:55.861477+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:30:18.004887+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T23:30:30.662979+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:30:31.780285+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T23:29:43.787716+0000', 'dead_epoch': 484}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:29:37.888583+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:29:58.358309+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:30:26.167473+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [6, 3]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/2581408231': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:6800/1136984393': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:6800/2443201864': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/1252671888': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/1451001371': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/4084814866': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:0/4076409285': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/2988977694': '2024-05-16T22:57:38.505263+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-15T23:30:39.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:30:39.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:39.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:30:39.668 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:30:39.694 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:39.714 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:30:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:39.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:30:39.819 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:30:39.868 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:39.916 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:30:39.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:39.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:30:40.021 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:30:40.083 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:40.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T23:30:40.209 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:30:40.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:40.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:30:40.252 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:30:40.298 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:30:40.316 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:30:40.336 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:30:40.343 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:40.362 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:30:40.363 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:30:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:40.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:30:40.395 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:30:40.419 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:30:40.452 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:30:40.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:30:40.484 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:30:40.513 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:40 smithi090 ceph-mon[44434]: pgmap v1568: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 133 B/s rd, 32 MiB/s wr, 758 op/s; 9641/288572 objects misplaced (3.341%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T23:30:40.514 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:40.514 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:30:40.549 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:30:40.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:40.555 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:30:40.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:40 smithi084 ceph-mon[39784]: pgmap v1568: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 133 B/s rd, 32 MiB/s wr, 758 op/s; 9641/288572 objects misplaced (3.341%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T23:30:40.556 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:30:40.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:40 smithi088 ceph-mon[44699]: pgmap v1568: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 133 B/s rd, 32 MiB/s wr, 758 op/s; 9641/288572 objects misplaced (3.341%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T23:30:40.584 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:30:40.616 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:30:40.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:30:40.662 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:30:40.668 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:30:40.668 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:30:40.668 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:30:40.668 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:30:40.668 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:30:40.668 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:30:40.668 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:30:40.668 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:30:40.668 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:30:40.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:30:40.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:30:40.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:30:40.714 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:40.743 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:30:40.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:40.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:30:40.752 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:30:40.853 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:30:40.879 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:40.898 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:30:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:40.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:30:41.002 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:30:41.029 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:41.049 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:30:41.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:41.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:30:41.075 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:30:41.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:30:41.194 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:41.227 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:30:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:41.231 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:30:41.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:30:41.371 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:41 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3079616074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:30:41.371 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.371 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.371 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.371 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.372 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.374 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:41.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:41 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3079616074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:30:41.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:41 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3079616074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:30:41.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:41.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:42.328 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:30:42.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:42.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:30:42.437 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:30:42.464 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:42.484 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:30:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:42.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:30:42.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:42.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:42.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:42.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:42 smithi090 ceph-mon[44434]: pgmap v1569: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 118 B/s rd, 24 MiB/s wr, 699 op/s; 9532/286994 objects misplaced (3.321%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T23:30:42.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:42.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:42.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:42.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:42 smithi088 ceph-mon[44699]: pgmap v1569: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 118 B/s rd, 24 MiB/s wr, 699 op/s; 9532/286994 objects misplaced (3.321%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T23:30:42.589 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:30:42.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:42.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:42.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:42.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:42 smithi084 ceph-mon[39784]: pgmap v1569: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 22 GiB used, 782 GiB / 805 GiB avail; 118 B/s rd, 24 MiB/s wr, 699 op/s; 9532/286994 objects misplaced (3.321%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T23:30:42.665 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:43.390 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:30:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:43.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:30:43.505 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:30:43.531 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:43.568 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:30:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:43.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:30:43.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:30:43.728 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:44.101 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:30:44.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:44.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:30:44.209 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:30:44.271 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:44.382 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:30:44.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:44.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:30:44.392 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:30:44.457 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:44 smithi090 ceph-mon[44434]: pgmap v1570: 17 pgs: 2 active+recovering, 15 active+clean; 8.6 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 102 B/s rd, 19 MiB/s wr, 881 op/s; 9296/280938 objects misplaced (3.309%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:30:44.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:30:44.552 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:44.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:44 smithi088 ceph-mon[44699]: pgmap v1570: 17 pgs: 2 active+recovering, 15 active+clean; 8.6 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 102 B/s rd, 19 MiB/s wr, 881 op/s; 9296/280938 objects misplaced (3.309%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:30:44.581 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:30:44.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:44.591 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:30:44.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:44 smithi084 ceph-mon[39784]: pgmap v1570: 17 pgs: 2 active+recovering, 15 active+clean; 8.6 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 102 B/s rd, 19 MiB/s wr, 881 op/s; 9296/280938 objects misplaced (3.309%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:30:44.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:30:44.732 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:44.848 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:30:44.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:44.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:30:44.914 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:30:44.915 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:30:44.953 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:30:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:45.000 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:30:45.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:45.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:30:45.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:30:45.159 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:45.192 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:30:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:45.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:30:45.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:30:45.349 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:45.427 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:30:46.288 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:46 smithi084 ceph-mon[39784]: pgmap v1571: 17 pgs: 2 active+recovering, 15 active+clean; 8.5 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 102 B/s rd, 16 MiB/s wr, 882 op/s; 9296/279828 objects misplaced (3.322%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:30:46.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:46 smithi090 ceph-mon[44434]: pgmap v1571: 17 pgs: 2 active+recovering, 15 active+clean; 8.5 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 102 B/s rd, 16 MiB/s wr, 882 op/s; 9296/279828 objects misplaced (3.322%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:30:46.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:46 smithi090 ceph-mon[44434]: from='client.36698 v1:172.21.15.84:0/545326945' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:30:46.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:46 smithi088 ceph-mon[44699]: pgmap v1571: 17 pgs: 2 active+recovering, 15 active+clean; 8.5 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 102 B/s rd, 16 MiB/s wr, 882 op/s; 9296/279828 objects misplaced (3.322%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:30:46.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:46 smithi088 ceph-mon[44699]: from='client.36698 v1:172.21.15.84:0/545326945' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:30:46.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:46 smithi084 ceph-mon[39784]: from='client.36698 v1:172.21.15.84:0/545326945' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:30:46.763 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:30:46.763 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:30:47.225 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:30:47.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:47.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:30:47.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T23:30:47.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:30:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:47 smithi090 ceph-mon[44434]: pgmap v1572: 17 pgs: 2 active+recovering, 15 active+clean; 8.4 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 7.8 MiB/s wr, 926 op/s; 8952/273886 objects misplaced (3.269%); 3.9 MiB/s, 62 objects/s recovering 2024-05-15T23:30:47.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:47 smithi084 ceph-mon[39784]: pgmap v1572: 17 pgs: 2 active+recovering, 15 active+clean; 8.4 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 7.8 MiB/s wr, 926 op/s; 8952/273886 objects misplaced (3.269%); 3.9 MiB/s, 62 objects/s recovering 2024-05-15T23:30:47.564 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:47.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:47 smithi088 ceph-mon[44699]: pgmap v1572: 17 pgs: 2 active+recovering, 15 active+clean; 8.4 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 7.8 MiB/s wr, 926 op/s; 8952/273886 objects misplaced (3.269%); 3.9 MiB/s, 62 objects/s recovering 2024-05-15T23:30:47.647 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:30:48.409 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:30:48.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:48.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:30:48.452 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1571,"stamp":"2024-05-15T23:30:45.133226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9169797688,"num_objects":139914,"num_object_clones":0,"num_object_copies":279828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9296,"num_objects_unfound":0,"num_objects_dirty":139914,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":152549,"num_write_kb":9356903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3228,"num_bytes_recovered":212743776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152520,"ondisk_log_size":152520,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":22978628,"kb_used_data":18176504,"kb_used_omap":16,"kb_used_meta":4801967,"kb_avail":820727740,"statfs":{"total":863955320832,"available":840425205760,"internally_reserved":0,"allocated":18612740096,"data_stored":18609318634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17176,"internal_metadata":4917214440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,3,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-241303528,"num_objects":-3681,"num_object_clones":0,"num_object_copies":-7362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-675,"num_objects_unfound":0,"num_objects_dirty":-3681,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8833,"num_write_kb":164801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":675,"num_bytes_recovered":44236800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.007527"},"pg_stats":[{"pgid":"7.f","version":"588'9426","reported_seq":9513,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-05-15T23:30:40.911630+0000","last_change":"2024-05-15T23:28:51.305473+0000","last_active":"2024-05-15T23:30:40.911630+0000","last_peered":"2024-05-15T23:30:40.911630+0000","last_clean":"2024-05-15T23:30:40.911630+0000","last_became_active":"2024-05-15T23:28:51.305173+0000","last_became_peered":"2024-05-15T23:28:51.305173+0000","last_unstale":"2024-05-15T23:30:40.911630+0000","last_undegraded":"2024-05-15T23:30:40.911630+0000","last_fullsized":"2024-05-15T23:30:40.911630+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":9426,"log_dups_size":0,"ondisk_log_size":9426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575668224,"num_objects":8784,"num_object_clones":0,"num_object_copies":17568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9426,"num_write_kb":582720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"588'9535","reported_seq":9713,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-05-15T23:30:44.050647+0000","last_change":"2024-05-15T23:30:19.696616+0000","last_active":"2024-05-15T23:30:44.050647+0000","last_peered":"2024-05-15T23:30:44.050647+0000","last_clean":"2024-05-15T23:30:44.050647+0000","last_became_active":"2024-05-15T23:28:51.519387+0000","last_became_peered":"2024-05-15T23:28:51.519387+0000","last_unstale":"2024-05-15T23:30:44.050647+0000","last_undegraded":"2024-05-15T23:30:44.050647+0000","last_fullsized":"2024-05-15T23:30:44.050647+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'8176","last_scrub_stamp":"2024-05-15T23:30:19.696456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:19.696456+0000","objects_scrubbed":8143,"log_size":9535,"log_dups_size":0,"ondisk_log_size":9535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:31:47.470011+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554500096,"num_objects":8461,"num_object_clones":0,"num_object_copies":16922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9535,"num_write_kb":575872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"588'9584","reported_seq":9672,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-05-15T23:30:41.130710+0000","last_change":"2024-05-15T23:28:51.306905+0000","last_active":"2024-05-15T23:30:41.130710+0000","last_peered":"2024-05-15T23:30:41.130710+0000","last_clean":"2024-05-15T23:30:41.130710+0000","last_became_active":"2024-05-15T23:28:51.306664+0000","last_became_peered":"2024-05-15T23:28:51.306664+0000","last_unstale":"2024-05-15T23:30:41.130710+0000","last_undegraded":"2024-05-15T23:30:41.130710+0000","last_fullsized":"2024-05-15T23:30:41.130710+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":9584,"log_dups_size":0,"ondisk_log_size":9584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":584777752,"num_objects":8924,"num_object_clones":0,"num_object_copies":17848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8924,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9584,"num_write_kb":592193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"588'9467","reported_seq":9638,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-05-15T23:30:41.387888+0000","last_change":"2024-05-15T23:30:09.921129+0000","last_active":"2024-05-15T23:30:41.387888+0000","last_peered":"2024-05-15T23:30:41.387888+0000","last_clean":"2024-05-15T23:30:41.387888+0000","last_became_active":"2024-05-15T23:28:51.305728+0000","last_became_peered":"2024-05-15T23:28:51.305728+0000","last_unstale":"2024-05-15T23:30:41.387888+0000","last_undegraded":"2024-05-15T23:30:41.387888+0000","last_fullsized":"2024-05-15T23:30:41.387888+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'7710","last_scrub_stamp":"2024-05-15T23:30:09.921025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:09.921025+0000","objects_scrubbed":7710,"log_size":9467,"log_dups_size":0,"ondisk_log_size":9467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:31:16.526902+0000","scrub_duration":673,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575209472,"num_objects":8777,"num_object_clones":0,"num_object_copies":17554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9467,"num_write_kb":583808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"588'9297","reported_seq":9467,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-05-15T23:30:41.162601+0000","last_change":"2024-05-15T23:30:11.062321+0000","last_active":"2024-05-15T23:30:41.162601+0000","last_peered":"2024-05-15T23:30:41.162601+0000","last_clean":"2024-05-15T23:30:41.162601+0000","last_became_active":"2024-05-15T23:28:51.306794+0000","last_became_peered":"2024-05-15T23:28:51.306794+0000","last_unstale":"2024-05-15T23:30:41.162601+0000","last_undegraded":"2024-05-15T23:30:41.162601+0000","last_fullsized":"2024-05-15T23:30:41.162601+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'7585","last_scrub_stamp":"2024-05-15T23:30:11.062191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:11.062191+0000","objects_scrubbed":7567,"log_size":9297,"log_dups_size":0,"ondisk_log_size":9297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:31:33.920474+0000","scrub_duration":1068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":562626560,"num_objects":8585,"num_object_clones":0,"num_object_copies":17170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9297,"num_write_kb":572224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"588'9581","reported_seq":9668,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-05-15T23:30:40.869775+0000","last_change":"2024-05-15T23:28:51.308505+0000","last_active":"2024-05-15T23:30:40.869775+0000","last_peered":"2024-05-15T23:30:40.869775+0000","last_clean":"2024-05-15T23:30:40.869775+0000","last_became_active":"2024-05-15T23:28:51.308216+0000","last_became_peered":"2024-05-15T23:28:51.308216+0000","last_unstale":"2024-05-15T23:30:40.869775+0000","last_undegraded":"2024-05-15T23:30:40.869775+0000","last_fullsized":"2024-05-15T23:30:40.869775+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":9581,"log_dups_size":0,"ondisk_log_size":9581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":580845568,"num_objects":8863,"num_object_clones":0,"num_object_copies":17726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9581,"num_write_kb":590208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"588'9627","reported_seq":9803,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-05-15T23:30:42.354240+0000","last_change":"2024-05-15T23:30:17.377631+0000","last_active":"2024-05-15T23:30:42.354240+0000","last_peered":"2024-05-15T23:30:42.354240+0000","last_clean":"2024-05-15T23:30:42.354240+0000","last_became_active":"2024-05-15T23:28:51.519752+0000","last_became_peered":"2024-05-15T23:28:51.519752+0000","last_unstale":"2024-05-15T23:30:42.354240+0000","last_undegraded":"2024-05-15T23:30:42.354240+0000","last_fullsized":"2024-05-15T23:30:42.354240+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"580'8175","last_scrub_stamp":"2024-05-15T23:30:17.377489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:17.377489+0000","objects_scrubbed":8113,"log_size":9627,"log_dups_size":0,"ondisk_log_size":9627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:31:39.238196+0000","scrub_duration":1869,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":569704448,"num_objects":8693,"num_object_clones":0,"num_object_copies":17386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9627,"num_write_kb":586240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"588'9777","reported_seq":9957,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-05-15T23:30:44.056471+0000","last_change":"2024-05-15T23:30:21.770656+0000","last_active":"2024-05-15T23:30:44.056471+0000","last_peered":"2024-05-15T23:30:44.056471+0000","last_clean":"2024-05-15T23:30:44.056471+0000","last_became_active":"2024-05-15T23:28:51.304930+0000","last_became_peered":"2024-05-15T23:28:51.304930+0000","last_unstale":"2024-05-15T23:30:44.056471+0000","last_undegraded":"2024-05-15T23:30:44.056471+0000","last_fullsized":"2024-05-15T23:30:44.056471+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'8556","last_scrub_stamp":"2024-05-15T23:30:21.770528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:21.770528+0000","objects_scrubbed":8519,"log_size":9777,"log_dups_size":0,"ondisk_log_size":9777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:31:23.768447+0000","scrub_duration":1444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575995904,"num_objects":8789,"num_object_clones":0,"num_object_copies":17578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9777,"num_write_kb":594112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"588'9441","reported_seq":9528,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-05-15T23:30:41.361719+0000","last_change":"2024-05-15T23:28:51.307151+0000","last_active":"2024-05-15T23:30:41.361719+0000","last_peered":"2024-05-15T23:30:41.361719+0000","last_clean":"2024-05-15T23:30:41.361719+0000","last_became_active":"2024-05-15T23:28:51.306983+0000","last_became_peered":"2024-05-15T23:28:51.306983+0000","last_unstale":"2024-05-15T23:30:41.361719+0000","last_undegraded":"2024-05-15T23:30:41.361719+0000","last_fullsized":"2024-05-15T23:30:41.361719+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":9441,"log_dups_size":0,"ondisk_log_size":9441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567738368,"num_objects":8663,"num_object_clones":0,"num_object_copies":17326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9441,"num_write_kb":579328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.5","version":"588'9612","reported_seq":9786,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-05-15T23:30:42.327417+0000","last_change":"2024-05-15T23:30:13.755483+0000","last_active":"2024-05-15T23:30:42.327417+0000","last_peered":"2024-05-15T23:30:42.327417+0000","last_clean":"2024-05-15T23:30:42.327417+0000","last_became_active":"2024-05-15T23:28:51.303581+0000","last_became_peered":"2024-05-15T23:28:51.303581+0000","last_unstale":"2024-05-15T23:30:42.327417+0000","last_undegraded":"2024-05-15T23:30:42.327417+0000","last_fullsized":"2024-05-15T23:30:42.327417+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"580'7953","last_scrub_stamp":"2024-05-15T23:30:13.755299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:13.755299+0000","objects_scrubbed":7919,"log_size":9612,"log_dups_size":0,"ondisk_log_size":9612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:31:34.976243+0000","scrub_duration":1278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":571080704,"num_objects":8714,"num_object_clones":0,"num_object_copies":17428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9612,"num_write_kb":586432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"588'9631","reported_seq":14882,"reported_epoch":588,"state":"active+recovering","last_fresh":"2024-05-15T23:30:41.189406+0000","last_change":"2024-05-15T23:30:10.297256+0000","last_active":"2024-05-15T23:30:41.189406+0000","last_peered":"2024-05-15T23:30:41.189406+0000","last_clean":"2024-05-15T23:29:30.779087+0000","last_became_active":"2024-05-15T23:30:10.242131+0000","last_became_peered":"2024-05-15T23:30:10.242131+0000","last_unstale":"2024-05-15T23:30:41.189406+0000","last_undegraded":"2024-05-15T23:30:41.189406+0000","last_fullsized":"2024-05-15T23:30:41.189406+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":9631,"log_dups_size":0,"ondisk_log_size":9631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":584908800,"num_objects":8925,"num_object_clones":0,"num_object_copies":17850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2282,"num_objects_unfound":0,"num_objects_dirty":8925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9631,"num_write_kb":593792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2580,"num_bytes_recovered":169082880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":2282},{"shards":"1,9","objects":6643}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"588'9607","reported_seq":9781,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-05-15T23:30:41.330525+0000","last_change":"2024-05-15T23:30:12.055284+0000","last_active":"2024-05-15T23:30:41.330525+0000","last_peered":"2024-05-15T23:30:41.330525+0000","last_clean":"2024-05-15T23:30:41.330525+0000","last_became_active":"2024-05-15T23:28:51.305691+0000","last_became_peered":"2024-05-15T23:28:51.305691+0000","last_unstale":"2024-05-15T23:30:41.330525+0000","last_undegraded":"2024-05-15T23:30:41.330525+0000","last_fullsized":"2024-05-15T23:30:41.330525+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'7899","last_scrub_stamp":"2024-05-15T23:30:12.055082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:12.055082+0000","objects_scrubbed":7846,"log_size":9607,"log_dups_size":0,"ondisk_log_size":9607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:31:27.901478+0000","scrub_duration":1764,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":577437696,"num_objects":8811,"num_object_clones":0,"num_object_copies":17622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9607,"num_write_kb":589376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"588'9552","reported_seq":9639,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-05-15T23:30:41.375870+0000","last_change":"2024-05-15T23:28:51.519631+0000","last_active":"2024-05-15T23:30:41.375870+0000","last_peered":"2024-05-15T23:30:41.375870+0000","last_clean":"2024-05-15T23:30:41.375870+0000","last_became_active":"2024-05-15T23:28:51.519343+0000","last_became_peered":"2024-05-15T23:28:51.519343+0000","last_unstale":"2024-05-15T23:30:41.375870+0000","last_undegraded":"2024-05-15T23:30:41.375870+0000","last_fullsized":"2024-05-15T23:30:41.375870+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":9552,"log_dups_size":0,"ondisk_log_size":9552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":577110016,"num_objects":8806,"num_object_clones":0,"num_object_copies":17612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9552,"num_write_kb":587456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"588'9394","reported_seq":10769,"reported_epoch":588,"state":"active+recovering","last_fresh":"2024-05-15T23:30:40.815451+0000","last_change":"2024-05-15T23:30:10.714218+0000","last_active":"2024-05-15T23:30:40.815451+0000","last_peered":"2024-05-15T23:30:40.815451+0000","last_clean":"2024-05-15T23:30:09.208910+0000","last_became_active":"2024-05-15T23:30:10.487853+0000","last_became_peered":"2024-05-15T23:30:10.487853+0000","last_unstale":"2024-05-15T23:30:40.815451+0000","last_undegraded":"2024-05-15T23:30:40.815451+0000","last_fullsized":"2024-05-15T23:30:40.815451+0000","mapping_epoch":578,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":9394,"log_dups_size":0,"ondisk_log_size":9394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574881792,"num_objects":8772,"num_object_clones":0,"num_object_copies":17544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7014,"num_objects_unfound":0,"num_objects_dirty":8772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9394,"num_write_kb":581312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":642,"num_bytes_recovered":42074112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":["10","4"],"object_location_counts":[{"shards":"3,10","objects":1758},{"shards":"4,10","objects":7014}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"588'9617","reported_seq":9794,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-05-15T23:30:42.962575+0000","last_change":"2024-05-15T23:30:18.018674+0000","last_active":"2024-05-15T23:30:42.962575+0000","last_peered":"2024-05-15T23:30:42.962575+0000","last_clean":"2024-05-15T23:30:42.962575+0000","last_became_active":"2024-05-15T23:28:51.303621+0000","last_became_peered":"2024-05-15T23:28:51.303621+0000","last_unstale":"2024-05-15T23:30:42.962575+0000","last_undegraded":"2024-05-15T23:30:42.962575+0000","last_fullsized":"2024-05-15T23:30:42.962575+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"580'8233","last_scrub_stamp":"2024-05-15T23:30:18.018497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:18.018497+0000","objects_scrubbed":8203,"log_size":9617,"log_dups_size":0,"ondisk_log_size":9617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:31:25.203224+0000","scrub_duration":1018,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572063744,"num_objects":8729,"num_object_clones":0,"num_object_copies":17458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9617,"num_write_kb":587072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"588'9278","reported_seq":9449,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-05-15T23:30:41.283538+0000","last_change":"2024-05-15T23:30:10.347103+0000","last_active":"2024-05-15T23:30:41.283538+0000","last_peered":"2024-05-15T23:30:41.283538+0000","last_clean":"2024-05-15T23:30:41.283538+0000","last_became_active":"2024-05-15T23:28:51.304317+0000","last_became_peered":"2024-05-15T23:28:51.304317+0000","last_unstale":"2024-05-15T23:30:41.283538+0000","last_undegraded":"2024-05-15T23:30:41.283538+0000","last_fullsized":"2024-05-15T23:30:41.283538+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'7605","last_scrub_stamp":"2024-05-15T23:30:10.347000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:10.347000+0000","objects_scrubbed":7603,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:31:12.094859+0000","scrub_duration":722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":564658176,"num_objects":8616,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9278,"num_write_kb":572608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1258,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-05-15T23:30:36.188992+0000","last_change":"2024-05-15T23:30:29.464644+0000","last_active":"2024-05-15T23:30:36.188992+0000","last_peered":"2024-05-15T23:30:36.188992+0000","last_clean":"2024-05-15T23:30:36.188992+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:30:36.188992+0000","last_undegraded":"2024-05-15T23:30:36.188992+0000","last_fullsized":"2024-05-15T23:30:36.188992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:30:29.464530+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:30:29.464530+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:31:37.479720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":9169207320,"num_objects":139912,"num_object_clones":0,"num_object_copies":279824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9296,"num_objects_unfound":0,"num_objects_dirty":139912,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152426,"num_write_kb":9354753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3222,"num_bytes_recovered":211156992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18552659968,"data_stored":18552651824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152426,"ondisk_log_size":152426,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351299,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2908144,"kb_used_data":2241128,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":90837008,"statfs":{"total":95995035648,"available":93017096192,"internally_reserved":0,"allocated":2294915072,"data_stored":2294533034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":683014596},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580232,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2636200,"kb_used_data":2248224,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":91108952,"statfs":{"total":95995035648,"available":93295566848,"internally_reserved":0,"allocated":2302181376,"data_stored":2301802910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":397277632},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776464,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3284008,"kb_used_data":2693344,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":90461144,"statfs":{"total":95995035648,"available":92632211456,"internally_reserved":0,"allocated":2757984256,"data_stored":2757605790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":604830144},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":0,"seq":2233382993968,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267032,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1669936,"kb_used_data":1129256,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92075216,"statfs":{"total":95995035648,"available":94285021184,"internally_reserved":0,"allocated":1156358144,"data_stored":1155976126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":553646538},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495967,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575225,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2927724,"kb_used_data":2250980,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":90817428,"statfs":{"total":95995035648,"available":92997046272,"internally_reserved":0,"allocated":2305003520,"data_stored":2304620982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":692976070},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016634,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483323,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2335208,"kb_used_data":2226464,"kb_used_omap":4,"kb_used_meta":108667,"kb_avail":91409944,"statfs":{"total":95995035648,"available":93603782656,"internally_reserved":0,"allocated":2279899136,"data_stored":2279520670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":111275715},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150064,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4198828,"kb_used_data":3481828,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":89546324,"statfs":{"total":95995035648,"available":91695435776,"internally_reserved":0,"allocated":3565391872,"data_stored":3565009334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":734198215},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986535,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1320364,"kb_used_data":801888,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92424788,"statfs":{"total":95995035648,"available":94642982912,"internally_reserved":0,"allocated":821133312,"data_stored":820754846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790308,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1698216,"kb_used_data":1103392,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92046936,"statfs":{"total":95995035648,"available":94256062464,"internally_reserved":0,"allocated":1129873408,"data_stored":1129494942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2300907520,"data_stored":2300903448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2265317376,"data_stored":2265317376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3560706048,"data_stored":3560701976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":820379648,"data_stored":820379648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1127350272,"data_stored":1127350272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1149239296,"data_stored":1149239296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2747990016,"data_stored":2747990016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2296971264,"data_stored":2296971264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2283798528,"data_stored":2283798528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:30:48.453 INFO:tasks.thrashosds.thrasher:Setting 7.8 to [6, 0] 2024-05-15T23:30:48.454 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '7.8', '6', '0'] 2024-05-15T23:30:48.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pg-upmap 7.8 6 0 2024-05-15T23:30:48.515 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:30:48.542 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:48.562 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:30:48.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:48.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:30:48.667 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:30:48.693 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:48.712 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:30:48.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:48.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:30:48.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:30:48.864 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:48.977 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:30:49.422 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:30:49.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:49.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:30:49.526 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:30:49.585 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:49.619 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:30:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:49.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:30:49.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:30:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:50.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:50 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:30:50.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:50 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:30:50.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:50 smithi090 ceph-mon[44434]: osdmap e589: 12 total, 12 up, 9 in 2024-05-15T23:30:50.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:50 smithi090 ceph-mon[44434]: pgmap v1574: 17 pgs: 2 active+recovering, 15 active+clean; 8.3 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 865 op/s; 8952/271250 objects misplaced (3.300%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:30:50.205 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:30:50.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:50.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:30:50.310 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:30:50.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:50 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:30:50.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:50 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:30:50.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:50 smithi088 ceph-mon[44699]: osdmap e589: 12 total, 12 up, 9 in 2024-05-15T23:30:50.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:50 smithi088 ceph-mon[44699]: pgmap v1574: 17 pgs: 2 active+recovering, 15 active+clean; 8.3 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 865 op/s; 8952/271250 objects misplaced (3.300%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:30:50.336 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:50.355 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:30:50.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:50.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:30:50.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:50 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:30:50.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:50 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:30:50.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:50 smithi084 ceph-mon[39784]: osdmap e589: 12 total, 12 up, 9 in 2024-05-15T23:30:50.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:50 smithi084 ceph-mon[39784]: pgmap v1574: 17 pgs: 2 active+recovering, 15 active+clean; 8.3 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 865 op/s; 8952/271250 objects misplaced (3.300%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:30:50.460 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:30:50.486 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:50.506 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:30:50.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:50.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:30:50.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:30:50.670 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:50.974 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:30:51.408 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:30:51.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:51 smithi090 ceph-mon[44434]: pgmap v1575: 17 pgs: 2 active+recovering, 15 active+clean; 8.2 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 863 op/s; 8837/269724 objects misplaced (3.276%); 4.3 MiB/s, 69 objects/s recovering 2024-05-15T23:30:51.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3395267079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:30:51.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:51 smithi090 ceph-mon[44434]: from='client.36704 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:30:51.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:51 smithi088 ceph-mon[44699]: pgmap v1575: 17 pgs: 2 active+recovering, 15 active+clean; 8.2 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 863 op/s; 8837/269724 objects misplaced (3.276%); 4.3 MiB/s, 69 objects/s recovering 2024-05-15T23:30:51.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3395267079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:30:51.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:51 smithi088 ceph-mon[44699]: from='client.36704 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:30:51.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:51 smithi084 ceph-mon[39784]: pgmap v1575: 17 pgs: 2 active+recovering, 15 active+clean; 8.2 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 0 B/s wr, 863 op/s; 8837/269724 objects misplaced (3.276%); 4.3 MiB/s, 69 objects/s recovering 2024-05-15T23:30:51.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3395267079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:30:51.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:51 smithi084 ceph-mon[39784]: from='client.36704 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:30:51.871 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:30:51.871 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:30:51.935 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:30:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:51.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:30:52.046 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:30:52.094 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:52.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:52 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:30:52.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:52 smithi090 ceph-mon[44434]: from='client.36704 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:30:52.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:52 smithi090 ceph-mon[44434]: osdmap e590: 12 total, 12 up, 9 in 2024-05-15T23:30:52.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:52 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:30:52.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:52 smithi088 ceph-mon[44699]: from='client.36704 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:30:52.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:52 smithi088 ceph-mon[44699]: osdmap e590: 12 total, 12 up, 9 in 2024-05-15T23:30:52.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:52 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:30:52.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:52 smithi084 ceph-mon[39784]: from='client.36704 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:30:52.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:52 smithi084 ceph-mon[39784]: osdmap e590: 12 total, 12 up, 9 in 2024-05-15T23:30:53.331 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T23:30:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:53 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:30:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:53 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:30:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:53 smithi090 ceph-mon[44434]: pgmap v1577: 17 pgs: 2 active+recovering, 15 active+clean; 8.1 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 983 op/s; 8581/264086 objects misplaced (3.249%); 5.6 MiB/s, 89 objects/s recovering 2024-05-15T23:30:53.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:53 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:30:53.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:53 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:30:53.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:53 smithi088 ceph-mon[44699]: pgmap v1577: 17 pgs: 2 active+recovering, 15 active+clean; 8.1 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 983 op/s; 8581/264086 objects misplaced (3.249%); 5.6 MiB/s, 89 objects/s recovering 2024-05-15T23:30:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:53 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:30:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:53 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:30:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:53 smithi084 ceph-mon[39784]: pgmap v1577: 17 pgs: 2 active+recovering, 15 active+clean; 8.1 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 983 op/s; 8581/264086 objects misplaced (3.249%); 5.6 MiB/s, 89 objects/s recovering 2024-05-15T23:30:53.742 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:30:54.227 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:54 smithi084 ceph-mon[39784]: osdmap e591: 12 total, 12 up, 9 in 2024-05-15T23:30:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:54 smithi090 ceph-mon[44434]: osdmap e591: 12 total, 12 up, 9 in 2024-05-15T23:30:54.566 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:30:54.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:54 smithi088 ceph-mon[44699]: osdmap e591: 12 total, 12 up, 9 in 2024-05-15T23:30:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:54.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:30:54.675 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:30:54.701 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:54.721 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:30:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:54.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:30:54.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:30:54.876 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:55.242 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:55 smithi090 ceph-mon[44434]: pgmap v1579: 17 pgs: 2 active+recovering, 15 active+clean; 8.0 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 755 op/s; 8581/262988 objects misplaced (3.263%); 3.8 MiB/s, 60 objects/s recovering 2024-05-15T23:30:55.346 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:30:55.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:55.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:30:55.452 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:30:55.478 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:55.497 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:30:55.499 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:55 smithi088 ceph-mon[44699]: pgmap v1579: 17 pgs: 2 active+recovering, 15 active+clean; 8.0 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 755 op/s; 8581/262988 objects misplaced (3.263%); 3.8 MiB/s, 60 objects/s recovering 2024-05-15T23:30:55.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:55.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:30:55.548 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:30:55.581 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:30:55.600 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:30:55.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:55 smithi084 ceph-mon[39784]: pgmap v1579: 17 pgs: 2 active+recovering, 15 active+clean; 8.0 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 755 op/s; 8581/262988 objects misplaced (3.263%); 3.8 MiB/s, 60 objects/s recovering 2024-05-15T23:30:55.620 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:30:55.627 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:55.647 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:30:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:55.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:30:55.660 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:30:55.723 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:30:55.731 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:30:55.753 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:30:55.756 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:30:55.782 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:30:55.817 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:30:55.840 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:30:55.874 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:55.898 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:30:55.936 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:30:55.946 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:30:55.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:30:55.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:30:55.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:30:55.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:30:55.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:30:55.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:30:55.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:30:55.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:30:55.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:30:55.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:30:55.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:30:56.240 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:30:56.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:56.243 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:30:56.246 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1782398304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:30:56.246 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:56.246 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:56.246 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:56.246 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:56.246 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:56.344 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:30:56.371 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:56.389 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:30:56.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:56.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:30:56.493 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:30:56.517 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1782398304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:30:56.517 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:56.518 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:56.518 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:56.518 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:56.518 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:56.519 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:30:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:56.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:30:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1782398304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:30:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:56.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:56.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:56.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:30:56.680 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:56.946 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:30:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:56.955 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:30:57.056 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:30:57.083 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:57.101 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:30:57.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:57.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:30:57.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:30:57.242 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:57.321 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:30:57.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:57.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:30:57.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:57 smithi084 ceph-mon[39784]: pgmap v1580: 17 pgs: 2 active+recovering, 15 active+clean; 7.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 874 op/s; 8049/257252 objects misplaced (3.129%); 7.1 MiB/s, 112 objects/s recovering 2024-05-15T23:30:57.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:30:57.459 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:57 smithi090 ceph-mon[44434]: pgmap v1580: 17 pgs: 2 active+recovering, 15 active+clean; 7.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 874 op/s; 8049/257252 objects misplaced (3.129%); 7.1 MiB/s, 112 objects/s recovering 2024-05-15T23:30:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:30:57.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:57 smithi088 ceph-mon[44699]: pgmap v1580: 17 pgs: 2 active+recovering, 15 active+clean; 7.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 874 op/s; 8049/257252 objects misplaced (3.129%); 7.1 MiB/s, 112 objects/s recovering 2024-05-15T23:30:57.615 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:30:57.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:57.619 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:30:57.720 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:30:57.749 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:57.769 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:30:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:57.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:30:57.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:30:57.913 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:57.940 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:30:57.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:57.953 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:30:58.053 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:30:58.080 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:30:58.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:58.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:30:58.206 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:30:58.232 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:58.253 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:30:58.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:58.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:30:58.357 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:30:58.398 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:58.529 INFO:teuthology.orchestra.run.smithi084.stderr:set 7.8 pg_upmap mapping to [6,0] 2024-05-15T23:30:58.649 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:30:58.649 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:30:58.649 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:30:58.649 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:30:58.649 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:30:58.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:58 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3421256864' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.8", "id": [6, 0]}]: dispatch 2024-05-15T23:30:58.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:58 smithi090 ceph-mon[44434]: from='client.36713 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.8", "id": [6, 0]}]: dispatch 2024-05-15T23:30:58.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:58 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3421256864' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.8", "id": [6, 0]}]: dispatch 2024-05-15T23:30:58.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:58 smithi088 ceph-mon[44699]: from='client.36713 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.8", "id": [6, 0]}]: dispatch 2024-05-15T23:30:58.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:58 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3421256864' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.8", "id": [6, 0]}]: dispatch 2024-05-15T23:30:58.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:58 smithi084 ceph-mon[39784]: from='client.36713 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.8", "id": [6, 0]}]: dispatch 2024-05-15T23:30:58.975 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:30:58.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:58.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:30:59.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:30:59.119 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:30:59.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:59.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:30:59.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T23:30:59.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:30:59.635 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:59 smithi084 ceph-mon[39784]: from='client.36713 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.8", "id": [6, 0]}]': finished 2024-05-15T23:30:59.635 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:59 smithi084 ceph-mon[39784]: osdmap e592: 12 total, 12 up, 9 in 2024-05-15T23:30:59.636 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:30:59 smithi084 ceph-mon[39784]: pgmap v1582: 17 pgs: 2 active+recovering, 15 active+clean; 7.8 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 855 op/s; 8049/254556 objects misplaced (3.162%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:30:59.640 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:59.662 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:30:59.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:59.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:30:59.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:30:59.775 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:30:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:59 smithi090 ceph-mon[44434]: from='client.36713 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.8", "id": [6, 0]}]': finished 2024-05-15T23:30:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:59 smithi090 ceph-mon[44434]: osdmap e592: 12 total, 12 up, 9 in 2024-05-15T23:30:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:30:59 smithi090 ceph-mon[44434]: pgmap v1582: 17 pgs: 2 active+recovering, 15 active+clean; 7.8 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 855 op/s; 8049/254556 objects misplaced (3.162%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:30:59.802 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:30:59.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:59 smithi088 ceph-mon[44699]: from='client.36713 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.8", "id": [6, 0]}]': finished 2024-05-15T23:30:59.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:59 smithi088 ceph-mon[44699]: osdmap e592: 12 total, 12 up, 9 in 2024-05-15T23:30:59.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:30:59 smithi088 ceph-mon[44699]: pgmap v1582: 17 pgs: 2 active+recovering, 15 active+clean; 7.8 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 855 op/s; 8049/254556 objects misplaced (3.162%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:30:59.832 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:30:59.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:30:59.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:30:59.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:30:59.955 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:30:59.981 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:00.017 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:31:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:00.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:31:00.129 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:31:00.179 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:00.215 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:31:00.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:00.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:31:00.245 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:31:00.320 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:31:00.391 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:00.542 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:31:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:00.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:31:00.597 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:31:00.647 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:31:00.726 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:00.849 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:31:00.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:00.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:31:00.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:31:00.995 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:01 smithi090 ceph-mon[44434]: pgmap v1583: 17 pgs: 2 active+recovering, 15 active+clean; 7.7 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 688 op/s; 7802/253062 objects misplaced (3.083%); 6.1 MiB/s, 97 objects/s recovering 2024-05-15T23:31:01.538 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:31:01.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:01.542 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:31:01.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:01 smithi088 ceph-mon[44699]: pgmap v1583: 17 pgs: 2 active+recovering, 15 active+clean; 7.7 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 688 op/s; 7802/253062 objects misplaced (3.083%); 6.1 MiB/s, 97 objects/s recovering 2024-05-15T23:31:01.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:01 smithi084 ceph-mon[39784]: pgmap v1583: 17 pgs: 2 active+recovering, 15 active+clean; 7.7 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 688 op/s; 7802/253062 objects misplaced (3.083%); 6.1 MiB/s, 97 objects/s recovering 2024-05-15T23:31:01.643 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:31:01.678 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:01.703 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:31:01.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:01.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:31:01.807 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:31:01.839 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:01.858 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:31:01.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:01.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:31:01.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:31:01.991 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:02.646 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:31:02.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:02.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:31:02.753 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:31:02.779 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:02.797 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:31:02.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:02.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:31:02.902 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:31:02.930 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:02.947 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:31:02.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:02.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:31:03.052 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:31:03.078 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:03.097 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:31:03.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:03.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:31:03.200 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:03.247 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:31:03.248 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:31:03.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:03.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:31:03.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:03 smithi090 ceph-mon[44434]: pgmap v1584: 17 pgs: 2 active+recovering, 15 active+clean; 7.6 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 839 op/s; 7484/247454 objects misplaced (3.024%); 6.9 MiB/s, 110 objects/s recovering 2024-05-15T23:31:03.350 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:31:03.374 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:03 smithi088 ceph-mon[44699]: pgmap v1584: 17 pgs: 2 active+recovering, 15 active+clean; 7.6 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 839 op/s; 7484/247454 objects misplaced (3.024%); 6.9 MiB/s, 110 objects/s recovering 2024-05-15T23:31:03.376 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:03.394 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:31:03.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:03.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:31:03.498 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:31:03.527 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:03.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:03 smithi084 ceph-mon[39784]: pgmap v1584: 17 pgs: 2 active+recovering, 15 active+clean; 7.6 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 839 op/s; 7484/247454 objects misplaced (3.024%); 6.9 MiB/s, 110 objects/s recovering 2024-05-15T23:31:03.545 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:31:03.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:03.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:31:03.649 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:31:03.677 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:03.695 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:31:03.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:03.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:31:03.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:31:03.832 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:04.192 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:31:04.192 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:31:04.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:31:04.192 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:31:04.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.193 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:137274 5.2 5:46696d63:::benchmark_data_smithi090_116470_object137273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e407)", 2024-05-15T23:31:04.193 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:21:09.055619+0000", 2024-05-15T23:31:04.193 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 595.13319377599998, 2024-05-15T23:31:04.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22972378399999999, 2024-05-15T23:31:04.193 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.193 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.193 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.193 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.193 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:31:04.193 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:31:04.193 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 137274 2024-05-15T23:31:04.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.193 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055619+0000", 2024-05-15T23:31:04.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055619+0000", 2024-05-15T23:31:04.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055620+0000", 2024-05-15T23:31:04.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9299999999999998e-07 2024-05-15T23:31:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055661+0000", 2024-05-15T23:31:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1100999999999998e-05 2024-05-15T23:31:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055667+0000", 2024-05-15T23:31:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3500000000000002e-06 2024-05-15T23:31:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055671+0000", 2024-05-15T23:31:04.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0289999999999999e-06 2024-05-15T23:31:04.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.057101+0000", 2024-05-15T23:31:04.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001429804 2024-05-15T23:31:04.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:31:04.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.057105+0000", 2024-05-15T23:31:04.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8840000000000004e-06 2024-05-15T23:31:04.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.245759+0000", 2024-05-15T23:31:04.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18865458700000001 2024-05-15T23:31:04.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.245843+0000", 2024-05-15T23:31:04.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4169e-05 2024-05-15T23:31:04.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:04.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.245943+0000", 2024-05-15T23:31:04.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9207000000000001e-05 2024-05-15T23:31:04.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.251870+0000", 2024-05-15T23:31:04.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0059269099999999996 2024-05-15T23:31:04.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.283457+0000", 2024-05-15T23:31:04.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031587478000000002 2024-05-15T23:31:04.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.283516+0000", 2024-05-15T23:31:04.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8899e-05 2024-05-15T23:31:04.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.285343+0000", 2024-05-15T23:31:04.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018268729999999999 2024-05-15T23:31:04.199 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.199 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.199 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.199 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:137275 5.2 5:47612e21:::benchmark_data_smithi090_116470_object137274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e407)", 2024-05-15T23:31:04.199 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:21:09.055676+0000", 2024-05-15T23:31:04.199 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 595.13313645300002, 2024-05-15T23:31:04.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.231978464, 2024-05-15T23:31:04.200 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.200 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.200 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.200 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.200 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:31:04.200 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:31:04.200 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 137275 2024-05-15T23:31:04.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.200 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055676+0000", 2024-05-15T23:31:04.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055676+0000", 2024-05-15T23:31:04.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055677+0000", 2024-05-15T23:31:04.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3000000000000001e-07 2024-05-15T23:31:04.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055718+0000", 2024-05-15T23:31:04.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1638999999999999e-05 2024-05-15T23:31:04.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055725+0000", 2024-05-15T23:31:04.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0759999999999999e-06 2024-05-15T23:31:04.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.055728+0000", 2024-05-15T23:31:04.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6449999999999998e-06 2024-05-15T23:31:04.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.057125+0000", 2024-05-15T23:31:04.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001396792 2024-05-15T23:31:04.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:31:04.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.057129+0000", 2024-05-15T23:31:04.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8010000000000001e-06 2024-05-15T23:31:04.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.247847+0000", 2024-05-15T23:31:04.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19071782100000001 2024-05-15T23:31:04.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.247962+0000", 2024-05-15T23:31:04.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011519 2024-05-15T23:31:04.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:04.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.248074+0000", 2024-05-15T23:31:04.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111953 2024-05-15T23:31:04.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.251873+0000", 2024-05-15T23:31:04.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037992970000000001 2024-05-15T23:31:04.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.285414+0000", 2024-05-15T23:31:04.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033540984000000003 2024-05-15T23:31:04.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.285472+0000", 2024-05-15T23:31:04.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7701000000000002e-05 2024-05-15T23:31:04.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:21:09.287655+0000", 2024-05-15T23:31:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021831350000000001 2024-05-15T23:31:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124079 7.1 7:857eceea:::benchmark_data_smithi090_143373_object124078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e579)", 2024-05-15T23:31:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.261908+0000", 2024-05-15T23:31:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.926904684999997, 2024-05-15T23:31:04.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26989557400000003, 2024-05-15T23:31:04.207 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.207 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.207 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.207 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.207 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.207 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.207 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124079 2024-05-15T23:31:04.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.208 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261908+0000", 2024-05-15T23:31:04.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261908+0000", 2024-05-15T23:31:04.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261911+0000", 2024-05-15T23:31:04.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4880000000000001e-06 2024-05-15T23:31:04.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261938+0000", 2024-05-15T23:31:04.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7135e-05 2024-05-15T23:31:04.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261952+0000", 2024-05-15T23:31:04.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4389999999999999e-05 2024-05-15T23:31:04.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261963+0000", 2024-05-15T23:31:04.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1094000000000001e-05 2024-05-15T23:31:04.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263369+0000", 2024-05-15T23:31:04.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014062160000000001 2024-05-15T23:31:04.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:31:04.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263378+0000", 2024-05-15T23:31:04.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4579999999999993e-06 2024-05-15T23:31:04.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.497009+0000", 2024-05-15T23:31:04.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23363141000000001 2024-05-15T23:31:04.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.499810+0000", 2024-05-15T23:31:04.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028003059999999998 2024-05-15T23:31:04.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.499987+0000", 2024-05-15T23:31:04.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176987 2024-05-15T23:31:04.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.510156+0000", 2024-05-15T23:31:04.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010169076000000001 2024-05-15T23:31:04.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.528308+0000", 2024-05-15T23:31:04.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018152166000000001 2024-05-15T23:31:04.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.528368+0000", 2024-05-15T23:31:04.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9679999999999998e-05 2024-05-15T23:31:04.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.531804+0000", 2024-05-15T23:31:04.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034361679999999999 2024-05-15T23:31:04.213 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.213 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.213 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.213 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124097 7.1 7:8475dbd8:::benchmark_data_smithi090_143373_object124096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:31:04.213 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.261987+0000", 2024-05-15T23:31:04.213 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.926825522999998, 2024-05-15T23:31:04.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28773406099999999, 2024-05-15T23:31:04.214 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.214 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.214 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.214 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.214 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.214 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.214 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124097 2024-05-15T23:31:04.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.214 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261987+0000", 2024-05-15T23:31:04.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261987+0000", 2024-05-15T23:31:04.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261988+0000", 2024-05-15T23:31:04.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3799999999999997e-07 2024-05-15T23:31:04.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262034+0000", 2024-05-15T23:31:04.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6520000000000002e-05 2024-05-15T23:31:04.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262041+0000", 2024-05-15T23:31:04.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5590000000000002e-06 2024-05-15T23:31:04.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262045+0000", 2024-05-15T23:31:04.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0110000000000002e-06 2024-05-15T23:31:04.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263404+0000", 2024-05-15T23:31:04.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013591110000000001 2024-05-15T23:31:04.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:31:04.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263410+0000", 2024-05-15T23:31:04.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780000000000003e-06 2024-05-15T23:31:04.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.506311+0000", 2024-05-15T23:31:04.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.242901425 2024-05-15T23:31:04.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.506407+0000", 2024-05-15T23:31:04.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5315999999999993e-05 2024-05-15T23:31:04.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.506505+0000", 2024-05-15T23:31:04.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8548999999999994e-05 2024-05-15T23:31:04.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.521629+0000", 2024-05-15T23:31:04.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015123319 2024-05-15T23:31:04.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.546928+0000", 2024-05-15T23:31:04.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025299687000000001 2024-05-15T23:31:04.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.546964+0000", 2024-05-15T23:31:04.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5552000000000001e-05 2024-05-15T23:31:04.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.549721+0000", 2024-05-15T23:31:04.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027575960000000002 2024-05-15T23:31:04.219 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.219 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.220 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124098 7.1 7:85fd872c:::benchmark_data_smithi090_143373_object124097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:31:04.220 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.262051+0000", 2024-05-15T23:31:04.220 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.926762142000001, 2024-05-15T23:31:04.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41649122300000002, 2024-05-15T23:31:04.220 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.220 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.220 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.220 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.220 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.220 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.220 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124098 2024-05-15T23:31:04.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.221 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262051+0000", 2024-05-15T23:31:04.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262051+0000", 2024-05-15T23:31:04.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262051+0000", 2024-05-15T23:31:04.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5699999999999998e-07 2024-05-15T23:31:04.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262088+0000", 2024-05-15T23:31:04.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6467000000000002e-05 2024-05-15T23:31:04.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262099+0000", 2024-05-15T23:31:04.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1422e-05 2024-05-15T23:31:04.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262111+0000", 2024-05-15T23:31:04.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1593000000000001e-05 2024-05-15T23:31:04.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263432+0000", 2024-05-15T23:31:04.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013212580000000001 2024-05-15T23:31:04.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:31:04.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263436+0000", 2024-05-15T23:31:04.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9310000000000001e-06 2024-05-15T23:31:04.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.510182+0000", 2024-05-15T23:31:04.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.24674654900000001 2024-05-15T23:31:04.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.510289+0000", 2024-05-15T23:31:04.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106941 2024-05-15T23:31:04.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.510417+0000", 2024-05-15T23:31:04.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012745399999999999 2024-05-15T23:31:04.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.521638+0000", 2024-05-15T23:31:04.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011221085 2024-05-15T23:31:04.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.675885+0000", 2024-05-15T23:31:04.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15424710599999999 2024-05-15T23:31:04.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.675949+0000", 2024-05-15T23:31:04.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4082000000000001e-05 2024-05-15T23:31:04.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.678542+0000", 2024-05-15T23:31:04.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025928779999999998 2024-05-15T23:31:04.226 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.226 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.226 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.226 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124106 7.1 7:8dcf5eef:::benchmark_data_smithi090_143373_object124105:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:31:04.226 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.262116+0000", 2024-05-15T23:31:04.226 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.926696763000002, 2024-05-15T23:31:04.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41946845300000002, 2024-05-15T23:31:04.227 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.227 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.227 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.227 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.227 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.227 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.227 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124106 2024-05-15T23:31:04.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.227 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262116+0000", 2024-05-15T23:31:04.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262116+0000", 2024-05-15T23:31:04.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262116+0000", 2024-05-15T23:31:04.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1399999999999997e-07 2024-05-15T23:31:04.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262149+0000", 2024-05-15T23:31:04.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2295999999999997e-05 2024-05-15T23:31:04.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262154+0000", 2024-05-15T23:31:04.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9130000000000002e-06 2024-05-15T23:31:04.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262158+0000", 2024-05-15T23:31:04.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0810000000000004e-06 2024-05-15T23:31:04.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263455+0000", 2024-05-15T23:31:04.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012971969999999999 2024-05-15T23:31:04.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:31:04.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263459+0000", 2024-05-15T23:31:04.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7859999999999998e-06 2024-05-15T23:31:04.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.513785+0000", 2024-05-15T23:31:04.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25032668899999999 2024-05-15T23:31:04.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.513908+0000", 2024-05-15T23:31:04.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012284899999999999 2024-05-15T23:31:04.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.514012+0000", 2024-05-15T23:31:04.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103525 2024-05-15T23:31:04.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.528239+0000", 2024-05-15T23:31:04.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014226946000000001 2024-05-15T23:31:04.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.678901+0000", 2024-05-15T23:31:04.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15066216900000001 2024-05-15T23:31:04.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.678961+0000", 2024-05-15T23:31:04.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0356000000000003e-05 2024-05-15T23:31:04.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.681585+0000", 2024-05-15T23:31:04.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026232320000000001 2024-05-15T23:31:04.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.232 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.233 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124122 7.1 7:88203a70:::benchmark_data_smithi090_143373_object124121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:31:04.233 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.262162+0000", 2024-05-15T23:31:04.233 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.926651024999998, 2024-05-15T23:31:04.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.42187396700000002, 2024-05-15T23:31:04.233 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.233 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.233 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.233 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.233 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.233 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.233 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124122 2024-05-15T23:31:04.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.234 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262162+0000", 2024-05-15T23:31:04.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262162+0000", 2024-05-15T23:31:04.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262162+0000", 2024-05-15T23:31:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2900000000000004e-07 2024-05-15T23:31:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262193+0000", 2024-05-15T23:31:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0460999999999999e-05 2024-05-15T23:31:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262208+0000", 2024-05-15T23:31:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5325e-05 2024-05-15T23:31:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262212+0000", 2024-05-15T23:31:04.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4809999999999998e-06 2024-05-15T23:31:04.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263485+0000", 2024-05-15T23:31:04.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001273176 2024-05-15T23:31:04.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:31:04.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263491+0000", 2024-05-15T23:31:04.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1480000000000003e-06 2024-05-15T23:31:04.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.516405+0000", 2024-05-15T23:31:04.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.252914099 2024-05-15T23:31:04.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.516538+0000", 2024-05-15T23:31:04.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013272499999999999 2024-05-15T23:31:04.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.516687+0000", 2024-05-15T23:31:04.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149693 2024-05-15T23:31:04.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.528245+0000", 2024-05-15T23:31:04.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011557596 2024-05-15T23:31:04.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.681654+0000", 2024-05-15T23:31:04.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15340879099999999 2024-05-15T23:31:04.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.681714+0000", 2024-05-15T23:31:04.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9806000000000003e-05 2024-05-15T23:31:04.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.684036+0000", 2024-05-15T23:31:04.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023221370000000002 2024-05-15T23:31:04.239 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.239 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.239 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.239 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124161 7.1 7:80dd3600:::benchmark_data_smithi090_143373_object124160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:31:04.239 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.262216+0000", 2024-05-15T23:31:04.239 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.926596695000001, 2024-05-15T23:31:04.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.42874030499999999, 2024-05-15T23:31:04.240 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.240 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.240 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.240 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124161 2024-05-15T23:31:04.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.240 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262216+0000", 2024-05-15T23:31:04.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262216+0000", 2024-05-15T23:31:04.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262217+0000", 2024-05-15T23:31:04.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8700000000000005e-07 2024-05-15T23:31:04.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262251+0000", 2024-05-15T23:31:04.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4162000000000003e-05 2024-05-15T23:31:04.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262256+0000", 2024-05-15T23:31:04.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5850000000000001e-06 2024-05-15T23:31:04.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262259+0000", 2024-05-15T23:31:04.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7790000000000002e-06 2024-05-15T23:31:04.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263546+0000", 2024-05-15T23:31:04.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012862450000000001 2024-05-15T23:31:04.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:31:04.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263552+0000", 2024-05-15T23:31:04.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9000000000000003e-06 2024-05-15T23:31:04.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.518749+0000", 2024-05-15T23:31:04.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25519711699999997 2024-05-15T23:31:04.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.518838+0000", 2024-05-15T23:31:04.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8896999999999999e-05 2024-05-15T23:31:04.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.518935+0000", 2024-05-15T23:31:04.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7421000000000001e-05 2024-05-15T23:31:04.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.535377+0000", 2024-05-15T23:31:04.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016442432 2024-05-15T23:31:04.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.687575+0000", 2024-05-15T23:31:04.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15219728799999999 2024-05-15T23:31:04.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.687629+0000", 2024-05-15T23:31:04.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4429999999999999e-05 2024-05-15T23:31:04.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.690956+0000", 2024-05-15T23:31:04.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033272620000000001 2024-05-15T23:31:04.246 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.246 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.246 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.246 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124174 7.1 7:856e5dbc:::benchmark_data_smithi090_143373_object124173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:31:04.246 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.262264+0000", 2024-05-15T23:31:04.246 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.926549170000001, 2024-05-15T23:31:04.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.43172603300000001, 2024-05-15T23:31:04.246 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.246 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.246 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.246 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.247 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.247 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.247 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124174 2024-05-15T23:31:04.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.247 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262264+0000", 2024-05-15T23:31:04.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262264+0000", 2024-05-15T23:31:04.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262264+0000", 2024-05-15T23:31:04.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0499999999999999e-07 2024-05-15T23:31:04.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262296+0000", 2024-05-15T23:31:04.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2254e-05 2024-05-15T23:31:04.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262301+0000", 2024-05-15T23:31:04.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.724e-06 2024-05-15T23:31:04.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262305+0000", 2024-05-15T23:31:04.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7469999999999999e-06 2024-05-15T23:31:04.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263583+0000", 2024-05-15T23:31:04.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012785819999999999 2024-05-15T23:31:04.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:31:04.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263587+0000", 2024-05-15T23:31:04.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.732e-06 2024-05-15T23:31:04.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.521673+0000", 2024-05-15T23:31:04.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25808625800000001 2024-05-15T23:31:04.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.521777+0000", 2024-05-15T23:31:04.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010374600000000001 2024-05-15T23:31:04.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.521947+0000", 2024-05-15T23:31:04.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017013099999999999 2024-05-15T23:31:04.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.535382+0000", 2024-05-15T23:31:04.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013435163 2024-05-15T23:31:04.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.691026+0000", 2024-05-15T23:31:04.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15564313299999999 2024-05-15T23:31:04.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.691082+0000", 2024-05-15T23:31:04.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6496000000000003e-05 2024-05-15T23:31:04.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.693990+0000", 2024-05-15T23:31:04.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029076620000000001 2024-05-15T23:31:04.252 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.252 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.252 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.252 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124212 7.1 7:81100074:::benchmark_data_smithi090_143373_object124211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.324932+0000", 2024-05-15T23:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.863881296000002, 2024-05-15T23:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37580514700000001, 2024-05-15T23:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124212 2024-05-15T23:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324932+0000", 2024-05-15T23:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324932+0000", 2024-05-15T23:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324933+0000", 2024-05-15T23:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.497e-06 2024-05-15T23:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324974+0000", 2024-05-15T23:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1214999999999998e-05 2024-05-15T23:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324984+0000", 2024-05-15T23:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7359999999999996e-06 2024-05-15T23:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324989+0000", 2024-05-15T23:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.259e-06 2024-05-15T23:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.325040+0000", 2024-05-15T23:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1172999999999999e-05 2024-05-15T23:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.325051+0000", 2024-05-15T23:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0317e-05 2024-05-15T23:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.524923+0000", 2024-05-15T23:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19987254500000001 2024-05-15T23:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.525080+0000", 2024-05-15T23:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015695999999999999 2024-05-15T23:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.525237+0000", 2024-05-15T23:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015635200000000001 2024-05-15T23:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.540733+0000", 2024-05-15T23:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01549625 2024-05-15T23:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.697259+0000", 2024-05-15T23:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15652601099999999 2024-05-15T23:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.697319+0000", 2024-05-15T23:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0210999999999997e-05 2024-05-15T23:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.700737+0000", 2024-05-15T23:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003417621 2024-05-15T23:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124224 7.1 7:81c65e53:::benchmark_data_smithi090_143373_object124223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.344409+0000", 2024-05-15T23:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.844403612999997, 2024-05-15T23:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.371311897, 2024-05-15T23:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124224 2024-05-15T23:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344409+0000", 2024-05-15T23:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344409+0000", 2024-05-15T23:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344412+0000", 2024-05-15T23:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3180000000000001e-06 2024-05-15T23:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344455+0000", 2024-05-15T23:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3247000000000001e-05 2024-05-15T23:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344466+0000", 2024-05-15T23:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0922e-05 2024-05-15T23:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344472+0000", 2024-05-15T23:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2790000000000004e-06 2024-05-15T23:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344531+0000", 2024-05-15T23:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9055000000000003e-05 2024-05-15T23:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344539+0000", 2024-05-15T23:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6040000000000004e-06 2024-05-15T23:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.531872+0000", 2024-05-15T23:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.187333584 2024-05-15T23:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.532039+0000", 2024-05-15T23:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016709099999999999 2024-05-15T23:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.532223+0000", 2024-05-15T23:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000183246 2024-05-15T23:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.545260+0000", 2024-05-15T23:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013037648000000001 2024-05-15T23:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.714171+0000", 2024-05-15T23:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.168910483 2024-05-15T23:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.714217+0000", 2024-05-15T23:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6461999999999997e-05 2024-05-15T23:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.715721+0000", 2024-05-15T23:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015039579999999999 2024-05-15T23:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124234 7.1 7:8b71c4c6:::benchmark_data_smithi090_143373_object124233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.366886+0000", 2024-05-15T23:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.821926699999999, 2024-05-15T23:31:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35046440299999998, 2024-05-15T23:31:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124234 2024-05-15T23:31:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366886+0000", 2024-05-15T23:31:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366886+0000", 2024-05-15T23:31:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366888+0000", 2024-05-15T23:31:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8920000000000001e-06 2024-05-15T23:31:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366934+0000", 2024-05-15T23:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5543000000000001e-05 2024-05-15T23:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366945+0000", 2024-05-15T23:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1345999999999999e-05 2024-05-15T23:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366951+0000", 2024-05-15T23:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3400000000000003e-06 2024-05-15T23:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.367009+0000", 2024-05-15T23:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7907e-05 2024-05-15T23:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.367022+0000", 2024-05-15T23:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2492e-05 2024-05-15T23:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.535417+0000", 2024-05-15T23:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16839585300000001 2024-05-15T23:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.535529+0000", 2024-05-15T23:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011196600000000001 2024-05-15T23:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.535644+0000", 2024-05-15T23:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011408099999999999 2024-05-15T23:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.545264+0000", 2024-05-15T23:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096206910000000007 2024-05-15T23:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.715777+0000", 2024-05-15T23:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17051261100000001 2024-05-15T23:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.715818+0000", 2024-05-15T23:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1119999999999999e-05 2024-05-15T23:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.717351+0000", 2024-05-15T23:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001532561 2024-05-15T23:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124269 7.1 7:84b5a5c1:::benchmark_data_smithi090_143373_object124268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.428950+0000", 2024-05-15T23:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.75986288, 2024-05-15T23:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31337896399999998, 2024-05-15T23:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124269 2024-05-15T23:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.428950+0000", 2024-05-15T23:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.428950+0000", 2024-05-15T23:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.428952+0000", 2024-05-15T23:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9989999999999998e-06 2024-05-15T23:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.428998+0000", 2024-05-15T23:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5986999999999999e-05 2024-05-15T23:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.429009+0000", 2024-05-15T23:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1008e-05 2024-05-15T23:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.429017+0000", 2024-05-15T23:31:04.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1110000000000008e-06 2024-05-15T23:31:04.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.429073+0000", 2024-05-15T23:31:04.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5998999999999997e-05 2024-05-15T23:31:04.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:31:04.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.429081+0000", 2024-05-15T23:31:04.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.943e-06 2024-05-15T23:31:04.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.538267+0000", 2024-05-15T23:31:04.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.109185886 2024-05-15T23:31:04.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.538366+0000", 2024-05-15T23:31:04.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8715000000000006e-05 2024-05-15T23:31:04.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.538461+0000", 2024-05-15T23:31:04.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5101999999999998e-05 2024-05-15T23:31:04.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.545270+0000", 2024-05-15T23:31:04.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0068090900000000003 2024-05-15T23:31:04.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.739358+0000", 2024-05-15T23:31:04.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.194088025 2024-05-15T23:31:04.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.739415+0000", 2024-05-15T23:31:04.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6719999999999999e-05 2024-05-15T23:31:04.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.742329+0000", 2024-05-15T23:31:04.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029143789999999999 2024-05-15T23:31:04.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.278 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.278 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124293 7.1 7:8a970b3a:::benchmark_data_smithi090_143373_object124292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:31:04.278 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.483940+0000", 2024-05-15T23:31:04.278 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.704872532000003, 2024-05-15T23:31:04.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27423620599999998, 2024-05-15T23:31:04.279 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.279 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.279 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.279 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124293 2024-05-15T23:31:04.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.279 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.483940+0000", 2024-05-15T23:31:04.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.483940+0000", 2024-05-15T23:31:04.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.483943+0000", 2024-05-15T23:31:04.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.322e-06 2024-05-15T23:31:04.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.483993+0000", 2024-05-15T23:31:04.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0241000000000002e-05 2024-05-15T23:31:04.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.484005+0000", 2024-05-15T23:31:04.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2024-05-15T23:31:04.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.484012+0000", 2024-05-15T23:31:04.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2849999999999997e-06 2024-05-15T23:31:04.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.540767+0000", 2024-05-15T23:31:04.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.056755011000000001 2024-05-15T23:31:04.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.540856+0000", 2024-05-15T23:31:04.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9073000000000006e-05 2024-05-15T23:31:04.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.540949+0000", 2024-05-15T23:31:04.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2831000000000006e-05 2024-05-15T23:31:04.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.545273+0000", 2024-05-15T23:31:04.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043239929999999999 2024-05-15T23:31:04.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.755350+0000", 2024-05-15T23:31:04.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.210076601 2024-05-15T23:31:04.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.755411+0000", 2024-05-15T23:31:04.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1271000000000003e-05 2024-05-15T23:31:04.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.758176+0000", 2024-05-15T23:31:04.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027657020000000001 2024-05-15T23:31:04.284 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.284 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.284 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.284 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124302 7.1 7:83d236b9:::benchmark_data_smithi090_143373_object124301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:31:04.284 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.505719+0000", 2024-05-15T23:31:04.284 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.683094189999998, 2024-05-15T23:31:04.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25507802400000001, 2024-05-15T23:31:04.284 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.284 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.284 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.284 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.285 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.285 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.285 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124302 2024-05-15T23:31:04.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.285 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505719+0000", 2024-05-15T23:31:04.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505719+0000", 2024-05-15T23:31:04.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505721+0000", 2024-05-15T23:31:04.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3760000000000002e-06 2024-05-15T23:31:04.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505778+0000", 2024-05-15T23:31:04.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6858999999999999e-05 2024-05-15T23:31:04.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505794+0000", 2024-05-15T23:31:04.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5926e-05 2024-05-15T23:31:04.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505804+0000", 2024-05-15T23:31:04.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0219e-05 2024-05-15T23:31:04.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.543199+0000", 2024-05-15T23:31:04.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.03739543 2024-05-15T23:31:04.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.543285+0000", 2024-05-15T23:31:04.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5523000000000001e-05 2024-05-15T23:31:04.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.543379+0000", 2024-05-15T23:31:04.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4110000000000005e-05 2024-05-15T23:31:04.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.545425+0000", 2024-05-15T23:31:04.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020457280000000001 2024-05-15T23:31:04.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.758248+0000", 2024-05-15T23:31:04.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.212822821 2024-05-15T23:31:04.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.758294+0000", 2024-05-15T23:31:04.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6774999999999999e-05 2024-05-15T23:31:04.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.760797+0000", 2024-05-15T23:31:04.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025022569999999999 2024-05-15T23:31:04.290 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.290 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.290 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.290 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:165512 7.1 7:81ed44b8:::benchmark_data_smithi090_143373_object19341:head [delete] snapc 0=[] ondisk+write+known_if_redirected e591)", 2024-05-15T23:31:04.290 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:57.374026+0000", 2024-05-15T23:31:04.290 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.8147866580000001, 2024-05-15T23:31:04.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.249226325, 2024-05-15T23:31:04.290 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.290 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.290 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.290 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.291 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.291 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.291 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 165512 2024-05-15T23:31:04.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.291 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374026+0000", 2024-05-15T23:31:04.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374026+0000", 2024-05-15T23:31:04.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374029+0000", 2024-05-15T23:31:04.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6060000000000001e-06 2024-05-15T23:31:04.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374037+0000", 2024-05-15T23:31:04.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3950000000000001e-06 2024-05-15T23:31:04.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374050+0000", 2024-05-15T23:31:04.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2826999999999999e-05 2024-05-15T23:31:04.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374060+0000", 2024-05-15T23:31:04.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9680000000000001e-06 2024-05-15T23:31:04.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.383163+0000", 2024-05-15T23:31:04.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091029890000000006 2024-05-15T23:31:04.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:31:04.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.383237+0000", 2024-05-15T23:31:04.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4456000000000007e-05 2024-05-15T23:31:04.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.567432+0000", 2024-05-15T23:31:04.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18419463999999999 2024-05-15T23:31:04.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.567524+0000", 2024-05-15T23:31:04.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2298000000000003e-05 2024-05-15T23:31:04.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.567612+0000", 2024-05-15T23:31:04.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7293999999999995e-05 2024-05-15T23:31:04.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.600864+0000", 2024-05-15T23:31:04.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033252149000000002 2024-05-15T23:31:04.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.620769+0000", 2024-05-15T23:31:04.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019905062000000001 2024-05-15T23:31:04.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.620810+0000", 2024-05-15T23:31:04.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0685000000000001e-05 2024-05-15T23:31:04.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.623252+0000", 2024-05-15T23:31:04.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002442956 2024-05-15T23:31:04.296 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.296 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.296 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.296 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:168938 7.1 7:87745b29:::benchmark_data_smithi090_143373_object22767:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:31:04.296 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:01.604430+0000", 2024-05-15T23:31:04.297 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.5843827460000002, 2024-05-15T23:31:04.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.246406922, 2024-05-15T23:31:04.297 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.297 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.297 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.297 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.297 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.297 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.297 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168938 2024-05-15T23:31:04.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.297 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604430+0000", 2024-05-15T23:31:04.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604430+0000", 2024-05-15T23:31:04.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604433+0000", 2024-05-15T23:31:04.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.661e-06 2024-05-15T23:31:04.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604442+0000", 2024-05-15T23:31:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1749999999999994e-06 2024-05-15T23:31:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604456+0000", 2024-05-15T23:31:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4354e-05 2024-05-15T23:31:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604468+0000", 2024-05-15T23:31:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1389000000000001e-05 2024-05-15T23:31:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.631305+0000", 2024-05-15T23:31:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02683758 2024-05-15T23:31:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:31:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.631363+0000", 2024-05-15T23:31:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8009e-05 2024-05-15T23:31:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.820785+0000", 2024-05-15T23:31:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.189421428 2024-05-15T23:31:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.820880+0000", 2024-05-15T23:31:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4852000000000006e-05 2024-05-15T23:31:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.821015+0000", 2024-05-15T23:31:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135709 2024-05-15T23:31:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.837594+0000", 2024-05-15T23:31:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016578446 2024-05-15T23:31:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.847339+0000", 2024-05-15T23:31:04.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0097453809999999991 2024-05-15T23:31:04.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.847400+0000", 2024-05-15T23:31:04.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1194999999999999e-05 2024-05-15T23:31:04.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.850837+0000", 2024-05-15T23:31:04.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034367429999999999 2024-05-15T23:31:04.302 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.303 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.303 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:168941 7.1 7:8a84f08b:::benchmark_data_smithi090_143373_object22770:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:31:04.303 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:01.604543+0000", 2024-05-15T23:31:04.303 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.5842698779999997, 2024-05-15T23:31:04.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.33646746300000002, 2024-05-15T23:31:04.303 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.303 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.303 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.303 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.303 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.303 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.304 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168941 2024-05-15T23:31:04.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.304 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604543+0000", 2024-05-15T23:31:04.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604543+0000", 2024-05-15T23:31:04.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604544+0000", 2024-05-15T23:31:04.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9500000000000001e-07 2024-05-15T23:31:04.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604548+0000", 2024-05-15T23:31:04.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0380000000000001e-06 2024-05-15T23:31:04.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604554+0000", 2024-05-15T23:31:04.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1940000000000003e-06 2024-05-15T23:31:04.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604559+0000", 2024-05-15T23:31:04.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8300000000000003e-06 2024-05-15T23:31:04.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634299+0000", 2024-05-15T23:31:04.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740566 2024-05-15T23:31:04.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:31:04.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634357+0000", 2024-05-15T23:31:04.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7884999999999999e-05 2024-05-15T23:31:04.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.916090+0000", 2024-05-15T23:31:04.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28173253799999998 2024-05-15T23:31:04.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.916206+0000", 2024-05-15T23:31:04.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116403 2024-05-15T23:31:04.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.916353+0000", 2024-05-15T23:31:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014637100000000001 2024-05-15T23:31:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.934169+0000", 2024-05-15T23:31:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017816117999999999 2024-05-15T23:31:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.939265+0000", 2024-05-15T23:31:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050958519999999997 2024-05-15T23:31:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.939322+0000", 2024-05-15T23:31:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7025999999999999e-05 2024-05-15T23:31:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.941010+0000", 2024-05-15T23:31:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001688747 2024-05-15T23:31:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPushReply(7.1 592/578 [PushReplyOp(7:8a84f08b:::benchmark_data_smithi090_143373_object22770:head)])", 2024-05-15T23:31:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:01.634777+0000", 2024-05-15T23:31:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.554035694, 2024-05-15T23:31:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28126164100000001, 2024-05-15T23:31:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-05-15T23:31:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634777+0000", 2024-05-15T23:31:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634777+0000", 2024-05-15T23:31:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634779+0000", 2024-05-15T23:31:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5099999999999999e-06 2024-05-15T23:31:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634789+0000", 2024-05-15T23:31:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0056e-05 2024-05-15T23:31:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634801+0000", 2024-05-15T23:31:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2532e-05 2024-05-15T23:31:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634809+0000", 2024-05-15T23:31:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8099999999999998e-06 2024-05-15T23:31:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.909849+0000", 2024-05-15T23:31:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27503971700000002 2024-05-15T23:31:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.916039+0000", 2024-05-15T23:31:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061900159999999996 2024-05-15T23:31:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:169812 7.1 7:817b8891:::benchmark_data_smithi090_143373_object23641:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:31:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:02.735245+0000", 2024-05-15T23:31:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.4535673929999999, 2024-05-15T23:31:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31279489399999999, 2024-05-15T23:31:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 169812 2024-05-15T23:31:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.735245+0000", 2024-05-15T23:31:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.735245+0000", 2024-05-15T23:31:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.735248+0000", 2024-05-15T23:31:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3549999999999999e-06 2024-05-15T23:31:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.735255+0000", 2024-05-15T23:31:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8029999999999999e-06 2024-05-15T23:31:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909550+0000", 2024-05-15T23:31:04.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.174295218 2024-05-15T23:31:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909565+0000", 2024-05-15T23:31:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-05-15T23:31:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909644+0000", 2024-05-15T23:31:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9839999999999995e-05 2024-05-15T23:31:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:31:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909718+0000", 2024-05-15T23:31:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3475999999999999e-05 2024-05-15T23:31:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.012591+0000", 2024-05-15T23:31:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102873279 2024-05-15T23:31:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.012700+0000", 2024-05-15T23:31:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108721 2024-05-15T23:31:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:31:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.012857+0000", 2024-05-15T23:31:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157305 2024-05-15T23:31:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.030528+0000", 2024-05-15T23:31:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017670788 2024-05-15T23:31:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.045291+0000", 2024-05-15T23:31:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014763108 2024-05-15T23:31:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.045348+0000", 2024-05-15T23:31:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6375000000000001e-05 2024-05-15T23:31:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.048040+0000", 2024-05-15T23:31:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002692776 2024-05-15T23:31:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:04.321 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:31:04.409 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10, 2] out_osds: [0, 6, 8] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:31:04.410 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:31:04.410 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-05-15T23:31:04.410 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-15T23:31:04.410 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-15T23:31:04.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 config set filestore_inject_stall 3 2024-05-15T23:31:04.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:04 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:31:04.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:04 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:31:04.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:04 smithi090 ceph-mon[44434]: osdmap e593: 12 total, 12 up, 9 in 2024-05-15T23:31:04.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:04 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:31:04.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:04 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:31:04.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:04 smithi084 ceph-mon[39784]: osdmap e593: 12 total, 12 up, 9 in 2024-05-15T23:31:04.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:04 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:31:04.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:04 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:31:04.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:04 smithi088 ceph-mon[44699]: osdmap e593: 12 total, 12 up, 9 in 2024-05-15T23:31:04.616 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:31:04.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:04.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:31:04.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:31:04.824 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:05.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:05 smithi090 ceph-mon[44434]: pgmap v1586: 17 pgs: 2 active+recovering, 15 active+clean; 7.5 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 672 op/s; 7484/246496 objects misplaced (3.036%); 4.4 MiB/s, 70 objects/s recovering 2024-05-15T23:31:05.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:05 smithi088 ceph-mon[44699]: pgmap v1586: 17 pgs: 2 active+recovering, 15 active+clean; 7.5 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 672 op/s; 7484/246496 objects misplaced (3.036%); 4.4 MiB/s, 70 objects/s recovering 2024-05-15T23:31:05.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:05 smithi084 ceph-mon[39784]: pgmap v1586: 17 pgs: 2 active+recovering, 15 active+clean; 7.5 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 672 op/s; 7484/246496 objects misplaced (3.036%); 4.4 MiB/s, 70 objects/s recovering 2024-05-15T23:31:05.908 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:31:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:05.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:31:05.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T23:31:06.001 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:31:06.013 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:31:06.069 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:06.287 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:31:06.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:06.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:31:06.358 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:31:06.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:31:06.459 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:06.506 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:31:06.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:06.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:31:06.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:06 smithi090 ceph-mon[44434]: from='client.26773 v1:172.21.15.84:0/849415330' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:31:06.517 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:06 smithi090 ceph-mon[44434]: from='client.26773 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:31:06.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:06 smithi088 ceph-mon[44699]: from='client.26773 v1:172.21.15.84:0/849415330' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:31:06.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:06 smithi088 ceph-mon[44699]: from='client.26773 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:31:06.616 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:31:06.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:06 smithi084 ceph-mon[39784]: from='client.26773 v1:172.21.15.84:0/849415330' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:31:06.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:06 smithi084 ceph-mon[39784]: from='client.26773 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:31:06.679 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:06.708 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:31:06.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:06.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:31:06.819 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:31:06.846 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:06.866 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:31:06.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:06.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:31:06.970 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:31:07.000 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:07.114 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:31:07.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:07.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:31:07.223 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:31:07.250 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:07.271 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:31:07.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:07.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:31:07.302 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:31:07.332 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:31:07.361 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:31:07.375 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:31:07.396 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:31:07.415 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:07.429 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:31:07.458 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:31:07.487 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:31:07.526 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:31:07.549 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:31:07.560 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:07 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:31:07.560 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:07 smithi090 ceph-mon[44434]: from='client.26773 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:31:07.560 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:07 smithi090 ceph-mon[44434]: osdmap e594: 12 total, 12 up, 9 in 2024-05-15T23:31:07.560 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:07 smithi090 ceph-mon[44434]: pgmap v1588: 17 pgs: 2 active+recovering, 15 active+clean; 7.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 828 op/s; 6859/241284 objects misplaced (2.843%); 9.3 MiB/s, 148 objects/s recovering 2024-05-15T23:31:07.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:07 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:31:07.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:07 smithi088 ceph-mon[44699]: from='client.26773 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:31:07.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:07 smithi088 ceph-mon[44699]: osdmap e594: 12 total, 12 up, 9 in 2024-05-15T23:31:07.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:07 smithi088 ceph-mon[44699]: pgmap v1588: 17 pgs: 2 active+recovering, 15 active+clean; 7.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 828 op/s; 6859/241284 objects misplaced (2.843%); 9.3 MiB/s, 148 objects/s recovering 2024-05-15T23:31:07.605 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:31:07.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:07 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:31:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:07 smithi084 ceph-mon[39784]: from='client.26773 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:31:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:07 smithi084 ceph-mon[39784]: osdmap e594: 12 total, 12 up, 9 in 2024-05-15T23:31:07.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:07 smithi084 ceph-mon[39784]: pgmap v1588: 17 pgs: 2 active+recovering, 15 active+clean; 7.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 828 op/s; 6859/241284 objects misplaced (2.843%); 9.3 MiB/s, 148 objects/s recovering 2024-05-15T23:31:07.625 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:31:07.655 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:31:07.661 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:31:07.661 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:31:07.661 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:31:07.661 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:31:07.661 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:31:07.661 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:31:07.661 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:31:07.662 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:31:07.662 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:31:07.662 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:31:07.662 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:31:07.662 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:31:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:31:08.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2904932629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:31:08.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:08 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:31:08.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2904932629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:31:08.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:08 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:31:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.578 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2904932629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:31:08.578 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.578 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.578 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.578 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.578 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.578 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:08 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:31:08.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:08.594 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:31:08.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:08.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:31:08.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:31:08.742 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:09.278 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:09.278 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:09.278 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:09.279 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:09 smithi090 ceph-mon[44434]: pgmap v1589: 17 pgs: 2 active+recovering, 15 active+clean; 7.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 892 op/s; 6859/238770 objects misplaced (2.873%); 7.4 MiB/s, 117 objects/s recovering 2024-05-15T23:31:09.306 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:31:09.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:09.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:31:09.410 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:31:09.436 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:09.458 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:31:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:09.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:31:09.561 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:31:09.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:09.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:09.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:09 smithi088 ceph-mon[44699]: pgmap v1589: 17 pgs: 2 active+recovering, 15 active+clean; 7.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 892 op/s; 6859/238770 objects misplaced (2.873%); 7.4 MiB/s, 117 objects/s recovering 2024-05-15T23:31:09.588 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:09.607 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:31:09.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:09.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:31:09.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:09.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:09.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:09.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:09 smithi084 ceph-mon[39784]: pgmap v1589: 17 pgs: 2 active+recovering, 15 active+clean; 7.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 892 op/s; 6859/238770 objects misplaced (2.873%); 7.4 MiB/s, 117 objects/s recovering 2024-05-15T23:31:09.691 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:31:09.691 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "" 2024-05-15T23:31:09.691 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:31:09.710 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:31:09.770 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:09.820 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:31:09.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:09.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:31:09.925 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:31:10.047 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:10.073 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:31:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:10.079 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:31:10.180 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:31:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:10.226 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:31:10.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:10.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:31:10.330 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:31:10.357 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:10.378 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:31:10.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:10.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:31:10.482 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:31:10.508 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:10.529 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:31:10.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:10.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:31:10.615 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:31:10.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:31:10.673 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:11.066 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:31:11.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:31:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:11 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:31:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:11 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:31:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:11 smithi084 ceph-mon[39784]: pgmap v1590: 17 pgs: 2 active+recovering, 15 active+clean; 7.2 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 635 op/s; 6616/237282 objects misplaced (2.788%); 6.8 MiB/s, 108 objects/s recovering 2024-05-15T23:31:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:11 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:31:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:11 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:31:12.036 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:11 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:31:12.036 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:11 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:31:12.036 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:11 smithi090 ceph-mon[44434]: pgmap v1590: 17 pgs: 2 active+recovering, 15 active+clean; 7.2 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 635 op/s; 6616/237282 objects misplaced (2.788%); 6.8 MiB/s, 108 objects/s recovering 2024-05-15T23:31:12.036 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:11 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:31:12.036 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:11 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:31:12.042 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:31:12.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:12.047 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:31:12.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:11 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:31:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:11 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:31:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:11 smithi088 ceph-mon[44699]: pgmap v1590: 17 pgs: 2 active+recovering, 15 active+clean; 7.2 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 635 op/s; 6616/237282 objects misplaced (2.788%); 6.8 MiB/s, 108 objects/s recovering 2024-05-15T23:31:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:11 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:31:12.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:11 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:31:12.147 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:31:12.188 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:12.412 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:31:12.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:12.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:31:12.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:31:12.557 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:12.650 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:31:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:12.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:31:12.756 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:31:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:12 smithi090 ceph-mon[44434]: osdmap e595: 12 total, 12 up, 9 in 2024-05-15T23:31:12.806 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:12.834 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:31:12.834 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:31:12.834 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:31:12.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:12.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:31:12.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:31:13.020 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:13.040 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:31:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:13.042 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:31:13.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:12 smithi088 ceph-mon[44699]: osdmap e595: 12 total, 12 up, 9 in 2024-05-15T23:31:13.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:12 smithi084 ceph-mon[39784]: osdmap e595: 12 total, 12 up, 9 in 2024-05-15T23:31:13.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:31:13.195 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:13.223 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:31:13.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:13.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:31:13.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:31:13.361 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:13.852 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:13 smithi084 ceph-mon[39784]: pgmap v1592: 17 pgs: 2 active+recovering, 15 active+clean; 7.1 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 917 op/s; 6263/231808 objects misplaced (2.702%); 9.5 MiB/s, 152 objects/s recovering 2024-05-15T23:31:13.911 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:31:14.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:13 smithi090 ceph-mon[44434]: pgmap v1592: 17 pgs: 2 active+recovering, 15 active+clean; 7.1 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 917 op/s; 6263/231808 objects misplaced (2.702%); 9.5 MiB/s, 152 objects/s recovering 2024-05-15T23:31:14.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:13 smithi088 ceph-mon[44699]: pgmap v1592: 17 pgs: 2 active+recovering, 15 active+clean; 7.1 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 917 op/s; 6263/231808 objects misplaced (2.702%); 9.5 MiB/s, 152 objects/s recovering 2024-05-15T23:31:14.720 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:31:14.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:14.728 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:31:14.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:31:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:15.033 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:14 smithi084 ceph-mon[39784]: 7.6 scrub starts 2024-05-15T23:31:15.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:14 smithi090 ceph-mon[44434]: 7.6 scrub starts 2024-05-15T23:31:15.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:14 smithi088 ceph-mon[44699]: 7.6 scrub starts 2024-05-15T23:31:15.248 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:31:15.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:15.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:31:15.305 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T23:31:15.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:31:15.450 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:15.480 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:31:15.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:15.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:31:15.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:31:15.637 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:15.673 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:31:15.752 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:31:16.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:15 smithi090 ceph-mon[44434]: 7.6 scrub ok 2024-05-15T23:31:16.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:15 smithi090 ceph-mon[44434]: pgmap v1593: 17 pgs: 2 active+recovering, 15 active+clean; 7.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 730 op/s; 6263/230744 objects misplaced (2.714%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:31:16.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:15 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/130984049' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:31:16.022 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:15 smithi090 ceph-mon[44434]: from='client.36728 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:31:16.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:15 smithi088 ceph-mon[44699]: 7.6 scrub ok 2024-05-15T23:31:16.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:15 smithi088 ceph-mon[44699]: pgmap v1593: 17 pgs: 2 active+recovering, 15 active+clean; 7.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 730 op/s; 6263/230744 objects misplaced (2.714%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:31:16.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:15 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/130984049' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:31:16.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:15 smithi088 ceph-mon[44699]: from='client.36728 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:31:16.106 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:31:16.108 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:15 smithi084 ceph-mon[39784]: 7.6 scrub ok 2024-05-15T23:31:16.108 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:15 smithi084 ceph-mon[39784]: pgmap v1593: 17 pgs: 2 active+recovering, 15 active+clean; 7.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 730 op/s; 6263/230744 objects misplaced (2.714%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:31:16.108 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:15 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/130984049' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:31:16.108 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:15 smithi084 ceph-mon[39784]: from='client.36728 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:31:16.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:16.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:31:16.215 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:31:16.287 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:16.340 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:31:16.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:16.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:31:16.446 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:31:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:16.494 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:31:16.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:16.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:31:16.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:31:16.607 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10, 2] out_osds: [0, 6, 8] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:31:16.608 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:31:16.608 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-05-15T23:31:16.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump -f json-pretty 2024-05-15T23:31:16.676 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:16.708 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:31:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:16.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:31:16.816 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:31:16.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:16 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:31:16.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:16 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:31:16.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:16 smithi088 ceph-mon[44699]: from='client.36728 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:31:16.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:16 smithi088 ceph-mon[44699]: osdmap e596: 12 total, 12 up, 9 in 2024-05-15T23:31:16.881 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:16.924 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:31:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:16.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:31:16.935 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:16 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:31:16.935 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:16 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:31:16.935 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:16 smithi084 ceph-mon[39784]: from='client.36728 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:31:16.935 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:16 smithi084 ceph-mon[39784]: osdmap e596: 12 total, 12 up, 9 in 2024-05-15T23:31:17.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:31:17.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:16 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:31:17.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:16 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:31:17.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:16 smithi090 ceph-mon[44434]: from='client.36728 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:31:17.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:16 smithi090 ceph-mon[44434]: osdmap e596: 12 total, 12 up, 9 in 2024-05-15T23:31:17.086 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:17.192 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:31:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:17.197 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:31:17.298 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:31:17.339 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:17.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:17 smithi090 ceph-mon[44434]: pgmap v1595: 17 pgs: 2 active+recovering, 15 active+clean; 6.9 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 853 op/s; 5654/225116 objects misplaced (2.512%); 9.4 MiB/s, 150 objects/s recovering 2024-05-15T23:31:17.938 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:31:17.939 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:31:17.971 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:17 smithi084 ceph-mon[39784]: pgmap v1595: 17 pgs: 2 active+recovering, 15 active+clean; 6.9 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 853 op/s; 5654/225116 objects misplaced (2.512%); 9.4 MiB/s, 150 objects/s recovering 2024-05-15T23:31:18.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:17 smithi088 ceph-mon[44699]: pgmap v1595: 17 pgs: 2 active+recovering, 15 active+clean; 6.9 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 853 op/s; 5654/225116 objects misplaced (2.512%); 9.4 MiB/s, 150 objects/s recovering 2024-05-15T23:31:18.712 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:31:18.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:18.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:31:18.747 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T23:31:18.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:31:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:18.894 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:31:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:18.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:31:18.979 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:18 smithi084 ceph-mon[39784]: 7.a scrub starts 2024-05-15T23:31:19.005 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:31:19.029 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:18 smithi088 ceph-mon[44699]: 7.a scrub starts 2024-05-15T23:31:19.032 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:19.055 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:31:19.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:18 smithi090 ceph-mon[44434]: 7.a scrub starts 2024-05-15T23:31:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:19.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:31:19.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:31:19.219 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:19.219 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:31:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:31:19.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:19.262 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:31:19.361 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:31:19.363 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:31:19.394 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:31:19.416 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:19.423 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:31:19.448 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:31:19.481 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:31:19.510 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:31:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:19.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:31:19.523 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:31:19.574 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:31:19.620 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:31:19.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:31:19.636 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:31:19.680 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:31:19.690 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:19.720 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:31:19.761 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:31:19.797 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:31:19.802 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:31:19.802 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:31:19.802 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:31:19.802 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:31:19.802 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:31:19.802 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:31:19.802 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:31:19.802 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:31:19.802 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:31:19.802 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:31:19.803 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:31:19.803 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:31:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:19 smithi090 ceph-mon[44434]: pgmap v1596: 17 pgs: 2 active+recovering, 15 active+clean; 6.8 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 920 op/s; 5654/222548 objects misplaced (2.541%); 7.5 MiB/s, 120 objects/s recovering 2024-05-15T23:31:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2641358441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:31:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:19 smithi090 ceph-mon[44434]: 7.f scrub starts 2024-05-15T23:31:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:19.827 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:19 smithi084 ceph-mon[39784]: pgmap v1596: 17 pgs: 2 active+recovering, 15 active+clean; 6.8 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 920 op/s; 5654/222548 objects misplaced (2.541%); 7.5 MiB/s, 120 objects/s recovering 2024-05-15T23:31:19.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2641358441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:31:19.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:19 smithi084 ceph-mon[39784]: 7.f scrub starts 2024-05-15T23:31:19.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:19.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:19.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:19.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:31:20.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:19 smithi088 ceph-mon[44699]: pgmap v1596: 17 pgs: 2 active+recovering, 15 active+clean; 6.8 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 920 op/s; 5654/222548 objects misplaced (2.541%); 7.5 MiB/s, 120 objects/s recovering 2024-05-15T23:31:20.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2641358441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:31:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:19 smithi088 ceph-mon[44699]: 7.f scrub starts 2024-05-15T23:31:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:20 smithi090 ceph-mon[44434]: 7.4 scrub starts 2024-05-15T23:31:21.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:20 smithi090 ceph-mon[44434]: 7.a scrub ok 2024-05-15T23:31:21.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:20 smithi088 ceph-mon[44699]: 7.4 scrub starts 2024-05-15T23:31:21.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:20 smithi088 ceph-mon[44699]: 7.a scrub ok 2024-05-15T23:31:21.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:20 smithi084 ceph-mon[39784]: 7.4 scrub starts 2024-05-15T23:31:21.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:20 smithi084 ceph-mon[39784]: 7.a scrub ok 2024-05-15T23:31:21.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:21.385 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:31:21.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:21.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:31:21.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:31:21.530 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:22.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:21 smithi090 ceph-mon[44434]: 7.4 scrub ok 2024-05-15T23:31:22.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:21 smithi090 ceph-mon[44434]: 7.f scrub ok 2024-05-15T23:31:22.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:21 smithi090 ceph-mon[44434]: pgmap v1597: 17 pgs: 2 active+recovering, 15 active+clean; 6.7 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 642 op/s; 5405/221084 objects misplaced (2.445%); 5.6 MiB/s, 90 objects/s recovering 2024-05-15T23:31:22.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:21 smithi088 ceph-mon[44699]: 7.4 scrub ok 2024-05-15T23:31:22.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:21 smithi088 ceph-mon[44699]: 7.f scrub ok 2024-05-15T23:31:22.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:21 smithi088 ceph-mon[44699]: pgmap v1597: 17 pgs: 2 active+recovering, 15 active+clean; 6.7 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 642 op/s; 5405/221084 objects misplaced (2.445%); 5.6 MiB/s, 90 objects/s recovering 2024-05-15T23:31:22.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:21 smithi084 ceph-mon[39784]: 7.4 scrub ok 2024-05-15T23:31:22.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:21 smithi084 ceph-mon[39784]: 7.f scrub ok 2024-05-15T23:31:22.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:21 smithi084 ceph-mon[39784]: pgmap v1597: 17 pgs: 2 active+recovering, 15 active+clean; 6.7 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 642 op/s; 5405/221084 objects misplaced (2.445%); 5.6 MiB/s, 90 objects/s recovering 2024-05-15T23:31:22.704 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:31:22.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:22.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:31:22.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:31:22.842 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:23.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:22 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:31:23.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:22 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:31:23.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:22 smithi090 ceph-mon[44434]: osdmap e597: 12 total, 12 up, 9 in 2024-05-15T23:31:23.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:22 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:31:23.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:22 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:31:23.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:22 smithi088 ceph-mon[44699]: osdmap e597: 12 total, 12 up, 9 in 2024-05-15T23:31:23.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:22 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:31:23.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:22 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:31:23.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:22 smithi084 ceph-mon[39784]: osdmap e597: 12 total, 12 up, 9 in 2024-05-15T23:31:23.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:23 smithi084 ceph-mon[39784]: pgmap v1599: 17 pgs: 1 active+recovering, 16 active+clean; 6.6 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 948 op/s; 5356/215550 objects misplaced (2.485%); 7.1 MiB/s, 113 objects/s recovering 2024-05-15T23:31:24.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:23 smithi090 ceph-mon[44434]: pgmap v1599: 17 pgs: 1 active+recovering, 16 active+clean; 6.6 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 948 op/s; 5356/215550 objects misplaced (2.485%); 7.1 MiB/s, 113 objects/s recovering 2024-05-15T23:31:24.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:23 smithi088 ceph-mon[44699]: pgmap v1599: 17 pgs: 1 active+recovering, 16 active+clean; 6.6 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 948 op/s; 5356/215550 objects misplaced (2.485%); 7.1 MiB/s, 113 objects/s recovering 2024-05-15T23:31:24.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:31:24.456 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:31:24.602 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:31:24.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:24.611 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:31:24.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:31:24.773 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:24.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:24 smithi090 ceph-mon[44434]: 7.8 deep-scrub starts 2024-05-15T23:31:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:31:24.881 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:31:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 597, 2024-05-15T23:31:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "3999fb7e-130e-11ef-bc99-c7b262605968", 2024-05-15T23:31:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-05-15T22:56:41.212493+0000", 2024-05-15T23:31:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-05-15T23:31:21.681471+0000", 2024-05-15T23:31:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-05-15T23:26:58.424794+0000", 2024-05-15T23:31:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-05-15T23:28:16.134275+0000", 2024-05-15T23:31:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-15T23:31:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5799936, 2024-05-15T23:31:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-05-15T23:31:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-05-15T23:31:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-05-15T23:31:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-05-15T23:31:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-05-15T23:31:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:31:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 30, 2024-05-15T23:31:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-05-15T23:31:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-15T23:31:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-15T23:31:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-05-15T23:31:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 7, 2024-05-15T23:31:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 12, 2024-05-15T23:31:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-05-15T23:31:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-05-15T23:31:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-05-15T23:31:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-05-15T23:31:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-05-15T23:31:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-05-15T23:31:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-05-15T23:31:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-05-15T22:59:38.603575+0000", 2024-05-15T23:31:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-05-15T23:31:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-05-15T23:31:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-05-15T23:31:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2024-05-15T23:31:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-05-15T23:31:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-05-15T23:31:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-05-15T23:31:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-05-15T23:31:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T23:31:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T23:31:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-05-15T23:31:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-05-15T23:31:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 1, 2024-05-15T23:31:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 1, 2024-05-15T23:31:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 1, 2024-05-15T23:31:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 1, 2024-05-15T23:31:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 1, 2024-05-15T23:31:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-05-15T23:31:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-05-15T23:31:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-05-15T23:31:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-05-15T23:31:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-05-15T23:31:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-05-15T23:31:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-05-15T23:31:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "16", 2024-05-15T23:31:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-05-15T23:31:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T23:31:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T23:31:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-05-15T23:31:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-05-15T23:31:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-05-15T23:31:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-05-15T23:31:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-05-15T23:31:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-05-15T23:31:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-05-15T23:31:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-05-15T23:31:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-05-15T23:31:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-05-15T23:31:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-05-15T23:31:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-05-15T23:31:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-05-15T23:31:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-05-15T23:31:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-05-15T23:31:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T23:31:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T23:31:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T23:31:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-05-15T23:31:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-05-15T23:31:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-05-15T23:31:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-05-15T23:31:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-05-15T23:31:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-05-15T23:31:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-05-15T23:31:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-05-15T23:31:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-05-15T23:31:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-05-15T23:31:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T23:31:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-05-15T23:31:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-05-15T23:31:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-05-15T23:31:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-05-15T23:31:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-05-15T23:31:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-05-15T23:31:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-05-15T23:31:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-05-15T23:31:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-05-15T23:31:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-05-15T23:31:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-05-15T23:31:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-05-15T23:31:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 11.760000228881836, 2024-05-15T23:31:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 11.760000228881836, 2024-05-15T23:31:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.17000000178813934, 2024-05-15T23:31:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-05-15T23:31:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-05-15T23:31:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.96207165718078613, 2024-05-15T23:31:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 0.95999997854232788, 2024-05-15T23:31:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.95999997854232788 2024-05-15T23:31:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-05-15T23:31:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_5", 2024-05-15T23:31:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-05-15T23:28:50.243463+0000", 2024-05-15T23:31:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-05-15T23:31:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-05-15T23:31:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-05-15T23:31:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2024-05-15T23:31:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-05-15T23:31:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-05-15T23:31:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-05-15T23:31:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-05-15T23:31:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T23:31:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T23:31:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-05-15T23:31:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-05-15T23:31:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 16, 2024-05-15T23:31:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 16, 2024-05-15T23:31:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 16, 2024-05-15T23:31:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 16, 2024-05-15T23:31:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 16, 2024-05-15T23:31:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-05-15T23:31:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-05-15T23:31:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-05-15T23:31:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-05-15T23:31:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-05-15T23:31:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-05-15T23:31:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-05-15T23:31:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "554", 2024-05-15T23:31:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-05-15T23:31:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T23:31:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T23:31:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-05-15T23:31:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-05-15T23:31:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-05-15T23:31:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-05-15T23:31:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-05-15T23:31:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-05-15T23:31:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-05-15T23:31:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-05-15T23:31:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-05-15T23:31:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-05-15T23:31:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-05-15T23:31:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-05-15T23:31:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-05-15T23:31:24.893 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-05-15T23:31:24.893 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-05-15T23:31:24.893 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T23:31:24.893 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T23:31:24.893 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T23:31:24.893 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-05-15T23:31:24.893 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-05-15T23:31:24.893 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-05-15T23:31:24.893 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-05-15T23:31:24.893 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-05-15T23:31:24.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.893 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-05-15T23:31:24.893 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-05-15T23:31:24.893 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-05-15T23:31:24.894 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-05-15T23:31:24.894 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-05-15T23:31:24.894 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T23:31:24.894 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-05-15T23:31:24.894 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-05-15T23:31:24.894 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-05-15T23:31:24.894 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-05-15T23:31:24.894 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-05-15T23:31:24.894 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-05-15T23:31:24.894 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-05-15T23:31:24.894 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-05-15T23:31:24.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.894 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-05-15T23:31:24.894 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-05-15T23:31:24.894 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.9900000095367432, 2024-05-15T23:31:24.895 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.9900000095367432, 2024-05-15T23:31:24.895 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.67000001668930054, 2024-05-15T23:31:24.895 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-05-15T23:31:24.895 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-05-15T23:31:24.895 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.96207165718078613, 2024-05-15T23:31:24.895 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 0.95999997854232788, 2024-05-15T23:31:24.895 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.95999997854232788 2024-05-15T23:31:24.895 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.895 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.895 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:31:24.895 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-05-15T23:31:24.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.895 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-05-15T23:31:24.895 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "80159cc8-1b6c-4aa6-a114-966786d7be1e", 2024-05-15T23:31:24.896 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:31:24.896 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-05-15T23:31:24.896 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-05-15T23:31:24.896 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:31:24.896 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 8, 2024-05-15T23:31:24.896 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 355, 2024-05-15T23:31:24.896 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 397, 2024-05-15T23:31:24.896 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 283, 2024-05-15T23:31:24.896 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 356, 2024-05-15T23:31:24.896 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:31:24.896 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:31:24.896 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.896 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.897 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-05-15T23:31:24.897 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:31:24.897 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.897 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.897 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:31:24.897 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.897 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.897 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-05-15T23:31:24.897 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:31:24.897 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.897 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.898 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:31:24.898 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.899 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.899 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-05-15T23:31:24.899 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:31:24.899 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.899 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.899 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:31:24.899 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.899 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.899 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-05-15T23:31:24.899 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:31:24.899 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.899 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.900 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/3077815271", 2024-05-15T23:31:24.900 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/3077815271", 2024-05-15T23:31:24.900 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/3077815271", 2024-05-15T23:31:24.900 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/3077815271", 2024-05-15T23:31:24.900 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:31:24.900 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:31:24.900 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:31:24.900 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.900 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-05-15T23:31:24.900 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "9c2c76c9-ce0f-4c2d-9d96-c87a10187607", 2024-05-15T23:31:24.900 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:31:24.901 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:31:24.901 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:31:24.901 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:31:24.901 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:31:24.901 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:31:24.901 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-05-15T23:31:24.901 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 578, 2024-05-15T23:31:24.901 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:31:24.901 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:31:24.901 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:31:24.902 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.902 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.902 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-05-15T23:31:24.902 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:31:24.902 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.902 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.902 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:31:24.902 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.902 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.902 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-05-15T23:31:24.903 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:31:24.903 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.903 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.903 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:31:24.903 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.903 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.903 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-05-15T23:31:24.903 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:31:24.903 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.903 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.903 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:31:24.904 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.904 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.904 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-05-15T23:31:24.904 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:31:24.904 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.904 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.904 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/2140830229", 2024-05-15T23:31:24.904 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/2140830229", 2024-05-15T23:31:24.904 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/2140830229", 2024-05-15T23:31:24.904 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/2140830229", 2024-05-15T23:31:24.904 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:31:24.904 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:31:24.905 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:31:24.905 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.905 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-05-15T23:31:24.905 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "e8514daa-b5f9-4f2f-a432-53f4a4eb86bc", 2024-05-15T23:31:24.905 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:31:24.905 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:31:24.905 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:31:24.905 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:31:24.905 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 20, 2024-05-15T23:31:24.905 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 261, 2024-05-15T23:31:24.905 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 291, 2024-05-15T23:31:24.905 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:31:24.906 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 262, 2024-05-15T23:31:24.906 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:31:24.906 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:31:24.906 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.906 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.906 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-05-15T23:31:24.906 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:31:24.906 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.906 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.906 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:31:24.906 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.907 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.907 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-05-15T23:31:24.907 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:31:24.907 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.907 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.907 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:31:24.907 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.907 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.907 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-05-15T23:31:24.907 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:31:24.907 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.907 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.908 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:31:24.908 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.908 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.908 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-05-15T23:31:24.908 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:31:24.908 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.908 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.908 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/941507493", 2024-05-15T23:31:24.908 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/941507493", 2024-05-15T23:31:24.908 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/941507493", 2024-05-15T23:31:24.908 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/941507493", 2024-05-15T23:31:24.909 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:31:24.909 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:31:24.909 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:31:24.909 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.909 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-05-15T23:31:24.909 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "5d9965f9-c0b2-48ce-869c-582990985623", 2024-05-15T23:31:24.909 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:31:24.909 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:31:24.909 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:31:24.909 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:31:24.909 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:31:24.910 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:31:24.910 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 26, 2024-05-15T23:31:24.910 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:31:24.910 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:31:24.910 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:31:24.910 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:31:24.910 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.910 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.910 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-05-15T23:31:24.910 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:31:24.910 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.910 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:31:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-05-15T23:31:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:31:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:31:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.912 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-05-15T23:31:24.912 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:31:24.912 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.912 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.912 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:31:24.912 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.912 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.912 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-05-15T23:31:24.912 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:31:24.912 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.912 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.913 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2204039828", 2024-05-15T23:31:24.913 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2204039828", 2024-05-15T23:31:24.913 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2204039828", 2024-05-15T23:31:24.913 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2204039828", 2024-05-15T23:31:24.913 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:31:24.913 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:31:24.913 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:31:24.913 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.913 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-05-15T23:31:24.913 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4bc50523-8480-44d2-8d76-31479e839c85", 2024-05-15T23:31:24.913 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:31:24.913 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:31:24.914 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0.396820068359375, 2024-05-15T23:31:24.914 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:31:24.914 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:31:24.914 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:31:24.914 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-05-15T23:31:24.914 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:31:24.914 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:31:24.914 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:31:24.914 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:31:24.914 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.914 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.914 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6800", 2024-05-15T23:31:24.914 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:31:24.915 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.915 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.915 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:31:24.915 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.915 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.915 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6801", 2024-05-15T23:31:24.915 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:31:24.915 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.915 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.915 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:31:24.915 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.916 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.916 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6803", 2024-05-15T23:31:24.916 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:31:24.916 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.916 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.916 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:31:24.916 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.916 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.916 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6802", 2024-05-15T23:31:24.916 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:31:24.916 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.917 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.917 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6800/2267955292", 2024-05-15T23:31:24.917 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6801/2267955292", 2024-05-15T23:31:24.917 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6803/2267955292", 2024-05-15T23:31:24.917 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6802/2267955292", 2024-05-15T23:31:24.917 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:31:24.917 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:31:24.917 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:31:24.917 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.917 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-05-15T23:31:24.917 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20", 2024-05-15T23:31:24.918 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:31:24.918 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:31:24.918 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:31:24.918 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:31:24.918 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:31:24.918 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:31:24.918 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 37, 2024-05-15T23:31:24.918 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:31:24.918 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:31:24.918 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:31:24.918 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:31:24.918 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.918 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.919 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6804", 2024-05-15T23:31:24.919 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:31:24.919 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.919 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.919 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:31:24.919 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.919 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.919 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6805", 2024-05-15T23:31:24.919 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:31:24.919 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.920 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.920 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:31:24.920 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.920 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.920 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6807", 2024-05-15T23:31:24.920 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:31:24.920 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.920 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.921 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:31:24.921 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.921 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.921 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6806", 2024-05-15T23:31:24.921 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:31:24.921 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.921 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.921 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6804/661041538", 2024-05-15T23:31:24.921 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6805/661041538", 2024-05-15T23:31:24.921 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6807/661041538", 2024-05-15T23:31:24.921 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6806/661041538", 2024-05-15T23:31:24.921 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:31:24.922 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:31:24.922 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:31:24.922 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.922 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-05-15T23:31:24.922 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f", 2024-05-15T23:31:24.922 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:31:24.922 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-05-15T23:31:24.922 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-05-15T23:31:24.922 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:31:24.922 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:31:24.922 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:31:24.923 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 46, 2024-05-15T23:31:24.923 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 374, 2024-05-15T23:31:24.923 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:31:24.923 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:31:24.923 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:31:24.923 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.923 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.923 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6808", 2024-05-15T23:31:24.923 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:31:24.923 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.923 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.923 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:31:24.924 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.924 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.924 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6809", 2024-05-15T23:31:24.924 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:31:24.924 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.924 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.924 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:31:24.924 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.924 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.924 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6811", 2024-05-15T23:31:24.924 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:31:24.925 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.925 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.925 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:31:24.925 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.925 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.925 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6810", 2024-05-15T23:31:24.925 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:31:24.925 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.925 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.925 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6808/380856869", 2024-05-15T23:31:24.925 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6809/380856869", 2024-05-15T23:31:24.926 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6811/380856869", 2024-05-15T23:31:24.926 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6810/380856869", 2024-05-15T23:31:24.926 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:31:24.926 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:31:24.926 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:31:24.926 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.926 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-05-15T23:31:24.926 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34", 2024-05-15T23:31:24.926 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:31:24.927 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:31:24.927 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:31:24.927 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:31:24.927 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:31:24.927 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:31:24.927 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 53, 2024-05-15T23:31:24.927 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:31:24.927 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:31:24.927 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:31:24.927 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:31:24.927 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.927 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.927 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6812", 2024-05-15T23:31:24.928 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:31:24.928 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.928 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.928 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:31:24.928 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.928 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.928 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6813", 2024-05-15T23:31:24.928 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:31:24.928 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.928 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.929 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:31:24.929 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.929 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.929 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6815", 2024-05-15T23:31:24.929 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:31:24.929 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.929 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.929 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:31:24.929 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.929 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.929 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6814", 2024-05-15T23:31:24.930 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:31:24.930 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.930 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.930 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6812/1044709063", 2024-05-15T23:31:24.930 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6813/1044709063", 2024-05-15T23:31:24.930 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6815/1044709063", 2024-05-15T23:31:24.930 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6814/1044709063", 2024-05-15T23:31:24.930 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:31:24.930 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:31:24.930 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:31:24.930 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.931 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-05-15T23:31:24.931 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "02cab286-d487-4a66-982c-3cf28f55e851", 2024-05-15T23:31:24.931 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:31:24.931 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-05-15T23:31:24.931 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-05-15T23:31:24.931 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 0.54486083984375, 2024-05-15T23:31:24.931 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 61, 2024-05-15T23:31:24.931 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 484, 2024-05-15T23:31:24.931 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 520, 2024-05-15T23:31:24.931 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 202, 2024-05-15T23:31:24.931 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 485, 2024-05-15T23:31:24.931 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:31:24.931 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:31:24.932 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.932 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.932 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6800", 2024-05-15T23:31:24.932 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:31:24.932 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.932 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.932 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:31:24.932 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.932 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.932 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6801", 2024-05-15T23:31:24.932 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:31:24.933 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.933 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.933 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:31:24.933 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.933 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.933 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6803", 2024-05-15T23:31:24.933 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:31:24.933 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.933 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.933 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:31:24.933 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.934 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.934 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6802", 2024-05-15T23:31:24.934 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:31:24.934 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.934 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.934 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6800/703598286", 2024-05-15T23:31:24.934 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6801/703598286", 2024-05-15T23:31:24.934 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6803/703598286", 2024-05-15T23:31:24.934 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6802/703598286", 2024-05-15T23:31:24.934 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:31:24.934 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-05-15T23:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613", 2024-05-15T23:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 68, 2024-05-15T23:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 565, 2024-05-15T23:31:24.935 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:31:24.936 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:31:24.936 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:31:24.936 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.936 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.936 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6804", 2024-05-15T23:31:24.936 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:31:24.936 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.936 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.936 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:31:24.936 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.937 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.937 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6805", 2024-05-15T23:31:24.937 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:31:24.937 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.937 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.937 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:31:24.937 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.937 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.937 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6807", 2024-05-15T23:31:24.937 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:31:24.937 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.937 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.938 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:31:24.938 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.938 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.938 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6806", 2024-05-15T23:31:24.938 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:31:24.938 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.938 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.938 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6804/2794571133", 2024-05-15T23:31:24.938 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6805/2794571133", 2024-05-15T23:31:24.938 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6807/2794571133", 2024-05-15T23:31:24.938 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6806/2794571133", 2024-05-15T23:31:24.939 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:31:24.939 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:31:24.939 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:31:24.939 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.939 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-05-15T23:31:24.939 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473", 2024-05-15T23:31:24.939 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:31:24.939 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:31:24.939 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:31:24.939 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:31:24.939 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:31:24.939 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:31:24.940 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 74, 2024-05-15T23:31:24.940 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 578, 2024-05-15T23:31:24.940 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:31:24.940 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:31:24.940 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:31:24.940 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.940 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.940 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6808", 2024-05-15T23:31:24.940 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:31:24.940 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.940 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.941 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:31:24.941 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.941 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.941 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6809", 2024-05-15T23:31:24.941 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:31:24.941 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.941 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.941 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:31:24.941 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.941 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.941 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6811", 2024-05-15T23:31:24.942 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:31:24.942 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.942 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.942 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:31:24.942 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.942 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.942 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6810", 2024-05-15T23:31:24.942 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:31:24.942 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.942 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.942 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6808/3735380744", 2024-05-15T23:31:24.943 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6809/3735380744", 2024-05-15T23:31:24.943 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6811/3735380744", 2024-05-15T23:31:24.943 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6810/3735380744", 2024-05-15T23:31:24.943 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:31:24.943 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:31:24.943 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:31:24.943 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.943 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-05-15T23:31:24.943 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5", 2024-05-15T23:31:24.943 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:31:24.943 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:31:24.943 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:31:24.944 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:31:24.944 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:31:24.944 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:31:24.944 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 81, 2024-05-15T23:31:24.944 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:31:24.944 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:31:24.944 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:31:24.944 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:31:24.944 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.944 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.944 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6812", 2024-05-15T23:31:24.944 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:31:24.944 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.944 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.945 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:31:24.945 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.945 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.945 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6813", 2024-05-15T23:31:24.945 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:31:24.945 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.945 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.945 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:31:24.945 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.945 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.946 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6815", 2024-05-15T23:31:24.946 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:31:24.946 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.946 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.946 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:31:24.946 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:31:24.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.946 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:31:24.946 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6814", 2024-05-15T23:31:24.946 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:31:24.946 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.946 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.947 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6812/2265615270", 2024-05-15T23:31:24.947 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6813/2265615270", 2024-05-15T23:31:24.947 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6815/2265615270", 2024-05-15T23:31:24.947 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6814/2265615270", 2024-05-15T23:31:24.947 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:31:24.947 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:31:24.947 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:31:24.947 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.947 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.947 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:31:24.947 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-05-15T23:31:24.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.947 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-05-15T23:31:24.947 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:31:24.947 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:31:24.948 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:31:24.948 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:31:24.948 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-05-15T23:31:24.948 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:30:21.891882+0000", 2024-05-15T23:31:24.948 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 355 2024-05-15T23:31:24.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.948 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-05-15T23:31:24.948 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:31:24.948 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:31:24.948 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:31:24.948 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:31:24.948 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:31:24.949 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:31:21.200544+0000", 2024-05-15T23:31:24.949 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:31:24.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.949 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-05-15T23:31:24.949 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:31:24.949 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:31:24.949 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:31:24.949 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:31:24.950 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:31:24.950 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:29:58.580698+0000", 2024-05-15T23:31:24.950 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 261 2024-05-15T23:31:24.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.950 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-05-15T23:31:24.950 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:31:24.950 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:31:24.950 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:31:24.950 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:31:24.950 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:31:24.950 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:30:19.343484+0000", 2024-05-15T23:31:24.950 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:31:24.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.951 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-05-15T23:31:24.951 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:31:24.951 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:31:24.951 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:31:24.951 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:31:24.951 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:31:24.951 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:31:03.073939+0000", 2024-05-15T23:31:24.951 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:31:24.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.951 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-05-15T23:31:24.952 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:31:24.952 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:31:24.952 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:31:24.952 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:31:24.952 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:31:24.952 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:30:18.004887+0000", 2024-05-15T23:31:24.952 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:31:24.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.952 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-05-15T23:31:24.952 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:31:24.952 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:31:24.952 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:31:24.952 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:31:24.952 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-05-15T23:31:24.953 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:30:30.662979+0000", 2024-05-15T23:31:24.953 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:31:24.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.953 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-05-15T23:31:24.953 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:31:24.953 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:31:24.953 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:31:24.953 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:31:24.953 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:31:24.953 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:30:31.780285+0000", 2024-05-15T23:31:24.953 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:31:24.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.954 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-05-15T23:31:24.954 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:31:24.954 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:31:24.954 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:31:24.954 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:31:24.954 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-05-15T23:31:24.954 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:30:52.538971+0000", 2024-05-15T23:31:24.954 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 484 2024-05-15T23:31:24.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.954 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-05-15T23:31:24.954 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:31:24.955 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:31:24.955 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:31:24.955 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:31:24.955 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:31:24.955 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:30:48.919641+0000", 2024-05-15T23:31:24.955 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:31:24.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.955 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-05-15T23:31:24.955 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:31:24.955 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:31:24.955 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:31:24.955 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:31:24.956 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:31:24.956 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:31:11.522605+0000", 2024-05-15T23:31:24.956 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:31:24.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.956 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-05-15T23:31:24.956 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:31:24.956 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:31:24.956 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:31:24.956 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:31:24.956 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:31:24.956 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:30:26.167473+0000", 2024-05-15T23:31:24.956 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:31:24.956 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.956 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:31:24.957 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2024-05-15T23:31:24.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.957 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "1.0", 2024-05-15T23:31:24.957 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-05-15T23:31:24.957 INFO:teuthology.orchestra.run.smithi084.stdout: 6, 2024-05-15T23:31:24.957 INFO:teuthology.orchestra.run.smithi084.stdout: 3 2024-05-15T23:31:24.957 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:24.957 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "7.8", 2024-05-15T23:31:24.957 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-05-15T23:31:24.957 INFO:teuthology.orchestra.run.smithi084.stdout: 6, 2024-05-15T23:31:24.957 INFO:teuthology.orchestra.run.smithi084.stdout: 0 2024-05-15T23:31:24.957 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:24.958 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.958 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:31:24.958 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [], 2024-05-15T23:31:24.958 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-05-15T23:31:24.958 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-05-15T23:31:24.958 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-05-15T23:31:24.958 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-05-15T23:31:24.958 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2581408231": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:31:24.958 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/1136984393": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:31:24.958 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2443201864": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:31:24.958 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1252671888": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:31:24.958 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1451001371": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:31:24.958 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4084814866": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:31:24.958 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4076409285": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:31:24.959 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2988977694": "2024-05-16T22:57:38.505263+0000" 2024-05-15T23:31:24.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.959 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-05-15T23:31:24.959 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-05-15T23:31:24.959 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-05-15T23:31:24.959 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-05-15T23:31:24.959 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-05-15T23:31:24.959 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-05-15T23:31:24.959 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-05-15T23:31:24.959 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-05-15T23:31:24.959 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:24.959 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [], 2024-05-15T23:31:24.959 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2024-05-15T23:31:24.960 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-05-15T23:31:24.960 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-05-15T23:31:24.960 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-05-15T23:31:24.960 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-05-15T23:31:24.960 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-05-15T23:31:24.960 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-05-15T23:31:24.960 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-05-15T23:31:24.960 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-05-15T23:31:24.960 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-05-15T23:31:24.960 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:24.960 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:31:25.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:24 smithi088 ceph-mon[44699]: 7.8 deep-scrub starts 2024-05-15T23:31:25.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:24 smithi084 ceph-mon[39784]: 7.8 deep-scrub starts 2024-05-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(5.9 426/407 [PushOp(5:915457ca:::benchmark_data_smithi090_116470_object33671:head, version: 382'2136, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:915457ca:::benchmark_data_smithi090_116470_object33671:head@382'2136, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-05-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:22:17.125971+0000", 2024-05-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 548.11685490499997, 2024-05-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12290139899999999, 2024-05-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.125971+0000", 2024-05-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.125971+0000", 2024-05-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.125972+0000", 2024-05-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.16e-07 2024-05-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.126038+0000", 2024-05-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5753999999999994e-05 2024-05-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.126050+0000", 2024-05-15T23:31:25.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2345e-05 2024-05-15T23:31:25.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.126063+0000", 2024-05-15T23:31:25.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2952000000000001e-05 2024-05-15T23:31:25.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.225456+0000", 2024-05-15T23:31:25.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099392709999999995 2024-05-15T23:31:25.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.225466+0000", 2024-05-15T23:31:25.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0152e-05 2024-05-15T23:31:25.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:22:17.248873+0000", 2024-05-15T23:31:25.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023406670000000001 2024-05-15T23:31:25.251 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.252 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.252 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.252 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:231953 5.d 5:b01a298e:::benchmark_data_smithi090_116470_object78393:head [delete] snapc 0=[] ondisk+write+known_if_redirected e445)", 2024-05-15T23:31:25.252 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:23:16.306799+0000", 2024-05-15T23:31:25.252 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 488.93602690099999, 2024-05-15T23:31:25.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11780294600000001, 2024-05-15T23:31:25.252 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.252 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.252 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.253 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.253 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:31:25.253 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:31:25.253 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 231953 2024-05-15T23:31:25.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.253 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306799+0000", 2024-05-15T23:31:25.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306799+0000", 2024-05-15T23:31:25.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306802+0000", 2024-05-15T23:31:25.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.157e-06 2024-05-15T23:31:25.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306811+0000", 2024-05-15T23:31:25.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1100000000000003e-06 2024-05-15T23:31:25.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306825+0000", 2024-05-15T23:31:25.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4642e-05 2024-05-15T23:31:25.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306835+0000", 2024-05-15T23:31:25.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0241000000000001e-05 2024-05-15T23:31:25.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306899+0000", 2024-05-15T23:31:25.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3317000000000004e-05 2024-05-15T23:31:25.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.306983+0000", 2024-05-15T23:31:25.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4265999999999996e-05 2024-05-15T23:31:25.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5", 2024-05-15T23:31:25.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.307055+0000", 2024-05-15T23:31:25.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2002000000000006e-05 2024-05-15T23:31:25.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.311076+0000", 2024-05-15T23:31:25.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040205149999999997 2024-05-15T23:31:25.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.424519+0000", 2024-05-15T23:31:25.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.113443878 2024-05-15T23:31:25.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.424556+0000", 2024-05-15T23:31:25.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6987999999999999e-05 2024-05-15T23:31:25.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:23:16.424602+0000", 2024-05-15T23:31:25.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.583e-05 2024-05-15T23:31:25.258 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.258 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.258 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.259 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:298436 5.8 5:193e65af:::benchmark_data_smithi090_116470_object144876:head [delete] snapc 0=[] ondisk+write+known_if_redirected e467)", 2024-05-15T23:31:25.259 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:28.011449+0000", 2024-05-15T23:31:25.259 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 417.23137770699998, 2024-05-15T23:31:25.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14426002900000001, 2024-05-15T23:31:25.259 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.259 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.259 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:31:25.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:31:25.259 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 298436 2024-05-15T23:31:25.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.260 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011449+0000", 2024-05-15T23:31:25.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011449+0000", 2024-05-15T23:31:25.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011451+0000", 2024-05-15T23:31:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.007e-06 2024-05-15T23:31:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011456+0000", 2024-05-15T23:31:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5890000000000002e-06 2024-05-15T23:31:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011466+0000", 2024-05-15T23:31:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8519999999999999e-06 2024-05-15T23:31:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011474+0000", 2024-05-15T23:31:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6109999999999996e-06 2024-05-15T23:31:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011530+0000", 2024-05-15T23:31:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.643e-05 2024-05-15T23:31:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011628+0000", 2024-05-15T23:31:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7677999999999996e-05 2024-05-15T23:31:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:31:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011701+0000", 2024-05-15T23:31:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3036999999999999e-05 2024-05-15T23:31:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.021745+0000", 2024-05-15T23:31:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010044628 2024-05-15T23:31:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155616+0000", 2024-05-15T23:31:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13387112800000001 2024-05-15T23:31:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155668+0000", 2024-05-15T23:31:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1533999999999998e-05 2024-05-15T23:31:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155709+0000", 2024-05-15T23:31:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0534999999999997e-05 2024-05-15T23:31:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:298437 5.8 5:16c84348:::benchmark_data_smithi090_116470_object144877:head [delete] snapc 0=[] ondisk+write+known_if_redirected e467)", 2024-05-15T23:31:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:28.011503+0000", 2024-05-15T23:31:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 417.23132331599999, 2024-05-15T23:31:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14441126500000001, 2024-05-15T23:31:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:31:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:31:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 298437 2024-05-15T23:31:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011503+0000", 2024-05-15T23:31:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011503+0000", 2024-05-15T23:31:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011504+0000", 2024-05-15T23:31:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7100000000000001e-07 2024-05-15T23:31:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011507+0000", 2024-05-15T23:31:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2969999999999999e-06 2024-05-15T23:31:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011513+0000", 2024-05-15T23:31:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2929999999999997e-06 2024-05-15T23:31:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.011518+0000", 2024-05-15T23:31:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0170000000000004e-06 2024-05-15T23:31:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.016632+0000", 2024-05-15T23:31:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005113345 2024-05-15T23:31:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.016718+0000", 2024-05-15T23:31:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5926000000000004e-05 2024-05-15T23:31:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:31:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.016829+0000", 2024-05-15T23:31:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011133000000000001 2024-05-15T23:31:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.031495+0000", 2024-05-15T23:31:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014666026 2024-05-15T23:31:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155838+0000", 2024-05-15T23:31:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.124343411 2024-05-15T23:31:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155881+0000", 2024-05-15T23:31:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2661999999999999e-05 2024-05-15T23:31:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155914+0000", 2024-05-15T23:31:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3186999999999999e-05 2024-05-15T23:31:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:298440 5.8 5:19a667cf:::benchmark_data_smithi090_116470_object144880:head [delete] snapc 0=[] ondisk+write+known_if_redirected e467)", 2024-05-15T23:31:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:28.017029+0000", 2024-05-15T23:31:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 417.22579702100001, 2024-05-15T23:31:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.138994741, 2024-05-15T23:31:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:31:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:31:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 298440 2024-05-15T23:31:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.017029+0000", 2024-05-15T23:31:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.017029+0000", 2024-05-15T23:31:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.017031+0000", 2024-05-15T23:31:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1169999999999998e-06 2024-05-15T23:31:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.017037+0000", 2024-05-15T23:31:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3020000000000002e-06 2024-05-15T23:31:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.017048+0000", 2024-05-15T23:31:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1418e-05 2024-05-15T23:31:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.017056+0000", 2024-05-15T23:31:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2390000000000002e-06 2024-05-15T23:31:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.021778+0000", 2024-05-15T23:31:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047214989999999997 2024-05-15T23:31:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.021903+0000", 2024-05-15T23:31:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012554299999999999 2024-05-15T23:31:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:31:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.022001+0000", 2024-05-15T23:31:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7408e-05 2024-05-15T23:31:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.031500+0000", 2024-05-15T23:31:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009499347 2024-05-15T23:31:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155953+0000", 2024-05-15T23:31:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.124453067 2024-05-15T23:31:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.155993+0000", 2024-05-15T23:31:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0021999999999997e-05 2024-05-15T23:31:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.156024+0000", 2024-05-15T23:31:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0778999999999998e-05 2024-05-15T23:31:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.25840.0:298453 5.8 5:19488f07:::benchmark_data_smithi090_116470_object144893:head [delete] snapc 0=[] ondisk+write+known_if_redirected e467)", 2024-05-15T23:31:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:24:28.023509+0000", 2024-05-15T23:31:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 417.219317644, 2024-05-15T23:31:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.132728863, 2024-05-15T23:31:25.280 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.280 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.280 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.25840", 2024-05-15T23:31:25.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/278901885", 2024-05-15T23:31:25.280 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 298453 2024-05-15T23:31:25.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.280 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.023509+0000", 2024-05-15T23:31:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.023509+0000", 2024-05-15T23:31:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.023511+0000", 2024-05-15T23:31:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3599999999999999e-06 2024-05-15T23:31:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.023517+0000", 2024-05-15T23:31:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7579999999999998e-06 2024-05-15T23:31:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.023527+0000", 2024-05-15T23:31:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0665999999999999e-05 2024-05-15T23:31:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.023535+0000", 2024-05-15T23:31:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5639999999999999e-06 2024-05-15T23:31:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.026406+0000", 2024-05-15T23:31:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002871401 2024-05-15T23:31:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.026516+0000", 2024-05-15T23:31:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109431 2024-05-15T23:31:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:31:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.026619+0000", 2024-05-15T23:31:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010360400000000001 2024-05-15T23:31:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.031659+0000", 2024-05-15T23:31:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050393149999999999 2024-05-15T23:31:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.156152+0000", 2024-05-15T23:31:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.124493797 2024-05-15T23:31:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.156202+0000", 2024-05-15T23:31:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9336000000000003e-05 2024-05-15T23:31:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:24:28.156237+0000", 2024-05-15T23:31:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5630999999999998e-05 2024-05-15T23:31:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:144922 7.9 7:909e0880:::benchmark_data_smithi090_143373_object144921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e587)", 2024-05-15T23:31:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:33.303923+0000", 2024-05-15T23:31:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.938902925999997, 2024-05-15T23:31:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13177601799999999, 2024-05-15T23:31:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 144922 2024-05-15T23:31:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.303923+0000", 2024-05-15T23:31:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.303923+0000", 2024-05-15T23:31:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.303926+0000", 2024-05-15T23:31:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.34e-06 2024-05-15T23:31:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.303946+0000", 2024-05-15T23:31:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0129e-05 2024-05-15T23:31:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.303957+0000", 2024-05-15T23:31:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1049e-05 2024-05-15T23:31:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.303963+0000", 2024-05-15T23:31:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2400000000000004e-06 2024-05-15T23:31:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.304021+0000", 2024-05-15T23:31:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7577000000000001e-05 2024-05-15T23:31:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.304154+0000", 2024-05-15T23:31:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013346699999999999 2024-05-15T23:31:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:31:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.304305+0000", 2024-05-15T23:31:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015088999999999999 2024-05-15T23:31:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.311182+0000", 2024-05-15T23:31:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00687749 2024-05-15T23:31:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.435598+0000", 2024-05-15T23:31:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12441527500000001 2024-05-15T23:31:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.435654+0000", 2024-05-15T23:31:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6137000000000001e-05 2024-05-15T23:31:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.435699+0000", 2024-05-15T23:31:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5423999999999997e-05 2024-05-15T23:31:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:144924 7.9 7:9cc672d6:::benchmark_data_smithi090_143373_object144923:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e587)", 2024-05-15T23:31:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:33.307231+0000", 2024-05-15T23:31:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.935595534999997, 2024-05-15T23:31:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12859894099999999, 2024-05-15T23:31:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 144924 2024-05-15T23:31:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.307231+0000", 2024-05-15T23:31:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.307231+0000", 2024-05-15T23:31:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.307233+0000", 2024-05-15T23:31:25.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1040000000000001e-06 2024-05-15T23:31:25.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.307252+0000", 2024-05-15T23:31:25.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9239e-05 2024-05-15T23:31:25.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.307268+0000", 2024-05-15T23:31:25.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5940999999999999e-05 2024-05-15T23:31:25.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.307275+0000", 2024-05-15T23:31:25.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6490000000000002e-06 2024-05-15T23:31:25.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.307763+0000", 2024-05-15T23:31:25.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048806499999999998 2024-05-15T23:31:25.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.307865+0000", 2024-05-15T23:31:25.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102128 2024-05-15T23:31:25.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:31:25.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.307959+0000", 2024-05-15T23:31:25.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3955000000000004e-05 2024-05-15T23:31:25.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.311326+0000", 2024-05-15T23:31:25.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033673119999999999 2024-05-15T23:31:25.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.435741+0000", 2024-05-15T23:31:25.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.124414724 2024-05-15T23:31:25.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.435794+0000", 2024-05-15T23:31:25.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3219e-05 2024-05-15T23:31:25.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:33.435830+0000", 2024-05-15T23:31:25.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5605000000000002e-05 2024-05-15T23:31:25.299 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.299 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.299 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.299 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:150642 7.9 7:9ed690b0:::benchmark_data_smithi090_143373_object4471:head [delete] snapc 0=[] ondisk+write+known_if_redirected e588)", 2024-05-15T23:31:25.299 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:39.696087+0000", 2024-05-15T23:31:25.299 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.546739095, 2024-05-15T23:31:25.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12134344800000001, 2024-05-15T23:31:25.300 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.300 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.300 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.300 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.300 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:25.300 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:25.300 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 150642 2024-05-15T23:31:25.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.300 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.696087+0000", 2024-05-15T23:31:25.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.696087+0000", 2024-05-15T23:31:25.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.696089+0000", 2024-05-15T23:31:25.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3379999999999999e-06 2024-05-15T23:31:25.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.696096+0000", 2024-05-15T23:31:25.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3419999999999996e-06 2024-05-15T23:31:25.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.696107+0000", 2024-05-15T23:31:25.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2024-05-15T23:31:25.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.696115+0000", 2024-05-15T23:31:25.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3909999999999998e-06 2024-05-15T23:31:25.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.702033+0000", 2024-05-15T23:31:25.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0059174470000000002 2024-05-15T23:31:25.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.702142+0000", 2024-05-15T23:31:25.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109751 2024-05-15T23:31:25.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:31:25.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.702254+0000", 2024-05-15T23:31:25.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112077 2024-05-15T23:31:25.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.817299+0000", 2024-05-15T23:31:25.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.115044642 2024-05-15T23:31:25.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.817332+0000", 2024-05-15T23:31:25.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2845000000000001e-05 2024-05-15T23:31:25.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.817384+0000", 2024-05-15T23:31:25.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2473999999999999e-05 2024-05-15T23:31:25.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.817431+0000", 2024-05-15T23:31:25.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6140999999999998e-05 2024-05-15T23:31:25.306 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.306 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.306 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.306 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.16764.0:150646 7.f e588/549 7:f5e2e4da:::benchmark_data_smithi090_143373_object4475:head v 588'9367, mlcod=588'9364)", 2024-05-15T23:31:25.306 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:39.696339+0000", 2024-05-15T23:31:25.306 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.546486975000001, 2024-05-15T23:31:25.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120920104, 2024-05-15T23:31:25.306 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.306 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.307 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.307 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.696339+0000", 2024-05-15T23:31:25.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.696339+0000", 2024-05-15T23:31:25.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.696341+0000", 2024-05-15T23:31:25.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5239999999999999e-06 2024-05-15T23:31:25.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.696349+0000", 2024-05-15T23:31:25.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1000000000000004e-06 2024-05-15T23:31:25.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.696358+0000", 2024-05-15T23:31:25.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2110000000000004e-06 2024-05-15T23:31:25.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.696365+0000", 2024-05-15T23:31:25.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6980000000000001e-06 2024-05-15T23:31:25.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.696428+0000", 2024-05-15T23:31:25.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3609999999999996e-05 2024-05-15T23:31:25.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.696448+0000", 2024-05-15T23:31:25.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9134999999999999e-05 2024-05-15T23:31:25.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.708231+0000", 2024-05-15T23:31:25.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011783752 2024-05-15T23:31:25.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.817259+0000", 2024-05-15T23:31:25.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.109028074 2024-05-15T23:31:25.311 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.311 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.311 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.311 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.16764.0:150649 7.f e588/549 7:fcfe3dbb:::benchmark_data_smithi090_143373_object4478:head v 588'9368, mlcod=588'9364)", 2024-05-15T23:31:25.311 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:39.700496+0000", 2024-05-15T23:31:25.311 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.542330511999999, 2024-05-15T23:31:25.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.123894262, 2024-05-15T23:31:25.312 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.312 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.312 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.312 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.700496+0000", 2024-05-15T23:31:25.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.700496+0000", 2024-05-15T23:31:25.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.700498+0000", 2024-05-15T23:31:25.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.65e-06 2024-05-15T23:31:25.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.700511+0000", 2024-05-15T23:31:25.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2259000000000001e-05 2024-05-15T23:31:25.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.700526+0000", 2024-05-15T23:31:25.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5688999999999998e-05 2024-05-15T23:31:25.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.700536+0000", 2024-05-15T23:31:25.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5910000000000002e-06 2024-05-15T23:31:25.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.701221+0000", 2024-05-15T23:31:25.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068530100000000003 2024-05-15T23:31:25.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.701240+0000", 2024-05-15T23:31:25.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8587999999999999e-05 2024-05-15T23:31:25.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.824346+0000", 2024-05-15T23:31:25.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.123106192 2024-05-15T23:31:25.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.824390+0000", 2024-05-15T23:31:25.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3992000000000002e-05 2024-05-15T23:31:25.316 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.316 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.316 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.317 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:150648 7.5 7:a38cdd40:::benchmark_data_smithi090_143373_object4477:head [delete] snapc 0=[] ondisk+write+known_if_redirected e588)", 2024-05-15T23:31:25.317 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:39.701903+0000", 2024-05-15T23:31:25.317 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.540923624000001, 2024-05-15T23:31:25.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120488106, 2024-05-15T23:31:25.317 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.317 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.317 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.317 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.317 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:25.317 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:25.317 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 150648 2024-05-15T23:31:25.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.318 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.701903+0000", 2024-05-15T23:31:25.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.701903+0000", 2024-05-15T23:31:25.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.701904+0000", 2024-05-15T23:31:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7719999999999999e-06 2024-05-15T23:31:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.701911+0000", 2024-05-15T23:31:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7469999999999999e-06 2024-05-15T23:31:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.701922+0000", 2024-05-15T23:31:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0992000000000001e-05 2024-05-15T23:31:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.701930+0000", 2024-05-15T23:31:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6029999999999999e-06 2024-05-15T23:31:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.701992+0000", 2024-05-15T23:31:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2277e-05 2024-05-15T23:31:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.702104+0000", 2024-05-15T23:31:25.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011181 2024-05-15T23:31:25.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 9", 2024-05-15T23:31:25.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.702209+0000", 2024-05-15T23:31:25.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000105037 2024-05-15T23:31:25.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.822276+0000", 2024-05-15T23:31:25.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120067384 2024-05-15T23:31:25.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.822289+0000", 2024-05-15T23:31:25.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2931e-05 2024-05-15T23:31:25.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.822342+0000", 2024-05-15T23:31:25.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2914000000000002e-05 2024-05-15T23:31:25.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.822391+0000", 2024-05-15T23:31:25.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8639e-05 2024-05-15T23:31:25.323 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.323 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.323 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.323 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:150650 7.5 7:a7053d85:::benchmark_data_smithi090_143373_object4479:head [delete] snapc 0=[] ondisk+write+known_if_redirected e588)", 2024-05-15T23:31:25.324 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:39.701969+0000", 2024-05-15T23:31:25.324 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.540857502000001, 2024-05-15T23:31:25.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12054617300000001, 2024-05-15T23:31:25.324 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.324 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.324 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.324 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.324 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:25.324 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:25.324 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 150650 2024-05-15T23:31:25.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.324 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.701969+0000", 2024-05-15T23:31:25.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.701969+0000", 2024-05-15T23:31:25.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.701969+0000", 2024-05-15T23:31:25.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1800000000000005e-07 2024-05-15T23:31:25.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.701973+0000", 2024-05-15T23:31:25.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5300000000000001e-06 2024-05-15T23:31:25.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.701979+0000", 2024-05-15T23:31:25.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2190000000000001e-06 2024-05-15T23:31:25.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.701984+0000", 2024-05-15T23:31:25.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1719999999999998e-06 2024-05-15T23:31:25.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.817425+0000", 2024-05-15T23:31:25.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.115440503 2024-05-15T23:31:25.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.817560+0000", 2024-05-15T23:31:25.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013491800000000001 2024-05-15T23:31:25.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 9", 2024-05-15T23:31:25.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.817665+0000", 2024-05-15T23:31:25.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104786 2024-05-15T23:31:25.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.822432+0000", 2024-05-15T23:31:25.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047674520000000001 2024-05-15T23:31:25.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.822443+0000", 2024-05-15T23:31:25.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1276e-05 2024-05-15T23:31:25.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.822481+0000", 2024-05-15T23:31:25.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8118000000000003e-05 2024-05-15T23:31:25.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.822515+0000", 2024-05-15T23:31:25.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3481e-05 2024-05-15T23:31:25.330 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.330 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.330 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.330 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:150657 7.5 7:a2882f87:::benchmark_data_smithi090_143373_object4486:head [delete] snapc 0=[] ondisk+write+known_if_redirected e588)", 2024-05-15T23:31:25.330 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:39.708465+0000", 2024-05-15T23:31:25.330 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.534361337999997, 2024-05-15T23:31:25.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12729938800000001, 2024-05-15T23:31:25.331 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.331 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.331 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.331 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.331 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:25.331 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:25.331 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 150657 2024-05-15T23:31:25.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.331 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.708465+0000", 2024-05-15T23:31:25.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.708465+0000", 2024-05-15T23:31:25.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.708468+0000", 2024-05-15T23:31:25.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6960000000000001e-06 2024-05-15T23:31:25.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.708474+0000", 2024-05-15T23:31:25.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8369999999999999e-06 2024-05-15T23:31:25.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.817223+0000", 2024-05-15T23:31:25.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10874874900000001 2024-05-15T23:31:25.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.817236+0000", 2024-05-15T23:31:25.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3237e-05 2024-05-15T23:31:25.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.822548+0000", 2024-05-15T23:31:25.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053110850000000001 2024-05-15T23:31:25.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.822644+0000", 2024-05-15T23:31:25.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6828000000000002e-05 2024-05-15T23:31:25.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 9", 2024-05-15T23:31:25.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.822710+0000", 2024-05-15T23:31:25.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5605999999999995e-05 2024-05-15T23:31:25.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.835583+0000", 2024-05-15T23:31:25.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012873242999999999 2024-05-15T23:31:25.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.835659+0000", 2024-05-15T23:31:25.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5994000000000005e-05 2024-05-15T23:31:25.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.835716+0000", 2024-05-15T23:31:25.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6705e-05 2024-05-15T23:31:25.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.835764+0000", 2024-05-15T23:31:25.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8408000000000001e-05 2024-05-15T23:31:25.336 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.337 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.337 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.337 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.16764.0:150655 7.f e588/549 7:f62ab310:::benchmark_data_smithi090_143373_object4484:head v 588'9369, mlcod=588'9365)", 2024-05-15T23:31:25.337 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:39.708697+0000", 2024-05-15T23:31:25.337 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.534129118999999, 2024-05-15T23:31:25.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120164655, 2024-05-15T23:31:25.337 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.337 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.337 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.337 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.708697+0000", 2024-05-15T23:31:25.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.708697+0000", 2024-05-15T23:31:25.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.708699+0000", 2024-05-15T23:31:25.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.654e-06 2024-05-15T23:31:25.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.708708+0000", 2024-05-15T23:31:25.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5610000000000004e-06 2024-05-15T23:31:25.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.817212+0000", 2024-05-15T23:31:25.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.108504087 2024-05-15T23:31:25.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.817229+0000", 2024-05-15T23:31:25.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.662e-05 2024-05-15T23:31:25.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.817287+0000", 2024-05-15T23:31:25.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7637999999999999e-05 2024-05-15T23:31:25.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.817307+0000", 2024-05-15T23:31:25.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0568999999999999e-05 2024-05-15T23:31:25.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.828825+0000", 2024-05-15T23:31:25.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011517627000000001 2024-05-15T23:31:25.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:39.828862+0000", 2024-05-15T23:31:25.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899e-05 2024-05-15T23:31:25.342 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.342 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.342 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.342 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:169805 7.9 7:9f062381:::benchmark_data_smithi090_143373_object23634:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:31:25.342 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:02.728364+0000", 2024-05-15T23:31:25.342 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.514462045999998, 2024-05-15T23:31:25.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18166183499999999, 2024-05-15T23:31:25.342 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.342 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.343 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.343 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.343 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:25.343 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:25.343 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 169805 2024-05-15T23:31:25.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.343 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728364+0000", 2024-05-15T23:31:25.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728364+0000", 2024-05-15T23:31:25.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728366+0000", 2024-05-15T23:31:25.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0250000000000001e-06 2024-05-15T23:31:25.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728373+0000", 2024-05-15T23:31:25.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4710000000000004e-06 2024-05-15T23:31:25.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728389+0000", 2024-05-15T23:31:25.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5892e-05 2024-05-15T23:31:25.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728398+0000", 2024-05-15T23:31:25.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6379999999999999e-06 2024-05-15T23:31:25.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728465+0000", 2024-05-15T23:31:25.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6455e-05 2024-05-15T23:31:25.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728586+0000", 2024-05-15T23:31:25.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121445 2024-05-15T23:31:25.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:31:25.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728705+0000", 2024-05-15T23:31:25.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118628 2024-05-15T23:31:25.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733796+0000", 2024-05-15T23:31:25.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00509144 2024-05-15T23:31:25.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909910+0000", 2024-05-15T23:31:25.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.176113774 2024-05-15T23:31:25.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909967+0000", 2024-05-15T23:31:25.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7312999999999998e-05 2024-05-15T23:31:25.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.910026+0000", 2024-05-15T23:31:25.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8754e-05 2024-05-15T23:31:25.349 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.349 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.349 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.349 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:169813 7.9 7:9f055894:::benchmark_data_smithi090_143373_object23642:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:31:25.349 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:02.733789+0000", 2024-05-15T23:31:25.349 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.509037545999998, 2024-05-15T23:31:25.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.192022421, 2024-05-15T23:31:25.349 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.349 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.349 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.349 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.350 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:25.350 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:25.350 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 169813 2024-05-15T23:31:25.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.350 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733789+0000", 2024-05-15T23:31:25.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733789+0000", 2024-05-15T23:31:25.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733791+0000", 2024-05-15T23:31:25.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.075e-06 2024-05-15T23:31:25.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733797+0000", 2024-05-15T23:31:25.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5830000000000003e-06 2024-05-15T23:31:25.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733808+0000", 2024-05-15T23:31:25.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0589e-05 2024-05-15T23:31:25.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733815+0000", 2024-05-15T23:31:25.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1930000000000004e-06 2024-05-15T23:31:25.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733865+0000", 2024-05-15T23:31:25.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0263999999999999e-05 2024-05-15T23:31:25.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733988+0000", 2024-05-15T23:31:25.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012234799999999999 2024-05-15T23:31:25.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:31:25.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.734102+0000", 2024-05-15T23:31:25.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113831 2024-05-15T23:31:25.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.745770+0000", 2024-05-15T23:31:25.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011668705 2024-05-15T23:31:25.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.925710+0000", 2024-05-15T23:31:25.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.179939708 2024-05-15T23:31:25.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.925762+0000", 2024-05-15T23:31:25.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2123999999999998e-05 2024-05-15T23:31:25.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.925811+0000", 2024-05-15T23:31:25.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9001000000000001e-05 2024-05-15T23:31:25.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.355 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.356 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:169814 7.9 7:9503e738:::benchmark_data_smithi090_143373_object23643:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:31:25.356 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:02.733837+0000", 2024-05-15T23:31:25.356 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.508989052, 2024-05-15T23:31:25.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19211088000000001, 2024-05-15T23:31:25.356 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.356 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.356 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:25.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:25.356 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 169814 2024-05-15T23:31:25.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.357 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733837+0000", 2024-05-15T23:31:25.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733837+0000", 2024-05-15T23:31:25.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733838+0000", 2024-05-15T23:31:25.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9400000000000004e-07 2024-05-15T23:31:25.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733842+0000", 2024-05-15T23:31:25.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8580000000000002e-06 2024-05-15T23:31:25.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733849+0000", 2024-05-15T23:31:25.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8770000000000004e-06 2024-05-15T23:31:25.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733854+0000", 2024-05-15T23:31:25.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.31e-06 2024-05-15T23:31:25.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.738245+0000", 2024-05-15T23:31:25.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043908159999999996 2024-05-15T23:31:25.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.738360+0000", 2024-05-15T23:31:25.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114731 2024-05-15T23:31:25.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:31:25.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.738470+0000", 2024-05-15T23:31:25.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011088500000000001 2024-05-15T23:31:25.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.745774+0000", 2024-05-15T23:31:25.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073033209999999998 2024-05-15T23:31:25.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.925860+0000", 2024-05-15T23:31:25.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18008618400000001 2024-05-15T23:31:25.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.925908+0000", 2024-05-15T23:31:25.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7673000000000002e-05 2024-05-15T23:31:25.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.925948+0000", 2024-05-15T23:31:25.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0431e-05 2024-05-15T23:31:25.362 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.362 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.362 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.362 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:169815 7.9 7:9b7f3b8f:::benchmark_data_smithi090_143373_object23644:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:31:25.362 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:02.733871+0000", 2024-05-15T23:31:25.363 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.508954741, 2024-05-15T23:31:25.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.193427022, 2024-05-15T23:31:25.363 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.363 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.363 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.363 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.363 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:25.363 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:25.363 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 169815 2024-05-15T23:31:25.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.363 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733871+0000", 2024-05-15T23:31:25.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733871+0000", 2024-05-15T23:31:25.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733872+0000", 2024-05-15T23:31:25.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3099999999999996e-07 2024-05-15T23:31:25.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733876+0000", 2024-05-15T23:31:25.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8360000000000002e-06 2024-05-15T23:31:25.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733883+0000", 2024-05-15T23:31:25.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7150000000000001e-06 2024-05-15T23:31:25.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733887+0000", 2024-05-15T23:31:25.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3120000000000004e-06 2024-05-15T23:31:25.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.741987+0000", 2024-05-15T23:31:25.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0080999220000000007 2024-05-15T23:31:25.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.742095+0000", 2024-05-15T23:31:25.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010813400000000001 2024-05-15T23:31:25.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:31:25.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.742213+0000", 2024-05-15T23:31:25.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011735000000000001 2024-05-15T23:31:25.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.745957+0000", 2024-05-15T23:31:25.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037446240000000002 2024-05-15T23:31:25.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.927206+0000", 2024-05-15T23:31:25.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.181248723 2024-05-15T23:31:25.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.927250+0000", 2024-05-15T23:31:25.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3733999999999998e-05 2024-05-15T23:31:25.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.927299+0000", 2024-05-15T23:31:25.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8741e-05 2024-05-15T23:31:25.369 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.369 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.369 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.369 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:169840 7.9 7:93ee92fe:::benchmark_data_smithi090_143373_object23669:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:31:25.369 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:02.756631+0000", 2024-05-15T23:31:25.369 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.486195015, 2024-05-15T23:31:25.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18059107899999999, 2024-05-15T23:31:25.369 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:31:25.370 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:31:25.370 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:25.370 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:31:25.370 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:31:25.370 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:25.370 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 169840 2024-05-15T23:31:25.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.370 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:31:25.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:31:25.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756631+0000", 2024-05-15T23:31:25.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:31:25.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756631+0000", 2024-05-15T23:31:25.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:31:25.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:31:25.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756633+0000", 2024-05-15T23:31:25.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0109999999999999e-06 2024-05-15T23:31:25.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:31:25.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756639+0000", 2024-05-15T23:31:25.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8440000000000003e-06 2024-05-15T23:31:25.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:31:25.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756649+0000", 2024-05-15T23:31:25.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0336999999999999e-05 2024-05-15T23:31:25.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:31:25.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756657+0000", 2024-05-15T23:31:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9349999999999994e-06 2024-05-15T23:31:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:31:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756716+0000", 2024-05-15T23:31:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8727000000000002e-05 2024-05-15T23:31:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:31:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756833+0000", 2024-05-15T23:31:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011724500000000001 2024-05-15T23:31:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:31:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756948+0000", 2024-05-15T23:31:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115159 2024-05-15T23:31:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:31:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.761100+0000", 2024-05-15T23:31:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041516579999999999 2024-05-15T23:31:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.937099+0000", 2024-05-15T23:31:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17599888899999999 2024-05-15T23:31:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:31:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.937162+0000", 2024-05-15T23:31:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2668000000000004e-05 2024-05-15T23:31:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:31:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:31:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:31:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.937222+0000", 2024-05-15T23:31:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0606000000000002e-05 2024-05-15T23:31:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.376 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.376 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.376 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:31:25.376 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:31:25.376 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:31:25.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:25.377 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:31:25.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:31:25.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:31:25.510 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:25.550 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:31:25.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:25.554 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:31:25.655 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:31:25.705 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:25.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:25 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2796770618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:31:25.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:25 smithi084 ceph-mon[39784]: pgmap v1600: 17 pgs: 1 active+recovering, 16 active+clean; 6.5 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 860 op/s; 5356/214408 objects misplaced (2.498%); 6.0 MiB/s, 95 objects/s recovering 2024-05-15T23:31:26.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:25 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2796770618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:31:26.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:25 smithi090 ceph-mon[44434]: pgmap v1600: 17 pgs: 1 active+recovering, 16 active+clean; 6.5 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 860 op/s; 5356/214408 objects misplaced (2.498%); 6.0 MiB/s, 95 objects/s recovering 2024-05-15T23:31:26.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:25 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2796770618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:31:26.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:25 smithi088 ceph-mon[44699]: pgmap v1600: 17 pgs: 1 active+recovering, 16 active+clean; 6.5 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 860 op/s; 5356/214408 objects misplaced (2.498%); 6.0 MiB/s, 95 objects/s recovering 2024-05-15T23:31:26.718 INFO:tasks.thrashosds.thrasher:j is {'epoch': 597, 'fsid': '3999fb7e-130e-11ef-bc99-c7b262605968', 'created': '2024-05-15T22:56:41.212493+0000', 'modified': '2024-05-15T23:31:21.681471+0000', 'last_up_change': '2024-05-15T23:26:58.424794+0000', 'last_in_change': '2024-05-15T23:28:16.134275+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 30, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T22:59:38.603575+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9620716571807861, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.9599999785423279}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-05-15T23:28:50.243463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '554', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.6700000166893005, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9620716571807861, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.9599999785423279}}], 'osds': [{'osd': 0, 'uuid': '80159cc8-1b6c-4aa6-a114-966786d7be1e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 355, 'up_from': 397, 'up_thru': 283, 'down_at': 356, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6801', 'nonce': 3077815271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6802', 'nonce': 3077815271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6804', 'nonce': 3077815271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6803', 'nonce': 3077815271}]}, 'public_addr': '172.21.15.84:6801/3077815271', 'cluster_addr': '172.21.15.84:6802/3077815271', 'heartbeat_back_addr': '172.21.15.84:6804/3077815271', 'heartbeat_front_addr': '172.21.15.84:6803/3077815271', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9c2c76c9-ce0f-4c2d-9d96-c87a10187607', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 578, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6805', 'nonce': 2140830229}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6806', 'nonce': 2140830229}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6808', 'nonce': 2140830229}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6807', 'nonce': 2140830229}]}, 'public_addr': '172.21.15.84:6805/2140830229', 'cluster_addr': '172.21.15.84:6806/2140830229', 'heartbeat_back_addr': '172.21.15.84:6808/2140830229', 'heartbeat_front_addr': '172.21.15.84:6807/2140830229', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e8514daa-b5f9-4f2f-a432-53f4a4eb86bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 261, 'up_from': 291, 'up_thru': 549, 'down_at': 262, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6809', 'nonce': 941507493}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6810', 'nonce': 941507493}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6812', 'nonce': 941507493}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6811', 'nonce': 941507493}]}, 'public_addr': '172.21.15.84:6809/941507493', 'cluster_addr': '172.21.15.84:6810/941507493', 'heartbeat_back_addr': '172.21.15.84:6812/941507493', 'heartbeat_front_addr': '172.21.15.84:6811/941507493', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5d9965f9-c0b2-48ce-869c-582990985623', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6813', 'nonce': 2204039828}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6814', 'nonce': 2204039828}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6816', 'nonce': 2204039828}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6815', 'nonce': 2204039828}]}, 'public_addr': '172.21.15.84:6813/2204039828', 'cluster_addr': '172.21.15.84:6814/2204039828', 'heartbeat_back_addr': '172.21.15.84:6816/2204039828', 'heartbeat_front_addr': '172.21.15.84:6815/2204039828', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4bc50523-8480-44d2-8d76-31479e839c85', 'up': 1, 'in': 1, 'weight': 0.396820068359375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6800', 'nonce': 2267955292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6801', 'nonce': 2267955292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6803', 'nonce': 2267955292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6802', 'nonce': 2267955292}]}, 'public_addr': '172.21.15.88:6800/2267955292', 'cluster_addr': '172.21.15.88:6801/2267955292', 'heartbeat_back_addr': '172.21.15.88:6803/2267955292', 'heartbeat_front_addr': '172.21.15.88:6802/2267955292', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'd4db8d7f-afc7-43dc-81fe-f70c290d8b20', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6804', 'nonce': 661041538}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6805', 'nonce': 661041538}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6807', 'nonce': 661041538}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6806', 'nonce': 661041538}]}, 'public_addr': '172.21.15.88:6804/661041538', 'cluster_addr': '172.21.15.88:6805/661041538', 'heartbeat_back_addr': '172.21.15.88:6807/661041538', 'heartbeat_front_addr': '172.21.15.88:6806/661041538', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'a2773454-2318-4a5b-8f31-0952b1b07f2f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6808', 'nonce': 380856869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6809', 'nonce': 380856869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6811', 'nonce': 380856869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6810', 'nonce': 380856869}]}, 'public_addr': '172.21.15.88:6808/380856869', 'cluster_addr': '172.21.15.88:6809/380856869', 'heartbeat_back_addr': '172.21.15.88:6811/380856869', 'heartbeat_front_addr': '172.21.15.88:6810/380856869', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '028ca7b6-3e00-4466-a33f-b726cd60cb34', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6812', 'nonce': 1044709063}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6813', 'nonce': 1044709063}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6815', 'nonce': 1044709063}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6814', 'nonce': 1044709063}]}, 'public_addr': '172.21.15.88:6812/1044709063', 'cluster_addr': '172.21.15.88:6813/1044709063', 'heartbeat_back_addr': '172.21.15.88:6815/1044709063', 'heartbeat_front_addr': '172.21.15.88:6814/1044709063', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '02cab286-d487-4a66-982c-3cf28f55e851', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.54486083984375, 'last_clean_begin': 61, 'last_clean_end': 484, 'up_from': 520, 'up_thru': 202, 'down_at': 485, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6800', 'nonce': 703598286}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 703598286}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 703598286}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6802', 'nonce': 703598286}]}, 'public_addr': '172.21.15.90:6800/703598286', 'cluster_addr': '172.21.15.90:6801/703598286', 'heartbeat_back_addr': '172.21.15.90:6803/703598286', 'heartbeat_front_addr': '172.21.15.90:6802/703598286', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '47d38bc7-39ba-4b35-af24-e282354dd613', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 68, 'up_thru': 565, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6804', 'nonce': 2794571133}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 2794571133}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 2794571133}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6806', 'nonce': 2794571133}]}, 'public_addr': '172.21.15.90:6804/2794571133', 'cluster_addr': '172.21.15.90:6805/2794571133', 'heartbeat_back_addr': '172.21.15.90:6807/2794571133', 'heartbeat_front_addr': '172.21.15.90:6806/2794571133', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'd7b2aeae-d012-41eb-9f91-e4c05c649473', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 578, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6808', 'nonce': 3735380744}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 3735380744}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 3735380744}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 3735380744}]}, 'public_addr': '172.21.15.90:6808/3735380744', 'cluster_addr': '172.21.15.90:6809/3735380744', 'heartbeat_back_addr': '172.21.15.90:6811/3735380744', 'heartbeat_front_addr': '172.21.15.90:6810/3735380744', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6812', 'nonce': 2265615270}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 2265615270}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 2265615270}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 2265615270}]}, 'public_addr': '172.21.15.90:6812/2265615270', 'cluster_addr': '172.21.15.90:6813/2265615270', 'heartbeat_back_addr': '172.21.15.90:6815/2265615270', 'heartbeat_front_addr': '172.21.15.90:6814/2265615270', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T23:30:21.891882+0000', 'dead_epoch': 355}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:31:21.200544+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:29:58.580698+0000', 'dead_epoch': 261}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:30:19.343484+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:31:03.073939+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:30:18.004887+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T23:30:30.662979+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:30:31.780285+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T23:30:52.538971+0000', 'dead_epoch': 484}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:30:48.919641+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:31:11.522605+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:30:26.167473+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [6, 3]}, {'pgid': '7.8', 'osds': [6, 0]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/2581408231': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:6800/1136984393': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:6800/2443201864': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/1252671888': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/1451001371': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/4084814866': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:0/4076409285': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/2988977694': '2024-05-16T22:57:38.505263+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-15T23:31:26.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:31:26.898 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:26 smithi090 ceph-mon[44434]: 7.d deep-scrub starts 2024-05-15T23:31:26.936 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T23:31:26.985 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:26 smithi084 ceph-mon[39784]: 7.d deep-scrub starts 2024-05-15T23:31:27.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:26 smithi088 ceph-mon[44699]: 7.d deep-scrub starts 2024-05-15T23:31:27.930 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:31:27.992 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:31:27.992 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:31:27.992 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:27 smithi084 ceph-mon[39784]: osdmap e598: 12 total, 12 up, 9 in 2024-05-15T23:31:27.992 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:27 smithi084 ceph-mon[39784]: 7.0 scrub starts 2024-05-15T23:31:27.992 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:27 smithi084 ceph-mon[39784]: pgmap v1602: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 841 op/s; 5114/209078 objects misplaced (2.446%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:31:27.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:27.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:31:28.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:31:28.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:31:28.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:27 smithi090 ceph-mon[44434]: osdmap e598: 12 total, 12 up, 9 in 2024-05-15T23:31:28.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:27 smithi090 ceph-mon[44434]: 7.0 scrub starts 2024-05-15T23:31:28.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:27 smithi090 ceph-mon[44434]: pgmap v1602: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 841 op/s; 5114/209078 objects misplaced (2.446%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:31:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:31:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:31:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:27 smithi088 ceph-mon[44699]: osdmap e598: 12 total, 12 up, 9 in 2024-05-15T23:31:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:27 smithi088 ceph-mon[44699]: 7.0 scrub starts 2024-05-15T23:31:28.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:27 smithi088 ceph-mon[44699]: pgmap v1602: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 841 op/s; 5114/209078 objects misplaced (2.446%); 4.2 MiB/s, 67 objects/s recovering 2024-05-15T23:31:28.095 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:31:28.164 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:28.323 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:31:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:28.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:31:28.427 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:31:28.518 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:28.851 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:31:28.900 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:31:28.904 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:31:28.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:28.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:31:28.910 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:31:29.011 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:31:29.047 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:28 smithi084 ceph-mon[39784]: 7.0 scrub ok 2024-05-15T23:31:29.047 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:28 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:31:29.048 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:28 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:31:29.048 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:28 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:31:29.048 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:28 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:31:29.048 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:31:29.048 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:28 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:31:29.048 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:28 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:31:29.051 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:28 smithi090 ceph-mon[44434]: 7.0 scrub ok 2024-05-15T23:31:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:28 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:31:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:28 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:31:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:28 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:31:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:28 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:31:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:31:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:28 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:31:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:28 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:31:29.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:28 smithi088 ceph-mon[44699]: 7.0 scrub ok 2024-05-15T23:31:29.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:28 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:31:29.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:28 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:31:29.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:28 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:31:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:28 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:31:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:31:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:28 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:31:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:28 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:31:29.189 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:31:29.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:29.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:31:29.298 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:31:29.324 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:29.343 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:31:29.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:29.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:31:29.448 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:31:29.474 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:29.494 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:31:29.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:29.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:31:29.599 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:31:29.663 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:29.783 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:29 smithi084 ceph-mon[39784]: osdmap e599: 12 total, 12 up, 9 in 2024-05-15T23:31:29.783 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:29 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:31:29.783 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:29 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:31:29.783 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:29 smithi084 ceph-mon[39784]: pgmap v1604: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.3 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 623 op/s; 5114/207776 objects misplaced (2.461%); 2.0 MiB/s, 32 objects/s recovering 2024-05-15T23:31:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:29 smithi090 ceph-mon[44434]: osdmap e599: 12 total, 12 up, 9 in 2024-05-15T23:31:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:29 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:31:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:29 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:31:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:29 smithi090 ceph-mon[44434]: pgmap v1604: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.3 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 623 op/s; 5114/207776 objects misplaced (2.461%); 2.0 MiB/s, 32 objects/s recovering 2024-05-15T23:31:30.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:29 smithi088 ceph-mon[44699]: osdmap e599: 12 total, 12 up, 9 in 2024-05-15T23:31:30.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:29 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:31:30.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:29 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:31:30.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:29 smithi088 ceph-mon[44699]: pgmap v1604: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.3 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 623 op/s; 5114/207776 objects misplaced (2.461%); 2.0 MiB/s, 32 objects/s recovering 2024-05-15T23:31:30.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:30 smithi084 ceph-mon[39784]: osdmap e600: 12 total, 12 up, 9 in 2024-05-15T23:31:30.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:30 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:31:30.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:30 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:31:31.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:30 smithi090 ceph-mon[44434]: osdmap e600: 12 total, 12 up, 9 in 2024-05-15T23:31:31.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:30 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:31:31.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:30 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:31:31.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:30 smithi088 ceph-mon[44699]: osdmap e600: 12 total, 12 up, 9 in 2024-05-15T23:31:31.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:30 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:31:31.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:30 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:31:31.856 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:31:32.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:31 smithi084 ceph-mon[39784]: osdmap e601: 12 total, 12 up, 9 in 2024-05-15T23:31:32.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:31 smithi084 ceph-mon[39784]: pgmap v1607: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.3 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 376 op/s; 4926/205744 objects misplaced (2.394%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T23:31:32.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3945234746' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:31:32.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:31 smithi088 ceph-mon[44699]: osdmap e601: 12 total, 12 up, 9 in 2024-05-15T23:31:32.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:31 smithi088 ceph-mon[44699]: pgmap v1607: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.3 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 376 op/s; 4926/205744 objects misplaced (2.394%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T23:31:32.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3945234746' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:31:32.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:31 smithi090 ceph-mon[44434]: osdmap e601: 12 total, 12 up, 9 in 2024-05-15T23:31:32.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:31 smithi090 ceph-mon[44434]: pgmap v1607: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.3 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 376 op/s; 4926/205744 objects misplaced (2.394%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T23:31:32.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3945234746' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:31:33.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:32 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:31:33.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3945234746' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:31:33.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:32 smithi088 ceph-mon[44699]: osdmap e602: 12 total, 12 up, 9 in 2024-05-15T23:31:33.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:32 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:31:33.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3945234746' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:31:33.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:32 smithi084 ceph-mon[39784]: osdmap e602: 12 total, 12 up, 9 in 2024-05-15T23:31:33.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:32 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:31:33.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3945234746' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:31:33.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:32 smithi090 ceph-mon[44434]: osdmap e602: 12 total, 12 up, 9 in 2024-05-15T23:31:33.821 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:33 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:31:33.821 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:33 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:31:33.821 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:33 smithi084 ceph-mon[39784]: pgmap v1609: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 736 op/s; 4926/201322 objects misplaced (2.447%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T23:31:33.821 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:33 smithi084 ceph-mon[39784]: osdmap e603: 12 total, 12 up, 9 in 2024-05-15T23:31:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:33 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:31:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:33 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:31:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:33 smithi090 ceph-mon[44434]: pgmap v1609: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 736 op/s; 4926/201322 objects misplaced (2.447%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T23:31:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:33 smithi090 ceph-mon[44434]: osdmap e603: 12 total, 12 up, 9 in 2024-05-15T23:31:34.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:33 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:31:34.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:33 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:31:34.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:33 smithi088 ceph-mon[44699]: pgmap v1609: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 736 op/s; 4926/201322 objects misplaced (2.447%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T23:31:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:33 smithi088 ceph-mon[44699]: osdmap e603: 12 total, 12 up, 9 in 2024-05-15T23:31:35.256 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:35 smithi084 ceph-mon[39784]: pgmap v1611: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 597 op/s; 4926/200524 objects misplaced (2.457%); 2.2 MiB/s, 34 objects/s recovering 2024-05-15T23:31:35.493 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:31:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:35.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:31:35.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:35 smithi090 ceph-mon[44434]: pgmap v1611: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 597 op/s; 4926/200524 objects misplaced (2.457%); 2.2 MiB/s, 34 objects/s recovering 2024-05-15T23:31:35.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:35 smithi088 ceph-mon[44699]: pgmap v1611: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 597 op/s; 4926/200524 objects misplaced (2.457%); 2.2 MiB/s, 34 objects/s recovering 2024-05-15T23:31:35.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:31:35.637 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:35.675 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:31:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:35.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:31:35.783 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:31:35.815 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:35.840 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:31:35.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:35.848 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:31:35.941 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:31:35.949 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:31:36.125 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:31:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:31:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(osd.2.0:26752 7.8 e603/549)", 2024-05-15T23:31:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.094665+0000", 2024-05-15T23:31:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.00053192300000000001, 2024-05-15T23:31:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00055756100000000004, 2024-05-15T23:31:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.126 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.094665+0000", 2024-05-15T23:31:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.094665+0000", 2024-05-15T23:31:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.094667+0000", 2024-05-15T23:31:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2060000000000001e-06 2024-05-15T23:31:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.094679+0000", 2024-05-15T23:31:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1813e-05 2024-05-15T23:31:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.094695+0000", 2024-05-15T23:31:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5458999999999999e-05 2024-05-15T23:31:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.094704+0000", 2024-05-15T23:31:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1379999999999996e-06 2024-05-15T23:31:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(client.16764.0:194755 7.8 e603/549)", 2024-05-15T23:31:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.009495+0000", 2024-05-15T23:31:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.085701971000000002, 2024-05-15T23:31:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.085779628999999996, 2024-05-15T23:31:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.130 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.009495+0000", 2024-05-15T23:31:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.009495+0000", 2024-05-15T23:31:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.009497+0000", 2024-05-15T23:31:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.499e-06 2024-05-15T23:31:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.009506+0000", 2024-05-15T23:31:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3149999999999998e-06 2024-05-15T23:31:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.009515+0000", 2024-05-15T23:31:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6059999999999992e-06 2024-05-15T23:31:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.009523+0000", 2024-05-15T23:31:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2420000000000003e-06 2024-05-15T23:31:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(osd.2.0:26736 7.8 e603/549)", 2024-05-15T23:31:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.034222+0000", 2024-05-15T23:31:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.060974961000000001, 2024-05-15T23:31:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.061098646999999999, 2024-05-15T23:31:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.133 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034222+0000", 2024-05-15T23:31:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034222+0000", 2024-05-15T23:31:36.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034224+0000", 2024-05-15T23:31:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.888e-06 2024-05-15T23:31:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034236+0000", 2024-05-15T23:31:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2243e-05 2024-05-15T23:31:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034246+0000", 2024-05-15T23:31:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0115999999999999e-05 2024-05-15T23:31:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034254+0000", 2024-05-15T23:31:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5549999999999997e-06 2024-05-15T23:31:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.136 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(osd.2.0:26737 7.8 e603/549)", 2024-05-15T23:31:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.034268+0000", 2024-05-15T23:31:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.060929090999999998, 2024-05-15T23:31:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.061098204000000003, 2024-05-15T23:31:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034268+0000", 2024-05-15T23:31:36.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034268+0000", 2024-05-15T23:31:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034268+0000", 2024-05-15T23:31:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3300000000000003e-07 2024-05-15T23:31:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034271+0000", 2024-05-15T23:31:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4499999999999998e-06 2024-05-15T23:31:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034275+0000", 2024-05-15T23:31:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.065e-06 2024-05-15T23:31:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034280+0000", 2024-05-15T23:31:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8620000000000002e-06 2024-05-15T23:31:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(osd.2.0:26738 7.8 e603/549)", 2024-05-15T23:31:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.034286+0000", 2024-05-15T23:31:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.060910680000000002, 2024-05-15T23:31:36.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.061132376000000002, 2024-05-15T23:31:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034286+0000", 2024-05-15T23:31:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034286+0000", 2024-05-15T23:31:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034287+0000", 2024-05-15T23:31:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1600000000000002e-07 2024-05-15T23:31:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034289+0000", 2024-05-15T23:31:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3499999999999999e-06 2024-05-15T23:31:36.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034293+0000", 2024-05-15T23:31:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9149999999999998e-06 2024-05-15T23:31:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.034297+0000", 2024-05-15T23:31:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.67e-06 2024-05-15T23:31:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.145 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(osd.2.0:26740 7.8 e603/549)", 2024-05-15T23:31:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.039420+0000", 2024-05-15T23:31:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.055776728999999997, 2024-05-15T23:31:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.056044480000000001, 2024-05-15T23:31:36.146 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039420+0000", 2024-05-15T23:31:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039420+0000", 2024-05-15T23:31:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039423+0000", 2024-05-15T23:31:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3549999999999999e-06 2024-05-15T23:31:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039430+0000", 2024-05-15T23:31:36.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.878e-06 2024-05-15T23:31:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039442+0000", 2024-05-15T23:31:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2517000000000001e-05 2024-05-15T23:31:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039451+0000", 2024-05-15T23:31:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9199999999999993e-06 2024-05-15T23:31:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.151 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(osd.2.0:26742 7.8 e603/549)", 2024-05-15T23:31:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.039465+0000", 2024-05-15T23:31:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.055732037999999998, 2024-05-15T23:31:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.056045095000000003, 2024-05-15T23:31:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.152 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039465+0000", 2024-05-15T23:31:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039465+0000", 2024-05-15T23:31:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039465+0000", 2024-05-15T23:31:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9999999999999998e-07 2024-05-15T23:31:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039468+0000", 2024-05-15T23:31:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.447e-06 2024-05-15T23:31:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039472+0000", 2024-05-15T23:31:36.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4920000000000004e-06 2024-05-15T23:31:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039477+0000", 2024-05-15T23:31:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1899999999999997e-06 2024-05-15T23:31:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(osd.2.0:26743 7.8 e603/549)", 2024-05-15T23:31:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.039481+0000", 2024-05-15T23:31:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.055715678999999997, 2024-05-15T23:31:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.056073378, 2024-05-15T23:31:36.155 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039481+0000", 2024-05-15T23:31:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039481+0000", 2024-05-15T23:31:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039482+0000", 2024-05-15T23:31:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4400000000000001e-07 2024-05-15T23:31:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039484+0000", 2024-05-15T23:31:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9750000000000001e-06 2024-05-15T23:31:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039487+0000", 2024-05-15T23:31:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.63e-06 2024-05-15T23:31:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039491+0000", 2024-05-15T23:31:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6500000000000002e-06 2024-05-15T23:31:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.159 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:194790 7.8 7.f8853968 (undecoded) ondisk+write+known_if_redirected e603)", 2024-05-15T23:31:36.159 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.039475+0000", 2024-05-15T23:31:36.159 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.055722163999999998, 2024-05-15T23:31:36.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.056128722999999998, 2024-05-15T23:31:36.159 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.159 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.159 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.159 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:36.159 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:36.159 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:36.159 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 194790 2024-05-15T23:31:36.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.159 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039475+0000", 2024-05-15T23:31:36.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039475+0000", 2024-05-15T23:31:36.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039477+0000", 2024-05-15T23:31:36.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5050000000000002e-06 2024-05-15T23:31:36.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039484+0000", 2024-05-15T23:31:36.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4489999999999996e-06 2024-05-15T23:31:36.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039497+0000", 2024-05-15T23:31:36.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3155e-05 2024-05-15T23:31:36.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.039510+0000", 2024-05-15T23:31:36.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3358e-05 2024-05-15T23:31:36.163 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.163 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.163 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.164 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(osd.2.0:26744 7.8 e603/549)", 2024-05-15T23:31:36.164 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.057433+0000", 2024-05-15T23:31:36.164 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.037763589, 2024-05-15T23:31:36.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.038223418000000002, 2024-05-15T23:31:36.164 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.164 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.164 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.164 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.057433+0000", 2024-05-15T23:31:36.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.057433+0000", 2024-05-15T23:31:36.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.057436+0000", 2024-05-15T23:31:36.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5799999999999999e-06 2024-05-15T23:31:36.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.057445+0000", 2024-05-15T23:31:36.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3980000000000005e-06 2024-05-15T23:31:36.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.057463+0000", 2024-05-15T23:31:36.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7554000000000002e-05 2024-05-15T23:31:36.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.057472+0000", 2024-05-15T23:31:36.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5000000000000005e-06 2024-05-15T23:31:36.168 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.168 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.168 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.168 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(osd.2.0:26745 7.8 e603/549)", 2024-05-15T23:31:36.168 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.064068+0000", 2024-05-15T23:31:36.168 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.031128939000000001, 2024-05-15T23:31:36.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.031636560000000001, 2024-05-15T23:31:36.168 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.169 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.169 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.169 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.064068+0000", 2024-05-15T23:31:36.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.064068+0000", 2024-05-15T23:31:36.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.064070+0000", 2024-05-15T23:31:36.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.742e-06 2024-05-15T23:31:36.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.064078+0000", 2024-05-15T23:31:36.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8599999999999993e-06 2024-05-15T23:31:36.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.064088+0000", 2024-05-15T23:31:36.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9629999999999993e-06 2024-05-15T23:31:36.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.064095+0000", 2024-05-15T23:31:36.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6790000000000005e-06 2024-05-15T23:31:36.171 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.171 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.171 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.171 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(osd.2.0:26746 7.8 e603/549)", 2024-05-15T23:31:36.172 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.066707+0000", 2024-05-15T23:31:36.172 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.028490534000000001, 2024-05-15T23:31:36.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.029046637, 2024-05-15T23:31:36.172 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.172 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.172 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.172 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.066707+0000", 2024-05-15T23:31:36.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.066707+0000", 2024-05-15T23:31:36.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.066708+0000", 2024-05-15T23:31:36.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.612e-06 2024-05-15T23:31:36.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.066715+0000", 2024-05-15T23:31:36.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5390000000000004e-06 2024-05-15T23:31:36.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.066730+0000", 2024-05-15T23:31:36.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5038000000000001e-05 2024-05-15T23:31:36.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.066738+0000", 2024-05-15T23:31:36.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9310000000000008e-06 2024-05-15T23:31:36.174 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.174 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.174 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.175 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(osd.2.0:26747 7.8 e603/549)", 2024-05-15T23:31:36.175 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.066814+0000", 2024-05-15T23:31:36.175 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.028382991, 2024-05-15T23:31:36.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028980598, 2024-05-15T23:31:36.175 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.175 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.175 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.175 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.066814+0000", 2024-05-15T23:31:36.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.066814+0000", 2024-05-15T23:31:36.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.066815+0000", 2024-05-15T23:31:36.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4300000000000002e-07 2024-05-15T23:31:36.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.066818+0000", 2024-05-15T23:31:36.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3299999999999999e-06 2024-05-15T23:31:36.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.066827+0000", 2024-05-15T23:31:36.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8219999999999996e-06 2024-05-15T23:31:36.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.066832+0000", 2024-05-15T23:31:36.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0189999999999997e-06 2024-05-15T23:31:36.177 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.177 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.178 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.178 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(osd.2.0:26748 7.8 e603/549)", 2024-05-15T23:31:36.178 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.086459+0000", 2024-05-15T23:31:36.178 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.008737903, 2024-05-15T23:31:36.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0093802120000000006, 2024-05-15T23:31:36.178 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.178 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.178 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.179 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086459+0000", 2024-05-15T23:31:36.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086459+0000", 2024-05-15T23:31:36.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086461+0000", 2024-05-15T23:31:36.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8160000000000001e-06 2024-05-15T23:31:36.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086468+0000", 2024-05-15T23:31:36.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0129999999999996e-06 2024-05-15T23:31:36.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086481+0000", 2024-05-15T23:31:36.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.31e-05 2024-05-15T23:31:36.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086489+0000", 2024-05-15T23:31:36.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9410000000000007e-06 2024-05-15T23:31:36.181 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.181 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.181 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.181 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:194746 7.8 7:15d745f4:::benchmark_data_smithi090_143373_object48575:head [delete] snapc 0=[] ondisk+write+known_if_redirected e603)", 2024-05-15T23:31:36.181 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:35.969302+0000", 2024-05-15T23:31:36.181 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.12589508499999999, 2024-05-15T23:31:36.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.12659205400000001, 2024-05-15T23:31:36.182 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.182 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.182 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:31:36.182 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:36.182 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:36.182 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:36.182 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 194746 2024-05-15T23:31:36.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.182 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969302+0000", 2024-05-15T23:31:36.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969302+0000", 2024-05-15T23:31:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969303+0000", 2024-05-15T23:31:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.33e-06 2024-05-15T23:31:36.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969312+0000", 2024-05-15T23:31:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2619999999999998e-06 2024-05-15T23:31:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969321+0000", 2024-05-15T23:31:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9140000000000003e-06 2024-05-15T23:31:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969336+0000", 2024-05-15T23:31:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4968e-05 2024-05-15T23:31:36.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.989969+0000", 2024-05-15T23:31:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020632265 2024-05-15T23:31:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.990085+0000", 2024-05-15T23:31:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000115883 2024-05-15T23:31:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.990200+0000", 2024-05-15T23:31:36.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114963 2024-05-15T23:31:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.007820+0000", 2024-05-15T23:31:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017620286999999998 2024-05-15T23:31:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(osd.2.0:26749 7.8 e603/549)", 2024-05-15T23:31:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.086503+0000", 2024-05-15T23:31:36.186 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.0086944880000000002, 2024-05-15T23:31:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094706979999999996, 2024-05-15T23:31:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086503+0000", 2024-05-15T23:31:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086503+0000", 2024-05-15T23:31:36.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086503+0000", 2024-05-15T23:31:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1600000000000001e-07 2024-05-15T23:31:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086506+0000", 2024-05-15T23:31:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8059999999999999e-06 2024-05-15T23:31:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086510+0000", 2024-05-15T23:31:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5229999999999997e-06 2024-05-15T23:31:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086515+0000", 2024-05-15T23:31:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1180000000000002e-06 2024-05-15T23:31:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(osd.2.0:26750 7.8 e603/549)", 2024-05-15T23:31:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.086519+0000", 2024-05-15T23:31:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.0086777020000000007, 2024-05-15T23:31:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0095042749999999995, 2024-05-15T23:31:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086519+0000", 2024-05-15T23:31:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086519+0000", 2024-05-15T23:31:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086520+0000", 2024-05-15T23:31:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4799999999999999e-07 2024-05-15T23:31:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086522+0000", 2024-05-15T23:31:36.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.04e-06 2024-05-15T23:31:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086525+0000", 2024-05-15T23:31:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.738e-06 2024-05-15T23:31:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.086529+0000", 2024-05-15T23:31:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7730000000000001e-06 2024-05-15T23:31:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:194755 7.8 7:14a1fd4f:::benchmark_data_smithi090_143373_object48584:head [delete] snapc 0=[] ondisk+write+known_if_redirected e603)", 2024-05-15T23:31:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:35.981439+0000", 2024-05-15T23:31:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.113758042, 2024-05-15T23:31:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.114644444, 2024-05-15T23:31:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:31:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 194755 2024-05-15T23:31:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981439+0000", 2024-05-15T23:31:36.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981439+0000", 2024-05-15T23:31:36.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981441+0000", 2024-05-15T23:31:36.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3740000000000001e-06 2024-05-15T23:31:36.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981449+0000", 2024-05-15T23:31:36.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5660000000000001e-06 2024-05-15T23:31:36.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981462+0000", 2024-05-15T23:31:36.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2795e-05 2024-05-15T23:31:36.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981468+0000", 2024-05-15T23:31:36.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4799999999999998e-06 2024-05-15T23:31:36.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:36.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.002522+0000", 2024-05-15T23:31:36.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021053477000000001 2024-05-15T23:31:36.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:36.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.002655+0000", 2024-05-15T23:31:36.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013380900000000001 2024-05-15T23:31:36.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:36.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.002781+0000", 2024-05-15T23:31:36.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000125022 2024-05-15T23:31:36.197 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.198 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.198 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.198 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop_reply(osd.2.0:26751 7.8 e603/549)", 2024-05-15T23:31:36.198 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.091958+0000", 2024-05-15T23:31:36.198 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.0032391659999999999, 2024-05-15T23:31:36.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044795629999999998, 2024-05-15T23:31:36.198 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.198 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.198 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.198 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.091958+0000", 2024-05-15T23:31:36.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.091958+0000", 2024-05-15T23:31:36.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.091959+0000", 2024-05-15T23:31:36.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5799999999999999e-06 2024-05-15T23:31:36.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.091968+0000", 2024-05-15T23:31:36.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9120000000000004e-06 2024-05-15T23:31:36.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.091984+0000", 2024-05-15T23:31:36.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5886e-05 2024-05-15T23:31:36.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.091994+0000", 2024-05-15T23:31:36.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8300000000000008e-06 2024-05-15T23:31:36.201 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.201 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.201 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.201 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:194823 7.8 7.23062348 (undecoded) ondisk+write+known_if_redirected e603)", 2024-05-15T23:31:36.201 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:36.091983+0000", 2024-05-15T23:31:36.201 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.0032144669999999999, 2024-05-15T23:31:36.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045400670000000001, 2024-05-15T23:31:36.201 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:36.201 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:36.202 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "queued for pg", 2024-05-15T23:31:36.202 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:36.202 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:36.202 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:36.202 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 194823 2024-05-15T23:31:36.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.202 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:36.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:36.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.091983+0000", 2024-05-15T23:31:36.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:36.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.091983+0000", 2024-05-15T23:31:36.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:36.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:36.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.091986+0000", 2024-05-15T23:31:36.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.033e-06 2024-05-15T23:31:36.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:36.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.091992+0000", 2024-05-15T23:31:36.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.742e-06 2024-05-15T23:31:36.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:36.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.092006+0000", 2024-05-15T23:31:36.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3354e-05 2024-05-15T23:31:36.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:36.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:36.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:36.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.092014+0000", 2024-05-15T23:31:36.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4409999999999993e-06 2024-05-15T23:31:36.204 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.204 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:36.204 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.204 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:36.204 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:31:36.204 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 20 2024-05-15T23:31:36.205 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:31:36.205 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/179132662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:31:36.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/179132662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:31:36.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/179132662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:31:36.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:31:36.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:31:36.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:31:36.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:31:36.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:31:36.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:31:36.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:31:36.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:31:36.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:31:36.808 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:31:36.808 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:31:36.808 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:31:36.808 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:31:36.808 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:31:36.808 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:31:36.808 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:31:36.808 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:31:36.808 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:31:36.808 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:31:36.808 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:31:36.808 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:31:36.809 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:31:36.809 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:31:36.809 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:31:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:37 smithi090 ceph-mon[44434]: pgmap v1612: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.0 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 857 op/s; 4716/196032 objects misplaced (2.406%); 3.9 MiB/s, 62 objects/s recovering 2024-05-15T23:31:37.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:37 smithi088 ceph-mon[44699]: pgmap v1612: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.0 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 857 op/s; 4716/196032 objects misplaced (2.406%); 3.9 MiB/s, 62 objects/s recovering 2024-05-15T23:31:37.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:37.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:37 smithi084 ceph-mon[39784]: pgmap v1612: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.0 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 857 op/s; 4716/196032 objects misplaced (2.406%); 3.9 MiB/s, 62 objects/s recovering 2024-05-15T23:31:38.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:38.273 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:38.273 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:38.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:38.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:38.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:38.769 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:31:38.771 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:31:39.487 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:31:39.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:39.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:31:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:39 smithi090 ceph-mon[44434]: from='client.36758 v1:172.21.15.84:0/1501637502' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:31:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:39 smithi090 ceph-mon[44434]: pgmap v1613: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 680 op/s; 4716/194848 objects misplaced (2.420%); 1.6 MiB/s, 26 objects/s recovering 2024-05-15T23:31:39.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:39 smithi088 ceph-mon[44699]: from='client.36758 v1:172.21.15.84:0/1501637502' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:31:39.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:39 smithi088 ceph-mon[44699]: pgmap v1613: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 680 op/s; 4716/194848 objects misplaced (2.420%); 1.6 MiB/s, 26 objects/s recovering 2024-05-15T23:31:39.598 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:31:39.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:39 smithi084 ceph-mon[39784]: from='client.36758 v1:172.21.15.84:0/1501637502' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:31:39.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:39 smithi084 ceph-mon[39784]: pgmap v1613: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 680 op/s; 4716/194848 objects misplaced (2.420%); 1.6 MiB/s, 26 objects/s recovering 2024-05-15T23:31:39.654 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:39.680 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:31:39.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:39.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:31:39.792 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:31:39.819 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:39.838 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:31:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:39.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:31:39.943 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:31:39.969 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:39.988 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:31:39.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:39.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:31:40.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T23:31:40.093 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:31:40.213 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:31:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:40.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:31:40.376 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:31:40.402 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:40.425 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:31:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:40.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:31:40.529 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:31:40.558 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:40.575 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:31:40.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:40.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:31:40.680 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:31:40.707 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:40.726 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:31:40.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:40.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:31:40.730 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:31:40.789 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1612,"stamp":"2024-05-15T23:31:37.159449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6423970360,"num_objects":98016,"num_object_clones":0,"num_object_copies":196032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4716,"num_objects_unfound":0,"num_objects_dirty":98016,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":194447,"num_write_kb":9356903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7808,"num_bytes_recovered":512898656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160940,"ondisk_log_size":160940,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":17761324,"kb_used_data":12786792,"kb_used_omap":16,"kb_used_meta":4974063,"kb_avail":825945044,"statfs":{"total":863955320832,"available":845767725056,"internally_reserved":0,"allocated":13093675008,"data_stored":13090424354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17176,"internal_metadata":5093440744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,4,3,4,0,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-357629952,"num_objects":-5457,"num_object_clones":0,"num_object_copies":-10914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-398,"num_objects_unfound":0,"num_objects_dirty":-5457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5457,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":398,"num_bytes_recovered":26083328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.360974"},"pg_stats":[{"pgid":"7.f","version":"603'12098","reported_seq":12293,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-05-15T23:31:35.892059+0000","last_change":"2024-05-15T23:31:20.669532+0000","last_active":"2024-05-15T23:31:35.892059+0000","last_peered":"2024-05-15T23:31:35.892059+0000","last_clean":"2024-05-15T23:31:35.892059+0000","last_became_active":"2024-05-15T23:28:51.305173+0000","last_became_peered":"2024-05-15T23:28:51.305173+0000","last_unstale":"2024-05-15T23:31:35.892059+0000","last_undegraded":"2024-05-15T23:31:35.892059+0000","last_fullsized":"2024-05-15T23:31:35.892059+0000","mapping_epoch":549,"log_start":"558'2000","ondisk_log_start":"558'2000","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'11414","last_scrub_stamp":"2024-05-15T23:31:20.669357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:31:20.669357+0000","objects_scrubbed":6848,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:32:30.158387+0000","scrub_duration":1343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400556032,"num_objects":6112,"num_object_clones":0,"num_object_copies":12224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":582720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"603'11985","reported_seq":12193,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-05-15T23:31:34.279604+0000","last_change":"2024-05-15T23:30:19.696616+0000","last_active":"2024-05-15T23:31:34.279604+0000","last_peered":"2024-05-15T23:31:34.279604+0000","last_clean":"2024-05-15T23:31:34.279604+0000","last_became_active":"2024-05-15T23:28:51.519387+0000","last_became_peered":"2024-05-15T23:28:51.519387+0000","last_unstale":"2024-05-15T23:31:34.279604+0000","last_undegraded":"2024-05-15T23:31:34.279604+0000","last_fullsized":"2024-05-15T23:31:34.279604+0000","mapping_epoch":549,"log_start":"558'1900","ondisk_log_start":"558'1900","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'8176","last_scrub_stamp":"2024-05-15T23:30:19.696456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:19.696456+0000","objects_scrubbed":8143,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:31:47.470011+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393936896,"num_objects":6011,"num_object_clones":0,"num_object_copies":12022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":575872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"603'13669","reported_seq":13824,"reported_epoch":603,"state":"active+clean+scrubbing+deep","last_fresh":"2024-05-15T23:31:36.474582+0000","last_change":"2024-05-15T23:31:24.977219+0000","last_active":"2024-05-15T23:31:36.474582+0000","last_peered":"2024-05-15T23:31:36.474582+0000","last_clean":"2024-05-15T23:31:36.474582+0000","last_became_active":"2024-05-15T23:28:51.306664+0000","last_became_peered":"2024-05-15T23:28:51.306664+0000","last_unstale":"2024-05-15T23:31:36.474582+0000","last_undegraded":"2024-05-15T23:31:36.474582+0000","last_fullsized":"2024-05-15T23:31:36.474582+0000","mapping_epoch":549,"log_start":"562'3601","ondisk_log_start":"562'3601","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":1449,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 11s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412024856,"num_objects":6288,"num_object_clones":0,"num_object_copies":12576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6288,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12220,"num_write_kb":592193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"603'12062","reported_seq":12265,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-05-15T23:31:36.462561+0000","last_change":"2024-05-15T23:30:09.921129+0000","last_active":"2024-05-15T23:31:36.462561+0000","last_peered":"2024-05-15T23:31:36.462561+0000","last_clean":"2024-05-15T23:31:36.462561+0000","last_became_active":"2024-05-15T23:28:51.305728+0000","last_became_peered":"2024-05-15T23:28:51.305728+0000","last_unstale":"2024-05-15T23:31:36.462561+0000","last_undegraded":"2024-05-15T23:31:36.462561+0000","last_fullsized":"2024-05-15T23:31:36.462561+0000","mapping_epoch":549,"log_start":"558'2000","ondisk_log_start":"558'2000","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'7710","last_scrub_stamp":"2024-05-15T23:30:09.921025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:09.921025+0000","objects_scrubbed":7710,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":673,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405143552,"num_objects":6182,"num_object_clones":0,"num_object_copies":12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12062,"num_write_kb":583808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"603'11947","reported_seq":12148,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-05-15T23:31:36.472323+0000","last_change":"2024-05-15T23:30:11.062321+0000","last_active":"2024-05-15T23:31:36.472323+0000","last_peered":"2024-05-15T23:31:36.472323+0000","last_clean":"2024-05-15T23:31:36.472323+0000","last_became_active":"2024-05-15T23:28:51.306794+0000","last_became_peered":"2024-05-15T23:28:51.306794+0000","last_unstale":"2024-05-15T23:31:36.472323+0000","last_undegraded":"2024-05-15T23:31:36.472323+0000","last_fullsized":"2024-05-15T23:31:36.472323+0000","mapping_epoch":549,"log_start":"558'1900","ondisk_log_start":"558'1900","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'7585","last_scrub_stamp":"2024-05-15T23:30:11.062191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:11.062191+0000","objects_scrubbed":7567,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:31:39.094770+0000","scrub_duration":1068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388956160,"num_objects":5935,"num_object_clones":0,"num_object_copies":11870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11947,"num_write_kb":572224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"603'12262","reported_seq":12457,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-05-15T23:31:35.955848+0000","last_change":"2024-05-15T23:31:19.550153+0000","last_active":"2024-05-15T23:31:35.955848+0000","last_peered":"2024-05-15T23:31:35.955848+0000","last_clean":"2024-05-15T23:31:35.955848+0000","last_became_active":"2024-05-15T23:28:51.308216+0000","last_became_peered":"2024-05-15T23:28:51.308216+0000","last_unstale":"2024-05-15T23:31:35.955848+0000","last_undegraded":"2024-05-15T23:31:35.955848+0000","last_fullsized":"2024-05-15T23:31:35.955848+0000","mapping_epoch":549,"log_start":"559'2200","ondisk_log_start":"559'2200","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'11559","last_scrub_stamp":"2024-05-15T23:31:19.550020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:31:19.550020+0000","objects_scrubbed":6914,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:32:41.658667+0000","scrub_duration":1247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405143552,"num_objects":6182,"num_object_clones":0,"num_object_copies":12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12262,"num_write_kb":590208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"602'12201","reported_seq":12404,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-05-15T23:31:32.366811+0000","last_change":"2024-05-15T23:30:17.377631+0000","last_active":"2024-05-15T23:31:32.366811+0000","last_peered":"2024-05-15T23:31:32.366811+0000","last_clean":"2024-05-15T23:31:32.366811+0000","last_became_active":"2024-05-15T23:28:51.519752+0000","last_became_peered":"2024-05-15T23:28:51.519752+0000","last_unstale":"2024-05-15T23:31:32.366811+0000","last_undegraded":"2024-05-15T23:31:32.366811+0000","last_fullsized":"2024-05-15T23:31:32.366811+0000","mapping_epoch":549,"log_start":"558'2200","ondisk_log_start":"558'2200","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"580'8175","last_scrub_stamp":"2024-05-15T23:30:17.377489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:17.377489+0000","objects_scrubbed":8113,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:31:39.238196+0000","scrub_duration":1869,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401014784,"num_objects":6119,"num_object_clones":0,"num_object_copies":12238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12201,"num_write_kb":586240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"603'13689","reported_seq":13959,"reported_epoch":603,"state":"active+clean+scrubbing+deep","last_fresh":"2024-05-15T23:31:34.275976+0000","last_change":"2024-05-15T23:31:24.137884+0000","last_active":"2024-05-15T23:31:34.275976+0000","last_peered":"2024-05-15T23:31:34.275976+0000","last_clean":"2024-05-15T23:31:34.275976+0000","last_became_active":"2024-05-15T23:28:51.304930+0000","last_became_peered":"2024-05-15T23:28:51.304930+0000","last_unstale":"2024-05-15T23:31:34.275976+0000","last_undegraded":"2024-05-15T23:31:34.275976+0000","last_fullsized":"2024-05-15T23:31:34.275976+0000","mapping_epoch":549,"log_start":"561'3600","ondisk_log_start":"561'3600","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'8556","last_scrub_stamp":"2024-05-15T23:30:21.770528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:21.770528+0000","objects_scrubbed":1374,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"deep scrubbing for 10s","scrub_duration":1444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409665536,"num_objects":6251,"num_object_clones":0,"num_object_copies":12502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12315,"num_write_kb":594112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"603'12063","reported_seq":12181,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-05-15T23:31:36.489325+0000","last_change":"2024-05-15T23:28:51.307151+0000","last_active":"2024-05-15T23:31:36.489325+0000","last_peered":"2024-05-15T23:31:36.489325+0000","last_clean":"2024-05-15T23:31:36.489325+0000","last_became_active":"2024-05-15T23:28:51.306983+0000","last_became_peered":"2024-05-15T23:28:51.306983+0000","last_unstale":"2024-05-15T23:31:36.489325+0000","last_undegraded":"2024-05-15T23:31:36.489325+0000","last_fullsized":"2024-05-15T23:31:36.489325+0000","mapping_epoch":549,"log_start":"558'2000","ondisk_log_start":"558'2000","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395902976,"num_objects":6041,"num_object_clones":0,"num_object_copies":12082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12063,"num_write_kb":579328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.5","version":"602'12085","reported_seq":12288,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-05-15T23:31:32.375120+0000","last_change":"2024-05-15T23:30:13.755483+0000","last_active":"2024-05-15T23:31:32.375120+0000","last_peered":"2024-05-15T23:31:32.375120+0000","last_clean":"2024-05-15T23:31:32.375120+0000","last_became_active":"2024-05-15T23:28:51.303581+0000","last_became_peered":"2024-05-15T23:28:51.303581+0000","last_unstale":"2024-05-15T23:31:32.375120+0000","last_undegraded":"2024-05-15T23:31:32.375120+0000","last_fullsized":"2024-05-15T23:31:32.375120+0000","mapping_epoch":549,"log_start":"558'2000","ondisk_log_start":"558'2000","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"580'7953","last_scrub_stamp":"2024-05-15T23:30:13.755299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:13.755299+0000","objects_scrubbed":7919,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:31:34.976243+0000","scrub_duration":1278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409010176,"num_objects":6241,"num_object_clones":0,"num_object_copies":12482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12085,"num_write_kb":586432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"603'12426","reported_seq":22352,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-05-15T23:31:36.535199+0000","last_change":"2024-05-15T23:31:20.292646+0000","last_active":"2024-05-15T23:31:36.535199+0000","last_peered":"2024-05-15T23:31:36.535199+0000","last_clean":"2024-05-15T23:31:36.535199+0000","last_became_active":"2024-05-15T23:30:10.242131+0000","last_became_peered":"2024-05-15T23:30:10.242131+0000","last_unstale":"2024-05-15T23:31:36.535199+0000","last_undegraded":"2024-05-15T23:31:36.535199+0000","last_fullsized":"2024-05-15T23:31:36.535199+0000","mapping_epoch":578,"log_start":"559'2400","ondisk_log_start":"559'2400","created":549,"last_epoch_clean":596,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'11691","last_scrub_stamp":"2024-05-15T23:31:20.292425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:31:20.292425+0000","objects_scrubbed":6889,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:32:31.562037+0000","scrub_duration":1379,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401735680,"num_objects":6130,"num_object_clones":0,"num_object_copies":12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12426,"num_write_kb":593792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4862,"num_bytes_recovered":318636032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"603'12323","reported_seq":12527,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-05-15T23:31:36.446635+0000","last_change":"2024-05-15T23:30:12.055284+0000","last_active":"2024-05-15T23:31:36.446635+0000","last_peered":"2024-05-15T23:31:36.446635+0000","last_clean":"2024-05-15T23:31:36.446635+0000","last_became_active":"2024-05-15T23:28:51.305691+0000","last_became_peered":"2024-05-15T23:28:51.305691+0000","last_unstale":"2024-05-15T23:31:36.446635+0000","last_undegraded":"2024-05-15T23:31:36.446635+0000","last_fullsized":"2024-05-15T23:31:36.446635+0000","mapping_epoch":549,"log_start":"559'2300","ondisk_log_start":"559'2300","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'7899","last_scrub_stamp":"2024-05-15T23:30:12.055082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:12.055082+0000","objects_scrubbed":7846,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1764,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399441920,"num_objects":6095,"num_object_clones":0,"num_object_copies":12190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12323,"num_write_kb":589376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"603'12341","reported_seq":12459,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-05-15T23:31:36.490127+0000","last_change":"2024-05-15T23:28:51.519631+0000","last_active":"2024-05-15T23:31:36.490127+0000","last_peered":"2024-05-15T23:31:36.490127+0000","last_clean":"2024-05-15T23:31:36.490127+0000","last_became_active":"2024-05-15T23:28:51.519343+0000","last_became_peered":"2024-05-15T23:28:51.519343+0000","last_unstale":"2024-05-15T23:31:36.490127+0000","last_undegraded":"2024-05-15T23:31:36.490127+0000","last_fullsized":"2024-05-15T23:31:36.490127+0000","mapping_epoch":549,"log_start":"559'2300","ondisk_log_start":"559'2300","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394330112,"num_objects":6017,"num_object_clones":0,"num_object_copies":12034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12341,"num_write_kb":587456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"603'12018","reported_seq":18019,"reported_epoch":603,"state":"active+recovering","last_fresh":"2024-05-15T23:31:35.941573+0000","last_change":"2024-05-15T23:30:10.714218+0000","last_active":"2024-05-15T23:31:35.941573+0000","last_peered":"2024-05-15T23:31:35.941573+0000","last_clean":"2024-05-15T23:30:09.208910+0000","last_became_active":"2024-05-15T23:30:10.487853+0000","last_became_peered":"2024-05-15T23:30:10.487853+0000","last_unstale":"2024-05-15T23:31:35.941573+0000","last_undegraded":"2024-05-15T23:31:35.941573+0000","last_fullsized":"2024-05-15T23:31:35.941573+0000","mapping_epoch":578,"log_start":"558'2000","ondisk_log_start":"558'2000","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402915328,"num_objects":6148,"num_object_clones":0,"num_object_copies":12296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4716,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12018,"num_write_kb":581312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2940,"num_bytes_recovered":192675840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":["10","4"],"object_location_counts":[{"shards":"3,10","objects":1432},{"shards":"4,10","objects":4716}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"602'12039","reported_seq":12318,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-05-15T23:31:33.019830+0000","last_change":"2024-05-15T23:31:27.462495+0000","last_active":"2024-05-15T23:31:33.019830+0000","last_peered":"2024-05-15T23:31:33.019830+0000","last_clean":"2024-05-15T23:31:33.019830+0000","last_became_active":"2024-05-15T23:28:51.303621+0000","last_became_peered":"2024-05-15T23:28:51.303621+0000","last_unstale":"2024-05-15T23:31:33.019830+0000","last_undegraded":"2024-05-15T23:31:33.019830+0000","last_fullsized":"2024-05-15T23:31:33.019830+0000","mapping_epoch":549,"log_start":"558'2000","ondisk_log_start":"558'2000","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"598'11835","last_scrub_stamp":"2024-05-15T23:31:27.462385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:31:27.462385+0000","objects_scrubbed":6518,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:32:42.197783+0000","scrub_duration":714,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413335552,"num_objects":6307,"num_object_clones":0,"num_object_copies":12614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12039,"num_write_kb":587072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"603'11939","reported_seq":12218,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-05-15T23:31:36.361049+0000","last_change":"2024-05-15T23:31:14.794949+0000","last_active":"2024-05-15T23:31:36.361049+0000","last_peered":"2024-05-15T23:31:36.361049+0000","last_clean":"2024-05-15T23:31:36.361049+0000","last_became_active":"2024-05-15T23:28:51.304317+0000","last_became_peered":"2024-05-15T23:28:51.304317+0000","last_unstale":"2024-05-15T23:31:36.361049+0000","last_undegraded":"2024-05-15T23:31:36.361049+0000","last_fullsized":"2024-05-15T23:31:36.361049+0000","mapping_epoch":549,"log_start":"558'1900","ondisk_log_start":"558'1900","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'10959","last_scrub_stamp":"2024-05-15T23:31:14.794848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:31:14.794848+0000","objects_scrubbed":6955,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:32:14.807739+0000","scrub_duration":985,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390266880,"num_objects":5955,"num_object_clones":0,"num_object_copies":11910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11939,"num_write_kb":572608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1288,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-05-15T23:31:33.268123+0000","last_change":"2024-05-15T23:30:29.464644+0000","last_active":"2024-05-15T23:31:33.268123+0000","last_peered":"2024-05-15T23:31:33.268123+0000","last_clean":"2024-05-15T23:31:33.268123+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:31:33.268123+0000","last_undegraded":"2024-05-15T23:31:33.268123+0000","last_fullsized":"2024-05-15T23:31:33.268123+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:30:29.464530+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:30:29.464530+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:31:37.479720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6423379992,"num_objects":98014,"num_object_clones":0,"num_object_copies":196028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4716,"num_objects_unfound":0,"num_objects_dirty":98014,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194324,"num_write_kb":9354753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7802,"num_bytes_recovered":511311872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13043310592,"data_stored":13043302448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160846,"ondisk_log_size":160846,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351309,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2296920,"kb_used_data":1607692,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":91448232,"statfs":{"total":95995035648,"available":93642989568,"internally_reserved":0,"allocated":1646276608,"data_stored":1645876782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":705755588},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580243,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1945516,"kb_used_data":1549220,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":91799636,"statfs":{"total":95995035648,"available":94002827264,"internally_reserved":0,"allocated":1586401280,"data_stored":1586053154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":405797312},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776475,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2574828,"kb_used_data":1975844,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":91170324,"statfs":{"total":95995035648,"available":93358411776,"internally_reserved":0,"allocated":2023264256,"data_stored":2022916130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":613349824},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":2233382993978,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267043,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1332276,"kb_used_data":777708,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92412876,"statfs":{"total":95995035648,"available":94630785024,"internally_reserved":0,"allocated":796372992,"data_stored":796021314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":567867850},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495977,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575236,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2269040,"kb_used_data":1565928,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":91476112,"statfs":{"total":95995035648,"available":93671538688,"internally_reserved":0,"allocated":1603510272,"data_stored":1603158074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":719976902},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016645,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483333,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1722664,"kb_used_data":1587492,"kb_used_omap":4,"kb_used_meta":135035,"kb_avail":92022488,"statfs":{"total":95995035648,"available":94231027712,"internally_reserved":0,"allocated":1625591808,"data_stored":1625243682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":138276547},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150075,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3199088,"kb_used_data":2437544,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":90546064,"statfs":{"total":95995035648,"available":92719169536,"internally_reserved":0,"allocated":2496045056,"data_stored":2495692858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":779680199},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1009256,"kb_used_data":490660,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92735896,"statfs":{"total":95995035648,"available":94961557504,"internally_reserved":0,"allocated":502435840,"data_stored":502087714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790318,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1411736,"kb_used_data":794704,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92333416,"statfs":{"total":95995035648,"available":94549417984,"internally_reserved":0,"allocated":813776896,"data_stored":813374646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1598492672,"data_stored":1598488600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1617887232,"data_stored":1617887232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2490306560,"data_stored":2490302488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":501743616,"data_stored":501743616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":811532288,"data_stored":811532288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":793640960,"data_stored":793640960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2012807168,"data_stored":2012807168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1580072960,"data_stored":1580072960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1636827136,"data_stored":1636827136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:31:40.792 INFO:tasks.thrashosds.thrasher:Setting 7.4 to [10, 11, 7, 9] 2024-05-15T23:31:40.792 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '7.4', '10', '11', '7', '9'] 2024-05-15T23:31:40.793 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pg-upmap-items 7.4 10 11 7 9 2024-05-15T23:31:40.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:31:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:31:40.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:40.918 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:31:41.019 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:31:41.045 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:41.065 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:31:41.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:41.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:31:41.170 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:31:41.198 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:41.219 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:31:41.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:41.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:31:41.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:41 smithi088 ceph-mon[44699]: pgmap v1614: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 510 op/s; 4524/192914 objects misplaced (2.345%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T23:31:41.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:31:41.350 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:41.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:41 smithi084 ceph-mon[39784]: pgmap v1614: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 510 op/s; 4524/192914 objects misplaced (2.345%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T23:31:41.387 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:31:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:41.391 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:31:41.492 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:31:41.519 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:41.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:31:41.539 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:31:41.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:41.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:31:41.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:41 smithi090 ceph-mon[44434]: pgmap v1614: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 510 op/s; 4524/192914 objects misplaced (2.345%); 2.7 MiB/s, 42 objects/s recovering 2024-05-15T23:31:41.643 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:31:41.670 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:41.691 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:31:41.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:41.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:31:41.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:31:41.827 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:41.855 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:31:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:41.861 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:31:41.962 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:31:41.991 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:43.317 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:31:43.325 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:31:43.354 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:31:43.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:43.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:31:43.399 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:31:43.464 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:31:43.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:43 smithi084 ceph-mon[39784]: pgmap v1615: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.8 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 644 op/s; 4524/188424 objects misplaced (2.401%); 2.5 MiB/s, 40 objects/s recovering 2024-05-15T23:31:43.548 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:43 smithi090 ceph-mon[44434]: pgmap v1615: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.8 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 644 op/s; 4524/188424 objects misplaced (2.401%); 2.5 MiB/s, 40 objects/s recovering 2024-05-15T23:31:43.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:43 smithi088 ceph-mon[44699]: pgmap v1615: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.8 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 644 op/s; 4524/188424 objects misplaced (2.401%); 2.5 MiB/s, 40 objects/s recovering 2024-05-15T23:31:43.613 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:31:43.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:43.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:31:43.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:31:43.758 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:31:43.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:43.782 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:31:43.882 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:31:43.909 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:43.928 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:31:43.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:43.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:31:44.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:31:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:31:44.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:44.098 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:31:44.199 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:31:44.226 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:44.226 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:44 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:31:44.248 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:31:44.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:44.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:31:44.353 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:31:44.379 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:44.399 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:31:44.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:44.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:31:44.505 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:31:44.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:44 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:31:44.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:44 smithi084 ceph-mon[39784]: osdmap e604: 12 total, 12 up, 9 in 2024-05-15T23:31:44.530 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:44 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:31:44.530 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:44 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:31:44.530 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:44 smithi088 ceph-mon[44699]: osdmap e604: 12 total, 12 up, 9 in 2024-05-15T23:31:44.532 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:44.551 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:31:44.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:44 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:31:44.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:44 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:31:44.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:44 smithi090 ceph-mon[44434]: osdmap e604: 12 total, 12 up, 9 in 2024-05-15T23:31:44.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:44.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:31:44.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:31:44.695 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:44.716 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:31:44.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:44.720 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:31:44.822 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:31:44.848 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:44.870 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:31:44.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:44.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:31:44.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:31:45.021 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:45.051 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:31:45.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:45.055 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:31:45.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:31:45.214 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:45.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:45 smithi090 ceph-mon[44434]: pgmap v1617: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.7 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 644 op/s; 4524/187620 objects misplaced (2.411%); 2.5 MiB/s, 40 objects/s recovering 2024-05-15T23:31:45.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:45 smithi088 ceph-mon[44699]: pgmap v1617: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.7 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 644 op/s; 4524/187620 objects misplaced (2.411%); 2.5 MiB/s, 40 objects/s recovering 2024-05-15T23:31:45.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:45 smithi084 ceph-mon[39784]: pgmap v1617: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.7 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 644 op/s; 4524/187620 objects misplaced (2.411%); 2.5 MiB/s, 40 objects/s recovering 2024-05-15T23:31:47.447 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:31:47.448 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:31:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:47 smithi090 ceph-mon[44434]: pgmap v1618: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 645 op/s; 4330/183106 objects misplaced (2.365%); 2.4 MiB/s, 38 objects/s recovering 2024-05-15T23:31:47.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:47 smithi088 ceph-mon[44699]: pgmap v1618: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 645 op/s; 4330/183106 objects misplaced (2.365%); 2.4 MiB/s, 38 objects/s recovering 2024-05-15T23:31:47.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:47 smithi084 ceph-mon[39784]: pgmap v1618: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 645 op/s; 4330/183106 objects misplaced (2.365%); 2.4 MiB/s, 38 objects/s recovering 2024-05-15T23:31:48.205 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:31:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:48.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:31:48.309 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:31:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:48.355 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:31:48.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:48.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:31:48.459 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:31:48.530 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:48.880 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:31:49.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/444916150' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:31:49.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:48 smithi088 ceph-mon[44699]: from='client.26809 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:31:49.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/444916150' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:31:49.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:48 smithi084 ceph-mon[39784]: from='client.26809 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:31:49.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/444916150' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:31:49.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:48 smithi090 ceph-mon[44434]: from='client.26809 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:31:49.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:49 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:31:49.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:49 smithi084 ceph-mon[39784]: from='client.26809 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:31:49.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:49 smithi084 ceph-mon[39784]: osdmap e605: 12 total, 12 up, 9 in 2024-05-15T23:31:49.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:49 smithi084 ceph-mon[39784]: pgmap v1620: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.5 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 691 op/s; 4330/181834 objects misplaced (2.381%); 1.5 MiB/s, 24 objects/s recovering 2024-05-15T23:31:50.018 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:31:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:50.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:31:50.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:49 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:31:50.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:49 smithi088 ceph-mon[44699]: from='client.26809 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:31:50.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:49 smithi088 ceph-mon[44699]: osdmap e605: 12 total, 12 up, 9 in 2024-05-15T23:31:50.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:49 smithi088 ceph-mon[44699]: pgmap v1620: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.5 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 691 op/s; 4330/181834 objects misplaced (2.381%); 1.5 MiB/s, 24 objects/s recovering 2024-05-15T23:31:50.128 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:31:50.195 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:50.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:49 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:31:50.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:49 smithi090 ceph-mon[44434]: from='client.26809 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:31:50.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:49 smithi090 ceph-mon[44434]: osdmap e605: 12 total, 12 up, 9 in 2024-05-15T23:31:50.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:49 smithi090 ceph-mon[44434]: pgmap v1620: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.5 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 691 op/s; 4330/181834 objects misplaced (2.381%); 1.5 MiB/s, 24 objects/s recovering 2024-05-15T23:31:50.561 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:31:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:50.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:31:50.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T23:31:50.668 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:31:50.695 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:50.714 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:31:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:50.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:31:50.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:31:50.850 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:50.874 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:31:50.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:50.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:31:50.979 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:31:51.006 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:51.025 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:31:51.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:51.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:31:51.129 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:31:51.155 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:51.175 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:31:51.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:51.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:31:51.279 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:31:51.322 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:51.369 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:31:51.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:51.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:31:51.473 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:31:51.501 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:51.520 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:31:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:51.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:31:51.624 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:31:51.650 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:51.670 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:31:51.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:51.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:31:51.775 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:31:51.801 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:51.821 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:31:51.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:51.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:31:51.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:31:51.999 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:52.029 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:31:52.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:52.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:31:52.137 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:31:52.164 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:52.185 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:31:52.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:52.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:31:52.289 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:31:52.316 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:52.338 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:31:52.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:52.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:31:52.377 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:52 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:31:52.377 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:52 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:31:52.377 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:52 smithi084 ceph-mon[39784]: pgmap v1621: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 554 op/s; 4103/179548 objects misplaced (2.285%); 3.3 MiB/s, 52 objects/s recovering 2024-05-15T23:31:52.377 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:52 smithi084 ceph-mon[39784]: osdmap e606: 12 total, 12 up, 9 in 2024-05-15T23:31:52.442 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:31:52.508 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:31:52.515 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:52.547 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:31:52.551 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:52 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:31:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:52 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:31:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:52 smithi090 ceph-mon[44434]: pgmap v1621: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 554 op/s; 4103/179548 objects misplaced (2.285%); 3.3 MiB/s, 52 objects/s recovering 2024-05-15T23:31:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:52 smithi090 ceph-mon[44434]: osdmap e606: 12 total, 12 up, 9 in 2024-05-15T23:31:52.554 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:31:52.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:52.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:31:52.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:52 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:31:52.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:52 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:31:52.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:52 smithi088 ceph-mon[44699]: pgmap v1621: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 554 op/s; 4103/179548 objects misplaced (2.285%); 3.3 MiB/s, 52 objects/s recovering 2024-05-15T23:31:52.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:52 smithi088 ceph-mon[44699]: osdmap e606: 12 total, 12 up, 9 in 2024-05-15T23:31:52.597 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:31:52.639 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:31:52.660 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:31:52.674 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:31:52.690 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:52.700 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:31:52.713 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:31:52.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:52.720 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:31:52.730 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:31:52.770 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:31:52.802 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:31:52.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:31:52.835 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:31:52.865 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:52.868 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:31:52.889 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:31:52.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:52.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:31:52.894 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:31:52.932 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:31:52.949 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:31:52.949 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:31:52.949 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:31:52.949 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:31:52.949 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:31:52.949 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:31:52.949 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:31:52.950 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:31:52.950 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:31:52.950 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:31:52.950 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:31:52.950 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:31:52.993 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:31:53.036 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:53.238 INFO:teuthology.orchestra.run.smithi084.stderr:set 7.4 pg_upmap_items mapping to [10->11,7->9] 2024-05-15T23:31:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:53 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2174434031' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [10, 11, 7, 9]}]: dispatch 2024-05-15T23:31:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:53 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2659451734' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:31:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:53.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:53 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2174434031' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [10, 11, 7, 9]}]: dispatch 2024-05-15T23:31:53.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:53 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2659451734' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:31:53.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:53.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:53.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:53.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:53 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2174434031' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [10, 11, 7, 9]}]: dispatch 2024-05-15T23:31:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:53 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2659451734' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:31:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:53.722 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:31:54.520 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:31:54.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:54 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:31:54.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:54 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:31:54.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:54 smithi084 ceph-mon[39784]: pgmap v1623: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 856 op/s; 4103/173896 objects misplaced (2.359%); 3.3 MiB/s, 52 objects/s recovering 2024-05-15T23:31:54.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2174434031' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [10, 11, 7, 9]}]': finished 2024-05-15T23:31:54.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:54 smithi084 ceph-mon[39784]: osdmap e607: 12 total, 12 up, 9 in 2024-05-15T23:31:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:54.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:31:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:54 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:31:54.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:54 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:31:54.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:54 smithi090 ceph-mon[44434]: pgmap v1623: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 856 op/s; 4103/173896 objects misplaced (2.359%); 3.3 MiB/s, 52 objects/s recovering 2024-05-15T23:31:54.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2174434031' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [10, 11, 7, 9]}]': finished 2024-05-15T23:31:54.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:54 smithi090 ceph-mon[44434]: osdmap e607: 12 total, 12 up, 9 in 2024-05-15T23:31:54.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:31:54.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:54 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:31:54.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:54 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:31:54.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:54 smithi088 ceph-mon[44699]: pgmap v1623: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 856 op/s; 4103/173896 objects misplaced (2.359%); 3.3 MiB/s, 52 objects/s recovering 2024-05-15T23:31:54.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2174434031' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [10, 11, 7, 9]}]': finished 2024-05-15T23:31:54.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:54 smithi088 ceph-mon[44699]: osdmap e607: 12 total, 12 up, 9 in 2024-05-15T23:31:54.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:31:54.757 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:54.789 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:31:54.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:54.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:31:54.893 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:31:54.920 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:54.940 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:31:54.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:54.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:31:55.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:31:55.082 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:55.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:55 smithi090 ceph-mon[44434]: pgmap v1625: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 719 op/s; 4103/172732 objects misplaced (2.375%); 2.2 MiB/s, 35 objects/s recovering 2024-05-15T23:31:55.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:55 smithi088 ceph-mon[44699]: pgmap v1625: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 719 op/s; 4103/172732 objects misplaced (2.375%); 2.2 MiB/s, 35 objects/s recovering 2024-05-15T23:31:55.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:55 smithi084 ceph-mon[39784]: pgmap v1625: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 719 op/s; 4103/172732 objects misplaced (2.375%); 2.2 MiB/s, 35 objects/s recovering 2024-05-15T23:31:56.954 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:31:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:56.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:31:57.061 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:31:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:31:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:57.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:31:57.211 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:31:57.272 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:57.434 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:31:57.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:57.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:31:57.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:31:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:57 smithi090 ceph-mon[44434]: pgmap v1626: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 944 op/s; 3823/166710 objects misplaced (2.293%); 4.0 MiB/s, 63 objects/s recovering 2024-05-15T23:31:57.564 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:31:57.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:57 smithi088 ceph-mon[44699]: pgmap v1626: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 944 op/s; 3823/166710 objects misplaced (2.293%); 4.0 MiB/s, 63 objects/s recovering 2024-05-15T23:31:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:57 smithi084 ceph-mon[39784]: pgmap v1626: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 944 op/s; 3823/166710 objects misplaced (2.293%); 4.0 MiB/s, 63 objects/s recovering 2024-05-15T23:31:57.646 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:31:57.646 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-05-15T23:31:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-05-15T23:31:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:31:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:169823 7.e 7:78b6abc8:::benchmark_data_smithi090_143373_object23652:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:31:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:02.738352+0000", 2024-05-15T23:31:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 54.880964445000004, 2024-05-15T23:31:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.179006257, 2024-05-15T23:31:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.647 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 169823 2024-05-15T23:31:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738352+0000", 2024-05-15T23:31:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738352+0000", 2024-05-15T23:31:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738354+0000", 2024-05-15T23:31:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4670000000000001e-06 2024-05-15T23:31:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738359+0000", 2024-05-15T23:31:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.113e-06 2024-05-15T23:31:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738368+0000", 2024-05-15T23:31:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9080000000000001e-06 2024-05-15T23:31:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738374+0000", 2024-05-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2319999999999998e-06 2024-05-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738442+0000", 2024-05-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7432000000000001e-05 2024-05-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738570+0000", 2024-05-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128147 2024-05-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 10", 2024-05-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738691+0000", 2024-05-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121704 2024-05-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.743746+0000", 2024-05-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050545909999999998 2024-05-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.917254+0000", 2024-05-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.173508254 2024-05-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.917310+0000", 2024-05-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5937999999999999e-05 2024-05-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.917359+0000", 2024-05-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8470999999999997e-05 2024-05-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:187219 7.8 7:17d271df:::benchmark_data_smithi090_143373_object41048:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-05-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:24.325708+0000", 2024-05-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 33.293608648000003, 2024-05-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22588118600000001, 2024-05-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 187219 2024-05-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.325708+0000", 2024-05-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.325708+0000", 2024-05-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.325709+0000", 2024-05-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.99e-07 2024-05-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.325712+0000", 2024-05-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9969999999999999e-06 2024-05-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.325717+0000", 2024-05-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1710000000000001e-06 2024-05-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.325722+0000", 2024-05-15T23:31:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.198e-06 2024-05-15T23:31:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.342296+0000", 2024-05-15T23:31:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016573944 2024-05-15T23:31:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.342433+0000", 2024-05-15T23:31:57.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000136339 2024-05-15T23:31:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.342557+0000", 2024-05-15T23:31:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012408300000000001 2024-05-15T23:31:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.347724+0000", 2024-05-15T23:31:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051678649999999998 2024-05-15T23:31:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.551460+0000", 2024-05-15T23:31:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.20373519600000001 2024-05-15T23:31:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.551519+0000", 2024-05-15T23:31:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8953999999999998e-05 2024-05-15T23:31:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.551589+0000", 2024-05-15T23:31:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0740000000000004e-05 2024-05-15T23:31:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:187251 7.8 7:182edd5f:::benchmark_data_smithi090_143373_object41080:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-05-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:24.357546+0000", 2024-05-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 33.261770675999998, 2024-05-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.224264303, 2024-05-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 187251 2024-05-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.357546+0000", 2024-05-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.357546+0000", 2024-05-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.357549+0000", 2024-05-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0120000000000001e-06 2024-05-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.357557+0000", 2024-05-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5070000000000003e-06 2024-05-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.357571+0000", 2024-05-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3896e-05 2024-05-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.357580+0000", 2024-05-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9500000000000007e-06 2024-05-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.384342+0000", 2024-05-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026762135999999999 2024-05-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.384470+0000", 2024-05-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012784999999999999 2024-05-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.384582+0000", 2024-05-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112768 2024-05-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394572+0000", 2024-05-15T23:31:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0099897700000000002 2024-05-15T23:31:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581707+0000", 2024-05-15T23:31:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.187134464 2024-05-15T23:31:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581756+0000", 2024-05-15T23:31:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9796999999999998e-05 2024-05-15T23:31:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581810+0000", 2024-05-15T23:31:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4153000000000001e-05 2024-05-15T23:31:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:187268 7.8 7:15214b14:::benchmark_data_smithi090_143373_object41097:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-05-15T23:31:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:24.382433+0000", 2024-05-15T23:31:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 33.236884285999999, 2024-05-15T23:31:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.199442491, 2024-05-15T23:31:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.667 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 187268 2024-05-15T23:31:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.382433+0000", 2024-05-15T23:31:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.382433+0000", 2024-05-15T23:31:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.382435+0000", 2024-05-15T23:31:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1019999999999999e-06 2024-05-15T23:31:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.382442+0000", 2024-05-15T23:31:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2969999999999998e-06 2024-05-15T23:31:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.382455+0000", 2024-05-15T23:31:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3443e-05 2024-05-15T23:31:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.382464+0000", 2024-05-15T23:31:57.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0159999999999997e-06 2024-05-15T23:31:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.389315+0000", 2024-05-15T23:31:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068508670000000001 2024-05-15T23:31:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.389433+0000", 2024-05-15T23:31:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011765 2024-05-15T23:31:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.389547+0000", 2024-05-15T23:31:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114188 2024-05-15T23:31:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394618+0000", 2024-05-15T23:31:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050712370000000001 2024-05-15T23:31:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581820+0000", 2024-05-15T23:31:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.187201271 2024-05-15T23:31:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581847+0000", 2024-05-15T23:31:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7373e-05 2024-05-15T23:31:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581875+0000", 2024-05-15T23:31:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8047000000000001e-05 2024-05-15T23:31:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:187277 7.8 7:1f6a15c7:::benchmark_data_smithi090_143373_object41106:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-05-15T23:31:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:24.390330+0000", 2024-05-15T23:31:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 33.228986411999998, 2024-05-15T23:31:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19160165600000001, 2024-05-15T23:31:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.677 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 187277 2024-05-15T23:31:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.390330+0000", 2024-05-15T23:31:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.390330+0000", 2024-05-15T23:31:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.390333+0000", 2024-05-15T23:31:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1380000000000001e-06 2024-05-15T23:31:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.390339+0000", 2024-05-15T23:31:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3949999999999998e-06 2024-05-15T23:31:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.390351+0000", 2024-05-15T23:31:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2028000000000001e-05 2024-05-15T23:31:57.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.390359+0000", 2024-05-15T23:31:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8320000000000006e-06 2024-05-15T23:31:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394646+0000", 2024-05-15T23:31:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042872860000000004 2024-05-15T23:31:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394773+0000", 2024-05-15T23:31:57.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012736400000000001 2024-05-15T23:31:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394896+0000", 2024-05-15T23:31:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000122791 2024-05-15T23:31:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.399676+0000", 2024-05-15T23:31:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047800589999999997 2024-05-15T23:31:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581882+0000", 2024-05-15T23:31:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18220559 2024-05-15T23:31:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581908+0000", 2024-05-15T23:31:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5604000000000001e-05 2024-05-15T23:31:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581932+0000", 2024-05-15T23:31:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4569000000000001e-05 2024-05-15T23:31:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.682 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:187279 7.8 7:174bce0b:::benchmark_data_smithi090_143373_object41108:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-05-15T23:31:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:24.394439+0000", 2024-05-15T23:31:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 33.224877700999997, 2024-05-15T23:31:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18754549300000001, 2024-05-15T23:31:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.683 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 187279 2024-05-15T23:31:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394439+0000", 2024-05-15T23:31:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394439+0000", 2024-05-15T23:31:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394442+0000", 2024-05-15T23:31:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1209999999999998e-06 2024-05-15T23:31:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394449+0000", 2024-05-15T23:31:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5259999999999999e-06 2024-05-15T23:31:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394458+0000", 2024-05-15T23:31:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4849999999999998e-06 2024-05-15T23:31:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394466+0000", 2024-05-15T23:31:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4580000000000004e-06 2024-05-15T23:31:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.399705+0000", 2024-05-15T23:31:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0052392350000000001 2024-05-15T23:31:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.399810+0000", 2024-05-15T23:31:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105234 2024-05-15T23:31:57.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.399916+0000", 2024-05-15T23:31:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000106261 2024-05-15T23:31:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.549470+0000", 2024-05-15T23:31:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14955318000000001 2024-05-15T23:31:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581939+0000", 2024-05-15T23:31:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032468887000000002 2024-05-15T23:31:57.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581961+0000", 2024-05-15T23:31:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2056999999999999e-05 2024-05-15T23:31:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581985+0000", 2024-05-15T23:31:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4049e-05 2024-05-15T23:31:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:188105 7.8 7:10ce08d9:::benchmark_data_smithi090_143373_object41934:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-05-15T23:31:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:25.700383+0000", 2024-05-15T23:31:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 31.918934264000001, 2024-05-15T23:31:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19831671200000001, 2024-05-15T23:31:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 188105 2024-05-15T23:31:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.689 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.700383+0000", 2024-05-15T23:31:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.700383+0000", 2024-05-15T23:31:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.700386+0000", 2024-05-15T23:31:57.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3699999999999999e-06 2024-05-15T23:31:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.700409+0000", 2024-05-15T23:31:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3142e-05 2024-05-15T23:31:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.700424+0000", 2024-05-15T23:31:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5024e-05 2024-05-15T23:31:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.700434+0000", 2024-05-15T23:31:57.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0176999999999999e-05 2024-05-15T23:31:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.708159+0000", 2024-05-15T23:31:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0077245250000000003 2024-05-15T23:31:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.708258+0000", 2024-05-15T23:31:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8877999999999998e-05 2024-05-15T23:31:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.708336+0000", 2024-05-15T23:31:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8468999999999997e-05 2024-05-15T23:31:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.882714+0000", 2024-05-15T23:31:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.17437766099999999 2024-05-15T23:31:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.898625+0000", 2024-05-15T23:31:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015911262999999998 2024-05-15T23:31:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.898658+0000", 2024-05-15T23:31:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2483000000000001e-05 2024-05-15T23:31:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.898699+0000", 2024-05-15T23:31:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.172e-05 2024-05-15T23:31:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:188117 7.8 7:10ee05b8:::benchmark_data_smithi090_143373_object41946:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-05-15T23:31:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:25.713894+0000", 2024-05-15T23:31:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 31.905423138, 2024-05-15T23:31:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.188971004, 2024-05-15T23:31:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 188117 2024-05-15T23:31:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.695 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.713894+0000", 2024-05-15T23:31:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.713894+0000", 2024-05-15T23:31:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.713896+0000", 2024-05-15T23:31:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.046e-06 2024-05-15T23:31:57.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.713901+0000", 2024-05-15T23:31:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3299999999999998e-06 2024-05-15T23:31:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.713915+0000", 2024-05-15T23:31:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3570000000000001e-05 2024-05-15T23:31:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.713924+0000", 2024-05-15T23:31:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0310000000000004e-06 2024-05-15T23:31:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.883514+0000", 2024-05-15T23:31:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16959009 2024-05-15T23:31:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.883636+0000", 2024-05-15T23:31:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121903 2024-05-15T23:31:57.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.883743+0000", 2024-05-15T23:31:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107105 2024-05-15T23:31:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.898512+0000", 2024-05-15T23:31:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014769521000000001 2024-05-15T23:31:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.902769+0000", 2024-05-15T23:31:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042569549999999998 2024-05-15T23:31:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.902815+0000", 2024-05-15T23:31:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6051999999999999e-05 2024-05-15T23:31:57.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.902865+0000", 2024-05-15T23:31:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9400999999999997e-05 2024-05-15T23:31:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190306 7.8 7:104adfd2:::benchmark_data_smithi090_143373_object44135:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:31:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.088517+0000", 2024-05-15T23:31:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 28.530799881, 2024-05-15T23:31:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.25864289899999998, 2024-05-15T23:31:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190306 2024-05-15T23:31:57.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.088517+0000", 2024-05-15T23:31:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.088517+0000", 2024-05-15T23:31:57.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.088518+0000", 2024-05-15T23:31:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9999999999999997e-07 2024-05-15T23:31:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.088520+0000", 2024-05-15T23:31:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6989999999999999e-06 2024-05-15T23:31:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.088526+0000", 2024-05-15T23:31:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1999999999999999e-06 2024-05-15T23:31:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.088532+0000", 2024-05-15T23:31:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3789999999999997e-06 2024-05-15T23:31:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.180699+0000", 2024-05-15T23:31:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.092167522000000002 2024-05-15T23:31:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.180835+0000", 2024-05-15T23:31:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013607499999999999 2024-05-15T23:31:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.180980+0000", 2024-05-15T23:31:57.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144726 2024-05-15T23:31:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.196243+0000", 2024-05-15T23:31:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015262805000000001 2024-05-15T23:31:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347041+0000", 2024-05-15T23:31:57.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15079778199999999 2024-05-15T23:31:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347097+0000", 2024-05-15T23:31:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5767999999999999e-05 2024-05-15T23:31:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347160+0000", 2024-05-15T23:31:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3343000000000007e-05 2024-05-15T23:31:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190324 7.8 7:1ecb0c0e:::benchmark_data_smithi090_143373_object44153:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:31:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.118138+0000", 2024-05-15T23:31:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 28.501178628000002, 2024-05-15T23:31:57.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22912729200000001, 2024-05-15T23:31:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190324 2024-05-15T23:31:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.118138+0000", 2024-05-15T23:31:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.118138+0000", 2024-05-15T23:31:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.118140+0000", 2024-05-15T23:31:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.694e-06 2024-05-15T23:31:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.118145+0000", 2024-05-15T23:31:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6059999999999996e-06 2024-05-15T23:31:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.118154+0000", 2024-05-15T23:31:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.465e-06 2024-05-15T23:31:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.118161+0000", 2024-05-15T23:31:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9689999999999997e-06 2024-05-15T23:31:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.185795+0000", 2024-05-15T23:31:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.067634 2024-05-15T23:31:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.185909+0000", 2024-05-15T23:31:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011406800000000001 2024-05-15T23:31:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.186023+0000", 2024-05-15T23:31:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113638 2024-05-15T23:31:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.196285+0000", 2024-05-15T23:31:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010262011 2024-05-15T23:31:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347169+0000", 2024-05-15T23:31:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.150883885 2024-05-15T23:31:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347215+0000", 2024-05-15T23:31:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6517999999999997e-05 2024-05-15T23:31:57.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347266+0000", 2024-05-15T23:31:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0438000000000001e-05 2024-05-15T23:31:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190337 7.8 7:1243b066:::benchmark_data_smithi090_143373_object44166:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:31:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.141526+0000", 2024-05-15T23:31:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 28.477790747, 2024-05-15T23:31:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.205845111, 2024-05-15T23:31:57.716 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.717 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.717 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.717 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.717 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.717 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.717 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190337 2024-05-15T23:31:57.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.717 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.141526+0000", 2024-05-15T23:31:57.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.141526+0000", 2024-05-15T23:31:57.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.141528+0000", 2024-05-15T23:31:57.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1179999999999999e-06 2024-05-15T23:31:57.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.141535+0000", 2024-05-15T23:31:57.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5400000000000001e-06 2024-05-15T23:31:57.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.141548+0000", 2024-05-15T23:31:57.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2974e-05 2024-05-15T23:31:57.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.141555+0000", 2024-05-15T23:31:57.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2490000000000004e-06 2024-05-15T23:31:57.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.191514+0000", 2024-05-15T23:31:57.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.049959274999999997 2024-05-15T23:31:57.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.191614+0000", 2024-05-15T23:31:57.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9795000000000003e-05 2024-05-15T23:31:57.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.191698+0000", 2024-05-15T23:31:57.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3929000000000002e-05 2024-05-15T23:31:57.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.196325+0000", 2024-05-15T23:31:57.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046271910000000001 2024-05-15T23:31:57.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347273+0000", 2024-05-15T23:31:57.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15094758799999999 2024-05-15T23:31:57.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347319+0000", 2024-05-15T23:31:57.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6467000000000001e-05 2024-05-15T23:31:57.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347371+0000", 2024-05-15T23:31:57.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1984999999999998e-05 2024-05-15T23:31:57.722 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.722 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.722 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.722 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190342 7.8 7:1167675d:::benchmark_data_smithi090_143373_object44171:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:31:57.722 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.147801+0000", 2024-05-15T23:31:57.722 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 28.471515413999999, 2024-05-15T23:31:57.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19967811099999999, 2024-05-15T23:31:57.722 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.722 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.722 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.722 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.723 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.723 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.723 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190342 2024-05-15T23:31:57.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.723 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.147801+0000", 2024-05-15T23:31:57.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.147801+0000", 2024-05-15T23:31:57.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.147804+0000", 2024-05-15T23:31:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3080000000000002e-06 2024-05-15T23:31:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.147810+0000", 2024-05-15T23:31:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4489999999999996e-06 2024-05-15T23:31:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.147823+0000", 2024-05-15T23:31:57.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2935999999999999e-05 2024-05-15T23:31:57.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.147831+0000", 2024-05-15T23:31:57.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3229999999999998e-06 2024-05-15T23:31:57.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.197606+0000", 2024-05-15T23:31:57.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.049774102000000001 2024-05-15T23:31:57.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.197743+0000", 2024-05-15T23:31:57.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013782000000000001 2024-05-15T23:31:57.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.197887+0000", 2024-05-15T23:31:57.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000143936 2024-05-15T23:31:57.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.207989+0000", 2024-05-15T23:31:57.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010101715000000001 2024-05-15T23:31:57.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347379+0000", 2024-05-15T23:31:57.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.139389546 2024-05-15T23:31:57.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347424+0000", 2024-05-15T23:31:57.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5210000000000003e-05 2024-05-15T23:31:57.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347480+0000", 2024-05-15T23:31:57.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5766000000000001e-05 2024-05-15T23:31:57.728 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.728 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.728 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.728 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190384 7.8 7:1cd23127:::benchmark_data_smithi090_143373_object44213:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:31:57.728 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.246952+0000", 2024-05-15T23:31:57.728 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 28.372364404999999, 2024-05-15T23:31:57.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29457159700000002, 2024-05-15T23:31:57.728 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.728 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.728 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.728 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.728 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.729 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190384 2024-05-15T23:31:57.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.729 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.246952+0000", 2024-05-15T23:31:57.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.246952+0000", 2024-05-15T23:31:57.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.246953+0000", 2024-05-15T23:31:57.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8200000000000008e-07 2024-05-15T23:31:57.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.246957+0000", 2024-05-15T23:31:57.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9009999999999996e-06 2024-05-15T23:31:57.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.246967+0000", 2024-05-15T23:31:57.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3170000000000008e-06 2024-05-15T23:31:57.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.246972+0000", 2024-05-15T23:31:57.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.451e-06 2024-05-15T23:31:57.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.337967+0000", 2024-05-15T23:31:57.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090994624999999996 2024-05-15T23:31:57.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.338085+0000", 2024-05-15T23:31:57.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000118177 2024-05-15T23:31:57.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.338199+0000", 2024-05-15T23:31:57.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113746 2024-05-15T23:31:57.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.357844+0000", 2024-05-15T23:31:57.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019645184999999999 2024-05-15T23:31:57.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541420+0000", 2024-05-15T23:31:57.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18357621800000001 2024-05-15T23:31:57.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541472+0000", 2024-05-15T23:31:57.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1743999999999998e-05 2024-05-15T23:31:57.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541524+0000", 2024-05-15T23:31:57.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2250999999999998e-05 2024-05-15T23:31:57.734 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.734 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.734 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.734 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190413 7.8 7:1fb699e8:::benchmark_data_smithi090_143373_object44242:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:31:57.734 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.298709+0000", 2024-05-15T23:31:57.735 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 28.320607695, 2024-05-15T23:31:57.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.242957495, 2024-05-15T23:31:57.735 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.735 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.735 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.735 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.735 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.735 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.735 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190413 2024-05-15T23:31:57.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.735 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.298709+0000", 2024-05-15T23:31:57.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.298709+0000", 2024-05-15T23:31:57.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.298711+0000", 2024-05-15T23:31:57.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2340000000000001e-06 2024-05-15T23:31:57.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.298718+0000", 2024-05-15T23:31:57.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5939999999999999e-06 2024-05-15T23:31:57.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.298729+0000", 2024-05-15T23:31:57.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1188e-05 2024-05-15T23:31:57.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.298737+0000", 2024-05-15T23:31:57.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.089e-06 2024-05-15T23:31:57.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.342556+0000", 2024-05-15T23:31:57.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043818548999999998 2024-05-15T23:31:57.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.342658+0000", 2024-05-15T23:31:57.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010268399999999999 2024-05-15T23:31:57.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.342765+0000", 2024-05-15T23:31:57.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000106654 2024-05-15T23:31:57.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.357850+0000", 2024-05-15T23:31:57.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015085334000000001 2024-05-15T23:31:57.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541574+0000", 2024-05-15T23:31:57.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18372361000000001 2024-05-15T23:31:57.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541621+0000", 2024-05-15T23:31:57.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7064999999999997e-05 2024-05-15T23:31:57.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541667+0000", 2024-05-15T23:31:57.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5494000000000002e-05 2024-05-15T23:31:57.740 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.741 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.741 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.741 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190443 7.8 7:1c1fbdd5:::benchmark_data_smithi090_143373_object44272:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:31:57.741 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.352457+0000", 2024-05-15T23:31:57.741 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 28.266859356000001, 2024-05-15T23:31:57.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18933589000000001, 2024-05-15T23:31:57.741 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.742 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.742 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.742 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.742 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.742 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.742 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190443 2024-05-15T23:31:57.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.742 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352457+0000", 2024-05-15T23:31:57.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352457+0000", 2024-05-15T23:31:57.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352459+0000", 2024-05-15T23:31:57.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.922e-06 2024-05-15T23:31:57.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352465+0000", 2024-05-15T23:31:57.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7819999999999999e-06 2024-05-15T23:31:57.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352479+0000", 2024-05-15T23:31:57.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3599e-05 2024-05-15T23:31:57.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352487+0000", 2024-05-15T23:31:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8830000000000006e-06 2024-05-15T23:31:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352537+0000", 2024-05-15T23:31:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0608000000000001e-05 2024-05-15T23:31:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352671+0000", 2024-05-15T23:31:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013372300000000001 2024-05-15T23:31:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352803+0000", 2024-05-15T23:31:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000131583 2024-05-15T23:31:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.357960+0000", 2024-05-15T23:31:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051577619999999998 2024-05-15T23:31:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541704+0000", 2024-05-15T23:31:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18374388999999999 2024-05-15T23:31:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541751+0000", 2024-05-15T23:31:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7067000000000002e-05 2024-05-15T23:31:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541793+0000", 2024-05-15T23:31:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2070999999999998e-05 2024-05-15T23:31:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190446 7.8 7:1d0b779c:::benchmark_data_smithi090_143373_object44275:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:31:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.358424+0000", 2024-05-15T23:31:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 28.260893301999999, 2024-05-15T23:31:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.183500423, 2024-05-15T23:31:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190446 2024-05-15T23:31:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358424+0000", 2024-05-15T23:31:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358424+0000", 2024-05-15T23:31:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358426+0000", 2024-05-15T23:31:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.289e-06 2024-05-15T23:31:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358432+0000", 2024-05-15T23:31:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.657e-06 2024-05-15T23:31:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358448+0000", 2024-05-15T23:31:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5849e-05 2024-05-15T23:31:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358459+0000", 2024-05-15T23:31:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0256e-05 2024-05-15T23:31:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358509+0000", 2024-05-15T23:31:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0121999999999999e-05 2024-05-15T23:31:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358647+0000", 2024-05-15T23:31:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013838000000000001 2024-05-15T23:31:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358770+0000", 2024-05-15T23:31:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000122774 2024-05-15T23:31:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.363625+0000", 2024-05-15T23:31:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048551280000000002 2024-05-15T23:31:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541831+0000", 2024-05-15T23:31:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.17820591199999999 2024-05-15T23:31:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541873+0000", 2024-05-15T23:31:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2531000000000003e-05 2024-05-15T23:31:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541924+0000", 2024-05-15T23:31:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0525000000000002e-05 2024-05-15T23:31:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:192687 7.8 7:17c8a2a5:::benchmark_data_smithi090_143373_object46516:head [delete] snapc 0=[] ondisk+write+known_if_redirected e602)", 2024-05-15T23:31:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:32.936204+0000", 2024-05-15T23:31:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 24.683112623, 2024-05-15T23:31:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.26211031600000001, 2024-05-15T23:31:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 192687 2024-05-15T23:31:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:32.936204+0000", 2024-05-15T23:31:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:32.936204+0000", 2024-05-15T23:31:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:32.936206+0000", 2024-05-15T23:31:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.815e-06 2024-05-15T23:31:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:32.936213+0000", 2024-05-15T23:31:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9959999999999996e-06 2024-05-15T23:31:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:32.936225+0000", 2024-05-15T23:31:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2442999999999999e-05 2024-05-15T23:31:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:32.936233+0000", 2024-05-15T23:31:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4549999999999998e-06 2024-05-15T23:31:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:33.022555+0000", 2024-05-15T23:31:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086322265999999995 2024-05-15T23:31:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:33.022652+0000", 2024-05-15T23:31:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6318999999999997e-05 2024-05-15T23:31:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:33.022750+0000", 2024-05-15T23:31:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8147999999999996e-05 2024-05-15T23:31:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:33.032462+0000", 2024-05-15T23:31:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0097126190000000005 2024-05-15T23:31:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:33.198199+0000", 2024-05-15T23:31:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16573666500000001 2024-05-15T23:31:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:33.198253+0000", 2024-05-15T23:31:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3559e-05 2024-05-15T23:31:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:33.198315+0000", 2024-05-15T23:31:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2031000000000003e-05 2024-05-15T23:31:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:194612 7.8 7:18f80d38:::benchmark_data_smithi090_143373_object48441:head [delete] snapc 0=[] ondisk+write+known_if_redirected e603)", 2024-05-15T23:31:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:35.760666+0000", 2024-05-15T23:31:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 21.858651035000001, 2024-05-15T23:31:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18154204400000001, 2024-05-15T23:31:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 194612 2024-05-15T23:31:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.760666+0000", 2024-05-15T23:31:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.760666+0000", 2024-05-15T23:31:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.760668+0000", 2024-05-15T23:31:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8560000000000001e-06 2024-05-15T23:31:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.760674+0000", 2024-05-15T23:31:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1750000000000002e-06 2024-05-15T23:31:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.760689+0000", 2024-05-15T23:31:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4677e-05 2024-05-15T23:31:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.760698+0000", 2024-05-15T23:31:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1190000000000003e-06 2024-05-15T23:31:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.787608+0000", 2024-05-15T23:31:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026910645 2024-05-15T23:31:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.787717+0000", 2024-05-15T23:31:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010851 2024-05-15T23:31:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.787823+0000", 2024-05-15T23:31:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0001066 2024-05-15T23:31:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.797974+0000", 2024-05-15T23:31:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010150648 2024-05-15T23:31:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.942101+0000", 2024-05-15T23:31:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14412665399999999 2024-05-15T23:31:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.942150+0000", 2024-05-15T23:31:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9190999999999997e-05 2024-05-15T23:31:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.942208+0000", 2024-05-15T23:31:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7969e-05 2024-05-15T23:31:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.767 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:194746 7.8 7:15d745f4:::benchmark_data_smithi090_143373_object48575:head [delete] snapc 0=[] ondisk+write+known_if_redirected e603)", 2024-05-15T23:31:57.767 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:35.969302+0000", 2024-05-15T23:31:57.767 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 21.650014898999999, 2024-05-15T23:31:57.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.194987882, 2024-05-15T23:31:57.767 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.767 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.767 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.767 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.767 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.767 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.767 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 194746 2024-05-15T23:31:57.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969302+0000", 2024-05-15T23:31:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969302+0000", 2024-05-15T23:31:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969303+0000", 2024-05-15T23:31:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.33e-06 2024-05-15T23:31:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969312+0000", 2024-05-15T23:31:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2619999999999998e-06 2024-05-15T23:31:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969321+0000", 2024-05-15T23:31:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9140000000000003e-06 2024-05-15T23:31:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969336+0000", 2024-05-15T23:31:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4968e-05 2024-05-15T23:31:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.989969+0000", 2024-05-15T23:31:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020632265 2024-05-15T23:31:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.990085+0000", 2024-05-15T23:31:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000115883 2024-05-15T23:31:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.990200+0000", 2024-05-15T23:31:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114963 2024-05-15T23:31:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.007820+0000", 2024-05-15T23:31:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017620286999999998 2024-05-15T23:31:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.164163+0000", 2024-05-15T23:31:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15634331700000001 2024-05-15T23:31:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.164222+0000", 2024-05-15T23:31:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9055000000000003e-05 2024-05-15T23:31:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.164290+0000", 2024-05-15T23:31:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7638000000000005e-05 2024-05-15T23:31:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:194755 7.8 7:14a1fd4f:::benchmark_data_smithi090_143373_object48584:head [delete] snapc 0=[] ondisk+write+known_if_redirected e603)", 2024-05-15T23:31:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:35.981439+0000", 2024-05-15T23:31:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 21.637877855999999, 2024-05-15T23:31:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.182957336, 2024-05-15T23:31:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:31:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:31:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:31:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:31:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:31:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:31:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 194755 2024-05-15T23:31:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:31:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:31:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981439+0000", 2024-05-15T23:31:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:31:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981439+0000", 2024-05-15T23:31:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:31:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:31:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981441+0000", 2024-05-15T23:31:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3740000000000001e-06 2024-05-15T23:31:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:31:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981449+0000", 2024-05-15T23:31:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5660000000000001e-06 2024-05-15T23:31:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:31:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981462+0000", 2024-05-15T23:31:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2795e-05 2024-05-15T23:31:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:31:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981468+0000", 2024-05-15T23:31:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4799999999999998e-06 2024-05-15T23:31:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:31:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.002522+0000", 2024-05-15T23:31:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021053477000000001 2024-05-15T23:31:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:31:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.002655+0000", 2024-05-15T23:31:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013380900000000001 2024-05-15T23:31:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:31:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.002781+0000", 2024-05-15T23:31:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000125022 2024-05-15T23:31:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:31:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.153024+0000", 2024-05-15T23:31:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.150243082 2024-05-15T23:31:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:31:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.164296+0000", 2024-05-15T23:31:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011272864 2024-05-15T23:31:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:31:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.164341+0000", 2024-05-15T23:31:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4332999999999997e-05 2024-05-15T23:31:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:31:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:31:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:31:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.164396+0000", 2024-05-15T23:31:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5534000000000001e-05 2024-05-15T23:31:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:31:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:31:57.780 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:31:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:57.782 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:57.782 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:31:57.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:31:57.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:31:57.912 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:57.949 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:31:57.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:57.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:31:58.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:31:58.132 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:58.158 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:31:58.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:58.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:31:58.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:31:58.321 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:58.790 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:31:58.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:58.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:31:58.813 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T23:31:58.897 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:31:58.962 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:58.964 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:31:59.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:31:59 smithi084 ceph-mon[39784]: pgmap v1627: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 908 op/s; 3823/164996 objects misplaced (2.317%); 2.2 MiB/s, 34 objects/s recovering 2024-05-15T23:31:59.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:31:59 smithi090 ceph-mon[44434]: pgmap v1627: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 908 op/s; 3823/164996 objects misplaced (2.317%); 2.2 MiB/s, 34 objects/s recovering 2024-05-15T23:31:59.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:31:59 smithi088 ceph-mon[44699]: pgmap v1627: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 908 op/s; 3823/164996 objects misplaced (2.317%); 2.2 MiB/s, 34 objects/s recovering 2024-05-15T23:31:59.753 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:31:59.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:59.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:31:59.860 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:31:59.888 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:31:59.908 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:31:59.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:31:59.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:32:00.012 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:32:00.038 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:00.060 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:32:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:00.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:32:00.077 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:00.164 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:32:00.224 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:00.273 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:32:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:00.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:32:00.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:32:00.419 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:00.457 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:32:00.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:00.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:32:00.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:00.605 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:00.637 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:00.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:00.741 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:32:00.767 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:00.788 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:32:00.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:00.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:32:00.805 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10, 2] out_osds: [0, 6, 8] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:32:00.805 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:32:00.806 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-15T23:32:00.806 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.413743 2024-05-15T23:32:00.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 1 0.4137430001445378 2024-05-15T23:32:00.892 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:00.937 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:32:00.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:00.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:32:01.045 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:01.102 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:32:01.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:01.106 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:32:01.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:01 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/373397182' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:32:01.206 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:32:01.234 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:01.253 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:32:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:01.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:32:01.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:32:01.397 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:01.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:01 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/373397182' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:32:01.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:01 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/373397182' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:32:02.170 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:32:02.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:02.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:32:02.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:32:02.308 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:02.329 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:32:02.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:02.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:32:02.372 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:32:02.434 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:32:02.458 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:02 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:32:02.458 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/373397182' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:32:02.459 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:02 smithi088 ceph-mon[44699]: osdmap e608: 12 total, 12 up, 9 in 2024-05-15T23:32:02.459 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:02 smithi088 ceph-mon[44699]: pgmap v1629: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 727 op/s; 3551/162248 objects misplaced (2.189%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:32:02.460 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:02.483 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:32:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:02.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:32:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:02 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:32:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/373397182' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:32:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:02 smithi090 ceph-mon[44434]: osdmap e608: 12 total, 12 up, 9 in 2024-05-15T23:32:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:02 smithi090 ceph-mon[44434]: pgmap v1629: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 727 op/s; 3551/162248 objects misplaced (2.189%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:32:02.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:02.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:02 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:32:02.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/373397182' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:32:02.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:02 smithi084 ceph-mon[39784]: osdmap e608: 12 total, 12 up, 9 in 2024-05-15T23:32:02.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:02 smithi084 ceph-mon[39784]: pgmap v1629: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 727 op/s; 3551/162248 objects misplaced (2.189%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:32:02.659 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:02.769 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:02.773 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:02.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:02.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:32:02.932 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:02.971 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:32:02.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:02.984 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:32:03.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:03.122 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:03.166 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:03.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:03.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:03.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:32:03.328 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:03.377 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:32:03.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:03.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:32:03.485 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:32:03.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:03 smithi084 ceph-mon[39784]: pgmap v1630: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 893 op/s; 3551/156082 objects misplaced (2.275%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T23:32:03.536 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:03.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:03 smithi090 ceph-mon[44434]: pgmap v1630: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 893 op/s; 3551/156082 objects misplaced (2.275%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T23:32:03.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:03 smithi088 ceph-mon[44699]: pgmap v1630: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 893 op/s; 3551/156082 objects misplaced (2.275%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T23:32:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:04 smithi090 ceph-mon[44434]: 7.d scrub starts 2024-05-15T23:32:04.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:04 smithi088 ceph-mon[44699]: 7.d scrub starts 2024-05-15T23:32:04.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:04 smithi084 ceph-mon[39784]: 7.d scrub starts 2024-05-15T23:32:05.315 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:32:05.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:05 smithi084 ceph-mon[39784]: 7.d scrub ok 2024-05-15T23:32:05.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:05 smithi084 ceph-mon[39784]: 7.5 scrub starts 2024-05-15T23:32:05.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:05 smithi084 ceph-mon[39784]: pgmap v1631: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 888 op/s; 3551/154954 objects misplaced (2.292%); 3.4 MiB/s, 55 objects/s recovering 2024-05-15T23:32:05.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:05.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:32:05.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:32:05.453 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:05.495 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:32:05.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:05.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:32:05.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:05 smithi090 ceph-mon[44434]: 7.d scrub ok 2024-05-15T23:32:05.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:05 smithi090 ceph-mon[44434]: 7.5 scrub starts 2024-05-15T23:32:05.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:05 smithi090 ceph-mon[44434]: pgmap v1631: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 888 op/s; 3551/154954 objects misplaced (2.292%); 3.4 MiB/s, 55 objects/s recovering 2024-05-15T23:32:05.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:05 smithi088 ceph-mon[44699]: 7.d scrub ok 2024-05-15T23:32:05.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:05 smithi088 ceph-mon[44699]: 7.5 scrub starts 2024-05-15T23:32:05.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:05 smithi088 ceph-mon[44699]: pgmap v1631: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 888 op/s; 3551/154954 objects misplaced (2.292%); 3.4 MiB/s, 55 objects/s recovering 2024-05-15T23:32:05.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:05.630 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:05.654 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:05.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:05.659 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:05.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:32:05.795 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:05.818 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:32:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:05.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:32:05.930 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:32:05.958 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:05.980 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:32:05.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:05.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:32:06.048 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:32:06.048 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:32:06.048 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:32:06.048 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:32:06.048 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:32:06.049 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:32:06.049 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:32:06.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:32:06.062 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:06.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:06.100 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:32:06.130 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:32:06.159 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:32:06.161 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:06.200 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:32:06.235 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:32:06.235 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:32:06.235 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:32:06.236 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:32:06.236 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:32:06.236 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:32:06.236 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:32:06.236 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:32:06.236 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:32:06.236 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:32:06.236 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:32:06.236 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:32:06.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:06 smithi090 ceph-mon[44434]: 7.5 scrub ok 2024-05-15T23:32:06.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:06 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1146996804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:32:06.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:06 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:06.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:06 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:06.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:06 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:06.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:06 smithi088 ceph-mon[44699]: 7.5 scrub ok 2024-05-15T23:32:06.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:06 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1146996804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:32:06.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:06 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:06.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:06 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:06.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:06 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:06.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:06 smithi084 ceph-mon[39784]: 7.5 scrub ok 2024-05-15T23:32:06.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:06 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1146996804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:32:06.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:06 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:06.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:06 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:06.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:06 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:06.661 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:06.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:06.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:06.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:06.799 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:06.839 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:06.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:06.846 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:06.947 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:32:06.973 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:06.995 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:32:06.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:06.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:32:07.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:32:07.099 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:32:07.125 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:07.144 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:32:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:07.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:32:07.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:07.287 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:07.293 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.293 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.293 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:07 smithi084 ceph-mon[39784]: 7.8 scrub starts 2024-05-15T23:32:07.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:07 smithi084 ceph-mon[39784]: pgmap v1632: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 901 op/s; 3286/148678 objects misplaced (2.210%); 3.4 MiB/s, 53 objects/s recovering 2024-05-15T23:32:07.317 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:07.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:07.323 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:07.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:07.454 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:07.484 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:07.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:07.491 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:07.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:07 smithi090 ceph-mon[44434]: 7.8 scrub starts 2024-05-15T23:32:07.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:07 smithi090 ceph-mon[44434]: pgmap v1632: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 901 op/s; 3286/148678 objects misplaced (2.210%); 3.4 MiB/s, 53 objects/s recovering 2024-05-15T23:32:07.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:07 smithi088 ceph-mon[44699]: 7.8 scrub starts 2024-05-15T23:32:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:07 smithi088 ceph-mon[44699]: pgmap v1632: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 901 op/s; 3286/148678 objects misplaced (2.210%); 3.4 MiB/s, 53 objects/s recovering 2024-05-15T23:32:07.592 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:32:07.629 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:08.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:08 smithi090 ceph-mon[44434]: 7.8 scrub ok 2024-05-15T23:32:08.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:08 smithi088 ceph-mon[44699]: 7.8 scrub ok 2024-05-15T23:32:08.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:08 smithi084 ceph-mon[39784]: 7.8 scrub ok 2024-05-15T23:32:08.789 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:32:08.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:08.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:32:08.901 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:32:08.940 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:09.306 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:32:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:09.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:32:09.420 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:09.482 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:09.568 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:09.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:09 smithi088 ceph-mon[44699]: 7.9 scrub starts 2024-05-15T23:32:09.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:09 smithi088 ceph-mon[44699]: pgmap v1633: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 898 op/s; 3286/147010 objects misplaced (2.235%); 3.4 MiB/s, 53 objects/s recovering 2024-05-15T23:32:09.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:09.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:09.577 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:09 smithi084 ceph-mon[39784]: 7.9 scrub starts 2024-05-15T23:32:09.577 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:09 smithi084 ceph-mon[39784]: pgmap v1633: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 898 op/s; 3286/147010 objects misplaced (2.235%); 3.4 MiB/s, 53 objects/s recovering 2024-05-15T23:32:09.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:32:09.734 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:09.789 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:32:09.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:09.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:32:09.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:09 smithi090 ceph-mon[44434]: 7.9 scrub starts 2024-05-15T23:32:09.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:09 smithi090 ceph-mon[44434]: pgmap v1633: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 898 op/s; 3286/147010 objects misplaced (2.235%); 3.4 MiB/s, 53 objects/s recovering 2024-05-15T23:32:09.893 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:09.935 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:09.977 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:32:09.977 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:32:10.068 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:10.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:10.174 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:32:10.210 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:10.250 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:32:10.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:10.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:32:10.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:10 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:32:10.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:10 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:32:10.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:10 smithi084 ceph-mon[39784]: 7.e scrub starts 2024-05-15T23:32:10.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:10 smithi084 ceph-mon[39784]: 7.9 scrub ok 2024-05-15T23:32:10.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:10.420 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:10.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:10 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:32:10.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:10 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:32:10.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:10 smithi090 ceph-mon[44434]: 7.e scrub starts 2024-05-15T23:32:10.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:10 smithi090 ceph-mon[44434]: 7.9 scrub ok 2024-05-15T23:32:10.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:10 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:32:10.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:10 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:32:10.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:10 smithi088 ceph-mon[44699]: 7.e scrub starts 2024-05-15T23:32:10.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:10 smithi088 ceph-mon[44699]: 7.9 scrub ok 2024-05-15T23:32:10.592 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:10.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:10.705 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:10.756 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:10.842 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:10.845 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:10.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:10.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:10.951 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:10.995 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:11.102 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:11.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:11.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:11.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:32:11.247 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:11.281 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:32:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:11.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:32:11.332 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:32:11.389 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:32:11.413 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:11 smithi084 ceph-mon[39784]: 7.e scrub ok 2024-05-15T23:32:11.413 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:11 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:32:11.413 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:11 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:32:11.413 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:11 smithi084 ceph-mon[39784]: pgmap v1634: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 896 op/s; 3002/144170 objects misplaced (2.082%); 3.4 MiB/s, 54 objects/s recovering 2024-05-15T23:32:11.415 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:11.436 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:32:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:11.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:32:11.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:32:11.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:11 smithi090 ceph-mon[44434]: 7.e scrub ok 2024-05-15T23:32:11.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:11 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:32:11.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:11 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:32:11.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:11 smithi090 ceph-mon[44434]: pgmap v1634: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 896 op/s; 3002/144170 objects misplaced (2.082%); 3.4 MiB/s, 54 objects/s recovering 2024-05-15T23:32:11.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:11 smithi088 ceph-mon[44699]: 7.e scrub ok 2024-05-15T23:32:11.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:11 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:32:11.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:11 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:32:11.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:11 smithi088 ceph-mon[44699]: pgmap v1634: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 896 op/s; 3002/144170 objects misplaced (2.082%); 3.4 MiB/s, 54 objects/s recovering 2024-05-15T23:32:11.573 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:11.599 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:32:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:11.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:32:11.712 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:32:11.739 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:11.758 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:32:11.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:11.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:32:11.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:32:11.947 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:12.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:12 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:32:12.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:12 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:32:12.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:12 smithi084 ceph-mon[39784]: osdmap e609: 12 total, 12 up, 9 in 2024-05-15T23:32:12.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:12 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3175335296' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.4137430001445378}]: dispatch 2024-05-15T23:32:12.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:12 smithi084 ceph-mon[39784]: from='client.36788 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.4137430001445378}]: dispatch 2024-05-15T23:32:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:12 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:32:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:12 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:32:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:12 smithi090 ceph-mon[44434]: osdmap e609: 12 total, 12 up, 9 in 2024-05-15T23:32:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:12 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3175335296' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.4137430001445378}]: dispatch 2024-05-15T23:32:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:12 smithi090 ceph-mon[44434]: from='client.36788 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.4137430001445378}]: dispatch 2024-05-15T23:32:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:12 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:32:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:12 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:32:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:12 smithi088 ceph-mon[44699]: osdmap e609: 12 total, 12 up, 9 in 2024-05-15T23:32:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:12 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3175335296' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.4137430001445378}]: dispatch 2024-05-15T23:32:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:12 smithi088 ceph-mon[44699]: from='client.36788 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.4137430001445378}]: dispatch 2024-05-15T23:32:12.948 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.1 primary-affinity to 0.413743 (8271152) 2024-05-15T23:32:13.213 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:32:13.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:13.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:32:13.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:32:13.350 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:13 smithi084 ceph-mon[39784]: from='client.36788 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.4137430001445378}]': finished 2024-05-15T23:32:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:13 smithi084 ceph-mon[39784]: osdmap e610: 12 total, 12 up, 9 in 2024-05-15T23:32:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:13 smithi084 ceph-mon[39784]: pgmap v1637: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 4.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 3002/138030 objects misplaced (2.175%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:32:13.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:13 smithi084 ceph-mon[39784]: osdmap e611: 12 total, 12 up, 9 in 2024-05-15T23:32:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:13 smithi090 ceph-mon[44434]: from='client.36788 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.4137430001445378}]': finished 2024-05-15T23:32:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:13 smithi090 ceph-mon[44434]: osdmap e610: 12 total, 12 up, 9 in 2024-05-15T23:32:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:13 smithi090 ceph-mon[44434]: pgmap v1637: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 4.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 3002/138030 objects misplaced (2.175%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:32:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:13 smithi090 ceph-mon[44434]: osdmap e611: 12 total, 12 up, 9 in 2024-05-15T23:32:13.807 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:32:13.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:13.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:32:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:13 smithi088 ceph-mon[44699]: from='client.36788 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.4137430001445378}]': finished 2024-05-15T23:32:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:13 smithi088 ceph-mon[44699]: osdmap e610: 12 total, 12 up, 9 in 2024-05-15T23:32:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:13 smithi088 ceph-mon[44699]: pgmap v1637: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 4.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 3002/138030 objects misplaced (2.175%); 4.3 MiB/s, 68 objects/s recovering 2024-05-15T23:32:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:13 smithi088 ceph-mon[44699]: osdmap e611: 12 total, 12 up, 9 in 2024-05-15T23:32:13.918 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:32:13.945 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:13.966 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:32:13.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:13.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:32:14.070 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:32:14.096 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:14.116 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:32:14.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:14.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:32:14.221 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:14.287 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:14.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:14 smithi084 ceph-mon[39784]: 7.b scrub starts 2024-05-15T23:32:14.864 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:14.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:14.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:14.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T23:32:14.978 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:32:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:15.025 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:32:15.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:15.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:32:15.129 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:15.158 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:14 smithi090 ceph-mon[44434]: 7.b scrub starts 2024-05-15T23:32:15.160 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:15.189 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:15.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:15.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:32:15.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:14 smithi088 ceph-mon[44699]: 7.b scrub starts 2024-05-15T23:32:15.338 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:15.394 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:32:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:15.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:32:15.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:15.569 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:15.598 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:15.638 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:15.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:15 smithi084 ceph-mon[39784]: pgmap v1639: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 851 op/s; 3002/136774 objects misplaced (2.195%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T23:32:15.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:15 smithi084 ceph-mon[39784]: 7.b scrub ok 2024-05-15T23:32:16.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:15 smithi090 ceph-mon[44434]: pgmap v1639: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 851 op/s; 3002/136774 objects misplaced (2.195%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T23:32:16.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:15 smithi090 ceph-mon[44434]: 7.b scrub ok 2024-05-15T23:32:16.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:15 smithi088 ceph-mon[44699]: pgmap v1639: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 851 op/s; 3002/136774 objects misplaced (2.195%); 3.0 MiB/s, 47 objects/s recovering 2024-05-15T23:32:16.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:15 smithi088 ceph-mon[44699]: 7.b scrub ok 2024-05-15T23:32:16.914 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:32:17.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:16 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2333322363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:32:17.189 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:16 smithi084 ceph-mon[39784]: from='client.36794 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:32:17.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:16 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2333322363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:32:17.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:16 smithi090 ceph-mon[44434]: from='client.36794 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:32:17.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:16 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2333322363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:32:17.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:16 smithi088 ceph-mon[44699]: from='client.36794 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:32:18.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:17 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:32:18.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:17 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:32:18.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:17 smithi084 ceph-mon[39784]: from='client.36794 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:32:18.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:17 smithi084 ceph-mon[39784]: osdmap e612: 12 total, 12 up, 9 in 2024-05-15T23:32:18.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:17 smithi084 ceph-mon[39784]: pgmap v1641: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 2703/131958 objects misplaced (2.048%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T23:32:18.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:17 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:32:18.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:17 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:32:18.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:17 smithi090 ceph-mon[44434]: from='client.36794 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:32:18.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:17 smithi090 ceph-mon[44434]: osdmap e612: 12 total, 12 up, 9 in 2024-05-15T23:32:18.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:17 smithi090 ceph-mon[44434]: pgmap v1641: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 2703/131958 objects misplaced (2.048%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T23:32:18.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:17 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:32:18.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:17 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:32:18.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:17 smithi088 ceph-mon[44699]: from='client.36794 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:32:18.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:17 smithi088 ceph-mon[44699]: osdmap e612: 12 total, 12 up, 9 in 2024-05-15T23:32:18.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:17 smithi088 ceph-mon[44699]: pgmap v1641: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 2703/131958 objects misplaced (2.048%); 3.2 MiB/s, 50 objects/s recovering 2024-05-15T23:32:18.820 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:18.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:18.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:18.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:32:18.977 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:19.186 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:32:19.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:19.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:32:19.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:32:19.335 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:19.372 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:32:19.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:19.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:32:19.476 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:32:19.503 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:19.524 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:32:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:19.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:32:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:19 smithi090 ceph-mon[44434]: pgmap v1642: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 667 op/s; 2703/130420 objects misplaced (2.073%); 2.7 MiB/s, 43 objects/s recovering 2024-05-15T23:32:19.564 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10, 2] out_osds: [0, 6, 8] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:32:19.564 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:32:19.564 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-05-15T23:32:19.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump -f json-pretty 2024-05-15T23:32:19.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:19 smithi088 ceph-mon[44699]: pgmap v1642: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 667 op/s; 2703/130420 objects misplaced (2.073%); 2.7 MiB/s, 43 objects/s recovering 2024-05-15T23:32:19.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:19 smithi084 ceph-mon[39784]: pgmap v1642: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 667 op/s; 2703/130420 objects misplaced (2.073%); 2.7 MiB/s, 43 objects/s recovering 2024-05-15T23:32:19.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:32:19.675 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:19.708 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:32:19.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:19.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:32:19.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:19.876 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:19.926 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:19.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:19.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:20.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:32:20.096 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:20.120 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:32:20.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:20.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:32:20.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:20.267 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:20.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:20.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:20.407 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:32:20.433 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:20.453 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:32:20.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:20.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:32:20.558 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:20.595 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:20.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:20.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:20.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:32:20.768 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:20.796 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:32:20.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:20.804 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:32:20.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:20.952 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:20.982 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:20.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:20.987 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:21.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:21.130 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:21.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:21.161 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:21.239 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:32:21.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:21.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:21 smithi090 ceph-mon[44434]: pgmap v1643: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 744 op/s; 2443/126102 objects misplaced (1.937%); 4.4 MiB/s, 69 objects/s recovering 2024-05-15T23:32:21.296 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:21.332 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:21.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:21.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:21.438 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:32:21.446 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:32:21.446 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:32:21.453 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:32:21.465 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:21.484 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:32:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:21.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:32:21.513 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:32:21.519 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:32:21.519 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:32:21.547 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:32:21.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:21 smithi088 ceph-mon[44699]: pgmap v1643: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 744 op/s; 2443/126102 objects misplaced (1.937%); 4.4 MiB/s, 69 objects/s recovering 2024-05-15T23:32:21.585 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:32:21.588 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:32:21.614 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:21.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:21 smithi084 ceph-mon[39784]: pgmap v1643: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 744 op/s; 2443/126102 objects misplaced (1.937%); 4.4 MiB/s, 69 objects/s recovering 2024-05-15T23:32:21.619 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:32:21.634 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:32:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:21.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:32:21.650 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:32:21.688 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:32:21.724 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:32:21.737 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:21.759 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:32:21.801 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:21.833 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:32:21.833 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:32:21.833 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:32:21.833 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:32:21.833 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:32:21.833 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:32:21.833 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:32:21.833 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:32:21.833 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:32:21.833 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:32:21.834 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:32:21.834 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:32:21.834 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:32:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:22 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1004640871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:32:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:22 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:22 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1004640871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:32:22.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:22 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:22 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1004640871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:32:22.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:22 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:22.655 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:22.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:22.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:22.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:22.802 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:22.840 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:22.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:22.946 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:22.981 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:23.263 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:23.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:23.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:23.301 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T23:32:23.374 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:32:23.402 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:23.421 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:32:23.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:23.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:32:23.525 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:32:23.552 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:23 smithi088 ceph-mon[44699]: pgmap v1644: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 876 op/s; 2443/120726 objects misplaced (2.024%); 3.5 MiB/s, 56 objects/s recovering 2024-05-15T23:32:23.573 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:32:23.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:23.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:32:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:23.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:23.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:23 smithi084 ceph-mon[39784]: pgmap v1644: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 876 op/s; 2443/120726 objects misplaced (2.024%); 3.5 MiB/s, 56 objects/s recovering 2024-05-15T23:32:23.678 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:32:23.711 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:23.711 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:23.711 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:23.711 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:23.711 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:23 smithi090 ceph-mon[44434]: pgmap v1644: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 876 op/s; 2443/120726 objects misplaced (2.024%); 3.5 MiB/s, 56 objects/s recovering 2024-05-15T23:32:23.715 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:23.744 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:32:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:23.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:32:23.852 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:32:23.879 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:23.899 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:32:23.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:23.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:32:23.919 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:24.003 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:32:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:24.064 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:32:24.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:24.067 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:32:24.168 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:32:24.195 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:24.215 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:32:24.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:24.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:32:24.261 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:24.319 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:32:24.347 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:24.367 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:32:24.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:24.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:32:24.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:32:24.513 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:24.545 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:32:24.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:24.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:32:24.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:32:24.694 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:24.735 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:32:24.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:24.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:32:24.839 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:32:24.887 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:25.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:25 smithi090 ceph-mon[44434]: pgmap v1645: 17 pgs: 1 active+recovering, 16 active+clean; 3.6 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 896 op/s; 2443/118850 objects misplaced (2.056%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T23:32:25.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:25 smithi088 ceph-mon[44699]: pgmap v1645: 17 pgs: 1 active+recovering, 16 active+clean; 3.6 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 896 op/s; 2443/118850 objects misplaced (2.056%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T23:32:25.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:25 smithi084 ceph-mon[39784]: pgmap v1645: 17 pgs: 1 active+recovering, 16 active+clean; 3.6 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 896 op/s; 2443/118850 objects misplaced (2.056%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T23:32:25.947 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:32:25.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:25.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:32:25.981 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:32:26.013 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:26.056 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:32:26.083 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:26.105 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:32:26.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:26.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:32:26.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:26.287 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:26.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:26 smithi084 ceph-mon[39784]: 7.6 scrub starts 2024-05-15T23:32:26.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:26 smithi084 ceph-mon[39784]: 7.6 scrub ok 2024-05-15T23:32:26.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:26 smithi090 ceph-mon[44434]: 7.6 scrub starts 2024-05-15T23:32:26.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:26 smithi090 ceph-mon[44434]: 7.6 scrub ok 2024-05-15T23:32:26.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:26 smithi088 ceph-mon[44699]: 7.6 scrub starts 2024-05-15T23:32:26.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:26 smithi088 ceph-mon[44699]: 7.6 scrub ok 2024-05-15T23:32:27.718 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:27.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:27.722 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:27 smithi084 ceph-mon[39784]: pgmap v1646: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 913 op/s; 2148/113114 objects misplaced (1.899%); 3.4 MiB/s, 53 objects/s recovering 2024-05-15T23:32:27.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:27 smithi090 ceph-mon[44434]: pgmap v1646: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 913 op/s; 2148/113114 objects misplaced (1.899%); 3.4 MiB/s, 53 objects/s recovering 2024-05-15T23:32:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:27 smithi088 ceph-mon[44699]: pgmap v1646: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 913 op/s; 2148/113114 objects misplaced (1.899%); 3.4 MiB/s, 53 objects/s recovering 2024-05-15T23:32:27.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:27.855 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:27.882 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:27.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:27.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:27.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:32:28.026 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:28.057 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:32:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:28.064 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:32:28.165 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:32:28.237 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:28.303 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:32:28.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:28.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:32:28.411 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:32:28.438 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:28.458 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:32:28.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:28.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:32:28.562 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:32:28.588 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3565467000' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:32:28.588 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:28 smithi084 ceph-mon[39784]: from='client.26839 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:32:28.590 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:28.601 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:32:28.611 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:32:28.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:28.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:32:28.716 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:32:28.770 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3565467000' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:32:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:28 smithi088 ceph-mon[44699]: from='client.26839 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:32:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3565467000' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:32:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:28 smithi090 ceph-mon[44434]: from='client.26839 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:32:29.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:29 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:32:29.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:29 smithi084 ceph-mon[39784]: from='client.26839 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:32:29.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:29 smithi084 ceph-mon[39784]: osdmap e613: 12 total, 12 up, 9 in 2024-05-15T23:32:29.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:29 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:32:29.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:29 smithi084 ceph-mon[39784]: pgmap v1648: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 955 op/s; 2148/111312 objects misplaced (1.930%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T23:32:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:29 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:32:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:29 smithi090 ceph-mon[44434]: from='client.26839 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:32:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:29 smithi090 ceph-mon[44434]: osdmap e613: 12 total, 12 up, 9 in 2024-05-15T23:32:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:29 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:32:30.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:29 smithi090 ceph-mon[44434]: pgmap v1648: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 955 op/s; 2148/111312 objects misplaced (1.930%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T23:32:30.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:29 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:32:30.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:29 smithi088 ceph-mon[44699]: from='client.26839 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:32:30.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:29 smithi088 ceph-mon[44699]: osdmap e613: 12 total, 12 up, 9 in 2024-05-15T23:32:30.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:29 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:32:30.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:29 smithi088 ceph-mon[44699]: pgmap v1648: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 955 op/s; 2148/111312 objects misplaced (1.930%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T23:32:31.419 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:31 smithi088 ceph-mon[44699]: pgmap v1649: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 947 op/s; 1851/107142 objects misplaced (1.728%); 3.7 MiB/s, 59 objects/s recovering 2024-05-15T23:32:31.515 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:32:31.516 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:32:31.516 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 613, 2024-05-15T23:32:31.516 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "3999fb7e-130e-11ef-bc99-c7b262605968", 2024-05-15T23:32:31.516 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-05-15T22:56:41.212493+0000", 2024-05-15T23:32:31.516 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-05-15T23:32:28.568314+0000", 2024-05-15T23:32:31.516 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-05-15T23:26:58.424794+0000", 2024-05-15T23:32:31.516 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-05-15T23:28:16.134275+0000", 2024-05-15T23:32:31.516 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-15T23:32:31.516 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5801984, 2024-05-15T23:32:31.516 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-05-15T23:32:31.516 INFO:teuthology.orchestra.run.smithi084.stdout: "noscrub", 2024-05-15T23:32:31.516 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-05-15T23:32:31.517 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-05-15T23:32:31.517 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-05-15T23:32:31.517 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-05-15T23:32:31.517 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:32:31.517 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 30, 2024-05-15T23:32:31.517 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-05-15T23:32:31.517 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-15T23:32:31.517 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-15T23:32:31.517 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-05-15T23:32:31.517 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 7, 2024-05-15T23:32:31.517 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 12, 2024-05-15T23:32:31.517 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-05-15T23:32:31.517 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-05-15T23:32:31.518 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-05-15T23:32:31.518 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-05-15T23:32:31.518 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-05-15T23:32:31.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.518 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-05-15T23:32:31.518 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-05-15T23:32:31.518 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-05-15T22:59:38.603575+0000", 2024-05-15T23:32:31.518 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-05-15T23:32:31.518 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-05-15T23:32:31.518 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-05-15T23:32:31.518 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2024-05-15T23:32:31.518 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-05-15T23:32:31.518 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-05-15T23:32:31.519 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-05-15T23:32:31.519 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-05-15T23:32:31.519 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T23:32:31.519 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T23:32:31.519 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-05-15T23:32:31.519 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-05-15T23:32:31.519 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 1, 2024-05-15T23:32:31.519 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 1, 2024-05-15T23:32:31.519 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 1, 2024-05-15T23:32:31.519 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 1, 2024-05-15T23:32:31.519 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 1, 2024-05-15T23:32:31.520 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-05-15T23:32:31.520 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-05-15T23:32:31.520 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-05-15T23:32:31.520 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-05-15T23:32:31.520 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-05-15T23:32:31.520 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-05-15T23:32:31.520 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-05-15T23:32:31.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.520 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "16", 2024-05-15T23:32:31.520 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-05-15T23:32:31.520 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T23:32:31.520 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T23:32:31.521 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-05-15T23:32:31.521 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-05-15T23:32:31.521 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-05-15T23:32:31.521 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-05-15T23:32:31.521 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-05-15T23:32:31.521 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-05-15T23:32:31.521 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-05-15T23:32:31.521 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-05-15T23:32:31.521 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-05-15T23:32:31.521 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-05-15T23:32:31.521 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-05-15T23:32:31.521 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-05-15T23:32:31.521 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-05-15T23:32:31.522 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-05-15T23:32:31.522 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-05-15T23:32:31.522 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T23:32:31.522 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T23:32:31.522 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T23:32:31.522 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-05-15T23:32:31.522 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-05-15T23:32:31.522 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-05-15T23:32:31.522 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-05-15T23:32:31.522 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-05-15T23:32:31.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.523 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-05-15T23:32:31.523 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-05-15T23:32:31.523 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-05-15T23:32:31.523 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-05-15T23:32:31.523 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-05-15T23:32:31.523 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T23:32:31.524 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-05-15T23:32:31.524 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-05-15T23:32:31.524 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-05-15T23:32:31.524 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-05-15T23:32:31.524 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-05-15T23:32:31.524 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-05-15T23:32:31.524 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-05-15T23:32:31.524 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-05-15T23:32:31.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.524 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-05-15T23:32:31.524 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-05-15T23:32:31.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.524 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-05-15T23:32:31.525 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-05-15T23:32:31.525 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 11.760000228881836, 2024-05-15T23:32:31.525 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 11.760000228881836, 2024-05-15T23:32:31.525 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.17000000178813934, 2024-05-15T23:32:31.525 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-05-15T23:32:31.525 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-05-15T23:32:31.525 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.91321688890457153, 2024-05-15T23:32:31.525 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 0.9100000262260437, 2024-05-15T23:32:31.526 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-05-15T23:32:31.526 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.526 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-05-15T23:32:31.526 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_5", 2024-05-15T23:32:31.526 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-05-15T23:28:50.243463+0000", 2024-05-15T23:32:31.526 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-05-15T23:32:31.526 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-05-15T23:32:31.526 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-05-15T23:32:31.526 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 2, 2024-05-15T23:32:31.527 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-05-15T23:32:31.527 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-05-15T23:32:31.527 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-05-15T23:32:31.527 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-05-15T23:32:31.527 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-05-15T23:32:31.527 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-15T23:32:31.527 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-05-15T23:32:31.527 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-05-15T23:32:31.527 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 16, 2024-05-15T23:32:31.527 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 16, 2024-05-15T23:32:31.527 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 16, 2024-05-15T23:32:31.527 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 16, 2024-05-15T23:32:31.527 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 16, 2024-05-15T23:32:31.527 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-05-15T23:32:31.528 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-05-15T23:32:31.528 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-05-15T23:32:31.528 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-05-15T23:32:31.528 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-05-15T23:32:31.528 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-05-15T23:32:31.528 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-05-15T23:32:31.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.528 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "554", 2024-05-15T23:32:31.528 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-05-15T23:32:31.528 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-15T23:32:31.528 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-05-15T23:32:31.529 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-05-15T23:32:31.529 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-05-15T23:32:31.529 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-05-15T23:32:31.529 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-05-15T23:32:31.529 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-05-15T23:32:31.529 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-05-15T23:32:31.529 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-05-15T23:32:31.529 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-05-15T23:32:31.529 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-05-15T23:32:31.529 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-05-15T23:32:31.529 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-05-15T23:32:31.529 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-05-15T23:32:31.530 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-05-15T23:32:31.530 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-05-15T23:32:31.530 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-05-15T23:32:31.530 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-15T23:32:31.530 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-15T23:32:31.530 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-15T23:32:31.530 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-05-15T23:32:31.530 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-05-15T23:32:31.530 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-05-15T23:32:31.530 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-05-15T23:32:31.530 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-05-15T23:32:31.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.531 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-05-15T23:32:31.531 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-05-15T23:32:31.531 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-05-15T23:32:31.531 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-05-15T23:32:31.531 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-05-15T23:32:31.531 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-05-15T23:32:31.531 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-05-15T23:32:31.531 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-05-15T23:32:31.531 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-05-15T23:32:31.531 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-05-15T23:32:31.531 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-05-15T23:32:31.531 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-05-15T23:32:31.531 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-05-15T23:32:31.531 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-05-15T23:32:31.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.532 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-05-15T23:32:31.532 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-05-15T23:32:31.532 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 3.7300000190734863, 2024-05-15T23:32:31.532 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 3.7300000190734863, 2024-05-15T23:32:31.532 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.67000001668930054, 2024-05-15T23:32:31.532 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.5, 2024-05-15T23:32:31.532 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.5, 2024-05-15T23:32:31.532 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.91321688890457153, 2024-05-15T23:32:31.532 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 0.9100000262260437, 2024-05-15T23:32:31.532 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-05-15T23:32:31.532 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.532 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.532 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:32:31.533 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-05-15T23:32:31.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.533 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-05-15T23:32:31.533 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "80159cc8-1b6c-4aa6-a114-966786d7be1e", 2024-05-15T23:32:31.533 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:32:31.533 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-05-15T23:32:31.533 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-05-15T23:32:31.534 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:32:31.534 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 8, 2024-05-15T23:32:31.534 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 355, 2024-05-15T23:32:31.534 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 397, 2024-05-15T23:32:31.534 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 283, 2024-05-15T23:32:31.534 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 356, 2024-05-15T23:32:31.534 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:32:31.534 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:32:31.534 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.535 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.535 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-05-15T23:32:31.535 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:32:31.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.535 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.535 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:32:31.535 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.535 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.535 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-05-15T23:32:31.535 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:32:31.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.536 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.536 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:32:31.536 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.536 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.536 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-05-15T23:32:31.536 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:32:31.536 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.536 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.536 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:32:31.536 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.537 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.537 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-05-15T23:32:31.537 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3077815271 2024-05-15T23:32:31.537 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.537 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.537 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/3077815271", 2024-05-15T23:32:31.537 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/3077815271", 2024-05-15T23:32:31.537 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/3077815271", 2024-05-15T23:32:31.537 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/3077815271", 2024-05-15T23:32:31.537 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:32:31.537 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:32:31.537 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:32:31.538 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.538 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-05-15T23:32:31.538 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "9c2c76c9-ce0f-4c2d-9d96-c87a10187607", 2024-05-15T23:32:31.538 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:32:31.538 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:32:31.538 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:32:31.538 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 0.4137420654296875, 2024-05-15T23:32:31.538 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:32:31.538 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:32:31.538 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-05-15T23:32:31.538 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 578, 2024-05-15T23:32:31.539 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:32:31.539 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:32:31.539 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:32:31.539 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.539 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.539 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-05-15T23:32:31.539 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:32:31.539 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.539 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.539 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:32:31.539 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.540 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.540 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-05-15T23:32:31.540 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:32:31.540 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.540 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.540 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:32:31.540 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.540 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.540 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-05-15T23:32:31.540 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:32:31.540 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.540 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.541 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:32:31.541 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.541 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.542 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-05-15T23:32:31.542 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2140830229 2024-05-15T23:32:31.542 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.542 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.542 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/2140830229", 2024-05-15T23:32:31.542 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/2140830229", 2024-05-15T23:32:31.542 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/2140830229", 2024-05-15T23:32:31.542 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/2140830229", 2024-05-15T23:32:31.542 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:32:31.542 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:32:31.542 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:32:31.542 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.543 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-05-15T23:32:31.543 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "e8514daa-b5f9-4f2f-a432-53f4a4eb86bc", 2024-05-15T23:32:31.543 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:32:31.543 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:32:31.543 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:32:31.543 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:32:31.543 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 20, 2024-05-15T23:32:31.543 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 261, 2024-05-15T23:32:31.543 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 291, 2024-05-15T23:32:31.543 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:32:31.544 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 262, 2024-05-15T23:32:31.544 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:32:31.544 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:32:31.544 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.544 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.544 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-05-15T23:32:31.545 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:32:31.545 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.545 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.545 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:32:31.545 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.545 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.545 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-05-15T23:32:31.545 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:32:31.545 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.545 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.546 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:32:31.546 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.546 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.546 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-05-15T23:32:31.546 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:32:31.546 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.546 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.546 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:32:31.546 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.547 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.547 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-05-15T23:32:31.547 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 941507493 2024-05-15T23:32:31.547 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.547 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.547 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/941507493", 2024-05-15T23:32:31.547 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/941507493", 2024-05-15T23:32:31.547 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/941507493", 2024-05-15T23:32:31.547 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/941507493", 2024-05-15T23:32:31.547 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:32:31.547 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:32:31.547 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:32:31.547 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.548 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-05-15T23:32:31.548 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "5d9965f9-c0b2-48ce-869c-582990985623", 2024-05-15T23:32:31.548 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:32:31.548 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:32:31.548 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:32:31.548 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:32:31.548 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:32:31.548 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:32:31.548 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 26, 2024-05-15T23:32:31.548 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 610, 2024-05-15T23:32:31.548 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:32:31.549 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:32:31.549 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:32:31.549 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.549 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.549 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-05-15T23:32:31.549 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:32:31.549 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.549 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.549 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:32:31.549 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.549 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.549 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-05-15T23:32:31.550 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:32:31.550 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.550 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.550 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:32:31.550 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.550 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.550 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-05-15T23:32:31.550 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:32:31.550 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.550 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.551 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:32:31.551 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.551 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.551 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-05-15T23:32:31.551 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2204039828 2024-05-15T23:32:31.551 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.551 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.551 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2204039828", 2024-05-15T23:32:31.551 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2204039828", 2024-05-15T23:32:31.551 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2204039828", 2024-05-15T23:32:31.551 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2204039828", 2024-05-15T23:32:31.551 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:32:31.552 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:32:31.552 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:32:31.552 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.552 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-05-15T23:32:31.552 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4bc50523-8480-44d2-8d76-31479e839c85", 2024-05-15T23:32:31.552 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:32:31.552 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:32:31.552 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0.396820068359375, 2024-05-15T23:32:31.553 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:32:31.553 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:32:31.553 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:32:31.553 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-05-15T23:32:31.553 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:32:31.553 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:32:31.553 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:32:31.553 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:32:31.553 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.553 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.553 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6800", 2024-05-15T23:32:31.553 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:32:31.553 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.554 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.554 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:32:31.554 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.554 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.554 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6801", 2024-05-15T23:32:31.555 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:32:31.555 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.555 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.555 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:32:31.555 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.556 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.556 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6803", 2024-05-15T23:32:31.556 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:32:31.556 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.556 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.556 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:32:31.556 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.556 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.556 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6802", 2024-05-15T23:32:31.556 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2267955292 2024-05-15T23:32:31.556 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.557 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.557 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6800/2267955292", 2024-05-15T23:32:31.557 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6801/2267955292", 2024-05-15T23:32:31.557 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6803/2267955292", 2024-05-15T23:32:31.557 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6802/2267955292", 2024-05-15T23:32:31.557 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:32:31.557 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:32:31.557 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:32:31.557 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.558 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-05-15T23:32:31.558 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d4db8d7f-afc7-43dc-81fe-f70c290d8b20", 2024-05-15T23:32:31.558 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:32:31.558 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:32:31.558 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:32:31.558 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:32:31.558 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:32:31.558 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:32:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 37, 2024-05-15T23:32:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:32:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:32:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:32:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:32:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6804", 2024-05-15T23:32:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:32:31.559 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:32:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6805", 2024-05-15T23:32:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:32:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:32:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6807", 2024-05-15T23:32:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:32:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:32:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6806", 2024-05-15T23:32:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 661041538 2024-05-15T23:32:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.561 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6804/661041538", 2024-05-15T23:32:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6805/661041538", 2024-05-15T23:32:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6807/661041538", 2024-05-15T23:32:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6806/661041538", 2024-05-15T23:32:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:32:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:32:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:32:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-05-15T23:32:31.562 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a2773454-2318-4a5b-8f31-0952b1b07f2f", 2024-05-15T23:32:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:32:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-05-15T23:32:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-05-15T23:32:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:32:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:32:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:32:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 46, 2024-05-15T23:32:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 374, 2024-05-15T23:32:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:32:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:32:31.563 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:32:31.564 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.564 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.564 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6808", 2024-05-15T23:32:31.564 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:32:31.564 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.564 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.564 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:32:31.564 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.565 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.565 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6809", 2024-05-15T23:32:31.565 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:32:31.565 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.565 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.565 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:32:31.565 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.566 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.566 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6811", 2024-05-15T23:32:31.566 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:32:31.566 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.566 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.566 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:32:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6810", 2024-05-15T23:32:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 380856869 2024-05-15T23:32:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6808/380856869", 2024-05-15T23:32:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6809/380856869", 2024-05-15T23:32:31.567 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6811/380856869", 2024-05-15T23:32:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6810/380856869", 2024-05-15T23:32:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:32:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:32:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:32:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-05-15T23:32:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "028ca7b6-3e00-4466-a33f-b726cd60cb34", 2024-05-15T23:32:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:32:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:32:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:32:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:32:31.568 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:32:31.569 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:32:31.569 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 53, 2024-05-15T23:32:31.569 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:32:31.569 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:32:31.569 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:32:31.569 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:32:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6812", 2024-05-15T23:32:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:32:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:32:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6813", 2024-05-15T23:32:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:32:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:32:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.571 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6815", 2024-05-15T23:32:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:32:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:32:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.88:6814", 2024-05-15T23:32:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1044709063 2024-05-15T23:32:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.572 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.88:6812/1044709063", 2024-05-15T23:32:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.88:6813/1044709063", 2024-05-15T23:32:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.88:6815/1044709063", 2024-05-15T23:32:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.88:6814/1044709063", 2024-05-15T23:32:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:32:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:32:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:32:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-05-15T23:32:31.573 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "02cab286-d487-4a66-982c-3cf28f55e851", 2024-05-15T23:32:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:32:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-05-15T23:32:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-05-15T23:32:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 0.54486083984375, 2024-05-15T23:32:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 61, 2024-05-15T23:32:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 484, 2024-05-15T23:32:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 520, 2024-05-15T23:32:31.574 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 202, 2024-05-15T23:32:31.575 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 485, 2024-05-15T23:32:31.575 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:32:31.575 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:32:31.575 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.575 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.576 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6800", 2024-05-15T23:32:31.576 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:32:31.576 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.576 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.576 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:32:31.576 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.576 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6801", 2024-05-15T23:32:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:32:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:32:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6803", 2024-05-15T23:32:31.577 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:32:31.578 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.578 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.578 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:32:31.578 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.579 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.579 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6802", 2024-05-15T23:32:31.579 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 703598286 2024-05-15T23:32:31.579 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.579 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.579 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6800/703598286", 2024-05-15T23:32:31.580 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6801/703598286", 2024-05-15T23:32:31.580 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6803/703598286", 2024-05-15T23:32:31.580 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6802/703598286", 2024-05-15T23:32:31.580 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:32:31.580 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:32:31.580 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:32:31.581 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.581 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-05-15T23:32:31.581 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "47d38bc7-39ba-4b35-af24-e282354dd613", 2024-05-15T23:32:31.581 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:32:31.581 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:32:31.582 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:32:31.582 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:32:31.582 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:32:31.582 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:32:31.582 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 68, 2024-05-15T23:32:31.582 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 610, 2024-05-15T23:32:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:32:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:32:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:32:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.583 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.584 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6804", 2024-05-15T23:32:31.584 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:32:31.584 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.584 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:32:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6805", 2024-05-15T23:32:31.585 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:32:31.586 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.586 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.586 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:32:31.586 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.587 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.587 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6807", 2024-05-15T23:32:31.587 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:32:31.587 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.587 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:32:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6806", 2024-05-15T23:32:31.588 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2794571133 2024-05-15T23:32:31.589 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.589 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.589 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6804/2794571133", 2024-05-15T23:32:31.589 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6805/2794571133", 2024-05-15T23:32:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6807/2794571133", 2024-05-15T23:32:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6806/2794571133", 2024-05-15T23:32:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:32:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:32:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:32:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.591 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-05-15T23:32:31.591 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d7b2aeae-d012-41eb-9f91-e4c05c649473", 2024-05-15T23:32:31.591 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:32:31.591 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:32:31.592 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:32:31.592 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:32:31.592 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:32:31.592 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:32:31.592 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 74, 2024-05-15T23:32:31.592 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 578, 2024-05-15T23:32:31.593 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:32:31.593 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:32:31.593 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:32:31.593 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.593 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.594 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6808", 2024-05-15T23:32:31.594 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:32:31.594 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.594 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.594 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:32:31.595 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.595 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.595 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6809", 2024-05-15T23:32:31.595 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:32:31.595 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.596 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.596 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:32:31.596 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.597 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.597 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6811", 2024-05-15T23:32:31.597 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:32:31.597 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.597 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.598 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:32:31.598 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.598 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.598 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6810", 2024-05-15T23:32:31.599 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3735380744 2024-05-15T23:32:31.599 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.599 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.599 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6808/3735380744", 2024-05-15T23:32:31.599 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6809/3735380744", 2024-05-15T23:32:31.600 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6811/3735380744", 2024-05-15T23:32:31.600 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6810/3735380744", 2024-05-15T23:32:31.600 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:32:31.600 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:32:31.600 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:32:31.600 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.601 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-05-15T23:32:31.601 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5", 2024-05-15T23:32:31.601 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-05-15T23:32:31.601 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-05-15T23:32:31.601 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-05-15T23:32:31.601 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-05-15T23:32:31.601 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-05-15T23:32:31.602 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-05-15T23:32:31.602 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 81, 2024-05-15T23:32:31.602 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 549, 2024-05-15T23:32:31.602 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-05-15T23:32:31.602 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-05-15T23:32:31.602 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-05-15T23:32:31.602 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.603 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.603 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6812", 2024-05-15T23:32:31.603 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:32:31.603 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.603 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.603 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-05-15T23:32:31.604 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.604 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.604 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6813", 2024-05-15T23:32:31.604 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:32:31.604 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.605 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.605 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-05-15T23:32:31.605 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.605 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.605 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6815", 2024-05-15T23:32:31.605 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:32:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-05-15T23:32:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-05-15T23:32:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-05-15T23:32:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.90:6814", 2024-05-15T23:32:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2265615270 2024-05-15T23:32:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.90:6812/2265615270", 2024-05-15T23:32:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.90:6813/2265615270", 2024-05-15T23:32:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.90:6815/2265615270", 2024-05-15T23:32:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.90:6814/2265615270", 2024-05-15T23:32:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-05-15T23:32:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-05-15T23:32:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-05-15T23:32:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:32:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-05-15T23:32:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-05-15T23:32:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:32:31.607 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:32:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:32:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:32:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-05-15T23:32:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:31:28.327492+0000", 2024-05-15T23:32:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 355 2024-05-15T23:32:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-05-15T23:32:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:32:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:32:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:32:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:32:31.608 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:32:31.609 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:31:21.200544+0000", 2024-05-15T23:32:31.609 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:32:31.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.609 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-05-15T23:32:31.609 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:32:31.609 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:32:31.609 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:32:31.609 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:32:31.609 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:32:31.609 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:31:28.894548+0000", 2024-05-15T23:32:31.609 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 261 2024-05-15T23:32:31.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.609 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-05-15T23:32:31.610 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:32:31.610 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:32:31.610 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:32:31.610 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:32:31.610 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:32:31.610 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:31:26.317814+0000", 2024-05-15T23:32:31.610 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:32:31.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.610 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-05-15T23:32:31.610 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:32:31.610 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:32:31.610 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:32:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:32:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:32:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:32:11.000054+0000", 2024-05-15T23:32:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:32:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-05-15T23:32:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:32:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:32:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:32:31.611 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:32:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:32:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:31:29.796298+0000", 2024-05-15T23:32:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:32:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-05-15T23:32:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:32:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:32:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:32:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:32:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-05-15T23:32:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:31:32.616758+0000", 2024-05-15T23:32:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:32:31.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-05-15T23:32:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:32:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:32:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:32:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:32:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:32:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:31:42.767268+0000", 2024-05-15T23:32:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:32:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-05-15T23:32:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:32:31.613 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:32:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:32:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:32:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-05-15T23:32:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:32:10.744669+0000", 2024-05-15T23:32:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 484 2024-05-15T23:32:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-05-15T23:32:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:32:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:32:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:32:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:32:31.614 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:32:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:31:52.934936+0000", 2024-05-15T23:32:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:32:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-05-15T23:32:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:32:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:32:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:32:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:32:31.615 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:32:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:31:11.522605+0000", 2024-05-15T23:32:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:32:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-05-15T23:32:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-05-15T23:32:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-05-15T23:32:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-05-15T23:32:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-05-15T23:32:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-05-15T23:32:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-05-15T23:31:51.137987+0000", 2024-05-15T23:32:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-05-15T23:32:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.616 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:32:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2024-05-15T23:32:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "1.0", 2024-05-15T23:32:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-05-15T23:32:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: 6, 2024-05-15T23:32:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: 3 2024-05-15T23:32:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:32:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "7.8", 2024-05-15T23:32:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-05-15T23:32:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: 6, 2024-05-15T23:32:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: 0 2024-05-15T23:32:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:32:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:32:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [], 2024-05-15T23:32:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-05-15T23:32:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-05-15T23:32:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-05-15T23:32:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-05-15T23:32:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2581408231": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:32:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/1136984393": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:32:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2443201864": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:32:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1252671888": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:32:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1451001371": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:32:31.618 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4084814866": "2024-05-16T22:57:38.505263+0000", 2024-05-15T23:32:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4076409285": "2024-05-16T22:57:09.403385+0000", 2024-05-15T23:32:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2988977694": "2024-05-16T22:57:38.505263+0000" 2024-05-15T23:32:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-05-15T23:32:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-05-15T23:32:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-05-15T23:32:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-05-15T23:32:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-05-15T23:32:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-05-15T23:32:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-05-15T23:32:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-05-15T23:32:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:32:31.619 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [], 2024-05-15T23:32:31.620 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2024-05-15T23:32:31.620 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-05-15T23:32:31.620 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-05-15T23:32:31.620 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-05-15T23:32:31.620 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-05-15T23:32:31.620 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-05-15T23:32:31.620 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-05-15T23:32:31.620 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-05-15T23:32:31.620 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-05-15T23:32:31.620 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-05-15T23:32:31.620 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:32:31.620 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:32:31.621 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:31 smithi090 ceph-mon[44434]: pgmap v1649: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 947 op/s; 1851/107142 objects misplaced (1.728%); 3.7 MiB/s, 59 objects/s recovering 2024-05-15T23:32:31.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:31 smithi084 ceph-mon[39784]: pgmap v1649: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 947 op/s; 1851/107142 objects misplaced (1.728%); 3.7 MiB/s, 59 objects/s recovering 2024-05-15T23:32:31.961 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:32:31.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:31.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:32:32.077 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:32.149 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:32.203 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:32.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:32.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:32.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:32:32.354 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:32.383 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:32:32.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:32.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:32:32.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:32:32.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2383029265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:32:32.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2383029265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:32:32.555 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:32.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2383029265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-05-15T23:32:32.647 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:32:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:32.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:32:32.700 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:32:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-05-15T23:32:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-05-15T23:32:32.700 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2024-05-15T23:32:32.701 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:32:32.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:32:32.821 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:32.961 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:32:33.147 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:33.171 INFO:tasks.thrashosds.thrasher:j is {'epoch': 613, 'fsid': '3999fb7e-130e-11ef-bc99-c7b262605968', 'created': '2024-05-15T22:56:41.212493+0000', 'modified': '2024-05-15T23:32:28.568314+0000', 'last_up_change': '2024-05-15T23:26:58.424794+0000', 'last_in_change': '2024-05-15T23:28:16.134275+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 30, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T22:59:38.603575+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9132168889045715, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-05-15T23:28:50.243463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '554', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.7300000190734863, 'score_stable': 3.7300000190734863, 'optimal_score': 0.6700000166893005, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9132168889045715, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}], 'osds': [{'osd': 0, 'uuid': '80159cc8-1b6c-4aa6-a114-966786d7be1e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 355, 'up_from': 397, 'up_thru': 283, 'down_at': 356, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6801', 'nonce': 3077815271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6802', 'nonce': 3077815271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6804', 'nonce': 3077815271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6803', 'nonce': 3077815271}]}, 'public_addr': '172.21.15.84:6801/3077815271', 'cluster_addr': '172.21.15.84:6802/3077815271', 'heartbeat_back_addr': '172.21.15.84:6804/3077815271', 'heartbeat_front_addr': '172.21.15.84:6803/3077815271', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9c2c76c9-ce0f-4c2d-9d96-c87a10187607', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.4137420654296875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 578, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6805', 'nonce': 2140830229}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6806', 'nonce': 2140830229}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6808', 'nonce': 2140830229}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6807', 'nonce': 2140830229}]}, 'public_addr': '172.21.15.84:6805/2140830229', 'cluster_addr': '172.21.15.84:6806/2140830229', 'heartbeat_back_addr': '172.21.15.84:6808/2140830229', 'heartbeat_front_addr': '172.21.15.84:6807/2140830229', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e8514daa-b5f9-4f2f-a432-53f4a4eb86bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 261, 'up_from': 291, 'up_thru': 549, 'down_at': 262, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6809', 'nonce': 941507493}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6810', 'nonce': 941507493}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6812', 'nonce': 941507493}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6811', 'nonce': 941507493}]}, 'public_addr': '172.21.15.84:6809/941507493', 'cluster_addr': '172.21.15.84:6810/941507493', 'heartbeat_back_addr': '172.21.15.84:6812/941507493', 'heartbeat_front_addr': '172.21.15.84:6811/941507493', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5d9965f9-c0b2-48ce-869c-582990985623', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 610, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6813', 'nonce': 2204039828}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6814', 'nonce': 2204039828}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6816', 'nonce': 2204039828}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.84:6815', 'nonce': 2204039828}]}, 'public_addr': '172.21.15.84:6813/2204039828', 'cluster_addr': '172.21.15.84:6814/2204039828', 'heartbeat_back_addr': '172.21.15.84:6816/2204039828', 'heartbeat_front_addr': '172.21.15.84:6815/2204039828', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4bc50523-8480-44d2-8d76-31479e839c85', 'up': 1, 'in': 1, 'weight': 0.396820068359375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6800', 'nonce': 2267955292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6801', 'nonce': 2267955292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6803', 'nonce': 2267955292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6802', 'nonce': 2267955292}]}, 'public_addr': '172.21.15.88:6800/2267955292', 'cluster_addr': '172.21.15.88:6801/2267955292', 'heartbeat_back_addr': '172.21.15.88:6803/2267955292', 'heartbeat_front_addr': '172.21.15.88:6802/2267955292', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'd4db8d7f-afc7-43dc-81fe-f70c290d8b20', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6804', 'nonce': 661041538}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6805', 'nonce': 661041538}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6807', 'nonce': 661041538}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6806', 'nonce': 661041538}]}, 'public_addr': '172.21.15.88:6804/661041538', 'cluster_addr': '172.21.15.88:6805/661041538', 'heartbeat_back_addr': '172.21.15.88:6807/661041538', 'heartbeat_front_addr': '172.21.15.88:6806/661041538', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'a2773454-2318-4a5b-8f31-0952b1b07f2f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6808', 'nonce': 380856869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6809', 'nonce': 380856869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6811', 'nonce': 380856869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6810', 'nonce': 380856869}]}, 'public_addr': '172.21.15.88:6808/380856869', 'cluster_addr': '172.21.15.88:6809/380856869', 'heartbeat_back_addr': '172.21.15.88:6811/380856869', 'heartbeat_front_addr': '172.21.15.88:6810/380856869', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '028ca7b6-3e00-4466-a33f-b726cd60cb34', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6812', 'nonce': 1044709063}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6813', 'nonce': 1044709063}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6815', 'nonce': 1044709063}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.88:6814', 'nonce': 1044709063}]}, 'public_addr': '172.21.15.88:6812/1044709063', 'cluster_addr': '172.21.15.88:6813/1044709063', 'heartbeat_back_addr': '172.21.15.88:6815/1044709063', 'heartbeat_front_addr': '172.21.15.88:6814/1044709063', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '02cab286-d487-4a66-982c-3cf28f55e851', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.54486083984375, 'last_clean_begin': 61, 'last_clean_end': 484, 'up_from': 520, 'up_thru': 202, 'down_at': 485, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6800', 'nonce': 703598286}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 703598286}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 703598286}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6802', 'nonce': 703598286}]}, 'public_addr': '172.21.15.90:6800/703598286', 'cluster_addr': '172.21.15.90:6801/703598286', 'heartbeat_back_addr': '172.21.15.90:6803/703598286', 'heartbeat_front_addr': '172.21.15.90:6802/703598286', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '47d38bc7-39ba-4b35-af24-e282354dd613', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 68, 'up_thru': 610, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6804', 'nonce': 2794571133}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 2794571133}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 2794571133}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6806', 'nonce': 2794571133}]}, 'public_addr': '172.21.15.90:6804/2794571133', 'cluster_addr': '172.21.15.90:6805/2794571133', 'heartbeat_back_addr': '172.21.15.90:6807/2794571133', 'heartbeat_front_addr': '172.21.15.90:6806/2794571133', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'd7b2aeae-d012-41eb-9f91-e4c05c649473', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 578, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6808', 'nonce': 3735380744}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 3735380744}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 3735380744}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 3735380744}]}, 'public_addr': '172.21.15.90:6808/3735380744', 'cluster_addr': '172.21.15.90:6809/3735380744', 'heartbeat_back_addr': '172.21.15.90:6811/3735380744', 'heartbeat_front_addr': '172.21.15.90:6810/3735380744', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 81, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6812', 'nonce': 2265615270}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 2265615270}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 2265615270}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 2265615270}]}, 'public_addr': '172.21.15.90:6812/2265615270', 'cluster_addr': '172.21.15.90:6813/2265615270', 'heartbeat_back_addr': '172.21.15.90:6815/2265615270', 'heartbeat_front_addr': '172.21.15.90:6814/2265615270', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T23:31:28.327492+0000', 'dead_epoch': 355}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:31:21.200544+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:31:28.894548+0000', 'dead_epoch': 261}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:31:26.317814+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:32:11.000054+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:31:29.796298+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T23:31:32.616758+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:31:42.767268+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-15T23:32:10.744669+0000', 'dead_epoch': 484}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:31:52.934936+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:31:11.522605+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-15T23:31:51.137987+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [6, 3]}, {'pgid': '7.8', 'osds': [6, 0]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/2581408231': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:6800/1136984393': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:6800/2443201864': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/1252671888': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/1451001371': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/4084814866': '2024-05-16T22:57:38.505263+0000', '172.21.15.84:0/4076409285': '2024-05-16T22:57:09.403385+0000', '172.21.15.84:0/2988977694': '2024-05-16T22:57:38.505263+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-15T23:32:33.172 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:32:33.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:33 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:32:33.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:33 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:32:33.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:33 smithi088 ceph-mon[44699]: pgmap v1650: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 946 op/s; 1851/101768 objects misplaced (1.819%); 3.7 MiB/s, 59 objects/s recovering 2024-05-15T23:32:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:33 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:32:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:33 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:32:33.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:33 smithi084 ceph-mon[39784]: pgmap v1650: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 946 op/s; 1851/101768 objects misplaced (1.819%); 3.7 MiB/s, 59 objects/s recovering 2024-05-15T23:32:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:33 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:32:33.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:33 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:32:33.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:33 smithi090 ceph-mon[44434]: pgmap v1650: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 946 op/s; 1851/101768 objects misplaced (1.819%); 3.7 MiB/s, 59 objects/s recovering 2024-05-15T23:32:34.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:34 smithi088 ceph-mon[44699]: osdmap e614: 12 total, 12 up, 9 in 2024-05-15T23:32:34.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:34 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:32:34.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:34 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:32:34.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:34 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:32:34.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:34 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:32:34.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:34 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:32:34.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:34 smithi084 ceph-mon[39784]: osdmap e614: 12 total, 12 up, 9 in 2024-05-15T23:32:34.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:34 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:32:34.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:34 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:32:34.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:34 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:32:34.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:34 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:32:34.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:34 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:32:34.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T23:32:34.717 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:34 smithi090 ceph-mon[44434]: osdmap e614: 12 total, 12 up, 9 in 2024-05-15T23:32:34.717 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:34 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:32:34.717 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:34 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:32:34.717 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:34 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:32:34.717 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:34 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:32:34.717 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:34 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:32:34.961 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:32:34.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:34.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:32:35.043 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:32:35.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:32:35.124 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:35.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:35 smithi084 ceph-mon[39784]: pgmap v1652: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 817 op/s; 1851/100024 objects misplaced (1.851%); 2.3 MiB/s, 37 objects/s recovering 2024-05-15T23:32:35.366 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:35 smithi090 ceph-mon[44434]: pgmap v1652: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 817 op/s; 1851/100024 objects misplaced (1.851%); 2.3 MiB/s, 37 objects/s recovering 2024-05-15T23:32:35.378 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:32:35.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:35.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:32:35.463 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:35.481 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:35.487 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:32:35.519 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:35.540 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:32:35.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:35.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:32:35.646 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:35.702 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:35.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:35 smithi088 ceph-mon[44699]: pgmap v1652: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 817 op/s; 1851/100024 objects misplaced (1.851%); 2.3 MiB/s, 37 objects/s recovering 2024-05-15T23:32:37.078 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:37.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:37.191 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:32:37.218 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:37.240 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:32:37.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:37.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:32:37.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:32:37.434 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:38 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:32:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:38 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:32:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:38 smithi090 ceph-mon[44434]: pgmap v1653: 17 pgs: 1 active+recovering, 16 active+clean; 2.9 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 999 op/s; 1570/94052 objects misplaced (1.669%); 4.2 MiB/s, 66 objects/s recovering 2024-05-15T23:32:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:38 smithi090 ceph-mon[44434]: osdmap e615: 12 total, 12 up, 9 in 2024-05-15T23:32:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:38 smithi090 ceph-mon[44434]: osdmap e616: 12 total, 12 up, 9 in 2024-05-15T23:32:38.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:38 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:32:38.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:38 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:32:38.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:38 smithi088 ceph-mon[44699]: pgmap v1653: 17 pgs: 1 active+recovering, 16 active+clean; 2.9 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 999 op/s; 1570/94052 objects misplaced (1.669%); 4.2 MiB/s, 66 objects/s recovering 2024-05-15T23:32:38.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:38 smithi088 ceph-mon[44699]: osdmap e615: 12 total, 12 up, 9 in 2024-05-15T23:32:38.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:38 smithi088 ceph-mon[44699]: osdmap e616: 12 total, 12 up, 9 in 2024-05-15T23:32:38.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:38 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:32:38.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:38 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:32:38.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:38 smithi084 ceph-mon[39784]: pgmap v1653: 17 pgs: 1 active+recovering, 16 active+clean; 2.9 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 999 op/s; 1570/94052 objects misplaced (1.669%); 4.2 MiB/s, 66 objects/s recovering 2024-05-15T23:32:38.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:38 smithi084 ceph-mon[39784]: osdmap e615: 12 total, 12 up, 9 in 2024-05-15T23:32:38.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:38 smithi084 ceph-mon[39784]: osdmap e616: 12 total, 12 up, 9 in 2024-05-15T23:32:39.218 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:32:39.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:39.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:32:39.326 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:32:39.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1346436544' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:32:39.351 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:39.352 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:39.355 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:39.375 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:32:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:39.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:32:39.479 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:32:39.505 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1346436544' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:32:39.505 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:39.505 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:39.506 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:39.527 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:32:39.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:39.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:32:39.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1346436544' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:32:39.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:39.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:39.631 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:32:39.658 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:39.678 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:32:39.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:39.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:32:39.750 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:32:39.750 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:32:39.750 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:32:39.750 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:32:39.750 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:32:39.751 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:32:39.751 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:32:39.751 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:32:39.751 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:32:39.751 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:32:39.751 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:32:39.751 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:32:39.751 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:32:39.751 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:32:39.751 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:32:39.751 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:32:39.751 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:32:39.752 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:32:39.752 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:32:39.752 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:32:39.752 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:32:39.752 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:32:39.752 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:32:39.752 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:32:39.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:32:39.820 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:32:40.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:40.105 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:32:40.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:32:40.245 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:40.264 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:32:40.265 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:32:40.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:40 smithi084 ceph-mon[39784]: pgmap v1656: 17 pgs: 1 active+recovering, 16 active+clean; 2.8 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 795 op/s; 1570/92208 objects misplaced (1.703%); 2.9 MiB/s, 46 objects/s recovering 2024-05-15T23:32:40.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:40 smithi090 ceph-mon[44434]: pgmap v1656: 17 pgs: 1 active+recovering, 16 active+clean; 2.8 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 795 op/s; 1570/92208 objects misplaced (1.703%); 2.9 MiB/s, 46 objects/s recovering 2024-05-15T23:32:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:40.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:40 smithi088 ceph-mon[44699]: pgmap v1656: 17 pgs: 1 active+recovering, 16 active+clean; 2.8 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 795 op/s; 1570/92208 objects misplaced (1.703%); 2.9 MiB/s, 46 objects/s recovering 2024-05-15T23:32:40.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:41.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:41 smithi084 ceph-mon[39784]: pgmap v1657: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 873 op/s; 1273/87970 objects misplaced (1.447%); 4.6 MiB/s, 73 objects/s recovering 2024-05-15T23:32:41.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:41 smithi090 ceph-mon[44434]: pgmap v1657: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 873 op/s; 1273/87970 objects misplaced (1.447%); 4.6 MiB/s, 73 objects/s recovering 2024-05-15T23:32:41.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:41 smithi088 ceph-mon[44699]: pgmap v1657: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 873 op/s; 1273/87970 objects misplaced (1.447%); 4.6 MiB/s, 73 objects/s recovering 2024-05-15T23:32:41.750 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:32:41.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:41.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:32:41.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:32:41.897 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:42.067 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:32:42.073 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:32:42.249 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:32:42.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:42 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:32:42.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:42 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:32:42.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:42 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1926069198' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:32:42.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:42 smithi084 ceph-mon[39784]: from='client.36818 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:32:42.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:42 smithi084 ceph-mon[39784]: from='client.16932 v1:172.21.15.84:0/1136720412' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:32:42.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:42 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:32:42.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:42 smithi084 ceph-mon[39784]: from='client.36818 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:32:42.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:42 smithi084 ceph-mon[39784]: osdmap e617: 12 total, 12 up, 9 in 2024-05-15T23:32:42.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:42 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:32:42.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:42 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:32:42.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:42 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1926069198' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:32:42.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:42 smithi090 ceph-mon[44434]: from='client.36818 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:32:42.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:42 smithi090 ceph-mon[44434]: from='client.16932 v1:172.21.15.84:0/1136720412' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:32:42.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:42 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:32:42.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:42 smithi090 ceph-mon[44434]: from='client.36818 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:32:42.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:42 smithi090 ceph-mon[44434]: osdmap e617: 12 total, 12 up, 9 in 2024-05-15T23:32:42.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:42 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:32:42.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:42 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:32:42.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:42 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1926069198' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:32:42.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:42 smithi088 ceph-mon[44699]: from='client.36818 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:32:42.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:42 smithi088 ceph-mon[44699]: from='client.16932 v1:172.21.15.84:0/1136720412' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:32:42.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:42 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:32:42.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:42 smithi088 ceph-mon[44699]: from='client.36818 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:32:42.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:42 smithi088 ceph-mon[44699]: osdmap e617: 12 total, 12 up, 9 in 2024-05-15T23:32:42.585 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:32:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:43 smithi090 ceph-mon[44434]: pgmap v1659: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 950 op/s; 1273/82626 objects misplaced (1.541%); 3.1 MiB/s, 49 objects/s recovering 2024-05-15T23:32:43.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:43 smithi088 ceph-mon[44699]: pgmap v1659: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 950 op/s; 1273/82626 objects misplaced (1.541%); 3.1 MiB/s, 49 objects/s recovering 2024-05-15T23:32:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:43 smithi084 ceph-mon[39784]: pgmap v1659: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 950 op/s; 1273/82626 objects misplaced (1.541%); 3.1 MiB/s, 49 objects/s recovering 2024-05-15T23:32:43.993 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1657,"stamp":"2024-05-15T23:32:41.218713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2882994744,"num_objects":43985,"num_object_clones":0,"num_object_copies":87970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1273,"num_objects_unfound":0,"num_objects_dirty":43985,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":248478,"num_write_kb":9356903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11251,"num_bytes_recovered":738539104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160895,"ondisk_log_size":160895,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":11169264,"kb_used_data":6064528,"kb_used_omap":16,"kb_used_meta":5104559,"kb_avail":832537104,"statfs":{"total":863955320832,"available":852517994496,"internally_reserved":0,"allocated":6210076672,"data_stored":6206266083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17176,"internal_metadata":5227068648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-452132864,"num_objects":-6899,"num_object_clones":0,"num_object_copies":-13798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-578,"num_objects_unfound":0,"num_objects_dirty":-6899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6899,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":37879808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.900034"},"pg_stats":[{"pgid":"7.f","version":"616'15583","reported_seq":15804,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-05-15T23:32:41.027049+0000","last_change":"2024-05-15T23:31:20.669532+0000","last_active":"2024-05-15T23:32:41.027049+0000","last_peered":"2024-05-15T23:32:41.027049+0000","last_clean":"2024-05-15T23:32:41.027049+0000","last_became_active":"2024-05-15T23:28:51.305173+0000","last_became_peered":"2024-05-15T23:28:51.305173+0000","last_unstale":"2024-05-15T23:32:41.027049+0000","last_undegraded":"2024-05-15T23:32:41.027049+0000","last_fullsized":"2024-05-15T23:32:41.027049+0000","mapping_epoch":549,"log_start":"570'5500","ondisk_log_start":"570'5500","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'11414","last_scrub_stamp":"2024-05-15T23:31:20.669357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:31:20.669357+0000","objects_scrubbed":6848,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:32:43.270585+0000","scrub_duration":1343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172163072,"num_objects":2627,"num_object_clones":0,"num_object_copies":5254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15583,"num_write_kb":582720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"616'15419","reported_seq":15706,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-05-15T23:32:39.473214+0000","last_change":"2024-05-15T23:32:09.972982+0000","last_active":"2024-05-15T23:32:39.473214+0000","last_peered":"2024-05-15T23:32:39.473214+0000","last_clean":"2024-05-15T23:32:39.473214+0000","last_became_active":"2024-05-15T23:28:51.519387+0000","last_became_peered":"2024-05-15T23:28:51.519387+0000","last_unstale":"2024-05-15T23:32:39.473214+0000","last_undegraded":"2024-05-15T23:32:39.473214+0000","last_fullsized":"2024-05-15T23:32:39.473214+0000","mapping_epoch":549,"log_start":"569'5400","ondisk_log_start":"569'5400","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'13643","last_scrub_stamp":"2024-05-15T23:32:09.972849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:09.972849+0000","objects_scrubbed":4379,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:33:20.896401+0000","scrub_duration":992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168886272,"num_objects":2577,"num_object_clones":0,"num_object_copies":5154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15419,"num_write_kb":575872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"614'18549","reported_seq":18875,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-05-15T23:32:36.711770+0000","last_change":"2024-05-15T23:32:13.361819+0000","last_active":"2024-05-15T23:32:36.711770+0000","last_peered":"2024-05-15T23:32:36.711770+0000","last_clean":"2024-05-15T23:32:36.711770+0000","last_became_active":"2024-05-15T23:32:13.327043+0000","last_became_peered":"2024-05-15T23:32:13.327043+0000","last_unstale":"2024-05-15T23:32:36.711770+0000","last_undegraded":"2024-05-15T23:32:36.711770+0000","last_fullsized":"2024-05-15T23:32:36.711770+0000","mapping_epoch":610,"log_start":"582'8501","ondisk_log_start":"582'8501","created":549,"last_epoch_clean":611,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'16611","last_scrub_stamp":"2024-05-15T23:32:04.371452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:04.371452+0000","objects_scrubbed":4852,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:33:27.916120+0000","scrub_duration":1003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189661208,"num_objects":2895,"num_object_clones":0,"num_object_copies":5790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2895,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15613,"num_write_kb":592193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.c","version":"614'15361","reported_seq":15586,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-05-15T23:32:36.787738+0000","last_change":"2024-05-15T23:30:09.921129+0000","last_active":"2024-05-15T23:32:36.787738+0000","last_peered":"2024-05-15T23:32:36.787738+0000","last_clean":"2024-05-15T23:32:36.787738+0000","last_became_active":"2024-05-15T23:28:51.305728+0000","last_became_peered":"2024-05-15T23:28:51.305728+0000","last_unstale":"2024-05-15T23:32:36.787738+0000","last_undegraded":"2024-05-15T23:32:36.787738+0000","last_fullsized":"2024-05-15T23:32:36.787738+0000","mapping_epoch":549,"log_start":"569'5300","ondisk_log_start":"569'5300","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'7710","last_scrub_stamp":"2024-05-15T23:30:09.921025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:09.921025+0000","objects_scrubbed":7710,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":673,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188940288,"num_objects":2883,"num_object_clones":0,"num_object_copies":5766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15361,"num_write_kb":583808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"614'15062","reported_seq":15333,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-05-15T23:32:36.760039+0000","last_change":"2024-05-15T23:32:15.224656+0000","last_active":"2024-05-15T23:32:36.760039+0000","last_peered":"2024-05-15T23:32:36.760039+0000","last_clean":"2024-05-15T23:32:36.760039+0000","last_became_active":"2024-05-15T23:28:51.306794+0000","last_became_peered":"2024-05-15T23:28:51.306794+0000","last_unstale":"2024-05-15T23:32:36.760039+0000","last_undegraded":"2024-05-15T23:32:36.760039+0000","last_fullsized":"2024-05-15T23:32:36.760039+0000","mapping_epoch":549,"log_start":"567'5000","ondisk_log_start":"567'5000","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"611'13861","last_scrub_stamp":"2024-05-15T23:32:15.224350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:15.224350+0000","objects_scrubbed":4042,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:33:21.117873+0000","scrub_duration":790,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184811520,"num_objects":2820,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15062,"num_write_kb":572224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"616'15834","reported_seq":16053,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-05-15T23:32:41.041239+0000","last_change":"2024-05-15T23:31:19.550153+0000","last_active":"2024-05-15T23:32:41.041239+0000","last_peered":"2024-05-15T23:32:41.041239+0000","last_clean":"2024-05-15T23:32:41.041239+0000","last_became_active":"2024-05-15T23:28:51.308216+0000","last_became_peered":"2024-05-15T23:28:51.308216+0000","last_unstale":"2024-05-15T23:32:41.041239+0000","last_undegraded":"2024-05-15T23:32:41.041239+0000","last_fullsized":"2024-05-15T23:32:41.041239+0000","mapping_epoch":549,"log_start":"570'5800","ondisk_log_start":"570'5800","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'11559","last_scrub_stamp":"2024-05-15T23:31:19.550020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:31:19.550020+0000","objects_scrubbed":6914,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:32:41.658667+0000","scrub_duration":1247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171048960,"num_objects":2610,"num_object_clones":0,"num_object_copies":5220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15834,"num_write_kb":590208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"616'15557","reported_seq":15844,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-05-15T23:32:37.405815+0000","last_change":"2024-05-15T23:32:09.654565+0000","last_active":"2024-05-15T23:32:37.405815+0000","last_peered":"2024-05-15T23:32:37.405815+0000","last_clean":"2024-05-15T23:32:37.405815+0000","last_became_active":"2024-05-15T23:28:51.519752+0000","last_became_peered":"2024-05-15T23:28:51.519752+0000","last_unstale":"2024-05-15T23:32:37.405815+0000","last_undegraded":"2024-05-15T23:32:37.405815+0000","last_fullsized":"2024-05-15T23:32:37.405815+0000","mapping_epoch":549,"log_start":"569'5500","ondisk_log_start":"569'5500","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'14001","last_scrub_stamp":"2024-05-15T23:32:09.654457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:09.654457+0000","objects_scrubbed":4350,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:33:33.501625+0000","scrub_duration":1216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181075968,"num_objects":2763,"num_object_clones":0,"num_object_copies":5526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15557,"num_write_kb":586240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"616'19206","reported_seq":19640,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-05-15T23:32:39.503319+0000","last_change":"2024-05-15T23:32:06.871453+0000","last_active":"2024-05-15T23:32:39.503319+0000","last_peered":"2024-05-15T23:32:39.503319+0000","last_clean":"2024-05-15T23:32:39.503319+0000","last_became_active":"2024-05-15T23:28:51.304930+0000","last_became_peered":"2024-05-15T23:28:51.304930+0000","last_unstale":"2024-05-15T23:32:39.503319+0000","last_undegraded":"2024-05-15T23:32:39.503319+0000","last_fullsized":"2024-05-15T23:32:39.503319+0000","mapping_epoch":549,"log_start":"587'9200","ondisk_log_start":"587'9200","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'17339","last_scrub_stamp":"2024-05-15T23:32:06.871247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:06.871247+0000","objects_scrubbed":4663,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:33:08.438282+0000","scrub_duration":887,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181600256,"num_objects":2771,"num_object_clones":0,"num_object_copies":5542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15795,"num_write_kb":594112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"614'15234","reported_seq":15374,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-05-15T23:32:36.751537+0000","last_change":"2024-05-15T23:28:51.307151+0000","last_active":"2024-05-15T23:32:36.751537+0000","last_peered":"2024-05-15T23:32:36.751537+0000","last_clean":"2024-05-15T23:32:36.751537+0000","last_became_active":"2024-05-15T23:28:51.306983+0000","last_became_peered":"2024-05-15T23:28:51.306983+0000","last_unstale":"2024-05-15T23:32:36.751537+0000","last_undegraded":"2024-05-15T23:32:36.751537+0000","last_fullsized":"2024-05-15T23:32:36.751537+0000","mapping_epoch":549,"log_start":"569'5200","ondisk_log_start":"569'5200","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188088320,"num_objects":2870,"num_object_clones":0,"num_object_copies":5740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15234,"num_write_kb":579328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.5","version":"616'15486","reported_seq":15771,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-05-15T23:32:37.403300+0000","last_change":"2024-05-15T23:32:05.298952+0000","last_active":"2024-05-15T23:32:37.403300+0000","last_peered":"2024-05-15T23:32:37.403300+0000","last_clean":"2024-05-15T23:32:37.403300+0000","last_became_active":"2024-05-15T23:28:51.303581+0000","last_became_peered":"2024-05-15T23:28:51.303581+0000","last_unstale":"2024-05-15T23:32:37.403300+0000","last_undegraded":"2024-05-15T23:32:37.403300+0000","last_fullsized":"2024-05-15T23:32:37.403300+0000","mapping_epoch":549,"log_start":"569'5400","ondisk_log_start":"569'5400","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'13647","last_scrub_stamp":"2024-05-15T23:32:05.298729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:05.298729+0000","objects_scrubbed":4699,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:33:08.559785+0000","scrub_duration":802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186122240,"num_objects":2840,"num_object_clones":0,"num_object_copies":5680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15486,"num_write_kb":586432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"616'16067","reported_seq":26025,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-05-15T23:32:41.009134+0000","last_change":"2024-05-15T23:32:13.361462+0000","last_active":"2024-05-15T23:32:41.009134+0000","last_peered":"2024-05-15T23:32:41.009134+0000","last_clean":"2024-05-15T23:32:41.009134+0000","last_became_active":"2024-05-15T23:32:13.327837+0000","last_became_peered":"2024-05-15T23:32:13.327837+0000","last_unstale":"2024-05-15T23:32:41.009134+0000","last_undegraded":"2024-05-15T23:32:41.009134+0000","last_fullsized":"2024-05-15T23:32:41.009134+0000","mapping_epoch":610,"log_start":"571'6000","ondisk_log_start":"571'6000","created":549,"last_epoch_clean":611,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'11691","last_scrub_stamp":"2024-05-15T23:31:20.292425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:31:20.292425+0000","objects_scrubbed":6889,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:32:43.270548+0000","scrub_duration":1379,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163119104,"num_objects":2489,"num_object_clones":0,"num_object_copies":4978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16067,"num_write_kb":593792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4862,"num_bytes_recovered":318636032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.3","version":"614'15535","reported_seq":15761,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-05-15T23:32:36.713831+0000","last_change":"2024-05-15T23:30:12.055284+0000","last_active":"2024-05-15T23:32:36.713831+0000","last_peered":"2024-05-15T23:32:36.713831+0000","last_clean":"2024-05-15T23:32:36.713831+0000","last_became_active":"2024-05-15T23:28:51.305691+0000","last_became_peered":"2024-05-15T23:28:51.305691+0000","last_unstale":"2024-05-15T23:32:36.713831+0000","last_undegraded":"2024-05-15T23:32:36.713831+0000","last_fullsized":"2024-05-15T23:32:36.713831+0000","mapping_epoch":549,"log_start":"569'5500","ondisk_log_start":"569'5500","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'7899","last_scrub_stamp":"2024-05-15T23:30:12.055082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:30:12.055082+0000","objects_scrubbed":7846,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1764,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188940288,"num_objects":2883,"num_object_clones":0,"num_object_copies":5766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15535,"num_write_kb":589376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"614'15541","reported_seq":15681,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-05-15T23:32:36.776453+0000","last_change":"2024-05-15T23:28:51.519631+0000","last_active":"2024-05-15T23:32:36.776453+0000","last_peered":"2024-05-15T23:32:36.776453+0000","last_clean":"2024-05-15T23:32:36.776453+0000","last_became_active":"2024-05-15T23:28:51.519343+0000","last_became_peered":"2024-05-15T23:28:51.519343+0000","last_unstale":"2024-05-15T23:32:36.776453+0000","last_undegraded":"2024-05-15T23:32:36.776453+0000","last_fullsized":"2024-05-15T23:32:36.776453+0000","mapping_epoch":549,"log_start":"569'5500","ondisk_log_start":"569'5500","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184614912,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15541,"num_write_kb":587456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"616'15459","reported_seq":28369,"reported_epoch":616,"state":"active+recovering","last_fresh":"2024-05-15T23:32:41.074790+0000","last_change":"2024-05-15T23:30:10.714218+0000","last_active":"2024-05-15T23:32:41.074790+0000","last_peered":"2024-05-15T23:32:41.074790+0000","last_clean":"2024-05-15T23:30:09.208910+0000","last_became_active":"2024-05-15T23:30:10.487853+0000","last_became_peered":"2024-05-15T23:30:10.487853+0000","last_unstale":"2024-05-15T23:32:41.074790+0000","last_undegraded":"2024-05-15T23:32:41.074790+0000","last_fullsized":"2024-05-15T23:32:41.074790+0000","mapping_epoch":578,"log_start":"569'5400","ondisk_log_start":"569'5400","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:28:50.293593+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177405952,"num_objects":2707,"num_object_clones":0,"num_object_copies":5414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1273,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15459,"num_write_kb":581312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6383,"num_bytes_recovered":418316288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":["10","4"],"object_location_counts":[{"shards":"3,10","objects":1434},{"shards":"4,10","objects":1273}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"616'15561","reported_seq":15868,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-05-15T23:32:38.005647+0000","last_change":"2024-05-15T23:31:27.462495+0000","last_active":"2024-05-15T23:32:38.005647+0000","last_peered":"2024-05-15T23:32:38.005647+0000","last_clean":"2024-05-15T23:32:38.005647+0000","last_became_active":"2024-05-15T23:28:51.303621+0000","last_became_peered":"2024-05-15T23:28:51.303621+0000","last_unstale":"2024-05-15T23:32:38.005647+0000","last_undegraded":"2024-05-15T23:32:38.005647+0000","last_fullsized":"2024-05-15T23:32:38.005647+0000","mapping_epoch":549,"log_start":"569'5500","ondisk_log_start":"569'5500","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"598'11835","last_scrub_stamp":"2024-05-15T23:31:27.462385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:31:27.462385+0000","objects_scrubbed":6518,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:32:42.197783+0000","scrub_duration":714,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182517760,"num_objects":2785,"num_object_clones":0,"num_object_copies":5570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15561,"num_write_kb":587072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"616'15248","reported_seq":15595,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-05-15T23:32:39.739743+0000","last_change":"2024-05-15T23:32:26.208670+0000","last_active":"2024-05-15T23:32:39.739743+0000","last_peered":"2024-05-15T23:32:39.739743+0000","last_clean":"2024-05-15T23:32:39.739743+0000","last_became_active":"2024-05-15T23:28:51.304317+0000","last_became_peered":"2024-05-15T23:28:51.304317+0000","last_unstale":"2024-05-15T23:32:39.739743+0000","last_undegraded":"2024-05-15T23:32:39.739743+0000","last_fullsized":"2024-05-15T23:32:39.739743+0000","mapping_epoch":549,"log_start":"569'5200","ondisk_log_start":"569'5200","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"612'14471","last_scrub_stamp":"2024-05-15T23:32:26.208565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:26.208565+0000","objects_scrubbed":3443,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:33:35.781604+0000","scrub_duration":576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173408256,"num_objects":2646,"num_object_clones":0,"num_object_copies":5292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15248,"num_write_kb":572608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1323,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-05-15T23:32:37.244560+0000","last_change":"2024-05-15T23:32:08.827746+0000","last_active":"2024-05-15T23:32:37.244560+0000","last_peered":"2024-05-15T23:32:37.244560+0000","last_clean":"2024-05-15T23:32:37.244560+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:32:37.244560+0000","last_undegraded":"2024-05-15T23:32:37.244560+0000","last_fullsized":"2024-05-15T23:32:37.244560+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:32:08.827664+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:32:08.827664+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:33:36.958374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2882404376,"num_objects":43983,"num_object_clones":0,"num_object_copies":87966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1273,"num_objects_unfound":0,"num_objects_dirty":43983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":248355,"num_write_kb":9354753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11245,"num_bytes_recovered":736952320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6149578752,"data_stored":6149570608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160801,"ondisk_log_size":160801,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351322,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1431352,"kb_used_data":737968,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":92313800,"statfs":{"total":95995035648,"available":94529331200,"internally_reserved":0,"allocated":755679232,"data_stored":755261919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":710015428},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580256,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1107976,"kb_used_data":671360,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92637176,"statfs":{"total":95995035648,"available":94860468224,"internally_reserved":0,"allocated":687472640,"data_stored":687046957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":447019456},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776488,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1480648,"kb_used_data":845568,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92264504,"statfs":{"total":95995035648,"available":94478852096,"internally_reserved":0,"allocated":865861632,"data_stored":865435949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":650312128},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":2233382993991,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267056,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903120,"kb_used_data":348552,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92842032,"statfs":{"total":95995035648,"available":95070240768,"internally_reserved":0,"allocated":356917248,"data_stored":356488013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":567867850},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495990,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575248,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1433716,"kb_used_data":726444,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":92311436,"statfs":{"total":95995035648,"available":94526910464,"internally_reserved":0,"allocated":743878656,"data_stored":743460843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":724236742},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016657,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483346,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":841168,"kb_used_data":696384,"kb_used_omap":4,"kb_used_meta":144763,"kb_avail":92903984,"statfs":{"total":95995035648,"available":95133679616,"internally_reserved":0,"allocated":713097216,"data_stored":712671533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":148238019},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150087,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1994100,"kb_used_data":1192428,"kb_used_omap":1,"kb_used_meta":801662,"kb_avail":91751052,"statfs":{"total":95995035648,"available":93953077248,"internally_reserved":0,"allocated":1221046272,"data_stored":1220628459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":820902343},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986559,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1009296,"kb_used_data":490816,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92735856,"statfs":{"total":95995035648,"available":94961516544,"internally_reserved":0,"allocated":502595584,"data_stored":502169901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":967888,"kb_used_data":355008,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92777264,"statfs":{"total":95995035648,"available":95003918336,"internally_reserved":0,"allocated":363528192,"data_stored":363102509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":742133760,"data_stored":742129688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":705167360,"data_stored":705167360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1219104768,"data_stored":1219100696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":501743616,"data_stored":501743616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":362414080,"data_stored":362414080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":353763328,"data_stored":353763328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":849412096,"data_stored":849412096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":675545088,"data_stored":675545088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":740294656,"data_stored":740294656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:32:43.995 INFO:tasks.thrashosds.thrasher:Setting 7.b to [1, 7] 2024-05-15T23:32:43.995 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '7.b', '1', '7'] 2024-05-15T23:32:43.995 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pg-upmap 7.b 1 7 2024-05-15T23:32:44.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:44 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:32:44.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:44 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:32:44.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:44 smithi090 ceph-mon[44434]: osdmap e618: 12 total, 12 up, 9 in 2024-05-15T23:32:44.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:44 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:32:44.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:44 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:32:44.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:44 smithi088 ceph-mon[44699]: osdmap e618: 12 total, 12 up, 9 in 2024-05-15T23:32:44.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:44 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:32:44.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:44 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:32:44.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:44 smithi084 ceph-mon[39784]: osdmap e618: 12 total, 12 up, 9 in 2024-05-15T23:32:44.709 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:45.280 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:45 smithi084 ceph-mon[39784]: pgmap v1661: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 825 op/s; 1273/80874 objects misplaced (1.574%); 2.3 MiB/s, 37 objects/s recovering 2024-05-15T23:32:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:45 smithi090 ceph-mon[44434]: pgmap v1661: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 825 op/s; 1273/80874 objects misplaced (1.574%); 2.3 MiB/s, 37 objects/s recovering 2024-05-15T23:32:45.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:45 smithi088 ceph-mon[44699]: pgmap v1661: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 825 op/s; 1273/80874 objects misplaced (1.574%); 2.3 MiB/s, 37 objects/s recovering 2024-05-15T23:32:46.206 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:46.287 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:32:46.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:46.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:32:46.307 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T23:32:46.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:32:46.455 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:46.481 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:32:46.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:46.493 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:32:46.594 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:32:46.621 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:46.639 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:32:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:46.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:32:46.705 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:32:46.712 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:32:46.744 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:32:46.770 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:46.791 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:32:46.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:46.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:32:46.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:32:46.936 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:47.296 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:47 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:32:47.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:47 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:32:47.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:47 smithi090 ceph-mon[44434]: osdmap e619: 12 total, 12 up, 9 in 2024-05-15T23:32:47.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:47 smithi090 ceph-mon[44434]: pgmap v1663: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 9.9 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.05k op/s; 1004/75294 objects misplaced (1.333%); 2.8 MiB/s, 44 objects/s recovering 2024-05-15T23:32:47.297 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:47 smithi090 ceph-mon[44434]: osdmap e620: 12 total, 12 up, 9 in 2024-05-15T23:32:47.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:47 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:32:47.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:47 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:32:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:47 smithi088 ceph-mon[44699]: osdmap e619: 12 total, 12 up, 9 in 2024-05-15T23:32:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:47 smithi088 ceph-mon[44699]: pgmap v1663: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 9.9 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.05k op/s; 1004/75294 objects misplaced (1.333%); 2.8 MiB/s, 44 objects/s recovering 2024-05-15T23:32:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:47 smithi088 ceph-mon[44699]: osdmap e620: 12 total, 12 up, 9 in 2024-05-15T23:32:47.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:47 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:32:47.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:47 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:32:47.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:47 smithi084 ceph-mon[39784]: osdmap e619: 12 total, 12 up, 9 in 2024-05-15T23:32:47.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:47 smithi084 ceph-mon[39784]: pgmap v1663: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 9.9 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.05k op/s; 1004/75294 objects misplaced (1.333%); 2.8 MiB/s, 44 objects/s recovering 2024-05-15T23:32:47.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:47 smithi084 ceph-mon[39784]: osdmap e620: 12 total, 12 up, 9 in 2024-05-15T23:32:48.109 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:32:48.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:48.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:32:48.214 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:32:48.240 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:48.262 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:32:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:48.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:32:48.307 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:32:48.360 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:32:48.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:32:48.394 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:32:48.400 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/466207903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:32:48.407 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:48.440 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:32:48.479 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:32:48.513 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:32:48.550 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:32:48.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/466207903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:32:48.587 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:32:48.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/466207903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:32:48.630 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:32:48.679 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:32:48.715 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:32:48.737 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:32:48.754 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:32:48.754 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:32:48.754 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:32:48.754 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:32:48.754 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:32:48.754 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:32:48.754 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:32:48.754 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:32:48.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:32:48.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:32:48.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:32:48.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:32:49.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:49 smithi088 ceph-mon[44699]: pgmap v1665: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 749 op/s; 1004/73634 objects misplaced (1.364%); 2.8 MiB/s, 44 objects/s recovering 2024-05-15T23:32:49.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:49 smithi084 ceph-mon[39784]: pgmap v1665: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 749 op/s; 1004/73634 objects misplaced (1.364%); 2.8 MiB/s, 44 objects/s recovering 2024-05-15T23:32:49.766 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:32:49.768 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.768 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.768 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.768 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.768 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.768 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.768 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.768 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.768 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:49.768 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:49 smithi090 ceph-mon[44434]: pgmap v1665: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 749 op/s; 1004/73634 objects misplaced (1.364%); 2.8 MiB/s, 44 objects/s recovering 2024-05-15T23:32:49.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:49.769 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:32:49.870 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:32:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:50.288 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:32:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:50.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:32:50.306 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:50.395 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:32:50.422 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:50.443 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:32:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:50.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:32:50.548 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:50.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:50.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:50 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:32:50.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:50 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:32:50.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:50.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:50.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:50 smithi090 ceph-mon[44434]: osdmap e621: 12 total, 12 up, 9 in 2024-05-15T23:32:50.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:50 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:32:50.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:50 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:32:50.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:50.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:50 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:32:50.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:50 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:32:50.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:50.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:50.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:50 smithi088 ceph-mon[44699]: osdmap e621: 12 total, 12 up, 9 in 2024-05-15T23:32:50.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:50 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:32:50.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:50 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:32:50.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:50.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:50 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:32:50.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:50 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:32:50.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:50.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:50.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:50 smithi084 ceph-mon[39784]: osdmap e621: 12 total, 12 up, 9 in 2024-05-15T23:32:50.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:50 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:32:50.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:50 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:32:50.630 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:50.681 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:50.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:50.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:50.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:32:50.864 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:51.000 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:32:51.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:51.007 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:32:51.108 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:32:51.127 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:32:51.127 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:32:51.127 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:32:51.127 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:32:51.127 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:32:51.155 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:51.204 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:32:51.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:51.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:32:51.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:32:51.348 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:51.368 INFO:teuthology.orchestra.run.smithi084.stderr:set 7.b pg_upmap mapping to [1,7] 2024-05-15T23:32:51.408 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:32:51.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:51.415 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:32:51.516 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:32:51.547 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:51 smithi084 ceph-mon[39784]: osdmap e622: 12 total, 12 up, 9 in 2024-05-15T23:32:51.547 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4243135967' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.b", "id": [1, 7]}]: dispatch 2024-05-15T23:32:51.547 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:51 smithi084 ceph-mon[39784]: pgmap v1668: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 916 op/s; 687/69554 objects misplaced (0.988%); 4.0 MiB/s, 64 objects/s recovering 2024-05-15T23:32:51.548 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:51 smithi090 ceph-mon[44434]: osdmap e622: 12 total, 12 up, 9 in 2024-05-15T23:32:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4243135967' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.b", "id": [1, 7]}]: dispatch 2024-05-15T23:32:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:51 smithi090 ceph-mon[44434]: pgmap v1668: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 916 op/s; 687/69554 objects misplaced (0.988%); 4.0 MiB/s, 64 objects/s recovering 2024-05-15T23:32:51.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:32:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:51 smithi088 ceph-mon[44699]: osdmap e622: 12 total, 12 up, 9 in 2024-05-15T23:32:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4243135967' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.b", "id": [1, 7]}]: dispatch 2024-05-15T23:32:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:51 smithi088 ceph-mon[44699]: pgmap v1668: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 916 op/s; 687/69554 objects misplaced (0.988%); 4.0 MiB/s, 64 objects/s recovering 2024-05-15T23:32:51.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T23:32:52.100 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:32:52.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:52.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:32:52.160 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:32:52.204 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:32:52.231 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:52.250 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:32:52.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:52.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:32:52.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:32:52.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:52 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4243135967' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.b", "id": [1, 7]}]': finished 2024-05-15T23:32:52.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:52 smithi084 ceph-mon[39784]: osdmap e623: 12 total, 12 up, 9 in 2024-05-15T23:32:52.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:52 smithi084 ceph-mon[39784]: osdmap e624: 12 total, 12 up, 9 in 2024-05-15T23:32:52.383 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:52.506 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:32:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:52.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:32:52.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:52 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4243135967' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.b", "id": [1, 7]}]': finished 2024-05-15T23:32:52.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:52 smithi088 ceph-mon[44699]: osdmap e623: 12 total, 12 up, 9 in 2024-05-15T23:32:52.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:52 smithi088 ceph-mon[44699]: osdmap e624: 12 total, 12 up, 9 in 2024-05-15T23:32:52.610 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:32:52.637 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:52.656 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:32:52.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:52.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:32:52.700 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:52 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4243135967' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.b", "id": [1, 7]}]': finished 2024-05-15T23:32:52.701 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:52 smithi090 ceph-mon[44434]: osdmap e623: 12 total, 12 up, 9 in 2024-05-15T23:32:52.701 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:52 smithi090 ceph-mon[44434]: osdmap e624: 12 total, 12 up, 9 in 2024-05-15T23:32:52.760 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:32:52.789 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:52.809 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:32:52.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:52.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:32:52.916 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:32:52.943 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:52.963 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:32:52.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:52.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:32:53.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:32:53.098 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:53.117 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:53.119 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:32:53.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:53.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:32:53.225 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:32:53.252 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:53.356 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:53 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:32:53.356 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:53 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:32:53.356 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:53 smithi084 ceph-mon[39784]: pgmap v1671: 17 pgs: 1 peering, 1 active+recovering, 15 active+clean; 2.0 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 687/64756 objects misplaced (1.061%); 4.9 MiB/s, 79 objects/s recovering 2024-05-15T23:32:53.356 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:53 smithi084 ceph-mon[39784]: osdmap e625: 12 total, 12 up, 9 in 2024-05-15T23:32:53.358 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:32:53.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:53.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:32:53.463 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:32:53.493 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:53.757 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:53 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:32:53.757 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:53 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:32:53.757 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:53 smithi090 ceph-mon[44434]: pgmap v1671: 17 pgs: 1 peering, 1 active+recovering, 15 active+clean; 2.0 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 687/64756 objects misplaced (1.061%); 4.9 MiB/s, 79 objects/s recovering 2024-05-15T23:32:53.757 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:53 smithi090 ceph-mon[44434]: osdmap e625: 12 total, 12 up, 9 in 2024-05-15T23:32:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:53 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:32:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:53 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:32:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:53 smithi088 ceph-mon[44699]: pgmap v1671: 17 pgs: 1 peering, 1 active+recovering, 15 active+clean; 2.0 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 687/64756 objects misplaced (1.061%); 4.9 MiB/s, 79 objects/s recovering 2024-05-15T23:32:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:53 smithi088 ceph-mon[44699]: osdmap e625: 12 total, 12 up, 9 in 2024-05-15T23:32:53.945 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:32:53.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:53.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:32:54.050 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:32:54.079 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:54.098 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:32:54.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:54.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:32:54.105 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:32:54.106 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:32:54.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:32:54.106 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [] 2024-05-15T23:32:54.106 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:32:54.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:54.246 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:54.361 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:54 smithi088 ceph-mon[44699]: 7.b starting backfill to osd.7 from (0'0,0'0] MAX to 622'15874 2024-05-15T23:32:54.361 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2070941465' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:32:54.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:54 smithi084 ceph-mon[39784]: 7.b starting backfill to osd.7 from (0'0,0'0] MAX to 622'15874 2024-05-15T23:32:54.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2070941465' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:32:54.366 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:54.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:54.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:54.376 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:32:54.476 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:32:54.504 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:54.526 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:32:54.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:54.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:32:54.530 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:54 smithi090 ceph-mon[44434]: 7.b starting backfill to osd.7 from (0'0,0'0] MAX to 622'15874 2024-05-15T23:32:54.530 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2070941465' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:32:54.630 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:54.707 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:54.816 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:54.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:54.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T23:32:54.929 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:32:55.052 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:55.201 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:32:55.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:55.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:32:55.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:32:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:55.374 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:32:55.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:55.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:32:55.482 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:32:55.509 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:55.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:55.532 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:32:55.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:32:55.620 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:32:55.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:55 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:32:55.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:55 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2070941465' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:32:55.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:55 smithi084 ceph-mon[39784]: osdmap e626: 12 total, 12 up, 9 in 2024-05-15T23:32:55.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:55 smithi084 ceph-mon[39784]: pgmap v1674: 17 pgs: 1 peering, 1 active+recovering, 15 active+clean; 1.9 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 710 op/s; 687/63868 objects misplaced (1.076%) 2024-05-15T23:32:55.634 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:32:55.659 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:55 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:32:55.659 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:55 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2070941465' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:32:55.659 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:55 smithi088 ceph-mon[44699]: osdmap e626: 12 total, 12 up, 9 in 2024-05-15T23:32:55.659 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:55 smithi088 ceph-mon[44699]: pgmap v1674: 17 pgs: 1 peering, 1 active+recovering, 15 active+clean; 1.9 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 710 op/s; 687/63868 objects misplaced (1.076%) 2024-05-15T23:32:55.661 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:55.671 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:32:55.680 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:32:55.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:55.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:32:55.709 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:32:55.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:32:55.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:32:55.796 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:32:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:55 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:32:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:55 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2070941465' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:32:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:55 smithi090 ceph-mon[44434]: osdmap e626: 12 total, 12 up, 9 in 2024-05-15T23:32:55.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:55 smithi090 ceph-mon[44434]: pgmap v1674: 17 pgs: 1 peering, 1 active+recovering, 15 active+clean; 1.9 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 710 op/s; 687/63868 objects misplaced (1.076%) 2024-05-15T23:32:55.817 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:55.832 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:32:55.845 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:32:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:55.855 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:32:55.860 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:32:55.895 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:32:55.942 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:32:55.956 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:32:55.984 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:56.007 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:32:56.010 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:32:56.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:56.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:32:56.037 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:32:56.070 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:56.076 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:32:56.095 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:32:56.095 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:32:56.095 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:32:56.095 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:32:56.095 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:32:56.096 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:32:56.096 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:32:56.096 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:32:56.096 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:32:56.096 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:32:56.096 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:32:56.096 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:32:56.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:56.172 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:56.351 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:56.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:56.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:56.464 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:32:56.491 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:56.513 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:32:56.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:56.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:32:56.617 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:32:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2892939656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:32:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:56 smithi084 ceph-mon[39784]: 7.0 scrub starts 2024-05-15T23:32:56.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:56 smithi084 ceph-mon[39784]: 7.0 scrub ok 2024-05-15T23:32:56.643 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2892939656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:32:56.643 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.643 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.643 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.643 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.643 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.644 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:56 smithi088 ceph-mon[44699]: 7.0 scrub starts 2024-05-15T23:32:56.644 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.644 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:56 smithi088 ceph-mon[44699]: 7.0 scrub ok 2024-05-15T23:32:56.645 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:56.664 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:32:56.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:56.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:32:56.770 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:32:56.797 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:56.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2892939656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:32:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:56 smithi090 ceph-mon[44434]: 7.0 scrub starts 2024-05-15T23:32:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:56 smithi090 ceph-mon[44434]: 7.0 scrub ok 2024-05-15T23:32:56.817 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:32:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:56.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:32:56.921 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:32:56.985 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:57.059 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:32:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:57.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:32:57.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:32:57.236 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:57.368 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:32:57.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:57.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:32:57.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:32:57.524 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10, 2] out_osds: [0, 6, 8] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:32:57.525 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:32:57.525 INFO:tasks.thrashosds.thrasher:Adding osd 8 2024-05-15T23:32:57.525 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd in 8 2024-05-15T23:32:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:57.550 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:32:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:57.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:32:57.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:57 smithi084 ceph-mon[39784]: pgmap v1675: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering, 15 active+clean; 1.8 GiB data, 9.1 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 778 op/s; 1327/60418 objects misplaced (2.196%); 13 MiB/s, 206 objects/s recovering 2024-05-15T23:32:57.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:57 smithi084 ceph-mon[39784]: osdmap e627: 12 total, 12 up, 9 in 2024-05-15T23:32:57.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:57.734 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:57 smithi090 ceph-mon[44434]: pgmap v1675: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering, 15 active+clean; 1.8 GiB data, 9.1 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 778 op/s; 1327/60418 objects misplaced (2.196%); 13 MiB/s, 206 objects/s recovering 2024-05-15T23:32:57.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:57 smithi090 ceph-mon[44434]: osdmap e627: 12 total, 12 up, 9 in 2024-05-15T23:32:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:32:57.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:57 smithi088 ceph-mon[44699]: pgmap v1675: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering, 15 active+clean; 1.8 GiB data, 9.1 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 778 op/s; 1327/60418 objects misplaced (2.196%); 13 MiB/s, 206 objects/s recovering 2024-05-15T23:32:57.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:57 smithi088 ceph-mon[44699]: osdmap e627: 12 total, 12 up, 9 in 2024-05-15T23:32:57.994 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:32:58.402 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:58.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:58.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:58.455 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:32:58.513 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:32:58.576 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:58.969 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:32:59.564 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:32:59.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:59.612 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:32:59 smithi088 ceph-mon[44699]: pgmap v1677: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering, 15 active+clean; 1.8 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 482 op/s; 1327/58962 objects misplaced (2.251%); 13 MiB/s, 202 objects/s recovering 2024-05-15T23:32:59.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:32:59.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:32:59 smithi084 ceph-mon[39784]: pgmap v1677: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering, 15 active+clean; 1.8 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 482 op/s; 1327/58962 objects misplaced (2.251%); 13 MiB/s, 202 objects/s recovering 2024-05-15T23:32:59.714 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:32:59.754 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:32:59.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:32:59 smithi090 ceph-mon[44434]: pgmap v1677: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering, 15 active+clean; 1.8 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 482 op/s; 1327/58962 objects misplaced (2.251%); 13 MiB/s, 202 objects/s recovering 2024-05-15T23:32:59.903 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:32:59.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:32:59.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:00.009 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:00.055 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:00.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:00 smithi084 ceph-mon[39784]: 7.4 scrub starts 2024-05-15T23:33:00.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:00 smithi084 ceph-mon[39784]: 7.4 scrub ok 2024-05-15T23:33:00.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:00 smithi084 ceph-mon[39784]: 7.1 scrub starts 2024-05-15T23:33:00.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:00 smithi088 ceph-mon[44699]: 7.4 scrub starts 2024-05-15T23:33:00.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:00 smithi088 ceph-mon[44699]: 7.4 scrub ok 2024-05-15T23:33:00.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:00 smithi088 ceph-mon[44699]: 7.1 scrub starts 2024-05-15T23:33:00.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:00 smithi090 ceph-mon[44434]: 7.4 scrub starts 2024-05-15T23:33:00.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:00 smithi090 ceph-mon[44434]: 7.4 scrub ok 2024-05-15T23:33:00.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:00 smithi090 ceph-mon[44434]: 7.1 scrub starts 2024-05-15T23:33:01.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:01 smithi088 ceph-mon[44699]: 7.1 scrub ok 2024-05-15T23:33:01.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:01 smithi088 ceph-mon[44699]: 7.f scrub starts 2024-05-15T23:33:01.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:01 smithi088 ceph-mon[44699]: 7.f scrub ok 2024-05-15T23:33:01.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:01 smithi088 ceph-mon[44699]: 7.2 scrub starts 2024-05-15T23:33:01.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:01 smithi088 ceph-mon[44699]: pgmap v1678: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.7 GiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 632 op/s; 1269/54670 objects misplaced (2.321%); 10 MiB/s, 159 objects/s recovering 2024-05-15T23:33:01.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:01 smithi084 ceph-mon[39784]: 7.1 scrub ok 2024-05-15T23:33:01.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:01 smithi084 ceph-mon[39784]: 7.f scrub starts 2024-05-15T23:33:01.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:01 smithi084 ceph-mon[39784]: 7.f scrub ok 2024-05-15T23:33:01.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:01 smithi084 ceph-mon[39784]: 7.2 scrub starts 2024-05-15T23:33:01.624 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:01 smithi084 ceph-mon[39784]: pgmap v1678: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.7 GiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 632 op/s; 1269/54670 objects misplaced (2.321%); 10 MiB/s, 159 objects/s recovering 2024-05-15T23:33:01.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:01 smithi090 ceph-mon[44434]: 7.1 scrub ok 2024-05-15T23:33:01.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:01 smithi090 ceph-mon[44434]: 7.f scrub starts 2024-05-15T23:33:01.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:01 smithi090 ceph-mon[44434]: 7.f scrub ok 2024-05-15T23:33:01.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:01 smithi090 ceph-mon[44434]: 7.2 scrub starts 2024-05-15T23:33:01.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:01 smithi090 ceph-mon[44434]: pgmap v1678: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.7 GiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 632 op/s; 1269/54670 objects misplaced (2.321%); 10 MiB/s, 159 objects/s recovering 2024-05-15T23:33:02.535 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:02.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:02.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:02.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:02 smithi084 ceph-mon[39784]: 7.a scrub starts 2024-05-15T23:33:02.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:02 smithi084 ceph-mon[39784]: 7.2 scrub ok 2024-05-15T23:33:02.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:02 smithi084 ceph-mon[39784]: 7.a scrub ok 2024-05-15T23:33:02.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:02 smithi084 ceph-mon[39784]: osdmap e628: 12 total, 12 up, 9 in 2024-05-15T23:33:02.639 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:02.664 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:02 smithi088 ceph-mon[44699]: 7.a scrub starts 2024-05-15T23:33:02.664 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:02 smithi088 ceph-mon[44699]: 7.2 scrub ok 2024-05-15T23:33:02.664 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:02 smithi088 ceph-mon[44699]: 7.a scrub ok 2024-05-15T23:33:02.664 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:02 smithi088 ceph-mon[44699]: osdmap e628: 12 total, 12 up, 9 in 2024-05-15T23:33:02.665 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:02.684 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:02.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:02.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:33:02.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:02 smithi090 ceph-mon[44434]: 7.a scrub starts 2024-05-15T23:33:02.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:02 smithi090 ceph-mon[44434]: 7.2 scrub ok 2024-05-15T23:33:02.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:02 smithi090 ceph-mon[44434]: 7.a scrub ok 2024-05-15T23:33:02.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:02 smithi090 ceph-mon[44434]: osdmap e628: 12 total, 12 up, 9 in 2024-05-15T23:33:02.805 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:02.815 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:33:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:02.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:33:02.848 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:33:02.849 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:33:02.936 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:02.965 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:03.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:03 smithi084 ceph-mon[39784]: pgmap v1680: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.5 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 911 op/s; 86/49286 objects misplaced (0.174%); 18 MiB/s, 288 objects/s recovering 2024-05-15T23:33:03.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:03 smithi084 ceph-mon[39784]: osdmap e629: 12 total, 12 up, 9 in 2024-05-15T23:33:03.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:03 smithi090 ceph-mon[44434]: pgmap v1680: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.5 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 911 op/s; 86/49286 objects misplaced (0.174%); 18 MiB/s, 288 objects/s recovering 2024-05-15T23:33:03.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:03 smithi090 ceph-mon[44434]: osdmap e629: 12 total, 12 up, 9 in 2024-05-15T23:33:03.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:03 smithi088 ceph-mon[44699]: pgmap v1680: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.5 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 911 op/s; 86/49286 objects misplaced (0.174%); 18 MiB/s, 288 objects/s recovering 2024-05-15T23:33:03.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:03 smithi088 ceph-mon[44699]: osdmap e629: 12 total, 12 up, 9 in 2024-05-15T23:33:04.031 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T23:33:04.358 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:04 smithi084 ceph-mon[39784]: 7.7 scrub starts 2024-05-15T23:33:04.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:04 smithi090 ceph-mon[44434]: 7.7 scrub starts 2024-05-15T23:33:04.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:04 smithi088 ceph-mon[44699]: 7.7 scrub starts 2024-05-15T23:33:05.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:05.162 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:05.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:05.263 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:05.292 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:05.311 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:05.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:05.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:33:05.416 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:33:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:05.463 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:33:05.464 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:05.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:05.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:33:05.566 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:05.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:05 smithi088 ceph-mon[44699]: 7.7 scrub ok 2024-05-15T23:33:05.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:05 smithi088 ceph-mon[44699]: pgmap v1682: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.5 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 802 op/s; 86/47624 objects misplaced (0.181%); 8.6 MiB/s, 137 objects/s recovering 2024-05-15T23:33:05.635 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:05 smithi084 ceph-mon[39784]: 7.7 scrub ok 2024-05-15T23:33:05.636 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:05 smithi084 ceph-mon[39784]: pgmap v1682: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.5 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 802 op/s; 86/47624 objects misplaced (0.181%); 8.6 MiB/s, 137 objects/s recovering 2024-05-15T23:33:05.642 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:05.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:05 smithi090 ceph-mon[44434]: 7.7 scrub ok 2024-05-15T23:33:05.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:05 smithi090 ceph-mon[44434]: pgmap v1682: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.5 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 802 op/s; 86/47624 objects misplaced (0.181%); 8.6 MiB/s, 137 objects/s recovering 2024-05-15T23:33:06.416 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:33:06.765 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:06 smithi084 ceph-mon[39784]: 7.c scrub starts 2024-05-15T23:33:06.765 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:06 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2743538267' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:33:06.765 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:06 smithi084 ceph-mon[39784]: from='client.36848 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:33:06.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:06 smithi090 ceph-mon[44434]: 7.c scrub starts 2024-05-15T23:33:06.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:06 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2743538267' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:33:06.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:06 smithi090 ceph-mon[44434]: from='client.36848 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:33:06.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:06 smithi088 ceph-mon[44699]: 7.c scrub starts 2024-05-15T23:33:06.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:06 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2743538267' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:33:06.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:06 smithi088 ceph-mon[44699]: from='client.36848 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:33:07.021 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:07.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:07.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:07.125 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:33:07.160 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:07.188 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:33:07.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:07.191 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:33:07.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:33:07.321 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:07.353 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:33:07.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:07.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:33:07.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:07.505 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:07.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:07 smithi090 ceph-mon[44434]: 7.c scrub ok 2024-05-15T23:33:07.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:07 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:33:07.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:07 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:33:07.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:07 smithi090 ceph-mon[44434]: from='client.36848 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:33:07.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:07 smithi090 ceph-mon[44434]: osdmap e630: 12 total, 12 up, 9 in 2024-05-15T23:33:07.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:07 smithi090 ceph-mon[44434]: pgmap v1684: 17 pgs: 17 active+clean; 1.3 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 12 MiB/s, 187 objects/s recovering 2024-05-15T23:33:07.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:07 smithi090 ceph-mon[44434]: osdmap e631: 12 total, 12 up, 9 in 2024-05-15T23:33:07.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:07 smithi088 ceph-mon[44699]: 7.c scrub ok 2024-05-15T23:33:07.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:07 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:33:07.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:07 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:33:07.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:07 smithi088 ceph-mon[44699]: from='client.36848 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:33:07.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:07 smithi088 ceph-mon[44699]: osdmap e630: 12 total, 12 up, 9 in 2024-05-15T23:33:07.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:07 smithi088 ceph-mon[44699]: pgmap v1684: 17 pgs: 17 active+clean; 1.3 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 12 MiB/s, 187 objects/s recovering 2024-05-15T23:33:07.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:07 smithi088 ceph-mon[44699]: osdmap e631: 12 total, 12 up, 9 in 2024-05-15T23:33:07.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:07 smithi084 ceph-mon[39784]: 7.c scrub ok 2024-05-15T23:33:07.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:07 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:33:07.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:07 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:33:07.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:07 smithi084 ceph-mon[39784]: from='client.36848 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:33:07.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:07 smithi084 ceph-mon[39784]: osdmap e630: 12 total, 12 up, 9 in 2024-05-15T23:33:07.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:07 smithi084 ceph-mon[39784]: pgmap v1684: 17 pgs: 17 active+clean; 1.3 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 12 MiB/s, 187 objects/s recovering 2024-05-15T23:33:07.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:07 smithi084 ceph-mon[39784]: osdmap e631: 12 total, 12 up, 9 in 2024-05-15T23:33:07.878 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:07.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:07.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:07.989 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:08.016 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:08.036 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:08.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:08.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:33:08.140 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:08.166 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:08.187 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:08.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:08.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:33:08.291 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:08.330 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:08.526 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:08.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:08.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:33:08.673 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:08.696 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:33:08.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:08.700 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:33:08.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:33:08.828 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:08.849 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:33:08.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:08.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:33:08.918 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:08.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:33:08.993 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:09.037 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:33:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:09.045 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:33:09.146 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:09.173 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:09.192 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:09.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:09.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:09.296 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:33:09.323 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:09.342 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:33:09.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:09.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:33:09.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:33:09.500 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:09.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:09 smithi088 ceph-mon[44699]: pgmap v1686: 17 pgs: 17 active+clean; 1.2 GiB data, 7.4 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 730 op/s; 885 KiB/s, 13 objects/s recovering 2024-05-15T23:33:09.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:09 smithi084 ceph-mon[39784]: pgmap v1686: 17 pgs: 17 active+clean; 1.2 GiB data, 7.4 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 730 op/s; 885 KiB/s, 13 objects/s recovering 2024-05-15T23:33:09.665 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:33:09.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:09.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:09.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:33:09.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:09 smithi090 ceph-mon[44434]: pgmap v1686: 17 pgs: 17 active+clean; 1.2 GiB data, 7.4 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 730 op/s; 885 KiB/s, 13 objects/s recovering 2024-05-15T23:33:09.818 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:09.858 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:33:09.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:09.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:33:09.962 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:33:10.035 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:10.165 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:33:10.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:10.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:10.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:33:10.314 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:10.316 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:10 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:33:10.312+0000 7fa0938b7700 -1 mon.a@0(leader).osd e631 definitely_dead 0 2024-05-15T23:33:10.349 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:33:10.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:10.355 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:33:10.357 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.8. 2024-05-15T23:33:10.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:33:10.483 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:10.502 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:33:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:10.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:33:10.529 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:33:10.548 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:33:10.567 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:33:10.586 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:33:10.605 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:33:10.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:10 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/314136378' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-05-15T23:33:10.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:33:10.637 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:10.649 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:33:10.669 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:33:10.688 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:33:10.707 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:33:10.727 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:33:10.748 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:33:10.749 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:33:10.749 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:33:10.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:33:10.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:33:10.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:33:10.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:33:10.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:33:10.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:33:10.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:33:10.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:33:10.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:33:10.750 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:33:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:10 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/314136378' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-05-15T23:33:10.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:10 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/314136378' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-05-15T23:33:10.969 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:33:11.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:11 smithi084 ceph-mon[39784]: 7.8 scrub starts 2024-05-15T23:33:11.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:11 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/314136378' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-05-15T23:33:11.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:11 smithi084 ceph-mon[39784]: osdmap e632: 12 total, 12 up, 10 in 2024-05-15T23:33:11.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:11 smithi084 ceph-mon[39784]: 7.8 scrub ok 2024-05-15T23:33:11.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:11 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/194086959' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:33:11.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:11 smithi084 ceph-mon[39784]: pgmap v1688: 17 pgs: 17 active+clean; 1.1 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 888 op/s; 885 KiB/s, 13 objects/s recovering 2024-05-15T23:33:11.665 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:11.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:11.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:11.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:11.797 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:11 smithi090 ceph-mon[44434]: 7.8 scrub starts 2024-05-15T23:33:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:11 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/314136378' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-05-15T23:33:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:11 smithi090 ceph-mon[44434]: osdmap e632: 12 total, 12 up, 10 in 2024-05-15T23:33:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:11 smithi090 ceph-mon[44434]: 7.8 scrub ok 2024-05-15T23:33:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:11 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/194086959' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:33:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:11 smithi090 ceph-mon[44434]: pgmap v1688: 17 pgs: 17 active+clean; 1.1 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 888 op/s; 885 KiB/s, 13 objects/s recovering 2024-05-15T23:33:11.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:11 smithi088 ceph-mon[44699]: 7.8 scrub starts 2024-05-15T23:33:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:11 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/314136378' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-05-15T23:33:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:11 smithi088 ceph-mon[44699]: osdmap e632: 12 total, 12 up, 10 in 2024-05-15T23:33:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:11 smithi088 ceph-mon[44699]: 7.8 scrub ok 2024-05-15T23:33:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:11 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/194086959' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:33:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:11 smithi088 ceph-mon[44699]: pgmap v1688: 17 pgs: 17 active+clean; 1.1 GiB data, 7.2 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 888 op/s; 885 KiB/s, 13 objects/s recovering 2024-05-15T23:33:12.145 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:12.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:12.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:12.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:33:12.278 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:12 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:12.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:12 smithi084 ceph-mon[39784]: osdmap e633: 12 total, 12 up, 10 in 2024-05-15T23:33:12.726 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:33:12.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:12.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:33:12.750 INFO:tasks.thrashosds.thrasher:Added osd 8 2024-05-15T23:33:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:12.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:12.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:12.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:12 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:12.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:12 smithi090 ceph-mon[44434]: osdmap e633: 12 total, 12 up, 10 in 2024-05-15T23:33:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:12 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:12 smithi088 ceph-mon[44699]: osdmap e633: 12 total, 12 up, 10 in 2024-05-15T23:33:12.831 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:33:12.860 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:12.881 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:33:12.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:12.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:33:12.986 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:13.013 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:13.031 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:13.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:13.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:33:13.180 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:13.212 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:13.239 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:33:13.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:13.247 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:33:13.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:33:13.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:13 smithi084 ceph-mon[39784]: 7.7 starting backfill to osd.8 from (0'0,0'0] MAX to 631'17087 2024-05-15T23:33:13.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:13 smithi084 ceph-mon[39784]: osdmap e634: 12 total, 12 up, 10 in 2024-05-15T23:33:13.363 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:13 smithi084 ceph-mon[39784]: 7.f starting backfill to osd.8 from (0'0,0'0] MAX to 631'17201 2024-05-15T23:33:13.363 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:13 smithi084 ceph-mon[39784]: 7.9 starting backfill to osd.8 from (0'0,0'0] MAX to 631'17294 2024-05-15T23:33:13.363 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:13 smithi084 ceph-mon[39784]: pgmap v1691: 17 pgs: 1 activating+remapped, 1 remapped+peering, 15 active+clean; 1.0 GiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 780 op/s; 1026/32920 objects misplaced (3.117%) 2024-05-15T23:33:13.404 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:13.449 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:33:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:13.455 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:13 smithi090 ceph-mon[44434]: 7.7 starting backfill to osd.8 from (0'0,0'0] MAX to 631'17087 2024-05-15T23:33:13.455 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:13 smithi090 ceph-mon[44434]: osdmap e634: 12 total, 12 up, 10 in 2024-05-15T23:33:13.455 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:13 smithi090 ceph-mon[44434]: 7.f starting backfill to osd.8 from (0'0,0'0] MAX to 631'17201 2024-05-15T23:33:13.455 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:13 smithi090 ceph-mon[44434]: 7.9 starting backfill to osd.8 from (0'0,0'0] MAX to 631'17294 2024-05-15T23:33:13.455 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:13 smithi090 ceph-mon[44434]: pgmap v1691: 17 pgs: 1 activating+remapped, 1 remapped+peering, 15 active+clean; 1.0 GiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 780 op/s; 1026/32920 objects misplaced (3.117%) 2024-05-15T23:33:13.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:33:13.557 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:33:13.583 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:13.602 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:33:13.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:13.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:33:13.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:13.756 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:13 smithi088 ceph-mon[44699]: 7.7 starting backfill to osd.8 from (0'0,0'0] MAX to 631'17087 2024-05-15T23:33:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:13 smithi088 ceph-mon[44699]: osdmap e634: 12 total, 12 up, 10 in 2024-05-15T23:33:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:13 smithi088 ceph-mon[44699]: 7.f starting backfill to osd.8 from (0'0,0'0] MAX to 631'17201 2024-05-15T23:33:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:13 smithi088 ceph-mon[44699]: 7.9 starting backfill to osd.8 from (0'0,0'0] MAX to 631'17294 2024-05-15T23:33:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:13 smithi088 ceph-mon[44699]: pgmap v1691: 17 pgs: 1 activating+remapped, 1 remapped+peering, 15 active+clean; 1.0 GiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 780 op/s; 1026/32920 objects misplaced (3.117%) 2024-05-15T23:33:13.845 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:13.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:13.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:13.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:33:13.998 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:14.015 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:33:14.015 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:33:14.015 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:33:14.015 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:33:14.015 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:33:14.045 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:33:14.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:14.053 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:33:14.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:33:14.154 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:33:14.181 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:14.199 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:33:14.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:14.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:33:14.303 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:14.330 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:14.350 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:14.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:14.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:33:14.453 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:33:14.481 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:14.500 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:33:14.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:14.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:33:14.604 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:33:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:14.651 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:33:14.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:14.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:33:14.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:33:14.803 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:14.833 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:33:14.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:14.837 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:33:14.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:14.990 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:15.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:14 smithi090 ceph-mon[44434]: 7.3 scrub starts 2024-05-15T23:33:15.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:14 smithi084 ceph-mon[39784]: 7.3 scrub starts 2024-05-15T23:33:15.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:14 smithi088 ceph-mon[44699]: 7.3 scrub starts 2024-05-15T23:33:16.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:15 smithi090 ceph-mon[44434]: 7.3 scrub ok 2024-05-15T23:33:16.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:15 smithi090 ceph-mon[44434]: pgmap v1692: 17 pgs: 1 activating+remapped, 1 remapped+peering, 15 active+clean; 1000 MiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 710 op/s; 1026/31978 objects misplaced (3.208%) 2024-05-15T23:33:16.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:15 smithi088 ceph-mon[44699]: 7.3 scrub ok 2024-05-15T23:33:16.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:15 smithi088 ceph-mon[44699]: pgmap v1692: 17 pgs: 1 activating+remapped, 1 remapped+peering, 15 active+clean; 1000 MiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 710 op/s; 1026/31978 objects misplaced (3.208%) 2024-05-15T23:33:16.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:15 smithi084 ceph-mon[39784]: 7.3 scrub ok 2024-05-15T23:33:16.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:15 smithi084 ceph-mon[39784]: pgmap v1692: 17 pgs: 1 activating+remapped, 1 remapped+peering, 15 active+clean; 1000 MiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 710 op/s; 1026/31978 objects misplaced (3.208%) 2024-05-15T23:33:17.192 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T23:33:17.545 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:17.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:17.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:17.651 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:33:17.695 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:17.745 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:33:17.751 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10, 2, 8] out_osds: [0, 6] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:33:17.751 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:33:17.751 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_5 2024-05-15T23:33:17.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:33:17.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:17.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:17.853 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:33:17.925 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:17.964 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:17.980 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:33:17.987 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:17.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:17.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:33:18.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:18 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:33:18.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:18 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:33:18.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:18 smithi084 ceph-mon[39784]: osdmap e635: 12 total, 12 up, 10 in 2024-05-15T23:33:18.079 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:18 smithi084 ceph-mon[39784]: pgmap v1694: 17 pgs: 2 active+remapped+backfilling, 1 activating+remapped, 14 active+clean; 846 MiB data, 6.7 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 824 op/s; 1848/27046 objects misplaced (6.833%); 8.9 MiB/s, 142 objects/s recovering 2024-05-15T23:33:18.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:33:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:33:18.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:18.171 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:33:18.272 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:18.300 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:18.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:18 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:33:18.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:18 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:33:18.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:18 smithi088 ceph-mon[44699]: osdmap e635: 12 total, 12 up, 10 in 2024-05-15T23:33:18.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:18 smithi088 ceph-mon[44699]: pgmap v1694: 17 pgs: 2 active+remapped+backfilling, 1 activating+remapped, 14 active+clean; 846 MiB data, 6.7 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 824 op/s; 1848/27046 objects misplaced (6.833%); 8.9 MiB/s, 142 objects/s recovering 2024-05-15T23:33:18.321 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:18.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:18.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:18.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:33:18.454 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:18 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:33:18.455 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:18 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:33:18.455 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:18 smithi090 ceph-mon[44434]: osdmap e635: 12 total, 12 up, 10 in 2024-05-15T23:33:18.455 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:18 smithi090 ceph-mon[44434]: pgmap v1694: 17 pgs: 2 active+remapped+backfilling, 1 activating+remapped, 14 active+clean; 846 MiB data, 6.7 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 824 op/s; 1848/27046 objects misplaced (6.833%); 8.9 MiB/s, 142 objects/s recovering 2024-05-15T23:33:18.457 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:18.498 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:33:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:18.508 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:33:18.609 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:33:18.648 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:19.119 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:33:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2733177971' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:33:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:19 smithi084 ceph-mon[39784]: from='client.26899 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:33:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:19 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:33:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:19 smithi084 ceph-mon[39784]: from='client.26899 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:33:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:19 smithi084 ceph-mon[39784]: osdmap e636: 12 total, 12 up, 10 in 2024-05-15T23:33:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2733177971' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:33:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:19 smithi090 ceph-mon[44434]: from='client.26899 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:33:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:19 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:33:19.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:19 smithi090 ceph-mon[44434]: from='client.26899 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:33:19.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:19 smithi090 ceph-mon[44434]: osdmap e636: 12 total, 12 up, 10 in 2024-05-15T23:33:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2733177971' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:33:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:19 smithi088 ceph-mon[44699]: from='client.26899 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:33:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:19 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:33:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:19 smithi088 ceph-mon[44699]: from='client.26899 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:33:19.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:19 smithi088 ceph-mon[44699]: osdmap e636: 12 total, 12 up, 10 in 2024-05-15T23:33:19.964 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:33:19.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:19.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:20.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:33:20.103 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:20.126 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:33:20.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:20.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:33:20.231 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:20.275 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:20 smithi084 ceph-mon[39784]: pgmap v1696: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 795 MiB data, 6.7 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 624 op/s; 970/25412 objects misplaced (3.817%); 14 MiB/s, 226 objects/s recovering 2024-05-15T23:33:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:20 smithi084 ceph-mon[39784]: osdmap e637: 12 total, 12 up, 10 in 2024-05-15T23:33:20.479 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:20.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:20 smithi090 ceph-mon[44434]: pgmap v1696: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 795 MiB data, 6.7 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 624 op/s; 970/25412 objects misplaced (3.817%); 14 MiB/s, 226 objects/s recovering 2024-05-15T23:33:20.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:20 smithi090 ceph-mon[44434]: osdmap e637: 12 total, 12 up, 10 in 2024-05-15T23:33:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:20 smithi088 ceph-mon[44699]: pgmap v1696: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 795 MiB data, 6.7 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 624 op/s; 970/25412 objects misplaced (3.817%); 14 MiB/s, 226 objects/s recovering 2024-05-15T23:33:20.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:20 smithi088 ceph-mon[44699]: osdmap e637: 12 total, 12 up, 10 in 2024-05-15T23:33:20.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:33:20.609 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:20.628 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:33:20.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:20.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:33:20.732 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:20.767 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:20.788 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:20.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:20.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:20.892 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:20.923 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:20.942 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:20.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:20.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:21.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:33:21.081 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:21.108 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:33:21.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:21.111 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:33:21.212 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:33:21.262 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:21.297 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:33:21.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:21.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:33:21.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:21.444 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:21.859 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:33:21.871 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:22.142 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:22.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:22.259 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:33:22.315 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:22.344 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:33:22.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:22.349 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:33:22.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:22 smithi090 ceph-mon[44434]: osdmap e638: 12 total, 12 up, 10 in 2024-05-15T23:33:22.349 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:22 smithi090 ceph-mon[44434]: pgmap v1699: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 710 MiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 518 op/s; 970/22702 objects misplaced (4.273%); 10 MiB/s, 167 objects/s recovering 2024-05-15T23:33:22.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:22 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:33:22.350 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:22 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:33:22.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:22 smithi084 ceph-mon[39784]: osdmap e638: 12 total, 12 up, 10 in 2024-05-15T23:33:22.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:22 smithi084 ceph-mon[39784]: pgmap v1699: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 710 MiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 518 op/s; 970/22702 objects misplaced (4.273%); 10 MiB/s, 167 objects/s recovering 2024-05-15T23:33:22.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:22 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:33:22.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:22 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:33:22.449 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:22.475 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:22 smithi088 ceph-mon[44699]: osdmap e638: 12 total, 12 up, 10 in 2024-05-15T23:33:22.475 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:22 smithi088 ceph-mon[44699]: pgmap v1699: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 710 MiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 518 op/s; 970/22702 objects misplaced (4.273%); 10 MiB/s, 167 objects/s recovering 2024-05-15T23:33:22.475 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:22 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:33:22.475 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:22 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:33:22.476 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:22.497 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:22.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:22.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:33:22.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:33:22.643 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:22.673 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:33:22.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:22.678 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:33:22.731 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:33:22.760 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:33:22.780 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:33:22.798 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:33:22.807 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:22.827 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:33:22.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:22.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:33:22.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:33:22.884 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:33:22.914 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:33:22.931 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:33:22.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:33:22.957 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:22.975 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:33:22.977 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:33:22.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:22.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:33:23.010 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:33:23.041 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:33:23.074 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:33:23.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:23.120 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:33:23.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:23 smithi084 ceph-mon[39784]: osdmap e639: 12 total, 12 up, 10 in 2024-05-15T23:33:23.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:23 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/759427723' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:33:23.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:23.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:23.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:23 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:23.136 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:23.138 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:33:23.138 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:33:23.138 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:33:23.138 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:33:23.138 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:33:23.138 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:33:23.138 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:33:23.138 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:33:23.138 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:33:23.138 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:33:23.138 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:33:23.138 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:33:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:23 smithi090 ceph-mon[44434]: osdmap e639: 12 total, 12 up, 10 in 2024-05-15T23:33:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:23 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/759427723' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:33:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:23.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:23.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:23 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:23 smithi088 ceph-mon[44699]: osdmap e639: 12 total, 12 up, 10 in 2024-05-15T23:33:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:23 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/759427723' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:33:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:23.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:23 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:23.812 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:23.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:23.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:23.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:23.989 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:24.158 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:24.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:24.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:24.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:33:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:24.337 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:33:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:24.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:33:24.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:24 smithi084 ceph-mon[39784]: pgmap v1701: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 612 MiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 701 op/s; 822/19558 objects misplaced (4.203%); 2.2 MiB/s, 34 objects/s recovering 2024-05-15T23:33:24.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:33:24.483 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:24.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:24 smithi090 ceph-mon[44434]: pgmap v1701: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 612 MiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 701 op/s; 822/19558 objects misplaced (4.203%); 2.2 MiB/s, 34 objects/s recovering 2024-05-15T23:33:24.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:24 smithi088 ceph-mon[44699]: pgmap v1701: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 612 MiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 701 op/s; 822/19558 objects misplaced (4.203%); 2.2 MiB/s, 34 objects/s recovering 2024-05-15T23:33:25.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:25 smithi084 ceph-mon[39784]: pgmap v1702: 17 pgs: 17 active+clean; 528 MiB data, 6.1 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 709 op/s; 9.2 MiB/s, 147 objects/s recovering 2024-05-15T23:33:25.627 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:33:25.627 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:33:25.635 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:33:25.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:25.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:25.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:33:25.798 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:25 smithi090 ceph-mon[44434]: pgmap v1702: 17 pgs: 17 active+clean; 528 MiB data, 6.1 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 709 op/s; 9.2 MiB/s, 147 objects/s recovering 2024-05-15T23:33:25.800 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:25.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:25 smithi088 ceph-mon[44699]: pgmap v1702: 17 pgs: 17 active+clean; 528 MiB data, 6.1 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 709 op/s; 9.2 MiB/s, 147 objects/s recovering 2024-05-15T23:33:25.827 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:33:25.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:25.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:33:25.935 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:33:25.961 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:25.980 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:33:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:25.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:33:26.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:26.140 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:26.228 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:26.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:26.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:26.335 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:33:26.363 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:26.383 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:33:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:26.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:33:26.488 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:26.515 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:26.534 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:26.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:33:26.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:33:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:26.722 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:33:26.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:26.725 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:33:26.826 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:26.830 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:26.852 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:26.872 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:26.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:26.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:33:26.980 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:27.007 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:27.027 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:27.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:27.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:33:27.080 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:33:27.094 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:33:27.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:33:27.169 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:27.200 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:33:27.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:27.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:33:27.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:33:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:27.387 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:33:27.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:27.400 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:33:27.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:33:27.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:33:27.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:27 smithi084 ceph-mon[39784]: osdmap e640: 12 total, 12 up, 10 in 2024-05-15T23:33:27.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:33:27.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:33:27.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:27 smithi084 ceph-mon[39784]: from='client.26908 v1:172.21.15.84:0/3555495906' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:33:27.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:27 smithi084 ceph-mon[39784]: pgmap v1704: 17 pgs: 17 active+clean; 365 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 987 op/s; 9.0 MiB/s, 143 objects/s recovering 2024-05-15T23:33:27.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:27 smithi084 ceph-mon[39784]: osdmap e641: 12 total, 12 up, 10 in 2024-05-15T23:33:27.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:33:27.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T23:33:27.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:33:27.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:33:27.535 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:27 smithi090 ceph-mon[44434]: osdmap e640: 12 total, 12 up, 10 in 2024-05-15T23:33:27.536 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:33:27.536 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:33:27.536 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:27 smithi090 ceph-mon[44434]: from='client.26908 v1:172.21.15.84:0/3555495906' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:33:27.536 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:27 smithi090 ceph-mon[44434]: pgmap v1704: 17 pgs: 17 active+clean; 365 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 987 op/s; 9.0 MiB/s, 143 objects/s recovering 2024-05-15T23:33:27.536 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:27 smithi090 ceph-mon[44434]: osdmap e641: 12 total, 12 up, 10 in 2024-05-15T23:33:27.538 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:27.565 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:33:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:27.568 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:33:27.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:33:27.712 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:27.754 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:33:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:27.761 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:33:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:33:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:33:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:27 smithi088 ceph-mon[44699]: osdmap e640: 12 total, 12 up, 10 in 2024-05-15T23:33:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:33:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:33:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:27 smithi088 ceph-mon[44699]: from='client.26908 v1:172.21.15.84:0/3555495906' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:33:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:27 smithi088 ceph-mon[44699]: pgmap v1704: 17 pgs: 17 active+clean; 365 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 987 op/s; 9.0 MiB/s, 143 objects/s recovering 2024-05-15T23:33:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:27 smithi088 ceph-mon[44699]: osdmap e641: 12 total, 12 up, 10 in 2024-05-15T23:33:27.862 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:33:27.911 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:28.003 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:33:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:28.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:28.108 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:28.147 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:28.181 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:28.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:28.306 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:28.333 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:28.354 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:28.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:28.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:28.458 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:33:28.509 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:28.597 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:33:28.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:28.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:33:28.629 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1703,"stamp":"2024-05-15T23:33:26.348044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":489554488,"num_objects":7464,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7464,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":284999,"num_write_kb":9356903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16677,"num_bytes_recovered":1094071904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160916,"ondisk_log_size":160916,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":937451520,"kb_used":6293688,"kb_used_data":1050632,"kb_used_omap":41,"kb_used_meta":5242838,"kb_avail":931157832,"statfs":{"total":959950356480,"available":953505619968,"internally_reserved":0,"allocated":1075847168,"data_stored":1070225514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42533,"internal_metadata":5368666587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,0,0,0,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-292487168,"num_objects":-4463,"num_object_clones":0,"num_object_copies":-8926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-970,"num_objects_unfound":0,"num_objects_dirty":-4463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4463,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":886,"num_bytes_recovered":58064896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.275982"},"pg_stats":[{"pgid":"7.f","version":"639'17752","reported_seq":19656,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-05-15T23:33:23.455228+0000","last_change":"2024-05-15T23:33:20.142232+0000","last_active":"2024-05-15T23:33:23.455228+0000","last_peered":"2024-05-15T23:33:23.455228+0000","last_clean":"2024-05-15T23:33:23.455228+0000","last_became_active":"2024-05-15T23:33:20.076302+0000","last_became_peered":"2024-05-15T23:33:20.076302+0000","last_unstale":"2024-05-15T23:33:23.455228+0000","last_undegraded":"2024-05-15T23:33:23.455228+0000","last_fullsized":"2024-05-15T23:33:23.455228+0000","mapping_epoch":636,"log_start":"577'7700","ondisk_log_start":"577'7700","created":549,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'16624","last_scrub_stamp":"2024-05-15T23:33:00.448056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:33:00.448056+0000","objects_scrubbed":1595,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:34:28.450626+0000","scrub_duration":362,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30015488,"num_objects":458,"num_object_clones":0,"num_object_copies":916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17752,"num_write_kb":582720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":799,"num_bytes_recovered":52297728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"639'17598","reported_seq":17931,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-05-15T23:33:24.570631+0000","last_change":"2024-05-15T23:32:09.972982+0000","last_active":"2024-05-15T23:33:24.570631+0000","last_peered":"2024-05-15T23:33:24.570631+0000","last_clean":"2024-05-15T23:33:24.570631+0000","last_became_active":"2024-05-15T23:28:51.519387+0000","last_became_peered":"2024-05-15T23:28:51.519387+0000","last_unstale":"2024-05-15T23:33:24.570631+0000","last_undegraded":"2024-05-15T23:33:24.570631+0000","last_fullsized":"2024-05-15T23:33:24.570631+0000","mapping_epoch":549,"log_start":"576'7500","ondisk_log_start":"576'7500","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'13643","last_scrub_stamp":"2024-05-15T23:32:09.972849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:09.972849+0000","objects_scrubbed":4379,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:33:26.177588+0000","scrub_duration":992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083328,"num_objects":398,"num_object_clones":0,"num_object_copies":796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17598,"num_write_kb":575872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"638'20840","reported_seq":21214,"reported_epoch":638,"state":"active+clean","last_fresh":"2024-05-15T23:33:21.873368+0000","last_change":"2024-05-15T23:32:13.361819+0000","last_active":"2024-05-15T23:33:21.873368+0000","last_peered":"2024-05-15T23:33:21.873368+0000","last_clean":"2024-05-15T23:33:21.873368+0000","last_became_active":"2024-05-15T23:32:13.327043+0000","last_became_peered":"2024-05-15T23:32:13.327043+0000","last_unstale":"2024-05-15T23:33:21.873368+0000","last_undegraded":"2024-05-15T23:33:21.873368+0000","last_fullsized":"2024-05-15T23:33:21.873368+0000","mapping_epoch":610,"log_start":"593'10801","ondisk_log_start":"593'10801","created":549,"last_epoch_clean":611,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'16611","last_scrub_stamp":"2024-05-15T23:32:04.371452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:04.371452+0000","objects_scrubbed":4852,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:33:27.916120+0000","scrub_duration":1003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39518232,"num_objects":604,"num_object_clones":0,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17904,"num_write_kb":592193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.c","version":"638'17652","reported_seq":17949,"reported_epoch":638,"state":"active+clean","last_fresh":"2024-05-15T23:33:21.849560+0000","last_change":"2024-05-15T23:33:06.235551+0000","last_active":"2024-05-15T23:33:21.849560+0000","last_peered":"2024-05-15T23:33:21.849560+0000","last_clean":"2024-05-15T23:33:21.849560+0000","last_became_active":"2024-05-15T23:28:51.305728+0000","last_became_peered":"2024-05-15T23:28:51.305728+0000","last_unstale":"2024-05-15T23:33:21.849560+0000","last_undegraded":"2024-05-15T23:33:21.849560+0000","last_fullsized":"2024-05-15T23:33:21.849560+0000","mapping_epoch":549,"log_start":"576'7600","ondisk_log_start":"576'7600","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"629'16925","last_scrub_stamp":"2024-05-15T23:33:06.235373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:33:06.235373+0000","objects_scrubbed":1335,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:34:34.201318+0000","scrub_duration":458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38797312,"num_objects":592,"num_object_clones":0,"num_object_copies":1184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17652,"num_write_kb":583808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"638'17312","reported_seq":21058,"reported_epoch":638,"state":"active+clean","last_fresh":"2024-05-15T23:33:21.880358+0000","last_change":"2024-05-15T23:33:03.303331+0000","last_active":"2024-05-15T23:33:21.880358+0000","last_peered":"2024-05-15T23:33:21.880358+0000","last_clean":"2024-05-15T23:33:21.880358+0000","last_became_active":"2024-05-15T23:33:03.268779+0000","last_became_peered":"2024-05-15T23:33:03.268779+0000","last_unstale":"2024-05-15T23:33:21.880358+0000","last_undegraded":"2024-05-15T23:33:21.880358+0000","last_fullsized":"2024-05-15T23:33:21.880358+0000","mapping_epoch":628,"log_start":"576'7300","ondisk_log_start":"576'7300","created":549,"last_epoch_clean":629,"parent":"0.0","parent_split_bits":0,"last_scrub":"611'13861","last_scrub_stamp":"2024-05-15T23:32:15.224350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:15.224350+0000","objects_scrubbed":4042,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:33:25.616637+0000","scrub_duration":790,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37355520,"num_objects":570,"num_object_clones":0,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17312,"num_write_kb":572224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1708,"num_bytes_recovered":111935488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"639'18150","reported_seq":18441,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-05-15T23:33:26.069183+0000","last_change":"2024-05-15T23:33:01.411982+0000","last_active":"2024-05-15T23:33:26.069183+0000","last_peered":"2024-05-15T23:33:26.069183+0000","last_clean":"2024-05-15T23:33:26.069183+0000","last_became_active":"2024-05-15T23:28:51.308216+0000","last_became_peered":"2024-05-15T23:28:51.308216+0000","last_unstale":"2024-05-15T23:33:26.069183+0000","last_undegraded":"2024-05-15T23:33:26.069183+0000","last_fullsized":"2024-05-15T23:33:26.069183+0000","mapping_epoch":549,"log_start":"580'8100","ondisk_log_start":"580'8100","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'16867","last_scrub_stamp":"2024-05-15T23:33:01.411089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:33:01.411089+0000","objects_scrubbed":1588,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:34:03.917265+0000","scrub_duration":297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19267584,"num_objects":294,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18150,"num_write_kb":590208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"639'17774","reported_seq":19815,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-05-15T23:33:22.422008+0000","last_change":"2024-05-15T23:33:20.118948+0000","last_active":"2024-05-15T23:33:22.422008+0000","last_peered":"2024-05-15T23:33:22.422008+0000","last_clean":"2024-05-15T23:33:22.422008+0000","last_became_active":"2024-05-15T23:33:20.076503+0000","last_became_peered":"2024-05-15T23:33:20.076503+0000","last_unstale":"2024-05-15T23:33:22.422008+0000","last_undegraded":"2024-05-15T23:33:22.422008+0000","last_fullsized":"2024-05-15T23:33:22.422008+0000","mapping_epoch":636,"log_start":"576'7700","ondisk_log_start":"576'7700","created":549,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'14001","last_scrub_stamp":"2024-05-15T23:32:09.654457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:09.654457+0000","objects_scrubbed":4350,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35782656,"num_objects":546,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17774,"num_write_kb":586240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":848,"num_bytes_recovered":55574528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"639'21576","reported_seq":22073,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-05-15T23:33:24.577136+0000","last_change":"2024-05-15T23:33:10.423414+0000","last_active":"2024-05-15T23:33:24.577136+0000","last_peered":"2024-05-15T23:33:24.577136+0000","last_clean":"2024-05-15T23:33:24.577136+0000","last_became_active":"2024-05-15T23:28:51.304930+0000","last_became_peered":"2024-05-15T23:28:51.304930+0000","last_unstale":"2024-05-15T23:33:24.577136+0000","last_undegraded":"2024-05-15T23:33:24.577136+0000","last_fullsized":"2024-05-15T23:33:24.577136+0000","mapping_epoch":549,"log_start":"596'11500","ondisk_log_start":"596'11500","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"632'20953","last_scrub_stamp":"2024-05-15T23:33:10.423249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:33:10.423249+0000","objects_scrubbed":1029,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:34:32.851043+0000","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26279936,"num_objects":401,"num_object_clones":0,"num_object_copies":802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18165,"num_write_kb":594112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"639'17617","reported_seq":19435,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-05-15T23:33:23.477660+0000","last_change":"2024-05-15T23:33:21.143328+0000","last_active":"2024-05-15T23:33:23.477660+0000","last_peered":"2024-05-15T23:33:23.477660+0000","last_clean":"2024-05-15T23:33:23.477660+0000","last_became_active":"2024-05-15T23:33:21.077514+0000","last_became_peered":"2024-05-15T23:33:21.077514+0000","last_unstale":"2024-05-15T23:33:23.477660+0000","last_undegraded":"2024-05-15T23:33:23.477660+0000","last_fullsized":"2024-05-15T23:33:23.477660+0000","mapping_epoch":637,"log_start":"576'7600","ondisk_log_start":"576'7600","created":549,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"629'16681","last_scrub_stamp":"2024-05-15T23:33:04.078201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:33:04.078201+0000","objects_scrubbed":1429,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:34:15.800168+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31916032,"num_objects":487,"num_object_clones":0,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17617,"num_write_kb":579328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":798,"num_bytes_recovered":52297728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"639'17761","reported_seq":18093,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-05-15T23:33:22.407686+0000","last_change":"2024-05-15T23:32:05.298952+0000","last_active":"2024-05-15T23:33:22.407686+0000","last_peered":"2024-05-15T23:33:22.407686+0000","last_clean":"2024-05-15T23:33:22.407686+0000","last_became_active":"2024-05-15T23:28:51.303581+0000","last_became_peered":"2024-05-15T23:28:51.303581+0000","last_unstale":"2024-05-15T23:33:22.407686+0000","last_undegraded":"2024-05-15T23:33:22.407686+0000","last_fullsized":"2024-05-15T23:33:22.407686+0000","mapping_epoch":549,"log_start":"577'7700","ondisk_log_start":"577'7700","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"608'13647","last_scrub_stamp":"2024-05-15T23:32:05.298729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:05.298729+0000","objects_scrubbed":4699,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:33:25.472348+0000","scrub_duration":802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37027840,"num_objects":565,"num_object_clones":0,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17761,"num_write_kb":586432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"639'18276","reported_seq":28304,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-05-15T23:33:26.064798+0000","last_change":"2024-05-15T23:32:59.414038+0000","last_active":"2024-05-15T23:33:26.064798+0000","last_peered":"2024-05-15T23:33:26.064798+0000","last_clean":"2024-05-15T23:33:26.064798+0000","last_became_active":"2024-05-15T23:32:13.327837+0000","last_became_peered":"2024-05-15T23:32:13.327837+0000","last_unstale":"2024-05-15T23:33:26.064798+0000","last_undegraded":"2024-05-15T23:33:26.064798+0000","last_fullsized":"2024-05-15T23:33:26.064798+0000","mapping_epoch":610,"log_start":"580'8200","ondisk_log_start":"580'8200","created":549,"last_epoch_clean":611,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'16964","last_scrub_stamp":"2024-05-15T23:32:59.413745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:59.413745+0000","objects_scrubbed":1599,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:34:09.828676+0000","scrub_duration":335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18350080,"num_objects":280,"num_object_clones":0,"num_object_copies":560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18276,"num_write_kb":593792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4862,"num_bytes_recovered":318636032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.3","version":"638'17861","reported_seq":18153,"reported_epoch":638,"state":"active+clean","last_fresh":"2024-05-15T23:33:21.899369+0000","last_change":"2024-05-15T23:33:14.966589+0000","last_active":"2024-05-15T23:33:21.899369+0000","last_peered":"2024-05-15T23:33:21.899369+0000","last_clean":"2024-05-15T23:33:21.899369+0000","last_became_active":"2024-05-15T23:28:51.305691+0000","last_became_peered":"2024-05-15T23:28:51.305691+0000","last_unstale":"2024-05-15T23:33:21.899369+0000","last_undegraded":"2024-05-15T23:33:21.899369+0000","last_fullsized":"2024-05-15T23:33:21.899369+0000","mapping_epoch":549,"log_start":"577'7800","ondisk_log_start":"577'7800","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"634'17542","last_scrub_stamp":"2024-05-15T23:33:14.966420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:33:14.966420+0000","objects_scrubbed":882,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:34:42.519621+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36503552,"num_objects":557,"num_object_clones":0,"num_object_copies":1114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17861,"num_write_kb":589376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"638'17803","reported_seq":18016,"reported_epoch":638,"state":"active+clean","last_fresh":"2024-05-15T23:33:21.873071+0000","last_change":"2024-05-15T23:33:01.218413+0000","last_active":"2024-05-15T23:33:21.873071+0000","last_peered":"2024-05-15T23:33:21.873071+0000","last_clean":"2024-05-15T23:33:21.873071+0000","last_became_active":"2024-05-15T23:28:51.519343+0000","last_became_peered":"2024-05-15T23:28:51.519343+0000","last_unstale":"2024-05-15T23:33:21.873071+0000","last_undegraded":"2024-05-15T23:33:21.873071+0000","last_fullsized":"2024-05-15T23:33:21.873071+0000","mapping_epoch":549,"log_start":"577'7800","ondisk_log_start":"577'7800","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'16821","last_scrub_stamp":"2024-05-15T23:33:01.218122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:33:01.218122+0000","objects_scrubbed":1546,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:34:02.707301+0000","scrub_duration":412,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36372480,"num_objects":555,"num_object_clones":0,"num_object_copies":1110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17803,"num_write_kb":587456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"639'17869","reported_seq":33403,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-05-15T23:33:26.124056+0000","last_change":"2024-05-15T23:32:59.911318+0000","last_active":"2024-05-15T23:33:26.124056+0000","last_peered":"2024-05-15T23:33:26.124056+0000","last_clean":"2024-05-15T23:33:26.124056+0000","last_became_active":"2024-05-15T23:30:10.487853+0000","last_became_peered":"2024-05-15T23:30:10.487853+0000","last_unstale":"2024-05-15T23:33:26.124056+0000","last_undegraded":"2024-05-15T23:33:26.124056+0000","last_fullsized":"2024-05-15T23:33:26.124056+0000","mapping_epoch":578,"log_start":"580'7800","ondisk_log_start":"580'7800","created":549,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'16489","last_scrub_stamp":"2024-05-15T23:32:59.911091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:59.911091+0000","objects_scrubbed":1688,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:34:03.773653+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19464192,"num_objects":297,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17869,"num_write_kb":581312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7656,"num_bytes_recovered":501743616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"639'17843","reported_seq":18225,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-05-15T23:33:23.060822+0000","last_change":"2024-05-15T23:32:56.208305+0000","last_active":"2024-05-15T23:33:23.060822+0000","last_peered":"2024-05-15T23:33:23.060822+0000","last_clean":"2024-05-15T23:33:23.060822+0000","last_became_active":"2024-05-15T23:28:51.303621+0000","last_became_peered":"2024-05-15T23:28:51.303621+0000","last_unstale":"2024-05-15T23:33:23.060822+0000","last_undegraded":"2024-05-15T23:33:23.060822+0000","last_fullsized":"2024-05-15T23:33:23.060822+0000","mapping_epoch":549,"log_start":"579'7800","ondisk_log_start":"579'7800","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'16471","last_scrub_stamp":"2024-05-15T23:32:56.208146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:56.208146+0000","objects_scrubbed":1882,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:34:12.645562+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32964608,"num_objects":503,"num_object_clones":0,"num_object_copies":1006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17843,"num_write_kb":587072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"639'17539","reported_seq":17933,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-05-15T23:33:24.803875+0000","last_change":"2024-05-15T23:32:26.208670+0000","last_active":"2024-05-15T23:33:24.803875+0000","last_peered":"2024-05-15T23:33:24.803875+0000","last_clean":"2024-05-15T23:33:24.803875+0000","last_became_active":"2024-05-15T23:28:51.304317+0000","last_became_peered":"2024-05-15T23:28:51.304317+0000","last_unstale":"2024-05-15T23:33:24.803875+0000","last_undegraded":"2024-05-15T23:33:24.803875+0000","last_fullsized":"2024-05-15T23:33:24.803875+0000","mapping_epoch":549,"log_start":"576'7500","ondisk_log_start":"576'7500","created":549,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"612'14471","last_scrub_stamp":"2024-05-15T23:32:26.208565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:28:50.293593+0000","last_clean_scrub_stamp":"2024-05-15T23:32:26.208565+0000","objects_scrubbed":3443,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:33:35.781604+0000","scrub_duration":576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23265280,"num_objects":355,"num_object_clones":0,"num_object_copies":710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17539,"num_write_kb":572608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1369,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-05-15T23:33:22.102933+0000","last_change":"2024-05-15T23:32:08.827746+0000","last_active":"2024-05-15T23:33:22.102933+0000","last_peered":"2024-05-15T23:33:22.102933+0000","last_clean":"2024-05-15T23:33:22.102933+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:33:22.102933+0000","last_undegraded":"2024-05-15T23:33:22.102933+0000","last_fullsized":"2024-05-15T23:33:22.102933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:32:08.827664+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:32:08.827664+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:33:36.958374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":488964120,"num_objects":7462,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7462,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":284876,"num_write_kb":9354753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16671,"num_bytes_recovered":1092485120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1519919104,"data_stored":1519910960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351332,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":803492,"kb_used_data":108700,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":92941660,"statfs":{"total":95995035648,"available":95172259840,"internally_reserved":0,"allocated":111308800,"data_stored":110743406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":711457220},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580265,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":532508,"kb_used_data":95956,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93212644,"statfs":{"total":95995035648,"available":95449747456,"internally_reserved":0,"allocated":98258944,"data_stored":97697122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":447019456},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776497,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":747036,"kb_used_data":116052,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92998116,"statfs":{"total":95995035648,"available":95230070784,"internally_reserved":0,"allocated":118837248,"data_stored":118275426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":646052288},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":520,"seq":2233382994000,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130012,"kb_used_data":93076,"kb_used_omap":24,"kb_used_meta":36903,"kb_avail":93615140,"statfs":{"total":95995035648,"available":95861903360,"internally_reserved":0,"allocated":95309824,"data_stored":94748002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25357,"internal_metadata":37788915},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,0,0,0,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267065,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":675108,"kb_used_data":76124,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93070044,"statfs":{"total":95995035648,"available":95303725056,"internally_reserved":0,"allocated":77950976,"data_stored":77385602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":613349834},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575257,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":897116,"kb_used_data":153740,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92848036,"statfs":{"total":95995035648,"available":95076388864,"internally_reserved":0,"allocated":157429760,"data_stored":156869783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":761199046},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016666,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483355,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":253788,"kb_used_data":109012,"kb_used_omap":4,"kb_used_meta":144763,"kb_avail":93491364,"statfs":{"total":95995035648,"available":95735156736,"internally_reserved":0,"allocated":111628288,"data_stored":111066466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":148238019},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150096,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1088404,"kb_used_data":263116,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92656748,"statfs":{"total":95995035648,"available":94880509952,"internally_reserved":0,"allocated":269430784,"data_stored":268870807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":845085127},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986568,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":519592,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":93225560,"statfs":{"total":95995035648,"available":95462973440,"internally_reserved":0,"allocated":1134592,"data_stored":572770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790340,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":646632,"kb_used_data":33748,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93098520,"statfs":{"total":95995035648,"available":95332884480,"internally_reserved":0,"allocated":34557952,"data_stored":33996130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152440832,"data_stored":152436760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104923136,"data_stored":104923136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":263917568,"data_stored":263913496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":501743616,"data_stored":501743616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32964608,"data_stored":32964608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":74973184,"data_stored":74973184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":94044160,"data_stored":94044160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":96731136,"data_stored":96731136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":88735744,"data_stored":88735744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":109445120,"data_stored":109445120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:33:28.630 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool set unique_pool_5 pgp_num 16 2024-05-15T23:33:28.706 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:28.733 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:28.754 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:28.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:28.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:28.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:33:28.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:33:28.899 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:28.929 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:33:28.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:28.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:33:29.036 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:33:29.086 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:29.128 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:33:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:29.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:33:29.237 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:29.264 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:29.283 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:29.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:29.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:33:29.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:29 smithi084 ceph-mon[39784]: pgmap v1706: 17 pgs: 17 active+clean; 261 MiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 868 op/s; 7.7 MiB/s, 123 objects/s recovering 2024-05-15T23:33:29.386 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:33:29.428 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:29.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:29 smithi088 ceph-mon[44699]: pgmap v1706: 17 pgs: 17 active+clean; 261 MiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 868 op/s; 7.7 MiB/s, 123 objects/s recovering 2024-05-15T23:33:29.689 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:33:29.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:29.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:33:29.728 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:29.771 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:29.795 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:29.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:29 smithi090 ceph-mon[44434]: pgmap v1706: 17 pgs: 17 active+clean; 261 MiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 868 op/s; 7.7 MiB/s, 123 objects/s recovering 2024-05-15T23:33:29.844 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:30.058 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:30.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:30.176 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:30.203 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:30.222 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:30.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:30.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:33:30.326 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:33:30.375 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:30.501 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:33:30.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:30.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:30.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:33:30.664 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:30.687 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:33:30.687 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Removed 146168 objects 2024-05-15T23:33:30.687 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Clean up completed and total clean up time :175.97 2024-05-15T23:33:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:30.689 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:33:30.790 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:30.819 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:30.917 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:30.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:31.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:33:31.048 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:31.067 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:33:31.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:31.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:33:31.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:33:31.198 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:31.217 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:33:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:31.220 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:33:31.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:33:31.347 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:31.366 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:33:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:31.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:33:31.470 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:31.499 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:31 smithi084 ceph-mon[39784]: pgmap v1707: 17 pgs: 17 active+clean; 147 MiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 928 op/s; 5.8 MiB/s, 92 objects/s recovering 2024-05-15T23:33:31.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:31 smithi090 ceph-mon[44434]: pgmap v1707: 17 pgs: 17 active+clean; 147 MiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 928 op/s; 5.8 MiB/s, 92 objects/s recovering 2024-05-15T23:33:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:31 smithi088 ceph-mon[44699]: pgmap v1707: 17 pgs: 17 active+clean; 147 MiB data, 5.4 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 928 op/s; 5.8 MiB/s, 92 objects/s recovering 2024-05-15T23:33:31.949 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:31.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:32.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:32.091 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:33.130 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:33.143 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:33.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:33.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:33.248 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:33.281 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:33.302 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:33.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:33.392 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:33:33.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:33:33.441 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:33.463 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:33:33.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:33.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:33:33.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:33 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/946334408' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:33:33.567 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:33:33.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:33 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/946334408' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:33:33.600 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:33.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:33 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/946334408' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:33:33.622 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:33:33.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:33.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:33:33.726 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:33:33.755 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:33.953 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:33:33.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:33.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:33:34.058 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:33:34.090 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:34.112 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:33:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:34.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:33:34.216 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:34.247 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:34.268 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:34.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:34.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:34 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:33:34.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:34 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/946334408' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:33:34.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:34 smithi084 ceph-mon[39784]: osdmap e642: 12 total, 12 up, 10 in 2024-05-15T23:33:34.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:34 smithi084 ceph-mon[39784]: pgmap v1709: 17 pgs: 17 active+clean; 31 MiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T23:33:34.372 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:33:34.400 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:34.524 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:33:34.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:34.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:34.551 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:34 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:33:34.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:34 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/946334408' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:33:34.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:34 smithi090 ceph-mon[44434]: osdmap e642: 12 total, 12 up, 10 in 2024-05-15T23:33:34.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:34 smithi090 ceph-mon[44434]: pgmap v1709: 17 pgs: 17 active+clean; 31 MiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T23:33:34.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:34 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:33:34.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:34 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/946334408' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:33:34.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:34 smithi088 ceph-mon[44699]: osdmap e642: 12 total, 12 up, 10 in 2024-05-15T23:33:34.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:34 smithi088 ceph-mon[44699]: pgmap v1709: 17 pgs: 17 active+clean; 31 MiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-05-15T23:33:34.628 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:34.654 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:34.675 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:34.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:34.779 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:34.805 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:34.927 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:34.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:34.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:35.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:33:35.058 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:35.077 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:33:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:35.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:33:35.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:33:35.207 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:35.225 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:33:35.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:35.228 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:33:35.328 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:35.355 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:35.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:35 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:33:35.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:35 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:33:35.554 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:35.556 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:35 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:33:35.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:35.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:33:35.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:35.689 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:36.437 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:36.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:36.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:36.549 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:33:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:36 smithi090 ceph-mon[44434]: pgmap v1710: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 727 op/s 2024-05-15T23:33:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:36 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:33:36.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:36 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:33:36.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:36 smithi088 ceph-mon[44699]: pgmap v1710: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 727 op/s 2024-05-15T23:33:36.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:36 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:33:36.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:36 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:33:36.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:36 smithi084 ceph-mon[39784]: pgmap v1710: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 727 op/s 2024-05-15T23:33:36.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:36 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:33:36.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:36 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:33:36.574 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:36.628 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:36.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:36.683 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:33:36.684 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:33:36.732 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:33:36.760 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:37.148 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:33:37.148 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:33:37.148 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:33:37.149 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:33:37.149 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:33:37.149 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:33:37.149 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:33:37.149 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:33:37.149 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:33:37.149 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:33:37.149 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:33:37.169 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:33:37.171 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:33:37.171 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:33:37.171 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:33:37.171 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:33:37.171 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:33:37.171 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:33:37.171 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:33:37.171 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:33:37.172 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:33:37.172 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:33:37.172 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:33:37.172 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:33:37.509 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:33:37.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:37 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:33:37.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:37 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:33:37.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:37 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:33:37.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:37 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1348074129' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:33:37.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:37.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:37.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:33:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:37 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:33:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:37 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:33:37.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:37 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:33:37.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:37 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1348074129' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:33:37.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:37.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:37.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:37 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:33:37.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:37 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:33:37.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:37 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:33:37.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:37 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1348074129' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:33:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:37.613 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:37.646 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:37.668 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:37.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:37.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:37.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:33:37.804 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:37.826 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:33:37.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:37.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:33:37.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:33:37.960 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:37.980 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:33:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:37.984 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:33:38.085 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:38.112 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:38.235 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:38.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:38.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:38.286 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 7 pgp_num to 16 2024-05-15T23:33:38.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:33:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:38 smithi084 ceph-mon[39784]: pgmap v1711: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 583 op/s 2024-05-15T23:33:38.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:38 smithi084 ceph-mon[39784]: osdmap e643: 12 total, 12 up, 10 in 2024-05-15T23:33:38.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:38 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3092821760' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-05-15T23:33:38.370 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:38.389 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:33:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:38.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:33:38.494 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:33:38.521 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:38 smithi090 ceph-mon[44434]: pgmap v1711: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 583 op/s 2024-05-15T23:33:38.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:38 smithi090 ceph-mon[44434]: osdmap e643: 12 total, 12 up, 10 in 2024-05-15T23:33:38.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:38 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3092821760' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-05-15T23:33:38.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:38.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:38 smithi088 ceph-mon[44699]: pgmap v1711: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 583 op/s 2024-05-15T23:33:38.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:38 smithi088 ceph-mon[44699]: osdmap e643: 12 total, 12 up, 10 in 2024-05-15T23:33:38.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:38 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3092821760' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-05-15T23:33:38.643 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:33:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:38.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:38.747 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:33:38.773 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:39.079 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-05-15T23:33:39.079 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-05-15T23:33:39.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3092821760' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-05-15T23:33:39.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:39 smithi084 ceph-mon[39784]: osdmap e644: 12 total, 12 up, 10 in 2024-05-15T23:33:39.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:39 smithi084 ceph-mon[39784]: pgmap v1714: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 80 op/s 2024-05-15T23:33:39.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T23:33:39.497 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:39.521 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:33:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:39.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:33:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3092821760' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-05-15T23:33:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:39 smithi090 ceph-mon[44434]: osdmap e644: 12 total, 12 up, 10 in 2024-05-15T23:33:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:39 smithi090 ceph-mon[44434]: pgmap v1714: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 80 op/s 2024-05-15T23:33:39.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3092821760' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-05-15T23:33:39.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:39 smithi088 ceph-mon[44699]: osdmap e644: 12 total, 12 up, 10 in 2024-05-15T23:33:39.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:39 smithi088 ceph-mon[44699]: pgmap v1714: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 80 op/s 2024-05-15T23:33:39.625 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:39.658 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:39.679 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:39.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:39.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:39.784 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:39.814 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:39.835 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:39.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:39.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:39.939 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:33:39.969 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:39.990 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:33:39.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:39.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:33:40.095 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:40.123 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:40.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:33:40.314 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:40.321 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:40.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:40.425 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:33:40.458 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:33:40.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:40.482 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:33:40.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:33:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:40.633 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:33:40.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:40.636 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:33:40.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:33:40.764 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:40.783 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:33:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:40.786 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:33:40.800 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:40.887 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:40.914 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:41.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=56,logm=4} crc {auth=1520374581,config=3313370793,health=179246142,kv=935763156,logm=3678511241}) 2024-05-15T23:33:41.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2426797887}) 2024-05-15T23:33:41.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870340251}) 2024-05-15T23:33:41.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1725249423}) 2024-05-15T23:33:41.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2146314195}) 2024-05-15T23:33:41.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3946242158}) 2024-05-15T23:33:41.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3520116053}) 2024-05-15T23:33:41.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1901508842}) 2024-05-15T23:33:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3187387302}) 2024-05-15T23:33:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=482389370}) 2024-05-15T23:33:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3428545426}) 2024-05-15T23:33:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764703124}) 2024-05-15T23:33:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=167838868}) 2024-05-15T23:33:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4288689889}) 2024-05-15T23:33:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2008805134}) 2024-05-15T23:33:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2276994830}) 2024-05-15T23:33:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999945854}) 2024-05-15T23:33:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3592033875}) 2024-05-15T23:33:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500388241}) 2024-05-15T23:33:41.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2286287071}) 2024-05-15T23:33:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1420265724}) 2024-05-15T23:33:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1234123452}) 2024-05-15T23:33:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2734481965}) 2024-05-15T23:33:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=232451552}) 2024-05-15T23:33:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33520000}) 2024-05-15T23:33:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1160898569}) 2024-05-15T23:33:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1165785341}) 2024-05-15T23:33:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4071730508}) 2024-05-15T23:33:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=821000861}) 2024-05-15T23:33:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4051512674}) 2024-05-15T23:33:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4241406160}) 2024-05-15T23:33:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3282414937}) 2024-05-15T23:33:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2568297646}) 2024-05-15T23:33:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2268511346}) 2024-05-15T23:33:41.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3019891359}) 2024-05-15T23:33:41.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3347760512}) 2024-05-15T23:33:41.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=717820527}) 2024-05-15T23:33:41.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=879936507}) 2024-05-15T23:33:41.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2003611838}) 2024-05-15T23:33:41.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2676785330}) 2024-05-15T23:33:41.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=801666774}) 2024-05-15T23:33:41.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1226552155}) 2024-05-15T23:33:41.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2749491030}) 2024-05-15T23:33:41.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4125425445}) 2024-05-15T23:33:41.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3329170317}) 2024-05-15T23:33:41.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1390995134}) 2024-05-15T23:33:41.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4266929623}) 2024-05-15T23:33:41.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2064200565}) 2024-05-15T23:33:41.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1107729905}) 2024-05-15T23:33:41.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2838066740}) 2024-05-15T23:33:41.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2875529902}) 2024-05-15T23:33:41.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1617420938}) 2024-05-15T23:33:41.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1682524283}) 2024-05-15T23:33:41.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2104946961}) 2024-05-15T23:33:41.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3746820722}) 2024-05-15T23:33:41.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3280245022}) 2024-05-15T23:33:41.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2999688925}) 2024-05-15T23:33:41.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1017477765}) 2024-05-15T23:33:41.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2701276287}) 2024-05-15T23:33:41.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3491321759}) 2024-05-15T23:33:41.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2945364778}) 2024-05-15T23:33:41.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3116399613}) 2024-05-15T23:33:41.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3305337980}) 2024-05-15T23:33:41.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3115709481}) 2024-05-15T23:33:41.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1441484028}) 2024-05-15T23:33:41.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214199673}) 2024-05-15T23:33:41.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2243014154}) 2024-05-15T23:33:41.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2227066885}) 2024-05-15T23:33:41.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2226716817}) 2024-05-15T23:33:41.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=599045550}) 2024-05-15T23:33:41.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2693544098}) 2024-05-15T23:33:41.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2703122293}) 2024-05-15T23:33:41.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196708190}) 2024-05-15T23:33:41.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1672815751}) 2024-05-15T23:33:41.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2086911295}) 2024-05-15T23:33:41.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3123145767}) 2024-05-15T23:33:41.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3999257099}) 2024-05-15T23:33:41.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=82,mdsmap=3,mgr=15} crc {logm=24591630,mdsmap=2840952586,mgr=1006343046}) 2024-05-15T23:33:41.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:40 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=77} crc {mgr=4028747438,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2403392517,mon_config_key=3906756719}) 2024-05-15T23:33:41.062 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=56,logm=4} crc {auth=1520374581,config=3313370793,health=179246142,kv=935763156,logm=3678511241}) 2024-05-15T23:33:41.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2426797887}) 2024-05-15T23:33:41.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870340251}) 2024-05-15T23:33:41.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1725249423}) 2024-05-15T23:33:41.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2146314195}) 2024-05-15T23:33:41.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3946242158}) 2024-05-15T23:33:41.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3520116053}) 2024-05-15T23:33:41.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1901508842}) 2024-05-15T23:33:41.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3187387302}) 2024-05-15T23:33:41.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=482389370}) 2024-05-15T23:33:41.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3428545426}) 2024-05-15T23:33:41.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764703124}) 2024-05-15T23:33:41.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=167838868}) 2024-05-15T23:33:41.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4288689889}) 2024-05-15T23:33:41.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2008805134}) 2024-05-15T23:33:41.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2276994830}) 2024-05-15T23:33:41.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999945854}) 2024-05-15T23:33:41.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3592033875}) 2024-05-15T23:33:41.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500388241}) 2024-05-15T23:33:41.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2286287071}) 2024-05-15T23:33:41.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1420265724}) 2024-05-15T23:33:41.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1234123452}) 2024-05-15T23:33:41.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2734481965}) 2024-05-15T23:33:41.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=232451552}) 2024-05-15T23:33:41.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33520000}) 2024-05-15T23:33:41.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1160898569}) 2024-05-15T23:33:41.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1165785341}) 2024-05-15T23:33:41.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4071730508}) 2024-05-15T23:33:41.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=821000861}) 2024-05-15T23:33:41.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4051512674}) 2024-05-15T23:33:41.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4241406160}) 2024-05-15T23:33:41.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3282414937}) 2024-05-15T23:33:41.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2568297646}) 2024-05-15T23:33:41.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2268511346}) 2024-05-15T23:33:41.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3019891359}) 2024-05-15T23:33:41.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3347760512}) 2024-05-15T23:33:41.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=717820527}) 2024-05-15T23:33:41.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=879936507}) 2024-05-15T23:33:41.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2003611838}) 2024-05-15T23:33:41.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2676785330}) 2024-05-15T23:33:41.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=801666774}) 2024-05-15T23:33:41.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1226552155}) 2024-05-15T23:33:41.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2749491030}) 2024-05-15T23:33:41.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4125425445}) 2024-05-15T23:33:41.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3329170317}) 2024-05-15T23:33:41.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1390995134}) 2024-05-15T23:33:41.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4266929623}) 2024-05-15T23:33:41.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2064200565}) 2024-05-15T23:33:41.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1107729905}) 2024-05-15T23:33:41.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2838066740}) 2024-05-15T23:33:41.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2875529902}) 2024-05-15T23:33:41.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1617420938}) 2024-05-15T23:33:41.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1682524283}) 2024-05-15T23:33:41.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2104946961}) 2024-05-15T23:33:41.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3746820722}) 2024-05-15T23:33:41.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3280245022}) 2024-05-15T23:33:41.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2999688925}) 2024-05-15T23:33:41.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1017477765}) 2024-05-15T23:33:41.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2701276287}) 2024-05-15T23:33:41.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3491321759}) 2024-05-15T23:33:41.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2945364778}) 2024-05-15T23:33:41.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3116399613}) 2024-05-15T23:33:41.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3305337980}) 2024-05-15T23:33:41.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3115709481}) 2024-05-15T23:33:41.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1441484028}) 2024-05-15T23:33:41.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214199673}) 2024-05-15T23:33:41.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2243014154}) 2024-05-15T23:33:41.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2227066885}) 2024-05-15T23:33:41.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2226716817}) 2024-05-15T23:33:41.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=599045550}) 2024-05-15T23:33:41.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2693544098}) 2024-05-15T23:33:41.068 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:41.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2703122293}) 2024-05-15T23:33:41.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196708190}) 2024-05-15T23:33:41.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1672815751}) 2024-05-15T23:33:41.069 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2086911295}) 2024-05-15T23:33:41.069 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3123145767}) 2024-05-15T23:33:41.069 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3999257099}) 2024-05-15T23:33:41.069 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=82,mdsmap=3,mgr=15} crc {logm=24591630,mdsmap=2840952586,mgr=1006343046}) 2024-05-15T23:33:41.069 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:40 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=77} crc {mgr=4028747438,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2403392517,mon_config_key=3906756719}) 2024-05-15T23:33:41.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:41.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:41.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=56,logm=4} crc {auth=1520374581,config=3313370793,health=179246142,kv=935763156,logm=3678511241}) 2024-05-15T23:33:41.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2426797887}) 2024-05-15T23:33:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870340251}) 2024-05-15T23:33:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1725249423}) 2024-05-15T23:33:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2146314195}) 2024-05-15T23:33:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3946242158}) 2024-05-15T23:33:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3520116053}) 2024-05-15T23:33:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1901508842}) 2024-05-15T23:33:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3187387302}) 2024-05-15T23:33:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=482389370}) 2024-05-15T23:33:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3428545426}) 2024-05-15T23:33:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764703124}) 2024-05-15T23:33:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=167838868}) 2024-05-15T23:33:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4288689889}) 2024-05-15T23:33:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2008805134}) 2024-05-15T23:33:41.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2276994830}) 2024-05-15T23:33:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999945854}) 2024-05-15T23:33:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3592033875}) 2024-05-15T23:33:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500388241}) 2024-05-15T23:33:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2286287071}) 2024-05-15T23:33:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1420265724}) 2024-05-15T23:33:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1234123452}) 2024-05-15T23:33:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2734481965}) 2024-05-15T23:33:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=232451552}) 2024-05-15T23:33:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33520000}) 2024-05-15T23:33:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1160898569}) 2024-05-15T23:33:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1165785341}) 2024-05-15T23:33:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4071730508}) 2024-05-15T23:33:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=821000861}) 2024-05-15T23:33:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4051512674}) 2024-05-15T23:33:41.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4241406160}) 2024-05-15T23:33:41.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3282414937}) 2024-05-15T23:33:41.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2568297646}) 2024-05-15T23:33:41.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2268511346}) 2024-05-15T23:33:41.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3019891359}) 2024-05-15T23:33:41.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3347760512}) 2024-05-15T23:33:41.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=717820527}) 2024-05-15T23:33:41.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=879936507}) 2024-05-15T23:33:41.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2003611838}) 2024-05-15T23:33:41.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2676785330}) 2024-05-15T23:33:41.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=801666774}) 2024-05-15T23:33:41.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1226552155}) 2024-05-15T23:33:41.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2749491030}) 2024-05-15T23:33:41.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4125425445}) 2024-05-15T23:33:41.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3329170317}) 2024-05-15T23:33:41.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1390995134}) 2024-05-15T23:33:41.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4266929623}) 2024-05-15T23:33:41.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2064200565}) 2024-05-15T23:33:41.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1107729905}) 2024-05-15T23:33:41.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2838066740}) 2024-05-15T23:33:41.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2875529902}) 2024-05-15T23:33:41.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1617420938}) 2024-05-15T23:33:41.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1682524283}) 2024-05-15T23:33:41.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2104946961}) 2024-05-15T23:33:41.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3746820722}) 2024-05-15T23:33:41.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3280245022}) 2024-05-15T23:33:41.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2999688925}) 2024-05-15T23:33:41.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1017477765}) 2024-05-15T23:33:41.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2701276287}) 2024-05-15T23:33:41.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3491321759}) 2024-05-15T23:33:41.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2945364778}) 2024-05-15T23:33:41.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3116399613}) 2024-05-15T23:33:41.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3305337980}) 2024-05-15T23:33:41.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3115709481}) 2024-05-15T23:33:41.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1441484028}) 2024-05-15T23:33:41.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214199673}) 2024-05-15T23:33:41.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2243014154}) 2024-05-15T23:33:41.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2227066885}) 2024-05-15T23:33:41.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2226716817}) 2024-05-15T23:33:41.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=599045550}) 2024-05-15T23:33:41.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2693544098}) 2024-05-15T23:33:41.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2703122293}) 2024-05-15T23:33:41.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196708190}) 2024-05-15T23:33:41.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1672815751}) 2024-05-15T23:33:41.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2086911295}) 2024-05-15T23:33:41.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3123145767}) 2024-05-15T23:33:41.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3999257099}) 2024-05-15T23:33:41.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=82,mdsmap=3,mgr=15} crc {logm=24591630,mdsmap=2840952586,mgr=1006343046}) 2024-05-15T23:33:41.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:40 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=77} crc {mgr=4028747438,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2403392517,mon_config_key=3906756719}) 2024-05-15T23:33:41.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:33:41.203 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:41.222 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:33:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:41.225 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:33:41.326 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:41.353 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:41.407 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:41.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:41.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:41.511 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:33:41.544 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:41.567 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:33:41.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:41.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:33:41.671 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:41.699 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:41.718 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:41.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:41.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4198178275}) 2024-05-15T23:33:41.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2058754015}) 2024-05-15T23:33:41.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=7,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=75} crc {mon_config_key=1739842792,monmap=1478146745,osd_metadata=2611665884,osd_pg_creating=2272183261,osdmap=3685776376}) 2024-05-15T23:33:41.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=98} crc {osdmap=1909477966}) 2024-05-15T23:33:41.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:41 smithi088 ceph-mon[44699]: pgmap v1715: 17 pgs: 17 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 60 op/s 2024-05-15T23:33:41.822 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:41.849 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:41.868 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:41.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:41.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:33:41.971 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:33:41.999 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4198178275}) 2024-05-15T23:33:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2058754015}) 2024-05-15T23:33:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=7,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=75} crc {mon_config_key=1739842792,monmap=1478146745,osd_metadata=2611665884,osd_pg_creating=2272183261,osdmap=3685776376}) 2024-05-15T23:33:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=98} crc {osdmap=1909477966}) 2024-05-15T23:33:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:41 smithi090 ceph-mon[44434]: pgmap v1715: 17 pgs: 17 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 60 op/s 2024-05-15T23:33:42.073 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:33:42.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4198178275}) 2024-05-15T23:33:42.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2058754015}) 2024-05-15T23:33:42.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=7,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=75} crc {mon_config_key=1739842792,monmap=1478146745,osd_metadata=2611665884,osd_pg_creating=2272183261,osdmap=3685776376}) 2024-05-15T23:33:42.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=98} crc {osdmap=1909477966}) 2024-05-15T23:33:42.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:41 smithi084 ceph-mon[39784]: pgmap v1715: 17 pgs: 17 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 60 op/s 2024-05-15T23:33:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:42.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:42.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:33:42.209 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:42.231 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:33:42.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:42.235 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:33:42.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:42.366 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:42.490 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:42.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:42.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:42.594 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:42.630 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:42.649 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:42.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:42.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:42.754 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:33:42.761 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:42.784 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:42.804 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:33:42.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:42.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:33:42.908 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:33:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:42.963 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:33:42.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:42.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:33:43.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:33:43.098 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:43.118 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:33:43.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:43.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:33:43.222 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:43.255 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:43.275 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:43.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:43.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:33:43.289 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:33:43.380 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:43.409 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:43 smithi090 ceph-mon[44434]: osdmap e645: 12 total, 12 up, 10 in 2024-05-15T23:33:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:43 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4064844736' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:33:43.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:43 smithi088 ceph-mon[44699]: osdmap e645: 12 total, 12 up, 10 in 2024-05-15T23:33:43.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:43 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4064844736' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:33:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:43 smithi084 ceph-mon[39784]: osdmap e645: 12 total, 12 up, 10 in 2024-05-15T23:33:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:43 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4064844736' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:33:44.079 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10, 2, 8] out_osds: [0, 6] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:33:44.080 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:33:44.080 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-15T23:33:44.080 INFO:tasks.thrashosds.thrasher:Setting osd 9 primary_affinity to 1.000000 2024-05-15T23:33:44.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 9 1 2024-05-15T23:33:44.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:44 smithi084 ceph-mon[39784]: pgmap v1717: 17 pgs: 17 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:44.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:44 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:33:44.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4064844736' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:33:44.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:44 smithi084 ceph-mon[39784]: osdmap e646: 12 total, 12 up, 10 in 2024-05-15T23:33:44.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:44 smithi090 ceph-mon[44434]: pgmap v1717: 17 pgs: 17 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:44.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:44 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:33:44.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4064844736' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:33:44.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:44 smithi090 ceph-mon[44434]: osdmap e646: 12 total, 12 up, 10 in 2024-05-15T23:33:44.558 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:44.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:44.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:44.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:44 smithi088 ceph-mon[44699]: pgmap v1717: 17 pgs: 17 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:44.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:44 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:33:44.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4064844736' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:33:44.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:44 smithi088 ceph-mon[44699]: osdmap e646: 12 total, 12 up, 10 in 2024-05-15T23:33:44.664 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:44.691 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:45 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:33:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:45 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:33:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:45 smithi090 ceph-mon[44434]: 7.b scrub starts 2024-05-15T23:33:45.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:45 smithi090 ceph-mon[44434]: 7.b scrub ok 2024-05-15T23:33:45.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:45 smithi090 ceph-mon[44434]: 7.9 scrub starts 2024-05-15T23:33:45.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:45 smithi090 ceph-mon[44434]: 7.9 scrub ok 2024-05-15T23:33:45.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:45 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:33:45.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:45 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:33:45.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/612868175' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:33:45.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:45 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:33:45.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:45 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:33:45.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:45 smithi088 ceph-mon[44699]: 7.b scrub starts 2024-05-15T23:33:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:45 smithi088 ceph-mon[44699]: 7.b scrub ok 2024-05-15T23:33:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:45 smithi088 ceph-mon[44699]: 7.9 scrub starts 2024-05-15T23:33:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:45 smithi088 ceph-mon[44699]: 7.9 scrub ok 2024-05-15T23:33:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:45 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:33:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:45 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:33:45.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/612868175' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:33:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:45 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:33:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:45 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:33:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:45 smithi084 ceph-mon[39784]: 7.b scrub starts 2024-05-15T23:33:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:45 smithi084 ceph-mon[39784]: 7.b scrub ok 2024-05-15T23:33:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:45 smithi084 ceph-mon[39784]: 7.9 scrub starts 2024-05-15T23:33:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:45 smithi084 ceph-mon[39784]: 7.9 scrub ok 2024-05-15T23:33:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:45 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:33:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:45 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:33:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/612868175' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:33:45.843 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_5' removed 2024-05-15T23:33:46.019 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:46.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:46.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:46.123 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:46.157 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:46.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:46.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:33:46.284 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:46.313 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:46.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:46 smithi090 ceph-mon[44434]: pgmap v1719: 17 pgs: 17 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:46.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:46 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/612868175' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:33:46.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:46 smithi090 ceph-mon[44434]: osdmap e647: 12 total, 12 up, 10 in 2024-05-15T23:33:46.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:46 smithi088 ceph-mon[44699]: pgmap v1719: 17 pgs: 17 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:46.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:46 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/612868175' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:33:46.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:46 smithi088 ceph-mon[44699]: osdmap e647: 12 total, 12 up, 10 in 2024-05-15T23:33:46.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:46 smithi084 ceph-mon[39784]: pgmap v1719: 17 pgs: 17 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:46.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:46 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/612868175' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:33:46.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:46 smithi084 ceph-mon[39784]: osdmap e647: 12 total, 12 up, 10 in 2024-05-15T23:33:46.824 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:33:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:33:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:33:46.824 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:33:46.824 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:33:46.841 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:46.862 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:46.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:46.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:33:46.996 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:47.095 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:33:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:47.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:47.199 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:33:47.226 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:47.324 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:33:47.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:47.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:33:47.330 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:33:47.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:33:47.461 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:47.483 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:33:47.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:47.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:33:47.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:33:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:33:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:47.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:33:47.710 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:33:47.710 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:33:47.710 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:33:47.711 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:33:47.711 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:33:47.711 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:33:47.724 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:33:47.742 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:33:47.744 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:33:47.763 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:33:47.773 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:47.783 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:33:47.793 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:33:47.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:47.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:33:47.803 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:33:47.823 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:33:47.824 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:33:47.824 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:33:47.824 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:33:47.824 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:33:47.824 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:33:47.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:33:47.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:33:47.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:33:47.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:33:47.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:33:47.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:33:47.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:33:47.897 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:47.925 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:47.946 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:47.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:48.049 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:33:48.079 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:48.099 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:33:48.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:48.102 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:33:48.153 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T23:33:48.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:33:48.231 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:48.251 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:33:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:48.254 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:33:48.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:48.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:48 smithi084 ceph-mon[39784]: pgmap v1721: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:48.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3803788260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:33:48.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.383 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.384 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:48.405 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:48.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:48.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:48.509 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:48.539 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:48 smithi088 ceph-mon[44699]: pgmap v1721: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:48.539 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3803788260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:33:48.539 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.539 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.539 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.539 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.539 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.539 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.541 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:48.551 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:48 smithi090 ceph-mon[44434]: pgmap v1721: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3803788260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:33:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:48.562 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:48.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:48.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:33:48.667 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:33:48.694 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:49.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:49.942 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:33:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:49.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:50.047 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:50.080 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:50.101 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:50.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:50.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:50.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:33:50.232 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:50.400 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:50.429 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:33:50.431 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T23:33:50.431 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T23:33:50.432 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-05-15T23:33:50.432 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-05-15T23:33:50.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:50.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:33:50.497 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:50 smithi084 ceph-mon[39784]: pgmap v1722: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:50.497 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:50 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:33:50.497 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:50 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:33:50.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:33:50.551 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:50 smithi090 ceph-mon[44434]: pgmap v1722: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:50.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:50 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:33:50.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:50 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:33:50.567 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:50.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:50 smithi088 ceph-mon[44699]: pgmap v1722: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:50.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:50 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:33:50.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:50 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:33:50.589 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:33:50.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:50.592 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:33:50.672 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:50.693 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:33:50.726 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:50.748 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:33:50.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:50.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:33:50.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:33:50.879 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:50.897 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:33:50.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:50.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:33:51.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:33:51.038 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:51.059 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:33:51.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:51.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:33:51.081 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:33:51.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:33:51.193 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:51.214 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:33:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:51.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:33:51.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:33:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:51.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:51 smithi088 ceph-mon[44699]: osdmap e648: 12 total, 12 up, 10 in 2024-05-15T23:33:51.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:51 smithi088 ceph-mon[44699]: pgmap v1724: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:51.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:51 smithi084 ceph-mon[39784]: osdmap e648: 12 total, 12 up, 10 in 2024-05-15T23:33:51.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:51 smithi084 ceph-mon[39784]: pgmap v1724: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:51 smithi090 ceph-mon[44434]: osdmap e648: 12 total, 12 up, 10 in 2024-05-15T23:33:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:51 smithi090 ceph-mon[44434]: pgmap v1724: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:52.312 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-05-15T23:33:52.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:52 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/224194153' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:33:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:52 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/224194153' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:33:52.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:52 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/224194153' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:33:53.496 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:33:53.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:53.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:33:53.510 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:33:53.510 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:33:53.532 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:53 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/224194153' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:33:53.532 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:53 smithi084 ceph-mon[39784]: osdmap e649: 12 total, 12 up, 10 in 2024-05-15T23:33:53.532 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:53 smithi084 ceph-mon[39784]: pgmap v1726: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:53.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:53 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/224194153' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:33:53.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:53 smithi088 ceph-mon[44699]: osdmap e649: 12 total, 12 up, 10 in 2024-05-15T23:33:53.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:53 smithi088 ceph-mon[44699]: pgmap v1726: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:53.602 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:33:53.633 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:53 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/224194153' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:33:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:53 smithi090 ceph-mon[44434]: osdmap e649: 12 total, 12 up, 10 in 2024-05-15T23:33:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:53 smithi090 ceph-mon[44434]: pgmap v1726: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:54.782 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:33:54.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:54.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:33:54.887 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:33:54.921 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:54.943 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:33:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:54.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:33:55.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:33:55.077 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:55.098 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:33:55.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:55.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:33:55.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:33:55.232 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:55.324 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:55.331 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:33:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:55.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:55.364 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:33:55.368 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T23:33:55.438 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:33:55.467 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:55.487 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:33:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:55.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:33:55.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:55 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:33:55.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:55 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:33:55.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:55 smithi088 ceph-mon[44699]: osdmap e650: 12 total, 12 up, 10 in 2024-05-15T23:33:55.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:55 smithi088 ceph-mon[44699]: pgmap v1728: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:55.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:55 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:33:55.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:55 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:33:55.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:55 smithi084 ceph-mon[39784]: osdmap e650: 12 total, 12 up, 10 in 2024-05-15T23:33:55.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:55 smithi084 ceph-mon[39784]: pgmap v1728: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:55.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:33:55.622 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:55 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:33:55.622 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:55 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:33:55.622 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:55 smithi090 ceph-mon[44434]: osdmap e650: 12 total, 12 up, 10 in 2024-05-15T23:33:55.622 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:55 smithi090 ceph-mon[44434]: pgmap v1728: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:55.624 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:55.646 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:33:55.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:55.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:33:55.649 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:33:55.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:33:55.779 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:55.798 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:33:55.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:55.802 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:33:55.903 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:33:55.930 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:55.986 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:33:55.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:55.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:33:56.094 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:33:56.123 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:56.471 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:33:56.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:56.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:33:56.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:33:56.602 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:56.624 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:33:56.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:56.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:33:56.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:33:56.771 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:57.554 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:33:57.555 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:33:57.936 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:33:57.937 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:33:57.937 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:33:57.937 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:33:57.937 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:33:57.937 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:33:57.937 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:33:57.937 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:33:57.937 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:33:57.937 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:33:57.937 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:33:57.937 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:33:57.938 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:33:57.938 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:33:57.938 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:33:57.938 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:33:57.938 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:33:57.938 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:33:57.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:33:57.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:33:57.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:33:57.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:33:57.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:33:57.940 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:33:57.972 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:57 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:33:57.972 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:57 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:33:57.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:57 smithi088 ceph-mon[44699]: osdmap e651: 12 total, 12 up, 10 in 2024-05-15T23:33:57.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:57 smithi088 ceph-mon[44699]: pgmap v1730: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:57.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4124325052' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:33:57.973 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2164239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:33:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:57 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:33:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:57 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:33:58.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:57 smithi090 ceph-mon[44434]: osdmap e651: 12 total, 12 up, 10 in 2024-05-15T23:33:58.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:57 smithi090 ceph-mon[44434]: pgmap v1730: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:58.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4124325052' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:33:58.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2164239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:33:58.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:57 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:33:58.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:57 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:33:58.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:57 smithi084 ceph-mon[39784]: osdmap e651: 12 total, 12 up, 10 in 2024-05-15T23:33:58.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:57 smithi084 ceph-mon[39784]: pgmap v1730: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:58.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4124325052' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:33:58.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2164239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:33:58.285 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:33:58.287 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10, 2, 8] out_osds: [0, 6] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:33:58.288 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:33:58.288 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-05-15T23:33:58.288 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-05-15T23:33:58.368 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T23:33:58.634 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:33:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:33:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:33:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:33:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:33:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4124325052' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:33:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:58 smithi090 ceph-mon[44434]: osdmap e652: 12 total, 12 up, 10 in 2024-05-15T23:33:59.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:33:59.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:33:59.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:33:59.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:33:59.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4124325052' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:33:59.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:58 smithi088 ceph-mon[44699]: osdmap e652: 12 total, 12 up, 10 in 2024-05-15T23:33:59.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:33:59.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:33:59.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.101 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:33:59.101 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:33:59.101 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:33:59.101 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4124325052' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:33:59.101 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:58 smithi084 ceph-mon[39784]: osdmap e652: 12 total, 12 up, 10 in 2024-05-15T23:33:59.121 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:33:59.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:59.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:59.225 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:33:59.253 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:59.601 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:33:59.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:59.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:33:59.704 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:33:59.734 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:33:59.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:33:59 smithi084 ceph-mon[39784]: pgmap v1732: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:59.971 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:33:59 smithi088 ceph-mon[44699]: pgmap v1732: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:33:59.983 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:33:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:33:59.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:34:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:33:59 smithi090 ceph-mon[44434]: pgmap v1732: 1 pgs: 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:00.087 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:34:00.114 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:00.245 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:34:00.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:00.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:34:00.350 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:34:00.377 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:00.439 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:00.474 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:34:00.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:00.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:34:00.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:34:00.595 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:34:00.596 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:34:00.596 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:34:00.596 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:34:00.596 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:34:00.612 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:00.635 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:34:00.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:00.638 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:34:00.739 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:34:00.767 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:00.797 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_6' created 2024-05-15T23:34:00.890 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:34:00.891 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:00 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:34:00.891 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:00 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:34:00.891 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:00 smithi088 ceph-mon[44699]: osdmap e653: 12 total, 12 up, 10 in 2024-05-15T23:34:00.892 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:00 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3041280267' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-05-15T23:34:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:00.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:34:00.957 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:00 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:34:00.957 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:00 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:34:00.957 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:00 smithi084 ceph-mon[39784]: osdmap e653: 12 total, 12 up, 10 in 2024-05-15T23:34:00.957 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:00 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3041280267' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-05-15T23:34:00.993 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:34:01.021 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:01.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:00 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:34:01.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:00 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:34:01.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:00 smithi090 ceph-mon[44434]: osdmap e653: 12 total, 12 up, 10 in 2024-05-15T23:34:01.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:00 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3041280267' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-05-15T23:34:01.129 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:34:01.455 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T23:34:01.470 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:34:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:01.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:34:01.574 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:34:01.600 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:01.626 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:34:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:01.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:34:01.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:34:01.742 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:34:01.763 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:01.784 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:34:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:01.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:34:01.918 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:34:01.946 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:02.010 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:01 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:34:02.011 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:01 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:34:02.011 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:01 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3041280267' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-05-15T23:34:02.011 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:01 smithi084 ceph-mon[39784]: osdmap e654: 12 total, 12 up, 10 in 2024-05-15T23:34:02.011 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:01 smithi084 ceph-mon[39784]: pgmap v1735: 17 pgs: 4 creating+peering, 12 unknown, 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:02.011 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:34:02.018 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:34:02.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:02.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:34:02.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:01 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:34:02.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:01 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:34:02.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:01 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3041280267' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-05-15T23:34:02.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:01 smithi090 ceph-mon[44434]: osdmap e654: 12 total, 12 up, 10 in 2024-05-15T23:34:02.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:01 smithi090 ceph-mon[44434]: pgmap v1735: 17 pgs: 4 creating+peering, 12 unknown, 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:02.063 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:01 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:34:02.063 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:01 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:34:02.063 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:01 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3041280267' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-05-15T23:34:02.064 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:01 smithi088 ceph-mon[44699]: osdmap e654: 12 total, 12 up, 10 in 2024-05-15T23:34:02.064 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:01 smithi088 ceph-mon[44699]: pgmap v1735: 17 pgs: 4 creating+peering, 12 unknown, 1 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:02.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:34:02.152 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:02.173 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:34:02.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:02.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:34:02.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:34:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:02.326 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:34:02.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:02.329 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:34:02.428 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-05-15T23:34:02.430 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:34:02.458 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:02.466 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:02.531 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:34:02.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:02.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:34:02.635 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:34:02.661 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:02.914 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:34:02.915 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:02 smithi084 ceph-mon[39784]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:34:02.915 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:02 smithi084 ceph-mon[39784]: osdmap e655: 12 total, 12 up, 10 in 2024-05-15T23:34:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:02.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:34:02.934 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:02 smithi088 ceph-mon[44699]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:34:02.934 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:02 smithi088 ceph-mon[44699]: osdmap e655: 12 total, 12 up, 10 in 2024-05-15T23:34:03.017 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:34:03.044 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:03.195 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:34:03.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:03.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:34:03.299 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:34:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:02 smithi090 ceph-mon[44434]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:34:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:02 smithi090 ceph-mon[44434]: osdmap e655: 12 total, 12 up, 10 in 2024-05-15T23:34:03.330 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:03.509 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:34:03.509 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:34:03.509 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:34:03.509 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:34:03.509 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.509 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:94654 6.e 6:74360dd9:::benchmark_data_smithi090_131278_object94653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e505)", 2024-05-15T23:34:03.509 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:26:08.245467+0000", 2024-05-15T23:34:03.510 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 475.24905047200002, 2024-05-15T23:34:03.510 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.117853851, 2024-05-15T23:34:03.510 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.510 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.510 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.510 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.510 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.510 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.510 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 94654 2024-05-15T23:34:03.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.510 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.511 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.511 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.511 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.245467+0000", 2024-05-15T23:34:03.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.511 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.511 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.511 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.511 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.245467+0000", 2024-05-15T23:34:03.511 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.511 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.511 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.512 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.512 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.245469+0000", 2024-05-15T23:34:03.512 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.984e-06 2024-05-15T23:34:03.512 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.512 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.512 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.512 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.245496+0000", 2024-05-15T23:34:03.512 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7506999999999999e-05 2024-05-15T23:34:03.512 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.512 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.512 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.512 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.245505+0000", 2024-05-15T23:34:03.513 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3030000000000005e-06 2024-05-15T23:34:03.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.513 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.513 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.513 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.245512+0000", 2024-05-15T23:34:03.513 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4590000000000003e-06 2024-05-15T23:34:03.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.513 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.513 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.513 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.284715+0000", 2024-05-15T23:34:03.513 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.039203100999999997 2024-05-15T23:34:03.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.514 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.514 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.514 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.284841+0000", 2024-05-15T23:34:03.514 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012596800000000001 2024-05-15T23:34:03.514 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.514 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.514 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:34:03.514 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.284960+0000", 2024-05-15T23:34:03.514 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011882100000000001 2024-05-15T23:34:03.514 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.515 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.515 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.515 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.360738+0000", 2024-05-15T23:34:03.515 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.075777757000000001 2024-05-15T23:34:03.515 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.515 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.515 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.515 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.363236+0000", 2024-05-15T23:34:03.515 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024987030000000001 2024-05-15T23:34:03.515 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.515 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.515 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.516 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.363283+0000", 2024-05-15T23:34:03.516 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6560000000000001e-05 2024-05-15T23:34:03.516 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.516 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.516 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.516 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.363320+0000", 2024-05-15T23:34:03.516 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7688000000000002e-05 2024-05-15T23:34:03.516 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.516 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.516 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.516 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.516 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.517 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:94707 6.e 6:7f82f31e:::benchmark_data_smithi090_131278_object94706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e505)", 2024-05-15T23:34:03.517 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:26:08.275948+0000", 2024-05-15T23:34:03.517 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 475.218568731, 2024-05-15T23:34:03.517 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.097771747000000006, 2024-05-15T23:34:03.517 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.517 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.517 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.517 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.517 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.517 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.517 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 94707 2024-05-15T23:34:03.517 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.518 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.518 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.518 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.518 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.275948+0000", 2024-05-15T23:34:03.518 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.518 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.518 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.518 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.518 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.275948+0000", 2024-05-15T23:34:03.518 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.518 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.518 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.519 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.519 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.275954+0000", 2024-05-15T23:34:03.519 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3020000000000002e-06 2024-05-15T23:34:03.519 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.519 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.519 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.519 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.276004+0000", 2024-05-15T23:34:03.519 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0299000000000001e-05 2024-05-15T23:34:03.519 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.519 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.519 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.519 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.276014+0000", 2024-05-15T23:34:03.519 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6630000000000005e-06 2024-05-15T23:34:03.520 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.520 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.520 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.520 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.276019+0000", 2024-05-15T23:34:03.520 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.631e-06 2024-05-15T23:34:03.520 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.520 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.520 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.520 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.287887+0000", 2024-05-15T23:34:03.520 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.011867857000000001 2024-05-15T23:34:03.520 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.521 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.521 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.521 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.288003+0000", 2024-05-15T23:34:03.521 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000115964 2024-05-15T23:34:03.521 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.521 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.521 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:34:03.521 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.357552+0000", 2024-05-15T23:34:03.521 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.069549269999999996 2024-05-15T23:34:03.521 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.521 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.521 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.521 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.365919+0000", 2024-05-15T23:34:03.522 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0083670350000000001 2024-05-15T23:34:03.522 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.522 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.522 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.522 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.373610+0000", 2024-05-15T23:34:03.522 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0076904340000000003 2024-05-15T23:34:03.522 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.522 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.522 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.522 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.373663+0000", 2024-05-15T23:34:03.522 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3018e-05 2024-05-15T23:34:03.522 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.523 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.523 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.523 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:08.373720+0000", 2024-05-15T23:34:03.523 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7274000000000001e-05 2024-05-15T23:34:03.523 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.523 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.523 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.523 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.523 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.523 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:96693 6.e 6:7e8532f2:::benchmark_data_smithi090_131278_object96692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e505)", 2024-05-15T23:34:03.523 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:26:09.669971+0000", 2024-05-15T23:34:03.523 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 473.82454578099998, 2024-05-15T23:34:03.523 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.103696179, 2024-05-15T23:34:03.524 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.524 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.524 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.524 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.524 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.524 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.524 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 96693 2024-05-15T23:34:03.524 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.524 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.524 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.524 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.524 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.669971+0000", 2024-05-15T23:34:03.524 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.525 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.525 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.525 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.525 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.669971+0000", 2024-05-15T23:34:03.525 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.525 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.525 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.525 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.525 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.669973+0000", 2024-05-15T23:34:03.525 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.886e-06 2024-05-15T23:34:03.525 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.526 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.526 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.526 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.670026+0000", 2024-05-15T23:34:03.526 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2735000000000002e-05 2024-05-15T23:34:03.526 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.526 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.526 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.526 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.670036+0000", 2024-05-15T23:34:03.526 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0361e-05 2024-05-15T23:34:03.526 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.526 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.526 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.527 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.670044+0000", 2024-05-15T23:34:03.527 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1419999999999993e-06 2024-05-15T23:34:03.527 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.527 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.527 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.527 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.763501+0000", 2024-05-15T23:34:03.527 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.093457042000000004 2024-05-15T23:34:03.527 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.527 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.527 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.527 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.763635+0000", 2024-05-15T23:34:03.527 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013397999999999999 2024-05-15T23:34:03.528 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.528 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.528 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:34:03.528 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.763785+0000", 2024-05-15T23:34:03.528 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014941800000000001 2024-05-15T23:34:03.528 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.528 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.528 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.528 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.767009+0000", 2024-05-15T23:34:03.528 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032239080000000002 2024-05-15T23:34:03.528 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.528 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.528 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.529 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.773589+0000", 2024-05-15T23:34:03.529 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0065803850000000002 2024-05-15T23:34:03.529 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.529 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.529 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.529 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.773626+0000", 2024-05-15T23:34:03.529 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.6761999999999998e-05 2024-05-15T23:34:03.529 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.529 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.529 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.529 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.773667+0000", 2024-05-15T23:34:03.529 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1560000000000002e-05 2024-05-15T23:34:03.530 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.530 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.530 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.530 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.530 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.530 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:96710 6.e 6:71303ed7:::benchmark_data_smithi090_131278_object96709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e505)", 2024-05-15T23:34:03.530 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:26:09.684225+0000", 2024-05-15T23:34:03.530 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 473.81029207400002, 2024-05-15T23:34:03.530 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.096060770000000004, 2024-05-15T23:34:03.530 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.530 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.530 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.531 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.531 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.531 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.531 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 96710 2024-05-15T23:34:03.531 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.531 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.531 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.531 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.531 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.684225+0000", 2024-05-15T23:34:03.532 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.532 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.532 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.532 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.532 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.684225+0000", 2024-05-15T23:34:03.532 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.532 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.532 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.532 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.532 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.684227+0000", 2024-05-15T23:34:03.532 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2280000000000001e-06 2024-05-15T23:34:03.532 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.533 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.533 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.533 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.684271+0000", 2024-05-15T23:34:03.533 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4103000000000001e-05 2024-05-15T23:34:03.533 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.533 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.533 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.533 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.684282+0000", 2024-05-15T23:34:03.533 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0283e-05 2024-05-15T23:34:03.533 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.533 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.533 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.533 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.684290+0000", 2024-05-15T23:34:03.534 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0839999999999992e-06 2024-05-15T23:34:03.534 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.534 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.534 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.534 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.767129+0000", 2024-05-15T23:34:03.534 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.082839641000000006 2024-05-15T23:34:03.534 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.534 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.534 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.534 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.767285+0000", 2024-05-15T23:34:03.534 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00015611600000000001 2024-05-15T23:34:03.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.535 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:34:03.535 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.767431+0000", 2024-05-15T23:34:03.535 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014538999999999999 2024-05-15T23:34:03.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.535 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.535 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.773717+0000", 2024-05-15T23:34:03.535 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0062857049999999999 2024-05-15T23:34:03.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.535 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.535 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.780198+0000", 2024-05-15T23:34:03.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0064813029999999999 2024-05-15T23:34:03.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.536 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.780255+0000", 2024-05-15T23:34:03.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7021000000000002e-05 2024-05-15T23:34:03.536 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.536 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.536 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.536 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:09.780286+0000", 2024-05-15T23:34:03.536 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0895999999999997e-05 2024-05-15T23:34:03.536 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.537 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.537 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.537 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.537 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.537 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:98301 6.e 6:7ed317cd:::benchmark_data_smithi090_131278_object98300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e506)", 2024-05-15T23:34:03.537 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:26:10.822000+0000", 2024-05-15T23:34:03.537 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 472.67251733799998, 2024-05-15T23:34:03.537 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.105313092, 2024-05-15T23:34:03.537 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.537 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.537 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.537 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.538 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.538 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.538 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 98301 2024-05-15T23:34:03.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.538 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.822000+0000", 2024-05-15T23:34:03.538 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.538 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.538 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.538 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.822000+0000", 2024-05-15T23:34:03.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.822001+0000", 2024-05-15T23:34:03.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.3799999999999996e-07 2024-05-15T23:34:03.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.539 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.539 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.539 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.822064+0000", 2024-05-15T23:34:03.539 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2910000000000006e-05 2024-05-15T23:34:03.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.540 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.822072+0000", 2024-05-15T23:34:03.540 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8869999999999998e-06 2024-05-15T23:34:03.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.540 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.822078+0000", 2024-05-15T23:34:03.540 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9070000000000004e-06 2024-05-15T23:34:03.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.540 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.540 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.540 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.882889+0000", 2024-05-15T23:34:03.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.06081023 2024-05-15T23:34:03.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.883017+0000", 2024-05-15T23:34:03.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000128498 2024-05-15T23:34:03.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.541 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.541 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:34:03.541 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.883154+0000", 2024-05-15T23:34:03.541 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013733199999999999 2024-05-15T23:34:03.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.892756+0000", 2024-05-15T23:34:03.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.009601666 2024-05-15T23:34:03.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.542 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.542 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.927240+0000", 2024-05-15T23:34:03.542 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.034483595999999998 2024-05-15T23:34:03.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.542 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.927272+0000", 2024-05-15T23:34:03.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2407000000000003e-05 2024-05-15T23:34:03.543 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.543 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.543 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.543 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:10.927313+0000", 2024-05-15T23:34:03.543 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0720999999999999e-05 2024-05-15T23:34:03.543 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.544 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.544 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.544 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.544 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:129201 6.2 6:495ff411:::benchmark_data_smithi090_131278_object129200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e514)", 2024-05-15T23:34:03.544 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:26:32.654127+0000", 2024-05-15T23:34:03.544 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 450.840389765, 2024-05-15T23:34:03.544 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.096044389999999993, 2024-05-15T23:34:03.544 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.544 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.544 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.545 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.545 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.545 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.545 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 129201 2024-05-15T23:34:03.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.545 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.545 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.545 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:32.654127+0000", 2024-05-15T23:34:03.545 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.545 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:32.654127+0000", 2024-05-15T23:34:03.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.546 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:32.654128+0000", 2024-05-15T23:34:03.546 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0299999999999997e-07 2024-05-15T23:34:03.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.546 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.546 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:32.654200+0000", 2024-05-15T23:34:03.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1793999999999997e-05 2024-05-15T23:34:03.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.547 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:32.654207+0000", 2024-05-15T23:34:03.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7889999999999997e-06 2024-05-15T23:34:03.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.547 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.547 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.547 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:32.654224+0000", 2024-05-15T23:34:03.547 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6704000000000001e-05 2024-05-15T23:34:03.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:32.658377+0000", 2024-05-15T23:34:03.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0041538820000000002 2024-05-15T23:34:03.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.548 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.548 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:32.658496+0000", 2024-05-15T23:34:03.548 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000118736 2024-05-15T23:34:03.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.548 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:34:03.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:32.658582+0000", 2024-05-15T23:34:03.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5416999999999999e-05 2024-05-15T23:34:03.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.549 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:32.742120+0000", 2024-05-15T23:34:03.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.083538812000000004 2024-05-15T23:34:03.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.549 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.549 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.549 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:32.750068+0000", 2024-05-15T23:34:03.549 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0079473540000000002 2024-05-15T23:34:03.550 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.550 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.550 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:32.750121+0000", 2024-05-15T23:34:03.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2893999999999999e-05 2024-05-15T23:34:03.550 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.550 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.550 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.550 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:26:32.750172+0000", 2024-05-15T23:34:03.550 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1104999999999999e-05 2024-05-15T23:34:03.550 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.550 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.551 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.551 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.551 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:243530 6.2 6:4995e3b1:::benchmark_data_smithi090_131278_object108336:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-05-15T23:34:03.551 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:28:08.089777+0000", 2024-05-15T23:34:03.551 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 355.40474010600002, 2024-05-15T23:34:03.551 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.057659500000000002, 2024-05-15T23:34:03.551 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.551 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.551 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.551 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.551 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.552 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.552 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 243530 2024-05-15T23:34:03.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.552 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.089777+0000", 2024-05-15T23:34:03.552 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.552 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.552 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.552 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.089777+0000", 2024-05-15T23:34:03.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.089779+0000", 2024-05-15T23:34:03.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3269999999999999e-06 2024-05-15T23:34:03.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.553 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.553 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.089787+0000", 2024-05-15T23:34:03.553 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2309999999999999e-06 2024-05-15T23:34:03.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.553 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.089802+0000", 2024-05-15T23:34:03.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5758999999999999e-05 2024-05-15T23:34:03.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.089811+0000", 2024-05-15T23:34:03.554 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.9449999999999999e-06 2024-05-15T23:34:03.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.554 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.554 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.554 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.089876+0000", 2024-05-15T23:34:03.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4553000000000003e-05 2024-05-15T23:34:03.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.090004+0000", 2024-05-15T23:34:03.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012781899999999999 2024-05-15T23:34:03.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.555 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:34:03.555 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.090123+0000", 2024-05-15T23:34:03.555 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000119151 2024-05-15T23:34:03.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.555 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.097018+0000", 2024-05-15T23:34:03.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068948589999999997 2024-05-15T23:34:03.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.147377+0000", 2024-05-15T23:34:03.556 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.050359525000000002 2024-05-15T23:34:03.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.556 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.556 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.147409+0000", 2024-05-15T23:34:03.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1420000000000001e-05 2024-05-15T23:34:03.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.557 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.557 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.147436+0000", 2024-05-15T23:34:03.557 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7911000000000001e-05 2024-05-15T23:34:03.557 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.557 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.557 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.557 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:243546 6.2 6:4e007fdc:::benchmark_data_smithi090_131278_object108352:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-05-15T23:34:03.557 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:28:08.097421+0000", 2024-05-15T23:34:03.558 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 355.39709606299999, 2024-05-15T23:34:03.558 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.050184484000000001, 2024-05-15T23:34:03.558 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.558 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.558 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.558 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.558 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.558 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.558 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 243546 2024-05-15T23:34:03.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.558 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.097421+0000", 2024-05-15T23:34:03.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.097421+0000", 2024-05-15T23:34:03.559 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.559 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.559 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.097423+0000", 2024-05-15T23:34:03.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0049999999999999e-06 2024-05-15T23:34:03.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.097428+0000", 2024-05-15T23:34:03.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2360000000000004e-06 2024-05-15T23:34:03.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.560 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.560 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.097441+0000", 2024-05-15T23:34:03.560 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.278e-05 2024-05-15T23:34:03.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.097449+0000", 2024-05-15T23:34:03.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6350000000000006e-06 2024-05-15T23:34:03.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.097496+0000", 2024-05-15T23:34:03.561 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7545e-05 2024-05-15T23:34:03.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.561 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.561 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.097610+0000", 2024-05-15T23:34:03.562 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011410599999999999 2024-05-15T23:34:03.562 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.562 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:34:03.562 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.097720+0000", 2024-05-15T23:34:03.562 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000109403 2024-05-15T23:34:03.562 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.562 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.562 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.105873+0000", 2024-05-15T23:34:03.562 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0081535080000000003 2024-05-15T23:34:03.562 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.563 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.147498+0000", 2024-05-15T23:34:03.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041625153999999998 2024-05-15T23:34:03.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.563 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.147562+0000", 2024-05-15T23:34:03.563 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3863999999999997e-05 2024-05-15T23:34:03.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.563 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.564 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.147606+0000", 2024-05-15T23:34:03.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3248000000000003e-05 2024-05-15T23:34:03.564 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.564 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.564 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.564 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:243550 6.2 6:4074e228:::benchmark_data_smithi090_131278_object108356:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-05-15T23:34:03.564 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:28:08.101409+0000", 2024-05-15T23:34:03.564 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 355.39310801200003, 2024-05-15T23:34:03.564 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.059247013000000001, 2024-05-15T23:34:03.564 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.565 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.565 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.565 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.565 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.565 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.565 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 243550 2024-05-15T23:34:03.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.565 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.565 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.565 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.101409+0000", 2024-05-15T23:34:03.565 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.101409+0000", 2024-05-15T23:34:03.566 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.566 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.101411+0000", 2024-05-15T23:34:03.566 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.26e-06 2024-05-15T23:34:03.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.566 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.101417+0000", 2024-05-15T23:34:03.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9000000000000003e-06 2024-05-15T23:34:03.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.101430+0000", 2024-05-15T23:34:03.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2349e-05 2024-05-15T23:34:03.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.567 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.567 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.101437+0000", 2024-05-15T23:34:03.567 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.875e-06 2024-05-15T23:34:03.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.568 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.568 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.102074+0000", 2024-05-15T23:34:03.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00063696900000000003 2024-05-15T23:34:03.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.568 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.568 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.102206+0000", 2024-05-15T23:34:03.568 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000131168 2024-05-15T23:34:03.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:34:03.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.102314+0000", 2024-05-15T23:34:03.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000108428 2024-05-15T23:34:03.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.109510+0000", 2024-05-15T23:34:03.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0071957660000000001 2024-05-15T23:34:03.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.569 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.569 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.160573+0000", 2024-05-15T23:34:03.569 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051063012999999997 2024-05-15T23:34:03.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.160608+0000", 2024-05-15T23:34:03.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5141000000000001e-05 2024-05-15T23:34:03.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.570 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.570 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.570 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.160656+0000", 2024-05-15T23:34:03.570 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8143999999999998e-05 2024-05-15T23:34:03.570 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.570 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.571 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.571 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.571 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:243553 6.2 6:4d319857:::benchmark_data_smithi090_131278_object108359:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-05-15T23:34:03.571 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:28:08.102685+0000", 2024-05-15T23:34:03.571 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 355.39183186700001, 2024-05-15T23:34:03.571 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058086275, 2024-05-15T23:34:03.571 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.571 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.571 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.571 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.571 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.572 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.572 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 243553 2024-05-15T23:34:03.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.572 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.102685+0000", 2024-05-15T23:34:03.572 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.572 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.572 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.572 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.572 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.102685+0000", 2024-05-15T23:34:03.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.102686+0000", 2024-05-15T23:34:03.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.209e-06 2024-05-15T23:34:03.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.573 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.573 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.573 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.102691+0000", 2024-05-15T23:34:03.573 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1999999999999996e-06 2024-05-15T23:34:03.573 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.574 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.574 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.102699+0000", 2024-05-15T23:34:03.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6780000000000006e-06 2024-05-15T23:34:03.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.574 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.574 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.102706+0000", 2024-05-15T23:34:03.574 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4999999999999996e-06 2024-05-15T23:34:03.574 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.574 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.105891+0000", 2024-05-15T23:34:03.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0031850749999999999 2024-05-15T23:34:03.575 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.575 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.575 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.575 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.105963+0000", 2024-05-15T23:34:03.575 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.1629000000000001e-05 2024-05-15T23:34:03.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:34:03.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.106039+0000", 2024-05-15T23:34:03.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6670999999999998e-05 2024-05-15T23:34:03.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.576 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.576 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.576 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.113083+0000", 2024-05-15T23:34:03.576 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.007043672 2024-05-15T23:34:03.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.160705+0000", 2024-05-15T23:34:03.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047622167 2024-05-15T23:34:03.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.160736+0000", 2024-05-15T23:34:03.577 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0630999999999999e-05 2024-05-15T23:34:03.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.577 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.577 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.577 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.160771+0000", 2024-05-15T23:34:03.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5843000000000002e-05 2024-05-15T23:34:03.578 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.578 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.578 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.578 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.578 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:243556 6.2 6:4e14268a:::benchmark_data_smithi090_131278_object108362:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-05-15T23:34:03.578 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:28:08.106526+0000", 2024-05-15T23:34:03.578 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 355.38799099300002, 2024-05-15T23:34:03.578 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058735392999999997, 2024-05-15T23:34:03.578 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.578 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.579 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.579 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.579 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.579 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.579 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 243556 2024-05-15T23:34:03.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.579 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.579 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.579 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.579 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.106526+0000", 2024-05-15T23:34:03.579 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.579 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.106526+0000", 2024-05-15T23:34:03.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.580 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.580 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.106528+0000", 2024-05-15T23:34:03.580 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6220000000000001e-06 2024-05-15T23:34:03.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.580 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.581 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.106533+0000", 2024-05-15T23:34:03.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0340000000000004e-06 2024-05-15T23:34:03.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.581 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.106543+0000", 2024-05-15T23:34:03.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7650000000000005e-06 2024-05-15T23:34:03.581 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.581 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.581 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.581 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.106549+0000", 2024-05-15T23:34:03.581 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2809999999999997e-06 2024-05-15T23:34:03.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.109526+0000", 2024-05-15T23:34:03.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029775079999999998 2024-05-15T23:34:03.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.582 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.582 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.109593+0000", 2024-05-15T23:34:03.582 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.7084999999999999e-05 2024-05-15T23:34:03.582 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.582 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:34:03.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.109657+0000", 2024-05-15T23:34:03.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3800999999999994e-05 2024-05-15T23:34:03.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.113196+0000", 2024-05-15T23:34:03.583 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0035392850000000001 2024-05-15T23:34:03.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.583 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.583 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.583 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.165148+0000", 2024-05-15T23:34:03.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051951360000000002 2024-05-15T23:34:03.584 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.584 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.584 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.584 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.165218+0000", 2024-05-15T23:34:03.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9970999999999998e-05 2024-05-15T23:34:03.584 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.584 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.584 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.584 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:08.165261+0000", 2024-05-15T23:34:03.584 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3680999999999997e-05 2024-05-15T23:34:03.584 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.584 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.585 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.585 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.585 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:245204 6.2 6:40901e09:::benchmark_data_smithi090_131278_object110010:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-05-15T23:34:03.585 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:28:09.435118+0000", 2024-05-15T23:34:03.585 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 354.05939904799999, 2024-05-15T23:34:03.585 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.063946732000000006, 2024-05-15T23:34:03.585 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.585 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.585 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.585 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.585 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.585 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.586 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 245204 2024-05-15T23:34:03.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.586 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.435118+0000", 2024-05-15T23:34:03.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.586 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.586 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.586 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.586 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.435118+0000", 2024-05-15T23:34:03.586 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.435120+0000", 2024-05-15T23:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.483e-06 2024-05-15T23:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.435125+0000", 2024-05-15T23:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8329999999999999e-06 2024-05-15T23:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.435138+0000", 2024-05-15T23:34:03.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2313999999999999e-05 2024-05-15T23:34:03.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.588 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.435145+0000", 2024-05-15T23:34:03.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7889999999999995e-06 2024-05-15T23:34:03.588 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.588 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.588 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.588 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.436839+0000", 2024-05-15T23:34:03.588 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0016939990000000001 2024-05-15T23:34:03.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.589 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.589 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.436961+0000", 2024-05-15T23:34:03.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012177599999999999 2024-05-15T23:34:03.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.589 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:34:03.589 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.437085+0000", 2024-05-15T23:34:03.589 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012330699999999999 2024-05-15T23:34:03.589 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.589 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.590 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.440928+0000", 2024-05-15T23:34:03.590 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0038429779999999999 2024-05-15T23:34:03.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.590 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.590 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.498959+0000", 2024-05-15T23:34:03.590 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.058031539 2024-05-15T23:34:03.590 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.590 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.590 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.590 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.499017+0000", 2024-05-15T23:34:03.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7853999999999999e-05 2024-05-15T23:34:03.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.591 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.591 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.499065+0000", 2024-05-15T23:34:03.591 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7859999999999999e-05 2024-05-15T23:34:03.591 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.591 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.591 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.591 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.591 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.591 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:245214 6.2 6:47c58aa8:::benchmark_data_smithi090_131278_object110020:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-05-15T23:34:03.592 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:28:09.443849+0000", 2024-05-15T23:34:03.592 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 354.05066801200002, 2024-05-15T23:34:03.592 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055345551, 2024-05-15T23:34:03.592 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.592 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.592 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.592 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.592 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.592 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.592 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 245214 2024-05-15T23:34:03.592 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.592 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.593 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.443849+0000", 2024-05-15T23:34:03.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.593 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.443849+0000", 2024-05-15T23:34:03.593 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.593 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.593 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.593 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.443851+0000", 2024-05-15T23:34:03.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9230000000000001e-06 2024-05-15T23:34:03.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.443857+0000", 2024-05-15T23:34:03.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0689999999999998e-06 2024-05-15T23:34:03.594 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.594 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.594 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.594 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.443869+0000", 2024-05-15T23:34:03.594 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2159e-05 2024-05-15T23:34:03.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.595 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.443877+0000", 2024-05-15T23:34:03.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2260000000000005e-06 2024-05-15T23:34:03.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.595 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.595 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.595 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.443929+0000", 2024-05-15T23:34:03.595 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1789000000000002e-05 2024-05-15T23:34:03.595 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.444055+0000", 2024-05-15T23:34:03.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012572000000000001 2024-05-15T23:34:03.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:34:03.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.444161+0000", 2024-05-15T23:34:03.596 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000106439 2024-05-15T23:34:03.596 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.596 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.596 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.596 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.449060+0000", 2024-05-15T23:34:03.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004898314 2024-05-15T23:34:03.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.499109+0000", 2024-05-15T23:34:03.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.050048794000000001 2024-05-15T23:34:03.597 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.597 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.597 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.597 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.499160+0000", 2024-05-15T23:34:03.597 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1038000000000002e-05 2024-05-15T23:34:03.598 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.598 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.598 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.598 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.499195+0000", 2024-05-15T23:34:03.598 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5080000000000003e-05 2024-05-15T23:34:03.598 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.598 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.598 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.599 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.599 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.599 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:245222 6.2 6:413dcf14:::benchmark_data_smithi090_131278_object110028:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-05-15T23:34:03.599 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:28:09.450576+0000", 2024-05-15T23:34:03.599 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 354.04394138100002, 2024-05-15T23:34:03.599 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052851948000000003, 2024-05-15T23:34:03.599 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.599 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.599 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.599 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.599 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.599 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.599 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 245222 2024-05-15T23:34:03.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.600 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.600 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.450576+0000", 2024-05-15T23:34:03.600 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.600 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.600 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.600 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.450576+0000", 2024-05-15T23:34:03.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.450577+0000", 2024-05-15T23:34:03.601 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.274e-06 2024-05-15T23:34:03.601 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.601 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.601 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.601 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.450582+0000", 2024-05-15T23:34:03.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6299999999999997e-06 2024-05-15T23:34:03.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.450591+0000", 2024-05-15T23:34:03.602 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.1279999999999997e-06 2024-05-15T23:34:03.602 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.602 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.602 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.602 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.450596+0000", 2024-05-15T23:34:03.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4709999999999998e-06 2024-05-15T23:34:03.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.450635+0000", 2024-05-15T23:34:03.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.913e-05 2024-05-15T23:34:03.603 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.603 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.603 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.603 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.450709+0000", 2024-05-15T23:34:03.603 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3269000000000006e-05 2024-05-15T23:34:03.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:34:03.604 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.450773+0000", 2024-05-15T23:34:03.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4616000000000006e-05 2024-05-15T23:34:03.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.604 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.454597+0000", 2024-05-15T23:34:03.604 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0038233260000000002 2024-05-15T23:34:03.604 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.604 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.604 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.605 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.503300+0000", 2024-05-15T23:34:03.605 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.048703808000000001 2024-05-15T23:34:03.605 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.605 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.605 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.503371+0000", 2024-05-15T23:34:03.605 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0959999999999998e-05 2024-05-15T23:34:03.605 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.605 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.605 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.605 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:09.503428+0000", 2024-05-15T23:34:03.605 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6335999999999997e-05 2024-05-15T23:34:03.606 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.606 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.606 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.606 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.606 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.606 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:246470 6.2 6:4160b113:::benchmark_data_smithi090_131278_object111276:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-05-15T23:34:03.606 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:28:10.460498+0000", 2024-05-15T23:34:03.606 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 353.034019091, 2024-05-15T23:34:03.606 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.039036661, 2024-05-15T23:34:03.606 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.606 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.606 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.606 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.607 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.607 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.607 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 246470 2024-05-15T23:34:03.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.607 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.607 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.607 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:10.460498+0000", 2024-05-15T23:34:03.607 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.607 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.607 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.607 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:10.460498+0000", 2024-05-15T23:34:03.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:10.460500+0000", 2024-05-15T23:34:03.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0990000000000001e-06 2024-05-15T23:34:03.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.608 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.608 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.608 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:10.460506+0000", 2024-05-15T23:34:03.608 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.0820000000000004e-06 2024-05-15T23:34:03.608 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:10.460518+0000", 2024-05-15T23:34:03.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1763000000000001e-05 2024-05-15T23:34:03.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:10.460523+0000", 2024-05-15T23:34:03.609 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5060000000000003e-06 2024-05-15T23:34:03.609 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.609 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.609 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.609 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:10.461335+0000", 2024-05-15T23:34:03.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00081136000000000001 2024-05-15T23:34:03.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.610 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.610 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:10.461452+0000", 2024-05-15T23:34:03.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000116981 2024-05-15T23:34:03.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.610 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.610 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:34:03.610 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:10.461563+0000", 2024-05-15T23:34:03.610 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011170700000000001 2024-05-15T23:34:03.610 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.611 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.611 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:10.465708+0000", 2024-05-15T23:34:03.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.004144249 2024-05-15T23:34:03.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.611 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.611 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:10.499441+0000", 2024-05-15T23:34:03.611 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.033733306999999997 2024-05-15T23:34:03.611 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.611 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.611 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.611 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:10.499494+0000", 2024-05-15T23:34:03.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2534999999999998e-05 2024-05-15T23:34:03.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.612 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.612 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:10.499535+0000", 2024-05-15T23:34:03.612 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1072000000000002e-05 2024-05-15T23:34:03.612 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.612 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.612 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.612 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.612 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.612 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:249084 6.e 6:7615e6d4:::benchmark_data_smithi090_131278_object113890:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-05-15T23:34:03.612 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:28:12.614914+0000", 2024-05-15T23:34:03.613 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 350.87960339699998, 2024-05-15T23:34:03.613 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.037155723000000002, 2024-05-15T23:34:03.613 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.613 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.613 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.613 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.613 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.613 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.613 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 249084 2024-05-15T23:34:03.613 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.613 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:12.614914+0000", 2024-05-15T23:34:03.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:12.614914+0000", 2024-05-15T23:34:03.614 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.614 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.614 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.614 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.614 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:12.614916+0000", 2024-05-15T23:34:03.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.852e-06 2024-05-15T23:34:03.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:12.614921+0000", 2024-05-15T23:34:03.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2390000000000002e-06 2024-05-15T23:34:03.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.615 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.615 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.615 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:12.614933+0000", 2024-05-15T23:34:03.615 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1749999999999999e-05 2024-05-15T23:34:03.615 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:12.614939+0000", 2024-05-15T23:34:03.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5719999999999999e-06 2024-05-15T23:34:03.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:12.614989+0000", 2024-05-15T23:34:03.616 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0297000000000003e-05 2024-05-15T23:34:03.616 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.616 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.616 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.616 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:12.615071+0000", 2024-05-15T23:34:03.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1168000000000006e-05 2024-05-15T23:34:03.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.617 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:34:03.617 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:12.615139+0000", 2024-05-15T23:34:03.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8100000000000002e-05 2024-05-15T23:34:03.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.617 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.617 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.617 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:12.627730+0000", 2024-05-15T23:34:03.617 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.012591803 2024-05-15T23:34:03.617 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:12.651980+0000", 2024-05-15T23:34:03.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024249975 2024-05-15T23:34:03.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.618 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:12.652029+0000", 2024-05-15T23:34:03.618 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8903000000000003e-05 2024-05-15T23:34:03.618 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.618 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.618 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.619 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:12.652069+0000", 2024-05-15T23:34:03.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0064000000000001e-05 2024-05-15T23:34:03.619 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.619 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.619 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.619 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.619 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.619 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:254975 6.e 6:72e4c363:::benchmark_data_smithi090_131278_object119781:head [delete] snapc 0=[] ondisk+write+known_if_redirected e540)", 2024-05-15T23:34:03.619 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:28:20.050617+0000", 2024-05-15T23:34:03.619 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 343.44390007099997, 2024-05-15T23:34:03.619 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.038020617999999999, 2024-05-15T23:34:03.619 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.619 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 254975 2024-05-15T23:34:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.050617+0000", 2024-05-15T23:34:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.620 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.050617+0000", 2024-05-15T23:34:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.050618+0000", 2024-05-15T23:34:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3880000000000001e-06 2024-05-15T23:34:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.621 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.050623+0000", 2024-05-15T23:34:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.011e-06 2024-05-15T23:34:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.050634+0000", 2024-05-15T23:34:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0881e-05 2024-05-15T23:34:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.050642+0000", 2024-05-15T23:34:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4080000000000001e-06 2024-05-15T23:34:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.065257+0000", 2024-05-15T23:34:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.014615704 2024-05-15T23:34:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.065386+0000", 2024-05-15T23:34:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012814099999999999 2024-05-15T23:34:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.623 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:34:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.065489+0000", 2024-05-15T23:34:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000103451 2024-05-15T23:34:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.079250+0000", 2024-05-15T23:34:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.013760807999999999 2024-05-15T23:34:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.088551+0000", 2024-05-15T23:34:03.624 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0093008210000000008 2024-05-15T23:34:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.088607+0000", 2024-05-15T23:34:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6660000000000003e-05 2024-05-15T23:34:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.088638+0000", 2024-05-15T23:34:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0345000000000002e-05 2024-05-15T23:34:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:255561 6.2 6:4ce57a52:::benchmark_data_smithi090_131278_object120367:head [delete] snapc 0=[] ondisk+write+known_if_redirected e540)", 2024-05-15T23:34:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:28:20.583445+0000", 2024-05-15T23:34:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 342.91107241700001, 2024-05-15T23:34:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.037410044000000003, 2024-05-15T23:34:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.626 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 255561 2024-05-15T23:34:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.583445+0000", 2024-05-15T23:34:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.583445+0000", 2024-05-15T23:34:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.627 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.583445+0000", 2024-05-15T23:34:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7000000000000004e-07 2024-05-15T23:34:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.583448+0000", 2024-05-15T23:34:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8490000000000002e-06 2024-05-15T23:34:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.628 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.583454+0000", 2024-05-15T23:34:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7520000000000002e-06 2024-05-15T23:34:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.583460+0000", 2024-05-15T23:34:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6370000000000004e-06 2024-05-15T23:34:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.583500+0000", 2024-05-15T23:34:03.629 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0219000000000002e-05 2024-05-15T23:34:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.583601+0000", 2024-05-15T23:34:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000101171 2024-05-15T23:34:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:34:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.583701+0000", 2024-05-15T23:34:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000100272 2024-05-15T23:34:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.630 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.592552+0000", 2024-05-15T23:34:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0088502189999999994 2024-05-15T23:34:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.620751+0000", 2024-05-15T23:34:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028199701000000001 2024-05-15T23:34:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.620804+0000", 2024-05-15T23:34:03.631 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3004999999999998e-05 2024-05-15T23:34:03.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.632 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.632 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.620855+0000", 2024-05-15T23:34:03.632 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0448999999999998e-05 2024-05-15T23:34:03.632 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.632 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.632 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.632 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.632 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.632 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:255566 6.2 6:47ca4ed5:::benchmark_data_smithi090_131278_object120372:head [delete] snapc 0=[] ondisk+write+known_if_redirected e540)", 2024-05-15T23:34:03.632 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:28:20.583851+0000", 2024-05-15T23:34:03.632 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 342.91066574500002, 2024-05-15T23:34:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044170731999999997, 2024-05-15T23:34:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 255566 2024-05-15T23:34:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.633 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.583851+0000", 2024-05-15T23:34:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.583851+0000", 2024-05-15T23:34:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.583852+0000", 2024-05-15T23:34:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.001e-06 2024-05-15T23:34:03.634 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.583856+0000", 2024-05-15T23:34:03.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0910000000000003e-06 2024-05-15T23:34:03.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.583865+0000", 2024-05-15T23:34:03.635 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.799e-06 2024-05-15T23:34:03.635 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.635 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.635 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.635 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.583871+0000", 2024-05-15T23:34:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1319999999999999e-06 2024-05-15T23:34:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.588805+0000", 2024-05-15T23:34:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0049340019999999998 2024-05-15T23:34:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.588901+0000", 2024-05-15T23:34:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5964000000000005e-05 2024-05-15T23:34:03.636 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:34:03.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.588974+0000", 2024-05-15T23:34:03.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2495000000000003e-05 2024-05-15T23:34:03.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.596378+0000", 2024-05-15T23:34:03.637 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0074045639999999998 2024-05-15T23:34:03.637 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.637 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.637 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.637 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.627917+0000", 2024-05-15T23:34:03.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031538608000000003 2024-05-15T23:34:03.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.627976+0000", 2024-05-15T23:34:03.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9407999999999998e-05 2024-05-15T23:34:03.638 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.638 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.638 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.638 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.628022+0000", 2024-05-15T23:34:03.638 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5667999999999997e-05 2024-05-15T23:34:03.638 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.639 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.639 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.639 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.639 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.639 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.26326.0:255568 6.2 6:4c2a0c5f:::benchmark_data_smithi090_131278_object120374:head [delete] snapc 0=[] ondisk+write+known_if_redirected e540)", 2024-05-15T23:34:03.639 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:28:20.588931+0000", 2024-05-15T23:34:03.639 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 342.90558582099999, 2024-05-15T23:34:03.639 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.039186250999999998, 2024-05-15T23:34:03.639 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:34:03.639 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:34:03.639 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:34:03.639 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:34:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.26326", 2024-05-15T23:34:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:34:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 255568 2024-05-15T23:34:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:34:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:34:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.588931+0000", 2024-05-15T23:34:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.640 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:34:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.588931+0000", 2024-05-15T23:34:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:34:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:34:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.588933+0000", 2024-05-15T23:34:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.1440000000000001e-06 2024-05-15T23:34:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:34:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.588939+0000", 2024-05-15T23:34:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1649999999999997e-06 2024-05-15T23:34:03.641 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:34:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.588948+0000", 2024-05-15T23:34:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0319999999999992e-06 2024-05-15T23:34:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:34:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.588955+0000", 2024-05-15T23:34:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3629999999999996e-06 2024-05-15T23:34:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.642 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:34:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.592586+0000", 2024-05-15T23:34:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036306989999999998 2024-05-15T23:34:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:34:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.592706+0000", 2024-05-15T23:34:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000120816 2024-05-15T23:34:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:34:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.592795+0000", 2024-05-15T23:34:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8782000000000005e-05 2024-05-15T23:34:03.643 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:34:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.596525+0000", 2024-05-15T23:34:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037293270000000002 2024-05-15T23:34:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:34:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.628059+0000", 2024-05-15T23:34:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031534768999999997 2024-05-15T23:34:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:34:03.644 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.628091+0000", 2024-05-15T23:34:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2067000000000003e-05 2024-05-15T23:34:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:34:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:34:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:34:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:28:20.628118+0000", 2024-05-15T23:34:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6086999999999999e-05 2024-05-15T23:34:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:34:03.645 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:34:03.645 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:34:03.683 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:03.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:34:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:03 smithi084 ceph-mon[39784]: osdmap e656: 12 total, 12 up, 10 in 2024-05-15T23:34:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:34:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:34:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:03 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:34:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:03 smithi084 ceph-mon[39784]: pgmap v1738: 17 pgs: 4 creating+peering, 3 unknown, 10 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:04.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:34:04.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:03 smithi088 ceph-mon[44699]: osdmap e656: 12 total, 12 up, 10 in 2024-05-15T23:34:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:34:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:34:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:03 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:34:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:03 smithi088 ceph-mon[44699]: pgmap v1738: 17 pgs: 4 creating+peering, 3 unknown, 10 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:04.230 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T23:34:04.281 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:34:04.281 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:03 smithi090 ceph-mon[44434]: osdmap e656: 12 total, 12 up, 10 in 2024-05-15T23:34:04.281 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:34:04.281 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:34:04.281 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:03 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:34:04.281 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:03 smithi090 ceph-mon[44434]: pgmap v1738: 17 pgs: 4 creating+peering, 3 unknown, 10 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:04.479 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:34:04.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:04.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:34:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:34:04.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:34:04.613 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:04.953 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:04 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:34:04.953 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:04 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:34:05.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-05-15T23:34:05.062 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:04 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:34:05.062 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:04 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:34:05.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:04 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:34:05.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:04 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:34:05.690 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:34:05.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:05.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:34:05.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:34:05.827 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:05.843 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:05 smithi088 ceph-mon[44699]: osdmap e657: 12 total, 12 up, 10 in 2024-05-15T23:34:05.843 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:05 smithi088 ceph-mon[44699]: pgmap v1740: 17 pgs: 4 creating+peering, 13 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:05.848 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:34:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:05.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:34:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:05 smithi084 ceph-mon[39784]: osdmap e657: 12 total, 12 up, 10 in 2024-05-15T23:34:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:05 smithi084 ceph-mon[39784]: pgmap v1740: 17 pgs: 4 creating+peering, 13 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:05.952 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:34:05.982 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:06.230 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:34:06.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:06.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:34:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:05 smithi090 ceph-mon[44434]: osdmap e657: 12 total, 12 up, 10 in 2024-05-15T23:34:06.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:05 smithi090 ceph-mon[44434]: pgmap v1740: 17 pgs: 4 creating+peering, 13 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:06.334 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:34:06.362 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:06.459 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:34:06.460 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:34:06.460 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:34:06.460 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2024-05-15T23:34:06.460 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:34:06.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:06.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:34:06.543 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:06.544 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:06.562 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:34:06.593 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:06.828 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:34:06.981 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:06 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1070125016' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:34:07.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:06 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1070125016' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:34:07.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:06 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1070125016' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:34:07.209 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T23:34:07.469 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:34:07.742 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:34:07.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:07.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:34:07.847 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:34:07.876 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:07.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:07 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:34:07.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:07 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1070125016' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:34:07.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:07 smithi088 ceph-mon[44699]: osdmap e658: 12 total, 12 up, 10 in 2024-05-15T23:34:07.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:34:07.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:34:07.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:34:07.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:34:07.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:34:07.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:07 smithi088 ceph-mon[44699]: pgmap v1742: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:08.011 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:34:08.030 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:34:08.048 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:34:08.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:07 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:34:08.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:07 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1070125016' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:34:08.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:07 smithi084 ceph-mon[39784]: osdmap e658: 12 total, 12 up, 10 in 2024-05-15T23:34:08.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:34:08.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:34:08.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:34:08.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:34:08.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:34:08.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:07 smithi084 ceph-mon[39784]: pgmap v1742: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:08.068 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:34:08.088 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:34:08.108 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:34:08.127 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:34:08.147 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:34:08.166 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:34:08.186 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:34:08.206 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:34:08.226 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:34:08.227 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:34:08.227 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:34:08.227 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:34:08.227 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:34:08.227 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:34:08.227 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:34:08.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:34:08.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:34:08.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:34:08.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:34:08.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:34:08.228 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:34:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:07 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:34:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:07 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1070125016' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:34:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:07 smithi090 ceph-mon[44434]: osdmap e658: 12 total, 12 up, 10 in 2024-05-15T23:34:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:34:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:34:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:34:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:34:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:34:08.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:07 smithi090 ceph-mon[44434]: pgmap v1742: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:08.996 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2843792616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:08.996 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:08.996 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:08.996 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:08.996 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:08.996 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:08.996 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:08.996 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:08.997 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.025 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:34:09.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:09.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:34:09.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2843792616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:09.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:34:09.161 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:09.214 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:34:09.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:09.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:34:09.293 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:34:09.294 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:34:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2843792616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:09.318 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:34:09.347 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:09.549 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:34:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:09.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:34:09.653 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:34:09.682 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:09.805 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:34:09.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:09.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:34:09.829 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-05-15T23:34:09.908 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:34:09.940 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:10.038 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:10.038 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:10.038 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:10.038 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:10.038 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:09 smithi088 ceph-mon[44699]: pgmap v1743: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:10.065 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T23:34:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:10.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:09 smithi084 ceph-mon[39784]: pgmap v1743: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:09 smithi090 ceph-mon[44434]: pgmap v1743: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:10.325 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:34:11.391 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:34:11.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:11.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:34:11.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:34:11.523 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:11.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:11 smithi088 ceph-mon[44699]: pgmap v1744: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:11.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:11 smithi084 ceph-mon[39784]: pgmap v1744: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:11 smithi090 ceph-mon[44434]: pgmap v1744: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:12.155 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:12.172 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:34:12.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:12.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:34:12.198 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:34:12.198 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:34:12.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:34:12.314 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:12.337 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:34:12.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:12.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:34:12.441 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:34:12.472 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:12.545 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:34:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:12.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:34:12.651 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:34:12.678 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:12.926 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:34:12.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:12.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:34:12.937 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T23:34:13.030 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:34:13.055 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:13.077 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:34:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:13.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:34:13.132 INFO:teuthology.orchestra.run.smithi084.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-05-15T23:34:13.181 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:34:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:13.236 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:34:13.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:13.239 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:34:13.249 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:34:13.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4021982412' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:34:13.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:34:13.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4021982412' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:34:13.370 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:13.390 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:34:13.392 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4021982412' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:34:13.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:13.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:34:13.424 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:34:13.494 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:34:13.526 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:13.545 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:34:13.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:13.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:34:13.648 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:34:13.677 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:13.696 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:34:13.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:13.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:34:13.800 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:34:13.827 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:14.103 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:34:14.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:14.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:34:14.140 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:34:14.211 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:34:14.237 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:14.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:14 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4021982412' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:34:14.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:14 smithi084 ceph-mon[39784]: osdmap e659: 12 total, 12 up, 10 in 2024-05-15T23:34:14.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:14 smithi084 ceph-mon[39784]: pgmap v1746: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:14.485 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:34:14.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:14.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:34:14.545 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:14.547 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:14 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4021982412' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:34:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:14 smithi090 ceph-mon[44434]: osdmap e659: 12 total, 12 up, 10 in 2024-05-15T23:34:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:14 smithi090 ceph-mon[44434]: pgmap v1746: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:14.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:14 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4021982412' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:34:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:14 smithi088 ceph-mon[44699]: osdmap e659: 12 total, 12 up, 10 in 2024-05-15T23:34:14.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:14 smithi088 ceph-mon[44699]: pgmap v1746: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:14.591 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:34:14.625 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:14.646 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:34:14.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:14.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:34:14.751 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:34:14.778 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:14.980 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:34:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:14.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:34:15.008 DEBUG:teuthology.orchestra.run.smithi090:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-05-15T23:34:15.011 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T23:34:15.011 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T23:34:15.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:34:15.110 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:hints = 1 2024-05-15T23:34:15.110 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T23:34:15.110 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object prefix: benchmark_data_smithi090_157496 2024-05-15T23:34:15.111 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:34:15.111 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 0 0 0 0 0 0 - 0 2024-05-15T23:34:15.116 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:15.148 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:34:15.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:15.155 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:34:15.242 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:34:15.242 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:34:15.256 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:34:15.265 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:15 smithi088 ceph-mon[44699]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:34:15.294 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:15.446 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:15 smithi084 ceph-mon[39784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:34:15.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:15 smithi090 ceph-mon[44434]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:34:16.022 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T23:34:16.110 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 1 16 4953 4937 308.537 308.562 0.00178606 0.00323268 2024-05-15T23:34:16.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:16 smithi088 ceph-mon[44699]: pgmap v1747: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:16.343 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:34:16.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:16 smithi084 ceph-mon[39784]: pgmap v1747: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:16 smithi090 ceph-mon[44434]: pgmap v1747: 17 pgs: 17 active+clean; 577 KiB data, 5.1 GiB used, 889 GiB / 894 GiB avail 2024-05-15T23:34:17.111 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 2 16 9510 9494 296.6 284.812 0.00276918 0.00336581 2024-05-15T23:34:17.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:17 smithi084 ceph-mon[39784]: pgmap v1748: 17 pgs: 17 active+clean; 144 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 14 MiB/s wr, 228 op/s 2024-05-15T23:34:17.703 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-05-15T23:34:17.718 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:17 smithi088 ceph-mon[44699]: pgmap v1748: 17 pgs: 17 active+clean; 144 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 14 MiB/s wr, 228 op/s 2024-05-15T23:34:17.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:17 smithi090 ceph-mon[44434]: pgmap v1748: 17 pgs: 17 active+clean; 144 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 14 MiB/s wr, 228 op/s 2024-05-15T23:34:17.953 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:34:17.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:17.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:34:18.057 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:34:18.106 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:18.111 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 3 16 13627 13611 283.468 257.312 0.00362799 0.00352358 2024-05-15T23:34:18.176 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:34:18.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:18.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:34:18.226 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:34:18.254 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:34:18.280 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:34:18.290 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:34:18.308 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:34:18.328 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:18.332 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:34:18.340 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:18 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1425587043' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:18.360 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:34:18.389 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:34:18.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:18.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:34:18.393 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:34:18.419 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:34:18.444 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:34:18.445 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:34:18.445 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:34:18.445 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:34:18.445 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:34:18.447 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:34:18.474 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:34:18.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:34:18.505 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:34:18.521 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:18.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:18 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1425587043' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:18.557 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:34:18.558 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:34:18.558 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:34:18.558 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:34:18.558 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:34:18.558 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:34:18.558 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:34:18.558 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:34:18.558 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:34:18.558 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:34:18.559 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:34:18.559 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:34:18.559 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:34:18.581 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:18 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1425587043' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:18.776 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:34:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:18.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:34:18.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:34:18.917 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:19.112 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 4 16 17398 17382 271.499 235.688 0.00315121 0.00367995 2024-05-15T23:34:19.200 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:34:19.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:19.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:34:19.262 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T23:34:19.305 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:34:19.340 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:19.470 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:34:19.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:19 smithi084 ceph-mon[39784]: pgmap v1749: 17 pgs: 17 active+clean; 582 MiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 58 MiB/s wr, 930 op/s 2024-05-15T23:34:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:19.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:34:19.488 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:19 smithi088 ceph-mon[44699]: pgmap v1749: 17 pgs: 17 active+clean; 582 MiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 58 MiB/s wr, 930 op/s 2024-05-15T23:34:19.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:34:19.593 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:34:19.628 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:19.629 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:19 smithi090 ceph-mon[44434]: pgmap v1749: 17 pgs: 17 active+clean; 582 MiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 58 MiB/s wr, 930 op/s 2024-05-15T23:34:19.635 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:19.668 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:34:19.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:19.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:34:19.773 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:34:19.810 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:19.885 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:34:19.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:19.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:34:19.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:34:20.034 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:20.112 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 5 16 20957 20941 261.637 222.438 0.00311559 0.00381794 2024-05-15T23:34:20.385 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:34:20.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:20.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:34:20.491 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:34:20.522 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:20.643 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:34:20.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:20 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3273964563' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:34:20.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:20 smithi088 ceph-mon[44699]: from='client.36989 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:34:21.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:20 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3273964563' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:34:21.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:20 smithi090 ceph-mon[44434]: from='client.36989 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:34:21.113 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 6 16 24397 24381 253.861 215 0.00435535 0.00393671 2024-05-15T23:34:21.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:20 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3273964563' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:34:21.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:20 smithi084 ceph-mon[39784]: from='client.36989 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:34:21.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:34:21.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:21 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:34:21.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:21 smithi084 ceph-mon[39784]: from='client.36989 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:34:21.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:21 smithi084 ceph-mon[39784]: osdmap e660: 12 total, 12 up, 10 in 2024-05-15T23:34:21.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:21 smithi084 ceph-mon[39784]: pgmap v1751: 17 pgs: 17 active+clean; 964 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-05-15T23:34:22.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:21 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:34:22.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:21 smithi090 ceph-mon[44434]: from='client.36989 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:34:22.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:21 smithi090 ceph-mon[44434]: osdmap e660: 12 total, 12 up, 10 in 2024-05-15T23:34:22.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:21 smithi090 ceph-mon[44434]: pgmap v1751: 17 pgs: 17 active+clean; 964 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-05-15T23:34:22.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:21 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:34:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:21 smithi088 ceph-mon[44699]: from='client.36989 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:34:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:21 smithi088 ceph-mon[44699]: osdmap e660: 12 total, 12 up, 10 in 2024-05-15T23:34:22.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:21 smithi088 ceph-mon[44699]: pgmap v1751: 17 pgs: 17 active+clean; 964 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-05-15T23:34:22.113 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 7 16 27486 27470 245.165 193.062 0.00323197 0.0040762 2024-05-15T23:34:22.176 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:34:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:22.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:34:22.280 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:34:22.314 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:22.442 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:34:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:22.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:34:22.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:34:22.595 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:22.613 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:22.629 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:34:22.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:22.636 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:34:22.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:34:22.773 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:22.809 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:34:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:22.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:34:22.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:34:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:22.995 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:34:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:23.003 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:34:23.104 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:34:23.114 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 8 16 30516 30500 238.172 189.375 0.00585883 0.00419614 2024-05-15T23:34:23.136 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:23.159 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:34:23.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:23.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:34:23.263 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:34:23.305 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:23.341 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:34:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:23.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:34:23.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:34:23.490 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:23.517 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:34:23.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:23.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:34:23.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:23 smithi090 ceph-mon[44434]: pgmap v1752: 17 pgs: 17 active+clean; 1.6 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-05-15T23:34:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:23 smithi088 ceph-mon[44699]: pgmap v1752: 17 pgs: 17 active+clean; 1.6 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-05-15T23:34:23.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:23 smithi084 ceph-mon[39784]: pgmap v1752: 17 pgs: 17 active+clean; 1.6 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-05-15T23:34:23.621 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:34:23.621 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:34:23.621 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:34:23.622 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2024-05-15T23:34:23.622 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:34:23.623 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:34:23.658 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:24.038 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:34:24.114 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 9 16 33455 33439 232.118 183.688 0.00842368 0.00430556 2024-05-15T23:34:24.210 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:34:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:24.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:34:24.314 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:34:24.345 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:25.114 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 10 16 36313 36297 226.769 178.625 0.00324189 0.00440777 2024-05-15T23:34:25.123 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T23:34:25.133 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:34:25.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:25.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:34:25.237 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:34:25.271 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:25.299 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:34:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:25.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:34:25.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:34:25.439 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:25.467 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:34:25.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:25 smithi088 ceph-mon[44699]: pgmap v1753: 17 pgs: 17 active+clean; 1.7 GiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 177 MiB/s wr, 2.84k op/s 2024-05-15T23:34:25.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:25 smithi084 ceph-mon[39784]: pgmap v1753: 17 pgs: 17 active+clean; 1.7 GiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 177 MiB/s wr, 2.84k op/s 2024-05-15T23:34:25.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:25 smithi090 ceph-mon[44434]: pgmap v1753: 17 pgs: 17 active+clean; 1.7 GiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 177 MiB/s wr, 2.84k op/s 2024-05-15T23:34:26.114 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 11 16 39089 39073 221.927 173.5 0.00231669 0.00450379 2024-05-15T23:34:27.116 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 12 15 41658 41643 216.791 160.625 0.011555 0.00461083 2024-05-15T23:34:27.244 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-05-15T23:34:27.292 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:34:27.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:27.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:34:27.397 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:34:27.432 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:27.455 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:34:27.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:27.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:34:27.560 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:34:27.592 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:27.617 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:34:27.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:27.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:34:27.683 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:34:27.683 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:34:27.683 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2024-05-15T23:34:27.683 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:34:27.721 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:34:27.761 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:27.789 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:28.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:34:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:34:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:27 smithi090 ceph-mon[44434]: osdmap e661: 12 total, 12 up, 10 in 2024-05-15T23:34:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:27 smithi090 ceph-mon[44434]: pgmap v1755: 17 pgs: 17 active+clean; 2.1 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 191 MiB/s wr, 3.06k op/s 2024-05-15T23:34:28.065 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:34:28.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:34:28.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:27 smithi084 ceph-mon[39784]: osdmap e661: 12 total, 12 up, 10 in 2024-05-15T23:34:28.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:27 smithi084 ceph-mon[39784]: pgmap v1755: 17 pgs: 17 active+clean; 2.1 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 191 MiB/s wr, 3.06k op/s 2024-05-15T23:34:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:34:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:34:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:27 smithi088 ceph-mon[44699]: osdmap e661: 12 total, 12 up, 10 in 2024-05-15T23:34:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:27 smithi088 ceph-mon[44699]: pgmap v1755: 17 pgs: 17 active+clean; 2.1 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 191 MiB/s wr, 3.06k op/s 2024-05-15T23:34:28.116 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 13 16 44246 44230 212.552 161.688 0.00577715 0.00470262 2024-05-15T23:34:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:34:28.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:28.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:34:28.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:34:28.964 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:28.996 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:34:29.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:29.005 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:34:29.106 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:34:29.118 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 14 16 46769 46753 208.606 157.688 0.0115315 0.00479044 2024-05-15T23:34:29.156 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:29.191 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:34:29.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:29.198 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:34:29.299 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:34:29.330 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T23:34:29.338 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:29.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:29 smithi088 ceph-mon[44699]: pgmap v1756: 17 pgs: 17 active+clean; 2.5 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 226 MiB/s wr, 3.61k op/s 2024-05-15T23:34:29.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:29 smithi084 ceph-mon[39784]: pgmap v1756: 17 pgs: 17 active+clean; 2.5 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 226 MiB/s wr, 3.61k op/s 2024-05-15T23:34:29.629 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:34:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:29 smithi090 ceph-mon[44434]: pgmap v1756: 17 pgs: 17 active+clean; 2.5 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 226 MiB/s wr, 3.61k op/s 2024-05-15T23:34:30.119 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 15 16 48877 48861 203.474 131.75 0.00420902 0.00491243 2024-05-15T23:34:30.122 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:34:30.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:30.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:34:30.226 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:34:30.261 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:30.368 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:30.421 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:34:30.446 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:34:30.478 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:34:30.504 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:34:30.534 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:34:30.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:34:30.587 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:34:30.616 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:34:30.655 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:34:30.688 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:34:30.717 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:34:30.746 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:34:30.746 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:34:30.746 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:34:30.747 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:34:30.747 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:34:30.747 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:34:30.747 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:34:30.747 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:34:30.747 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:34:30.747 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:34:30.747 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:34:30.747 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:34:30.747 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:34:30.769 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:30 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1027221033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:30 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1027221033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:30.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:30 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1027221033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:31.015 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:34:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:31.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:34:31.119 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:34:31.119 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 16 15 51249 51234 200.026 148.312 0.00338958 0.00499803 2024-05-15T23:34:31.160 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:31.315 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:34:31.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:31.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:34:31.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:34:31.449 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.450 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.450 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872955322265625]}]: dispatch 2024-05-15T23:34:31.450 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10162714320340317]}]: dispatch 2024-05-15T23:34:31.450 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.1089795271252694]}]: dispatch 2024-05-15T23:34:31.450 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.12466008122195914]}]: dispatch 2024-05-15T23:34:31.450 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08749702190754423]}]: dispatch 2024-05-15T23:34:31.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10162714320340317]}]: dispatch 2024-05-15T23:34:31.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872955322265625]}]: dispatch 2024-05-15T23:34:31.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.1089795271252694]}]: dispatch 2024-05-15T23:34:31.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.1089795271252694]}]: dispatch 2024-05-15T23:34:31.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.1089795271252694]}]: dispatch 2024-05-15T23:34:31.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.10162714320340317]}]: dispatch 2024-05-15T23:34:31.452 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09458974547795972]}]: dispatch 2024-05-15T23:34:31.452 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"4\": \"26333\"}"}]: dispatch 2024-05-15T23:34:31.452 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.452 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.452 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.452 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.453 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.453 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.453 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.453 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: pgmap v1757: 17 pgs: 17 active+clean; 2.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 188 MiB/s wr, 3.00k op/s 2024-05-15T23:34:31.453 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4099850340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:34:31.453 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:31 smithi084 ceph-mon[39784]: from='client.37007 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:34:31.462 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:34:31.468 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:31.501 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:34:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:31.505 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:34:31.606 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:34:31.637 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872955322265625]}]: dispatch 2024-05-15T23:34:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10162714320340317]}]: dispatch 2024-05-15T23:34:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.1089795271252694]}]: dispatch 2024-05-15T23:34:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.12466008122195914]}]: dispatch 2024-05-15T23:34:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08749702190754423]}]: dispatch 2024-05-15T23:34:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10162714320340317]}]: dispatch 2024-05-15T23:34:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872955322265625]}]: dispatch 2024-05-15T23:34:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.1089795271252694]}]: dispatch 2024-05-15T23:34:31.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.1089795271252694]}]: dispatch 2024-05-15T23:34:31.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.1089795271252694]}]: dispatch 2024-05-15T23:34:31.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.10162714320340317]}]: dispatch 2024-05-15T23:34:31.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09458974547795972]}]: dispatch 2024-05-15T23:34:31.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"4\": \"26333\"}"}]: dispatch 2024-05-15T23:34:31.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: pgmap v1757: 17 pgs: 17 active+clean; 2.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 188 MiB/s wr, 3.00k op/s 2024-05-15T23:34:31.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4099850340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:34:31.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:31 smithi090 ceph-mon[44434]: from='client.37007 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:34:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872955322265625]}]: dispatch 2024-05-15T23:34:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10162714320340317]}]: dispatch 2024-05-15T23:34:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.1089795271252694]}]: dispatch 2024-05-15T23:34:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.12466008122195914]}]: dispatch 2024-05-15T23:34:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08749702190754423]}]: dispatch 2024-05-15T23:34:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10162714320340317]}]: dispatch 2024-05-15T23:34:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872955322265625]}]: dispatch 2024-05-15T23:34:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.1089795271252694]}]: dispatch 2024-05-15T23:34:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.1089795271252694]}]: dispatch 2024-05-15T23:34:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.1089795271252694]}]: dispatch 2024-05-15T23:34:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.10162714320340317]}]: dispatch 2024-05-15T23:34:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09458974547795972]}]: dispatch 2024-05-15T23:34:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"4\": \"26333\"}"}]: dispatch 2024-05-15T23:34:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:31.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: pgmap v1757: 17 pgs: 17 active+clean; 2.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 188 MiB/s wr, 3.00k op/s 2024-05-15T23:34:31.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4099850340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:34:31.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:31 smithi088 ceph-mon[44699]: from='client.37007 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:34:32.119 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 17 16 53577 53561 196.815 145.438 0.00794236 0.00507855 2024-05-15T23:34:32.302 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:34:32.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:32.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:34:32.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:34:32.435 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:32.495 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:34:32.495 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:34:32.559 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:34:32.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:32.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:32.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:32.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:34:32.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:34:32.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:34:32.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872955322265625]}]': finished 2024-05-15T23:34:32.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10162714320340317]}]': finished 2024-05-15T23:34:32.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.1089795271252694]}]': finished 2024-05-15T23:34:32.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.12466008122195914]}]': finished 2024-05-15T23:34:32.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08749702190754423]}]': finished 2024-05-15T23:34:32.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10162714320340317]}]': finished 2024-05-15T23:34:32.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872955322265625]}]': finished 2024-05-15T23:34:32.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.1089795271252694]}]': finished 2024-05-15T23:34:32.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.1089795271252694]}]': finished 2024-05-15T23:34:32.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.1089795271252694]}]': finished 2024-05-15T23:34:32.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.10162714320340317]}]': finished 2024-05-15T23:34:32.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09458974547795972]}]': finished 2024-05-15T23:34:32.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"4\": \"26333\"}"}]': finished 2024-05-15T23:34:32.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: from='client.37007 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:34:32.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: osdmap e662: 12 total, 12 up, 10 in 2024-05-15T23:34:32.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:32 smithi084 ceph-mon[39784]: osdmap e663: 12 total, 12 up, 10 in 2024-05-15T23:34:32.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:32.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:34:32.665 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:34:32.698 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:32.698 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:32.698 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:32.698 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:34:32.698 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:34:32.698 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:34:32.698 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872955322265625]}]': finished 2024-05-15T23:34:32.698 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10162714320340317]}]': finished 2024-05-15T23:34:32.698 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.1089795271252694]}]': finished 2024-05-15T23:34:32.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.12466008122195914]}]': finished 2024-05-15T23:34:32.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08749702190754423]}]': finished 2024-05-15T23:34:32.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10162714320340317]}]': finished 2024-05-15T23:34:32.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872955322265625]}]': finished 2024-05-15T23:34:32.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.1089795271252694]}]': finished 2024-05-15T23:34:32.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.1089795271252694]}]': finished 2024-05-15T23:34:32.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.1089795271252694]}]': finished 2024-05-15T23:34:32.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.10162714320340317]}]': finished 2024-05-15T23:34:32.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09458974547795972]}]': finished 2024-05-15T23:34:32.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"4\": \"26333\"}"}]': finished 2024-05-15T23:34:32.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: from='client.37007 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:34:32.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: osdmap e662: 12 total, 12 up, 10 in 2024-05-15T23:34:32.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:32 smithi088 ceph-mon[44699]: osdmap e663: 12 total, 12 up, 10 in 2024-05-15T23:34:32.700 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:32.812 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:32.812 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:32.812 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:34:32.812 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:34:32.812 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:34:32.812 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872955322265625]}]': finished 2024-05-15T23:34:32.812 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10162714320340317]}]': finished 2024-05-15T23:34:32.812 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.1089795271252694]}]': finished 2024-05-15T23:34:32.812 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.12466008122195914]}]': finished 2024-05-15T23:34:32.813 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08749702190754423]}]': finished 2024-05-15T23:34:32.813 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10162714320340317]}]': finished 2024-05-15T23:34:32.813 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872955322265625]}]': finished 2024-05-15T23:34:32.813 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.1089795271252694]}]': finished 2024-05-15T23:34:32.813 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.1089795271252694]}]': finished 2024-05-15T23:34:32.813 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.1089795271252694]}]': finished 2024-05-15T23:34:32.813 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.10162714320340317]}]': finished 2024-05-15T23:34:32.813 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09458974547795972]}]': finished 2024-05-15T23:34:32.813 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"4\": \"26333\"}"}]': finished 2024-05-15T23:34:32.813 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: from='client.37007 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:34:32.813 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: osdmap e662: 12 total, 12 up, 10 in 2024-05-15T23:34:32.813 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:32 smithi090 ceph-mon[44434]: osdmap e663: 12 total, 12 up, 10 in 2024-05-15T23:34:33.119 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 18 16 55493 55477 192.534 119.75 0.00233188 0.00517399 2024-05-15T23:34:33.352 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:34:33.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:33.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:34:33.458 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:34:33.494 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:33.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:33 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:34:33.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:33 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:34:33.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:33 smithi084 ceph-mon[39784]: pgmap v1760: 17 pgs: 17 active+clean; 3.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-05-15T23:34:33.751 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:34:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:33.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:34:33.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:33 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:34:33.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:33 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:34:33.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:33 smithi090 ceph-mon[44434]: pgmap v1760: 17 pgs: 17 active+clean; 3.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-05-15T23:34:33.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:33 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:34:33.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:33 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:34:33.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:33 smithi088 ceph-mon[44699]: pgmap v1760: 17 pgs: 17 active+clean; 3.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-05-15T23:34:33.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:34:33.890 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:33.930 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:34:33.958 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T23:34:34.094 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:34:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:34.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:34:34.119 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 19 16 57591 57575 189.303 131.125 0.00490538 0.00528017 2024-05-15T23:34:34.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:34:34.248 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:34.272 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:34:34.274 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:34:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:34.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:34:34.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:34:34.432 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:34:34.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:34.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:34:34.573 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:34:34.608 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:34.635 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:34:34.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:34.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:34:34.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:34:34.771 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:34.804 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:34:34.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:34.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:34:34.825 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:34.829 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-05-15T23:34:34.910 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:34:34.940 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:35.017 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:34:35.119 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:34:35.119677 min lat: 0.00114746 max lat: 0.212087 avg lat: 0.00537118 2024-05-15T23:34:35.119 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:34:35.119 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 20 16 59596 59580 186.104 125.312 0.0032745 0.00537118 2024-05-15T23:34:35.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:35 smithi084 ceph-mon[39784]: pgmap v1761: 17 pgs: 17 active+clean; 3.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 158 MiB/s wr, 2.52k op/s 2024-05-15T23:34:35.798 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:34:35.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:35.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:34:35.804 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:35 smithi088 ceph-mon[44699]: pgmap v1761: 17 pgs: 17 active+clean; 3.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 158 MiB/s wr, 2.52k op/s 2024-05-15T23:34:35.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:35 smithi090 ceph-mon[44434]: pgmap v1761: 17 pgs: 17 active+clean; 3.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 158 MiB/s wr, 2.52k op/s 2024-05-15T23:34:35.904 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:34:35.935 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:35.959 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:34:35.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:35.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:34:36.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:34:36.111 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:36.119 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 21 16 61371 61355 182.525 110.938 0.0066206 0.00547633 2024-05-15T23:34:36.143 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:34:36.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:36.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:34:36.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:34:36.299 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:36.343 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:34:36.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:36.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:34:36.451 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:34:36.487 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:36.512 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:34:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:36.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:34:36.617 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:34:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:36.724 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:34:36.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:36.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:34:36.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:34:36.889 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:36.923 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:34:36.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:36.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:34:37.029 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:34:37.064 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:37.066 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:37.091 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:34:37.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:37.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:34:37.119 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 22 16 63420 63404 180.05 128.062 0.00415607 0.00555203 2024-05-15T23:34:37.123 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:34:37.123 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:34:37.196 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:34:37.231 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:37.385 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:34:37.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:37.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:34:37.490 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:34:37.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:37 smithi084 ceph-mon[39784]: pgmap v1762: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-05-15T23:34:37.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:37 smithi084 ceph-mon[39784]: osdmap e664: 12 total, 12 up, 10 in 2024-05-15T23:34:37.524 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:37 smithi090 ceph-mon[44434]: pgmap v1762: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-05-15T23:34:37.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:37 smithi090 ceph-mon[44434]: osdmap e664: 12 total, 12 up, 10 in 2024-05-15T23:34:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:37 smithi088 ceph-mon[44699]: pgmap v1762: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-05-15T23:34:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:37 smithi088 ceph-mon[44699]: osdmap e664: 12 total, 12 up, 10 in 2024-05-15T23:34:37.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T23:34:38.119 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 23 15 65375 65360 177.537 122.25 0.0132289 0.00562972 2024-05-15T23:34:38.345 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:34:39.120 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 24 16 66898 66882 174.104 95.125 0.00317527 0.00573332 2024-05-15T23:34:39.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4096553950' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4096553950' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:39.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4096553950' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:39.578 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:34:39.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:39.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:34:39.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:34:39.717 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:39.776 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:34:39.776 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:34:39.776 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:34:39.776 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:34:39.776 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:34:39.777 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:34:39.777 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:34:39.777 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:34:39.777 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:34:39.778 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:34:39.778 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:34:39.778 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:34:39.778 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:34:39.779 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:34:39.779 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:34:39.779 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:34:39.779 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:34:39.779 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:34:39.779 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:34:39.780 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:34:39.780 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:34:39.780 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:34:39.780 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:34:39.780 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:34:39.978 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:40.121 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 25 16 68650 68634 171.512 109.5 0.00350806 0.00582889 2024-05-15T23:34:40.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:40 smithi084 ceph-mon[39784]: pgmap v1764: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-05-15T23:34:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:40 smithi090 ceph-mon[44434]: pgmap v1764: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-05-15T23:34:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.571 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:34:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:40 smithi088 ceph-mon[44699]: pgmap v1764: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-05-15T23:34:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:40.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:40.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:34:40.677 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:34:40.705 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:40.724 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:34:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:40.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:34:40.827 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:34:40.878 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:41.079 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:34:41.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:41.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:34:41.121 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 26 16 70354 70338 169.012 106.5 0.00726101 0.00591473 2024-05-15T23:34:41.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:34:41.225 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:41.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:41.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:41.381 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:34:41.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:41.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:34:41.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:34:41.524 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:41.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:41.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:41.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:42.121 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 27 16 72215 72199 167.06 116.312 0.00536778 0.00598425 2024-05-15T23:34:42.238 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:42 smithi090 ceph-mon[44434]: pgmap v1765: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 135 MiB/s wr, 2.15k op/s 2024-05-15T23:34:42.261 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-05-15T23:34:42.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:42 smithi088 ceph-mon[44699]: pgmap v1765: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 135 MiB/s wr, 2.15k op/s 2024-05-15T23:34:42.580 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:34:42.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:42 smithi084 ceph-mon[39784]: pgmap v1765: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 135 MiB/s wr, 2.15k op/s 2024-05-15T23:34:42.980 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:34:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:42.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:34:43.086 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:43.087 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:34:43.104 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:43.120 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:43.121 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 28 16 73989 73973 165.054 110.875 0.00628243 0.00605714 2024-05-15T23:34:43.256 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:34:43.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:43.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:34:43.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:34:43.482 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:43.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:43 smithi088 ceph-mon[44699]: osdmap e665: 12 total, 12 up, 10 in 2024-05-15T23:34:43.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:43 smithi088 ceph-mon[44699]: pgmap v1767: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-05-15T23:34:43.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:43 smithi084 ceph-mon[39784]: osdmap e665: 12 total, 12 up, 10 in 2024-05-15T23:34:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:43 smithi084 ceph-mon[39784]: pgmap v1767: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-05-15T23:34:43.638 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:34:43.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:43.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:34:43.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:34:43.775 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:43.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:43 smithi090 ceph-mon[44434]: osdmap e665: 12 total, 12 up, 10 in 2024-05-15T23:34:43.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:43 smithi090 ceph-mon[44434]: pgmap v1767: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-05-15T23:34:44.034 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:34:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:44.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:34:44.123 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 29 16 75791 75775 163.236 112.625 0.0106611 0.00612382 2024-05-15T23:34:44.138 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:34:44.170 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:44.191 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:34:44.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:44.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:34:44.297 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:34:44.319 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:34:44.319 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:34:44.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.319 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:76121 8.c 8:3d0be417:::benchmark_data_smithi090_157496_object76120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e665)", 2024-05-15T23:34:44.319 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:34:44.311758+0000", 2024-05-15T23:34:44.320 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0066252469999999999, 2024-05-15T23:34:44.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0066605969999999999, 2024-05-15T23:34:44.320 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:34:44.320 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:34:44.320 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:34:44.320 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:34:44.320 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:34:44.320 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:34:44.320 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 76121 2024-05-15T23:34:44.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.320 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:34:44.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:34:44.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311758+0000", 2024-05-15T23:34:44.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:34:44.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:34:44.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311758+0000", 2024-05-15T23:34:44.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:34:44.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:34:44.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311760+0000", 2024-05-15T23:34:44.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2520000000000002e-06 2024-05-15T23:34:44.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:34:44.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311795+0000", 2024-05-15T23:34:44.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5442999999999999e-05 2024-05-15T23:34:44.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:34:44.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311811+0000", 2024-05-15T23:34:44.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5866e-05 2024-05-15T23:34:44.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:34:44.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311820+0000", 2024-05-15T23:34:44.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9940000000000006e-06 2024-05-15T23:34:44.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311876+0000", 2024-05-15T23:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.592e-05 2024-05-15T23:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.312104+0000", 2024-05-15T23:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022772499999999999 2024-05-15T23:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 8", 2024-05-15T23:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.312243+0000", 2024-05-15T23:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139056 2024-05-15T23:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.316888+0000", 2024-05-15T23:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0046446530000000003 2024-05-15T23:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:76123 8.c 8:39603431:::benchmark_data_smithi090_157496_object76122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e665)", 2024-05-15T23:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:34:44.311844+0000", 2024-05-15T23:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0065390839999999997, 2024-05-15T23:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0066651510000000002, 2024-05-15T23:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 76123 2024-05-15T23:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311844+0000", 2024-05-15T23:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311844+0000", 2024-05-15T23:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311845+0000", 2024-05-15T23:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.02e-07 2024-05-15T23:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311867+0000", 2024-05-15T23:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2096999999999999e-05 2024-05-15T23:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311874+0000", 2024-05-15T23:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2969999999999998e-06 2024-05-15T23:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311881+0000", 2024-05-15T23:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7499999999999997e-06 2024-05-15T23:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.314720+0000", 2024-05-15T23:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028394169999999999 2024-05-15T23:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.314823+0000", 2024-05-15T23:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102237 2024-05-15T23:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 8", 2024-05-15T23:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.314951+0000", 2024-05-15T23:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128319 2024-05-15T23:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.17073.0:76114 8.4 e665/654 8:23d20865:::benchmark_data_smithi090_157496_object76113:head v 665'4793, mlcod=665'4791)", 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:34:44.311973+0000", 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0064099029999999998, 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0066135179999999997, 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311973+0000", 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:34:44.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311973+0000", 2024-05-15T23:34:44.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:34:44.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:34:44.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.311975+0000", 2024-05-15T23:34:44.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6840000000000001e-06 2024-05-15T23:34:44.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:34:44.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.312002+0000", 2024-05-15T23:34:44.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7688e-05 2024-05-15T23:34:44.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:34:44.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.312012+0000", 2024-05-15T23:34:44.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8260000000000005e-06 2024-05-15T23:34:44.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:34:44.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.312019+0000", 2024-05-15T23:34:44.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5189999999999998e-06 2024-05-15T23:34:44.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:34:44.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.314704+0000", 2024-05-15T23:34:44.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026847960000000001 2024-05-15T23:34:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:34:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.314724+0000", 2024-05-15T23:34:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0463999999999999e-05 2024-05-15T23:34:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:34:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:34:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:34:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.17073.0:76127 8.1 e665/654 8:836696b2:::benchmark_data_smithi090_157496_object76126:head v 665'4741, mlcod=665'4739)", 2024-05-15T23:34:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:34:44.318171+0000", 2024-05-15T23:34:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.000211571, 2024-05-15T23:34:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049903800000000004, 2024-05-15T23:34:44.333 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:34:44.333 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:34:44.333 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:34:44.333 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:34:44.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:34:44.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.318171+0000", 2024-05-15T23:34:44.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:34:44.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:34:44.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.318171+0000", 2024-05-15T23:34:44.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:34:44.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:34:44.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.318181+0000", 2024-05-15T23:34:44.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3249999999999997e-06 2024-05-15T23:34:44.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:34:44.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.318232+0000", 2024-05-15T23:34:44.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1236000000000002e-05 2024-05-15T23:34:44.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:34:44.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.318247+0000", 2024-05-15T23:34:44.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5021000000000001e-05 2024-05-15T23:34:44.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:34:44.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.318254+0000", 2024-05-15T23:34:44.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.447e-06 2024-05-15T23:34:44.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:34:44.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.318352+0000", 2024-05-15T23:34:44.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7411000000000006e-05 2024-05-15T23:34:44.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:34:44.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:34:44.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:34:44.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:34:44.318374+0000", 2024-05-15T23:34:44.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2677999999999999e-05 2024-05-15T23:34:44.336 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:34:44.336 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:34:44.336 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:34:44.336 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:34:44.336 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-05-15T23:34:44.336 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 4 2024-05-15T23:34:44.336 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:34:44.336 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:44.496 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:34:44.520 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/773292545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:34:44.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/773292545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:34:44.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/773292545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:34:45.124 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 30 16 77361 77345 161.061 98.125 0.00705279 0.00620712 2024-05-15T23:34:45.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:45 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:34:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:45 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:34:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/773292545' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:34:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:45 smithi084 ceph-mon[39784]: osdmap e666: 12 total, 12 up, 10 in 2024-05-15T23:34:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:45 smithi084 ceph-mon[39784]: pgmap v1769: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-05-15T23:34:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:45 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:34:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:45 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:34:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/773292545' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:34:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:45 smithi090 ceph-mon[44434]: osdmap e666: 12 total, 12 up, 10 in 2024-05-15T23:34:45.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:45 smithi090 ceph-mon[44434]: pgmap v1769: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-05-15T23:34:45.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:45 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:34:45.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:45 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:34:45.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/773292545' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:34:45.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:45 smithi088 ceph-mon[44699]: osdmap e666: 12 total, 12 up, 10 in 2024-05-15T23:34:45.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:45 smithi088 ceph-mon[44699]: pgmap v1769: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-05-15T23:34:46.125 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 31 16 78766 78750 158.696 87.8125 0.111153 0.00628475 2024-05-15T23:34:46.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:46 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:34:46.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:46 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:34:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:46 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:34:46.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:46 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:34:46.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:46 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:34:46.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:46 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:34:46.895 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:34:46.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:46.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:34:46.999 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:34:47.033 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:47.125 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 32 15 80358 80343 156.848 99.5625 0.00696018 0.006374 2024-05-15T23:34:47.139 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:34:47.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:47.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:34:47.244 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:34:47.272 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:47.294 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:34:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:47.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:34:47.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:34:47.428 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:34:47.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:47.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:34:47.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:34:47.592 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:47.648 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:34:47.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:47.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:34:47.752 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:34:47.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:47 smithi084 ceph-mon[39784]: osdmap e667: 12 total, 12 up, 10 in 2024-05-15T23:34:47.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:47 smithi084 ceph-mon[39784]: pgmap v1771: 17 pgs: 17 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-05-15T23:34:47.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:47 smithi084 ceph-mon[39784]: osdmap e668: 12 total, 12 up, 10 in 2024-05-15T23:34:47.782 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:47.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:47 smithi090 ceph-mon[44434]: osdmap e667: 12 total, 12 up, 10 in 2024-05-15T23:34:47.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:47 smithi090 ceph-mon[44434]: pgmap v1771: 17 pgs: 17 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-05-15T23:34:47.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:47 smithi090 ceph-mon[44434]: osdmap e668: 12 total, 12 up, 10 in 2024-05-15T23:34:47.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:47 smithi088 ceph-mon[44699]: osdmap e667: 12 total, 12 up, 10 in 2024-05-15T23:34:47.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:47 smithi088 ceph-mon[44699]: pgmap v1771: 17 pgs: 17 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-05-15T23:34:47.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:47 smithi088 ceph-mon[44699]: osdmap e668: 12 total, 12 up, 10 in 2024-05-15T23:34:48.085 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-05-15T23:34:48.125 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 33 16 82037 82021 155.274 104.875 0.0189297 0.00643859 2024-05-15T23:34:48.337 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:34:48.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:48.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:34:48.348 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:34:48.377 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:34:48.408 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:34:48.439 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:34:48.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:34:48.475 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:34:48.497 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:48.505 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:48.509 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:34:48.526 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:34:48.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:48.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:34:48.539 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:34:48.570 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:34:48.593 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:34:48.614 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:34:48.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2990253751' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:48.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:48.630 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:34:48.657 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:48.662 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:34:48.676 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:34:48.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:48.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:34:48.728 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:34:48.753 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:34:48.754 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:34:48.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:34:48.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:34:48.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:34:48.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:34:48.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:34:48.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:34:48.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:34:48.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:34:48.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:34:48.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:34:48.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:34:48.780 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:34:48.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2990253751' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:48.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:48.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2990253751' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:48.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:48.831 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:48.996 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:34:48.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:48.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:34:49.099 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:34:49.125 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 34 15 83503 83488 153.404 91.6875 0.00523188 0.00651649 2024-05-15T23:34:49.140 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:49.216 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:34:49.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:49.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:34:49.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:34:49.361 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:49 smithi090 ceph-mon[44434]: pgmap v1773: 17 pgs: 17 active+clean; 4.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-05-15T23:34:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:49 smithi088 ceph-mon[44699]: pgmap v1773: 17 pgs: 17 active+clean; 4.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-05-15T23:34:49.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:49 smithi084 ceph-mon[39784]: pgmap v1773: 17 pgs: 17 active+clean; 4.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-05-15T23:34:50.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:50.043 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:34:50.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:34:50.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T23:34:50.125 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 35 16 85026 85010 151.739 95.125 0.00958784 0.00658834 2024-05-15T23:34:50.144 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:34:50.171 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:50.194 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:34:50.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:50.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:34:50.304 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:34:50.333 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:50.411 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:34:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:50.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:34:50.431 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:50.517 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:34:50.542 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:50 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:50.542 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:50 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:34:50.542 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:50 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:34:50.542 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:50 smithi088 ceph-mon[44699]: osdmap e669: 12 total, 12 up, 10 in 2024-05-15T23:34:50.546 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:50.560 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:34:50.568 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:34:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:50.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:34:50.672 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:34:50.727 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:50.728 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:50 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:50.728 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:50 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:34:50.728 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:50 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:34:50.728 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:50 smithi084 ceph-mon[39784]: osdmap e669: 12 total, 12 up, 10 in 2024-05-15T23:34:50.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:50 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:50.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:50 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:34:50.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:50 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:34:50.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:50 smithi090 ceph-mon[44434]: osdmap e669: 12 total, 12 up, 10 in 2024-05-15T23:34:50.850 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:34:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:50.859 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:34:50.960 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:34:51.000 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:51.035 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:34:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:51.040 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:34:51.126 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 36 16 86510 86494 150.098 92.75 0.0055018 0.00666135 2024-05-15T23:34:51.141 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:34:51.175 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:51.327 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:34:51.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:51.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:34:51.434 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:34:51.463 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:51.490 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:34:51.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:51.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:34:51.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:51 smithi084 ceph-mon[39784]: pgmap v1775: 17 pgs: 17 active+clean; 5.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T23:34:51.593 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:34:51.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:34:51.622 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:51.644 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:34:51.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:51.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:34:51.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:34:51.787 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:51 smithi090 ceph-mon[44434]: pgmap v1775: 17 pgs: 17 active+clean; 5.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T23:34:51.808 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:51 smithi088 ceph-mon[44699]: pgmap v1775: 17 pgs: 17 active+clean; 5.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-05-15T23:34:52.126 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 37 16 88136 88120 148.789 101.625 0.00646429 0.0067192 2024-05-15T23:34:53.127 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 38 15 89800 89785 147.607 104.062 0.0102775 0.00677277 2024-05-15T23:34:53.214 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:53.223 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-05-15T23:34:53.522 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:53 smithi084 ceph-mon[39784]: osdmap e670: 12 total, 12 up, 10 in 2024-05-15T23:34:53.522 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:53 smithi084 ceph-mon[39784]: pgmap v1777: 17 pgs: 17 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-05-15T23:34:53.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:53 smithi088 ceph-mon[44699]: osdmap e670: 12 total, 12 up, 10 in 2024-05-15T23:34:53.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:53 smithi088 ceph-mon[44699]: pgmap v1777: 17 pgs: 17 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-05-15T23:34:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:53 smithi090 ceph-mon[44434]: osdmap e670: 12 total, 12 up, 10 in 2024-05-15T23:34:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:53 smithi090 ceph-mon[44434]: pgmap v1777: 17 pgs: 17 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-05-15T23:34:54.127 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 39 16 91304 91288 146.231 93.9375 0.00291725 0.00683544 2024-05-15T23:34:54.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:54 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:34:54.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:54 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:34:54.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:54 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:34:54.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:54 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:34:54.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:54 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:34:54.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:54 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:34:54.743 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:34:54.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:54.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:34:54.848 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:34:54.876 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:54.901 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:34:54.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:54.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:34:54.970 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:55.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:34:55.039 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:55.122 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:34:55.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:55.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:34:55.128 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:34:55.129089 min lat: 0.00114746 max lat: 0.212087 avg lat: 0.00691703 2024-05-15T23:34:55.129 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:34:55.129 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 40 16 92560 92544 144.533 78.5 0.0136809 0.00691703 2024-05-15T23:34:55.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:34:55.288 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:55.344 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:34:55.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:55 smithi084 ceph-mon[39784]: osdmap e671: 12 total, 12 up, 10 in 2024-05-15T23:34:55.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:55 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4251724148' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:34:55.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:55 smithi084 ceph-mon[39784]: pgmap v1779: 17 pgs: 17 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-05-15T23:34:55.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:55 smithi090 ceph-mon[44434]: osdmap e671: 12 total, 12 up, 10 in 2024-05-15T23:34:55.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:55 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4251724148' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:34:55.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:55 smithi090 ceph-mon[44434]: pgmap v1779: 17 pgs: 17 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-05-15T23:34:55.647 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:34:55.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:55.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:34:55.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:34:55.812 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:34:55.812 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:34:55.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:55 smithi088 ceph-mon[44699]: osdmap e671: 12 total, 12 up, 10 in 2024-05-15T23:34:55.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:55 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4251724148' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:34:55.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:55 smithi088 ceph-mon[44699]: pgmap v1779: 17 pgs: 17 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-05-15T23:34:55.833 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:56.098 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:34:56.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:56.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:34:56.132 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 41 16 94010 93994 143.206 90.625 0.00578425 0.0069809 2024-05-15T23:34:56.205 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:34:56.232 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:56.256 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:34:56.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:56.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:34:56.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:34:56.398 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:56 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:34:56.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4251724148' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:34:56.399 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:56 smithi090 ceph-mon[44434]: osdmap e672: 12 total, 12 up, 10 in 2024-05-15T23:34:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:56.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:56 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:34:56.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4251724148' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:34:56.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:56 smithi088 ceph-mon[44699]: osdmap e672: 12 total, 12 up, 10 in 2024-05-15T23:34:56.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:56 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:34:56.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4251724148' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:34:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:56 smithi084 ceph-mon[39784]: osdmap e672: 12 total, 12 up, 10 in 2024-05-15T23:34:56.628 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:34:56.651 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:34:56.676 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:34:56.703 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:34:56.727 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:34:56.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:34:56.775 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:34:56.803 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:34:56.830 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:34:56.858 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:34:56.880 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:34:56.909 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:34:56.910 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:34:56.910 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:34:56.910 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:34:56.911 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:34:56.911 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:34:56.911 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:34:56.911 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:34:56.911 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:34:56.911 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:34:56.911 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:34:56.911 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:34:56.911 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:34:57.132 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 42 16 95553 95537 142.093 96.4375 0.00624164 0.00703653 2024-05-15T23:34:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1487145261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:57 smithi084 ceph-mon[39784]: pgmap v1781: 17 pgs: 17 active+clean; 5.6 GiB data, 17 GiB used, 878 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-05-15T23:34:57.770 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1487145261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:57.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:57 smithi090 ceph-mon[44434]: pgmap v1781: 17 pgs: 17 active+clean; 5.6 GiB data, 17 GiB used, 878 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-05-15T23:34:57.796 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:34:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:57.806 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:34:57.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T23:34:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1487145261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:34:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:57 smithi088 ceph-mon[44699]: pgmap v1781: 17 pgs: 17 active+clean; 5.6 GiB data, 17 GiB used, 878 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-05-15T23:34:57.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:34:57.984 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:58.019 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:34:58.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:58.023 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:34:58.123 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:34:58.132 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 43 16 96870 96854 140.703 82.3125 0.0128488 0.00710548 2024-05-15T23:34:58.152 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:58.173 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:34:58.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:58.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:34:58.192 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:34:58.269 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:34:58.277 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:34:58.304 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:58.328 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:34:58.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:58.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:34:58.431 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:34:58.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:58.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:58.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:58.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:58.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:58.462 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:58.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:58.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:58.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:58.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:58.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:58.596 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:34:58.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:58.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:34:58.609 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:58.609 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:58.610 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:58.610 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:58.610 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:34:58.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:34:58.753 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:58.909 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:34:58.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:58.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:34:58.936 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:34:59.012 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:34:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:59.065 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:34:59.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:59.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:34:59.132 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 44 16 98259 98243 139.479 86.8125 0.0112455 0.00716788 2024-05-15T23:34:59.169 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:34:59.223 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:34:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:34:59 smithi090 ceph-mon[44434]: pgmap v1782: 17 pgs: 17 active+clean; 5.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 105 MiB/s wr, 1.67k op/s 2024-05-15T23:34:59.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:59.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:34:59.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:34:59 smithi084 ceph-mon[39784]: pgmap v1782: 17 pgs: 17 active+clean; 5.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 105 MiB/s wr, 1.67k op/s 2024-05-15T23:34:59.613 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-05-15T23:34:59.655 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:34:59.682 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:34:59 smithi088 ceph-mon[44699]: pgmap v1782: 17 pgs: 17 active+clean; 5.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 105 MiB/s wr, 1.67k op/s 2024-05-15T23:34:59.684 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:59.713 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:34:59.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:59.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:34:59.820 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:34:59.851 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:34:59.874 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:34:59.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:34:59.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:34:59.980 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:00.010 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:00.033 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:00.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:00.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:00.133 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 45 16 99693 99677 138.371 89.625 0.0106768 0.00722543 2024-05-15T23:35:00.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:35:00.170 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:00.211 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:35:00.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:00.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:35:00.325 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:00.357 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:00.380 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:00.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:00.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:00.485 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:35:00.512 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:00.540 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:35:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:00.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:35:00.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:35:00.646 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:35:00.720 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:00.747 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:00.949 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:35:00.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:00.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:35:01.058 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:01.085 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:01.106 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:01.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:01.133 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 46 16 101154 101138 137.349 91.3125 0.00735188 0.00727902 2024-05-15T23:35:01.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:01.257 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:01.290 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:01.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:01.294 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:01.395 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:35:01.440 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:01.473 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:35:01.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:01.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:35:01.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:35:01.617 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:01.915 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:35:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:35:02.122 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:35:02.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:35:02.122 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:35:02.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.122 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9707 6.9 6:9754036e:::benchmark_data_smithi090_131278_object9706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.109351+0000", 2024-05-15T23:35:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.008304264, 2024-05-15T23:35:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93008897099999999, 2024-05-15T23:35:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9707 2024-05-15T23:35:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109351+0000", 2024-05-15T23:35:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109351+0000", 2024-05-15T23:35:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109353+0000", 2024-05-15T23:35:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5840000000000002e-06 2024-05-15T23:35:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109383+0000", 2024-05-15T23:35:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0077999999999999e-05 2024-05-15T23:35:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109399+0000", 2024-05-15T23:35:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6019999999999999e-05 2024-05-15T23:35:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109422+0000", 2024-05-15T23:35:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2385e-05 2024-05-15T23:35:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109556+0000", 2024-05-15T23:35:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013386999999999999 2024-05-15T23:35:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109571+0000", 2024-05-15T23:35:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.524e-05 2024-05-15T23:35:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.035494+0000", 2024-05-15T23:35:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.925922892 2024-05-15T23:35:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.035642+0000", 2024-05-15T23:35:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148245 2024-05-15T23:35:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.035759+0000", 2024-05-15T23:35:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116655 2024-05-15T23:35:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.039112+0000", 2024-05-15T23:35:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033537879999999999 2024-05-15T23:35:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.039158+0000", 2024-05-15T23:35:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5238000000000003e-05 2024-05-15T23:35:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.039208+0000", 2024-05-15T23:35:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0482000000000002e-05 2024-05-15T23:35:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.039440+0000", 2024-05-15T23:35:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231494 2024-05-15T23:35:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9733 6.9 6:948d81d7:::benchmark_data_smithi090_131278_object9732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.109443+0000", 2024-05-15T23:35:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.00821158300005, 2024-05-15T23:35:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93032885799999998, 2024-05-15T23:35:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9733 2024-05-15T23:35:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109443+0000", 2024-05-15T23:35:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109443+0000", 2024-05-15T23:35:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109444+0000", 2024-05-15T23:35:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.145e-06 2024-05-15T23:35:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109475+0000", 2024-05-15T23:35:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2024-05-15T23:35:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109486+0000", 2024-05-15T23:35:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0434999999999999e-05 2024-05-15T23:35:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109494+0000", 2024-05-15T23:35:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6290000000000001e-06 2024-05-15T23:35:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109608+0000", 2024-05-15T23:35:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011493900000000001 2024-05-15T23:35:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109622+0000", 2024-05-15T23:35:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3151e-05 2024-05-15T23:35:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.036660+0000", 2024-05-15T23:35:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.92703861200000004 2024-05-15T23:35:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.036757+0000", 2024-05-15T23:35:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6810000000000003e-05 2024-05-15T23:35:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.036896+0000", 2024-05-15T23:35:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013938899999999999 2024-05-15T23:35:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.039120+0000", 2024-05-15T23:35:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002223123 2024-05-15T23:35:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.039486+0000", 2024-05-15T23:35:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000366485 2024-05-15T23:35:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.039533+0000", 2024-05-15T23:35:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7017000000000001e-05 2024-05-15T23:35:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.039772+0000", 2024-05-15T23:35:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023906700000000001 2024-05-15T23:35:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9753 6.9 6:97490a28:::benchmark_data_smithi090_131278_object9752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.109504+0000", 2024-05-15T23:35:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.00815104799995, 2024-05-15T23:35:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94141924600000004, 2024-05-15T23:35:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9753 2024-05-15T23:35:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109504+0000", 2024-05-15T23:35:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109504+0000", 2024-05-15T23:35:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109505+0000", 2024-05-15T23:35:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4499999999999995e-07 2024-05-15T23:35:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109536+0000", 2024-05-15T23:35:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1590000000000001e-05 2024-05-15T23:35:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109546+0000", 2024-05-15T23:35:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6239999999999997e-06 2024-05-15T23:35:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109554+0000", 2024-05-15T23:35:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0919999999999998e-06 2024-05-15T23:35:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109653+0000", 2024-05-15T23:35:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9423e-05 2024-05-15T23:35:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109665+0000", 2024-05-15T23:35:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2028000000000001e-05 2024-05-15T23:35:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.046577+0000", 2024-05-15T23:35:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93691109400000006 2024-05-15T23:35:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.046710+0000", 2024-05-15T23:35:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000132977 2024-05-15T23:35:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.046848+0000", 2024-05-15T23:35:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013824900000000001 2024-05-15T23:35:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.048294+0000", 2024-05-15T23:35:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001446262 2024-05-15T23:35:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.050383+0000", 2024-05-15T23:35:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020884620000000001 2024-05-15T23:35:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.050443+0000", 2024-05-15T23:35:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0451000000000001e-05 2024-05-15T23:35:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.050923+0000", 2024-05-15T23:35:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048004900000000002 2024-05-15T23:35:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9767 6.9 6:9cea39df:::benchmark_data_smithi090_131278_object9766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.109573+0000", 2024-05-15T23:35:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.00808167299999, 2024-05-15T23:35:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.941860794, 2024-05-15T23:35:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9767 2024-05-15T23:35:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109573+0000", 2024-05-15T23:35:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109573+0000", 2024-05-15T23:35:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109574+0000", 2024-05-15T23:35:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0869999999999999e-06 2024-05-15T23:35:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109604+0000", 2024-05-15T23:35:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9560000000000002e-05 2024-05-15T23:35:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109611+0000", 2024-05-15T23:35:02.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6480000000000003e-06 2024-05-15T23:35:02.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109626+0000", 2024-05-15T23:35:02.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4722999999999999e-05 2024-05-15T23:35:02.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109712+0000", 2024-05-15T23:35:02.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5511000000000002e-05 2024-05-15T23:35:02.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109724+0000", 2024-05-15T23:35:02.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1995e-05 2024-05-15T23:35:02.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.047477+0000", 2024-05-15T23:35:02.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93775379699999994 2024-05-15T23:35:02.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.047599+0000", 2024-05-15T23:35:02.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121768 2024-05-15T23:35:02.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.047753+0000", 2024-05-15T23:35:02.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000153301 2024-05-15T23:35:02.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.050947+0000", 2024-05-15T23:35:02.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031944220000000001 2024-05-15T23:35:02.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.051009+0000", 2024-05-15T23:35:02.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2502000000000006e-05 2024-05-15T23:35:02.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.051089+0000", 2024-05-15T23:35:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9973000000000002e-05 2024-05-15T23:35:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.051434+0000", 2024-05-15T23:35:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034450699999999997 2024-05-15T23:35:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9775 6.9 6:9c0e2b69:::benchmark_data_smithi090_131278_object9774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.155 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.109643+0000", 2024-05-15T23:35:02.155 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.00801216299999, 2024-05-15T23:35:02.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.942130354, 2024-05-15T23:35:02.155 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.155 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.155 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.155 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.155 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.156 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.156 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9775 2024-05-15T23:35:02.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.156 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109643+0000", 2024-05-15T23:35:02.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109643+0000", 2024-05-15T23:35:02.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109644+0000", 2024-05-15T23:35:02.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8400000000000002e-07 2024-05-15T23:35:02.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109674+0000", 2024-05-15T23:35:02.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0141000000000002e-05 2024-05-15T23:35:02.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109681+0000", 2024-05-15T23:35:02.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6310000000000003e-06 2024-05-15T23:35:02.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109688+0000", 2024-05-15T23:35:02.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0160000000000003e-06 2024-05-15T23:35:02.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109771+0000", 2024-05-15T23:35:02.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2731999999999993e-05 2024-05-15T23:35:02.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109784+0000", 2024-05-15T23:35:02.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3294e-05 2024-05-15T23:35:02.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.048329+0000", 2024-05-15T23:35:02.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93854469699999998 2024-05-15T23:35:02.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.048437+0000", 2024-05-15T23:35:02.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107563 2024-05-15T23:35:02.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.048560+0000", 2024-05-15T23:35:02.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012356199999999999 2024-05-15T23:35:02.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.051490+0000", 2024-05-15T23:35:02.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029295710000000002 2024-05-15T23:35:02.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.051499+0000", 2024-05-15T23:35:02.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5999999999999996e-06 2024-05-15T23:35:02.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.051532+0000", 2024-05-15T23:35:02.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2663999999999998e-05 2024-05-15T23:35:02.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.051773+0000", 2024-05-15T23:35:02.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024089900000000001 2024-05-15T23:35:02.161 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.161 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.161 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.161 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9786 6.9 6:997790a3:::benchmark_data_smithi090_131278_object9785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.161 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.109704+0000", 2024-05-15T23:35:02.162 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.00795111399998, 2024-05-15T23:35:02.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94241746800000004, 2024-05-15T23:35:02.162 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.162 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.162 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.162 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.162 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.162 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.162 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9786 2024-05-15T23:35:02.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.162 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109704+0000", 2024-05-15T23:35:02.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109704+0000", 2024-05-15T23:35:02.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109705+0000", 2024-05-15T23:35:02.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4600000000000003e-07 2024-05-15T23:35:02.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109732+0000", 2024-05-15T23:35:02.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7007000000000001e-05 2024-05-15T23:35:02.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109740+0000", 2024-05-15T23:35:02.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1829999999999994e-06 2024-05-15T23:35:02.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109747+0000", 2024-05-15T23:35:02.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.742e-06 2024-05-15T23:35:02.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109813+0000", 2024-05-15T23:35:02.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6296000000000004e-05 2024-05-15T23:35:02.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109820+0000", 2024-05-15T23:35:02.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6709999999999999e-06 2024-05-15T23:35:02.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.048953+0000", 2024-05-15T23:35:02.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93913215100000003 2024-05-15T23:35:02.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.049106+0000", 2024-05-15T23:35:02.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015367399999999999 2024-05-15T23:35:02.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.049276+0000", 2024-05-15T23:35:02.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169758 2024-05-15T23:35:02.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.051781+0000", 2024-05-15T23:35:02.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025051660000000001 2024-05-15T23:35:02.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.051830+0000", 2024-05-15T23:35:02.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9234999999999999e-05 2024-05-15T23:35:02.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.051886+0000", 2024-05-15T23:35:02.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5729999999999997e-05 2024-05-15T23:35:02.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.052121+0000", 2024-05-15T23:35:02.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000235009 2024-05-15T23:35:02.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.170 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.170 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9799 6.9 6:96bb59fd:::benchmark_data_smithi090_131278_object9798:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.170 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.109767+0000", 2024-05-15T23:35:02.170 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.00788823100004, 2024-05-15T23:35:02.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94274228599999998, 2024-05-15T23:35:02.170 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.171 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.171 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.171 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.171 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.171 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.171 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9799 2024-05-15T23:35:02.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.171 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109767+0000", 2024-05-15T23:35:02.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109767+0000", 2024-05-15T23:35:02.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109768+0000", 2024-05-15T23:35:02.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6200000000000006e-07 2024-05-15T23:35:02.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109798+0000", 2024-05-15T23:35:02.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0181000000000001e-05 2024-05-15T23:35:02.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109803+0000", 2024-05-15T23:35:02.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-05-15T23:35:02.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109807+0000", 2024-05-15T23:35:02.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1899999999999997e-06 2024-05-15T23:35:02.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109852+0000", 2024-05-15T23:35:02.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4332000000000002e-05 2024-05-15T23:35:02.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109859+0000", 2024-05-15T23:35:02.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4309999999999997e-06 2024-05-15T23:35:02.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.049701+0000", 2024-05-15T23:35:02.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93984147600000001 2024-05-15T23:35:02.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.049810+0000", 2024-05-15T23:35:02.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109413 2024-05-15T23:35:02.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.049908+0000", 2024-05-15T23:35:02.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7838999999999996e-05 2024-05-15T23:35:02.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.051785+0000", 2024-05-15T23:35:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001877426 2024-05-15T23:35:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.052184+0000", 2024-05-15T23:35:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039892999999999998 2024-05-15T23:35:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.052266+0000", 2024-05-15T23:35:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1846000000000002e-05 2024-05-15T23:35:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.052509+0000", 2024-05-15T23:35:02.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000242704 2024-05-15T23:35:02.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.177 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.177 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9813 6.9 6:90a74fca:::benchmark_data_smithi090_131278_object9812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.177 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.109815+0000", 2024-05-15T23:35:02.177 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.00783981699999, 2024-05-15T23:35:02.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94727946699999999, 2024-05-15T23:35:02.177 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.177 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.178 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.178 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9813 2024-05-15T23:35:02.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.178 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109815+0000", 2024-05-15T23:35:02.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109815+0000", 2024-05-15T23:35:02.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109816+0000", 2024-05-15T23:35:02.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4700000000000001e-07 2024-05-15T23:35:02.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109844+0000", 2024-05-15T23:35:02.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8419999999999999e-05 2024-05-15T23:35:02.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109851+0000", 2024-05-15T23:35:02.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.114e-06 2024-05-15T23:35:02.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109858+0000", 2024-05-15T23:35:02.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.268e-06 2024-05-15T23:35:02.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109922+0000", 2024-05-15T23:35:02.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4294000000000005e-05 2024-05-15T23:35:02.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.109931+0000", 2024-05-15T23:35:02.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7509999999999997e-06 2024-05-15T23:35:02.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.052636+0000", 2024-05-15T23:35:02.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94270564099999998 2024-05-15T23:35:02.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.052778+0000", 2024-05-15T23:35:02.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014178699999999999 2024-05-15T23:35:02.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.052887+0000", 2024-05-15T23:35:02.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108942 2024-05-15T23:35:02.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.055768+0000", 2024-05-15T23:35:02.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028809500000000002 2024-05-15T23:35:02.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.056515+0000", 2024-05-15T23:35:02.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00074742899999999998 2024-05-15T23:35:02.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.056566+0000", 2024-05-15T23:35:02.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0646000000000003e-05 2024-05-15T23:35:02.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.057094+0000", 2024-05-15T23:35:02.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052847799999999998 2024-05-15T23:35:02.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.184 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.184 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.185 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9714 6.1 6:83dab7a1:::benchmark_data_smithi090_131278_object9713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.185 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.110079+0000", 2024-05-15T23:35:02.185 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.00757593200001, 2024-05-15T23:35:02.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.930064153, 2024-05-15T23:35:02.185 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.185 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.185 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.185 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.185 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.185 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.185 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9714 2024-05-15T23:35:02.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.186 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110079+0000", 2024-05-15T23:35:02.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110079+0000", 2024-05-15T23:35:02.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110081+0000", 2024-05-15T23:35:02.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.747e-06 2024-05-15T23:35:02.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110120+0000", 2024-05-15T23:35:02.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9041000000000001e-05 2024-05-15T23:35:02.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110138+0000", 2024-05-15T23:35:02.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8660000000000001e-05 2024-05-15T23:35:02.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110147+0000", 2024-05-15T23:35:02.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4300000000000006e-06 2024-05-15T23:35:02.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110209+0000", 2024-05-15T23:35:02.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1780999999999997e-05 2024-05-15T23:35:02.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110216+0000", 2024-05-15T23:35:02.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2024-05-15T23:35:02.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.038023+0000", 2024-05-15T23:35:02.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.92780699499999997 2024-05-15T23:35:02.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.038114+0000", 2024-05-15T23:35:02.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1135000000000001e-05 2024-05-15T23:35:02.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.038218+0000", 2024-05-15T23:35:02.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010425900000000001 2024-05-15T23:35:02.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.039453+0000", 2024-05-15T23:35:02.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012343 2024-05-15T23:35:02.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.039838+0000", 2024-05-15T23:35:02.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038587000000000002 2024-05-15T23:35:02.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.039894+0000", 2024-05-15T23:35:02.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5736999999999998e-05 2024-05-15T23:35:02.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.040143+0000", 2024-05-15T23:35:02.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000248898 2024-05-15T23:35:02.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.191 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.191 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9722 6.1 6:8ccec00d:::benchmark_data_smithi090_131278_object9721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.191 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.110164+0000", 2024-05-15T23:35:02.191 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.00749117400005, 2024-05-15T23:35:02.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93276001600000003, 2024-05-15T23:35:02.191 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.191 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.192 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.192 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9722 2024-05-15T23:35:02.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.192 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110164+0000", 2024-05-15T23:35:02.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110164+0000", 2024-05-15T23:35:02.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110165+0000", 2024-05-15T23:35:02.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.15e-06 2024-05-15T23:35:02.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110216+0000", 2024-05-15T23:35:02.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0821e-05 2024-05-15T23:35:02.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110226+0000", 2024-05-15T23:35:02.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0047000000000001e-05 2024-05-15T23:35:02.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110235+0000", 2024-05-15T23:35:02.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7820000000000005e-06 2024-05-15T23:35:02.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110282+0000", 2024-05-15T23:35:02.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6588000000000002e-05 2024-05-15T23:35:02.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110287+0000", 2024-05-15T23:35:02.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2920000000000003e-06 2024-05-15T23:35:02.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.040188+0000", 2024-05-15T23:35:02.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.92990107799999999 2024-05-15T23:35:02.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.040342+0000", 2024-05-15T23:35:02.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015361300000000001 2024-05-15T23:35:02.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.040524+0000", 2024-05-15T23:35:02.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000181841 2024-05-15T23:35:02.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.042514+0000", 2024-05-15T23:35:02.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019898530000000002 2024-05-15T23:35:02.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.042533+0000", 2024-05-15T23:35:02.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9017000000000001e-05 2024-05-15T23:35:02.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.042584+0000", 2024-05-15T23:35:02.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0963e-05 2024-05-15T23:35:02.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.042924+0000", 2024-05-15T23:35:02.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000339971 2024-05-15T23:35:02.199 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.199 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.200 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.200 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9730 6.1 6:867b5b0c:::benchmark_data_smithi090_131278_object9729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.200 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.110249+0000", 2024-05-15T23:35:02.200 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.00740632600002, 2024-05-15T23:35:02.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93305730399999998, 2024-05-15T23:35:02.200 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.200 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.200 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.200 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.200 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.200 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.200 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9730 2024-05-15T23:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110249+0000", 2024-05-15T23:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110249+0000", 2024-05-15T23:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110249+0000", 2024-05-15T23:35:02.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7000000000000003e-07 2024-05-15T23:35:02.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110287+0000", 2024-05-15T23:35:02.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7428999999999997e-05 2024-05-15T23:35:02.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110296+0000", 2024-05-15T23:35:02.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2520000000000005e-06 2024-05-15T23:35:02.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110303+0000", 2024-05-15T23:35:02.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0029999999999997e-06 2024-05-15T23:35:02.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110345+0000", 2024-05-15T23:35:02.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2382999999999997e-05 2024-05-15T23:35:02.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110354+0000", 2024-05-15T23:35:02.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1440000000000003e-06 2024-05-15T23:35:02.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.041008+0000", 2024-05-15T23:35:02.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93065464499999995 2024-05-15T23:35:02.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.041147+0000", 2024-05-15T23:35:02.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013925100000000001 2024-05-15T23:35:02.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.041315+0000", 2024-05-15T23:35:02.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000167314 2024-05-15T23:35:02.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.042997+0000", 2024-05-15T23:35:02.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016822779999999999 2024-05-15T23:35:02.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.043062+0000", 2024-05-15T23:35:02.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4857000000000006e-05 2024-05-15T23:35:02.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.043095+0000", 2024-05-15T23:35:02.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2914999999999999e-05 2024-05-15T23:35:02.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.043306+0000", 2024-05-15T23:35:02.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021096299999999999 2024-05-15T23:35:02.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.206 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.206 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9755 6.1 6:8a354ca5:::benchmark_data_smithi090_131278_object9754:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.207 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.110316+0000", 2024-05-15T23:35:02.207 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.00733878699998, 2024-05-15T23:35:02.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93329116899999998, 2024-05-15T23:35:02.207 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.207 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.207 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.207 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.207 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.207 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.207 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9755 2024-05-15T23:35:02.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.207 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110316+0000", 2024-05-15T23:35:02.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110316+0000", 2024-05-15T23:35:02.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110317+0000", 2024-05-15T23:35:02.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6200000000000006e-07 2024-05-15T23:35:02.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110359+0000", 2024-05-15T23:35:02.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2284999999999999e-05 2024-05-15T23:35:02.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110369+0000", 2024-05-15T23:35:02.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4029999999999996e-06 2024-05-15T23:35:02.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110377+0000", 2024-05-15T23:35:02.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2490000000000001e-06 2024-05-15T23:35:02.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110414+0000", 2024-05-15T23:35:02.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7273e-05 2024-05-15T23:35:02.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110418+0000", 2024-05-15T23:35:02.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2749999999999997e-06 2024-05-15T23:35:02.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.041718+0000", 2024-05-15T23:35:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.931299867 2024-05-15T23:35:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.041835+0000", 2024-05-15T23:35:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116444 2024-05-15T23:35:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.041952+0000", 2024-05-15T23:35:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011723299999999999 2024-05-15T23:35:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.043046+0000", 2024-05-15T23:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010942619999999999 2024-05-15T23:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.043316+0000", 2024-05-15T23:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026935199999999999 2024-05-15T23:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.043346+0000", 2024-05-15T23:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0440999999999999e-05 2024-05-15T23:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.043607+0000", 2024-05-15T23:35:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026112299999999999 2024-05-15T23:35:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9757 6.1 6:80463661:::benchmark_data_smithi090_131278_object9756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.110389+0000", 2024-05-15T23:35:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.00726600899998, 2024-05-15T23:35:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93669891800000005, 2024-05-15T23:35:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9757 2024-05-15T23:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110389+0000", 2024-05-15T23:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110389+0000", 2024-05-15T23:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110390+0000", 2024-05-15T23:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999997e-07 2024-05-15T23:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110420+0000", 2024-05-15T23:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0787000000000002e-05 2024-05-15T23:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110427+0000", 2024-05-15T23:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3509999999999998e-06 2024-05-15T23:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110431+0000", 2024-05-15T23:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5279999999999997e-06 2024-05-15T23:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110465+0000", 2024-05-15T23:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3238999999999998e-05 2024-05-15T23:35:02.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110469+0000", 2024-05-15T23:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1679999999999997e-06 2024-05-15T23:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.043662+0000", 2024-05-15T23:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93319312399999998 2024-05-15T23:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.043757+0000", 2024-05-15T23:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4780999999999999e-05 2024-05-15T23:35:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.043933+0000", 2024-05-15T23:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017640900000000001 2024-05-15T23:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.046538+0000", 2024-05-15T23:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00260523 2024-05-15T23:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.046613+0000", 2024-05-15T23:35:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4733000000000004e-05 2024-05-15T23:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.046674+0000", 2024-05-15T23:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1205999999999996e-05 2024-05-15T23:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.047088+0000", 2024-05-15T23:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041357600000000003 2024-05-15T23:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9765 6.1 6:80abe3b5:::benchmark_data_smithi090_131278_object9764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.110439+0000", 2024-05-15T23:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.00721539899996, 2024-05-15T23:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93747247899999997, 2024-05-15T23:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9765 2024-05-15T23:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110439+0000", 2024-05-15T23:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110439+0000", 2024-05-15T23:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110440+0000", 2024-05-15T23:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.99e-07 2024-05-15T23:35:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110464+0000", 2024-05-15T23:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4111999999999999e-05 2024-05-15T23:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110469+0000", 2024-05-15T23:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1490000000000002e-06 2024-05-15T23:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110473+0000", 2024-05-15T23:35:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9480000000000001e-06 2024-05-15T23:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110510+0000", 2024-05-15T23:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6514999999999999e-05 2024-05-15T23:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110516+0000", 2024-05-15T23:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5649999999999998e-06 2024-05-15T23:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.044452+0000", 2024-05-15T23:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.933935819 2024-05-15T23:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.044544+0000", 2024-05-15T23:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1378000000000005e-05 2024-05-15T23:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.044641+0000", 2024-05-15T23:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7566999999999996e-05 2024-05-15T23:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.047422+0000", 2024-05-15T23:35:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027806240000000002 2024-05-15T23:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.047470+0000", 2024-05-15T23:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8717000000000001e-05 2024-05-15T23:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.047517+0000", 2024-05-15T23:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6965999999999997e-05 2024-05-15T23:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.047912+0000", 2024-05-15T23:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039451999999999998 2024-05-15T23:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9790 6.1 6:8296d3e6:::benchmark_data_smithi090_131278_object9789:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.110481+0000", 2024-05-15T23:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.007174136, 2024-05-15T23:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93780108200000001, 2024-05-15T23:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9790 2024-05-15T23:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110481+0000", 2024-05-15T23:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110481+0000", 2024-05-15T23:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110481+0000", 2024-05-15T23:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7599999999999997e-07 2024-05-15T23:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110505+0000", 2024-05-15T23:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4273999999999998e-05 2024-05-15T23:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110511+0000", 2024-05-15T23:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0980000000000001e-06 2024-05-15T23:35:02.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110514+0000", 2024-05-15T23:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.777e-06 2024-05-15T23:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110542+0000", 2024-05-15T23:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7898999999999998e-05 2024-05-15T23:35:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110549+0000", 2024-05-15T23:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8249999999999999e-06 2024-05-15T23:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.045112+0000", 2024-05-15T23:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93456253899999997 2024-05-15T23:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.045287+0000", 2024-05-15T23:35:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017588400000000001 2024-05-15T23:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.045444+0000", 2024-05-15T23:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015623300000000001 2024-05-15T23:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.047427+0000", 2024-05-15T23:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019835040000000001 2024-05-15T23:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.047962+0000", 2024-05-15T23:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053510600000000004 2024-05-15T23:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.048021+0000", 2024-05-15T23:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9045999999999997e-05 2024-05-15T23:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.048282+0000", 2024-05-15T23:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026042099999999999 2024-05-15T23:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:9806 6.1 6:8f2dd2d0:::benchmark_data_smithi090_131278_object9805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e492)", 2024-05-15T23:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:25:33.110521+0000", 2024-05-15T23:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 569.00713387200005, 2024-05-15T23:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93813413199999995, 2024-05-15T23:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 9806 2024-05-15T23:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110521+0000", 2024-05-15T23:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110521+0000", 2024-05-15T23:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110521+0000", 2024-05-15T23:35:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1300000000000001e-07 2024-05-15T23:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110544+0000", 2024-05-15T23:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2699000000000001e-05 2024-05-15T23:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110550+0000", 2024-05-15T23:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0290000000000002e-06 2024-05-15T23:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110554+0000", 2024-05-15T23:35:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.675e-06 2024-05-15T23:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110589+0000", 2024-05-15T23:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4986e-05 2024-05-15T23:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:33.110594+0000", 2024-05-15T23:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6239999999999999e-06 2024-05-15T23:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.045803+0000", 2024-05-15T23:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93520885600000003 2024-05-15T23:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.045937+0000", 2024-05-15T23:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353200000000001 2024-05-15T23:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.046108+0000", 2024-05-15T23:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170779 2024-05-15T23:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.047924+0000", 2024-05-15T23:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018167070000000001 2024-05-15T23:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.048338+0000", 2024-05-15T23:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041324599999999998 2024-05-15T23:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.048391+0000", 2024-05-15T23:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.308e-05 2024-05-15T23:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:25:34.048655+0000", 2024-05-15T23:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026450600000000002 2024-05-15T23:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:253399 6.9 6.70c2b459 (undecoded) ondisk+write+known_if_redirected e537)", 2024-05-15T23:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:28:16.191642+0000", 2024-05-15T23:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 405.92601299500001, 2024-05-15T23:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.96622732700000002, 2024-05-15T23:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-05-15T23:35:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 253399 2024-05-15T23:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.191642+0000", 2024-05-15T23:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.191642+0000", 2024-05-15T23:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.191644+0000", 2024-05-15T23:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7689999999999999e-06 2024-05-15T23:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.191648+0000", 2024-05-15T23:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5800000000000002e-06 2024-05-15T23:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.191658+0000", 2024-05-15T23:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0145e-05 2024-05-15T23:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.191664+0000", 2024-05-15T23:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8939999999999998e-06 2024-05-15T23:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.191710+0000", 2024-05-15T23:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5385e-05 2024-05-15T23:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.191717+0000", 2024-05-15T23:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1369999999999996e-06 2024-05-15T23:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:17.157836+0000", 2024-05-15T23:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.96611937800000003 2024-05-15T23:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:17.157869+0000", 2024-05-15T23:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3039e-05 2024-05-15T23:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:253413 6.9 6.23bac5d9 (undecoded) ondisk+write+known_if_redirected e537)", 2024-05-15T23:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:28:16.207918+0000", 2024-05-15T23:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 405.90973679899997, 2024-05-15T23:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.949991739, 2024-05-15T23:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-05-15T23:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 253413 2024-05-15T23:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.207918+0000", 2024-05-15T23:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.207918+0000", 2024-05-15T23:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.207920+0000", 2024-05-15T23:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8509999999999999e-06 2024-05-15T23:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.207926+0000", 2024-05-15T23:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.852e-06 2024-05-15T23:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.207936+0000", 2024-05-15T23:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0564000000000001e-05 2024-05-15T23:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.207943+0000", 2024-05-15T23:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9820000000000002e-06 2024-05-15T23:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.207990+0000", 2024-05-15T23:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6610000000000003e-05 2024-05-15T23:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.207997+0000", 2024-05-15T23:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3139999999999998e-06 2024-05-15T23:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:17.157901+0000", 2024-05-15T23:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.94990424799999995 2024-05-15T23:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:17.157910+0000", 2024-05-15T23:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3180000000000007e-06 2024-05-15T23:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:253438 6.9 6.7f7b4689 (undecoded) ondisk+write+known_if_redirected e537)", 2024-05-15T23:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:28:16.260969+0000", 2024-05-15T23:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 405.85668552999999, 2024-05-15T23:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.89703333200000002, 2024-05-15T23:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-05-15T23:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 253438 2024-05-15T23:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.260969+0000", 2024-05-15T23:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.260969+0000", 2024-05-15T23:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.260971+0000", 2024-05-15T23:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.801e-06 2024-05-15T23:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.260977+0000", 2024-05-15T23:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2489999999999998e-06 2024-05-15T23:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.260988+0000", 2024-05-15T23:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090999999999999e-05 2024-05-15T23:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.260994+0000", 2024-05-15T23:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0070000000000003e-06 2024-05-15T23:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.261042+0000", 2024-05-15T23:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8032999999999999e-05 2024-05-15T23:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:16.261051+0000", 2024-05-15T23:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4859999999999997e-06 2024-05-15T23:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:17.157994+0000", 2024-05-15T23:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.89694287800000005 2024-05-15T23:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:17.158003+0000", 2024-05-15T23:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7870000000000007e-06 2024-05-15T23:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.253 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.26326.0:253442 6.f 6:ff65e2a2:::benchmark_data_smithi090_131278_object118248:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e539)", 2024-05-15T23:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:28:18.183816+0000", 2024-05-15T23:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 403.93383891600001, 2024-05-15T23:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.523354506, 2024-05-15T23:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.26326", 2024-05-15T23:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/3286321417", 2024-05-15T23:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 253442 2024-05-15T23:35:02.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.183816+0000", 2024-05-15T23:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.183816+0000", 2024-05-15T23:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.183817+0000", 2024-05-15T23:35:02.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8100000000000002e-07 2024-05-15T23:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.183819+0000", 2024-05-15T23:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.813e-06 2024-05-15T23:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.183825+0000", 2024-05-15T23:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6980000000000003e-06 2024-05-15T23:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.183830+0000", 2024-05-15T23:35:02.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3489999999999999e-06 2024-05-15T23:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.184032+0000", 2024-05-15T23:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000201571 2024-05-15T23:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.184042+0000", 2024-05-15T23:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.008e-05 2024-05-15T23:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.683761+0000", 2024-05-15T23:35:02.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.499718889 2024-05-15T23:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.683888+0000", 2024-05-15T23:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127105 2024-05-15T23:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,10", 2024-05-15T23:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.684027+0000", 2024-05-15T23:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139114 2024-05-15T23:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.688820+0000", 2024-05-15T23:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047927300000000003 2024-05-15T23:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.698383+0000", 2024-05-15T23:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095628510000000007 2024-05-15T23:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.706986+0000", 2024-05-15T23:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0086031830000000004 2024-05-15T23:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:02.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.707038+0000", 2024-05-15T23:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2435999999999997e-05 2024-05-15T23:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:28:18.707170+0000", 2024-05-15T23:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000132006 2024-05-15T23:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:02.260 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:35:02.260 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 47 16 102570 102554 136.309 88.5 0.0118441 0.00733423 2024-05-15T23:35:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:01 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:35:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:01 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:35:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:01 smithi090 ceph-mon[44434]: osdmap e673: 12 total, 12 up, 10 in 2024-05-15T23:35:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:01 smithi090 ceph-mon[44434]: pgmap v1784: 17 pgs: 17 active+clean; 6.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-05-15T23:35:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:01 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3286343801' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:01 smithi090 ceph-mon[44434]: from='client.37067 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:01 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:35:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:01 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:35:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:01 smithi088 ceph-mon[44699]: osdmap e673: 12 total, 12 up, 10 in 2024-05-15T23:35:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:01 smithi088 ceph-mon[44699]: pgmap v1784: 17 pgs: 17 active+clean; 6.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-05-15T23:35:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:01 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3286343801' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:02.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:01 smithi088 ceph-mon[44699]: from='client.37067 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:02.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:01 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:35:02.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:01 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:35:02.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:01 smithi084 ceph-mon[39784]: osdmap e673: 12 total, 12 up, 10 in 2024-05-15T23:35:02.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:01 smithi084 ceph-mon[39784]: pgmap v1784: 17 pgs: 17 active+clean; 6.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-05-15T23:35:02.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:01 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3286343801' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:02.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:01 smithi084 ceph-mon[39784]: from='client.37067 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:02.382 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:35:02.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:02.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:35:02.488 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:02.516 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:02.537 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:02.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:02.607 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:02.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:35:02.676 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:02.718 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:35:02.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:02.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:35:02.828 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:02.861 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:02.881 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:02.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:02.912 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:02 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:35:02.912 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:02 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:35:02.912 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:02 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:35:02.912 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:02 smithi090 ceph-mon[44434]: from='client.37067 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:35:02.912 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:02 smithi090 ceph-mon[44434]: osdmap e674: 12 total, 12 up, 10 in 2024-05-15T23:35:02.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:35:03.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:02 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:35:03.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:02 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:35:03.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:02 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:35:03.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:02 smithi084 ceph-mon[39784]: from='client.37067 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:35:03.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:02 smithi084 ceph-mon[39784]: osdmap e674: 12 total, 12 up, 10 in 2024-05-15T23:35:03.019 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:03.057 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:35:03.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:03.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:35:03.133 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 48 16 103982 103966 135.308 88.25 0.00477537 0.00738887 2024-05-15T23:35:03.163 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:35:03.197 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:03.280 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:35:03.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:03.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:35:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:02 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:35:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:02 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:35:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:02 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:35:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:02 smithi088 ceph-mon[44699]: from='client.37067 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:35:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:02 smithi088 ceph-mon[44699]: osdmap e674: 12 total, 12 up, 10 in 2024-05-15T23:35:03.384 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:35:03.416 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:03.474 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:35:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:03.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:35:03.578 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:03.630 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:03.744 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:03.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:03.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T23:35:03.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:35:03.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:03 smithi084 ceph-mon[39784]: pgmap v1786: 17 pgs: 17 active+clean; 6.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-05-15T23:35:03.989 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:04.136 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 49 15 105387 105372 134.341 87.875 0.0332232 0.00744197 2024-05-15T23:35:04.140 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:35:04.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:04.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:35:04.246 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:35:04.276 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:35:04.283 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:04.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:03 smithi090 ceph-mon[44434]: pgmap v1786: 17 pgs: 17 active+clean; 6.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-05-15T23:35:04.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:03 smithi088 ceph-mon[44699]: pgmap v1786: 17 pgs: 17 active+clean; 6.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-05-15T23:35:04.339 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:35:04.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:04.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:35:04.448 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:35:04.480 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:05.060 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:35:05.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:05.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:35:05.077 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:35:05.134 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 50 16 106721 106705 133.318 83.3125 0.0123926 0.00749867 2024-05-15T23:35:05.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:35:05.216 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:05.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:05 smithi084 ceph-mon[39784]: pgmap v1787: 17 pgs: 17 active+clean; 6.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-05-15T23:35:05.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:05 smithi090 ceph-mon[44434]: pgmap v1787: 17 pgs: 17 active+clean; 6.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-05-15T23:35:05.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:05 smithi088 ceph-mon[44699]: pgmap v1787: 17 pgs: 17 active+clean; 6.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-05-15T23:35:06.079 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:35:06.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:06.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:35:06.119 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:06.134 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 51 16 108140 108124 132.444 88.6875 0.011731 0.00754897 2024-05-15T23:35:06.184 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:06.218 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:06.236 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:06.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:06.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:06.340 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:06.370 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:06.391 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:06.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:06.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:06.521 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:06 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:35:06.522 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:06 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:35:06.524 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:06.547 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:06.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:06.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:06.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:06 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:35:06.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:06 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:35:06.652 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:35:06.680 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:06.704 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:35:06.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:06.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:35:06.808 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:35:06.812 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:06 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:35:06.812 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:06 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:35:06.836 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:06.859 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:35:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:06.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:35:06.963 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:35:06.992 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:07.014 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:35:07.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:07.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:35:07.121 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:07.134 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 52 16 109376 109360 131.382 77.25 0.0124134 0.00760977 2024-05-15T23:35:07.149 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:07.170 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:07.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:07.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:07.213 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:35:07.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:35:07.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:35:07.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:35:07.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:35:07.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:35:07.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:35:07.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:35:07.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:35:07.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:35:07.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:35:07.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:35:07.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:35:07.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:35:07.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:35:07.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:35:07.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:35:07.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:35:07.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:35:07.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:35:07.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:35:07.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:35:07.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:35:07.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:35:07.278 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:07.310 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:07.335 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:07.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:07.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:07.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:35:07.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:07 smithi090 ceph-mon[44434]: osdmap e675: 12 total, 12 up, 10 in 2024-05-15T23:35:07.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:07 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3960814430' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:35:07.503 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:07 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:07 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:35:07.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:07 smithi090 ceph-mon[44434]: pgmap v1789: 17 pgs: 17 active+clean; 6.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-05-15T23:35:07.504 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:07 smithi090 ceph-mon[44434]: osdmap e676: 12 total, 12 up, 10 in 2024-05-15T23:35:07.507 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:07.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:07 smithi088 ceph-mon[44699]: osdmap e675: 12 total, 12 up, 10 in 2024-05-15T23:35:07.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:07 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3960814430' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:35:07.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:07 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:07 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:35:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:07 smithi088 ceph-mon[44699]: pgmap v1789: 17 pgs: 17 active+clean; 6.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-05-15T23:35:07.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:07 smithi088 ceph-mon[44699]: osdmap e676: 12 total, 12 up, 10 in 2024-05-15T23:35:07.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:07 smithi084 ceph-mon[39784]: osdmap e675: 12 total, 12 up, 10 in 2024-05-15T23:35:07.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:07 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3960814430' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:35:07.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:07 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:07.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:07 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:35:07.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:07 smithi084 ceph-mon[39784]: pgmap v1789: 17 pgs: 17 active+clean; 6.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-05-15T23:35:07.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:07 smithi084 ceph-mon[39784]: osdmap e676: 12 total, 12 up, 10 in 2024-05-15T23:35:07.729 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:35:07.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:07.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:35:07.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:35:07.954 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:07.996 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:35:07.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:08.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:35:08.101 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:35:08.131 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:08.138 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 53 16 110595 110579 130.335 76.1875 0.0115719 0.00767125 2024-05-15T23:35:08.153 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:35:08.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:08.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:35:08.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:35:08.295 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:08.480 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:35:08.480 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:35:08.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:08.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:08.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:08.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:08.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:08.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:08.746 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:35:08.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:08.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:35:08.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:08.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:08.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:08.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:08.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:08.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:35:08.896 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:08.934 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:35:08.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:08.941 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:35:09.043 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:09.072 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:09.094 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:09.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:09.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:09.137 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 54 16 111938 111922 129.476 83.9375 0.0052585 0.00772187 2024-05-15T23:35:09.197 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:35:09.228 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:09.252 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:35:09.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:09.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:35:09.356 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:35:09.378 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:35:09.384 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:09.406 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:35:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:09.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:35:09.509 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:35:09.539 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:09.560 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:35:09.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:09.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:35:09.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:09 smithi084 ceph-mon[39784]: pgmap v1791: 17 pgs: 17 active+clean; 6.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-05-15T23:35:09.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:09 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3626333529' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:35:09.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:09 smithi084 ceph-mon[39784]: from='client.37085 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:35:09.665 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:09.717 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:09 smithi090 ceph-mon[44434]: pgmap v1791: 17 pgs: 17 active+clean; 6.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-05-15T23:35:09.717 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:09 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3626333529' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:35:09.717 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:09 smithi090 ceph-mon[44434]: from='client.37085 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:35:09.718 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:09.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:09 smithi088 ceph-mon[44699]: pgmap v1791: 17 pgs: 17 active+clean; 6.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-05-15T23:35:09.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:09 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3626333529' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:35:09.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:09 smithi088 ceph-mon[44699]: from='client.37085 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:35:10.100 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 11, 9, 10, 2, 8] out_osds: [0, 6] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:35:10.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 1 1 2024-05-15T23:35:10.137 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 55 16 112934 112918 128.254 62.25 0.00412922 0.00779579 2024-05-15T23:35:10.383 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:10.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:10.387 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:10.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:10.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:35:10.538 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:10.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:10 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:35:10.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:10 smithi084 ceph-mon[39784]: from='client.37085 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:35:10.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:10 smithi084 ceph-mon[39784]: osdmap e677: 12 total, 12 up, 10 in 2024-05-15T23:35:10.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:10 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:35:10.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:10 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:35:10.784 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:10 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:35:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:10 smithi090 ceph-mon[44434]: from='client.37085 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:35:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:10 smithi090 ceph-mon[44434]: osdmap e677: 12 total, 12 up, 10 in 2024-05-15T23:35:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:10 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:35:10.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:10 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:35:10.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:10 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:35:10.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:10 smithi088 ceph-mon[44699]: from='client.37085 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:35:10.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:10 smithi088 ceph-mon[44699]: osdmap e677: 12 total, 12 up, 10 in 2024-05-15T23:35:10.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:10 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:35:10.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:10 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:35:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:10.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:10.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:35:10.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T23:35:11.002 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:11.043 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:35:11.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:11.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:35:11.137 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 56 15 114020 114005 127.177 67.9375 0.00975507 0.00786136 2024-05-15T23:35:11.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:35:11.183 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:11.474 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:35:11.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:11 smithi084 ceph-mon[39784]: 8.9 scrub starts 2024-05-15T23:35:11.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:11 smithi084 ceph-mon[39784]: osdmap e678: 12 total, 12 up, 10 in 2024-05-15T23:35:11.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:11 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:35:11.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:11 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:35:11.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:11 smithi084 ceph-mon[39784]: pgmap v1794: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-05-15T23:35:11.637 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:35:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:11.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:35:11.687 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:11.697 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:11.742 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:35:11.775 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:11 smithi090 ceph-mon[44434]: 8.9 scrub starts 2024-05-15T23:35:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:11 smithi090 ceph-mon[44434]: osdmap e678: 12 total, 12 up, 10 in 2024-05-15T23:35:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:11 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:35:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:11 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:35:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:11 smithi090 ceph-mon[44434]: pgmap v1794: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-05-15T23:35:11.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:11 smithi088 ceph-mon[44699]: 8.9 scrub starts 2024-05-15T23:35:11.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:11 smithi088 ceph-mon[44699]: osdmap e678: 12 total, 12 up, 10 in 2024-05-15T23:35:11.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:11 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:35:11.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:11 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:35:11.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:11 smithi088 ceph-mon[44699]: pgmap v1794: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-05-15T23:35:11.836 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:35:11.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:11.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:35:11.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:35:11.972 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:12.137 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 57 16 115229 115213 126.27 75.5 0.0132004 0.00791776 2024-05-15T23:35:12.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:12 smithi090 ceph-mon[44434]: 8.9 scrub ok 2024-05-15T23:35:12.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:12 smithi090 ceph-mon[44434]: 8.8 scrub starts 2024-05-15T23:35:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:12 smithi088 ceph-mon[44699]: 8.9 scrub ok 2024-05-15T23:35:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:12 smithi088 ceph-mon[44699]: 8.8 scrub starts 2024-05-15T23:35:12.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:12 smithi084 ceph-mon[39784]: 8.9 scrub ok 2024-05-15T23:35:12.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:12 smithi084 ceph-mon[39784]: 8.8 scrub starts 2024-05-15T23:35:13.137 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 58 16 116285 116269 125.231 66 0.00995827 0.00798249 2024-05-15T23:35:13.629 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:35:13.655 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:13 smithi084 ceph-mon[39784]: 8.b scrub starts 2024-05-15T23:35:13.655 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:13 smithi084 ceph-mon[39784]: 8.8 scrub ok 2024-05-15T23:35:13.655 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:13 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:35:13.655 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:13 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:35:13.656 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:13 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:35:13.656 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:13 smithi084 ceph-mon[39784]: pgmap v1795: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-05-15T23:35:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:13 smithi090 ceph-mon[44434]: 8.b scrub starts 2024-05-15T23:35:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:13 smithi090 ceph-mon[44434]: 8.8 scrub ok 2024-05-15T23:35:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:13 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:35:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:13 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:35:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:13 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:35:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:13 smithi090 ceph-mon[44434]: pgmap v1795: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-05-15T23:35:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:13 smithi088 ceph-mon[44699]: 8.b scrub starts 2024-05-15T23:35:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:13 smithi088 ceph-mon[44699]: 8.8 scrub ok 2024-05-15T23:35:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:13 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:35:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:13 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:35:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:13 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:35:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:13 smithi088 ceph-mon[44699]: pgmap v1795: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-05-15T23:35:13.830 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:35:13.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:13.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:35:13.941 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:13.970 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:14.006 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:14.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:14.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:14.110 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:14.137 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 59 16 117180 117164 124.057 55.9375 0.0234926 0.0080595 2024-05-15T23:35:14.138 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:14.160 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:14.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:14.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:14.265 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:14.295 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:14.315 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:14.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:14.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:14.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:35:14.472 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:15.047 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:35:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:15.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:35:15.138 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:35:15.137877 min lat: 0.00114746 max lat: 0.266312 avg lat: 0.00811039 2024-05-15T23:35:15.138 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:35:15.138 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 60 16 118418 118402 123.28 77.375 0.00631678 0.00811039 2024-05-15T23:35:15.154 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:15.181 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:15.187 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:15.205 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:15.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:15.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:15.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:15 smithi090 ceph-mon[44434]: 8.a scrub starts 2024-05-15T23:35:15.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:15 smithi090 ceph-mon[44434]: 8.b scrub ok 2024-05-15T23:35:15.307 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:15 smithi090 ceph-mon[44434]: 8.6 scrub starts 2024-05-15T23:35:15.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:35:15.378 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:15.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:15 smithi088 ceph-mon[44699]: 8.a scrub starts 2024-05-15T23:35:15.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:15 smithi088 ceph-mon[44699]: 8.b scrub ok 2024-05-15T23:35:15.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:15 smithi088 ceph-mon[44699]: 8.6 scrub starts 2024-05-15T23:35:15.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:15 smithi084 ceph-mon[39784]: 8.a scrub starts 2024-05-15T23:35:15.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:15 smithi084 ceph-mon[39784]: 8.b scrub ok 2024-05-15T23:35:15.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:15 smithi084 ceph-mon[39784]: 8.6 scrub starts 2024-05-15T23:35:16.138 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 61 16 119450 119434 122.316 64.5 0.00384284 0.00815692 2024-05-15T23:35:16.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:16 smithi090 ceph-mon[44434]: 8.a scrub ok 2024-05-15T23:35:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:16 smithi090 ceph-mon[44434]: pgmap v1796: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-05-15T23:35:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:35:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:35:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:35:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:35:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:35:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:35:16.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:35:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:16 smithi088 ceph-mon[44699]: 8.a scrub ok 2024-05-15T23:35:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:16 smithi088 ceph-mon[44699]: pgmap v1796: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-05-15T23:35:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:35:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:35:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:35:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:35:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:35:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:35:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:35:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:16 smithi084 ceph-mon[39784]: 8.a scrub ok 2024-05-15T23:35:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:16 smithi084 ceph-mon[39784]: pgmap v1796: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-05-15T23:35:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:35:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:35:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:35:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:35:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:35:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:35:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:35:17.076 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:35:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:17.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:35:17.138 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 62 16 120582 120566 121.485 70.75 0.00516993 0.00822953 2024-05-15T23:35:17.181 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:17.459 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:17 smithi090 ceph-mon[44434]: 8.6 scrub ok 2024-05-15T23:35:17.469 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:17.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:17.474 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:17.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:17 smithi088 ceph-mon[44699]: 8.6 scrub ok 2024-05-15T23:35:17.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:17.603 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:17.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:17 smithi084 ceph-mon[39784]: 8.6 scrub ok 2024-05-15T23:35:17.623 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:17.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:17.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:17.727 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:35:17.768 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:17.975 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:35:17.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:17.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:35:18.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:18.138 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 63 15 121775 121760 120.74 74.625 0.028193 0.00828071 2024-05-15T23:35:18.143 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:18.174 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:18.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:18.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:18.281 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:18.318 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2024-05-15T23:35:18.338 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:18.366 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:18.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:18.373 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:18.474 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:18.502 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:35:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:35:18.521 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:18.523 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:18 smithi088 ceph-mon[44699]: pgmap v1797: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-05-15T23:35:18.523 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:18 smithi088 ceph-mon[44699]: 8.d scrub starts 2024-05-15T23:35:18.523 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:18 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2817355725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:35:18.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:18.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:18.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:18 smithi090 ceph-mon[44434]: pgmap v1797: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-05-15T23:35:18.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:18 smithi090 ceph-mon[44434]: 8.d scrub starts 2024-05-15T23:35:18.526 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:18 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2817355725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:35:18.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:18 smithi084 ceph-mon[39784]: pgmap v1797: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-05-15T23:35:18.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:18 smithi084 ceph-mon[39784]: 8.d scrub starts 2024-05-15T23:35:18.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:18 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2817355725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:35:18.624 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:18.652 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:18.676 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:18.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:18.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:18.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:35:18.809 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:18.842 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:35:18.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:18.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:35:18.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:35:18.997 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:19.141 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 64 16 122600 122584 119.654 51.5 0.0342983 0.00835622 2024-05-15T23:35:19.324 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:35:19.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 3 1 2024-05-15T23:35:19.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:19 smithi088 ceph-mon[44699]: 8.1 scrub starts 2024-05-15T23:35:19.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2817355725' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:35:19.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:19 smithi088 ceph-mon[44699]: osdmap e679: 12 total, 12 up, 10 in 2024-05-15T23:35:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3515014500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:35:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1654072315' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:19 smithi088 ceph-mon[44699]: from='client.37097 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:19 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:19 smithi088 ceph-mon[44699]: pgmap v1799: 17 pgs: 17 active+clean; 7.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T23:35:19.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:19 smithi084 ceph-mon[39784]: 8.1 scrub starts 2024-05-15T23:35:19.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2817355725' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:35:19.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:19 smithi084 ceph-mon[39784]: osdmap e679: 12 total, 12 up, 10 in 2024-05-15T23:35:19.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3515014500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:35:19.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1654072315' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:19.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:19 smithi084 ceph-mon[39784]: from='client.37097 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:19.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:19 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:19 smithi084 ceph-mon[39784]: pgmap v1799: 17 pgs: 17 active+clean; 7.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T23:35:19.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:19 smithi090 ceph-mon[44434]: 8.1 scrub starts 2024-05-15T23:35:19.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2817355725' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:35:19.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:19 smithi090 ceph-mon[44434]: osdmap e679: 12 total, 12 up, 10 in 2024-05-15T23:35:19.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3515014500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:35:19.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1654072315' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:19.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:19 smithi090 ceph-mon[44434]: from='client.37097 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:19.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:19 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:19.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:19 smithi090 ceph-mon[44434]: pgmap v1799: 17 pgs: 17 active+clean; 7.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-05-15T23:35:19.891 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:35:19.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:19.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:35:20.004 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:35:20.034 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:20.056 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:35:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:20.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:35:20.143 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 65 16 123531 123515 118.704 58.1875 0.0149688 0.00842281 2024-05-15T23:35:20.161 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:20.189 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:20.210 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:20.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:20.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:20.215 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:35:20.215 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:35:20.215 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:35:20.216 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:35:20.216 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:35:20.216 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:35:20.216 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:35:20.216 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:35:20.216 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:35:20.216 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:35:20.216 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:35:20.216 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:35:20.216 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:35:20.216 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:35:20.216 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:35:20.217 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:35:20.217 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:35:20.217 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:35:20.217 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:35:20.217 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:35:20.217 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:35:20.217 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:35:20.217 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:35:20.217 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:35:20.315 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:35:20.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:20 smithi088 ceph-mon[44699]: 8.1 scrub ok 2024-05-15T23:35:20.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:20 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:35:20.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:20 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:35:20.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:20 smithi088 ceph-mon[44699]: from='client.37097 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:35:20.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:20 smithi088 ceph-mon[44699]: osdmap e680: 12 total, 12 up, 10 in 2024-05-15T23:35:20.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:20 smithi088 ceph-mon[44699]: 8.3 scrub starts 2024-05-15T23:35:20.343 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:20.365 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:35:20.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:20 smithi084 ceph-mon[39784]: 8.1 scrub ok 2024-05-15T23:35:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:20 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:35:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:20 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:35:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:20 smithi084 ceph-mon[39784]: from='client.37097 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:35:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:20 smithi084 ceph-mon[39784]: osdmap e680: 12 total, 12 up, 10 in 2024-05-15T23:35:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:20 smithi084 ceph-mon[39784]: 8.3 scrub starts 2024-05-15T23:35:20.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:20.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:35:20.471 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:20.519 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:20.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:20.623 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:20.672 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:20.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:20.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:20.776 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:20.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:20.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:20 smithi090 ceph-mon[44434]: 8.1 scrub ok 2024-05-15T23:35:20.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:20 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:35:20.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:20 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:35:20.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:20 smithi090 ceph-mon[44434]: from='client.37097 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:35:20.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:20 smithi090 ceph-mon[44434]: osdmap e680: 12 total, 12 up, 10 in 2024-05-15T23:35:20.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:20 smithi090 ceph-mon[44434]: 8.3 scrub starts 2024-05-15T23:35:20.848 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:20.995 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:21.143 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 66 16 124440 124424 117.767 56.8125 0.010765 0.00848291 2024-05-15T23:35:21.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T23:35:21.419 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:21.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:21.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:21.528 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:21.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:21 smithi084 ceph-mon[39784]: 8.5 scrub starts 2024-05-15T23:35:21.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:21 smithi084 ceph-mon[39784]: 8.d scrub ok 2024-05-15T23:35:21.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:21 smithi084 ceph-mon[39784]: pgmap v1801: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 60 MiB/s wr, 964 op/s 2024-05-15T23:35:21.560 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:21.582 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:21.584 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:21 smithi088 ceph-mon[44699]: 8.5 scrub starts 2024-05-15T23:35:21.584 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:21 smithi088 ceph-mon[44699]: 8.d scrub ok 2024-05-15T23:35:21.584 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:21 smithi088 ceph-mon[44699]: pgmap v1801: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 60 MiB/s wr, 964 op/s 2024-05-15T23:35:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:21.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:21.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:35:21.717 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:21 smithi090 ceph-mon[44434]: 8.5 scrub starts 2024-05-15T23:35:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:21 smithi090 ceph-mon[44434]: 8.d scrub ok 2024-05-15T23:35:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:21 smithi090 ceph-mon[44434]: pgmap v1801: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 60 MiB/s wr, 964 op/s 2024-05-15T23:35:21.866 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:35:21.866 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:35:21.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:21.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:35:21.973 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:22.002 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:22.025 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:22.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:22.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:22.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:35:22.136 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:35:22.143 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 67 16 125473 125457 116.973 64.5625 0.0151016 0.00854732 2024-05-15T23:35:22.197 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:22.233 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:35:22.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:22.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:35:22.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:35:22.372 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:22.407 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:35:22.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:22.412 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:35:22.420 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:22 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:35:22.420 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:22 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:35:22.421 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:22 smithi090 ceph-mon[44434]: 8.5 scrub ok 2024-05-15T23:35:22.421 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:22 smithi090 ceph-mon[44434]: 8.3 scrub ok 2024-05-15T23:35:22.421 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:22 smithi090 ceph-mon[44434]: 8.f scrub starts 2024-05-15T23:35:22.493 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:22.513 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:22.543 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:22.565 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:22.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:22.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:22.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:22 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:35:22.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:22 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:35:22.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:22 smithi084 ceph-mon[39784]: 8.5 scrub ok 2024-05-15T23:35:22.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:22 smithi084 ceph-mon[39784]: 8.3 scrub ok 2024-05-15T23:35:22.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:22 smithi084 ceph-mon[39784]: 8.f scrub starts 2024-05-15T23:35:22.673 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:22.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:22 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:35:22.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:22 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:35:22.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:22 smithi088 ceph-mon[44699]: 8.5 scrub ok 2024-05-15T23:35:22.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:22 smithi088 ceph-mon[44699]: 8.3 scrub ok 2024-05-15T23:35:22.699 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:22 smithi088 ceph-mon[44699]: 8.f scrub starts 2024-05-15T23:35:22.701 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:22.722 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:22.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:22.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:35:22.884 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:22.890 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:35:23.147 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 68 16 126538 126522 116.226 66.5625 0.00978029 0.00860231 2024-05-15T23:35:23.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:23 smithi084 ceph-mon[39784]: 8.4 scrub starts 2024-05-15T23:35:23.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:23 smithi084 ceph-mon[39784]: pgmap v1802: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T23:35:23.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:23 smithi090 ceph-mon[44434]: 8.4 scrub starts 2024-05-15T23:35:23.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:23 smithi090 ceph-mon[44434]: pgmap v1802: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T23:35:23.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:23 smithi088 ceph-mon[44699]: 8.4 scrub starts 2024-05-15T23:35:23.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:23 smithi088 ceph-mon[44699]: pgmap v1802: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-05-15T23:35:24.147 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 69 16 127694 127678 115.589 72.25 0.00683033 0.00864972 2024-05-15T23:35:24.163 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:35:24.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:24.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:35:24.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:35:24.301 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:24.328 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:24.372 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-05-15T23:35:24.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:24 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/154708281' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:35:24.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:24 smithi090 ceph-mon[44434]: 8.f scrub ok 2024-05-15T23:35:24.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:24 smithi090 ceph-mon[44434]: 8.4 scrub ok 2024-05-15T23:35:24.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:24 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/154708281' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:35:24.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:24 smithi084 ceph-mon[39784]: 8.f scrub ok 2024-05-15T23:35:24.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:24 smithi084 ceph-mon[39784]: 8.4 scrub ok 2024-05-15T23:35:24.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:24 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/154708281' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:35:24.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:24 smithi088 ceph-mon[44699]: 8.f scrub ok 2024-05-15T23:35:24.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:24 smithi088 ceph-mon[44699]: 8.4 scrub ok 2024-05-15T23:35:24.999 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:35:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:25.006 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:35:25.106 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:35:25.148 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 70 16 128856 128840 114.974 72.625 0.00775431 0.00869599 2024-05-15T23:35:25.149 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:25.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:25 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/154708281' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:35:25.705 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:25 smithi090 ceph-mon[44434]: osdmap e681: 12 total, 12 up, 10 in 2024-05-15T23:35:25.706 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:25 smithi090 ceph-mon[44434]: pgmap v1804: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 53 MiB/s wr, 852 op/s 2024-05-15T23:35:25.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:25 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/154708281' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:35:25.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:25 smithi088 ceph-mon[44699]: osdmap e681: 12 total, 12 up, 10 in 2024-05-15T23:35:25.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:25 smithi088 ceph-mon[44699]: pgmap v1804: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 53 MiB/s wr, 852 op/s 2024-05-15T23:35:25.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:25 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/154708281' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:35:25.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:25 smithi084 ceph-mon[39784]: osdmap e681: 12 total, 12 up, 10 in 2024-05-15T23:35:25.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:25 smithi084 ceph-mon[39784]: pgmap v1804: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 53 MiB/s wr, 852 op/s 2024-05-15T23:35:25.871 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 4 1 2024-05-15T23:35:26.148 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 71 16 129832 129816 114.214 61 0.0279186 0.0087529 2024-05-15T23:35:26.303 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:35:26.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:26.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:35:26.395 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:35:26.411 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:26.439 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:26.462 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:26.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:26.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:26.567 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:35:26.600 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:26.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:26 smithi084 ceph-mon[39784]: 8.2 scrub starts 2024-05-15T23:35:26.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:26 smithi084 ceph-mon[39784]: 8.c scrub starts 2024-05-15T23:35:26.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:26 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3868454329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:35:26.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:26 smithi084 ceph-mon[39784]: from='client.37112 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:35:26.623 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:35:26.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:26.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:35:26.682 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:35:26.682 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:35:26.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:35:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:35:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124079 7.1 7:857eceea:::benchmark_data_smithi090_143373_object124078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e579)", 2024-05-15T23:35:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.261908+0000", 2024-05-15T23:35:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.41681733299998, 2024-05-15T23:35:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26989557400000003, 2024-05-15T23:35:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124079 2024-05-15T23:35:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261908+0000", 2024-05-15T23:35:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261908+0000", 2024-05-15T23:35:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261911+0000", 2024-05-15T23:35:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4880000000000001e-06 2024-05-15T23:35:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261938+0000", 2024-05-15T23:35:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7135e-05 2024-05-15T23:35:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261952+0000", 2024-05-15T23:35:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4389999999999999e-05 2024-05-15T23:35:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261963+0000", 2024-05-15T23:35:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1094000000000001e-05 2024-05-15T23:35:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263369+0000", 2024-05-15T23:35:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014062160000000001 2024-05-15T23:35:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263378+0000", 2024-05-15T23:35:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4579999999999993e-06 2024-05-15T23:35:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.497009+0000", 2024-05-15T23:35:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23363141000000001 2024-05-15T23:35:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.499810+0000", 2024-05-15T23:35:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028003059999999998 2024-05-15T23:35:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.499987+0000", 2024-05-15T23:35:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176987 2024-05-15T23:35:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.510156+0000", 2024-05-15T23:35:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010169076000000001 2024-05-15T23:35:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.528308+0000", 2024-05-15T23:35:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018152166000000001 2024-05-15T23:35:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.528368+0000", 2024-05-15T23:35:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9679999999999998e-05 2024-05-15T23:35:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.531804+0000", 2024-05-15T23:35:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034361679999999999 2024-05-15T23:35:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124097 7.1 7:8475dbd8:::benchmark_data_smithi090_143373_object124096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:35:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.261987+0000", 2024-05-15T23:35:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.41673817100002, 2024-05-15T23:35:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28773406099999999, 2024-05-15T23:35:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124097 2024-05-15T23:35:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261987+0000", 2024-05-15T23:35:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261987+0000", 2024-05-15T23:35:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261988+0000", 2024-05-15T23:35:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3799999999999997e-07 2024-05-15T23:35:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262034+0000", 2024-05-15T23:35:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6520000000000002e-05 2024-05-15T23:35:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262041+0000", 2024-05-15T23:35:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5590000000000002e-06 2024-05-15T23:35:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262045+0000", 2024-05-15T23:35:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0110000000000002e-06 2024-05-15T23:35:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263404+0000", 2024-05-15T23:35:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013591110000000001 2024-05-15T23:35:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263410+0000", 2024-05-15T23:35:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780000000000003e-06 2024-05-15T23:35:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.506311+0000", 2024-05-15T23:35:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.242901425 2024-05-15T23:35:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.506407+0000", 2024-05-15T23:35:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5315999999999993e-05 2024-05-15T23:35:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.506505+0000", 2024-05-15T23:35:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8548999999999994e-05 2024-05-15T23:35:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.521629+0000", 2024-05-15T23:35:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015123319 2024-05-15T23:35:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.546928+0000", 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025299687000000001 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.546964+0000", 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5552000000000001e-05 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.549721+0000", 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027575960000000002 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124098 7.1 7:85fd872c:::benchmark_data_smithi090_143373_object124097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:35:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.262051+0000", 2024-05-15T23:35:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.41667479, 2024-05-15T23:35:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41649122300000002, 2024-05-15T23:35:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124098 2024-05-15T23:35:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262051+0000", 2024-05-15T23:35:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262051+0000", 2024-05-15T23:35:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262051+0000", 2024-05-15T23:35:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5699999999999998e-07 2024-05-15T23:35:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262088+0000", 2024-05-15T23:35:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6467000000000002e-05 2024-05-15T23:35:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262099+0000", 2024-05-15T23:35:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1422e-05 2024-05-15T23:35:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262111+0000", 2024-05-15T23:35:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1593000000000001e-05 2024-05-15T23:35:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263432+0000", 2024-05-15T23:35:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013212580000000001 2024-05-15T23:35:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263436+0000", 2024-05-15T23:35:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9310000000000001e-06 2024-05-15T23:35:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.510182+0000", 2024-05-15T23:35:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.24674654900000001 2024-05-15T23:35:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.510289+0000", 2024-05-15T23:35:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106941 2024-05-15T23:35:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.510417+0000", 2024-05-15T23:35:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012745399999999999 2024-05-15T23:35:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.521638+0000", 2024-05-15T23:35:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011221085 2024-05-15T23:35:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.675885+0000", 2024-05-15T23:35:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15424710599999999 2024-05-15T23:35:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.675949+0000", 2024-05-15T23:35:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4082000000000001e-05 2024-05-15T23:35:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.678542+0000", 2024-05-15T23:35:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025928779999999998 2024-05-15T23:35:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.702 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.702 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124106 7.1 7:8dcf5eef:::benchmark_data_smithi090_143373_object124105:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:35:26.702 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.262116+0000", 2024-05-15T23:35:26.702 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.41660941100002, 2024-05-15T23:35:26.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41946845300000002, 2024-05-15T23:35:26.702 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.702 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.702 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.702 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.702 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.702 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.702 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124106 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262116+0000", 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262116+0000", 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262116+0000", 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1399999999999997e-07 2024-05-15T23:35:26.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262149+0000", 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2295999999999997e-05 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262154+0000", 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9130000000000002e-06 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262158+0000", 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0810000000000004e-06 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263455+0000", 2024-05-15T23:35:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012971969999999999 2024-05-15T23:35:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263459+0000", 2024-05-15T23:35:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7859999999999998e-06 2024-05-15T23:35:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.513785+0000", 2024-05-15T23:35:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25032668899999999 2024-05-15T23:35:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.513908+0000", 2024-05-15T23:35:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012284899999999999 2024-05-15T23:35:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.514012+0000", 2024-05-15T23:35:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103525 2024-05-15T23:35:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.528239+0000", 2024-05-15T23:35:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014226946000000001 2024-05-15T23:35:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.678901+0000", 2024-05-15T23:35:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15066216900000001 2024-05-15T23:35:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.678961+0000", 2024-05-15T23:35:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0356000000000003e-05 2024-05-15T23:35:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.681585+0000", 2024-05-15T23:35:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026232320000000001 2024-05-15T23:35:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124122 7.1 7:88203a70:::benchmark_data_smithi090_143373_object124121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:35:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.262162+0000", 2024-05-15T23:35:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.41656367299998, 2024-05-15T23:35:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.42187396700000002, 2024-05-15T23:35:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124122 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262162+0000", 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262162+0000", 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262162+0000", 2024-05-15T23:35:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2900000000000004e-07 2024-05-15T23:35:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262193+0000", 2024-05-15T23:35:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0460999999999999e-05 2024-05-15T23:35:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262208+0000", 2024-05-15T23:35:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5325e-05 2024-05-15T23:35:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262212+0000", 2024-05-15T23:35:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4809999999999998e-06 2024-05-15T23:35:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263485+0000", 2024-05-15T23:35:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001273176 2024-05-15T23:35:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263491+0000", 2024-05-15T23:35:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1480000000000003e-06 2024-05-15T23:35:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.516405+0000", 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.252914099 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.516538+0000", 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013272499999999999 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.516687+0000", 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149693 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.528245+0000", 2024-05-15T23:35:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011557596 2024-05-15T23:35:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.681654+0000", 2024-05-15T23:35:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15340879099999999 2024-05-15T23:35:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.681714+0000", 2024-05-15T23:35:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9806000000000003e-05 2024-05-15T23:35:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.684036+0000", 2024-05-15T23:35:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023221370000000002 2024-05-15T23:35:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124161 7.1 7:80dd3600:::benchmark_data_smithi090_143373_object124160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:35:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.262216+0000", 2024-05-15T23:35:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.41650934299997, 2024-05-15T23:35:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.42874030499999999, 2024-05-15T23:35:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124161 2024-05-15T23:35:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262216+0000", 2024-05-15T23:35:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262216+0000", 2024-05-15T23:35:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262217+0000", 2024-05-15T23:35:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8700000000000005e-07 2024-05-15T23:35:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262251+0000", 2024-05-15T23:35:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4162000000000003e-05 2024-05-15T23:35:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262256+0000", 2024-05-15T23:35:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5850000000000001e-06 2024-05-15T23:35:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262259+0000", 2024-05-15T23:35:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7790000000000002e-06 2024-05-15T23:35:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263546+0000", 2024-05-15T23:35:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012862450000000001 2024-05-15T23:35:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263552+0000", 2024-05-15T23:35:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9000000000000003e-06 2024-05-15T23:35:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.518749+0000", 2024-05-15T23:35:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25519711699999997 2024-05-15T23:35:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.518838+0000", 2024-05-15T23:35:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8896999999999999e-05 2024-05-15T23:35:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.518935+0000", 2024-05-15T23:35:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7421000000000001e-05 2024-05-15T23:35:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.535377+0000", 2024-05-15T23:35:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016442432 2024-05-15T23:35:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.687575+0000", 2024-05-15T23:35:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15219728799999999 2024-05-15T23:35:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.687629+0000", 2024-05-15T23:35:26.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4429999999999999e-05 2024-05-15T23:35:26.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.690956+0000", 2024-05-15T23:35:26.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033272620000000001 2024-05-15T23:35:26.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.720 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.720 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124174 7.1 7:856e5dbc:::benchmark_data_smithi090_143373_object124173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:35:26.720 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.262264+0000", 2024-05-15T23:35:26.720 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.41646181800002, 2024-05-15T23:35:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.43172603300000001, 2024-05-15T23:35:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124174 2024-05-15T23:35:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262264+0000", 2024-05-15T23:35:26.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262264+0000", 2024-05-15T23:35:26.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262264+0000", 2024-05-15T23:35:26.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0499999999999999e-07 2024-05-15T23:35:26.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262296+0000", 2024-05-15T23:35:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2254e-05 2024-05-15T23:35:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262301+0000", 2024-05-15T23:35:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.724e-06 2024-05-15T23:35:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262305+0000", 2024-05-15T23:35:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7469999999999999e-06 2024-05-15T23:35:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263583+0000", 2024-05-15T23:35:26.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012785819999999999 2024-05-15T23:35:26.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:26.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263587+0000", 2024-05-15T23:35:26.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.732e-06 2024-05-15T23:35:26.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.521673+0000", 2024-05-15T23:35:26.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25808625800000001 2024-05-15T23:35:26.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.521777+0000", 2024-05-15T23:35:26.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010374600000000001 2024-05-15T23:35:26.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.521947+0000", 2024-05-15T23:35:26.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017013099999999999 2024-05-15T23:35:26.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.535382+0000", 2024-05-15T23:35:26.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013435163 2024-05-15T23:35:26.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.691026+0000", 2024-05-15T23:35:26.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15564313299999999 2024-05-15T23:35:26.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.691082+0000", 2024-05-15T23:35:26.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6496000000000003e-05 2024-05-15T23:35:26.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.693990+0000", 2024-05-15T23:35:26.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029076620000000001 2024-05-15T23:35:26.727 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.727 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.727 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.727 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124212 7.1 7:81100074:::benchmark_data_smithi090_143373_object124211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:35:26.727 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.324932+0000", 2024-05-15T23:35:26.727 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.35379394400002, 2024-05-15T23:35:26.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37580514700000001, 2024-05-15T23:35:26.727 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.727 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.727 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.728 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.728 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.728 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124212 2024-05-15T23:35:26.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.728 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324932+0000", 2024-05-15T23:35:26.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324932+0000", 2024-05-15T23:35:26.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324933+0000", 2024-05-15T23:35:26.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.497e-06 2024-05-15T23:35:26.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324974+0000", 2024-05-15T23:35:26.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1214999999999998e-05 2024-05-15T23:35:26.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324984+0000", 2024-05-15T23:35:26.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7359999999999996e-06 2024-05-15T23:35:26.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324989+0000", 2024-05-15T23:35:26.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.259e-06 2024-05-15T23:35:26.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.325040+0000", 2024-05-15T23:35:26.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1172999999999999e-05 2024-05-15T23:35:26.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.325051+0000", 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0317e-05 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.524923+0000", 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19987254500000001 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.525080+0000", 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015695999999999999 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.525237+0000", 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015635200000000001 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.540733+0000", 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01549625 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.697259+0000", 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15652601099999999 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.697319+0000", 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0210999999999997e-05 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.700737+0000", 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003417621 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124224 7.1 7:81c65e53:::benchmark_data_smithi090_143373_object124223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.344409+0000", 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.33431626100003, 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.371311897, 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.734 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.734 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.734 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.734 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.734 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124224 2024-05-15T23:35:26.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.734 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344409+0000", 2024-05-15T23:35:26.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344409+0000", 2024-05-15T23:35:26.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344412+0000", 2024-05-15T23:35:26.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3180000000000001e-06 2024-05-15T23:35:26.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344455+0000", 2024-05-15T23:35:26.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3247000000000001e-05 2024-05-15T23:35:26.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344466+0000", 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0922e-05 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344472+0000", 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2790000000000004e-06 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344531+0000", 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9055000000000003e-05 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:26.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344539+0000", 2024-05-15T23:35:26.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6040000000000004e-06 2024-05-15T23:35:26.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.531872+0000", 2024-05-15T23:35:26.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.187333584 2024-05-15T23:35:26.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.532039+0000", 2024-05-15T23:35:26.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016709099999999999 2024-05-15T23:35:26.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.532223+0000", 2024-05-15T23:35:26.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000183246 2024-05-15T23:35:26.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.545260+0000", 2024-05-15T23:35:26.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013037648000000001 2024-05-15T23:35:26.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.714171+0000", 2024-05-15T23:35:26.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.168910483 2024-05-15T23:35:26.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.714217+0000", 2024-05-15T23:35:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6461999999999997e-05 2024-05-15T23:35:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.715721+0000", 2024-05-15T23:35:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015039579999999999 2024-05-15T23:35:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124234 7.1 7:8b71c4c6:::benchmark_data_smithi090_143373_object124233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:35:26.740 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.366886+0000", 2024-05-15T23:35:26.740 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.31183934799998, 2024-05-15T23:35:26.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35046440299999998, 2024-05-15T23:35:26.740 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.740 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.740 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.740 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.740 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.740 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.740 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124234 2024-05-15T23:35:26.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.740 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366886+0000", 2024-05-15T23:35:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366886+0000", 2024-05-15T23:35:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366888+0000", 2024-05-15T23:35:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8920000000000001e-06 2024-05-15T23:35:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366934+0000", 2024-05-15T23:35:26.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5543000000000001e-05 2024-05-15T23:35:26.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366945+0000", 2024-05-15T23:35:26.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1345999999999999e-05 2024-05-15T23:35:26.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366951+0000", 2024-05-15T23:35:26.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3400000000000003e-06 2024-05-15T23:35:26.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.367009+0000", 2024-05-15T23:35:26.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7907e-05 2024-05-15T23:35:26.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:26.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.367022+0000", 2024-05-15T23:35:26.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2492e-05 2024-05-15T23:35:26.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.535417+0000", 2024-05-15T23:35:26.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16839585300000001 2024-05-15T23:35:26.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.535529+0000", 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011196600000000001 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.535644+0000", 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011408099999999999 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.545264+0000", 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096206910000000007 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.715777+0000", 2024-05-15T23:35:26.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17051261100000001 2024-05-15T23:35:26.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.715818+0000", 2024-05-15T23:35:26.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1119999999999999e-05 2024-05-15T23:35:26.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.717351+0000", 2024-05-15T23:35:26.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001532561 2024-05-15T23:35:26.745 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.745 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.745 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124269 7.1 7:84b5a5c1:::benchmark_data_smithi090_143373_object124268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:35:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.428950+0000", 2024-05-15T23:35:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.24977552799999, 2024-05-15T23:35:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31337896399999998, 2024-05-15T23:35:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124269 2024-05-15T23:35:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.428950+0000", 2024-05-15T23:35:26.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.428950+0000", 2024-05-15T23:35:26.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.428952+0000", 2024-05-15T23:35:26.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9989999999999998e-06 2024-05-15T23:35:26.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.428998+0000", 2024-05-15T23:35:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5986999999999999e-05 2024-05-15T23:35:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.429009+0000", 2024-05-15T23:35:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1008e-05 2024-05-15T23:35:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.429017+0000", 2024-05-15T23:35:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1110000000000008e-06 2024-05-15T23:35:26.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.429073+0000", 2024-05-15T23:35:26.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5998999999999997e-05 2024-05-15T23:35:26.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:26.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.429081+0000", 2024-05-15T23:35:26.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.943e-06 2024-05-15T23:35:26.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.538267+0000", 2024-05-15T23:35:26.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.109185886 2024-05-15T23:35:26.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.538366+0000", 2024-05-15T23:35:26.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8715000000000006e-05 2024-05-15T23:35:26.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.538461+0000", 2024-05-15T23:35:26.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5101999999999998e-05 2024-05-15T23:35:26.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.545270+0000", 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0068090900000000003 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.739358+0000", 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.194088025 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.739415+0000", 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6719999999999999e-05 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.742329+0000", 2024-05-15T23:35:26.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029143789999999999 2024-05-15T23:35:26.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.752 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.752 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124293 7.1 7:8a970b3a:::benchmark_data_smithi090_143373_object124292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:35:26.752 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.483940+0000", 2024-05-15T23:35:26.752 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.19478518, 2024-05-15T23:35:26.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27423620599999998, 2024-05-15T23:35:26.752 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.752 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.752 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.752 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.752 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.752 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.753 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124293 2024-05-15T23:35:26.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.753 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.483940+0000", 2024-05-15T23:35:26.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.483940+0000", 2024-05-15T23:35:26.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.483943+0000", 2024-05-15T23:35:26.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.322e-06 2024-05-15T23:35:26.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.483993+0000", 2024-05-15T23:35:26.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0241000000000002e-05 2024-05-15T23:35:26.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.484005+0000", 2024-05-15T23:35:26.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2024-05-15T23:35:26.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.484012+0000", 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2849999999999997e-06 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.540767+0000", 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.056755011000000001 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.540856+0000", 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9073000000000006e-05 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.540949+0000", 2024-05-15T23:35:26.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2831000000000006e-05 2024-05-15T23:35:26.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.545273+0000", 2024-05-15T23:35:26.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043239929999999999 2024-05-15T23:35:26.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.755350+0000", 2024-05-15T23:35:26.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.210076601 2024-05-15T23:35:26.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.755411+0000", 2024-05-15T23:35:26.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1271000000000003e-05 2024-05-15T23:35:26.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.758176+0000", 2024-05-15T23:35:26.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027657020000000001 2024-05-15T23:35:26.757 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.757 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.757 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.757 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124302 7.1 7:83d236b9:::benchmark_data_smithi090_143373_object124301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:35:26.757 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.505719+0000", 2024-05-15T23:35:26.757 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.17300683799999, 2024-05-15T23:35:26.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25507802400000001, 2024-05-15T23:35:26.758 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.758 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.758 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.758 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.758 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.758 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.758 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124302 2024-05-15T23:35:26.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.758 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505719+0000", 2024-05-15T23:35:26.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505719+0000", 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505721+0000", 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3760000000000002e-06 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505778+0000", 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6858999999999999e-05 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505794+0000", 2024-05-15T23:35:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5926e-05 2024-05-15T23:35:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505804+0000", 2024-05-15T23:35:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0219e-05 2024-05-15T23:35:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.543199+0000", 2024-05-15T23:35:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.03739543 2024-05-15T23:35:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.543285+0000", 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5523000000000001e-05 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.543379+0000", 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4110000000000005e-05 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.545425+0000", 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020457280000000001 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.758248+0000", 2024-05-15T23:35:26.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.212822821 2024-05-15T23:35:26.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.758294+0000", 2024-05-15T23:35:26.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6774999999999999e-05 2024-05-15T23:35:26.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.760797+0000", 2024-05-15T23:35:26.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025022569999999999 2024-05-15T23:35:26.762 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.762 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:165512 7.1 7:81ed44b8:::benchmark_data_smithi090_143373_object19341:head [delete] snapc 0=[] ondisk+write+known_if_redirected e591)", 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:57.374026+0000", 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 269.30469930599997, 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.249226325, 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 165512 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.763 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374026+0000", 2024-05-15T23:35:26.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374026+0000", 2024-05-15T23:35:26.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374029+0000", 2024-05-15T23:35:26.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6060000000000001e-06 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374037+0000", 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3950000000000001e-06 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374050+0000", 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2826999999999999e-05 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374060+0000", 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9680000000000001e-06 2024-05-15T23:35:26.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.383163+0000", 2024-05-15T23:35:26.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091029890000000006 2024-05-15T23:35:26.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:35:26.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.383237+0000", 2024-05-15T23:35:26.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4456000000000007e-05 2024-05-15T23:35:26.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.567432+0000", 2024-05-15T23:35:26.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18419463999999999 2024-05-15T23:35:26.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.567524+0000", 2024-05-15T23:35:26.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2298000000000003e-05 2024-05-15T23:35:26.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.567612+0000", 2024-05-15T23:35:26.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7293999999999995e-05 2024-05-15T23:35:26.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.600864+0000", 2024-05-15T23:35:26.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033252149000000002 2024-05-15T23:35:26.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.620769+0000", 2024-05-15T23:35:26.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019905062000000001 2024-05-15T23:35:26.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.620810+0000", 2024-05-15T23:35:26.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0685000000000001e-05 2024-05-15T23:35:26.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.623252+0000", 2024-05-15T23:35:26.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002442956 2024-05-15T23:35:26.769 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.769 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.769 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.769 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:168938 7.1 7:87745b29:::benchmark_data_smithi090_143373_object22767:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:35:26.769 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:01.604430+0000", 2024-05-15T23:35:26.769 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 265.07429539399999, 2024-05-15T23:35:26.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.246406922, 2024-05-15T23:35:26.769 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.769 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.769 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.770 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168938 2024-05-15T23:35:26.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.770 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604430+0000", 2024-05-15T23:35:26.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604430+0000", 2024-05-15T23:35:26.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604433+0000", 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.661e-06 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604442+0000", 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1749999999999994e-06 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604456+0000", 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4354e-05 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604468+0000", 2024-05-15T23:35:26.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1389000000000001e-05 2024-05-15T23:35:26.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.631305+0000", 2024-05-15T23:35:26.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02683758 2024-05-15T23:35:26.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:35:26.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.631363+0000", 2024-05-15T23:35:26.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8009e-05 2024-05-15T23:35:26.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.820785+0000", 2024-05-15T23:35:26.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.189421428 2024-05-15T23:35:26.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.820880+0000", 2024-05-15T23:35:26.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4852000000000006e-05 2024-05-15T23:35:26.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.821015+0000", 2024-05-15T23:35:26.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135709 2024-05-15T23:35:26.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.837594+0000", 2024-05-15T23:35:26.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016578446 2024-05-15T23:35:26.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.847339+0000", 2024-05-15T23:35:26.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0097453809999999991 2024-05-15T23:35:26.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.847400+0000", 2024-05-15T23:35:26.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1194999999999999e-05 2024-05-15T23:35:26.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.850837+0000", 2024-05-15T23:35:26.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034367429999999999 2024-05-15T23:35:26.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.775 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.775 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:168941 7.1 7:8a84f08b:::benchmark_data_smithi090_143373_object22770:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:35:26.775 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:01.604543+0000", 2024-05-15T23:35:26.775 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 265.07418252600002, 2024-05-15T23:35:26.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.33646746300000002, 2024-05-15T23:35:26.775 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.775 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.775 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.775 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.775 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.776 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168941 2024-05-15T23:35:26.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.776 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604543+0000", 2024-05-15T23:35:26.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604543+0000", 2024-05-15T23:35:26.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604544+0000", 2024-05-15T23:35:26.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9500000000000001e-07 2024-05-15T23:35:26.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604548+0000", 2024-05-15T23:35:26.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0380000000000001e-06 2024-05-15T23:35:26.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604554+0000", 2024-05-15T23:35:26.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1940000000000003e-06 2024-05-15T23:35:26.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604559+0000", 2024-05-15T23:35:26.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8300000000000003e-06 2024-05-15T23:35:26.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634299+0000", 2024-05-15T23:35:26.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740566 2024-05-15T23:35:26.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:35:26.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634357+0000", 2024-05-15T23:35:26.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7884999999999999e-05 2024-05-15T23:35:26.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.916090+0000", 2024-05-15T23:35:26.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28173253799999998 2024-05-15T23:35:26.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.916206+0000", 2024-05-15T23:35:26.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116403 2024-05-15T23:35:26.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.916353+0000", 2024-05-15T23:35:26.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014637100000000001 2024-05-15T23:35:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.934169+0000", 2024-05-15T23:35:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017816117999999999 2024-05-15T23:35:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.939265+0000", 2024-05-15T23:35:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050958519999999997 2024-05-15T23:35:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.939322+0000", 2024-05-15T23:35:26.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7025999999999999e-05 2024-05-15T23:35:26.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.941010+0000", 2024-05-15T23:35:26.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001688747 2024-05-15T23:35:26.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.781 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.781 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPushReply(7.1 592/578 [PushReplyOp(7:8a84f08b:::benchmark_data_smithi090_143373_object22770:head)])", 2024-05-15T23:35:26.781 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:01.634777+0000", 2024-05-15T23:35:26.781 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 265.04394834200002, 2024-05-15T23:35:26.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28126164100000001, 2024-05-15T23:35:26.781 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.782 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.782 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-05-15T23:35:26.782 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634777+0000", 2024-05-15T23:35:26.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634777+0000", 2024-05-15T23:35:26.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634779+0000", 2024-05-15T23:35:26.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5099999999999999e-06 2024-05-15T23:35:26.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634789+0000", 2024-05-15T23:35:26.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0056e-05 2024-05-15T23:35:26.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634801+0000", 2024-05-15T23:35:26.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2532e-05 2024-05-15T23:35:26.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634809+0000", 2024-05-15T23:35:26.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8099999999999998e-06 2024-05-15T23:35:26.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.909849+0000", 2024-05-15T23:35:26.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27503971700000002 2024-05-15T23:35:26.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.916039+0000", 2024-05-15T23:35:26.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061900159999999996 2024-05-15T23:35:26.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.784 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.785 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:169812 7.1 7:817b8891:::benchmark_data_smithi090_143373_object23641:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:35:26.785 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:02.735245+0000", 2024-05-15T23:35:26.785 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 263.94348004099999, 2024-05-15T23:35:26.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31279489399999999, 2024-05-15T23:35:26.785 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.785 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:35:26.786 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.786 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.786 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.787 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.787 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 169812 2024-05-15T23:35:26.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.787 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.735245+0000", 2024-05-15T23:35:26.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.735245+0000", 2024-05-15T23:35:26.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.735248+0000", 2024-05-15T23:35:26.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3549999999999999e-06 2024-05-15T23:35:26.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.735255+0000", 2024-05-15T23:35:26.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8029999999999999e-06 2024-05-15T23:35:26.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909550+0000", 2024-05-15T23:35:26.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.174295218 2024-05-15T23:35:26.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909565+0000", 2024-05-15T23:35:26.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-05-15T23:35:26.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909644+0000", 2024-05-15T23:35:26.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9839999999999995e-05 2024-05-15T23:35:26.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:35:26.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909718+0000", 2024-05-15T23:35:26.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3475999999999999e-05 2024-05-15T23:35:26.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.012591+0000", 2024-05-15T23:35:26.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102873279 2024-05-15T23:35:26.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.012700+0000", 2024-05-15T23:35:26.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108721 2024-05-15T23:35:26.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.012857+0000", 2024-05-15T23:35:26.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157305 2024-05-15T23:35:26.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.030528+0000", 2024-05-15T23:35:26.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017670788 2024-05-15T23:35:26.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.045291+0000", 2024-05-15T23:35:26.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014763108 2024-05-15T23:35:26.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.045348+0000", 2024-05-15T23:35:26.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6375000000000001e-05 2024-05-15T23:35:26.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.048040+0000", 2024-05-15T23:35:26.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002692776 2024-05-15T23:35:26.792 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.792 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.792 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.792 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:208781 7.1 7:87e9d900:::benchmark_data_smithi090_143373_object62610:head [delete] snapc 0=[] ondisk+write+known_if_redirected e607)", 2024-05-15T23:35:26.792 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:55.190408+0000", 2024-05-15T23:35:26.792 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 211.488316965, 2024-05-15T23:35:26.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.24431366500000001, 2024-05-15T23:35:26.793 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.793 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.793 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.793 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 208781 2024-05-15T23:35:26.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.793 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.190408+0000", 2024-05-15T23:35:26.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.190408+0000", 2024-05-15T23:35:26.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.190409+0000", 2024-05-15T23:35:26.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8899999999999998e-07 2024-05-15T23:35:26.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.190412+0000", 2024-05-15T23:35:26.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9689999999999999e-06 2024-05-15T23:35:26.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.190418+0000", 2024-05-15T23:35:26.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8969999999999996e-06 2024-05-15T23:35:26.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.190424+0000", 2024-05-15T23:35:26.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5890000000000002e-06 2024-05-15T23:35:26.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.215086+0000", 2024-05-15T23:35:26.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024662133999999999 2024-05-15T23:35:26.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:35:26.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.215146+0000", 2024-05-15T23:35:26.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9737000000000001e-05 2024-05-15T23:35:26.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.251337+0000", 2024-05-15T23:35:26.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036191359999999999 2024-05-15T23:35:26.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.251394+0000", 2024-05-15T23:35:26.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7102999999999999e-05 2024-05-15T23:35:26.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.251470+0000", 2024-05-15T23:35:26.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5964999999999995e-05 2024-05-15T23:35:26.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.262512+0000", 2024-05-15T23:35:26.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011041751000000001 2024-05-15T23:35:26.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.432850+0000", 2024-05-15T23:35:26.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.170338194 2024-05-15T23:35:26.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.432915+0000", 2024-05-15T23:35:26.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4789999999999995e-05 2024-05-15T23:35:26.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.434722+0000", 2024-05-15T23:35:26.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018072870000000001 2024-05-15T23:35:26.798 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.798 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.798 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.799 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:248392 7.1 7:88e83ff0:::benchmark_data_smithi090_143373_object102221:head [delete] snapc 0=[] ondisk+write+known_if_redirected e616)", 2024-05-15T23:35:26.799 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:32:38.458728+0000", 2024-05-15T23:35:26.799 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 168.21999708600001, 2024-05-15T23:35:26.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23309224200000001, 2024-05-15T23:35:26.799 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:26.799 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:26.799 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:26.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:26.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:26.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:26.799 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 248392 2024-05-15T23:35:26.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.458728+0000", 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.458728+0000", 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.458731+0000", 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.379e-06 2024-05-15T23:35:26.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:26.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.458737+0000", 2024-05-15T23:35:26.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5690000000000001e-06 2024-05-15T23:35:26.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:26.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.458749+0000", 2024-05-15T23:35:26.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1450999999999999e-05 2024-05-15T23:35:26.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:26.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.458757+0000", 2024-05-15T23:35:26.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0350000000000001e-06 2024-05-15T23:35:26.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.612089+0000", 2024-05-15T23:35:26.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15333242999999999 2024-05-15T23:35:26.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:35:26.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.612703+0000", 2024-05-15T23:35:26.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061384099999999995 2024-05-15T23:35:26.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:26.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.641419+0000", 2024-05-15T23:35:26.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028715898 2024-05-15T23:35:26.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:26.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.641479+0000", 2024-05-15T23:35:26.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9755e-05 2024-05-15T23:35:26.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:35:26.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.641580+0000", 2024-05-15T23:35:26.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100875 2024-05-15T23:35:26.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:26.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.654633+0000", 2024-05-15T23:35:26.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013053021999999999 2024-05-15T23:35:26.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:26.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.690959+0000", 2024-05-15T23:35:26.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036326439000000002 2024-05-15T23:35:26.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:26.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.690997+0000", 2024-05-15T23:35:26.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7543000000000003e-05 2024-05-15T23:35:26.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:26.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:26.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:26.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.691821+0000", 2024-05-15T23:35:26.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082400500000000005 2024-05-15T23:35:26.805 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.805 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.805 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.805 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:26.805 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:26.805 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:35:26.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:26 smithi090 ceph-mon[44434]: 8.2 scrub starts 2024-05-15T23:35:26.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:26 smithi090 ceph-mon[44434]: 8.c scrub starts 2024-05-15T23:35:26.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:26 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3868454329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:35:26.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:26 smithi090 ceph-mon[44434]: from='client.37112 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:35:26.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:26 smithi088 ceph-mon[44699]: 8.2 scrub starts 2024-05-15T23:35:26.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:26 smithi088 ceph-mon[44699]: 8.c scrub starts 2024-05-15T23:35:26.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:26 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3868454329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:35:26.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:26 smithi088 ceph-mon[44699]: from='client.37112 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:35:26.853 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:26.893 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:35:26.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:26.898 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:35:26.999 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:27.028 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:27.056 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:27.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:27.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:27.149 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 72 16 130982 130966 113.625 71.875 0.0126758 0.00879936 2024-05-15T23:35:27.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:35:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:27.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:27 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:35:27.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:27 smithi090 ceph-mon[44434]: from='client.37112 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:35:27.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:27 smithi090 ceph-mon[44434]: osdmap e682: 12 total, 12 up, 10 in 2024-05-15T23:35:27.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:27 smithi090 ceph-mon[44434]: pgmap v1806: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 7.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 58 MiB/s wr, 922 op/s 2024-05-15T23:35:27.799 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:27 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:35:27.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:27 smithi084 ceph-mon[39784]: from='client.37112 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:35:27.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:27 smithi084 ceph-mon[39784]: osdmap e682: 12 total, 12 up, 10 in 2024-05-15T23:35:27.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:27 smithi084 ceph-mon[39784]: pgmap v1806: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 7.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 58 MiB/s wr, 922 op/s 2024-05-15T23:35:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:27 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:35:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:27 smithi088 ceph-mon[44699]: from='client.37112 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:35:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:27 smithi088 ceph-mon[44699]: osdmap e682: 12 total, 12 up, 10 in 2024-05-15T23:35:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:27 smithi088 ceph-mon[44699]: pgmap v1806: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 7.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 58 MiB/s wr, 922 op/s 2024-05-15T23:35:28.151 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 73 16 132156 132140 113.071 73.375 0.0162501 0.00884249 2024-05-15T23:35:28.166 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:28.211 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:35:28.244 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:35:28.271 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:35:28.293 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:35:28.315 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:35:28.343 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:35:28.374 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:35:28.403 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:35:28.430 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:35:28.458 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:35:28.486 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:35:28.524 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:35:28.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:35:28.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:35:28.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:35:28.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:35:28.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:35:28.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:35:28.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:35:28.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:35:28.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:35:28.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:35:28.526 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:35:28.526 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:35:28.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:28 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:35:28.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:28 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:35:28.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:28 smithi090 ceph-mon[44434]: osdmap e683: 12 total, 12 up, 10 in 2024-05-15T23:35:28.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1463881044' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:35:28.761 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:28 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:35:28.761 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:28 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:35:28.761 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:28 smithi084 ceph-mon[39784]: osdmap e683: 12 total, 12 up, 10 in 2024-05-15T23:35:28.761 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1463881044' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:35:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:28 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:35:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:28 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:35:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:28 smithi088 ceph-mon[44699]: osdmap e683: 12 total, 12 up, 10 in 2024-05-15T23:35:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1463881044' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:35:29.154 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 74 16 133357 133341 112.553 75.0625 0.00912482 0.00888349 2024-05-15T23:35:29.184 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:35:29.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:29.191 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:35:29.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T23:35:29.292 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:35:29.319 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:29.340 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:35:29.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:29.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:35:29.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.419 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.448 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:29.476 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:29.498 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:29.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:29.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:29.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:29 smithi088 ceph-mon[44699]: pgmap v1808: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-05-15T23:35:29.602 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:35:29.603 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:35:29.630 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:29.652 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:35:29.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:29.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:35:29.686 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:35:29.757 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:35:29.786 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:29.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:29 smithi084 ceph-mon[39784]: pgmap v1808: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-05-15T23:35:29.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:29 smithi090 ceph-mon[44434]: pgmap v1808: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-05-15T23:35:29.808 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:35:29.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:29.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:35:29.914 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:29.956 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:29.969 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:29.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:30.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:30.146 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:30.154 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 75 16 134430 134414 111.947 67.0625 0.0157921 0.0089309 2024-05-15T23:35:30.176 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:30.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:30.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:30.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:30.312 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:30.333 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:30.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:30.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:30.437 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:30.466 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:30.494 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:30.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:30.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:30.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:30 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:35:30.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:30 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:35:30.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:30 smithi088 ceph-mon[44699]: osdmap e684: 12 total, 12 up, 10 in 2024-05-15T23:35:30.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:35:30.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:30 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:35:30.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:30 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:35:30.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:30 smithi084 ceph-mon[39784]: osdmap e684: 12 total, 12 up, 10 in 2024-05-15T23:35:30.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:30 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:35:30.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:30 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:35:30.634 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:30 smithi090 ceph-mon[44434]: osdmap e684: 12 total, 12 up, 10 in 2024-05-15T23:35:30.647 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:30.659 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:30.718 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:35:30.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:30.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:35:30.824 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:35:30.864 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:30.942 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:35:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:30.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:35:31.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:31.107 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:31.157 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 76 16 135463 135447 111.319 64.5625 0.0087736 0.00898105 2024-05-15T23:35:31.423 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:31 smithi090 ceph-mon[44434]: pgmap v1810: 17 pgs: 17 active+clean; 8.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-05-15T23:35:31.576 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:31.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:31.581 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:31 smithi084 ceph-mon[39784]: pgmap v1810: 17 pgs: 17 active+clean; 8.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-05-15T23:35:31.682 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:35:31.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:31 smithi088 ceph-mon[44699]: pgmap v1810: 17 pgs: 17 active+clean; 8.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-05-15T23:35:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:31.733 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:35:31.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:31.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:35:31.822 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:35:31.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:31.893 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:31.938 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:31.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:31.943 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:32.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:32.101 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:32.157 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 77 16 136590 136574 110.789 70.4375 0.00652503 0.00902488 2024-05-15T23:35:32.262 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:32.330 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:32.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:32.332 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:32.433 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:35:32.443 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2024-05-15T23:35:32.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872955322265625]}]: dispatch 2024-05-15T23:35:32.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10162549532694229]}]: dispatch 2024-05-15T23:35:32.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10898246420242876]}]: dispatch 2024-05-15T23:35:32.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.12465862612183218]}]: dispatch 2024-05-15T23:35:32.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08749217455573334]}]: dispatch 2024-05-15T23:35:32.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10162549532694229]}]: dispatch 2024-05-15T23:35:32.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872955322265625]}]: dispatch 2024-05-15T23:35:32.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.10898246420242876]}]: dispatch 2024-05-15T23:35:32.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.10898246420242876]}]: dispatch 2024-05-15T23:35:32.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.10898246420242876]}]: dispatch 2024-05-15T23:35:32.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.10162549532694229]}]: dispatch 2024-05-15T23:35:32.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09458924325064234]}]: dispatch 2024-05-15T23:35:32.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:32 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"4\": \"26660\"}"}]: dispatch 2024-05-15T23:35:32.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1357806458' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:35:32.464 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:32.505 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:35:32.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:32.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:35:32.609 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:35:32.626 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:35:32.626 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:35:32.638 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:32.645 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872955322265625]}]: dispatch 2024-05-15T23:35:32.645 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10162549532694229]}]: dispatch 2024-05-15T23:35:32.645 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10898246420242876]}]: dispatch 2024-05-15T23:35:32.645 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.12465862612183218]}]: dispatch 2024-05-15T23:35:32.645 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08749217455573334]}]: dispatch 2024-05-15T23:35:32.645 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10162549532694229]}]: dispatch 2024-05-15T23:35:32.645 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872955322265625]}]: dispatch 2024-05-15T23:35:32.645 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.10898246420242876]}]: dispatch 2024-05-15T23:35:32.646 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.10898246420242876]}]: dispatch 2024-05-15T23:35:32.646 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.10898246420242876]}]: dispatch 2024-05-15T23:35:32.646 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.10162549532694229]}]: dispatch 2024-05-15T23:35:32.646 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09458924325064234]}]: dispatch 2024-05-15T23:35:32.646 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:32 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"4\": \"26660\"}"}]: dispatch 2024-05-15T23:35:32.646 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1357806458' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:35:32.711 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:35:32.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:32.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:35:32.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872955322265625]}]: dispatch 2024-05-15T23:35:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10162549532694229]}]: dispatch 2024-05-15T23:35:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10898246420242876]}]: dispatch 2024-05-15T23:35:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.12465862612183218]}]: dispatch 2024-05-15T23:35:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08749217455573334]}]: dispatch 2024-05-15T23:35:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10162549532694229]}]: dispatch 2024-05-15T23:35:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872955322265625]}]: dispatch 2024-05-15T23:35:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.10898246420242876]}]: dispatch 2024-05-15T23:35:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.10898246420242876]}]: dispatch 2024-05-15T23:35:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.10898246420242876]}]: dispatch 2024-05-15T23:35:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.10162549532694229]}]: dispatch 2024-05-15T23:35:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09458924325064234]}]: dispatch 2024-05-15T23:35:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:32 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"4\": \"26660\"}"}]: dispatch 2024-05-15T23:35:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1357806458' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:35:32.842 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:32.860 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:32.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:32.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:32.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:35:32.993 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:33.157 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 78 16 137022 137006 109.715 27 0.0053512 0.0090367 2024-05-15T23:35:33.424 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 5 1 2024-05-15T23:35:33.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872955322265625]}]': finished 2024-05-15T23:35:33.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10162549532694229]}]': finished 2024-05-15T23:35:33.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10898246420242876]}]': finished 2024-05-15T23:35:33.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.12465862612183218]}]': finished 2024-05-15T23:35:33.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08749217455573334]}]': finished 2024-05-15T23:35:33.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10162549532694229]}]': finished 2024-05-15T23:35:33.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872955322265625]}]': finished 2024-05-15T23:35:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.10898246420242876]}]': finished 2024-05-15T23:35:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.10898246420242876]}]': finished 2024-05-15T23:35:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.10898246420242876]}]': finished 2024-05-15T23:35:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.10162549532694229]}]': finished 2024-05-15T23:35:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09458924325064234]}]': finished 2024-05-15T23:35:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"4\": \"26660\"}"}]': finished 2024-05-15T23:35:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1357806458' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:35:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: osdmap e685: 12 total, 12 up, 10 in 2024-05-15T23:35:33.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:33 smithi090 ceph-mon[44434]: pgmap v1812: 17 pgs: 17 active+clean; 8.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-05-15T23:35:33.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872955322265625]}]': finished 2024-05-15T23:35:33.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10162549532694229]}]': finished 2024-05-15T23:35:33.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10898246420242876]}]': finished 2024-05-15T23:35:33.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.12465862612183218]}]': finished 2024-05-15T23:35:33.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08749217455573334]}]': finished 2024-05-15T23:35:33.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10162549532694229]}]': finished 2024-05-15T23:35:33.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872955322265625]}]': finished 2024-05-15T23:35:33.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.10898246420242876]}]': finished 2024-05-15T23:35:33.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.10898246420242876]}]': finished 2024-05-15T23:35:33.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.10898246420242876]}]': finished 2024-05-15T23:35:33.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.10162549532694229]}]': finished 2024-05-15T23:35:33.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09458924325064234]}]': finished 2024-05-15T23:35:33.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"4\": \"26660\"}"}]': finished 2024-05-15T23:35:33.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1357806458' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:35:33.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: osdmap e685: 12 total, 12 up, 10 in 2024-05-15T23:35:33.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:33 smithi088 ceph-mon[44699]: pgmap v1812: 17 pgs: 17 active+clean; 8.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-05-15T23:35:33.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872955322265625]}]': finished 2024-05-15T23:35:33.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10162549532694229]}]': finished 2024-05-15T23:35:33.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10898246420242876]}]': finished 2024-05-15T23:35:33.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.12465862612183218]}]': finished 2024-05-15T23:35:33.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08749217455573334]}]': finished 2024-05-15T23:35:33.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.10162549532694229]}]': finished 2024-05-15T23:35:33.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872955322265625]}]': finished 2024-05-15T23:35:33.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.10898246420242876]}]': finished 2024-05-15T23:35:33.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.10898246420242876]}]': finished 2024-05-15T23:35:33.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.10898246420242876]}]': finished 2024-05-15T23:35:33.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.10162549532694229]}]': finished 2024-05-15T23:35:33.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09458924325064234]}]': finished 2024-05-15T23:35:33.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"4\": \"26660\"}"}]': finished 2024-05-15T23:35:33.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1357806458' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:35:33.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: osdmap e685: 12 total, 12 up, 10 in 2024-05-15T23:35:33.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:33 smithi084 ceph-mon[39784]: pgmap v1812: 17 pgs: 17 active+clean; 8.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-05-15T23:35:33.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T23:35:33.943 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:35:33.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:33.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:35:34.049 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:35:34.092 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:34.157 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:35:34.157 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 79 16 137291 137275 108.539 16.8125 0.0366242 0.00921112 2024-05-15T23:35:34.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:34.162 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:35:34.263 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:35:34.295 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:34.298 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:34.316 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:35:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:34.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:35:34.339 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:35:34.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:35:34.468 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:34.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:34 smithi088 ceph-mon[44699]: osdmap e686: 12 total, 12 up, 10 in 2024-05-15T23:35:34.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:34 smithi090 ceph-mon[44434]: osdmap e686: 12 total, 12 up, 10 in 2024-05-15T23:35:34.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:34 smithi084 ceph-mon[39784]: osdmap e686: 12 total, 12 up, 10 in 2024-05-15T23:35:35.158 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:35:35.158203 min lat: 0.00114746 max lat: 1.42997 avg lat: 0.0092659 2024-05-15T23:35:35.158 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:35:35.158 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 80 16 138212 138196 107.902 57.5625 0.0204363 0.0092659 2024-05-15T23:35:35.463 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:35:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:35 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2172165876' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:35 smithi090 ceph-mon[44434]: from='client.37127 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:35 smithi090 ceph-mon[44434]: pgmap v1814: 17 pgs: 1 activating, 1 active+recovering, 15 active+clean; 8.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 58 MiB/s wr, 927 op/s; 8454/273182 objects degraded (3.095%); 8388/273182 objects misplaced (3.070%); 235 KiB/s, 3 objects/s recovering 2024-05-15T23:35:35.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:35 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2172165876' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:35.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:35 smithi088 ceph-mon[44699]: from='client.37127 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:35.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:35 smithi088 ceph-mon[44699]: pgmap v1814: 17 pgs: 1 activating, 1 active+recovering, 15 active+clean; 8.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 58 MiB/s wr, 927 op/s; 8454/273182 objects degraded (3.095%); 8388/273182 objects misplaced (3.070%); 235 KiB/s, 3 objects/s recovering 2024-05-15T23:35:35.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:35 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2172165876' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:35.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:35 smithi084 ceph-mon[39784]: from='client.37127 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:35:35.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:35 smithi084 ceph-mon[39784]: pgmap v1814: 17 pgs: 1 activating, 1 active+recovering, 15 active+clean; 8.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 58 MiB/s wr, 927 op/s; 8454/273182 objects degraded (3.095%); 8388/273182 objects misplaced (3.070%); 235 KiB/s, 3 objects/s recovering 2024-05-15T23:35:36.158 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 81 16 139106 139090 107.259 55.875 0.0346845 0.00932026 2024-05-15T23:35:36.699 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:36 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:35:36.699 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:36 smithi090 ceph-mon[44434]: from='client.37127 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:35:36.699 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:36 smithi090 ceph-mon[44434]: osdmap e687: 12 total, 12 up, 10 in 2024-05-15T23:35:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:36 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:35:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:36 smithi088 ceph-mon[44699]: from='client.37127 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:35:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:36 smithi088 ceph-mon[44699]: osdmap e687: 12 total, 12 up, 10 in 2024-05-15T23:35:36.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:36 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:35:36.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:36 smithi084 ceph-mon[39784]: from='client.37127 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:35:36.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:36 smithi084 ceph-mon[39784]: osdmap e687: 12 total, 12 up, 10 in 2024-05-15T23:35:36.922 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:35:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:36.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:35:37.027 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:35:37.060 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:37.159 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 82 16 139942 139926 106.588 52.25 0.0132253 0.00937973 2024-05-15T23:35:37.463 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:37 smithi090 ceph-mon[44434]: pgmap v1816: 17 pgs: 1 activating, 1 active+recovering, 15 active+clean; 8.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 53 MiB/s wr, 842 op/s; 8454/275736 objects degraded (3.066%); 8388/275736 objects misplaced (3.042%); 235 KiB/s, 3 objects/s recovering 2024-05-15T23:35:37.701 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:35:37.702 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:35:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:37 smithi088 ceph-mon[44699]: pgmap v1816: 17 pgs: 1 activating, 1 active+recovering, 15 active+clean; 8.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 53 MiB/s wr, 842 op/s; 8454/275736 objects degraded (3.066%); 8388/275736 objects misplaced (3.042%); 235 KiB/s, 3 objects/s recovering 2024-05-15T23:35:37.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:37 smithi084 ceph-mon[39784]: pgmap v1816: 17 pgs: 1 activating, 1 active+recovering, 15 active+clean; 8.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 53 MiB/s wr, 842 op/s; 8454/275736 objects degraded (3.066%); 8388/275736 objects misplaced (3.042%); 235 KiB/s, 3 objects/s recovering 2024-05-15T23:35:38.159 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 83 16 140866 140850 106 57.75 0.0148429 0.00943182 2024-05-15T23:35:38.712 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:35:38.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:38.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:35:38.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:35:38.847 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:38.893 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:35:38.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:38.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:35:38.999 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:35:39.029 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:39.159 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 84 16 141864 141848 105.481 62.375 0.043271 0.00947849 2024-05-15T23:35:39.322 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:35:39.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:39.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:35:39.427 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:39.455 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:39.482 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:39.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:39.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:39.590 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:39.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:39 smithi084 ceph-mon[39784]: pgmap v1817: 17 pgs: 2 active+recovering, 15 active+clean; 8.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 59 MiB/s wr, 940 op/s; 16765/279906 objects misplaced (5.990%); 925 KiB/s, 14 objects/s recovering 2024-05-15T23:35:39.635 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:39 smithi090 ceph-mon[44434]: pgmap v1817: 17 pgs: 2 active+recovering, 15 active+clean; 8.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 59 MiB/s wr, 940 op/s; 16765/279906 objects misplaced (5.990%); 925 KiB/s, 14 objects/s recovering 2024-05-15T23:35:39.636 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:39.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:39 smithi088 ceph-mon[44699]: pgmap v1817: 17 pgs: 2 active+recovering, 15 active+clean; 8.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 59 MiB/s wr, 940 op/s; 16765/279906 objects misplaced (5.990%); 925 KiB/s, 14 objects/s recovering 2024-05-15T23:35:40.159 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 85 16 142810 142794 104.935 59.125 0.012298 0.00952753 2024-05-15T23:35:40.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T23:35:40.210 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:40.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:40.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:40.319 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:40.348 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:40.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:40.478 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:40.505 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:40.527 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:40.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:40.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:40 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:35:40.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:40 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:35:40.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:40 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:35:40.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:40 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:35:40.604 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:35:40.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:40 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:35:40.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:40 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:35:40.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:40 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:35:40.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:40 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:35:40.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:35:40.666 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:40.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:40 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:35:40.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:40 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:35:40.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:40 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:35:40.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:40 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:35:41.159 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 86 16 143665 143649 104.337 53.4375 0.00512501 0.00958093 2024-05-15T23:35:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:41 smithi090 ceph-mon[44434]: osdmap e688: 12 total, 12 up, 10 in 2024-05-15T23:35:41.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:41 smithi090 ceph-mon[44434]: pgmap v1819: 17 pgs: 2 active+recovering, 15 active+clean; 8.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 44 MiB/s wr, 708 op/s; 16622/282728 objects misplaced (5.879%); 1.9 MiB/s, 30 objects/s recovering 2024-05-15T23:35:41.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:41 smithi084 ceph-mon[39784]: osdmap e688: 12 total, 12 up, 10 in 2024-05-15T23:35:41.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:41 smithi084 ceph-mon[39784]: pgmap v1819: 17 pgs: 2 active+recovering, 15 active+clean; 8.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 44 MiB/s wr, 708 op/s; 16622/282728 objects misplaced (5.879%); 1.9 MiB/s, 30 objects/s recovering 2024-05-15T23:35:41.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:41 smithi088 ceph-mon[44699]: osdmap e688: 12 total, 12 up, 10 in 2024-05-15T23:35:41.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:41 smithi088 ceph-mon[44699]: pgmap v1819: 17 pgs: 2 active+recovering, 15 active+clean; 8.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 44 MiB/s wr, 708 op/s; 16622/282728 objects misplaced (5.879%); 1.9 MiB/s, 30 objects/s recovering 2024-05-15T23:35:42.112 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:35:42.141 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:35:42.159 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 87 16 144594 144578 103.805 58.0625 0.00942987 0.00963102 2024-05-15T23:35:42.165 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:35:42.191 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:35:42.215 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:35:42.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:35:42.262 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:35:42.287 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:35:42.316 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:35:42.337 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:35:42.365 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:35:42.444 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:35:42.446 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:35:42.446 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:35:42.446 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:35:42.446 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:35:42.446 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:35:42.446 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:35:42.446 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:35:42.446 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:35:42.446 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:35:42.447 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:35:42.447 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:35:42.447 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:35:42.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:42 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/196823233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:35:42.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:42.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:42.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:42 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/196823233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:35:42.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:42.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:42.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:42 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/196823233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:35:42.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:42.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.162 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 88 16 145382 145366 103.185 49.25 0.0943835 0.00968847 2024-05-15T23:35:43.363 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:35:43.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:43.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:35:43.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:35:43.502 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:43.602 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:35:43.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:43.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:35:43.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:43 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:43 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:43 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:43 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:43 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:43 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:43 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:43 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:43 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:43 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:43 smithi084 ceph-mon[39784]: pgmap v1820: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 58 MiB/s wr, 922 op/s; 16622/287940 objects misplaced (5.773%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T23:35:43.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:35:43.755 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:43 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.755 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:43 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.755 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:43 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.756 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:43 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.756 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:43 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.756 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:43 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.756 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:43 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.756 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:43 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.756 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:43 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.756 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:43 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.756 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:43 smithi090 ceph-mon[44434]: pgmap v1820: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 58 MiB/s wr, 922 op/s; 16622/287940 objects misplaced (5.773%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T23:35:43.758 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:43.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:43 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:43 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:43 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:43 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:43 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:43 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:43 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:43 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:43 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:43 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:43.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:43 smithi088 ceph-mon[44699]: pgmap v1820: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 58 MiB/s wr, 922 op/s; 16622/287940 objects misplaced (5.773%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T23:35:44.159 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 89 16 146219 146203 102.614 52.3125 0.00576843 0.00973402 2024-05-15T23:35:44.628 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:35:44.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:44.634 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:35:44.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:35:44.788 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:44.805 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-05-15T23:35:44.913 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:35:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:44.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:35:44.917 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/276966513' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:35:44.917 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:44 smithi090 ceph-mon[44434]: from='client.37139 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:35:45.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:35:45.058 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:45.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/276966513' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:35:45.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:44 smithi088 ceph-mon[44699]: from='client.37139 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:35:45.100 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:35:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:45.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:35:45.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/276966513' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:35:45.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:44 smithi084 ceph-mon[39784]: from='client.37139 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:35:45.159 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total time run: 90.0335 2024-05-15T23:35:45.160 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total writes made: 146935 2024-05-15T23:35:45.160 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Write size: 65536 2024-05-15T23:35:45.160 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object size: 65536 2024-05-15T23:35:45.160 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Bandwidth (MB/sec): 102 2024-05-15T23:35:45.160 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Bandwidth: 54.9573 2024-05-15T23:35:45.160 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max bandwidth (MB/sec): 308.562 2024-05-15T23:35:45.160 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min bandwidth (MB/sec): 16.8125 2024-05-15T23:35:45.160 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average IOPS: 1632 2024-05-15T23:35:45.160 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev IOPS: 879.317 2024-05-15T23:35:45.160 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max IOPS: 4937 2024-05-15T23:35:45.160 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min IOPS: 269 2024-05-15T23:35:45.160 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average Latency(s): 0.00980312 2024-05-15T23:35:45.160 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Latency(s): 0.0176631 2024-05-15T23:35:45.161 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max latency(s): 1.42997 2024-05-15T23:35:45.161 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min latency(s): 0.00114746 2024-05-15T23:35:45.167 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T23:35:45.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:35:45.245 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:45.279 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:35:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:35:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:35:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:35:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:169805 7.9 7:9f062381:::benchmark_data_smithi090_143373_object23634:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:35:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:02.728364+0000", 2024-05-15T23:35:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 282.54775764800002, 2024-05-15T23:35:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18166183499999999, 2024-05-15T23:35:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 169805 2024-05-15T23:35:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728364+0000", 2024-05-15T23:35:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728364+0000", 2024-05-15T23:35:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728366+0000", 2024-05-15T23:35:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0250000000000001e-06 2024-05-15T23:35:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728373+0000", 2024-05-15T23:35:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4710000000000004e-06 2024-05-15T23:35:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728389+0000", 2024-05-15T23:35:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5892e-05 2024-05-15T23:35:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728398+0000", 2024-05-15T23:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6379999999999999e-06 2024-05-15T23:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728465+0000", 2024-05-15T23:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6455e-05 2024-05-15T23:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728586+0000", 2024-05-15T23:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121445 2024-05-15T23:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.728705+0000", 2024-05-15T23:35:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118628 2024-05-15T23:35:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733796+0000", 2024-05-15T23:35:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00509144 2024-05-15T23:35:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909910+0000", 2024-05-15T23:35:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.176113774 2024-05-15T23:35:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909967+0000", 2024-05-15T23:35:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7312999999999998e-05 2024-05-15T23:35:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.910026+0000", 2024-05-15T23:35:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8754e-05 2024-05-15T23:35:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:169813 7.9 7:9f055894:::benchmark_data_smithi090_143373_object23642:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:35:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:02.733789+0000", 2024-05-15T23:35:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 282.54233314800001, 2024-05-15T23:35:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.192022421, 2024-05-15T23:35:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 169813 2024-05-15T23:35:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733789+0000", 2024-05-15T23:35:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733789+0000", 2024-05-15T23:35:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733791+0000", 2024-05-15T23:35:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.075e-06 2024-05-15T23:35:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733797+0000", 2024-05-15T23:35:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5830000000000003e-06 2024-05-15T23:35:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733808+0000", 2024-05-15T23:35:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0589e-05 2024-05-15T23:35:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733815+0000", 2024-05-15T23:35:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1930000000000004e-06 2024-05-15T23:35:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733865+0000", 2024-05-15T23:35:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0263999999999999e-05 2024-05-15T23:35:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733988+0000", 2024-05-15T23:35:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012234799999999999 2024-05-15T23:35:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.734102+0000", 2024-05-15T23:35:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113831 2024-05-15T23:35:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.745770+0000", 2024-05-15T23:35:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011668705 2024-05-15T23:35:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.925710+0000", 2024-05-15T23:35:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.179939708 2024-05-15T23:35:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.925762+0000", 2024-05-15T23:35:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2123999999999998e-05 2024-05-15T23:35:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.925811+0000", 2024-05-15T23:35:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9001000000000001e-05 2024-05-15T23:35:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:169814 7.9 7:9503e738:::benchmark_data_smithi090_143373_object23643:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:35:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:02.733837+0000", 2024-05-15T23:35:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 282.54228465400001, 2024-05-15T23:35:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19211088000000001, 2024-05-15T23:35:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 169814 2024-05-15T23:35:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733837+0000", 2024-05-15T23:35:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733837+0000", 2024-05-15T23:35:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733838+0000", 2024-05-15T23:35:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9400000000000004e-07 2024-05-15T23:35:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733842+0000", 2024-05-15T23:35:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8580000000000002e-06 2024-05-15T23:35:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733849+0000", 2024-05-15T23:35:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8770000000000004e-06 2024-05-15T23:35:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733854+0000", 2024-05-15T23:35:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.31e-06 2024-05-15T23:35:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.738245+0000", 2024-05-15T23:35:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043908159999999996 2024-05-15T23:35:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.738360+0000", 2024-05-15T23:35:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114731 2024-05-15T23:35:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.738470+0000", 2024-05-15T23:35:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011088500000000001 2024-05-15T23:35:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.745774+0000", 2024-05-15T23:35:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073033209999999998 2024-05-15T23:35:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.925860+0000", 2024-05-15T23:35:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18008618400000001 2024-05-15T23:35:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.925908+0000", 2024-05-15T23:35:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7673000000000002e-05 2024-05-15T23:35:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.925948+0000", 2024-05-15T23:35:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0431e-05 2024-05-15T23:35:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:169815 7.9 7:9b7f3b8f:::benchmark_data_smithi090_143373_object23644:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:35:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:02.733871+0000", 2024-05-15T23:35:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 282.54225034299998, 2024-05-15T23:35:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.193427022, 2024-05-15T23:35:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 169815 2024-05-15T23:35:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733871+0000", 2024-05-15T23:35:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733871+0000", 2024-05-15T23:35:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733872+0000", 2024-05-15T23:35:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3099999999999996e-07 2024-05-15T23:35:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733876+0000", 2024-05-15T23:35:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8360000000000002e-06 2024-05-15T23:35:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733883+0000", 2024-05-15T23:35:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7150000000000001e-06 2024-05-15T23:35:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.733887+0000", 2024-05-15T23:35:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3120000000000004e-06 2024-05-15T23:35:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.741987+0000", 2024-05-15T23:35:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0080999220000000007 2024-05-15T23:35:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.742095+0000", 2024-05-15T23:35:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010813400000000001 2024-05-15T23:35:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.742213+0000", 2024-05-15T23:35:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011735000000000001 2024-05-15T23:35:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.745957+0000", 2024-05-15T23:35:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037446240000000002 2024-05-15T23:35:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.927206+0000", 2024-05-15T23:35:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.181248723 2024-05-15T23:35:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.927250+0000", 2024-05-15T23:35:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3733999999999998e-05 2024-05-15T23:35:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.927299+0000", 2024-05-15T23:35:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8741e-05 2024-05-15T23:35:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:169840 7.9 7:93ee92fe:::benchmark_data_smithi090_143373_object23669:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:35:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:02.756631+0000", 2024-05-15T23:35:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 282.51949061699997, 2024-05-15T23:35:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18059107899999999, 2024-05-15T23:35:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 169840 2024-05-15T23:35:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756631+0000", 2024-05-15T23:35:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756631+0000", 2024-05-15T23:35:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756633+0000", 2024-05-15T23:35:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0109999999999999e-06 2024-05-15T23:35:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756639+0000", 2024-05-15T23:35:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8440000000000003e-06 2024-05-15T23:35:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756649+0000", 2024-05-15T23:35:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0336999999999999e-05 2024-05-15T23:35:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756657+0000", 2024-05-15T23:35:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9349999999999994e-06 2024-05-15T23:35:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756716+0000", 2024-05-15T23:35:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8727000000000002e-05 2024-05-15T23:35:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756833+0000", 2024-05-15T23:35:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011724500000000001 2024-05-15T23:35:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.756948+0000", 2024-05-15T23:35:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115159 2024-05-15T23:35:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.761100+0000", 2024-05-15T23:35:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041516579999999999 2024-05-15T23:35:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.937099+0000", 2024-05-15T23:35:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17599888899999999 2024-05-15T23:35:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.937162+0000", 2024-05-15T23:35:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2668000000000004e-05 2024-05-15T23:35:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.937222+0000", 2024-05-15T23:35:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0606000000000002e-05 2024-05-15T23:35:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:276228 7.9 7:99c05770:::benchmark_data_smithi090_143373_object130057:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e634)", 2024-05-15T23:35:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:33:12.371613+0000", 2024-05-15T23:35:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.904508362, 2024-05-15T23:35:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35507692800000001, 2024-05-15T23:35:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 276228 2024-05-15T23:35:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371613+0000", 2024-05-15T23:35:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371613+0000", 2024-05-15T23:35:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371617+0000", 2024-05-15T23:35:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8500000000000004e-06 2024-05-15T23:35:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371626+0000", 2024-05-15T23:35:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1330000000000005e-06 2024-05-15T23:35:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371643+0000", 2024-05-15T23:35:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6674e-05 2024-05-15T23:35:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371658+0000", 2024-05-15T23:35:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4473e-05 2024-05-15T23:35:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371732+0000", 2024-05-15T23:35:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4478000000000001e-05 2024-05-15T23:35:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371745+0000", 2024-05-15T23:35:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3073e-05 2024-05-15T23:35:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.711975+0000", 2024-05-15T23:35:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.340230278 2024-05-15T23:35:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.712119+0000", 2024-05-15T23:35:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014348099999999999 2024-05-15T23:35:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 8,10", 2024-05-15T23:35:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.712240+0000", 2024-05-15T23:35:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121478 2024-05-15T23:35:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.717558+0000", 2024-05-15T23:35:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053173689999999997 2024-05-15T23:35:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.726261+0000", 2024-05-15T23:35:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087035789999999995 2024-05-15T23:35:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.726512+0000", 2024-05-15T23:35:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000251077 2024-05-15T23:35:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.726579+0000", 2024-05-15T23:35:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6598000000000002e-05 2024-05-15T23:35:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.726690+0000", 2024-05-15T23:35:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111387 2024-05-15T23:35:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:276229 7.9 7:9d1850b3:::benchmark_data_smithi090_143373_object130058:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e634)", 2024-05-15T23:35:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:33:12.371688+0000", 2024-05-15T23:35:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.904433587, 2024-05-15T23:35:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35527329600000002, 2024-05-15T23:35:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 276229 2024-05-15T23:35:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371688+0000", 2024-05-15T23:35:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371688+0000", 2024-05-15T23:35:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371689+0000", 2024-05-15T23:35:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1699999999999997e-07 2024-05-15T23:35:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371694+0000", 2024-05-15T23:35:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-05-15T23:35:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371701+0000", 2024-05-15T23:35:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6619999999999999e-06 2024-05-15T23:35:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371707+0000", 2024-05-15T23:35:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6429999999999998e-06 2024-05-15T23:35:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371774+0000", 2024-05-15T23:35:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6521999999999998e-05 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371782+0000", 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0709999999999995e-06 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.717608+0000", 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.34582564199999999 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.717740+0000", 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000132115 2024-05-15T23:35:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 8,10", 2024-05-15T23:35:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.717876+0000", 2024-05-15T23:35:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013646599999999999 2024-05-15T23:35:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.726707+0000", 2024-05-15T23:35:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0088309159999999994 2024-05-15T23:35:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.726746+0000", 2024-05-15T23:35:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8622999999999999e-05 2024-05-15T23:35:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.726796+0000", 2024-05-15T23:35:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0154000000000001e-05 2024-05-15T23:35:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.726852+0000", 2024-05-15T23:35:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6619999999999997e-05 2024-05-15T23:35:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.726962+0000", 2024-05-15T23:35:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109097 2024-05-15T23:35:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:276247 7.9 7:9d4ca9d7:::benchmark_data_smithi090_143373_object130076:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e634)", 2024-05-15T23:35:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:33:12.371716+0000", 2024-05-15T23:35:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.90440546100001, 2024-05-15T23:35:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36618598600000002, 2024-05-15T23:35:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 276247 2024-05-15T23:35:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371716+0000", 2024-05-15T23:35:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371716+0000", 2024-05-15T23:35:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371717+0000", 2024-05-15T23:35:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8500000000000001e-07 2024-05-15T23:35:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371720+0000", 2024-05-15T23:35:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4960000000000001e-06 2024-05-15T23:35:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371727+0000", 2024-05-15T23:35:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4679999999999997e-06 2024-05-15T23:35:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371732+0000", 2024-05-15T23:35:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5079999999999996e-06 2024-05-15T23:35:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371807+0000", 2024-05-15T23:35:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4449999999999994e-05 2024-05-15T23:35:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371814+0000", 2024-05-15T23:35:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6359999999999997e-06 2024-05-15T23:35:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.722297+0000", 2024-05-15T23:35:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35048311399999998 2024-05-15T23:35:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.722404+0000", 2024-05-15T23:35:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107786 2024-05-15T23:35:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 8,10", 2024-05-15T23:35:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.722490+0000", 2024-05-15T23:35:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5131999999999997e-05 2024-05-15T23:35:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.726758+0000", 2024-05-15T23:35:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004268679 2024-05-15T23:35:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.727004+0000", 2024-05-15T23:35:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024537099999999998 2024-05-15T23:35:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.737766+0000", 2024-05-15T23:35:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010762893000000001 2024-05-15T23:35:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.737821+0000", 2024-05-15T23:35:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4153000000000001e-05 2024-05-15T23:35:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.737902+0000", 2024-05-15T23:35:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1714999999999999e-05 2024-05-15T23:35:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:276280 7.9 7:91893f9b:::benchmark_data_smithi090_143373_object130109:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e634)", 2024-05-15T23:35:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:33:12.371738+0000", 2024-05-15T23:35:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.90438360499999, 2024-05-15T23:35:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36635805199999999, 2024-05-15T23:35:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 276280 2024-05-15T23:35:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371738+0000", 2024-05-15T23:35:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371738+0000", 2024-05-15T23:35:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371739+0000", 2024-05-15T23:35:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.51e-07 2024-05-15T23:35:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371742+0000", 2024-05-15T23:35:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8679999999999999e-06 2024-05-15T23:35:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371746+0000", 2024-05-15T23:35:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7219999999999999e-06 2024-05-15T23:35:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371751+0000", 2024-05-15T23:35:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5179999999999998e-06 2024-05-15T23:35:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371919+0000", 2024-05-15T23:35:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016848000000000001 2024-05-15T23:35:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371928+0000", 2024-05-15T23:35:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2600000000000005e-06 2024-05-15T23:35:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.727187+0000", 2024-05-15T23:35:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35525923199999998 2024-05-15T23:35:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.727316+0000", 2024-05-15T23:35:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129183 2024-05-15T23:35:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 8,10", 2024-05-15T23:35:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.727447+0000", 2024-05-15T23:35:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013054700000000001 2024-05-15T23:35:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.737729+0000", 2024-05-15T23:35:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010282367000000001 2024-05-15T23:35:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.737918+0000", 2024-05-15T23:35:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018870099999999999 2024-05-15T23:35:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.737958+0000", 2024-05-15T23:35:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0550000000000003e-05 2024-05-15T23:35:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.738007+0000", 2024-05-15T23:35:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8863999999999998e-05 2024-05-15T23:35:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.738096+0000", 2024-05-15T23:35:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9209000000000006e-05 2024-05-15T23:35:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:276284 7.9 7:97b85e8a:::benchmark_data_smithi090_143373_object130113:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e634)", 2024-05-15T23:35:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:33:12.371889+0000", 2024-05-15T23:35:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.904233076, 2024-05-15T23:35:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39513651, 2024-05-15T23:35:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 276284 2024-05-15T23:35:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371889+0000", 2024-05-15T23:35:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371889+0000", 2024-05-15T23:35:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371890+0000", 2024-05-15T23:35:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.516e-06 2024-05-15T23:35:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371897+0000", 2024-05-15T23:35:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0620000000000003e-06 2024-05-15T23:35:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371910+0000", 2024-05-15T23:35:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2904999999999999e-05 2024-05-15T23:35:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371921+0000", 2024-05-15T23:35:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0796e-05 2024-05-15T23:35:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371985+0000", 2024-05-15T23:35:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4085999999999996e-05 2024-05-15T23:35:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371992+0000", 2024-05-15T23:35:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.182e-06 2024-05-15T23:35:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.732507+0000", 2024-05-15T23:35:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36051508500000001 2024-05-15T23:35:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.732662+0000", 2024-05-15T23:35:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000154623 2024-05-15T23:35:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 8,10", 2024-05-15T23:35:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.732798+0000", 2024-05-15T23:35:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013550300000000001 2024-05-15T23:35:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.738107+0000", 2024-05-15T23:35:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053096619999999997 2024-05-15T23:35:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.758753+0000", 2024-05-15T23:35:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020645875000000001 2024-05-15T23:35:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.766841+0000", 2024-05-15T23:35:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0080876679999999992 2024-05-15T23:35:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.766911+0000", 2024-05-15T23:35:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0662999999999998e-05 2024-05-15T23:35:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.767025+0000", 2024-05-15T23:35:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113884 2024-05-15T23:35:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:276286 7.9 7:949647bb:::benchmark_data_smithi090_143373_object130115:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e634)", 2024-05-15T23:35:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:33:12.371939+0000", 2024-05-15T23:35:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.90418248399999, 2024-05-15T23:35:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39531272200000001, 2024-05-15T23:35:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:35:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:35:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 276286 2024-05-15T23:35:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371939+0000", 2024-05-15T23:35:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371939+0000", 2024-05-15T23:35:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371940+0000", 2024-05-15T23:35:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7800000000000001e-07 2024-05-15T23:35:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371944+0000", 2024-05-15T23:35:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5710000000000002e-06 2024-05-15T23:35:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371953+0000", 2024-05-15T23:35:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.867e-06 2024-05-15T23:35:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.371959+0000", 2024-05-15T23:35:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.438e-06 2024-05-15T23:35:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.372019+0000", 2024-05-15T23:35:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9524999999999997e-05 2024-05-15T23:35:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:35:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.372025+0000", 2024-05-15T23:35:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4670000000000001e-06 2024-05-15T23:35:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.738141+0000", 2024-05-15T23:35:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36611574899999999 2024-05-15T23:35:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.738264+0000", 2024-05-15T23:35:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012281299999999999 2024-05-15T23:35:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 8,10", 2024-05-15T23:35:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.738368+0000", 2024-05-15T23:35:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103965 2024-05-15T23:35:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.758798+0000", 2024-05-15T23:35:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020430045000000001 2024-05-15T23:35:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.763763+0000", 2024-05-15T23:35:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049650939999999998 2024-05-15T23:35:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.767083+0000", 2024-05-15T23:35:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033198659999999999 2024-05-15T23:35:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.767143+0000", 2024-05-15T23:35:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0574e-05 2024-05-15T23:35:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:33:12.767252+0000", 2024-05-15T23:35:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010897 2024-05-15T23:35:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:122795 8.7 8:e651989f:::benchmark_data_smithi090_157496_object122794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e680)", 2024-05-15T23:35:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:19.316754+0000", 2024-05-15T23:35:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.959367389000001, 2024-05-15T23:35:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21256903299999999, 2024-05-15T23:35:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:35:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:35:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 122795 2024-05-15T23:35:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316754+0000", 2024-05-15T23:35:45.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316754+0000", 2024-05-15T23:35:45.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316757+0000", 2024-05-15T23:35:45.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1449999999999998e-06 2024-05-15T23:35:45.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316784+0000", 2024-05-15T23:35:45.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7304000000000002e-05 2024-05-15T23:35:45.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316795+0000", 2024-05-15T23:35:45.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1185e-05 2024-05-15T23:35:45.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316803+0000", 2024-05-15T23:35:45.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3210000000000005e-06 2024-05-15T23:35:45.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316882+0000", 2024-05-15T23:35:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8652000000000005e-05 2024-05-15T23:35:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-05-15T23:35:45.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316886+0000", 2024-05-15T23:35:45.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5250000000000001e-06 2024-05-15T23:35:45.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.507295+0000", 2024-05-15T23:35:45.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19040950100000001 2024-05-15T23:35:45.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.507421+0000", 2024-05-15T23:35:45.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012542200000000001 2024-05-15T23:35:45.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:35:45.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.507555+0000", 2024-05-15T23:35:45.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013485600000000001 2024-05-15T23:35:45.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.511512+0000", 2024-05-15T23:35:45.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039569549999999998 2024-05-15T23:35:45.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.529220+0000", 2024-05-15T23:35:45.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017707191000000001 2024-05-15T23:35:45.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.529284+0000", 2024-05-15T23:35:45.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4355000000000003e-05 2024-05-15T23:35:45.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.529323+0000", 2024-05-15T23:35:45.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9620999999999999e-05 2024-05-15T23:35:45.382 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:122797 8.7 8:e3e7a3a9:::benchmark_data_smithi090_157496_object122796:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e680)", 2024-05-15T23:35:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:19.316829+0000", 2024-05-15T23:35:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.959292587, 2024-05-15T23:35:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21617704600000001, 2024-05-15T23:35:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:35:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:35:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 122797 2024-05-15T23:35:45.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.384 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316829+0000", 2024-05-15T23:35:45.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316829+0000", 2024-05-15T23:35:45.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316834+0000", 2024-05-15T23:35:45.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3420000000000001e-06 2024-05-15T23:35:45.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316854+0000", 2024-05-15T23:35:45.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9979999999999998e-05 2024-05-15T23:35:45.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316862+0000", 2024-05-15T23:35:45.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0590000000000003e-06 2024-05-15T23:35:45.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316868+0000", 2024-05-15T23:35:45.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0599999999999996e-06 2024-05-15T23:35:45.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316933+0000", 2024-05-15T23:35:45.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5474000000000004e-05 2024-05-15T23:35:45.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2024-05-15T23:35:45.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316939+0000", 2024-05-15T23:35:45.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6069999999999998e-06 2024-05-15T23:35:45.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.511542+0000", 2024-05-15T23:35:45.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.194603215 2024-05-15T23:35:45.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.511660+0000", 2024-05-15T23:35:45.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011786900000000001 2024-05-15T23:35:45.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:35:45.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.511794+0000", 2024-05-15T23:35:45.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013374700000000001 2024-05-15T23:35:45.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.518928+0000", 2024-05-15T23:35:45.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071346580000000003 2024-05-15T23:35:45.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.532913+0000", 2024-05-15T23:35:45.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013984992999999999 2024-05-15T23:35:45.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.532966+0000", 2024-05-15T23:35:45.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3179e-05 2024-05-15T23:35:45.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.533006+0000", 2024-05-15T23:35:45.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9863000000000001e-05 2024-05-15T23:35:45.389 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.389 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.389 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.389 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:122799 8.3 8:caf838ab:::benchmark_data_smithi090_157496_object122798:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e680)", 2024-05-15T23:35:45.389 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:19.316900+0000", 2024-05-15T23:35:45.390 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.959221653, 2024-05-15T23:35:45.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19896370599999999, 2024-05-15T23:35:45.390 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.390 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.390 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:35:45.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:35:45.390 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 122799 2024-05-15T23:35:45.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.390 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316900+0000", 2024-05-15T23:35:45.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316900+0000", 2024-05-15T23:35:45.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316902+0000", 2024-05-15T23:35:45.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.575e-06 2024-05-15T23:35:45.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316929+0000", 2024-05-15T23:35:45.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7463000000000001e-05 2024-05-15T23:35:45.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316943+0000", 2024-05-15T23:35:45.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3757999999999999e-05 2024-05-15T23:35:45.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.316951+0000", 2024-05-15T23:35:45.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9650000000000008e-06 2024-05-15T23:35:45.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.317010+0000", 2024-05-15T23:35:45.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9323000000000001e-05 2024-05-15T23:35:45.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-05-15T23:35:45.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.317022+0000", 2024-05-15T23:35:45.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2170999999999999e-05 2024-05-15T23:35:45.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.507193+0000", 2024-05-15T23:35:45.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19017014199999999 2024-05-15T23:35:45.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.507353+0000", 2024-05-15T23:35:45.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016011199999999999 2024-05-15T23:35:45.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:35:45.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.507485+0000", 2024-05-15T23:35:45.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000132225 2024-05-15T23:35:45.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.515763+0000", 2024-05-15T23:35:45.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0082783609999999997 2024-05-15T23:35:45.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.515782+0000", 2024-05-15T23:35:45.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8969000000000001e-05 2024-05-15T23:35:45.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.515826+0000", 2024-05-15T23:35:45.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3349000000000001e-05 2024-05-15T23:35:45.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.515864+0000", 2024-05-15T23:35:45.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8293000000000001e-05 2024-05-15T23:35:45.396 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.396 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.396 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.396 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:122802 8.6 8:6830764c:::benchmark_data_smithi090_157496_object122801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e680)", 2024-05-15T23:35:45.396 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:19.317008+0000", 2024-05-15T23:35:45.396 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.959113423000002, 2024-05-15T23:35:45.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206172512, 2024-05-15T23:35:45.396 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.396 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.396 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.396 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.396 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:35:45.396 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:35:45.397 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 122802 2024-05-15T23:35:45.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.397 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.317008+0000", 2024-05-15T23:35:45.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.317008+0000", 2024-05-15T23:35:45.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.317009+0000", 2024-05-15T23:35:45.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5700000000000002e-07 2024-05-15T23:35:45.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.317030+0000", 2024-05-15T23:35:45.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1243e-05 2024-05-15T23:35:45.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.317037+0000", 2024-05-15T23:35:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6490000000000002e-06 2024-05-15T23:35:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.317044+0000", 2024-05-15T23:35:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6889999999999998e-06 2024-05-15T23:35:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.317102+0000", 2024-05-15T23:35:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.808e-05 2024-05-15T23:35:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2024-05-15T23:35:45.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.317107+0000", 2024-05-15T23:35:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7099999999999998e-06 2024-05-15T23:35:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.507236+0000", 2024-05-15T23:35:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19012927499999999 2024-05-15T23:35:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.507340+0000", 2024-05-15T23:35:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103978 2024-05-15T23:35:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.507469+0000", 2024-05-15T23:35:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012887400000000001 2024-05-15T23:35:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.523040+0000", 2024-05-15T23:35:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015571457 2024-05-15T23:35:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.523083+0000", 2024-05-15T23:35:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3207999999999997e-05 2024-05-15T23:35:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.523137+0000", 2024-05-15T23:35:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4060000000000001e-05 2024-05-15T23:35:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.523181+0000", 2024-05-15T23:35:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3532000000000003e-05 2024-05-15T23:35:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:122804 8.7 8:e3dd9e3d:::benchmark_data_smithi090_157496_object122803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e680)", 2024-05-15T23:35:45.402 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:19.321063+0000", 2024-05-15T23:35:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.955059331000001, 2024-05-15T23:35:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21202526599999999, 2024-05-15T23:35:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:35:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:35:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 122804 2024-05-15T23:35:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.321063+0000", 2024-05-15T23:35:45.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.321063+0000", 2024-05-15T23:35:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.321065+0000", 2024-05-15T23:35:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.982e-06 2024-05-15T23:35:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.321092+0000", 2024-05-15T23:35:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6577999999999999e-05 2024-05-15T23:35:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.505814+0000", 2024-05-15T23:35:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.184722418 2024-05-15T23:35:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.505830+0000", 2024-05-15T23:35:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5307000000000001e-05 2024-05-15T23:35:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.506143+0000", 2024-05-15T23:35:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031339600000000002 2024-05-15T23:35:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2024-05-15T23:35:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.506148+0000", 2024-05-15T23:35:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0200000000000002e-06 2024-05-15T23:35:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.515272+0000", 2024-05-15T23:35:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091242879999999995 2024-05-15T23:35:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.515398+0000", 2024-05-15T23:35:45.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012523999999999999 2024-05-15T23:35:45.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:35:45.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.515545+0000", 2024-05-15T23:35:45.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014741799999999999 2024-05-15T23:35:45.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.518959+0000", 2024-05-15T23:35:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003413475 2024-05-15T23:35:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.533014+0000", 2024-05-15T23:35:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014055444 2024-05-15T23:35:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.533055+0000", 2024-05-15T23:35:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0720999999999999e-05 2024-05-15T23:35:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:19.533088+0000", 2024-05-15T23:35:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2978999999999997e-05 2024-05-15T23:35:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:146164 8.6 8:6a59e7ef:::benchmark_data_smithi090_157496_object146163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e688)", 2024-05-15T23:35:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:44.018534+0000", 2024-05-15T23:35:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.2575879080000001, 2024-05-15T23:35:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.154419378, 2024-05-15T23:35:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:35:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:35:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 146164 2024-05-15T23:35:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.018534+0000", 2024-05-15T23:35:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.018534+0000", 2024-05-15T23:35:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.018536+0000", 2024-05-15T23:35:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.875e-06 2024-05-15T23:35:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.018570+0000", 2024-05-15T23:35:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3964000000000002e-05 2024-05-15T23:35:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.018579+0000", 2024-05-15T23:35:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9099999999999994e-06 2024-05-15T23:35:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.018586+0000", 2024-05-15T23:35:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2400000000000001e-06 2024-05-15T23:35:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.018642+0000", 2024-05-15T23:35:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6062e-05 2024-05-15T23:35:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.018763+0000", 2024-05-15T23:35:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121221 2024-05-15T23:35:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.018881+0000", 2024-05-15T23:35:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118058 2024-05-15T23:35:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.167877+0000", 2024-05-15T23:35:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.148996237 2024-05-15T23:35:45.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.172860+0000", 2024-05-15T23:35:45.417 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:35:45.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049822809999999999 2024-05-15T23:35:45.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.172911+0000", 2024-05-15T23:35:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0779999999999998e-05 2024-05-15T23:35:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.172953+0000", 2024-05-15T23:35:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2750000000000002e-05 2024-05-15T23:35:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:146322 8.f 8:f22d6116:::benchmark_data_smithi090_157496_object146321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e688)", 2024-05-15T23:35:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:44.275635+0000", 2024-05-15T23:35:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.000486888, 2024-05-15T23:35:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15314660399999999, 2024-05-15T23:35:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:35:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:35:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 146322 2024-05-15T23:35:45.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.275635+0000", 2024-05-15T23:35:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.275635+0000", 2024-05-15T23:35:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.275638+0000", 2024-05-15T23:35:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.909e-06 2024-05-15T23:35:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.275661+0000", 2024-05-15T23:35:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3264999999999998e-05 2024-05-15T23:35:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.275674+0000", 2024-05-15T23:35:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2577e-05 2024-05-15T23:35:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.275682+0000", 2024-05-15T23:35:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6810000000000004e-06 2024-05-15T23:35:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.275860+0000", 2024-05-15T23:35:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017771599999999999 2024-05-15T23:35:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.275963+0000", 2024-05-15T23:35:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010304199999999999 2024-05-15T23:35:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-05-15T23:35:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.276049+0000", 2024-05-15T23:35:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5572e-05 2024-05-15T23:35:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.283147+0000", 2024-05-15T23:35:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0070978359999999997 2024-05-15T23:35:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.428685+0000", 2024-05-15T23:35:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14553895 2024-05-15T23:35:45.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.428732+0000", 2024-05-15T23:35:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6405e-05 2024-05-15T23:35:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.428782+0000", 2024-05-15T23:35:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9651000000000003e-05 2024-05-15T23:35:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:146326 8.7 8:e59c7b5a:::benchmark_data_smithi090_157496_object146325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e688)", 2024-05-15T23:35:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:44.278528+0000", 2024-05-15T23:35:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.99759428400000005, 2024-05-15T23:35:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15693320999999999, 2024-05-15T23:35:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:35:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:35:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 146326 2024-05-15T23:35:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.278528+0000", 2024-05-15T23:35:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.278528+0000", 2024-05-15T23:35:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.278532+0000", 2024-05-15T23:35:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1300000000000003e-06 2024-05-15T23:35:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.278573+0000", 2024-05-15T23:35:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0918999999999999e-05 2024-05-15T23:35:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.278588+0000", 2024-05-15T23:35:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5458e-05 2024-05-15T23:35:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.278598+0000", 2024-05-15T23:35:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0057999999999999e-05 2024-05-15T23:35:45.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.280103+0000", 2024-05-15T23:35:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015049449999999999 2024-05-15T23:35:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.280242+0000", 2024-05-15T23:35:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139219 2024-05-15T23:35:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:35:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.280367+0000", 2024-05-15T23:35:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012452399999999999 2024-05-15T23:35:45.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.420320+0000", 2024-05-15T23:35:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13995302800000001 2024-05-15T23:35:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.435371+0000", 2024-05-15T23:35:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015050717999999999 2024-05-15T23:35:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.435421+0000", 2024-05-15T23:35:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0918999999999998e-05 2024-05-15T23:35:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.435461+0000", 2024-05-15T23:35:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9292000000000003e-05 2024-05-15T23:35:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:146332 8.6 8:6b4e03c5:::benchmark_data_smithi090_157496_object146331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e688)", 2024-05-15T23:35:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:44.285430+0000", 2024-05-15T23:35:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.99069221600000001, 2024-05-15T23:35:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15492555899999999, 2024-05-15T23:35:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:35:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:35:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:35:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:35:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:35:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:35:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 146332 2024-05-15T23:35:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:35:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:35:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.285430+0000", 2024-05-15T23:35:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:35:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.285430+0000", 2024-05-15T23:35:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:35:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:35:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.285431+0000", 2024-05-15T23:35:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.652e-06 2024-05-15T23:35:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:35:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.285468+0000", 2024-05-15T23:35:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7209000000000002e-05 2024-05-15T23:35:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:35:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.285478+0000", 2024-05-15T23:35:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0610000000000002e-06 2024-05-15T23:35:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:35:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.285484+0000", 2024-05-15T23:35:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.584e-06 2024-05-15T23:35:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:35:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.420310+0000", 2024-05-15T23:35:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.134825844 2024-05-15T23:35:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:35:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.420469+0000", 2024-05-15T23:35:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015900300000000001 2024-05-15T23:35:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 10", 2024-05-15T23:35:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.420597+0000", 2024-05-15T23:35:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012834599999999999 2024-05-15T23:35:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:35:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.434677+0000", 2024-05-15T23:35:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014079342 2024-05-15T23:35:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:35:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.440244+0000", 2024-05-15T23:35:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055668920000000004 2024-05-15T23:35:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:35:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.440304+0000", 2024-05-15T23:35:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0643999999999997e-05 2024-05-15T23:35:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:35:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:35:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:35:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:44.440355+0000", 2024-05-15T23:35:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0982000000000001e-05 2024-05-15T23:35:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:35:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:35:45.437 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:35:46.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:45 smithi090 ceph-mon[44434]: from='client.37139 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:35:46.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:45 smithi090 ceph-mon[44434]: osdmap e689: 12 total, 12 up, 10 in 2024-05-15T23:35:46.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:45 smithi090 ceph-mon[44434]: pgmap v1822: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 54 MiB/s wr, 867 op/s; 16398/289624 objects misplaced (5.662%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T23:35:46.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:45 smithi088 ceph-mon[44699]: from='client.37139 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:35:46.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:45 smithi088 ceph-mon[44699]: osdmap e689: 12 total, 12 up, 10 in 2024-05-15T23:35:46.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:45 smithi088 ceph-mon[44699]: pgmap v1822: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 54 MiB/s wr, 867 op/s; 16398/289624 objects misplaced (5.662%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T23:35:46.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:45 smithi084 ceph-mon[39784]: from='client.37139 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:35:46.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:45 smithi084 ceph-mon[39784]: osdmap e689: 12 total, 12 up, 10 in 2024-05-15T23:35:46.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:45 smithi084 ceph-mon[39784]: pgmap v1822: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 54 MiB/s wr, 867 op/s; 16398/289624 objects misplaced (5.662%); 3.5 MiB/s, 55 objects/s recovering 2024-05-15T23:35:46.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:35:46.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T23:35:46.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 7 1 2024-05-15T23:35:46.753 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:35:46.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:46.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:35:46.801 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:46.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:35:46.881 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:46.904 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:35:46.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:46.971 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:35:47.073 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:35:47.122 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:47.160 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:35:47.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:47.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:35:47.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:35:47.303 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:35:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:47.351 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:35:47.454 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:35:47.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:47 smithi084 ceph-mon[39784]: pgmap v1823: 17 pgs: 2 active+recovering, 15 active+clean; 8.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 47 MiB/s wr, 804 op/s; 16398/291038 objects misplaced (5.634%); 2.9 MiB/s, 45 objects/s recovering 2024-05-15T23:35:47.485 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:47.514 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:35:47.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:47.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:35:47.551 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:47 smithi090 ceph-mon[44434]: pgmap v1823: 17 pgs: 2 active+recovering, 15 active+clean; 8.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 47 MiB/s wr, 804 op/s; 16398/291038 objects misplaced (5.634%); 2.9 MiB/s, 45 objects/s recovering 2024-05-15T23:35:47.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:35:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:47.708 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:35:47.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:47.715 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:35:47.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:35:47.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:47 smithi088 ceph-mon[44699]: pgmap v1823: 17 pgs: 2 active+recovering, 15 active+clean; 8.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 47 MiB/s wr, 804 op/s; 16398/291038 objects misplaced (5.634%); 2.9 MiB/s, 45 objects/s recovering 2024-05-15T23:35:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:47.870 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:35:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:47.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:35:47.974 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:48.003 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:48.028 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:48.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:48.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:48.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:35:48.174 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:48.202 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:35:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:48.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:35:48.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:48.348 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:48.382 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:48.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:48.391 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:48.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:35:48.544 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:48.578 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:35:48.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:48.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:35:48.691 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:35:48.720 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:49.228 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:49.273 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:35:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:49.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:35:49.379 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:49.407 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:49.430 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:49.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:49.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:49.536 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:35:49.549 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:49 smithi084 ceph-mon[39784]: pgmap v1824: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 114 B/s rd, 47 MiB/s wr, 966 op/s; 16301/289978 objects misplaced (5.621%); 2.2 MiB/s, 35 objects/s recovering 2024-05-15T23:35:49.567 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:49.591 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:35:49.597 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:49 smithi088 ceph-mon[44699]: pgmap v1824: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 114 B/s rd, 47 MiB/s wr, 966 op/s; 16301/289978 objects misplaced (5.621%); 2.2 MiB/s, 35 objects/s recovering 2024-05-15T23:35:49.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:49.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:35:49.700 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:49.729 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:49.750 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:49.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:49.765 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:35:49.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:49 smithi090 ceph-mon[44434]: pgmap v1824: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 114 B/s rd, 47 MiB/s wr, 966 op/s; 16301/289978 objects misplaced (5.621%); 2.2 MiB/s, 35 objects/s recovering 2024-05-15T23:35:49.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:49.897 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:49.926 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:49.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:49.931 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:50.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:35:50.069 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:50.105 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:35:50.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:50.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:35:50.214 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:35:50.244 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:50.272 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:35:50.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:50.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:35:50.377 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:35:50.409 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:50.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:50 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:35:50.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:50 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:35:50.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:50 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:35:50.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:50 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:35:50.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:50 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:35:50.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:50 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:35:51.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:51 smithi084 ceph-mon[39784]: osdmap e690: 12 total, 12 up, 10 in 2024-05-15T23:35:51.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:51 smithi084 ceph-mon[39784]: pgmap v1826: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 23 MiB/s wr, 782 op/s; 16167/287296 objects misplaced (5.627%); 3.6 MiB/s, 56 objects/s recovering 2024-05-15T23:35:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:51 smithi090 ceph-mon[44434]: osdmap e690: 12 total, 12 up, 10 in 2024-05-15T23:35:51.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:51 smithi090 ceph-mon[44434]: pgmap v1826: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 23 MiB/s wr, 782 op/s; 16167/287296 objects misplaced (5.627%); 3.6 MiB/s, 56 objects/s recovering 2024-05-15T23:35:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:51 smithi088 ceph-mon[44699]: osdmap e690: 12 total, 12 up, 10 in 2024-05-15T23:35:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:51 smithi088 ceph-mon[44699]: pgmap v1826: 17 pgs: 2 active+recovering, 15 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 23 MiB/s wr, 782 op/s; 16167/287296 objects misplaced (5.627%); 3.6 MiB/s, 56 objects/s recovering 2024-05-15T23:35:52.559 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:35:52.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:52.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:35:52.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:35:52.714 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:52.741 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:35:52.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:52.744 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:35:52.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:35:52.878 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:53.082 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:35:53.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:53.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:35:53.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:35:53.226 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:53.259 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:35:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:53.263 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:35:53.364 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:35:53.394 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:53.422 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:35:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:53.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:35:53.528 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:35:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:53 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:35:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:53 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:35:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:53 smithi090 ceph-mon[44434]: pgmap v1827: 17 pgs: 2 active+recovering, 15 active+clean; 8.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 120 B/s rd, 16 MiB/s wr, 934 op/s; 16167/282256 objects misplaced (5.728%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T23:35:53.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:53 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:35:53.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:53 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:35:53.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:53 smithi084 ceph-mon[39784]: pgmap v1827: 17 pgs: 2 active+recovering, 15 active+clean; 8.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 120 B/s rd, 16 MiB/s wr, 934 op/s; 16167/282256 objects misplaced (5.728%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T23:35:53.560 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:53 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:35:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:53 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:35:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:53 smithi088 ceph-mon[44699]: pgmap v1827: 17 pgs: 2 active+recovering, 15 active+clean; 8.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 120 B/s rd, 16 MiB/s wr, 934 op/s; 16167/282256 objects misplaced (5.728%); 1.7 MiB/s, 27 objects/s recovering 2024-05-15T23:35:54.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:54 smithi084 ceph-mon[39784]: osdmap e691: 12 total, 12 up, 10 in 2024-05-15T23:35:54.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:54 smithi090 ceph-mon[44434]: osdmap e691: 12 total, 12 up, 10 in 2024-05-15T23:35:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:54 smithi088 ceph-mon[44699]: osdmap e691: 12 total, 12 up, 10 in 2024-05-15T23:35:55.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:55 smithi084 ceph-mon[39784]: pgmap v1829: 17 pgs: 2 active+recovering, 15 active+clean; 8.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 7.7 MiB/s wr, 894 op/s; 15807/280672 objects misplaced (5.632%); 4.6 MiB/s, 73 objects/s recovering 2024-05-15T23:35:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:55 smithi090 ceph-mon[44434]: pgmap v1829: 17 pgs: 2 active+recovering, 15 active+clean; 8.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 7.7 MiB/s wr, 894 op/s; 15807/280672 objects misplaced (5.632%); 4.6 MiB/s, 73 objects/s recovering 2024-05-15T23:35:55.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:55 smithi088 ceph-mon[44699]: pgmap v1829: 17 pgs: 2 active+recovering, 15 active+clean; 8.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 7.7 MiB/s wr, 894 op/s; 15807/280672 objects misplaced (5.632%); 4.6 MiB/s, 73 objects/s recovering 2024-05-15T23:35:55.848 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:35:55.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:55.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:35:55.954 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:35:55.987 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:56.009 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:35:56.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:56.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:35:56.115 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:56.147 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:56.172 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:56.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:56.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:56.248 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:35:56.248 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:35:56.277 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:35:56.304 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:35:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:56.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:35:56.351 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:35:56.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:35:56.472 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:56.512 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:35:56.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:56.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:35:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1807680725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:35:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3014597844' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:35:56.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.572 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:35:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1807680725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:35:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3014597844' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:35:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.643 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1807680725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:35:56.643 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.643 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.644 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.644 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.644 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.644 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.644 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.644 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3014597844' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:35:56.644 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.644 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.644 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.644 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:56.646 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:56.669 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:35:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:56.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:35:56.776 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:35:56.806 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:56.833 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:35:56.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:56.838 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:35:56.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:35:56.972 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:57.058 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:35:57.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:57.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:35:57.162 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:35:57.193 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:57.227 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:35:57.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:57.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:35:57.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:35:57.366 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:57.389 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:35:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:57.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:35:57.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:35:57.532 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:57 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:35:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3014597844' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:35:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:57 smithi090 ceph-mon[44434]: osdmap e692: 12 total, 12 up, 10 in 2024-05-15T23:35:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:57 smithi090 ceph-mon[44434]: pgmap v1831: 17 pgs: 2 active+recovering, 15 active+clean; 8.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 872 op/s; 15807/277398 objects misplaced (5.698%); 3.2 MiB/s, 51 objects/s recovering 2024-05-15T23:35:57.560 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:35:57.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:57.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:35:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:57.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:57 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:35:57.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3014597844' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:35:57.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:57 smithi084 ceph-mon[39784]: osdmap e692: 12 total, 12 up, 10 in 2024-05-15T23:35:57.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:57 smithi084 ceph-mon[39784]: pgmap v1831: 17 pgs: 2 active+recovering, 15 active+clean; 8.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 872 op/s; 15807/277398 objects misplaced (5.698%); 3.2 MiB/s, 51 objects/s recovering 2024-05-15T23:35:57.664 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:35:57.725 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:57.753 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:35:57.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:57.757 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:35:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:35:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:57 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:35:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3014597844' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:35:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:57 smithi088 ceph-mon[44699]: osdmap e692: 12 total, 12 up, 10 in 2024-05-15T23:35:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:57 smithi088 ceph-mon[44699]: pgmap v1831: 17 pgs: 2 active+recovering, 15 active+clean; 8.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 872 op/s; 15807/277398 objects misplaced (5.698%); 3.2 MiB/s, 51 objects/s recovering 2024-05-15T23:35:57.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:35:57.888 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:57.965 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:35:57.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:57.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:35:58.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:35:58.103 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:58.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:58 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/395178205' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:35:58.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:58 smithi090 ceph-mon[44434]: from='client.37157 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:35:58.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:58 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/395178205' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:35:58.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:58 smithi088 ceph-mon[44699]: from='client.37157 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:35:58.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:58 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/395178205' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:35:58.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:58 smithi084 ceph-mon[39784]: from='client.37157 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:35:59.460 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:35:59.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:59.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:35:59.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:35:59.605 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:59.622 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:35:59.622 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:35:59.622 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:35:59.622 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:35:59.623 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:35:59.623 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:35:59.623 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:35:59.623 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:35:59.623 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:35:59.623 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:35:59.623 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:35:59.623 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:35:59.623 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:35:59.623 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:35:59.623 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:35:59.624 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:35:59.624 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:35:59.624 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:35:59.624 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:35:59.624 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:35:59.624 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:35:59.624 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:35:59.624 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:35:59.624 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:35:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:35:59.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:59.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:35:59.737 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:35:59.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:59 smithi084 ceph-mon[39784]: from='client.37157 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:35:59.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:59 smithi084 ceph-mon[39784]: osdmap e693: 12 total, 12 up, 10 in 2024-05-15T23:35:59.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:59 smithi084 ceph-mon[39784]: 8.c scrub starts 2024-05-15T23:35:59.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:59 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:35:59.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:59 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:35:59.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:35:59 smithi084 ceph-mon[39784]: pgmap v1833: 17 pgs: 2 active+recovering, 15 active+clean; 8.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 780 op/s; 15558/272886 objects misplaced (5.701%); 6.3 MiB/s, 101 objects/s recovering 2024-05-15T23:35:59.766 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:35:59.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:59 smithi090 ceph-mon[44434]: from='client.37157 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:35:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:59 smithi090 ceph-mon[44434]: osdmap e693: 12 total, 12 up, 10 in 2024-05-15T23:35:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:59 smithi090 ceph-mon[44434]: 8.c scrub starts 2024-05-15T23:35:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:59 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:35:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:59 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:35:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:35:59 smithi090 ceph-mon[44434]: pgmap v1833: 17 pgs: 2 active+recovering, 15 active+clean; 8.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 780 op/s; 15558/272886 objects misplaced (5.701%); 6.3 MiB/s, 101 objects/s recovering 2024-05-15T23:35:59.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:59 smithi088 ceph-mon[44699]: from='client.37157 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:35:59.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:59 smithi088 ceph-mon[44699]: osdmap e693: 12 total, 12 up, 10 in 2024-05-15T23:35:59.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:59 smithi088 ceph-mon[44699]: 8.c scrub starts 2024-05-15T23:35:59.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:59 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:35:59.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:59 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:35:59.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:35:59 smithi088 ceph-mon[44699]: pgmap v1833: 17 pgs: 2 active+recovering, 15 active+clean; 8.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 780 op/s; 15558/272886 objects misplaced (5.701%); 6.3 MiB/s, 101 objects/s recovering 2024-05-15T23:35:59.892 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:35:59.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:35:59.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:35:59.996 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:36:00.066 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:00.109 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:36:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:00.114 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:36:00.215 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:36:00.246 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:00.274 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:36:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:00.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:36:00.292 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-05-15T23:36:00.379 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:36:00.409 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:00.467 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:36:00.473 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:00 smithi090 ceph-mon[44434]: osdmap e694: 12 total, 12 up, 10 in 2024-05-15T23:36:00.473 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:00 smithi090 ceph-mon[44434]: 8.7 scrub starts 2024-05-15T23:36:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:00.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:36:00.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:00 smithi088 ceph-mon[44699]: osdmap e694: 12 total, 12 up, 10 in 2024-05-15T23:36:00.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:00 smithi088 ceph-mon[44699]: 8.7 scrub starts 2024-05-15T23:36:00.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:36:00.621 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:00.664 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:36:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:00.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:36:00.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:36:00.856 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:00.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:00 smithi084 ceph-mon[39784]: osdmap e694: 12 total, 12 up, 10 in 2024-05-15T23:36:00.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:00 smithi084 ceph-mon[39784]: 8.7 scrub starts 2024-05-15T23:36:00.911 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:36:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:00.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:36:01.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:36:01.103 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:01.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:36:01.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:01 smithi090 ceph-mon[44434]: 8.2 scrub starts 2024-05-15T23:36:01.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:01 smithi090 ceph-mon[44434]: 8.c scrub ok 2024-05-15T23:36:01.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:01 smithi090 ceph-mon[44434]: pgmap v1835: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 8.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 847 op/s; 15389/270496 objects misplaced (5.689%); 4.4 MiB/s, 69 objects/s recovering 2024-05-15T23:36:01.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:01 smithi088 ceph-mon[44699]: 8.2 scrub starts 2024-05-15T23:36:01.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:01 smithi088 ceph-mon[44699]: 8.c scrub ok 2024-05-15T23:36:01.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:01 smithi088 ceph-mon[44699]: pgmap v1835: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 8.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 847 op/s; 15389/270496 objects misplaced (5.689%); 4.4 MiB/s, 69 objects/s recovering 2024-05-15T23:36:01.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:01 smithi084 ceph-mon[39784]: 8.2 scrub starts 2024-05-15T23:36:01.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:01 smithi084 ceph-mon[39784]: 8.c scrub ok 2024-05-15T23:36:01.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:01 smithi084 ceph-mon[39784]: pgmap v1835: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 8.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 847 op/s; 15389/270496 objects misplaced (5.689%); 4.4 MiB/s, 69 objects/s recovering 2024-05-15T23:36:02.654 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:36:02.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:02.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:36:02.762 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:36:02.791 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:02 smithi088 ceph-mon[44699]: 8.2 scrub ok 2024-05-15T23:36:02.791 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:02 smithi088 ceph-mon[44699]: 8.7 scrub ok 2024-05-15T23:36:02.791 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:02 smithi088 ceph-mon[44699]: 8.0 scrub starts 2024-05-15T23:36:02.793 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:02.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:02 smithi090 ceph-mon[44434]: 8.2 scrub ok 2024-05-15T23:36:02.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:02 smithi090 ceph-mon[44434]: 8.7 scrub ok 2024-05-15T23:36:02.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:02 smithi090 ceph-mon[44434]: 8.0 scrub starts 2024-05-15T23:36:02.818 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:36:02.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:02.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:36:02.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:02 smithi084 ceph-mon[39784]: 8.2 scrub ok 2024-05-15T23:36:02.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:02 smithi084 ceph-mon[39784]: 8.7 scrub ok 2024-05-15T23:36:02.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:02 smithi084 ceph-mon[39784]: 8.0 scrub starts 2024-05-15T23:36:02.868 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T23:36:02.921 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:36:02.952 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:02.977 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:36:02.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:02.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:36:03.080 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:36:03.118 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:03.142 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:36:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:03.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:36:03.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 11 1 2024-05-15T23:36:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:36:03.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:36:03.280 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:03.341 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:36:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:03.343 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:36:03.365 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:36:03.444 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:36:03.474 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:03.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:03 smithi084 ceph-mon[39784]: pgmap v1836: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 8.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 872 op/s; 15389/266374 objects misplaced (5.777%); 3.8 MiB/s, 60 objects/s recovering 2024-05-15T23:36:03.503 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:36:03.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:03.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:36:03.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:03 smithi090 ceph-mon[44434]: pgmap v1836: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 8.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 872 op/s; 15389/266374 objects misplaced (5.777%); 3.8 MiB/s, 60 objects/s recovering 2024-05-15T23:36:03.607 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:36:03.636 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:03.658 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:36:03.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:03.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:36:03.762 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:36:03.791 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:03 smithi088 ceph-mon[44699]: pgmap v1836: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 8.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 872 op/s; 15389/266374 objects misplaced (5.777%); 3.8 MiB/s, 60 objects/s recovering 2024-05-15T23:36:03.793 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:03.823 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:36:03.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:03.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:36:03.927 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:36:03.956 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:03.982 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:36:03.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:03.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:36:04.089 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:36:04.121 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:04.471 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:36:04.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:04.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:36:04.575 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:36:04.606 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:04.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:04 smithi090 ceph-mon[44434]: 8.0 scrub ok 2024-05-15T23:36:04.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:04 smithi088 ceph-mon[44699]: 8.0 scrub ok 2024-05-15T23:36:04.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:04 smithi084 ceph-mon[39784]: 8.0 scrub ok 2024-05-15T23:36:04.998 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:36:05.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:05 smithi090 ceph-mon[44434]: pgmap v1837: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 8.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 771 op/s; 14935/265046 objects misplaced (5.635%); 6.8 MiB/s, 108 objects/s recovering 2024-05-15T23:36:05.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:05 smithi088 ceph-mon[44699]: pgmap v1837: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 8.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 771 op/s; 14935/265046 objects misplaced (5.635%); 6.8 MiB/s, 108 objects/s recovering 2024-05-15T23:36:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:05 smithi084 ceph-mon[39784]: pgmap v1837: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 8.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 771 op/s; 14935/265046 objects misplaced (5.635%); 6.8 MiB/s, 108 objects/s recovering 2024-05-15T23:36:06.561 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:36:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:06.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:36:06.576 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:36:06.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:36:06.716 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:06.884 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:36:06.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:06.889 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:36:06.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:36:07.018 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:07.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:07 smithi084 ceph-mon[39784]: pgmap v1838: 17 pgs: 2 active+recovering, 15 active+clean; 8.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 667 op/s; 14935/262124 objects misplaced (5.698%); 6.2 MiB/s, 98 objects/s recovering 2024-05-15T23:36:07.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:07 smithi090 ceph-mon[44434]: pgmap v1838: 17 pgs: 2 active+recovering, 15 active+clean; 8.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 667 op/s; 14935/262124 objects misplaced (5.698%); 6.2 MiB/s, 98 objects/s recovering 2024-05-15T23:36:07.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:07 smithi088 ceph-mon[44699]: pgmap v1838: 17 pgs: 2 active+recovering, 15 active+clean; 8.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 667 op/s; 14935/262124 objects misplaced (5.698%); 6.2 MiB/s, 98 objects/s recovering 2024-05-15T23:36:07.901 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:36:09.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:09 smithi084 ceph-mon[39784]: pgmap v1839: 17 pgs: 2 active+recovering, 15 active+clean; 7.9 GiB data, 22 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 766 op/s; 14687/257548 objects misplaced (5.703%); 5.4 MiB/s, 87 objects/s recovering 2024-05-15T23:36:09.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:09 smithi090 ceph-mon[44434]: pgmap v1839: 17 pgs: 2 active+recovering, 15 active+clean; 7.9 GiB data, 22 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 766 op/s; 14687/257548 objects misplaced (5.703%); 5.4 MiB/s, 87 objects/s recovering 2024-05-15T23:36:09.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:09 smithi088 ceph-mon[44699]: pgmap v1839: 17 pgs: 2 active+recovering, 15 active+clean; 7.9 GiB data, 22 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 766 op/s; 14687/257548 objects misplaced (5.703%); 5.4 MiB/s, 87 objects/s recovering 2024-05-15T23:36:10.459 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:36:10.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:10.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:36:10.565 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:36:10.594 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:10.616 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:36:10.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:10.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:36:10.721 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:36:10.770 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:11.702 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:36:11.703 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:36:12.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:11 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:36:12.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:11 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:36:12.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:11 smithi090 ceph-mon[44434]: osdmap e695: 12 total, 12 up, 10 in 2024-05-15T23:36:12.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:11 smithi090 ceph-mon[44434]: pgmap v1841: 17 pgs: 2 active+recovering, 15 active+clean; 7.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 778 op/s; 14446/254926 objects misplaced (5.667%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T23:36:12.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:11 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:36:12.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:11 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:36:12.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:11 smithi088 ceph-mon[44699]: osdmap e695: 12 total, 12 up, 10 in 2024-05-15T23:36:12.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:11 smithi088 ceph-mon[44699]: pgmap v1841: 17 pgs: 2 active+recovering, 15 active+clean; 7.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 778 op/s; 14446/254926 objects misplaced (5.667%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T23:36:12.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:11 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:36:12.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:11 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:36:12.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:11 smithi084 ceph-mon[39784]: osdmap e695: 12 total, 12 up, 10 in 2024-05-15T23:36:12.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:11 smithi084 ceph-mon[39784]: pgmap v1841: 17 pgs: 2 active+recovering, 15 active+clean; 7.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 778 op/s; 14446/254926 objects misplaced (5.667%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T23:36:13.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3421086774' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:36:13.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3421086774' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:36:13.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3421086774' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:36:14.162 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:36:14.365 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:36:14.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:14 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:36:14.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:14 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:36:14.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:14 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3421086774' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:36:14.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:14 smithi084 ceph-mon[39784]: osdmap e696: 12 total, 12 up, 10 in 2024-05-15T23:36:14.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:14 smithi084 ceph-mon[39784]: pgmap v1843: 17 pgs: 2 active+recovering, 15 active+clean; 7.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 931 op/s; 14446/250142 objects misplaced (5.775%); 3.8 MiB/s, 61 objects/s recovering 2024-05-15T23:36:14.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:14.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:36:14.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:36:14.528 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:14.551 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:14 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:36:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:14 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:36:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:14 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3421086774' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:36:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:14 smithi090 ceph-mon[44434]: osdmap e696: 12 total, 12 up, 10 in 2024-05-15T23:36:14.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:14 smithi090 ceph-mon[44434]: pgmap v1843: 17 pgs: 2 active+recovering, 15 active+clean; 7.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 931 op/s; 14446/250142 objects misplaced (5.775%); 3.8 MiB/s, 61 objects/s recovering 2024-05-15T23:36:14.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:14 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:36:14.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:14 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:36:14.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:14 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3421086774' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:36:14.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:14 smithi088 ceph-mon[44699]: osdmap e696: 12 total, 12 up, 10 in 2024-05-15T23:36:14.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:14 smithi088 ceph-mon[44699]: pgmap v1843: 17 pgs: 2 active+recovering, 15 active+clean; 7.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 931 op/s; 14446/250142 objects misplaced (5.775%); 3.8 MiB/s, 61 objects/s recovering 2024-05-15T23:36:15.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:15 smithi084 ceph-mon[39784]: pgmap v1844: 17 pgs: 2 active+recovering, 15 active+clean; 7.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 846 op/s; 13948/248584 objects misplaced (5.611%); 7.7 MiB/s, 123 objects/s recovering 2024-05-15T23:36:15.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:15 smithi090 ceph-mon[44434]: pgmap v1844: 17 pgs: 2 active+recovering, 15 active+clean; 7.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 846 op/s; 13948/248584 objects misplaced (5.611%); 7.7 MiB/s, 123 objects/s recovering 2024-05-15T23:36:15.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:15 smithi088 ceph-mon[44699]: pgmap v1844: 17 pgs: 2 active+recovering, 15 active+clean; 7.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 846 op/s; 13948/248584 objects misplaced (5.611%); 7.7 MiB/s, 123 objects/s recovering 2024-05-15T23:36:16.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:16 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:36:16.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:16 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:36:16.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:36:16.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:16 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:36:16.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:16 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:36:16.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:36:16.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:16 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:36:16.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:16 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:36:16.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:36:17.218 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T23:36:17.399 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.11 to 1 (10000) 2024-05-15T23:36:17.444 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:17 smithi084 ceph-mon[39784]: osdmap e697: 12 total, 12 up, 10 in 2024-05-15T23:36:17.444 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:17 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1097610219' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:36:17.444 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:17 smithi084 ceph-mon[39784]: pgmap v1846: 17 pgs: 2 active+recovering, 15 active+clean; 7.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 894 op/s; 13948/245646 objects misplaced (5.678%); 4.9 MiB/s, 78 objects/s recovering 2024-05-15T23:36:17.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:17 smithi090 ceph-mon[44434]: osdmap e697: 12 total, 12 up, 10 in 2024-05-15T23:36:17.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:17 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1097610219' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:36:17.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:17 smithi090 ceph-mon[44434]: pgmap v1846: 17 pgs: 2 active+recovering, 15 active+clean; 7.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 894 op/s; 13948/245646 objects misplaced (5.678%); 4.9 MiB/s, 78 objects/s recovering 2024-05-15T23:36:17.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:17 smithi088 ceph-mon[44699]: osdmap e697: 12 total, 12 up, 10 in 2024-05-15T23:36:17.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:17 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1097610219' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:36:17.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:17 smithi088 ceph-mon[44699]: pgmap v1846: 17 pgs: 2 active+recovering, 15 active+clean; 7.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 894 op/s; 13948/245646 objects misplaced (5.678%); 4.9 MiB/s, 78 objects/s recovering 2024-05-15T23:36:18.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:36:18.063 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:36:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:18.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:36:18.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:36:18.231 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:18.295 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:36:18.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:18.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:36:18.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:36:18.433 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:18.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:18 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1097610219' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:36:18.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:18 smithi090 ceph-mon[44434]: osdmap e698: 12 total, 12 up, 10 in 2024-05-15T23:36:18.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:36:18.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:36:18.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:36:18.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:18 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1097610219' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:36:18.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:18 smithi088 ceph-mon[44699]: osdmap e698: 12 total, 12 up, 10 in 2024-05-15T23:36:18.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:36:18.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:36:18.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:36:18.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:18 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1097610219' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:36:18.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:18 smithi084 ceph-mon[39784]: osdmap e698: 12 total, 12 up, 10 in 2024-05-15T23:36:18.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:36:18.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:36:18.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:36:19.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 9 1 2024-05-15T23:36:19.184 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:36:19.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:19.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:36:19.289 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:36:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:19.338 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:36:19.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:19.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:36:19.442 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:36:19.470 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:19.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:19 smithi088 ceph-mon[44699]: pgmap v1848: 17 pgs: 2 active+recovering, 15 active+clean; 7.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 802 op/s; 13689/241260 objects misplaced (5.674%); 7.6 MiB/s, 122 objects/s recovering 2024-05-15T23:36:19.601 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:36:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:19.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:36:19.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:36:19.735 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:19 smithi090 ceph-mon[44434]: pgmap v1848: 17 pgs: 2 active+recovering, 15 active+clean; 7.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 802 op/s; 13689/241260 objects misplaced (5.674%); 7.6 MiB/s, 122 objects/s recovering 2024-05-15T23:36:19.737 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:19.753 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:36:19.801 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:36:19.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:19.806 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:36:19.813 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:36:19.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:19 smithi084 ceph-mon[39784]: pgmap v1848: 17 pgs: 2 active+recovering, 15 active+clean; 7.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 802 op/s; 13689/241260 objects misplaced (5.674%); 7.6 MiB/s, 122 objects/s recovering 2024-05-15T23:36:19.907 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:36:19.937 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:20.017 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:36:20.039 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:36:20.069 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:36:20.090 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:36:20.121 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:36:20.147 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:36:20.170 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:36:20.193 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:36:20.225 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:36:20.248 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:36:20.273 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:36:20.322 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:36:20.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:36:20.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:36:20.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:36:20.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:36:20.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:36:20.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:36:20.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:36:20.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:36:20.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:36:20.323 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:36:20.323 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:36:20.323 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:36:20.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:20 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2149144652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:36:20.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:20.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:20.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:20 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:20 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2149144652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:36:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:20 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:20.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:20 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2149144652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:36:20.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:20.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:20.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:20 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:21 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:21 smithi084 ceph-mon[39784]: pgmap v1849: 17 pgs: 2 active+recovering, 15 active+clean; 7.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 709 op/s; 13430/238788 objects misplaced (5.624%); 7.9 MiB/s, 126 objects/s recovering 2024-05-15T23:36:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:21 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:21 smithi090 ceph-mon[44434]: pgmap v1849: 17 pgs: 2 active+recovering, 15 active+clean; 7.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 709 op/s; 13430/238788 objects misplaced (5.624%); 7.9 MiB/s, 126 objects/s recovering 2024-05-15T23:36:21.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:21 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:21.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:21 smithi088 ceph-mon[44699]: pgmap v1849: 17 pgs: 2 active+recovering, 15 active+clean; 7.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 709 op/s; 13430/238788 objects misplaced (5.624%); 7.9 MiB/s, 126 objects/s recovering 2024-05-15T23:36:23.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:36:23.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:36:23.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:23 smithi084 ceph-mon[39784]: pgmap v1850: 17 pgs: 2 active+recovering, 15 active+clean; 7.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 904 op/s; 13430/234110 objects misplaced (5.737%); 4.0 MiB/s, 64 objects/s recovering 2024-05-15T23:36:23.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:36:23.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:36:23.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:23 smithi090 ceph-mon[44434]: pgmap v1850: 17 pgs: 2 active+recovering, 15 active+clean; 7.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 904 op/s; 13430/234110 objects misplaced (5.737%); 4.0 MiB/s, 64 objects/s recovering 2024-05-15T23:36:23.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:36:23.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:36:23.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:23 smithi088 ceph-mon[44699]: pgmap v1850: 17 pgs: 2 active+recovering, 15 active+clean; 7.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 904 op/s; 13430/234110 objects misplaced (5.737%); 4.0 MiB/s, 64 objects/s recovering 2024-05-15T23:36:23.998 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:36:24.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:24.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:36:24.103 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:36:24.132 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:24.156 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:36:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:24.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:36:24.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:36:24.292 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:24.380 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:36:24.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:24 smithi088 ceph-mon[44699]: osdmap e699: 12 total, 12 up, 10 in 2024-05-15T23:36:24.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:24 smithi084 ceph-mon[39784]: osdmap e699: 12 total, 12 up, 10 in 2024-05-15T23:36:24.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:24 smithi090 ceph-mon[44434]: osdmap e699: 12 total, 12 up, 10 in 2024-05-15T23:36:25.549 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:36:25.549 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:36:25.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:25.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:36:25.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:25 smithi084 ceph-mon[39784]: pgmap v1852: 17 pgs: 2 active+recovering, 15 active+clean; 7.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 816 op/s; 12913/232582 objects misplaced (5.552%); 8.1 MiB/s, 129 objects/s recovering 2024-05-15T23:36:25.656 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:36:25.686 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:25.709 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:36:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:25.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:36:25.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:25 smithi090 ceph-mon[44434]: pgmap v1852: 17 pgs: 2 active+recovering, 15 active+clean; 7.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 816 op/s; 12913/232582 objects misplaced (5.552%); 8.1 MiB/s, 129 objects/s recovering 2024-05-15T23:36:25.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:36:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:25 smithi088 ceph-mon[44699]: pgmap v1852: 17 pgs: 2 active+recovering, 15 active+clean; 7.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 816 op/s; 12913/232582 objects misplaced (5.552%); 8.1 MiB/s, 129 objects/s recovering 2024-05-15T23:36:25.850 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:27.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:27 smithi084 ceph-mon[39784]: pgmap v1853: 17 pgs: 2 active+recovering, 15 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 819 op/s; 12913/229396 objects misplaced (5.629%); 6.5 MiB/s, 104 objects/s recovering 2024-05-15T23:36:27.624 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:36:27.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:27.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:36:27.633 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:36:27.731 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:36:27.760 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:27 smithi088 ceph-mon[44699]: pgmap v1853: 17 pgs: 2 active+recovering, 15 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 819 op/s; 12913/229396 objects misplaced (5.629%); 6.5 MiB/s, 104 objects/s recovering 2024-05-15T23:36:27.763 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:27.784 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:36:27.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:27.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:36:27.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:27 smithi090 ceph-mon[44434]: pgmap v1853: 17 pgs: 2 active+recovering, 15 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 819 op/s; 12913/229396 objects misplaced (5.629%); 6.5 MiB/s, 104 objects/s recovering 2024-05-15T23:36:27.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:36:27.920 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:28.144 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:36:28.144 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-05-15T23:36:28.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-05-15T23:36:28.145 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:36:28.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.145 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:169823 7.e 7:78b6abc8:::benchmark_data_smithi090_143373_object23652:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:36:28.145 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:02.738352+0000", 2024-05-15T23:36:28.145 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 325.40269956700001, 2024-05-15T23:36:28.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.179006257, 2024-05-15T23:36:28.145 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.145 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.145 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.145 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.145 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.145 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.145 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 169823 2024-05-15T23:36:28.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.146 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738352+0000", 2024-05-15T23:36:28.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738352+0000", 2024-05-15T23:36:28.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738354+0000", 2024-05-15T23:36:28.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4670000000000001e-06 2024-05-15T23:36:28.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738359+0000", 2024-05-15T23:36:28.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.113e-06 2024-05-15T23:36:28.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738368+0000", 2024-05-15T23:36:28.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9080000000000001e-06 2024-05-15T23:36:28.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738374+0000", 2024-05-15T23:36:28.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2319999999999998e-06 2024-05-15T23:36:28.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738442+0000", 2024-05-15T23:36:28.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7432000000000001e-05 2024-05-15T23:36:28.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738570+0000", 2024-05-15T23:36:28.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000128147 2024-05-15T23:36:28.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 10", 2024-05-15T23:36:28.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.738691+0000", 2024-05-15T23:36:28.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121704 2024-05-15T23:36:28.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.743746+0000", 2024-05-15T23:36:28.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050545909999999998 2024-05-15T23:36:28.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.917254+0000", 2024-05-15T23:36:28.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.173508254 2024-05-15T23:36:28.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.917310+0000", 2024-05-15T23:36:28.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5937999999999999e-05 2024-05-15T23:36:28.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:02.917359+0000", 2024-05-15T23:36:28.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8470999999999997e-05 2024-05-15T23:36:28.150 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.150 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.150 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.151 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:187219 7.8 7:17d271df:::benchmark_data_smithi090_143373_object41048:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-05-15T23:36:28.151 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:24.325708+0000", 2024-05-15T23:36:28.151 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 303.81534377000003, 2024-05-15T23:36:28.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22588118600000001, 2024-05-15T23:36:28.151 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.151 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.151 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.151 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 187219 2024-05-15T23:36:28.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.151 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.325708+0000", 2024-05-15T23:36:28.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.325708+0000", 2024-05-15T23:36:28.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.325709+0000", 2024-05-15T23:36:28.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.99e-07 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.325712+0000", 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9969999999999999e-06 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.325717+0000", 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1710000000000001e-06 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.325722+0000", 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.198e-06 2024-05-15T23:36:28.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.342296+0000", 2024-05-15T23:36:28.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016573944 2024-05-15T23:36:28.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.342433+0000", 2024-05-15T23:36:28.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000136339 2024-05-15T23:36:28.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.342557+0000", 2024-05-15T23:36:28.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012408300000000001 2024-05-15T23:36:28.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.347724+0000", 2024-05-15T23:36:28.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051678649999999998 2024-05-15T23:36:28.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.551460+0000", 2024-05-15T23:36:28.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.20373519600000001 2024-05-15T23:36:28.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.551519+0000", 2024-05-15T23:36:28.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8953999999999998e-05 2024-05-15T23:36:28.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.551589+0000", 2024-05-15T23:36:28.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0740000000000004e-05 2024-05-15T23:36:28.157 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.157 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.157 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.157 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:187251 7.8 7:182edd5f:::benchmark_data_smithi090_143373_object41080:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-05-15T23:36:28.157 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:24.357546+0000", 2024-05-15T23:36:28.157 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 303.78350579800002, 2024-05-15T23:36:28.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.224264303, 2024-05-15T23:36:28.158 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.158 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.158 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.158 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.158 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.158 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.158 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 187251 2024-05-15T23:36:28.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.158 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.357546+0000", 2024-05-15T23:36:28.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.357546+0000", 2024-05-15T23:36:28.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.357549+0000", 2024-05-15T23:36:28.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0120000000000001e-06 2024-05-15T23:36:28.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.357557+0000", 2024-05-15T23:36:28.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5070000000000003e-06 2024-05-15T23:36:28.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.357571+0000", 2024-05-15T23:36:28.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3896e-05 2024-05-15T23:36:28.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.357580+0000", 2024-05-15T23:36:28.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9500000000000007e-06 2024-05-15T23:36:28.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.384342+0000", 2024-05-15T23:36:28.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026762135999999999 2024-05-15T23:36:28.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.384470+0000", 2024-05-15T23:36:28.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012784999999999999 2024-05-15T23:36:28.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.384582+0000", 2024-05-15T23:36:28.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112768 2024-05-15T23:36:28.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394572+0000", 2024-05-15T23:36:28.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0099897700000000002 2024-05-15T23:36:28.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581707+0000", 2024-05-15T23:36:28.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.187134464 2024-05-15T23:36:28.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581756+0000", 2024-05-15T23:36:28.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9796999999999998e-05 2024-05-15T23:36:28.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581810+0000", 2024-05-15T23:36:28.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4153000000000001e-05 2024-05-15T23:36:28.164 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.164 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.164 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.165 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:187268 7.8 7:15214b14:::benchmark_data_smithi090_143373_object41097:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-05-15T23:36:28.165 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:24.382433+0000", 2024-05-15T23:36:28.165 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 303.75861940800002, 2024-05-15T23:36:28.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.199442491, 2024-05-15T23:36:28.165 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.165 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.165 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.165 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.165 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.165 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.165 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 187268 2024-05-15T23:36:28.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.165 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.382433+0000", 2024-05-15T23:36:28.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.382433+0000", 2024-05-15T23:36:28.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.382435+0000", 2024-05-15T23:36:28.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1019999999999999e-06 2024-05-15T23:36:28.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.382442+0000", 2024-05-15T23:36:28.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2969999999999998e-06 2024-05-15T23:36:28.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.382455+0000", 2024-05-15T23:36:28.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3443e-05 2024-05-15T23:36:28.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.382464+0000", 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0159999999999997e-06 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.389315+0000", 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068508670000000001 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.389433+0000", 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011765 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.389547+0000", 2024-05-15T23:36:28.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114188 2024-05-15T23:36:28.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394618+0000", 2024-05-15T23:36:28.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050712370000000001 2024-05-15T23:36:28.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581820+0000", 2024-05-15T23:36:28.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.187201271 2024-05-15T23:36:28.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581847+0000", 2024-05-15T23:36:28.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7373e-05 2024-05-15T23:36:28.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581875+0000", 2024-05-15T23:36:28.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8047000000000001e-05 2024-05-15T23:36:28.170 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.170 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.170 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.170 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:187277 7.8 7:1f6a15c7:::benchmark_data_smithi090_143373_object41106:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-05-15T23:36:28.170 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:24.390330+0000", 2024-05-15T23:36:28.170 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 303.75072153399998, 2024-05-15T23:36:28.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19160165600000001, 2024-05-15T23:36:28.170 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 187277 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.390330+0000", 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.390330+0000", 2024-05-15T23:36:28.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.390333+0000", 2024-05-15T23:36:28.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1380000000000001e-06 2024-05-15T23:36:28.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.390339+0000", 2024-05-15T23:36:28.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3949999999999998e-06 2024-05-15T23:36:28.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.390351+0000", 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2028000000000001e-05 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.390359+0000", 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8320000000000006e-06 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394646+0000", 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042872860000000004 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394773+0000", 2024-05-15T23:36:28.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012736400000000001 2024-05-15T23:36:28.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394896+0000", 2024-05-15T23:36:28.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000122791 2024-05-15T23:36:28.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.399676+0000", 2024-05-15T23:36:28.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047800589999999997 2024-05-15T23:36:28.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581882+0000", 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18220559 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581908+0000", 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5604000000000001e-05 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581932+0000", 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4569000000000001e-05 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:187279 7.8 7:174bce0b:::benchmark_data_smithi090_143373_object41108:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:24.394439+0000", 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 303.74661282300002, 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18754549300000001, 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 187279 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394439+0000", 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394439+0000", 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394442+0000", 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1209999999999998e-06 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394449+0000", 2024-05-15T23:36:28.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5259999999999999e-06 2024-05-15T23:36:28.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394458+0000", 2024-05-15T23:36:28.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4849999999999998e-06 2024-05-15T23:36:28.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.394466+0000", 2024-05-15T23:36:28.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4580000000000004e-06 2024-05-15T23:36:28.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.399705+0000", 2024-05-15T23:36:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0052392350000000001 2024-05-15T23:36:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.399810+0000", 2024-05-15T23:36:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105234 2024-05-15T23:36:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.399916+0000", 2024-05-15T23:36:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000106261 2024-05-15T23:36:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.549470+0000", 2024-05-15T23:36:28.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14955318000000001 2024-05-15T23:36:28.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581939+0000", 2024-05-15T23:36:28.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032468887000000002 2024-05-15T23:36:28.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581961+0000", 2024-05-15T23:36:28.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2056999999999999e-05 2024-05-15T23:36:28.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:24.581985+0000", 2024-05-15T23:36:28.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4049e-05 2024-05-15T23:36:28.181 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.181 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.181 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.181 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:188105 7.8 7:10ce08d9:::benchmark_data_smithi090_143373_object41934:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-05-15T23:36:28.181 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:25.700383+0000", 2024-05-15T23:36:28.181 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 302.44066938600002, 2024-05-15T23:36:28.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19831671200000001, 2024-05-15T23:36:28.182 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.182 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.182 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.182 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.182 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.182 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.182 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 188105 2024-05-15T23:36:28.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.182 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.700383+0000", 2024-05-15T23:36:28.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.700383+0000", 2024-05-15T23:36:28.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.700386+0000", 2024-05-15T23:36:28.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3699999999999999e-06 2024-05-15T23:36:28.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.700409+0000", 2024-05-15T23:36:28.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3142e-05 2024-05-15T23:36:28.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.700424+0000", 2024-05-15T23:36:28.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5024e-05 2024-05-15T23:36:28.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.700434+0000", 2024-05-15T23:36:28.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0176999999999999e-05 2024-05-15T23:36:28.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.708159+0000", 2024-05-15T23:36:28.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0077245250000000003 2024-05-15T23:36:28.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.708258+0000", 2024-05-15T23:36:28.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8877999999999998e-05 2024-05-15T23:36:28.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.708336+0000", 2024-05-15T23:36:28.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8468999999999997e-05 2024-05-15T23:36:28.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.882714+0000", 2024-05-15T23:36:28.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.17437766099999999 2024-05-15T23:36:28.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.898625+0000", 2024-05-15T23:36:28.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015911262999999998 2024-05-15T23:36:28.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.898658+0000", 2024-05-15T23:36:28.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2483000000000001e-05 2024-05-15T23:36:28.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.898699+0000", 2024-05-15T23:36:28.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.172e-05 2024-05-15T23:36:28.187 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.187 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.187 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.187 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:188117 7.8 7:10ee05b8:::benchmark_data_smithi090_143373_object41946:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-05-15T23:36:28.188 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:25.713894+0000", 2024-05-15T23:36:28.188 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 302.42715826, 2024-05-15T23:36:28.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.188971004, 2024-05-15T23:36:28.188 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.188 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.188 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.188 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.188 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.188 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.188 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 188117 2024-05-15T23:36:28.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.189 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.713894+0000", 2024-05-15T23:36:28.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.713894+0000", 2024-05-15T23:36:28.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.713896+0000", 2024-05-15T23:36:28.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.046e-06 2024-05-15T23:36:28.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.713901+0000", 2024-05-15T23:36:28.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3299999999999998e-06 2024-05-15T23:36:28.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.713915+0000", 2024-05-15T23:36:28.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3570000000000001e-05 2024-05-15T23:36:28.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.713924+0000", 2024-05-15T23:36:28.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0310000000000004e-06 2024-05-15T23:36:28.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.883514+0000", 2024-05-15T23:36:28.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16959009 2024-05-15T23:36:28.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.883636+0000", 2024-05-15T23:36:28.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121903 2024-05-15T23:36:28.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.883743+0000", 2024-05-15T23:36:28.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107105 2024-05-15T23:36:28.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.898512+0000", 2024-05-15T23:36:28.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014769521000000001 2024-05-15T23:36:28.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.902769+0000", 2024-05-15T23:36:28.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042569549999999998 2024-05-15T23:36:28.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.902815+0000", 2024-05-15T23:36:28.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6051999999999999e-05 2024-05-15T23:36:28.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:25.902865+0000", 2024-05-15T23:36:28.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9400999999999997e-05 2024-05-15T23:36:28.194 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.194 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.194 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.195 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190306 7.8 7:104adfd2:::benchmark_data_smithi090_143373_object44135:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:36:28.195 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.088517+0000", 2024-05-15T23:36:28.195 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 299.052535003, 2024-05-15T23:36:28.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.25864289899999998, 2024-05-15T23:36:28.195 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.195 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.195 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.195 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.195 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.196 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.196 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190306 2024-05-15T23:36:28.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.196 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.088517+0000", 2024-05-15T23:36:28.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.088517+0000", 2024-05-15T23:36:28.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.088518+0000", 2024-05-15T23:36:28.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9999999999999997e-07 2024-05-15T23:36:28.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.088520+0000", 2024-05-15T23:36:28.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6989999999999999e-06 2024-05-15T23:36:28.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.088526+0000", 2024-05-15T23:36:28.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1999999999999999e-06 2024-05-15T23:36:28.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.088532+0000", 2024-05-15T23:36:28.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3789999999999997e-06 2024-05-15T23:36:28.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.180699+0000", 2024-05-15T23:36:28.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.092167522000000002 2024-05-15T23:36:28.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.180835+0000", 2024-05-15T23:36:28.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013607499999999999 2024-05-15T23:36:28.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.180980+0000", 2024-05-15T23:36:28.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000144726 2024-05-15T23:36:28.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.196243+0000", 2024-05-15T23:36:28.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015262805000000001 2024-05-15T23:36:28.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347041+0000", 2024-05-15T23:36:28.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15079778199999999 2024-05-15T23:36:28.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347097+0000", 2024-05-15T23:36:28.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5767999999999999e-05 2024-05-15T23:36:28.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347160+0000", 2024-05-15T23:36:28.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3343000000000007e-05 2024-05-15T23:36:28.201 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.201 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.201 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.201 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190324 7.8 7:1ecb0c0e:::benchmark_data_smithi090_143373_object44153:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:36:28.201 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.118138+0000", 2024-05-15T23:36:28.201 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 299.02291374999999, 2024-05-15T23:36:28.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.22912729200000001, 2024-05-15T23:36:28.201 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.201 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.201 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.201 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.202 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.202 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.202 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190324 2024-05-15T23:36:28.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.202 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.118138+0000", 2024-05-15T23:36:28.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.118138+0000", 2024-05-15T23:36:28.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.118140+0000", 2024-05-15T23:36:28.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.694e-06 2024-05-15T23:36:28.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.118145+0000", 2024-05-15T23:36:28.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6059999999999996e-06 2024-05-15T23:36:28.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.118154+0000", 2024-05-15T23:36:28.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.465e-06 2024-05-15T23:36:28.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.118161+0000", 2024-05-15T23:36:28.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9689999999999997e-06 2024-05-15T23:36:28.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.185795+0000", 2024-05-15T23:36:28.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.067634 2024-05-15T23:36:28.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.185909+0000", 2024-05-15T23:36:28.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011406800000000001 2024-05-15T23:36:28.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.186023+0000", 2024-05-15T23:36:28.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113638 2024-05-15T23:36:28.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.196285+0000", 2024-05-15T23:36:28.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010262011 2024-05-15T23:36:28.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347169+0000", 2024-05-15T23:36:28.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.150883885 2024-05-15T23:36:28.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347215+0000", 2024-05-15T23:36:28.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6517999999999997e-05 2024-05-15T23:36:28.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347266+0000", 2024-05-15T23:36:28.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0438000000000001e-05 2024-05-15T23:36:28.208 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.208 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.208 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.208 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190337 7.8 7:1243b066:::benchmark_data_smithi090_143373_object44166:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:36:28.208 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.141526+0000", 2024-05-15T23:36:28.208 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 298.99952586900002, 2024-05-15T23:36:28.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.205845111, 2024-05-15T23:36:28.208 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.208 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.208 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.208 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.209 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.209 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.209 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190337 2024-05-15T23:36:28.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.209 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.141526+0000", 2024-05-15T23:36:28.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.141526+0000", 2024-05-15T23:36:28.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.141528+0000", 2024-05-15T23:36:28.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1179999999999999e-06 2024-05-15T23:36:28.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.141535+0000", 2024-05-15T23:36:28.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5400000000000001e-06 2024-05-15T23:36:28.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.141548+0000", 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2974e-05 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.141555+0000", 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2490000000000004e-06 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.191514+0000", 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.049959274999999997 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.191614+0000", 2024-05-15T23:36:28.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9795000000000003e-05 2024-05-15T23:36:28.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.191698+0000", 2024-05-15T23:36:28.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3929000000000002e-05 2024-05-15T23:36:28.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.196325+0000", 2024-05-15T23:36:28.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046271910000000001 2024-05-15T23:36:28.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347273+0000", 2024-05-15T23:36:28.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15094758799999999 2024-05-15T23:36:28.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347319+0000", 2024-05-15T23:36:28.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6467000000000001e-05 2024-05-15T23:36:28.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347371+0000", 2024-05-15T23:36:28.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1984999999999998e-05 2024-05-15T23:36:28.213 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.213 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.213 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.214 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190342 7.8 7:1167675d:::benchmark_data_smithi090_143373_object44171:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:36:28.214 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.147801+0000", 2024-05-15T23:36:28.214 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 298.99325053600001, 2024-05-15T23:36:28.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19967811099999999, 2024-05-15T23:36:28.214 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.214 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.214 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.214 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.214 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.214 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.214 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190342 2024-05-15T23:36:28.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.214 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.147801+0000", 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.147801+0000", 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.147804+0000", 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3080000000000002e-06 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.147810+0000", 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4489999999999996e-06 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.147823+0000", 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2935999999999999e-05 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.147831+0000", 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3229999999999998e-06 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.197606+0000", 2024-05-15T23:36:28.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.049774102000000001 2024-05-15T23:36:28.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.197743+0000", 2024-05-15T23:36:28.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013782000000000001 2024-05-15T23:36:28.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.197887+0000", 2024-05-15T23:36:28.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000143936 2024-05-15T23:36:28.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.207989+0000", 2024-05-15T23:36:28.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010101715000000001 2024-05-15T23:36:28.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347379+0000", 2024-05-15T23:36:28.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.139389546 2024-05-15T23:36:28.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347424+0000", 2024-05-15T23:36:28.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5210000000000003e-05 2024-05-15T23:36:28.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.347480+0000", 2024-05-15T23:36:28.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5766000000000001e-05 2024-05-15T23:36:28.219 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.219 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.219 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.220 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190384 7.8 7:1cd23127:::benchmark_data_smithi090_143373_object44213:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:36:28.220 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.246952+0000", 2024-05-15T23:36:28.220 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 298.89409952699998, 2024-05-15T23:36:28.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.29457159700000002, 2024-05-15T23:36:28.220 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.220 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.220 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.220 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.220 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.220 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.220 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190384 2024-05-15T23:36:28.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.220 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.246952+0000", 2024-05-15T23:36:28.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.246952+0000", 2024-05-15T23:36:28.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.246953+0000", 2024-05-15T23:36:28.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8200000000000008e-07 2024-05-15T23:36:28.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.246957+0000", 2024-05-15T23:36:28.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9009999999999996e-06 2024-05-15T23:36:28.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.246967+0000", 2024-05-15T23:36:28.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3170000000000008e-06 2024-05-15T23:36:28.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.246972+0000", 2024-05-15T23:36:28.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.451e-06 2024-05-15T23:36:28.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.337967+0000", 2024-05-15T23:36:28.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.090994624999999996 2024-05-15T23:36:28.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.338085+0000", 2024-05-15T23:36:28.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000118177 2024-05-15T23:36:28.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.338199+0000", 2024-05-15T23:36:28.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113746 2024-05-15T23:36:28.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.357844+0000", 2024-05-15T23:36:28.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019645184999999999 2024-05-15T23:36:28.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541420+0000", 2024-05-15T23:36:28.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18357621800000001 2024-05-15T23:36:28.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541472+0000", 2024-05-15T23:36:28.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1743999999999998e-05 2024-05-15T23:36:28.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541524+0000", 2024-05-15T23:36:28.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2250999999999998e-05 2024-05-15T23:36:28.226 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.226 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.226 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.227 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190413 7.8 7:1fb699e8:::benchmark_data_smithi090_143373_object44242:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:36:28.227 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.298709+0000", 2024-05-15T23:36:28.227 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 298.84234281699997, 2024-05-15T23:36:28.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.242957495, 2024-05-15T23:36:28.227 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.227 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.227 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.227 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.227 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.228 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.228 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190413 2024-05-15T23:36:28.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.228 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.298709+0000", 2024-05-15T23:36:28.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.298709+0000", 2024-05-15T23:36:28.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.298711+0000", 2024-05-15T23:36:28.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2340000000000001e-06 2024-05-15T23:36:28.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.298718+0000", 2024-05-15T23:36:28.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5939999999999999e-06 2024-05-15T23:36:28.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.298729+0000", 2024-05-15T23:36:28.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1188e-05 2024-05-15T23:36:28.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.298737+0000", 2024-05-15T23:36:28.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.089e-06 2024-05-15T23:36:28.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.342556+0000", 2024-05-15T23:36:28.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.043818548999999998 2024-05-15T23:36:28.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.342658+0000", 2024-05-15T23:36:28.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010268399999999999 2024-05-15T23:36:28.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.342765+0000", 2024-05-15T23:36:28.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000106654 2024-05-15T23:36:28.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.357850+0000", 2024-05-15T23:36:28.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015085334000000001 2024-05-15T23:36:28.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541574+0000", 2024-05-15T23:36:28.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18372361000000001 2024-05-15T23:36:28.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541621+0000", 2024-05-15T23:36:28.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7064999999999997e-05 2024-05-15T23:36:28.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541667+0000", 2024-05-15T23:36:28.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5494000000000002e-05 2024-05-15T23:36:28.234 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.234 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.234 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.234 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190443 7.8 7:1c1fbdd5:::benchmark_data_smithi090_143373_object44272:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:36:28.234 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.352457+0000", 2024-05-15T23:36:28.234 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 298.78859447799999, 2024-05-15T23:36:28.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18933589000000001, 2024-05-15T23:36:28.235 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.235 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.235 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.235 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.235 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.235 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.235 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190443 2024-05-15T23:36:28.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.235 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352457+0000", 2024-05-15T23:36:28.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352457+0000", 2024-05-15T23:36:28.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352459+0000", 2024-05-15T23:36:28.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.922e-06 2024-05-15T23:36:28.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352465+0000", 2024-05-15T23:36:28.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7819999999999999e-06 2024-05-15T23:36:28.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352479+0000", 2024-05-15T23:36:28.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3599e-05 2024-05-15T23:36:28.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352487+0000", 2024-05-15T23:36:28.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8830000000000006e-06 2024-05-15T23:36:28.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352537+0000", 2024-05-15T23:36:28.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0608000000000001e-05 2024-05-15T23:36:28.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352671+0000", 2024-05-15T23:36:28.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013372300000000001 2024-05-15T23:36:28.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.352803+0000", 2024-05-15T23:36:28.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000131583 2024-05-15T23:36:28.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.357960+0000", 2024-05-15T23:36:28.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051577619999999998 2024-05-15T23:36:28.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541704+0000", 2024-05-15T23:36:28.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18374388999999999 2024-05-15T23:36:28.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541751+0000", 2024-05-15T23:36:28.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7067000000000002e-05 2024-05-15T23:36:28.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541793+0000", 2024-05-15T23:36:28.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2070999999999998e-05 2024-05-15T23:36:28.240 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.240 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.240 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.240 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:190446 7.8 7:1d0b779c:::benchmark_data_smithi090_143373_object44275:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-05-15T23:36:28.240 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:29.358424+0000", 2024-05-15T23:36:28.240 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 298.78262842399999, 2024-05-15T23:36:28.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.183500423, 2024-05-15T23:36:28.241 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.241 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.241 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.241 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.241 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.241 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.241 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 190446 2024-05-15T23:36:28.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.241 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358424+0000", 2024-05-15T23:36:28.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358424+0000", 2024-05-15T23:36:28.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358426+0000", 2024-05-15T23:36:28.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.289e-06 2024-05-15T23:36:28.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358432+0000", 2024-05-15T23:36:28.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.657e-06 2024-05-15T23:36:28.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358448+0000", 2024-05-15T23:36:28.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5849e-05 2024-05-15T23:36:28.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358459+0000", 2024-05-15T23:36:28.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0256e-05 2024-05-15T23:36:28.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358509+0000", 2024-05-15T23:36:28.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0121999999999999e-05 2024-05-15T23:36:28.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358647+0000", 2024-05-15T23:36:28.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013838000000000001 2024-05-15T23:36:28.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.358770+0000", 2024-05-15T23:36:28.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000122774 2024-05-15T23:36:28.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.363625+0000", 2024-05-15T23:36:28.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048551280000000002 2024-05-15T23:36:28.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541831+0000", 2024-05-15T23:36:28.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.17820591199999999 2024-05-15T23:36:28.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541873+0000", 2024-05-15T23:36:28.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2531000000000003e-05 2024-05-15T23:36:28.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:29.541924+0000", 2024-05-15T23:36:28.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0525000000000002e-05 2024-05-15T23:36:28.246 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.246 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.246 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.246 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:192687 7.8 7:17c8a2a5:::benchmark_data_smithi090_143373_object46516:head [delete] snapc 0=[] ondisk+write+known_if_redirected e602)", 2024-05-15T23:36:28.246 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:32.936204+0000", 2024-05-15T23:36:28.246 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 295.204847745, 2024-05-15T23:36:28.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.26211031600000001, 2024-05-15T23:36:28.246 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.246 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.246 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.246 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.247 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.247 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.247 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 192687 2024-05-15T23:36:28.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.247 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:32.936204+0000", 2024-05-15T23:36:28.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:32.936204+0000", 2024-05-15T23:36:28.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:32.936206+0000", 2024-05-15T23:36:28.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.815e-06 2024-05-15T23:36:28.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:32.936213+0000", 2024-05-15T23:36:28.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9959999999999996e-06 2024-05-15T23:36:28.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:32.936225+0000", 2024-05-15T23:36:28.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2442999999999999e-05 2024-05-15T23:36:28.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:32.936233+0000", 2024-05-15T23:36:28.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4549999999999998e-06 2024-05-15T23:36:28.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:33.022555+0000", 2024-05-15T23:36:28.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.086322265999999995 2024-05-15T23:36:28.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:33.022652+0000", 2024-05-15T23:36:28.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6318999999999997e-05 2024-05-15T23:36:28.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:33.022750+0000", 2024-05-15T23:36:28.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8147999999999996e-05 2024-05-15T23:36:28.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:33.032462+0000", 2024-05-15T23:36:28.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0097126190000000005 2024-05-15T23:36:28.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:33.198199+0000", 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.16573666500000001 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:33.198253+0000", 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3559e-05 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:33.198315+0000", 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2031000000000003e-05 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.251 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.252 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:194612 7.8 7:18f80d38:::benchmark_data_smithi090_143373_object48441:head [delete] snapc 0=[] ondisk+write+known_if_redirected e603)", 2024-05-15T23:36:28.252 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:35.760666+0000", 2024-05-15T23:36:28.252 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 292.38038615699998, 2024-05-15T23:36:28.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.18154204400000001, 2024-05-15T23:36:28.252 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.252 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.252 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.252 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.252 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.252 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.252 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 194612 2024-05-15T23:36:28.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.252 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.760666+0000", 2024-05-15T23:36:28.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.760666+0000", 2024-05-15T23:36:28.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.760668+0000", 2024-05-15T23:36:28.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8560000000000001e-06 2024-05-15T23:36:28.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.254 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.254 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.760674+0000", 2024-05-15T23:36:28.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1750000000000002e-06 2024-05-15T23:36:28.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.254 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.254 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.760689+0000", 2024-05-15T23:36:28.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4677e-05 2024-05-15T23:36:28.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.760698+0000", 2024-05-15T23:36:28.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1190000000000003e-06 2024-05-15T23:36:28.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.787608+0000", 2024-05-15T23:36:28.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026910645 2024-05-15T23:36:28.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.787717+0000", 2024-05-15T23:36:28.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010851 2024-05-15T23:36:28.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.787823+0000", 2024-05-15T23:36:28.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0001066 2024-05-15T23:36:28.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.797974+0000", 2024-05-15T23:36:28.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010150648 2024-05-15T23:36:28.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.942101+0000", 2024-05-15T23:36:28.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.14412665399999999 2024-05-15T23:36:28.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.942150+0000", 2024-05-15T23:36:28.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9190999999999997e-05 2024-05-15T23:36:28.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.942208+0000", 2024-05-15T23:36:28.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7969e-05 2024-05-15T23:36:28.258 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.258 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.258 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.258 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:194746 7.8 7:15d745f4:::benchmark_data_smithi090_143373_object48575:head [delete] snapc 0=[] ondisk+write+known_if_redirected e603)", 2024-05-15T23:36:28.258 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:35.969302+0000", 2024-05-15T23:36:28.258 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 292.17175002099998, 2024-05-15T23:36:28.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.194987882, 2024-05-15T23:36:28.258 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.259 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.259 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.259 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.259 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.259 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.259 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 194746 2024-05-15T23:36:28.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.259 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969302+0000", 2024-05-15T23:36:28.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969302+0000", 2024-05-15T23:36:28.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969303+0000", 2024-05-15T23:36:28.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.33e-06 2024-05-15T23:36:28.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969312+0000", 2024-05-15T23:36:28.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2619999999999998e-06 2024-05-15T23:36:28.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969321+0000", 2024-05-15T23:36:28.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9140000000000003e-06 2024-05-15T23:36:28.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.969336+0000", 2024-05-15T23:36:28.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4968e-05 2024-05-15T23:36:28.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.989969+0000", 2024-05-15T23:36:28.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020632265 2024-05-15T23:36:28.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.990085+0000", 2024-05-15T23:36:28.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000115883 2024-05-15T23:36:28.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.990200+0000", 2024-05-15T23:36:28.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114963 2024-05-15T23:36:28.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.007820+0000", 2024-05-15T23:36:28.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017620286999999998 2024-05-15T23:36:28.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.164163+0000", 2024-05-15T23:36:28.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.15634331700000001 2024-05-15T23:36:28.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.164222+0000", 2024-05-15T23:36:28.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9055000000000003e-05 2024-05-15T23:36:28.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.164290+0000", 2024-05-15T23:36:28.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7638000000000005e-05 2024-05-15T23:36:28.264 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.264 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.264 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.264 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:194755 7.8 7:14a1fd4f:::benchmark_data_smithi090_143373_object48584:head [delete] snapc 0=[] ondisk+write+known_if_redirected e603)", 2024-05-15T23:36:28.264 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:31:35.981439+0000", 2024-05-15T23:36:28.264 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 292.15961297799998, 2024-05-15T23:36:28.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.182957336, 2024-05-15T23:36:28.265 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:36:28.265 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:36:28.265 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:36:28.265 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:36:28.265 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:36:28.265 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:36:28.265 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 194755 2024-05-15T23:36:28.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.265 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:36:28.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:36:28.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981439+0000", 2024-05-15T23:36:28.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:36:28.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981439+0000", 2024-05-15T23:36:28.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:36:28.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:36:28.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981441+0000", 2024-05-15T23:36:28.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3740000000000001e-06 2024-05-15T23:36:28.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:36:28.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981449+0000", 2024-05-15T23:36:28.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5660000000000001e-06 2024-05-15T23:36:28.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:36:28.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981462+0000", 2024-05-15T23:36:28.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2795e-05 2024-05-15T23:36:28.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.268 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:36:28.268 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:35.981468+0000", 2024-05-15T23:36:28.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4799999999999998e-06 2024-05-15T23:36:28.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.268 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:36:28.268 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.002522+0000", 2024-05-15T23:36:28.268 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021053477000000001 2024-05-15T23:36:28.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:36:28.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.002655+0000", 2024-05-15T23:36:28.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013380900000000001 2024-05-15T23:36:28.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7", 2024-05-15T23:36:28.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.002781+0000", 2024-05-15T23:36:28.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000125022 2024-05-15T23:36:28.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:36:28.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.153024+0000", 2024-05-15T23:36:28.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.150243082 2024-05-15T23:36:28.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:36:28.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.164296+0000", 2024-05-15T23:36:28.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011272864 2024-05-15T23:36:28.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:36:28.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.164341+0000", 2024-05-15T23:36:28.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4332999999999997e-05 2024-05-15T23:36:28.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:36:28.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:36:28.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:36:28.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:31:36.164396+0000", 2024-05-15T23:36:28.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5534000000000001e-05 2024-05-15T23:36:28.272 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.272 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.272 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.272 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:36:28.272 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:36:28.272 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:36:28.496 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:36:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2445899570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:36:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:28 smithi090 ceph-mon[44434]: from='client.37181 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:36:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2445899570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:36:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:28 smithi088 ceph-mon[44699]: from='client.37181 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:36:28.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2445899570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:36:28.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:28 smithi084 ceph-mon[39784]: from='client.37181 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:36:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:29 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:36:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:29 smithi090 ceph-mon[44434]: from='client.37181 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:36:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:29 smithi090 ceph-mon[44434]: osdmap e700: 12 total, 12 up, 10 in 2024-05-15T23:36:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:29 smithi090 ceph-mon[44434]: pgmap v1855: 17 pgs: 2 active+recovering, 15 active+clean; 6.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 880 op/s; 12636/224692 objects misplaced (5.624%); 6.2 MiB/s, 99 objects/s recovering 2024-05-15T23:36:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:29 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:36:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:29 smithi088 ceph-mon[44699]: from='client.37181 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:36:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:29 smithi088 ceph-mon[44699]: osdmap e700: 12 total, 12 up, 10 in 2024-05-15T23:36:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:29 smithi088 ceph-mon[44699]: pgmap v1855: 17 pgs: 2 active+recovering, 15 active+clean; 6.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 880 op/s; 12636/224692 objects misplaced (5.624%); 6.2 MiB/s, 99 objects/s recovering 2024-05-15T23:36:29.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:29 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:36:29.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:29 smithi084 ceph-mon[39784]: from='client.37181 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:36:29.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:29 smithi084 ceph-mon[39784]: osdmap e700: 12 total, 12 up, 10 in 2024-05-15T23:36:29.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:29 smithi084 ceph-mon[39784]: pgmap v1855: 17 pgs: 2 active+recovering, 15 active+clean; 6.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 880 op/s; 12636/224692 objects misplaced (5.624%); 6.2 MiB/s, 99 objects/s recovering 2024-05-15T23:36:31.578 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:36:31.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:31.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:36:31.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:31 smithi084 ceph-mon[39784]: pgmap v1856: 17 pgs: 2 active+recovering, 15 active+clean; 6.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 654 op/s; 12385/223638 objects misplaced (5.538%); 8.2 MiB/s, 130 objects/s recovering 2024-05-15T23:36:31.683 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:36:31.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:31 smithi088 ceph-mon[44699]: pgmap v1856: 17 pgs: 2 active+recovering, 15 active+clean; 6.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 654 op/s; 12385/223638 objects misplaced (5.538%); 8.2 MiB/s, 130 objects/s recovering 2024-05-15T23:36:31.713 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:31.739 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:36:31.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:31.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:36:31.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:31 smithi090 ceph-mon[44434]: pgmap v1856: 17 pgs: 2 active+recovering, 15 active+clean; 6.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 654 op/s; 12385/223638 objects misplaced (5.538%); 8.2 MiB/s, 130 objects/s recovering 2024-05-15T23:36:31.847 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:36:31.880 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:31.902 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:36:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:31.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:36:32.007 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:36:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:33.494 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:36:33.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:33.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:36:33.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:36:33.641 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:33.710 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:36:33.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:33.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:36:33.819 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:36:33.856 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:33.880 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.9 to 1 (10000) 2024-05-15T23:36:34.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T23:36:34.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:33 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:36:34.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:33 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:36:34.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:33 smithi090 ceph-mon[44434]: osdmap e701: 12 total, 12 up, 10 in 2024-05-15T23:36:34.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:33 smithi090 ceph-mon[44434]: pgmap v1858: 17 pgs: 2 active+recovering, 15 active+clean; 6.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 970 op/s; 12385/217044 objects misplaced (5.706%); 4.1 MiB/s, 65 objects/s recovering 2024-05-15T23:36:34.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:33 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2433114393' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:36:34.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:33 smithi090 ceph-mon[44434]: from='client.37187 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:36:34.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:33 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:36:34.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:33 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:36:34.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:33 smithi084 ceph-mon[39784]: osdmap e701: 12 total, 12 up, 10 in 2024-05-15T23:36:34.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:33 smithi084 ceph-mon[39784]: pgmap v1858: 17 pgs: 2 active+recovering, 15 active+clean; 6.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 970 op/s; 12385/217044 objects misplaced (5.706%); 4.1 MiB/s, 65 objects/s recovering 2024-05-15T23:36:34.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:33 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2433114393' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:36:34.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:33 smithi084 ceph-mon[39784]: from='client.37187 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:36:34.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:33 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:36:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:33 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:36:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:33 smithi088 ceph-mon[44699]: osdmap e701: 12 total, 12 up, 10 in 2024-05-15T23:36:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:33 smithi088 ceph-mon[44699]: pgmap v1858: 17 pgs: 2 active+recovering, 15 active+clean; 6.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 970 op/s; 12385/217044 objects misplaced (5.706%); 4.1 MiB/s, 65 objects/s recovering 2024-05-15T23:36:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:33 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2433114393' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:36:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:33 smithi088 ceph-mon[44699]: from='client.37187 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:36:34.410 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:36:34.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:34.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:36:34.436 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:36:34.523 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:36:34.567 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:34 smithi090 ceph-mon[44434]: from='client.37187 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:36:35.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:34 smithi090 ceph-mon[44434]: osdmap e702: 12 total, 12 up, 10 in 2024-05-15T23:36:35.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:34 smithi088 ceph-mon[44699]: from='client.37187 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:36:35.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:34 smithi088 ceph-mon[44699]: osdmap e702: 12 total, 12 up, 10 in 2024-05-15T23:36:35.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:34 smithi084 ceph-mon[39784]: from='client.37187 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:36:35.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:34 smithi084 ceph-mon[39784]: osdmap e702: 12 total, 12 up, 10 in 2024-05-15T23:36:36.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:35 smithi090 ceph-mon[44434]: pgmap v1860: 17 pgs: 2 active+recovering, 15 active+clean; 6.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 761 op/s; 11859/215406 objects misplaced (5.505%); 9.7 MiB/s, 154 objects/s recovering 2024-05-15T23:36:36.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:35 smithi088 ceph-mon[44699]: pgmap v1860: 17 pgs: 2 active+recovering, 15 active+clean; 6.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 761 op/s; 11859/215406 objects misplaced (5.505%); 9.7 MiB/s, 154 objects/s recovering 2024-05-15T23:36:36.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:35 smithi084 ceph-mon[39784]: pgmap v1860: 17 pgs: 2 active+recovering, 15 active+clean; 6.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 761 op/s; 11859/215406 objects misplaced (5.505%); 9.7 MiB/s, 154 objects/s recovering 2024-05-15T23:36:36.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 10 1 2024-05-15T23:36:36.626 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:36:36.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:36.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:36:36.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:36:36.790 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:36.819 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:36:36.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:36.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:36:36.831 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:36:36.923 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:36:36.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:36:36.953 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:36.979 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:36:36.999 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:36:37.008 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:36:37.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:37.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:36:37.039 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:36:37.065 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:36:37.089 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:36:37.116 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:36:37.118 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:36:37.142 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:36:37.150 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:37.172 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:36:37.176 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:36:37.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:37.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:36:37.201 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:36:37.225 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:36:37.252 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:36:37.254 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:36:37.254 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:36:37.254 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:36:37.254 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:36:37.254 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:36:37.254 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:36:37.255 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:36:37.255 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:36:37.255 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:36:37.255 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:36:37.255 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:36:37.255 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:36:37.282 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:36:37.313 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:37 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:36:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:37 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:36:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:37 smithi090 ceph-mon[44434]: osdmap e703: 12 total, 12 up, 10 in 2024-05-15T23:36:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:37 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4095731922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:36:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:37 smithi090 ceph-mon[44434]: pgmap v1862: 17 pgs: 2 active+recovering, 15 active+clean; 6.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 947 op/s; 11859/212268 objects misplaced (5.587%); 5.5 MiB/s, 87 objects/s recovering 2024-05-15T23:36:38.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:37 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:36:38.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:37 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:36:38.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:37 smithi088 ceph-mon[44699]: osdmap e703: 12 total, 12 up, 10 in 2024-05-15T23:36:38.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:37 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4095731922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:36:38.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:37 smithi088 ceph-mon[44699]: pgmap v1862: 17 pgs: 2 active+recovering, 15 active+clean; 6.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 947 op/s; 11859/212268 objects misplaced (5.587%); 5.5 MiB/s, 87 objects/s recovering 2024-05-15T23:36:38.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:37 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:36:38.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:37 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:36:38.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:37 smithi084 ceph-mon[39784]: osdmap e703: 12 total, 12 up, 10 in 2024-05-15T23:36:38.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:37 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4095731922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:36:38.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:38.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:37 smithi084 ceph-mon[39784]: pgmap v1862: 17 pgs: 2 active+recovering, 15 active+clean; 6.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 947 op/s; 11859/212268 objects misplaced (5.587%); 5.5 MiB/s, 87 objects/s recovering 2024-05-15T23:36:38.372 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:36:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:38.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:36:38.474 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:36:38.506 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:39.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:39.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:39.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:39.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:39.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:39.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:39.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:39.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:39.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:39.759 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:36:39.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:39.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:36:39.866 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:36:39.907 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:39.939 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:36:39.940 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:36:39.941 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:36:39.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:39.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:36:40.044 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:39 smithi084 ceph-mon[39784]: pgmap v1863: 17 pgs: 2 active+recovering, 15 active+clean; 6.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 831 op/s; 11585/207434 objects misplaced (5.585%); 7.8 MiB/s, 124 objects/s recovering 2024-05-15T23:36:40.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:36:40.077 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:40.107 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:36:40.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:40.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:36:40.211 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:36:40.247 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:39 smithi088 ceph-mon[44699]: pgmap v1863: 17 pgs: 2 active+recovering, 15 active+clean; 6.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 831 op/s; 11585/207434 objects misplaced (5.585%); 7.8 MiB/s, 124 objects/s recovering 2024-05-15T23:36:40.248 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:40.278 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:36:40.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:40.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:36:40.301 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:39 smithi090 ceph-mon[44434]: pgmap v1863: 17 pgs: 2 active+recovering, 15 active+clean; 6.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 831 op/s; 11585/207434 objects misplaced (5.585%); 7.8 MiB/s, 124 objects/s recovering 2024-05-15T23:36:40.384 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:36:40.417 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:40.490 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:36:40.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:40.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:36:40.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:36:40.646 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:40.684 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:36:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:40.695 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:36:40.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:36:40.833 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:40.864 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:36:40.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:40.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:36:40.921 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:40 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:36:40.921 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:40 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:36:40.921 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:40 smithi090 ceph-mon[44434]: osdmap e704: 12 total, 12 up, 10 in 2024-05-15T23:36:40.921 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:40 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:36:40.921 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:40 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:36:40.971 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:36:41.001 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:40 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:36:41.001 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:40 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:36:41.001 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:40 smithi084 ceph-mon[39784]: osdmap e704: 12 total, 12 up, 10 in 2024-05-15T23:36:41.001 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:40 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:36:41.001 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:40 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:36:41.004 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:41.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:40 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:36:41.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:40 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:36:41.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:40 smithi088 ceph-mon[44699]: osdmap e704: 12 total, 12 up, 10 in 2024-05-15T23:36:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:40 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:36:41.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:40 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:36:41.960 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:36:41.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:41.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:36:41.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:36:42.034 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:41 smithi084 ceph-mon[39784]: osdmap e705: 12 total, 12 up, 10 in 2024-05-15T23:36:42.034 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:41 smithi084 ceph-mon[39784]: pgmap v1866: 17 pgs: 2 active+recovering, 15 active+clean; 6.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 750 op/s; 11323/206404 objects misplaced (5.486%); 5.6 MiB/s, 89 objects/s recovering 2024-05-15T23:36:42.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:36:42.106 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:42.187 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:36:42.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:42.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:36:42.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:36:42.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:41 smithi090 ceph-mon[44434]: osdmap e705: 12 total, 12 up, 10 in 2024-05-15T23:36:42.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:41 smithi090 ceph-mon[44434]: pgmap v1866: 17 pgs: 2 active+recovering, 15 active+clean; 6.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 750 op/s; 11323/206404 objects misplaced (5.486%); 5.6 MiB/s, 89 objects/s recovering 2024-05-15T23:36:42.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:41 smithi088 ceph-mon[44699]: osdmap e705: 12 total, 12 up, 10 in 2024-05-15T23:36:42.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:41 smithi088 ceph-mon[44699]: pgmap v1866: 17 pgs: 2 active+recovering, 15 active+clean; 6.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 750 op/s; 11323/206404 objects misplaced (5.486%); 5.6 MiB/s, 89 objects/s recovering 2024-05-15T23:36:42.348 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:42.377 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:36:42.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:42.385 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:36:42.486 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:36:42.513 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:42.538 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:36:42.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:42.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:36:42.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:36:42.674 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:43.012 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:36:43.013 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:36:43.027 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:36:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:43.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:36:43.133 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:36:43.163 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:43.289 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:36:43.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:43.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:36:43.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:36:43.431 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:43.470 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:36:43.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:43.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:36:43.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:36:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:43.607 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:43 smithi090 ceph-mon[44434]: pgmap v1867: 17 pgs: 2 active+recovering, 15 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s; 11323/200020 objects misplaced (5.661%); 5.2 MiB/s, 83 objects/s recovering 2024-05-15T23:36:43.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:43 smithi084 ceph-mon[39784]: pgmap v1867: 17 pgs: 2 active+recovering, 15 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s; 11323/200020 objects misplaced (5.661%); 5.2 MiB/s, 83 objects/s recovering 2024-05-15T23:36:43.634 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:36:43.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:43.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:36:43.741 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:36:43.768 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:43 smithi088 ceph-mon[44699]: pgmap v1867: 17 pgs: 2 active+recovering, 15 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s; 11323/200020 objects misplaced (5.661%); 5.2 MiB/s, 83 objects/s recovering 2024-05-15T23:36:43.771 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:43.796 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:36:43.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:43.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:36:43.901 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:36:43.938 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:45.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:45 smithi084 ceph-mon[39784]: pgmap v1868: 17 pgs: 2 active+recovering, 15 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 866 op/s; 10766/198400 objects misplaced (5.426%); 8.5 MiB/s, 136 objects/s recovering 2024-05-15T23:36:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:45 smithi090 ceph-mon[44434]: pgmap v1868: 17 pgs: 2 active+recovering, 15 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 866 op/s; 10766/198400 objects misplaced (5.426%); 8.5 MiB/s, 136 objects/s recovering 2024-05-15T23:36:45.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:45 smithi088 ceph-mon[44699]: pgmap v1868: 17 pgs: 2 active+recovering, 15 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 866 op/s; 10766/198400 objects misplaced (5.426%); 8.5 MiB/s, 136 objects/s recovering 2024-05-15T23:36:46.097 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:36:46.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:46.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:36:46.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:36:46.238 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:46.339 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:36:46.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:46.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:36:46.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:36:46.487 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:46.517 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:36:46.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:46.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:36:46.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:36:46.688 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:46.724 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:36:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:46.728 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:36:46.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:36:46.866 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:46.897 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:36:46.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:46.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:36:47.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:36:47.039 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:47.240 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:36:47.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:47.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:36:47.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:36:47.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:47 smithi084 ceph-mon[39784]: pgmap v1869: 17 pgs: 2 active+recovering, 15 active+clean; 6.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 756 op/s; 10766/195326 objects misplaced (5.512%); 6.4 MiB/s, 102 objects/s recovering 2024-05-15T23:36:47.402 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:47.425 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:36:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:47.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:36:47.533 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:36:47.562 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:47.587 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:36:47.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:47.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:36:47.691 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:36:47.722 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:47 smithi088 ceph-mon[44699]: pgmap v1869: 17 pgs: 2 active+recovering, 15 active+clean; 6.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 756 op/s; 10766/195326 objects misplaced (5.512%); 6.4 MiB/s, 102 objects/s recovering 2024-05-15T23:36:47.729 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:47.757 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:36:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:47.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:36:47.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:47 smithi090 ceph-mon[44434]: pgmap v1869: 17 pgs: 2 active+recovering, 15 active+clean; 6.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 756 op/s; 10766/195326 objects misplaced (5.512%); 6.4 MiB/s, 102 objects/s recovering 2024-05-15T23:36:47.861 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:36:47.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T23:36:47.884 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:36:47.890 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:36:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:47.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:36:48.020 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:36:48.054 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:48.077 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:36:48.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:48.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:36:48.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:36:48.234 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:48.277 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:36:48.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:48.282 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:36:48.373 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:36:48.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:36:48.384 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.10 to 1 (10000) 2024-05-15T23:36:48.403 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3026512795' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:36:48.404 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2697339186' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:36:48.404 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:48 smithi084 ceph-mon[39784]: from='client.37202 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:36:48.428 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:48.454 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:36:48.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:48.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:36:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3026512795' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:36:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2697339186' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:36:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:48 smithi090 ceph-mon[44434]: from='client.37202 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:36:48.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:36:48.605 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:48.640 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:36:48.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:48.647 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:36:48.748 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:36:48.775 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3026512795' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:36:48.775 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2697339186' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:36:48.775 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:48 smithi088 ceph-mon[44699]: from='client.37202 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:36:48.777 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:48.804 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:36:48.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:48.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:36:48.908 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:36:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:48.973 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:36:48.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:48.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:36:49.077 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:36:49.105 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:49.559 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:36:49.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:49.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:36:49.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:49 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:36:49.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:49 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3026512795' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:36:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:49 smithi084 ceph-mon[39784]: from='client.37202 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:36:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:49 smithi084 ceph-mon[39784]: osdmap e706: 12 total, 12 up, 10 in 2024-05-15T23:36:49.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:49 smithi084 ceph-mon[39784]: pgmap v1871: 17 pgs: 2 active+recovering, 15 active+clean; 5.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 984 op/s; 10498/190504 objects misplaced (5.511%); 6.2 MiB/s, 99 objects/s recovering 2024-05-15T23:36:49.670 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:36:49.700 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:49 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:36:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:49 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3026512795' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:36:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:49 smithi090 ceph-mon[44434]: from='client.37202 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:36:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:49 smithi090 ceph-mon[44434]: osdmap e706: 12 total, 12 up, 10 in 2024-05-15T23:36:49.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:49 smithi090 ceph-mon[44434]: pgmap v1871: 17 pgs: 2 active+recovering, 15 active+clean; 5.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 984 op/s; 10498/190504 objects misplaced (5.511%); 6.2 MiB/s, 99 objects/s recovering 2024-05-15T23:36:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:49 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:36:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:49 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3026512795' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:36:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:49 smithi088 ceph-mon[44699]: from='client.37202 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:36:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:49 smithi088 ceph-mon[44699]: osdmap e706: 12 total, 12 up, 10 in 2024-05-15T23:36:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:49 smithi088 ceph-mon[44699]: pgmap v1871: 17 pgs: 2 active+recovering, 15 active+clean; 5.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 984 op/s; 10498/190504 objects misplaced (5.511%); 6.2 MiB/s, 99 objects/s recovering 2024-05-15T23:36:50.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:50 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:36:50.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:50 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:36:50.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:50 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:36:50.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:50 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:36:50.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:50 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:36:50.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:50 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:36:51.754 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:36:51.758 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:51 smithi084 ceph-mon[39784]: osdmap e707: 12 total, 12 up, 10 in 2024-05-15T23:36:51.758 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:51 smithi084 ceph-mon[39784]: pgmap v1873: 17 pgs: 2 active+recovering, 15 active+clean; 5.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 657 op/s; 10239/189500 objects misplaced (5.403%); 8.5 MiB/s, 135 objects/s recovering 2024-05-15T23:36:51.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:51.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:36:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:51 smithi090 ceph-mon[44434]: osdmap e707: 12 total, 12 up, 10 in 2024-05-15T23:36:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:51 smithi090 ceph-mon[44434]: pgmap v1873: 17 pgs: 2 active+recovering, 15 active+clean; 5.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 657 op/s; 10239/189500 objects misplaced (5.403%); 8.5 MiB/s, 135 objects/s recovering 2024-05-15T23:36:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:51 smithi088 ceph-mon[44699]: osdmap e707: 12 total, 12 up, 10 in 2024-05-15T23:36:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:51 smithi088 ceph-mon[44699]: pgmap v1873: 17 pgs: 2 active+recovering, 15 active+clean; 5.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 657 op/s; 10239/189500 objects misplaced (5.403%); 8.5 MiB/s, 135 objects/s recovering 2024-05-15T23:36:51.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:36:51.919 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:36:51.925 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:51.957 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:36:51.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:51.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:36:52.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:36:52.108 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:52.143 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:36:52.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:52.146 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:36:52.247 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:36:52.283 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:52.725 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:36:52.745 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:36:52.768 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:36:52.796 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:36:52.816 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:36:52.836 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:36:52.861 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:36:52.888 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:36:52.915 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:36:52.939 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:36:52.971 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:36:52.994 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:36:52.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:36:52.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:36:52.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:36:52.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:36:52.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:36:52.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:36:52.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:36:52.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:36:52.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:36:52.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:36:52.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:36:52.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:36:53.235 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:36:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:53.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:36:53.271 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 2 1 2024-05-15T23:36:53.347 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:36:53.379 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:53.410 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:36:53.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:53.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:36:53.518 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:36:53.549 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:53 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2689054697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:36:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:53 smithi084 ceph-mon[39784]: pgmap v1874: 17 pgs: 2 active+recovering, 15 active+clean; 5.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 952 op/s; 10239/183152 objects misplaced (5.590%); 4.1 MiB/s, 65 objects/s recovering 2024-05-15T23:36:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:53 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2689054697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:36:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:53 smithi090 ceph-mon[44434]: pgmap v1874: 17 pgs: 2 active+recovering, 15 active+clean; 5.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 952 op/s; 10239/183152 objects misplaced (5.590%); 4.1 MiB/s, 65 objects/s recovering 2024-05-15T23:36:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:53 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2689054697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:36:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:53 smithi088 ceph-mon[44699]: pgmap v1874: 17 pgs: 2 active+recovering, 15 active+clean; 5.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 952 op/s; 10239/183152 objects misplaced (5.590%); 4.1 MiB/s, 65 objects/s recovering 2024-05-15T23:36:54.007 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:36:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:54.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:36:54.117 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:36:54.151 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:54.181 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:36:54.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:54.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:36:54.286 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:36:54.314 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:54.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:36:54.865 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:36:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:54.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:36:54.976 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:36:55.034 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:55.078 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:36:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:55.084 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:36:55.185 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:36:55.216 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:55.236 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:36:55.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:55.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:36:55.341 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:36:55.372 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:55.489 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:36:55.501 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:36:55.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:55.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:36:55.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:55 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:36:55.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:55 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:36:55.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:55 smithi084 ceph-mon[39784]: osdmap e708: 12 total, 12 up, 10 in 2024-05-15T23:36:55.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:55 smithi084 ceph-mon[39784]: pgmap v1876: 17 pgs: 2 active+recovering, 15 active+clean; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 727 op/s; 9701/181494 objects misplaced (5.345%); 9.6 MiB/s, 153 objects/s recovering 2024-05-15T23:36:55.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:55 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:36:55.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:55 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:36:55.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:55 smithi088 ceph-mon[44699]: osdmap e708: 12 total, 12 up, 10 in 2024-05-15T23:36:55.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:55 smithi088 ceph-mon[44699]: pgmap v1876: 17 pgs: 2 active+recovering, 15 active+clean; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 727 op/s; 9701/181494 objects misplaced (5.345%); 9.6 MiB/s, 153 objects/s recovering 2024-05-15T23:36:55.605 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:36:55.634 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:55.656 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:36:55.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:55.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:36:55.763 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:36:55.797 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:55.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:55 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:36:55.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:55 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:36:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:55 smithi090 ceph-mon[44434]: osdmap e708: 12 total, 12 up, 10 in 2024-05-15T23:36:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:55 smithi090 ceph-mon[44434]: pgmap v1876: 17 pgs: 2 active+recovering, 15 active+clean; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 727 op/s; 9701/181494 objects misplaced (5.345%); 9.6 MiB/s, 153 objects/s recovering 2024-05-15T23:36:55.858 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:36:55.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:55.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:36:55.970 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:36:55.999 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:56.026 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:36:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:56.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:36:56.037 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:36:56.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:36:56.170 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:36:56.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:56.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:36:56.305 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:36:56.342 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:56.369 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:36:56.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:56.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:36:56.473 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:36:56.505 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:56.538 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:36:56.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:56.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:36:56.648 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:36:56.661 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:36:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:57.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:57 smithi084 ceph-mon[39784]: pgmap v1877: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 774 op/s; 9701/178106 objects misplaced (5.447%); 6.2 MiB/s, 99 objects/s recovering 2024-05-15T23:36:57.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:57 smithi090 ceph-mon[44434]: pgmap v1877: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 774 op/s; 9701/178106 objects misplaced (5.447%); 6.2 MiB/s, 99 objects/s recovering 2024-05-15T23:36:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:57 smithi088 ceph-mon[44699]: pgmap v1877: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 774 op/s; 9701/178106 objects misplaced (5.447%); 6.2 MiB/s, 99 objects/s recovering 2024-05-15T23:36:58.043 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:36:58.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:58.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:36:58.149 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:36:58.187 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:58.189 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:36:58.190 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:36:58.190 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:36:58.190 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:36:58.190 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:36:58.212 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:36:58.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:58.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:36:58.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:36:58.361 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:58.421 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:36:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:58.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:36:58.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:36:58.595 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:58.642 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:36:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:58.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:36:58.746 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:36:58.776 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:58.803 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:36:58.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:36:58.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:36:58.908 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:36:58.941 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:36:59.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:36:59 smithi084 ceph-mon[39784]: pgmap v1878: 17 pgs: 2 active+recovering, 15 active+clean; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 961 op/s; 9430/172834 objects misplaced (5.456%); 5.7 MiB/s, 90 objects/s recovering 2024-05-15T23:36:59.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:36:59 smithi090 ceph-mon[44434]: pgmap v1878: 17 pgs: 2 active+recovering, 15 active+clean; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 961 op/s; 9430/172834 objects misplaced (5.456%); 5.7 MiB/s, 90 objects/s recovering 2024-05-15T23:36:59.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:36:59 smithi088 ceph-mon[44699]: pgmap v1878: 17 pgs: 2 active+recovering, 15 active+clean; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 961 op/s; 9430/172834 objects misplaced (5.456%); 5.7 MiB/s, 90 objects/s recovering 2024-05-15T23:37:01.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:01 smithi084 ceph-mon[39784]: pgmap v1879: 17 pgs: 2 active+recovering, 15 active+clean; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 885 op/s; 9173/171796 objects misplaced (5.339%); 6.7 MiB/s, 106 objects/s recovering 2024-05-15T23:37:01.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:01 smithi090 ceph-mon[44434]: pgmap v1879: 17 pgs: 2 active+recovering, 15 active+clean; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 885 op/s; 9173/171796 objects misplaced (5.339%); 6.7 MiB/s, 106 objects/s recovering 2024-05-15T23:37:01.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:01 smithi088 ceph-mon[44699]: pgmap v1879: 17 pgs: 2 active+recovering, 15 active+clean; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 885 op/s; 9173/171796 objects misplaced (5.339%); 6.7 MiB/s, 106 objects/s recovering 2024-05-15T23:37:02.020 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T23:37:02.398 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:37:02.446 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:37:02.869 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:37:02.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:02.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:37:02.975 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:37:03.002 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:03.034 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:37:03.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:03.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:37:03.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:37:03.182 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:03.225 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:37:03.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:03.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:37:03.330 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:37:03.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/777847411' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T23:37:03.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:03 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:37:03.339 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:03 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:37:03.339 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:03 smithi084 ceph-mon[39784]: pgmap v1880: 17 pgs: 2 active+recovering, 15 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 915 op/s; 9173/164844 objects misplaced (5.565%); 6.7 MiB/s, 106 objects/s recovering 2024-05-15T23:37:03.342 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2024-05-15T23:37:03.364 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:03.389 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:37:03.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:03.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:37:03.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:37:03.533 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:03.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/777847411' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T23:37:03.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:03 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:37:03.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:03 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:37:03.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:03 smithi090 ceph-mon[44434]: pgmap v1880: 17 pgs: 2 active+recovering, 15 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 915 op/s; 9173/164844 objects misplaced (5.565%); 6.7 MiB/s, 106 objects/s recovering 2024-05-15T23:37:03.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/777847411' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T23:37:03.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:03 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:37:03.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:03 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:37:03.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:03 smithi088 ceph-mon[44699]: pgmap v1880: 17 pgs: 2 active+recovering, 15 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 915 op/s; 9173/164844 objects misplaced (5.565%); 6.7 MiB/s, 106 objects/s recovering 2024-05-15T23:37:04.314 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 8 1 2024-05-15T23:37:04.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/777847411' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T23:37:04.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:04 smithi084 ceph-mon[39784]: osdmap e709: 12 total, 12 up, 10 in 2024-05-15T23:37:04.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/777847411' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T23:37:04.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:04 smithi090 ceph-mon[44434]: osdmap e709: 12 total, 12 up, 10 in 2024-05-15T23:37:04.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/777847411' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T23:37:04.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:04 smithi088 ceph-mon[44699]: osdmap e709: 12 total, 12 up, 10 in 2024-05-15T23:37:05.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:05 smithi084 ceph-mon[39784]: pgmap v1882: 17 pgs: 2 active+recovering, 15 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 919 op/s; 8599/163104 objects misplaced (5.272%); 6.9 MiB/s, 110 objects/s recovering 2024-05-15T23:37:05.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:05 smithi090 ceph-mon[44434]: pgmap v1882: 17 pgs: 2 active+recovering, 15 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 919 op/s; 8599/163104 objects misplaced (5.272%); 6.9 MiB/s, 110 objects/s recovering 2024-05-15T23:37:05.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:05 smithi088 ceph-mon[44699]: pgmap v1882: 17 pgs: 2 active+recovering, 15 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 919 op/s; 8599/163104 objects misplaced (5.272%); 6.9 MiB/s, 110 objects/s recovering 2024-05-15T23:37:06.888 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:37:06.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:06.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:37:06.919 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:37:07.001 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:37:07.031 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:07.056 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:37:07.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:07.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:37:07.160 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:37:07.199 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:07.223 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:37:07.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:07.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:37:07.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:37:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:07.394 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:37:07.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:07.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:37:07.498 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:37:07.530 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:07.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:07 smithi084 ceph-mon[39784]: pgmap v1883: 17 pgs: 2 active+recovering, 15 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 968 op/s; 8599/158742 objects misplaced (5.417%); 6.9 MiB/s, 110 objects/s recovering 2024-05-15T23:37:07.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:07 smithi090 ceph-mon[44434]: pgmap v1883: 17 pgs: 2 active+recovering, 15 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 968 op/s; 8599/158742 objects misplaced (5.417%); 6.9 MiB/s, 110 objects/s recovering 2024-05-15T23:37:07.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:07 smithi088 ceph-mon[44699]: pgmap v1883: 17 pgs: 2 active+recovering, 15 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 968 op/s; 8599/158742 objects misplaced (5.417%); 6.9 MiB/s, 110 objects/s recovering 2024-05-15T23:37:08.974 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:37:09.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:09 smithi084 ceph-mon[39784]: pgmap v1884: 17 pgs: 2 active+recovering, 15 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 905 op/s; 8325/154728 objects misplaced (5.380%); 6.9 MiB/s, 110 objects/s recovering 2024-05-15T23:37:09.796 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:37:09.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:09 smithi090 ceph-mon[44434]: pgmap v1884: 17 pgs: 2 active+recovering, 15 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 905 op/s; 8325/154728 objects misplaced (5.380%); 6.9 MiB/s, 110 objects/s recovering 2024-05-15T23:37:09.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:09 smithi088 ceph-mon[44699]: pgmap v1884: 17 pgs: 2 active+recovering, 15 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 905 op/s; 8325/154728 objects misplaced (5.380%); 6.9 MiB/s, 110 objects/s recovering 2024-05-15T23:37:09.820 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:37:09.855 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:37:09.881 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:37:09.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:37:09.934 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:37:09.960 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:37:09.983 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:37:10.015 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:37:10.038 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:37:10.062 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:37:10.097 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:37:10.099 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:37:10.099 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:37:10.099 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:37:10.099 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:37:10.099 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:37:10.099 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:37:10.099 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:37:10.100 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:37:10.100 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:37:10.100 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:37:10.100 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:37:10.100 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:37:10.486 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:37:10.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:10 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:37:10.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:10 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:37:10.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:10 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3821360505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:37:10.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:10.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:37:10.590 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:37:10.616 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:10 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:37:10.617 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:10 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:37:10.617 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:10 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3821360505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:37:10.617 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.617 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.617 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.617 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.617 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.617 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.617 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.618 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:10.644 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:37:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:10.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:37:10.749 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:37:10.778 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:10 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:37:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:10 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:37:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:10 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3821360505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:37:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:10.806 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:37:10.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:10.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:37:10.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:10.957 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:10.999 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:11.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:11.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:11.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:37:11.149 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:11.412 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:37:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:11 smithi090 ceph-mon[44434]: osdmap e710: 12 total, 12 up, 10 in 2024-05-15T23:37:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:11 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2057214288' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:37:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:11 smithi090 ceph-mon[44434]: from='client.37226 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:37:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:11 smithi090 ceph-mon[44434]: pgmap v1886: 17 pgs: 2 active+recovering, 15 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 705 op/s; 8039/153554 objects misplaced (5.235%); 8.9 MiB/s, 141 objects/s recovering 2024-05-15T23:37:11.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:11 smithi088 ceph-mon[44699]: osdmap e710: 12 total, 12 up, 10 in 2024-05-15T23:37:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:11 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2057214288' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:37:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:11 smithi088 ceph-mon[44699]: from='client.37226 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:37:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:11 smithi088 ceph-mon[44699]: pgmap v1886: 17 pgs: 2 active+recovering, 15 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 705 op/s; 8039/153554 objects misplaced (5.235%); 8.9 MiB/s, 141 objects/s recovering 2024-05-15T23:37:11.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:11 smithi084 ceph-mon[39784]: osdmap e710: 12 total, 12 up, 10 in 2024-05-15T23:37:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:11 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2057214288' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:37:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:11 smithi084 ceph-mon[39784]: from='client.37226 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:37:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:11 smithi084 ceph-mon[39784]: pgmap v1886: 17 pgs: 2 active+recovering, 15 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 705 op/s; 8039/153554 objects misplaced (5.235%); 8.9 MiB/s, 141 objects/s recovering 2024-05-15T23:37:12.588 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:37:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:12.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:37:12.693 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:37:12.725 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:12 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:37:12.725 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:12 smithi084 ceph-mon[39784]: from='client.37226 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:37:12.725 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:12 smithi084 ceph-mon[39784]: osdmap e711: 12 total, 12 up, 10 in 2024-05-15T23:37:12.725 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:37:12.725 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:37:12.726 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:37:12.726 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:37:12.726 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:37:12.726 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:37:12.726 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:37:12.726 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:37:12.726 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:37:12.726 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:12 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:37:12.726 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:12 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:37:12.726 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:12 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:37:12.728 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:12 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:37:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:12 smithi090 ceph-mon[44434]: from='client.37226 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:37:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:12 smithi090 ceph-mon[44434]: osdmap e711: 12 total, 12 up, 10 in 2024-05-15T23:37:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:37:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:37:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:37:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:37:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:37:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:37:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:37:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:37:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:37:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:12 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:37:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:12 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:37:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:12 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:37:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:12 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:37:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:12 smithi088 ceph-mon[44699]: from='client.37226 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:37:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:12 smithi088 ceph-mon[44699]: osdmap e711: 12 total, 12 up, 10 in 2024-05-15T23:37:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:37:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:37:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:37:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:37:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:37:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:37:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:37:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:37:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:37:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:12 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:37:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:12 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:37:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:12 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:37:12.983 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:37:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:12.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:37:13.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:13.150 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:13.175 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:13.178 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:13.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:37:13.309 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:13.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:13 smithi084 ceph-mon[39784]: 8.c scrub starts 2024-05-15T23:37:13.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:13 smithi084 ceph-mon[39784]: 8.1 scrub starts 2024-05-15T23:37:13.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:13 smithi084 ceph-mon[39784]: 8.c scrub ok 2024-05-15T23:37:13.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:13 smithi084 ceph-mon[39784]: 8.d scrub starts 2024-05-15T23:37:13.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:13 smithi084 ceph-mon[39784]: pgmap v1888: 17 pgs: 2 active+recovering, 15 active+clean; 4.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s; 8039/146528 objects misplaced (5.486%); 4.4 MiB/s, 69 objects/s recovering 2024-05-15T23:37:13.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:13 smithi084 ceph-mon[39784]: osdmap e712: 12 total, 12 up, 10 in 2024-05-15T23:37:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:13 smithi090 ceph-mon[44434]: 8.c scrub starts 2024-05-15T23:37:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:13 smithi090 ceph-mon[44434]: 8.1 scrub starts 2024-05-15T23:37:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:13 smithi090 ceph-mon[44434]: 8.c scrub ok 2024-05-15T23:37:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:13 smithi090 ceph-mon[44434]: 8.d scrub starts 2024-05-15T23:37:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:13 smithi090 ceph-mon[44434]: pgmap v1888: 17 pgs: 2 active+recovering, 15 active+clean; 4.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s; 8039/146528 objects misplaced (5.486%); 4.4 MiB/s, 69 objects/s recovering 2024-05-15T23:37:13.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:13 smithi090 ceph-mon[44434]: osdmap e712: 12 total, 12 up, 10 in 2024-05-15T23:37:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:13 smithi088 ceph-mon[44699]: 8.c scrub starts 2024-05-15T23:37:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:13 smithi088 ceph-mon[44699]: 8.1 scrub starts 2024-05-15T23:37:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:13 smithi088 ceph-mon[44699]: 8.c scrub ok 2024-05-15T23:37:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:13 smithi088 ceph-mon[44699]: 8.d scrub starts 2024-05-15T23:37:13.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:13 smithi088 ceph-mon[44699]: pgmap v1888: 17 pgs: 2 active+recovering, 15 active+clean; 4.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s; 8039/146528 objects misplaced (5.486%); 4.4 MiB/s, 69 objects/s recovering 2024-05-15T23:37:13.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:13 smithi088 ceph-mon[44699]: osdmap e712: 12 total, 12 up, 10 in 2024-05-15T23:37:14.590 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:37:14.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:14.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:37:14.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:14 smithi084 ceph-mon[39784]: 8.1 scrub ok 2024-05-15T23:37:14.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:14 smithi084 ceph-mon[39784]: 8.d scrub ok 2024-05-15T23:37:14.694 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:37:14.720 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:14 smithi088 ceph-mon[44699]: 8.1 scrub ok 2024-05-15T23:37:14.720 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:14 smithi088 ceph-mon[44699]: 8.d scrub ok 2024-05-15T23:37:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:14.749 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:37:14.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:14.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:37:14.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:14 smithi090 ceph-mon[44434]: 8.1 scrub ok 2024-05-15T23:37:14.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:14 smithi090 ceph-mon[44434]: 8.d scrub ok 2024-05-15T23:37:14.856 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:37:14.887 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:14.911 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:37:14.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:14.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:37:15.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:37:15.046 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:15.316 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:37:15.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:15.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:37:15.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:37:15.461 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:15.492 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:37:15.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:15.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:37:15.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:37:15.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:15 smithi084 ceph-mon[39784]: 8.3 scrub starts 2024-05-15T23:37:15.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:15 smithi084 ceph-mon[39784]: pgmap v1890: 17 pgs: 2 active+recovering, 15 active+clean; 4.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 822 op/s; 7490/144856 objects misplaced (5.171%); 8.7 MiB/s, 139 objects/s recovering 2024-05-15T23:37:15.633 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:15.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:15 smithi090 ceph-mon[44434]: 8.3 scrub starts 2024-05-15T23:37:15.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:15 smithi090 ceph-mon[44434]: pgmap v1890: 17 pgs: 2 active+recovering, 15 active+clean; 4.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 822 op/s; 7490/144856 objects misplaced (5.171%); 8.7 MiB/s, 139 objects/s recovering 2024-05-15T23:37:15.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:15 smithi088 ceph-mon[44699]: 8.3 scrub starts 2024-05-15T23:37:15.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:15 smithi088 ceph-mon[44699]: pgmap v1890: 17 pgs: 2 active+recovering, 15 active+clean; 4.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 822 op/s; 7490/144856 objects misplaced (5.171%); 8.7 MiB/s, 139 objects/s recovering 2024-05-15T23:37:16.187 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:37:16.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:16.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:37:16.291 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:37:16.317 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:37:16.317 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:37:16.369 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:16.401 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:37:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:16.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:37:16.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:37:16.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:16 smithi084 ceph-mon[39784]: 8.3 scrub ok 2024-05-15T23:37:16.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:16 smithi084 ceph-mon[39784]: 8.6 scrub starts 2024-05-15T23:37:16.536 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:16.567 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:37:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:16.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:37:16.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:37:16.675 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:37:16.703 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:16 smithi088 ceph-mon[44699]: 8.3 scrub ok 2024-05-15T23:37:16.703 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:16 smithi088 ceph-mon[44699]: 8.6 scrub starts 2024-05-15T23:37:16.705 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:16.727 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:37:16.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:16.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:37:16.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:16 smithi090 ceph-mon[44434]: 8.3 scrub ok 2024-05-15T23:37:16.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:16 smithi090 ceph-mon[44434]: 8.6 scrub starts 2024-05-15T23:37:16.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:16.896 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:16.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:16.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:17.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:37:17.103 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:17.140 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:37:17.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:17.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:37:17.245 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:37:17.280 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:17.334 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:37:17.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:17 smithi084 ceph-mon[39784]: 8.8 scrub starts 2024-05-15T23:37:17.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:17 smithi084 ceph-mon[39784]: 8.6 scrub ok 2024-05-15T23:37:17.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:17 smithi084 ceph-mon[39784]: 8.f scrub starts 2024-05-15T23:37:17.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:17 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:37:17.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:17 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:37:17.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:17 smithi084 ceph-mon[39784]: pgmap v1891: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 987 op/s; 7490/140504 objects misplaced (5.331%); 5.0 MiB/s, 79 objects/s recovering 2024-05-15T23:37:17.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:17 smithi090 ceph-mon[44434]: 8.8 scrub starts 2024-05-15T23:37:17.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:17 smithi090 ceph-mon[44434]: 8.6 scrub ok 2024-05-15T23:37:17.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:17 smithi090 ceph-mon[44434]: 8.f scrub starts 2024-05-15T23:37:17.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:17 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:37:17.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:17 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:37:17.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:17 smithi090 ceph-mon[44434]: pgmap v1891: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 987 op/s; 7490/140504 objects misplaced (5.331%); 5.0 MiB/s, 79 objects/s recovering 2024-05-15T23:37:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:17 smithi088 ceph-mon[44699]: 8.8 scrub starts 2024-05-15T23:37:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:17 smithi088 ceph-mon[44699]: 8.6 scrub ok 2024-05-15T23:37:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:17 smithi088 ceph-mon[44699]: 8.f scrub starts 2024-05-15T23:37:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:17 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:37:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:17 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:37:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:17 smithi088 ceph-mon[44699]: pgmap v1891: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 987 op/s; 7490/140504 objects misplaced (5.331%); 5.0 MiB/s, 79 objects/s recovering 2024-05-15T23:37:18.135 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:37:18.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:18.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:37:18.240 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:37:18.270 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:18.523 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:37:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:18.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:37:18.638 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:37:18.669 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:18.695 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:37:18.698 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:18 smithi088 ceph-mon[44699]: 8.8 scrub ok 2024-05-15T23:37:18.698 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:18 smithi088 ceph-mon[44699]: 8.4 scrub starts 2024-05-15T23:37:18.698 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:18 smithi088 ceph-mon[44699]: 8.f scrub ok 2024-05-15T23:37:18.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:18.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:37:18.800 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:37:18.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:18 smithi090 ceph-mon[44434]: 8.8 scrub ok 2024-05-15T23:37:18.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:18 smithi090 ceph-mon[44434]: 8.4 scrub starts 2024-05-15T23:37:18.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:18 smithi090 ceph-mon[44434]: 8.f scrub ok 2024-05-15T23:37:18.829 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:18 smithi084 ceph-mon[39784]: 8.8 scrub ok 2024-05-15T23:37:18.829 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:18 smithi084 ceph-mon[39784]: 8.4 scrub starts 2024-05-15T23:37:18.829 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:18 smithi084 ceph-mon[39784]: 8.f scrub ok 2024-05-15T23:37:18.831 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:19.309 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:37:19.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:19.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:37:19.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T23:37:19.334 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:37:19.334 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:37:19.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:37:19.505 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:19.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:19 smithi084 ceph-mon[39784]: 8.9 scrub starts 2024-05-15T23:37:19.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:19 smithi084 ceph-mon[39784]: 8.4 scrub ok 2024-05-15T23:37:19.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:19 smithi084 ceph-mon[39784]: 8.9 scrub ok 2024-05-15T23:37:19.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:19 smithi084 ceph-mon[39784]: 8.7 scrub starts 2024-05-15T23:37:19.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:19 smithi084 ceph-mon[39784]: pgmap v1892: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 4.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 7217/136644 objects misplaced (5.282%); 6.4 MiB/s, 102 objects/s recovering 2024-05-15T23:37:19.568 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:37:19.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:19.576 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:37:19.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:19.718 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:19 smithi090 ceph-mon[44434]: 8.9 scrub starts 2024-05-15T23:37:19.718 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:19 smithi090 ceph-mon[44434]: 8.4 scrub ok 2024-05-15T23:37:19.718 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:19 smithi090 ceph-mon[44434]: 8.9 scrub ok 2024-05-15T23:37:19.718 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:19 smithi090 ceph-mon[44434]: 8.7 scrub starts 2024-05-15T23:37:19.718 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:19 smithi090 ceph-mon[44434]: pgmap v1892: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 4.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 7217/136644 objects misplaced (5.282%); 6.4 MiB/s, 102 objects/s recovering 2024-05-15T23:37:19.718 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:19.766 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:19.773 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:19.812 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:37:19.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:19 smithi088 ceph-mon[44699]: 8.9 scrub starts 2024-05-15T23:37:19.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:19 smithi088 ceph-mon[44699]: 8.4 scrub ok 2024-05-15T23:37:19.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:19 smithi088 ceph-mon[44699]: 8.9 scrub ok 2024-05-15T23:37:19.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:19 smithi088 ceph-mon[44699]: 8.7 scrub starts 2024-05-15T23:37:19.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:19 smithi088 ceph-mon[44699]: pgmap v1892: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 4.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 7217/136644 objects misplaced (5.282%); 6.4 MiB/s, 102 objects/s recovering 2024-05-15T23:37:19.874 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:37:19.916 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:20.022 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:37:20.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:20.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:37:20.127 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:37:20.154 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:20.175 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:37:20.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:20.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:37:20.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:20.333 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:21.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:21 smithi090 ceph-mon[44434]: 8.7 scrub ok 2024-05-15T23:37:21.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:21 smithi088 ceph-mon[44699]: 8.7 scrub ok 2024-05-15T23:37:21.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:21 smithi084 ceph-mon[39784]: 8.7 scrub ok 2024-05-15T23:37:22.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:22 smithi090 ceph-mon[44434]: pgmap v1893: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 4.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 813 op/s; 6923/135618 objects misplaced (5.105%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T23:37:22.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:22 smithi088 ceph-mon[44699]: pgmap v1893: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 4.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 813 op/s; 6923/135618 objects misplaced (5.105%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T23:37:22.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:22 smithi084 ceph-mon[39784]: pgmap v1893: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering, 13 active+clean; 4.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 813 op/s; 6923/135618 objects misplaced (5.105%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T23:37:22.637 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:22.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:22.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:22.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:37:22.791 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:22.829 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:37:22.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:22.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:37:22.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:37:22.973 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:23.003 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:37:23.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:23.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:37:23.109 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:37:23.139 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:23.246 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:37:23.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:23.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:37:23.351 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:23.408 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:23.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:23 smithi084 ceph-mon[39784]: 8.5 scrub starts 2024-05-15T23:37:23.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:37:23.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:37:23.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:23 smithi084 ceph-mon[39784]: osdmap e713: 12 total, 12 up, 10 in 2024-05-15T23:37:23.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:23 smithi084 ceph-mon[39784]: 8.5 scrub ok 2024-05-15T23:37:23.541 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:23.543 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:23 smithi090 ceph-mon[44434]: 8.5 scrub starts 2024-05-15T23:37:23.543 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:37:23.543 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:37:23.543 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:23 smithi090 ceph-mon[44434]: osdmap e713: 12 total, 12 up, 10 in 2024-05-15T23:37:23.544 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:23 smithi090 ceph-mon[44434]: 8.5 scrub ok 2024-05-15T23:37:23.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:23.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:23 smithi088 ceph-mon[44699]: 8.5 scrub starts 2024-05-15T23:37:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:37:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:37:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:23 smithi088 ceph-mon[44699]: osdmap e713: 12 total, 12 up, 10 in 2024-05-15T23:37:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:23 smithi088 ceph-mon[44699]: 8.5 scrub ok 2024-05-15T23:37:23.645 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:37:23.676 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:23.702 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:37:23.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:23.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:37:23.808 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:37:23.842 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:23.863 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:37:23.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:23.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:37:23.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:37:24.034 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:24.285 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:24 smithi090 ceph-mon[44434]: pgmap v1895: 17 pgs: 2 active+recovering, 15 active+clean; 3.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 869 op/s; 6923/129212 objects misplaced (5.358%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T23:37:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:37:24.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:24.513 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:37:24.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:24 smithi088 ceph-mon[44699]: pgmap v1895: 17 pgs: 2 active+recovering, 15 active+clean; 3.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 869 op/s; 6923/129212 objects misplaced (5.358%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T23:37:24.614 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:37:24.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:24 smithi084 ceph-mon[39784]: pgmap v1895: 17 pgs: 2 active+recovering, 15 active+clean; 3.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 869 op/s; 6923/129212 objects misplaced (5.358%); 7.0 MiB/s, 112 objects/s recovering 2024-05-15T23:37:24.650 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:24.673 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:37:24.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:24.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:37:24.778 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:37:24.804 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:24.826 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:37:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:24.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:37:24.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:24.998 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:25.053 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:25.057 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:25.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:25.224 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:25.260 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:25.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:25.263 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:25.296 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.8 to 1 (10000) 2024-05-15T23:37:25.363 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:37:25.397 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:25.418 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:37:25.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:25.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:37:25.527 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:37:25.545 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:37:25.545 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:37:25.551 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:25 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3071586472' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:37:25.555 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:25 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3071586472' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:37:25.557 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:25.579 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:37:25.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:25.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:37:25.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:25 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3071586472' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:37:25.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:25.741 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:25.837 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:25.844 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:25.945 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:37:25.981 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:26.007 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:37:26.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:26.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:37:26.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:37:26.151 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:26.301 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:26 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3071586472' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:37:26.301 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:26 smithi088 ceph-mon[44699]: osdmap e714: 12 total, 12 up, 10 in 2024-05-15T23:37:26.301 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:26 smithi088 ceph-mon[44699]: pgmap v1897: 17 pgs: 2 active+recovering, 15 active+clean; 3.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 805 op/s; 6321/127618 objects misplaced (4.953%); 9.1 MiB/s, 146 objects/s recovering 2024-05-15T23:37:26.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:26 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3071586472' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:37:26.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:26 smithi090 ceph-mon[44434]: osdmap e714: 12 total, 12 up, 10 in 2024-05-15T23:37:26.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:26 smithi090 ceph-mon[44434]: pgmap v1897: 17 pgs: 2 active+recovering, 15 active+clean; 3.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 805 op/s; 6321/127618 objects misplaced (4.953%); 9.1 MiB/s, 146 objects/s recovering 2024-05-15T23:37:26.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:26 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3071586472' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:37:26.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:26 smithi084 ceph-mon[39784]: osdmap e714: 12 total, 12 up, 10 in 2024-05-15T23:37:26.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:26 smithi084 ceph-mon[39784]: pgmap v1897: 17 pgs: 2 active+recovering, 15 active+clean; 3.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 805 op/s; 6321/127618 objects misplaced (4.953%); 9.1 MiB/s, 146 objects/s recovering 2024-05-15T23:37:27.020 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-05-15T23:37:27.020 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-05-15T23:37:27.020 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-15T23:37:27.021 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump --format=json 2024-05-15T23:37:27.305 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:27 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/14649614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:37:27.305 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:27.305 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:27.305 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:27.305 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:27.305 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:27 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:27.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:27 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/14649614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:37:27.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:27.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:27.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:27.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:27.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:27 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:27.651 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:37:27.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:27.658 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:37:27.667 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:27 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/14649614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:37:27.667 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:27.667 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:27.667 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:27.667 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:27.667 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:27 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:27.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T23:37:27.759 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:37:27.790 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:28.054 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:37:28.054 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:37:28.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:37:28.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:37:28.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:37:28.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:37:28.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:37:28.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:37:28.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:37:28.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:37:28.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:37:28.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:37:28.055 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:37:28.055 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:37:28.056 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:37:28.056 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:37:28.056 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:37:28.056 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:37:28.056 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:37:28.056 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:37:28.056 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:37:28.056 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:37:28.056 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:37:28.056 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:37:28.203 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:37:28.326 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:37:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:28 smithi084 ceph-mon[39784]: 8.2 scrub starts 2024-05-15T23:37:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:28 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:28 smithi084 ceph-mon[39784]: pgmap v1898: 17 pgs: 2 active+recovering, 15 active+clean; 3.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 827 op/s; 6321/123394 objects misplaced (5.123%); 7.0 MiB/s, 111 objects/s recovering 2024-05-15T23:37:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1964176015' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:37:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:28 smithi084 ceph-mon[39784]: 8.2 scrub ok 2024-05-15T23:37:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:28 smithi090 ceph-mon[44434]: 8.2 scrub starts 2024-05-15T23:37:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:28 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:28 smithi090 ceph-mon[44434]: pgmap v1898: 17 pgs: 2 active+recovering, 15 active+clean; 3.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 827 op/s; 6321/123394 objects misplaced (5.123%); 7.0 MiB/s, 111 objects/s recovering 2024-05-15T23:37:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1964176015' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:37:28.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:28 smithi090 ceph-mon[44434]: 8.2 scrub ok 2024-05-15T23:37:28.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:28 smithi088 ceph-mon[44699]: 8.2 scrub starts 2024-05-15T23:37:28.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:28 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:28.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:28 smithi088 ceph-mon[44699]: pgmap v1898: 17 pgs: 2 active+recovering, 15 active+clean; 3.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 827 op/s; 6321/123394 objects misplaced (5.123%); 7.0 MiB/s, 111 objects/s recovering 2024-05-15T23:37:28.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1964176015' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:37:28.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:28 smithi088 ceph-mon[44699]: 8.2 scrub ok 2024-05-15T23:37:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:29 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:37:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:29 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:37:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:29 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1964176015' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:37:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:29 smithi084 ceph-mon[39784]: osdmap e715: 12 total, 12 up, 10 in 2024-05-15T23:37:29.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:29 smithi084 ceph-mon[39784]: pgmap v1900: 17 pgs: 2 active+recovering, 15 active+clean; 3.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 870 op/s; 6062/119616 objects misplaced (5.068%); 7.6 MiB/s, 122 objects/s recovering 2024-05-15T23:37:29.783 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:37:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:29 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:37:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:29 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:37:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:29 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1964176015' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:37:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:29 smithi090 ceph-mon[44434]: osdmap e715: 12 total, 12 up, 10 in 2024-05-15T23:37:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:29 smithi090 ceph-mon[44434]: pgmap v1900: 17 pgs: 2 active+recovering, 15 active+clean; 3.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 870 op/s; 6062/119616 objects misplaced (5.068%); 7.6 MiB/s, 122 objects/s recovering 2024-05-15T23:37:29.813 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:37:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:29 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:37:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:29 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:37:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:29 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1964176015' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:37:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:29 smithi088 ceph-mon[44699]: osdmap e715: 12 total, 12 up, 10 in 2024-05-15T23:37:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:29 smithi088 ceph-mon[44699]: pgmap v1900: 17 pgs: 2 active+recovering, 15 active+clean; 3.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 870 op/s; 6062/119616 objects misplaced (5.068%); 7.6 MiB/s, 122 objects/s recovering 2024-05-15T23:37:29.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:29.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:37:29.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:29.988 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:31.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:31.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:31.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:37:31.219 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:31.573 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:37:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:31.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:37:31.677 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:37:31.709 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:31 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:37:31.709 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:31 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:37:31.709 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:31 smithi084 ceph-mon[39784]: osdmap e716: 12 total, 12 up, 10 in 2024-05-15T23:37:31.709 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:31 smithi084 ceph-mon[39784]: pgmap v1902: 17 pgs: 2 active+recovering, 15 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 751 op/s; 5753/118582 objects misplaced (4.851%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T23:37:31.710 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:31.773 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:37:31.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:31 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:37:31.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:31 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:37:31.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:31 smithi090 ceph-mon[44434]: osdmap e716: 12 total, 12 up, 10 in 2024-05-15T23:37:31.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:31 smithi090 ceph-mon[44434]: pgmap v1902: 17 pgs: 2 active+recovering, 15 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 751 op/s; 5753/118582 objects misplaced (4.851%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T23:37:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:31 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:37:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:31 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:37:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:31 smithi088 ceph-mon[44699]: osdmap e716: 12 total, 12 up, 10 in 2024-05-15T23:37:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:31 smithi088 ceph-mon[44699]: pgmap v1902: 17 pgs: 2 active+recovering, 15 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 751 op/s; 5753/118582 objects misplaced (4.851%); 5.9 MiB/s, 94 objects/s recovering 2024-05-15T23:37:31.911 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:37:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:31.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:37:32.018 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:37:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:32.074 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:37:32.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:32.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:37:32.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:37:32.213 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:32.490 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:37:33.200 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:37:33.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:33.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:37:33.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:37:33.330 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:33.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:33 smithi090 ceph-mon[44434]: pgmap v1903: 17 pgs: 2 active+recovering, 15 active+clean; 3.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 983 op/s; 5753/111870 objects misplaced (5.143%); 4.4 MiB/s, 70 objects/s recovering 2024-05-15T23:37:33.603 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:37:33.608 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:37:33.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:33 smithi084 ceph-mon[39784]: pgmap v1903: 17 pgs: 2 active+recovering, 15 active+clean; 3.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 983 op/s; 5753/111870 objects misplaced (5.143%); 4.4 MiB/s, 70 objects/s recovering 2024-05-15T23:37:33.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:33 smithi088 ceph-mon[44699]: pgmap v1903: 17 pgs: 2 active+recovering, 15 active+clean; 3.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 983 op/s; 5753/111870 objects misplaced (5.143%); 4.4 MiB/s, 70 objects/s recovering 2024-05-15T23:37:34.048 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:37:34.049 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":716,"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","created":"2024-05-15T22:56:41.212493+0000","modified":"2024-05-15T23:37:30.412912+0000","last_up_change":"2024-05-15T23:26:58.424794+0000","last_in_change":"2024-05-15T23:33:10.315546+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":32,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T22:59:38.603575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.91321688890457153,"average_primary_affinity":0.9100000262260437,"average_primary_affinity_weighted":0.9100000262260437}},{"pool":8,"pool_name":"unique_pool_6","create_time":"2024-05-15T23:34:00.714362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"659","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.7699999809265137,"score_stable":3.7699999809265137,"optimal_score":0.82999998331069946,"raw_score_acting":3.130000114440918,"raw_score_stable":3.130000114440918,"primary_affinity_weighted":0.91321688890457153,"average_primary_affinity":0.9100000262260437,"average_primary_affinity_weighted":0.9100000262260437}}],"osds":[{"osd":0,"uuid":"80159cc8-1b6c-4aa6-a114-966786d7be1e","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":355,"up_from":397,"up_thru":283,"down_at":356,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6801","nonce":3077815271}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6802","nonce":3077815271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6804","nonce":3077815271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6803","nonce":3077815271}]},"public_addr":"172.21.15.84:6801/3077815271","cluster_addr":"172.21.15.84:6802/3077815271","heartbeat_back_addr":"172.21.15.84:6804/3077815271","heartbeat_front_addr":"172.21.15.84:6803/3077815271","state":["exists","up"]},{"osd":1,"uuid":"9c2c76c9-ce0f-4c2d-9d96-c87a10187607","up":1,"in":1,"weight":1,"primary_affinity":0.4137420654296875,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":654,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6805","nonce":2140830229}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6806","nonce":2140830229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6808","nonce":2140830229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6807","nonce":2140830229}]},"public_addr":"172.21.15.84:6805/2140830229","cluster_addr":"172.21.15.84:6806/2140830229","heartbeat_back_addr":"172.21.15.84:6808/2140830229","heartbeat_front_addr":"172.21.15.84:6807/2140830229","state":["exists","up"]},{"osd":2,"uuid":"e8514daa-b5f9-4f2f-a432-53f4a4eb86bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":261,"up_from":291,"up_thru":685,"down_at":262,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6809","nonce":941507493}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6810","nonce":941507493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6812","nonce":941507493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6811","nonce":941507493}]},"public_addr":"172.21.15.84:6809/941507493","cluster_addr":"172.21.15.84:6810/941507493","heartbeat_back_addr":"172.21.15.84:6812/941507493","heartbeat_front_addr":"172.21.15.84:6811/941507493","state":["exists","up"]},{"osd":3,"uuid":"5d9965f9-c0b2-48ce-869c-582990985623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":654,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6813","nonce":2204039828}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6814","nonce":2204039828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6816","nonce":2204039828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6815","nonce":2204039828}]},"public_addr":"172.21.15.84:6813/2204039828","cluster_addr":"172.21.15.84:6814/2204039828","heartbeat_back_addr":"172.21.15.84:6816/2204039828","heartbeat_front_addr":"172.21.15.84:6815/2204039828","state":["exists","up"]},{"osd":4,"uuid":"4bc50523-8480-44d2-8d76-31479e839c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":549,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6800","nonce":2267955292}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":2267955292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":2267955292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":2267955292}]},"public_addr":"172.21.15.88:6800/2267955292","cluster_addr":"172.21.15.88:6801/2267955292","heartbeat_back_addr":"172.21.15.88:6803/2267955292","heartbeat_front_addr":"172.21.15.88:6802/2267955292","state":["exists","up"]},{"osd":5,"uuid":"d4db8d7f-afc7-43dc-81fe-f70c290d8b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":654,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":661041538}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":661041538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":661041538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":661041538}]},"public_addr":"172.21.15.88:6804/661041538","cluster_addr":"172.21.15.88:6805/661041538","heartbeat_back_addr":"172.21.15.88:6807/661041538","heartbeat_front_addr":"172.21.15.88:6806/661041538","state":["exists","up"]},{"osd":6,"uuid":"a2773454-2318-4a5b-8f31-0952b1b07f2f","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":374,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":380856869}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":380856869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":380856869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":380856869}]},"public_addr":"172.21.15.88:6808/380856869","cluster_addr":"172.21.15.88:6809/380856869","heartbeat_back_addr":"172.21.15.88:6811/380856869","heartbeat_front_addr":"172.21.15.88:6810/380856869","state":["exists","up"]},{"osd":7,"uuid":"028ca7b6-3e00-4466-a33f-b726cd60cb34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":549,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":1044709063}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":1044709063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":1044709063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":1044709063}]},"public_addr":"172.21.15.88:6812/1044709063","cluster_addr":"172.21.15.88:6813/1044709063","heartbeat_back_addr":"172.21.15.88:6815/1044709063","heartbeat_front_addr":"172.21.15.88:6814/1044709063","state":["exists","up"]},{"osd":8,"uuid":"02cab286-d487-4a66-982c-3cf28f55e851","up":1,"in":1,"weight":1,"primary_affinity":0.54486083984375,"last_clean_begin":61,"last_clean_end":484,"up_from":520,"up_thru":685,"down_at":485,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6800","nonce":703598286}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6801","nonce":703598286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6803","nonce":703598286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6802","nonce":703598286}]},"public_addr":"172.21.15.90:6800/703598286","cluster_addr":"172.21.15.90:6801/703598286","heartbeat_back_addr":"172.21.15.90:6803/703598286","heartbeat_front_addr":"172.21.15.90:6802/703598286","state":["exists","up"]},{"osd":9,"uuid":"47d38bc7-39ba-4b35-af24-e282354dd613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":654,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6804","nonce":2794571133}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6805","nonce":2794571133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6807","nonce":2794571133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6806","nonce":2794571133}]},"public_addr":"172.21.15.90:6804/2794571133","cluster_addr":"172.21.15.90:6805/2794571133","heartbeat_back_addr":"172.21.15.90:6807/2794571133","heartbeat_front_addr":"172.21.15.90:6806/2794571133","state":["exists","up"]},{"osd":10,"uuid":"d7b2aeae-d012-41eb-9f91-e4c05c649473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":654,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6808","nonce":3735380744}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6809","nonce":3735380744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6811","nonce":3735380744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6810","nonce":3735380744}]},"public_addr":"172.21.15.90:6808/3735380744","cluster_addr":"172.21.15.90:6809/3735380744","heartbeat_back_addr":"172.21.15.90:6811/3735380744","heartbeat_front_addr":"172.21.15.90:6810/3735380744","state":["exists","up"]},{"osd":11,"uuid":"2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":654,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6812","nonce":2265615270}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6813","nonce":2265615270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6815","nonce":2265615270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6814","nonce":2265615270}]},"public_addr":"172.21.15.90:6812/2265615270","cluster_addr":"172.21.15.90:6813/2265615270","heartbeat_back_addr":"172.21.15.90:6815/2265615270","heartbeat_front_addr":"172.21.15.90:6814/2265615270","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-05-15T23:36:39.524711+0000","dead_epoch":355},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:37:30.362201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:37:22.099798+0000","dead_epoch":261},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:36:36.181896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:36:40.231969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:36:15.736480+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-05-15T23:37:09.402088+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:36:32.811730+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:37:12.334572+0000","dead_epoch":484},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:36:53.723780+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:36:49.665108+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:37:03.179831+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[6,3]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/2581408231":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/1136984393":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/2443201864":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1252671888":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1451001371":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/4084814866":"2024-05-16T22:57:38.505263+0000","172.21.15.84:0/4076409285":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/2988977694":"2024-05-16T22:57:38.505263+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T23:37:34.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:34 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1124103153' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:37:34.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:34 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1124103153' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:37:34.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:34 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1124103153' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:37:35.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:35 smithi084 ceph-mon[39784]: 8.0 scrub starts 2024-05-15T23:37:35.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:35 smithi084 ceph-mon[39784]: pgmap v1904: 17 pgs: 2 active+recovering, 15 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 867 op/s; 5127/109516 objects misplaced (4.682%); 9.3 MiB/s, 149 objects/s recovering 2024-05-15T23:37:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:35 smithi090 ceph-mon[44434]: 8.0 scrub starts 2024-05-15T23:37:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:35 smithi090 ceph-mon[44434]: pgmap v1904: 17 pgs: 2 active+recovering, 15 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 867 op/s; 5127/109516 objects misplaced (4.682%); 9.3 MiB/s, 149 objects/s recovering 2024-05-15T23:37:35.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:35 smithi088 ceph-mon[44699]: 8.0 scrub starts 2024-05-15T23:37:35.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:35 smithi088 ceph-mon[44699]: pgmap v1904: 17 pgs: 2 active+recovering, 15 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 867 op/s; 5127/109516 objects misplaced (4.682%); 9.3 MiB/s, 149 objects/s recovering 2024-05-15T23:37:36.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:36 smithi090 ceph-mon[44434]: 8.0 scrub ok 2024-05-15T23:37:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:36 smithi088 ceph-mon[44699]: 8.0 scrub ok 2024-05-15T23:37:36.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:36 smithi084 ceph-mon[39784]: 8.0 scrub ok 2024-05-15T23:37:36.940 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-15T23:37:37.379 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:37:37.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:37.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:37:37.486 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:37:37.531 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:37.784 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:37:37.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:37 smithi084 ceph-mon[39784]: pgmap v1905: 17 pgs: 2 active+recovering, 15 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 847 op/s; 5127/105524 objects misplaced (4.859%); 8.3 MiB/s, 132 objects/s recovering 2024-05-15T23:37:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:37.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:37:37.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:37 smithi090 ceph-mon[44434]: pgmap v1905: 17 pgs: 2 active+recovering, 15 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 847 op/s; 5127/105524 objects misplaced (4.859%); 8.3 MiB/s, 132 objects/s recovering 2024-05-15T23:37:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:37 smithi088 ceph-mon[44699]: pgmap v1905: 17 pgs: 2 active+recovering, 15 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 847 op/s; 5127/105524 objects misplaced (4.859%); 8.3 MiB/s, 132 objects/s recovering 2024-05-15T23:37:37.889 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:37:37.917 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:38.568 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:37:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:38.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:37:38.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:37:38.705 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:38.877 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:37:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:38.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:37:38.984 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:37:39.017 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:39.038 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:37:39.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:39.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:37:39.143 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:37:39.173 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:39.272 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:37:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:39.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:37:39.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:37:39.432 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:39.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:37:39.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:37:39.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:37:39.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:37:39.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:37:39.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:37:39.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:37:39.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:37:39.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:39 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:37:39.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:39 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:37:39.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:39 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:37:39.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:39 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:37:40.204 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:37:40.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:40.207 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:37:40.308 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:37:40.338 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:40.361 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:37:40.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:40.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:37:40.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:37:40.483 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:37:40.534 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:40.535 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:37:40.561 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:37:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:40.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:37:40.582 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:37:40.614 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:37:40.639 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:37:40.666 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:37:40.667 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:37:40.696 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:40 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2155902408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:37:40.697 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:40.699 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:40.702 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:37:40.724 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:37:40.726 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:37:40.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:40.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:37:40.750 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:37:40.783 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:37:40.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:40 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2155902408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:37:40.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:40.808 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:37:40.828 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:37:40.836 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:37:40.837 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:37:40.837 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:37:40.838 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:37:40.838 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:37:40.838 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:37:40.838 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:37:40.838 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:37:40.838 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:37:40.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:37:40.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:37:40.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:37:40.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:37:40.858 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:40 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2155902408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:37:40.858 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:40.859 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:40.882 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:37:40.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:40.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:37:40.986 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:37:41.023 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:41.046 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:37:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:41.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:37:41.150 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:37:41.178 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:41.202 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:37:41.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:41.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:37:41.290 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:37:41.290 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:37:41.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:41.378 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:41.590 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:37:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:41.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:41.605 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:41.606 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.606 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:37:41.606 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:37:41.606 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:37:41.606 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.606 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.606 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.606 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.606 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.606 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.607 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:41 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2328059153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:37:41.607 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:41 smithi090 ceph-mon[44434]: from='client.37256 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:37:41.607 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:41 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.607 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:41 smithi090 ceph-mon[44434]: pgmap v1907: 17 pgs: 2 active+recovering, 15 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 834 op/s; 4502/100414 objects misplaced (4.483%); 7.2 MiB/s, 114 objects/s recovering 2024-05-15T23:37:41.707 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:37:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:41.767 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:37:41.773 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.773 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:37:41.773 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:37:41.773 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:37:41.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:41 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2328059153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:37:41.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:41 smithi088 ceph-mon[44699]: from='client.37256 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:37:41.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:41 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:41 smithi088 ceph-mon[44699]: pgmap v1907: 17 pgs: 2 active+recovering, 15 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 834 op/s; 4502/100414 objects misplaced (4.483%); 7.2 MiB/s, 114 objects/s recovering 2024-05-15T23:37:41.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:41.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:37:41.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:37:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:37:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:37:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:41 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2328059153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:37:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:41 smithi084 ceph-mon[39784]: from='client.37256 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:37:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:41 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:41 smithi084 ceph-mon[39784]: pgmap v1907: 17 pgs: 2 active+recovering, 15 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 834 op/s; 4502/100414 objects misplaced (4.483%); 7.2 MiB/s, 114 objects/s recovering 2024-05-15T23:37:41.876 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:37:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:41.939 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:37:41.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:41.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:37:42.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:37:42.101 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:42.218 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:37:42.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:42.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:37:42.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T23:37:42.328 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:37:42.362 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:42.384 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:37:42.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:42.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:37:42.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:37:42.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:42.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:42.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:42 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:42.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:42 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:37:42.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:42 smithi090 ceph-mon[44434]: from='client.37256 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:37:42.519 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:42 smithi090 ceph-mon[44434]: osdmap e717: 12 total, 12 up, 10 in 2024-05-15T23:37:42.520 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:42 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:37:42.520 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:42 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:37:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:42.617 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:37:42.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:42.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:42.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:42 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:42.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:42 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:37:42.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:42 smithi084 ceph-mon[39784]: from='client.37256 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:37:42.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:42 smithi084 ceph-mon[39784]: osdmap e717: 12 total, 12 up, 10 in 2024-05-15T23:37:42.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:42 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:37:42.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:42 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:37:42.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:42.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:37:42.722 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:37:42.739 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:37:42.753 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:42.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:42.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:42.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:42 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:42.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:42 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:37:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:42 smithi088 ceph-mon[44699]: from='client.37256 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:37:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:42 smithi088 ceph-mon[44699]: osdmap e717: 12 total, 12 up, 10 in 2024-05-15T23:37:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:42 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:37:42.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:42 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:37:43.006 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:37:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:43.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:37:43.112 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:37:43.148 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:43.180 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:37:43.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:43.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:37:43.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:37:43.345 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:43.608 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:43 smithi090 ceph-mon[44434]: osdmap e718: 12 total, 12 up, 10 in 2024-05-15T23:37:43.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:43 smithi084 ceph-mon[39784]: osdmap e718: 12 total, 12 up, 10 in 2024-05-15T23:37:43.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:43 smithi084 ceph-mon[39784]: pgmap v1910: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s; 4502/93440 objects misplaced (4.818%); 4.9 MiB/s, 78 objects/s recovering 2024-05-15T23:37:44.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:43 smithi090 ceph-mon[44434]: pgmap v1910: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s; 4502/93440 objects misplaced (4.818%); 4.9 MiB/s, 78 objects/s recovering 2024-05-15T23:37:44.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:43 smithi088 ceph-mon[44699]: osdmap e718: 12 total, 12 up, 10 in 2024-05-15T23:37:44.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:43 smithi088 ceph-mon[44699]: pgmap v1910: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s; 4502/93440 objects misplaced (4.818%); 4.9 MiB/s, 78 objects/s recovering 2024-05-15T23:37:44.111 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:37:44.451 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:37:44.604 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:37:44.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:44.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:37:44.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:37:44.757 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:45.007 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:37:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:45 smithi090 ceph-mon[44434]: pgmap v1911: 17 pgs: 2 active+recovering, 15 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 892 op/s; 3824/91234 objects misplaced (4.191%); 10 MiB/s, 162 objects/s recovering 2024-05-15T23:37:45.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:45 smithi088 ceph-mon[44699]: pgmap v1911: 17 pgs: 2 active+recovering, 15 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 892 op/s; 3824/91234 objects misplaced (4.191%); 10 MiB/s, 162 objects/s recovering 2024-05-15T23:37:45.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:45 smithi084 ceph-mon[39784]: pgmap v1911: 17 pgs: 2 active+recovering, 15 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 892 op/s; 3824/91234 objects misplaced (4.191%); 10 MiB/s, 162 objects/s recovering 2024-05-15T23:37:46.593 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:37:47.309 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:37:47.334 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:37:47.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:37:47.394 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:37:47.418 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:37:47.445 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:37:47.467 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:37:47.492 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:37:47.518 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:37:47.546 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:37:47.574 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:37:47.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:47 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3424392156' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:37:47.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:47 smithi084 ceph-mon[39784]: pgmap v1912: 17 pgs: 2 active+recovering, 15 active+clean; 2.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 899 op/s; 3824/87108 objects misplaced (4.390%); 8.1 MiB/s, 129 objects/s recovering 2024-05-15T23:37:47.664 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:37:47.664 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:37:47.664 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:37:47.664 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:37:47.664 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:37:47.664 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:37:47.664 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:37:47.664 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:37:47.665 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:37:47.665 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:37:47.665 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:37:47.665 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:37:47.665 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:37:47.748 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:37:47.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:47.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:37:47.751 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:47 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3424392156' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:37:47.751 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:47 smithi090 ceph-mon[44434]: pgmap v1912: 17 pgs: 2 active+recovering, 15 active+clean; 2.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 899 op/s; 3824/87108 objects misplaced (4.390%); 8.1 MiB/s, 129 objects/s recovering 2024-05-15T23:37:47.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:47 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3424392156' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:37:47.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:47 smithi088 ceph-mon[44699]: pgmap v1912: 17 pgs: 2 active+recovering, 15 active+clean; 2.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 899 op/s; 3824/87108 objects misplaced (4.390%); 8.1 MiB/s, 129 objects/s recovering 2024-05-15T23:37:47.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:47.896 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:48.014 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:48.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:48.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:48.125 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:37:48.156 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:48.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:48.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:49.609 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:37:49.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:49.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:49 smithi084 ceph-mon[39784]: pgmap v1913: 17 pgs: 2 active+recovering, 15 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s; 3507/82862 objects misplaced (4.232%); 7.8 MiB/s, 124 objects/s recovering 2024-05-15T23:37:49.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:49.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:37:49.714 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:49.714 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:49 smithi090 ceph-mon[44434]: pgmap v1913: 17 pgs: 2 active+recovering, 15 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s; 3507/82862 objects misplaced (4.232%); 7.8 MiB/s, 124 objects/s recovering 2024-05-15T23:37:49.714 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:37:49.755 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:49.755 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:49 smithi088 ceph-mon[44699]: pgmap v1913: 17 pgs: 2 active+recovering, 15 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s; 3507/82862 objects misplaced (4.232%); 7.8 MiB/s, 124 objects/s recovering 2024-05-15T23:37:49.756 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:49.782 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:37:49.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:49.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:37:49.887 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:37:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:49.939 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:37:49.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:49.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:37:50.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:37:50.079 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:50.106 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:37:50.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:50.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:37:50.212 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:37:50.274 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:50.320 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:37:50.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:50.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:37:50.360 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:37:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:37:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:37:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:37:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:151542 8.b 8:d1f2434c:::benchmark_data_smithi090_157496_object4604:head [delete] snapc 0=[] ondisk+write+known_if_redirected e689)", 2024-05-15T23:37:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:50.138359+0000", 2024-05-15T23:37:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 120.217445352, 2024-05-15T23:37:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29638827200000001, 2024-05-15T23:37:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151542 2024-05-15T23:37:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.138359+0000", 2024-05-15T23:37:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.138359+0000", 2024-05-15T23:37:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.138361+0000", 2024-05-15T23:37:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4360000000000001e-06 2024-05-15T23:37:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.138368+0000", 2024-05-15T23:37:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2180000000000002e-06 2024-05-15T23:37:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.138383+0000", 2024-05-15T23:37:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5133999999999999e-05 2024-05-15T23:37:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.138393+0000", 2024-05-15T23:37:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1579999999999994e-06 2024-05-15T23:37:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.179570+0000", 2024-05-15T23:37:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041177333000000003 2024-05-15T23:37:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.179625+0000", 2024-05-15T23:37:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4666000000000001e-05 2024-05-15T23:37:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.356910+0000", 2024-05-15T23:37:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.177285628 2024-05-15T23:37:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.357002+0000", 2024-05-15T23:37:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1973999999999997e-05 2024-05-15T23:37:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.357122+0000", 2024-05-15T23:37:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011967 2024-05-15T23:37:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.385565+0000", 2024-05-15T23:37:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028443141000000002 2024-05-15T23:37:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.430699+0000", 2024-05-15T23:37:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045134223000000001 2024-05-15T23:37:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.430760+0000", 2024-05-15T23:37:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0705000000000002e-05 2024-05-15T23:37:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.434747+0000", 2024-05-15T23:37:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039869859999999997 2024-05-15T23:37:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:151619 8.b 8:d15d1694:::benchmark_data_smithi090_157496_object4681:head [delete] snapc 0=[] ondisk+write+known_if_redirected e689)", 2024-05-15T23:37:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:50.238992+0000", 2024-05-15T23:37:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 120.116811478, 2024-05-15T23:37:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25909452599999999, 2024-05-15T23:37:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151619 2024-05-15T23:37:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.238992+0000", 2024-05-15T23:37:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.238992+0000", 2024-05-15T23:37:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.238994+0000", 2024-05-15T23:37:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.756e-06 2024-05-15T23:37:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.239000+0000", 2024-05-15T23:37:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1229999999999997e-06 2024-05-15T23:37:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.239011+0000", 2024-05-15T23:37:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0764999999999999e-05 2024-05-15T23:37:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.239019+0000", 2024-05-15T23:37:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7330000000000003e-06 2024-05-15T23:37:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.329727+0000", 2024-05-15T23:37:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090707967 2024-05-15T23:37:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.329792+0000", 2024-05-15T23:37:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5390999999999998e-05 2024-05-15T23:37:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.463313+0000", 2024-05-15T23:37:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.133520635 2024-05-15T23:37:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.463403+0000", 2024-05-15T23:37:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0635000000000002e-05 2024-05-15T23:37:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.463521+0000", 2024-05-15T23:37:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117984 2024-05-15T23:37:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.493488+0000", 2024-05-15T23:37:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029966046999999999 2024-05-15T23:37:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.493543+0000", 2024-05-15T23:37:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5509e-05 2024-05-15T23:37:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.493595+0000", 2024-05-15T23:37:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1498000000000001e-05 2024-05-15T23:37:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.498087+0000", 2024-05-15T23:37:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044924830000000002 2024-05-15T23:37:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPushReply(8.b 689/685 [PushReplyOp(8:d15d1694:::benchmark_data_smithi090_157496_object4681:head)])", 2024-05-15T23:37:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:50.247344+0000", 2024-05-15T23:37:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 120.10845967100001, 2024-05-15T23:37:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.215917252, 2024-05-15T23:37:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-05-15T23:37:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.247344+0000", 2024-05-15T23:37:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.247344+0000", 2024-05-15T23:37:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.247345+0000", 2024-05-15T23:37:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2100000000000002e-07 2024-05-15T23:37:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.247347+0000", 2024-05-15T23:37:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2029999999999999e-06 2024-05-15T23:37:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.247351+0000", 2024-05-15T23:37:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3889999999999998e-06 2024-05-15T23:37:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.247356+0000", 2024-05-15T23:37:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5549999999999996e-06 2024-05-15T23:37:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.455399+0000", 2024-05-15T23:37:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20804351400000001 2024-05-15T23:37:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.463262+0000", 2024-05-15T23:37:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0078621699999999999 2024-05-15T23:37:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:151789 8.b 8:dc0c5050:::benchmark_data_smithi090_157496_object4851:head [delete] snapc 0=[] ondisk+write+known_if_redirected e690)", 2024-05-15T23:37:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:50.478656+0000", 2024-05-15T23:37:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 119.87714814900001, 2024-05-15T23:37:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22747393900000001, 2024-05-15T23:37:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151789 2024-05-15T23:37:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.478656+0000", 2024-05-15T23:37:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.478656+0000", 2024-05-15T23:37:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.478658+0000", 2024-05-15T23:37:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.137e-06 2024-05-15T23:37:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.478663+0000", 2024-05-15T23:37:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1340000000000003e-06 2024-05-15T23:37:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.478675+0000", 2024-05-15T23:37:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1687e-05 2024-05-15T23:37:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.478682+0000", 2024-05-15T23:37:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.832e-06 2024-05-15T23:37:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.599025+0000", 2024-05-15T23:37:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120343856 2024-05-15T23:37:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.599082+0000", 2024-05-15T23:37:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6286999999999998e-05 2024-05-15T23:37:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.625536+0000", 2024-05-15T23:37:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026454086000000002 2024-05-15T23:37:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.625618+0000", 2024-05-15T23:37:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2280000000000005e-05 2024-05-15T23:37:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.625723+0000", 2024-05-15T23:37:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010528599999999999 2024-05-15T23:37:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.651756+0000", 2024-05-15T23:37:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026032204 2024-05-15T23:37:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.702090+0000", 2024-05-15T23:37:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050334062999999998 2024-05-15T23:37:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.702151+0000", 2024-05-15T23:37:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1568000000000004e-05 2024-05-15T23:37:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.706130+0000", 2024-05-15T23:37:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039785189999999998 2024-05-15T23:37:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:151792 8.b 8:dda4993a:::benchmark_data_smithi090_157496_object4854:head [delete] snapc 0=[] ondisk+write+known_if_redirected e690)", 2024-05-15T23:37:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:50.481774+0000", 2024-05-15T23:37:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 119.874030019, 2024-05-15T23:37:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22870373699999999, 2024-05-15T23:37:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151792 2024-05-15T23:37:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.481774+0000", 2024-05-15T23:37:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.481774+0000", 2024-05-15T23:37:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.481775+0000", 2024-05-15T23:37:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5460000000000001e-06 2024-05-15T23:37:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.481781+0000", 2024-05-15T23:37:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1499999999999998e-06 2024-05-15T23:37:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.481790+0000", 2024-05-15T23:37:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3989999999999993e-06 2024-05-15T23:37:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.481796+0000", 2024-05-15T23:37:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-05-15T23:37:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.599121+0000", 2024-05-15T23:37:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11732537 2024-05-15T23:37:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.599758+0000", 2024-05-15T23:37:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063657099999999995 2024-05-15T23:37:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.637884+0000", 2024-05-15T23:37:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038126733000000003 2024-05-15T23:37:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.637977+0000", 2024-05-15T23:37:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2764999999999997e-05 2024-05-15T23:37:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.638104+0000", 2024-05-15T23:37:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012648999999999999 2024-05-15T23:37:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.664887+0000", 2024-05-15T23:37:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026783162999999999 2024-05-15T23:37:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.706219+0000", 2024-05-15T23:37:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041332596999999999 2024-05-15T23:37:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.706279+0000", 2024-05-15T23:37:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9874999999999998e-05 2024-05-15T23:37:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.710478+0000", 2024-05-15T23:37:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041984279999999997 2024-05-15T23:37:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:151795 8.b 8:d8d6d855:::benchmark_data_smithi090_157496_object4857:head [delete] snapc 0=[] ondisk+write+known_if_redirected e690)", 2024-05-15T23:37:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:50.484079+0000", 2024-05-15T23:37:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 119.87172478399999, 2024-05-15T23:37:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23097083299999999, 2024-05-15T23:37:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151795 2024-05-15T23:37:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.484079+0000", 2024-05-15T23:37:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.484079+0000", 2024-05-15T23:37:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.484081+0000", 2024-05-15T23:37:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6339999999999999e-06 2024-05-15T23:37:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.484086+0000", 2024-05-15T23:37:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.091e-06 2024-05-15T23:37:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.484096+0000", 2024-05-15T23:37:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0206e-05 2024-05-15T23:37:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.484104+0000", 2024-05-15T23:37:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6639999999999998e-06 2024-05-15T23:37:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.599799+0000", 2024-05-15T23:37:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11569531500000001 2024-05-15T23:37:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.600377+0000", 2024-05-15T23:37:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000577576 2024-05-15T23:37:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.651787+0000", 2024-05-15T23:37:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.051410697999999998 2024-05-15T23:37:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.651851+0000", 2024-05-15T23:37:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4077000000000003e-05 2024-05-15T23:37:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.651933+0000", 2024-05-15T23:37:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1117000000000003e-05 2024-05-15T23:37:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.675641+0000", 2024-05-15T23:37:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.023708814000000002 2024-05-15T23:37:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.710525+0000", 2024-05-15T23:37:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034883391 2024-05-15T23:37:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.710558+0000", 2024-05-15T23:37:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3049999999999997e-05 2024-05-15T23:37:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.715050+0000", 2024-05-15T23:37:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044922 2024-05-15T23:37:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:151807 8.b 8:d9be9618:::benchmark_data_smithi090_157496_object4869:head [delete] snapc 0=[] ondisk+write+known_if_redirected e690)", 2024-05-15T23:37:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:50.503411+0000", 2024-05-15T23:37:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 119.852392935, 2024-05-15T23:37:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21544803500000001, 2024-05-15T23:37:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151807 2024-05-15T23:37:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.503411+0000", 2024-05-15T23:37:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.503411+0000", 2024-05-15T23:37:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.503414+0000", 2024-05-15T23:37:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.571e-06 2024-05-15T23:37:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.503421+0000", 2024-05-15T23:37:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.041e-06 2024-05-15T23:37:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.503436+0000", 2024-05-15T23:37:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5536000000000002e-05 2024-05-15T23:37:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.503446+0000", 2024-05-15T23:37:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3500000000000003e-06 2024-05-15T23:37:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.600424+0000", 2024-05-15T23:37:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.096978910000000002 2024-05-15T23:37:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.600951+0000", 2024-05-15T23:37:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052681199999999998 2024-05-15T23:37:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.664930+0000", 2024-05-15T23:37:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0639792 2024-05-15T23:37:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.665038+0000", 2024-05-15T23:37:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010759 2024-05-15T23:37:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.665182+0000", 2024-05-15T23:37:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014357700000000001 2024-05-15T23:37:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.686713+0000", 2024-05-15T23:37:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021531840999999999 2024-05-15T23:37:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.715138+0000", 2024-05-15T23:37:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028424207 2024-05-15T23:37:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.715204+0000", 2024-05-15T23:37:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5874e-05 2024-05-15T23:37:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.718859+0000", 2024-05-15T23:37:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036555260000000001 2024-05-15T23:37:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:151982 8.b 8:d7da07d6:::benchmark_data_smithi090_157496_object5044:head [delete] snapc 0=[] ondisk+write+known_if_redirected e690)", 2024-05-15T23:37:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:50.738160+0000", 2024-05-15T23:37:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 119.61764443600001, 2024-05-15T23:37:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22251390700000001, 2024-05-15T23:37:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151982 2024-05-15T23:37:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.738160+0000", 2024-05-15T23:37:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.738160+0000", 2024-05-15T23:37:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.738162+0000", 2024-05-15T23:37:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8430000000000001e-06 2024-05-15T23:37:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.738170+0000", 2024-05-15T23:37:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5719999999999997e-06 2024-05-15T23:37:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.738187+0000", 2024-05-15T23:37:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7408e-05 2024-05-15T23:37:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.738202+0000", 2024-05-15T23:37:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4832e-05 2024-05-15T23:37:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.829485+0000", 2024-05-15T23:37:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.091282471000000004 2024-05-15T23:37:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.829552+0000", 2024-05-15T23:37:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7712999999999994e-05 2024-05-15T23:37:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.857424+0000", 2024-05-15T23:37:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027871967000000001 2024-05-15T23:37:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.857498+0000", 2024-05-15T23:37:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3691999999999998e-05 2024-05-15T23:37:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.857624+0000", 2024-05-15T23:37:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012606100000000001 2024-05-15T23:37:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.883111+0000", 2024-05-15T23:37:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025486992 2024-05-15T23:37:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.955902+0000", 2024-05-15T23:37:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072790925000000006 2024-05-15T23:37:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.955944+0000", 2024-05-15T23:37:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1557999999999998e-05 2024-05-15T23:37:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.960673+0000", 2024-05-15T23:37:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0047298729999999999 2024-05-15T23:37:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:151986 8.b 8:d87bcf68:::benchmark_data_smithi090_157496_object5048:head [delete] snapc 0=[] ondisk+write+known_if_redirected e690)", 2024-05-15T23:37:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:50.744570+0000", 2024-05-15T23:37:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 119.61123435099999, 2024-05-15T23:37:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21973810899999999, 2024-05-15T23:37:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 151986 2024-05-15T23:37:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.744570+0000", 2024-05-15T23:37:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.744570+0000", 2024-05-15T23:37:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.744572+0000", 2024-05-15T23:37:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5189999999999999e-06 2024-05-15T23:37:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.744579+0000", 2024-05-15T23:37:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1269999999999997e-06 2024-05-15T23:37:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.744595+0000", 2024-05-15T23:37:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5849999999999999e-05 2024-05-15T23:37:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.744603+0000", 2024-05-15T23:37:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1510000000000004e-06 2024-05-15T23:37:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.829585+0000", 2024-05-15T23:37:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.084981944000000004 2024-05-15T23:37:50.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.830190+0000", 2024-05-15T23:37:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060442799999999995 2024-05-15T23:37:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.871206+0000", 2024-05-15T23:37:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041016334000000002 2024-05-15T23:37:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.871304+0000", 2024-05-15T23:37:50.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8103000000000006e-05 2024-05-15T23:37:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.871434+0000", 2024-05-15T23:37:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130257 2024-05-15T23:37:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.896636+0000", 2024-05-15T23:37:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025201550999999999 2024-05-15T23:37:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.960766+0000", 2024-05-15T23:37:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.064130002000000005 2024-05-15T23:37:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.960827+0000", 2024-05-15T23:37:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0730000000000003e-05 2024-05-15T23:37:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:50.964308+0000", 2024-05-15T23:37:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034811130000000001 2024-05-15T23:37:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:153950 8.b 8:deb8bd36:::benchmark_data_smithi090_157496_object7012:head [delete] snapc 0=[] ondisk+write+known_if_redirected e690)", 2024-05-15T23:37:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:52.955807+0000", 2024-05-15T23:37:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 117.39999688100001, 2024-05-15T23:37:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.273199317, 2024-05-15T23:37:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 153950 2024-05-15T23:37:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.420 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:52.955807+0000", 2024-05-15T23:37:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:52.955807+0000", 2024-05-15T23:37:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:52.955809+0000", 2024-05-15T23:37:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0140000000000001e-06 2024-05-15T23:37:50.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:52.955814+0000", 2024-05-15T23:37:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4090000000000002e-06 2024-05-15T23:37:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:52.955825+0000", 2024-05-15T23:37:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0682e-05 2024-05-15T23:37:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:52.955835+0000", 2024-05-15T23:37:50.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0018e-05 2024-05-15T23:37:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.045180+0000", 2024-05-15T23:37:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089345103999999995 2024-05-15T23:37:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.045220+0000", 2024-05-15T23:37:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9931000000000001e-05 2024-05-15T23:37:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.180401+0000", 2024-05-15T23:37:50.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.135180577 2024-05-15T23:37:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.180479+0000", 2024-05-15T23:37:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8378000000000001e-05 2024-05-15T23:37:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.180557+0000", 2024-05-15T23:37:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7426e-05 2024-05-15T23:37:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.205527+0000", 2024-05-15T23:37:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024970283999999999 2024-05-15T23:37:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.224519+0000", 2024-05-15T23:37:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018992266000000001 2024-05-15T23:37:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.224576+0000", 2024-05-15T23:37:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7336999999999997e-05 2024-05-15T23:37:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.229006+0000", 2024-05-15T23:37:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004429891 2024-05-15T23:37:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:154458 8.b 8:d825cf3b:::benchmark_data_smithi090_157496_object7520:head [delete] snapc 0=[] ondisk+write+known_if_redirected e691)", 2024-05-15T23:37:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:53.568448+0000", 2024-05-15T23:37:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 116.787355703, 2024-05-15T23:37:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22699090899999999, 2024-05-15T23:37:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.426 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 154458 2024-05-15T23:37:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.568448+0000", 2024-05-15T23:37:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.568448+0000", 2024-05-15T23:37:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.568451+0000", 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4889999999999998e-06 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.568459+0000", 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9249999999999995e-06 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.568472+0000", 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3699e-05 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.568481+0000", 2024-05-15T23:37:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9420000000000001e-06 2024-05-15T23:37:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.579795+0000", 2024-05-15T23:37:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011313456 2024-05-15T23:37:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.579839+0000", 2024-05-15T23:37:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4239000000000001e-05 2024-05-15T23:37:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.739582+0000", 2024-05-15T23:37:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15974308300000001 2024-05-15T23:37:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.739653+0000", 2024-05-15T23:37:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0819000000000001e-05 2024-05-15T23:37:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.739759+0000", 2024-05-15T23:37:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010590399999999999 2024-05-15T23:37:50.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.774692+0000", 2024-05-15T23:37:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034933383999999998 2024-05-15T23:37:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.791227+0000", 2024-05-15T23:37:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016534414000000001 2024-05-15T23:37:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.791283+0000", 2024-05-15T23:37:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6234999999999999e-05 2024-05-15T23:37:50.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.795439+0000", 2024-05-15T23:37:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041563199999999998 2024-05-15T23:37:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:154501 8.b 8:dedcfcce:::benchmark_data_smithi090_157496_object7563:head [delete] snapc 0=[] ondisk+write+known_if_redirected e691)", 2024-05-15T23:37:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:53.610456+0000", 2024-05-15T23:37:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 116.74534774200001, 2024-05-15T23:37:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26700531199999999, 2024-05-15T23:37:50.432 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 154501 2024-05-15T23:37:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.610456+0000", 2024-05-15T23:37:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.610456+0000", 2024-05-15T23:37:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.610457+0000", 2024-05-15T23:37:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1999999999999999e-06 2024-05-15T23:37:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.610462+0000", 2024-05-15T23:37:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4989999999999996e-06 2024-05-15T23:37:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.610471+0000", 2024-05-15T23:37:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2760000000000006e-06 2024-05-15T23:37:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.610476+0000", 2024-05-15T23:37:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0350000000000001e-06 2024-05-15T23:37:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.659063+0000", 2024-05-15T23:37:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.048587000999999998 2024-05-15T23:37:50.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.659099+0000", 2024-05-15T23:37:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6278000000000001e-05 2024-05-15T23:37:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.824796+0000", 2024-05-15T23:37:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16569647000000001 2024-05-15T23:37:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.824882+0000", 2024-05-15T23:37:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5665000000000001e-05 2024-05-15T23:37:50.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.824994+0000", 2024-05-15T23:37:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112018 2024-05-15T23:37:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.860098+0000", 2024-05-15T23:37:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035104344000000003 2024-05-15T23:37:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.872552+0000", 2024-05-15T23:37:50.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012453578 2024-05-15T23:37:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.872616+0000", 2024-05-15T23:37:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4548000000000006e-05 2024-05-15T23:37:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:53.877462+0000", 2024-05-15T23:37:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048453999999999997 2024-05-15T23:37:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:155969 8.b 8:d4f81049:::benchmark_data_smithi090_157496_object9031:head [delete] snapc 0=[] ondisk+write+known_if_redirected e691)", 2024-05-15T23:37:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:35:55.304461+0000", 2024-05-15T23:37:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 115.05134336899999, 2024-05-15T23:37:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25164376300000002, 2024-05-15T23:37:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 155969 2024-05-15T23:37:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:55.304461+0000", 2024-05-15T23:37:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:55.304461+0000", 2024-05-15T23:37:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:55.304463+0000", 2024-05-15T23:37:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7219999999999999e-06 2024-05-15T23:37:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:55.304469+0000", 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8150000000000002e-06 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:55.304484+0000", 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5099000000000001e-05 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:55.304493+0000", 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6049999999999998e-06 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:55.353885+0000", 2024-05-15T23:37:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049392654000000001 2024-05-15T23:37:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:55.353939+0000", 2024-05-15T23:37:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3730000000000002e-05 2024-05-15T23:37:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:55.451535+0000", 2024-05-15T23:37:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097595541999999993 2024-05-15T23:37:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:55.451612+0000", 2024-05-15T23:37:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7142000000000001e-05 2024-05-15T23:37:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:55.451689+0000", 2024-05-15T23:37:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6946000000000005e-05 2024-05-15T23:37:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:55.485459+0000", 2024-05-15T23:37:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033769795999999998 2024-05-15T23:37:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:55.552587+0000", 2024-05-15T23:37:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067128139000000003 2024-05-15T23:37:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:55.552625+0000", 2024-05-15T23:37:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7858999999999998e-05 2024-05-15T23:37:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:35:55.556104+0000", 2024-05-15T23:37:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034797140000000001 2024-05-15T23:37:50.444 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:175664 8.b 8:dc173445:::benchmark_data_smithi090_157496_object28726:head [delete] snapc 0=[] ondisk+write+known_if_redirected e698)", 2024-05-15T23:37:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:36:20.254329+0000", 2024-05-15T23:37:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.101474951, 2024-05-15T23:37:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.222012336, 2024-05-15T23:37:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.445 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 175664 2024-05-15T23:37:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:20.254329+0000", 2024-05-15T23:37:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:20.254329+0000", 2024-05-15T23:37:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:20.254332+0000", 2024-05-15T23:37:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6900000000000001e-06 2024-05-15T23:37:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:20.254338+0000", 2024-05-15T23:37:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6089999999999997e-06 2024-05-15T23:37:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:20.254352+0000", 2024-05-15T23:37:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4047e-05 2024-05-15T23:37:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:20.254360+0000", 2024-05-15T23:37:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7090000000000002e-06 2024-05-15T23:37:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:20.287752+0000", 2024-05-15T23:37:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033391496999999999 2024-05-15T23:37:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:20.287799+0000", 2024-05-15T23:37:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7058999999999998e-05 2024-05-15T23:37:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:20.440344+0000", 2024-05-15T23:37:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15254551999999999 2024-05-15T23:37:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:20.440442+0000", 2024-05-15T23:37:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8361000000000002e-05 2024-05-15T23:37:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:20.440587+0000", 2024-05-15T23:37:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014449499999999999 2024-05-15T23:37:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:20.468309+0000", 2024-05-15T23:37:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027722455 2024-05-15T23:37:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:20.472280+0000", 2024-05-15T23:37:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039709200000000002 2024-05-15T23:37:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:20.472342+0000", 2024-05-15T23:37:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1773000000000006e-05 2024-05-15T23:37:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:20.476341+0000", 2024-05-15T23:37:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003999201 2024-05-15T23:37:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:178934 8.b 8:d4dccb92:::benchmark_data_smithi090_157496_object31996:head [delete] snapc 0=[] ondisk+write+known_if_redirected e699)", 2024-05-15T23:37:50.451 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:36:24.083743+0000", 2024-05-15T23:37:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.272060732, 2024-05-15T23:37:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30037772699999998, 2024-05-15T23:37:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 178934 2024-05-15T23:37:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.083743+0000", 2024-05-15T23:37:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.083743+0000", 2024-05-15T23:37:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.083746+0000", 2024-05-15T23:37:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3029999999999998e-06 2024-05-15T23:37:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.083751+0000", 2024-05-15T23:37:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7570000000000001e-06 2024-05-15T23:37:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.083762+0000", 2024-05-15T23:37:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0905e-05 2024-05-15T23:37:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.083769+0000", 2024-05-15T23:37:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0079999999999996e-06 2024-05-15T23:37:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.154165+0000", 2024-05-15T23:37:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.070395903999999995 2024-05-15T23:37:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.154232+0000", 2024-05-15T23:37:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7274000000000007e-05 2024-05-15T23:37:50.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.343222+0000", 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18898947599999999 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.343314+0000", 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2558000000000004e-05 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.343449+0000", 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134117 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.365499+0000", 2024-05-15T23:37:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02205095 2024-05-15T23:37:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.381024+0000", 2024-05-15T23:37:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015524356 2024-05-15T23:37:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.381072+0000", 2024-05-15T23:37:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8356000000000003e-05 2024-05-15T23:37:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.384121+0000", 2024-05-15T23:37:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030487629999999999 2024-05-15T23:37:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPushReply(8.b 699/685 [PushReplyOp(8:d4dccb92:::benchmark_data_smithi090_157496_object31996:head)])", 2024-05-15T23:37:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:36:24.083777+0000", 2024-05-15T23:37:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.272027019000006, 2024-05-15T23:37:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25938465500000002, 2024-05-15T23:37:50.458 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-05-15T23:37:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.083777+0000", 2024-05-15T23:37:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.083777+0000", 2024-05-15T23:37:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.083777+0000", 2024-05-15T23:37:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.72e-07 2024-05-15T23:37:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.083780+0000", 2024-05-15T23:37:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2670000000000001e-06 2024-05-15T23:37:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.083784+0000", 2024-05-15T23:37:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7119999999999999e-06 2024-05-15T23:37:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.083789+0000", 2024-05-15T23:37:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5909999999999998e-06 2024-05-15T23:37:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.337653+0000", 2024-05-15T23:37:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25386384000000001 2024-05-15T23:37:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.343162+0000", 2024-05-15T23:37:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055088730000000001 2024-05-15T23:37:50.461 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:179021 8.b 8:dba08fd9:::benchmark_data_smithi090_157496_object32083:head [delete] snapc 0=[] ondisk+write+known_if_redirected e699)", 2024-05-15T23:37:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:36:24.220796+0000", 2024-05-15T23:37:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.135007841999993, 2024-05-15T23:37:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22116612599999999, 2024-05-15T23:37:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 179021 2024-05-15T23:37:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.220796+0000", 2024-05-15T23:37:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.220796+0000", 2024-05-15T23:37:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.220797+0000", 2024-05-15T23:37:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0299999999999998e-07 2024-05-15T23:37:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.220801+0000", 2024-05-15T23:37:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7110000000000001e-06 2024-05-15T23:37:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.220806+0000", 2024-05-15T23:37:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6319999999999996e-06 2024-05-15T23:37:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.220812+0000", 2024-05-15T23:37:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.259e-06 2024-05-15T23:37:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.257585+0000", 2024-05-15T23:37:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.03677329 2024-05-15T23:37:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.257650+0000", 2024-05-15T23:37:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4877999999999998e-05 2024-05-15T23:37:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.412658+0000", 2024-05-15T23:37:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.155007858 2024-05-15T23:37:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.412755+0000", 2024-05-15T23:37:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7267000000000002e-05 2024-05-15T23:37:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.412901+0000", 2024-05-15T23:37:50.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014581699999999999 2024-05-15T23:37:50.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.432985+0000", 2024-05-15T23:37:50.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020084557999999999 2024-05-15T23:37:50.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.437972+0000", 2024-05-15T23:37:50.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049871109999999998 2024-05-15T23:37:50.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.438032+0000", 2024-05-15T23:37:50.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9991000000000002e-05 2024-05-15T23:37:50.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:24.441962+0000", 2024-05-15T23:37:50.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039299510000000001 2024-05-15T23:37:50.468 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.468 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.468 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.468 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:181762 8.b 8:dd01bef9:::benchmark_data_smithi090_157496_object34824:head [delete] snapc 0=[] ondisk+write+known_if_redirected e699)", 2024-05-15T23:37:50.468 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:36:27.489218+0000", 2024-05-15T23:37:50.469 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.866585977, 2024-05-15T23:37:50.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23801483300000001, 2024-05-15T23:37:50.469 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.469 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.469 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.469 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.469 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.469 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.469 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 181762 2024-05-15T23:37:50.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.469 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:27.489218+0000", 2024-05-15T23:37:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:27.489218+0000", 2024-05-15T23:37:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:27.489220+0000", 2024-05-15T23:37:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.736e-06 2024-05-15T23:37:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:27.489226+0000", 2024-05-15T23:37:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8459999999999996e-06 2024-05-15T23:37:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:27.489237+0000", 2024-05-15T23:37:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1749999999999999e-05 2024-05-15T23:37:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:27.489245+0000", 2024-05-15T23:37:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5639999999999999e-06 2024-05-15T23:37:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.472 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:37:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:27.499122+0000", 2024-05-15T23:37:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009877594 2024-05-15T23:37:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:27.499157+0000", 2024-05-15T23:37:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4980000000000001e-05 2024-05-15T23:37:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:27.666557+0000", 2024-05-15T23:37:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16739910599999999 2024-05-15T23:37:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:27.666663+0000", 2024-05-15T23:37:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010625500000000001 2024-05-15T23:37:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:27.666818+0000", 2024-05-15T23:37:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155348 2024-05-15T23:37:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:27.691167+0000", 2024-05-15T23:37:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024349193000000002 2024-05-15T23:37:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:27.723189+0000", 2024-05-15T23:37:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032021382000000001 2024-05-15T23:37:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:27.723255+0000", 2024-05-15T23:37:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6138999999999998e-05 2024-05-15T23:37:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:27.727233+0000", 2024-05-15T23:37:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039779400000000001 2024-05-15T23:37:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:187780 8.b 8:d969f854:::benchmark_data_smithi090_157496_object40842:head [delete] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-05-15T23:37:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:36:34.447451+0000", 2024-05-15T23:37:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.908353359000003, 2024-05-15T23:37:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23138165599999999, 2024-05-15T23:37:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 187780 2024-05-15T23:37:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.447451+0000", 2024-05-15T23:37:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.447451+0000", 2024-05-15T23:37:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.447453+0000", 2024-05-15T23:37:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0509999999999999e-06 2024-05-15T23:37:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.447459+0000", 2024-05-15T23:37:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9059999999999999e-06 2024-05-15T23:37:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.447469+0000", 2024-05-15T23:37:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0715e-05 2024-05-15T23:37:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.447477+0000", 2024-05-15T23:37:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7880000000000007e-06 2024-05-15T23:37:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.498824+0000", 2024-05-15T23:37:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.051347275999999997 2024-05-15T23:37:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.498890+0000", 2024-05-15T23:37:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6103000000000001e-05 2024-05-15T23:37:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.658111+0000", 2024-05-15T23:37:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15922008000000001 2024-05-15T23:37:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.658199+0000", 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8385000000000002e-05 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.658332+0000", 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013318100000000001 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.675803+0000", 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017471362000000001 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.676389+0000", 2024-05-15T23:37:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058587500000000002 2024-05-15T23:37:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.676431+0000", 2024-05-15T23:37:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1900999999999997e-05 2024-05-15T23:37:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.678832+0000", 2024-05-15T23:37:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024010329999999999 2024-05-15T23:37:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.17073.0:187848 8.b 8:d83d8822:::benchmark_data_smithi090_157496_object40910:head [delete] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-05-15T23:37:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:36:34.522364+0000", 2024-05-15T23:37:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.833440296000006, 2024-05-15T23:37:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21914118299999999, 2024-05-15T23:37:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:37:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:37:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:37:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.17073", 2024-05-15T23:37:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 187848 2024-05-15T23:37:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:37:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:37:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.522364+0000", 2024-05-15T23:37:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:37:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.522364+0000", 2024-05-15T23:37:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:37:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:37:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.522366+0000", 2024-05-15T23:37:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.311e-06 2024-05-15T23:37:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:37:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.522371+0000", 2024-05-15T23:37:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3669999999999996e-06 2024-05-15T23:37:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:37:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.522381+0000", 2024-05-15T23:37:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005e-05 2024-05-15T23:37:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:37:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.522389+0000", 2024-05-15T23:37:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3989999999999998e-06 2024-05-15T23:37:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.571040+0000", 2024-05-15T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.048650769000000003 2024-05-15T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.571096+0000", 2024-05-15T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6446999999999997e-05 2024-05-15T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.722280+0000", 2024-05-15T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15118393299999999 2024-05-15T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:37:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.722358+0000", 2024-05-15T23:37:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8368000000000006e-05 2024-05-15T23:37:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-05-15T23:37:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.722474+0000", 2024-05-15T23:37:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115314 2024-05-15T23:37:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:37:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.738522+0000", 2024-05-15T23:37:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016048539000000001 2024-05-15T23:37:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.738570+0000", 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8216000000000001e-05 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.738626+0000", 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5702000000000003e-05 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:36:34.741505+0000", 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028787679999999999 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:37:50.490 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:37:50.490 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:37:50.505 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:51.078 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:37:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:51.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:37:51.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:37:51.243 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:51.249 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T23:37:51.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:51 smithi090 ceph-mon[44434]: pgmap v1914: 17 pgs: 2 active+recovering, 15 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 872 op/s; 3078/81658 objects misplaced (3.769%); 9.1 MiB/s, 146 objects/s recovering 2024-05-15T23:37:51.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:51 smithi084 ceph-mon[39784]: pgmap v1914: 17 pgs: 2 active+recovering, 15 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 872 op/s; 3078/81658 objects misplaced (3.769%); 9.1 MiB/s, 146 objects/s recovering 2024-05-15T23:37:51.693 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:37:51.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:51.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:37:51.734 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:37:51.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:37:51.828 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:51 smithi088 ceph-mon[44699]: pgmap v1914: 17 pgs: 2 active+recovering, 15 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 872 op/s; 3078/81658 objects misplaced (3.769%); 9.1 MiB/s, 146 objects/s recovering 2024-05-15T23:37:51.849 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:51.882 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:37:51.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:51.885 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:37:51.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:37:52.028 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:52.238 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:37:52.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:52.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:37:52.342 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:37:52.375 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:52.400 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:37:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:52.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:37:52.505 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:37:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:52.565 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:37:52.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:52.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:37:52.673 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:37:52.709 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:52.737 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:37:52.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:52.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:37:52.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:52.896 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:53.099 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:53.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:53.106 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:53.116 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:37:53.207 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:37:53.275 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:53.313 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:37:53.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:53.315 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:37:53.416 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:37:53.458 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:53.480 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:37:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:53.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:37:53.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:53 smithi090 ceph-mon[44434]: pgmap v1915: 17 pgs: 2 active+recovering, 15 active+clean; 2.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s; 3078/74468 objects misplaced (4.133%); 8.3 MiB/s, 132 objects/s recovering 2024-05-15T23:37:53.587 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:37:53.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:53 smithi084 ceph-mon[39784]: pgmap v1915: 17 pgs: 2 active+recovering, 15 active+clean; 2.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s; 3078/74468 objects misplaced (4.133%); 8.3 MiB/s, 132 objects/s recovering 2024-05-15T23:37:53.625 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:53 smithi088 ceph-mon[44699]: pgmap v1915: 17 pgs: 2 active+recovering, 15 active+clean; 2.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s; 3078/74468 objects misplaced (4.133%); 8.3 MiB/s, 132 objects/s recovering 2024-05-15T23:37:53.625 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:53.647 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:37:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:53.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:37:53.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:37:53.819 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:53.859 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:37:53.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:53.862 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:37:53.963 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:37:53.999 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:54.033 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:37:54.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:54.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:37:54.137 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:37:54.166 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:54.193 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:37:54.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:54.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:37:54.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:37:54.348 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:54.382 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:37:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:54.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:37:54.442 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:37:54.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:54.542 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:54.574 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:54.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:54.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:54.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:54 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:37:54.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:54 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:37:54.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:54 smithi084 ceph-mon[39784]: osdmap e719: 12 total, 12 up, 10 in 2024-05-15T23:37:54.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:37:54.740 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:54 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:37:54.740 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:54 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:37:54.740 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:54 smithi090 ceph-mon[44434]: osdmap e719: 12 total, 12 up, 10 in 2024-05-15T23:37:54.741 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:54.765 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:37:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:54.771 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:37:54.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:54 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:37:54.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:54 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:37:54.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:54 smithi088 ceph-mon[44699]: osdmap e719: 12 total, 12 up, 10 in 2024-05-15T23:37:54.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:37:54.911 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:54.989 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:37:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:54.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:37:55.093 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:37:55.126 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:55.356 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:55 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:37:55.356 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:55 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:37:55.356 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:55 smithi084 ceph-mon[39784]: osdmap e720: 12 total, 12 up, 10 in 2024-05-15T23:37:55.356 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:55 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4109796661' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:37:55.356 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:55 smithi084 ceph-mon[39784]: pgmap v1918: 17 pgs: 2 active+recovering, 15 active+clean; 2.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 944 op/s; 2291/71992 objects misplaced (3.182%); 12 MiB/s, 191 objects/s recovering 2024-05-15T23:37:55.376 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:37:55.431 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:37:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:37:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.17073.0:260387 8.1 8:8d76059a:::benchmark_data_smithi090_157496_object113449:head [delete] snapc 0=[] ondisk+write+known_if_redirected e721)", 2024-05-15T23:37:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:37:55.409968+0000", 2024-05-15T23:37:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.021140504000000001, 2024-05-15T23:37:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021172943999999999, 2024-05-15T23:37:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:37:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:37:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:37:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:37:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.17073", 2024-05-15T23:37:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260387 2024-05-15T23:37:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:37:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:37:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.409968+0000", 2024-05-15T23:37:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:37:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:37:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.409968+0000", 2024-05-15T23:37:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:37:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:37:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.409970+0000", 2024-05-15T23:37:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.99e-06 2024-05-15T23:37:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:37:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.409976+0000", 2024-05-15T23:37:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7690000000000002e-06 2024-05-15T23:37:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:37:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.409988+0000", 2024-05-15T23:37:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2544000000000001e-05 2024-05-15T23:37:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:37:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.409996+0000", 2024-05-15T23:37:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0160000000000008e-06 2024-05-15T23:37:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:37:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.410051+0000", 2024-05-15T23:37:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4740000000000001e-05 2024-05-15T23:37:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:37:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.410184+0000", 2024-05-15T23:37:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0001327 2024-05-15T23:37:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:37:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.410301+0000", 2024-05-15T23:37:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011781399999999999 2024-05-15T23:37:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:37:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.414865+0000", 2024-05-15T23:37:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045632370000000004 2024-05-15T23:37:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:37:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.431068+0000", 2024-05-15T23:37:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016203594000000002 2024-05-15T23:37:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:37:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.431126+0000", 2024-05-15T23:37:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8291000000000001e-05 2024-05-15T23:37:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:37:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:37:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:37:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.17073.0:260393 8.1 8:821ba6fd:::benchmark_data_smithi090_157496_object113455:head [delete] snapc 0=[] ondisk+write+known_if_redirected e721)", 2024-05-15T23:37:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:37:55.421379+0000", 2024-05-15T23:37:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.0097289660000000003, 2024-05-15T23:37:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0098503319999999998, 2024-05-15T23:37:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:37:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:37:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:37:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:37:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.17073", 2024-05-15T23:37:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:37:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260393 2024-05-15T23:37:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:37:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:37:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.421379+0000", 2024-05-15T23:37:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:37:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:37:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.421379+0000", 2024-05-15T23:37:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:37:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:37:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.421381+0000", 2024-05-15T23:37:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9709999999999998e-06 2024-05-15T23:37:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:37:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.421388+0000", 2024-05-15T23:37:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3840000000000002e-06 2024-05-15T23:37:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:37:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.421407+0000", 2024-05-15T23:37:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.95e-05 2024-05-15T23:37:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:37:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.421414+0000", 2024-05-15T23:37:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2250000000000003e-06 2024-05-15T23:37:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:37:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.421455+0000", 2024-05-15T23:37:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0448999999999998e-05 2024-05-15T23:37:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:37:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.421547+0000", 2024-05-15T23:37:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2299000000000005e-05 2024-05-15T23:37:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:37:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.421624+0000", 2024-05-15T23:37:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7072999999999999e-05 2024-05-15T23:37:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:37:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.425761+0000", 2024-05-15T23:37:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041363759999999998 2024-05-15T23:37:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:37:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:37:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:37:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.17073.0:260401 8.7 e721/654 8:e5dc5a61:::benchmark_data_smithi090_157496_object113463:head v 721'16490, mlcod=720'16488)", 2024-05-15T23:37:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:37:55.428357+0000", 2024-05-15T23:37:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.0027509520000000001, 2024-05-15T23:37:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029646120000000001, 2024-05-15T23:37:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:37:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:37:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-05-15T23:37:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:37:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:37:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.428357+0000", 2024-05-15T23:37:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:37:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:37:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.428357+0000", 2024-05-15T23:37:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:37:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:37:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.428360+0000", 2024-05-15T23:37:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6259999999999999e-06 2024-05-15T23:37:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:37:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.428374+0000", 2024-05-15T23:37:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3584000000000001e-05 2024-05-15T23:37:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:37:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.428394+0000", 2024-05-15T23:37:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0191e-05 2024-05-15T23:37:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:37:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.428405+0000", 2024-05-15T23:37:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1562999999999999e-05 2024-05-15T23:37:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:37:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.428475+0000", 2024-05-15T23:37:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9387999999999994e-05 2024-05-15T23:37:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:37:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:37:55.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:37:55.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:37:55.428500+0000", 2024-05-15T23:37:55.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5114999999999999e-05 2024-05-15T23:37:55.446 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:37:55.446 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:37:55.446 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:37:55.446 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:37:55.446 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-05-15T23:37:55.446 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 3 2024-05-15T23:37:55.446 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:37:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:55 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:37:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:55 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:37:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:55 smithi090 ceph-mon[44434]: osdmap e720: 12 total, 12 up, 10 in 2024-05-15T23:37:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:55 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4109796661' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:37:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:55 smithi090 ceph-mon[44434]: pgmap v1918: 17 pgs: 2 active+recovering, 15 active+clean; 2.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 944 op/s; 2291/71992 objects misplaced (3.182%); 12 MiB/s, 191 objects/s recovering 2024-05-15T23:37:55.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:55 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:37:55.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:55 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:37:55.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:55 smithi088 ceph-mon[44699]: osdmap e720: 12 total, 12 up, 10 in 2024-05-15T23:37:55.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:55 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4109796661' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:37:55.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:55 smithi088 ceph-mon[44699]: pgmap v1918: 17 pgs: 2 active+recovering, 15 active+clean; 2.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 944 op/s; 2291/71992 objects misplaced (3.182%); 12 MiB/s, 191 objects/s recovering 2024-05-15T23:37:56.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:56 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:37:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4109796661' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:37:56.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:56 smithi084 ceph-mon[39784]: osdmap e721: 12 total, 12 up, 10 in 2024-05-15T23:37:56.686 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:37:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:56.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:37:56.693 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T23:37:56.793 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:37:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:56 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:37:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4109796661' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:37:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:56 smithi090 ceph-mon[44434]: osdmap e721: 12 total, 12 up, 10 in 2024-05-15T23:37:56.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:56 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:37:56.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4109796661' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:37:56.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:56 smithi088 ceph-mon[44699]: osdmap e721: 12 total, 12 up, 10 in 2024-05-15T23:37:56.825 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:56.858 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:37:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:56.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:37:56.941 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-15T23:37:56.941 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:37:56.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:37:57.025 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:57.064 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:37:57.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:57.072 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:37:57.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:57.220 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:57.249 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:57.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:57.258 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:57.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:57.407 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:57.452 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:57.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:57.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:57.566 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:37:57.598 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:57.632 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:37:57.635 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:57 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:37:57.635 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:57 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:37:57.635 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:57 smithi088 ceph-mon[44699]: pgmap v1920: 17 pgs: 2 active+recovering, 15 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s; 2291/67648 objects misplaced (3.387%); 8.2 MiB/s, 131 objects/s recovering 2024-05-15T23:37:57.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:57.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:37:57.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:37:57.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:57 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:37:57.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:57 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:37:57.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:57 smithi084 ceph-mon[39784]: pgmap v1920: 17 pgs: 2 active+recovering, 15 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s; 2291/67648 objects misplaced (3.387%); 8.2 MiB/s, 131 objects/s recovering 2024-05-15T23:37:57.769 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:57 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:37:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:57 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:37:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:57 smithi090 ceph-mon[44434]: pgmap v1920: 17 pgs: 2 active+recovering, 15 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s; 2291/67648 objects misplaced (3.387%); 8.2 MiB/s, 131 objects/s recovering 2024-05-15T23:37:57.977 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:37:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:57.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:37:58.023 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:37:58.083 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:37:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:58.146 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:37:58.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:58.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:37:58.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:37:58.308 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:37:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:58.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:37:58.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:37:58.491 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:58.515 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:37:58.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:58 smithi084 ceph-mon[39784]: osdmap e722: 12 total, 12 up, 10 in 2024-05-15T23:37:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:58.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:37:58.619 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:37:58.651 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:58.764 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:37:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:58.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:37:58.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:58 smithi090 ceph-mon[44434]: osdmap e722: 12 total, 12 up, 10 in 2024-05-15T23:37:58.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:58 smithi088 ceph-mon[44699]: osdmap e722: 12 total, 12 up, 10 in 2024-05-15T23:37:58.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:37:58.872 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:37:58.872 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:37:58.872 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:37:58.879 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:37:58.895 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:37:58.917 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:37:58.942 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:37:58.948 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:58.971 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:37:58.991 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:37:58.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:58.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:37:59.000 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:37:59.038 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:37:59.062 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:37:59.097 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:37:59.103 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:37:59.130 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:59.138 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:37:59.143 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:37:59.143 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:37:59.143 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:37:59.143 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:37:59.143 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:37:59.143 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:37:59.143 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:37:59.143 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:37:59.143 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:37:59.144 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:37:59.144 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:37:59.144 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:37:59.227 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:37:59.381 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:59 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2302006030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:37:59.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.653 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.653 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.653 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.653 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:59 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.653 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:37:59 smithi084 ceph-mon[39784]: pgmap v1922: 17 pgs: 2 active+recovering, 15 active+clean; 1.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 935 op/s; 1951/63318 objects misplaced (3.081%); 12 MiB/s, 189 objects/s recovering 2024-05-15T23:37:59.686 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:37:59.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:59.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:37:59.795 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:37:59.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:59 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2302006030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:37:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:59 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:37:59 smithi090 ceph-mon[44434]: pgmap v1922: 17 pgs: 2 active+recovering, 15 active+clean; 1.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 935 op/s; 1951/63318 objects misplaced (3.081%); 12 MiB/s, 189 objects/s recovering 2024-05-15T23:37:59.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:59 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2302006030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:37:59.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.824 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:59 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:37:59.825 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:37:59 smithi088 ceph-mon[44699]: pgmap v1922: 17 pgs: 2 active+recovering, 15 active+clean; 1.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 935 op/s; 1951/63318 objects misplaced (3.081%); 12 MiB/s, 189 objects/s recovering 2024-05-15T23:37:59.828 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:37:59.854 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:37:59.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:37:59.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:37:59.962 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:37:59.994 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:00 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:00 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.857 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.857 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.857 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.858 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.858 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:00.858 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:00 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:01.249 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:38:01.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:01.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:38:01.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:38:01.412 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:01.442 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:38:01.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:01.449 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:38:01.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:38:01.590 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:01.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:01 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:38:01.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:01 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:38:01.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:01 smithi084 ceph-mon[39784]: osdmap e723: 12 total, 12 up, 10 in 2024-05-15T23:38:01.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:01 smithi084 ceph-mon[39784]: pgmap v1924: 17 pgs: 2 active+recovering, 15 active+clean; 1.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 824 op/s; 1361/62094 objects misplaced (2.192%); 9.7 MiB/s, 154 objects/s recovering 2024-05-15T23:38:01.635 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:38:01.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:01.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:38:01.713 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:01.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:38:01.783 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:01.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:01 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:38:01.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:01 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:38:01.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:01 smithi090 ceph-mon[44434]: osdmap e723: 12 total, 12 up, 10 in 2024-05-15T23:38:01.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:01 smithi090 ceph-mon[44434]: pgmap v1924: 17 pgs: 2 active+recovering, 15 active+clean; 1.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 824 op/s; 1361/62094 objects misplaced (2.192%); 9.7 MiB/s, 154 objects/s recovering 2024-05-15T23:38:01.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:01 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:38:01.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:01 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:38:01.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:01 smithi088 ceph-mon[44699]: osdmap e723: 12 total, 12 up, 10 in 2024-05-15T23:38:01.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:01 smithi088 ceph-mon[44699]: pgmap v1924: 17 pgs: 2 active+recovering, 15 active+clean; 1.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 824 op/s; 1361/62094 objects misplaced (2.192%); 9.7 MiB/s, 154 objects/s recovering 2024-05-15T23:38:02.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:38:02.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:02 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:38:02.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:02 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:38:02.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:02 smithi084 ceph-mon[39784]: osdmap e724: 12 total, 12 up, 10 in 2024-05-15T23:38:02.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:02 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:38:02.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:02 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:38:02.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:02 smithi090 ceph-mon[44434]: osdmap e724: 12 total, 12 up, 10 in 2024-05-15T23:38:02.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:02 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:38:02.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:02 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:38:02.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:02 smithi088 ceph-mon[44699]: osdmap e724: 12 total, 12 up, 10 in 2024-05-15T23:38:02.867 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:38:02.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:02.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:38:02.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:03.032 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:03.073 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:03.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:03.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:03.178 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:03.214 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:03.240 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:03.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:03.345 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:38:03.373 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:03.399 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:38:03.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:03.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:38:03.485 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:38:03.486 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:38:03.504 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:38:03.532 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:03 smithi088 ceph-mon[44699]: from='client.37277 v1:172.21.15.84:0/2442641212' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:38:03.532 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:03 smithi088 ceph-mon[44699]: pgmap v1926: 17 pgs: 2 active+recovering, 15 active+clean; 1.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s; 1361/54536 objects misplaced (2.496%); 9.7 MiB/s, 154 objects/s recovering 2024-05-15T23:38:03.535 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:03.565 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:38:03.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:03.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:38:03.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:03 smithi084 ceph-mon[39784]: from='client.37277 v1:172.21.15.84:0/2442641212' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:38:03.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:03 smithi084 ceph-mon[39784]: pgmap v1926: 17 pgs: 2 active+recovering, 15 active+clean; 1.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s; 1361/54536 objects misplaced (2.496%); 9.7 MiB/s, 154 objects/s recovering 2024-05-15T23:38:03.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:38:03.705 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:03.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:03 smithi090 ceph-mon[44434]: from='client.37277 v1:172.21.15.84:0/2442641212' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:38:03.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:03 smithi090 ceph-mon[44434]: pgmap v1926: 17 pgs: 2 active+recovering, 15 active+clean; 1.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s; 1361/54536 objects misplaced (2.496%); 9.7 MiB/s, 154 objects/s recovering 2024-05-15T23:38:03.918 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:38:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:03.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:38:04.028 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:38:04.062 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:04.092 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:38:04.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:04.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:38:04.197 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:04.227 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:04.249 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:04.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:04.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:04.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:38:04.409 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:04.412 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:38:04.412 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:38:04.437 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:38:04.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:04.442 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:38:04.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:38:04.575 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:04.878 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1925,"stamp":"2024-05-15T23:38:01.426395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1973879352,"num_objects":30113,"num_object_clones":0,"num_object_copies":60226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1361,"num_objects_unfound":0,"num_objects_dirty":30113,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":263884,"num_write_kb":9405991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15509,"num_bytes_recovered":1017591392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160855,"ondisk_log_size":160855,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":10996792,"kb_used_data":4877780,"kb_used_omap":41,"kb_used_meta":6118870,"kb_avail":926454728,"statfs":{"total":959950356480,"available":948689641472,"internally_reserved":0,"allocated":4994846720,"data_stored":4990967057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42533,"internal_metadata":6265723355},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,9,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-385548288,"num_objects":-5883,"num_object_clones":0,"num_object_copies":-11766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-930,"num_objects_unfound":0,"num_objects_dirty":-5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5883,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":930,"num_bytes_recovered":60948480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.050394"},"pg_stats":[{"pgid":"8.f","version":"722'16003","reported_seq":16285,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-05-15T23:37:57.802885+0000","last_change":"2024-05-15T23:37:17.929589+0000","last_active":"2024-05-15T23:37:57.802885+0000","last_peered":"2024-05-15T23:37:57.802885+0000","last_clean":"2024-05-15T23:37:57.802885+0000","last_became_active":"2024-05-15T23:34:01.801268+0000","last_became_peered":"2024-05-15T23:34:01.801268+0000","last_unstale":"2024-05-15T23:37:57.802885+0000","last_undegraded":"2024-05-15T23:37:57.802885+0000","last_fullsized":"2024-05-15T23:37:57.802885+0000","mapping_epoch":654,"log_start":"672'6000","ondisk_log_start":"672'6000","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'13799","last_scrub_stamp":"2024-05-15T23:37:17.929445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:17.929445+0000","objects_scrubbed":4116,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:37.474269+0000","scrub_duration":824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123928576,"num_objects":1891,"num_object_clones":0,"num_object_copies":3782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16003,"num_write_kb":572608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.e","version":"722'16282","reported_seq":32884,"reported_epoch":722,"state":"active+recovering","last_fresh":"2024-05-15T23:38:00.000464+0000","last_change":"2024-05-15T23:35:33.882254+0000","last_active":"2024-05-15T23:38:00.000464+0000","last_peered":"2024-05-15T23:38:00.000464+0000","last_clean":"2024-05-15T23:35:32.407085+0000","last_became_active":"2024-05-15T23:35:33.794835+0000","last_became_peered":"2024-05-15T23:35:33.794835+0000","last_unstale":"2024-05-15T23:38:00.000464+0000","last_undegraded":"2024-05-15T23:38:00.000464+0000","last_fullsized":"2024-05-15T23:38:00.000464+0000","mapping_epoch":685,"log_start":"673'6200","ondisk_log_start":"673'6200","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:34:00.786907+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116391936,"num_objects":1776,"num_object_clones":0,"num_object_copies":3552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":184,"num_objects_unfound":0,"num_objects_dirty":1776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16282,"num_write_kb":577856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8226,"num_bytes_recovered":539099136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,4","objects":1592},{"shards":"2,7","objects":184}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"722'16578","reported_seq":16863,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-05-15T23:37:57.792386+0000","last_change":"2024-05-15T23:37:13.985832+0000","last_active":"2024-05-15T23:37:57.792386+0000","last_peered":"2024-05-15T23:37:57.792386+0000","last_clean":"2024-05-15T23:37:57.792386+0000","last_became_active":"2024-05-15T23:34:01.993980+0000","last_became_peered":"2024-05-15T23:34:01.993980+0000","last_unstale":"2024-05-15T23:37:57.792386+0000","last_undegraded":"2024-05-15T23:37:57.792386+0000","last_fullsized":"2024-05-15T23:37:57.792386+0000","mapping_epoch":654,"log_start":"674'6500","ondisk_log_start":"674'6500","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14037","last_scrub_stamp":"2024-05-15T23:37:13.985583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:13.985583+0000","objects_scrubbed":4477,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:30.662338+0000","scrub_duration":973,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124846104,"num_objects":1906,"num_object_clones":0,"num_object_copies":3812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1906,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16578,"num_write_kb":591425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"722'16544","reported_seq":16903,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-05-15T23:38:00.028378+0000","last_change":"2024-05-15T23:37:12.677782+0000","last_active":"2024-05-15T23:38:00.028378+0000","last_peered":"2024-05-15T23:38:00.028378+0000","last_clean":"2024-05-15T23:38:00.028378+0000","last_became_active":"2024-05-15T23:34:01.805322+0000","last_became_peered":"2024-05-15T23:34:01.805322+0000","last_unstale":"2024-05-15T23:38:00.028378+0000","last_undegraded":"2024-05-15T23:38:00.028378+0000","last_fullsized":"2024-05-15T23:38:00.028378+0000","mapping_epoch":654,"log_start":"674'6500","ondisk_log_start":"674'6500","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'13803","last_scrub_stamp":"2024-05-15T23:37:12.677680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:12.677680+0000","objects_scrubbed":4570,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:21.090707+0000","scrub_duration":855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118751232,"num_objects":1812,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16544,"num_write_kb":587392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"722'16339","reported_seq":31123,"reported_epoch":722,"state":"active+recovering","last_fresh":"2024-05-15T23:37:59.211956+0000","last_change":"2024-05-15T23:35:33.908885+0000","last_active":"2024-05-15T23:37:59.211956+0000","last_peered":"2024-05-15T23:37:59.211956+0000","last_clean":"2024-05-15T23:35:32.482713+0000","last_became_active":"2024-05-15T23:35:33.804849+0000","last_became_peered":"2024-05-15T23:35:33.804849+0000","last_unstale":"2024-05-15T23:37:59.211956+0000","last_undegraded":"2024-05-15T23:37:59.211956+0000","last_fullsized":"2024-05-15T23:37:59.211956+0000","mapping_epoch":685,"log_start":"674'6300","ondisk_log_start":"674'6300","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'7203","last_scrub_stamp":"2024-05-15T23:35:13.697763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:35:13.697763+0000","objects_scrubbed":7163,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123404288,"num_objects":1883,"num_object_clones":0,"num_object_copies":3766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1177,"num_objects_unfound":0,"num_objects_dirty":1883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16339,"num_write_kb":583104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7277,"num_bytes_recovered":476905472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8","1"],"object_location_counts":[{"shards":"1,8","objects":1177},{"shards":"4,8","objects":706}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"722'16618","reported_seq":16847,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-05-15T23:37:59.202125+0000","last_change":"2024-05-15T23:35:14.555676+0000","last_active":"2024-05-15T23:37:59.202125+0000","last_peered":"2024-05-15T23:37:59.202125+0000","last_clean":"2024-05-15T23:37:59.202125+0000","last_became_active":"2024-05-15T23:34:01.800290+0000","last_became_peered":"2024-05-15T23:34:01.800290+0000","last_unstale":"2024-05-15T23:37:59.202125+0000","last_undegraded":"2024-05-15T23:37:59.202125+0000","last_fullsized":"2024-05-15T23:37:59.202125+0000","mapping_epoch":654,"log_start":"674'6600","ondisk_log_start":"674'6600","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'7418","last_scrub_stamp":"2024-05-15T23:35:14.555528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:35:14.555528+0000","objects_scrubbed":7384,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119406592,"num_objects":1822,"num_object_clones":0,"num_object_copies":3644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16618,"num_write_kb":590080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"721'16612","reported_seq":16889,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-05-15T23:37:57.210819+0000","last_change":"2024-05-15T23:37:17.558370+0000","last_active":"2024-05-15T23:37:57.210819+0000","last_peered":"2024-05-15T23:37:57.210819+0000","last_clean":"2024-05-15T23:37:57.210819+0000","last_became_active":"2024-05-15T23:34:01.802958+0000","last_became_peered":"2024-05-15T23:34:01.802958+0000","last_unstale":"2024-05-15T23:37:57.210819+0000","last_undegraded":"2024-05-15T23:37:57.210819+0000","last_fullsized":"2024-05-15T23:37:57.210819+0000","mapping_epoch":654,"log_start":"674'6600","ondisk_log_start":"674'6600","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14323","last_scrub_stamp":"2024-05-15T23:37:17.558198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:17.558198+0000","objects_scrubbed":4282,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:31.112904+0000","scrub_duration":663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129368064,"num_objects":1974,"num_object_clones":0,"num_object_copies":3948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":594752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"722'16620","reported_seq":16909,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-05-15T23:37:57.744120+0000","last_change":"2024-05-15T23:37:20.443433+0000","last_active":"2024-05-15T23:37:57.744120+0000","last_peered":"2024-05-15T23:37:57.744120+0000","last_clean":"2024-05-15T23:37:57.744120+0000","last_became_active":"2024-05-15T23:34:01.801309+0000","last_became_peered":"2024-05-15T23:34:01.801309+0000","last_unstale":"2024-05-15T23:37:57.744120+0000","last_undegraded":"2024-05-15T23:37:57.744120+0000","last_fullsized":"2024-05-15T23:37:57.744120+0000","mapping_epoch":654,"log_start":"674'6600","ondisk_log_start":"674'6600","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14483","last_scrub_stamp":"2024-05-15T23:37:20.443208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:20.443208+0000","objects_scrubbed":4160,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:49.450668+0000","scrub_duration":1397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130154496,"num_objects":1986,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16620,"num_write_kb":595392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.6","version":"722'16290","reported_seq":16568,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-05-15T23:37:57.764717+0000","last_change":"2024-05-15T23:37:16.969610+0000","last_active":"2024-05-15T23:37:57.764717+0000","last_peered":"2024-05-15T23:37:57.764717+0000","last_clean":"2024-05-15T23:37:57.764717+0000","last_became_active":"2024-05-15T23:34:01.801861+0000","last_became_peered":"2024-05-15T23:34:01.801861+0000","last_unstale":"2024-05-15T23:37:57.764717+0000","last_undegraded":"2024-05-15T23:37:57.764717+0000","last_fullsized":"2024-05-15T23:37:57.764717+0000","mapping_epoch":654,"log_start":"672'6200","ondisk_log_start":"672'6200","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'13988","last_scrub_stamp":"2024-05-15T23:37:16.969449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:16.969449+0000","objects_scrubbed":4218,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:18.292206+0000","scrub_duration":851,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124518400,"num_objects":1900,"num_object_clones":0,"num_object_copies":3800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16290,"num_write_kb":582080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"723'16754","reported_seq":17037,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-05-15T23:38:01.363004+0000","last_change":"2024-05-15T23:37:22.767028+0000","last_active":"2024-05-15T23:38:01.363004+0000","last_peered":"2024-05-15T23:38:01.363004+0000","last_clean":"2024-05-15T23:38:01.363004+0000","last_became_active":"2024-05-15T23:34:02.002490+0000","last_became_peered":"2024-05-15T23:34:02.002490+0000","last_unstale":"2024-05-15T23:38:01.363004+0000","last_undegraded":"2024-05-15T23:38:01.363004+0000","last_fullsized":"2024-05-15T23:38:01.363004+0000","mapping_epoch":654,"log_start":"674'6700","ondisk_log_start":"674'6700","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'14430","last_scrub_stamp":"2024-05-15T23:37:22.766704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:22.766704+0000","objects_scrubbed":4080,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:25.000957+0000","scrub_duration":891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114294784,"num_objects":1744,"num_object_clones":0,"num_object_copies":3488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16754,"num_write_kb":591936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"721'16452","reported_seq":16735,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-05-15T23:37:57.244740+0000","last_change":"2024-05-15T23:37:18.603196+0000","last_active":"2024-05-15T23:37:57.244740+0000","last_peered":"2024-05-15T23:37:57.244740+0000","last_clean":"2024-05-15T23:37:57.244740+0000","last_became_active":"2024-05-15T23:34:01.800457+0000","last_became_peered":"2024-05-15T23:34:01.800457+0000","last_unstale":"2024-05-15T23:37:57.244740+0000","last_undegraded":"2024-05-15T23:37:57.244740+0000","last_fullsized":"2024-05-15T23:37:57.244740+0000","mapping_epoch":654,"log_start":"673'6400","ondisk_log_start":"673'6400","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14248","last_scrub_stamp":"2024-05-15T23:37:18.602932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:18.602932+0000","objects_scrubbed":4247,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:37.953179+0000","scrub_duration":726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132907008,"num_objects":2028,"num_object_clones":0,"num_object_copies":4056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16452,"num_write_kb":591360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"722'16636","reported_seq":16920,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-05-15T23:37:57.818171+0000","last_change":"2024-05-15T23:37:15.959332+0000","last_active":"2024-05-15T23:37:57.818171+0000","last_peered":"2024-05-15T23:37:57.818171+0000","last_clean":"2024-05-15T23:37:57.818171+0000","last_became_active":"2024-05-15T23:34:01.801285+0000","last_became_peered":"2024-05-15T23:34:01.801285+0000","last_unstale":"2024-05-15T23:37:57.818171+0000","last_undegraded":"2024-05-15T23:37:57.818171+0000","last_fullsized":"2024-05-15T23:37:57.818171+0000","mapping_epoch":654,"log_start":"674'6600","ondisk_log_start":"674'6600","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14151","last_scrub_stamp":"2024-05-15T23:37:15.959223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:15.959223+0000","objects_scrubbed":4455,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:21.746827+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127533056,"num_objects":1946,"num_object_clones":0,"num_object_copies":3892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16636,"num_write_kb":594624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.2","version":"723'16841","reported_seq":17209,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-05-15T23:38:01.362649+0000","last_change":"2024-05-15T23:37:27.724613+0000","last_active":"2024-05-15T23:38:01.362649+0000","last_peered":"2024-05-15T23:38:01.362649+0000","last_clean":"2024-05-15T23:38:01.362649+0000","last_became_active":"2024-05-15T23:34:02.002460+0000","last_became_peered":"2024-05-15T23:34:02.002460+0000","last_unstale":"2024-05-15T23:38:01.362649+0000","last_undegraded":"2024-05-15T23:38:01.362649+0000","last_fullsized":"2024-05-15T23:38:01.362649+0000","mapping_epoch":654,"log_start":"675'6800","ondisk_log_start":"675'6800","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'14872","last_scrub_stamp":"2024-05-15T23:37:27.724324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:27.724324+0000","objects_scrubbed":3814,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:29.437384+0000","scrub_duration":761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119734272,"num_objects":1827,"num_object_clones":0,"num_object_copies":3654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16841,"num_write_kb":597376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"721'16359","reported_seq":16642,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-05-15T23:37:56.897531+0000","last_change":"2024-05-15T23:37:13.004231+0000","last_active":"2024-05-15T23:37:56.897531+0000","last_peered":"2024-05-15T23:37:56.897531+0000","last_clean":"2024-05-15T23:37:56.897531+0000","last_became_active":"2024-05-15T23:34:01.801345+0000","last_became_peered":"2024-05-15T23:34:01.801345+0000","last_unstale":"2024-05-15T23:37:56.897531+0000","last_undegraded":"2024-05-15T23:37:56.897531+0000","last_fullsized":"2024-05-15T23:37:56.897531+0000","mapping_epoch":654,"log_start":"672'6300","ondisk_log_start":"672'6300","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'13817","last_scrub_stamp":"2024-05-15T23:37:13.004123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:13.004123+0000","objects_scrubbed":4537,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:19.464099+0000","scrub_duration":690,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129564672,"num_objects":1977,"num_object_clones":0,"num_object_copies":3954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16359,"num_write_kb":586752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"721'16361","reported_seq":16636,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-05-15T23:37:56.977912+0000","last_change":"2024-05-15T23:37:35.218638+0000","last_active":"2024-05-15T23:37:56.977912+0000","last_peered":"2024-05-15T23:37:56.977912+0000","last_clean":"2024-05-15T23:37:56.977912+0000","last_became_active":"2024-05-15T23:34:01.803171+0000","last_became_peered":"2024-05-15T23:34:01.803171+0000","last_unstale":"2024-05-15T23:37:56.977912+0000","last_undegraded":"2024-05-15T23:37:56.977912+0000","last_fullsized":"2024-05-15T23:37:56.977912+0000","mapping_epoch":654,"log_start":"673'6300","ondisk_log_start":"673'6300","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'15038","last_scrub_stamp":"2024-05-15T23:37:35.218511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:35.218511+0000","objects_scrubbed":3337,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:37.435272+0000","scrub_duration":561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131137536,"num_objects":2001,"num_object_clones":0,"num_object_copies":4002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16361,"num_write_kb":587584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"723'16472","reported_seq":16747,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-05-15T23:38:01.364172+0000","last_change":"2024-05-15T23:37:18.618832+0000","last_active":"2024-05-15T23:38:01.364172+0000","last_peered":"2024-05-15T23:38:01.364172+0000","last_clean":"2024-05-15T23:38:01.364172+0000","last_became_active":"2024-05-15T23:34:01.803136+0000","last_became_peered":"2024-05-15T23:34:01.803136+0000","last_unstale":"2024-05-15T23:38:01.364172+0000","last_undegraded":"2024-05-15T23:38:01.364172+0000","last_fullsized":"2024-05-15T23:38:01.364172+0000","mapping_epoch":654,"log_start":"674'6400","ondisk_log_start":"674'6400","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14069","last_scrub_stamp":"2024-05-15T23:37:18.618655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:18.618655+0000","objects_scrubbed":4063,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:29.384529+0000","scrub_duration":734,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107347968,"num_objects":1638,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16472,"num_write_kb":579520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1561,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-05-15T23:37:57.400751+0000","last_change":"2024-05-15T23:37:17.136503+0000","last_active":"2024-05-15T23:37:57.400751+0000","last_peered":"2024-05-15T23:37:57.400751+0000","last_clean":"2024-05-15T23:37:57.400751+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:37:57.400751+0000","last_undegraded":"2024-05-15T23:37:57.400751+0000","last_fullsized":"2024-05-15T23:37:57.400751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:37:17.136378+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:37:17.136378+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:22.171455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1973288984,"num_objects":30111,"num_object_clones":0,"num_object_copies":60222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1361,"num_objects_unfound":0,"num_objects_dirty":30111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263761,"num_write_kb":9403841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15503,"num_bytes_recovered":1016004608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4939653120,"data_stored":4939644976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160761,"ondisk_log_size":160761,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351387,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1457212,"kb_used_data":636080,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92287940,"statfs":{"total":95995035648,"available":94502850560,"internally_reserved":0,"allocated":651345920,"data_stored":650956019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":840825284},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019272,"kb_used_data":446652,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92725880,"statfs":{"total":95995035648,"available":94951301120,"internally_reserved":0,"allocated":457371648,"data_stored":456977441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":586348992},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776552,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1061500,"kb_used_data":350064,"kb_used_omap":1,"kb_used_meta":711422,"kb_avail":92683652,"statfs":{"total":95995035648,"available":94908059648,"internally_reserved":0,"allocated":358465536,"data_stored":358077304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":728496576},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":520,"seq":2233382994055,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":472120,"kb_used_data":358832,"kb_used_omap":24,"kb_used_meta":113255,"kb_avail":93273032,"statfs":{"total":95995035648,"available":95511584768,"internally_reserved":0,"allocated":367443968,"data_stored":367055736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25357,"internal_metadata":115973363},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267121,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1277636,"kb_used_data":655036,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92467516,"statfs":{"total":95995035648,"available":94686736384,"internally_reserved":0,"allocated":670756864,"data_stored":670361008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":637532618},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568496054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575312,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1734700,"kb_used_data":931620,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":92010452,"statfs":{"total":95995035648,"available":94218702848,"internally_reserved":0,"allocated":953978880,"data_stored":953596623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":822344134},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016721,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483410,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545404,"kb_used_data":342320,"kb_used_omap":4,"kb_used_meta":203067,"kb_avail":93199748,"statfs":{"total":95995035648,"available":95436541952,"internally_reserved":0,"allocated":350535680,"data_stored":350147448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":207941315},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1133292,"kb_used_data":258020,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92611860,"statfs":{"total":95995035648,"available":94834544640,"internally_reserved":0,"allocated":264212480,"data_stored":263830223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":896268743},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986623,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1067324,"kb_used_data":391984,"kb_used_omap":1,"kb_used_meta":675326,"kb_avail":92677828,"statfs":{"total":95995035648,"available":94902095872,"internally_reserved":0,"allocated":401391616,"data_stored":401003384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":691534277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790395,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1228332,"kb_used_data":507172,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92516820,"statfs":{"total":95995035648,"available":94737223680,"internally_reserved":0,"allocated":519344128,"data_stored":518961871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":948043776,"data_stored":948043776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":348061696,"data_stored":348061696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":261357568,"data_stored":261357568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":398852096,"data_stored":398852096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":515637248,"data_stored":515637248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":667357184,"data_stored":667353112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":365297664,"data_stored":365297664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":354353152,"data_stored":354353152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":450101248,"data_stored":450101248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":630591488,"data_stored":630587416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:38:04.879 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:38:05.210 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:05.245 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:05.707 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:05 smithi084 ceph-mon[39784]: pgmap v1927: 17 pgs: 1 active+recovering, 16 active+clean; 1.6 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 990 op/s; 772/51928 objects misplaced (1.487%); 12 MiB/s, 191 objects/s recovering 2024-05-15T23:38:05.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:05 smithi090 ceph-mon[44434]: pgmap v1927: 17 pgs: 1 active+recovering, 16 active+clean; 1.6 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 990 op/s; 772/51928 objects misplaced (1.487%); 12 MiB/s, 191 objects/s recovering 2024-05-15T23:38:05.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:05 smithi088 ceph-mon[44699]: pgmap v1927: 17 pgs: 1 active+recovering, 16 active+clean; 1.6 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 990 op/s; 772/51928 objects misplaced (1.487%); 12 MiB/s, 191 objects/s recovering 2024-05-15T23:38:06.531 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:38:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:06.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:38:06.639 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:38:06.669 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:06.691 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:38:06.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:06.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:38:06.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:38:06.840 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:06.918 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:38:06.923 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:06 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/338753255' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:38:06.995 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:38:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:07.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:38:07.103 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:38:07.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:06 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/338753255' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:38:07.133 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:07.158 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:38:07.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:07.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:38:07.270 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:38:07.297 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:07.301 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:06 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/338753255' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:38:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:07 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:38:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:07 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/338753255' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:38:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:07 smithi090 ceph-mon[44434]: osdmap e725: 12 total, 12 up, 10 in 2024-05-15T23:38:08.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:07 smithi090 ceph-mon[44434]: pgmap v1929: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 772/47294 objects misplaced (1.632%); 5.3 MiB/s, 85 objects/s recovering 2024-05-15T23:38:08.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:07 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:38:08.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:07 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/338753255' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:38:08.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:07 smithi088 ceph-mon[44699]: osdmap e725: 12 total, 12 up, 10 in 2024-05-15T23:38:08.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:07 smithi088 ceph-mon[44699]: pgmap v1929: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 772/47294 objects misplaced (1.632%); 5.3 MiB/s, 85 objects/s recovering 2024-05-15T23:38:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:07 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:38:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:07 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/338753255' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:38:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:07 smithi084 ceph-mon[39784]: osdmap e725: 12 total, 12 up, 10 in 2024-05-15T23:38:08.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:07 smithi084 ceph-mon[39784]: pgmap v1929: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 772/47294 objects misplaced (1.632%); 5.3 MiB/s, 85 objects/s recovering 2024-05-15T23:38:08.390 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:38:08.411 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:38:08.436 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:38:08.456 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:38:08.479 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:38:08.503 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:38:08.525 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:38:08.546 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:38:08.579 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:38:08.604 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:38:08.633 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:38:08.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:38:08.658 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:38:08.658 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:38:08.658 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:38:08.658 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:38:08.659 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:38:08.659 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:38:08.659 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:38:08.659 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:38:08.659 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:38:08.659 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:38:08.659 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:38:08.659 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:38:09.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4010398400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:38:09.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:08 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4010398400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:38:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:08 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4010398400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:38:09.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:08 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:09.456 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:38:09.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:09.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:38:09.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:38:09.633 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:09.670 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:38:09.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:09.677 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:38:09.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:09.814 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:09.866 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:09.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:09.873 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:09.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:38:10.036 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:10.076 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:38:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:10.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:38:10.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:38:10.206 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.207 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.207 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.207 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.207 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.207 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:09 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.207 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:09 smithi084 ceph-mon[39784]: pgmap v1930: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s; 309/42852 objects misplaced (0.721%); 8.2 MiB/s, 131 objects/s recovering 2024-05-15T23:38:10.210 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:09 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:09 smithi090 ceph-mon[44434]: pgmap v1930: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s; 309/42852 objects misplaced (0.721%); 8.2 MiB/s, 131 objects/s recovering 2024-05-15T23:38:10.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:09 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:10.324 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:09 smithi088 ceph-mon[44699]: pgmap v1930: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.20k op/s; 309/42852 objects misplaced (0.721%); 8.2 MiB/s, 131 objects/s recovering 2024-05-15T23:38:10.364 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:38:10.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:10.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:38:10.390 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T23:38:10.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:38:10.504 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:10.530 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:38:10.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:10.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:38:10.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:38:10.675 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:10.700 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:38:10.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:10.708 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:38:10.809 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:38:10.840 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:10.865 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:38:10.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:10.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:38:10.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:10.983 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:11.018 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:11.040 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:11.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:11.047 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:11.148 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:38:11.193 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:11.215 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:38:11.218 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:38:11.222 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:38:11.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:11.225 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:38:11.326 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:38:11.360 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:11.380 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:38:11.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:11.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:38:11.485 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:38:11.491 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:38:11.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:11 smithi084 ceph-mon[39784]: from='client.17361 v1:172.21.15.84:0/3373375050' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:38:11.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:11 smithi084 ceph-mon[39784]: pgmap v1931: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 940 op/s; 309/41626 objects misplaced (0.742%); 6.7 MiB/s, 106 objects/s recovering 2024-05-15T23:38:11.518 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:11 smithi090 ceph-mon[44434]: from='client.17361 v1:172.21.15.84:0/3373375050' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:38:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:11 smithi090 ceph-mon[44434]: pgmap v1931: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 940 op/s; 309/41626 objects misplaced (0.742%); 6.7 MiB/s, 106 objects/s recovering 2024-05-15T23:38:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:11 smithi088 ceph-mon[44699]: from='client.17361 v1:172.21.15.84:0/3373375050' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:38:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:11 smithi088 ceph-mon[44699]: pgmap v1931: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 940 op/s; 309/41626 objects misplaced (0.742%); 6.7 MiB/s, 106 objects/s recovering 2024-05-15T23:38:11.951 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:38:11.973 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:38:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:11.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:38:12.031 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:12.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:38:12.130 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:12.156 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:38:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:12.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:38:12.263 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:12.292 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:12.317 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:12.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:12.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:12.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:38:12.484 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:12.526 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:38:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:12.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:38:12.633 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:38:12.662 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:12.683 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:38:12.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:12.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:38:12.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:12.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:12 smithi090 ceph-mon[44434]: 8.e scrub starts 2024-05-15T23:38:12.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:12 smithi090 ceph-mon[44434]: 8.e scrub ok 2024-05-15T23:38:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:12.874 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:12.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:12.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:12.902 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:12 smithi084 ceph-mon[39784]: 8.e scrub starts 2024-05-15T23:38:12.902 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:12 smithi084 ceph-mon[39784]: 8.e scrub ok 2024-05-15T23:38:12.978 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:38:13.007 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:13.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:12 smithi088 ceph-mon[44699]: 8.e scrub starts 2024-05-15T23:38:13.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:12 smithi088 ceph-mon[44699]: 8.e scrub ok 2024-05-15T23:38:13.208 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:38:13.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:13.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:38:13.257 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1930,"stamp":"2024-05-15T23:38:09.313152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1404568120,"num_objects":21426,"num_object_clones":0,"num_object_copies":42852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":309,"num_objects_unfound":0,"num_objects_dirty":21426,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":272571,"num_write_kb":9405991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16561,"num_bytes_recovered":1086535264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160942,"ondisk_log_size":160942,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":10040660,"kb_used_data":3917248,"kb_used_omap":41,"kb_used_meta":6123030,"kb_avail":927410860,"statfs":{"total":959950356480,"available":949668720640,"internally_reserved":0,"allocated":4011261952,"data_stored":4007232814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42533,"internal_metadata":6269983195},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-630521856,"num_objects":-9621,"num_object_clones":0,"num_object_copies":-19242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1052,"num_objects_unfound":0,"num_objects_dirty":-9621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9621,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1052,"num_bytes_recovered":68943872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001435"},"pg_stats":[{"pgid":"8.f","version":"725'16646","reported_seq":16934,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-05-15T23:38:07.858497+0000","last_change":"2024-05-15T23:37:17.929589+0000","last_active":"2024-05-15T23:38:07.858497+0000","last_peered":"2024-05-15T23:38:07.858497+0000","last_clean":"2024-05-15T23:38:07.858497+0000","last_became_active":"2024-05-15T23:34:01.801268+0000","last_became_peered":"2024-05-15T23:34:01.801268+0000","last_unstale":"2024-05-15T23:38:07.858497+0000","last_undegraded":"2024-05-15T23:38:07.858497+0000","last_fullsized":"2024-05-15T23:38:07.858497+0000","mapping_epoch":654,"log_start":"675'6600","ondisk_log_start":"675'6600","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'13799","last_scrub_stamp":"2024-05-15T23:37:17.929445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:17.929445+0000","objects_scrubbed":4116,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:37.474269+0000","scrub_duration":824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81788928,"num_objects":1248,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16646,"num_write_kb":572608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.e","version":"724'16598","reported_seq":33575,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-05-15T23:38:04.952132+0000","last_change":"2024-05-15T23:38:01.445941+0000","last_active":"2024-05-15T23:38:04.952132+0000","last_peered":"2024-05-15T23:38:04.952132+0000","last_clean":"2024-05-15T23:38:04.952132+0000","last_became_active":"2024-05-15T23:35:33.794835+0000","last_became_peered":"2024-05-15T23:35:33.794835+0000","last_unstale":"2024-05-15T23:38:04.952132+0000","last_undegraded":"2024-05-15T23:38:04.952132+0000","last_fullsized":"2024-05-15T23:38:04.952132+0000","mapping_epoch":685,"log_start":"674'6500","ondisk_log_start":"674'6500","created":654,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:34:00.786907+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:07.803486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95682560,"num_objects":1460,"num_object_clones":0,"num_object_copies":2920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16598,"num_write_kb":577856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8410,"num_bytes_recovered":551157760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"725'17221","reported_seq":17512,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-05-15T23:38:07.858460+0000","last_change":"2024-05-15T23:37:13.985832+0000","last_active":"2024-05-15T23:38:07.858460+0000","last_peered":"2024-05-15T23:38:07.858460+0000","last_clean":"2024-05-15T23:38:07.858460+0000","last_became_active":"2024-05-15T23:34:01.993980+0000","last_became_peered":"2024-05-15T23:34:01.993980+0000","last_unstale":"2024-05-15T23:38:07.858460+0000","last_undegraded":"2024-05-15T23:38:07.858460+0000","last_fullsized":"2024-05-15T23:38:07.858460+0000","mapping_epoch":654,"log_start":"678'7200","ondisk_log_start":"678'7200","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14037","last_scrub_stamp":"2024-05-15T23:37:13.985583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:13.985583+0000","objects_scrubbed":4477,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:30.662338+0000","scrub_duration":973,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82706456,"num_objects":1263,"num_object_clones":0,"num_object_copies":2526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1263,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17221,"num_write_kb":591425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"724'16920","reported_seq":17283,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-05-15T23:38:05.086594+0000","last_change":"2024-05-15T23:37:12.677782+0000","last_active":"2024-05-15T23:38:05.086594+0000","last_peered":"2024-05-15T23:38:05.086594+0000","last_clean":"2024-05-15T23:38:05.086594+0000","last_became_active":"2024-05-15T23:34:01.805322+0000","last_became_peered":"2024-05-15T23:34:01.805322+0000","last_unstale":"2024-05-15T23:38:05.086594+0000","last_undegraded":"2024-05-15T23:38:05.086594+0000","last_fullsized":"2024-05-15T23:38:05.086594+0000","mapping_epoch":654,"log_start":"676'6900","ondisk_log_start":"676'6900","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'13803","last_scrub_stamp":"2024-05-15T23:37:12.677680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:12.677680+0000","objects_scrubbed":4570,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:21.090707+0000","scrub_duration":855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94109696,"num_objects":1436,"num_object_clones":0,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16920,"num_write_kb":587392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"725'16950","reported_seq":33475,"reported_epoch":725,"state":"active+recovering","last_fresh":"2024-05-15T23:38:09.218032+0000","last_change":"2024-05-15T23:35:33.908885+0000","last_active":"2024-05-15T23:38:09.218032+0000","last_peered":"2024-05-15T23:38:09.218032+0000","last_clean":"2024-05-15T23:35:32.482713+0000","last_became_active":"2024-05-15T23:35:33.804849+0000","last_became_peered":"2024-05-15T23:35:33.804849+0000","last_unstale":"2024-05-15T23:38:09.218032+0000","last_undegraded":"2024-05-15T23:38:09.218032+0000","last_fullsized":"2024-05-15T23:38:09.218032+0000","mapping_epoch":685,"log_start":"676'6900","ondisk_log_start":"676'6900","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'7203","last_scrub_stamp":"2024-05-15T23:35:13.697763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:35:13.697763+0000","objects_scrubbed":7163,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83361792,"num_objects":1272,"num_object_clones":0,"num_object_copies":2544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":309,"num_objects_unfound":0,"num_objects_dirty":1272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16950,"num_write_kb":583104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8145,"num_bytes_recovered":533790720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8","1"],"object_location_counts":[{"shards":"1,8","objects":309},{"shards":"4,8","objects":963}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"725'17238","reported_seq":17473,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-05-15T23:38:09.173857+0000","last_change":"2024-05-15T23:35:14.555676+0000","last_active":"2024-05-15T23:38:09.173857+0000","last_peered":"2024-05-15T23:38:09.173857+0000","last_clean":"2024-05-15T23:38:09.173857+0000","last_became_active":"2024-05-15T23:34:01.800290+0000","last_became_peered":"2024-05-15T23:34:01.800290+0000","last_unstale":"2024-05-15T23:38:09.173857+0000","last_undegraded":"2024-05-15T23:38:09.173857+0000","last_fullsized":"2024-05-15T23:38:09.173857+0000","mapping_epoch":654,"log_start":"678'7200","ondisk_log_start":"678'7200","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'7418","last_scrub_stamp":"2024-05-15T23:35:14.555528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:35:14.555528+0000","objects_scrubbed":7384,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78774272,"num_objects":1202,"num_object_clones":0,"num_object_copies":2404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17238,"num_write_kb":590080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"725'17234","reported_seq":17519,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-05-15T23:38:07.230997+0000","last_change":"2024-05-15T23:37:17.558370+0000","last_active":"2024-05-15T23:38:07.230997+0000","last_peered":"2024-05-15T23:38:07.230997+0000","last_clean":"2024-05-15T23:38:07.230997+0000","last_became_active":"2024-05-15T23:34:01.802958+0000","last_became_peered":"2024-05-15T23:34:01.802958+0000","last_unstale":"2024-05-15T23:38:07.230997+0000","last_undegraded":"2024-05-15T23:38:07.230997+0000","last_fullsized":"2024-05-15T23:38:07.230997+0000","mapping_epoch":654,"log_start":"678'7200","ondisk_log_start":"678'7200","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14323","last_scrub_stamp":"2024-05-15T23:37:17.558198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:17.558198+0000","objects_scrubbed":4282,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:31.112904+0000","scrub_duration":663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88604672,"num_objects":1352,"num_object_clones":0,"num_object_copies":2704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17234,"num_write_kb":594752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"725'17278","reported_seq":17572,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-05-15T23:38:07.847926+0000","last_change":"2024-05-15T23:37:20.443433+0000","last_active":"2024-05-15T23:38:07.847926+0000","last_peered":"2024-05-15T23:38:07.847926+0000","last_clean":"2024-05-15T23:38:07.847926+0000","last_became_active":"2024-05-15T23:34:01.801309+0000","last_became_peered":"2024-05-15T23:34:01.801309+0000","last_unstale":"2024-05-15T23:38:07.847926+0000","last_undegraded":"2024-05-15T23:38:07.847926+0000","last_fullsized":"2024-05-15T23:38:07.847926+0000","mapping_epoch":654,"log_start":"678'7200","ondisk_log_start":"678'7200","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14483","last_scrub_stamp":"2024-05-15T23:37:20.443208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:20.443208+0000","objects_scrubbed":4160,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:49.450668+0000","scrub_duration":1397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87031808,"num_objects":1328,"num_object_clones":0,"num_object_copies":2656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17278,"num_write_kb":595392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.6","version":"725'16958","reported_seq":17242,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-05-15T23:38:07.861538+0000","last_change":"2024-05-15T23:37:16.969610+0000","last_active":"2024-05-15T23:38:07.861538+0000","last_peered":"2024-05-15T23:38:07.861538+0000","last_clean":"2024-05-15T23:38:07.861538+0000","last_became_active":"2024-05-15T23:34:01.801861+0000","last_became_peered":"2024-05-15T23:34:01.801861+0000","last_unstale":"2024-05-15T23:38:07.861538+0000","last_undegraded":"2024-05-15T23:38:07.861538+0000","last_fullsized":"2024-05-15T23:38:07.861538+0000","mapping_epoch":654,"log_start":"676'6900","ondisk_log_start":"676'6900","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'13988","last_scrub_stamp":"2024-05-15T23:37:16.969449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:16.969449+0000","objects_scrubbed":4218,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:18.292206+0000","scrub_duration":851,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80740352,"num_objects":1232,"num_object_clones":0,"num_object_copies":2464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16958,"num_write_kb":582080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"724'17094","reported_seq":17379,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-05-15T23:38:06.358306+0000","last_change":"2024-05-15T23:37:22.767028+0000","last_active":"2024-05-15T23:38:06.358306+0000","last_peered":"2024-05-15T23:38:06.358306+0000","last_clean":"2024-05-15T23:38:06.358306+0000","last_became_active":"2024-05-15T23:34:02.002490+0000","last_became_peered":"2024-05-15T23:34:02.002490+0000","last_unstale":"2024-05-15T23:38:06.358306+0000","last_undegraded":"2024-05-15T23:38:06.358306+0000","last_fullsized":"2024-05-15T23:38:06.358306+0000","mapping_epoch":654,"log_start":"676'7000","ondisk_log_start":"676'7000","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'14430","last_scrub_stamp":"2024-05-15T23:37:22.766704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:22.766704+0000","objects_scrubbed":4080,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:25.000957+0000","scrub_duration":891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92012544,"num_objects":1404,"num_object_clones":0,"num_object_copies":2808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17094,"num_write_kb":591936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"725'17128","reported_seq":17419,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-05-15T23:38:07.234189+0000","last_change":"2024-05-15T23:37:18.603196+0000","last_active":"2024-05-15T23:38:07.234189+0000","last_peered":"2024-05-15T23:38:07.234189+0000","last_clean":"2024-05-15T23:38:07.234189+0000","last_became_active":"2024-05-15T23:34:01.800457+0000","last_became_peered":"2024-05-15T23:34:01.800457+0000","last_unstale":"2024-05-15T23:38:07.234189+0000","last_undegraded":"2024-05-15T23:38:07.234189+0000","last_fullsized":"2024-05-15T23:38:07.234189+0000","mapping_epoch":654,"log_start":"678'7100","ondisk_log_start":"678'7100","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14248","last_scrub_stamp":"2024-05-15T23:37:18.602932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:18.602932+0000","objects_scrubbed":4247,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:37.953179+0000","scrub_duration":726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88604672,"num_objects":1352,"num_object_clones":0,"num_object_copies":2704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17128,"num_write_kb":591360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"725'17207","reported_seq":17498,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-05-15T23:38:07.854370+0000","last_change":"2024-05-15T23:37:15.959332+0000","last_active":"2024-05-15T23:38:07.854370+0000","last_peered":"2024-05-15T23:38:07.854370+0000","last_clean":"2024-05-15T23:38:07.854370+0000","last_became_active":"2024-05-15T23:34:01.801285+0000","last_became_peered":"2024-05-15T23:34:01.801285+0000","last_unstale":"2024-05-15T23:38:07.854370+0000","last_undegraded":"2024-05-15T23:38:07.854370+0000","last_fullsized":"2024-05-15T23:38:07.854370+0000","mapping_epoch":654,"log_start":"678'7200","ondisk_log_start":"678'7200","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14151","last_scrub_stamp":"2024-05-15T23:37:15.959223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:15.959223+0000","objects_scrubbed":4455,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:21.746827+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90112000,"num_objects":1375,"num_object_clones":0,"num_object_copies":2750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17207,"num_write_kb":594624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.2","version":"724'17174","reported_seq":17546,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-05-15T23:38:06.400050+0000","last_change":"2024-05-15T23:37:27.724613+0000","last_active":"2024-05-15T23:38:06.400050+0000","last_peered":"2024-05-15T23:38:06.400050+0000","last_clean":"2024-05-15T23:38:06.400050+0000","last_became_active":"2024-05-15T23:34:02.002460+0000","last_became_peered":"2024-05-15T23:34:02.002460+0000","last_unstale":"2024-05-15T23:38:06.400050+0000","last_undegraded":"2024-05-15T23:38:06.400050+0000","last_fullsized":"2024-05-15T23:38:06.400050+0000","mapping_epoch":654,"log_start":"677'7100","ondisk_log_start":"677'7100","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'14872","last_scrub_stamp":"2024-05-15T23:37:27.724324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:27.724324+0000","objects_scrubbed":3814,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:29.437384+0000","scrub_duration":761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97910784,"num_objects":1494,"num_object_clones":0,"num_object_copies":2988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17174,"num_write_kb":597376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"725'17028","reported_seq":17319,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-05-15T23:38:06.955900+0000","last_change":"2024-05-15T23:37:13.004231+0000","last_active":"2024-05-15T23:38:06.955900+0000","last_peered":"2024-05-15T23:38:06.955900+0000","last_clean":"2024-05-15T23:38:06.955900+0000","last_became_active":"2024-05-15T23:34:01.801345+0000","last_became_peered":"2024-05-15T23:34:01.801345+0000","last_unstale":"2024-05-15T23:38:06.955900+0000","last_undegraded":"2024-05-15T23:38:06.955900+0000","last_fullsized":"2024-05-15T23:38:06.955900+0000","mapping_epoch":654,"log_start":"676'7000","ondisk_log_start":"676'7000","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'13817","last_scrub_stamp":"2024-05-15T23:37:13.004123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:13.004123+0000","objects_scrubbed":4537,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:19.464099+0000","scrub_duration":690,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85721088,"num_objects":1308,"num_object_clones":0,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17028,"num_write_kb":586752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"725'16991","reported_seq":17275,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-05-15T23:38:06.928362+0000","last_change":"2024-05-15T23:37:35.218638+0000","last_active":"2024-05-15T23:38:06.928362+0000","last_peered":"2024-05-15T23:38:06.928362+0000","last_clean":"2024-05-15T23:38:06.928362+0000","last_became_active":"2024-05-15T23:34:01.803171+0000","last_became_peered":"2024-05-15T23:34:01.803171+0000","last_unstale":"2024-05-15T23:38:06.928362+0000","last_undegraded":"2024-05-15T23:38:06.928362+0000","last_fullsized":"2024-05-15T23:38:06.928362+0000","mapping_epoch":654,"log_start":"676'6900","ondisk_log_start":"676'6900","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'15038","last_scrub_stamp":"2024-05-15T23:37:35.218511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:35.218511+0000","objects_scrubbed":3337,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:37.435272+0000","scrub_duration":561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89849856,"num_objects":1371,"num_object_clones":0,"num_object_copies":2742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16991,"num_write_kb":587584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"724'16783","reported_seq":17061,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-05-15T23:38:06.336155+0000","last_change":"2024-05-15T23:37:18.618832+0000","last_active":"2024-05-15T23:38:06.336155+0000","last_peered":"2024-05-15T23:38:06.336155+0000","last_clean":"2024-05-15T23:38:06.336155+0000","last_became_active":"2024-05-15T23:34:01.803136+0000","last_became_peered":"2024-05-15T23:34:01.803136+0000","last_unstale":"2024-05-15T23:38:06.336155+0000","last_undegraded":"2024-05-15T23:38:06.336155+0000","last_fullsized":"2024-05-15T23:38:06.336155+0000","mapping_epoch":654,"log_start":"674'6700","ondisk_log_start":"674'6700","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14069","last_scrub_stamp":"2024-05-15T23:37:18.618655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:18.618655+0000","objects_scrubbed":4063,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:29.384529+0000","scrub_duration":734,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86966272,"num_objects":1327,"num_object_clones":0,"num_object_copies":2654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16783,"num_write_kb":579520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1565,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-05-15T23:38:01.449037+0000","last_change":"2024-05-15T23:37:17.136503+0000","last_active":"2024-05-15T23:38:01.449037+0000","last_peered":"2024-05-15T23:38:01.449037+0000","last_clean":"2024-05-15T23:38:01.449037+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:38:01.449037+0000","last_undegraded":"2024-05-15T23:38:01.449037+0000","last_fullsized":"2024-05-15T23:38:01.449037+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:37:17.136378+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:37:17.136378+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:22.171455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1403977752,"num_objects":21424,"num_object_clones":0,"num_object_copies":42848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":309,"num_objects_unfound":0,"num_objects_dirty":21424,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272448,"num_write_kb":9403841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16555,"num_bytes_recovered":1084948480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3946782720,"data_stored":3946774576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351389,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1268252,"kb_used_data":434516,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":92476900,"statfs":{"total":95995035648,"available":94696345600,"internally_reserved":0,"allocated":444944384,"data_stored":444536558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":853604804},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580321,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940240,"kb_used_data":363336,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92804912,"statfs":{"total":95995035648,"available":95032229888,"internally_reserved":0,"allocated":372056064,"data_stored":371655882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":590608832},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776553,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":982224,"kb_used_data":283272,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":92762928,"statfs":{"total":95995035648,"available":94989238272,"internally_reserved":0,"allocated":290070528,"data_stored":289670346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":715717056},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":520,"seq":2233382994057,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":341340,"kb_used_data":236372,"kb_used_omap":24,"kb_used_meta":104935,"kb_avail":93403812,"statfs":{"total":95995035648,"available":95645503488,"internally_reserved":0,"allocated":242044928,"data_stored":241638721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25357,"internal_metadata":107453683},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267122,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1258076,"kb_used_data":631316,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92487076,"statfs":{"total":95995035648,"available":94706765824,"internally_reserved":0,"allocated":646467584,"data_stored":646059778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":641792458},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568496056,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575314,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1619920,"kb_used_data":808520,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92125232,"statfs":{"total":95995035648,"available":94336237568,"internally_reserved":0,"allocated":827924480,"data_stored":827524298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":830863814},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016723,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483411,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":489680,"kb_used_data":282440,"kb_used_omap":4,"kb_used_meta":207227,"kb_avail":93255472,"statfs":{"total":95995035648,"available":95493603328,"internally_reserved":0,"allocated":289218560,"data_stored":288818378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":212201155},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150153,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1062480,"kb_used_data":178888,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":92682672,"statfs":{"total":95995035648,"available":94907056128,"internally_reserved":0,"allocated":183181312,"data_stored":182781130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":904788423},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986624,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1022416,"kb_used_data":355400,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":92722736,"statfs":{"total":95995035648,"available":94948081664,"internally_reserved":0,"allocated":363929600,"data_stored":363529418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790397,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1056032,"kb_used_data":343188,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":92689120,"statfs":{"total":95995035648,"available":94913658880,"internally_reserved":0,"allocated":351424512,"data_stored":351018305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":823656448,"data_stored":823656448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":286392320,"data_stored":286392320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":179634176,"data_stored":179634176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":355336192,"data_stored":355336192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":348585984,"data_stored":348585984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":640094208,"data_stored":640090136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":240779264,"data_stored":240779264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":286457856,"data_stored":286457856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":363790336,"data_stored":363790336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":422055936,"data_stored":422051864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:38:13.258 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:38:13.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:13.366 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:13.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:13.515 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:13.543 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:13.564 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:13.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:13.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:13.661 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:13 smithi090 ceph-mon[44434]: pgmap v1932: 17 pgs: 1 active+recovering, 16 active+clean; 1.0 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s; 309/33648 objects misplaced (0.918%); 6.6 MiB/s, 105 objects/s recovering 2024-05-15T23:38:13.668 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:38:13.698 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:13.726 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:38:13.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:13.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:38:13.761 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:13 smithi084 ceph-mon[39784]: pgmap v1932: 17 pgs: 1 active+recovering, 16 active+clean; 1.0 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s; 309/33648 objects misplaced (0.918%); 6.6 MiB/s, 105 objects/s recovering 2024-05-15T23:38:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:13 smithi088 ceph-mon[44699]: pgmap v1932: 17 pgs: 1 active+recovering, 16 active+clean; 1.0 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s; 309/33648 objects misplaced (0.918%); 6.6 MiB/s, 105 objects/s recovering 2024-05-15T23:38:13.840 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:38:13.870 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:13.901 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:38:13.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:13.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:38:14.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:38:14.057 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:14.092 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:38:14.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:14.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:38:14.202 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:38:14.235 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:14.259 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:38:14.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:14.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:38:14.363 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:38:14.395 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:14.418 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:38:14.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:14.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:38:14.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:14.566 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:14.604 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:14.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:14.713 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:38:14.742 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:14 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:38:14.742 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:14 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:38:14.745 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:14.946 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:38:14.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:14.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:38:15.053 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:38:15.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:14 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:38:15.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:14 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:38:15.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:14 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:38:15.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:14 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:38:15.083 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:15.104 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:38:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:15.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:38:15.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:38:15.267 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:15.304 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:38:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:15.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:38:15.408 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:15.436 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:15.459 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:15.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:15.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:15.493 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:38:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-05-15T23:38:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-05-15T23:38:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-05-15T23:38:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:74911 7.4 7:2b8b218d:::benchmark_data_smithi090_143373_object74910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e566)", 2024-05-15T23:38:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:29:32.827895+0000", 2024-05-15T23:38:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 521.28218675400001, 2024-05-15T23:38:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.56446615499999997, 2024-05-15T23:38:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.493 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 74911 2024-05-15T23:38:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.827895+0000", 2024-05-15T23:38:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.827895+0000", 2024-05-15T23:38:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.827896+0000", 2024-05-15T23:38:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2699999999999998e-07 2024-05-15T23:38:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.827950+0000", 2024-05-15T23:38:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4614000000000003e-05 2024-05-15T23:38:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.827959+0000", 2024-05-15T23:38:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3559999999999993e-06 2024-05-15T23:38:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.827964+0000", 2024-05-15T23:38:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1710000000000001e-06 2024-05-15T23:38:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828795+0000", 2024-05-15T23:38:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00083052099999999997 2024-05-15T23:38:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828798+0000", 2024-05-15T23:38:15.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5540000000000002e-06 2024-05-15T23:38:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.355542+0000", 2024-05-15T23:38:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.52674400099999996 2024-05-15T23:38:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.355657+0000", 2024-05-15T23:38:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114431 2024-05-15T23:38:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:38:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.355770+0000", 2024-05-15T23:38:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113174 2024-05-15T23:38:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.366773+0000", 2024-05-15T23:38:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011003647 2024-05-15T23:38:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.390259+0000", 2024-05-15T23:38:15.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.02348563 2024-05-15T23:38:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.390315+0000", 2024-05-15T23:38:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6098999999999999e-05 2024-05-15T23:38:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.392361+0000", 2024-05-15T23:38:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020461300000000002 2024-05-15T23:38:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.500 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:74930 7.4 7:20f92e45:::benchmark_data_smithi090_143373_object74929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e566)", 2024-05-15T23:38:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:29:32.827981+0000", 2024-05-15T23:38:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 521.28210111400006, 2024-05-15T23:38:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.56619529700000004, 2024-05-15T23:38:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.501 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 74930 2024-05-15T23:38:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.827981+0000", 2024-05-15T23:38:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.827981+0000", 2024-05-15T23:38:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.827981+0000", 2024-05-15T23:38:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2200000000000003e-07 2024-05-15T23:38:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828054+0000", 2024-05-15T23:38:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2621999999999997e-05 2024-05-15T23:38:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828060+0000", 2024-05-15T23:38:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1340000000000001e-06 2024-05-15T23:38:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828066+0000", 2024-05-15T23:38:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4659999999999998e-06 2024-05-15T23:38:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828817+0000", 2024-05-15T23:38:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075177999999999996 2024-05-15T23:38:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828821+0000", 2024-05-15T23:38:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8920000000000002e-06 2024-05-15T23:38:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.358413+0000", 2024-05-15T23:38:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.52959214399999999 2024-05-15T23:38:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.358576+0000", 2024-05-15T23:38:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000162788 2024-05-15T23:38:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:38:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.358756+0000", 2024-05-15T23:38:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000180092 2024-05-15T23:38:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.366786+0000", 2024-05-15T23:38:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080298780000000007 2024-05-15T23:38:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.392433+0000", 2024-05-15T23:38:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025646873000000001 2024-05-15T23:38:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.392494+0000", 2024-05-15T23:38:15.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0980000000000002e-05 2024-05-15T23:38:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.394176+0000", 2024-05-15T23:38:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001681926 2024-05-15T23:38:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:74938 7.4 7:27c4493b:::benchmark_data_smithi090_143373_object74937:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e566)", 2024-05-15T23:38:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:29:32.828080+0000", 2024-05-15T23:38:15.508 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 521.28200179800001, 2024-05-15T23:38:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.57289789899999999, 2024-05-15T23:38:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 74938 2024-05-15T23:38:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828080+0000", 2024-05-15T23:38:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828080+0000", 2024-05-15T23:38:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828081+0000", 2024-05-15T23:38:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4000000000000001e-07 2024-05-15T23:38:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828163+0000", 2024-05-15T23:38:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1916999999999995e-05 2024-05-15T23:38:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828169+0000", 2024-05-15T23:38:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6100000000000002e-06 2024-05-15T23:38:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828174+0000", 2024-05-15T23:38:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4129999999999999e-06 2024-05-15T23:38:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828840+0000", 2024-05-15T23:38:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066651199999999996 2024-05-15T23:38:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828844+0000", 2024-05-15T23:38:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.517e-06 2024-05-15T23:38:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.360754+0000", 2024-05-15T23:38:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.53190996599999996 2024-05-15T23:38:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.360881+0000", 2024-05-15T23:38:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012711299999999999 2024-05-15T23:38:15.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:38:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.361015+0000", 2024-05-15T23:38:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013427799999999999 2024-05-15T23:38:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.371370+0000", 2024-05-15T23:38:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01035466 2024-05-15T23:38:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.398875+0000", 2024-05-15T23:38:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027505732000000001 2024-05-15T23:38:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.398931+0000", 2024-05-15T23:38:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5682e-05 2024-05-15T23:38:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.400978+0000", 2024-05-15T23:38:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020468589999999998 2024-05-15T23:38:15.515 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:74954 7.4 7:2fce5d89:::benchmark_data_smithi090_143373_object74953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e566)", 2024-05-15T23:38:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:29:32.828190+0000", 2024-05-15T23:38:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 521.28189202999999, 2024-05-15T23:38:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.57424599899999995, 2024-05-15T23:38:15.516 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 74954 2024-05-15T23:38:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828190+0000", 2024-05-15T23:38:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828190+0000", 2024-05-15T23:38:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828191+0000", 2024-05-15T23:38:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.102e-06 2024-05-15T23:38:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828253+0000", 2024-05-15T23:38:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2216000000000002e-05 2024-05-15T23:38:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828260+0000", 2024-05-15T23:38:15.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9829999999999999e-06 2024-05-15T23:38:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828264+0000", 2024-05-15T23:38:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3479999999999997e-06 2024-05-15T23:38:15.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828862+0000", 2024-05-15T23:38:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00059808500000000004 2024-05-15T23:38:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828866+0000", 2024-05-15T23:38:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5999999999999998e-06 2024-05-15T23:38:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.371418+0000", 2024-05-15T23:38:15.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.54255162999999995 2024-05-15T23:38:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.371578+0000", 2024-05-15T23:38:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016067499999999999 2024-05-15T23:38:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:38:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.371746+0000", 2024-05-15T23:38:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016718899999999999 2024-05-15T23:38:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.382608+0000", 2024-05-15T23:38:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010862790000000001 2024-05-15T23:38:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.401041+0000", 2024-05-15T23:38:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018433050999999999 2024-05-15T23:38:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.401098+0000", 2024-05-15T23:38:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6632000000000003e-05 2024-05-15T23:38:15.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.402436+0000", 2024-05-15T23:38:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001337698 2024-05-15T23:38:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:74955 7.4 7:281390c0:::benchmark_data_smithi090_143373_object74954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e566)", 2024-05-15T23:38:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:29:32.828269+0000", 2024-05-15T23:38:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 521.28181244999996, 2024-05-15T23:38:15.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.59213755599999995, 2024-05-15T23:38:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 74955 2024-05-15T23:38:15.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828269+0000", 2024-05-15T23:38:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828269+0000", 2024-05-15T23:38:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828270+0000", 2024-05-15T23:38:15.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6400000000000003e-07 2024-05-15T23:38:15.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828316+0000", 2024-05-15T23:38:15.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6233999999999998e-05 2024-05-15T23:38:15.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828320+0000", 2024-05-15T23:38:15.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9489999999999998e-06 2024-05-15T23:38:15.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828323+0000", 2024-05-15T23:38:15.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0620000000000001e-06 2024-05-15T23:38:15.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828884+0000", 2024-05-15T23:38:15.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00056118200000000002 2024-05-15T23:38:15.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828888+0000", 2024-05-15T23:38:15.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.478e-06 2024-05-15T23:38:15.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.376980+0000", 2024-05-15T23:38:15.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.54809250099999995 2024-05-15T23:38:15.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.377149+0000", 2024-05-15T23:38:15.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016831699999999999 2024-05-15T23:38:15.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:38:15.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.377332+0000", 2024-05-15T23:38:15.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018318300000000001 2024-05-15T23:38:15.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.387600+0000", 2024-05-15T23:38:15.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010268368999999999 2024-05-15T23:38:15.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.417493+0000", 2024-05-15T23:38:15.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.029892572999999999 2024-05-15T23:38:15.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.417550+0000", 2024-05-15T23:38:15.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7494999999999998e-05 2024-05-15T23:38:15.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.420407+0000", 2024-05-15T23:38:15.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002856749 2024-05-15T23:38:15.534 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.534 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.534 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.534 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:74987 7.4 7:2a714f6d:::benchmark_data_smithi090_143373_object74986:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e566)", 2024-05-15T23:38:15.534 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:29:32.828340+0000", 2024-05-15T23:38:15.534 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 521.28174216100001, 2024-05-15T23:38:15.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.59395777100000002, 2024-05-15T23:38:15.535 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.535 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.535 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.535 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.535 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.535 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.535 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 74987 2024-05-15T23:38:15.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.535 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828340+0000", 2024-05-15T23:38:15.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828340+0000", 2024-05-15T23:38:15.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828340+0000", 2024-05-15T23:38:15.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7199999999999999e-07 2024-05-15T23:38:15.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828386+0000", 2024-05-15T23:38:15.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5670000000000002e-05 2024-05-15T23:38:15.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828405+0000", 2024-05-15T23:38:15.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8950999999999999e-05 2024-05-15T23:38:15.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828418+0000", 2024-05-15T23:38:15.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3464e-05 2024-05-15T23:38:15.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828908+0000", 2024-05-15T23:38:15.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00048965300000000005 2024-05-15T23:38:15.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828911+0000", 2024-05-15T23:38:15.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5700000000000001e-06 2024-05-15T23:38:15.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.379809+0000", 2024-05-15T23:38:15.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.55089798300000004 2024-05-15T23:38:15.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.379939+0000", 2024-05-15T23:38:15.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000129369 2024-05-15T23:38:15.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:38:15.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.380084+0000", 2024-05-15T23:38:15.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145585 2024-05-15T23:38:15.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.392378+0000", 2024-05-15T23:38:15.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012293379 2024-05-15T23:38:15.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.420511+0000", 2024-05-15T23:38:15.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028132832 2024-05-15T23:38:15.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.420566+0000", 2024-05-15T23:38:15.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5482000000000002e-05 2024-05-15T23:38:15.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.422297+0000", 2024-05-15T23:38:15.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017313610000000001 2024-05-15T23:38:15.543 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.543 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.543 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.543 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:74996 7.4 7:2c4ff30f:::benchmark_data_smithi090_143373_object74995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e566)", 2024-05-15T23:38:15.543 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:29:32.828425+0000", 2024-05-15T23:38:15.543 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 521.28165649499999, 2024-05-15T23:38:15.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.59533426300000003, 2024-05-15T23:38:15.543 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.543 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.544 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.544 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.544 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.544 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.544 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 74996 2024-05-15T23:38:15.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.544 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828425+0000", 2024-05-15T23:38:15.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828425+0000", 2024-05-15T23:38:15.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828426+0000", 2024-05-15T23:38:15.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1699999999999997e-07 2024-05-15T23:38:15.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828477+0000", 2024-05-15T23:38:15.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1412000000000002e-05 2024-05-15T23:38:15.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828482+0000", 2024-05-15T23:38:15.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7840000000000003e-06 2024-05-15T23:38:15.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828486+0000", 2024-05-15T23:38:15.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6399999999999999e-06 2024-05-15T23:38:15.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828924+0000", 2024-05-15T23:38:15.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00043855899999999999 2024-05-15T23:38:15.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:32.828928+0000", 2024-05-15T23:38:15.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4910000000000001e-06 2024-05-15T23:38:15.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.387644+0000", 2024-05-15T23:38:15.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.55871632800000004 2024-05-15T23:38:15.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.387760+0000", 2024-05-15T23:38:15.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000115739 2024-05-15T23:38:15.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 9", 2024-05-15T23:38:15.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.387870+0000", 2024-05-15T23:38:15.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110231 2024-05-15T23:38:15.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.396575+0000", 2024-05-15T23:38:15.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087044489999999995 2024-05-15T23:38:15.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.422362+0000", 2024-05-15T23:38:15.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.025787094 2024-05-15T23:38:15.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.422429+0000", 2024-05-15T23:38:15.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6798999999999995e-05 2024-05-15T23:38:15.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:29:33.423760+0000", 2024-05-15T23:38:15.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00133102 2024-05-15T23:38:15.550 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.550 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.550 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.551 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:260260 7.b 7:de4f5d47:::benchmark_data_smithi090_143373_object114089:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e624)", 2024-05-15T23:38:15.551 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:32:53.273269+0000", 2024-05-15T23:38:15.551 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 320.83681252700001, 2024-05-15T23:38:15.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.56331034700000004, 2024-05-15T23:38:15.551 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.551 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.551 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.551 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.551 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.551 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.552 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260260 2024-05-15T23:38:15.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.552 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273269+0000", 2024-05-15T23:38:15.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273269+0000", 2024-05-15T23:38:15.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273270+0000", 2024-05-15T23:38:15.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0399999999999996e-07 2024-05-15T23:38:15.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273272+0000", 2024-05-15T23:38:15.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6199999999999999e-06 2024-05-15T23:38:15.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273278+0000", 2024-05-15T23:38:15.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6269999999999997e-06 2024-05-15T23:38:15.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273284+0000", 2024-05-15T23:38:15.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7880000000000003e-06 2024-05-15T23:38:15.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273380+0000", 2024-05-15T23:38:15.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5832e-05 2024-05-15T23:38:15.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273384+0000", 2024-05-15T23:38:15.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7919999999999999e-06 2024-05-15T23:38:15.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.825042+0000", 2024-05-15T23:38:15.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.55165879799999995 2024-05-15T23:38:15.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.825118+0000", 2024-05-15T23:38:15.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5685000000000005e-05 2024-05-15T23:38:15.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,7", 2024-05-15T23:38:15.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.825207+0000", 2024-05-15T23:38:15.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9291999999999998e-05 2024-05-15T23:38:15.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.836303+0000", 2024-05-15T23:38:15.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011095403 2024-05-15T23:38:15.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.836372+0000", 2024-05-15T23:38:15.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9697999999999996e-05 2024-05-15T23:38:15.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.836400+0000", 2024-05-15T23:38:15.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7750999999999999e-05 2024-05-15T23:38:15.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.836455+0000", 2024-05-15T23:38:15.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4880000000000003e-05 2024-05-15T23:38:15.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.836580+0000", 2024-05-15T23:38:15.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012457700000000001 2024-05-15T23:38:15.558 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.558 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.558 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.559 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:260277 7.b 7:d27f329a:::benchmark_data_smithi090_143373_object114106:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e624)", 2024-05-15T23:38:15.559 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:32:53.273307+0000", 2024-05-15T23:38:15.559 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 320.836774703, 2024-05-15T23:38:15.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.57352379799999997, 2024-05-15T23:38:15.559 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.559 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.559 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.559 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.559 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.559 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.560 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260277 2024-05-15T23:38:15.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.560 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273307+0000", 2024-05-15T23:38:15.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273307+0000", 2024-05-15T23:38:15.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273308+0000", 2024-05-15T23:38:15.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8899999999999999e-07 2024-05-15T23:38:15.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273311+0000", 2024-05-15T23:38:15.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9229999999999998e-06 2024-05-15T23:38:15.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273317+0000", 2024-05-15T23:38:15.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.793e-06 2024-05-15T23:38:15.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273323+0000", 2024-05-15T23:38:15.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8109999999999999e-06 2024-05-15T23:38:15.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273418+0000", 2024-05-15T23:38:15.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4600999999999998e-05 2024-05-15T23:38:15.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273422+0000", 2024-05-15T23:38:15.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0910000000000003e-06 2024-05-15T23:38:15.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.836621+0000", 2024-05-15T23:38:15.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.56319852400000003 2024-05-15T23:38:15.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.836745+0000", 2024-05-15T23:38:15.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012404899999999999 2024-05-15T23:38:15.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,7", 2024-05-15T23:38:15.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.836877+0000", 2024-05-15T23:38:15.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013213499999999999 2024-05-15T23:38:15.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.846566+0000", 2024-05-15T23:38:15.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096889779999999991 2024-05-15T23:38:15.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.846634+0000", 2024-05-15T23:38:15.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7929999999999995e-05 2024-05-15T23:38:15.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.846652+0000", 2024-05-15T23:38:15.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.806e-05 2024-05-15T23:38:15.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.846711+0000", 2024-05-15T23:38:15.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8943000000000001e-05 2024-05-15T23:38:15.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.846831+0000", 2024-05-15T23:38:15.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120271 2024-05-15T23:38:15.566 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.566 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.567 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.567 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:260301 7.b 7:dd22a92e:::benchmark_data_smithi090_143373_object114130:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e624)", 2024-05-15T23:38:15.567 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:32:53.273343+0000", 2024-05-15T23:38:15.567 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 320.83673884299998, 2024-05-15T23:38:15.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.57373485899999999, 2024-05-15T23:38:15.567 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.568 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.568 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.568 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.568 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.568 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.568 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260301 2024-05-15T23:38:15.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.568 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273343+0000", 2024-05-15T23:38:15.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273343+0000", 2024-05-15T23:38:15.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273344+0000", 2024-05-15T23:38:15.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8899999999999999e-07 2024-05-15T23:38:15.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273347+0000", 2024-05-15T23:38:15.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0249999999999998e-06 2024-05-15T23:38:15.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273353+0000", 2024-05-15T23:38:15.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6800000000000004e-06 2024-05-15T23:38:15.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273359+0000", 2024-05-15T23:38:15.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8930000000000002e-06 2024-05-15T23:38:15.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273442+0000", 2024-05-15T23:38:15.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2610999999999999e-05 2024-05-15T23:38:15.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273445+0000", 2024-05-15T23:38:15.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.641e-06 2024-05-15T23:38:15.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.841439+0000", 2024-05-15T23:38:15.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.56799324100000004 2024-05-15T23:38:15.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.841557+0000", 2024-05-15T23:38:15.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011851800000000001 2024-05-15T23:38:15.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,7", 2024-05-15T23:38:15.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.841666+0000", 2024-05-15T23:38:15.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000109071 2024-05-15T23:38:15.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.846876+0000", 2024-05-15T23:38:15.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0052094029999999996 2024-05-15T23:38:15.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.846889+0000", 2024-05-15T23:38:15.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3592e-05 2024-05-15T23:38:15.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.846929+0000", 2024-05-15T23:38:15.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9477000000000002e-05 2024-05-15T23:38:15.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.846979+0000", 2024-05-15T23:38:15.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0512e-05 2024-05-15T23:38:15.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.847078+0000", 2024-05-15T23:38:15.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8605999999999997e-05 2024-05-15T23:38:15.575 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.575 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.575 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.575 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:260318 7.b 7:d3ac58a6:::benchmark_data_smithi090_143373_object114147:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e624)", 2024-05-15T23:38:15.575 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:32:53.273382+0000", 2024-05-15T23:38:15.575 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 320.83669972400003, 2024-05-15T23:38:15.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.58913567499999997, 2024-05-15T23:38:15.575 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.576 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.576 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.576 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.576 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.576 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.576 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260318 2024-05-15T23:38:15.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.576 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273382+0000", 2024-05-15T23:38:15.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273382+0000", 2024-05-15T23:38:15.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273383+0000", 2024-05-15T23:38:15.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6799999999999999e-07 2024-05-15T23:38:15.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273386+0000", 2024-05-15T23:38:15.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.337e-06 2024-05-15T23:38:15.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273402+0000", 2024-05-15T23:38:15.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6036e-05 2024-05-15T23:38:15.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273409+0000", 2024-05-15T23:38:15.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4999999999999996e-06 2024-05-15T23:38:15.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273465+0000", 2024-05-15T23:38:15.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6535e-05 2024-05-15T23:38:15.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273469+0000", 2024-05-15T23:38:15.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.721e-06 2024-05-15T23:38:15.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.847112+0000", 2024-05-15T23:38:15.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.57364335300000002 2024-05-15T23:38:15.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.847232+0000", 2024-05-15T23:38:15.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000119496 2024-05-15T23:38:15.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,7", 2024-05-15T23:38:15.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.847361+0000", 2024-05-15T23:38:15.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012903799999999999 2024-05-15T23:38:15.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.851888+0000", 2024-05-15T23:38:15.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045274219999999997 2024-05-15T23:38:15.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.861102+0000", 2024-05-15T23:38:15.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0092132370000000009 2024-05-15T23:38:15.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.862408+0000", 2024-05-15T23:38:15.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0013067090000000001 2024-05-15T23:38:15.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.862447+0000", 2024-05-15T23:38:15.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8396000000000003e-05 2024-05-15T23:38:15.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.862518+0000", 2024-05-15T23:38:15.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1126999999999998e-05 2024-05-15T23:38:15.583 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.583 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.583 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.583 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:260348 7.b 7:d1be7b2e:::benchmark_data_smithi090_143373_object114177:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e624)", 2024-05-15T23:38:15.583 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:32:53.273417+0000", 2024-05-15T23:38:15.583 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 320.83666526600001, 2024-05-15T23:38:15.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.58926874299999998, 2024-05-15T23:38:15.584 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.584 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.584 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.584 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.584 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.584 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.584 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260348 2024-05-15T23:38:15.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.584 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273417+0000", 2024-05-15T23:38:15.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273417+0000", 2024-05-15T23:38:15.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273417+0000", 2024-05-15T23:38:15.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2099999999999996e-07 2024-05-15T23:38:15.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273420+0000", 2024-05-15T23:38:15.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0359999999999998e-06 2024-05-15T23:38:15.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273426+0000", 2024-05-15T23:38:15.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1029999999999998e-06 2024-05-15T23:38:15.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273432+0000", 2024-05-15T23:38:15.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7259999999999999e-06 2024-05-15T23:38:15.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273488+0000", 2024-05-15T23:38:15.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5872999999999999e-05 2024-05-15T23:38:15.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273492+0000", 2024-05-15T23:38:15.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5559999999999999e-06 2024-05-15T23:38:15.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.851942+0000", 2024-05-15T23:38:15.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.57845026700000002 2024-05-15T23:38:15.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.852081+0000", 2024-05-15T23:38:15.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000139648 2024-05-15T23:38:15.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,7", 2024-05-15T23:38:15.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.852208+0000", 2024-05-15T23:38:15.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012676899999999999 2024-05-15T23:38:15.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.862530+0000", 2024-05-15T23:38:15.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010321813000000001 2024-05-15T23:38:15.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.862565+0000", 2024-05-15T23:38:15.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4547e-05 2024-05-15T23:38:15.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.862598+0000", 2024-05-15T23:38:15.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3036e-05 2024-05-15T23:38:15.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.862628+0000", 2024-05-15T23:38:15.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0238000000000001e-05 2024-05-15T23:38:15.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.862685+0000", 2024-05-15T23:38:15.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7410000000000001e-05 2024-05-15T23:38:15.591 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.591 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.591 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.591 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:260353 7.b 7:d4539185:::benchmark_data_smithi090_143373_object114182:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e624)", 2024-05-15T23:38:15.592 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:32:53.273443+0000", 2024-05-15T23:38:15.592 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 320.83663905399999, 2024-05-15T23:38:15.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.58939907599999997, 2024-05-15T23:38:15.592 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.592 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.592 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.592 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260353 2024-05-15T23:38:15.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.592 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273443+0000", 2024-05-15T23:38:15.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273443+0000", 2024-05-15T23:38:15.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273443+0000", 2024-05-15T23:38:15.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7700000000000004e-07 2024-05-15T23:38:15.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273446+0000", 2024-05-15T23:38:15.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.543e-06 2024-05-15T23:38:15.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273451+0000", 2024-05-15T23:38:15.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2650000000000004e-06 2024-05-15T23:38:15.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273459+0000", 2024-05-15T23:38:15.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4399999999999999e-06 2024-05-15T23:38:15.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273510+0000", 2024-05-15T23:38:15.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1790999999999999e-05 2024-05-15T23:38:15.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273514+0000", 2024-05-15T23:38:15.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5300000000000001e-06 2024-05-15T23:38:15.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.856969+0000", 2024-05-15T23:38:15.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.58345509699999998 2024-05-15T23:38:15.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.857114+0000", 2024-05-15T23:38:15.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000145029 2024-05-15T23:38:15.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,7", 2024-05-15T23:38:15.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.857264+0000", 2024-05-15T23:38:15.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000149772 2024-05-15T23:38:15.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.862695+0000", 2024-05-15T23:38:15.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054317080000000004 2024-05-15T23:38:15.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.862727+0000", 2024-05-15T23:38:15.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1430999999999998e-05 2024-05-15T23:38:15.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.862762+0000", 2024-05-15T23:38:15.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4749999999999998e-05 2024-05-15T23:38:15.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.862790+0000", 2024-05-15T23:38:15.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8422999999999999e-05 2024-05-15T23:38:15.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.862842+0000", 2024-05-15T23:38:15.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1719999999999999e-05 2024-05-15T23:38:15.599 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.599 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.599 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.599 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:260361 7.b 7:d80c8594:::benchmark_data_smithi090_143373_object114190:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e624)", 2024-05-15T23:38:15.600 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:32:53.273466+0000", 2024-05-15T23:38:15.600 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 320.836615865, 2024-05-15T23:38:15.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.60681571700000003, 2024-05-15T23:38:15.600 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.600 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.600 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.600 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.600 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.600 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.600 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260361 2024-05-15T23:38:15.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.600 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273466+0000", 2024-05-15T23:38:15.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273466+0000", 2024-05-15T23:38:15.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273466+0000", 2024-05-15T23:38:15.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6100000000000001e-07 2024-05-15T23:38:15.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273469+0000", 2024-05-15T23:38:15.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7870000000000002e-06 2024-05-15T23:38:15.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273476+0000", 2024-05-15T23:38:15.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4880000000000004e-06 2024-05-15T23:38:15.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273482+0000", 2024-05-15T23:38:15.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6980000000000001e-06 2024-05-15T23:38:15.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273533+0000", 2024-05-15T23:38:15.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0112999999999999e-05 2024-05-15T23:38:15.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273536+0000", 2024-05-15T23:38:15.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4470000000000002e-06 2024-05-15T23:38:15.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.862872+0000", 2024-05-15T23:38:15.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.58933552700000003 2024-05-15T23:38:15.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.862946+0000", 2024-05-15T23:38:15.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4037999999999998e-05 2024-05-15T23:38:15.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,7", 2024-05-15T23:38:15.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.863019+0000", 2024-05-15T23:38:15.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3868999999999994e-05 2024-05-15T23:38:15.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.871194+0000", 2024-05-15T23:38:15.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081748340000000006 2024-05-15T23:38:15.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.871270+0000", 2024-05-15T23:38:15.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5395999999999994e-05 2024-05-15T23:38:15.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.880105+0000", 2024-05-15T23:38:15.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0088353319999999996 2024-05-15T23:38:15.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.880176+0000", 2024-05-15T23:38:15.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1218999999999997e-05 2024-05-15T23:38:15.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.880282+0000", 2024-05-15T23:38:15.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105408 2024-05-15T23:38:15.607 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.607 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.607 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.607 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:260375 7.b 7:db332fd9:::benchmark_data_smithi090_143373_object114204:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e624)", 2024-05-15T23:38:15.607 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:32:53.273489+0000", 2024-05-15T23:38:15.607 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 320.83659299599998, 2024-05-15T23:38:15.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.607099151, 2024-05-15T23:38:15.608 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.608 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.608 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.608 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.608 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.608 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.608 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260375 2024-05-15T23:38:15.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.608 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273489+0000", 2024-05-15T23:38:15.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273489+0000", 2024-05-15T23:38:15.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273489+0000", 2024-05-15T23:38:15.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.68e-07 2024-05-15T23:38:15.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273492+0000", 2024-05-15T23:38:15.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5390000000000002e-06 2024-05-15T23:38:15.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273497+0000", 2024-05-15T23:38:15.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1549999999999998e-06 2024-05-15T23:38:15.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273503+0000", 2024-05-15T23:38:15.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.84e-06 2024-05-15T23:38:15.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273555+0000", 2024-05-15T23:38:15.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2296000000000002e-05 2024-05-15T23:38:15.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273559+0000", 2024-05-15T23:38:15.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3330000000000001e-06 2024-05-15T23:38:15.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.866899+0000", 2024-05-15T23:38:15.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.59334006100000003 2024-05-15T23:38:15.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.867000+0000", 2024-05-15T23:38:15.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101692 2024-05-15T23:38:15.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,7", 2024-05-15T23:38:15.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.867088+0000", 2024-05-15T23:38:15.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7925999999999998e-05 2024-05-15T23:38:15.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.871260+0000", 2024-05-15T23:38:15.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00417206 2024-05-15T23:38:15.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.880378+0000", 2024-05-15T23:38:15.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0091179739999999992 2024-05-15T23:38:15.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.880426+0000", 2024-05-15T23:38:15.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8060999999999999e-05 2024-05-15T23:38:15.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.880492+0000", 2024-05-15T23:38:15.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5395999999999995e-05 2024-05-15T23:38:15.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.880588+0000", 2024-05-15T23:38:15.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6249999999999995e-05 2024-05-15T23:38:15.615 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.615 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.615 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.615 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:260388 7.b 7:d59184f2:::benchmark_data_smithi090_143373_object114217:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e624)", 2024-05-15T23:38:15.615 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:32:53.273509+0000", 2024-05-15T23:38:15.615 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 320.83657256599997, 2024-05-15T23:38:15.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.62155916, 2024-05-15T23:38:15.615 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.616 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.616 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.616 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.616 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.616 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.616 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260388 2024-05-15T23:38:15.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.616 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273509+0000", 2024-05-15T23:38:15.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273509+0000", 2024-05-15T23:38:15.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273510+0000", 2024-05-15T23:38:15.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6000000000000004e-07 2024-05-15T23:38:15.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273512+0000", 2024-05-15T23:38:15.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4509999999999999e-06 2024-05-15T23:38:15.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273518+0000", 2024-05-15T23:38:15.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1940000000000003e-06 2024-05-15T23:38:15.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273524+0000", 2024-05-15T23:38:15.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2959999999999998e-06 2024-05-15T23:38:15.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273579+0000", 2024-05-15T23:38:15.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4772999999999999e-05 2024-05-15T23:38:15.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273582+0000", 2024-05-15T23:38:15.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.501e-06 2024-05-15T23:38:15.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.871304+0000", 2024-05-15T23:38:15.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.59772159599999997 2024-05-15T23:38:15.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.871434+0000", 2024-05-15T23:38:15.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013026199999999999 2024-05-15T23:38:15.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,7", 2024-05-15T23:38:15.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.871561+0000", 2024-05-15T23:38:15.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000127362 2024-05-15T23:38:15.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.880363+0000", 2024-05-15T23:38:15.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0088013629999999995 2024-05-15T23:38:15.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.880666+0000", 2024-05-15T23:38:15.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000303851 2024-05-15T23:38:15.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.894917+0000", 2024-05-15T23:38:15.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014250406 2024-05-15T23:38:15.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.894960+0000", 2024-05-15T23:38:15.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3118000000000003e-05 2024-05-15T23:38:15.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895068+0000", 2024-05-15T23:38:15.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000108427 2024-05-15T23:38:15.623 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.623 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.623 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.624 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:260401 7.b 7:d8658415:::benchmark_data_smithi090_143373_object114230:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e624)", 2024-05-15T23:38:15.624 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:32:53.273530+0000", 2024-05-15T23:38:15.624 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 320.83655203199999, 2024-05-15T23:38:15.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.62166882700000003, 2024-05-15T23:38:15.624 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.624 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.624 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.624 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.624 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.624 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.624 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260401 2024-05-15T23:38:15.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.625 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273530+0000", 2024-05-15T23:38:15.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273530+0000", 2024-05-15T23:38:15.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273530+0000", 2024-05-15T23:38:15.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3000000000000001e-07 2024-05-15T23:38:15.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273533+0000", 2024-05-15T23:38:15.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7329999999999999e-06 2024-05-15T23:38:15.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273538+0000", 2024-05-15T23:38:15.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0889999999999999e-06 2024-05-15T23:38:15.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273543+0000", 2024-05-15T23:38:15.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2850000000000002e-06 2024-05-15T23:38:15.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273601+0000", 2024-05-15T23:38:15.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7352999999999998e-05 2024-05-15T23:38:15.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273604+0000", 2024-05-15T23:38:15.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.405e-06 2024-05-15T23:38:15.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.876034+0000", 2024-05-15T23:38:15.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.60242976699999995 2024-05-15T23:38:15.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.876109+0000", 2024-05-15T23:38:15.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5164000000000001e-05 2024-05-15T23:38:15.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,7", 2024-05-15T23:38:15.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.876181+0000", 2024-05-15T23:38:15.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1966999999999997e-05 2024-05-15T23:38:15.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.880648+0000", 2024-05-15T23:38:15.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044669339999999997 2024-05-15T23:38:15.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.880675+0000", 2024-05-15T23:38:15.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6913000000000001e-05 2024-05-15T23:38:15.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895111+0000", 2024-05-15T23:38:15.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01443643 2024-05-15T23:38:15.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895144+0000", 2024-05-15T23:38:15.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2288e-05 2024-05-15T23:38:15.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895199+0000", 2024-05-15T23:38:15.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4969000000000002e-05 2024-05-15T23:38:15.631 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.631 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.631 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.632 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:260404 7.b 7:df48844c:::benchmark_data_smithi090_143373_object114233:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e624)", 2024-05-15T23:38:15.632 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:32:53.273551+0000", 2024-05-15T23:38:15.632 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 320.83653097799998, 2024-05-15T23:38:15.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.62182125200000005, 2024-05-15T23:38:15.632 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.632 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.632 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.632 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.632 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.632 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.632 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260404 2024-05-15T23:38:15.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.633 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273551+0000", 2024-05-15T23:38:15.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273551+0000", 2024-05-15T23:38:15.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273551+0000", 2024-05-15T23:38:15.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9699999999999996e-07 2024-05-15T23:38:15.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273554+0000", 2024-05-15T23:38:15.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4789999999999999e-06 2024-05-15T23:38:15.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273559+0000", 2024-05-15T23:38:15.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5860000000000004e-06 2024-05-15T23:38:15.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273565+0000", 2024-05-15T23:38:15.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2769999999999996e-06 2024-05-15T23:38:15.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273622+0000", 2024-05-15T23:38:15.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7546000000000001e-05 2024-05-15T23:38:15.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273626+0000", 2024-05-15T23:38:15.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4769999999999999e-06 2024-05-15T23:38:15.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.880715+0000", 2024-05-15T23:38:15.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.60708927899999998 2024-05-15T23:38:15.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.880829+0000", 2024-05-15T23:38:15.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011410599999999999 2024-05-15T23:38:15.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,7", 2024-05-15T23:38:15.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.880955+0000", 2024-05-15T23:38:15.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012564499999999999 2024-05-15T23:38:15.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.894853+0000", 2024-05-15T23:38:15.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013897853 2024-05-15T23:38:15.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895209+0000", 2024-05-15T23:38:15.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00035635799999999998 2024-05-15T23:38:15.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895286+0000", 2024-05-15T23:38:15.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7124000000000002e-05 2024-05-15T23:38:15.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895319+0000", 2024-05-15T23:38:15.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2645999999999999e-05 2024-05-15T23:38:15.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895372+0000", 2024-05-15T23:38:15.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3378999999999998e-05 2024-05-15T23:38:15.639 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.639 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.639 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.640 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:260406 7.b 7:d5fb46d7:::benchmark_data_smithi090_143373_object114235:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e624)", 2024-05-15T23:38:15.640 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:32:53.273579+0000", 2024-05-15T23:38:15.640 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 320.83650313999999, 2024-05-15T23:38:15.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.62196866799999995, 2024-05-15T23:38:15.640 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.640 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.640 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.640 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.640 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.640 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.640 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260406 2024-05-15T23:38:15.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.641 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273579+0000", 2024-05-15T23:38:15.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273579+0000", 2024-05-15T23:38:15.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273579+0000", 2024-05-15T23:38:15.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8400000000000004e-07 2024-05-15T23:38:15.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273582+0000", 2024-05-15T23:38:15.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.661e-06 2024-05-15T23:38:15.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273589+0000", 2024-05-15T23:38:15.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8129999999999998e-06 2024-05-15T23:38:15.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273595+0000", 2024-05-15T23:38:15.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9900000000000002e-06 2024-05-15T23:38:15.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273643+0000", 2024-05-15T23:38:15.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8523999999999998e-05 2024-05-15T23:38:15.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273647+0000", 2024-05-15T23:38:15.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3469999999999999e-06 2024-05-15T23:38:15.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.885765+0000", 2024-05-15T23:38:15.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.61211885099999996 2024-05-15T23:38:15.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.885882+0000", 2024-05-15T23:38:15.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011620699999999999 2024-05-15T23:38:15.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,7", 2024-05-15T23:38:15.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.885992+0000", 2024-05-15T23:38:15.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110549 2024-05-15T23:38:15.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895217+0000", 2024-05-15T23:38:15.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0092248190000000004 2024-05-15T23:38:15.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895253+0000", 2024-05-15T23:38:15.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5865000000000003e-05 2024-05-15T23:38:15.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895458+0000", 2024-05-15T23:38:15.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000204933 2024-05-15T23:38:15.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895491+0000", 2024-05-15T23:38:15.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3526999999999999e-05 2024-05-15T23:38:15.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895547+0000", 2024-05-15T23:38:15.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5998000000000002e-05 2024-05-15T23:38:15.647 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.647 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.647 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.648 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.16764.0:260421 7.b 7:d39183d1:::benchmark_data_smithi090_143373_object114250:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e624)", 2024-05-15T23:38:15.648 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-05-15T23:32:53.273601+0000", 2024-05-15T23:38:15.648 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 320.83648036900001, 2024-05-15T23:38:15.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.62219092600000003, 2024-05-15T23:38:15.648 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-05-15T23:38:15.648 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-05-15T23:38:15.648 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:38:15.648 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-05-15T23:38:15.648 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.16764", 2024-05-15T23:38:15.648 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:38:15.649 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 260421 2024-05-15T23:38:15.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.649 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-05-15T23:38:15.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-05-15T23:38:15.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273601+0000", 2024-05-15T23:38:15.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-05-15T23:38:15.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273601+0000", 2024-05-15T23:38:15.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-05-15T23:38:15.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-05-15T23:38:15.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273602+0000", 2024-05-15T23:38:15.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7100000000000002e-07 2024-05-15T23:38:15.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-05-15T23:38:15.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273605+0000", 2024-05-15T23:38:15.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.57e-06 2024-05-15T23:38:15.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-05-15T23:38:15.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273610+0000", 2024-05-15T23:38:15.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6799999999999998e-06 2024-05-15T23:38:15.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-05-15T23:38:15.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273616+0000", 2024-05-15T23:38:15.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4489999999999998e-06 2024-05-15T23:38:15.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273659+0000", 2024-05-15T23:38:15.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3545999999999999e-05 2024-05-15T23:38:15.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-05-15T23:38:15.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.273663+0000", 2024-05-15T23:38:15.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.484e-06 2024-05-15T23:38:15.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-05-15T23:38:15.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.890630+0000", 2024-05-15T23:38:15.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.61696769900000004 2024-05-15T23:38:15.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-05-15T23:38:15.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.890739+0000", 2024-05-15T23:38:15.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000108541 2024-05-15T23:38:15.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,7", 2024-05-15T23:38:15.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.890865+0000", 2024-05-15T23:38:15.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000126122 2024-05-15T23:38:15.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-05-15T23:38:15.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895382+0000", 2024-05-15T23:38:15.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045165459999999998 2024-05-15T23:38:15.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895422+0000", 2024-05-15T23:38:15.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0672e-05 2024-05-15T23:38:15.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:38:15.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895618+0000", 2024-05-15T23:38:15.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000195804 2024-05-15T23:38:15.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-05-15T23:38:15.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895681+0000", 2024-05-15T23:38:15.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2915000000000003e-05 2024-05-15T23:38:15.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-05-15T23:38:15.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-05-15T23:38:15.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-05-15T23:38:15.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-05-15T23:32:53.895792+0000", 2024-05-15T23:38:15.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111327 2024-05-15T23:38:15.655 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.655 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.655 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.655 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-05-15T23:38:15.655 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-05-15T23:38:15.656 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:38:15.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:38:15.704 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:15.710 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:15.744 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:38:15.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:15.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:38:15.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:15 smithi088 ceph-mon[44699]: osdmap e726: 12 total, 12 up, 10 in 2024-05-15T23:38:15.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:15 smithi088 ceph-mon[44699]: pgmap v1934: 17 pgs: 17 active+clean; 966 MiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.13k op/s; 5.7 MiB/s, 91 objects/s recovering 2024-05-15T23:38:15.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:15 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1855500489' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:38:15.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:15.852 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:15.885 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:15.907 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:15.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:15.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:16.013 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:38:16.041 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:16.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:15 smithi090 ceph-mon[44434]: osdmap e726: 12 total, 12 up, 10 in 2024-05-15T23:38:16.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:15 smithi090 ceph-mon[44434]: pgmap v1934: 17 pgs: 17 active+clean; 966 MiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.13k op/s; 5.7 MiB/s, 91 objects/s recovering 2024-05-15T23:38:16.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:15 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1855500489' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:38:16.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:16.063 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:38:16.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:16.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:38:16.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:15 smithi084 ceph-mon[39784]: osdmap e726: 12 total, 12 up, 10 in 2024-05-15T23:38:16.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:15 smithi084 ceph-mon[39784]: pgmap v1934: 17 pgs: 17 active+clean; 966 MiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.13k op/s; 5.7 MiB/s, 91 objects/s recovering 2024-05-15T23:38:16.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:15 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1855500489' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:38:16.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:16.169 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:38:16.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T23:38:16.199 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:16.220 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:38:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:16.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:38:16.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:38:16.390 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:38:16.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:16.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:38:16.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:38:16.540 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:38:16.540 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:38:16.540 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:38:16.540 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:38:16.540 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:38:16.540 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:38:16.541 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:38:16.541 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:38:16.541 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:38:16.541 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:38:16.541 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:38:16.541 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:38:16.541 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:38:16.541 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:38:16.541 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:38:16.541 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:38:16.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:38:16.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:38:16.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:38:16.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:38:16.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:38:16.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:38:16.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:38:16.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:38:16.564 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:16.740 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:16.859 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:16.859 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:16.859 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:16.859 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:16.859 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:16.859 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:16.859 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:16.859 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:16.860 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.615 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:38:17.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:17.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:38:17.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:17.761 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:17.785 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:17.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:17.794 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:17.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:17.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:17 smithi084 ceph-mon[39784]: pgmap v1935: 17 pgs: 17 active+clean; 817 MiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 4.8 MiB/s, 77 objects/s recovering 2024-05-15T23:38:17.894 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:38:17.933 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:18.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:18.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:18.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:17 smithi090 ceph-mon[44434]: pgmap v1935: 17 pgs: 17 active+clean; 817 MiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 4.8 MiB/s, 77 objects/s recovering 2024-05-15T23:38:18.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:18.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:18.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:17 smithi088 ceph-mon[44699]: pgmap v1935: 17 pgs: 17 active+clean; 817 MiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 4.8 MiB/s, 77 objects/s recovering 2024-05-15T23:38:18.085 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:38:18.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:18.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:38:18.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:38:18.229 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:18.260 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:38:18.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:18.266 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:38:18.368 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:38:18.400 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:18.423 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:38:18.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:18.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:38:18.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:18.581 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:18.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:18.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:38:18.777 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:18.800 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:38:18.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:18.806 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:38:18.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:38:18.948 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:18.987 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:38:18.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:18.997 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:38:19.097 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:38:19.124 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:18 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/916111071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:38:19.127 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:19.154 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:38:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:19.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:38:19.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:38:19.294 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:19.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:18 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/916111071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:38:19.324 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:38:19.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:19.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:38:19.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:18 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/916111071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:38:19.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:38:19.474 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:19.521 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:38:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:19.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:38:19.624 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:19.651 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:19.671 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:19.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:19.775 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:38:19.809 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:19 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:38:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:19 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:38:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/916111071' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:38:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:19 smithi088 ceph-mon[44699]: osdmap e727: 12 total, 12 up, 10 in 2024-05-15T23:38:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:19 smithi088 ceph-mon[44699]: 8.6 scrub starts 2024-05-15T23:38:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:19 smithi088 ceph-mon[44699]: 8.6 scrub ok 2024-05-15T23:38:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:19 smithi088 ceph-mon[44699]: pgmap v1937: 17 pgs: 17 active+clean; 675 MiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s; 2.4 MiB/s, 38 objects/s recovering 2024-05-15T23:38:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:19 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:38:20.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:19 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:38:20.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:19 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:38:20.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:19 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:38:20.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/916111071' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:38:20.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:19 smithi084 ceph-mon[39784]: osdmap e727: 12 total, 12 up, 10 in 2024-05-15T23:38:20.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:19 smithi084 ceph-mon[39784]: 8.6 scrub starts 2024-05-15T23:38:20.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:19 smithi084 ceph-mon[39784]: 8.6 scrub ok 2024-05-15T23:38:20.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:19 smithi084 ceph-mon[39784]: pgmap v1937: 17 pgs: 17 active+clean; 675 MiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s; 2.4 MiB/s, 38 objects/s recovering 2024-05-15T23:38:20.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:19 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:38:20.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:19 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:38:20.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:19 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:38:20.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:19 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:38:20.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/916111071' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:38:20.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:19 smithi090 ceph-mon[44434]: osdmap e727: 12 total, 12 up, 10 in 2024-05-15T23:38:20.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:19 smithi090 ceph-mon[44434]: 8.6 scrub starts 2024-05-15T23:38:20.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:19 smithi090 ceph-mon[44434]: 8.6 scrub ok 2024-05-15T23:38:20.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:19 smithi090 ceph-mon[44434]: pgmap v1937: 17 pgs: 17 active+clean; 675 MiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s; 2.4 MiB/s, 38 objects/s recovering 2024-05-15T23:38:20.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:19 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:38:20.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:19 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:38:21.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:20 smithi090 ceph-mon[44434]: osdmap e728: 12 total, 12 up, 10 in 2024-05-15T23:38:21.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:20 smithi088 ceph-mon[44699]: osdmap e728: 12 total, 12 up, 10 in 2024-05-15T23:38:21.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:20 smithi084 ceph-mon[39784]: osdmap e728: 12 total, 12 up, 10 in 2024-05-15T23:38:21.731 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:38:22.301 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:21 smithi090 ceph-mon[44434]: 8.b scrub starts 2024-05-15T23:38:22.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:21 smithi090 ceph-mon[44434]: 8.b scrub ok 2024-05-15T23:38:22.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:21 smithi090 ceph-mon[44434]: pgmap v1939: 17 pgs: 17 active+clean; 634 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 899 op/s 2024-05-15T23:38:22.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:21 smithi088 ceph-mon[44699]: 8.b scrub starts 2024-05-15T23:38:22.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:21 smithi088 ceph-mon[44699]: 8.b scrub ok 2024-05-15T23:38:22.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:21 smithi088 ceph-mon[44699]: pgmap v1939: 17 pgs: 17 active+clean; 634 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 899 op/s 2024-05-15T23:38:22.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:21 smithi084 ceph-mon[39784]: 8.b scrub starts 2024-05-15T23:38:22.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:21 smithi084 ceph-mon[39784]: 8.b scrub ok 2024-05-15T23:38:22.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:21 smithi084 ceph-mon[39784]: pgmap v1939: 17 pgs: 17 active+clean; 634 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 899 op/s 2024-05-15T23:38:22.469 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:38:22.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:22.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:38:22.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:38:22.641 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:22.679 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:38:22.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:22.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:38:22.787 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:38:22.818 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:22.890 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:22 smithi090 ceph-mon[44434]: 8.1 deep-scrub starts 2024-05-15T23:38:22.890 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:22 smithi090 ceph-mon[44434]: osdmap e729: 12 total, 12 up, 10 in 2024-05-15T23:38:23.020 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:38:23.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:23.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:38:23.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:22 smithi084 ceph-mon[39784]: 8.1 deep-scrub starts 2024-05-15T23:38:23.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:22 smithi084 ceph-mon[39784]: osdmap e729: 12 total, 12 up, 10 in 2024-05-15T23:38:23.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:38:23.173 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:23.211 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:38:23.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:23.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:38:23.314 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:38:23.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:22 smithi088 ceph-mon[44699]: 8.1 deep-scrub starts 2024-05-15T23:38:23.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:22 smithi088 ceph-mon[44699]: osdmap e729: 12 total, 12 up, 10 in 2024-05-15T23:38:23.345 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:23.366 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:38:23.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:23.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:38:23.471 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:23.500 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:23.521 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:23.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:23.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:23.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:38:23.679 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:23.722 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:38:23.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:23.727 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:38:23.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:38:23.866 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:23.899 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:38:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:23.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:38:24.003 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:38:24.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:23 smithi090 ceph-mon[44434]: 8.3 deep-scrub starts 2024-05-15T23:38:24.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:23 smithi090 ceph-mon[44434]: 8.a scrub starts 2024-05-15T23:38:24.040 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:23 smithi090 ceph-mon[44434]: 8.a scrub ok 2024-05-15T23:38:24.041 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:23 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:38:24.041 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:23 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:38:24.041 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:23 smithi090 ceph-mon[44434]: pgmap v1941: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 388 MiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:38:24.042 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:24.073 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:38:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:24.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:38:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:23 smithi084 ceph-mon[39784]: 8.3 deep-scrub starts 2024-05-15T23:38:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:23 smithi084 ceph-mon[39784]: 8.a scrub starts 2024-05-15T23:38:24.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:23 smithi084 ceph-mon[39784]: 8.a scrub ok 2024-05-15T23:38:24.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:23 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:38:24.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:23 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:38:24.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:23 smithi084 ceph-mon[39784]: pgmap v1941: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 388 MiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:38:24.184 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:38:24.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:23 smithi088 ceph-mon[44699]: 8.3 deep-scrub starts 2024-05-15T23:38:24.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:23 smithi088 ceph-mon[44699]: 8.a scrub starts 2024-05-15T23:38:24.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:23 smithi088 ceph-mon[44699]: 8.a scrub ok 2024-05-15T23:38:24.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:23 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:38:24.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:23 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:38:24.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:23 smithi088 ceph-mon[44699]: pgmap v1941: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 388 MiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-05-15T23:38:24.217 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:24.235 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:24.236 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:38:24.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:24.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:38:24.340 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:38:24.368 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:24.388 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:38:24.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:24.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:38:24.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:38:24.507 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:38:24.524 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:24.548 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:38:24.548 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:38:24.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:24.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:38:24.551 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:38:24.652 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:24.681 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:24.703 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:24.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:24.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:24.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:24.850 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:24.880 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:24.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:24.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:24.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:38:25.034 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:25.073 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:38:25.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:25.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:38:25.182 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:25.210 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:25.236 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:25.242 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:24 smithi088 ceph-mon[44699]: 8.1 deep-scrub ok 2024-05-15T23:38:25.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:25.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:25.305 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:24 smithi090 ceph-mon[44434]: 8.1 deep-scrub ok 2024-05-15T23:38:25.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:25.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:24 smithi084 ceph-mon[39784]: 8.1 deep-scrub ok 2024-05-15T23:38:25.396 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:25.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:25.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:25.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:38:25.575 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:25.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:38:26.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:25 smithi084 ceph-mon[39784]: from='client.37313 v1:172.21.15.84:0/3224871834' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:38:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:25 smithi084 ceph-mon[39784]: 8.3 deep-scrub ok 2024-05-15T23:38:26.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:25 smithi084 ceph-mon[39784]: pgmap v1942: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 316 MiB data, 7.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 994 op/s 2024-05-15T23:38:26.231 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:38:26.241 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:26.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:26.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:38:26.265 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1941,"stamp":"2024-05-15T23:38:23.315571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":406389304,"num_objects":6195,"num_object_clones":0,"num_object_copies":12390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":287802,"num_write_kb":9405991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16870,"num_bytes_recovered":1106785888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160897,"ondisk_log_size":160897,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":8170112,"kb_used_data":1935704,"kb_used_omap":41,"kb_used_meta":6234070,"kb_avail":929281408,"statfs":{"total":959950356480,"available":951584161792,"internally_reserved":0,"allocated":1982160896,"data_stored":1977927708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42533,"internal_metadata":6383688155},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-450494464,"num_objects":-6874,"num_object_clones":0,"num_object_copies":-13748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6874,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000990"},"pg_stats":[{"pgid":"8.f","version":"729'17605","reported_seq":17901,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-05-15T23:38:22.845565+0000","last_change":"2024-05-15T23:37:17.929589+0000","last_active":"2024-05-15T23:38:22.845565+0000","last_peered":"2024-05-15T23:38:22.845565+0000","last_clean":"2024-05-15T23:38:22.845565+0000","last_became_active":"2024-05-15T23:34:01.801268+0000","last_became_peered":"2024-05-15T23:34:01.801268+0000","last_unstale":"2024-05-15T23:38:22.845565+0000","last_undegraded":"2024-05-15T23:38:22.845565+0000","last_fullsized":"2024-05-15T23:38:22.845565+0000","mapping_epoch":654,"log_start":"680'7600","ondisk_log_start":"680'7600","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'13799","last_scrub_stamp":"2024-05-15T23:37:17.929445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:17.929445+0000","objects_scrubbed":4116,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:37.474269+0000","scrub_duration":824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18939904,"num_objects":289,"num_object_clones":0,"num_object_copies":578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17605,"num_write_kb":572608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.e","version":"728'17572","reported_seq":34574,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-05-15T23:38:19.969816+0000","last_change":"2024-05-15T23:38:11.874799+0000","last_active":"2024-05-15T23:38:19.969816+0000","last_peered":"2024-05-15T23:38:19.969816+0000","last_clean":"2024-05-15T23:38:19.969816+0000","last_became_active":"2024-05-15T23:35:33.794835+0000","last_became_peered":"2024-05-15T23:35:33.794835+0000","last_unstale":"2024-05-15T23:38:19.969816+0000","last_undegraded":"2024-05-15T23:38:19.969816+0000","last_fullsized":"2024-05-15T23:38:19.969816+0000","mapping_epoch":685,"log_start":"678'7500","ondisk_log_start":"678'7500","created":654,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"725'17063","last_scrub_stamp":"2024-05-15T23:38:11.874645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:38:11.874645+0000","objects_scrubbed":997,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:39:23.219445+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31850496,"num_objects":486,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17572,"num_write_kb":577856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8410,"num_bytes_recovered":551157760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"729'18190","reported_seq":18489,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-05-15T23:38:22.886602+0000","last_change":"2024-05-15T23:37:13.985832+0000","last_active":"2024-05-15T23:38:22.886602+0000","last_peered":"2024-05-15T23:38:22.886602+0000","last_clean":"2024-05-15T23:38:22.886602+0000","last_became_active":"2024-05-15T23:34:01.993980+0000","last_became_peered":"2024-05-15T23:34:01.993980+0000","last_unstale":"2024-05-15T23:38:22.886602+0000","last_undegraded":"2024-05-15T23:38:22.886602+0000","last_fullsized":"2024-05-15T23:38:22.886602+0000","mapping_epoch":654,"log_start":"681'8100","ondisk_log_start":"681'8100","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14037","last_scrub_stamp":"2024-05-15T23:37:13.985583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:13.985583+0000","objects_scrubbed":4477,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:30.662338+0000","scrub_duration":973,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19202072,"num_objects":294,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":294,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18190,"num_write_kb":591425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"728'17876","reported_seq":18244,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-05-15T23:38:20.092440+0000","last_change":"2024-05-15T23:37:12.677782+0000","last_active":"2024-05-15T23:38:20.092440+0000","last_peered":"2024-05-15T23:38:20.092440+0000","last_clean":"2024-05-15T23:38:20.092440+0000","last_became_active":"2024-05-15T23:34:01.805322+0000","last_became_peered":"2024-05-15T23:34:01.805322+0000","last_unstale":"2024-05-15T23:38:20.092440+0000","last_undegraded":"2024-05-15T23:38:20.092440+0000","last_fullsized":"2024-05-15T23:38:20.092440+0000","mapping_epoch":654,"log_start":"680'7800","ondisk_log_start":"680'7800","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'13803","last_scrub_stamp":"2024-05-15T23:37:12.677680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:12.677680+0000","objects_scrubbed":4570,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:21.090707+0000","scrub_duration":855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31457280,"num_objects":480,"num_object_clones":0,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17876,"num_write_kb":587392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"727'17647","reported_seq":34798,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-05-15T23:38:19.198637+0000","last_change":"2024-05-15T23:38:12.386534+0000","last_active":"2024-05-15T23:38:19.198637+0000","last_peered":"2024-05-15T23:38:19.198637+0000","last_clean":"2024-05-15T23:38:19.198637+0000","last_became_active":"2024-05-15T23:35:33.804849+0000","last_became_peered":"2024-05-15T23:35:33.804849+0000","last_unstale":"2024-05-15T23:38:19.198637+0000","last_undegraded":"2024-05-15T23:38:19.198637+0000","last_fullsized":"2024-05-15T23:38:19.198637+0000","mapping_epoch":685,"log_start":"680'7600","ondisk_log_start":"680'7600","created":654,"last_epoch_clean":725,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'7203","last_scrub_stamp":"2024-05-15T23:35:13.697763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:35:13.697763+0000","objects_scrubbed":7163,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37683200,"num_objects":575,"num_object_clones":0,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17647,"num_write_kb":583104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8454,"num_bytes_recovered":554041344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"727'17929","reported_seq":18168,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-05-15T23:38:19.209545+0000","last_change":"2024-05-15T23:35:14.555676+0000","last_active":"2024-05-15T23:38:19.209545+0000","last_peered":"2024-05-15T23:38:19.209545+0000","last_clean":"2024-05-15T23:38:19.209545+0000","last_became_active":"2024-05-15T23:34:01.800290+0000","last_became_peered":"2024-05-15T23:34:01.800290+0000","last_unstale":"2024-05-15T23:38:19.209545+0000","last_undegraded":"2024-05-15T23:38:19.209545+0000","last_fullsized":"2024-05-15T23:38:19.209545+0000","mapping_epoch":654,"log_start":"680'7900","ondisk_log_start":"680'7900","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'7418","last_scrub_stamp":"2024-05-15T23:35:14.555528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:35:14.555528+0000","objects_scrubbed":7384,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33488896,"num_objects":511,"num_object_clones":0,"num_object_copies":1022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17929,"num_write_kb":590080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"728'18253","reported_seq":18544,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-05-15T23:38:22.254595+0000","last_change":"2024-05-15T23:37:17.558370+0000","last_active":"2024-05-15T23:38:22.254595+0000","last_peered":"2024-05-15T23:38:22.254595+0000","last_clean":"2024-05-15T23:38:22.254595+0000","last_became_active":"2024-05-15T23:34:01.802958+0000","last_became_peered":"2024-05-15T23:34:01.802958+0000","last_unstale":"2024-05-15T23:38:22.254595+0000","last_undegraded":"2024-05-15T23:38:22.254595+0000","last_fullsized":"2024-05-15T23:38:22.254595+0000","mapping_epoch":654,"log_start":"681'8200","ondisk_log_start":"681'8200","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14323","last_scrub_stamp":"2024-05-15T23:37:17.558198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:17.558198+0000","objects_scrubbed":4282,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:31.112904+0000","scrub_duration":663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21823488,"num_objects":333,"num_object_clones":0,"num_object_copies":666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18253,"num_write_kb":594752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"729'18296","reported_seq":18600,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-05-15T23:38:22.909820+0000","last_change":"2024-05-15T23:37:20.443433+0000","last_active":"2024-05-15T23:38:22.909820+0000","last_peered":"2024-05-15T23:38:22.909820+0000","last_clean":"2024-05-15T23:38:22.909820+0000","last_became_active":"2024-05-15T23:34:01.801309+0000","last_became_peered":"2024-05-15T23:34:01.801309+0000","last_unstale":"2024-05-15T23:38:22.909820+0000","last_undegraded":"2024-05-15T23:38:22.909820+0000","last_fullsized":"2024-05-15T23:38:22.909820+0000","mapping_epoch":654,"log_start":"681'8200","ondisk_log_start":"681'8200","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14483","last_scrub_stamp":"2024-05-15T23:37:20.443208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:20.443208+0000","objects_scrubbed":4160,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:49.450668+0000","scrub_duration":1397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20316160,"num_objects":310,"num_object_clones":0,"num_object_copies":620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18296,"num_write_kb":595392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.6","version":"729'17889","reported_seq":18197,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-05-15T23:38:22.898739+0000","last_change":"2024-05-15T23:38:19.242117+0000","last_active":"2024-05-15T23:38:22.898739+0000","last_peered":"2024-05-15T23:38:22.898739+0000","last_clean":"2024-05-15T23:38:22.898739+0000","last_became_active":"2024-05-15T23:34:01.801861+0000","last_became_peered":"2024-05-15T23:34:01.801861+0000","last_unstale":"2024-05-15T23:38:22.898739+0000","last_undegraded":"2024-05-15T23:38:22.898739+0000","last_fullsized":"2024-05-15T23:38:22.898739+0000","mapping_epoch":654,"log_start":"680'7800","ondisk_log_start":"680'7800","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'17651","last_scrub_stamp":"2024-05-15T23:38:19.242005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:38:19.242005+0000","objects_scrubbed":549,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:39:31.148235+0000","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19726336,"num_objects":301,"num_object_clones":0,"num_object_copies":602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17889,"num_write_kb":582080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"728'18118","reported_seq":18411,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-05-15T23:38:21.393449+0000","last_change":"2024-05-15T23:37:22.767028+0000","last_active":"2024-05-15T23:38:21.393449+0000","last_peered":"2024-05-15T23:38:21.393449+0000","last_clean":"2024-05-15T23:38:21.393449+0000","last_became_active":"2024-05-15T23:34:02.002490+0000","last_became_peered":"2024-05-15T23:34:02.002490+0000","last_unstale":"2024-05-15T23:38:21.393449+0000","last_undegraded":"2024-05-15T23:38:21.393449+0000","last_fullsized":"2024-05-15T23:38:21.393449+0000","mapping_epoch":654,"log_start":"681'8100","ondisk_log_start":"681'8100","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'14430","last_scrub_stamp":"2024-05-15T23:37:22.766704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:22.766704+0000","objects_scrubbed":4080,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:25.000957+0000","scrub_duration":891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24903680,"num_objects":380,"num_object_clones":0,"num_object_copies":760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18118,"num_write_kb":591936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"728'18144","reported_seq":18441,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-05-15T23:38:22.245780+0000","last_change":"2024-05-15T23:37:18.603196+0000","last_active":"2024-05-15T23:38:22.245780+0000","last_peered":"2024-05-15T23:38:22.245780+0000","last_clean":"2024-05-15T23:38:22.245780+0000","last_became_active":"2024-05-15T23:34:01.800457+0000","last_became_peered":"2024-05-15T23:34:01.800457+0000","last_unstale":"2024-05-15T23:38:22.245780+0000","last_undegraded":"2024-05-15T23:38:22.245780+0000","last_fullsized":"2024-05-15T23:38:22.245780+0000","mapping_epoch":654,"log_start":"681'8100","ondisk_log_start":"681'8100","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14248","last_scrub_stamp":"2024-05-15T23:37:18.602932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:18.602932+0000","objects_scrubbed":4247,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:37.953179+0000","scrub_duration":726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22020096,"num_objects":336,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18144,"num_write_kb":591360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"729'18243","reported_seq":18546,"reported_epoch":729,"state":"active+clean+scrubbing+deep","last_fresh":"2024-05-15T23:38:22.900526+0000","last_change":"2024-05-15T23:38:22.900009+0000","last_active":"2024-05-15T23:38:22.900526+0000","last_peered":"2024-05-15T23:38:22.900526+0000","last_clean":"2024-05-15T23:38:22.900526+0000","last_became_active":"2024-05-15T23:34:01.801285+0000","last_became_peered":"2024-05-15T23:34:01.801285+0000","last_unstale":"2024-05-15T23:38:22.900526+0000","last_undegraded":"2024-05-15T23:38:22.900526+0000","last_fullsized":"2024-05-15T23:38:22.900526+0000","mapping_epoch":654,"log_start":"682'8200","ondisk_log_start":"682'8200","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14151","last_scrub_stamp":"2024-05-15T23:37:15.959223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:15.959223+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22216704,"num_objects":339,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18243,"num_write_kb":594624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.2","version":"728'18213","reported_seq":18593,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-05-15T23:38:21.374420+0000","last_change":"2024-05-15T23:37:27.724613+0000","last_active":"2024-05-15T23:38:21.374420+0000","last_peered":"2024-05-15T23:38:21.374420+0000","last_clean":"2024-05-15T23:38:21.374420+0000","last_became_active":"2024-05-15T23:34:02.002460+0000","last_became_peered":"2024-05-15T23:34:02.002460+0000","last_unstale":"2024-05-15T23:38:21.374420+0000","last_undegraded":"2024-05-15T23:38:21.374420+0000","last_fullsized":"2024-05-15T23:38:21.374420+0000","mapping_epoch":654,"log_start":"682'8200","ondisk_log_start":"682'8200","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'14872","last_scrub_stamp":"2024-05-15T23:37:27.724324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:27.724324+0000","objects_scrubbed":3814,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:29.437384+0000","scrub_duration":761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29818880,"num_objects":455,"num_object_clones":0,"num_object_copies":910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18213,"num_write_kb":597376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"728'18009","reported_seq":18311,"reported_epoch":728,"state":"active+clean+scrubbing+deep","last_fresh":"2024-05-15T23:38:21.979889+0000","last_change":"2024-05-15T23:38:21.824521+0000","last_active":"2024-05-15T23:38:21.979889+0000","last_peered":"2024-05-15T23:38:21.979889+0000","last_clean":"2024-05-15T23:38:21.979889+0000","last_became_active":"2024-05-15T23:34:01.801345+0000","last_became_peered":"2024-05-15T23:34:01.801345+0000","last_unstale":"2024-05-15T23:38:21.979889+0000","last_undegraded":"2024-05-15T23:38:21.979889+0000","last_fullsized":"2024-05-15T23:38:21.979889+0000","mapping_epoch":654,"log_start":"680'8000","ondisk_log_start":"680'8000","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'13817","last_scrub_stamp":"2024-05-15T23:37:13.004123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:13.004123+0000","objects_scrubbed":24,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":690,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23003136,"num_objects":351,"num_object_clones":0,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17985,"num_write_kb":586752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"728'17979","reported_seq":18269,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-05-15T23:38:21.984989+0000","last_change":"2024-05-15T23:37:35.218638+0000","last_active":"2024-05-15T23:38:21.984989+0000","last_peered":"2024-05-15T23:38:21.984989+0000","last_clean":"2024-05-15T23:38:21.984989+0000","last_became_active":"2024-05-15T23:34:01.803171+0000","last_became_peered":"2024-05-15T23:34:01.803171+0000","last_unstale":"2024-05-15T23:38:21.984989+0000","last_undegraded":"2024-05-15T23:38:21.984989+0000","last_fullsized":"2024-05-15T23:38:21.984989+0000","mapping_epoch":654,"log_start":"680'7900","ondisk_log_start":"680'7900","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'15038","last_scrub_stamp":"2024-05-15T23:37:35.218511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:35.218511+0000","objects_scrubbed":3337,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:37.435272+0000","scrub_duration":561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25100288,"num_objects":383,"num_object_clones":0,"num_object_copies":766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17979,"num_write_kb":587584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"728'17740","reported_seq":18026,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-05-15T23:38:21.393662+0000","last_change":"2024-05-15T23:37:18.618832+0000","last_active":"2024-05-15T23:38:21.393662+0000","last_peered":"2024-05-15T23:38:21.393662+0000","last_clean":"2024-05-15T23:38:21.393662+0000","last_became_active":"2024-05-15T23:34:01.803136+0000","last_became_peered":"2024-05-15T23:34:01.803136+0000","last_unstale":"2024-05-15T23:38:21.393662+0000","last_undegraded":"2024-05-15T23:38:21.393662+0000","last_fullsized":"2024-05-15T23:38:21.393662+0000","mapping_epoch":654,"log_start":"680'7700","ondisk_log_start":"680'7700","created":654,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'14069","last_scrub_stamp":"2024-05-15T23:37:18.618655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:34:00.786907+0000","last_clean_scrub_stamp":"2024-05-15T23:37:18.618655+0000","objects_scrubbed":4063,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:29.384529+0000","scrub_duration":734,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24248320,"num_objects":370,"num_object_clones":0,"num_object_copies":740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17740,"num_write_kb":579520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1571,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-05-15T23:38:18.866622+0000","last_change":"2024-05-15T23:37:17.136503+0000","last_active":"2024-05-15T23:38:18.866622+0000","last_peered":"2024-05-15T23:38:18.866622+0000","last_clean":"2024-05-15T23:38:18.866622+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:38:18.866622+0000","last_undegraded":"2024-05-15T23:38:18.866622+0000","last_fullsized":"2024-05-15T23:38:18.866622+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:37:17.136378+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:37:17.136378+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:38:22.171455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":405798936,"num_objects":6193,"num_object_clones":0,"num_object_copies":12386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6193,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":287679,"num_write_kb":9403841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16864,"num_bytes_recovered":1105199104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1927880704,"data_stored":1927872560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160803,"ondisk_log_size":160803,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351392,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":933144,"kb_used_data":99536,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":92812008,"statfs":{"total":95995035648,"available":95039496192,"internally_reserved":0,"allocated":101924864,"data_stored":101487013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":853604804},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580324,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":687808,"kb_used_data":111032,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93057344,"statfs":{"total":95995035648,"available":95290720256,"internally_reserved":0,"allocated":113696768,"data_stored":113272586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":590608832},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":817800,"kb_used_data":92280,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92927352,"statfs":{"total":95995035648,"available":95157608448,"internally_reserved":0,"allocated":94494720,"data_stored":94070538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":742717888},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":520,"seq":2233382994059,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":219000,"kb_used_data":105708,"kb_used_omap":24,"kb_used_meta":113255,"kb_avail":93526152,"statfs":{"total":95995035648,"available":95770779648,"internally_reserved":0,"allocated":108244992,"data_stored":107826785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25357,"internal_metadata":115973363},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267125,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1170948,"kb_used_data":540024,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92574204,"statfs":{"total":95995035648,"available":94795984896,"internally_reserved":0,"allocated":552984576,"data_stored":552558745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":646052298},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568496059,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575317,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1403776,"kb_used_data":596536,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92341376,"statfs":{"total":95995035648,"available":94557569024,"internally_reserved":0,"allocated":610852864,"data_stored":610428682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":826603974},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016726,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483414,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317112,"kb_used_data":95980,"kb_used_omap":4,"kb_used_meta":221115,"kb_avail":93428040,"statfs":{"total":95995035648,"available":95670312960,"internally_reserved":0,"allocated":98283520,"data_stored":97865313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":226422467},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150156,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":942852,"kb_used_data":50936,"kb_used_omap":1,"kb_used_meta":891902,"kb_avail":92802300,"statfs":{"total":95995035648,"available":95029555200,"internally_reserved":0,"allocated":52158464,"data_stored":51734282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":913308103},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986627,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":855528,"kb_used_data":152416,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92889624,"statfs":{"total":95995035648,"available":95118974976,"internally_reserved":0,"allocated":156073984,"data_stored":155661802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790400,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":822144,"kb_used_data":91256,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92923008,"statfs":{"total":95995035648,"available":95153160192,"internally_reserved":0,"allocated":93446144,"data_stored":93021962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":607584256,"data_stored":607584256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93454336,"data_stored":93454336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":48431104,"data_stored":48431104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":140640256,"data_stored":140640256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":89128960,"data_stored":89128960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":545918976,"data_stored":545914904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":106037248,"data_stored":106037248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":91291648,"data_stored":91291648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":104988672,"data_stored":104988672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":100405248,"data_stored":100401176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:38:26.266 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.3 is not active+clean 2024-05-15T23:38:26.266 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.3', 'version': "729'18243", 'reported_seq': 18546, 'reported_epoch': 729, 'state': 'active+clean+scrubbing+deep', 'last_fresh': '2024-05-15T23:38:22.900526+0000', 'last_change': '2024-05-15T23:38:22.900009+0000', 'last_active': '2024-05-15T23:38:22.900526+0000', 'last_peered': '2024-05-15T23:38:22.900526+0000', 'last_clean': '2024-05-15T23:38:22.900526+0000', 'last_became_active': '2024-05-15T23:34:01.801285+0000', 'last_became_peered': '2024-05-15T23:34:01.801285+0000', 'last_unstale': '2024-05-15T23:38:22.900526+0000', 'last_undegraded': '2024-05-15T23:38:22.900526+0000', 'last_fullsized': '2024-05-15T23:38:22.900526+0000', 'mapping_epoch': 654, 'log_start': "682'8200", 'ondisk_log_start': "682'8200", 'created': 654, 'last_epoch_clean': 655, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "712'14151", 'last_scrub_stamp': '2024-05-15T23:37:15.959223+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-15T23:34:00.786907+0000', 'last_clean_scrub_stamp': '2024-05-15T23:37:15.959223+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'deep scrubbing for 0s', 'scrub_duration': 889, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22216704, 'num_objects': 339, 'num_object_clones': 0, 'num_object_copies': 678, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18243, 'num_write_kb': 594624, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [11, 1], 'acting': [11, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 11, 'acting_primary': 11, 'purged_snaps': []} 2024-05-15T23:38:26.267 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.1 is not active+clean 2024-05-15T23:38:26.267 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.1', 'version': "728'18009", 'reported_seq': 18311, 'reported_epoch': 728, 'state': 'active+clean+scrubbing+deep', 'last_fresh': '2024-05-15T23:38:21.979889+0000', 'last_change': '2024-05-15T23:38:21.824521+0000', 'last_active': '2024-05-15T23:38:21.979889+0000', 'last_peered': '2024-05-15T23:38:21.979889+0000', 'last_clean': '2024-05-15T23:38:21.979889+0000', 'last_became_active': '2024-05-15T23:34:01.801345+0000', 'last_became_peered': '2024-05-15T23:34:01.801345+0000', 'last_unstale': '2024-05-15T23:38:21.979889+0000', 'last_undegraded': '2024-05-15T23:38:21.979889+0000', 'last_fullsized': '2024-05-15T23:38:21.979889+0000', 'mapping_epoch': 654, 'log_start': "680'8000", 'ondisk_log_start': "680'8000", 'created': 654, 'last_epoch_clean': 655, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "711'13817", 'last_scrub_stamp': '2024-05-15T23:37:13.004123+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-15T23:34:00.786907+0000', 'last_clean_scrub_stamp': '2024-05-15T23:37:13.004123+0000', 'objects_scrubbed': 24, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'deep scrubbing for 0s', 'scrub_duration': 690, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23003136, 'num_objects': 351, 'num_object_clones': 0, 'num_object_copies': 702, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17985, 'num_write_kb': 586752, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 9], 'acting': [1, 9], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-15T23:38:26.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph status --format=json 2024-05-15T23:38:26.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:25 smithi090 ceph-mon[44434]: from='client.37313 v1:172.21.15.84:0/3224871834' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:38:26.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:25 smithi090 ceph-mon[44434]: 8.3 deep-scrub ok 2024-05-15T23:38:26.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:25 smithi090 ceph-mon[44434]: pgmap v1942: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 316 MiB data, 7.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 994 op/s 2024-05-15T23:38:26.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:25 smithi088 ceph-mon[44699]: from='client.37313 v1:172.21.15.84:0/3224871834' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:38:26.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:25 smithi088 ceph-mon[44699]: 8.3 deep-scrub ok 2024-05-15T23:38:26.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:25 smithi088 ceph-mon[44699]: pgmap v1942: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 316 MiB data, 7.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 994 op/s 2024-05-15T23:38:26.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:38:26.411 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:26.452 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:38:26.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:26.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:38:26.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:38:26.606 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:26.635 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:38:26.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:26.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:38:26.741 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:38:26.773 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:26.795 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:38:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:26.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:38:26.903 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:38:26.949 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:38:26.949 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:38:26.950 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:27.439 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:38:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:27.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:38:27.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:38:27.577 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:28.040 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Removed 146935 objects 2024-05-15T23:38:28.040 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Clean up completed and total clean up time :162.873 2024-05-15T23:38:28.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T23:38:28.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:28 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:38:28.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:28 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:38:28.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:28 smithi090 ceph-mon[44434]: osdmap e730: 12 total, 12 up, 10 in 2024-05-15T23:38:28.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:28 smithi090 ceph-mon[44434]: pgmap v1944: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 188 MiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-05-15T23:38:28.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:28 smithi090 ceph-mon[44434]: osdmap e731: 12 total, 12 up, 10 in 2024-05-15T23:38:28.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:28 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:38:28.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:28 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:38:28.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:28 smithi088 ceph-mon[44699]: osdmap e730: 12 total, 12 up, 10 in 2024-05-15T23:38:28.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:28 smithi088 ceph-mon[44699]: pgmap v1944: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 188 MiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-05-15T23:38:28.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:28 smithi088 ceph-mon[44699]: osdmap e731: 12 total, 12 up, 10 in 2024-05-15T23:38:28.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:28 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:38:28.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:28 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:38:28.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:28 smithi084 ceph-mon[39784]: osdmap e730: 12 total, 12 up, 10 in 2024-05-15T23:38:28.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:28 smithi084 ceph-mon[39784]: pgmap v1944: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 188 MiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-05-15T23:38:28.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:28 smithi084 ceph-mon[39784]: osdmap e731: 12 total, 12 up, 10 in 2024-05-15T23:38:28.383 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:29.687 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:29.789 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:29 smithi084 ceph-mon[39784]: pgmap v1946: 17 pgs: 17 active+clean; 64 MiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 986 op/s 2024-05-15T23:38:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:29 smithi090 ceph-mon[44434]: pgmap v1946: 17 pgs: 17 active+clean; 64 MiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 986 op/s 2024-05-15T23:38:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:29 smithi088 ceph-mon[44699]: pgmap v1946: 17 pgs: 17 active+clean; 64 MiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 986 op/s 2024-05-15T23:38:30.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:30 smithi084 ceph-mon[39784]: 8.5 scrub starts 2024-05-15T23:38:30.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:30 smithi084 ceph-mon[39784]: 8.5 scrub ok 2024-05-15T23:38:30.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:30 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:38:30.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:30 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:38:30.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:30 smithi084 ceph-mon[39784]: osdmap e732: 12 total, 12 up, 10 in 2024-05-15T23:38:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:30 smithi090 ceph-mon[44434]: 8.5 scrub starts 2024-05-15T23:38:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:30 smithi090 ceph-mon[44434]: 8.5 scrub ok 2024-05-15T23:38:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:30 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:38:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:30 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:38:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:30 smithi090 ceph-mon[44434]: osdmap e732: 12 total, 12 up, 10 in 2024-05-15T23:38:30.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:30 smithi088 ceph-mon[44699]: 8.5 scrub starts 2024-05-15T23:38:30.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:30 smithi088 ceph-mon[44699]: 8.5 scrub ok 2024-05-15T23:38:30.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:30 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:38:30.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:30 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:38:30.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:30 smithi088 ceph-mon[44699]: osdmap e732: 12 total, 12 up, 10 in 2024-05-15T23:38:31.233 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:38:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:31.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:38:31.302 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-05-15T23:38:31.302 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-05-15T23:38:31.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:38:31.370 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:31.391 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:38:31.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:31.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:38:31.410 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:38:31.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:38:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:31.543 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:38:31.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:31.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:38:31.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:31 smithi084 ceph-mon[39784]: 8.9 scrub starts 2024-05-15T23:38:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:31 smithi084 ceph-mon[39784]: 8.9 scrub ok 2024-05-15T23:38:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/837476221' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:38:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:31 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:38:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:31 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:38:31.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:31 smithi084 ceph-mon[39784]: pgmap v1948: 17 pgs: 17 active+clean; 38 MiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 741 op/s 2024-05-15T23:38:31.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:31.673 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:31 smithi090 ceph-mon[44434]: 8.9 scrub starts 2024-05-15T23:38:31.673 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:31 smithi090 ceph-mon[44434]: 8.9 scrub ok 2024-05-15T23:38:31.673 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/837476221' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:38:31.673 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:31 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:38:31.673 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:31 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:38:31.673 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:31 smithi090 ceph-mon[44434]: pgmap v1948: 17 pgs: 17 active+clean; 38 MiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 741 op/s 2024-05-15T23:38:31.675 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:31.694 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:31.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:31.797 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:38:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:31 smithi088 ceph-mon[44699]: 8.9 scrub starts 2024-05-15T23:38:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:31 smithi088 ceph-mon[44699]: 8.9 scrub ok 2024-05-15T23:38:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/837476221' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:38:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:31 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:38:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:31 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:38:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:31 smithi088 ceph-mon[44699]: pgmap v1948: 17 pgs: 17 active+clean; 38 MiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 741 op/s 2024-05-15T23:38:31.825 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:32.650 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:32.675 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:38:32.678 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:32 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:38:32.678 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/837476221' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:38:32.678 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:32 smithi084 ceph-mon[39784]: osdmap e733: 12 total, 12 up, 10 in 2024-05-15T23:38:32.679 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:32 smithi084 ceph-mon[39784]: 8.d scrub starts 2024-05-15T23:38:32.679 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:32 smithi084 ceph-mon[39784]: 8.d scrub ok 2024-05-15T23:38:32.679 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:32 smithi084 ceph-mon[39784]: osdmap e734: 12 total, 12 up, 10 in 2024-05-15T23:38:32.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:32.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:38:32.780 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:38:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:32 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:38:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/837476221' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:38:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:32 smithi090 ceph-mon[44434]: osdmap e733: 12 total, 12 up, 10 in 2024-05-15T23:38:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:32 smithi090 ceph-mon[44434]: 8.d scrub starts 2024-05-15T23:38:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:32 smithi090 ceph-mon[44434]: 8.d scrub ok 2024-05-15T23:38:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:32 smithi090 ceph-mon[44434]: osdmap e734: 12 total, 12 up, 10 in 2024-05-15T23:38:32.809 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:32 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:38:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/837476221' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:38:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:32 smithi088 ceph-mon[44699]: osdmap e733: 12 total, 12 up, 10 in 2024-05-15T23:38:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:32 smithi088 ceph-mon[44699]: 8.d scrub starts 2024-05-15T23:38:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:32 smithi088 ceph-mon[44699]: 8.d scrub ok 2024-05-15T23:38:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:32 smithi088 ceph-mon[44699]: osdmap e734: 12 total, 12 up, 10 in 2024-05-15T23:38:32.959 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:38:32.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:32.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:38:33.063 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:33.096 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:33.118 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:33.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:33.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:33.222 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:38:33.251 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:33.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:33 smithi084 ceph-mon[39784]: pgmap v1951: 17 pgs: 17 active+clean; 577 KiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 503 op/s 2024-05-15T23:38:33.638 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:38:33.639 INFO:teuthology.orchestra.run.smithi084.stdout:{"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2393,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":734,"num_osds":12,"num_up_osds":12,"osd_up_since":1715815618,"num_in_osds":10,"osd_in_since":1715815990,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17}],"num_pgs":17,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":7374925824,"bytes_avail":952575430656,"bytes_total":959950356480,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":503},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.84:8443/"}},"servicemap":{"epoch":50,"modified":"2024-05-15T23:38:31.317002+0000","services":{}},"progress_events":{}} 2024-05-15T23:38:33.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:33 smithi090 ceph-mon[44434]: pgmap v1951: 17 pgs: 17 active+clean; 577 KiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 503 op/s 2024-05-15T23:38:33.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:33 smithi088 ceph-mon[44699]: pgmap v1951: 17 pgs: 17 active+clean; 577 KiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 503 op/s 2024-05-15T23:38:33.901 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:38:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:33.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:38:34.005 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:38:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:34.060 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:38:34.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:34.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:38:34.165 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:34.198 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:34.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:34.224 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:34.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:38:34.357 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:34.379 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:38:34.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:34.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:38:34.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:34.511 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:34 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2062893923' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T23:38:34.513 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:34.534 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:34.537 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:34.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:34 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2062893923' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T23:38:34.638 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:38:34.667 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:34.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:38:34.688 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:38:34.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:34.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:38:34.791 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:38:34.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:34 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2062893923' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T23:38:34.824 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:34.846 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:38:34.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:34.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:38:34.950 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:34.980 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:34.992 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:38:35.001 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:35.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:35.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:35.005 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:38:35.024 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:38:35.044 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:38:35.064 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:38:35.084 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:38:35.103 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:38:35.105 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:38:35.123 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:38:35.133 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:35.148 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:38:35.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:38:35.190 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:38:35.210 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:38:35.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:38:35.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:38:35.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:38:35.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:38:35.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:38:35.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:38:35.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:38:35.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:38:35.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:38:35.212 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:38:35.212 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:38:35.212 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:38:35.551 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-05-15T23:38:35.552 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:38:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:35 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4212472419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:38:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:35.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:35 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:35.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:35 smithi090 ceph-mon[44434]: pgmap v1952: 17 pgs: 17 active+clean; 577 KiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 170 op/s 2024-05-15T23:38:35.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:35 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4212472419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:38:35.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:35.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:35.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:35 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:35.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:35 smithi088 ceph-mon[44699]: pgmap v1952: 17 pgs: 17 active+clean; 577 KiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 170 op/s 2024-05-15T23:38:35.867 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:35 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4212472419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:38:35.867 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:35.867 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:35.867 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:35 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:35.867 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:35 smithi084 ceph-mon[39784]: pgmap v1952: 17 pgs: 17 active+clean; 577 KiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 170 op/s 2024-05-15T23:38:36.245 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:36.251 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:36.465 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:38:36.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:36.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:38:36.571 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:38:36.598 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:36.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:36.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:37.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:37 smithi084 ceph-mon[39784]: pgmap v1953: 17 pgs: 17 active+clean; 577 KiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 128 op/s 2024-05-15T23:38:37.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:37 smithi090 ceph-mon[44434]: pgmap v1953: 17 pgs: 17 active+clean; 577 KiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 128 op/s 2024-05-15T23:38:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:37 smithi088 ceph-mon[44699]: pgmap v1953: 17 pgs: 17 active+clean; 577 KiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 128 op/s 2024-05-15T23:38:37.903 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:38.251 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:38:38.449 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:38:38.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:38.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:38:38.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:38:38.581 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:38.602 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:38:38.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:38.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:38:38.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:38:38.736 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:39.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:39 smithi090 ceph-mon[44434]: pgmap v1954: 17 pgs: 17 active+clean; 577 KiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 75 op/s 2024-05-15T23:38:39.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:39 smithi088 ceph-mon[44699]: pgmap v1954: 17 pgs: 17 active+clean; 577 KiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 75 op/s 2024-05-15T23:38:39.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:39 smithi084 ceph-mon[39784]: pgmap v1954: 17 pgs: 17 active+clean; 577 KiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 75 op/s 2024-05-15T23:38:40.669 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_6' removed 2024-05-15T23:38:41.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:40 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/862525227' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:38:41.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:40 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/862525227' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:38:41.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:40 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/862525227' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:38:41.805 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:38:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:38:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:38:41.805 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:38:41.805 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:38:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/862525227' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:38:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: osdmap e735: 12 total, 12 up, 10 in 2024-05-15T23:38:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=56,logm=3} crc {auth=2538864823,config=3313370793,health=3340005642,kv=1756326304,logm=3302533902}) 2024-05-15T23:38:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1516971614}) 2024-05-15T23:38:41.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3313148636}) 2024-05-15T23:38:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2630970466}) 2024-05-15T23:38:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=477590076}) 2024-05-15T23:38:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3352644419}) 2024-05-15T23:38:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:38:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2992031702}) 2024-05-15T23:38:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3600092471}) 2024-05-15T23:38:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1174693991}) 2024-05-15T23:38:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2536281528}) 2024-05-15T23:38:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=215922433}) 2024-05-15T23:38:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=696617330}) 2024-05-15T23:38:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562507268}) 2024-05-15T23:38:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1712230194}) 2024-05-15T23:38:41.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=711005070}) 2024-05-15T23:38:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2085043503}) 2024-05-15T23:38:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2109388669}) 2024-05-15T23:38:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1495540415}) 2024-05-15T23:38:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=527203025}) 2024-05-15T23:38:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4261639031}) 2024-05-15T23:38:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=480255780}) 2024-05-15T23:38:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=467216706}) 2024-05-15T23:38:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1171914202}) 2024-05-15T23:38:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2029844523}) 2024-05-15T23:38:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=132951594}) 2024-05-15T23:38:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3651731516}) 2024-05-15T23:38:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1453179408}) 2024-05-15T23:38:41.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2154214262}) 2024-05-15T23:38:41.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=665283240}) 2024-05-15T23:38:41.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2179004607}) 2024-05-15T23:38:41.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1514450303}) 2024-05-15T23:38:41.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3571246206}) 2024-05-15T23:38:41.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4139153056}) 2024-05-15T23:38:41.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1827969687}) 2024-05-15T23:38:41.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2558902243}) 2024-05-15T23:38:41.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1047544489}) 2024-05-15T23:38:41.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1698467322}) 2024-05-15T23:38:41.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3876236850}) 2024-05-15T23:38:41.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=942885613}) 2024-05-15T23:38:41.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2683101278}) 2024-05-15T23:38:41.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3643496636}) 2024-05-15T23:38:41.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2286337707}) 2024-05-15T23:38:41.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202549060}) 2024-05-15T23:38:41.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968047973}) 2024-05-15T23:38:41.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4226514128}) 2024-05-15T23:38:41.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1459247832}) 2024-05-15T23:38:41.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3036832040}) 2024-05-15T23:38:41.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2944797012}) 2024-05-15T23:38:41.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=568273031}) 2024-05-15T23:38:41.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2799409944}) 2024-05-15T23:38:41.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3478843253}) 2024-05-15T23:38:41.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3175543000}) 2024-05-15T23:38:41.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2397237210}) 2024-05-15T23:38:41.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1997689058}) 2024-05-15T23:38:41.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2448813612}) 2024-05-15T23:38:41.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=608211991}) 2024-05-15T23:38:41.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2981950653}) 2024-05-15T23:38:41.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=659553293}) 2024-05-15T23:38:41.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500194075}) 2024-05-15T23:38:41.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2208270612}) 2024-05-15T23:38:41.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3118061493}) 2024-05-15T23:38:41.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=211595091}) 2024-05-15T23:38:41.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1990260840}) 2024-05-15T23:38:41.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3021702763}) 2024-05-15T23:38:41.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=788479431}) 2024-05-15T23:38:41.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=660456026}) 2024-05-15T23:38:41.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=531371118}) 2024-05-15T23:38:41.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=23822197}) 2024-05-15T23:38:41.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3451836111}) 2024-05-15T23:38:41.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290605568}) 2024-05-15T23:38:41.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1043003639}) 2024-05-15T23:38:41.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3471072391}) 2024-05-15T23:38:41.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3801771928}) 2024-05-15T23:38:41.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3725338290}) 2024-05-15T23:38:41.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=514949436}) 2024-05-15T23:38:41.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2041402937}) 2024-05-15T23:38:41.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4081566008}) 2024-05-15T23:38:41.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2062474620}) 2024-05-15T23:38:41.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=368420169}) 2024-05-15T23:38:41.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=420034205}) 2024-05-15T23:38:41.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1546418961}) 2024-05-15T23:38:41.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1044882330}) 2024-05-15T23:38:41.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1910854631}) 2024-05-15T23:38:41.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=302810579}) 2024-05-15T23:38:41.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1878859647}) 2024-05-15T23:38:41.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3056923279}) 2024-05-15T23:38:41.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=941071573}) 2024-05-15T23:38:41.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2771371855}) 2024-05-15T23:38:41.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=17,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=42} crc {logm=828830188,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=68947745,mon_config_key=4172026194}) 2024-05-15T23:38:41.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1592419060}) 2024-05-15T23:38:41.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=741324769}) 2024-05-15T23:38:41.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=42,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=40} crc {mon_config_key=942257344,monmap=1478146745,osd_metadata=2611665884,osd_pg_creating=392134948,osdmap=849019563}) 2024-05-15T23:38:41.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=67} crc {osdmap=2272871416}) 2024-05-15T23:38:41.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: pgmap v1956: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:41.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:38:41.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:41 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:38:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/862525227' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:38:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: osdmap e735: 12 total, 12 up, 10 in 2024-05-15T23:38:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=56,logm=3} crc {auth=2538864823,config=3313370793,health=3340005642,kv=1756326304,logm=3302533902}) 2024-05-15T23:38:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1516971614}) 2024-05-15T23:38:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3313148636}) 2024-05-15T23:38:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2630970466}) 2024-05-15T23:38:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=477590076}) 2024-05-15T23:38:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3352644419}) 2024-05-15T23:38:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:38:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2992031702}) 2024-05-15T23:38:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3600092471}) 2024-05-15T23:38:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1174693991}) 2024-05-15T23:38:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2536281528}) 2024-05-15T23:38:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=215922433}) 2024-05-15T23:38:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=696617330}) 2024-05-15T23:38:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562507268}) 2024-05-15T23:38:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1712230194}) 2024-05-15T23:38:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=711005070}) 2024-05-15T23:38:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2085043503}) 2024-05-15T23:38:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2109388669}) 2024-05-15T23:38:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1495540415}) 2024-05-15T23:38:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=527203025}) 2024-05-15T23:38:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4261639031}) 2024-05-15T23:38:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=480255780}) 2024-05-15T23:38:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=467216706}) 2024-05-15T23:38:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1171914202}) 2024-05-15T23:38:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2029844523}) 2024-05-15T23:38:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=132951594}) 2024-05-15T23:38:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3651731516}) 2024-05-15T23:38:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1453179408}) 2024-05-15T23:38:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2154214262}) 2024-05-15T23:38:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=665283240}) 2024-05-15T23:38:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2179004607}) 2024-05-15T23:38:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1514450303}) 2024-05-15T23:38:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3571246206}) 2024-05-15T23:38:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4139153056}) 2024-05-15T23:38:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1827969687}) 2024-05-15T23:38:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2558902243}) 2024-05-15T23:38:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1047544489}) 2024-05-15T23:38:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1698467322}) 2024-05-15T23:38:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3876236850}) 2024-05-15T23:38:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=942885613}) 2024-05-15T23:38:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2683101278}) 2024-05-15T23:38:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3643496636}) 2024-05-15T23:38:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2286337707}) 2024-05-15T23:38:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202549060}) 2024-05-15T23:38:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968047973}) 2024-05-15T23:38:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4226514128}) 2024-05-15T23:38:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1459247832}) 2024-05-15T23:38:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3036832040}) 2024-05-15T23:38:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2944797012}) 2024-05-15T23:38:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=568273031}) 2024-05-15T23:38:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2799409944}) 2024-05-15T23:38:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3478843253}) 2024-05-15T23:38:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3175543000}) 2024-05-15T23:38:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2397237210}) 2024-05-15T23:38:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1997689058}) 2024-05-15T23:38:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2448813612}) 2024-05-15T23:38:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=608211991}) 2024-05-15T23:38:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2981950653}) 2024-05-15T23:38:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=659553293}) 2024-05-15T23:38:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500194075}) 2024-05-15T23:38:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2208270612}) 2024-05-15T23:38:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3118061493}) 2024-05-15T23:38:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=211595091}) 2024-05-15T23:38:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1990260840}) 2024-05-15T23:38:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3021702763}) 2024-05-15T23:38:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=788479431}) 2024-05-15T23:38:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=660456026}) 2024-05-15T23:38:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=531371118}) 2024-05-15T23:38:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=23822197}) 2024-05-15T23:38:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3451836111}) 2024-05-15T23:38:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290605568}) 2024-05-15T23:38:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1043003639}) 2024-05-15T23:38:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3471072391}) 2024-05-15T23:38:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3801771928}) 2024-05-15T23:38:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3725338290}) 2024-05-15T23:38:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=514949436}) 2024-05-15T23:38:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2041402937}) 2024-05-15T23:38:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4081566008}) 2024-05-15T23:38:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2062474620}) 2024-05-15T23:38:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=368420169}) 2024-05-15T23:38:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=420034205}) 2024-05-15T23:38:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1546418961}) 2024-05-15T23:38:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1044882330}) 2024-05-15T23:38:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1910854631}) 2024-05-15T23:38:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=302810579}) 2024-05-15T23:38:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1878859647}) 2024-05-15T23:38:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3056923279}) 2024-05-15T23:38:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=941071573}) 2024-05-15T23:38:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2771371855}) 2024-05-15T23:38:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=17,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=42} crc {logm=828830188,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=68947745,mon_config_key=4172026194}) 2024-05-15T23:38:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1592419060}) 2024-05-15T23:38:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=741324769}) 2024-05-15T23:38:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=42,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=40} crc {mon_config_key=942257344,monmap=1478146745,osd_metadata=2611665884,osd_pg_creating=392134948,osdmap=849019563}) 2024-05-15T23:38:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=67} crc {osdmap=2272871416}) 2024-05-15T23:38:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: pgmap v1956: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:38:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:41 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:38:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/862525227' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:38:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: osdmap e735: 12 total, 12 up, 10 in 2024-05-15T23:38:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=56,logm=3} crc {auth=2538864823,config=3313370793,health=3340005642,kv=1756326304,logm=3302533902}) 2024-05-15T23:38:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1516971614}) 2024-05-15T23:38:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3313148636}) 2024-05-15T23:38:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2630970466}) 2024-05-15T23:38:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=477590076}) 2024-05-15T23:38:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3352644419}) 2024-05-15T23:38:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:38:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2992031702}) 2024-05-15T23:38:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3600092471}) 2024-05-15T23:38:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1174693991}) 2024-05-15T23:38:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2536281528}) 2024-05-15T23:38:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=215922433}) 2024-05-15T23:38:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=696617330}) 2024-05-15T23:38:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562507268}) 2024-05-15T23:38:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1712230194}) 2024-05-15T23:38:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=711005070}) 2024-05-15T23:38:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2085043503}) 2024-05-15T23:38:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2109388669}) 2024-05-15T23:38:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1495540415}) 2024-05-15T23:38:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=527203025}) 2024-05-15T23:38:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4261639031}) 2024-05-15T23:38:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=480255780}) 2024-05-15T23:38:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=467216706}) 2024-05-15T23:38:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1171914202}) 2024-05-15T23:38:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2029844523}) 2024-05-15T23:38:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=132951594}) 2024-05-15T23:38:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3651731516}) 2024-05-15T23:38:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1453179408}) 2024-05-15T23:38:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2154214262}) 2024-05-15T23:38:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=665283240}) 2024-05-15T23:38:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2179004607}) 2024-05-15T23:38:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1514450303}) 2024-05-15T23:38:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3571246206}) 2024-05-15T23:38:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4139153056}) 2024-05-15T23:38:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1827969687}) 2024-05-15T23:38:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2558902243}) 2024-05-15T23:38:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1047544489}) 2024-05-15T23:38:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1698467322}) 2024-05-15T23:38:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3876236850}) 2024-05-15T23:38:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=942885613}) 2024-05-15T23:38:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2683101278}) 2024-05-15T23:38:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3643496636}) 2024-05-15T23:38:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2286337707}) 2024-05-15T23:38:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202549060}) 2024-05-15T23:38:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968047973}) 2024-05-15T23:38:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4226514128}) 2024-05-15T23:38:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1459247832}) 2024-05-15T23:38:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3036832040}) 2024-05-15T23:38:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2944797012}) 2024-05-15T23:38:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=568273031}) 2024-05-15T23:38:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2799409944}) 2024-05-15T23:38:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3478843253}) 2024-05-15T23:38:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3175543000}) 2024-05-15T23:38:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2397237210}) 2024-05-15T23:38:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1997689058}) 2024-05-15T23:38:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2448813612}) 2024-05-15T23:38:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=608211991}) 2024-05-15T23:38:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2981950653}) 2024-05-15T23:38:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=659553293}) 2024-05-15T23:38:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500194075}) 2024-05-15T23:38:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2208270612}) 2024-05-15T23:38:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3118061493}) 2024-05-15T23:38:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=211595091}) 2024-05-15T23:38:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1990260840}) 2024-05-15T23:38:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3021702763}) 2024-05-15T23:38:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=788479431}) 2024-05-15T23:38:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=660456026}) 2024-05-15T23:38:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=531371118}) 2024-05-15T23:38:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=23822197}) 2024-05-15T23:38:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3451836111}) 2024-05-15T23:38:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290605568}) 2024-05-15T23:38:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1043003639}) 2024-05-15T23:38:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3471072391}) 2024-05-15T23:38:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3801771928}) 2024-05-15T23:38:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3725338290}) 2024-05-15T23:38:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=514949436}) 2024-05-15T23:38:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2041402937}) 2024-05-15T23:38:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4081566008}) 2024-05-15T23:38:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2062474620}) 2024-05-15T23:38:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=368420169}) 2024-05-15T23:38:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=420034205}) 2024-05-15T23:38:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1546418961}) 2024-05-15T23:38:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1044882330}) 2024-05-15T23:38:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1910854631}) 2024-05-15T23:38:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=302810579}) 2024-05-15T23:38:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1878859647}) 2024-05-15T23:38:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3056923279}) 2024-05-15T23:38:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=941071573}) 2024-05-15T23:38:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2771371855}) 2024-05-15T23:38:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=17,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=42} crc {logm=828830188,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=68947745,mon_config_key=4172026194}) 2024-05-15T23:38:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1592419060}) 2024-05-15T23:38:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=741324769}) 2024-05-15T23:38:42.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=42,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=40} crc {mon_config_key=942257344,monmap=1478146745,osd_metadata=2611665884,osd_pg_creating=392134948,osdmap=849019563}) 2024-05-15T23:38:42.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=67} crc {osdmap=2272871416}) 2024-05-15T23:38:42.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: pgmap v1956: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:42.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:38:42.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:41 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:38:43.122 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:38:43.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:43.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:38:43.227 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:38:43.260 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:43.283 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:38:43.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:43.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:38:43.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:43.421 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:43.444 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:43.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:43.446 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:43.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:38:43.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:43 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:38:43.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:43 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:38:43.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:43 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:38:43.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:43 smithi088 ceph-mon[44699]: pgmap v1957: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:43.576 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:43.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:43 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:38:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:43 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:38:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:43 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:38:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:43 smithi084 ceph-mon[39784]: pgmap v1957: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:43.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:43 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:38:43.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:43 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:38:43.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:43 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:38:43.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:43 smithi090 ceph-mon[44434]: pgmap v1957: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:43.922 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-05-15T23:38:43.923 INFO:tasks.radosbench:Beginning radosbench... 2024-05-15T23:38:43.924 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-05-15T23:38:43.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool create unique_pool_7 16 2024-05-15T23:38:43.925 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:38:43.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:43.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:38:44.029 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:38:44.055 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:44.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T23:38:44.190 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:44.206 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:38:44.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:44.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:38:44.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:38:44.336 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:44.370 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:38:44.522 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/685067068' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:38:44.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/685067068' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:38:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/685067068' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:38:45.114 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:45.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:45 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:38:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/685067068' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:38:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:45 smithi084 ceph-mon[39784]: osdmap e736: 12 total, 12 up, 10 in 2024-05-15T23:38:45.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:45 smithi084 ceph-mon[39784]: pgmap v1959: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:45 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:38:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/685067068' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:38:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:45 smithi090 ceph-mon[44434]: osdmap e736: 12 total, 12 up, 10 in 2024-05-15T23:38:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:45 smithi090 ceph-mon[44434]: pgmap v1959: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:45.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:45 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:38:45.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/685067068' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:38:45.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:45 smithi088 ceph-mon[44699]: osdmap e736: 12 total, 12 up, 10 in 2024-05-15T23:38:45.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:45 smithi088 ceph-mon[44699]: pgmap v1959: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:45.988 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:38:45.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:45.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:38:46.093 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:38:46.124 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:47.769 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:47 smithi084 ceph-mon[39784]: pgmap v1960: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:47.773 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:38:47.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:47.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:38:47.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:47 smithi090 ceph-mon[44434]: pgmap v1960: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:47.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:47 smithi088 ceph-mon[44699]: pgmap v1960: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:47.879 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:38:47.893 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:38:47.894 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:38:47.908 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:48.297 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:38:48.297 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:38:48.297 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:38:48.297 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:38:48.297 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:38:48.297 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:38:48.298 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:38:48.298 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:38:48.309 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:38:48.326 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:38:48.346 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:38:48.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:38:48.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:38:48.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:38:48.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:38:48.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:38:48.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:38:48.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:38:48.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:38:48.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:38:48.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:38:48.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:38:48.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:38:48.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:38:48.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:48 smithi084 ceph-mon[39784]: from='client.17415 v1:172.21.15.84:0/3191240400' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:38:48.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/994834433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:38:48.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:48.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:48 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:48.666 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:38:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:48 smithi090 ceph-mon[44434]: from='client.17415 v1:172.21.15.84:0/3191240400' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:38:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/994834433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:38:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:48 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:48.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:48 smithi088 ceph-mon[44699]: from='client.17415 v1:172.21.15.84:0/3191240400' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:38:48.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/994834433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:38:48.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:48.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:48 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.410 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:49.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:49 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:49 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/982616975' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-05-15T23:38:49.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:49 smithi084 ceph-mon[39784]: from='client.37355 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-05-15T23:38:49.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:49 smithi084 ceph-mon[39784]: pgmap v1961: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:49.587 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:38:49.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:49.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:38:49.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:38:49.722 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:49.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:49 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:49 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/982616975' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-05-15T23:38:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:49 smithi090 ceph-mon[44434]: from='client.37355 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-05-15T23:38:49.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:49 smithi090 ceph-mon[44434]: pgmap v1961: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:49 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:49 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/982616975' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-05-15T23:38:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:49 smithi088 ceph-mon[44699]: from='client.37355 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-05-15T23:38:49.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:49 smithi088 ceph-mon[44699]: pgmap v1961: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:49.988 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1960,"stamp":"2024-05-15T23:38:47.320334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":3,"kb":937451520,"kb_used":7143976,"kb_used_data":937788,"kb_used_omap":41,"kb_used_meta":6204950,"kb_avail":930307544,"statfs":{"total":959950356480,"available":952634925056,"internally_reserved":0,"allocated":960294912,"data_stored":956633308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42533,"internal_metadata":6353869275},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-293872,"num_write_kb":-9403841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-16864,"num_bytes_recovered":-1105199104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001592"},"pg_stats":[{"pgid":"1.0","version":"57'94","reported_seq":1596,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-05-15T23:38:41.007597+0000","last_change":"2024-05-15T23:38:23.156562+0000","last_active":"2024-05-15T23:38:41.007597+0000","last_peered":"2024-05-15T23:38:41.007597+0000","last_clean":"2024-05-15T23:38:41.007597+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:38:41.007597+0000","last_undegraded":"2024-05-15T23:38:41.007597+0000","last_fullsized":"2024-05-15T23:38:41.007597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:38:23.156474+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:38:23.156474+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:39:44.796181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351396,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":826724,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92918428,"statfs":{"total":95995035648,"available":95148470272,"internally_reserved":0,"allocated":1282048,"data_stored":898206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":845085124},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580329,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":581684,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93163468,"statfs":{"total":95995035648,"available":95399391232,"internally_reserved":0,"allocated":688128,"data_stored":337502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":594868672},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776561,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":717748,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93027404,"statfs":{"total":95995035648,"available":95260061696,"internally_reserved":0,"allocated":688128,"data_stored":337502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":734198208},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":8,"up_from":520,"seq":2233382994064,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":118364,"kb_used_data":732,"kb_used_omap":24,"kb_used_meta":117415,"kb_avail":93626788,"statfs":{"total":95995035648,"available":95873830912,"internally_reserved":0,"allocated":749568,"data_stored":369298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25357,"internal_metadata":120233203},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267130,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1087980,"kb_used_data":457060,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92657172,"statfs":{"total":95995035648,"available":94880944128,"internally_reserved":0,"allocated":468029440,"data_stored":467645618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":646052298},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":197568496063,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575322,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1290228,"kb_used_data":474592,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92454924,"statfs":{"total":95995035648,"available":94673842176,"internally_reserved":0,"allocated":485982208,"data_stored":485631582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":835123654},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016731,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483419,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":209564,"kb_used_data":732,"kb_used_omap":4,"kb_used_meta":208635,"kb_avail":93535588,"statfs":{"total":95995035648,"available":95780442112,"internally_reserved":0,"allocated":749568,"data_stored":369298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":213642947},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150161,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":876020,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92869132,"statfs":{"total":95995035648,"available":95097991168,"internally_reserved":0,"allocated":688128,"data_stored":337502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":896268743},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986632,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":699868,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":93045284,"statfs":{"total":95995035648,"available":95278370816,"internally_reserved":0,"allocated":749568,"data_stored":369298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790405,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":735796,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":93009356,"statfs":{"total":95995035648,"available":95241580544,"internally_reserved":0,"allocated":688128,"data_stored":337502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":752679365},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:38:50.057 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_7' created 2024-05-15T23:38:50.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:50 smithi084 ceph-mon[39784]: from='client.37355 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-05-15T23:38:50.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:50 smithi084 ceph-mon[39784]: osdmap e737: 12 total, 12 up, 10 in 2024-05-15T23:38:50.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-05-15T23:38:50.671 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:38:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:50.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:38:50.690 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:50.775 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:50.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:50 smithi090 ceph-mon[44434]: from='client.37355 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-05-15T23:38:50.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:50 smithi090 ceph-mon[44434]: osdmap e737: 12 total, 12 up, 10 in 2024-05-15T23:38:50.805 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:50 smithi088 ceph-mon[44699]: from='client.37355 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-05-15T23:38:50.805 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:50 smithi088 ceph-mon[44699]: osdmap e737: 12 total, 12 up, 10 in 2024-05-15T23:38:50.810 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:50.829 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:50.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:50.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:38:50.961 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:51.199 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:38:51.604 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:51 smithi084 ceph-mon[39784]: osdmap e738: 12 total, 12 up, 10 in 2024-05-15T23:38:51.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:51 smithi084 ceph-mon[39784]: pgmap v1964: 17 pgs: 4 creating+peering, 1 active+clean, 12 unknown; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:51.610 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:38:51.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:51.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:38:51.715 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:51.745 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:51 smithi090 ceph-mon[44434]: osdmap e738: 12 total, 12 up, 10 in 2024-05-15T23:38:51.745 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:51 smithi090 ceph-mon[44434]: pgmap v1964: 17 pgs: 4 creating+peering, 1 active+clean, 12 unknown; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:51.748 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:51.770 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:51.773 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:51 smithi088 ceph-mon[44699]: osdmap e738: 12 total, 12 up, 10 in 2024-05-15T23:38:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:51 smithi088 ceph-mon[44699]: pgmap v1964: 17 pgs: 4 creating+peering, 1 active+clean, 12 unknown; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:51.874 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:38:51.901 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:52.150 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:38:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:52.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:38:52.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:38:52.288 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:38:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:52.312 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:38:52.413 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:38:52.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:52 smithi084 ceph-mon[39784]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:38:52.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:52 smithi084 ceph-mon[39784]: osdmap e739: 12 total, 12 up, 10 in 2024-05-15T23:38:52.440 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:52 smithi090 ceph-mon[44434]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:38:52.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:52 smithi090 ceph-mon[44434]: osdmap e739: 12 total, 12 up, 10 in 2024-05-15T23:38:52.789 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:38:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:52.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:38:52.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:52 smithi088 ceph-mon[44699]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T23:38:52.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:52 smithi088 ceph-mon[44699]: osdmap e739: 12 total, 12 up, 10 in 2024-05-15T23:38:52.894 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:38:52.922 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:52.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:38:53.124 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:38:53.125 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:53.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:53.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:38:53.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:38:53.259 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:53.688 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:53 smithi084 ceph-mon[39784]: pgmap v1966: 17 pgs: 4 creating+peering, 12 active+clean, 1 unknown; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:53.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:53 smithi090 ceph-mon[44434]: pgmap v1966: 17 pgs: 4 creating+peering, 12 active+clean, 1 unknown; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:53.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:53 smithi088 ceph-mon[44699]: pgmap v1966: 17 pgs: 4 creating+peering, 12 active+clean, 1 unknown; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:54.108 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:38:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:54.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:38:54.213 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:54.249 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:54.271 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:54.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:54.375 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:38:54.407 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:54.607 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:38:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:54.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:38:54.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:38:54.746 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:54.768 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:38:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:54.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:38:54.872 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:54.905 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:54.927 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:54.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:55.031 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:38:55.061 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:55.081 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:38:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:55.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:38:55.185 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:38:55.214 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:55.252 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:38:55.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:55.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:38:55.356 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:38:55.389 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:55.410 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:38:55.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:55.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:38:55.472 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:55.514 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:38:55.522 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:38:55.522 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:38:55.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:55 smithi084 ceph-mon[39784]: pgmap v1967: 17 pgs: 17 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:55.547 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:55.570 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:38:55.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:55.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:38:55.675 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:38:55.703 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:55 smithi088 ceph-mon[44699]: pgmap v1967: 17 pgs: 17 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:55.726 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:38:55.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:55.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:38:55.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:55 smithi090 ceph-mon[44434]: pgmap v1967: 17 pgs: 17 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:55.829 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:38:55.847 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:38:55.858 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:55.866 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:38:55.878 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:38:55.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:55.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:38:55.885 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:38:55.905 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:38:55.925 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:38:55.945 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:38:55.967 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:38:55.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:38:55.987 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:38:56.008 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:38:56.016 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:56.028 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:38:56.038 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:38:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:56.042 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:38:56.048 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:38:56.073 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:38:56.074 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:38:56.074 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:38:56.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:38:56.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:38:56.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:38:56.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:38:56.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:38:56.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:38:56.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:38:56.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:38:56.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:38:56.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:38:56.143 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:38:56.173 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:56.387 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:38:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/281482450' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:38:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4090143971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:38:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:56 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:56 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:38:56.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:56 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:38:56.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/281482450' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:38:56.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4090143971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:38:56.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:56 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:56 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:38:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:56 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:38:56.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/281482450' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:38:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4090143971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:38:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:56 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:56 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:38:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:56 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:38:57.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:57 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:38:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/281482450' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:38:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:57 smithi084 ceph-mon[39784]: osdmap e740: 12 total, 12 up, 10 in 2024-05-15T23:38:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:57 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:38:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:57 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:38:57.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:57 smithi084 ceph-mon[39784]: pgmap v1969: 17 pgs: 17 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:57 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:38:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/281482450' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:38:57.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:57 smithi090 ceph-mon[44434]: osdmap e740: 12 total, 12 up, 10 in 2024-05-15T23:38:57.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:57 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:38:57.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:57 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:38:57.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:57 smithi090 ceph-mon[44434]: pgmap v1969: 17 pgs: 17 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:38:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:57 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:38:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/281482450' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:38:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:57 smithi088 ceph-mon[44699]: osdmap e740: 12 total, 12 up, 10 in 2024-05-15T23:38:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:57 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:38:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:57 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:38:57.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:57 smithi088 ceph-mon[44699]: pgmap v1969: 17 pgs: 17 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:57.925 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:38:57.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:57.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:38:57.951 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T23:38:58.028 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:38:58.056 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:58.076 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:38:58.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:58.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:38:58.181 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:38:58.208 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:58.241 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:38:58.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:58.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:38:58.251 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:38:58.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:38:58.382 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:58.559 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:58 smithi088 ceph-mon[44699]: osdmap e741: 12 total, 12 up, 10 in 2024-05-15T23:38:58.599 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:38:58.711 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:58 smithi084 ceph-mon[39784]: osdmap e741: 12 total, 12 up, 10 in 2024-05-15T23:38:58.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:58 smithi090 ceph-mon[44434]: osdmap e741: 12 total, 12 up, 10 in 2024-05-15T23:38:59.239 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:38:59.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:38:59.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:38:59.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:38:59.373 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:38:59.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:59 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:38:59.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:59 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:38:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:59 smithi088 ceph-mon[44699]: osdmap e742: 12 total, 12 up, 10 in 2024-05-15T23:38:59.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:38:59 smithi088 ceph-mon[44699]: pgmap v1972: 17 pgs: 17 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:59 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:38:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:59 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:38:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:59 smithi090 ceph-mon[44434]: osdmap e742: 12 total, 12 up, 10 in 2024-05-15T23:38:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:38:59 smithi090 ceph-mon[44434]: pgmap v1972: 17 pgs: 17 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:38:59.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:59 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:38:59.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:59 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:38:59.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:59 smithi084 ceph-mon[39784]: osdmap e742: 12 total, 12 up, 10 in 2024-05-15T23:38:59.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:38:59 smithi084 ceph-mon[39784]: pgmap v1972: 17 pgs: 17 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:39:00.227 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:39:00.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:00.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:39:00.332 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:39:00.364 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:00.384 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:39:00.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:00.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:39:00.411 INFO:teuthology.orchestra.run.smithi084.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-05-15T23:39:00.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:39:00.483 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:39:00.484 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:39:00.484 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2024-05-15T23:39:00.484 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:39:00.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:39:00.518 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:00.538 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:39:00.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:00.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:39:00.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:00 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1389290275' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:39:00.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:39:00.669 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:00 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1389290275' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:39:00.671 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:00.691 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:39:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:00.694 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:39:00.795 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:39:00.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:00 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1389290275' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T23:39:00.822 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:01.023 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:39:01.255 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T23:39:01.271 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:39:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:01.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:39:01.376 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:39:01.403 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:01.424 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:39:01.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:01.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:39:01.490 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:39:01.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:39:01.535 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:39:01.561 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:01.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:01 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1389290275' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:39:01.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:01 smithi088 ceph-mon[44699]: osdmap e743: 12 total, 12 up, 10 in 2024-05-15T23:39:01.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:01 smithi088 ceph-mon[44699]: pgmap v1974: 17 pgs: 17 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:39:01.582 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:39:01.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:01.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:39:01.672 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:01 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1389290275' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:39:01.672 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:01 smithi084 ceph-mon[39784]: osdmap e743: 12 total, 12 up, 10 in 2024-05-15T23:39:01.672 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:01 smithi084 ceph-mon[39784]: pgmap v1974: 17 pgs: 17 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:39:01.686 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:39:01.713 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:01.785 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:39:01.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:01.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:39:01.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:01 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1389290275' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-05-15T23:39:01.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:01 smithi090 ceph-mon[44434]: osdmap e743: 12 total, 12 up, 10 in 2024-05-15T23:39:01.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:01 smithi090 ceph-mon[44434]: pgmap v1974: 17 pgs: 17 active+clean; 577 KiB data, 6.8 GiB used, 887 GiB / 894 GiB avail 2024-05-15T23:39:01.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:39:01.922 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:01.943 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:39:01.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:01.946 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:39:02.048 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:39:02.076 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:02.175 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:39:02.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:02.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:39:02.280 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:39:02.309 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:02.444 DEBUG:teuthology.orchestra.run.smithi090:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-05-15T23:39:02.447 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-05-15T23:39:02.447 INFO:teuthology.orchestra.run:waiting for 3000 2024-05-15T23:39:02.553 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:hints = 1 2024-05-15T23:39:02.554 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-05-15T23:39:02.554 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object prefix: benchmark_data_smithi090_166897 2024-05-15T23:39:02.554 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:39:02.554 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 0 0 0 0 0 0 - 0 2024-05-15T23:39:02.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:02 smithi088 ceph-mon[44699]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:39:02.588 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:02 smithi084 ceph-mon[39784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:39:02.600 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:39:02.600 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:39:02.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:02 smithi090 ceph-mon[44434]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T23:39:02.812 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:39:02.860 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:39:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:02.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:39:02.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:39:03.005 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:03.037 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:39:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:03.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:39:03.141 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:03.178 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:03.207 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:03.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:03.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:03.311 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:39:03.351 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:03.395 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:39:03.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:03.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:39:03.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:39:03.540 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:03.554 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 1 16 5707 5691 355.557 355.688 0.00297848 0.00280418 2024-05-15T23:39:03.568 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:39:03.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:03 smithi088 ceph-mon[44699]: from='client.17448 v1:172.21.15.84:0/3743709126' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:39:03.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:03 smithi088 ceph-mon[44699]: pgmap v1975: 17 pgs: 17 active+clean; 19 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 3.1 MiB/s wr, 49 op/s 2024-05-15T23:39:03.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:03.572 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:39:03.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:39:03.706 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:39:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:39:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:39:03.706 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:39:03.706 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:39:03.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:03 smithi090 ceph-mon[44434]: from='client.17448 v1:172.21.15.84:0/3743709126' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:39:03.709 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:03 smithi090 ceph-mon[44434]: pgmap v1975: 17 pgs: 17 active+clean; 19 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 3.1 MiB/s wr, 49 op/s 2024-05-15T23:39:03.711 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:03.741 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:39:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:03.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:39:03.831 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:03 smithi084 ceph-mon[39784]: from='client.17448 v1:172.21.15.84:0/3743709126' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:39:03.831 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:03 smithi084 ceph-mon[39784]: pgmap v1975: 17 pgs: 17 active+clean; 19 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 3.1 MiB/s wr, 49 op/s 2024-05-15T23:39:03.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:39:03.879 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1974,"stamp":"2024-05-15T23:39:01.323464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":937451520,"kb_used":7097108,"kb_used_data":891972,"kb_used_omap":41,"kb_used_meta":6204950,"kb_avail":930354412,"statfs":{"total":959950356480,"available":952682917888,"internally_reserved":0,"allocated":913379328,"data_stored":909590638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42533,"internal_metadata":6353869275},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.939574"},"pg_stats":[{"pgid":"9.f","version":"0'0","reported_seq":14,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-05-15T23:38:56.385369+0000","last_change":"2024-05-15T23:38:50.401719+0000","last_active":"2024-05-15T23:38:56.385369+0000","last_peered":"2024-05-15T23:38:56.385369+0000","last_clean":"2024-05-15T23:38:56.385369+0000","last_became_active":"2024-05-15T23:38:50.401351+0000","last_became_peered":"2024-05-15T23:38:50.401351+0000","last_unstale":"2024-05-15T23:38:56.385369+0000","last_undegraded":"2024-05-15T23:38:56.385369+0000","last_fullsized":"2024-05-15T23:38:56.385369+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:40:04.277012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":18,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T23:38:58.626028+0000","last_change":"2024-05-15T23:38:50.403048+0000","last_active":"2024-05-15T23:38:58.626028+0000","last_peered":"2024-05-15T23:38:58.626028+0000","last_clean":"2024-05-15T23:38:58.626028+0000","last_became_active":"2024-05-15T23:38:50.402740+0000","last_became_peered":"2024-05-15T23:38:50.402740+0000","last_unstale":"2024-05-15T23:38:58.626028+0000","last_undegraded":"2024-05-15T23:38:58.626028+0000","last_fullsized":"2024-05-15T23:38:58.626028+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:40:01.199213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":18,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T23:38:58.626017+0000","last_change":"2024-05-15T23:38:50.401477+0000","last_active":"2024-05-15T23:38:58.626017+0000","last_peered":"2024-05-15T23:38:58.626017+0000","last_clean":"2024-05-15T23:38:58.626017+0000","last_became_active":"2024-05-15T23:38:50.401123+0000","last_became_peered":"2024-05-15T23:38:50.401123+0000","last_unstale":"2024-05-15T23:38:58.626017+0000","last_undegraded":"2024-05-15T23:38:58.626017+0000","last_fullsized":"2024-05-15T23:38:58.626017+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:39:59.496522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-05-15T23:38:56.873768+0000","last_change":"2024-05-15T23:38:50.402679+0000","last_active":"2024-05-15T23:38:56.873768+0000","last_peered":"2024-05-15T23:38:56.873768+0000","last_clean":"2024-05-15T23:38:56.873768+0000","last_became_active":"2024-05-15T23:38:50.402428+0000","last_became_peered":"2024-05-15T23:38:50.402428+0000","last_unstale":"2024-05-15T23:38:56.873768+0000","last_undegraded":"2024-05-15T23:38:56.873768+0000","last_fullsized":"2024-05-15T23:38:56.873768+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:39:55.630579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":18,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T23:38:58.826506+0000","last_change":"2024-05-15T23:38:50.402430+0000","last_active":"2024-05-15T23:38:58.826506+0000","last_peered":"2024-05-15T23:38:58.826506+0000","last_clean":"2024-05-15T23:38:58.826506+0000","last_became_active":"2024-05-15T23:38:50.401815+0000","last_became_peered":"2024-05-15T23:38:50.401815+0000","last_unstale":"2024-05-15T23:38:58.826506+0000","last_undegraded":"2024-05-15T23:38:58.826506+0000","last_fullsized":"2024-05-15T23:38:58.826506+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:40:02.566191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-05-15T23:38:56.385359+0000","last_change":"2024-05-15T23:38:50.401827+0000","last_active":"2024-05-15T23:38:56.385359+0000","last_peered":"2024-05-15T23:38:56.385359+0000","last_clean":"2024-05-15T23:38:56.385359+0000","last_became_active":"2024-05-15T23:38:50.401544+0000","last_became_peered":"2024-05-15T23:38:50.401544+0000","last_unstale":"2024-05-15T23:38:56.385359+0000","last_undegraded":"2024-05-15T23:38:56.385359+0000","last_fullsized":"2024-05-15T23:38:56.385359+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:39:54.768962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-05-15T23:38:52.225568+0000","last_change":"2024-05-15T23:38:50.404483+0000","last_active":"2024-05-15T23:38:52.225568+0000","last_peered":"2024-05-15T23:38:52.225568+0000","last_clean":"2024-05-15T23:38:52.225568+0000","last_became_active":"2024-05-15T23:38:50.404189+0000","last_became_peered":"2024-05-15T23:38:50.404189+0000","last_unstale":"2024-05-15T23:38:52.225568+0000","last_undegraded":"2024-05-15T23:38:52.225568+0000","last_fullsized":"2024-05-15T23:38:52.225568+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:40:01.741122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-05-15T23:38:52.225798+0000","last_change":"2024-05-15T23:38:50.401707+0000","last_active":"2024-05-15T23:38:52.225798+0000","last_peered":"2024-05-15T23:38:52.225798+0000","last_clean":"2024-05-15T23:38:52.225798+0000","last_became_active":"2024-05-15T23:38:50.401416+0000","last_became_peered":"2024-05-15T23:38:50.401416+0000","last_unstale":"2024-05-15T23:38:52.225798+0000","last_undegraded":"2024-05-15T23:38:52.225798+0000","last_fullsized":"2024-05-15T23:38:52.225798+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:40:12.044197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":14,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-05-15T23:38:56.385365+0000","last_change":"2024-05-15T23:38:50.402125+0000","last_active":"2024-05-15T23:38:56.385365+0000","last_peered":"2024-05-15T23:38:56.385365+0000","last_clean":"2024-05-15T23:38:56.385365+0000","last_became_active":"2024-05-15T23:38:50.401441+0000","last_became_peered":"2024-05-15T23:38:50.401441+0000","last_unstale":"2024-05-15T23:38:56.385365+0000","last_undegraded":"2024-05-15T23:38:56.385365+0000","last_fullsized":"2024-05-15T23:38:56.385365+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:39:54.558791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-05-15T23:38:57.250397+0000","last_change":"2024-05-15T23:38:50.401955+0000","last_active":"2024-05-15T23:38:57.250397+0000","last_peered":"2024-05-15T23:38:57.250397+0000","last_clean":"2024-05-15T23:38:57.250397+0000","last_became_active":"2024-05-15T23:38:50.401629+0000","last_became_peered":"2024-05-15T23:38:50.401629+0000","last_unstale":"2024-05-15T23:38:57.250397+0000","last_undegraded":"2024-05-15T23:38:57.250397+0000","last_fullsized":"2024-05-15T23:38:57.250397+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:39:52.029002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-05-15T23:38:56.386177+0000","last_change":"2024-05-15T23:38:50.398678+0000","last_active":"2024-05-15T23:38:56.386177+0000","last_peered":"2024-05-15T23:38:56.386177+0000","last_clean":"2024-05-15T23:38:56.386177+0000","last_became_active":"2024-05-15T23:38:50.398410+0000","last_became_peered":"2024-05-15T23:38:50.398410+0000","last_unstale":"2024-05-15T23:38:56.386177+0000","last_undegraded":"2024-05-15T23:38:56.386177+0000","last_fullsized":"2024-05-15T23:38:56.386177+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:39:55.800898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":18,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T23:38:58.626465+0000","last_change":"2024-05-15T23:38:50.402064+0000","last_active":"2024-05-15T23:38:58.626465+0000","last_peered":"2024-05-15T23:38:58.626465+0000","last_clean":"2024-05-15T23:38:58.626465+0000","last_became_active":"2024-05-15T23:38:50.401779+0000","last_became_peered":"2024-05-15T23:38:50.401779+0000","last_unstale":"2024-05-15T23:38:58.626465+0000","last_undegraded":"2024-05-15T23:38:58.626465+0000","last_fullsized":"2024-05-15T23:38:58.626465+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:40:09.493600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-05-15T23:38:56.873777+0000","last_change":"2024-05-15T23:38:50.402094+0000","last_active":"2024-05-15T23:38:56.873777+0000","last_peered":"2024-05-15T23:38:56.873777+0000","last_clean":"2024-05-15T23:38:56.873777+0000","last_became_active":"2024-05-15T23:38:50.401872+0000","last_became_peered":"2024-05-15T23:38:50.401872+0000","last_unstale":"2024-05-15T23:38:56.873777+0000","last_undegraded":"2024-05-15T23:38:56.873777+0000","last_fullsized":"2024-05-15T23:38:56.873777+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:40:00.685635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-05-15T23:38:56.873761+0000","last_change":"2024-05-15T23:38:50.401343+0000","last_active":"2024-05-15T23:38:56.873761+0000","last_peered":"2024-05-15T23:38:56.873761+0000","last_clean":"2024-05-15T23:38:56.873761+0000","last_became_active":"2024-05-15T23:38:50.401074+0000","last_became_peered":"2024-05-15T23:38:50.401074+0000","last_unstale":"2024-05-15T23:38:56.873761+0000","last_undegraded":"2024-05-15T23:38:56.873761+0000","last_fullsized":"2024-05-15T23:38:56.873761+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:40:03.323115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-05-15T23:38:57.250397+0000","last_change":"2024-05-15T23:38:50.402309+0000","last_active":"2024-05-15T23:38:57.250397+0000","last_peered":"2024-05-15T23:38:57.250397+0000","last_clean":"2024-05-15T23:38:57.250397+0000","last_became_active":"2024-05-15T23:38:50.401998+0000","last_became_peered":"2024-05-15T23:38:50.401998+0000","last_unstale":"2024-05-15T23:38:57.250397+0000","last_undegraded":"2024-05-15T23:38:57.250397+0000","last_fullsized":"2024-05-15T23:38:57.250397+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:39:52.098036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":18,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T23:38:58.626463+0000","last_change":"2024-05-15T23:38:50.404457+0000","last_active":"2024-05-15T23:38:58.626463+0000","last_peered":"2024-05-15T23:38:58.626463+0000","last_clean":"2024-05-15T23:38:58.626463+0000","last_became_active":"2024-05-15T23:38:50.404155+0000","last_became_peered":"2024-05-15T23:38:50.404155+0000","last_unstale":"2024-05-15T23:38:58.626463+0000","last_undegraded":"2024-05-15T23:38:58.626463+0000","last_fullsized":"2024-05-15T23:38:58.626463+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:40:08.996050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1610,"reported_epoch":742,"state":"active+clean","last_fresh":"2024-05-15T23:38:58.626066+0000","last_change":"2024-05-15T23:38:23.156562+0000","last_active":"2024-05-15T23:38:58.626066+0000","last_peered":"2024-05-15T23:38:58.626066+0000","last_clean":"2024-05-15T23:38:58.626066+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:38:58.626066+0000","last_undegraded":"2024-05-15T23:38:58.626066+0000","last_fullsized":"2024-05-15T23:38:58.626066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:38:23.156474+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:38:23.156474+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:39:44.796181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351399,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":826532,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92918620,"statfs":{"total":95995035648,"available":95148666880,"internally_reserved":0,"allocated":1269760,"data_stored":892223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":845085124},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580331,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":581648,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93163504,"statfs":{"total":95995035648,"available":95399428096,"internally_reserved":0,"allocated":724992,"data_stored":357015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":594868672},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776564,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":717748,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93027404,"statfs":{"total":95995035648,"available":95260061696,"internally_reserved":0,"allocated":761856,"data_stored":375915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":734198208},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":520,"seq":2233382994067,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":118188,"kb_used_data":732,"kb_used_omap":24,"kb_used_meta":117415,"kb_avail":93626964,"statfs":{"total":95995035648,"available":95874011136,"internally_reserved":0,"allocated":749568,"data_stored":369615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25357,"internal_metadata":120233203},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267133,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1061116,"kb_used_data":430192,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92684036,"statfs":{"total":95995035648,"available":94908452864,"internally_reserved":0,"allocated":440516608,"data_stored":440127115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":646052298},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568496066,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575324,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1271004,"kb_used_data":455440,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92474148,"statfs":{"total":95995035648,"available":94693527552,"internally_reserved":0,"allocated":466370560,"data_stored":465996595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":835123654},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016733,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483422,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":209404,"kb_used_data":744,"kb_used_omap":4,"kb_used_meta":208635,"kb_avail":93535748,"statfs":{"total":95995035648,"available":95780605952,"internally_reserved":0,"allocated":761856,"data_stored":375915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":213642947},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150163,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":875988,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92869164,"statfs":{"total":95995035648,"available":95098023936,"internally_reserved":0,"allocated":724992,"data_stored":357015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":896268743},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986635,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":699708,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":93045444,"statfs":{"total":95995035648,"available":95278534656,"internally_reserved":0,"allocated":761856,"data_stored":375915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790407,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":735772,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":93009380,"statfs":{"total":95995035648,"available":95241605120,"internally_reserved":0,"allocated":737280,"data_stored":363315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":752679365},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:39:03.881 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-15T23:39:03.900 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:04.153 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:39:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:04.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:39:04.259 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:39:04.300 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:04.330 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:39:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:04.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:39:04.433 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:04 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:39:04.433 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:04 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:39:04.436 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:39:04.476 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:04.495 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:39:04.521 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:39:04.551 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:39:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:04 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:39:04.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:04 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:39:04.554 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 2 16 10788 10772 336.549 317.562 0.0019686 0.00296712 2024-05-15T23:39:04.562 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T23:39:04.582 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:39:04.611 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:39:04.644 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:39:04.682 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:39:04.683 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:39:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:04.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:39:04.687 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:04 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:39:04.688 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:04 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:39:04.712 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:39:04.740 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:39:04.768 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:39:04.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:39:04.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:39:04.826 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:04.828 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:39:04.830 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:39:04.830 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:39:04.830 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:39:04.830 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:39:04.831 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:39:04.831 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:39:04.831 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:39:04.831 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:39:04.831 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:39:04.831 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:39:04.831 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:39:04.831 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:39:04.851 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:39:04.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:04.856 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:39:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:39:04.957 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:39:04.993 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:05.451 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:39:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:05.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:39:05.554 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 3 15 15513 15498 322.818 295.375 0.00198954 0.00309451 2024-05-15T23:39:05.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:39:05.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:05 smithi088 ceph-mon[44699]: osdmap e744: 12 total, 12 up, 10 in 2024-05-15T23:39:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:05 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1322534839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:05 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:05 smithi088 ceph-mon[44699]: pgmap v1977: 17 pgs: 17 active+clean; 259 MiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 37 MiB/s wr, 595 op/s 2024-05-15T23:39:05.602 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:05.638 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:39:05.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:05.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:39:05.743 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:39:05.779 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:05.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:05 smithi090 ceph-mon[44434]: osdmap e744: 12 total, 12 up, 10 in 2024-05-15T23:39:05.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:05 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1322534839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:05.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:05 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:05 smithi090 ceph-mon[44434]: pgmap v1977: 17 pgs: 17 active+clean; 259 MiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 37 MiB/s wr, 595 op/s 2024-05-15T23:39:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:05 smithi084 ceph-mon[39784]: osdmap e744: 12 total, 12 up, 10 in 2024-05-15T23:39:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:05 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1322534839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:05 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:05 smithi084 ceph-mon[39784]: pgmap v1977: 17 pgs: 17 active+clean; 259 MiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 37 MiB/s wr, 595 op/s 2024-05-15T23:39:06.554 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 4 16 19337 19321 301.845 238.938 0.00808099 0.00331103 2024-05-15T23:39:06.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:06 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:06.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:06 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:06.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:06 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:06.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:06 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:06.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:06 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:06.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:06 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:06.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:06 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:06.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:06 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:06.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:06 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:07.554 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 5 15 22975 22960 286.961 227.438 0.00191745 0.00348209 2024-05-15T23:39:07.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:07 smithi090 ceph-mon[44434]: pgmap v1978: 17 pgs: 17 active+clean; 993 MiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-05-15T23:39:07.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:07 smithi088 ceph-mon[44699]: pgmap v1978: 17 pgs: 17 active+clean; 993 MiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-05-15T23:39:07.839 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:39:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:07.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:39:07.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:07 smithi084 ceph-mon[39784]: pgmap v1978: 17 pgs: 17 active+clean; 993 MiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-05-15T23:39:07.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:39:07.990 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:39:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:08.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:39:08.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:39:08.124 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:39:08.163 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:08.362 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:39:08.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:08.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:39:08.449 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3892439683' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:39:08.467 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:39:08.468 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:39:08.469 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:39:08.502 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:08.554 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 6 16 26670 26654 277.607 230.875 0.00251173 0.00357361 2024-05-15T23:39:08.684 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:39:08.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:08.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:39:08.789 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:08.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3892439683' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:39:08.826 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:08.853 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:08.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:08.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:08.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3892439683' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:39:08.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:39:09.000 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:09.028 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:39:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:09.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:39:09.134 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:39:09.175 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:09.203 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:39:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:09.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:39:09.309 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:39:09.310 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:39:09.311 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:39:09.350 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:09.554 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 7 15 30348 30333 270.794 229.938 0.00633803 0.00369075 2024-05-15T23:39:09.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:09 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:39:09.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:09 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:39:09.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:09 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3892439683' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:39:09.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:09 smithi084 ceph-mon[39784]: osdmap e745: 12 total, 12 up, 10 in 2024-05-15T23:39:09.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:09 smithi084 ceph-mon[39784]: pgmap v1980: 17 pgs: 17 active+clean; 1.2 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-05-15T23:39:09.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:09 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:39:09.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:09 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:39:09.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:09 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3892439683' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:39:09.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:09 smithi090 ceph-mon[44434]: osdmap e745: 12 total, 12 up, 10 in 2024-05-15T23:39:09.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:09 smithi090 ceph-mon[44434]: pgmap v1980: 17 pgs: 17 active+clean; 1.2 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-05-15T23:39:09.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:09 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:39:09.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:09 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:39:09.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:09 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3892439683' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:39:09.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:09 smithi088 ceph-mon[44699]: osdmap e745: 12 total, 12 up, 10 in 2024-05-15T23:39:09.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:09 smithi088 ceph-mon[44699]: pgmap v1980: 17 pgs: 17 active+clean; 1.2 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-05-15T23:39:10.142 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T23:39:10.496 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:39:10.555 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 8 16 33502 33486 261.575 197.062 0.00197415 0.00379015 2024-05-15T23:39:10.705 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:39:10.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:10.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:39:10.810 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:39:10.850 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:10.986 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:39:10.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:10.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:39:11.090 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:39:11.130 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:11.555 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 9 16 36540 36524 253.606 189.875 0.00167527 0.00392735 2024-05-15T23:39:11.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:11 smithi084 ceph-mon[39784]: pgmap v1981: 17 pgs: 17 active+clean; 1.5 GiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 191 MiB/s wr, 3.05k op/s 2024-05-15T23:39:11.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:11 smithi090 ceph-mon[44434]: pgmap v1981: 17 pgs: 17 active+clean; 1.5 GiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 191 MiB/s wr, 3.05k op/s 2024-05-15T23:39:11.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:11 smithi088 ceph-mon[44699]: pgmap v1981: 17 pgs: 17 active+clean; 1.5 GiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 191 MiB/s wr, 3.05k op/s 2024-05-15T23:39:12.555 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 10 15 39630 39615 247.562 193.188 0.00395908 0.0040376 2024-05-15T23:39:12.652 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:39:12.652 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:39:12.669 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:39:12.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:12.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:39:12.773 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:39:12.812 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:12.917 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:39:12.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:12.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:39:13.024 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5 2024-05-15T23:39:13.061 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:13.470 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T23:39:13.514 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5" found: no such container 2024-05-15T23:39:13.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:13.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.5' 2024-05-15T23:39:13.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:13 smithi084 ceph-mon[39784]: pgmap v1982: 17 pgs: 17 active+clean; 2.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 248 MiB/s wr, 3.97k op/s 2024-05-15T23:39:13.555 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 11 16 42712 42696 242.561 192.562 0.00205573 0.00412049 2024-05-15T23:39:13.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:39:13.659 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:13 smithi090 ceph-mon[44434]: pgmap v1982: 17 pgs: 17 active+clean; 2.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 248 MiB/s wr, 3.97k op/s 2024-05-15T23:39:13.660 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:13.686 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:39:13.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:13.691 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:39:13.792 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:13.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:13 smithi088 ceph-mon[44699]: pgmap v1982: 17 pgs: 17 active+clean; 2.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 248 MiB/s wr, 3.97k op/s 2024-05-15T23:39:13.832 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:13.861 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:13.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:13.888 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:39:13.968 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:39:14.004 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:14.086 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:39:14.115 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:39:14.142 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:39:14.162 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:39:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:14.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:39:14.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:39:14.202 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:39:14.233 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:39:14.263 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:39:14.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:39:14.297 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:39:14.314 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:14.327 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:39:14.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:39:14.354 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:39:14.383 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:39:14.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:14.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:39:14.394 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:39:14.430 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:39:14.442 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:39:14.442 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:39:14.442 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:39:14.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:39:14.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:39:14.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:39:14.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:39:14.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:39:14.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:39:14.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:39:14.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:39:14.444 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:39:14.485 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:14 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3363768412' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:14.485 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:14.485 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:14 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:14.492 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:39:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:14.555 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 12 16 45563 45547 237.196 178.188 0.0105687 0.00421301 2024-05-15T23:39:14.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:14 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3363768412' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:14.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:14.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:14 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:14.738 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:39:14.788 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:39:14.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:14.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:39:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:14 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3363768412' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:14 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:14.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:39:14.927 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:15.555 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 13 16 48292 48276 232.069 170.562 0.00699217 0.0043078 2024-05-15T23:39:15.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:15 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:15 smithi088 ceph-mon[44699]: pgmap v1983: 17 pgs: 17 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 229 MiB/s wr, 3.67k op/s 2024-05-15T23:39:15.788 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:39:15.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:15.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:39:15.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:15 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:15 smithi084 ceph-mon[39784]: pgmap v1983: 17 pgs: 17 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 229 MiB/s wr, 3.67k op/s 2024-05-15T23:39:15.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:15 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:15.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:15 smithi090 ceph-mon[44434]: pgmap v1983: 17 pgs: 17 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 229 MiB/s wr, 3.67k op/s 2024-05-15T23:39:15.894 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:39:15.933 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:15.979 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:39:15.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:15.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:39:16.085 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:39:16.127 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:16.182 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:39:16.182 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:39:16.182 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:39:16.182 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:39:16.183 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:39:16.552 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:16 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:39:16.552 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:16 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:39:16.558 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 14 16 51001 50985 227.586 169.312 0.00493534 0.00439197 2024-05-15T23:39:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:16 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:39:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:16 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:39:16.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:16 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:39:16.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:16 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:39:17.006 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T23:39:17.370 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:39:17.403 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:39:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:17.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:39:17.509 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:17.547 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:17.556 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 15 16 53652 53636 223.447 165.688 0.0032195 0.00447399 2024-05-15T23:39:17.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:17 smithi084 ceph-mon[39784]: osdmap e746: 12 total, 12 up, 10 in 2024-05-15T23:39:17.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:17 smithi084 ceph-mon[39784]: pgmap v1985: 17 pgs: 17 active+clean; 3.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 213 MiB/s wr, 3.41k op/s 2024-05-15T23:39:17.631 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:17.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:17.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:17.692 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:17 smithi088 ceph-mon[44699]: osdmap e746: 12 total, 12 up, 10 in 2024-05-15T23:39:17.692 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:17 smithi088 ceph-mon[44699]: pgmap v1985: 17 pgs: 17 active+clean; 3.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 213 MiB/s wr, 3.41k op/s 2024-05-15T23:39:17.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:39:17.774 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:17.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:17 smithi090 ceph-mon[44434]: osdmap e746: 12 total, 12 up, 10 in 2024-05-15T23:39:17.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:17 smithi090 ceph-mon[44434]: pgmap v1985: 17 pgs: 17 active+clean; 3.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 213 MiB/s wr, 3.41k op/s 2024-05-15T23:39:17.823 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:39:17.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:17.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:39:17.932 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:39:17.969 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:18.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:18 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:39:18.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:18 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:39:18.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:18 smithi084 ceph-mon[39784]: osdmap e747: 12 total, 12 up, 10 in 2024-05-15T23:39:18.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:39:18.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:39:18.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:39:18.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:39:18.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:39:18.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:18 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:39:18.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:18 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:39:18.558 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 16 16 56182 56166 219.344 158.125 0.00991197 0.00455751 2024-05-15T23:39:18.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:18 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:39:18.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:18 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:39:18.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:18 smithi090 ceph-mon[44434]: osdmap e747: 12 total, 12 up, 10 in 2024-05-15T23:39:18.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:39:18.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:39:18.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:39:18.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:39:18.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:39:18.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:18 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:39:18.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:18 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:39:18.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:18 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:39:18.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:18 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:39:18.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:18 smithi088 ceph-mon[44699]: osdmap e747: 12 total, 12 up, 10 in 2024-05-15T23:39:18.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:39:18.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:39:18.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:39:18.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:39:18.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:39:18.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:18 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:39:18.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:18 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:39:19.439 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:39:19.558 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 17 16 58602 58586 215.339 151.25 0.00500921 0.00464271 2024-05-15T23:39:19.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3632321556' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:39:19.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:19 smithi090 ceph-mon[44434]: from='client.27313 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:39:19.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:19 smithi090 ceph-mon[44434]: pgmap v1987: 17 pgs: 17 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 209 MiB/s wr, 3.35k op/s 2024-05-15T23:39:19.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3632321556' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:39:19.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:19 smithi088 ceph-mon[44699]: from='client.27313 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:39:19.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:19 smithi088 ceph-mon[44699]: pgmap v1987: 17 pgs: 17 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 209 MiB/s wr, 3.35k op/s 2024-05-15T23:39:19.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:39:19.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3632321556' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:39:19.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:19 smithi084 ceph-mon[39784]: from='client.27313 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:39:19.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:19 smithi084 ceph-mon[39784]: pgmap v1987: 17 pgs: 17 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 209 MiB/s wr, 3.35k op/s 2024-05-15T23:39:20.328 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:39:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:20.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:39:20.434 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:20.470 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:20.558 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 18 16 60845 60829 211.163 140.188 0.00267783 0.00471886 2024-05-15T23:39:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:20 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:39:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:20 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:39:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:20 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:39:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:20 smithi090 ceph-mon[44434]: from='client.27313 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:39:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:20 smithi090 ceph-mon[44434]: osdmap e748: 12 total, 12 up, 10 in 2024-05-15T23:39:20.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:20 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:39:20.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:20 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:39:20.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:20 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:39:20.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:20 smithi088 ceph-mon[44699]: from='client.27313 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:39:20.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:20 smithi088 ceph-mon[44699]: osdmap e748: 12 total, 12 up, 10 in 2024-05-15T23:39:20.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:20 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:39:20.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:20 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:39:20.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:20 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:39:20.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:20 smithi084 ceph-mon[39784]: from='client.27313 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:39:20.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:20 smithi084 ceph-mon[39784]: osdmap e748: 12 total, 12 up, 10 in 2024-05-15T23:39:21.425 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:39:21.558 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 19 16 63181 63165 207.733 146 0.00476994 0.0048121 2024-05-15T23:39:21.690 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:21 smithi084 ceph-mon[39784]: pgmap v1989: 17 pgs: 17 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-05-15T23:39:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:21 smithi090 ceph-mon[44434]: pgmap v1989: 17 pgs: 17 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-05-15T23:39:21.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:21 smithi088 ceph-mon[44699]: pgmap v1989: 17 pgs: 17 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-05-15T23:39:22.156 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:22.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:22.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:22.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:39:22.313 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:22.341 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:39:22.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:22.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:39:22.447 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:39:22.486 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:22.511 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:39:22.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:22.515 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:39:22.558 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:39:22.558467 min lat: 0.00103205 max lat: 0.205005 avg lat: 0.00490889 2024-05-15T23:39:22.558 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:39:22.558 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 20 16 65198 65182 203.65 126.062 0.0051686 0.00490889 2024-05-15T23:39:22.615 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:39:22.658 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:22.794 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:39:22.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:22.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:39:22.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:39:22.935 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:22.971 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:39:22.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:22.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:39:23.078 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:39:23.114 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:23.147 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:39:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:23.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:39:23.248 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:39:23.248 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:39:23.248 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:39:23.248 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:39:23.248 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.248 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:208426 7.0 7:069186d7:::benchmark_data_smithi090_143373_object62255:head [delete] snapc 0=[] ondisk+write+known_if_redirected e607)", 2024-05-15T23:39:23.248 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:31:54.763469+0000", 2024-05-15T23:39:23.248 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 448.48123301599998, 2024-05-15T23:39:23.249 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.068690670999999995, 2024-05-15T23:39:23.249 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.249 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.249 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.249 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.249 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:39:23.249 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:39:23.249 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 208426 2024-05-15T23:39:23.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.249 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.249 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.249 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.763469+0000", 2024-05-15T23:39:23.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.763469+0000", 2024-05-15T23:39:23.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.250 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.250 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.250 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.763470+0000", 2024-05-15T23:39:23.250 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2109999999999999e-06 2024-05-15T23:39:23.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.763475+0000", 2024-05-15T23:39:23.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4429999999999996e-06 2024-05-15T23:39:23.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.763486+0000", 2024-05-15T23:39:23.251 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1219e-05 2024-05-15T23:39:23.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.251 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.251 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.251 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.763491+0000", 2024-05-15T23:39:23.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6629999999999998e-06 2024-05-15T23:39:23.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.763550+0000", 2024-05-15T23:39:23.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8863999999999997e-05 2024-05-15T23:39:23.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.252 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.252 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.763629+0000", 2024-05-15T23:39:23.252 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8320999999999998e-05 2024-05-15T23:39:23.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.252 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:39:23.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.763698+0000", 2024-05-15T23:39:23.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8975999999999998e-05 2024-05-15T23:39:23.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.766716+0000", 2024-05-15T23:39:23.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0030180419999999999 2024-05-15T23:39:23.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.253 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.253 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.253 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.832093+0000", 2024-05-15T23:39:23.253 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.065377085000000001 2024-05-15T23:39:23.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.832127+0000", 2024-05-15T23:39:23.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4541e-05 2024-05-15T23:39:23.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.254 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.254 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.254 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.832160+0000", 2024-05-15T23:39:23.254 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2305999999999999e-05 2024-05-15T23:39:23.254 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.254 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.254 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.255 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.255 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:208441 7.0 7:0fb5bef7:::benchmark_data_smithi090_143373_object62270:head [delete] snapc 0=[] ondisk+write+known_if_redirected e607)", 2024-05-15T23:39:23.255 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:31:54.781295+0000", 2024-05-15T23:39:23.255 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 448.46340659100002, 2024-05-15T23:39:23.255 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.069236746000000002, 2024-05-15T23:39:23.255 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.255 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.255 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.255 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.255 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:39:23.255 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:39:23.255 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 208441 2024-05-15T23:39:23.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.255 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.781295+0000", 2024-05-15T23:39:23.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.781295+0000", 2024-05-15T23:39:23.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.256 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.256 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.256 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.781297+0000", 2024-05-15T23:39:23.256 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.336e-06 2024-05-15T23:39:23.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.781301+0000", 2024-05-15T23:39:23.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7140000000000001e-06 2024-05-15T23:39:23.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.781311+0000", 2024-05-15T23:39:23.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6039999999999999e-06 2024-05-15T23:39:23.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.257 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.257 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.257 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.781317+0000", 2024-05-15T23:39:23.257 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1550000000000004e-06 2024-05-15T23:39:23.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.781368+0000", 2024-05-15T23:39:23.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1165999999999998e-05 2024-05-15T23:39:23.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.781483+0000", 2024-05-15T23:39:23.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000114974 2024-05-15T23:39:23.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.258 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.258 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:39:23.258 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.781598+0000", 2024-05-15T23:39:23.258 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011436 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.785965+0000", 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0043676369999999997 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.850430+0000", 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.064465092000000002 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.850486+0000", 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6002999999999999e-05 2024-05-15T23:39:23.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:54.850532+0000", 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5704999999999997e-05 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:211293 7.0 7:0a0a72e3:::benchmark_data_smithi090_143373_object65122:head [delete] snapc 0=[] ondisk+write+known_if_redirected e607)", 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:31:58.027447+0000", 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 445.21725483199998, 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.120775434, 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.260 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 211293 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:58.027447+0000", 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:58.027447+0000", 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.261 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:58.027449+0000", 2024-05-15T23:39:23.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.649e-06 2024-05-15T23:39:23.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:58.027454+0000", 2024-05-15T23:39:23.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.254e-06 2024-05-15T23:39:23.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.262 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.262 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.262 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:58.027467+0000", 2024-05-15T23:39:23.262 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3114e-05 2024-05-15T23:39:23.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:58.027476+0000", 2024-05-15T23:39:23.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.0450000000000006e-06 2024-05-15T23:39:23.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:58.027538+0000", 2024-05-15T23:39:23.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2022999999999999e-05 2024-05-15T23:39:23.263 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.263 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.263 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.263 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:58.027637+0000", 2024-05-15T23:39:23.263 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8763000000000003e-05 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:58.027707+0000", 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.9819000000000004e-05 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:58.031421+0000", 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037137899999999998 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:58.148082+0000", 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.116661952 2024-05-15T23:39:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:58.148169+0000", 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6211999999999994e-05 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:58.148223+0000", 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3813000000000001e-05 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.17073.0:24586 8.8 8:174f9712:::benchmark_data_smithi090_157496_object24585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e660)", 2024-05-15T23:39:23.265 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:34:21.168522+0000", 2024-05-15T23:39:23.266 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 302.07617993299999, 2024-05-15T23:39:23.266 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.095676910000000004, 2024-05-15T23:39:23.266 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.266 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.266 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.266 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.266 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.17073", 2024-05-15T23:39:23.266 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:39:23.266 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 24586 2024-05-15T23:39:23.266 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.266 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.266 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.266 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.266 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.168522+0000", 2024-05-15T23:39:23.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.168522+0000", 2024-05-15T23:39:23.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.168523+0000", 2024-05-15T23:39:23.267 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1260000000000001e-06 2024-05-15T23:39:23.267 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.267 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.267 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.267 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.168581+0000", 2024-05-15T23:39:23.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7398999999999997e-05 2024-05-15T23:39:23.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.168589+0000", 2024-05-15T23:39:23.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0039999999999999e-06 2024-05-15T23:39:23.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.168596+0000", 2024-05-15T23:39:23.268 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.075e-06 2024-05-15T23:39:23.268 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.268 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.268 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.268 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.260545+0000", 2024-05-15T23:39:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091949271999999999 2024-05-15T23:39:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.260708+0000", 2024-05-15T23:39:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00016363599999999999 2024-05-15T23:39:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:39:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.260832+0000", 2024-05-15T23:39:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012394399999999999 2024-05-15T23:39:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.263624+0000", 2024-05-15T23:39:23.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0027916659999999999 2024-05-15T23:39:23.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.264109+0000", 2024-05-15T23:39:23.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000485212 2024-05-15T23:39:23.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.270 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.270 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.270 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.264162+0000", 2024-05-15T23:39:23.270 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2357e-05 2024-05-15T23:39:23.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.271 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.271 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.264199+0000", 2024-05-15T23:39:23.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7218999999999997e-05 2024-05-15T23:39:23.271 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.271 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.271 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.271 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.271 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.17073.0:24585 8.0 e660/654 8:06aa9e7e:::benchmark_data_smithi090_157496_object24584:head v 660'1506, mlcod=660'1504)", 2024-05-15T23:39:23.271 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:34:21.169055+0000", 2024-05-15T23:39:23.271 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 302.07564710600002, 2024-05-15T23:39:23.271 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.094558575000000006, 2024-05-15T23:39:23.271 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.272 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.272 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.272 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.169055+0000", 2024-05-15T23:39:23.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.272 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.169055+0000", 2024-05-15T23:39:23.272 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.272 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.272 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.272 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.169056+0000", 2024-05-15T23:39:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.215e-06 2024-05-15T23:39:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.169167+0000", 2024-05-15T23:39:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000111031 2024-05-15T23:39:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.260499+0000", 2024-05-15T23:39:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.091331627999999998 2024-05-15T23:39:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.273 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.260511+0000", 2024-05-15T23:39:23.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2019e-05 2024-05-15T23:39:23.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.260578+0000", 2024-05-15T23:39:23.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6863e-05 2024-05-15T23:39:23.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.274 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.274 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.260599+0000", 2024-05-15T23:39:23.274 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.1787e-05 2024-05-15T23:39:23.274 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.274 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.263550+0000", 2024-05-15T23:39:23.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0029511260000000001 2024-05-15T23:39:23.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.275 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.275 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.263613+0000", 2024-05-15T23:39:23.275 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2905999999999997e-05 2024-05-15T23:39:23.275 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.275 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.275 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.275 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.275 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.276 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.17073.0:24596 8.4 8:267a8f0f:::benchmark_data_smithi090_157496_object24595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e660)", 2024-05-15T23:39:23.276 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:34:21.170785+0000", 2024-05-15T23:39:23.276 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 302.07391738000001, 2024-05-15T23:39:23.276 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.092233006000000006, 2024-05-15T23:39:23.276 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.276 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.276 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.276 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.276 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.17073", 2024-05-15T23:39:23.276 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:39:23.276 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 24596 2024-05-15T23:39:23.276 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.276 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.276 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.276 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.170785+0000", 2024-05-15T23:39:23.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.170785+0000", 2024-05-15T23:39:23.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.277 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.170786+0000", 2024-05-15T23:39:23.277 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1659999999999999e-06 2024-05-15T23:39:23.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.277 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.277 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.170858+0000", 2024-05-15T23:39:23.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2414000000000002e-05 2024-05-15T23:39:23.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.260514+0000", 2024-05-15T23:39:23.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.089656352999999994 2024-05-15T23:39:23.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.278 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.278 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.278 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.260526+0000", 2024-05-15T23:39:23.278 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1114000000000001e-05 2024-05-15T23:39:23.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.260598+0000", 2024-05-15T23:39:23.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2230999999999993e-05 2024-05-15T23:39:23.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.260734+0000", 2024-05-15T23:39:23.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013645399999999999 2024-05-15T23:39:23.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.279 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.279 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:39:23.279 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.260882+0000", 2024-05-15T23:39:23.279 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014783099999999999 2024-05-15T23:39:23.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.262602+0000", 2024-05-15T23:39:23.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0017195979999999999 2024-05-15T23:39:23.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.262902+0000", 2024-05-15T23:39:23.280 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00030015499999999997 2024-05-15T23:39:23.280 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.280 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.280 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.280 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.262960+0000", 2024-05-15T23:39:23.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7951000000000001e-05 2024-05-15T23:39:23.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.281 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.281 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:21.263018+0000", 2024-05-15T23:39:23.281 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.7738999999999997e-05 2024-05-15T23:39:23.281 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.281 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.281 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.281 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.281 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.17073.0:98450 8.4 8:2be161a1:::benchmark_data_smithi090_157496_object98449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e672)", 2024-05-15T23:39:23.281 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:34:59.276432+0000", 2024-05-15T23:39:23.281 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 263.968270351, 2024-05-15T23:39:23.282 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.051972404999999999, 2024-05-15T23:39:23.282 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.282 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.282 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.282 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.282 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.17073", 2024-05-15T23:39:23.282 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:39:23.282 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 98450 2024-05-15T23:39:23.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.283 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.276432+0000", 2024-05-15T23:39:23.283 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.283 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.283 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.283 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.276432+0000", 2024-05-15T23:39:23.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.276433+0000", 2024-05-15T23:39:23.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.127e-06 2024-05-15T23:39:23.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.284 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.284 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.276503+0000", 2024-05-15T23:39:23.284 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0195000000000001e-05 2024-05-15T23:39:23.284 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.284 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.276511+0000", 2024-05-15T23:39:23.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.5779999999999999e-06 2024-05-15T23:39:23.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.276517+0000", 2024-05-15T23:39:23.285 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2809999999999999e-06 2024-05-15T23:39:23.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.285 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.285 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.285 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.276555+0000", 2024-05-15T23:39:23.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8260000000000003e-05 2024-05-15T23:39:23.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.276653+0000", 2024-05-15T23:39:23.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8073999999999996e-05 2024-05-15T23:39:23.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:39:23.286 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.276774+0000", 2024-05-15T23:39:23.286 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000120622 2024-05-15T23:39:23.286 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.286 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.286 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.279191+0000", 2024-05-15T23:39:23.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00241771 2024-05-15T23:39:23.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.328313+0000", 2024-05-15T23:39:23.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.049121642 2024-05-15T23:39:23.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.287 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.328354+0000", 2024-05-15T23:39:23.287 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0976999999999997e-05 2024-05-15T23:39:23.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.287 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.287 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.288 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.328404+0000", 2024-05-15T23:39:23.288 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9938999999999997e-05 2024-05-15T23:39:23.288 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.288 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.288 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.288 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.288 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.288 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.17073.0:98457 8.4 8:2d0e2131:::benchmark_data_smithi090_157496_object98456:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e672)", 2024-05-15T23:39:23.288 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:34:59.280202+0000", 2024-05-15T23:39:23.288 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 263.96449943200003, 2024-05-15T23:39:23.288 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.048497693000000001, 2024-05-15T23:39:23.288 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.288 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.288 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.289 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.289 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.17073", 2024-05-15T23:39:23.289 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:39:23.289 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 98457 2024-05-15T23:39:23.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.289 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.289 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.289 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.280202+0000", 2024-05-15T23:39:23.289 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.289 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.289 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.289 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.289 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.280202+0000", 2024-05-15T23:39:23.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.280203+0000", 2024-05-15T23:39:23.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6699999999999999e-07 2024-05-15T23:39:23.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.290 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.280236+0000", 2024-05-15T23:39:23.290 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3263000000000003e-05 2024-05-15T23:39:23.290 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.290 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.290 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.280241+0000", 2024-05-15T23:39:23.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5430000000000004e-06 2024-05-15T23:39:23.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.280245+0000", 2024-05-15T23:39:23.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4599999999999996e-06 2024-05-15T23:39:23.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.291 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.291 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.282727+0000", 2024-05-15T23:39:23.291 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0024815610000000002 2024-05-15T23:39:23.291 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.291 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.282830+0000", 2024-05-15T23:39:23.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000103538 2024-05-15T23:39:23.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:39:23.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.282915+0000", 2024-05-15T23:39:23.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.4975999999999994e-05 2024-05-15T23:39:23.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.292 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.292 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.292 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.284990+0000", 2024-05-15T23:39:23.292 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0020749850000000001 2024-05-15T23:39:23.292 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.328619+0000", 2024-05-15T23:39:23.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043628263 2024-05-15T23:39:23.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.328675+0000", 2024-05-15T23:39:23.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6084e-05 2024-05-15T23:39:23.293 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.293 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.293 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.293 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:34:59.328700+0000", 2024-05-15T23:39:23.293 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5553000000000001e-05 2024-05-15T23:39:23.293 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.294 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.294 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.294 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.294 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.294 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.17073.0:238444 8.4 8:29fa7c54:::benchmark_data_smithi090_157496_object91506:head [delete] snapc 0=[] ondisk+write+known_if_redirected e716)", 2024-05-15T23:39:23.294 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:37:32.044625+0000", 2024-05-15T23:39:23.294 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 111.20007645699999, 2024-05-15T23:39:23.294 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.093602757999999994, 2024-05-15T23:39:23.294 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.294 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.294 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.294 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.294 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.17073", 2024-05-15T23:39:23.294 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:39:23.295 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 238444 2024-05-15T23:39:23.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.295 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.044625+0000", 2024-05-15T23:39:23.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.295 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.295 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.044625+0000", 2024-05-15T23:39:23.295 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.295 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.295 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.044627+0000", 2024-05-15T23:39:23.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9099999999999999e-06 2024-05-15T23:39:23.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.044634+0000", 2024-05-15T23:39:23.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2179999999999996e-06 2024-05-15T23:39:23.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.296 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.296 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.296 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.044645+0000", 2024-05-15T23:39:23.296 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1764e-05 2024-05-15T23:39:23.296 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.044655+0000", 2024-05-15T23:39:23.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.1560000000000001e-06 2024-05-15T23:39:23.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.044720+0000", 2024-05-15T23:39:23.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5900000000000003e-05 2024-05-15T23:39:23.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.297 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.297 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.297 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.044849+0000", 2024-05-15T23:39:23.297 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012831299999999999 2024-05-15T23:39:23.297 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:39:23.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.044981+0000", 2024-05-15T23:39:23.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013217700000000001 2024-05-15T23:39:23.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.298 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.049110+0000", 2024-05-15T23:39:23.298 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0041295699999999999 2024-05-15T23:39:23.298 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.298 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.298 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.138110+0000", 2024-05-15T23:39:23.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.088999358000000001 2024-05-15T23:39:23.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.138174+0000", 2024-05-15T23:39:23.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3961000000000006e-05 2024-05-15T23:39:23.302 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.302 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.302 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.302 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.138228+0000", 2024-05-15T23:39:23.302 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4431000000000001e-05 2024-05-15T23:39:23.302 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.303 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.303 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.303 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.303 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.303 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.17073.0:238466 8.4 8:27d08635:::benchmark_data_smithi090_157496_object91528:head [delete] snapc 0=[] ondisk+write+known_if_redirected e716)", 2024-05-15T23:39:23.303 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:37:32.073138+0000", 2024-05-15T23:39:23.303 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 111.17156351, 2024-05-15T23:39:23.303 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.071409920000000002, 2024-05-15T23:39:23.303 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.303 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.303 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.303 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.303 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.17073", 2024-05-15T23:39:23.303 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:39:23.303 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 238466 2024-05-15T23:39:23.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.304 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.073138+0000", 2024-05-15T23:39:23.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.073138+0000", 2024-05-15T23:39:23.304 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.304 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.304 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.304 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.304 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.073141+0000", 2024-05-15T23:39:23.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2340000000000001e-06 2024-05-15T23:39:23.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.073147+0000", 2024-05-15T23:39:23.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.1840000000000004e-06 2024-05-15T23:39:23.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.073157+0000", 2024-05-15T23:39:23.305 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.8630000000000003e-06 2024-05-15T23:39:23.305 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.305 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.305 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.305 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.073163+0000", 2024-05-15T23:39:23.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3520000000000003e-06 2024-05-15T23:39:23.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.073226+0000", 2024-05-15T23:39:23.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3244999999999995e-05 2024-05-15T23:39:23.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.306 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.306 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.073364+0000", 2024-05-15T23:39:23.306 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013747600000000001 2024-05-15T23:39:23.306 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.306 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:39:23.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.073496+0000", 2024-05-15T23:39:23.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013244299999999999 2024-05-15T23:39:23.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.077832+0000", 2024-05-15T23:39:23.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0043357170000000002 2024-05-15T23:39:23.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.307 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.307 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.307 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.144477+0000", 2024-05-15T23:39:23.307 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066645108999999994 2024-05-15T23:39:23.307 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.144514+0000", 2024-05-15T23:39:23.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7126000000000003e-05 2024-05-15T23:39:23.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.308 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.308 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.144548+0000", 2024-05-15T23:39:23.308 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4171000000000002e-05 2024-05-15T23:39:23.308 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.308 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.308 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.308 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.308 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.309 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.17073.0:238476 8.4 8:237e8e3e:::benchmark_data_smithi090_157496_object91538:head [delete] snapc 0=[] ondisk+write+known_if_redirected e716)", 2024-05-15T23:39:23.309 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:37:32.096682+0000", 2024-05-15T23:39:23.309 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 111.148019914, 2024-05-15T23:39:23.309 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.057191872999999997, 2024-05-15T23:39:23.309 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.309 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.309 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.309 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.309 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.17073", 2024-05-15T23:39:23.309 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:39:23.309 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 238476 2024-05-15T23:39:23.309 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.309 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.309 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.309 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.096682+0000", 2024-05-15T23:39:23.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.096682+0000", 2024-05-15T23:39:23.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.310 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.310 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.096683+0000", 2024-05-15T23:39:23.310 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.302e-06 2024-05-15T23:39:23.310 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.310 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.096688+0000", 2024-05-15T23:39:23.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2370000000000003e-06 2024-05-15T23:39:23.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.096698+0000", 2024-05-15T23:39:23.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.049e-05 2024-05-15T23:39:23.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.311 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.311 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.311 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.096706+0000", 2024-05-15T23:39:23.311 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2630000000000003e-06 2024-05-15T23:39:23.311 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.096762+0000", 2024-05-15T23:39:23.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5432000000000001e-05 2024-05-15T23:39:23.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.096844+0000", 2024-05-15T23:39:23.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2515000000000005e-05 2024-05-15T23:39:23.312 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.312 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.312 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:39:23.312 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.096917+0000", 2024-05-15T23:39:23.312 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2775999999999996e-05 2024-05-15T23:39:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.100758+0000", 2024-05-15T23:39:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0038405290000000001 2024-05-15T23:39:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.153790+0000", 2024-05-15T23:39:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.053031994999999998 2024-05-15T23:39:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.153833+0000", 2024-05-15T23:39:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3420000000000001e-05 2024-05-15T23:39:23.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.314 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.314 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.153874+0000", 2024-05-15T23:39:23.314 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0914000000000002e-05 2024-05-15T23:39:23.314 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.314 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.314 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.314 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.314 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.314 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.17073.0:238477 8.4 8:29a86e28:::benchmark_data_smithi090_157496_object91539:head [delete] snapc 0=[] ondisk+write+known_if_redirected e716)", 2024-05-15T23:39:23.314 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:37:32.101460+0000", 2024-05-15T23:39:23.314 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 111.143241457, 2024-05-15T23:39:23.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.057859170000000001, 2024-05-15T23:39:23.315 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.315 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.315 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.315 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.315 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.17073", 2024-05-15T23:39:23.315 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:39:23.315 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 238477 2024-05-15T23:39:23.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.315 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.315 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.315 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.315 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.101460+0000", 2024-05-15T23:39:23.315 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.315 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.101460+0000", 2024-05-15T23:39:23.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.101462+0000", 2024-05-15T23:39:23.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.79e-06 2024-05-15T23:39:23.316 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.316 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.316 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.316 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.101468+0000", 2024-05-15T23:39:23.316 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4049999999999999e-06 2024-05-15T23:39:23.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.101479+0000", 2024-05-15T23:39:23.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1749000000000001e-05 2024-05-15T23:39:23.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.101489+0000", 2024-05-15T23:39:23.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6430000000000007e-06 2024-05-15T23:39:23.317 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.317 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.317 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.317 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.101532+0000", 2024-05-15T23:39:23.317 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3346000000000001e-05 2024-05-15T23:39:23.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.101657+0000", 2024-05-15T23:39:23.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000124533 2024-05-15T23:39:23.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:39:23.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.101782+0000", 2024-05-15T23:39:23.318 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012468200000000001 2024-05-15T23:39:23.318 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.318 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.318 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.318 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.106447+0000", 2024-05-15T23:39:23.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0046654349999999999 2024-05-15T23:39:23.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.159267+0000", 2024-05-15T23:39:23.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052819563 2024-05-15T23:39:23.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.319 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.319 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.159296+0000", 2024-05-15T23:39:23.319 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9456999999999999e-05 2024-05-15T23:39:23.319 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.319 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.320 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.320 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:32.159320+0000", 2024-05-15T23:39:23.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3567e-05 2024-05-15T23:39:23.320 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.320 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.320 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.320 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.320 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.320 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.17073.0:259292 8.4 8:245380c2:::benchmark_data_smithi090_157496_object112354:head [delete] snapc 0=[] ondisk+write+known_if_redirected e719)", 2024-05-15T23:39:23.320 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:37:54.289538+0000", 2024-05-15T23:39:23.320 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 88.955164103000001, 2024-05-15T23:39:23.320 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046771551000000001, 2024-05-15T23:39:23.320 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.320 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.320 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.321 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.321 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.17073", 2024-05-15T23:39:23.321 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/1074821035", 2024-05-15T23:39:23.321 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 259292 2024-05-15T23:39:23.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.321 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.321 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.321 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:54.289538+0000", 2024-05-15T23:39:23.321 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.321 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.321 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.321 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.321 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:54.289538+0000", 2024-05-15T23:39:23.321 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.322 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.322 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.322 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:54.289539+0000", 2024-05-15T23:39:23.322 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6899999999999996e-07 2024-05-15T23:39:23.322 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.322 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.322 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:54.289542+0000", 2024-05-15T23:39:23.322 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7330000000000001e-06 2024-05-15T23:39:23.322 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.322 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.322 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.322 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:54.289547+0000", 2024-05-15T23:39:23.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6249999999999998e-06 2024-05-15T23:39:23.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:54.289551+0000", 2024-05-15T23:39:23.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.844e-06 2024-05-15T23:39:23.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.323 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:54.294748+0000", 2024-05-15T23:39:23.323 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0051974539999999998 2024-05-15T23:39:23.323 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.323 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.323 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:54.294879+0000", 2024-05-15T23:39:23.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013109600000000001 2024-05-15T23:39:23.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 9", 2024-05-15T23:39:23.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:54.294962+0000", 2024-05-15T23:39:23.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3044e-05 2024-05-15T23:39:23.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.324 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.324 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:54.303821+0000", 2024-05-15T23:39:23.324 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0088589140000000007 2024-05-15T23:39:23.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.324 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:54.336184+0000", 2024-05-15T23:39:23.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.032362677999999999 2024-05-15T23:39:23.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:54.336258+0000", 2024-05-15T23:39:23.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.3825000000000006e-05 2024-05-15T23:39:23.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.325 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.325 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.325 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:37:54.336309+0000", 2024-05-15T23:39:23.325 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1569e-05 2024-05-15T23:39:23.325 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.325 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.326 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.326 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.326 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.27292.0:58455 9.a 9:528c1472:::benchmark_data_smithi090_166897_object58454:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e748)", 2024-05-15T23:39:23.326 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:39:19.472556+0000", 2024-05-15T23:39:23.326 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 3.7721458910000001, 2024-05-15T23:39:23.326 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047613197000000003, 2024-05-15T23:39:23.326 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.326 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.326 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.326 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.326 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.27292", 2024-05-15T23:39:23.326 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2136483304", 2024-05-15T23:39:23.327 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 58455 2024-05-15T23:39:23.327 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.327 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.327 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.327 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.327 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.472556+0000", 2024-05-15T23:39:23.327 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.327 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.327 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.327 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.327 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.472556+0000", 2024-05-15T23:39:23.327 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.327 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.327 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.328 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.328 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.472558+0000", 2024-05-15T23:39:23.328 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5230000000000001e-06 2024-05-15T23:39:23.328 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.328 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.328 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.328 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.472612+0000", 2024-05-15T23:39:23.328 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4107000000000002e-05 2024-05-15T23:39:23.328 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.328 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.328 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.328 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.472625+0000", 2024-05-15T23:39:23.328 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3720000000000001e-05 2024-05-15T23:39:23.328 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.328 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.472631+0000", 2024-05-15T23:39:23.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6670000000000001e-06 2024-05-15T23:39:23.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.472697+0000", 2024-05-15T23:39:23.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5852999999999994e-05 2024-05-15T23:39:23.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.329 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.329 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.472824+0000", 2024-05-15T23:39:23.329 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000126818 2024-05-15T23:39:23.329 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.329 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:39:23.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.472946+0000", 2024-05-15T23:39:23.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012249299999999999 2024-05-15T23:39:23.330 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.330 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.520051+0000", 2024-05-15T23:39:23.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047105086999999997 2024-05-15T23:39:23.330 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.330 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.330 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.330 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.520072+0000", 2024-05-15T23:39:23.330 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0548999999999999e-05 2024-05-15T23:39:23.330 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.520123+0000", 2024-05-15T23:39:23.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0691e-05 2024-05-15T23:39:23.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.331 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.331 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.520169+0000", 2024-05-15T23:39:23.331 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.6689e-05 2024-05-15T23:39:23.331 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.331 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.331 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.331 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.331 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.27292.0:58459 9.a 9:57fa53e1:::benchmark_data_smithi090_166897_object58458:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e748)", 2024-05-15T23:39:23.332 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:39:19.472659+0000", 2024-05-15T23:39:23.332 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 3.7720433209999999, 2024-05-15T23:39:23.332 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.049688607000000003, 2024-05-15T23:39:23.332 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.332 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.332 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.332 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.332 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.27292", 2024-05-15T23:39:23.332 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2136483304", 2024-05-15T23:39:23.332 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 58459 2024-05-15T23:39:23.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.332 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.332 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.333 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.472659+0000", 2024-05-15T23:39:23.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.333 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.333 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.472659+0000", 2024-05-15T23:39:23.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.333 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.333 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.472659+0000", 2024-05-15T23:39:23.333 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3699999999999999e-07 2024-05-15T23:39:23.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.334 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.334 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.472688+0000", 2024-05-15T23:39:23.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9051e-05 2024-05-15T23:39:23.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.334 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.334 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.334 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.472694+0000", 2024-05-15T23:39:23.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6200000000000004e-06 2024-05-15T23:39:23.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.334 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.334 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.334 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.472698+0000", 2024-05-15T23:39:23.334 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0389999999999998e-06 2024-05-15T23:39:23.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.474716+0000", 2024-05-15T23:39:23.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002018312 2024-05-15T23:39:23.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.474840+0000", 2024-05-15T23:39:23.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012370299999999999 2024-05-15T23:39:23.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.335 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:39:23.335 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.518052+0000", 2024-05-15T23:39:23.335 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043211725999999999 2024-05-15T23:39:23.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.520221+0000", 2024-05-15T23:39:23.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0021690500000000001 2024-05-15T23:39:23.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.522262+0000", 2024-05-15T23:39:23.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.002041218 2024-05-15T23:39:23.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.336 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.336 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.522309+0000", 2024-05-15T23:39:23.336 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.7250000000000003e-05 2024-05-15T23:39:23.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.337 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.337 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.522347+0000", 2024-05-15T23:39:23.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7800999999999999e-05 2024-05-15T23:39:23.337 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.337 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.337 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.337 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.27292.0:58461 9.5 e748/737 9:a5301ffd:::benchmark_data_smithi090_166897_object58460:head v 748'3544, mlcod=748'3542)", 2024-05-15T23:39:23.337 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:39:19.474679+0000", 2024-05-15T23:39:23.337 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 3.770022956, 2024-05-15T23:39:23.337 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.048927855999999999, 2024-05-15T23:39:23.337 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.338 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.338 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.338 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.338 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.338 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.474679+0000", 2024-05-15T23:39:23.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.338 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.338 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.474679+0000", 2024-05-15T23:39:23.338 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.474680+0000", 2024-05-15T23:39:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.159e-06 2024-05-15T23:39:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.474754+0000", 2024-05-15T23:39:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4334999999999999e-05 2024-05-15T23:39:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.474762+0000", 2024-05-15T23:39:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.6129999999999998e-06 2024-05-15T23:39:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.474767+0000", 2024-05-15T23:39:23.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.9080000000000003e-06 2024-05-15T23:39:23.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.474812+0000", 2024-05-15T23:39:23.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5302000000000001e-05 2024-05-15T23:39:23.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.340 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.340 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.474833+0000", 2024-05-15T23:39:23.340 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.1060000000000002e-05 2024-05-15T23:39:23.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.523564+0000", 2024-05-15T23:39:23.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.048730242999999999 2024-05-15T23:39:23.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.341 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.341 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.523607+0000", 2024-05-15T23:39:23.341 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3235999999999997e-05 2024-05-15T23:39:23.341 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.341 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.341 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.341 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.27292.0:58465 9.a 9:5cb1e7db:::benchmark_data_smithi090_166897_object58464:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e748)", 2024-05-15T23:39:23.342 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:39:19.475661+0000", 2024-05-15T23:39:23.342 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 3.7690407500000003, 2024-05-15T23:39:23.342 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.048699157, 2024-05-15T23:39:23.342 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.342 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.342 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.342 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.342 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.27292", 2024-05-15T23:39:23.342 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2136483304", 2024-05-15T23:39:23.342 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 58465 2024-05-15T23:39:23.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.342 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.342 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.475661+0000", 2024-05-15T23:39:23.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.475661+0000", 2024-05-15T23:39:23.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.343 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.475663+0000", 2024-05-15T23:39:23.343 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.393e-06 2024-05-15T23:39:23.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.343 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.475694+0000", 2024-05-15T23:39:23.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1890999999999997e-05 2024-05-15T23:39:23.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.344 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.518888+0000", 2024-05-15T23:39:23.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043193249000000003 2024-05-15T23:39:23.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.344 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.344 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.518900+0000", 2024-05-15T23:39:23.344 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2328e-05 2024-05-15T23:39:23.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.344 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.520252+0000", 2024-05-15T23:39:23.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001352429 2024-05-15T23:39:23.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.520363+0000", 2024-05-15T23:39:23.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000110763 2024-05-15T23:39:23.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.345 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:39:23.345 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.520483+0000", 2024-05-15T23:39:23.345 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000120141 2024-05-15T23:39:23.345 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.345 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.346 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.346 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.524242+0000", 2024-05-15T23:39:23.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037587459999999999 2024-05-15T23:39:23.346 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.346 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.346 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.346 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.524301+0000", 2024-05-15T23:39:23.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9002000000000002e-05 2024-05-15T23:39:23.346 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.346 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.346 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.346 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.524336+0000", 2024-05-15T23:39:23.346 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4654999999999999e-05 2024-05-15T23:39:23.346 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.347 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.347 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.524360+0000", 2024-05-15T23:39:23.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4559999999999999e-05 2024-05-15T23:39:23.347 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.347 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.347 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.347 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.347 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.347 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.27292.0:60826 9.6 9:6c66e853:::benchmark_data_smithi090_166897_object60825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e748)", 2024-05-15T23:39:23.347 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:39:20.496460+0000", 2024-05-15T23:39:23.347 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 2.7482422680000003, 2024-05-15T23:39:23.347 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.069091808000000005, 2024-05-15T23:39:23.347 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.348 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.348 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.348 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.348 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.27292", 2024-05-15T23:39:23.348 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2136483304", 2024-05-15T23:39:23.348 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 60826 2024-05-15T23:39:23.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.348 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.348 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.496460+0000", 2024-05-15T23:39:23.348 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.348 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.348 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.348 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.349 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.496460+0000", 2024-05-15T23:39:23.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.349 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.349 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.496461+0000", 2024-05-15T23:39:23.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.854e-06 2024-05-15T23:39:23.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.349 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.349 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.496550+0000", 2024-05-15T23:39:23.349 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.8733999999999994e-05 2024-05-15T23:39:23.349 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.349 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.496562+0000", 2024-05-15T23:39:23.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1343e-05 2024-05-15T23:39:23.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.496570+0000", 2024-05-15T23:39:23.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.884e-06 2024-05-15T23:39:23.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.350 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.350 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.350 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.496633+0000", 2024-05-15T23:39:23.350 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.2436999999999999e-05 2024-05-15T23:39:23.350 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.496793+0000", 2024-05-15T23:39:23.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000160156 2024-05-15T23:39:23.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:39:23.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.496992+0000", 2024-05-15T23:39:23.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00019869399999999999 2024-05-15T23:39:23.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.351 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.351 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.351 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.498722+0000", 2024-05-15T23:39:23.351 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0017306349999999999 2024-05-15T23:39:23.351 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.565455+0000", 2024-05-15T23:39:23.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.066732993000000004 2024-05-15T23:39:23.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.565510+0000", 2024-05-15T23:39:23.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4591e-05 2024-05-15T23:39:23.352 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.352 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.352 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.352 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.565551+0000", 2024-05-15T23:39:23.352 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1486999999999998e-05 2024-05-15T23:39:23.352 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.353 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.353 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.353 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.353 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.353 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.27292.0:60839 9.f 9:ffb8d952:::benchmark_data_smithi090_166897_object60838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e748)", 2024-05-15T23:39:23.353 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:39:20.503852+0000", 2024-05-15T23:39:23.353 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 2.7408503959999999, 2024-05-15T23:39:23.353 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.057781802, 2024-05-15T23:39:23.353 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.353 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.353 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.353 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.353 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.27292", 2024-05-15T23:39:23.353 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2136483304", 2024-05-15T23:39:23.354 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 60839 2024-05-15T23:39:23.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.354 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.503852+0000", 2024-05-15T23:39:23.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.354 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.503852+0000", 2024-05-15T23:39:23.354 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.354 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.354 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.354 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.503855+0000", 2024-05-15T23:39:23.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.066e-06 2024-05-15T23:39:23.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.503904+0000", 2024-05-15T23:39:23.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8946000000000002e-05 2024-05-15T23:39:23.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.355 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.355 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.503929+0000", 2024-05-15T23:39:23.355 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5876000000000001e-05 2024-05-15T23:39:23.355 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.355 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.503945+0000", 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5486e-05 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.504052+0000", 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000107055 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.504243+0000", 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00019088500000000001 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:39:23.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.504332+0000", 2024-05-15T23:39:23.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.9653000000000004e-05 2024-05-15T23:39:23.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.357 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.505940+0000", 2024-05-15T23:39:23.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001607523 2024-05-15T23:39:23.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.357 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.357 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.561533+0000", 2024-05-15T23:39:23.357 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.055593324999999999 2024-05-15T23:39:23.357 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.357 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.357 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.358 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.561588+0000", 2024-05-15T23:39:23.358 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4177e-05 2024-05-15T23:39:23.358 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.358 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.358 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.561633+0000", 2024-05-15T23:39:23.358 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5809999999999997e-05 2024-05-15T23:39:23.358 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.358 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.358 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.358 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.358 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.358 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.27292.0:60845 9.6 9:67d49a53:::benchmark_data_smithi090_166897_object60844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e748)", 2024-05-15T23:39:23.358 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:39:20.513533+0000", 2024-05-15T23:39:23.359 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 2.7311687149999999, 2024-05-15T23:39:23.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.052140965999999997, 2024-05-15T23:39:23.359 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:23.359 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:23.359 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:23.359 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:23.359 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.27292", 2024-05-15T23:39:23.359 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2136483304", 2024-05-15T23:39:23.359 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 60845 2024-05-15T23:39:23.359 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.359 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:23.359 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.359 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:23.359 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.513533+0000", 2024-05-15T23:39:23.359 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:23.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.513533+0000", 2024-05-15T23:39:23.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:23.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:23.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.513534+0000", 2024-05-15T23:39:23.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.178e-06 2024-05-15T23:39:23.360 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.360 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.360 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:23.360 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.513556+0000", 2024-05-15T23:39:23.360 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.1735000000000001e-05 2024-05-15T23:39:23.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:23.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.513566+0000", 2024-05-15T23:39:23.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.6490000000000003e-06 2024-05-15T23:39:23.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:23.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.513573+0000", 2024-05-15T23:39:23.361 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7090000000000002e-06 2024-05-15T23:39:23.361 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.361 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.361 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:23.361 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.513616+0000", 2024-05-15T23:39:23.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2704999999999999e-05 2024-05-15T23:39:23.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:23.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.513726+0000", 2024-05-15T23:39:23.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000109858 2024-05-15T23:39:23.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:39:23.362 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.513815+0000", 2024-05-15T23:39:23.362 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.9177000000000004e-05 2024-05-15T23:39:23.362 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.362 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.362 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:23.363 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.515418+0000", 2024-05-15T23:39:23.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001602926 2024-05-15T23:39:23.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.363 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:23.363 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.565593+0000", 2024-05-15T23:39:23.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.050175143999999998 2024-05-15T23:39:23.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.363 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.363 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:23.363 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.565641+0000", 2024-05-15T23:39:23.363 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.8021e-05 2024-05-15T23:39:23.363 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:23.364 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:23.364 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:23.364 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:20.565674+0000", 2024-05-15T23:39:23.364 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2864000000000002e-05 2024-05-15T23:39:23.364 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.364 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.364 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.364 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:23.364 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:23.364 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:39:23.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:39:23.410 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:23.423 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:39:23.440 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:39:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:23.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:39:23.508 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:39:23.541 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:39:23.545 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:23.558 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 21 16 67360 67344 200.386 135.125 0.0158707 0.0049889 2024-05-15T23:39:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:23 smithi088 ceph-mon[44699]: pgmap v1990: 17 pgs: 17 active+clean; 3.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 199 MiB/s wr, 3.18k op/s 2024-05-15T23:39:23.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:39:23.579 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:23.621 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:39:23.660 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:39:23.692 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-05-15T23:39:23.723 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:39:23.756 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:39:23.786 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:39:23.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:23 smithi090 ceph-mon[44434]: pgmap v1990: 17 pgs: 17 active+clean; 3.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 199 MiB/s wr, 3.18k op/s 2024-05-15T23:39:23.817 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:39:23.849 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:39:23.852 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:39:23.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:23 smithi084 ceph-mon[39784]: pgmap v1990: 17 pgs: 17 active+clean; 3.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 199 MiB/s wr, 3.18k op/s 2024-05-15T23:39:23.873 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:39:23.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:39:23.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:39:23.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:39:23.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:39:23.877 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:39:23.877 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-15T23:39:23.877 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:39:23.877 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:39:23.877 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:39:23.877 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:39:23.877 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:39:23.877 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:39:23.881 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-05-15T23:39:23.881 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 0, 8] 2024-05-15T23:39:23.881 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-15T23:39:23.882 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.5 2024-05-15T23:39:24.320 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:39:23 smithi088 systemd[1]: Stopping Ceph osd.5 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:39:24.510 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:24 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:39:24.536 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:24 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:39:24.536 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:24 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1523082147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:24.536 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.536 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.536 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.536 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.536 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.537 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.537 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.537 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.564 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 22 16 69253 69237 196.656 118.312 0.00369386 0.00508351 2024-05-15T23:39:24.743 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:24 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:39:24.743 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:24 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:39:24.743 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:24 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1523082147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:24.743 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.743 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.743 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.744 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.744 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.744 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.744 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.744 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.781 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T23:39:24.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:24 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:39:24.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:24 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:39:24.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:24 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1523082147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:24.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:24.834 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:24.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:24.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:24.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:39:24.983 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:25.150 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:39:25.398 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:25.398 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:25.398 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:25.399 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:25.399 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:25 smithi088 ceph-mon[44699]: osdmap e749: 12 total, 12 up, 10 in 2024-05-15T23:39:25.399 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:25 smithi088 ceph-mon[44699]: osd.5 marked itself down and dead 2024-05-15T23:39:25.399 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:25 smithi088 ceph-mon[44699]: pgmap v1992: 17 pgs: 17 active+clean; 4.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-05-15T23:39:25.399 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:39:25 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[57847]: 2024-05-15T23:39:25.091+0000 7f0a04e16700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:39:25.399 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:39:25 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[57847]: 2024-05-15T23:39:25.091+0000 7f0a04e16700 -1 osd.5 749 *** Got signal Terminated *** 2024-05-15T23:39:25.399 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:39:25 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[57847]: 2024-05-15T23:39:25.091+0000 7f0a04e16700 -1 osd.5 749 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:39:25.558 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 23 16 71152 71136 193.265 118.688 0.00319373 0.00514313 2024-05-15T23:39:25.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:25.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:25.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:25.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:25.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:25 smithi090 ceph-mon[44434]: osdmap e749: 12 total, 12 up, 10 in 2024-05-15T23:39:25.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:25 smithi090 ceph-mon[44434]: osd.5 marked itself down and dead 2024-05-15T23:39:25.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:25 smithi090 ceph-mon[44434]: pgmap v1992: 17 pgs: 17 active+clean; 4.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-05-15T23:39:25.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:25.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:25.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:25.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:25.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:25 smithi084 ceph-mon[39784]: osdmap e749: 12 total, 12 up, 10 in 2024-05-15T23:39:25.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:25 smithi084 ceph-mon[39784]: osd.5 marked itself down and dead 2024-05-15T23:39:25.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:25 smithi084 ceph-mon[39784]: pgmap v1992: 17 pgs: 17 active+clean; 4.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-05-15T23:39:26.371 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:39:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:26.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:39:26.475 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:26.505 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:26.559 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 24 16 71152 71136 185.213 0 - 0.00514313 2024-05-15T23:39:26.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:26 smithi084 ceph-mon[39784]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-15T23:39:26.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:26 smithi084 ceph-mon[39784]: osdmap e750: 12 total, 11 up, 10 in 2024-05-15T23:39:26.719 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:26 smithi088 ceph-mon[44699]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-15T23:39:26.719 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:26 smithi088 ceph-mon[44699]: osdmap e750: 12 total, 11 up, 10 in 2024-05-15T23:39:26.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:26 smithi090 ceph-mon[44434]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-15T23:39:26.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:26 smithi090 ceph-mon[44434]: osdmap e750: 12 total, 11 up, 10 in 2024-05-15T23:39:27.014 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:39:26 smithi088 bash[167267]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5 2024-05-15T23:39:27.054 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:27.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:27.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:27.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:39:27.191 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:27.212 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:39:27.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:27.216 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:39:27.317 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:39:27.346 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:27.545 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:39:27.547 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:27 smithi084 ceph-mon[39784]: osdmap e751: 12 total, 11 up, 10 in 2024-05-15T23:39:27.547 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:27 smithi084 ceph-mon[39784]: pgmap v1995: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 160 MiB/s wr, 2.57k op/s; 13135/140240 objects degraded (9.366%) 2024-05-15T23:39:27.547 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:27 smithi084 ceph-mon[39784]: osdmap e752: 12 total, 11 up, 10 in 2024-05-15T23:39:27.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:27.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:39:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:27 smithi090 ceph-mon[44434]: osdmap e751: 12 total, 11 up, 10 in 2024-05-15T23:39:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:27 smithi090 ceph-mon[44434]: pgmap v1995: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 160 MiB/s wr, 2.57k op/s; 13135/140240 objects degraded (9.366%) 2024-05-15T23:39:27.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:27 smithi090 ceph-mon[44434]: osdmap e752: 12 total, 11 up, 10 in 2024-05-15T23:39:27.558 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 25 16 71152 71136 177.805 0 - 0.00514313 2024-05-15T23:39:27.582 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:39:27.582 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:39:27.649 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:39:27.676 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:27.695 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:39:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:27.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:39:27.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:39:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:27 smithi088 ceph-mon[44699]: osdmap e751: 12 total, 11 up, 10 in 2024-05-15T23:39:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:27 smithi088 ceph-mon[44699]: pgmap v1995: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 160 MiB/s wr, 2.57k op/s; 13135/140240 objects degraded (9.366%) 2024-05-15T23:39:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:27 smithi088 ceph-mon[44699]: osdmap e752: 12 total, 11 up, 10 in 2024-05-15T23:39:27.825 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:27.844 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:39:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:27.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:39:27.948 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:39:27.979 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:28.178 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:39:28.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:28.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:39:28.190 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T23:39:28.282 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:39:28.310 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:28.329 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:39:28.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:28.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:39:28.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:39:28.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:28 smithi088 ceph-mon[44699]: Health check failed: Degraded data redundancy: 13135/140240 objects degraded (9.366%), 3 pgs degraded (PG_DEGRADED) 2024-05-15T23:39:28.417 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/290117808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:39:28.424 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:39:28.433 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:39:28.463 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:28.559 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 26 16 71152 71136 170.967 0 - 0.00514313 2024-05-15T23:39:28.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:28 smithi084 ceph-mon[39784]: Health check failed: Degraded data redundancy: 13135/140240 objects degraded (9.366%), 3 pgs degraded (PG_DEGRADED) 2024-05-15T23:39:28.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/290117808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:39:28.661 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:39:28.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:28.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:39:28.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:39:28.796 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:28 smithi090 ceph-mon[44434]: Health check failed: Degraded data redundancy: 13135/140240 objects degraded (9.366%), 3 pgs degraded (PG_DEGRADED) 2024-05-15T23:39:28.796 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/290117808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:39:28.798 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:28.820 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:39:28.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:28.825 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:39:28.926 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:39:28.954 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:28.957 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:39:29.260 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:39:29.559 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 27 16 71152 71136 164.635 0 - 0.00514313 2024-05-15T23:39:29.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:29 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:39:29.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:29 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/290117808' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:39:29.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:29 smithi084 ceph-mon[39784]: osdmap e753: 12 total, 11 up, 10 in 2024-05-15T23:39:29.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:29 smithi084 ceph-mon[39784]: pgmap v1998: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s; 17500/141822 objects degraded (12.339%) 2024-05-15T23:39:29.603 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:39:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:29.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:39:29.641 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:29 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:39:29.642 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:29 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/290117808' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:39:29.642 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:29 smithi088 ceph-mon[44699]: osdmap e753: 12 total, 11 up, 10 in 2024-05-15T23:39:29.642 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:29 smithi088 ceph-mon[44699]: pgmap v1998: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s; 17500/141822 objects degraded (12.339%) 2024-05-15T23:39:29.707 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:39:29.739 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:29.789 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:39:29.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:29.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:39:29.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:29 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:39:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:29 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/290117808' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:39:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:29 smithi090 ceph-mon[44434]: osdmap e753: 12 total, 11 up, 10 in 2024-05-15T23:39:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:29 smithi090 ceph-mon[44434]: pgmap v1998: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s; 17500/141822 objects degraded (12.339%) 2024-05-15T23:39:29.894 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:39:29.920 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:30.120 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:39:30.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:30.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:39:30.224 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:39:30.257 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:30.278 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:39:30.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:30.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:39:30.382 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:39:30.410 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:30.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:30 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:39:30.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:30 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:39:30.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:30 smithi090 ceph-mon[44434]: osdmap e754: 12 total, 11 up, 10 in 2024-05-15T23:39:30.559 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 28 16 71152 71136 158.756 0 - 0.00514313 2024-05-15T23:39:30.569 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:30 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:39:30.569 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:30 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:39:30.569 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:30 smithi088 ceph-mon[44699]: osdmap e754: 12 total, 11 up, 10 in 2024-05-15T23:39:30.593 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:39:30.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:30.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:39:30.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:39:30.731 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:39:30.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:30.757 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:39:30.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:30 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:39:30.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:30 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:39:30.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:30 smithi084 ceph-mon[39784]: osdmap e754: 12 total, 11 up, 10 in 2024-05-15T23:39:30.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:39:30.886 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:31.035 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.5.service' 2024-05-15T23:39:31.085 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:39:31.085 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-15T23:39:31.085 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd down 5 2024-05-15T23:39:31.126 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:39:31.136 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:39:31.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:31.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:39:31.146 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:39:31.165 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:39:31.185 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:39:31.204 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:39:31.224 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:39:31.240 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:31.243 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:39:31.263 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:39:31.264 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:39:31.272 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:31.283 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:39:31.293 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:31.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:31.312 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:39:31.336 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:39:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:39:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:39:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:39:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:39:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:39:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-05-15T23:39:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:39:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:39:31.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:39:31.338 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:39:31.338 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:39:31.338 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:39:31.396 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:31.425 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:31.523 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:31.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:31.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:31.542 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:39:31.543 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:39:31.543 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:39:31.543 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:39:31.543 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:39:31.559 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 29 16 71152 71136 153.281 0 - 0.00514313 2024-05-15T23:39:31.567 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/868102996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:31.567 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:31 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:31.567 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:31 smithi088 ceph-mon[44699]: pgmap v2000: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 63 MiB/s wr, 1.01k op/s; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:31.590 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:39:31.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:39:31.661 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:31.682 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:39:31.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:31.686 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:39:31.787 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:39:31.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/868102996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:31.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:31 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:31.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:31 smithi090 ceph-mon[44434]: pgmap v2000: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 63 MiB/s wr, 1.01k op/s; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:31.813 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/868102996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:31.813 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:31 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:31.813 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:31 smithi084 ceph-mon[39784]: pgmap v2000: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 63 MiB/s wr, 1.01k op/s; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:31.817 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:32.163 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T23:39:32.164 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:39:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:32.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:39:32.268 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:32.313 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:32.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:32.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:32.417 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:39:32.428 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:39:32.437 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.438 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.448 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:32.559 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 30 16 71152 71136 148.172 0 - 0.00514313 2024-05-15T23:39:32.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:32 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:32 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:39:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:32 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:39:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:32 smithi084 ceph-mon[39784]: osdmap e755: 12 total, 11 up, 10 in 2024-05-15T23:39:32.614 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:39:32.614 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:39:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:32 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:32 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:39:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:32 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:39:32.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:32 smithi090 ceph-mon[44434]: osdmap e755: 12 total, 11 up, 10 in 2024-05-15T23:39:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:32 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:32 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:39:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:32 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:39:32.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:32 smithi088 ceph-mon[44699]: osdmap e755: 12 total, 11 up, 10 in 2024-05-15T23:39:32.850 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:39:33.098 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:39:33.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:33.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:39:33.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:39:33.230 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:33.303 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:39:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:33.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:39:33.407 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:33.438 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:33.459 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:33.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:33.560 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 31 16 71152 71136 143.393 0 - 0.00514313 2024-05-15T23:39:33.563 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:39:33.590 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:33.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:33 smithi084 ceph-mon[39784]: pgmap v2002: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 11 MiB/s wr, 171 op/s; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:33.737 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:39:33.739 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:33 smithi088 ceph-mon[44699]: pgmap v2002: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 11 MiB/s wr, 171 op/s; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:33.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:33.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:39:33.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:33 smithi090 ceph-mon[44434]: pgmap v2002: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 11 MiB/s wr, 171 op/s; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:33.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:39:33.874 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:33.896 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:39:33.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:33.899 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:39:34.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:39:34.028 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:34.042 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:39:34.042 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:39:34.042 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:39:34.042 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [ 2024-05-15T23:39:34.042 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.042 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(osd.2.0:25281 7.8 e599/549 7:11abdc70:::benchmark_data_smithi090_143373_object93718:head v 599'12819, mlcod=599'12789)", 2024-05-15T23:39:34.042 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:31:29.440530+0000", 2024-05-15T23:39:34.042 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 484.598737019, 2024-05-15T23:39:34.043 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.100520923, 2024-05-15T23:39:34.043 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.043 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.043 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.043 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.043 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.043 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.043 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440530+0000", 2024-05-15T23:39:34.043 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.043 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.043 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.043 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.043 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440530+0000", 2024-05-15T23:39:34.043 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.044 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.044 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.044 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440531+0000", 2024-05-15T23:39:34.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6400000000000001e-07 2024-05-15T23:39:34.044 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.044 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.044 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440535+0000", 2024-05-15T23:39:34.044 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5159999999999996e-06 2024-05-15T23:39:34.044 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.044 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.045 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440538+0000", 2024-05-15T23:39:34.045 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3440000000000001e-06 2024-05-15T23:39:34.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.045 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.045 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440541+0000", 2024-05-15T23:39:34.045 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5450000000000002e-06 2024-05-15T23:39:34.045 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.045 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.511113+0000", 2024-05-15T23:39:34.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.070572326000000005 2024-05-15T23:39:34.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.511128+0000", 2024-05-15T23:39:34.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4202999999999999e-05 2024-05-15T23:39:34.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.046 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.046 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.046 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.541038+0000", 2024-05-15T23:39:34.046 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.029910438000000001 2024-05-15T23:39:34.046 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.047 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.047 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.541051+0000", 2024-05-15T23:39:34.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3387000000000001e-05 2024-05-15T23:39:34.047 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.047 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.047 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.047 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.047 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.047 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.16764.0:190384 7.8 e599/549 7:1cd23127:::benchmark_data_smithi090_143373_object44213:head v 599'12820, mlcod=599'12789)", 2024-05-15T23:39:34.047 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:31:29.440544+0000", 2024-05-15T23:39:34.047 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 484.59872333599998, 2024-05-15T23:39:34.047 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.100521839, 2024-05-15T23:39:34.047 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.048 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.048 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.048 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440544+0000", 2024-05-15T23:39:34.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.048 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.048 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.048 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440544+0000", 2024-05-15T23:39:34.048 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.048 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.049 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.049 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440544+0000", 2024-05-15T23:39:34.049 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.74e-07 2024-05-15T23:39:34.049 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.049 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.049 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440547+0000", 2024-05-15T23:39:34.049 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3470000000000001e-06 2024-05-15T23:39:34.049 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.049 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440550+0000", 2024-05-15T23:39:34.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2839999999999998e-06 2024-05-15T23:39:34.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.050 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440552+0000", 2024-05-15T23:39:34.050 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5600000000000001e-06 2024-05-15T23:39:34.050 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.050 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.050 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.050 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.513081+0000", 2024-05-15T23:39:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.072528867999999996 2024-05-15T23:39:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.513095+0000", 2024-05-15T23:39:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3638000000000001e-05 2024-05-15T23:39:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.541053+0000", 2024-05-15T23:39:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.027958199 2024-05-15T23:39:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.541066+0000", 2024-05-15T23:39:34.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2768999999999999e-05 2024-05-15T23:39:34.052 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.052 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.052 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.052 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.052 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.16764.0:190413 7.8 e599/549 7:1fb699e8:::benchmark_data_smithi090_143373_object44242:head v 599'12821, mlcod=599'12789)", 2024-05-15T23:39:34.052 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:31:29.440556+0000", 2024-05-15T23:39:34.052 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 484.59871149600002, 2024-05-15T23:39:34.052 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.100523216, 2024-05-15T23:39:34.052 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.052 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.052 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.052 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.052 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440556+0000", 2024-05-15T23:39:34.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440556+0000", 2024-05-15T23:39:34.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.053 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.053 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440556+0000", 2024-05-15T23:39:34.053 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.67e-07 2024-05-15T23:39:34.053 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.053 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.054 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440559+0000", 2024-05-15T23:39:34.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6410000000000002e-06 2024-05-15T23:39:34.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.054 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.054 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440562+0000", 2024-05-15T23:39:34.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.1370000000000002e-06 2024-05-15T23:39:34.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.054 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.054 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.054 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440564+0000", 2024-05-15T23:39:34.054 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.5950000000000001e-06 2024-05-15T23:39:34.054 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.055 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.055 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.515035+0000", 2024-05-15T23:39:34.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.074470050999999995 2024-05-15T23:39:34.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.055 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.055 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.515045+0000", 2024-05-15T23:39:34.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0835999999999999e-05 2024-05-15T23:39:34.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.055 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.055 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.055 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.541068+0000", 2024-05-15T23:39:34.055 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.026022234000000002 2024-05-15T23:39:34.055 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.056 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.056 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.056 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.541079+0000", 2024-05-15T23:39:34.056 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1555e-05 2024-05-15T23:39:34.056 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.056 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.056 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.056 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.056 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.056 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.16764.0:190443 7.8 e599/549 7:1c1fbdd5:::benchmark_data_smithi090_143373_object44272:head v 599'12822, mlcod=599'12794)", 2024-05-15T23:39:34.056 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:31:29.440567+0000", 2024-05-15T23:39:34.056 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 484.59870014000001, 2024-05-15T23:39:34.056 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.100526606, 2024-05-15T23:39:34.056 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.056 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.057 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.057 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440567+0000", 2024-05-15T23:39:34.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440567+0000", 2024-05-15T23:39:34.057 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.057 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.057 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.057 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.057 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440567+0000", 2024-05-15T23:39:34.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6899999999999999e-07 2024-05-15T23:39:34.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440571+0000", 2024-05-15T23:39:34.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.9230000000000004e-06 2024-05-15T23:39:34.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.058 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440575+0000", 2024-05-15T23:39:34.058 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.145e-06 2024-05-15T23:39:34.058 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.058 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.058 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440577+0000", 2024-05-15T23:39:34.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.8250000000000001e-06 2024-05-15T23:39:34.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.516870+0000", 2024-05-15T23:39:34.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.076292834000000004 2024-05-15T23:39:34.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.059 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.516881+0000", 2024-05-15T23:39:34.059 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0855e-05 2024-05-15T23:39:34.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.059 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.059 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.060 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.541080+0000", 2024-05-15T23:39:34.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.024199353999999999 2024-05-15T23:39:34.060 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.060 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.060 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.060 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.541094+0000", 2024-05-15T23:39:34.060 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.3501e-05 2024-05-15T23:39:34.060 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.060 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.060 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.061 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.061 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.16764.0:190446 7.8 e599/549 7:1d0b779c:::benchmark_data_smithi090_143373_object44275:head v 599'12823, mlcod=599'12794)", 2024-05-15T23:39:34.061 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:31:29.440580+0000", 2024-05-15T23:39:34.061 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 484.59868706899999, 2024-05-15T23:39:34.061 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.100532153, 2024-05-15T23:39:34.061 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.061 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.061 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.061 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.062 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.062 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.062 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440580+0000", 2024-05-15T23:39:34.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.062 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.062 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.062 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.062 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440580+0000", 2024-05-15T23:39:34.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.062 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.062 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.062 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.062 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440581+0000", 2024-05-15T23:39:34.062 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.91e-07 2024-05-15T23:39:34.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440583+0000", 2024-05-15T23:39:34.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.181e-06 2024-05-15T23:39:34.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440586+0000", 2024-05-15T23:39:34.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.117e-06 2024-05-15T23:39:34.063 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.063 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.063 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.063 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440588+0000", 2024-05-15T23:39:34.063 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.4090000000000001e-06 2024-05-15T23:39:34.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.518940+0000", 2024-05-15T23:39:34.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.078351749999999998 2024-05-15T23:39:34.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.518962+0000", 2024-05-15T23:39:34.064 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.1829999999999999e-05 2024-05-15T23:39:34.064 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.064 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.064 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.064 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.541096+0000", 2024-05-15T23:39:34.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.022134542 2024-05-15T23:39:34.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.065 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.065 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.541113+0000", 2024-05-15T23:39:34.065 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.6133e-05 2024-05-15T23:39:34.065 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.065 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.065 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.065 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.065 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.065 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.16764.0:190481 7.8 e599/549 7:131155ad:::benchmark_data_smithi090_143373_object44310:head v 599'12824, mlcod=599'12794)", 2024-05-15T23:39:34.065 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:31:29.440591+0000", 2024-05-15T23:39:34.065 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 484.598676463, 2024-05-15T23:39:34.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.100533972, 2024-05-15T23:39:34.066 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.066 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.066 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.066 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.066 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.066 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440591+0000", 2024-05-15T23:39:34.066 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.066 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.066 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.066 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.066 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440591+0000", 2024-05-15T23:39:34.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440591+0000", 2024-05-15T23:39:34.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9600000000000001e-07 2024-05-15T23:39:34.067 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.067 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.067 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.067 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440596+0000", 2024-05-15T23:39:34.067 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.0259999999999998e-06 2024-05-15T23:39:34.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440599+0000", 2024-05-15T23:39:34.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.14e-06 2024-05-15T23:39:34.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440602+0000", 2024-05-15T23:39:34.068 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2520000000000002e-06 2024-05-15T23:39:34.068 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.068 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.068 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.068 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.521757+0000", 2024-05-15T23:39:34.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.081155295000000002 2024-05-15T23:39:34.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.069 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.521772+0000", 2024-05-15T23:39:34.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5051999999999999e-05 2024-05-15T23:39:34.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.069 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.541114+0000", 2024-05-15T23:39:34.069 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.019341702999999998 2024-05-15T23:39:34.069 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.069 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.069 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.069 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.541125+0000", 2024-05-15T23:39:34.070 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1308000000000001e-05 2024-05-15T23:39:34.070 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.070 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.070 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.070 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.070 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.070 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_repop(client.16764.0:190495 7.8 e599/549 7:16b101d0:::benchmark_data_smithi090_143373_object44324:head v 599'12825, mlcod=599'12794)", 2024-05-15T23:39:34.070 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:31:29.440604+0000", 2024-05-15T23:39:34.070 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 484.59866312299999, 2024-05-15T23:39:34.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.10053464199999999, 2024-05-15T23:39:34.071 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.071 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.071 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.071 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440604+0000", 2024-05-15T23:39:34.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.071 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.071 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.071 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.071 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440604+0000", 2024-05-15T23:39:34.071 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440604+0000", 2024-05-15T23:39:34.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9000000000000001e-07 2024-05-15T23:39:34.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440609+0000", 2024-05-15T23:39:34.072 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2819999999999998e-06 2024-05-15T23:39:34.072 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.072 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.072 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.072 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440612+0000", 2024-05-15T23:39:34.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2629999999999999e-06 2024-05-15T23:39:34.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.440615+0000", 2024-05-15T23:39:34.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.7590000000000002e-06 2024-05-15T23:39:34.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.525582+0000", 2024-05-15T23:39:34.073 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.084967272999999996 2024-05-15T23:39:34.073 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.073 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.073 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.073 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.525601+0000", 2024-05-15T23:39:34.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.9037000000000001e-05 2024-05-15T23:39:34.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.541126+0000", 2024-05-15T23:39:34.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.015524862 2024-05-15T23:39:34.074 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.074 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.074 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.074 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:31:29.541139+0000", 2024-05-15T23:39:34.074 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2976000000000001e-05 2024-05-15T23:39:34.074 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.074 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.074 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.075 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.075 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.075 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:214419 7.6 7:6cfd8502:::benchmark_data_smithi090_143373_object68248:head [delete] snapc 0=[] ondisk+write+known_if_redirected e608)", 2024-05-15T23:39:34.075 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:32:01.593022+0000", 2024-05-15T23:39:34.075 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 452.44624529999999, 2024-05-15T23:39:34.075 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.054925863999999998, 2024-05-15T23:39:34.075 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.075 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.075 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.075 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:34.075 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:39:34.075 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:39:34.075 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 214419 2024-05-15T23:39:34.075 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.075 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.076 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.076 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:01.593022+0000", 2024-05-15T23:39:34.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.076 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.076 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.076 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.076 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:01.593022+0000", 2024-05-15T23:39:34.076 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.076 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:01.593024+0000", 2024-05-15T23:39:34.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0949999999999998e-06 2024-05-15T23:39:34.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:01.593029+0000", 2024-05-15T23:39:34.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3270000000000002e-06 2024-05-15T23:39:34.077 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.077 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.077 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.077 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:01.593039+0000", 2024-05-15T23:39:34.077 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0112000000000001e-05 2024-05-15T23:39:34.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:01.593044+0000", 2024-05-15T23:39:34.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.818e-06 2024-05-15T23:39:34.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.078 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:01.593100+0000", 2024-05-15T23:39:34.078 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.5553000000000002e-05 2024-05-15T23:39:34.078 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.078 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.078 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.079 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:01.593228+0000", 2024-05-15T23:39:34.079 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000127496 2024-05-15T23:39:34.079 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.079 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.079 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:39:34.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:01.593349+0000", 2024-05-15T23:39:34.081 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000121087 2024-05-15T23:39:34.081 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.081 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.081 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:34.081 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:01.596991+0000", 2024-05-15T23:39:34.082 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0036421230000000001 2024-05-15T23:39:34.082 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.082 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.082 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:34.082 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:01.647833+0000", 2024-05-15T23:39:34.082 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.050842007000000002 2024-05-15T23:39:34.082 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.082 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.082 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.082 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:01.647885+0000", 2024-05-15T23:39:34.082 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1946e-05 2024-05-15T23:39:34.082 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.082 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.082 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:01.647948+0000", 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.3299999999999994e-05 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:236538 7.6 7:65fa90a1:::benchmark_data_smithi090_143373_object90367:head [delete] snapc 0=[] ondisk+write+known_if_redirected e612)", 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:32:26.023475+0000", 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 428.015792713, 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046125289999999999, 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:34.083 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:39:34.084 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:39:34.084 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 236538 2024-05-15T23:39:34.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.084 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.023475+0000", 2024-05-15T23:39:34.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.084 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.084 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.084 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.023475+0000", 2024-05-15T23:39:34.084 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.084 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.023479+0000", 2024-05-15T23:39:34.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0369999999999996e-06 2024-05-15T23:39:34.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.023491+0000", 2024-05-15T23:39:34.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2235999999999999e-05 2024-05-15T23:39:34.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.085 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.085 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.085 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.023514+0000", 2024-05-15T23:39:34.085 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.2643e-05 2024-05-15T23:39:34.085 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.023528+0000", 2024-05-15T23:39:34.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.4697e-05 2024-05-15T23:39:34.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.023930+0000", 2024-05-15T23:39:34.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00040136300000000001 2024-05-15T23:39:34.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.086 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.086 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.086 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.024013+0000", 2024-05-15T23:39:34.086 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.3521000000000002e-05 2024-05-15T23:39:34.086 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.087 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:39:34.087 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.024092+0000", 2024-05-15T23:39:34.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.8830000000000002e-05 2024-05-15T23:39:34.087 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.087 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:34.087 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.032821+0000", 2024-05-15T23:39:34.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0087290549999999995 2024-05-15T23:39:34.087 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.087 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.087 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:34.087 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.069523+0000", 2024-05-15T23:39:34.087 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036701839 2024-05-15T23:39:34.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.069557+0000", 2024-05-15T23:39:34.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3590000000000002e-05 2024-05-15T23:39:34.088 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.088 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.088 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.088 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.069600+0000", 2024-05-15T23:39:34.088 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.3479000000000002e-05 2024-05-15T23:39:34.088 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.088 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.088 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.089 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.089 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:236596 7.6 7:66ca9e2f:::benchmark_data_smithi090_143373_object90425:head [delete] snapc 0=[] ondisk+write+known_if_redirected e612)", 2024-05-15T23:39:34.089 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:32:26.093944+0000", 2024-05-15T23:39:34.089 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 427.94532304900002, 2024-05-15T23:39:34.089 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.042003549000000001, 2024-05-15T23:39:34.089 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.089 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.089 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.089 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:34.089 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:39:34.089 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:39:34.089 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 236596 2024-05-15T23:39:34.089 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.090 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.093944+0000", 2024-05-15T23:39:34.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.090 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.093944+0000", 2024-05-15T23:39:34.090 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.090 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.090 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.090 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.093946+0000", 2024-05-15T23:39:34.091 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.1069999999999999e-06 2024-05-15T23:39:34.091 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.091 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.091 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.091 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.093953+0000", 2024-05-15T23:39:34.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5100000000000004e-06 2024-05-15T23:39:34.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.093963+0000", 2024-05-15T23:39:34.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0321e-05 2024-05-15T23:39:34.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.092 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.092 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.092 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.093974+0000", 2024-05-15T23:39:34.092 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0678e-05 2024-05-15T23:39:34.092 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.098743+0000", 2024-05-15T23:39:34.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0047690470000000002 2024-05-15T23:39:34.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.098840+0000", 2024-05-15T23:39:34.093 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.7065999999999995e-05 2024-05-15T23:39:34.093 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.093 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.093 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:39:34.093 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.098929+0000", 2024-05-15T23:39:34.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.9277000000000006e-05 2024-05-15T23:39:34.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:34.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.107680+0000", 2024-05-15T23:39:34.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0087504729999999999 2024-05-15T23:39:34.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.094 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.094 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:34.094 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.135822+0000", 2024-05-15T23:39:34.094 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.028142332999999999 2024-05-15T23:39:34.094 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.095 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.095 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.095 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.135882+0000", 2024-05-15T23:39:34.096 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.9772999999999998e-05 2024-05-15T23:39:34.096 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.097 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.097 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:26.135948+0000", 2024-05-15T23:39:34.097 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.5963999999999994e-05 2024-05-15T23:39:34.097 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.097 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.097 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.097 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.097 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.097 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:243754 7.6 7:6fb94a62:::benchmark_data_smithi090_143373_object97583:head [delete] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-05-15T23:39:34.098 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:32:33.732503+0000", 2024-05-15T23:39:34.098 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 420.30676464599998, 2024-05-15T23:39:34.098 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.047256048000000002, 2024-05-15T23:39:34.098 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.098 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.098 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.098 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:34.098 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:39:34.098 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:39:34.098 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 243754 2024-05-15T23:39:34.098 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.098 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:33.732503+0000", 2024-05-15T23:39:34.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:33.732503+0000", 2024-05-15T23:39:34.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.099 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.099 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.099 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.099 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:33.732504+0000", 2024-05-15T23:39:34.099 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.699e-06 2024-05-15T23:39:34.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:33.732510+0000", 2024-05-15T23:39:34.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3879999999999999e-06 2024-05-15T23:39:34.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:33.732520+0000", 2024-05-15T23:39:34.100 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0348e-05 2024-05-15T23:39:34.100 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.100 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.100 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.100 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:33.732526+0000", 2024-05-15T23:39:34.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.631e-06 2024-05-15T23:39:34.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:33.732566+0000", 2024-05-15T23:39:34.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.0567e-05 2024-05-15T23:39:34.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.101 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:33.732644+0000", 2024-05-15T23:39:34.101 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.7220999999999998e-05 2024-05-15T23:39:34.101 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.101 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.101 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:39:34.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:33.732712+0000", 2024-05-15T23:39:34.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8378000000000002e-05 2024-05-15T23:39:34.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:34.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:33.737487+0000", 2024-05-15T23:39:34.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0047746610000000004 2024-05-15T23:39:34.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.102 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:34.102 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:33.779688+0000", 2024-05-15T23:39:34.102 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.042201277000000002 2024-05-15T23:39:34.102 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.102 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:33.779725+0000", 2024-05-15T23:39:34.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7483e-05 2024-05-15T23:39:34.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.103 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.103 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:33.779759+0000", 2024-05-15T23:39:34.103 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3395000000000001e-05 2024-05-15T23:39:34.103 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.103 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.103 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.103 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.103 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:247497 7.6 7:670881ae:::benchmark_data_smithi090_143373_object101326:head [delete] snapc 0=[] ondisk+write+known_if_redirected e616)", 2024-05-15T23:39:34.103 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:32:37.577057+0000", 2024-05-15T23:39:34.104 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 416.46221026000001, 2024-05-15T23:39:34.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046478852000000001, 2024-05-15T23:39:34.104 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.104 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.104 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.104 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:34.104 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:39:34.104 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:39:34.104 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 247497 2024-05-15T23:39:34.104 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.104 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.104 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.104 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.104 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.577057+0000", 2024-05-15T23:39:34.104 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.577057+0000", 2024-05-15T23:39:34.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.105 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.577059+0000", 2024-05-15T23:39:34.105 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.5409999999999999e-06 2024-05-15T23:39:34.105 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.105 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.105 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.577063+0000", 2024-05-15T23:39:34.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.5739999999999997e-06 2024-05-15T23:39:34.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.577075+0000", 2024-05-15T23:39:34.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1361e-05 2024-05-15T23:39:34.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.106 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.106 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.577083+0000", 2024-05-15T23:39:34.106 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.1189999999999997e-06 2024-05-15T23:39:34.106 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.106 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.577137+0000", 2024-05-15T23:39:34.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.3817999999999999e-05 2024-05-15T23:39:34.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.577261+0000", 2024-05-15T23:39:34.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00012490299999999999 2024-05-15T23:39:34.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.107 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:39:34.107 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.577375+0000", 2024-05-15T23:39:34.107 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00011388200000000001 2024-05-15T23:39:34.107 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.108 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:34.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.584913+0000", 2024-05-15T23:39:34.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0075380660000000004 2024-05-15T23:39:34.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.108 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:34.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.623467+0000", 2024-05-15T23:39:34.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.038553657999999998 2024-05-15T23:39:34.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.108 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.108 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.108 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.623501+0000", 2024-05-15T23:39:34.108 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4381999999999997e-05 2024-05-15T23:39:34.108 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.109 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.109 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.109 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.623536+0000", 2024-05-15T23:39:34.109 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4548000000000002e-05 2024-05-15T23:39:34.109 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.109 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.109 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.109 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.110 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.110 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:247500 7.6 7:6d7b4a75:::benchmark_data_smithi090_143373_object101329:head [delete] snapc 0=[] ondisk+write+known_if_redirected e616)", 2024-05-15T23:39:34.110 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:32:37.578681+0000", 2024-05-15T23:39:34.110 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 416.46058685600002, 2024-05-15T23:39:34.110 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.044966789, 2024-05-15T23:39:34.110 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:39:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:39:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 247500 2024-05-15T23:39:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.578681+0000", 2024-05-15T23:39:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.578681+0000", 2024-05-15T23:39:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.578682+0000", 2024-05-15T23:39:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.327e-06 2024-05-15T23:39:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.578686+0000", 2024-05-15T23:39:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4730000000000002e-06 2024-05-15T23:39:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.578696+0000", 2024-05-15T23:39:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9369999999999999e-06 2024-05-15T23:39:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.578702+0000", 2024-05-15T23:39:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.054e-06 2024-05-15T23:39:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.581942+0000", 2024-05-15T23:39:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0032397179999999999 2024-05-15T23:39:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.582035+0000", 2024-05-15T23:39:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2727999999999997e-05 2024-05-15T23:39:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:39:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.582109+0000", 2024-05-15T23:39:34.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4074999999999998e-05 2024-05-15T23:39:34.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.115 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:34.115 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.591938+0000", 2024-05-15T23:39:34.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0098290659999999992 2024-05-15T23:39:34.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.115 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:34.115 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.623577+0000", 2024-05-15T23:39:34.115 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031639132 2024-05-15T23:39:34.115 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.115 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.116 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.116 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.623609+0000", 2024-05-15T23:39:34.116 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2128000000000001e-05 2024-05-15T23:39:34.116 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.116 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.116 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.116 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.623647+0000", 2024-05-15T23:39:34.116 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8151000000000001e-05 2024-05-15T23:39:34.116 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.116 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.116 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.116 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.116 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.117 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:247501 7.6 7:61ff65ba:::benchmark_data_smithi090_143373_object101330:head [delete] snapc 0=[] ondisk+write+known_if_redirected e616)", 2024-05-15T23:39:34.117 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:32:37.580706+0000", 2024-05-15T23:39:34.117 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 416.45856143499998, 2024-05-15T23:39:34.117 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.043012441999999998, 2024-05-15T23:39:34.117 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.117 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.117 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.117 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:34.117 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:39:34.117 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:39:34.117 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 247501 2024-05-15T23:39:34.117 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.117 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.580706+0000", 2024-05-15T23:39:34.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.118 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.580706+0000", 2024-05-15T23:39:34.118 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.118 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.118 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.118 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.580707+0000", 2024-05-15T23:39:34.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.2720000000000001e-06 2024-05-15T23:39:34.119 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.580712+0000", 2024-05-15T23:39:34.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.4719999999999997e-06 2024-05-15T23:39:34.119 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.119 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.119 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.580721+0000", 2024-05-15T23:39:34.119 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.5689999999999994e-06 2024-05-15T23:39:34.119 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.119 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.580741+0000", 2024-05-15T23:39:34.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0156000000000001e-05 2024-05-15T23:39:34.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.584963+0000", 2024-05-15T23:39:34.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042213700000000003 2024-05-15T23:39:34.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.120 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.120 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.120 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.585035+0000", 2024-05-15T23:39:34.120 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.2436999999999998e-05 2024-05-15T23:39:34.120 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.121 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:39:34.121 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.585104+0000", 2024-05-15T23:39:34.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.8554999999999997e-05 2024-05-15T23:39:34.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.121 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:34.121 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.591944+0000", 2024-05-15T23:39:34.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0068398490000000003 2024-05-15T23:39:34.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.121 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.121 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:34.121 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.623672+0000", 2024-05-15T23:39:34.121 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.031728433 2024-05-15T23:39:34.121 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.122 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.122 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.623698+0000", 2024-05-15T23:39:34.122 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.6207000000000002e-05 2024-05-15T23:39:34.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.122 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.122 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.122 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.623718+0000", 2024-05-15T23:39:34.122 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.0122000000000001e-05 2024-05-15T23:39:34.122 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.122 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.122 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.122 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.123 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.123 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:247507 7.6 7:6d26d07f:::benchmark_data_smithi090_143373_object101336:head [delete] snapc 0=[] ondisk+write+known_if_redirected e616)", 2024-05-15T23:39:34.123 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:32:37.586884+0000", 2024-05-15T23:39:34.123 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 416.45238293199998, 2024-05-15T23:39:34.123 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.038884393000000003, 2024-05-15T23:39:34.123 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.123 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.123 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.123 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:34.123 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:39:34.124 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:39:34.124 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 247507 2024-05-15T23:39:34.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.124 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.586884+0000", 2024-05-15T23:39:34.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.124 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.124 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.124 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.586884+0000", 2024-05-15T23:39:34.124 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.124 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.586886+0000", 2024-05-15T23:39:34.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.238e-06 2024-05-15T23:39:34.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.586890+0000", 2024-05-15T23:39:34.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.2339999999999996e-06 2024-05-15T23:39:34.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.125 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.125 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.125 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.586900+0000", 2024-05-15T23:39:34.125 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.9110000000000005e-06 2024-05-15T23:39:34.125 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.586905+0000", 2024-05-15T23:39:34.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.254e-06 2024-05-15T23:39:34.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.588518+0000", 2024-05-15T23:39:34.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.001613361 2024-05-15T23:39:34.126 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.126 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.126 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.126 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.588623+0000", 2024-05-15T23:39:34.126 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000104348 2024-05-15T23:39:34.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:39:34.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.588739+0000", 2024-05-15T23:39:34.127 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000115931 2024-05-15T23:39:34.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:34.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.592044+0000", 2024-05-15T23:39:34.127 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0033051880000000001 2024-05-15T23:39:34.127 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.127 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.127 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:34.127 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.625716+0000", 2024-05-15T23:39:34.127 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.033672050000000002 2024-05-15T23:39:34.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.128 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.625745+0000", 2024-05-15T23:39:34.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9096000000000001e-05 2024-05-15T23:39:34.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.128 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.128 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:37.625769+0000", 2024-05-15T23:39:34.128 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.3782e-05 2024-05-15T23:39:34.128 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.128 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.128 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.128 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.128 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.129 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:257832 7.6 7:6c13544a:::benchmark_data_smithi090_143373_object111661:head [delete] snapc 0=[] ondisk+write+known_if_redirected e620)", 2024-05-15T23:39:34.129 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:32:48.815435+0000", 2024-05-15T23:39:34.129 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 405.22383239599998, 2024-05-15T23:39:34.129 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.046686667000000001, 2024-05-15T23:39:34.129 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.129 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.129 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.129 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:34.129 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:39:34.129 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:39:34.129 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 257832 2024-05-15T23:39:34.129 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.129 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.129 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.129 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.130 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:48.815435+0000", 2024-05-15T23:39:34.130 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.130 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.130 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.130 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.130 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:48.815435+0000", 2024-05-15T23:39:34.130 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.130 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.130 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.130 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.130 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:48.815437+0000", 2024-05-15T23:39:34.130 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.052e-06 2024-05-15T23:39:34.130 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.130 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:48.815443+0000", 2024-05-15T23:39:34.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6319999999999996e-06 2024-05-15T23:39:34.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:48.815454+0000", 2024-05-15T23:39:34.131 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.1258e-05 2024-05-15T23:39:34.131 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.131 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.131 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.131 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:48.815461+0000", 2024-05-15T23:39:34.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.0570000000000004e-06 2024-05-15T23:39:34.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:48.815513+0000", 2024-05-15T23:39:34.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2061000000000002e-05 2024-05-15T23:39:34.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.132 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.132 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:48.815598+0000", 2024-05-15T23:39:34.132 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.4487000000000006e-05 2024-05-15T23:39:34.132 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.132 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:39:34.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:48.815672+0000", 2024-05-15T23:39:34.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.4437000000000006e-05 2024-05-15T23:39:34.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:34.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:48.819422+0000", 2024-05-15T23:39:34.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0037496080000000002 2024-05-15T23:39:34.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.133 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.133 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:34.133 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:48.862059+0000", 2024-05-15T23:39:34.133 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.042636891000000003 2024-05-15T23:39:34.133 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:48.862092+0000", 2024-05-15T23:39:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.3079e-05 2024-05-15T23:39:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:48.862122+0000", 2024-05-15T23:39:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0105000000000001e-05 2024-05-15T23:39:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.134 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.135 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:261626 7.6 7:6f518239:::benchmark_data_smithi090_143373_object115455:head [delete] snapc 0=[] ondisk+write+known_if_redirected e626)", 2024-05-15T23:39:34.135 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:32:55.069876+0000", 2024-05-15T23:39:34.135 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 398.96939191199999, 2024-05-15T23:39:34.135 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.039997051999999998, 2024-05-15T23:39:34.135 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.135 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.135 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.135 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:34.135 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:39:34.135 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:39:34.135 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 261626 2024-05-15T23:39:34.135 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.135 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:55.069876+0000", 2024-05-15T23:39:34.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.136 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:55.069876+0000", 2024-05-15T23:39:34.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.136 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.136 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.136 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.136 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:55.069877+0000", 2024-05-15T23:39:34.136 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.8780000000000001e-06 2024-05-15T23:39:34.136 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.137 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.137 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:55.069882+0000", 2024-05-15T23:39:34.137 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.1520000000000002e-06 2024-05-15T23:39:34.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.137 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.137 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:55.069891+0000", 2024-05-15T23:39:34.137 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.4590000000000004e-06 2024-05-15T23:39:34.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.137 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.137 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.137 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:55.069897+0000", 2024-05-15T23:39:34.137 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.8499999999999999e-06 2024-05-15T23:39:34.137 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.138 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:55.069934+0000", 2024-05-15T23:39:34.138 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.7076000000000002e-05 2024-05-15T23:39:34.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.138 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:55.070020+0000", 2024-05-15T23:39:34.138 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.6470000000000004e-05 2024-05-15T23:39:34.138 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.138 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.138 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:39:34.138 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:55.070103+0000", 2024-05-15T23:39:34.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2253e-05 2024-05-15T23:39:34.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:34.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:55.073713+0000", 2024-05-15T23:39:34.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003609943 2024-05-15T23:39:34.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:34.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:55.109802+0000", 2024-05-15T23:39:34.139 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036089743000000001 2024-05-15T23:39:34.139 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.139 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.139 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.139 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:55.109837+0000", 2024-05-15T23:39:34.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.4785e-05 2024-05-15T23:39:34.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.140 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.140 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:32:55.109873+0000", 2024-05-15T23:39:34.140 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.5442999999999999e-05 2024-05-15T23:39:34.140 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.140 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.140 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.140 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.140 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.140 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.16764.0:281088 7.6 7:6cd5438b:::benchmark_data_smithi090_143373_object134917:head [delete] snapc 0=[] ondisk+write+known_if_redirected e635)", 2024-05-15T23:39:34.140 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:33:18.057930+0000", 2024-05-15T23:39:34.140 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 375.981337435, 2024-05-15T23:39:34.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.041984581999999999, 2024-05-15T23:39:34.141 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.141 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.141 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.141 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:34.141 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.16764", 2024-05-15T23:39:34.141 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:39:34.141 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 281088 2024-05-15T23:39:34.141 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.141 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.141 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.141 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.141 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:33:18.057930+0000", 2024-05-15T23:39:34.141 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:33:18.057930+0000", 2024-05-15T23:39:34.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:33:18.057932+0000", 2024-05-15T23:39:34.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.689e-06 2024-05-15T23:39:34.142 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.142 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.142 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.142 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:33:18.057936+0000", 2024-05-15T23:39:34.142 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 4.758e-06 2024-05-15T23:39:34.143 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.143 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.143 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.143 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:33:18.057947+0000", 2024-05-15T23:39:34.143 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.0169999999999999e-05 2024-05-15T23:39:34.143 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.143 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.143 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.143 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:33:18.057952+0000", 2024-05-15T23:39:34.143 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.6289999999999998e-06 2024-05-15T23:39:34.143 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.143 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.143 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.143 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:33:18.058027+0000", 2024-05-15T23:39:34.143 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 7.5048000000000004e-05 2024-05-15T23:39:34.144 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.144 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.144 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.144 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:33:18.058153+0000", 2024-05-15T23:39:34.144 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000125606 2024-05-15T23:39:34.144 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.144 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.144 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 3", 2024-05-15T23:39:34.144 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:33:18.058285+0000", 2024-05-15T23:39:34.144 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00013194699999999999 2024-05-15T23:39:34.144 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.144 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.144 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:34.144 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:33:18.061765+0000", 2024-05-15T23:39:34.144 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.003480637 2024-05-15T23:39:34.145 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.145 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.145 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:34.145 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:33:18.099807+0000", 2024-05-15T23:39:34.145 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.038041548000000001 2024-05-15T23:39:34.145 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.145 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.145 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.145 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:33:18.099862+0000", 2024-05-15T23:39:34.145 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.4750999999999998e-05 2024-05-15T23:39:34.145 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.145 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.145 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.145 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:33:18.099915+0000", 2024-05-15T23:39:34.145 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.2799000000000001e-05 2024-05-15T23:39:34.146 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.146 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.146 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.146 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.146 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.146 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.27292.0:58479 9.d 9:b6ac325e:::benchmark_data_smithi090_166897_object58478:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e748)", 2024-05-15T23:39:34.146 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:39:19.483118+0000", 2024-05-15T23:39:34.146 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 14.5561493, 2024-05-15T23:39:34.146 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.040881892000000003, 2024-05-15T23:39:34.146 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.146 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.146 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.146 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:34.146 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.27292", 2024-05-15T23:39:34.147 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2136483304", 2024-05-15T23:39:34.147 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 58479 2024-05-15T23:39:34.147 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.147 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.147 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.147 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.147 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.483118+0000", 2024-05-15T23:39:34.147 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.147 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.147 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.147 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.147 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.483118+0000", 2024-05-15T23:39:34.147 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.147 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.147 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.483119+0000", 2024-05-15T23:39:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.009e-06 2024-05-15T23:39:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.483263+0000", 2024-05-15T23:39:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000143814 2024-05-15T23:39:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.483270+0000", 2024-05-15T23:39:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.6039999999999998e-06 2024-05-15T23:39:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.483275+0000", 2024-05-15T23:39:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 5.1549999999999998e-06 2024-05-15T23:39:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.483313+0000", 2024-05-15T23:39:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8099999999999998e-05 2024-05-15T23:39:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.483396+0000", 2024-05-15T23:39:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2837000000000007e-05 2024-05-15T23:39:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 1", 2024-05-15T23:39:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.483476+0000", 2024-05-15T23:39:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.0128000000000003e-05 2024-05-15T23:39:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.519669+0000", 2024-05-15T23:39:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036192997999999997 2024-05-15T23:39:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.523929+0000", 2024-05-15T23:39:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0042602889999999996 2024-05-15T23:39:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.523961+0000", 2024-05-15T23:39:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.2181000000000002e-05 2024-05-15T23:39:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:19.524000+0000", 2024-05-15T23:39:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.8776999999999998e-05 2024-05-15T23:39:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "description": "osd_op(client.27292.0:65063 9.e 9:774d667b:::benchmark_data_smithi090_166897_object65062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e748)", 2024-05-15T23:39:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "initiated_at": "2024-05-15T23:39:22.482723+0000", 2024-05-15T23:39:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "age": 11.556544889, 2024-05-15T23:39:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.038957044000000003, 2024-05-15T23:39:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "continuous": false, 2024-05-15T23:39:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "type_data": { 2024-05-15T23:39:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:39:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "client_info": { 2024-05-15T23:39:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "client": "client.27292", 2024-05-15T23:39:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "client_addr": "v1:172.21.15.90:0/2136483304", 2024-05-15T23:39:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "tid": 65063 2024-05-15T23:39:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "events": [ 2024-05-15T23:39:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "initiated", 2024-05-15T23:39:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:22.482723+0000", 2024-05-15T23:39:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "header_read", 2024-05-15T23:39:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:22.482723+0000", 2024-05-15T23:39:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0 2024-05-15T23:39:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "throttled", 2024-05-15T23:39:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:22.482725+0000", 2024-05-15T23:39:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.323e-06 2024-05-15T23:39:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.154 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "all_read", 2024-05-15T23:39:34.154 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:22.482818+0000", 2024-05-15T23:39:34.154 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 9.2804000000000001e-05 2024-05-15T23:39:34.154 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.154 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.154 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "dispatched", 2024-05-15T23:39:34.154 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:22.482836+0000", 2024-05-15T23:39:34.154 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 1.7915999999999999e-05 2024-05-15T23:39:34.154 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.154 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.154 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "queued_for_pg", 2024-05-15T23:39:34.154 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:22.482844+0000", 2024-05-15T23:39:34.154 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 8.2879999999999993e-06 2024-05-15T23:39:34.154 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.154 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "reached_pg", 2024-05-15T23:39:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:22.482908+0000", 2024-05-15T23:39:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 6.4287000000000003e-05 2024-05-15T23:39:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "started", 2024-05-15T23:39:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:22.483056+0000", 2024-05-15T23:39:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.000147687 2024-05-15T23:39:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "waiting for subops from 10", 2024-05-15T23:39:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:22.483200+0000", 2024-05-15T23:39:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.00014408699999999999 2024-05-15T23:39:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.156 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "op_commit", 2024-05-15T23:39:34.156 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:22.485009+0000", 2024-05-15T23:39:34.156 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.0018087470000000001 2024-05-15T23:39:34.156 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.156 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.156 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:39:34.156 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:22.521620+0000", 2024-05-15T23:39:34.156 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 0.036610845000000003 2024-05-15T23:39:34.156 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.156 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.156 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "commit_sent", 2024-05-15T23:39:34.156 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:22.521650+0000", 2024-05-15T23:39:34.156 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 3.0632000000000001e-05 2024-05-15T23:39:34.156 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-15T23:39:34.156 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-05-15T23:39:34.157 INFO:teuthology.orchestra.run.smithi088.stdout: "event": "done", 2024-05-15T23:39:34.157 INFO:teuthology.orchestra.run.smithi088.stdout: "time": "2024-05-15T23:39:22.521680+0000", 2024-05-15T23:39:34.157 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 2.9428e-05 2024-05-15T23:39:34.157 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.157 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.157 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.157 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-15T23:39:34.157 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-05-15T23:39:34.157 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:39:34.226 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:39:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:34.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:39:34.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:39:34.363 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:34.385 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:39:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:34.388 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:39:34.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:39:34.518 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:34.537 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:39:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:34.540 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:39:34.560 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 32 16 71152 71136 138.912 0 - 0.00514313 2024-05-15T23:39:34.615 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:39:34.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:34 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:39:34.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:34 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:39:34.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:39:34.685 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:34 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:39:34.685 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:34 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:39:34.686 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:34.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T23:39:34.734 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:34 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:39:34.735 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:34 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:39:34.785 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:39:34.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:34.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:39:34.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:39:34.917 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:34.937 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:39:34.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:34.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:39:34.970 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:39:35.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:39:35.070 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:35.262 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5 is already down. 2024-05-15T23:39:35.267 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:39:35.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:35.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:39:35.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:39:35.400 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:35.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:35 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:39:35.256+0000 7fa0938b7700 -1 mon.a@0(leader).osd e756 definitely_dead 0 2024-05-15T23:39:35.503 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:39:35.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:35.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:39:35.560 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 33 16 71152 71136 134.702 0 - 0.00514313 2024-05-15T23:39:35.608 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:39:35.638 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:35.658 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:39:35.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:35.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:39:35.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:39:35.789 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:35 smithi090 ceph-mon[44434]: osdmap e756: 12 total, 11 up, 10 in 2024-05-15T23:39:35.789 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:35 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1875401399' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["5"]}]: dispatch 2024-05-15T23:39:35.789 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:35 smithi090 ceph-mon[44434]: pgmap v2004: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 2.4 MiB/s wr, 38 op/s; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:35.792 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:35.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:35 smithi088 ceph-mon[44699]: osdmap e756: 12 total, 11 up, 10 in 2024-05-15T23:39:35.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:35 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1875401399' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["5"]}]: dispatch 2024-05-15T23:39:35.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:35 smithi088 ceph-mon[44699]: pgmap v2004: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 2.4 MiB/s wr, 38 op/s; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:35.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:35 smithi084 ceph-mon[39784]: osdmap e756: 12 total, 11 up, 10 in 2024-05-15T23:39:35.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:35 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1875401399' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["5"]}]: dispatch 2024-05-15T23:39:35.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:35 smithi084 ceph-mon[39784]: pgmap v2004: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 2.4 MiB/s wr, 38 op/s; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:39:35.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:35.894 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:39:35.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:39:36.021 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:36.276 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:39:36.305 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [1, 3, 4, 5, 7, 11, 9, 10, 2, 8] 2024-05-15T23:39:36.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd out 5 2024-05-15T23:39:36.462 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:39:36.469 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:39:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:36.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:39:36.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/705083361' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:39:36.560 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 34 16 71152 71136 130.741 0 - 0.00514313 2024-05-15T23:39:36.574 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:39:36.609 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:36.630 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:39:36.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:36.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:39:36.730 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/705083361' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:39:36.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:39:36.764 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/705083361' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:39:36.862 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:39:36.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:36.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:39:36.967 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:39:37.001 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:37.022 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:39:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:37.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:39:37.107 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:39:37.108 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:39:37.108 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-05-15T23:39:37.108 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:39:37.126 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:37.156 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:37.176 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:37.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:37.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:37.281 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:39:37.312 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:37.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:37 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:39:37.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:37 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/705083361' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:39:37.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:37 smithi084 ceph-mon[39784]: osdmap e757: 12 total, 11 up, 10 in 2024-05-15T23:39:37.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:37 smithi084 ceph-mon[39784]: pgmap v2006: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:37.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:37 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 17500/142284 objects degraded (12.299%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:39:37.560 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 35 16 71152 71136 127.006 0 - 0.00514313 2024-05-15T23:39:37.751 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:37 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:39:37.751 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:37 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/705083361' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:39:37.751 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:37 smithi090 ceph-mon[44434]: osdmap e757: 12 total, 11 up, 10 in 2024-05-15T23:39:37.751 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:37 smithi090 ceph-mon[44434]: pgmap v2006: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:37.751 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:37 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 17500/142284 objects degraded (12.299%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:39:37.763 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:39:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:37.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:39:37.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T23:39:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:37 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:39:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:37 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/705083361' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:39:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:37 smithi088 ceph-mon[44699]: osdmap e757: 12 total, 11 up, 10 in 2024-05-15T23:39:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:37 smithi088 ceph-mon[44699]: pgmap v2006: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:37 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 17500/142284 objects degraded (12.299%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:39:37.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:39:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:37.914 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:39:37.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:37.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:39:38.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:39:38.046 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:38.066 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:39:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:38.070 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:39:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:39:38.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:39:38.198 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:38.345 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:39:38.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:38.349 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:39:38.450 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:39:38.477 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:38.498 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:39:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:38.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:39:38.560 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 36 16 71152 71136 123.478 0 - 0.00514313 2024-05-15T23:39:38.601 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:39:38.634 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:38.643 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:38 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2275280618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:38.643 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:38.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:39:38.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:39:38.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:39:38.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:39:38.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:39:38.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:39:38.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:39:38.657 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:39:38.657 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:39:38.657 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:39:38.657 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:39:38.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:39:38.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:39:38.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:39:38.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:39:38.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:39:38.657 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-05-15T23:39:38.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:39:38.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:39:38.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:39:38.658 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:39:38.658 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:39:38.658 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:39:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:39:38.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:38.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:39:38.760 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:39:38.786 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:38 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2275280618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:38.786 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:38.788 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:38.808 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:39:38.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:38.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:39:38.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:38 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2275280618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:38.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:38.912 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:39:38.941 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:39.390 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:39:39.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:39.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:39:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:39:39.401 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:39:39.441 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:39:39.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:39:39.522 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:39.560 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 37 16 71152 71136 120.141 0 - 0.00514313 2024-05-15T23:39:39.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.641 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.642 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.642 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.642 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:39 smithi090 ceph-mon[44434]: pgmap v2007: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:39.646 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:39:39.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:39.649 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:39:39.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.661 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.661 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.661 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.661 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.661 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.661 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.661 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.661 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.661 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:39 smithi084 ceph-mon[39784]: pgmap v2007: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:39.750 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:39:39.781 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.781 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.781 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.781 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.781 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.781 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.782 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.782 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.782 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:39.782 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:39 smithi088 ceph-mon[44699]: pgmap v2007: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:39.783 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:39.805 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:39:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:39.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:39:39.910 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:39:39.940 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:39.964 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:39:39.964 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:39:40.139 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:39:40.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:40.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:39:40.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:39:40.274 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:40.347 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:39:40.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:40.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:39:40.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:39:40.488 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:40.560 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 38 16 71152 71136 116.979 0 - 0.00514313 2024-05-15T23:39:40.561 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:39:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:40.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:39:40.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:39:40.667 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:40.679 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T23:39:40.699 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:40.721 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:40.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:40.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:39:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:40.871 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:39:40.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:40.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:39:40.943 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:39:40.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:39:41.001 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:41.204 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:39:41.250 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:39:41.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:41.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:39:41.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:39:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:41.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:39:41.458 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:39:41.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:41.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:39:41.551 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:41 smithi090 ceph-mon[44434]: pgmap v2008: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:41.560 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 39 16 71152 71136 113.98 0 - 0.00514313 2024-05-15T23:39:41.572 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:39:41.600 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:41.713 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:41 smithi084 ceph-mon[39784]: pgmap v2008: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:41.714 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:41 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:39:41.413+0000 7fa0938b7700 -1 mon.a@0(leader).osd e757 definitely_dead 0 2024-05-15T23:39:41.724 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:39:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:41.727 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:39:41.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:41 smithi088 ceph-mon[44699]: pgmap v2008: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:41.828 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:41.860 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:41.882 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:41.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:41.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:41.986 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:39:42.013 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:39:42.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:42.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:39:42.365 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:39:42.399 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:42.404 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.5. 2024-05-15T23:39:42.419 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:39:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:42.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:39:42.427 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:42 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2766782683' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-05-15T23:39:42.427 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:42 smithi084 ceph-mon[39784]: Client client.admin marked osd.5 out, after it was down for 16 seconds 2024-05-15T23:39:42.509 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:42 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2766782683' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-05-15T23:39:42.510 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:42 smithi090 ceph-mon[44434]: Client client.admin marked osd.5 out, after it was down for 16 seconds 2024-05-15T23:39:42.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:39:42.552 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:42.561 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:39:42.561078 min lat: 0.00103205 max lat: 0.213385 avg lat: 0.00514313 2024-05-15T23:39:42.561 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:39:42.561 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 40 16 71152 71136 111.131 0 - 0.00514313 2024-05-15T23:39:42.650 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:39:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:42.653 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:39:42.754 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:39:42.781 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:39:42.781 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:39:42.782 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:42 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2766782683' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-05-15T23:39:42.782 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:42 smithi088 ceph-mon[44699]: Client client.admin marked osd.5 out, after it was down for 16 seconds 2024-05-15T23:39:42.784 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:42.805 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:39:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:42.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:39:42.909 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:39:42.938 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:43.238 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-05-15T23:39:43.238 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi088.front.sepia.ceph.com 2024-05-15T23:39:43.238 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -n osd.5 --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-05-15T23:39:43.447 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:43 smithi090 ceph-mon[44434]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-15T23:39:43.447 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:43 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2766782683' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-05-15T23:39:43.447 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:43 smithi090 ceph-mon[44434]: osdmap e758: 12 total, 11 up, 9 in 2024-05-15T23:39:43.447 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:43 smithi090 ceph-mon[44434]: pgmap v2010: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:43.469 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/osd.5/config 2024-05-15T23:39:43.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T23:39:43.487 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:39:43.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:43 smithi084 ceph-mon[39784]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-15T23:39:43.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:43 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2766782683' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-05-15T23:39:43.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:43 smithi084 ceph-mon[39784]: osdmap e758: 12 total, 11 up, 9 in 2024-05-15T23:39:43.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:43 smithi084 ceph-mon[39784]: pgmap v2010: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:43.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:39:43.561 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 41 16 71169 71153 108.446 0.0590278 0.00618943 0.00590643 2024-05-15T23:39:43.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:43 smithi088 ceph-mon[44699]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-15T23:39:43.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:43 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2766782683' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-05-15T23:39:43.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:43 smithi088 ceph-mon[44699]: osdmap e758: 12 total, 11 up, 9 in 2024-05-15T23:39:43.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:43 smithi088 ceph-mon[44699]: pgmap v2010: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.3 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 17500/142284 objects degraded (12.299%) 2024-05-15T23:39:43.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:39:43.618 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:43.716 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:39:43.716 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:39:43.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:43.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:39:43.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:39:43.827 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:39:43.852 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:44.108 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:39:44.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:44.111 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:39:44.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:39:44.242 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:44.267 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:39:44.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:44.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:39:44.373 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:39:44.402 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:44.427 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:39:44.561 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 42 16 71330 71314 106.104 10.0625 19.1476 0.00875354 2024-05-15T23:39:44.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:44 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:39:44.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:44 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:39:44.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:44 smithi084 ceph-mon[39784]: osdmap e759: 12 total, 11 up, 9 in 2024-05-15T23:39:44.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1687811596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:39:44.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:44 smithi084 ceph-mon[39784]: from='client.37448 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:39:44.768 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:44 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:39:44.769 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:44 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:39:44.769 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:44 smithi088 ceph-mon[44699]: osdmap e759: 12 total, 11 up, 9 in 2024-05-15T23:39:44.769 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1687811596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:39:44.769 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:44 smithi088 ceph-mon[44699]: from='client.37448 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:39:44.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:44 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:39:44.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:44 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:39:44.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:44 smithi090 ceph-mon[44434]: osdmap e759: 12 total, 11 up, 9 in 2024-05-15T23:39:44.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1687811596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:39:44.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:44 smithi090 ceph-mon[44434]: from='client.37448 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:39:44.828 INFO:teuthology.orchestra.run.smithi088.stderr:Creating an OSD daemon form without an OSD FSID value 2024-05-15T23:39:45.410 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:45 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:39:45.410 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:45 smithi088 ceph-mon[44699]: from='client.37448 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:39:45.410 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:45 smithi088 ceph-mon[44699]: osdmap e760: 12 total, 11 up, 9 in 2024-05-15T23:39:45.410 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:45 smithi088 ceph-mon[44699]: pgmap v2013: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 2.1 MiB/s wr, 33 op/s; 17500/142692 objects degraded (12.264%) 2024-05-15T23:39:45.561 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 43 16 72637 72621 105.536 81.6875 0.0111657 0.00947283 2024-05-15T23:39:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:45 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:39:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:45 smithi090 ceph-mon[44434]: from='client.37448 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:39:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:45 smithi090 ceph-mon[44434]: osdmap e760: 12 total, 11 up, 9 in 2024-05-15T23:39:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:45 smithi090 ceph-mon[44434]: pgmap v2013: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 2.1 MiB/s wr, 33 op/s; 17500/142692 objects degraded (12.264%) 2024-05-15T23:39:45.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:45 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:39:45.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:45 smithi084 ceph-mon[39784]: from='client.37448 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:39:45.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:45 smithi084 ceph-mon[39784]: osdmap e760: 12 total, 11 up, 9 in 2024-05-15T23:39:45.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:45 smithi084 ceph-mon[39784]: pgmap v2013: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 2.1 MiB/s wr, 33 op/s; 17500/142692 objects degraded (12.264%) 2024-05-15T23:39:46.561 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 44 16 73866 73850 104.883 76.8125 0.00727041 0.00953305 2024-05-15T23:39:46.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:46 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 17500/142692 objects degraded (12.264%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:39:46.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:46 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 17500/142692 objects degraded (12.264%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:39:46.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:46 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 17500/142692 objects degraded (12.264%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:39:47.561 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 45 16 75058 75042 104.208 74.5 0.00293299 0.00958883 2024-05-15T23:39:48.561 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 46 16 76242 76226 103.551 74 0.0343339 0.00965419 2024-05-15T23:39:48.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:48 smithi090 ceph-mon[44434]: pgmap v2014: 17 pgs: 1 active+recovering+degraded+remapped, 2 active+recovering+degraded, 1 undersized+degraded+peered, 13 active+clean; 4.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 20 MiB/s wr, 323 op/s; 17049/146166 objects degraded (11.664%); 4.7 MiB/s, 75 objects/s recovering 2024-05-15T23:39:48.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:48 smithi090 ceph-mon[44434]: osdmap e761: 12 total, 11 up, 9 in 2024-05-15T23:39:48.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:48 smithi088 ceph-mon[44699]: pgmap v2014: 17 pgs: 1 active+recovering+degraded+remapped, 2 active+recovering+degraded, 1 undersized+degraded+peered, 13 active+clean; 4.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 20 MiB/s wr, 323 op/s; 17049/146166 objects degraded (11.664%); 4.7 MiB/s, 75 objects/s recovering 2024-05-15T23:39:48.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:48 smithi088 ceph-mon[44699]: osdmap e761: 12 total, 11 up, 9 in 2024-05-15T23:39:48.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:48 smithi084 ceph-mon[39784]: pgmap v2014: 17 pgs: 1 active+recovering+degraded+remapped, 2 active+recovering+degraded, 1 undersized+degraded+peered, 13 active+clean; 4.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 20 MiB/s wr, 323 op/s; 17049/146166 objects degraded (11.664%); 4.7 MiB/s, 75 objects/s recovering 2024-05-15T23:39:48.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:48 smithi084 ceph-mon[39784]: osdmap e761: 12 total, 11 up, 9 in 2024-05-15T23:39:49.376 INFO:teuthology.orchestra.run.smithi088.stdout:9.f 2024-05-15T23:39:49.376 INFO:teuthology.orchestra.run.smithi088.stdout:9.a 2024-05-15T23:39:49.376 INFO:teuthology.orchestra.run.smithi088.stdout:9.5 2024-05-15T23:39:49.376 INFO:teuthology.orchestra.run.smithi088.stdout:9.6 2024-05-15T23:39:49.563 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 47 16 77510 77494 103.033 79.25 0.015864 0.00970322 2024-05-15T23:39:49.769 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:39:49.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:49.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:39:49.876 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:39:49.913 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:49.966 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:39:49.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:49.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:39:50.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:39:50.124 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:50.155 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:39:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:50.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:39:50.260 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:39:50.300 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:50.333 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:39:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:50.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:39:50.383 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:50 smithi088 ceph-mon[44699]: pgmap v2016: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 4.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 30 MiB/s wr, 483 op/s; 16808/148090 objects degraded (11.350%); 7.2 MiB/s, 115 objects/s recovering 2024-05-15T23:39:50.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:39:50.476 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -n osd.5 --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 9.6 --file /var/log/ceph/exp.9.6.5 2024-05-15T23:39:50.487 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:50.533 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:39:50.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:50.535 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:39:50.562 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 48 16 78613 78597 102.322 68.9375 0.0174892 0.00977162 2024-05-15T23:39:50.589 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:39:50.590 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:39:50.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:50 smithi084 ceph-mon[39784]: pgmap v2016: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 4.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 30 MiB/s wr, 483 op/s; 16808/148090 objects degraded (11.350%); 7.2 MiB/s, 115 objects/s recovering 2024-05-15T23:39:50.636 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:39:50.666 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:50.693 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:39:50.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:50.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:39:50.726 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/osd.5/config 2024-05-15T23:39:50.797 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:39:50.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:50 smithi090 ceph-mon[44434]: pgmap v2016: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 4.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 30 MiB/s wr, 483 op/s; 16808/148090 objects degraded (11.350%); 7.2 MiB/s, 115 objects/s recovering 2024-05-15T23:39:50.840 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:50.868 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:39:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:50.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:39:50.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:39:51.025 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:51.395 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:39:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:51.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:39:51.499 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:39:51.552 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:51.565 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 49 15 79765 79750 101.699 72.0625 0.0175911 0.00983039 2024-05-15T23:39:51.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:51 smithi088 ceph-mon[44699]: pgmap v2017: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 4.6 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 35 MiB/s wr, 560 op/s; 16808/151166 objects degraded (11.119%); 5.5 MiB/s, 87 objects/s recovering 2024-05-15T23:39:51.625 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:39:51.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:51.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:39:51.742 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:51.774 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:51.785 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:39:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:51 smithi090 ceph-mon[44434]: pgmap v2017: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 4.6 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 35 MiB/s wr, 560 op/s; 16808/151166 objects degraded (11.119%); 5.5 MiB/s, 87 objects/s recovering 2024-05-15T23:39:51.821 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:51 smithi084 ceph-mon[39784]: pgmap v2017: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 4.6 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 35 MiB/s wr, 560 op/s; 16808/151166 objects degraded (11.119%); 5.5 MiB/s, 87 objects/s recovering 2024-05-15T23:39:51.974 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:51.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:51.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:52.011 INFO:teuthology.orchestra.run.smithi088.stderr:Creating an OSD daemon form without an OSD FSID value 2024-05-15T23:39:52.077 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:39:52.107 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:52.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T23:39:52.245 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:39:52.307 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:39:52.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:52.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:39:52.411 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:39:52.412 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:52 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 16808/151166 objects degraded (11.119%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:39:52.457 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:39:52.568 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 50 16 80879 80863 101.05 69.5625 0.00992541 0.00989344 2024-05-15T23:39:52.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:52 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 16808/151166 objects degraded (11.119%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:39:52.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:52 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 16808/151166 objects degraded (11.119%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:39:53.155 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:39:53.186 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:39:53.251 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:39:53.255 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:39:53.289 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:39:53.322 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:39:53.359 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:39:53.391 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:39:53.423 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:53 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2342875625' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:53.424 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:53 smithi090 ceph-mon[44434]: pgmap v2018: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 4.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 59 MiB/s wr, 946 op/s; 15922/159182 objects degraded (10.002%); 11 MiB/s, 176 objects/s recovering 2024-05-15T23:39:53.424 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:53.428 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:39:53.465 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:39:53.508 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:39:53.509 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:39:53.509 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:39:53.509 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:39:53.509 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:39:53.509 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:39:53.509 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-05-15T23:39:53.509 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:39:53.509 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:39:53.509 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:39:53.509 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:39:53.510 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:39:53.510 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:39:53.568 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 51 16 82098 82082 100.563 76.1875 0.0115689 0.00994274 2024-05-15T23:39:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:53 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2342875625' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:53 smithi088 ceph-mon[44699]: pgmap v2018: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 4.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 59 MiB/s wr, 946 op/s; 15922/159182 objects degraded (10.002%); 11 MiB/s, 176 objects/s recovering 2024-05-15T23:39:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:53.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:53 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2342875625' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:39:53.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:53 smithi084 ceph-mon[39784]: pgmap v2018: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 4.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 59 MiB/s wr, 946 op/s; 15922/159182 objects degraded (10.002%); 11 MiB/s, 176 objects/s recovering 2024-05-15T23:39:53.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.568 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 52 15 83295 83280 100.069 74.875 0.00630788 0.00999101 2024-05-15T23:39:54.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:54 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.823 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:54 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:54 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:39:55.569 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 53 16 84365 84349 99.4412 66.8125 0.0247796 0.0100543 2024-05-15T23:39:55.578 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:39:55.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:55.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:39:55.585 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:55 smithi084 ceph-mon[39784]: pgmap v2019: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 63 MiB/s wr, 1.01k op/s; 15922/162842 objects degraded (9.778%); 9.9 MiB/s, 157 objects/s recovering 2024-05-15T23:39:55.684 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:39:55.723 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:55 smithi088 ceph-mon[44699]: pgmap v2019: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 63 MiB/s wr, 1.01k op/s; 15922/162842 objects degraded (9.778%); 9.9 MiB/s, 157 objects/s recovering 2024-05-15T23:39:55.725 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:55.749 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:39:55.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:55.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:39:55.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:55 smithi090 ceph-mon[44434]: pgmap v2019: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 63 MiB/s wr, 1.01k op/s; 15922/162842 objects degraded (9.778%); 9.9 MiB/s, 157 objects/s recovering 2024-05-15T23:39:55.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:39:55.913 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:55.945 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:39:55.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:39:55.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:39:56.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:39:56.110 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:39:56.390 INFO:teuthology.orchestra.run.smithi088.stderr:Exporting 9.6 info 9.6( v 749'4391 (0'0,749'4391] local-lis/les=737/738 n=4391 ec=737/737 lis/c=737/737 les/c/f=738/738/0 sis=737) 2024-05-15T23:39:56.390 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:600043ad:::benchmark_data_smithi090_166897_object41622:head# 2024-05-15T23:39:56.390 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6002fc21:::benchmark_data_smithi090_166897_object14756:head# 2024-05-15T23:39:56.390 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60038712:::benchmark_data_smithi090_166897_object2532:head# 2024-05-15T23:39:56.390 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6003a46f:::benchmark_data_smithi090_166897_object23915:head# 2024-05-15T23:39:56.390 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60043b15:::benchmark_data_smithi090_166897_object20640:head# 2024-05-15T23:39:56.390 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:600614fe:::benchmark_data_smithi090_166897_object65594:head# 2024-05-15T23:39:56.390 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:600692de:::benchmark_data_smithi090_166897_object25263:head# 2024-05-15T23:39:56.391 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6006f4c8:::benchmark_data_smithi090_166897_object62680:head# 2024-05-15T23:39:56.391 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60080f34:::benchmark_data_smithi090_166897_object10663:head# 2024-05-15T23:39:56.391 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6008d99a:::benchmark_data_smithi090_166897_object35902:head# 2024-05-15T23:39:56.391 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6008f364:::benchmark_data_smithi090_166897_object64021:head# 2024-05-15T23:39:56.391 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60094baf:::benchmark_data_smithi090_166897_object21769:head# 2024-05-15T23:39:56.391 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6009dfe8:::benchmark_data_smithi090_166897_object17006:head# 2024-05-15T23:39:56.391 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6009e268:::benchmark_data_smithi090_166897_object60612:head# 2024-05-15T23:39:56.391 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:600adbfc:::benchmark_data_smithi090_166897_object45954:head# 2024-05-15T23:39:56.391 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:600b1509:::benchmark_data_smithi090_166897_object19152:head# 2024-05-15T23:39:56.391 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:600b8fe2:::benchmark_data_smithi090_166897_object53053:head# 2024-05-15T23:39:56.391 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:600c210f:::benchmark_data_smithi090_166897_object11646:head# 2024-05-15T23:39:56.391 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:600c5f3f:::benchmark_data_smithi090_166897_object68521:head# 2024-05-15T23:39:56.391 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:600da86d:::benchmark_data_smithi090_166897_object41617:head# 2024-05-15T23:39:56.392 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:600dd719:::benchmark_data_smithi090_166897_object63737:head# 2024-05-15T23:39:56.392 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:600e66ba:::benchmark_data_smithi090_166897_object67065:head# 2024-05-15T23:39:56.392 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:600fdd65:::benchmark_data_smithi090_166897_object12014:head# 2024-05-15T23:39:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60108bb6:::benchmark_data_smithi090_166897_object12150:head# 2024-05-15T23:39:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6012dd58:::benchmark_data_smithi090_166897_object48212:head# 2024-05-15T23:39:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6015b4db:::benchmark_data_smithi090_166897_object2945:head# 2024-05-15T23:39:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6016a135:::benchmark_data_smithi090_166897_object12886:head# 2024-05-15T23:39:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60176914:::benchmark_data_smithi090_166897_object22474:head# 2024-05-15T23:39:56.394 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6019ea65:::benchmark_data_smithi090_166897_object65965:head# 2024-05-15T23:39:56.394 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:601c1b3a:::benchmark_data_smithi090_166897_object29401:head# 2024-05-15T23:39:56.394 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:601c848b:::benchmark_data_smithi090_166897_object11387:head# 2024-05-15T23:39:56.395 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:601d496b:::benchmark_data_smithi090_166897_object27716:head# 2024-05-15T23:39:56.395 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:601d898b:::benchmark_data_smithi090_166897_object22708:head# 2024-05-15T23:39:56.395 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:601ff967:::benchmark_data_smithi090_166897_object24763:head# 2024-05-15T23:39:56.396 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60204340:::benchmark_data_smithi090_166897_object67063:head# 2024-05-15T23:39:56.396 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60225d83:::benchmark_data_smithi090_166897_object39597:head# 2024-05-15T23:39:56.397 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6023c1e9:::benchmark_data_smithi090_166897_object70715:head# 2024-05-15T23:39:56.397 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6023c32f:::benchmark_data_smithi090_166897_object65783:head# 2024-05-15T23:39:56.398 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:602701cc:::benchmark_data_smithi090_166897_object26430:head# 2024-05-15T23:39:56.398 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:602ab461:::benchmark_data_smithi090_166897_object19908:head# 2024-05-15T23:39:56.398 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:602df5f4:::benchmark_data_smithi090_166897_object2500:head# 2024-05-15T23:39:56.399 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:602e9aea:::benchmark_data_smithi090_166897_object70129:head# 2024-05-15T23:39:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:602f0a43:::benchmark_data_smithi090_166897_object33762:head# 2024-05-15T23:39:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60309446:::benchmark_data_smithi090_166897_object12603:head# 2024-05-15T23:39:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6032eb9e:::benchmark_data_smithi090_166897_object2791:head# 2024-05-15T23:39:56.402 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60333bfe:::benchmark_data_smithi090_166897_object51378:head# 2024-05-15T23:39:56.406 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6034f97c:::benchmark_data_smithi090_166897_object62263:head# 2024-05-15T23:39:56.406 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60354e21:::benchmark_data_smithi090_166897_object44130:head# 2024-05-15T23:39:56.406 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6035ebcd:::benchmark_data_smithi090_166897_object1943:head# 2024-05-15T23:39:56.406 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60366d3d:::benchmark_data_smithi090_166897_object31798:head# 2024-05-15T23:39:56.406 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6037d39a:::benchmark_data_smithi090_166897_object259:head# 2024-05-15T23:39:56.406 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:603abae4:::benchmark_data_smithi090_166897_object62302:head# 2024-05-15T23:39:56.406 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:603b207b:::benchmark_data_smithi090_166897_object24784:head# 2024-05-15T23:39:56.407 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:603bb70b:::benchmark_data_smithi090_166897_object49264:head# 2024-05-15T23:39:56.407 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:603e2699:::benchmark_data_smithi090_166897_object14818:head# 2024-05-15T23:39:56.407 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:603f4336:::benchmark_data_smithi090_166897_object17971:head# 2024-05-15T23:39:56.407 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60402e8f:::benchmark_data_smithi090_166897_object56787:head# 2024-05-15T23:39:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60430fb8:::benchmark_data_smithi090_166897_object34672:head# 2024-05-15T23:39:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6043ecb8:::benchmark_data_smithi090_166897_object49075:head# 2024-05-15T23:39:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6044d078:::benchmark_data_smithi090_166897_object17932:head# 2024-05-15T23:39:56.409 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6045181e:::benchmark_data_smithi090_166897_object50981:head# 2024-05-15T23:39:56.409 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:604533e4:::benchmark_data_smithi090_166897_object52933:head# 2024-05-15T23:39:56.410 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:604576ca:::benchmark_data_smithi090_166897_object27631:head# 2024-05-15T23:39:56.410 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6046590d:::benchmark_data_smithi090_166897_object62222:head# 2024-05-15T23:39:56.411 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60465987:::benchmark_data_smithi090_166897_object57365:head# 2024-05-15T23:39:56.411 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:604704c3:::benchmark_data_smithi090_166897_object34658:head# 2024-05-15T23:39:56.412 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:604857f8:::benchmark_data_smithi090_166897_object69641:head# 2024-05-15T23:39:56.412 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6048d91f:::benchmark_data_smithi090_166897_object40349:head# 2024-05-15T23:39:56.413 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6049e87c:::benchmark_data_smithi090_166897_object24095:head# 2024-05-15T23:39:56.413 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:604d56fc:::benchmark_data_smithi090_166897_object22301:head# 2024-05-15T23:39:56.414 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:604ee3d9:::benchmark_data_smithi090_166897_object70120:head# 2024-05-15T23:39:56.414 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:604eec0e:::benchmark_data_smithi090_166897_object25322:head# 2024-05-15T23:39:56.415 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:604ef649:::benchmark_data_smithi090_166897_object32775:head# 2024-05-15T23:39:56.415 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:604fc945:::benchmark_data_smithi090_166897_object34756:head# 2024-05-15T23:39:56.415 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:604fd567:::benchmark_data_smithi090_166897_object33388:head# 2024-05-15T23:39:56.416 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:605060c5:::benchmark_data_smithi090_166897_object12459:head# 2024-05-15T23:39:56.416 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6050a5c2:::benchmark_data_smithi090_166897_object42219:head# 2024-05-15T23:39:56.417 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6050f4a5:::benchmark_data_smithi090_166897_object51565:head# 2024-05-15T23:39:56.417 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60511293:::benchmark_data_smithi090_166897_object55513:head# 2024-05-15T23:39:56.418 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:605186e8:::benchmark_data_smithi090_166897_object46617:head# 2024-05-15T23:39:56.418 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60530427:::benchmark_data_smithi090_166897_object22290:head# 2024-05-15T23:39:56.418 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60533bea:::benchmark_data_smithi090_166897_object1795:head# 2024-05-15T23:39:56.419 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6053454a:::benchmark_data_smithi090_166897_object40760:head# 2024-05-15T23:39:56.419 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:605378ce:::benchmark_data_smithi090_166897_object20252:head# 2024-05-15T23:39:56.420 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60555685:::benchmark_data_smithi090_166897_object17558:head# 2024-05-15T23:39:56.420 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6057bd36:::benchmark_data_smithi090_166897_object55190:head# 2024-05-15T23:39:56.420 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60581384:::benchmark_data_smithi090_166897_object52320:head# 2024-05-15T23:39:56.421 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60594be5:::benchmark_data_smithi090_166897_object50795:head# 2024-05-15T23:39:56.421 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:605a0465:::benchmark_data_smithi090_166897_object13697:head# 2024-05-15T23:39:56.424 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:605a1df2:::benchmark_data_smithi090_166897_object24376:head# 2024-05-15T23:39:56.424 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:605d5db4:::benchmark_data_smithi090_166897_object47058:head# 2024-05-15T23:39:56.425 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:605dbbcc:::benchmark_data_smithi090_166897_object47069:head# 2024-05-15T23:39:56.425 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:605ebc40:::benchmark_data_smithi090_166897_object3010:head# 2024-05-15T23:39:56.426 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:605edd0a:::benchmark_data_smithi090_166897_object29253:head# 2024-05-15T23:39:56.426 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60613041:::benchmark_data_smithi090_166897_object34679:head# 2024-05-15T23:39:56.427 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60628c79:::benchmark_data_smithi090_166897_object42877:head# 2024-05-15T23:39:56.427 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60628ed3:::benchmark_data_smithi090_166897_object10045:head# 2024-05-15T23:39:56.427 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6063458a:::benchmark_data_smithi090_166897_object65703:head# 2024-05-15T23:39:56.428 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60652f06:::benchmark_data_smithi090_166897_object58556:head# 2024-05-15T23:39:56.431 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:606581cc:::benchmark_data_smithi090_166897_object11251:head# 2024-05-15T23:39:56.431 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6066f396:::benchmark_data_smithi090_166897_object54555:head# 2024-05-15T23:39:56.431 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:606700cf:::benchmark_data_smithi090_166897_object54066:head# 2024-05-15T23:39:56.431 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60672d81:::benchmark_data_smithi090_166897_object28133:head# 2024-05-15T23:39:56.431 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60680bfe:::benchmark_data_smithi090_166897_object19890:head# 2024-05-15T23:39:56.431 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:606865e3:::benchmark_data_smithi090_166897_object16107:head# 2024-05-15T23:39:56.432 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:606ae3d9:::benchmark_data_smithi090_166897_object18200:head# 2024-05-15T23:39:56.432 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:606b4616:::benchmark_data_smithi090_166897_object54015:head# 2024-05-15T23:39:56.433 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:606c39c3:::benchmark_data_smithi090_166897_object38428:head# 2024-05-15T23:39:56.433 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:606cce6b:::benchmark_data_smithi090_166897_object5528:head# 2024-05-15T23:39:56.433 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:606d0342:::benchmark_data_smithi090_166897_object58228:head# 2024-05-15T23:39:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:606d26d9:::benchmark_data_smithi090_166897_object1585:head# 2024-05-15T23:39:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:606f4c31:::benchmark_data_smithi090_166897_object54953:head# 2024-05-15T23:39:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:606f6cf4:::benchmark_data_smithi090_166897_object56779:head# 2024-05-15T23:39:56.436 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:606f8dc9:::benchmark_data_smithi090_166897_object199:head# 2024-05-15T23:39:56.436 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60700638:::benchmark_data_smithi090_166897_object57047:head# 2024-05-15T23:39:56.436 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:607108d4:::benchmark_data_smithi090_166897_object55820:head# 2024-05-15T23:39:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6075b927:::benchmark_data_smithi090_166897_object32403:head# 2024-05-15T23:39:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:607659fa:::benchmark_data_smithi090_166897_object3133:head# 2024-05-15T23:39:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6076f32e:::benchmark_data_smithi090_166897_object35823:head# 2024-05-15T23:39:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:607776bf:::benchmark_data_smithi090_166897_object1374:head# 2024-05-15T23:39:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6078d3cb:::benchmark_data_smithi090_166897_object18261:head# 2024-05-15T23:39:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6079a984:::benchmark_data_smithi090_166897_object31356:head# 2024-05-15T23:39:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6079bf95:::benchmark_data_smithi090_166897_object66331:head# 2024-05-15T23:39:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:607a5284:::benchmark_data_smithi090_166897_object11218:head# 2024-05-15T23:39:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:607c5316:::benchmark_data_smithi090_166897_object69532:head# 2024-05-15T23:39:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:607ca04d:::benchmark_data_smithi090_166897_object6821:head# 2024-05-15T23:39:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:607d3518:::benchmark_data_smithi090_166897_object8681:head# 2024-05-15T23:39:56.442 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:607f510b:::benchmark_data_smithi090_166897_object34420:head# 2024-05-15T23:39:56.443 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:608120b2:::benchmark_data_smithi090_166897_object45652:head# 2024-05-15T23:39:56.443 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60813bcb:::benchmark_data_smithi090_166897_object8413:head# 2024-05-15T23:39:56.443 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6081db8e:::benchmark_data_smithi090_166897_object44163:head# 2024-05-15T23:39:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6083f21f:::benchmark_data_smithi090_166897_object64074:head# 2024-05-15T23:39:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6085311b:::benchmark_data_smithi090_166897_object57937:head# 2024-05-15T23:39:56.445 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6085dd7b:::benchmark_data_smithi090_166897_object54938:head# 2024-05-15T23:39:56.445 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60862f57:::benchmark_data_smithi090_166897_object63431:head# 2024-05-15T23:39:56.445 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:608714b0:::benchmark_data_smithi090_166897_object27622:head# 2024-05-15T23:39:56.446 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6088845b:::benchmark_data_smithi090_166897_object4115:head# 2024-05-15T23:39:56.447 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6088f749:::benchmark_data_smithi090_166897_object70487:head# 2024-05-15T23:39:56.447 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:608dbbec:::benchmark_data_smithi090_166897_object57293:head# 2024-05-15T23:39:56.447 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:608e9473:::benchmark_data_smithi090_166897_object6442:head# 2024-05-15T23:39:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:608feaff:::benchmark_data_smithi090_166897_object66616:head# 2024-05-15T23:39:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6090f2ca:::benchmark_data_smithi090_166897_object36605:head# 2024-05-15T23:39:56.449 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:609148d4:::benchmark_data_smithi090_166897_object60759:head# 2024-05-15T23:39:56.449 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6091c6d6:::benchmark_data_smithi090_166897_object35634:head# 2024-05-15T23:39:56.450 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6092405f:::benchmark_data_smithi090_166897_object62665:head# 2024-05-15T23:39:56.450 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60939c83:::benchmark_data_smithi090_166897_object23503:head# 2024-05-15T23:39:56.451 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6094e863:::benchmark_data_smithi090_166897_object35702:head# 2024-05-15T23:39:56.451 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6098b814:::benchmark_data_smithi090_166897_object53954:head# 2024-05-15T23:39:56.452 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:609c3767:::benchmark_data_smithi090_166897_object56853:head# 2024-05-15T23:39:56.453 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:609c838f:::benchmark_data_smithi090_166897_object32798:head# 2024-05-15T23:39:56.453 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:609dd238:::benchmark_data_smithi090_166897_object63637:head# 2024-05-15T23:39:56.454 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:609edc23:::benchmark_data_smithi090_166897_object36882:head# 2024-05-15T23:39:56.454 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:609f82b8:::benchmark_data_smithi090_166897_object60892:head# 2024-05-15T23:39:56.455 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60a03328:::benchmark_data_smithi090_166897_object69914:head# 2024-05-15T23:39:56.455 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60a0cb2b:::benchmark_data_smithi090_166897_object2578:head# 2024-05-15T23:39:56.456 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60a0d24d:::benchmark_data_smithi090_166897_object71010:head# 2024-05-15T23:39:56.457 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60a1f9d1:::benchmark_data_smithi090_166897_object19624:head# 2024-05-15T23:39:56.457 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60a24ee9:::benchmark_data_smithi090_166897_object52501:head# 2024-05-15T23:39:56.457 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60a25b4e:::benchmark_data_smithi090_166897_object34800:head# 2024-05-15T23:39:56.458 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60a2e915:::benchmark_data_smithi090_166897_object58624:head# 2024-05-15T23:39:56.458 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60a33113:::benchmark_data_smithi090_166897_object11413:head# 2024-05-15T23:39:56.459 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60a475f7:::benchmark_data_smithi090_166897_object29334:head# 2024-05-15T23:39:56.459 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60a492cb:::benchmark_data_smithi090_166897_object378:head# 2024-05-15T23:39:56.460 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60a4bcc0:::benchmark_data_smithi090_166897_object14626:head# 2024-05-15T23:39:56.460 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60a69b41:::benchmark_data_smithi090_166897_object8994:head# 2024-05-15T23:39:56.461 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60a7ade0:::benchmark_data_smithi090_166897_object58208:head# 2024-05-15T23:39:56.461 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60a7d8e7:::benchmark_data_smithi090_166897_object67081:head# 2024-05-15T23:39:56.463 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60a96002:::benchmark_data_smithi090_166897_object2974:head# 2024-05-15T23:39:56.463 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60aa21c7:::benchmark_data_smithi090_166897_object23892:head# 2024-05-15T23:39:56.464 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60aaf5ae:::benchmark_data_smithi090_166897_object38870:head# 2024-05-15T23:39:56.464 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60abcb98:::benchmark_data_smithi090_166897_object41995:head# 2024-05-15T23:39:56.465 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60acd683:::benchmark_data_smithi090_166897_object27510:head# 2024-05-15T23:39:56.465 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ad114a:::benchmark_data_smithi090_166897_object70184:head# 2024-05-15T23:39:56.466 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ad943e:::benchmark_data_smithi090_166897_object11994:head# 2024-05-15T23:39:56.467 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ae2489:::benchmark_data_smithi090_166897_object69022:head# 2024-05-15T23:39:56.468 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ae3a70:::benchmark_data_smithi090_166897_object43833:head# 2024-05-15T23:39:56.469 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60af6756:::benchmark_data_smithi090_166897_object68107:head# 2024-05-15T23:39:56.470 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60afb20b:::benchmark_data_smithi090_166897_object35919:head# 2024-05-15T23:39:56.471 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60b0c3fe:::benchmark_data_smithi090_166897_object68899:head# 2024-05-15T23:39:56.471 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60b22d4b:::benchmark_data_smithi090_166897_object51316:head# 2024-05-15T23:39:56.471 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60b2a636:::benchmark_data_smithi090_166897_object13844:head# 2024-05-15T23:39:56.471 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60b2d7ae:::benchmark_data_smithi090_166897_object30800:head# 2024-05-15T23:39:56.473 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60b44999:::benchmark_data_smithi090_166897_object29556:head# 2024-05-15T23:39:56.473 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60b56593:::benchmark_data_smithi090_166897_object31381:head# 2024-05-15T23:39:56.473 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60b5956f:::benchmark_data_smithi090_166897_object19212:head# 2024-05-15T23:39:56.473 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60b86990:::benchmark_data_smithi090_166897_object14200:head# 2024-05-15T23:39:56.473 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60b953da:::benchmark_data_smithi090_166897_object7012:head# 2024-05-15T23:39:56.474 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60b984e6:::benchmark_data_smithi090_166897_object17290:head# 2024-05-15T23:39:56.474 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60b99f29:::benchmark_data_smithi090_166897_object49757:head# 2024-05-15T23:39:56.475 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60b9b28c:::benchmark_data_smithi090_166897_object41145:head# 2024-05-15T23:39:56.476 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ba6f11:::benchmark_data_smithi090_166897_object53593:head# 2024-05-15T23:39:56.476 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60bc0fa3:::benchmark_data_smithi090_166897_object41702:head# 2024-05-15T23:39:56.476 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60bc50f6:::benchmark_data_smithi090_166897_object61678:head# 2024-05-15T23:39:56.477 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60bc57f6:::benchmark_data_smithi090_166897_object20335:head# 2024-05-15T23:39:56.477 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60bc8c96:::benchmark_data_smithi090_166897_object18078:head# 2024-05-15T23:39:56.477 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60bca1bf:::benchmark_data_smithi090_166897_object48225:head# 2024-05-15T23:39:56.478 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60bdf74a:::benchmark_data_smithi090_166897_object16659:head# 2024-05-15T23:39:56.478 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60bf5313:::benchmark_data_smithi090_166897_object27481:head# 2024-05-15T23:39:56.479 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60c2d72a:::benchmark_data_smithi090_166897_object44789:head# 2024-05-15T23:39:56.479 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60c3f2d5:::benchmark_data_smithi090_166897_object34506:head# 2024-05-15T23:39:56.480 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60c50864:::benchmark_data_smithi090_166897_object5018:head# 2024-05-15T23:39:56.480 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60c69f4c:::benchmark_data_smithi090_166897_object13636:head# 2024-05-15T23:39:56.481 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60c8298c:::benchmark_data_smithi090_166897_object55750:head# 2024-05-15T23:39:56.481 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60c97135:::benchmark_data_smithi090_166897_object36387:head# 2024-05-15T23:39:56.482 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60c9b450:::benchmark_data_smithi090_166897_object50571:head# 2024-05-15T23:39:56.482 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60c9f432:::benchmark_data_smithi090_166897_object60760:head# 2024-05-15T23:39:56.483 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ca0682:::benchmark_data_smithi090_166897_object47831:head# 2024-05-15T23:39:56.483 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60cb3d4d:::benchmark_data_smithi090_166897_object29913:head# 2024-05-15T23:39:56.484 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60cc8161:::benchmark_data_smithi090_166897_object22392:head# 2024-05-15T23:39:56.486 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ce7b6f:::benchmark_data_smithi090_166897_object54009:head# 2024-05-15T23:39:56.487 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ce8514:::benchmark_data_smithi090_166897_object14247:head# 2024-05-15T23:39:56.487 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ceaeed:::benchmark_data_smithi090_166897_object69851:head# 2024-05-15T23:39:56.487 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60cef96b:::benchmark_data_smithi090_166897_object13437:head# 2024-05-15T23:39:56.487 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60cfbce5:::benchmark_data_smithi090_166897_object45583:head# 2024-05-15T23:39:56.491 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60d123bb:::benchmark_data_smithi090_166897_object20123:head# 2024-05-15T23:39:56.491 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60d15757:::benchmark_data_smithi090_166897_object56780:head# 2024-05-15T23:39:56.491 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60d3ae15:::benchmark_data_smithi090_166897_object13093:head# 2024-05-15T23:39:56.491 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60d3f759:::benchmark_data_smithi090_166897_object3310:head# 2024-05-15T23:39:56.491 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60d48877:::benchmark_data_smithi090_166897_object55347:head# 2024-05-15T23:39:56.492 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60d59c84:::benchmark_data_smithi090_166897_object48366:head# 2024-05-15T23:39:56.492 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60d65099:::benchmark_data_smithi090_166897_object41235:head# 2024-05-15T23:39:56.492 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60d6e162:::benchmark_data_smithi090_166897_object34544:head# 2024-05-15T23:39:56.492 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60d8fef7:::benchmark_data_smithi090_166897_object17287:head# 2024-05-15T23:39:56.492 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60d91924:::benchmark_data_smithi090_166897_object61029:head# 2024-05-15T23:39:56.492 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60da6110:::benchmark_data_smithi090_166897_object3178:head# 2024-05-15T23:39:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60dba8dc:::benchmark_data_smithi090_166897_object45195:head# 2024-05-15T23:39:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60dc310d:::benchmark_data_smithi090_166897_object34673:head# 2024-05-15T23:39:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60dc34ee:::benchmark_data_smithi090_166897_object65391:head# 2024-05-15T23:39:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60dd01bc:::benchmark_data_smithi090_166897_object23976:head# 2024-05-15T23:39:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60dee7ef:::benchmark_data_smithi090_166897_object67819:head# 2024-05-15T23:39:56.494 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60df1a4c:::benchmark_data_smithi090_166897_object60038:head# 2024-05-15T23:39:56.495 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60e17703:::benchmark_data_smithi090_166897_object910:head# 2024-05-15T23:39:56.496 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60e1f8ac:::benchmark_data_smithi090_166897_object18180:head# 2024-05-15T23:39:56.496 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60e3383c:::benchmark_data_smithi090_166897_object34256:head# 2024-05-15T23:39:56.501 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60e3eff5:::benchmark_data_smithi090_166897_object6252:head# 2024-05-15T23:39:56.501 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60e44ce6:::benchmark_data_smithi090_166897_object28614:head# 2024-05-15T23:39:56.501 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60e481fa:::benchmark_data_smithi090_166897_object29873:head# 2024-05-15T23:39:56.502 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60e59df4:::benchmark_data_smithi090_166897_object21826:head# 2024-05-15T23:39:56.502 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60e59fc8:::benchmark_data_smithi090_166897_object58991:head# 2024-05-15T23:39:56.502 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60e5d6f6:::benchmark_data_smithi090_166897_object52978:head# 2024-05-15T23:39:56.502 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60e65557:::benchmark_data_smithi090_166897_object30589:head# 2024-05-15T23:39:56.502 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60e977f1:::benchmark_data_smithi090_166897_object22214:head# 2024-05-15T23:39:56.503 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ea09f3:::benchmark_data_smithi090_166897_object19681:head# 2024-05-15T23:39:56.503 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ea7772:::benchmark_data_smithi090_166897_object33977:head# 2024-05-15T23:39:56.504 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ebb1b7:::benchmark_data_smithi090_166897_object16602:head# 2024-05-15T23:39:56.504 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ec25fb:::benchmark_data_smithi090_166897_object58259:head# 2024-05-15T23:39:56.504 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ed4832:::benchmark_data_smithi090_166897_object45073:head# 2024-05-15T23:39:56.506 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60edd713:::benchmark_data_smithi090_166897_object69947:head# 2024-05-15T23:39:56.506 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ef17e5:::benchmark_data_smithi090_166897_object56638:head# 2024-05-15T23:39:56.506 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ef32ab:::benchmark_data_smithi090_166897_object66797:head# 2024-05-15T23:39:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60f0c63e:::benchmark_data_smithi090_166897_object38688:head# 2024-05-15T23:39:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60f16281:::benchmark_data_smithi090_166897_object9076:head# 2024-05-15T23:39:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60f1dd2f:::benchmark_data_smithi090_166897_object20054:head# 2024-05-15T23:39:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60f22908:::benchmark_data_smithi090_166897_object26193:head# 2024-05-15T23:39:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60f29f9a:::benchmark_data_smithi090_166897_object36279:head# 2024-05-15T23:39:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60f3851b:::benchmark_data_smithi090_166897_object45190:head# 2024-05-15T23:39:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60f5ab35:::benchmark_data_smithi090_166897_object47957:head# 2024-05-15T23:39:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60f5e8b7:::benchmark_data_smithi090_166897_object17303:head# 2024-05-15T23:39:56.508 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60f6d8b8:::benchmark_data_smithi090_166897_object3409:head# 2024-05-15T23:39:56.510 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60f73f41:::benchmark_data_smithi090_166897_object30222:head# 2024-05-15T23:39:56.510 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60f83198:::benchmark_data_smithi090_166897_object57744:head# 2024-05-15T23:39:56.511 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60f891fc:::benchmark_data_smithi090_166897_object23152:head# 2024-05-15T23:39:56.511 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60fb958e:::benchmark_data_smithi090_166897_object26942:head# 2024-05-15T23:39:56.513 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60fbc10e:::benchmark_data_smithi090_166897_object52092:head# 2024-05-15T23:39:56.513 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60fc4e8b:::benchmark_data_smithi090_166897_object9146:head# 2024-05-15T23:39:56.513 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60fdcbe4:::benchmark_data_smithi090_166897_object48339:head# 2024-05-15T23:39:56.513 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60fe44fb:::benchmark_data_smithi090_166897_object27551:head# 2024-05-15T23:39:56.513 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:60ffbde9:::benchmark_data_smithi090_166897_object29144:head# 2024-05-15T23:39:56.514 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61008425:::benchmark_data_smithi090_166897_object56542:head# 2024-05-15T23:39:56.514 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6104c8e5:::benchmark_data_smithi090_166897_object21115:head# 2024-05-15T23:39:56.515 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61054929:::benchmark_data_smithi090_166897_object54691:head# 2024-05-15T23:39:56.515 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6107f9af:::benchmark_data_smithi090_166897_object49513:head# 2024-05-15T23:39:56.516 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:610b6627:::benchmark_data_smithi090_166897_object30748:head# 2024-05-15T23:39:56.516 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:610c35ea:::benchmark_data_smithi090_166897_object12270:head# 2024-05-15T23:39:56.516 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:610c80d0:::benchmark_data_smithi090_166897_object9719:head# 2024-05-15T23:39:56.517 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:610ebc92:::benchmark_data_smithi090_166897_object3034:head# 2024-05-15T23:39:56.517 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:610f1c36:::benchmark_data_smithi090_166897_object22625:head# 2024-05-15T23:39:56.518 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:610f8873:::benchmark_data_smithi090_166897_object56207:head# 2024-05-15T23:39:56.518 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:610fe05e:::benchmark_data_smithi090_166897_object49847:head# 2024-05-15T23:39:56.519 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6111741f:::benchmark_data_smithi090_166897_object54809:head# 2024-05-15T23:39:56.519 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6111edae:::benchmark_data_smithi090_166897_object30668:head# 2024-05-15T23:39:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6111f85f:::benchmark_data_smithi090_166897_object26854:head# 2024-05-15T23:39:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:611237bb:::benchmark_data_smithi090_166897_object69751:head# 2024-05-15T23:39:56.521 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61142088:::benchmark_data_smithi090_166897_object23039:head# 2024-05-15T23:39:56.521 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61143cb3:::benchmark_data_smithi090_166897_object58723:head# 2024-05-15T23:39:56.522 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6114a873:::benchmark_data_smithi090_166897_object44492:head# 2024-05-15T23:39:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:611620dd:::benchmark_data_smithi090_166897_object30383:head# 2024-05-15T23:39:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6116acce:::benchmark_data_smithi090_166897_object4400:head# 2024-05-15T23:39:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6117d233:::benchmark_data_smithi090_166897_object19575:head# 2024-05-15T23:39:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:611818f9:::benchmark_data_smithi090_166897_object13991:head# 2024-05-15T23:39:56.524 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:611844b0:::benchmark_data_smithi090_166897_object63224:head# 2024-05-15T23:39:56.524 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6119a688:::benchmark_data_smithi090_166897_object11773:head# 2024-05-15T23:39:56.525 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:611be89a:::benchmark_data_smithi090_166897_object38280:head# 2024-05-15T23:39:56.525 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:611c8631:::benchmark_data_smithi090_166897_object48070:head# 2024-05-15T23:39:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:611d83e3:::benchmark_data_smithi090_166897_object55156:head# 2024-05-15T23:39:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:611db2ed:::benchmark_data_smithi090_166897_object13646:head# 2024-05-15T23:39:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:611e595a:::benchmark_data_smithi090_166897_object50387:head# 2024-05-15T23:39:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:611f1b12:::benchmark_data_smithi090_166897_object63691:head# 2024-05-15T23:39:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:612029f8:::benchmark_data_smithi090_166897_object8772:head# 2024-05-15T23:39:56.530 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6120b255:::benchmark_data_smithi090_166897_object35719:head# 2024-05-15T23:39:56.530 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61213fb6:::benchmark_data_smithi090_166897_object15564:head# 2024-05-15T23:39:56.530 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61221aaf:::benchmark_data_smithi090_166897_object27768:head# 2024-05-15T23:39:56.530 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61232602:::benchmark_data_smithi090_166897_object33958:head# 2024-05-15T23:39:56.531 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61235fb7:::benchmark_data_smithi090_166897_object23775:head# 2024-05-15T23:39:56.531 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6123791d:::benchmark_data_smithi090_166897_object62811:head# 2024-05-15T23:39:56.531 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61243113:::benchmark_data_smithi090_166897_object66757:head# 2024-05-15T23:39:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6125de85:::benchmark_data_smithi090_166897_object61713:head# 2024-05-15T23:39:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61289494:::benchmark_data_smithi090_166897_object855:head# 2024-05-15T23:39:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:612969e5:::benchmark_data_smithi090_166897_object38685:head# 2024-05-15T23:39:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:612a0960:::benchmark_data_smithi090_166897_object10491:head# 2024-05-15T23:39:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:612a7dc9:::benchmark_data_smithi090_166897_object12414:head# 2024-05-15T23:39:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:612ab5f7:::benchmark_data_smithi090_166897_object55361:head# 2024-05-15T23:39:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:612b955d:::benchmark_data_smithi090_166897_object22653:head# 2024-05-15T23:39:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:612be218:::benchmark_data_smithi090_166897_object53835:head# 2024-05-15T23:39:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:612c55b3:::benchmark_data_smithi090_166897_object32045:head# 2024-05-15T23:39:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:612d854b:::benchmark_data_smithi090_166897_object65231:head# 2024-05-15T23:39:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:612d8c53:::benchmark_data_smithi090_166897_object53230:head# 2024-05-15T23:39:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:612e9488:::benchmark_data_smithi090_166897_object20254:head# 2024-05-15T23:39:56.539 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:612ebfb6:::benchmark_data_smithi090_166897_object48516:head# 2024-05-15T23:39:56.539 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:613035bf:::benchmark_data_smithi090_166897_object70671:head# 2024-05-15T23:39:56.539 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6130f03a:::benchmark_data_smithi090_166897_object17910:head# 2024-05-15T23:39:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61316cec:::benchmark_data_smithi090_166897_object52932:head# 2024-05-15T23:39:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61330365:::benchmark_data_smithi090_166897_object17843:head# 2024-05-15T23:39:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6135274c:::benchmark_data_smithi090_166897_object10448:head# 2024-05-15T23:39:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:613572a5:::benchmark_data_smithi090_166897_object27350:head# 2024-05-15T23:39:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6135994a:::benchmark_data_smithi090_166897_object37116:head# 2024-05-15T23:39:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:613635f5:::benchmark_data_smithi090_166897_object8983:head# 2024-05-15T23:39:56.543 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6136800c:::benchmark_data_smithi090_166897_object59570:head# 2024-05-15T23:39:56.543 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:613684fe:::benchmark_data_smithi090_166897_object7600:head# 2024-05-15T23:39:56.545 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6136ec69:::benchmark_data_smithi090_166897_object40150:head# 2024-05-15T23:39:56.545 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:613afcb6:::benchmark_data_smithi090_166897_object25006:head# 2024-05-15T23:39:56.546 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:613b451b:::benchmark_data_smithi090_166897_object52016:head# 2024-05-15T23:39:56.546 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:613b5088:::benchmark_data_smithi090_166897_object40359:head# 2024-05-15T23:39:56.546 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:613d179b:::benchmark_data_smithi090_166897_object30209:head# 2024-05-15T23:39:56.547 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:613da6bd:::benchmark_data_smithi090_166897_object69884:head# 2024-05-15T23:39:56.547 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:613f2838:::benchmark_data_smithi090_166897_object50378:head# 2024-05-15T23:39:56.549 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:613f53bd:::benchmark_data_smithi090_166897_object6190:head# 2024-05-15T23:39:56.550 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:613f64a7:::benchmark_data_smithi090_166897_object16405:head# 2024-05-15T23:39:56.550 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:613f912c:::benchmark_data_smithi090_166897_object32423:head# 2024-05-15T23:39:56.551 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6140324b:::benchmark_data_smithi090_166897_object62877:head# 2024-05-15T23:39:56.551 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6142990a:::benchmark_data_smithi090_166897_object54356:head# 2024-05-15T23:39:56.552 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61429a05:::benchmark_data_smithi090_166897_object17690:head# 2024-05-15T23:39:56.552 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6145aa9d:::benchmark_data_smithi090_166897_object13577:head# 2024-05-15T23:39:56.554 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61476283:::benchmark_data_smithi090_166897_object46452:head# 2024-05-15T23:39:56.555 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6148a516:::benchmark_data_smithi090_166897_object70303:head# 2024-05-15T23:39:56.555 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:614b2105:::benchmark_data_smithi090_166897_object29232:head# 2024-05-15T23:39:56.556 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:614b4c15:::benchmark_data_smithi090_166897_object7299:head# 2024-05-15T23:39:56.556 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:614c7342:::benchmark_data_smithi090_166897_object12182:head# 2024-05-15T23:39:56.556 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:614c89d3:::benchmark_data_smithi090_166897_object24480:head# 2024-05-15T23:39:56.557 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:614ce1fc:::benchmark_data_smithi090_166897_object32566:head# 2024-05-15T23:39:56.557 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:614dcb2e:::benchmark_data_smithi090_166897_object46837:head# 2024-05-15T23:39:56.558 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:614de7cb:::benchmark_data_smithi090_166897_object61611:head# 2024-05-15T23:39:56.558 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:615018ff:::benchmark_data_smithi090_166897_object20673:head# 2024-05-15T23:39:56.559 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61511120:::benchmark_data_smithi090_166897_object36951:head# 2024-05-15T23:39:56.561 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61517d54:::benchmark_data_smithi090_166897_object11733:head# 2024-05-15T23:39:56.563 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6151ab82:::benchmark_data_smithi090_166897_object3815:head# 2024-05-15T23:39:56.564 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61526a91:::benchmark_data_smithi090_166897_object64622:head# 2024-05-15T23:39:56.564 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:615353e5:::benchmark_data_smithi090_166897_object4980:head# 2024-05-15T23:39:56.564 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6153d1b8:::benchmark_data_smithi090_166897_object6121:head# 2024-05-15T23:39:56.565 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61547ccc:::benchmark_data_smithi090_166897_object34757:head# 2024-05-15T23:39:56.565 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6154f289:::benchmark_data_smithi090_166897_object29014:head# 2024-05-15T23:39:56.566 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6155a349:::benchmark_data_smithi090_166897_object3391:head# 2024-05-15T23:39:56.566 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61566729:::benchmark_data_smithi090_166897_object48550:head# 2024-05-15T23:39:56.568 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6156f677:::benchmark_data_smithi090_166897_object6261:head# 2024-05-15T23:39:56.568 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 54 16 85492 85476 98.9041 70.4375 0.00333573 0.0101069 2024-05-15T23:39:56.569 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61574837:::benchmark_data_smithi090_166897_object2671:head# 2024-05-15T23:39:56.569 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:615770a7:::benchmark_data_smithi090_166897_object31741:head# 2024-05-15T23:39:56.572 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:615a86a1:::benchmark_data_smithi090_166897_object41763:head# 2024-05-15T23:39:56.572 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:615b7dd5:::benchmark_data_smithi090_166897_object2965:head# 2024-05-15T23:39:56.572 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:615cc162:::benchmark_data_smithi090_166897_object43046:head# 2024-05-15T23:39:56.573 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:615dae6f:::benchmark_data_smithi090_166897_object33760:head# 2024-05-15T23:39:56.574 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:615dbeff:::benchmark_data_smithi090_166897_object44696:head# 2024-05-15T23:39:56.574 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:615dddd6:::benchmark_data_smithi090_166897_object24680:head# 2024-05-15T23:39:56.574 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:615ebc64:::benchmark_data_smithi090_166897_object35704:head# 2024-05-15T23:39:56.575 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:615ec775:::benchmark_data_smithi090_166897_object819:head# 2024-05-15T23:39:56.575 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:615ec8fe:::benchmark_data_smithi090_166897_object65001:head# 2024-05-15T23:39:56.575 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:615ef31c:::benchmark_data_smithi090_166897_object59790:head# 2024-05-15T23:39:56.576 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:616020b0:::benchmark_data_smithi090_166897_object45442:head# 2024-05-15T23:39:56.576 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61604775:::benchmark_data_smithi090_166897_object12079:head# 2024-05-15T23:39:56.577 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61614357:::benchmark_data_smithi090_166897_object59602:head# 2024-05-15T23:39:56.579 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6165ffcf:::benchmark_data_smithi090_166897_object21803:head# 2024-05-15T23:39:56.580 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61670c83:::benchmark_data_smithi090_166897_object19473:head# 2024-05-15T23:39:56.581 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6167245a:::benchmark_data_smithi090_166897_object756:head# 2024-05-15T23:39:56.581 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:616a2f4c:::benchmark_data_smithi090_166897_object52272:head# 2024-05-15T23:39:56.581 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:616b9c6f:::benchmark_data_smithi090_166897_object13974:head# 2024-05-15T23:39:56.582 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:616c7bd3:::benchmark_data_smithi090_166897_object36578:head# 2024-05-15T23:39:56.582 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:616d9881:::benchmark_data_smithi090_166897_object63649:head# 2024-05-15T23:39:56.582 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:616f3938:::benchmark_data_smithi090_166897_object14096:head# 2024-05-15T23:39:56.583 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:616f8b2a:::benchmark_data_smithi090_166897_object19898:head# 2024-05-15T23:39:56.583 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61706a8e:::benchmark_data_smithi090_166897_object62274:head# 2024-05-15T23:39:56.584 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61710fe3:::benchmark_data_smithi090_166897_object68382:head# 2024-05-15T23:39:56.584 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61715ba9:::benchmark_data_smithi090_166897_object21548:head# 2024-05-15T23:39:56.585 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6171e8de:::benchmark_data_smithi090_166897_object3571:head# 2024-05-15T23:39:56.585 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6171f22a:::benchmark_data_smithi090_166897_object38159:head# 2024-05-15T23:39:56.586 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61722985:::benchmark_data_smithi090_166897_object60175:head# 2024-05-15T23:39:56.586 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6172763d:::benchmark_data_smithi090_166897_object48525:head# 2024-05-15T23:39:56.587 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6174aaaa:::benchmark_data_smithi090_166897_object20113:head# 2024-05-15T23:39:56.587 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:617530c1:::benchmark_data_smithi090_166897_object3904:head# 2024-05-15T23:39:56.588 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61754f65:::benchmark_data_smithi090_166897_object41577:head# 2024-05-15T23:39:56.589 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61755d21:::benchmark_data_smithi090_166897_object24569:head# 2024-05-15T23:39:56.589 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61761e7b:::benchmark_data_smithi090_166897_object14903:head# 2024-05-15T23:39:56.589 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6176f5dc:::benchmark_data_smithi090_166897_object20355:head# 2024-05-15T23:39:56.589 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6177a175:::benchmark_data_smithi090_166897_object33847:head# 2024-05-15T23:39:56.591 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6177ae49:::benchmark_data_smithi090_166897_object38653:head# 2024-05-15T23:39:56.591 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6178fb8f:::benchmark_data_smithi090_166897_object25009:head# 2024-05-15T23:39:56.592 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:617950bb:::benchmark_data_smithi090_166897_object32908:head# 2024-05-15T23:39:56.592 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:617b1a56:::benchmark_data_smithi090_166897_object13076:head# 2024-05-15T23:39:56.593 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:617b8206:::benchmark_data_smithi090_166897_object24098:head# 2024-05-15T23:39:56.593 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:617c7a82:::benchmark_data_smithi090_166897_object68533:head# 2024-05-15T23:39:56.594 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:617cab4c:::benchmark_data_smithi090_166897_object45596:head# 2024-05-15T23:39:56.594 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:617db779:::benchmark_data_smithi090_166897_object5195:head# 2024-05-15T23:39:56.595 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:617f8d6c:::benchmark_data_smithi090_166897_object8791:head# 2024-05-15T23:39:56.595 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:617fb4e6:::benchmark_data_smithi090_166897_object59851:head# 2024-05-15T23:39:56.596 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6180ed6a:::benchmark_data_smithi090_166897_object11992:head# 2024-05-15T23:39:56.596 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61817a1a:::benchmark_data_smithi090_166897_object26225:head# 2024-05-15T23:39:56.598 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61823d14:::benchmark_data_smithi090_166897_object28934:head# 2024-05-15T23:39:56.599 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:618265bd:::benchmark_data_smithi090_166897_object45349:head# 2024-05-15T23:39:56.599 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6182ae38:::benchmark_data_smithi090_166897_object66634:head# 2024-05-15T23:39:56.600 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61831c5e:::benchmark_data_smithi090_166897_object41178:head# 2024-05-15T23:39:56.600 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6184f3be:::benchmark_data_smithi090_166897_object70490:head# 2024-05-15T23:39:56.601 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6185bbae:::benchmark_data_smithi090_166897_object42117:head# 2024-05-15T23:39:56.601 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6187081b:::benchmark_data_smithi090_166897_object18600:head# 2024-05-15T23:39:56.601 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61881017:::benchmark_data_smithi090_166897_object56682:head# 2024-05-15T23:39:56.602 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6188a3c3:::benchmark_data_smithi090_166897_object8176:head# 2024-05-15T23:39:56.602 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:618aee23:::benchmark_data_smithi090_166897_object66480:head# 2024-05-15T23:39:56.603 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:618aff3b:::benchmark_data_smithi090_166897_object13860:head# 2024-05-15T23:39:56.603 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:618b6202:::benchmark_data_smithi090_166897_object64901:head# 2024-05-15T23:39:56.604 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:618c17e3:::benchmark_data_smithi090_166897_object54949:head# 2024-05-15T23:39:56.604 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:618da72a:::benchmark_data_smithi090_166897_object2770:head# 2024-05-15T23:39:56.605 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:618e61db:::benchmark_data_smithi090_166897_object68332:head# 2024-05-15T23:39:56.605 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:618ef06a:::benchmark_data_smithi090_166897_object43389:head# 2024-05-15T23:39:56.605 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:618f0c01:::benchmark_data_smithi090_166897_object62564:head# 2024-05-15T23:39:56.606 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:618f474b:::benchmark_data_smithi090_166897_object45074:head# 2024-05-15T23:39:56.607 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6190a935:::benchmark_data_smithi090_166897_object10639:head# 2024-05-15T23:39:56.607 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61923ad3:::benchmark_data_smithi090_166897_object61410:head# 2024-05-15T23:39:56.608 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:619247b1:::benchmark_data_smithi090_166897_object39044:head# 2024-05-15T23:39:56.608 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6192740b:::benchmark_data_smithi090_166897_object10807:head# 2024-05-15T23:39:56.609 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6192837b:::benchmark_data_smithi090_166897_object13941:head# 2024-05-15T23:39:56.609 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6192e1e8:::benchmark_data_smithi090_166897_object43927:head# 2024-05-15T23:39:56.609 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6193778f:::benchmark_data_smithi090_166897_object69812:head# 2024-05-15T23:39:56.610 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6194ae84:::benchmark_data_smithi090_166897_object60352:head# 2024-05-15T23:39:56.610 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61954ae5:::benchmark_data_smithi090_166897_object34793:head# 2024-05-15T23:39:56.610 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61965227:::benchmark_data_smithi090_166897_object17235:head# 2024-05-15T23:39:56.611 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:619f8e90:::benchmark_data_smithi090_166897_object40391:head# 2024-05-15T23:39:56.611 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61a00af1:::benchmark_data_smithi090_166897_object48098:head# 2024-05-15T23:39:56.612 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61a0fc10:::benchmark_data_smithi090_166897_object37874:head# 2024-05-15T23:39:56.612 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61a16ea9:::benchmark_data_smithi090_166897_object10865:head# 2024-05-15T23:39:56.613 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61a19a4e:::benchmark_data_smithi090_166897_object52897:head# 2024-05-15T23:39:56.613 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61a1dd72:::benchmark_data_smithi090_166897_object24360:head# 2024-05-15T23:39:56.614 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61a24027:::benchmark_data_smithi090_166897_object23771:head# 2024-05-15T23:39:56.614 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61a2643d:::benchmark_data_smithi090_166897_object268:head# 2024-05-15T23:39:56.614 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61a3d7eb:::benchmark_data_smithi090_166897_object38659:head# 2024-05-15T23:39:56.615 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61a63121:::benchmark_data_smithi090_166897_object44626:head# 2024-05-15T23:39:56.615 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61a828a5:::benchmark_data_smithi090_166897_object18908:head# 2024-05-15T23:39:56.616 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61a86b0f:::benchmark_data_smithi090_166897_object59584:head# 2024-05-15T23:39:56.616 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61a9c5a4:::benchmark_data_smithi090_166897_object16994:head# 2024-05-15T23:39:56.617 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61a9f41d:::benchmark_data_smithi090_166897_object60227:head# 2024-05-15T23:39:56.618 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61aa0c92:::benchmark_data_smithi090_166897_object33666:head# 2024-05-15T23:39:56.618 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61aad786:::benchmark_data_smithi090_166897_object57257:head# 2024-05-15T23:39:56.619 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61ab7768:::benchmark_data_smithi090_166897_object27969:head# 2024-05-15T23:39:56.619 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61abeae5:::benchmark_data_smithi090_166897_object58693:head# 2024-05-15T23:39:56.620 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61add859:::benchmark_data_smithi090_166897_object53381:head# 2024-05-15T23:39:56.620 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61af2663:::benchmark_data_smithi090_166897_object31588:head# 2024-05-15T23:39:56.621 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61b18d3d:::benchmark_data_smithi090_166897_object60974:head# 2024-05-15T23:39:56.621 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61b4cdea:::benchmark_data_smithi090_166897_object21203:head# 2024-05-15T23:39:56.622 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61b6beb5:::benchmark_data_smithi090_166897_object49862:head# 2024-05-15T23:39:56.622 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61b7b6d4:::benchmark_data_smithi090_166897_object415:head# 2024-05-15T23:39:56.622 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61b80e70:::benchmark_data_smithi090_166897_object13833:head# 2024-05-15T23:39:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61baa641:::benchmark_data_smithi090_166897_object30238:head# 2024-05-15T23:39:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61bac318:::benchmark_data_smithi090_166897_object21744:head# 2024-05-15T23:39:56.624 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61bb7b14:::benchmark_data_smithi090_166897_object34954:head# 2024-05-15T23:39:56.624 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61bd25d3:::benchmark_data_smithi090_166897_object53711:head# 2024-05-15T23:39:56.624 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61bed7c4:::benchmark_data_smithi090_166897_object15798:head# 2024-05-15T23:39:56.625 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61c01c63:::benchmark_data_smithi090_166897_object7275:head# 2024-05-15T23:39:56.625 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61c0e50b:::benchmark_data_smithi090_166897_object26437:head# 2024-05-15T23:39:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61c18c66:::benchmark_data_smithi090_166897_object46914:head# 2024-05-15T23:39:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61c303b7:::benchmark_data_smithi090_166897_object29412:head# 2024-05-15T23:39:56.627 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61c32ce3:::benchmark_data_smithi090_166897_object25715:head# 2024-05-15T23:39:56.627 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61c3448f:::benchmark_data_smithi090_166897_object37635:head# 2024-05-15T23:39:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61c47c0f:::benchmark_data_smithi090_166897_object42948:head# 2024-05-15T23:39:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61c48a8c:::benchmark_data_smithi090_166897_object36726:head# 2024-05-15T23:39:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61c57977:::benchmark_data_smithi090_166897_object37204:head# 2024-05-15T23:39:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61c707fb:::benchmark_data_smithi090_166897_object43282:head# 2024-05-15T23:39:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61c867d2:::benchmark_data_smithi090_166897_object33448:head# 2024-05-15T23:39:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61c9e66a:::benchmark_data_smithi090_166897_object29866:head# 2024-05-15T23:39:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61cabf5d:::benchmark_data_smithi090_166897_object13758:head# 2024-05-15T23:39:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61cb0bcb:::benchmark_data_smithi090_166897_object59828:head# 2024-05-15T23:39:56.631 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61cc486a:::benchmark_data_smithi090_166897_object53967:head# 2024-05-15T23:39:56.631 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61ce9993:::benchmark_data_smithi090_166897_object66127:head# 2024-05-15T23:39:56.632 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61cfe9d8:::benchmark_data_smithi090_166897_object42153:head# 2024-05-15T23:39:56.632 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61d08847:::benchmark_data_smithi090_166897_object27214:head# 2024-05-15T23:39:56.632 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61d30894:::benchmark_data_smithi090_166897_object33712:head# 2024-05-15T23:39:56.633 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61d3212b:::benchmark_data_smithi090_166897_object39069:head# 2024-05-15T23:39:56.633 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61d36601:::benchmark_data_smithi090_166897_object11065:head# 2024-05-15T23:39:56.634 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61d38416:::benchmark_data_smithi090_166897_object13392:head# 2024-05-15T23:39:56.634 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61d47a50:::benchmark_data_smithi090_166897_object17444:head# 2024-05-15T23:39:56.634 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61d5722c:::benchmark_data_smithi090_166897_object4992:head# 2024-05-15T23:39:56.635 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61d84dd8:::benchmark_data_smithi090_166897_object50880:head# 2024-05-15T23:39:56.635 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61dc74a5:::benchmark_data_smithi090_166897_object57243:head# 2024-05-15T23:39:56.636 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61ddaa9c:::benchmark_data_smithi090_166897_object37464:head# 2024-05-15T23:39:56.636 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61ddb97c:::benchmark_data_smithi090_166897_object48238:head# 2024-05-15T23:39:56.637 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61ddba66:::benchmark_data_smithi090_166897_object19067:head# 2024-05-15T23:39:56.637 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61de1607:::benchmark_data_smithi090_166897_object16395:head# 2024-05-15T23:39:56.637 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61df396f:::benchmark_data_smithi090_166897_object167:head# 2024-05-15T23:39:56.640 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61dfe9cf:::benchmark_data_smithi090_166897_object7401:head# 2024-05-15T23:39:56.640 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61e11b44:::benchmark_data_smithi090_166897_object4176:head# 2024-05-15T23:39:56.641 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61e11f4a:::benchmark_data_smithi090_166897_object42258:head# 2024-05-15T23:39:56.641 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61e121ce:::benchmark_data_smithi090_166897_object13540:head# 2024-05-15T23:39:56.641 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61e1720e:::benchmark_data_smithi090_166897_object64669:head# 2024-05-15T23:39:56.641 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61e1be76:::benchmark_data_smithi090_166897_object51799:head# 2024-05-15T23:39:56.641 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61e24a5c:::benchmark_data_smithi090_166897_object59083:head# 2024-05-15T23:39:56.641 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61e49f22:::benchmark_data_smithi090_166897_object16507:head# 2024-05-15T23:39:56.641 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61e545cf:::benchmark_data_smithi090_166897_object34178:head# 2024-05-15T23:39:56.642 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61e58d52:::benchmark_data_smithi090_166897_object70840:head# 2024-05-15T23:39:56.642 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61e82af9:::benchmark_data_smithi090_166897_object26062:head# 2024-05-15T23:39:56.643 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61e86434:::benchmark_data_smithi090_166897_object34129:head# 2024-05-15T23:39:56.643 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61e879aa:::benchmark_data_smithi090_166897_object13960:head# 2024-05-15T23:39:56.643 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61e992a5:::benchmark_data_smithi090_166897_object64527:head# 2024-05-15T23:39:56.644 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61e9eb9e:::benchmark_data_smithi090_166897_object37733:head# 2024-05-15T23:39:56.644 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61e9f89b:::benchmark_data_smithi090_166897_object49751:head# 2024-05-15T23:39:56.645 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61ea8f3c:::benchmark_data_smithi090_166897_object69304:head# 2024-05-15T23:39:56.645 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61ebaf71:::benchmark_data_smithi090_166897_object32771:head# 2024-05-15T23:39:56.645 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61ebe33b:::benchmark_data_smithi090_166897_object18419:head# 2024-05-15T23:39:56.646 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61ec4c67:::benchmark_data_smithi090_166897_object24123:head# 2024-05-15T23:39:56.646 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61ed8c58:::benchmark_data_smithi090_166897_object58537:head# 2024-05-15T23:39:56.647 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61edb473:::benchmark_data_smithi090_166897_object30843:head# 2024-05-15T23:39:56.647 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61edd8e5:::benchmark_data_smithi090_166897_object63547:head# 2024-05-15T23:39:56.648 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61eef977:::benchmark_data_smithi090_166897_object34374:head# 2024-05-15T23:39:56.649 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61efc979:::benchmark_data_smithi090_166897_object920:head# 2024-05-15T23:39:56.649 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61efff27:::benchmark_data_smithi090_166897_object31675:head# 2024-05-15T23:39:56.649 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61f0c438:::benchmark_data_smithi090_166897_object32281:head# 2024-05-15T23:39:56.650 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61f32a42:::benchmark_data_smithi090_166897_object27361:head# 2024-05-15T23:39:56.650 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61f37262:::benchmark_data_smithi090_166897_object35563:head# 2024-05-15T23:39:56.651 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61f3af8e:::benchmark_data_smithi090_166897_object17719:head# 2024-05-15T23:39:56.651 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61f41f9e:::benchmark_data_smithi090_166897_object17202:head# 2024-05-15T23:39:56.652 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61f58df1:::benchmark_data_smithi090_166897_object395:head# 2024-05-15T23:39:56.652 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61f683ca:::benchmark_data_smithi090_166897_object29461:head# 2024-05-15T23:39:56.653 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61f6cf42:::benchmark_data_smithi090_166897_object28184:head# 2024-05-15T23:39:56.653 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61f72399:::benchmark_data_smithi090_166897_object50862:head# 2024-05-15T23:39:56.653 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61f90b91:::benchmark_data_smithi090_166897_object26467:head# 2024-05-15T23:39:56.654 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61fa20c6:::benchmark_data_smithi090_166897_object14023:head# 2024-05-15T23:39:56.654 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61fa2299:::benchmark_data_smithi090_166897_object41975:head# 2024-05-15T23:39:56.655 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61fa3b6d:::benchmark_data_smithi090_166897_object21087:head# 2024-05-15T23:39:56.655 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61fa6b3f:::benchmark_data_smithi090_166897_object38695:head# 2024-05-15T23:39:56.660 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61fc21eb:::benchmark_data_smithi090_166897_object63517:head# 2024-05-15T23:39:56.660 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61fcb914:::benchmark_data_smithi090_166897_object2674:head# 2024-05-15T23:39:56.660 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61fd7645:::benchmark_data_smithi090_166897_object31696:head# 2024-05-15T23:39:56.660 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61fd952a:::benchmark_data_smithi090_166897_object48812:head# 2024-05-15T23:39:56.660 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:61fe8307:::benchmark_data_smithi090_166897_object25137:head# 2024-05-15T23:39:56.660 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62010446:::benchmark_data_smithi090_166897_object31730:head# 2024-05-15T23:39:56.661 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6201574a:::benchmark_data_smithi090_166897_object48213:head# 2024-05-15T23:39:56.661 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62023f6d:::benchmark_data_smithi090_166897_object26098:head# 2024-05-15T23:39:56.661 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6202efae:::benchmark_data_smithi090_166897_object19176:head# 2024-05-15T23:39:56.661 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62034ec4:::benchmark_data_smithi090_166897_object44852:head# 2024-05-15T23:39:56.661 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62039225:::benchmark_data_smithi090_166897_object10960:head# 2024-05-15T23:39:56.661 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62040acc:::benchmark_data_smithi090_166897_object32026:head# 2024-05-15T23:39:56.661 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62042eb0:::benchmark_data_smithi090_166897_object58731:head# 2024-05-15T23:39:56.662 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62047770:::benchmark_data_smithi090_166897_object59555:head# 2024-05-15T23:39:56.662 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62049ef5:::benchmark_data_smithi090_166897_object36673:head# 2024-05-15T23:39:56.663 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62062493:::benchmark_data_smithi090_166897_object53857:head# 2024-05-15T23:39:56.663 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6206e3e1:::benchmark_data_smithi090_166897_object32138:head# 2024-05-15T23:39:56.664 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62070ad9:::benchmark_data_smithi090_166897_object65614:head# 2024-05-15T23:39:56.664 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62070b76:::benchmark_data_smithi090_166897_object30197:head# 2024-05-15T23:39:56.664 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:620736e2:::benchmark_data_smithi090_166897_object28125:head# 2024-05-15T23:39:56.665 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6207ae42:::benchmark_data_smithi090_166897_object57473:head# 2024-05-15T23:39:56.665 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:620904ec:::benchmark_data_smithi090_166897_object43194:head# 2024-05-15T23:39:56.666 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62091c64:::benchmark_data_smithi090_166897_object49979:head# 2024-05-15T23:39:56.666 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6209726d:::benchmark_data_smithi090_166897_object984:head# 2024-05-15T23:39:56.667 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:620ae72f:::benchmark_data_smithi090_166897_object30234:head# 2024-05-15T23:39:56.667 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:620af87d:::benchmark_data_smithi090_166897_object38146:head# 2024-05-15T23:39:56.668 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:620c47f0:::benchmark_data_smithi090_166897_object89:head# 2024-05-15T23:39:56.668 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:620cf1ca:::benchmark_data_smithi090_166897_object14558:head# 2024-05-15T23:39:56.669 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:620d84a0:::benchmark_data_smithi090_166897_object56547:head# 2024-05-15T23:39:56.669 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:620e54e5:::benchmark_data_smithi090_166897_object31858:head# 2024-05-15T23:39:56.670 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:620f3c0b:::benchmark_data_smithi090_166897_object66822:head# 2024-05-15T23:39:56.670 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:621384a3:::benchmark_data_smithi090_166897_object19547:head# 2024-05-15T23:39:56.670 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6214c427:::benchmark_data_smithi090_166897_object10849:head# 2024-05-15T23:39:56.671 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:621500f6:::benchmark_data_smithi090_166897_object62805:head# 2024-05-15T23:39:56.672 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:621549b9:::benchmark_data_smithi090_166897_object49802:head# 2024-05-15T23:39:56.672 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62175e02:::benchmark_data_smithi090_166897_object48127:head# 2024-05-15T23:39:56.672 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62198a86:::benchmark_data_smithi090_166897_object14347:head# 2024-05-15T23:39:56.673 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:621a5959:::benchmark_data_smithi090_166897_object39764:head# 2024-05-15T23:39:56.673 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:621b927e:::benchmark_data_smithi090_166897_object43673:head# 2024-05-15T23:39:56.674 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:621bdb82:::benchmark_data_smithi090_166897_object19743:head# 2024-05-15T23:39:56.675 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:621c5b63:::benchmark_data_smithi090_166897_object17274:head# 2024-05-15T23:39:56.675 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:621c7bfd:::benchmark_data_smithi090_166897_object14186:head# 2024-05-15T23:39:56.675 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:621e2d5f:::benchmark_data_smithi090_166897_object40963:head# 2024-05-15T23:39:56.676 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:621e402f:::benchmark_data_smithi090_166897_object70870:head# 2024-05-15T23:39:56.676 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6220e335:::benchmark_data_smithi090_166897_object7507:head# 2024-05-15T23:39:56.677 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6221bce6:::benchmark_data_smithi090_166897_object60584:head# 2024-05-15T23:39:56.677 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62230ea5:::benchmark_data_smithi090_166897_object36410:head# 2024-05-15T23:39:56.678 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6224cde0:::benchmark_data_smithi090_166897_object56750:head# 2024-05-15T23:39:56.678 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6224cf11:::benchmark_data_smithi090_166897_object49370:head# 2024-05-15T23:39:56.678 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62253601:::benchmark_data_smithi090_166897_object39064:head# 2024-05-15T23:39:56.679 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:622676c4:::benchmark_data_smithi090_166897_object60947:head# 2024-05-15T23:39:56.679 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62269e37:::benchmark_data_smithi090_166897_object65299:head# 2024-05-15T23:39:56.680 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62281a43:::benchmark_data_smithi090_166897_object60001:head# 2024-05-15T23:39:56.680 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6229277a:::benchmark_data_smithi090_166897_object20948:head# 2024-05-15T23:39:56.681 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6229da49:::benchmark_data_smithi090_166897_object33180:head# 2024-05-15T23:39:56.681 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:622a66ce:::benchmark_data_smithi090_166897_object5473:head# 2024-05-15T23:39:56.682 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:622bb25c:::benchmark_data_smithi090_166897_object6943:head# 2024-05-15T23:39:56.682 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:622beaac:::benchmark_data_smithi090_166897_object19036:head# 2024-05-15T23:39:56.683 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:622d0cbb:::benchmark_data_smithi090_166897_object23954:head# 2024-05-15T23:39:56.683 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:622d7ee9:::benchmark_data_smithi090_166897_object40307:head# 2024-05-15T23:39:56.684 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:622db15f:::benchmark_data_smithi090_166897_object27524:head# 2024-05-15T23:39:56.684 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:622e046f:::benchmark_data_smithi090_166897_object17698:head# 2024-05-15T23:39:56.685 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:623089e9:::benchmark_data_smithi090_166897_object43071:head# 2024-05-15T23:39:56.685 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62333a55:::benchmark_data_smithi090_166897_object51177:head# 2024-05-15T23:39:56.685 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62338a8b:::benchmark_data_smithi090_166897_object11272:head# 2024-05-15T23:39:56.686 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62354027:::benchmark_data_smithi090_166897_object49194:head# 2024-05-15T23:39:56.686 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6236d269:::benchmark_data_smithi090_166897_object55029:head# 2024-05-15T23:39:56.688 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:623913bc:::benchmark_data_smithi090_166897_object29860:head# 2024-05-15T23:39:56.691 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6239830e:::benchmark_data_smithi090_166897_object7880:head# 2024-05-15T23:39:56.692 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:623a5017:::benchmark_data_smithi090_166897_object28373:head# 2024-05-15T23:39:56.692 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:623a734e:::benchmark_data_smithi090_166897_object31743:head# 2024-05-15T23:39:56.692 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:623bc17d:::benchmark_data_smithi090_166897_object20392:head# 2024-05-15T23:39:56.693 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:623d0116:::benchmark_data_smithi090_166897_object67120:head# 2024-05-15T23:39:56.694 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:623e2e28:::benchmark_data_smithi090_166897_object54239:head# 2024-05-15T23:39:56.694 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62410d72:::benchmark_data_smithi090_166897_object28832:head# 2024-05-15T23:39:56.694 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62418dc9:::benchmark_data_smithi090_166897_object24169:head# 2024-05-15T23:39:56.695 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:624334d2:::benchmark_data_smithi090_166897_object3742:head# 2024-05-15T23:39:56.695 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:624578ed:::benchmark_data_smithi090_166897_object7934:head# 2024-05-15T23:39:56.696 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6245ce6d:::benchmark_data_smithi090_166897_object21911:head# 2024-05-15T23:39:56.697 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6245e34e:::benchmark_data_smithi090_166897_object25568:head# 2024-05-15T23:39:56.697 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62464831:::benchmark_data_smithi090_166897_object10968:head# 2024-05-15T23:39:56.698 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62467bb7:::benchmark_data_smithi090_166897_object42493:head# 2024-05-15T23:39:56.699 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62468302:::benchmark_data_smithi090_166897_object29492:head# 2024-05-15T23:39:56.699 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62485de1:::benchmark_data_smithi090_166897_object8125:head# 2024-05-15T23:39:56.702 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62486113:::benchmark_data_smithi090_166897_object70312:head# 2024-05-15T23:39:56.702 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62495447:::benchmark_data_smithi090_166897_object25267:head# 2024-05-15T23:39:56.703 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:624a2abb:::benchmark_data_smithi090_166897_object69389:head# 2024-05-15T23:39:56.703 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:624b027c:::benchmark_data_smithi090_166897_object4537:head# 2024-05-15T23:39:56.704 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:624b5a6e:::benchmark_data_smithi090_166897_object44829:head# 2024-05-15T23:39:56.704 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:624c3dec:::benchmark_data_smithi090_166897_object8939:head# 2024-05-15T23:39:56.705 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:624ca9d3:::benchmark_data_smithi090_166897_object20246:head# 2024-05-15T23:39:56.705 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:624cedf7:::benchmark_data_smithi090_166897_object44859:head# 2024-05-15T23:39:56.708 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:624ea0bf:::benchmark_data_smithi090_166897_object52536:head# 2024-05-15T23:39:56.708 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62502605:::benchmark_data_smithi090_166897_object10947:head# 2024-05-15T23:39:56.708 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62502abc:::benchmark_data_smithi090_166897_object25738:head# 2024-05-15T23:39:56.709 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62506e7c:::benchmark_data_smithi090_166897_object27911:head# 2024-05-15T23:39:56.710 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6250813a:::benchmark_data_smithi090_166897_object61847:head# 2024-05-15T23:39:56.710 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6251064b:::benchmark_data_smithi090_166897_object36424:head# 2024-05-15T23:39:56.710 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6253047c:::benchmark_data_smithi090_166897_object20998:head# 2024-05-15T23:39:56.711 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62532702:::benchmark_data_smithi090_166897_object40212:head# 2024-05-15T23:39:56.714 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62534deb:::benchmark_data_smithi090_166897_object6840:head# 2024-05-15T23:39:56.714 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62539109:::benchmark_data_smithi090_166897_object24830:head# 2024-05-15T23:39:56.715 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62544023:::benchmark_data_smithi090_166897_object39004:head# 2024-05-15T23:39:56.715 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6255182b:::benchmark_data_smithi090_166897_object6278:head# 2024-05-15T23:39:56.715 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6255f47d:::benchmark_data_smithi090_166897_object65959:head# 2024-05-15T23:39:56.716 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6255ff2a:::benchmark_data_smithi090_166897_object15975:head# 2024-05-15T23:39:56.716 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62564dc7:::benchmark_data_smithi090_166897_object25064:head# 2024-05-15T23:39:56.717 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62568c29:::benchmark_data_smithi090_166897_object2457:head# 2024-05-15T23:39:56.717 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6256c392:::benchmark_data_smithi090_166897_object63836:head# 2024-05-15T23:39:56.718 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:625a3d5c:::benchmark_data_smithi090_166897_object6833:head# 2024-05-15T23:39:56.718 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:625a76b4:::benchmark_data_smithi090_166897_object25022:head# 2024-05-15T23:39:56.718 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:625da19b:::benchmark_data_smithi090_166897_object42257:head# 2024-05-15T23:39:56.719 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:625e2cac:::benchmark_data_smithi090_166897_object23040:head# 2024-05-15T23:39:56.721 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:625e6e6b:::benchmark_data_smithi090_166897_object49651:head# 2024-05-15T23:39:56.722 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:625eed8d:::benchmark_data_smithi090_166897_object20586:head# 2024-05-15T23:39:56.722 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6261bc80:::benchmark_data_smithi090_166897_object43985:head# 2024-05-15T23:39:56.723 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62634a57:::benchmark_data_smithi090_166897_object55690:head# 2024-05-15T23:39:56.724 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6263f844:::benchmark_data_smithi090_166897_object37662:head# 2024-05-15T23:39:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62646a08:::benchmark_data_smithi090_166897_object52703:head# 2024-05-15T23:39:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6265ad03:::benchmark_data_smithi090_166897_object9216:head# 2024-05-15T23:39:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6266d1a6:::benchmark_data_smithi090_166897_object37234:head# 2024-05-15T23:39:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62670587:::benchmark_data_smithi090_166897_object32475:head# 2024-05-15T23:39:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62674ff5:::benchmark_data_smithi090_166897_object51477:head# 2024-05-15T23:39:56.726 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62696f7f:::benchmark_data_smithi090_166897_object27971:head# 2024-05-15T23:39:56.726 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:626a891a:::benchmark_data_smithi090_166897_object55563:head# 2024-05-15T23:39:56.726 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:626bbb26:::benchmark_data_smithi090_166897_object48675:head# 2024-05-15T23:39:56.728 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:626be7ef:::benchmark_data_smithi090_166897_object44792:head# 2024-05-15T23:39:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:626c2459:::benchmark_data_smithi090_166897_object30181:head# 2024-05-15T23:39:56.731 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:626c2602:::benchmark_data_smithi090_166897_object63247:head# 2024-05-15T23:39:56.734 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:626ceaad:::benchmark_data_smithi090_166897_object21793:head# 2024-05-15T23:39:56.734 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:626d6a36:::benchmark_data_smithi090_166897_object65679:head# 2024-05-15T23:39:56.735 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62721046:::benchmark_data_smithi090_166897_object35052:head# 2024-05-15T23:39:56.735 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:627376a3:::benchmark_data_smithi090_166897_object62992:head# 2024-05-15T23:39:56.735 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6274816c:::benchmark_data_smithi090_166897_object29780:head# 2024-05-15T23:39:56.736 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62754cb4:::benchmark_data_smithi090_166897_object7684:head# 2024-05-15T23:39:56.736 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6278efed:::benchmark_data_smithi090_166897_object67203:head# 2024-05-15T23:39:56.737 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62795330:::benchmark_data_smithi090_166897_object9297:head# 2024-05-15T23:39:56.737 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6279a815:::benchmark_data_smithi090_166897_object10784:head# 2024-05-15T23:39:56.738 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:627b40db:::benchmark_data_smithi090_166897_object12045:head# 2024-05-15T23:39:56.738 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:627bc5ea:::benchmark_data_smithi090_166897_object42184:head# 2024-05-15T23:39:56.738 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:628045ba:::benchmark_data_smithi090_166897_object25096:head# 2024-05-15T23:39:56.739 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6281bee2:::benchmark_data_smithi090_166897_object49817:head# 2024-05-15T23:39:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62830d1f:::benchmark_data_smithi090_166897_object22441:head# 2024-05-15T23:39:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6283e95e:::benchmark_data_smithi090_166897_object70415:head# 2024-05-15T23:39:56.741 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6284a699:::benchmark_data_smithi090_166897_object67279:head# 2024-05-15T23:39:56.741 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6284c95e:::benchmark_data_smithi090_166897_object36173:head# 2024-05-15T23:39:56.741 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6285c6a0:::benchmark_data_smithi090_166897_object31794:head# 2024-05-15T23:39:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62861624:::benchmark_data_smithi090_166897_object69295:head# 2024-05-15T23:39:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6288246b:::benchmark_data_smithi090_166897_object42278:head# 2024-05-15T23:39:56.743 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6288bfc3:::benchmark_data_smithi090_166897_object51900:head# 2024-05-15T23:39:56.743 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:628acd4e:::benchmark_data_smithi090_166897_object44315:head# 2024-05-15T23:39:56.744 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:628c651a:::benchmark_data_smithi090_166897_object36875:head# 2024-05-15T23:39:56.744 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:628c7e1f:::benchmark_data_smithi090_166897_object34833:head# 2024-05-15T23:39:56.745 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:628dabce:::benchmark_data_smithi090_166897_object23189:head# 2024-05-15T23:39:56.745 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:628e9866:::benchmark_data_smithi090_166897_object29666:head# 2024-05-15T23:39:56.746 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:628e9f1f:::benchmark_data_smithi090_166897_object68488:head# 2024-05-15T23:39:56.746 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6291dd19:::benchmark_data_smithi090_166897_object46467:head# 2024-05-15T23:39:56.747 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6291edd7:::benchmark_data_smithi090_166897_object41761:head# 2024-05-15T23:39:56.747 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:629204c1:::benchmark_data_smithi090_166897_object15355:head# 2024-05-15T23:39:56.748 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:629684d0:::benchmark_data_smithi090_166897_object25530:head# 2024-05-15T23:39:56.748 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6296bc49:::benchmark_data_smithi090_166897_object65838:head# 2024-05-15T23:39:56.749 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6297cc3f:::benchmark_data_smithi090_166897_object17376:head# 2024-05-15T23:39:56.749 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6297d5d6:::benchmark_data_smithi090_166897_object53766:head# 2024-05-15T23:39:56.750 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:629a033b:::benchmark_data_smithi090_166897_object3125:head# 2024-05-15T23:39:56.752 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:629a2e56:::benchmark_data_smithi090_166897_object50495:head# 2024-05-15T23:39:56.752 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:629abc2e:::benchmark_data_smithi090_166897_object48501:head# 2024-05-15T23:39:56.753 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:629c0c41:::benchmark_data_smithi090_166897_object25760:head# 2024-05-15T23:39:56.753 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:629d569a:::benchmark_data_smithi090_166897_object17060:head# 2024-05-15T23:39:56.754 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:629d5a60:::benchmark_data_smithi090_166897_object23483:head# 2024-05-15T23:39:56.754 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:629e2c42:::benchmark_data_smithi090_166897_object1728:head# 2024-05-15T23:39:56.755 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:629e90d1:::benchmark_data_smithi090_166897_object63502:head# 2024-05-15T23:39:56.755 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:629ffce2:::benchmark_data_smithi090_166897_object29545:head# 2024-05-15T23:39:56.756 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62a20dc9:::benchmark_data_smithi090_166897_object49396:head# 2024-05-15T23:39:56.756 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62a27132:::benchmark_data_smithi090_166897_object19306:head# 2024-05-15T23:39:56.756 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62a46e82:::benchmark_data_smithi090_166897_object44056:head# 2024-05-15T23:39:56.757 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62a4a217:::benchmark_data_smithi090_166897_object29476:head# 2024-05-15T23:39:56.758 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62a4c6f0:::benchmark_data_smithi090_166897_object23425:head# 2024-05-15T23:39:56.758 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62a4c918:::benchmark_data_smithi090_166897_object42229:head# 2024-05-15T23:39:56.758 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62a57022:::benchmark_data_smithi090_166897_object25076:head# 2024-05-15T23:39:56.759 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62a6c611:::benchmark_data_smithi090_166897_object14257:head# 2024-05-15T23:39:56.759 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62a78406:::benchmark_data_smithi090_166897_object46539:head# 2024-05-15T23:39:56.760 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62a837db:::benchmark_data_smithi090_166897_object11569:head# 2024-05-15T23:39:56.761 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62a95b21:::benchmark_data_smithi090_166897_object44293:head# 2024-05-15T23:39:56.761 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62a9fe2a:::benchmark_data_smithi090_166897_object68434:head# 2024-05-15T23:39:56.763 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62aa439c:::benchmark_data_smithi090_166897_object35154:head# 2024-05-15T23:39:56.763 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62aa6896:::benchmark_data_smithi090_166897_object6422:head# 2024-05-15T23:39:56.764 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62aaa8f1:::benchmark_data_smithi090_166897_object34085:head# 2024-05-15T23:39:56.764 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62aaaaa1:::benchmark_data_smithi090_166897_object69646:head# 2024-05-15T23:39:56.764 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62ac7980:::benchmark_data_smithi090_166897_object45611:head# 2024-05-15T23:39:56.765 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62b0382f:::benchmark_data_smithi090_166897_object9737:head# 2024-05-15T23:39:56.765 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62b1813b:::benchmark_data_smithi090_166897_object56011:head# 2024-05-15T23:39:56.766 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62b20273:::benchmark_data_smithi090_166897_object31380:head# 2024-05-15T23:39:56.766 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62b20611:::benchmark_data_smithi090_166897_object7399:head# 2024-05-15T23:39:56.767 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62b34e26:::benchmark_data_smithi090_166897_object5682:head# 2024-05-15T23:39:56.767 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62b58adc:::benchmark_data_smithi090_166897_object57873:head# 2024-05-15T23:39:56.767 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62b5b7f9:::benchmark_data_smithi090_166897_object29567:head# 2024-05-15T23:39:56.768 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62b7501c:::benchmark_data_smithi090_166897_object42784:head# 2024-05-15T23:39:56.769 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62b83f9d:::benchmark_data_smithi090_166897_object55250:head# 2024-05-15T23:39:56.769 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62b8c44c:::benchmark_data_smithi090_166897_object27319:head# 2024-05-15T23:39:56.770 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62b8e8fe:::benchmark_data_smithi090_166897_object25710:head# 2024-05-15T23:39:56.770 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62b91426:::benchmark_data_smithi090_166897_object4133:head# 2024-05-15T23:39:56.772 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62bc159f:::benchmark_data_smithi090_166897_object35678:head# 2024-05-15T23:39:56.772 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62bc5ecf:::benchmark_data_smithi090_166897_object29008:head# 2024-05-15T23:39:56.772 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62bc9385:::benchmark_data_smithi090_166897_object33938:head# 2024-05-15T23:39:56.773 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62bcd764:::benchmark_data_smithi090_166897_object37348:head# 2024-05-15T23:39:56.773 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62bd8dfd:::benchmark_data_smithi090_166897_object30463:head# 2024-05-15T23:39:56.774 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62bf6fa2:::benchmark_data_smithi090_166897_object3027:head# 2024-05-15T23:39:56.774 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62bf9b65:::benchmark_data_smithi090_166897_object67740:head# 2024-05-15T23:39:56.775 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62c01598:::benchmark_data_smithi090_166897_object34003:head# 2024-05-15T23:39:56.775 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62c10883:::benchmark_data_smithi090_166897_object50766:head# 2024-05-15T23:39:56.776 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62c11f0e:::benchmark_data_smithi090_166897_object3892:head# 2024-05-15T23:39:56.776 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62c1dc97:::benchmark_data_smithi090_166897_object36093:head# 2024-05-15T23:39:56.777 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62c2a984:::benchmark_data_smithi090_166897_object15486:head# 2024-05-15T23:39:56.777 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62c2bb3c:::benchmark_data_smithi090_166897_object735:head# 2024-05-15T23:39:56.777 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62c36181:::benchmark_data_smithi090_166897_object14877:head# 2024-05-15T23:39:56.778 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62c53f83:::benchmark_data_smithi090_166897_object64487:head# 2024-05-15T23:39:56.778 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62c6fe23:::benchmark_data_smithi090_166897_object66538:head# 2024-05-15T23:39:56.779 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62c90a3f:::benchmark_data_smithi090_166897_object61430:head# 2024-05-15T23:39:56.779 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62c97796:::benchmark_data_smithi090_166897_object46870:head# 2024-05-15T23:39:56.780 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62c9e046:::benchmark_data_smithi090_166897_object11036:head# 2024-05-15T23:39:56.780 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62ca35dc:::benchmark_data_smithi090_166897_object14437:head# 2024-05-15T23:39:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62cb3ccc:::benchmark_data_smithi090_166897_object61244:head# 2024-05-15T23:39:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62cb7a3f:::benchmark_data_smithi090_166897_object43145:head# 2024-05-15T23:39:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62cd6d65:::benchmark_data_smithi090_166897_object67017:head# 2024-05-15T23:39:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62cfd4d6:::benchmark_data_smithi090_166897_object33546:head# 2024-05-15T23:39:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62cfe989:::benchmark_data_smithi090_166897_object48949:head# 2024-05-15T23:39:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d00718:::benchmark_data_smithi090_166897_object15498:head# 2024-05-15T23:39:56.784 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d05579:::benchmark_data_smithi090_166897_object34189:head# 2024-05-15T23:39:56.784 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d0d2b7:::benchmark_data_smithi090_166897_object66772:head# 2024-05-15T23:39:56.784 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d0e1f5:::benchmark_data_smithi090_166897_object59686:head# 2024-05-15T23:39:56.785 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d1bde8:::benchmark_data_smithi090_166897_object1982:head# 2024-05-15T23:39:56.785 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d29575:::benchmark_data_smithi090_166897_object34609:head# 2024-05-15T23:39:56.786 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d4389f:::benchmark_data_smithi090_166897_object42356:head# 2024-05-15T23:39:56.786 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d46c13:::benchmark_data_smithi090_166897_object28204:head# 2024-05-15T23:39:56.787 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d47f57:::benchmark_data_smithi090_166897_object57346:head# 2024-05-15T23:39:56.787 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d601bc:::benchmark_data_smithi090_166897_object65825:head# 2024-05-15T23:39:56.788 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d6317e:::benchmark_data_smithi090_166897_object19640:head# 2024-05-15T23:39:56.788 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d65371:::benchmark_data_smithi090_166897_object31600:head# 2024-05-15T23:39:56.789 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d71963:::benchmark_data_smithi090_166897_object12274:head# 2024-05-15T23:39:56.789 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d78f84:::benchmark_data_smithi090_166897_object37432:head# 2024-05-15T23:39:56.790 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d7a777:::benchmark_data_smithi090_166897_object20922:head# 2024-05-15T23:39:56.790 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d91c3a:::benchmark_data_smithi090_166897_object25884:head# 2024-05-15T23:39:56.791 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62d99a77:::benchmark_data_smithi090_166897_object36917:head# 2024-05-15T23:39:56.791 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62da5cfe:::benchmark_data_smithi090_166897_object11117:head# 2024-05-15T23:39:56.794 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62dbc8b1:::benchmark_data_smithi090_166897_object56545:head# 2024-05-15T23:39:56.794 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62dd06da:::benchmark_data_smithi090_166897_object39772:head# 2024-05-15T23:39:56.794 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62decf26:::benchmark_data_smithi090_166897_object3991:head# 2024-05-15T23:39:56.794 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62df8ccc:::benchmark_data_smithi090_166897_object13174:head# 2024-05-15T23:39:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62e1956b:::benchmark_data_smithi090_166897_object48748:head# 2024-05-15T23:39:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62e23af8:::benchmark_data_smithi090_166897_object8303:head# 2024-05-15T23:39:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62e2eab7:::benchmark_data_smithi090_166897_object7790:head# 2024-05-15T23:39:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62e348fa:::benchmark_data_smithi090_166897_object12388:head# 2024-05-15T23:39:56.796 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62e3a248:::benchmark_data_smithi090_166897_object23743:head# 2024-05-15T23:39:56.796 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62e3ed46:::benchmark_data_smithi090_166897_object69064:head# 2024-05-15T23:39:56.796 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62e42292:::benchmark_data_smithi090_166897_object37479:head# 2024-05-15T23:39:56.797 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62e50d6b:::benchmark_data_smithi090_166897_object67537:head# 2024-05-15T23:39:56.797 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62e55a19:::benchmark_data_smithi090_166897_object52423:head# 2024-05-15T23:39:56.798 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62e60251:::benchmark_data_smithi090_166897_object9247:head# 2024-05-15T23:39:56.798 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62e64f63:::benchmark_data_smithi090_166897_object46327:head# 2024-05-15T23:39:56.799 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62ea3e98:::benchmark_data_smithi090_166897_object22907:head# 2024-05-15T23:39:56.799 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62ec3fdb:::benchmark_data_smithi090_166897_object28140:head# 2024-05-15T23:39:56.799 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62ec68dd:::benchmark_data_smithi090_166897_object18386:head# 2024-05-15T23:39:56.800 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62ec78c3:::benchmark_data_smithi090_166897_object55419:head# 2024-05-15T23:39:56.800 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62ece5c4:::benchmark_data_smithi090_166897_object44967:head# 2024-05-15T23:39:56.802 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62ed6e16:::benchmark_data_smithi090_166897_object7159:head# 2024-05-15T23:39:56.802 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62eda42f:::benchmark_data_smithi090_166897_object59526:head# 2024-05-15T23:39:56.802 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62f1a6c3:::benchmark_data_smithi090_166897_object18495:head# 2024-05-15T23:39:56.803 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62f3f782:::benchmark_data_smithi090_166897_object65770:head# 2024-05-15T23:39:56.803 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62f53682:::benchmark_data_smithi090_166897_object23336:head# 2024-05-15T23:39:56.804 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62f5f83b:::benchmark_data_smithi090_166897_object69832:head# 2024-05-15T23:39:56.804 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62f6c6e3:::benchmark_data_smithi090_166897_object56366:head# 2024-05-15T23:39:56.804 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62f7bd0f:::benchmark_data_smithi090_166897_object6456:head# 2024-05-15T23:39:56.805 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62f7cf6e:::benchmark_data_smithi090_166897_object21662:head# 2024-05-15T23:39:56.805 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62f8438a:::benchmark_data_smithi090_166897_object47901:head# 2024-05-15T23:39:56.806 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62fa3249:::benchmark_data_smithi090_166897_object3580:head# 2024-05-15T23:39:56.806 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62fa6bad:::benchmark_data_smithi090_166897_object45862:head# 2024-05-15T23:39:56.809 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62fbbab2:::benchmark_data_smithi090_166897_object13966:head# 2024-05-15T23:39:56.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:56 smithi090 ceph-mon[44434]: 9.7 scrub starts 2024-05-15T23:39:56.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:56 smithi090 ceph-mon[44434]: 9.7 scrub ok 2024-05-15T23:39:56.810 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62fc338d:::benchmark_data_smithi090_166897_object1201:head# 2024-05-15T23:39:56.810 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62fd5662:::benchmark_data_smithi090_166897_object40867:head# 2024-05-15T23:39:56.810 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62ff4473:::benchmark_data_smithi090_166897_object560:head# 2024-05-15T23:39:56.810 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:62ff9347:::benchmark_data_smithi090_166897_object9577:head# 2024-05-15T23:39:56.811 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:630099ee:::benchmark_data_smithi090_166897_object53319:head# 2024-05-15T23:39:56.811 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6301a66c:::benchmark_data_smithi090_166897_object41112:head# 2024-05-15T23:39:56.811 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63061539:::benchmark_data_smithi090_166897_object53211:head# 2024-05-15T23:39:56.812 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63066a2f:::benchmark_data_smithi090_166897_object28398:head# 2024-05-15T23:39:56.812 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63079dc7:::benchmark_data_smithi090_166897_object22552:head# 2024-05-15T23:39:56.812 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63079e1d:::benchmark_data_smithi090_166897_object6374:head# 2024-05-15T23:39:56.814 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6307b2c1:::benchmark_data_smithi090_166897_object60793:head# 2024-05-15T23:39:56.814 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6307dc83:::benchmark_data_smithi090_166897_object46803:head# 2024-05-15T23:39:56.815 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:630a9d03:::benchmark_data_smithi090_166897_object20362:head# 2024-05-15T23:39:56.815 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:630ad801:::benchmark_data_smithi090_166897_object51545:head# 2024-05-15T23:39:56.816 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:630b62fa:::benchmark_data_smithi090_166897_object68911:head# 2024-05-15T23:39:56.816 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:630bbfcc:::benchmark_data_smithi090_166897_object18952:head# 2024-05-15T23:39:56.818 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:630c358b:::benchmark_data_smithi090_166897_object24920:head# 2024-05-15T23:39:56.819 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:630cfac4:::benchmark_data_smithi090_166897_object27135:head# 2024-05-15T23:39:56.819 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:630ec903:::benchmark_data_smithi090_166897_object46196:head# 2024-05-15T23:39:56.819 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:630ee629:::benchmark_data_smithi090_166897_object29007:head# 2024-05-15T23:39:56.820 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:630f2fb4:::benchmark_data_smithi090_166897_object52331:head# 2024-05-15T23:39:56.820 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63103015:::benchmark_data_smithi090_166897_object57229:head# 2024-05-15T23:39:56.821 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631103e9:::benchmark_data_smithi090_166897_object70386:head# 2024-05-15T23:39:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:56 smithi088 ceph-mon[44699]: 9.7 scrub starts 2024-05-15T23:39:56.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:56 smithi088 ceph-mon[44699]: 9.7 scrub ok 2024-05-15T23:39:56.822 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6311cad7:::benchmark_data_smithi090_166897_object49215:head# 2024-05-15T23:39:56.822 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63122c2b:::benchmark_data_smithi090_166897_object41749:head# 2024-05-15T23:39:56.822 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63132eb5:::benchmark_data_smithi090_166897_object1920:head# 2024-05-15T23:39:56.823 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63150063:::benchmark_data_smithi090_166897_object32801:head# 2024-05-15T23:39:56.823 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6315e7cc:::benchmark_data_smithi090_166897_object7382:head# 2024-05-15T23:39:56.824 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6316f73f:::benchmark_data_smithi090_166897_object55833:head# 2024-05-15T23:39:56.824 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6317494b:::benchmark_data_smithi090_166897_object16842:head# 2024-05-15T23:39:56.825 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63175360:::benchmark_data_smithi090_166897_object41437:head# 2024-05-15T23:39:56.825 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63178ed6:::benchmark_data_smithi090_166897_object7613:head# 2024-05-15T23:39:56.826 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6317d64d:::benchmark_data_smithi090_166897_object50639:head# 2024-05-15T23:39:56.826 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631803ab:::benchmark_data_smithi090_166897_object23449:head# 2024-05-15T23:39:56.826 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6318f42b:::benchmark_data_smithi090_166897_object28116:head# 2024-05-15T23:39:56.827 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6319e121:::benchmark_data_smithi090_166897_object41291:head# 2024-05-15T23:39:56.827 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631ae23d:::benchmark_data_smithi090_166897_object7813:head# 2024-05-15T23:39:56.828 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631afd4a:::benchmark_data_smithi090_166897_object39351:head# 2024-05-15T23:39:56.828 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631bb9d4:::benchmark_data_smithi090_166897_object28876:head# 2024-05-15T23:39:56.829 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631bcac3:::benchmark_data_smithi090_166897_object48875:head# 2024-05-15T23:39:56.829 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631c3b77:::benchmark_data_smithi090_166897_object39884:head# 2024-05-15T23:39:56.830 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631ddc55:::benchmark_data_smithi090_166897_object68419:head# 2024-05-15T23:39:56.831 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631dec7b:::benchmark_data_smithi090_166897_object21794:head# 2024-05-15T23:39:56.831 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631defc0:::benchmark_data_smithi090_166897_object59239:head# 2024-05-15T23:39:56.832 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631e5437:::benchmark_data_smithi090_166897_object8813:head# 2024-05-15T23:39:56.832 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631ebd85:::benchmark_data_smithi090_166897_object45651:head# 2024-05-15T23:39:56.834 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631f2883:::benchmark_data_smithi090_166897_object45067:head# 2024-05-15T23:39:56.835 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631fa7b1:::benchmark_data_smithi090_166897_object42035:head# 2024-05-15T23:39:56.835 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631fed18:::benchmark_data_smithi090_166897_object70650:head# 2024-05-15T23:39:56.836 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:631ff41b:::benchmark_data_smithi090_166897_object41405:head# 2024-05-15T23:39:56.836 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63208fe3:::benchmark_data_smithi090_166897_object2626:head# 2024-05-15T23:39:56.837 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6320e17a:::benchmark_data_smithi090_166897_object877:head# 2024-05-15T23:39:56.837 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63212502:::benchmark_data_smithi090_166897_object6575:head# 2024-05-15T23:39:56.838 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6321ca7b:::benchmark_data_smithi090_166897_object9558:head# 2024-05-15T23:39:56.838 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6321e583:::benchmark_data_smithi090_166897_object45576:head# 2024-05-15T23:39:56.839 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6323c680:::benchmark_data_smithi090_166897_object51798:head# 2024-05-15T23:39:56.839 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63245c4f:::benchmark_data_smithi090_166897_object58446:head# 2024-05-15T23:39:56.840 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6325f676:::benchmark_data_smithi090_166897_object29524:head# 2024-05-15T23:39:56.840 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:632659ba:::benchmark_data_smithi090_166897_object34387:head# 2024-05-15T23:39:56.841 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:632689d4:::benchmark_data_smithi090_166897_object57211:head# 2024-05-15T23:39:56.843 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63282431:::benchmark_data_smithi090_166897_object7533:head# 2024-05-15T23:39:56.844 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63290241:::benchmark_data_smithi090_166897_object29837:head# 2024-05-15T23:39:56.844 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:632b4c44:::benchmark_data_smithi090_166897_object34837:head# 2024-05-15T23:39:56.845 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:632b7827:::benchmark_data_smithi090_166897_object18132:head# 2024-05-15T23:39:56.845 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:632bbf8d:::benchmark_data_smithi090_166897_object64731:head# 2024-05-15T23:39:56.846 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:632c062b:::benchmark_data_smithi090_166897_object39054:head# 2024-05-15T23:39:56.847 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:632de2bf:::benchmark_data_smithi090_166897_object38200:head# 2024-05-15T23:39:56.847 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:632e672a:::benchmark_data_smithi090_166897_object45993:head# 2024-05-15T23:39:56.847 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:632ee713:::benchmark_data_smithi090_166897_object12829:head# 2024-05-15T23:39:56.848 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:632f68f0:::benchmark_data_smithi090_166897_object53484:head# 2024-05-15T23:39:56.848 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:632fed66:::benchmark_data_smithi090_166897_object15403:head# 2024-05-15T23:39:56.848 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6330859f:::benchmark_data_smithi090_166897_object52995:head# 2024-05-15T23:39:56.849 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6332a1c9:::benchmark_data_smithi090_166897_object56816:head# 2024-05-15T23:39:56.849 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63332286:::benchmark_data_smithi090_166897_object3635:head# 2024-05-15T23:39:56.849 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63338271:::benchmark_data_smithi090_166897_object61281:head# 2024-05-15T23:39:56.851 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63356f34:::benchmark_data_smithi090_166897_object35104:head# 2024-05-15T23:39:56.851 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6335ca85:::benchmark_data_smithi090_166897_object40917:head# 2024-05-15T23:39:56.851 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:633630ee:::benchmark_data_smithi090_166897_object38386:head# 2024-05-15T23:39:56.852 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63367421:::benchmark_data_smithi090_166897_object49091:head# 2024-05-15T23:39:56.853 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6337d7bc:::benchmark_data_smithi090_166897_object29814:head# 2024-05-15T23:39:56.853 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6339f4d4:::benchmark_data_smithi090_166897_object18299:head# 2024-05-15T23:39:56.854 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:633a4cd7:::benchmark_data_smithi090_166897_object37995:head# 2024-05-15T23:39:56.854 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:633b0cf4:::benchmark_data_smithi090_166897_object49439:head# 2024-05-15T23:39:56.855 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:633b30ea:::benchmark_data_smithi090_166897_object46440:head# 2024-05-15T23:39:56.855 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:633d5154:::benchmark_data_smithi090_166897_object37699:head# 2024-05-15T23:39:56.856 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:633d9048:::benchmark_data_smithi090_166897_object50359:head# 2024-05-15T23:39:56.856 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:633da023:::benchmark_data_smithi090_166897_object60511:head# 2024-05-15T23:39:56.857 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:633de340:::benchmark_data_smithi090_166897_object59465:head# 2024-05-15T23:39:56.857 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:633f5937:::benchmark_data_smithi090_166897_object31189:head# 2024-05-15T23:39:56.857 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:633fb536:::benchmark_data_smithi090_166897_object51441:head# 2024-05-15T23:39:56.858 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:633fbe1c:::benchmark_data_smithi090_166897_object9185:head# 2024-05-15T23:39:56.862 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:634007ff:::benchmark_data_smithi090_166897_object5700:head# 2024-05-15T23:39:56.863 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6340b54d:::benchmark_data_smithi090_166897_object44601:head# 2024-05-15T23:39:56.863 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6341f49a:::benchmark_data_smithi090_166897_object41408:head# 2024-05-15T23:39:56.863 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6342238f:::benchmark_data_smithi090_166897_object21851:head# 2024-05-15T23:39:56.863 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:634843be:::benchmark_data_smithi090_166897_object55245:head# 2024-05-15T23:39:56.866 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:634844a8:::benchmark_data_smithi090_166897_object14382:head# 2024-05-15T23:39:56.866 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63488b4c:::benchmark_data_smithi090_166897_object69930:head# 2024-05-15T23:39:56.867 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6349b5cf:::benchmark_data_smithi090_166897_object725:head# 2024-05-15T23:39:56.867 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:634aa6d1:::benchmark_data_smithi090_166897_object25127:head# 2024-05-15T23:39:56.868 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:634b1ec1:::benchmark_data_smithi090_166897_object55540:head# 2024-05-15T23:39:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:56 smithi084 ceph-mon[39784]: 9.7 scrub starts 2024-05-15T23:39:56.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:56 smithi084 ceph-mon[39784]: 9.7 scrub ok 2024-05-15T23:39:56.870 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:634e1c2f:::benchmark_data_smithi090_166897_object35946:head# 2024-05-15T23:39:56.870 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63520c8e:::benchmark_data_smithi090_166897_object7678:head# 2024-05-15T23:39:56.870 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63560216:::benchmark_data_smithi090_166897_object19496:head# 2024-05-15T23:39:56.870 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6356e8d0:::benchmark_data_smithi090_166897_object12612:head# 2024-05-15T23:39:56.871 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63575273:::benchmark_data_smithi090_166897_object25455:head# 2024-05-15T23:39:56.871 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63575825:::benchmark_data_smithi090_166897_object35013:head# 2024-05-15T23:39:56.871 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6357bad5:::benchmark_data_smithi090_166897_object22421:head# 2024-05-15T23:39:56.872 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6359a4bd:::benchmark_data_smithi090_166897_object54111:head# 2024-05-15T23:39:56.872 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:635a7354:::benchmark_data_smithi090_166897_object24989:head# 2024-05-15T23:39:56.873 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:635b7e20:::benchmark_data_smithi090_166897_object60014:head# 2024-05-15T23:39:56.873 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:635bb402:::benchmark_data_smithi090_166897_object62621:head# 2024-05-15T23:39:56.874 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:635e2239:::benchmark_data_smithi090_166897_object43857:head# 2024-05-15T23:39:56.878 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:635eb67e:::benchmark_data_smithi090_166897_object31544:head# 2024-05-15T23:39:56.878 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:635f1c32:::benchmark_data_smithi090_166897_object31233:head# 2024-05-15T23:39:56.878 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:635f4ab5:::benchmark_data_smithi090_166897_object55509:head# 2024-05-15T23:39:56.878 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:636020b2:::benchmark_data_smithi090_166897_object12018:head# 2024-05-15T23:39:56.879 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6360db69:::benchmark_data_smithi090_166897_object69161:head# 2024-05-15T23:39:56.880 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63626b06:::benchmark_data_smithi090_166897_object28385:head# 2024-05-15T23:39:56.880 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6362addb:::benchmark_data_smithi090_166897_object44886:head# 2024-05-15T23:39:56.880 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:636439d7:::benchmark_data_smithi090_166897_object13738:head# 2024-05-15T23:39:56.881 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6364f0a9:::benchmark_data_smithi090_166897_object2353:head# 2024-05-15T23:39:56.881 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63664822:::benchmark_data_smithi090_166897_object18451:head# 2024-05-15T23:39:56.881 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6366f6ab:::benchmark_data_smithi090_166897_object49320:head# 2024-05-15T23:39:56.881 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6367854f:::benchmark_data_smithi090_166897_object41192:head# 2024-05-15T23:39:56.882 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63699a3e:::benchmark_data_smithi090_166897_object6289:head# 2024-05-15T23:39:56.882 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:636b1585:::benchmark_data_smithi090_166897_object8858:head# 2024-05-15T23:39:56.883 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:636bf47e:::benchmark_data_smithi090_166897_object24114:head# 2024-05-15T23:39:56.883 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:636c084c:::benchmark_data_smithi090_166897_object65463:head# 2024-05-15T23:39:56.886 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:636c4cc6:::benchmark_data_smithi090_166897_object22031:head# 2024-05-15T23:39:56.886 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63704495:::benchmark_data_smithi090_166897_object58784:head# 2024-05-15T23:39:56.887 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:637170f1:::benchmark_data_smithi090_166897_object16789:head# 2024-05-15T23:39:56.887 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6371e83e:::benchmark_data_smithi090_166897_object29644:head# 2024-05-15T23:39:56.888 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6372177b:::benchmark_data_smithi090_166897_object63193:head# 2024-05-15T23:39:56.888 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6372cef8:::benchmark_data_smithi090_166897_object51576:head# 2024-05-15T23:39:56.889 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6372d8e8:::benchmark_data_smithi090_166897_object27521:head# 2024-05-15T23:39:56.889 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:637368e6:::benchmark_data_smithi090_166897_object55711:head# 2024-05-15T23:39:56.890 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63738a7c:::benchmark_data_smithi090_166897_object5121:head# 2024-05-15T23:39:56.890 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6376dd5e:::benchmark_data_smithi090_166897_object25391:head# 2024-05-15T23:39:56.890 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6378c762:::benchmark_data_smithi090_166897_object43035:head# 2024-05-15T23:39:56.891 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6379dc50:::benchmark_data_smithi090_166897_object39075:head# 2024-05-15T23:39:56.891 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:637c0741:::benchmark_data_smithi090_166897_object51428:head# 2024-05-15T23:39:56.893 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:637c498e:::benchmark_data_smithi090_166897_object1889:head# 2024-05-15T23:39:56.893 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:637df01d:::benchmark_data_smithi090_166897_object2841:head# 2024-05-15T23:39:56.894 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6381700a:::benchmark_data_smithi090_166897_object3800:head# 2024-05-15T23:39:56.895 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63835146:::benchmark_data_smithi090_166897_object15:head# 2024-05-15T23:39:56.895 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6385428f:::benchmark_data_smithi090_166897_object47695:head# 2024-05-15T23:39:56.896 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63867c68:::benchmark_data_smithi090_166897_object51656:head# 2024-05-15T23:39:56.896 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6386a3d9:::benchmark_data_smithi090_166897_object45445:head# 2024-05-15T23:39:56.898 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63875d49:::benchmark_data_smithi090_166897_object11360:head# 2024-05-15T23:39:56.899 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:638941f5:::benchmark_data_smithi090_166897_object5523:head# 2024-05-15T23:39:56.899 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:638ab0aa:::benchmark_data_smithi090_166897_object43037:head# 2024-05-15T23:39:56.900 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:638b0ccd:::benchmark_data_smithi090_166897_object9909:head# 2024-05-15T23:39:56.900 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:638b1611:::benchmark_data_smithi090_166897_object59044:head# 2024-05-15T23:39:56.900 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:638bbe5d:::benchmark_data_smithi090_166897_object676:head# 2024-05-15T23:39:56.901 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:638d4d48:::benchmark_data_smithi090_166897_object33193:head# 2024-05-15T23:39:56.901 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:638d54d9:::benchmark_data_smithi090_166897_object40216:head# 2024-05-15T23:39:56.902 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63902368:::benchmark_data_smithi090_166897_object33501:head# 2024-05-15T23:39:56.902 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6391832b:::benchmark_data_smithi090_166897_object49961:head# 2024-05-15T23:39:56.903 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63932339:::benchmark_data_smithi090_166897_object27606:head# 2024-05-15T23:39:56.903 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63935979:::benchmark_data_smithi090_166897_object36472:head# 2024-05-15T23:39:56.904 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6393a2f2:::benchmark_data_smithi090_166897_object16098:head# 2024-05-15T23:39:56.904 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63949937:::benchmark_data_smithi090_166897_object27018:head# 2024-05-15T23:39:56.904 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6395978e:::benchmark_data_smithi090_166897_object10493:head# 2024-05-15T23:39:56.905 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63970c7c:::benchmark_data_smithi090_166897_object10852:head# 2024-05-15T23:39:56.905 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6397a030:::benchmark_data_smithi090_166897_object39433:head# 2024-05-15T23:39:56.906 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63996223:::benchmark_data_smithi090_166897_object35994:head# 2024-05-15T23:39:56.906 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63996e24:::benchmark_data_smithi090_166897_object47796:head# 2024-05-15T23:39:56.907 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6399b152:::benchmark_data_smithi090_166897_object59303:head# 2024-05-15T23:39:56.907 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6399b1d5:::benchmark_data_smithi090_166897_object27543:head# 2024-05-15T23:39:56.907 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:639aaca9:::benchmark_data_smithi090_166897_object16295:head# 2024-05-15T23:39:56.908 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:639ad1aa:::benchmark_data_smithi090_166897_object25462:head# 2024-05-15T23:39:56.909 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:639c1efa:::benchmark_data_smithi090_166897_object13624:head# 2024-05-15T23:39:56.909 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:639cdbf2:::benchmark_data_smithi090_166897_object59510:head# 2024-05-15T23:39:56.909 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63a1e757:::benchmark_data_smithi090_166897_object33531:head# 2024-05-15T23:39:56.910 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63a2b776:::benchmark_data_smithi090_166897_object67136:head# 2024-05-15T23:39:56.910 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63a3733e:::benchmark_data_smithi090_166897_object10785:head# 2024-05-15T23:39:56.911 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63a45322:::benchmark_data_smithi090_166897_object55757:head# 2024-05-15T23:39:56.911 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63a71878:::benchmark_data_smithi090_166897_object59270:head# 2024-05-15T23:39:56.912 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63ab1c50:::benchmark_data_smithi090_166897_object37826:head# 2024-05-15T23:39:56.912 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63ab7a50:::benchmark_data_smithi090_166897_object39337:head# 2024-05-15T23:39:56.915 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63ac39a4:::benchmark_data_smithi090_166897_object26954:head# 2024-05-15T23:39:56.915 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63acebb7:::benchmark_data_smithi090_166897_object22321:head# 2024-05-15T23:39:56.916 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63ada610:::benchmark_data_smithi090_166897_object34423:head# 2024-05-15T23:39:56.916 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63ae00bd:::benchmark_data_smithi090_166897_object58686:head# 2024-05-15T23:39:56.916 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63af20c5:::benchmark_data_smithi090_166897_object17076:head# 2024-05-15T23:39:56.917 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63afef5a:::benchmark_data_smithi090_166897_object32467:head# 2024-05-15T23:39:56.917 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63b0125d:::benchmark_data_smithi090_166897_object14502:head# 2024-05-15T23:39:56.918 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63b193b8:::benchmark_data_smithi090_166897_object54473:head# 2024-05-15T23:39:56.918 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63b22f1f:::benchmark_data_smithi090_166897_object18063:head# 2024-05-15T23:39:56.919 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63b2941a:::benchmark_data_smithi090_166897_object31502:head# 2024-05-15T23:39:56.919 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63b2d861:::benchmark_data_smithi090_166897_object26514:head# 2024-05-15T23:39:56.920 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63b37ed2:::benchmark_data_smithi090_166897_object56336:head# 2024-05-15T23:39:56.920 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63b455b7:::benchmark_data_smithi090_166897_object53445:head# 2024-05-15T23:39:56.920 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63b47313:::benchmark_data_smithi090_166897_object4411:head# 2024-05-15T23:39:56.921 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63b593ba:::benchmark_data_smithi090_166897_object27304:head# 2024-05-15T23:39:56.921 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63b67a4a:::benchmark_data_smithi090_166897_object55155:head# 2024-05-15T23:39:56.922 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63b7f740:::benchmark_data_smithi090_166897_object51410:head# 2024-05-15T23:39:56.922 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63b97d08:::benchmark_data_smithi090_166897_object37620:head# 2024-05-15T23:39:56.922 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63b9d69b:::benchmark_data_smithi090_166897_object34295:head# 2024-05-15T23:39:56.923 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63bb2b69:::benchmark_data_smithi090_166897_object4092:head# 2024-05-15T23:39:56.924 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63bb4af5:::benchmark_data_smithi090_166897_object40559:head# 2024-05-15T23:39:56.924 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63bb738a:::benchmark_data_smithi090_166897_object6215:head# 2024-05-15T23:39:56.925 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63bc21d1:::benchmark_data_smithi090_166897_object59196:head# 2024-05-15T23:39:56.925 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63bcd852:::benchmark_data_smithi090_166897_object55551:head# 2024-05-15T23:39:56.926 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63bef727:::benchmark_data_smithi090_166897_object5693:head# 2024-05-15T23:39:56.927 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63bf3ec6:::benchmark_data_smithi090_166897_object7398:head# 2024-05-15T23:39:56.927 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63c0608d:::benchmark_data_smithi090_166897_object5550:head# 2024-05-15T23:39:56.928 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63c0db89:::benchmark_data_smithi090_166897_object55603:head# 2024-05-15T23:39:56.928 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63c18e2b:::benchmark_data_smithi090_166897_object18806:head# 2024-05-15T23:39:56.929 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63c1c4e6:::benchmark_data_smithi090_166897_object52638:head# 2024-05-15T23:39:56.929 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63c283b2:::benchmark_data_smithi090_166897_object40630:head# 2024-05-15T23:39:56.929 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63c4b039:::benchmark_data_smithi090_166897_object21191:head# 2024-05-15T23:39:56.930 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63c51acd:::benchmark_data_smithi090_166897_object37997:head# 2024-05-15T23:39:56.930 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63c5b99e:::benchmark_data_smithi090_166897_object29770:head# 2024-05-15T23:39:56.931 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63ca3527:::benchmark_data_smithi090_166897_object44978:head# 2024-05-15T23:39:56.931 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63ca6618:::benchmark_data_smithi090_166897_object52337:head# 2024-05-15T23:39:56.932 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63caf2e3:::benchmark_data_smithi090_166897_object30502:head# 2024-05-15T23:39:56.932 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63cb00cb:::benchmark_data_smithi090_166897_object48076:head# 2024-05-15T23:39:56.932 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63cb78f1:::benchmark_data_smithi090_166897_object36151:head# 2024-05-15T23:39:56.933 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63cd89de:::benchmark_data_smithi090_166897_object67781:head# 2024-05-15T23:39:56.933 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63cdb2a2:::benchmark_data_smithi090_166897_object16950:head# 2024-05-15T23:39:56.934 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63ce48c9:::benchmark_data_smithi090_166897_object33220:head# 2024-05-15T23:39:56.934 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63ce9312:::benchmark_data_smithi090_166897_object6360:head# 2024-05-15T23:39:56.935 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63ceefaf:::benchmark_data_smithi090_166897_object50150:head# 2024-05-15T23:39:56.935 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63d04e8a:::benchmark_data_smithi090_166897_object55974:head# 2024-05-15T23:39:56.936 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63d07723:::benchmark_data_smithi090_166897_object45957:head# 2024-05-15T23:39:56.936 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63d0a371:::benchmark_data_smithi090_166897_object67717:head# 2024-05-15T23:39:56.937 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63d14582:::benchmark_data_smithi090_166897_object29791:head# 2024-05-15T23:39:56.937 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63d18984:::benchmark_data_smithi090_166897_object45547:head# 2024-05-15T23:39:56.938 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63d33192:::benchmark_data_smithi090_166897_object3038:head# 2024-05-15T23:39:56.938 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63d3701f:::benchmark_data_smithi090_166897_object22098:head# 2024-05-15T23:39:56.939 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63d3aad4:::benchmark_data_smithi090_166897_object58024:head# 2024-05-15T23:39:56.939 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63d4dc12:::benchmark_data_smithi090_166897_object57046:head# 2024-05-15T23:39:56.939 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63d52d16:::benchmark_data_smithi090_166897_object11107:head# 2024-05-15T23:39:56.940 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63d6395b:::benchmark_data_smithi090_166897_object40079:head# 2024-05-15T23:39:56.941 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63d879ee:::benchmark_data_smithi090_166897_object53210:head# 2024-05-15T23:39:56.941 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63d88a69:::benchmark_data_smithi090_166897_object10505:head# 2024-05-15T23:39:56.942 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63d9ca33:::benchmark_data_smithi090_166897_object2996:head# 2024-05-15T23:39:56.942 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63d9dc3e:::benchmark_data_smithi090_166897_object26875:head# 2024-05-15T23:39:56.943 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63dc5baa:::benchmark_data_smithi090_166897_object58614:head# 2024-05-15T23:39:56.944 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63de1ad9:::benchmark_data_smithi090_166897_object49522:head# 2024-05-15T23:39:56.944 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63df0b36:::benchmark_data_smithi090_166897_object59055:head# 2024-05-15T23:39:56.944 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63dfdd8d:::benchmark_data_smithi090_166897_object1096:head# 2024-05-15T23:39:56.945 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63e008cb:::benchmark_data_smithi090_166897_object53093:head# 2024-05-15T23:39:56.945 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63e05f28:::benchmark_data_smithi090_166897_object53654:head# 2024-05-15T23:39:56.945 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63e19b70:::benchmark_data_smithi090_166897_object1926:head# 2024-05-15T23:39:56.946 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63e25d7a:::benchmark_data_smithi090_166897_object68878:head# 2024-05-15T23:39:56.946 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63e294f8:::benchmark_data_smithi090_166897_object15601:head# 2024-05-15T23:39:56.947 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63e3af5c:::benchmark_data_smithi090_166897_object4838:head# 2024-05-15T23:39:56.947 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63e4a2f1:::benchmark_data_smithi090_166897_object66854:head# 2024-05-15T23:39:56.948 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63e6995e:::benchmark_data_smithi090_166897_object69347:head# 2024-05-15T23:39:56.948 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63e6ffd0:::benchmark_data_smithi090_166897_object64464:head# 2024-05-15T23:39:56.948 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63ea23f0:::benchmark_data_smithi090_166897_object54633:head# 2024-05-15T23:39:56.949 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63ebc0c0:::benchmark_data_smithi090_166897_object53963:head# 2024-05-15T23:39:56.949 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63ef5816:::benchmark_data_smithi090_166897_object61083:head# 2024-05-15T23:39:56.950 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63efd570:::benchmark_data_smithi090_166897_object17916:head# 2024-05-15T23:39:56.950 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63f02f78:::benchmark_data_smithi090_166897_object22570:head# 2024-05-15T23:39:56.951 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63f0a532:::benchmark_data_smithi090_166897_object35128:head# 2024-05-15T23:39:56.951 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63f0f552:::benchmark_data_smithi090_166897_object13961:head# 2024-05-15T23:39:56.951 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63f1392f:::benchmark_data_smithi090_166897_object40096:head# 2024-05-15T23:39:56.953 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63f1465e:::benchmark_data_smithi090_166897_object18539:head# 2024-05-15T23:39:56.953 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63f1c1e2:::benchmark_data_smithi090_166897_object69439:head# 2024-05-15T23:39:56.953 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63f2b62f:::benchmark_data_smithi090_166897_object19201:head# 2024-05-15T23:39:56.955 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63f2b74c:::benchmark_data_smithi090_166897_object42875:head# 2024-05-15T23:39:56.955 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63f437a2:::benchmark_data_smithi090_166897_object5386:head# 2024-05-15T23:39:56.955 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63f440cb:::benchmark_data_smithi090_166897_object30563:head# 2024-05-15T23:39:56.955 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63f5580e:::benchmark_data_smithi090_166897_object19538:head# 2024-05-15T23:39:56.955 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63f6baef:::benchmark_data_smithi090_166897_object61780:head# 2024-05-15T23:39:56.955 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63f6d10e:::benchmark_data_smithi090_166897_object66961:head# 2024-05-15T23:39:56.956 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63f8eb13:::benchmark_data_smithi090_166897_object64262:head# 2024-05-15T23:39:56.956 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63fd8a90:::benchmark_data_smithi090_166897_object59958:head# 2024-05-15T23:39:56.957 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63fdf5e9:::benchmark_data_smithi090_166897_object43715:head# 2024-05-15T23:39:56.957 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:63fe75ee:::benchmark_data_smithi090_166897_object50581:head# 2024-05-15T23:39:56.958 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64026ed7:::benchmark_data_smithi090_166897_object13963:head# 2024-05-15T23:39:56.958 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:640388ea:::benchmark_data_smithi090_166897_object45788:head# 2024-05-15T23:39:56.959 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6404d5fc:::benchmark_data_smithi090_166897_object32327:head# 2024-05-15T23:39:56.959 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64062d08:::benchmark_data_smithi090_166897_object44846:head# 2024-05-15T23:39:56.960 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6406bda5:::benchmark_data_smithi090_166897_object19834:head# 2024-05-15T23:39:56.960 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6407153a:::benchmark_data_smithi090_166897_object67903:head# 2024-05-15T23:39:56.961 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6407962d:::benchmark_data_smithi090_166897_object66316:head# 2024-05-15T23:39:56.961 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6407ca25:::benchmark_data_smithi090_166897_object59032:head# 2024-05-15T23:39:56.962 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64085541:::benchmark_data_smithi090_166897_object38174:head# 2024-05-15T23:39:56.962 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:640deb9e:::benchmark_data_smithi090_166897_object41197:head# 2024-05-15T23:39:56.962 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:640fac1d:::benchmark_data_smithi090_166897_object65189:head# 2024-05-15T23:39:56.963 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6410564e:::benchmark_data_smithi090_166897_object19901:head# 2024-05-15T23:39:56.963 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6412e03e:::benchmark_data_smithi090_166897_object17318:head# 2024-05-15T23:39:56.963 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64130c63:::benchmark_data_smithi090_166897_object46537:head# 2024-05-15T23:39:56.964 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64133228:::benchmark_data_smithi090_166897_object15917:head# 2024-05-15T23:39:56.964 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64145af9:::benchmark_data_smithi090_166897_object40192:head# 2024-05-15T23:39:56.966 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6414add8:::benchmark_data_smithi090_166897_object31673:head# 2024-05-15T23:39:56.966 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64160372:::benchmark_data_smithi090_166897_object62026:head# 2024-05-15T23:39:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:641682ce:::benchmark_data_smithi090_166897_object43368:head# 2024-05-15T23:39:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6416b347:::benchmark_data_smithi090_166897_object35254:head# 2024-05-15T23:39:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64174cac:::benchmark_data_smithi090_166897_object53474:head# 2024-05-15T23:39:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6417c941:::benchmark_data_smithi090_166897_object37594:head# 2024-05-15T23:39:56.969 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:641b2f82:::benchmark_data_smithi090_166897_object21982:head# 2024-05-15T23:39:56.969 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:641ba343:::benchmark_data_smithi090_166897_object32986:head# 2024-05-15T23:39:56.970 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:641bb5fb:::benchmark_data_smithi090_166897_object47798:head# 2024-05-15T23:39:56.970 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:641c34f1:::benchmark_data_smithi090_166897_object41220:head# 2024-05-15T23:39:56.972 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:641c56e2:::benchmark_data_smithi090_166897_object39494:head# 2024-05-15T23:39:56.973 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:641ca232:::benchmark_data_smithi090_166897_object2940:head# 2024-05-15T23:39:56.973 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:641d1b6b:::benchmark_data_smithi090_166897_object28683:head# 2024-05-15T23:39:56.974 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:641ee0a3:::benchmark_data_smithi090_166897_object19662:head# 2024-05-15T23:39:56.975 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:641f40bb:::benchmark_data_smithi090_166897_object27015:head# 2024-05-15T23:39:56.975 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:641f6abd:::benchmark_data_smithi090_166897_object55644:head# 2024-05-15T23:39:56.975 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:642157ea:::benchmark_data_smithi090_166897_object7350:head# 2024-05-15T23:39:56.976 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64215fd1:::benchmark_data_smithi090_166897_object10084:head# 2024-05-15T23:39:56.976 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6421c6c9:::benchmark_data_smithi090_166897_object10847:head# 2024-05-15T23:39:56.977 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6421f663:::benchmark_data_smithi090_166897_object33438:head# 2024-05-15T23:39:56.977 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64225ac0:::benchmark_data_smithi090_166897_object4122:head# 2024-05-15T23:39:56.977 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64234076:::benchmark_data_smithi090_166897_object4255:head# 2024-05-15T23:39:56.978 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64234640:::benchmark_data_smithi090_166897_object63832:head# 2024-05-15T23:39:56.978 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64255162:::benchmark_data_smithi090_166897_object15309:head# 2024-05-15T23:39:56.979 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6426de6a:::benchmark_data_smithi090_166897_object47523:head# 2024-05-15T23:39:56.980 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6426fa48:::benchmark_data_smithi090_166897_object22724:head# 2024-05-15T23:39:56.981 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6427fbea:::benchmark_data_smithi090_166897_object1537:head# 2024-05-15T23:39:56.981 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6429ca9a:::benchmark_data_smithi090_166897_object35114:head# 2024-05-15T23:39:56.981 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6429f8d4:::benchmark_data_smithi090_166897_object31813:head# 2024-05-15T23:39:56.981 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:642ca5c4:::benchmark_data_smithi090_166897_object17155:head# 2024-05-15T23:39:56.981 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:642cbcdf:::benchmark_data_smithi090_166897_object49294:head# 2024-05-15T23:39:56.982 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:642d200b:::benchmark_data_smithi090_166897_object25548:head# 2024-05-15T23:39:56.982 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6430f793:::benchmark_data_smithi090_166897_object32238:head# 2024-05-15T23:39:56.983 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6431159a:::benchmark_data_smithi090_166897_object7250:head# 2024-05-15T23:39:56.984 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:643174e3:::benchmark_data_smithi090_166897_object42662:head# 2024-05-15T23:39:56.984 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6433518f:::benchmark_data_smithi090_166897_object36325:head# 2024-05-15T23:39:56.985 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6433babc:::benchmark_data_smithi090_166897_object54901:head# 2024-05-15T23:39:56.985 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64354abb:::benchmark_data_smithi090_166897_object66664:head# 2024-05-15T23:39:56.985 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64354d1b:::benchmark_data_smithi090_166897_object29188:head# 2024-05-15T23:39:56.986 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6435f2b0:::benchmark_data_smithi090_166897_object3816:head# 2024-05-15T23:39:56.986 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6437deef:::benchmark_data_smithi090_166897_object6298:head# 2024-05-15T23:39:56.987 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6437e620:::benchmark_data_smithi090_166897_object45658:head# 2024-05-15T23:39:56.987 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:643830ae:::benchmark_data_smithi090_166897_object42949:head# 2024-05-15T23:39:56.988 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6438e0b9:::benchmark_data_smithi090_166897_object6435:head# 2024-05-15T23:39:56.991 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:643a2361:::benchmark_data_smithi090_166897_object7241:head# 2024-05-15T23:39:56.991 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:643af633:::benchmark_data_smithi090_166897_object69477:head# 2024-05-15T23:39:56.992 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:643d5d09:::benchmark_data_smithi090_166897_object14013:head# 2024-05-15T23:39:56.992 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:643ecbc4:::benchmark_data_smithi090_166897_object14515:head# 2024-05-15T23:39:56.993 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:643eed68:::benchmark_data_smithi090_166897_object58443:head# 2024-05-15T23:39:56.993 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:643f6102:::benchmark_data_smithi090_166897_object59841:head# 2024-05-15T23:39:56.994 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6440208e:::benchmark_data_smithi090_166897_object69047:head# 2024-05-15T23:39:56.994 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64423419:::benchmark_data_smithi090_166897_object23373:head# 2024-05-15T23:39:56.995 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64427d11:::benchmark_data_smithi090_166897_object43912:head# 2024-05-15T23:39:56.995 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:644299c5:::benchmark_data_smithi090_166897_object124:head# 2024-05-15T23:39:56.996 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:644621d4:::benchmark_data_smithi090_166897_object39117:head# 2024-05-15T23:39:56.996 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64464ef9:::benchmark_data_smithi090_166897_object64814:head# 2024-05-15T23:39:56.997 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6446eb80:::benchmark_data_smithi090_166897_object62264:head# 2024-05-15T23:39:56.997 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:644793bb:::benchmark_data_smithi090_166897_object46282:head# 2024-05-15T23:39:56.999 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6448cb75:::benchmark_data_smithi090_166897_object48763:head# 2024-05-15T23:39:57.000 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6449ef0d:::benchmark_data_smithi090_166897_object18579:head# 2024-05-15T23:39:57.000 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:644a7160:::benchmark_data_smithi090_166897_object55016:head# 2024-05-15T23:39:57.001 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:644a72d0:::benchmark_data_smithi090_166897_object48566:head# 2024-05-15T23:39:57.001 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:644ad655:::benchmark_data_smithi090_166897_object56425:head# 2024-05-15T23:39:57.002 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:644c466b:::benchmark_data_smithi090_166897_object17402:head# 2024-05-15T23:39:57.002 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:644c80ef:::benchmark_data_smithi090_166897_object30867:head# 2024-05-15T23:39:57.003 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:644c94e0:::benchmark_data_smithi090_166897_object48736:head# 2024-05-15T23:39:57.003 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:644d5063:::benchmark_data_smithi090_166897_object34119:head# 2024-05-15T23:39:57.003 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:644e4d94:::benchmark_data_smithi090_166897_object34819:head# 2024-05-15T23:39:57.004 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:644e8b27:::benchmark_data_smithi090_166897_object16512:head# 2024-05-15T23:39:57.007 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:644eab3c:::benchmark_data_smithi090_166897_object51630:head# 2024-05-15T23:39:57.007 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:645032cf:::benchmark_data_smithi090_166897_object11206:head# 2024-05-15T23:39:57.008 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6453c646:::benchmark_data_smithi090_166897_object54818:head# 2024-05-15T23:39:57.008 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64541911:::benchmark_data_smithi090_166897_object42023:head# 2024-05-15T23:39:57.009 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:645456fa:::benchmark_data_smithi090_166897_object28641:head# 2024-05-15T23:39:57.009 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6455ce3d:::benchmark_data_smithi090_166897_object66598:head# 2024-05-15T23:39:57.009 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64571387:::benchmark_data_smithi090_166897_object39339:head# 2024-05-15T23:39:57.011 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64572981:::benchmark_data_smithi090_166897_object37162:head# 2024-05-15T23:39:57.012 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6457b8d5:::benchmark_data_smithi090_166897_object30988:head# 2024-05-15T23:39:57.012 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:645842f7:::benchmark_data_smithi090_166897_object59426:head# 2024-05-15T23:39:57.015 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6458a7fa:::benchmark_data_smithi090_166897_object25527:head# 2024-05-15T23:39:57.015 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:645914fe:::benchmark_data_smithi090_166897_object20051:head# 2024-05-15T23:39:57.016 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64595464:::benchmark_data_smithi090_166897_object17143:head# 2024-05-15T23:39:57.017 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:645a4c1e:::benchmark_data_smithi090_166897_object48395:head# 2024-05-15T23:39:57.017 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:645a6a0d:::benchmark_data_smithi090_166897_object17858:head# 2024-05-15T23:39:57.018 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:645c763e:::benchmark_data_smithi090_166897_object58350:head# 2024-05-15T23:39:57.018 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:645d42ea:::benchmark_data_smithi090_166897_object30856:head# 2024-05-15T23:39:57.019 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:645e1e83:::benchmark_data_smithi090_166897_object64185:head# 2024-05-15T23:39:57.019 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:645e5c7e:::benchmark_data_smithi090_166897_object18958:head# 2024-05-15T23:39:57.020 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:645f57ba:::benchmark_data_smithi090_166897_object9715:head# 2024-05-15T23:39:57.021 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64613c1a:::benchmark_data_smithi090_166897_object39526:head# 2024-05-15T23:39:57.021 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64619379:::benchmark_data_smithi090_166897_object40040:head# 2024-05-15T23:39:57.022 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6461b376:::benchmark_data_smithi090_166897_object47521:head# 2024-05-15T23:39:57.023 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6461fd61:::benchmark_data_smithi090_166897_object49467:head# 2024-05-15T23:39:57.023 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6463b5f0:::benchmark_data_smithi090_166897_object23190:head# 2024-05-15T23:39:57.025 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64643fee:::benchmark_data_smithi090_166897_object45336:head# 2024-05-15T23:39:57.025 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64646365:::benchmark_data_smithi090_166897_object5377:head# 2024-05-15T23:39:57.026 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64656240:::benchmark_data_smithi090_166897_object63777:head# 2024-05-15T23:39:57.027 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6467677b:::benchmark_data_smithi090_166897_object21921:head# 2024-05-15T23:39:57.028 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64686629:::benchmark_data_smithi090_166897_object42720:head# 2024-05-15T23:39:57.028 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:646aa77e:::benchmark_data_smithi090_166897_object32605:head# 2024-05-15T23:39:57.029 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:646b3c85:::benchmark_data_smithi090_166897_object47164:head# 2024-05-15T23:39:57.030 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:646b8b7c:::benchmark_data_smithi090_166897_object12677:head# 2024-05-15T23:39:57.030 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:646be35d:::benchmark_data_smithi090_166897_object60489:head# 2024-05-15T23:39:57.030 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:646c46ce:::benchmark_data_smithi090_166897_object23021:head# 2024-05-15T23:39:57.031 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:646c8b81:::benchmark_data_smithi090_166897_object60425:head# 2024-05-15T23:39:57.031 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:646c9231:::benchmark_data_smithi090_166897_object25829:head# 2024-05-15T23:39:57.032 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:646dce77:::benchmark_data_smithi090_166897_object42812:head# 2024-05-15T23:39:57.034 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:646e6e1d:::benchmark_data_smithi090_166897_object5798:head# 2024-05-15T23:39:57.034 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:646e80e3:::benchmark_data_smithi090_166897_object14290:head# 2024-05-15T23:39:57.035 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:646ee326:::benchmark_data_smithi090_166897_object2511:head# 2024-05-15T23:39:57.035 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:646f4c50:::benchmark_data_smithi090_166897_object38079:head# 2024-05-15T23:39:57.035 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6470a4c9:::benchmark_data_smithi090_166897_object45710:head# 2024-05-15T23:39:57.036 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6470a7b2:::benchmark_data_smithi090_166897_object6811:head# 2024-05-15T23:39:57.036 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6471ae12:::benchmark_data_smithi090_166897_object55686:head# 2024-05-15T23:39:57.037 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6471c867:::benchmark_data_smithi090_166897_object61163:head# 2024-05-15T23:39:57.039 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64727b95:::benchmark_data_smithi090_166897_object66597:head# 2024-05-15T23:39:57.039 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64737f36:::benchmark_data_smithi090_166897_object51070:head# 2024-05-15T23:39:57.040 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:647472a5:::benchmark_data_smithi090_166897_object34865:head# 2024-05-15T23:39:57.040 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6474af77:::benchmark_data_smithi090_166897_object2660:head# 2024-05-15T23:39:57.041 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64763d74:::benchmark_data_smithi090_166897_object32596:head# 2024-05-15T23:39:57.043 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:647674dd:::benchmark_data_smithi090_166897_object7699:head# 2024-05-15T23:39:57.044 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6476a7ad:::benchmark_data_smithi090_166897_object32020:head# 2024-05-15T23:39:57.044 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6476dd52:::benchmark_data_smithi090_166897_object5082:head# 2024-05-15T23:39:57.047 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6477021f:::benchmark_data_smithi090_166897_object8316:head# 2024-05-15T23:39:57.047 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64777d3a:::benchmark_data_smithi090_166897_object51241:head# 2024-05-15T23:39:57.048 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64786a87:::benchmark_data_smithi090_166897_object13329:head# 2024-05-15T23:39:57.049 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64799338:::benchmark_data_smithi090_166897_object1369:head# 2024-05-15T23:39:57.051 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64799c1f:::benchmark_data_smithi090_166897_object38302:head# 2024-05-15T23:39:57.051 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6479d625:::benchmark_data_smithi090_166897_object45389:head# 2024-05-15T23:39:57.052 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:647b154b:::benchmark_data_smithi090_166897_object43648:head# 2024-05-15T23:39:57.054 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:647e0e23:::benchmark_data_smithi090_166897_object5916:head# 2024-05-15T23:39:57.054 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:647eb90c:::benchmark_data_smithi090_166897_object19594:head# 2024-05-15T23:39:57.055 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64805e5c:::benchmark_data_smithi090_166897_object3186:head# 2024-05-15T23:39:57.055 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6482134e:::benchmark_data_smithi090_166897_object57508:head# 2024-05-15T23:39:57.055 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64826cd3:::benchmark_data_smithi090_166897_object35243:head# 2024-05-15T23:39:57.056 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:648436a3:::benchmark_data_smithi090_166897_object10845:head# 2024-05-15T23:39:57.056 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6484a38e:::benchmark_data_smithi090_166897_object33070:head# 2024-05-15T23:39:57.057 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6486ace8:::benchmark_data_smithi090_166897_object7625:head# 2024-05-15T23:39:57.057 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6487d8ea:::benchmark_data_smithi090_166897_object27706:head# 2024-05-15T23:39:57.057 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64881d81:::benchmark_data_smithi090_166897_object21153:head# 2024-05-15T23:39:57.058 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64892f6e:::benchmark_data_smithi090_166897_object31052:head# 2024-05-15T23:39:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:648958ed:::benchmark_data_smithi090_166897_object39136:head# 2024-05-15T23:39:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6489b453:::benchmark_data_smithi090_166897_object50217:head# 2024-05-15T23:39:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:648aabfc:::benchmark_data_smithi090_166897_object28121:head# 2024-05-15T23:39:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:648abb28:::benchmark_data_smithi090_166897_object17599:head# 2024-05-15T23:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:648b0b2e:::benchmark_data_smithi090_166897_object17926:head# 2024-05-15T23:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:648b83f7:::benchmark_data_smithi090_166897_object3967:head# 2024-05-15T23:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:648c20e4:::benchmark_data_smithi090_166897_object57780:head# 2024-05-15T23:39:57.063 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:648cacee:::benchmark_data_smithi090_166897_object67543:head# 2024-05-15T23:39:57.067 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:648d7dff:::benchmark_data_smithi090_166897_object40037:head# 2024-05-15T23:39:57.068 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:648d8d8c:::benchmark_data_smithi090_166897_object31744:head# 2024-05-15T23:39:57.068 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:648df259:::benchmark_data_smithi090_166897_object12320:head# 2024-05-15T23:39:57.068 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:648df361:::benchmark_data_smithi090_166897_object505:head# 2024-05-15T23:39:57.068 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:648ea54a:::benchmark_data_smithi090_166897_object42717:head# 2024-05-15T23:39:57.068 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:648f2e2a:::benchmark_data_smithi090_166897_object7686:head# 2024-05-15T23:39:57.068 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64900c68:::benchmark_data_smithi090_166897_object9198:head# 2024-05-15T23:39:57.068 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6490493d:::benchmark_data_smithi090_166897_object66370:head# 2024-05-15T23:39:57.068 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64914d51:::benchmark_data_smithi090_166897_object43722:head# 2024-05-15T23:39:57.069 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6492ca2a:::benchmark_data_smithi090_166897_object19274:head# 2024-05-15T23:39:57.069 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6493ed51:::benchmark_data_smithi090_166897_object5101:head# 2024-05-15T23:39:57.069 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64947738:::benchmark_data_smithi090_166897_object28091:head# 2024-05-15T23:39:57.070 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6494e67c:::benchmark_data_smithi090_166897_object27162:head# 2024-05-15T23:39:57.070 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6496235a:::benchmark_data_smithi090_166897_object63614:head# 2024-05-15T23:39:57.071 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64962806:::benchmark_data_smithi090_166897_object61484:head# 2024-05-15T23:39:57.071 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64975c88:::benchmark_data_smithi090_166897_object36589:head# 2024-05-15T23:39:57.072 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6499567a:::benchmark_data_smithi090_166897_object33910:head# 2024-05-15T23:39:57.072 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:649a6a3b:::benchmark_data_smithi090_166897_object24598:head# 2024-05-15T23:39:57.073 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:649c6fe0:::benchmark_data_smithi090_166897_object18767:head# 2024-05-15T23:39:57.074 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:649dd75b:::benchmark_data_smithi090_166897_object70535:head# 2024-05-15T23:39:57.074 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:649e355e:::benchmark_data_smithi090_166897_object28810:head# 2024-05-15T23:39:57.075 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:649f0977:::benchmark_data_smithi090_166897_object18576:head# 2024-05-15T23:39:57.075 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64a021ec:::benchmark_data_smithi090_166897_object69083:head# 2024-05-15T23:39:57.075 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64a1d2da:::benchmark_data_smithi090_166897_object68326:head# 2024-05-15T23:39:57.076 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64a1f9b2:::benchmark_data_smithi090_166897_object54281:head# 2024-05-15T23:39:57.076 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64a2d629:::benchmark_data_smithi090_166897_object59972:head# 2024-05-15T23:39:57.077 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64a2d7c5:::benchmark_data_smithi090_166897_object19301:head# 2024-05-15T23:39:57.077 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64a48d7b:::benchmark_data_smithi090_166897_object14564:head# 2024-05-15T23:39:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64a6e2cb:::benchmark_data_smithi090_166897_object1407:head# 2024-05-15T23:39:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64a846d6:::benchmark_data_smithi090_166897_object31859:head# 2024-05-15T23:39:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64a95108:::benchmark_data_smithi090_166897_object33857:head# 2024-05-15T23:39:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64a99f4b:::benchmark_data_smithi090_166897_object65655:head# 2024-05-15T23:39:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64aa7f02:::benchmark_data_smithi090_166897_object61684:head# 2024-05-15T23:39:57.080 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64aaf55a:::benchmark_data_smithi090_166897_object939:head# 2024-05-15T23:39:57.080 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64aaf984:::benchmark_data_smithi090_166897_object35868:head# 2024-05-15T23:39:57.081 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64aba15a:::benchmark_data_smithi090_166897_object3420:head# 2024-05-15T23:39:57.081 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64b05084:::benchmark_data_smithi090_166897_object59904:head# 2024-05-15T23:39:57.081 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64b18359:::benchmark_data_smithi090_166897_object44895:head# 2024-05-15T23:39:57.082 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64b190d8:::benchmark_data_smithi090_166897_object53514:head# 2024-05-15T23:39:57.082 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64b2cc15:::benchmark_data_smithi090_166897_object43834:head# 2024-05-15T23:39:57.083 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64b2cd0b:::benchmark_data_smithi090_166897_object43285:head# 2024-05-15T23:39:57.083 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64b3aab4:::benchmark_data_smithi090_166897_object18922:head# 2024-05-15T23:39:57.084 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64b4b292:::benchmark_data_smithi090_166897_object40026:head# 2024-05-15T23:39:57.084 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64b527b7:::benchmark_data_smithi090_166897_object42442:head# 2024-05-15T23:39:57.084 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64b528e5:::benchmark_data_smithi090_166897_object66936:head# 2024-05-15T23:39:57.085 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64b570f6:::benchmark_data_smithi090_166897_object48167:head# 2024-05-15T23:39:57.085 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64be5ea6:::benchmark_data_smithi090_166897_object14513:head# 2024-05-15T23:39:57.086 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64beb9f4:::benchmark_data_smithi090_166897_object43971:head# 2024-05-15T23:39:57.086 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64bf06b5:::benchmark_data_smithi090_166897_object38267:head# 2024-05-15T23:39:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64bf7c1c:::benchmark_data_smithi090_166897_object36174:head# 2024-05-15T23:39:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64c076e9:::benchmark_data_smithi090_166897_object51405:head# 2024-05-15T23:39:57.088 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64c13297:::benchmark_data_smithi090_166897_object13135:head# 2024-05-15T23:39:57.088 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64c19fbf:::benchmark_data_smithi090_166897_object27237:head# 2024-05-15T23:39:57.088 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64c1f662:::benchmark_data_smithi090_166897_object39739:head# 2024-05-15T23:39:57.089 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64c30650:::benchmark_data_smithi090_166897_object23767:head# 2024-05-15T23:39:57.089 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64c32f6b:::benchmark_data_smithi090_166897_object61657:head# 2024-05-15T23:39:57.089 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64c35caf:::benchmark_data_smithi090_166897_object25170:head# 2024-05-15T23:39:57.090 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64c430be:::benchmark_data_smithi090_166897_object21960:head# 2024-05-15T23:39:57.090 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64c4ebb9:::benchmark_data_smithi090_166897_object45886:head# 2024-05-15T23:39:57.091 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64c81457:::benchmark_data_smithi090_166897_object64846:head# 2024-05-15T23:39:57.091 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64c83de1:::benchmark_data_smithi090_166897_object37011:head# 2024-05-15T23:39:57.092 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64c89926:::benchmark_data_smithi090_166897_object48677:head# 2024-05-15T23:39:57.092 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64c89ed5:::benchmark_data_smithi090_166897_object70150:head# 2024-05-15T23:39:57.093 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64ca5263:::benchmark_data_smithi090_166897_object21834:head# 2024-05-15T23:39:57.093 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64cb70ae:::benchmark_data_smithi090_166897_object7854:head# 2024-05-15T23:39:57.093 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64cc26de:::benchmark_data_smithi090_166897_object21763:head# 2024-05-15T23:39:57.094 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64cc3478:::benchmark_data_smithi090_166897_object49863:head# 2024-05-15T23:39:57.095 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64cd6463:::benchmark_data_smithi090_166897_object23092:head# 2024-05-15T23:39:57.095 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64cd76e1:::benchmark_data_smithi090_166897_object1911:head# 2024-05-15T23:39:57.096 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64cfdcd0:::benchmark_data_smithi090_166897_object6334:head# 2024-05-15T23:39:57.096 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64d0c059:::benchmark_data_smithi090_166897_object57278:head# 2024-05-15T23:39:57.097 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64d0d2bc:::benchmark_data_smithi090_166897_object55810:head# 2024-05-15T23:39:57.097 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64d1ae08:::benchmark_data_smithi090_166897_object19946:head# 2024-05-15T23:39:57.097 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:39:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:39:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:39:57.098 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:39:57.098 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:39:57.098 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64d1aeef:::benchmark_data_smithi090_166897_object64475:head# 2024-05-15T23:39:57.098 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64d1f95b:::benchmark_data_smithi090_166897_object70369:head# 2024-05-15T23:39:57.099 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64d2f06d:::benchmark_data_smithi090_166897_object13894:head# 2024-05-15T23:39:57.099 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64d3606b:::benchmark_data_smithi090_166897_object48209:head# 2024-05-15T23:39:57.099 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64d54e4e:::benchmark_data_smithi090_166897_object12310:head# 2024-05-15T23:39:57.100 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64d7e8b8:::benchmark_data_smithi090_166897_object52084:head# 2024-05-15T23:39:57.100 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64d8f50a:::benchmark_data_smithi090_166897_object28263:head# 2024-05-15T23:39:57.101 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64db0b7a:::benchmark_data_smithi090_166897_object44543:head# 2024-05-15T23:39:57.101 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64db2156:::benchmark_data_smithi090_166897_object6301:head# 2024-05-15T23:39:57.101 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64dbcf31:::benchmark_data_smithi090_166897_object59042:head# 2024-05-15T23:39:57.102 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64dbf494:::benchmark_data_smithi090_166897_object66162:head# 2024-05-15T23:39:57.102 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64dc702d:::benchmark_data_smithi090_166897_object21330:head# 2024-05-15T23:39:57.103 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64dd5e82:::benchmark_data_smithi090_166897_object40406:head# 2024-05-15T23:39:57.103 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64defb8f:::benchmark_data_smithi090_166897_object24492:head# 2024-05-15T23:39:57.104 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64e2bb4e:::benchmark_data_smithi090_166897_object24773:head# 2024-05-15T23:39:57.104 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64e56c49:::benchmark_data_smithi090_166897_object15274:head# 2024-05-15T23:39:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64e6ab34:::benchmark_data_smithi090_166897_object14764:head# 2024-05-15T23:39:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64e8bc33:::benchmark_data_smithi090_166897_object55582:head# 2024-05-15T23:39:57.106 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64e9698d:::benchmark_data_smithi090_166897_object15373:head# 2024-05-15T23:39:57.106 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64e9a58d:::benchmark_data_smithi090_166897_object60580:head# 2024-05-15T23:39:57.106 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64ecab1f:::benchmark_data_smithi090_166897_object42268:head# 2024-05-15T23:39:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64ed551e:::benchmark_data_smithi090_166897_object33150:head# 2024-05-15T23:39:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64ed744c:::benchmark_data_smithi090_166897_object1351:head# 2024-05-15T23:39:57.108 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64f08731:::benchmark_data_smithi090_166897_object23934:head# 2024-05-15T23:39:57.108 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64f1e9e5:::benchmark_data_smithi090_166897_object43225:head# 2024-05-15T23:39:57.109 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64f20cd3:::benchmark_data_smithi090_166897_object15621:head# 2024-05-15T23:39:57.109 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64f3f41e:::benchmark_data_smithi090_166897_object21212:head# 2024-05-15T23:39:57.109 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64f876fa:::benchmark_data_smithi090_166897_object33246:head# 2024-05-15T23:39:57.110 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64f90963:::benchmark_data_smithi090_166897_object11459:head# 2024-05-15T23:39:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64fa1e42:::benchmark_data_smithi090_166897_object2068:head# 2024-05-15T23:39:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64fa526f:::benchmark_data_smithi090_166897_object36281:head# 2024-05-15T23:39:57.112 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64fa5b99:::benchmark_data_smithi090_166897_object20127:head# 2024-05-15T23:39:57.112 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64fb354d:::benchmark_data_smithi090_166897_object25777:head# 2024-05-15T23:39:57.113 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64fb9864:::benchmark_data_smithi090_166897_object30488:head# 2024-05-15T23:39:57.113 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64fc501e:::benchmark_data_smithi090_166897_object59705:head# 2024-05-15T23:39:57.113 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64fcb38c:::benchmark_data_smithi090_166897_object19174:head# 2024-05-15T23:39:57.114 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64fd655d:::benchmark_data_smithi090_166897_object90:head# 2024-05-15T23:39:57.114 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64fd892f:::benchmark_data_smithi090_166897_object64163:head# 2024-05-15T23:39:57.115 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64fd9dc9:::benchmark_data_smithi090_166897_object47304:head# 2024-05-15T23:39:57.115 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64fe487c:::benchmark_data_smithi090_166897_object39860:head# 2024-05-15T23:39:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64fe8372:::benchmark_data_smithi090_166897_object66977:head# 2024-05-15T23:39:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64ff1c88:::benchmark_data_smithi090_166897_object16167:head# 2024-05-15T23:39:57.117 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:64ffd203:::benchmark_data_smithi090_166897_object53142:head# 2024-05-15T23:39:57.117 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65000c6d:::benchmark_data_smithi090_166897_object6700:head# 2024-05-15T23:39:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65004f35:::benchmark_data_smithi090_166897_object65758:head# 2024-05-15T23:39:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65020ef2:::benchmark_data_smithi090_166897_object35139:head# 2024-05-15T23:39:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65034232:::benchmark_data_smithi090_166897_object5308:head# 2024-05-15T23:39:57.119 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65038880:::benchmark_data_smithi090_166897_object245:head# 2024-05-15T23:39:57.119 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65042b9f:::benchmark_data_smithi090_166897_object51573:head# 2024-05-15T23:39:57.119 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6504f09b:::benchmark_data_smithi090_166897_object56663:head# 2024-05-15T23:39:57.120 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6505ebb2:::benchmark_data_smithi090_166897_object62258:head# 2024-05-15T23:39:57.120 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65060497:::benchmark_data_smithi090_166897_object21429:head# 2024-05-15T23:39:57.121 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65062db3:::benchmark_data_smithi090_166897_object46023:head# 2024-05-15T23:39:57.121 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65078ef7:::benchmark_data_smithi090_166897_object55268:head# 2024-05-15T23:39:57.121 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6508877a:::benchmark_data_smithi090_166897_object59337:head# 2024-05-15T23:39:57.122 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65088ea8:::benchmark_data_smithi090_166897_object8499:head# 2024-05-15T23:39:57.122 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:650a7d0a:::benchmark_data_smithi090_166897_object66053:head# 2024-05-15T23:39:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:650af4df:::benchmark_data_smithi090_166897_object68721:head# 2024-05-15T23:39:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:650c0d2b:::benchmark_data_smithi090_166897_object37258:head# 2024-05-15T23:39:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:650cebd5:::benchmark_data_smithi090_166897_object52036:head# 2024-05-15T23:39:57.124 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:650d6612:::benchmark_data_smithi090_166897_object20879:head# 2024-05-15T23:39:57.125 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:650db7e5:::benchmark_data_smithi090_166897_object48729:head# 2024-05-15T23:39:57.125 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:650e6c93:::benchmark_data_smithi090_166897_object28673:head# 2024-05-15T23:39:57.125 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:650f3417:::benchmark_data_smithi090_166897_object37509:head# 2024-05-15T23:39:57.126 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:650f540b:::benchmark_data_smithi090_166897_object55465:head# 2024-05-15T23:39:57.127 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:650fbf41:::benchmark_data_smithi090_166897_object52243:head# 2024-05-15T23:39:57.127 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:650fca30:::benchmark_data_smithi090_166897_object13933:head# 2024-05-15T23:39:57.128 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65149199:::benchmark_data_smithi090_166897_object28113:head# 2024-05-15T23:39:57.128 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6515bed9:::benchmark_data_smithi090_166897_object56813:head# 2024-05-15T23:39:57.129 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6517851f:::benchmark_data_smithi090_166897_object10398:head# 2024-05-15T23:39:57.129 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65181b53:::benchmark_data_smithi090_166897_object50498:head# 2024-05-15T23:39:57.131 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6518a3f7:::benchmark_data_smithi090_166897_object24437:head# 2024-05-15T23:39:57.132 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65191384:::benchmark_data_smithi090_166897_object46106:head# 2024-05-15T23:39:57.132 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65196553:::benchmark_data_smithi090_166897_object39370:head# 2024-05-15T23:39:57.134 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65197843:::benchmark_data_smithi090_166897_object10519:head# 2024-05-15T23:39:57.135 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:651998d5:::benchmark_data_smithi090_166897_object69070:head# 2024-05-15T23:39:57.135 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:651a39b7:::benchmark_data_smithi090_166897_object63568:head# 2024-05-15T23:39:57.136 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:651b0acd:::benchmark_data_smithi090_166897_object30880:head# 2024-05-15T23:39:57.136 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:651bbc88:::benchmark_data_smithi090_166897_object39676:head# 2024-05-15T23:39:57.137 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:651d86ab:::benchmark_data_smithi090_166897_object5558:head# 2024-05-15T23:39:57.137 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:652091d7:::benchmark_data_smithi090_166897_object53682:head# 2024-05-15T23:39:57.138 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6520bb5c:::benchmark_data_smithi090_166897_object40249:head# 2024-05-15T23:39:57.138 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65211b11:::benchmark_data_smithi090_166897_object62381:head# 2024-05-15T23:39:57.138 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65218f3b:::benchmark_data_smithi090_166897_object16241:head# 2024-05-15T23:39:57.139 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65226aaa:::benchmark_data_smithi090_166897_object9432:head# 2024-05-15T23:39:57.139 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6523c289:::benchmark_data_smithi090_166897_object59759:head# 2024-05-15T23:39:57.140 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6523ef8b:::benchmark_data_smithi090_166897_object56412:head# 2024-05-15T23:39:57.140 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6524862f:::benchmark_data_smithi090_166897_object25156:head# 2024-05-15T23:39:57.141 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6525efc7:::benchmark_data_smithi090_166897_object23458:head# 2024-05-15T23:39:57.141 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65263705:::benchmark_data_smithi090_166897_object29956:head# 2024-05-15T23:39:57.141 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65263b6a:::benchmark_data_smithi090_166897_object26020:head# 2024-05-15T23:39:57.142 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65264a6d:::benchmark_data_smithi090_166897_object53109:head# 2024-05-15T23:39:57.142 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:652686a3:::benchmark_data_smithi090_166897_object60517:head# 2024-05-15T23:39:57.143 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6526bcfe:::benchmark_data_smithi090_166897_object33548:head# 2024-05-15T23:39:57.144 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6526debd:::benchmark_data_smithi090_166897_object49146:head# 2024-05-15T23:39:57.144 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65280f82:::benchmark_data_smithi090_166897_object55235:head# 2024-05-15T23:39:57.145 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6528111e:::benchmark_data_smithi090_166897_object54948:head# 2024-05-15T23:39:57.145 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65285fa2:::benchmark_data_smithi090_166897_object11038:head# 2024-05-15T23:39:57.146 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6529b973:::benchmark_data_smithi090_166897_object20374:head# 2024-05-15T23:39:57.146 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6529da40:::benchmark_data_smithi090_166897_object38552:head# 2024-05-15T23:39:57.147 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:652b5b09:::benchmark_data_smithi090_166897_object48607:head# 2024-05-15T23:39:57.147 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:652be0e1:::benchmark_data_smithi090_166897_object70932:head# 2024-05-15T23:39:57.148 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:652bec84:::benchmark_data_smithi090_166897_object18675:head# 2024-05-15T23:39:57.148 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:652c14a1:::benchmark_data_smithi090_166897_object52925:head# 2024-05-15T23:39:57.149 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:652c6a67:::benchmark_data_smithi090_166897_object4390:head# 2024-05-15T23:39:57.149 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:652d872c:::benchmark_data_smithi090_166897_object45604:head# 2024-05-15T23:39:57.150 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:652d8bbe:::benchmark_data_smithi090_166897_object17532:head# 2024-05-15T23:39:57.150 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:652efd39:::benchmark_data_smithi090_166897_object28788:head# 2024-05-15T23:39:57.150 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:652f40b2:::benchmark_data_smithi090_166897_object2943:head# 2024-05-15T23:39:57.151 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:652f5fa6:::benchmark_data_smithi090_166897_object49126:head# 2024-05-15T23:39:57.151 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65300c5b:::benchmark_data_smithi090_166897_object138:head# 2024-05-15T23:39:57.152 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65311af1:::benchmark_data_smithi090_166897_object8911:head# 2024-05-15T23:39:57.152 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6531bae5:::benchmark_data_smithi090_166897_object5293:head# 2024-05-15T23:39:57.153 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6531c298:::benchmark_data_smithi090_166897_object14313:head# 2024-05-15T23:39:57.153 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65321cb5:::benchmark_data_smithi090_166897_object65625:head# 2024-05-15T23:39:57.154 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65335589:::benchmark_data_smithi090_166897_object39893:head# 2024-05-15T23:39:57.154 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65352c1e:::benchmark_data_smithi090_166897_object15308:head# 2024-05-15T23:39:57.154 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65355270:::benchmark_data_smithi090_166897_object65378:head# 2024-05-15T23:39:57.155 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:653575d9:::benchmark_data_smithi090_166897_object65464:head# 2024-05-15T23:39:57.155 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65379736:::benchmark_data_smithi090_166897_object63602:head# 2024-05-15T23:39:57.156 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6538359f:::benchmark_data_smithi090_166897_object25343:head# 2024-05-15T23:39:57.156 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6539319f:::benchmark_data_smithi090_166897_object11190:head# 2024-05-15T23:39:57.157 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6539a1fb:::benchmark_data_smithi090_166897_object39882:head# 2024-05-15T23:39:57.157 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:653c2d8e:::benchmark_data_smithi090_166897_object17182:head# 2024-05-15T23:39:57.158 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:653c77c3:::benchmark_data_smithi090_166897_object1447:head# 2024-05-15T23:39:57.158 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:653deca8:::benchmark_data_smithi090_166897_object67856:head# 2024-05-15T23:39:57.159 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:653e31aa:::benchmark_data_smithi090_166897_object40846:head# 2024-05-15T23:39:57.159 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:653ecc04:::benchmark_data_smithi090_166897_object62918:head# 2024-05-15T23:39:57.159 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:653f5cff:::benchmark_data_smithi090_166897_object24551:head# 2024-05-15T23:39:57.160 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:653f948e:::benchmark_data_smithi090_166897_object42652:head# 2024-05-15T23:39:57.160 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65400a47:::benchmark_data_smithi090_166897_object56099:head# 2024-05-15T23:39:57.161 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6541d09e:::benchmark_data_smithi090_166897_object62143:head# 2024-05-15T23:39:57.161 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6543b1df:::benchmark_data_smithi090_166897_object63929:head# 2024-05-15T23:39:57.162 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6543b4a5:::benchmark_data_smithi090_166897_object52390:head# 2024-05-15T23:39:57.162 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65447a47:::benchmark_data_smithi090_166897_object60679:head# 2024-05-15T23:39:57.163 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65468d5b:::benchmark_data_smithi090_166897_object64339:head# 2024-05-15T23:39:57.163 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6546a724:::benchmark_data_smithi090_166897_object56599:head# 2024-05-15T23:39:57.166 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65472bca:::benchmark_data_smithi090_166897_object67366:head# 2024-05-15T23:39:57.166 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6548a48c:::benchmark_data_smithi090_166897_object54625:head# 2024-05-15T23:39:57.167 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6549adcf:::benchmark_data_smithi090_166897_object65606:head# 2024-05-15T23:39:57.167 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:654bd46e:::benchmark_data_smithi090_166897_object33155:head# 2024-05-15T23:39:57.168 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:654c2639:::benchmark_data_smithi090_166897_object55718:head# 2024-05-15T23:39:57.169 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:654cdef1:::benchmark_data_smithi090_166897_object48123:head# 2024-05-15T23:39:57.169 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:654d4c8b:::benchmark_data_smithi090_166897_object19694:head# 2024-05-15T23:39:57.170 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:654da810:::benchmark_data_smithi090_166897_object8660:head# 2024-05-15T23:39:57.170 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:654eabc5:::benchmark_data_smithi090_166897_object69042:head# 2024-05-15T23:39:57.173 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:654ec508:::benchmark_data_smithi090_166897_object42633:head# 2024-05-15T23:39:57.173 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:655056a9:::benchmark_data_smithi090_166897_object53284:head# 2024-05-15T23:39:57.173 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6551276e:::benchmark_data_smithi090_166897_object56164:head# 2024-05-15T23:39:57.174 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6553b779:::benchmark_data_smithi090_166897_object10063:head# 2024-05-15T23:39:57.174 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6553ea8d:::benchmark_data_smithi090_166897_object31688:head# 2024-05-15T23:39:57.175 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6554a7c9:::benchmark_data_smithi090_166897_object8292:head# 2024-05-15T23:39:57.175 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6554c436:::benchmark_data_smithi090_166897_object40560:head# 2024-05-15T23:39:57.176 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65552908:::benchmark_data_smithi090_166897_object25100:head# 2024-05-15T23:39:57.176 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65556c79:::benchmark_data_smithi090_166897_object19822:head# 2024-05-15T23:39:57.177 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6556a667:::benchmark_data_smithi090_166897_object35303:head# 2024-05-15T23:39:57.177 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:655ab98d:::benchmark_data_smithi090_166897_object31470:head# 2024-05-15T23:39:57.177 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:655d4120:::benchmark_data_smithi090_166897_object6271:head# 2024-05-15T23:39:57.178 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:655d7152:::benchmark_data_smithi090_166897_object45697:head# 2024-05-15T23:39:57.178 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:655ec561:::benchmark_data_smithi090_166897_object21891:head# 2024-05-15T23:39:57.179 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:655f75c6:::benchmark_data_smithi090_166897_object69256:head# 2024-05-15T23:39:57.179 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:655fd207:::benchmark_data_smithi090_166897_object23717:head# 2024-05-15T23:39:57.180 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6560216d:::benchmark_data_smithi090_166897_object39157:head# 2024-05-15T23:39:57.180 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:656031b7:::benchmark_data_smithi090_166897_object24786:head# 2024-05-15T23:39:57.181 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65629f51:::benchmark_data_smithi090_166897_object14074:head# 2024-05-15T23:39:57.181 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6563f5f0:::benchmark_data_smithi090_166897_object9283:head# 2024-05-15T23:39:57.181 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65643f0a:::benchmark_data_smithi090_166897_object56108:head# 2024-05-15T23:39:57.182 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65646a8e:::benchmark_data_smithi090_166897_object4357:head# 2024-05-15T23:39:57.183 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65647816:::benchmark_data_smithi090_166897_object61599:head# 2024-05-15T23:39:57.183 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65659d3c:::benchmark_data_smithi090_166897_object46210:head# 2024-05-15T23:39:57.183 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6565fab0:::benchmark_data_smithi090_166897_object58702:head# 2024-05-15T23:39:57.184 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6566a7b0:::benchmark_data_smithi090_166897_object51047:head# 2024-05-15T23:39:57.184 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65671fa1:::benchmark_data_smithi090_166897_object53885:head# 2024-05-15T23:39:57.185 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65682947:::benchmark_data_smithi090_166897_object1941:head# 2024-05-15T23:39:57.185 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65699fd6:::benchmark_data_smithi090_166897_object32636:head# 2024-05-15T23:39:57.185 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6569a412:::benchmark_data_smithi090_166897_object38339:head# 2024-05-15T23:39:57.186 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:656ac10d:::benchmark_data_smithi090_166897_object17587:head# 2024-05-15T23:39:57.186 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:656b38e0:::benchmark_data_smithi090_166897_object45119:head# 2024-05-15T23:39:57.187 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:656b8c08:::benchmark_data_smithi090_166897_object27549:head# 2024-05-15T23:39:57.187 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:656baff9:::benchmark_data_smithi090_166897_object31144:head# 2024-05-15T23:39:57.188 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:656dde87:::benchmark_data_smithi090_166897_object3622:head# 2024-05-15T23:39:57.188 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:656ea656:::benchmark_data_smithi090_166897_object36859:head# 2024-05-15T23:39:57.189 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:656f45a2:::benchmark_data_smithi090_166897_object39563:head# 2024-05-15T23:39:57.193 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:656f9b5e:::benchmark_data_smithi090_166897_object8833:head# 2024-05-15T23:39:57.193 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6570c3fe:::benchmark_data_smithi090_166897_object1027:head# 2024-05-15T23:39:57.194 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6570cfcc:::benchmark_data_smithi090_166897_object51202:head# 2024-05-15T23:39:57.195 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6571f35f:::benchmark_data_smithi090_166897_object9953:head# 2024-05-15T23:39:57.195 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65726fa8:::benchmark_data_smithi090_166897_object52831:head# 2024-05-15T23:39:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:657276b7:::benchmark_data_smithi090_166897_object23959:head# 2024-05-15T23:39:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65741715:::benchmark_data_smithi090_166897_object17019:head# 2024-05-15T23:39:57.198 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65748004:::benchmark_data_smithi090_166897_object54392:head# 2024-05-15T23:39:57.198 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:657668e5:::benchmark_data_smithi090_166897_object52579:head# 2024-05-15T23:39:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65767dbe:::benchmark_data_smithi090_166897_object67030:head# 2024-05-15T23:39:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6577d812:::benchmark_data_smithi090_166897_object6451:head# 2024-05-15T23:39:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6578639b:::benchmark_data_smithi090_166897_object14749:head# 2024-05-15T23:39:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6578b2da:::benchmark_data_smithi090_166897_object18286:head# 2024-05-15T23:39:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6579b03c:::benchmark_data_smithi090_166897_object24679:head# 2024-05-15T23:39:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6579c8fd:::benchmark_data_smithi090_166897_object20399:head# 2024-05-15T23:39:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:657a3183:::benchmark_data_smithi090_166897_object65053:head# 2024-05-15T23:39:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:657bc5d8:::benchmark_data_smithi090_166897_object67348:head# 2024-05-15T23:39:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:657be90e:::benchmark_data_smithi090_166897_object8665:head# 2024-05-15T23:39:57.204 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:657c0394:::benchmark_data_smithi090_166897_object2454:head# 2024-05-15T23:39:57.204 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:657d110e:::benchmark_data_smithi090_166897_object17415:head# 2024-05-15T23:39:57.204 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:657d6b84:::benchmark_data_smithi090_166897_object61964:head# 2024-05-15T23:39:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:657e42b2:::benchmark_data_smithi090_166897_object3329:head# 2024-05-15T23:39:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:657f2577:::benchmark_data_smithi090_166897_object24858:head# 2024-05-15T23:39:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:657f4445:::benchmark_data_smithi090_166897_object5024:head# 2024-05-15T23:39:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:657f5725:::benchmark_data_smithi090_166897_object64069:head# 2024-05-15T23:39:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6582945b:::benchmark_data_smithi090_166897_object41994:head# 2024-05-15T23:39:57.208 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65836349:::benchmark_data_smithi090_166897_object17260:head# 2024-05-15T23:39:57.208 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65848003:::benchmark_data_smithi090_166897_object8709:head# 2024-05-15T23:39:57.209 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6584b8e8:::benchmark_data_smithi090_166897_object52861:head# 2024-05-15T23:39:57.209 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:658513d4:::benchmark_data_smithi090_166897_object65458:head# 2024-05-15T23:39:57.209 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:658808a0:::benchmark_data_smithi090_166897_object27937:head# 2024-05-15T23:39:57.209 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:658932b0:::benchmark_data_smithi090_166897_object38836:head# 2024-05-15T23:39:57.210 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:658973e5:::benchmark_data_smithi090_166897_object33990:head# 2024-05-15T23:39:57.210 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:658b48ba:::benchmark_data_smithi090_166897_object12925:head# 2024-05-15T23:39:57.211 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:658ca16f:::benchmark_data_smithi090_166897_object32704:head# 2024-05-15T23:39:57.211 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:658edbcb:::benchmark_data_smithi090_166897_object24649:head# 2024-05-15T23:39:57.212 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:658fdea7:::benchmark_data_smithi090_166897_object44309:head# 2024-05-15T23:39:57.212 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:658ff20b:::benchmark_data_smithi090_166897_object55740:head# 2024-05-15T23:39:57.213 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65926c48:::benchmark_data_smithi090_166897_object41350:head# 2024-05-15T23:39:57.213 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65929e96:::benchmark_data_smithi090_166897_object47327:head# 2024-05-15T23:39:57.213 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6594462a:::benchmark_data_smithi090_166897_object18990:head# 2024-05-15T23:39:57.214 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65950612:::benchmark_data_smithi090_166897_object35224:head# 2024-05-15T23:39:57.214 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65950aa8:::benchmark_data_smithi090_166897_object70435:head# 2024-05-15T23:39:57.215 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65966bee:::benchmark_data_smithi090_166897_object65718:head# 2024-05-15T23:39:57.215 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:659706bb:::benchmark_data_smithi090_166897_object31593:head# 2024-05-15T23:39:57.216 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6598a10a:::benchmark_data_smithi090_166897_object9054:head# 2024-05-15T23:39:57.216 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6598e18d:::benchmark_data_smithi090_166897_object65036:head# 2024-05-15T23:39:57.216 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6599aea6:::benchmark_data_smithi090_166897_object33350:head# 2024-05-15T23:39:57.217 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:659c0b4e:::benchmark_data_smithi090_166897_object70677:head# 2024-05-15T23:39:57.219 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:659c4faa:::benchmark_data_smithi090_166897_object63115:head# 2024-05-15T23:39:57.220 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:659da724:::benchmark_data_smithi090_166897_object55720:head# 2024-05-15T23:39:57.220 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:659f5a3b:::benchmark_data_smithi090_166897_object7739:head# 2024-05-15T23:39:57.220 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65a003eb:::benchmark_data_smithi090_166897_object47008:head# 2024-05-15T23:39:57.220 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65a03cb3:::benchmark_data_smithi090_166897_object36264:head# 2024-05-15T23:39:57.221 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65a1a72b:::benchmark_data_smithi090_166897_object52065:head# 2024-05-15T23:39:57.222 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65a273b7:::benchmark_data_smithi090_166897_object11924:head# 2024-05-15T23:39:57.223 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65a3879b:::benchmark_data_smithi090_166897_object47378:head# 2024-05-15T23:39:57.223 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65a4dd3e:::benchmark_data_smithi090_166897_object49573:head# 2024-05-15T23:39:57.224 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65a53e53:::benchmark_data_smithi090_166897_object13770:head# 2024-05-15T23:39:57.226 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65a55ca0:::benchmark_data_smithi090_166897_object11473:head# 2024-05-15T23:39:57.227 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65a58152:::benchmark_data_smithi090_166897_object33087:head# 2024-05-15T23:39:57.228 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65a67667:::benchmark_data_smithi090_166897_object55818:head# 2024-05-15T23:39:57.228 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65a69da8:::benchmark_data_smithi090_166897_object59567:head# 2024-05-15T23:39:57.228 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65a890e9:::benchmark_data_smithi090_166897_object23512:head# 2024-05-15T23:39:57.229 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65a9f7fa:::benchmark_data_smithi090_166897_object44270:head# 2024-05-15T23:39:57.229 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65aa9b36:::benchmark_data_smithi090_166897_object25971:head# 2024-05-15T23:39:57.232 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65aac95f:::benchmark_data_smithi090_166897_object23437:head# 2024-05-15T23:39:57.232 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65ab2090:::benchmark_data_smithi090_166897_object62185:head# 2024-05-15T23:39:57.233 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65ac1171:::benchmark_data_smithi090_166897_object7418:head# 2024-05-15T23:39:57.233 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65ac7468:::benchmark_data_smithi090_166897_object17460:head# 2024-05-15T23:39:57.234 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65aca308:::benchmark_data_smithi090_166897_object44416:head# 2024-05-15T23:39:57.234 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65aed303:::benchmark_data_smithi090_166897_object57454:head# 2024-05-15T23:39:57.235 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65b07717:::benchmark_data_smithi090_166897_object61085:head# 2024-05-15T23:39:57.236 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65b0fe89:::benchmark_data_smithi090_166897_object51953:head# 2024-05-15T23:39:57.236 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65b111f8:::benchmark_data_smithi090_166897_object11628:head# 2024-05-15T23:39:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65b24ad8:::benchmark_data_smithi090_166897_object65077:head# 2024-05-15T23:39:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65b2bcae:::benchmark_data_smithi090_166897_object4896:head# 2024-05-15T23:39:57.239 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65b2f7c5:::benchmark_data_smithi090_166897_object5705:head# 2024-05-15T23:39:57.241 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65b504a5:::benchmark_data_smithi090_166897_object30216:head# 2024-05-15T23:39:57.241 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65b61619:::benchmark_data_smithi090_166897_object69700:head# 2024-05-15T23:39:57.242 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65b6d8e6:::benchmark_data_smithi090_166897_object62502:head# 2024-05-15T23:39:57.242 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65b6e7d1:::benchmark_data_smithi090_166897_object27060:head# 2024-05-15T23:39:57.243 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65b7112e:::benchmark_data_smithi090_166897_object4466:head# 2024-05-15T23:39:57.244 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65b86a82:::benchmark_data_smithi090_166897_object51063:head# 2024-05-15T23:39:57.244 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65baf405:::benchmark_data_smithi090_166897_object69718:head# 2024-05-15T23:39:57.244 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65bbfb42:::benchmark_data_smithi090_166897_object21390:head# 2024-05-15T23:39:57.245 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65bc99c6:::benchmark_data_smithi090_166897_object4524:head# 2024-05-15T23:39:57.245 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65c06096:::benchmark_data_smithi090_166897_object61473:head# 2024-05-15T23:39:57.245 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65c07756:::benchmark_data_smithi090_166897_object49834:head# 2024-05-15T23:39:57.246 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65c0939f:::benchmark_data_smithi090_166897_object58465:head# 2024-05-15T23:39:57.246 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65c3185a:::benchmark_data_smithi090_166897_object48695:head# 2024-05-15T23:39:57.247 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65c3d108:::benchmark_data_smithi090_166897_object14660:head# 2024-05-15T23:39:57.248 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65c43110:::benchmark_data_smithi090_166897_object41026:head# 2024-05-15T23:39:57.249 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65c5d8b8:::benchmark_data_smithi090_166897_object64088:head# 2024-05-15T23:39:57.250 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65c696dd:::benchmark_data_smithi090_166897_object56084:head# 2024-05-15T23:39:57.253 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65c6cfd8:::benchmark_data_smithi090_166897_object39636:head# 2024-05-15T23:39:57.253 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65c95fbd:::benchmark_data_smithi090_166897_object29790:head# 2024-05-15T23:39:57.254 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65c9e521:::benchmark_data_smithi090_166897_object19677:head# 2024-05-15T23:39:57.254 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65cab72b:::benchmark_data_smithi090_166897_object47674:head# 2024-05-15T23:39:57.255 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65caeabd:::benchmark_data_smithi090_166897_object6524:head# 2024-05-15T23:39:57.255 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65cc1f58:::benchmark_data_smithi090_166897_object66675:head# 2024-05-15T23:39:57.255 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65ce17d6:::benchmark_data_smithi090_166897_object1244:head# 2024-05-15T23:39:57.256 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65d07907:::benchmark_data_smithi090_166897_object50040:head# 2024-05-15T23:39:57.256 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65d29644:::benchmark_data_smithi090_166897_object1339:head# 2024-05-15T23:39:57.257 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65d31aa8:::benchmark_data_smithi090_166897_object54952:head# 2024-05-15T23:39:57.258 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65d56c41:::benchmark_data_smithi090_166897_object54190:head# 2024-05-15T23:39:57.258 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65d5ae44:::benchmark_data_smithi090_166897_object23468:head# 2024-05-15T23:39:57.259 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65d66a9c:::benchmark_data_smithi090_166897_object43554:head# 2024-05-15T23:39:57.259 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65d6ffcf:::benchmark_data_smithi090_166897_object5345:head# 2024-05-15T23:39:57.259 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65d7db53:::benchmark_data_smithi090_166897_object46071:head# 2024-05-15T23:39:57.260 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65d84a4a:::benchmark_data_smithi090_166897_object54781:head# 2024-05-15T23:39:57.260 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65d84d53:::benchmark_data_smithi090_166897_object41754:head# 2024-05-15T23:39:57.261 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65d89600:::benchmark_data_smithi090_166897_object35885:head# 2024-05-15T23:39:57.262 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65d9f74c:::benchmark_data_smithi090_166897_object33756:head# 2024-05-15T23:39:57.262 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65db3f86:::benchmark_data_smithi090_166897_object62127:head# 2024-05-15T23:39:57.263 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65dd0f75:::benchmark_data_smithi090_166897_object64836:head# 2024-05-15T23:39:57.263 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65dd1a15:::benchmark_data_smithi090_166897_object63835:head# 2024-05-15T23:39:57.264 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65df8cf3:::benchmark_data_smithi090_166897_object9316:head# 2024-05-15T23:39:57.264 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65e0a109:::benchmark_data_smithi090_166897_object64808:head# 2024-05-15T23:39:57.264 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65e11828:::benchmark_data_smithi090_166897_object29574:head# 2024-05-15T23:39:57.265 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65e145d2:::benchmark_data_smithi090_166897_object29915:head# 2024-05-15T23:39:57.265 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65e1836b:::benchmark_data_smithi090_166897_object6884:head# 2024-05-15T23:39:57.266 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65e1a77a:::benchmark_data_smithi090_166897_object34345:head# 2024-05-15T23:39:57.266 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65e39300:::benchmark_data_smithi090_166897_object6141:head# 2024-05-15T23:39:57.268 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65e461be:::benchmark_data_smithi090_166897_object32601:head# 2024-05-15T23:39:57.268 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65e5a7c9:::benchmark_data_smithi090_166897_object37961:head# 2024-05-15T23:39:57.269 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65e5aea2:::benchmark_data_smithi090_166897_object54961:head# 2024-05-15T23:39:57.269 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65e69586:::benchmark_data_smithi090_166897_object2920:head# 2024-05-15T23:39:57.270 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65e6b815:::benchmark_data_smithi090_166897_object45256:head# 2024-05-15T23:39:57.270 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65e6ecb2:::benchmark_data_smithi090_166897_object69567:head# 2024-05-15T23:39:57.271 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65e739cd:::benchmark_data_smithi090_166897_object60684:head# 2024-05-15T23:39:57.271 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65e79708:::benchmark_data_smithi090_166897_object34812:head# 2024-05-15T23:39:57.271 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65e9e669:::benchmark_data_smithi090_166897_object54849:head# 2024-05-15T23:39:57.272 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65eb230c:::benchmark_data_smithi090_166897_object20207:head# 2024-05-15T23:39:57.272 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65edbf8d:::benchmark_data_smithi090_166897_object48588:head# 2024-05-15T23:39:57.273 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65eec8b0:::benchmark_data_smithi090_166897_object22720:head# 2024-05-15T23:39:57.273 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65efa81e:::benchmark_data_smithi090_166897_object46598:head# 2024-05-15T23:39:57.274 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65f0a1cf:::benchmark_data_smithi090_166897_object68703:head# 2024-05-15T23:39:57.274 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65f0c3e8:::benchmark_data_smithi090_166897_object53521:head# 2024-05-15T23:39:57.274 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65f0d78e:::benchmark_data_smithi090_166897_object3217:head# 2024-05-15T23:39:57.275 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65f2a63b:::benchmark_data_smithi090_166897_object65256:head# 2024-05-15T23:39:57.275 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65f5fcb1:::benchmark_data_smithi090_166897_object61101:head# 2024-05-15T23:39:57.276 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65f71865:::benchmark_data_smithi090_166897_object18295:head# 2024-05-15T23:39:57.276 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65f71d40:::benchmark_data_smithi090_166897_object3940:head# 2024-05-15T23:39:57.277 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65f7564c:::benchmark_data_smithi090_166897_object48151:head# 2024-05-15T23:39:57.277 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65f7a8d8:::benchmark_data_smithi090_166897_object6743:head# 2024-05-15T23:39:57.278 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65f7b534:::benchmark_data_smithi090_166897_object10465:head# 2024-05-15T23:39:57.278 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65f8dd1e:::benchmark_data_smithi090_166897_object17472:head# 2024-05-15T23:39:57.278 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65f8ddbe:::benchmark_data_smithi090_166897_object65206:head# 2024-05-15T23:39:57.279 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65fa1d54:::benchmark_data_smithi090_166897_object45717:head# 2024-05-15T23:39:57.279 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65fb617a:::benchmark_data_smithi090_166897_object61046:head# 2024-05-15T23:39:57.280 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65fc2a1a:::benchmark_data_smithi090_166897_object67152:head# 2024-05-15T23:39:57.281 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65fcaad2:::benchmark_data_smithi090_166897_object29796:head# 2024-05-15T23:39:57.281 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65fcb79c:::benchmark_data_smithi090_166897_object56707:head# 2024-05-15T23:39:57.281 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:65feff69:::benchmark_data_smithi090_166897_object43174:head# 2024-05-15T23:39:57.282 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6600d3c7:::benchmark_data_smithi090_166897_object55856:head# 2024-05-15T23:39:57.282 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6602490a:::benchmark_data_smithi090_166897_object5703:head# 2024-05-15T23:39:57.283 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6603e6ad:::benchmark_data_smithi090_166897_object33479:head# 2024-05-15T23:39:57.283 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66047b53:::benchmark_data_smithi090_166897_object1507:head# 2024-05-15T23:39:57.284 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6604fcf8:::benchmark_data_smithi090_166897_object63318:head# 2024-05-15T23:39:57.285 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6606164e:::benchmark_data_smithi090_166897_object45107:head# 2024-05-15T23:39:57.285 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66063c48:::benchmark_data_smithi090_166897_object39937:head# 2024-05-15T23:39:57.286 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66092d3b:::benchmark_data_smithi090_166897_object23848:head# 2024-05-15T23:39:57.286 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:660a685d:::benchmark_data_smithi090_166897_object64388:head# 2024-05-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:660aadd5:::benchmark_data_smithi090_166897_object4941:head# 2024-05-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:660ab92f:::benchmark_data_smithi090_166897_object15655:head# 2024-05-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:660c1f13:::benchmark_data_smithi090_166897_object11845:head# 2024-05-15T23:39:57.288 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:660c2de1:::benchmark_data_smithi090_166897_object49260:head# 2024-05-15T23:39:57.288 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:660d3a0b:::benchmark_data_smithi090_166897_object70968:head# 2024-05-15T23:39:57.291 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:660d701a:::benchmark_data_smithi090_166897_object12957:head# 2024-05-15T23:39:57.291 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:660f84b4:::benchmark_data_smithi090_166897_object41014:head# 2024-05-15T23:39:57.292 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:660fa740:::benchmark_data_smithi090_166897_object45470:head# 2024-05-15T23:39:57.292 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:660fe249:::benchmark_data_smithi090_166897_object66054:head# 2024-05-15T23:39:57.292 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:661096a0:::benchmark_data_smithi090_166897_object10520:head# 2024-05-15T23:39:57.293 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6610cc7b:::benchmark_data_smithi090_166897_object24961:head# 2024-05-15T23:39:57.294 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6610faa2:::benchmark_data_smithi090_166897_object66932:head# 2024-05-15T23:39:57.295 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6611527a:::benchmark_data_smithi090_166897_object68093:head# 2024-05-15T23:39:57.295 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6611cbd6:::benchmark_data_smithi090_166897_object40158:head# 2024-05-15T23:39:57.296 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66158ad9:::benchmark_data_smithi090_166897_object59060:head# 2024-05-15T23:39:57.296 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6616f736:::benchmark_data_smithi090_166897_object22095:head# 2024-05-15T23:39:57.296 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66177984:::benchmark_data_smithi090_166897_object62776:head# 2024-05-15T23:39:57.297 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6617aa40:::benchmark_data_smithi090_166897_object52687:head# 2024-05-15T23:39:57.297 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6617dd7c:::benchmark_data_smithi090_166897_object51073:head# 2024-05-15T23:39:57.298 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66188100:::benchmark_data_smithi090_166897_object1296:head# 2024-05-15T23:39:57.298 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66198a3d:::benchmark_data_smithi090_166897_object31505:head# 2024-05-15T23:39:57.299 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6619cf7c:::benchmark_data_smithi090_166897_object10380:head# 2024-05-15T23:39:57.299 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:661dbcdb:::benchmark_data_smithi090_166897_object3121:head# 2024-05-15T23:39:57.300 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:661fd420:::benchmark_data_smithi090_166897_object20224:head# 2024-05-15T23:39:57.300 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66208294:::benchmark_data_smithi090_166897_object50806:head# 2024-05-15T23:39:57.301 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6620bb11:::benchmark_data_smithi090_166897_object20801:head# 2024-05-15T23:39:57.301 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6621308a:::benchmark_data_smithi090_166897_object4788:head# 2024-05-15T23:39:57.302 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66216be7:::benchmark_data_smithi090_166897_object59813:head# 2024-05-15T23:39:57.302 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:662184f4:::benchmark_data_smithi090_166897_object45988:head# 2024-05-15T23:39:57.302 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66221165:::benchmark_data_smithi090_166897_object46962:head# 2024-05-15T23:39:57.303 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66226208:::benchmark_data_smithi090_166897_object7082:head# 2024-05-15T23:39:57.303 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66229250:::benchmark_data_smithi090_166897_object38230:head# 2024-05-15T23:39:57.304 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66233c19:::benchmark_data_smithi090_166897_object15391:head# 2024-05-15T23:39:57.305 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6623c35c:::benchmark_data_smithi090_166897_object29788:head# 2024-05-15T23:39:57.305 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66261b83:::benchmark_data_smithi090_166897_object6882:head# 2024-05-15T23:39:57.305 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6629d6ef:::benchmark_data_smithi090_166897_object28819:head# 2024-05-15T23:39:57.306 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:662a338e:::benchmark_data_smithi090_166897_object63171:head# 2024-05-15T23:39:57.306 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:662a545d:::benchmark_data_smithi090_166897_object36708:head# 2024-05-15T23:39:57.306 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:662d4620:::benchmark_data_smithi090_166897_object16731:head# 2024-05-15T23:39:57.307 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:662d7ca3:::benchmark_data_smithi090_166897_object18273:head# 2024-05-15T23:39:57.307 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:662dda76:::benchmark_data_smithi090_166897_object2395:head# 2024-05-15T23:39:57.308 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:662e8fcc:::benchmark_data_smithi090_166897_object1879:head# 2024-05-15T23:39:57.308 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:662fa48d:::benchmark_data_smithi090_166897_object26607:head# 2024-05-15T23:39:57.309 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:662fcacf:::benchmark_data_smithi090_166897_object10034:head# 2024-05-15T23:39:57.309 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6630052a:::benchmark_data_smithi090_166897_object790:head# 2024-05-15T23:39:57.309 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6630c1a0:::benchmark_data_smithi090_166897_object13566:head# 2024-05-15T23:39:57.310 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66322cd7:::benchmark_data_smithi090_166897_object22493:head# 2024-05-15T23:39:57.310 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:663263e3:::benchmark_data_smithi090_166897_object38590:head# 2024-05-15T23:39:57.311 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:663333d9:::benchmark_data_smithi090_166897_object55874:head# 2024-05-15T23:39:57.311 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66337ad6:::benchmark_data_smithi090_166897_object51720:head# 2024-05-15T23:39:57.311 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:663498a2:::benchmark_data_smithi090_166897_object55389:head# 2024-05-15T23:39:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6634a18f:::benchmark_data_smithi090_166897_object57335:head# 2024-05-15T23:39:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6634bcf0:::benchmark_data_smithi090_166897_object63392:head# 2024-05-15T23:39:57.314 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66355d9a:::benchmark_data_smithi090_166897_object67129:head# 2024-05-15T23:39:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:663625ce:::benchmark_data_smithi090_166897_object67295:head# 2024-05-15T23:39:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6637b7db:::benchmark_data_smithi090_166897_object32025:head# 2024-05-15T23:39:57.316 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6637ff16:::benchmark_data_smithi090_166897_object15332:head# 2024-05-15T23:39:57.316 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:663a809f:::benchmark_data_smithi090_166897_object43033:head# 2024-05-15T23:39:57.317 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:663c12b5:::benchmark_data_smithi090_166897_object2009:head# 2024-05-15T23:39:57.319 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:663ccfe7:::benchmark_data_smithi090_166897_object51147:head# 2024-05-15T23:39:57.319 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:663cdfc2:::benchmark_data_smithi090_166897_object45428:head# 2024-05-15T23:39:57.320 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:663d7c35:::benchmark_data_smithi090_166897_object58971:head# 2024-05-15T23:39:57.320 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:664103ed:::benchmark_data_smithi090_166897_object11635:head# 2024-05-15T23:39:57.320 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6641bcc3:::benchmark_data_smithi090_166897_object31725:head# 2024-05-15T23:39:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66421ab2:::benchmark_data_smithi090_166897_object40230:head# 2024-05-15T23:39:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66444976:::benchmark_data_smithi090_166897_object20438:head# 2024-05-15T23:39:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:664475b6:::benchmark_data_smithi090_166897_object8330:head# 2024-05-15T23:39:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:664791f1:::benchmark_data_smithi090_166897_object9019:head# 2024-05-15T23:39:57.324 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6647a0d7:::benchmark_data_smithi090_166897_object6067:head# 2024-05-15T23:39:57.324 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6647f87e:::benchmark_data_smithi090_166897_object26796:head# 2024-05-15T23:39:57.325 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66481e3a:::benchmark_data_smithi090_166897_object59698:head# 2024-05-15T23:39:57.325 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6648b697:::benchmark_data_smithi090_166897_object3955:head# 2024-05-15T23:39:57.326 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66490d20:::benchmark_data_smithi090_166897_object38236:head# 2024-05-15T23:39:57.326 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:664bc451:::benchmark_data_smithi090_166897_object67121:head# 2024-05-15T23:39:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:664f95e4:::benchmark_data_smithi090_166897_object16716:head# 2024-05-15T23:39:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:664fb970:::benchmark_data_smithi090_166897_object8812:head# 2024-05-15T23:39:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66504e02:::benchmark_data_smithi090_166897_object37163:head# 2024-05-15T23:39:57.328 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66505a22:::benchmark_data_smithi090_166897_object56958:head# 2024-05-15T23:39:57.328 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6650d417:::benchmark_data_smithi090_166897_object59466:head# 2024-05-15T23:39:57.329 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66521475:::benchmark_data_smithi090_166897_object34199:head# 2024-05-15T23:39:57.330 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66530391:::benchmark_data_smithi090_166897_object36813:head# 2024-05-15T23:39:57.330 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66546e2b:::benchmark_data_smithi090_166897_object42449:head# 2024-05-15T23:39:57.331 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66550de1:::benchmark_data_smithi090_166897_object26344:head# 2024-05-15T23:39:57.331 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66577095:::benchmark_data_smithi090_166897_object1088:head# 2024-05-15T23:39:57.333 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6657a889:::benchmark_data_smithi090_166897_object14661:head# 2024-05-15T23:39:57.334 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6657b6d6:::benchmark_data_smithi090_166897_object13815:head# 2024-05-15T23:39:57.334 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6659bc83:::benchmark_data_smithi090_166897_object63334:head# 2024-05-15T23:39:57.334 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:665a0cd5:::benchmark_data_smithi090_166897_object20391:head# 2024-05-15T23:39:57.337 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:665a25a0:::benchmark_data_smithi090_166897_object56003:head# 2024-05-15T23:39:57.337 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:665ab40e:::benchmark_data_smithi090_166897_object61579:head# 2024-05-15T23:39:57.338 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:665e8f4d:::benchmark_data_smithi090_166897_object34454:head# 2024-05-15T23:39:57.338 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:665f82f5:::benchmark_data_smithi090_166897_object52504:head# 2024-05-15T23:39:57.338 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:665ff639:::benchmark_data_smithi090_166897_object63633:head# 2024-05-15T23:39:57.339 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6660d5e0:::benchmark_data_smithi090_166897_object28911:head# 2024-05-15T23:39:57.339 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66618123:::benchmark_data_smithi090_166897_object62291:head# 2024-05-15T23:39:57.339 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6662ec9d:::benchmark_data_smithi090_166897_object46744:head# 2024-05-15T23:39:57.340 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6662feca:::benchmark_data_smithi090_166897_object59708:head# 2024-05-15T23:39:57.341 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:666468b6:::benchmark_data_smithi090_166897_object23733:head# 2024-05-15T23:39:57.341 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6664bd45:::benchmark_data_smithi090_166897_object14407:head# 2024-05-15T23:39:57.342 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6664d98c:::benchmark_data_smithi090_166897_object43929:head# 2024-05-15T23:39:57.342 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66654a45:::benchmark_data_smithi090_166897_object36121:head# 2024-05-15T23:39:57.343 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66663005:::benchmark_data_smithi090_166897_object30270:head# 2024-05-15T23:39:57.343 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:666839ce:::benchmark_data_smithi090_166897_object12845:head# 2024-05-15T23:39:57.343 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66685c9c:::benchmark_data_smithi090_166897_object28135:head# 2024-05-15T23:39:57.344 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6668825e:::benchmark_data_smithi090_166897_object28141:head# 2024-05-15T23:39:57.344 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66695922:::benchmark_data_smithi090_166897_object26207:head# 2024-05-15T23:39:57.345 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:666aa11e:::benchmark_data_smithi090_166897_object40270:head# 2024-05-15T23:39:57.345 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:666b2b09:::benchmark_data_smithi090_166897_object64370:head# 2024-05-15T23:39:57.345 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:666d03d9:::benchmark_data_smithi090_166897_object14698:head# 2024-05-15T23:39:57.346 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:666d0955:::benchmark_data_smithi090_166897_object44707:head# 2024-05-15T23:39:57.346 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:666d9d14:::benchmark_data_smithi090_166897_object53036:head# 2024-05-15T23:39:57.347 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:666dd72c:::benchmark_data_smithi090_166897_object63964:head# 2024-05-15T23:39:57.347 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:666e09e4:::benchmark_data_smithi090_166897_object43999:head# 2024-05-15T23:39:57.350 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:666e6447:::benchmark_data_smithi090_166897_object4693:head# 2024-05-15T23:39:57.350 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6670ea9f:::benchmark_data_smithi090_166897_object40166:head# 2024-05-15T23:39:57.351 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6670edfb:::benchmark_data_smithi090_166897_object31031:head# 2024-05-15T23:39:57.352 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66751f95:::benchmark_data_smithi090_166897_object69803:head# 2024-05-15T23:39:57.354 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66753550:::benchmark_data_smithi090_166897_object37072:head# 2024-05-15T23:39:57.354 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:667620bd:::benchmark_data_smithi090_166897_object46944:head# 2024-05-15T23:39:57.354 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6678d499:::benchmark_data_smithi090_166897_object21644:head# 2024-05-15T23:39:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:667943f5:::benchmark_data_smithi090_166897_object10904:head# 2024-05-15T23:39:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:667a686a:::benchmark_data_smithi090_166897_object10941:head# 2024-05-15T23:39:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:667aef36:::benchmark_data_smithi090_166897_object30955:head# 2024-05-15T23:39:57.359 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:667b831e:::benchmark_data_smithi090_166897_object52728:head# 2024-05-15T23:39:57.360 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:667be894:::benchmark_data_smithi090_166897_object62930:head# 2024-05-15T23:39:57.360 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:667bf74a:::benchmark_data_smithi090_166897_object38046:head# 2024-05-15T23:39:57.360 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:667c9908:::benchmark_data_smithi090_166897_object57731:head# 2024-05-15T23:39:57.361 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:667d04e7:::benchmark_data_smithi090_166897_object62003:head# 2024-05-15T23:39:57.362 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:667d17f1:::benchmark_data_smithi090_166897_object32730:head# 2024-05-15T23:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:667d8011:::benchmark_data_smithi090_166897_object45269:head# 2024-05-15T23:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:667e961c:::benchmark_data_smithi090_166897_object37708:head# 2024-05-15T23:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:667eb162:::benchmark_data_smithi090_166897_object61270:head# 2024-05-15T23:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66805cff:::benchmark_data_smithi090_166897_object9476:head# 2024-05-15T23:39:57.364 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6681b2ab:::benchmark_data_smithi090_166897_object10670:head# 2024-05-15T23:39:57.364 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66822820:::benchmark_data_smithi090_166897_object47316:head# 2024-05-15T23:39:57.365 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6682d22c:::benchmark_data_smithi090_166897_object54287:head# 2024-05-15T23:39:57.365 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66839875:::benchmark_data_smithi090_166897_object8628:head# 2024-05-15T23:39:57.365 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6685c072:::benchmark_data_smithi090_166897_object65004:head# 2024-05-15T23:39:57.366 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6686788c:::benchmark_data_smithi090_166897_object28252:head# 2024-05-15T23:39:57.366 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66876e52:::benchmark_data_smithi090_166897_object9912:head# 2024-05-15T23:39:57.367 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6688557d:::benchmark_data_smithi090_166897_object44241:head# 2024-05-15T23:39:57.367 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6688bb3b:::benchmark_data_smithi090_166897_object26316:head# 2024-05-15T23:39:57.368 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:668936b7:::benchmark_data_smithi090_166897_object7365:head# 2024-05-15T23:39:57.368 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6689b297:::benchmark_data_smithi090_166897_object41808:head# 2024-05-15T23:39:57.368 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:668a836c:::benchmark_data_smithi090_166897_object52118:head# 2024-05-15T23:39:57.369 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:668bdfb8:::benchmark_data_smithi090_166897_object69082:head# 2024-05-15T23:39:57.369 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:668c2784:::benchmark_data_smithi090_166897_object10089:head# 2024-05-15T23:39:57.370 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:668d0b7d:::benchmark_data_smithi090_166897_object40075:head# 2024-05-15T23:39:57.370 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:668dfbc3:::benchmark_data_smithi090_166897_object50930:head# 2024-05-15T23:39:57.371 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:668fa5a2:::benchmark_data_smithi090_166897_object65455:head# 2024-05-15T23:39:57.371 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66904f3e:::benchmark_data_smithi090_166897_object39788:head# 2024-05-15T23:39:57.374 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:669090d8:::benchmark_data_smithi090_166897_object23027:head# 2024-05-15T23:39:57.374 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:669250ec:::benchmark_data_smithi090_166897_object40080:head# 2024-05-15T23:39:57.375 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:669528e7:::benchmark_data_smithi090_166897_object30640:head# 2024-05-15T23:39:57.375 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:669734a2:::benchmark_data_smithi090_166897_object50861:head# 2024-05-15T23:39:57.377 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6697abc9:::benchmark_data_smithi090_166897_object56880:head# 2024-05-15T23:39:57.380 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6698f1a0:::benchmark_data_smithi090_166897_object14754:head# 2024-05-15T23:39:57.380 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66990910:::benchmark_data_smithi090_166897_object68616:head# 2024-05-15T23:39:57.381 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:669a12fb:::benchmark_data_smithi090_166897_object8573:head# 2024-05-15T23:39:57.381 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:669cb481:::benchmark_data_smithi090_166897_object41992:head# 2024-05-15T23:39:57.381 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:669e8b76:::benchmark_data_smithi090_166897_object1011:head# 2024-05-15T23:39:57.382 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:669f79fc:::benchmark_data_smithi090_166897_object64305:head# 2024-05-15T23:39:57.382 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:669ffe5a:::benchmark_data_smithi090_166897_object46107:head# 2024-05-15T23:39:57.383 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66a19837:::benchmark_data_smithi090_166897_object50062:head# 2024-05-15T23:39:57.383 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66a1e911:::benchmark_data_smithi090_166897_object26304:head# 2024-05-15T23:39:57.384 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66a25cd8:::benchmark_data_smithi090_166897_object68187:head# 2024-05-15T23:39:57.384 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66a2f754:::benchmark_data_smithi090_166897_object39481:head# 2024-05-15T23:39:57.384 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66a3644a:::benchmark_data_smithi090_166897_object35014:head# 2024-05-15T23:39:57.385 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66a37d49:::benchmark_data_smithi090_166897_object53159:head# 2024-05-15T23:39:57.386 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66a44d08:::benchmark_data_smithi090_166897_object37720:head# 2024-05-15T23:39:57.386 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66a4c4a8:::benchmark_data_smithi090_166897_object28915:head# 2024-05-15T23:39:57.387 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66a5a049:::benchmark_data_smithi090_166897_object18713:head# 2024-05-15T23:39:57.387 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66a692b1:::benchmark_data_smithi090_166897_object68177:head# 2024-05-15T23:39:57.388 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66a77b4c:::benchmark_data_smithi090_166897_object69506:head# 2024-05-15T23:39:57.388 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66a7a845:::benchmark_data_smithi090_166897_object34272:head# 2024-05-15T23:39:57.388 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66a7c1f8:::benchmark_data_smithi090_166897_object53250:head# 2024-05-15T23:39:57.389 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66a8565c:::benchmark_data_smithi090_166897_object26043:head# 2024-05-15T23:39:57.389 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66a9b615:::benchmark_data_smithi090_166897_object14540:head# 2024-05-15T23:39:57.390 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66aa2d98:::benchmark_data_smithi090_166897_object47400:head# 2024-05-15T23:39:57.390 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66ab147a:::benchmark_data_smithi090_166897_object2734:head# 2024-05-15T23:39:57.390 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66af3d86:::benchmark_data_smithi090_166897_object13130:head# 2024-05-15T23:39:57.391 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66b0704c:::benchmark_data_smithi090_166897_object4811:head# 2024-05-15T23:39:57.391 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66b2e407:::benchmark_data_smithi090_166897_object63993:head# 2024-05-15T23:39:57.392 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66b4807e:::benchmark_data_smithi090_166897_object49500:head# 2024-05-15T23:39:57.392 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66b50953:::benchmark_data_smithi090_166897_object55558:head# 2024-05-15T23:39:57.393 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66b5b709:::benchmark_data_smithi090_166897_object25874:head# 2024-05-15T23:39:57.393 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66b63728:::benchmark_data_smithi090_166897_object51995:head# 2024-05-15T23:39:57.394 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66b9abf3:::benchmark_data_smithi090_166897_object51599:head# 2024-05-15T23:39:57.394 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66baab14:::benchmark_data_smithi090_166897_object32669:head# 2024-05-15T23:39:57.394 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66bb3897:::benchmark_data_smithi090_166897_object26457:head# 2024-05-15T23:39:57.395 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66bb9cf3:::benchmark_data_smithi090_166897_object48301:head# 2024-05-15T23:39:57.395 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66bc0cb1:::benchmark_data_smithi090_166897_object15110:head# 2024-05-15T23:39:57.395 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66bcddfc:::benchmark_data_smithi090_166897_object11090:head# 2024-05-15T23:39:57.396 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66bd2669:::benchmark_data_smithi090_166897_object25481:head# 2024-05-15T23:39:57.396 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66bd5eaf:::benchmark_data_smithi090_166897_object63336:head# 2024-05-15T23:39:57.397 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66bd7619:::benchmark_data_smithi090_166897_object52019:head# 2024-05-15T23:39:57.397 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66bd8ce2:::benchmark_data_smithi090_166897_object54865:head# 2024-05-15T23:39:57.398 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66bdab22:::benchmark_data_smithi090_166897_object66532:head# 2024-05-15T23:39:57.398 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66bdf51c:::benchmark_data_smithi090_166897_object27114:head# 2024-05-15T23:39:57.399 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66bf95fa:::benchmark_data_smithi090_166897_object31410:head# 2024-05-15T23:39:57.399 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66bff6bc:::benchmark_data_smithi090_166897_object49924:head# 2024-05-15T23:39:57.400 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66c11f72:::benchmark_data_smithi090_166897_object59331:head# 2024-05-15T23:39:57.400 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66c26e94:::benchmark_data_smithi090_166897_object20513:head# 2024-05-15T23:39:57.400 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66c39d0d:::benchmark_data_smithi090_166897_object41965:head# 2024-05-15T23:39:57.400 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66c48502:::benchmark_data_smithi090_166897_object67775:head# 2024-05-15T23:39:57.401 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66c54df5:::benchmark_data_smithi090_166897_object18726:head# 2024-05-15T23:39:57.401 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66c90c89:::benchmark_data_smithi090_166897_object12564:head# 2024-05-15T23:39:57.402 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66c963f8:::benchmark_data_smithi090_166897_object54663:head# 2024-05-15T23:39:57.402 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66c9efff:::benchmark_data_smithi090_166897_object65325:head# 2024-05-15T23:39:57.403 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66ca8ac7:::benchmark_data_smithi090_166897_object5256:head# 2024-05-15T23:39:57.403 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66ca9f44:::benchmark_data_smithi090_166897_object71074:head# 2024-05-15T23:39:57.404 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66cc44c5:::benchmark_data_smithi090_166897_object460:head# 2024-05-15T23:39:57.404 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66cc98dc:::benchmark_data_smithi090_166897_object53750:head# 2024-05-15T23:39:57.405 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66cea109:::benchmark_data_smithi090_166897_object27144:head# 2024-05-15T23:39:57.405 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66d18df7:::benchmark_data_smithi090_166897_object22222:head# 2024-05-15T23:39:57.408 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66d1c1a8:::benchmark_data_smithi090_166897_object49131:head# 2024-05-15T23:39:57.408 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66d1e3c7:::benchmark_data_smithi090_166897_object10666:head# 2024-05-15T23:39:57.408 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66d2f865:::benchmark_data_smithi090_166897_object3839:head# 2024-05-15T23:39:57.409 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66d5a954:::benchmark_data_smithi090_166897_object39580:head# 2024-05-15T23:39:57.409 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66d6324e:::benchmark_data_smithi090_166897_object8987:head# 2024-05-15T23:39:57.409 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66d6a3b9:::benchmark_data_smithi090_166897_object26966:head# 2024-05-15T23:39:57.409 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66d717f4:::benchmark_data_smithi090_166897_object12646:head# 2024-05-15T23:39:57.410 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66da41c0:::benchmark_data_smithi090_166897_object65677:head# 2024-05-15T23:39:57.410 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66db6185:::benchmark_data_smithi090_166897_object70136:head# 2024-05-15T23:39:57.411 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66dd3687:::benchmark_data_smithi090_166897_object6513:head# 2024-05-15T23:39:57.411 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66de4daa:::benchmark_data_smithi090_166897_object65795:head# 2024-05-15T23:39:57.412 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66df4ebe:::benchmark_data_smithi090_166897_object58927:head# 2024-05-15T23:39:57.412 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66e040fc:::benchmark_data_smithi090_166897_object59484:head# 2024-05-15T23:39:57.412 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66e08af2:::benchmark_data_smithi090_166897_object63769:head# 2024-05-15T23:39:57.413 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66e3a6f1:::benchmark_data_smithi090_166897_object2182:head# 2024-05-15T23:39:57.413 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66e3c317:::benchmark_data_smithi090_166897_object8555:head# 2024-05-15T23:39:57.414 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66e49a97:::benchmark_data_smithi090_166897_object65287:head# 2024-05-15T23:39:57.414 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66e5bc2c:::benchmark_data_smithi090_166897_object38460:head# 2024-05-15T23:39:57.415 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66e69444:::benchmark_data_smithi090_166897_object61604:head# 2024-05-15T23:39:57.415 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66e73849:::benchmark_data_smithi090_166897_object51300:head# 2024-05-15T23:39:57.416 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66e7a56e:::benchmark_data_smithi090_166897_object47486:head# 2024-05-15T23:39:57.416 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66e85bb0:::benchmark_data_smithi090_166897_object59838:head# 2024-05-15T23:39:57.417 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66e92a09:::benchmark_data_smithi090_166897_object40558:head# 2024-05-15T23:39:57.417 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66e9b400:::benchmark_data_smithi090_166897_object45700:head# 2024-05-15T23:39:57.418 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66eb76f2:::benchmark_data_smithi090_166897_object2015:head# 2024-05-15T23:39:57.418 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66ebb73d:::benchmark_data_smithi090_166897_object38189:head# 2024-05-15T23:39:57.419 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66ec1d2e:::benchmark_data_smithi090_166897_object69025:head# 2024-05-15T23:39:57.419 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66edba8c:::benchmark_data_smithi090_166897_object2565:head# 2024-05-15T23:39:57.420 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66ee552f:::benchmark_data_smithi090_166897_object23753:head# 2024-05-15T23:39:57.420 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66ee6822:::benchmark_data_smithi090_166897_object38654:head# 2024-05-15T23:39:57.421 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66ef984d:::benchmark_data_smithi090_166897_object16119:head# 2024-05-15T23:39:57.421 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66ef99bf:::benchmark_data_smithi090_166897_object59310:head# 2024-05-15T23:39:57.421 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66f14040:::benchmark_data_smithi090_166897_object52696:head# 2024-05-15T23:39:57.422 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66f1ad6e:::benchmark_data_smithi090_166897_object9926:head# 2024-05-15T23:39:57.423 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66f28dd0:::benchmark_data_smithi090_166897_object52050:head# 2024-05-15T23:39:57.423 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66f2b6f5:::benchmark_data_smithi090_166897_object24839:head# 2024-05-15T23:39:57.424 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66f42da0:::benchmark_data_smithi090_166897_object22288:head# 2024-05-15T23:39:57.424 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66f53ecd:::benchmark_data_smithi090_166897_object12378:head# 2024-05-15T23:39:57.424 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66f561d4:::benchmark_data_smithi090_166897_object5670:head# 2024-05-15T23:39:57.425 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66f5c81c:::benchmark_data_smithi090_166897_object62180:head# 2024-05-15T23:39:57.425 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66f74305:::benchmark_data_smithi090_166897_object69153:head# 2024-05-15T23:39:57.426 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66f7b573:::benchmark_data_smithi090_166897_object50045:head# 2024-05-15T23:39:57.426 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66f7b583:::benchmark_data_smithi090_166897_object15012:head# 2024-05-15T23:39:57.427 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66f8b600:::benchmark_data_smithi090_166897_object61578:head# 2024-05-15T23:39:57.427 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66f96c25:::benchmark_data_smithi090_166897_object62600:head# 2024-05-15T23:39:57.428 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66fb7a5f:::benchmark_data_smithi090_166897_object5244:head# 2024-05-15T23:39:57.428 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66fd7b62:::benchmark_data_smithi090_166897_object26451:head# 2024-05-15T23:39:57.429 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66fe3786:::benchmark_data_smithi090_166897_object58811:head# 2024-05-15T23:39:57.430 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66fecf35:::benchmark_data_smithi090_166897_object36346:head# 2024-05-15T23:39:57.430 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:66ff7c68:::benchmark_data_smithi090_166897_object22256:head# 2024-05-15T23:39:57.430 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67028daa:::benchmark_data_smithi090_166897_object23460:head# 2024-05-15T23:39:57.430 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67038067:::benchmark_data_smithi090_166897_object8758:head# 2024-05-15T23:39:57.431 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67047fcb:::benchmark_data_smithi090_166897_object58449:head# 2024-05-15T23:39:57.432 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67069da7:::benchmark_data_smithi090_166897_object17565:head# 2024-05-15T23:39:57.432 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67089433:::benchmark_data_smithi090_166897_object65930:head# 2024-05-15T23:39:57.433 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:670a27f3:::benchmark_data_smithi090_166897_object8063:head# 2024-05-15T23:39:57.433 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:670b35f5:::benchmark_data_smithi090_166897_object32456:head# 2024-05-15T23:39:57.434 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:670b5c00:::benchmark_data_smithi090_166897_object1286:head# 2024-05-15T23:39:57.434 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:670bd3b0:::benchmark_data_smithi090_166897_object63304:head# 2024-05-15T23:39:57.435 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:670c155d:::benchmark_data_smithi090_166897_object64986:head# 2024-05-15T23:39:57.435 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:670df0a7:::benchmark_data_smithi090_166897_object13589:head# 2024-05-15T23:39:57.436 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:670e0fa4:::benchmark_data_smithi090_166897_object40562:head# 2024-05-15T23:39:57.436 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6714ad91:::benchmark_data_smithi090_166897_object19090:head# 2024-05-15T23:39:57.437 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6714e444:::benchmark_data_smithi090_166897_object30543:head# 2024-05-15T23:39:57.437 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67164d6a:::benchmark_data_smithi090_166897_object65054:head# 2024-05-15T23:39:57.437 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6716a9b5:::benchmark_data_smithi090_166897_object16000:head# 2024-05-15T23:39:57.438 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6718c50d:::benchmark_data_smithi090_166897_object20566:head# 2024-05-15T23:39:57.438 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6718c95c:::benchmark_data_smithi090_166897_object6649:head# 2024-05-15T23:39:57.439 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67193c5a:::benchmark_data_smithi090_166897_object32747:head# 2024-05-15T23:39:57.439 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:671f8ec0:::benchmark_data_smithi090_166897_object43334:head# 2024-05-15T23:39:57.440 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67200eaa:::benchmark_data_smithi090_166897_object26227:head# 2024-05-15T23:39:57.440 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6722ee10:::benchmark_data_smithi090_166897_object30811:head# 2024-05-15T23:39:57.440 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6723553d:::benchmark_data_smithi090_166897_object28117:head# 2024-05-15T23:39:57.441 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67239283:::benchmark_data_smithi090_166897_object49436:head# 2024-05-15T23:39:57.441 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6723f18d:::benchmark_data_smithi090_166897_object25675:head# 2024-05-15T23:39:57.442 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67240e40:::benchmark_data_smithi090_166897_object48894:head# 2024-05-15T23:39:57.442 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6724568b:::benchmark_data_smithi090_166897_object40265:head# 2024-05-15T23:39:57.443 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6724ee7e:::benchmark_data_smithi090_166897_object52432:head# 2024-05-15T23:39:57.443 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6726aba9:::benchmark_data_smithi090_166897_object10806:head# 2024-05-15T23:39:57.446 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6726ff69:::benchmark_data_smithi090_166897_object64130:head# 2024-05-15T23:39:57.448 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67271827:::benchmark_data_smithi090_166897_object37434:head# 2024-05-15T23:39:57.450 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67275a99:::benchmark_data_smithi090_166897_object19250:head# 2024-05-15T23:39:57.451 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67279ef4:::benchmark_data_smithi090_166897_object3999:head# 2024-05-15T23:39:57.454 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:672adc2e:::benchmark_data_smithi090_166897_object64811:head# 2024-05-15T23:39:57.454 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:672c7c23:::benchmark_data_smithi090_166897_object31424:head# 2024-05-15T23:39:57.455 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:672cd52f:::benchmark_data_smithi090_166897_object24713:head# 2024-05-15T23:39:57.455 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:672e4c71:::benchmark_data_smithi090_166897_object65205:head# 2024-05-15T23:39:57.455 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:672f52f1:::benchmark_data_smithi090_166897_object46823:head# 2024-05-15T23:39:57.457 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:673010ca:::benchmark_data_smithi090_166897_object54881:head# 2024-05-15T23:39:57.457 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6730f951:::benchmark_data_smithi090_166897_object24848:head# 2024-05-15T23:39:57.457 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67323849:::benchmark_data_smithi090_166897_object29337:head# 2024-05-15T23:39:57.458 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6732f380:::benchmark_data_smithi090_166897_object34762:head# 2024-05-15T23:39:57.458 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67333714:::benchmark_data_smithi090_166897_object9067:head# 2024-05-15T23:39:57.459 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6733a2e2:::benchmark_data_smithi090_166897_object32665:head# 2024-05-15T23:39:57.459 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67349f9f:::benchmark_data_smithi090_166897_object49547:head# 2024-05-15T23:39:57.460 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:673ab56f:::benchmark_data_smithi090_166897_object6386:head# 2024-05-15T23:39:57.460 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:673b82dd:::benchmark_data_smithi090_166897_object58726:head# 2024-05-15T23:39:57.460 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:673c4683:::benchmark_data_smithi090_166897_object36186:head# 2024-05-15T23:39:57.461 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:673cb670:::benchmark_data_smithi090_166897_object41442:head# 2024-05-15T23:39:57.461 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:673cc0fc:::benchmark_data_smithi090_166897_object10206:head# 2024-05-15T23:39:57.462 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:673d0586:::benchmark_data_smithi090_166897_object2317:head# 2024-05-15T23:39:57.462 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:673d4e12:::benchmark_data_smithi090_166897_object5123:head# 2024-05-15T23:39:57.463 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:673e05e1:::benchmark_data_smithi090_166897_object49360:head# 2024-05-15T23:39:57.466 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67414849:::benchmark_data_smithi090_166897_object57109:head# 2024-05-15T23:39:57.466 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67418df8:::benchmark_data_smithi090_166897_object45384:head# 2024-05-15T23:39:57.466 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6741dae9:::benchmark_data_smithi090_166897_object26867:head# 2024-05-15T23:39:57.467 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67427619:::benchmark_data_smithi090_166897_object64376:head# 2024-05-15T23:39:57.467 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67429067:::benchmark_data_smithi090_166897_object11571:head# 2024-05-15T23:39:57.468 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6742a2fb:::benchmark_data_smithi090_166897_object45938:head# 2024-05-15T23:39:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6742c05f:::benchmark_data_smithi090_166897_object46468:head# 2024-05-15T23:39:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6743f6d2:::benchmark_data_smithi090_166897_object6822:head# 2024-05-15T23:39:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6744f312:::benchmark_data_smithi090_166897_object30112:head# 2024-05-15T23:39:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67486156:::benchmark_data_smithi090_166897_object28709:head# 2024-05-15T23:39:57.470 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67493243:::benchmark_data_smithi090_166897_object55728:head# 2024-05-15T23:39:57.470 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:674a24ee:::benchmark_data_smithi090_166897_object55128:head# 2024-05-15T23:39:57.471 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:674b870f:::benchmark_data_smithi090_166897_object17871:head# 2024-05-15T23:39:57.471 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:674cc837:::benchmark_data_smithi090_166897_object54604:head# 2024-05-15T23:39:57.472 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:674d2979:::benchmark_data_smithi090_166897_object2880:head# 2024-05-15T23:39:57.474 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6754b75e:::benchmark_data_smithi090_166897_object24491:head# 2024-05-15T23:39:57.475 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6754d1bb:::benchmark_data_smithi090_166897_object7484:head# 2024-05-15T23:39:57.475 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67552e34:::benchmark_data_smithi090_166897_object47091:head# 2024-05-15T23:39:57.477 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67556062:::benchmark_data_smithi090_166897_object4734:head# 2024-05-15T23:39:57.480 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6756e76c:::benchmark_data_smithi090_166897_object51928:head# 2024-05-15T23:39:57.482 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67594d08:::benchmark_data_smithi090_166897_object45254:head# 2024-05-15T23:39:57.485 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6759acbb:::benchmark_data_smithi090_166897_object497:head# 2024-05-15T23:39:57.485 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6759e000:::benchmark_data_smithi090_166897_object53476:head# 2024-05-15T23:39:57.486 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:675a0a23:::benchmark_data_smithi090_166897_object11842:head# 2024-05-15T23:39:57.486 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:675a4257:::benchmark_data_smithi090_166897_object39490:head# 2024-05-15T23:39:57.486 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:675ab3eb:::benchmark_data_smithi090_166897_object32972:head# 2024-05-15T23:39:57.487 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:675b02e6:::benchmark_data_smithi090_166897_object1833:head# 2024-05-15T23:39:57.488 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:675b48ad:::benchmark_data_smithi090_166897_object52545:head# 2024-05-15T23:39:57.488 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:675b8bcb:::benchmark_data_smithi090_166897_object33956:head# 2024-05-15T23:39:57.489 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:675c3a6c:::benchmark_data_smithi090_166897_object61534:head# 2024-05-15T23:39:57.489 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:675c8cbe:::benchmark_data_smithi090_166897_object18266:head# 2024-05-15T23:39:57.490 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:675cfbd4:::benchmark_data_smithi090_166897_object41488:head# 2024-05-15T23:39:57.490 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6764a7fc:::benchmark_data_smithi090_166897_object52789:head# 2024-05-15T23:39:57.491 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:676641be:::benchmark_data_smithi090_166897_object17037:head# 2024-05-15T23:39:57.491 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67687e4e:::benchmark_data_smithi090_166897_object22308:head# 2024-05-15T23:39:57.492 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67688bdd:::benchmark_data_smithi090_166897_object12637:head# 2024-05-15T23:39:57.492 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:676acd54:::benchmark_data_smithi090_166897_object30786:head# 2024-05-15T23:39:57.493 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:676b0b46:::benchmark_data_smithi090_166897_object54530:head# 2024-05-15T23:39:57.493 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:676cb7f6:::benchmark_data_smithi090_166897_object47827:head# 2024-05-15T23:39:57.494 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:676d269c:::benchmark_data_smithi090_166897_object37406:head# 2024-05-15T23:39:57.497 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:676e09ad:::benchmark_data_smithi090_166897_object50017:head# 2024-05-15T23:39:57.498 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:676f0add:::benchmark_data_smithi090_166897_object4779:head# 2024-05-15T23:39:57.498 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:676f30a7:::benchmark_data_smithi090_166897_object69358:head# 2024-05-15T23:39:57.500 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:676fa6d5:::benchmark_data_smithi090_166897_object8788:head# 2024-05-15T23:39:57.500 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67703d2e:::benchmark_data_smithi090_166897_object2767:head# 2024-05-15T23:39:57.501 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:677121dc:::benchmark_data_smithi090_166897_object27572:head# 2024-05-15T23:39:57.501 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67717b18:::benchmark_data_smithi090_166897_object1987:head# 2024-05-15T23:39:57.502 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67721c04:::benchmark_data_smithi090_166897_object8373:head# 2024-05-15T23:39:57.502 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67728450:::benchmark_data_smithi090_166897_object10976:head# 2024-05-15T23:39:57.503 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67783802:::benchmark_data_smithi090_166897_object37630:head# 2024-05-15T23:39:57.503 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:677a7d22:::benchmark_data_smithi090_166897_object54944:head# 2024-05-15T23:39:57.503 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:677a8867:::benchmark_data_smithi090_166897_object27020:head# 2024-05-15T23:39:57.504 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:677b3dee:::benchmark_data_smithi090_166897_object48562:head# 2024-05-15T23:39:57.504 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:677b7b90:::benchmark_data_smithi090_166897_object36261:head# 2024-05-15T23:39:57.505 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:677c4e5c:::benchmark_data_smithi090_166897_object36070:head# 2024-05-15T23:39:57.505 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:677c9b1c:::benchmark_data_smithi090_166897_object42487:head# 2024-05-15T23:39:57.505 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:677d672c:::benchmark_data_smithi090_166897_object67849:head# 2024-05-15T23:39:57.506 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:677ec246:::benchmark_data_smithi090_166897_object50992:head# 2024-05-15T23:39:57.506 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:677fcb3e:::benchmark_data_smithi090_166897_object16928:head# 2024-05-15T23:39:57.507 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:677fd4c8:::benchmark_data_smithi090_166897_object66201:head# 2024-05-15T23:39:57.510 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67814091:::benchmark_data_smithi090_166897_object46229:head# 2024-05-15T23:39:57.510 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67817b6a:::benchmark_data_smithi090_166897_object44731:head# 2024-05-15T23:39:57.510 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6782d985:::benchmark_data_smithi090_166897_object55624:head# 2024-05-15T23:39:57.511 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6785795c:::benchmark_data_smithi090_166897_object51005:head# 2024-05-15T23:39:57.512 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67861fc2:::benchmark_data_smithi090_166897_object65864:head# 2024-05-15T23:39:57.512 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6786e268:::benchmark_data_smithi090_166897_object2831:head# 2024-05-15T23:39:57.513 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6786e41a:::benchmark_data_smithi090_166897_object34161:head# 2024-05-15T23:39:57.513 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:678933cf:::benchmark_data_smithi090_166897_object60202:head# 2024-05-15T23:39:57.514 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:678ac0c2:::benchmark_data_smithi090_166897_object50547:head# 2024-05-15T23:39:57.514 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:678acde6:::benchmark_data_smithi090_166897_object69945:head# 2024-05-15T23:39:57.514 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:678ba2fd:::benchmark_data_smithi090_166897_object66339:head# 2024-05-15T23:39:57.515 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:678c8e37:::benchmark_data_smithi090_166897_object2448:head# 2024-05-15T23:39:57.515 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:679118dc:::benchmark_data_smithi090_166897_object45321:head# 2024-05-15T23:39:57.516 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67918a73:::benchmark_data_smithi090_166897_object45037:head# 2024-05-15T23:39:57.516 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6792be2c:::benchmark_data_smithi090_166897_object62577:head# 2024-05-15T23:39:57.517 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6792f9e4:::benchmark_data_smithi090_166897_object157:head# 2024-05-15T23:39:57.517 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:679314f7:::benchmark_data_smithi090_166897_object18733:head# 2024-05-15T23:39:57.517 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67931cbf:::benchmark_data_smithi090_166897_object27501:head# 2024-05-15T23:39:57.518 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6793ee7c:::benchmark_data_smithi090_166897_object31734:head# 2024-05-15T23:39:57.518 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6797c7d0:::benchmark_data_smithi090_166897_object6317:head# 2024-05-15T23:39:57.519 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6797fea1:::benchmark_data_smithi090_166897_object30968:head# 2024-05-15T23:39:57.519 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67982819:::benchmark_data_smithi090_166897_object37117:head# 2024-05-15T23:39:57.519 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6799833b:::benchmark_data_smithi090_166897_object30193:head# 2024-05-15T23:39:57.520 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:679ca3ae:::benchmark_data_smithi090_166897_object40695:head# 2024-05-15T23:39:57.520 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:679d0f68:::benchmark_data_smithi090_166897_object41861:head# 2024-05-15T23:39:57.521 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:679db75a:::benchmark_data_smithi090_166897_object38741:head# 2024-05-15T23:39:57.521 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:679e0e51:::benchmark_data_smithi090_166897_object35801:head# 2024-05-15T23:39:57.521 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:679e2b2f:::benchmark_data_smithi090_166897_object65683:head# 2024-05-15T23:39:57.522 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67a09885:::benchmark_data_smithi090_166897_object54502:head# 2024-05-15T23:39:57.522 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67a1c969:::benchmark_data_smithi090_166897_object52281:head# 2024-05-15T23:39:57.523 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67a1eddb:::benchmark_data_smithi090_166897_object12521:head# 2024-05-15T23:39:57.523 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67a2a315:::benchmark_data_smithi090_166897_object32130:head# 2024-05-15T23:39:57.524 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67a2d4f1:::benchmark_data_smithi090_166897_object46472:head# 2024-05-15T23:39:57.525 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67a3f342:::benchmark_data_smithi090_166897_object28746:head# 2024-05-15T23:39:57.525 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67a4d0d5:::benchmark_data_smithi090_166897_object35693:head# 2024-05-15T23:39:57.525 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67a7e943:::benchmark_data_smithi090_166897_object55487:head# 2024-05-15T23:39:57.526 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67a8d6d8:::benchmark_data_smithi090_166897_object19522:head# 2024-05-15T23:39:57.526 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67a9d0f5:::benchmark_data_smithi090_166897_object39208:head# 2024-05-15T23:39:57.527 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67acdf21:::benchmark_data_smithi090_166897_object7047:head# 2024-05-15T23:39:57.527 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67ad2111:::benchmark_data_smithi090_166897_object44536:head# 2024-05-15T23:39:57.527 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67ad80ee:::benchmark_data_smithi090_166897_object5905:head# 2024-05-15T23:39:57.528 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67adfaab:::benchmark_data_smithi090_166897_object38582:head# 2024-05-15T23:39:57.528 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67aef080:::benchmark_data_smithi090_166897_object30479:head# 2024-05-15T23:39:57.529 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67af28f2:::benchmark_data_smithi090_166897_object3947:head# 2024-05-15T23:39:57.529 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67b09586:::benchmark_data_smithi090_166897_object993:head# 2024-05-15T23:39:57.530 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67b11d54:::benchmark_data_smithi090_166897_object31610:head# 2024-05-15T23:39:57.530 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67b1d10e:::benchmark_data_smithi090_166897_object44644:head# 2024-05-15T23:39:57.531 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67b1d4e7:::benchmark_data_smithi090_166897_object7857:head# 2024-05-15T23:39:57.531 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67b23967:::benchmark_data_smithi090_166897_object13965:head# 2024-05-15T23:39:57.532 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67b276e5:::benchmark_data_smithi090_166897_object59932:head# 2024-05-15T23:39:57.532 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67b2adf4:::benchmark_data_smithi090_166897_object49085:head# 2024-05-15T23:39:57.532 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67b418d7:::benchmark_data_smithi090_166897_object10167:head# 2024-05-15T23:39:57.533 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67b4f63c:::benchmark_data_smithi090_166897_object25889:head# 2024-05-15T23:39:57.533 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67b5a0f2:::benchmark_data_smithi090_166897_object44419:head# 2024-05-15T23:39:57.534 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67b6816e:::benchmark_data_smithi090_166897_object40691:head# 2024-05-15T23:39:57.534 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67ba1a02:::benchmark_data_smithi090_166897_object17801:head# 2024-05-15T23:39:57.534 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67bace5c:::benchmark_data_smithi090_166897_object27624:head# 2024-05-15T23:39:57.535 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67bb129b:::benchmark_data_smithi090_166897_object25365:head# 2024-05-15T23:39:57.535 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67bc10f3:::benchmark_data_smithi090_166897_object70086:head# 2024-05-15T23:39:57.536 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67bc18e5:::benchmark_data_smithi090_166897_object61647:head# 2024-05-15T23:39:57.536 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67bd59c9:::benchmark_data_smithi090_166897_object11565:head# 2024-05-15T23:39:57.536 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67be4e5d:::benchmark_data_smithi090_166897_object52368:head# 2024-05-15T23:39:57.537 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67c06908:::benchmark_data_smithi090_166897_object37813:head# 2024-05-15T23:39:57.537 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67c1800a:::benchmark_data_smithi090_166897_object13923:head# 2024-05-15T23:39:57.538 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67c1942f:::benchmark_data_smithi090_166897_object44375:head# 2024-05-15T23:39:57.538 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67c226b1:::benchmark_data_smithi090_166897_object65437:head# 2024-05-15T23:39:57.539 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67c4aa33:::benchmark_data_smithi090_166897_object14773:head# 2024-05-15T23:39:57.539 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67c678ea:::benchmark_data_smithi090_166897_object20643:head# 2024-05-15T23:39:57.539 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67c858da:::benchmark_data_smithi090_166897_object36543:head# 2024-05-15T23:39:57.540 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67c8af54:::benchmark_data_smithi090_166897_object26452:head# 2024-05-15T23:39:57.541 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67c95388:::benchmark_data_smithi090_166897_object37463:head# 2024-05-15T23:39:57.541 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67cdcd9c:::benchmark_data_smithi090_166897_object21488:head# 2024-05-15T23:39:57.541 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67ceadc7:::benchmark_data_smithi090_166897_object68335:head# 2024-05-15T23:39:57.542 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67cefb3d:::benchmark_data_smithi090_166897_object11605:head# 2024-05-15T23:39:57.543 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67cf1aae:::benchmark_data_smithi090_166897_object67982:head# 2024-05-15T23:39:57.543 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67cf928c:::benchmark_data_smithi090_166897_object1104:head# 2024-05-15T23:39:57.544 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67cf95e8:::benchmark_data_smithi090_166897_object14376:head# 2024-05-15T23:39:57.544 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67d100d2:::benchmark_data_smithi090_166897_object52817:head# 2024-05-15T23:39:57.545 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67d140c1:::benchmark_data_smithi090_166897_object25357:head# 2024-05-15T23:39:57.545 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67d1665d:::benchmark_data_smithi090_166897_object20373:head# 2024-05-15T23:39:57.546 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67d1ca70:::benchmark_data_smithi090_166897_object36622:head# 2024-05-15T23:39:57.546 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67d2ccd6:::benchmark_data_smithi090_166897_object32175:head# 2024-05-15T23:39:57.546 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67d339e7:::benchmark_data_smithi090_166897_object39527:head# 2024-05-15T23:39:57.547 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67d34a93:::benchmark_data_smithi090_166897_object60925:head# 2024-05-15T23:39:57.547 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67d386a5:::benchmark_data_smithi090_166897_object15467:head# 2024-05-15T23:39:57.548 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67d49a53:::benchmark_data_smithi090_166897_object60844:head# 2024-05-15T23:39:57.548 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67d6cadd:::benchmark_data_smithi090_166897_object36900:head# 2024-05-15T23:39:57.549 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67d78696:::benchmark_data_smithi090_166897_object69968:head# 2024-05-15T23:39:57.549 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67d7fc44:::benchmark_data_smithi090_166897_object55530:head# 2024-05-15T23:39:57.549 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67d8b244:::benchmark_data_smithi090_166897_object10333:head# 2024-05-15T23:39:57.550 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67d9a9d1:::benchmark_data_smithi090_166897_object39246:head# 2024-05-15T23:39:57.550 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67d9f468:::benchmark_data_smithi090_166897_object18485:head# 2024-05-15T23:39:57.551 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67da1d79:::benchmark_data_smithi090_166897_object11105:head# 2024-05-15T23:39:57.552 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67db77ed:::benchmark_data_smithi090_166897_object6739:head# 2024-05-15T23:39:57.552 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67db989f:::benchmark_data_smithi090_166897_object7690:head# 2024-05-15T23:39:57.552 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67dcf617:::benchmark_data_smithi090_166897_object57989:head# 2024-05-15T23:39:57.553 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67de2648:::benchmark_data_smithi090_166897_object45138:head# 2024-05-15T23:39:57.553 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67debaa5:::benchmark_data_smithi090_166897_object27411:head# 2024-05-15T23:39:57.554 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67e2f161:::benchmark_data_smithi090_166897_object70524:head# 2024-05-15T23:39:57.554 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67e53e39:::benchmark_data_smithi090_166897_object46906:head# 2024-05-15T23:39:57.554 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67e5deb7:::benchmark_data_smithi090_166897_object29616:head# 2024-05-15T23:39:57.555 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67e85809:::benchmark_data_smithi090_166897_object54943:head# 2024-05-15T23:39:57.555 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67e872fd:::benchmark_data_smithi090_166897_object23234:head# 2024-05-15T23:39:57.556 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67e8ca8a:::benchmark_data_smithi090_166897_object9236:head# 2024-05-15T23:39:57.556 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67e9477a:::benchmark_data_smithi090_166897_object46610:head# 2024-05-15T23:39:57.557 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67ea04a8:::benchmark_data_smithi090_166897_object38957:head# 2024-05-15T23:39:57.557 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67ea44a4:::benchmark_data_smithi090_166897_object62984:head# 2024-05-15T23:39:57.558 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67ea6387:::benchmark_data_smithi090_166897_object61364:head# 2024-05-15T23:39:57.558 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67ea6439:::benchmark_data_smithi090_166897_object6129:head# 2024-05-15T23:39:57.558 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67eba059:::benchmark_data_smithi090_166897_object52618:head# 2024-05-15T23:39:57.559 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67ec1ef0:::benchmark_data_smithi090_166897_object62916:head# 2024-05-15T23:39:57.559 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67ed75dd:::benchmark_data_smithi090_166897_object48782:head# 2024-05-15T23:39:57.560 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67ee5922:::benchmark_data_smithi090_166897_object54428:head# 2024-05-15T23:39:57.560 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67f18eb7:::benchmark_data_smithi090_166897_object53286:head# 2024-05-15T23:39:57.560 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67f353a0:::benchmark_data_smithi090_166897_object57090:head# 2024-05-15T23:39:57.561 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67f8f0b4:::benchmark_data_smithi090_166897_object46092:head# 2024-05-15T23:39:57.561 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67f9a0de:::benchmark_data_smithi090_166897_object32094:head# 2024-05-15T23:39:57.562 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67fa5a90:::benchmark_data_smithi090_166897_object7380:head# 2024-05-15T23:39:57.562 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67fbe16f:::benchmark_data_smithi090_166897_object10299:head# 2024-05-15T23:39:57.563 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67fc55e8:::benchmark_data_smithi090_166897_object26009:head# 2024-05-15T23:39:57.563 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67fc670c:::benchmark_data_smithi090_166897_object28407:head# 2024-05-15T23:39:57.564 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67fd9821:::benchmark_data_smithi090_166897_object48595:head# 2024-05-15T23:39:57.564 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67ff30d4:::benchmark_data_smithi090_166897_object37685:head# 2024-05-15T23:39:57.565 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:67ff96d4:::benchmark_data_smithi090_166897_object52610:head# 2024-05-15T23:39:57.565 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6802732d:::benchmark_data_smithi090_166897_object53999:head# 2024-05-15T23:39:57.566 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68038ca5:::benchmark_data_smithi090_166897_object24563:head# 2024-05-15T23:39:57.566 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6804c3ed:::benchmark_data_smithi090_166897_object16434:head# 2024-05-15T23:39:57.567 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6804e2f5:::benchmark_data_smithi090_166897_object32647:head# 2024-05-15T23:39:57.567 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6804fb7a:::benchmark_data_smithi090_166897_object61012:head# 2024-05-15T23:39:57.567 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68051902:::benchmark_data_smithi090_166897_object67829:head# 2024-05-15T23:39:57.568 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68067297:::benchmark_data_smithi090_166897_object62452:head# 2024-05-15T23:39:57.568 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6806bc89:::benchmark_data_smithi090_166897_object11455:head# 2024-05-15T23:39:57.569 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68088049:::benchmark_data_smithi090_166897_object65156:head# 2024-05-15T23:39:57.569 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:680b27ff:::benchmark_data_smithi090_166897_object40731:head# 2024-05-15T23:39:57.569 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:680b4e0b:::benchmark_data_smithi090_166897_object4590:head# 2024-05-15T23:39:57.570 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:680c3067:::benchmark_data_smithi090_166897_object38639:head# 2024-05-15T23:39:57.571 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 55 16 86683 86667 98.4553 74.4375 0.0141233 0.0101542 2024-05-15T23:39:57.571 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:680c3673:::benchmark_data_smithi090_166897_object11104:head# 2024-05-15T23:39:57.571 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:680c52cd:::benchmark_data_smithi090_166897_object50165:head# 2024-05-15T23:39:57.572 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:680c56a7:::benchmark_data_smithi090_166897_object50616:head# 2024-05-15T23:39:57.572 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:680c59e1:::benchmark_data_smithi090_166897_object31652:head# 2024-05-15T23:39:57.573 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:680c5c62:::benchmark_data_smithi090_166897_object41290:head# 2024-05-15T23:39:57.573 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:680cce03:::benchmark_data_smithi090_166897_object47810:head# 2024-05-15T23:39:57.573 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:680cdda3:::benchmark_data_smithi090_166897_object62415:head# 2024-05-15T23:39:57.574 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:680d94d4:::benchmark_data_smithi090_166897_object22733:head# 2024-05-15T23:39:57.575 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:680f2925:::benchmark_data_smithi090_166897_object14983:head# 2024-05-15T23:39:57.575 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:680f363c:::benchmark_data_smithi090_166897_object65917:head# 2024-05-15T23:39:57.575 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6813e6ae:::benchmark_data_smithi090_166897_object22211:head# 2024-05-15T23:39:57.576 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6813f3c4:::benchmark_data_smithi090_166897_object20758:head# 2024-05-15T23:39:57.576 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68159f9a:::benchmark_data_smithi090_166897_object5855:head# 2024-05-15T23:39:57.577 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68163fbf:::benchmark_data_smithi090_166897_object64505:head# 2024-05-15T23:39:57.577 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:681694ac:::benchmark_data_smithi090_166897_object61920:head# 2024-05-15T23:39:57.577 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:681721c5:::benchmark_data_smithi090_166897_object24018:head# 2024-05-15T23:39:57.578 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:681dc021:::benchmark_data_smithi090_166897_object39218:head# 2024-05-15T23:39:57.578 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68201329:::benchmark_data_smithi090_166897_object43557:head# 2024-05-15T23:39:57.579 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68201676:::benchmark_data_smithi090_166897_object33520:head# 2024-05-15T23:39:57.579 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68207426:::benchmark_data_smithi090_166897_object9123:head# 2024-05-15T23:39:57.580 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:682077ff:::benchmark_data_smithi090_166897_object23656:head# 2024-05-15T23:39:57.580 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68226bc5:::benchmark_data_smithi090_166897_object64538:head# 2024-05-15T23:39:57.580 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6822cc5c:::benchmark_data_smithi090_166897_object60621:head# 2024-05-15T23:39:57.581 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68242722:::benchmark_data_smithi090_166897_object66808:head# 2024-05-15T23:39:57.581 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6824a7d6:::benchmark_data_smithi090_166897_object37247:head# 2024-05-15T23:39:57.582 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6824bee4:::benchmark_data_smithi090_166897_object21462:head# 2024-05-15T23:39:57.582 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:682568aa:::benchmark_data_smithi090_166897_object14969:head# 2024-05-15T23:39:57.582 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:682798a0:::benchmark_data_smithi090_166897_object43959:head# 2024-05-15T23:39:57.583 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6827d2f0:::benchmark_data_smithi090_166897_object11996:head# 2024-05-15T23:39:57.583 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6829f170:::benchmark_data_smithi090_166897_object35368:head# 2024-05-15T23:39:57.583 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:682a3e34:::benchmark_data_smithi090_166897_object30887:head# 2024-05-15T23:39:57.584 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:682b33f1:::benchmark_data_smithi090_166897_object45595:head# 2024-05-15T23:39:57.584 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:682b9c2f:::benchmark_data_smithi090_166897_object63122:head# 2024-05-15T23:39:57.585 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:682d8800:::benchmark_data_smithi090_166897_object70332:head# 2024-05-15T23:39:57.585 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:682dce95:::benchmark_data_smithi090_166897_object12562:head# 2024-05-15T23:39:57.586 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:682de348:::benchmark_data_smithi090_166897_object3854:head# 2024-05-15T23:39:57.586 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:682e25d6:::benchmark_data_smithi090_166897_object48411:head# 2024-05-15T23:39:57.586 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:682e4523:::benchmark_data_smithi090_166897_object56774:head# 2024-05-15T23:39:57.587 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:682f8079:::benchmark_data_smithi090_166897_object53808:head# 2024-05-15T23:39:57.587 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68311fa2:::benchmark_data_smithi090_166897_object57550:head# 2024-05-15T23:39:57.588 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68315c00:::benchmark_data_smithi090_166897_object3080:head# 2024-05-15T23:39:57.589 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68319327:::benchmark_data_smithi090_166897_object55706:head# 2024-05-15T23:39:57.590 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6831e87d:::benchmark_data_smithi090_166897_object43409:head# 2024-05-15T23:39:57.590 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6832417f:::benchmark_data_smithi090_166897_object24529:head# 2024-05-15T23:39:57.591 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68325c44:::benchmark_data_smithi090_166897_object3016:head# 2024-05-15T23:39:57.591 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:683404a2:::benchmark_data_smithi090_166897_object39226:head# 2024-05-15T23:39:57.592 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6834fed8:::benchmark_data_smithi090_166897_object61682:head# 2024-05-15T23:39:57.592 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68350b70:::benchmark_data_smithi090_166897_object19978:head# 2024-05-15T23:39:57.593 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68364fb5:::benchmark_data_smithi090_166897_object67322:head# 2024-05-15T23:39:57.593 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68373664:::benchmark_data_smithi090_166897_object28888:head# 2024-05-15T23:39:57.594 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:683793ab:::benchmark_data_smithi090_166897_object58924:head# 2024-05-15T23:39:57.594 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6837d60f:::benchmark_data_smithi090_166897_object45127:head# 2024-05-15T23:39:57.594 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68383152:::benchmark_data_smithi090_166897_object4661:head# 2024-05-15T23:39:57.595 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6839c03e:::benchmark_data_smithi090_166897_object10991:head# 2024-05-15T23:39:57.595 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:683af3e8:::benchmark_data_smithi090_166897_object7235:head# 2024-05-15T23:39:57.596 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:683bad7d:::benchmark_data_smithi090_166897_object28940:head# 2024-05-15T23:39:57.596 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:683bb6de:::benchmark_data_smithi090_166897_object61849:head# 2024-05-15T23:39:57.597 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:683ce5c3:::benchmark_data_smithi090_166897_object56533:head# 2024-05-15T23:39:57.597 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:683fd482:::benchmark_data_smithi090_166897_object13999:head# 2024-05-15T23:39:57.597 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6840f820:::benchmark_data_smithi090_166897_object4793:head# 2024-05-15T23:39:57.599 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:684283ed:::benchmark_data_smithi090_166897_object38421:head# 2024-05-15T23:39:57.599 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68470af2:::benchmark_data_smithi090_166897_object16041:head# 2024-05-15T23:39:57.600 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68479e9d:::benchmark_data_smithi090_166897_object60048:head# 2024-05-15T23:39:57.601 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68481436:::benchmark_data_smithi090_166897_object1446:head# 2024-05-15T23:39:57.601 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6849d50a:::benchmark_data_smithi090_166897_object36055:head# 2024-05-15T23:39:57.601 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:684a7585:::benchmark_data_smithi090_166897_object56326:head# 2024-05-15T23:39:57.602 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:684b77fd:::benchmark_data_smithi090_166897_object2811:head# 2024-05-15T23:39:57.602 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:684b9b40:::benchmark_data_smithi090_166897_object54157:head# 2024-05-15T23:39:57.603 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:684bb348:::benchmark_data_smithi090_166897_object65229:head# 2024-05-15T23:39:57.604 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:684c6a5f:::benchmark_data_smithi090_166897_object48032:head# 2024-05-15T23:39:57.604 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:684d1ffe:::benchmark_data_smithi090_166897_object57327:head# 2024-05-15T23:39:57.607 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:684d60bf:::benchmark_data_smithi090_166897_object44985:head# 2024-05-15T23:39:57.607 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:684dce68:::benchmark_data_smithi090_166897_object11479:head# 2024-05-15T23:39:57.607 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:685050cd:::benchmark_data_smithi090_166897_object46089:head# 2024-05-15T23:39:57.607 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68505fb0:::benchmark_data_smithi090_166897_object8388:head# 2024-05-15T23:39:57.608 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:685089ad:::benchmark_data_smithi090_166897_object27798:head# 2024-05-15T23:39:57.608 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68509ea1:::benchmark_data_smithi090_166897_object12300:head# 2024-05-15T23:39:57.608 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6850c1bb:::benchmark_data_smithi090_166897_object1622:head# 2024-05-15T23:39:57.609 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68531ef6:::benchmark_data_smithi090_166897_object12724:head# 2024-05-15T23:39:57.609 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6857cb49:::benchmark_data_smithi090_166897_object67946:head# 2024-05-15T23:39:57.610 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6857dfed:::benchmark_data_smithi090_166897_object10970:head# 2024-05-15T23:39:57.610 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6858954d:::benchmark_data_smithi090_166897_object27865:head# 2024-05-15T23:39:57.610 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68592e6c:::benchmark_data_smithi090_166897_object52274:head# 2024-05-15T23:39:57.611 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:685d2d28:::benchmark_data_smithi090_166897_object19807:head# 2024-05-15T23:39:57.611 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:685d8a8c:::benchmark_data_smithi090_166897_object71019:head# 2024-05-15T23:39:57.612 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:685e704f:::benchmark_data_smithi090_166897_object48014:head# 2024-05-15T23:39:57.612 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:685f8978:::benchmark_data_smithi090_166897_object35788:head# 2024-05-15T23:39:57.612 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:685fe610:::benchmark_data_smithi090_166897_object37973:head# 2024-05-15T23:39:57.613 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68627a7c:::benchmark_data_smithi090_166897_object63904:head# 2024-05-15T23:39:57.613 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6863c694:::benchmark_data_smithi090_166897_object43034:head# 2024-05-15T23:39:57.615 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68662fae:::benchmark_data_smithi090_166897_object41794:head# 2024-05-15T23:39:57.615 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68665bbb:::benchmark_data_smithi090_166897_object48973:head# 2024-05-15T23:39:57.616 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6866a872:::benchmark_data_smithi090_166897_object1955:head# 2024-05-15T23:39:57.616 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6866d88e:::benchmark_data_smithi090_166897_object13413:head# 2024-05-15T23:39:57.617 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6867c99b:::benchmark_data_smithi090_166897_object5822:head# 2024-05-15T23:39:57.617 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68686551:::benchmark_data_smithi090_166897_object61371:head# 2024-05-15T23:39:57.618 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6868b954:::benchmark_data_smithi090_166897_object50640:head# 2024-05-15T23:39:57.618 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68695a78:::benchmark_data_smithi090_166897_object64601:head# 2024-05-15T23:39:57.619 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6869bff7:::benchmark_data_smithi090_166897_object54060:head# 2024-05-15T23:39:57.619 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:686b4cf1:::benchmark_data_smithi090_166897_object19887:head# 2024-05-15T23:39:57.620 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:686b8892:::benchmark_data_smithi090_166897_object26953:head# 2024-05-15T23:39:57.620 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:686eb02a:::benchmark_data_smithi090_166897_object32726:head# 2024-05-15T23:39:57.620 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:686f498b:::benchmark_data_smithi090_166897_object10482:head# 2024-05-15T23:39:57.620 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68705b81:::benchmark_data_smithi090_166897_object10999:head# 2024-05-15T23:39:57.621 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:687130c8:::benchmark_data_smithi090_166897_object58782:head# 2024-05-15T23:39:57.621 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68713e97:::benchmark_data_smithi090_166897_object982:head# 2024-05-15T23:39:57.622 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68719665:::benchmark_data_smithi090_166897_object23000:head# 2024-05-15T23:39:57.622 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68720322:::benchmark_data_smithi090_166897_object55942:head# 2024-05-15T23:39:57.623 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6872a592:::benchmark_data_smithi090_166897_object34237:head# 2024-05-15T23:39:57.623 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6872fbc6:::benchmark_data_smithi090_166897_object13555:head# 2024-05-15T23:39:57.624 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68746145:::benchmark_data_smithi090_166897_object59034:head# 2024-05-15T23:39:57.624 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68758edb:::benchmark_data_smithi090_166897_object68572:head# 2024-05-15T23:39:57.625 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:687621a9:::benchmark_data_smithi090_166897_object51516:head# 2024-05-15T23:39:57.625 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6876b7f8:::benchmark_data_smithi090_166897_object9114:head# 2024-05-15T23:39:57.626 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68772327:::benchmark_data_smithi090_166897_object55631:head# 2024-05-15T23:39:57.626 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:687736fc:::benchmark_data_smithi090_166897_object64111:head# 2024-05-15T23:39:57.627 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6878b2e0:::benchmark_data_smithi090_166897_object43841:head# 2024-05-15T23:39:57.627 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6878dbd8:::benchmark_data_smithi090_166897_object26353:head# 2024-05-15T23:39:57.627 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:687941b9:::benchmark_data_smithi090_166897_object3238:head# 2024-05-15T23:39:57.628 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:687b1443:::benchmark_data_smithi090_166897_object63511:head# 2024-05-15T23:39:57.628 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:687d1830:::benchmark_data_smithi090_166897_object63561:head# 2024-05-15T23:39:57.628 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68807cc2:::benchmark_data_smithi090_166897_object8377:head# 2024-05-15T23:39:57.629 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68817ea3:::benchmark_data_smithi090_166897_object57080:head# 2024-05-15T23:39:57.629 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6881aaf8:::benchmark_data_smithi090_166897_object49254:head# 2024-05-15T23:39:57.630 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6883933c:::benchmark_data_smithi090_166897_object39600:head# 2024-05-15T23:39:57.630 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6883a52c:::benchmark_data_smithi090_166897_object27356:head# 2024-05-15T23:39:57.630 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6883e039:::benchmark_data_smithi090_166897_object51654:head# 2024-05-15T23:39:57.631 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68851a28:::benchmark_data_smithi090_166897_object1988:head# 2024-05-15T23:39:57.631 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:688569a1:::benchmark_data_smithi090_166897_object18465:head# 2024-05-15T23:39:57.632 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6886ab68:::benchmark_data_smithi090_166897_object55950:head# 2024-05-15T23:39:57.632 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68880f52:::benchmark_data_smithi090_166897_object5103:head# 2024-05-15T23:39:57.633 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68882d0f:::benchmark_data_smithi090_166897_object13976:head# 2024-05-15T23:39:57.633 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:688861fd:::benchmark_data_smithi090_166897_object3850:head# 2024-05-15T23:39:57.634 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68888688:::benchmark_data_smithi090_166897_object70264:head# 2024-05-15T23:39:57.634 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6888ade1:::benchmark_data_smithi090_166897_object18418:head# 2024-05-15T23:39:57.635 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6889e832:::benchmark_data_smithi090_166897_object47722:head# 2024-05-15T23:39:57.635 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:688ab657:::benchmark_data_smithi090_166897_object23649:head# 2024-05-15T23:39:57.636 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:688b3341:::benchmark_data_smithi090_166897_object2833:head# 2024-05-15T23:39:57.636 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:688b7159:::benchmark_data_smithi090_166897_object10859:head# 2024-05-15T23:39:57.637 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:688c85df:::benchmark_data_smithi090_166897_object68303:head# 2024-05-15T23:39:57.637 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:688cb299:::benchmark_data_smithi090_166897_object52690:head# 2024-05-15T23:39:57.638 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:688df29c:::benchmark_data_smithi090_166897_object30775:head# 2024-05-15T23:39:57.638 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:688e6be0:::benchmark_data_smithi090_166897_object44925:head# 2024-05-15T23:39:57.639 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:688f7bdc:::benchmark_data_smithi090_166897_object22079:head# 2024-05-15T23:39:57.639 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68902311:::benchmark_data_smithi090_166897_object43830:head# 2024-05-15T23:39:57.640 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6890cce6:::benchmark_data_smithi090_166897_object26265:head# 2024-05-15T23:39:57.640 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68910bee:::benchmark_data_smithi090_166897_object38332:head# 2024-05-15T23:39:57.640 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68911630:::benchmark_data_smithi090_166897_object30160:head# 2024-05-15T23:39:57.641 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6891fd56:::benchmark_data_smithi090_166897_object49214:head# 2024-05-15T23:39:57.642 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6892900c:::benchmark_data_smithi090_166897_object34174:head# 2024-05-15T23:39:57.642 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:689463d4:::benchmark_data_smithi090_166897_object43105:head# 2024-05-15T23:39:57.643 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6894afea:::benchmark_data_smithi090_166897_object22854:head# 2024-05-15T23:39:57.643 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68962a05:::benchmark_data_smithi090_166897_object62279:head# 2024-05-15T23:39:57.644 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6898cec7:::benchmark_data_smithi090_166897_object21113:head# 2024-05-15T23:39:57.645 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6899174f:::benchmark_data_smithi090_166897_object42250:head# 2024-05-15T23:39:57.645 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:689924a2:::benchmark_data_smithi090_166897_object34938:head# 2024-05-15T23:39:57.646 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6899ae3b:::benchmark_data_smithi090_166897_object26715:head# 2024-05-15T23:39:57.647 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6899b67f:::benchmark_data_smithi090_166897_object18896:head# 2024-05-15T23:39:57.647 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6899eb88:::benchmark_data_smithi090_166897_object12073:head# 2024-05-15T23:39:57.648 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:689a10c3:::benchmark_data_smithi090_166897_object28938:head# 2024-05-15T23:39:57.648 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:689a44a0:::benchmark_data_smithi090_166897_object23700:head# 2024-05-15T23:39:57.649 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:689acb09:::benchmark_data_smithi090_166897_object67408:head# 2024-05-15T23:39:57.649 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:689b30bc:::benchmark_data_smithi090_166897_object45357:head# 2024-05-15T23:39:57.649 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:689bf64a:::benchmark_data_smithi090_166897_object33473:head# 2024-05-15T23:39:57.650 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:689d6472:::benchmark_data_smithi090_166897_object55062:head# 2024-05-15T23:39:57.651 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:689da9cc:::benchmark_data_smithi090_166897_object28081:head# 2024-05-15T23:39:57.651 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:689de3d0:::benchmark_data_smithi090_166897_object27379:head# 2024-05-15T23:39:57.652 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:689fadbf:::benchmark_data_smithi090_166897_object5610:head# 2024-05-15T23:39:57.653 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68a1f1d8:::benchmark_data_smithi090_166897_object58132:head# 2024-05-15T23:39:57.653 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68a3d22e:::benchmark_data_smithi090_166897_object50693:head# 2024-05-15T23:39:57.654 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68a50338:::benchmark_data_smithi090_166897_object743:head# 2024-05-15T23:39:57.654 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68a5eb8c:::benchmark_data_smithi090_166897_object36452:head# 2024-05-15T23:39:57.654 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68a701a9:::benchmark_data_smithi090_166897_object54002:head# 2024-05-15T23:39:57.655 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68a9e2a2:::benchmark_data_smithi090_166897_object7049:head# 2024-05-15T23:39:57.655 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68abc8d7:::benchmark_data_smithi090_166897_object48756:head# 2024-05-15T23:39:57.656 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68abf870:::benchmark_data_smithi090_166897_object46463:head# 2024-05-15T23:39:57.658 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68ac27cd:::benchmark_data_smithi090_166897_object66187:head# 2024-05-15T23:39:57.658 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68ac6389:::benchmark_data_smithi090_166897_object38120:head# 2024-05-15T23:39:57.659 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68ace43b:::benchmark_data_smithi090_166897_object63253:head# 2024-05-15T23:39:57.659 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68b051d8:::benchmark_data_smithi090_166897_object12566:head# 2024-05-15T23:39:57.659 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68b0aadc:::benchmark_data_smithi090_166897_object12898:head# 2024-05-15T23:39:57.660 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68b20de2:::benchmark_data_smithi090_166897_object59350:head# 2024-05-15T23:39:57.660 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68b22cc6:::benchmark_data_smithi090_166897_object66341:head# 2024-05-15T23:39:57.661 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68b2840e:::benchmark_data_smithi090_166897_object18069:head# 2024-05-15T23:39:57.661 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68b2cba2:::benchmark_data_smithi090_166897_object40127:head# 2024-05-15T23:39:57.661 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68b5b7a4:::benchmark_data_smithi090_166897_object7922:head# 2024-05-15T23:39:57.662 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68b608db:::benchmark_data_smithi090_166897_object69686:head# 2024-05-15T23:39:57.662 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68b63f1d:::benchmark_data_smithi090_166897_object48647:head# 2024-05-15T23:39:57.662 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68b8930d:::benchmark_data_smithi090_166897_object29931:head# 2024-05-15T23:39:57.663 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68b8ca1b:::benchmark_data_smithi090_166897_object71036:head# 2024-05-15T23:39:57.663 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68b9723e:::benchmark_data_smithi090_166897_object54595:head# 2024-05-15T23:39:57.664 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68bad3f8:::benchmark_data_smithi090_166897_object18222:head# 2024-05-15T23:39:57.665 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68baff1a:::benchmark_data_smithi090_166897_object41548:head# 2024-05-15T23:39:57.665 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68bb11dd:::benchmark_data_smithi090_166897_object44756:head# 2024-05-15T23:39:57.665 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68bb8f10:::benchmark_data_smithi090_166897_object15505:head# 2024-05-15T23:39:57.666 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68bc447f:::benchmark_data_smithi090_166897_object20821:head# 2024-05-15T23:39:57.666 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68bd8095:::benchmark_data_smithi090_166897_object29180:head# 2024-05-15T23:39:57.667 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68bedd14:::benchmark_data_smithi090_166897_object6755:head# 2024-05-15T23:39:57.667 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68bef27a:::benchmark_data_smithi090_166897_object29623:head# 2024-05-15T23:39:57.667 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68c00861:::benchmark_data_smithi090_166897_object10568:head# 2024-05-15T23:39:57.668 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68c0da69:::benchmark_data_smithi090_166897_object10905:head# 2024-05-15T23:39:57.668 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68c27514:::benchmark_data_smithi090_166897_object50993:head# 2024-05-15T23:39:57.669 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68c371af:::benchmark_data_smithi090_166897_object70359:head# 2024-05-15T23:39:57.669 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68c4a5ef:::benchmark_data_smithi090_166897_object8311:head# 2024-05-15T23:39:57.669 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68c4f26f:::benchmark_data_smithi090_166897_object54670:head# 2024-05-15T23:39:57.670 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68c54edd:::benchmark_data_smithi090_166897_object1317:head# 2024-05-15T23:39:57.671 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68c6a752:::benchmark_data_smithi090_166897_object22943:head# 2024-05-15T23:39:57.671 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68c74895:::benchmark_data_smithi090_166897_object65281:head# 2024-05-15T23:39:57.672 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68c775e3:::benchmark_data_smithi090_166897_object3093:head# 2024-05-15T23:39:57.672 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68c82170:::benchmark_data_smithi090_166897_object64326:head# 2024-05-15T23:39:57.672 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68c901a5:::benchmark_data_smithi090_166897_object34784:head# 2024-05-15T23:39:57.673 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68c94848:::benchmark_data_smithi090_166897_object2869:head# 2024-05-15T23:39:57.673 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68c9a752:::benchmark_data_smithi090_166897_object63009:head# 2024-05-15T23:39:57.674 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68ca6305:::benchmark_data_smithi090_166897_object18424:head# 2024-05-15T23:39:57.674 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68ca8f1d:::benchmark_data_smithi090_166897_object67736:head# 2024-05-15T23:39:57.675 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68cb08a0:::benchmark_data_smithi090_166897_object20876:head# 2024-05-15T23:39:57.675 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68cb16d0:::benchmark_data_smithi090_166897_object61762:head# 2024-05-15T23:39:57.676 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68cb2b16:::benchmark_data_smithi090_166897_object2825:head# 2024-05-15T23:39:57.676 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68cb3f51:::benchmark_data_smithi090_166897_object17616:head# 2024-05-15T23:39:57.677 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68cb506a:::benchmark_data_smithi090_166897_object31227:head# 2024-05-15T23:39:57.677 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68cb569f:::benchmark_data_smithi090_166897_object58953:head# 2024-05-15T23:39:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68cba4bf:::benchmark_data_smithi090_166897_object39631:head# 2024-05-15T23:39:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68cc2cac:::benchmark_data_smithi090_166897_object8010:head# 2024-05-15T23:39:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68cd1134:::benchmark_data_smithi090_166897_object6627:head# 2024-05-15T23:39:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68cda7d8:::benchmark_data_smithi090_166897_object29766:head# 2024-05-15T23:39:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68cde388:::benchmark_data_smithi090_166897_object60261:head# 2024-05-15T23:39:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68cfa734:::benchmark_data_smithi090_166897_object9681:head# 2024-05-15T23:39:57.682 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68d10211:::benchmark_data_smithi090_166897_object68261:head# 2024-05-15T23:39:57.682 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68d3a072:::benchmark_data_smithi090_166897_object3958:head# 2024-05-15T23:39:57.682 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68d699e1:::benchmark_data_smithi090_166897_object36178:head# 2024-05-15T23:39:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68d6f9a9:::benchmark_data_smithi090_166897_object12998:head# 2024-05-15T23:39:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68d72855:::benchmark_data_smithi090_166897_object15869:head# 2024-05-15T23:39:57.684 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68d7c878:::benchmark_data_smithi090_166897_object20422:head# 2024-05-15T23:39:57.684 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68d863dd:::benchmark_data_smithi090_166897_object18027:head# 2024-05-15T23:39:57.685 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68d91573:::benchmark_data_smithi090_166897_object34576:head# 2024-05-15T23:39:57.685 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68d9a3c0:::benchmark_data_smithi090_166897_object17055:head# 2024-05-15T23:39:57.686 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68d9f56d:::benchmark_data_smithi090_166897_object59811:head# 2024-05-15T23:39:57.686 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68da0ef4:::benchmark_data_smithi090_166897_object63925:head# 2024-05-15T23:39:57.687 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68dbb1c5:::benchmark_data_smithi090_166897_object9979:head# 2024-05-15T23:39:57.687 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68dd98a9:::benchmark_data_smithi090_166897_object36658:head# 2024-05-15T23:39:57.688 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68ddd5a1:::benchmark_data_smithi090_166897_object64275:head# 2024-05-15T23:39:57.688 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68e04fb3:::benchmark_data_smithi090_166897_object11908:head# 2024-05-15T23:39:57.689 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68e0c151:::benchmark_data_smithi090_166897_object31656:head# 2024-05-15T23:39:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68e1547b:::benchmark_data_smithi090_166897_object65459:head# 2024-05-15T23:39:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68e26ff3:::benchmark_data_smithi090_166897_object48415:head# 2024-05-15T23:39:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68e2a49d:::benchmark_data_smithi090_166897_object51732:head# 2024-05-15T23:39:57.691 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68e3c5a4:::benchmark_data_smithi090_166897_object35756:head# 2024-05-15T23:39:57.691 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68e534ce:::benchmark_data_smithi090_166897_object48450:head# 2024-05-15T23:39:57.691 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68e55f5f:::benchmark_data_smithi090_166897_object61716:head# 2024-05-15T23:39:57.692 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68e58728:::benchmark_data_smithi090_166897_object25290:head# 2024-05-15T23:39:57.692 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68e7b86d:::benchmark_data_smithi090_166897_object17051:head# 2024-05-15T23:39:57.693 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68e83c64:::benchmark_data_smithi090_166897_object58142:head# 2024-05-15T23:39:57.693 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68e88296:::benchmark_data_smithi090_166897_object4280:head# 2024-05-15T23:39:57.693 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68e8e6bb:::benchmark_data_smithi090_166897_object3292:head# 2024-05-15T23:39:57.694 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68eb089a:::benchmark_data_smithi090_166897_object50531:head# 2024-05-15T23:39:57.694 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68ebc35b:::benchmark_data_smithi090_166897_object17890:head# 2024-05-15T23:39:57.695 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68ebf869:::benchmark_data_smithi090_166897_object37905:head# 2024-05-15T23:39:57.695 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68eca068:::benchmark_data_smithi090_166897_object66564:head# 2024-05-15T23:39:57.696 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68ef6614:::benchmark_data_smithi090_166897_object30516:head# 2024-05-15T23:39:57.696 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68f0c91e:::benchmark_data_smithi090_166897_object57934:head# 2024-05-15T23:39:57.697 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68f1e881:::benchmark_data_smithi090_166897_object65034:head# 2024-05-15T23:39:57.697 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68f1fdbe:::benchmark_data_smithi090_166897_object33585:head# 2024-05-15T23:39:57.698 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68f22f50:::benchmark_data_smithi090_166897_object67895:head# 2024-05-15T23:39:57.698 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68f33fe9:::benchmark_data_smithi090_166897_object64579:head# 2024-05-15T23:39:57.699 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68f34391:::benchmark_data_smithi090_166897_object43700:head# 2024-05-15T23:39:57.700 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68f443a9:::benchmark_data_smithi090_166897_object21294:head# 2024-05-15T23:39:57.700 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68f57424:::benchmark_data_smithi090_166897_object37865:head# 2024-05-15T23:39:57.700 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68f589ec:::benchmark_data_smithi090_166897_object39716:head# 2024-05-15T23:39:57.701 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68f7824c:::benchmark_data_smithi090_166897_object54842:head# 2024-05-15T23:39:57.701 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68f83df3:::benchmark_data_smithi090_166897_object61149:head# 2024-05-15T23:39:57.701 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68f8c822:::benchmark_data_smithi090_166897_object14156:head# 2024-05-15T23:39:57.702 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68f9b85f:::benchmark_data_smithi090_166897_object35650:head# 2024-05-15T23:39:57.702 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68fc2a2f:::benchmark_data_smithi090_166897_object1146:head# 2024-05-15T23:39:57.702 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68fd4077:::benchmark_data_smithi090_166897_object17952:head# 2024-05-15T23:39:57.703 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68fdb50c:::benchmark_data_smithi090_166897_object296:head# 2024-05-15T23:39:57.704 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68fe2bea:::benchmark_data_smithi090_166897_object22353:head# 2024-05-15T23:39:57.704 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68fe5f85:::benchmark_data_smithi090_166897_object61152:head# 2024-05-15T23:39:57.705 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68fef56a:::benchmark_data_smithi090_166897_object32480:head# 2024-05-15T23:39:57.705 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:68ff9db1:::benchmark_data_smithi090_166897_object3170:head# 2024-05-15T23:39:57.706 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69006905:::benchmark_data_smithi090_166897_object31261:head# 2024-05-15T23:39:57.706 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:690089da:::benchmark_data_smithi090_166897_object48300:head# 2024-05-15T23:39:57.707 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6903172b:::benchmark_data_smithi090_166897_object63918:head# 2024-05-15T23:39:57.707 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6903193d:::benchmark_data_smithi090_166897_object9075:head# 2024-05-15T23:39:57.707 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6903659c:::benchmark_data_smithi090_166897_object7340:head# 2024-05-15T23:39:57.708 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6903fddf:::benchmark_data_smithi090_166897_object2341:head# 2024-05-15T23:39:57.708 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69041481:::benchmark_data_smithi090_166897_object41582:head# 2024-05-15T23:39:57.709 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6904c877:::benchmark_data_smithi090_166897_object30965:head# 2024-05-15T23:39:57.709 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6905897b:::benchmark_data_smithi090_166897_object9364:head# 2024-05-15T23:39:57.710 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6905cff3:::benchmark_data_smithi090_166897_object30512:head# 2024-05-15T23:39:57.710 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6905e151:::benchmark_data_smithi090_166897_object22842:head# 2024-05-15T23:39:57.710 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:690681a1:::benchmark_data_smithi090_166897_object52157:head# 2024-05-15T23:39:57.713 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:690733d3:::benchmark_data_smithi090_166897_object68539:head# 2024-05-15T23:39:57.713 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69091091:::benchmark_data_smithi090_166897_object63147:head# 2024-05-15T23:39:57.714 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6909d99f:::benchmark_data_smithi090_166897_object67449:head# 2024-05-15T23:39:57.715 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:690a55b2:::benchmark_data_smithi090_166897_object29643:head# 2024-05-15T23:39:57.715 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:690a9c4c:::benchmark_data_smithi090_166897_object37392:head# 2024-05-15T23:39:57.715 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69131d8c:::benchmark_data_smithi090_166897_object13850:head# 2024-05-15T23:39:57.716 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69136da6:::benchmark_data_smithi090_166897_object813:head# 2024-05-15T23:39:57.716 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:691438e6:::benchmark_data_smithi090_166897_object14996:head# 2024-05-15T23:39:57.717 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6914459a:::benchmark_data_smithi090_166897_object50958:head# 2024-05-15T23:39:57.719 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6914c4bd:::benchmark_data_smithi090_166897_object70667:head# 2024-05-15T23:39:57.719 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69154cdc:::benchmark_data_smithi090_166897_object46048:head# 2024-05-15T23:39:57.720 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:691579db:::benchmark_data_smithi090_166897_object44328:head# 2024-05-15T23:39:57.720 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6917b9ac:::benchmark_data_smithi090_166897_object45778:head# 2024-05-15T23:39:57.721 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:691821ad:::benchmark_data_smithi090_166897_object59864:head# 2024-05-15T23:39:57.721 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6918d350:::benchmark_data_smithi090_166897_object69088:head# 2024-05-15T23:39:57.721 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69193d15:::benchmark_data_smithi090_166897_object31387:head# 2024-05-15T23:39:57.722 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6919897b:::benchmark_data_smithi090_166897_object45830:head# 2024-05-15T23:39:57.722 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6919fadf:::benchmark_data_smithi090_166897_object12239:head# 2024-05-15T23:39:57.723 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:691b3a78:::benchmark_data_smithi090_166897_object24861:head# 2024-05-15T23:39:57.723 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:691cb19a:::benchmark_data_smithi090_166897_object56965:head# 2024-05-15T23:39:57.724 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:691e6728:::benchmark_data_smithi090_166897_object34603:head# 2024-05-15T23:39:57.724 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:691f7af5:::benchmark_data_smithi090_166897_object34948:head# 2024-05-15T23:39:57.725 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6921378b:::benchmark_data_smithi090_166897_object37748:head# 2024-05-15T23:39:57.725 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:692155ed:::benchmark_data_smithi090_166897_object47004:head# 2024-05-15T23:39:57.725 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69218ab2:::benchmark_data_smithi090_166897_object16854:head# 2024-05-15T23:39:57.726 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6922387d:::benchmark_data_smithi090_166897_object70002:head# 2024-05-15T23:39:57.726 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:692327ae:::benchmark_data_smithi090_166897_object44530:head# 2024-05-15T23:39:57.727 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69232c28:::benchmark_data_smithi090_166897_object56931:head# 2024-05-15T23:39:57.728 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:692365ed:::benchmark_data_smithi090_166897_object33885:head# 2024-05-15T23:39:57.728 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:692431d4:::benchmark_data_smithi090_166897_object9016:head# 2024-05-15T23:39:57.728 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6924dfcb:::benchmark_data_smithi090_166897_object28862:head# 2024-05-15T23:39:57.729 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6925ee67:::benchmark_data_smithi090_166897_object377:head# 2024-05-15T23:39:57.729 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:692bd369:::benchmark_data_smithi090_166897_object19086:head# 2024-05-15T23:39:57.730 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:692c1345:::benchmark_data_smithi090_166897_object35326:head# 2024-05-15T23:39:57.730 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:692cd662:::benchmark_data_smithi090_166897_object16764:head# 2024-05-15T23:39:57.731 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:692f2dbf:::benchmark_data_smithi090_166897_object70459:head# 2024-05-15T23:39:57.731 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:692f7806:::benchmark_data_smithi090_166897_object56863:head# 2024-05-15T23:39:57.732 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:693353a8:::benchmark_data_smithi090_166897_object42695:head# 2024-05-15T23:39:57.732 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:693368be:::benchmark_data_smithi090_166897_object1314:head# 2024-05-15T23:39:57.733 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6935759f:::benchmark_data_smithi090_166897_object70774:head# 2024-05-15T23:39:57.733 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6936ab06:::benchmark_data_smithi090_166897_object37298:head# 2024-05-15T23:39:57.733 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6936d260:::benchmark_data_smithi090_166897_object65264:head# 2024-05-15T23:39:57.734 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:693aa704:::benchmark_data_smithi090_166897_object33935:head# 2024-05-15T23:39:57.734 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:693b0269:::benchmark_data_smithi090_166897_object48950:head# 2024-05-15T23:39:57.735 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:693ba396:::benchmark_data_smithi090_166897_object53622:head# 2024-05-15T23:39:57.735 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:693e3d57:::benchmark_data_smithi090_166897_object14786:head# 2024-05-15T23:39:57.736 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:693f4536:::benchmark_data_smithi090_166897_object20186:head# 2024-05-15T23:39:57.736 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6942d647:::benchmark_data_smithi090_166897_object9282:head# 2024-05-15T23:39:57.737 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69430b2f:::benchmark_data_smithi090_166897_object49543:head# 2024-05-15T23:39:57.737 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:694386ba:::benchmark_data_smithi090_166897_object3051:head# 2024-05-15T23:39:57.737 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6943a353:::benchmark_data_smithi090_166897_object8887:head# 2024-05-15T23:39:57.738 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6943d770:::benchmark_data_smithi090_166897_object61555:head# 2024-05-15T23:39:57.738 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:694431a3:::benchmark_data_smithi090_166897_object43608:head# 2024-05-15T23:39:57.739 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69485dec:::benchmark_data_smithi090_166897_object59022:head# 2024-05-15T23:39:57.739 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:694936e9:::benchmark_data_smithi090_166897_object19447:head# 2024-05-15T23:39:57.740 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69493bcb:::benchmark_data_smithi090_166897_object57318:head# 2024-05-15T23:39:57.740 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69495619:::benchmark_data_smithi090_166897_object43844:head# 2024-05-15T23:39:57.741 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6949b11e:::benchmark_data_smithi090_166897_object65727:head# 2024-05-15T23:39:57.741 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:694abdb3:::benchmark_data_smithi090_166897_object43741:head# 2024-05-15T23:39:57.742 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:694b69a3:::benchmark_data_smithi090_166897_object48436:head# 2024-05-15T23:39:57.742 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:694ba53f:::benchmark_data_smithi090_166897_object46503:head# 2024-05-15T23:39:57.742 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:694ead04:::benchmark_data_smithi090_166897_object48596:head# 2024-05-15T23:39:57.743 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6954597d:::benchmark_data_smithi090_166897_object14242:head# 2024-05-15T23:39:57.743 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6954e43c:::benchmark_data_smithi090_166897_object68004:head# 2024-05-15T23:39:57.744 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:695559fb:::benchmark_data_smithi090_166897_object24777:head# 2024-05-15T23:39:57.744 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:695625d3:::benchmark_data_smithi090_166897_object2478:head# 2024-05-15T23:39:57.744 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69580ec2:::benchmark_data_smithi090_166897_object56601:head# 2024-05-15T23:39:57.745 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69582245:::benchmark_data_smithi090_166897_object24728:head# 2024-05-15T23:39:57.745 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69589950:::benchmark_data_smithi090_166897_object22331:head# 2024-05-15T23:39:57.746 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:695a94a8:::benchmark_data_smithi090_166897_object48565:head# 2024-05-15T23:39:57.746 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:695abfb3:::benchmark_data_smithi090_166897_object52977:head# 2024-05-15T23:39:57.747 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:695b6754:::benchmark_data_smithi090_166897_object31935:head# 2024-05-15T23:39:57.747 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:695be8a6:::benchmark_data_smithi090_166897_object12422:head# 2024-05-15T23:39:57.748 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:695cd563:::benchmark_data_smithi090_166897_object2496:head# 2024-05-15T23:39:57.752 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69617df1:::benchmark_data_smithi090_166897_object38972:head# 2024-05-15T23:39:57.752 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:696184c0:::benchmark_data_smithi090_166897_object27385:head# 2024-05-15T23:39:57.752 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6961f791:::benchmark_data_smithi090_166897_object60043:head# 2024-05-15T23:39:57.752 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:696353f8:::benchmark_data_smithi090_166897_object44413:head# 2024-05-15T23:39:57.752 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6963551d:::benchmark_data_smithi090_166897_object11030:head# 2024-05-15T23:39:57.752 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6964570b:::benchmark_data_smithi090_166897_object63360:head# 2024-05-15T23:39:57.752 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:696592e6:::benchmark_data_smithi090_166897_object22281:head# 2024-05-15T23:39:57.752 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69664953:::benchmark_data_smithi090_166897_object55343:head# 2024-05-15T23:39:57.753 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6966bd6b:::benchmark_data_smithi090_166897_object40151:head# 2024-05-15T23:39:57.753 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:696707e2:::benchmark_data_smithi090_166897_object59586:head# 2024-05-15T23:39:57.753 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69679216:::benchmark_data_smithi090_166897_object36131:head# 2024-05-15T23:39:57.753 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69689225:::benchmark_data_smithi090_166897_object29962:head# 2024-05-15T23:39:57.754 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6969e791:::benchmark_data_smithi090_166897_object16526:head# 2024-05-15T23:39:57.754 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:696c651d:::benchmark_data_smithi090_166897_object51022:head# 2024-05-15T23:39:57.754 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:696ce774:::benchmark_data_smithi090_166897_object66110:head# 2024-05-15T23:39:57.755 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:696ce7b8:::benchmark_data_smithi090_166897_object18511:head# 2024-05-15T23:39:57.755 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:696f04d3:::benchmark_data_smithi090_166897_object43391:head# 2024-05-15T23:39:57.755 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:696fd9e3:::benchmark_data_smithi090_166897_object66254:head# 2024-05-15T23:39:57.756 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:696ff249:::benchmark_data_smithi090_166897_object32081:head# 2024-05-15T23:39:57.756 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:697123f8:::benchmark_data_smithi090_166897_object64094:head# 2024-05-15T23:39:57.756 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69716677:::benchmark_data_smithi090_166897_object40801:head# 2024-05-15T23:39:57.757 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69721512:::benchmark_data_smithi090_166897_object70257:head# 2024-05-15T23:39:57.757 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6977135c:::benchmark_data_smithi090_166897_object63671:head# 2024-05-15T23:39:57.758 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6977d708:::benchmark_data_smithi090_166897_object65272:head# 2024-05-15T23:39:57.758 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:697abf16:::benchmark_data_smithi090_166897_object102:head# 2024-05-15T23:39:57.759 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:697ae5fc:::benchmark_data_smithi090_166897_object37372:head# 2024-05-15T23:39:57.759 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:697b5065:::benchmark_data_smithi090_166897_object8016:head# 2024-05-15T23:39:57.759 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:697bbd53:::benchmark_data_smithi090_166897_object67388:head# 2024-05-15T23:39:57.760 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:697be02b:::benchmark_data_smithi090_166897_object67987:head# 2024-05-15T23:39:57.760 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:697e2e22:::benchmark_data_smithi090_166897_object54118:head# 2024-05-15T23:39:57.762 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:697e6e91:::benchmark_data_smithi090_166897_object7566:head# 2024-05-15T23:39:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:697f0dcd:::benchmark_data_smithi090_166897_object17950:head# 2024-05-15T23:39:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:697f1f18:::benchmark_data_smithi090_166897_object37924:head# 2024-05-15T23:39:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6980eed0:::benchmark_data_smithi090_166897_object55283:head# 2024-05-15T23:39:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6980f268:::benchmark_data_smithi090_166897_object48900:head# 2024-05-15T23:39:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6981affa:::benchmark_data_smithi090_166897_object68156:head# 2024-05-15T23:39:57.764 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6984b5b0:::benchmark_data_smithi090_166897_object64954:head# 2024-05-15T23:39:57.764 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69861e4e:::benchmark_data_smithi090_166897_object68668:head# 2024-05-15T23:39:57.765 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:698631ec:::benchmark_data_smithi090_166897_object43663:head# 2024-05-15T23:39:57.765 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:698959b8:::benchmark_data_smithi090_166897_object2597:head# 2024-05-15T23:39:57.766 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:698a0dcd:::benchmark_data_smithi090_166897_object44670:head# 2024-05-15T23:39:57.766 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:698a35ee:::benchmark_data_smithi090_166897_object43770:head# 2024-05-15T23:39:57.766 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:698db010:::benchmark_data_smithi090_166897_object28771:head# 2024-05-15T23:39:57.767 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:698f0a6b:::benchmark_data_smithi090_166897_object51723:head# 2024-05-15T23:39:57.767 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69906023:::benchmark_data_smithi090_166897_object36380:head# 2024-05-15T23:39:57.768 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6990ea90:::benchmark_data_smithi090_166897_object61207:head# 2024-05-15T23:39:57.768 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6992af4b:::benchmark_data_smithi090_166897_object26849:head# 2024-05-15T23:39:57.769 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6992f9c8:::benchmark_data_smithi090_166897_object15385:head# 2024-05-15T23:39:57.769 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6993c00a:::benchmark_data_smithi090_166897_object16466:head# 2024-05-15T23:39:57.769 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:699461fc:::benchmark_data_smithi090_166897_object12147:head# 2024-05-15T23:39:57.770 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6994b0c9:::benchmark_data_smithi090_166897_object16190:head# 2024-05-15T23:39:57.770 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:699542e0:::benchmark_data_smithi090_166897_object41505:head# 2024-05-15T23:39:57.771 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6996a18f:::benchmark_data_smithi090_166897_object2910:head# 2024-05-15T23:39:57.771 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6996f14f:::benchmark_data_smithi090_166897_object41229:head# 2024-05-15T23:39:57.772 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6998534e:::benchmark_data_smithi090_166897_object56322:head# 2024-05-15T23:39:57.772 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69987545:::benchmark_data_smithi090_166897_object10139:head# 2024-05-15T23:39:57.773 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6998935b:::benchmark_data_smithi090_166897_object28243:head# 2024-05-15T23:39:57.773 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6999d48a:::benchmark_data_smithi090_166897_object51121:head# 2024-05-15T23:39:57.774 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:699a0b53:::benchmark_data_smithi090_166897_object5507:head# 2024-05-15T23:39:57.774 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:699b98fe:::benchmark_data_smithi090_166897_object64411:head# 2024-05-15T23:39:57.775 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:699be1c5:::benchmark_data_smithi090_166897_object8575:head# 2024-05-15T23:39:57.775 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:699bf465:::benchmark_data_smithi090_166897_object16047:head# 2024-05-15T23:39:57.776 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:699c8774:::benchmark_data_smithi090_166897_object57234:head# 2024-05-15T23:39:57.777 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:699d158f:::benchmark_data_smithi090_166897_object33566:head# 2024-05-15T23:39:57.777 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:699e1bf2:::benchmark_data_smithi090_166897_object27369:head# 2024-05-15T23:39:57.777 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:699ece7e:::benchmark_data_smithi090_166897_object29921:head# 2024-05-15T23:39:57.778 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:699f4ea7:::benchmark_data_smithi090_166897_object39620:head# 2024-05-15T23:39:57.778 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69a16ea6:::benchmark_data_smithi090_166897_object59292:head# 2024-05-15T23:39:57.779 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69a26369:::benchmark_data_smithi090_166897_object22338:head# 2024-05-15T23:39:57.779 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69a38088:::benchmark_data_smithi090_166897_object66121:head# 2024-05-15T23:39:57.779 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69a38bc3:::benchmark_data_smithi090_166897_object52636:head# 2024-05-15T23:39:57.780 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69a3bdde:::benchmark_data_smithi090_166897_object39464:head# 2024-05-15T23:39:57.780 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69a4229d:::benchmark_data_smithi090_166897_object3381:head# 2024-05-15T23:39:57.781 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69a47f1a:::benchmark_data_smithi090_166897_object42088:head# 2024-05-15T23:39:57.781 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69a5ea15:::benchmark_data_smithi090_166897_object20663:head# 2024-05-15T23:39:57.782 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69a6164c:::benchmark_data_smithi090_166897_object4223:head# 2024-05-15T23:39:57.782 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69a6dd22:::benchmark_data_smithi090_166897_object25409:head# 2024-05-15T23:39:57.783 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69a7968b:::benchmark_data_smithi090_166897_object62225:head# 2024-05-15T23:39:57.783 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69a8c229:::benchmark_data_smithi090_166897_object25138:head# 2024-05-15T23:39:57.783 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69a914fb:::benchmark_data_smithi090_166897_object697:head# 2024-05-15T23:39:57.784 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69a96b59:::benchmark_data_smithi090_166897_object45211:head# 2024-05-15T23:39:57.784 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69a97041:::benchmark_data_smithi090_166897_object16729:head# 2024-05-15T23:39:57.787 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69aacbdb:::benchmark_data_smithi090_166897_object36438:head# 2024-05-15T23:39:57.787 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69ac3d52:::benchmark_data_smithi090_166897_object53000:head# 2024-05-15T23:39:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69aca965:::benchmark_data_smithi090_166897_object20932:head# 2024-05-15T23:39:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69acba68:::benchmark_data_smithi090_166897_object56162:head# 2024-05-15T23:39:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69ad3474:::benchmark_data_smithi090_166897_object25442:head# 2024-05-15T23:39:57.789 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69ad4549:::benchmark_data_smithi090_166897_object21428:head# 2024-05-15T23:39:57.789 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69adcd88:::benchmark_data_smithi090_166897_object31757:head# 2024-05-15T23:39:57.791 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69aee348:::benchmark_data_smithi090_166897_object36881:head# 2024-05-15T23:39:57.791 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69af5cb8:::benchmark_data_smithi090_166897_object63914:head# 2024-05-15T23:39:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69af7645:::benchmark_data_smithi090_166897_object9790:head# 2024-05-15T23:39:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69afcd5d:::benchmark_data_smithi090_166897_object49335:head# 2024-05-15T23:39:57.793 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69afdf62:::benchmark_data_smithi090_166897_object31558:head# 2024-05-15T23:39:57.793 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b00517:::benchmark_data_smithi090_166897_object67088:head# 2024-05-15T23:39:57.794 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b16d20:::benchmark_data_smithi090_166897_object9331:head# 2024-05-15T23:39:57.794 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b1c866:::benchmark_data_smithi090_166897_object13564:head# 2024-05-15T23:39:57.794 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b3ad0a:::benchmark_data_smithi090_166897_object15069:head# 2024-05-15T23:39:57.795 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b41531:::benchmark_data_smithi090_166897_object17523:head# 2024-05-15T23:39:57.795 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b4bedf:::benchmark_data_smithi090_166897_object22702:head# 2024-05-15T23:39:57.796 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b50d6c:::benchmark_data_smithi090_166897_object54370:head# 2024-05-15T23:39:57.797 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b571f8:::benchmark_data_smithi090_166897_object34908:head# 2024-05-15T23:39:57.797 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b5f285:::benchmark_data_smithi090_166897_object43273:head# 2024-05-15T23:39:57.797 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b66e5e:::benchmark_data_smithi090_166897_object57649:head# 2024-05-15T23:39:57.798 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b68e9d:::benchmark_data_smithi090_166897_object59279:head# 2024-05-15T23:39:57.798 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b7715b:::benchmark_data_smithi090_166897_object14148:head# 2024-05-15T23:39:57.799 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b7ee42:::benchmark_data_smithi090_166897_object36016:head# 2024-05-15T23:39:57.799 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b7f6b3:::benchmark_data_smithi090_166897_object6156:head# 2024-05-15T23:39:57.799 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b8525b:::benchmark_data_smithi090_166897_object69367:head# 2024-05-15T23:39:57.800 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b8c3ae:::benchmark_data_smithi090_166897_object47408:head# 2024-05-15T23:39:57.800 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69b98b74:::benchmark_data_smithi090_166897_object29519:head# 2024-05-15T23:39:57.801 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69bc2ed7:::benchmark_data_smithi090_166897_object3551:head# 2024-05-15T23:39:57.801 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69bcf21a:::benchmark_data_smithi090_166897_object42135:head# 2024-05-15T23:39:57.802 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69be0ba9:::benchmark_data_smithi090_166897_object51438:head# 2024-05-15T23:39:57.802 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69becfbf:::benchmark_data_smithi090_166897_object8691:head# 2024-05-15T23:39:57.803 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69bf475d:::benchmark_data_smithi090_166897_object70047:head# 2024-05-15T23:39:57.803 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69bf935d:::benchmark_data_smithi090_166897_object15280:head# 2024-05-15T23:39:57.804 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69bfee4e:::benchmark_data_smithi090_166897_object63580:head# 2024-05-15T23:39:57.805 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69c058a7:::benchmark_data_smithi090_166897_object34839:head# 2024-05-15T23:39:57.805 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69c19795:::benchmark_data_smithi090_166897_object3431:head# 2024-05-15T23:39:57.805 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69c1983a:::benchmark_data_smithi090_166897_object40883:head# 2024-05-15T23:39:57.806 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69c21108:::benchmark_data_smithi090_166897_object31861:head# 2024-05-15T23:39:57.806 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69c276dc:::benchmark_data_smithi090_166897_object41911:head# 2024-05-15T23:39:57.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:57 smithi090 ceph-mon[44434]: pgmap v2020: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s; 15243/169206 objects degraded (9.009%); 11 MiB/s, 180 objects/s recovering 2024-05-15T23:39:57.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:57 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 15922/162842 objects degraded (9.778%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:39:57.808 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69c5d468:::benchmark_data_smithi090_166897_object23877:head# 2024-05-15T23:39:57.808 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69c86206:::benchmark_data_smithi090_166897_object61206:head# 2024-05-15T23:39:57.808 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69c8c42e:::benchmark_data_smithi090_166897_object69318:head# 2024-05-15T23:39:57.809 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69c8d507:::benchmark_data_smithi090_166897_object42281:head# 2024-05-15T23:39:57.809 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69c945cb:::benchmark_data_smithi090_166897_object40375:head# 2024-05-15T23:39:57.810 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69c9b0a1:::benchmark_data_smithi090_166897_object15053:head# 2024-05-15T23:39:57.813 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69caf5a1:::benchmark_data_smithi090_166897_object12809:head# 2024-05-15T23:39:57.813 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69ce9406:::benchmark_data_smithi090_166897_object65665:head# 2024-05-15T23:39:57.814 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69cef1c1:::benchmark_data_smithi090_166897_object4623:head# 2024-05-15T23:39:57.814 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d04748:::benchmark_data_smithi090_166897_object33490:head# 2024-05-15T23:39:57.814 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d1f74c:::benchmark_data_smithi090_166897_object54729:head# 2024-05-15T23:39:57.815 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d1f93f:::benchmark_data_smithi090_166897_object14701:head# 2024-05-15T23:39:57.815 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d209fc:::benchmark_data_smithi090_166897_object32529:head# 2024-05-15T23:39:57.815 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d27867:::benchmark_data_smithi090_166897_object57306:head# 2024-05-15T23:39:57.816 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d37634:::benchmark_data_smithi090_166897_object53214:head# 2024-05-15T23:39:57.817 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d3bf82:::benchmark_data_smithi090_166897_object19884:head# 2024-05-15T23:39:57.817 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d4b796:::benchmark_data_smithi090_166897_object40528:head# 2024-05-15T23:39:57.817 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d57afc:::benchmark_data_smithi090_166897_object57969:head# 2024-05-15T23:39:57.818 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d57e08:::benchmark_data_smithi090_166897_object69017:head# 2024-05-15T23:39:57.818 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d58bc7:::benchmark_data_smithi090_166897_object7883:head# 2024-05-15T23:39:57.819 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d6b83e:::benchmark_data_smithi090_166897_object61601:head# 2024-05-15T23:39:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:57 smithi088 ceph-mon[44699]: pgmap v2020: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s; 15243/169206 objects degraded (9.009%); 11 MiB/s, 180 objects/s recovering 2024-05-15T23:39:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:57 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 15922/162842 objects degraded (9.778%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:39:57.821 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d7bdb9:::benchmark_data_smithi090_166897_object616:head# 2024-05-15T23:39:57.823 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d7f067:::benchmark_data_smithi090_166897_object1856:head# 2024-05-15T23:39:57.823 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d8717f:::benchmark_data_smithi090_166897_object37564:head# 2024-05-15T23:39:57.824 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d88e95:::benchmark_data_smithi090_166897_object22667:head# 2024-05-15T23:39:57.824 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d99ec7:::benchmark_data_smithi090_166897_object6382:head# 2024-05-15T23:39:57.825 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69d9a70e:::benchmark_data_smithi090_166897_object52392:head# 2024-05-15T23:39:57.825 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69da6f00:::benchmark_data_smithi090_166897_object37860:head# 2024-05-15T23:39:57.826 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69daa6e4:::benchmark_data_smithi090_166897_object5065:head# 2024-05-15T23:39:57.826 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69dc89f3:::benchmark_data_smithi090_166897_object22759:head# 2024-05-15T23:39:57.826 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69de8053:::benchmark_data_smithi090_166897_object20816:head# 2024-05-15T23:39:57.827 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69df610b:::benchmark_data_smithi090_166897_object7816:head# 2024-05-15T23:39:57.827 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69e09667:::benchmark_data_smithi090_166897_object17554:head# 2024-05-15T23:39:57.827 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69e1ef36:::benchmark_data_smithi090_166897_object8632:head# 2024-05-15T23:39:57.828 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69e21ce5:::benchmark_data_smithi090_166897_object40661:head# 2024-05-15T23:39:57.828 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69e7307e:::benchmark_data_smithi090_166897_object38857:head# 2024-05-15T23:39:57.829 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69e9aeaf:::benchmark_data_smithi090_166897_object22238:head# 2024-05-15T23:39:57.829 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69ea6a98:::benchmark_data_smithi090_166897_object28827:head# 2024-05-15T23:39:57.830 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69eb7437:::benchmark_data_smithi090_166897_object70198:head# 2024-05-15T23:39:57.830 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69ed7960:::benchmark_data_smithi090_166897_object33012:head# 2024-05-15T23:39:57.831 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69ee523d:::benchmark_data_smithi090_166897_object12782:head# 2024-05-15T23:39:57.831 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69ee8eb7:::benchmark_data_smithi090_166897_object65659:head# 2024-05-15T23:39:57.831 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69ee8f62:::benchmark_data_smithi090_166897_object8982:head# 2024-05-15T23:39:57.832 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69f02e73:::benchmark_data_smithi090_166897_object23573:head# 2024-05-15T23:39:57.833 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69f1e041:::benchmark_data_smithi090_166897_object49892:head# 2024-05-15T23:39:57.833 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69f2efc3:::benchmark_data_smithi090_166897_object4172:head# 2024-05-15T23:39:57.833 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69f3b8ca:::benchmark_data_smithi090_166897_object26159:head# 2024-05-15T23:39:57.834 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69f3bb1c:::benchmark_data_smithi090_166897_object58709:head# 2024-05-15T23:39:57.834 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69f40e06:::benchmark_data_smithi090_166897_object59638:head# 2024-05-15T23:39:57.835 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69f78d9d:::benchmark_data_smithi090_166897_object13323:head# 2024-05-15T23:39:57.836 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69f96fcb:::benchmark_data_smithi090_166897_object70531:head# 2024-05-15T23:39:57.836 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69fb4080:::benchmark_data_smithi090_166897_object24499:head# 2024-05-15T23:39:57.838 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69fc5662:::benchmark_data_smithi090_166897_object53269:head# 2024-05-15T23:39:57.838 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:69fd7405:::benchmark_data_smithi090_166897_object29298:head# 2024-05-15T23:39:57.839 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a006b7d:::benchmark_data_smithi090_166897_object58716:head# 2024-05-15T23:39:57.839 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a00c109:::benchmark_data_smithi090_166897_object32512:head# 2024-05-15T23:39:57.840 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a00fa46:::benchmark_data_smithi090_166897_object33786:head# 2024-05-15T23:39:57.840 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a01550c:::benchmark_data_smithi090_166897_object20773:head# 2024-05-15T23:39:57.841 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a01ac91:::benchmark_data_smithi090_166897_object2799:head# 2024-05-15T23:39:57.841 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a02ef21:::benchmark_data_smithi090_166897_object16497:head# 2024-05-15T23:39:57.841 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a02fba6:::benchmark_data_smithi090_166897_object50831:head# 2024-05-15T23:39:57.842 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a0318e7:::benchmark_data_smithi090_166897_object35643:head# 2024-05-15T23:39:57.842 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a0325c1:::benchmark_data_smithi090_166897_object1340:head# 2024-05-15T23:39:57.845 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a039969:::benchmark_data_smithi090_166897_object30296:head# 2024-05-15T23:39:57.845 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a040d41:::benchmark_data_smithi090_166897_object10611:head# 2024-05-15T23:39:57.846 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a053510:::benchmark_data_smithi090_166897_object58203:head# 2024-05-15T23:39:57.846 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a059746:::benchmark_data_smithi090_166897_object51758:head# 2024-05-15T23:39:57.847 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a07feef:::benchmark_data_smithi090_166897_object52006:head# 2024-05-15T23:39:57.847 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a09cf2b:::benchmark_data_smithi090_166897_object42981:head# 2024-05-15T23:39:57.847 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a0a2c93:::benchmark_data_smithi090_166897_object63218:head# 2024-05-15T23:39:57.848 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a0a4f23:::benchmark_data_smithi090_166897_object63858:head# 2024-05-15T23:39:57.848 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a0f65aa:::benchmark_data_smithi090_166897_object2742:head# 2024-05-15T23:39:57.849 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a0fb374:::benchmark_data_smithi090_166897_object4387:head# 2024-05-15T23:39:57.849 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a11595c:::benchmark_data_smithi090_166897_object20111:head# 2024-05-15T23:39:57.850 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a121ccb:::benchmark_data_smithi090_166897_object64612:head# 2024-05-15T23:39:57.850 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a125d09:::benchmark_data_smithi090_166897_object36209:head# 2024-05-15T23:39:57.851 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a13e492:::benchmark_data_smithi090_166897_object59284:head# 2024-05-15T23:39:57.851 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a15c6dd:::benchmark_data_smithi090_166897_object24484:head# 2024-05-15T23:39:57.851 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a17d015:::benchmark_data_smithi090_166897_object30366:head# 2024-05-15T23:39:57.852 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a191799:::benchmark_data_smithi090_166897_object17151:head# 2024-05-15T23:39:57.852 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a1a7693:::benchmark_data_smithi090_166897_object48686:head# 2024-05-15T23:39:57.853 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a1bf47c:::benchmark_data_smithi090_166897_object28481:head# 2024-05-15T23:39:57.856 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a1c2307:::benchmark_data_smithi090_166897_object68603:head# 2024-05-15T23:39:57.856 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a1d2ad2:::benchmark_data_smithi090_166897_object4308:head# 2024-05-15T23:39:57.856 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a1dbe6a:::benchmark_data_smithi090_166897_object69068:head# 2024-05-15T23:39:57.857 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a1dd0ae:::benchmark_data_smithi090_166897_object35988:head# 2024-05-15T23:39:57.858 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a1f991c:::benchmark_data_smithi090_166897_object60362:head# 2024-05-15T23:39:57.858 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a210623:::benchmark_data_smithi090_166897_object5789:head# 2024-05-15T23:39:57.858 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a232f16:::benchmark_data_smithi090_166897_object71051:head# 2024-05-15T23:39:57.859 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a24dd4f:::benchmark_data_smithi090_166897_object22901:head# 2024-05-15T23:39:57.859 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a24f05e:::benchmark_data_smithi090_166897_object38618:head# 2024-05-15T23:39:57.860 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a26d589:::benchmark_data_smithi090_166897_object60575:head# 2024-05-15T23:39:57.862 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a2745da:::benchmark_data_smithi090_166897_object32567:head# 2024-05-15T23:39:57.863 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a274651:::benchmark_data_smithi090_166897_object12697:head# 2024-05-15T23:39:57.863 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a278921:::benchmark_data_smithi090_166897_object67768:head# 2024-05-15T23:39:57.863 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a286179:::benchmark_data_smithi090_166897_object65122:head# 2024-05-15T23:39:57.864 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a28b77d:::benchmark_data_smithi090_166897_object28327:head# 2024-05-15T23:39:57.864 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a28f736:::benchmark_data_smithi090_166897_object26314:head# 2024-05-15T23:39:57.865 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a29fb3d:::benchmark_data_smithi090_166897_object54791:head# 2024-05-15T23:39:57.865 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a2beef3:::benchmark_data_smithi090_166897_object6798:head# 2024-05-15T23:39:57.868 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a2c7991:::benchmark_data_smithi090_166897_object56688:head# 2024-05-15T23:39:57.868 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a2da965:::benchmark_data_smithi090_166897_object50750:head# 2024-05-15T23:39:57.869 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a2db7b6:::benchmark_data_smithi090_166897_object47772:head# 2024-05-15T23:39:57.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:57 smithi084 ceph-mon[39784]: pgmap v2020: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s; 15243/169206 objects degraded (9.009%); 11 MiB/s, 180 objects/s recovering 2024-05-15T23:39:57.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:57 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 15922/162842 objects degraded (9.778%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:39:57.870 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a2dca35:::benchmark_data_smithi090_166897_object28111:head# 2024-05-15T23:39:57.870 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a2e7784:::benchmark_data_smithi090_166897_object25258:head# 2024-05-15T23:39:57.870 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a2ed805:::benchmark_data_smithi090_166897_object17896:head# 2024-05-15T23:39:57.870 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a2f4951:::benchmark_data_smithi090_166897_object59105:head# 2024-05-15T23:39:57.871 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a30774d:::benchmark_data_smithi090_166897_object3750:head# 2024-05-15T23:39:57.871 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a30cc13:::benchmark_data_smithi090_166897_object58744:head# 2024-05-15T23:39:57.872 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a316e83:::benchmark_data_smithi090_166897_object58134:head# 2024-05-15T23:39:57.872 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a328ab7:::benchmark_data_smithi090_166897_object22085:head# 2024-05-15T23:39:57.872 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a32c27a:::benchmark_data_smithi090_166897_object23054:head# 2024-05-15T23:39:57.873 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a339b3c:::benchmark_data_smithi090_166897_object57285:head# 2024-05-15T23:39:57.873 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a34b3c9:::benchmark_data_smithi090_166897_object13381:head# 2024-05-15T23:39:57.874 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3528cb:::benchmark_data_smithi090_166897_object20935:head# 2024-05-15T23:39:57.874 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3567c7:::benchmark_data_smithi090_166897_object26247:head# 2024-05-15T23:39:57.875 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a371f93:::benchmark_data_smithi090_166897_object5248:head# 2024-05-15T23:39:57.875 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a372567:::benchmark_data_smithi090_166897_object5404:head# 2024-05-15T23:39:57.876 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a373943:::benchmark_data_smithi090_166897_object31637:head# 2024-05-15T23:39:57.876 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a381416:::benchmark_data_smithi090_166897_object34972:head# 2024-05-15T23:39:57.876 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a382add:::benchmark_data_smithi090_166897_object30835:head# 2024-05-15T23:39:57.876 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3887ad:::benchmark_data_smithi090_166897_object16889:head# 2024-05-15T23:39:57.877 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3aa601:::benchmark_data_smithi090_166897_object52032:head# 2024-05-15T23:39:57.877 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3b18f3:::benchmark_data_smithi090_166897_object64042:head# 2024-05-15T23:39:57.878 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3bca67:::benchmark_data_smithi090_166897_object12287:head# 2024-05-15T23:39:57.878 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3bcea7:::benchmark_data_smithi090_166897_object32070:head# 2024-05-15T23:39:57.878 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3cd630:::benchmark_data_smithi090_166897_object29108:head# 2024-05-15T23:39:57.882 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3d3d53:::benchmark_data_smithi090_166897_object695:head# 2024-05-15T23:39:57.883 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3d532d:::benchmark_data_smithi090_166897_object32103:head# 2024-05-15T23:39:57.883 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3d711f:::benchmark_data_smithi090_166897_object9180:head# 2024-05-15T23:39:57.883 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3d791f:::benchmark_data_smithi090_166897_object33927:head# 2024-05-15T23:39:57.883 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3e5690:::benchmark_data_smithi090_166897_object58111:head# 2024-05-15T23:39:57.883 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3ed491:::benchmark_data_smithi090_166897_object32299:head# 2024-05-15T23:39:57.883 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3f3c57:::benchmark_data_smithi090_166897_object26842:head# 2024-05-15T23:39:57.883 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a3f9e97:::benchmark_data_smithi090_166897_object57571:head# 2024-05-15T23:39:57.884 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a41713d:::benchmark_data_smithi090_166897_object68067:head# 2024-05-15T23:39:57.884 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a421cd0:::benchmark_data_smithi090_166897_object20797:head# 2024-05-15T23:39:57.884 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a431597:::benchmark_data_smithi090_166897_object35357:head# 2024-05-15T23:39:57.884 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a438146:::benchmark_data_smithi090_166897_object19269:head# 2024-05-15T23:39:57.885 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a448adc:::benchmark_data_smithi090_166897_object20799:head# 2024-05-15T23:39:57.885 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a455417:::benchmark_data_smithi090_166897_object5958:head# 2024-05-15T23:39:57.885 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a468874:::benchmark_data_smithi090_166897_object17757:head# 2024-05-15T23:39:57.886 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a482e9e:::benchmark_data_smithi090_166897_object38584:head# 2024-05-15T23:39:57.886 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a485a3f:::benchmark_data_smithi090_166897_object16496:head# 2024-05-15T23:39:57.886 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a48b7cc:::benchmark_data_smithi090_166897_object43757:head# 2024-05-15T23:39:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a4a42de:::benchmark_data_smithi090_166897_object63804:head# 2024-05-15T23:39:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a4cf7f2:::benchmark_data_smithi090_166897_object42025:head# 2024-05-15T23:39:57.888 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a4eb07f:::benchmark_data_smithi090_166897_object21792:head# 2024-05-15T23:39:57.888 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a4ed6f4:::benchmark_data_smithi090_166897_object41050:head# 2024-05-15T23:39:57.889 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a4ef4ee:::benchmark_data_smithi090_166897_object63624:head# 2024-05-15T23:39:57.889 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a4f35c2:::benchmark_data_smithi090_166897_object31396:head# 2024-05-15T23:39:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a50f16d:::benchmark_data_smithi090_166897_object41971:head# 2024-05-15T23:39:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a513e53:::benchmark_data_smithi090_166897_object16499:head# 2024-05-15T23:39:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a513fb9:::benchmark_data_smithi090_166897_object56888:head# 2024-05-15T23:39:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a556751:::benchmark_data_smithi090_166897_object40841:head# 2024-05-15T23:39:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a55e29e:::benchmark_data_smithi090_166897_object14587:head# 2024-05-15T23:39:57.892 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a560525:::benchmark_data_smithi090_166897_object54543:head# 2024-05-15T23:39:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a5679fc:::benchmark_data_smithi090_166897_object22844:head# 2024-05-15T23:39:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a56876d:::benchmark_data_smithi090_166897_object60359:head# 2024-05-15T23:39:57.894 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a57cec5:::benchmark_data_smithi090_166897_object32485:head# 2024-05-15T23:39:57.894 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a5a07d0:::benchmark_data_smithi090_166897_object20476:head# 2024-05-15T23:39:57.895 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a5a1200:::benchmark_data_smithi090_166897_object23995:head# 2024-05-15T23:39:57.895 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a5ad6f3:::benchmark_data_smithi090_166897_object20819:head# 2024-05-15T23:39:57.895 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a5c7faa:::benchmark_data_smithi090_166897_object62734:head# 2024-05-15T23:39:57.896 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a5c98f4:::benchmark_data_smithi090_166897_object27087:head# 2024-05-15T23:39:57.896 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a5d4c2b:::benchmark_data_smithi090_166897_object10164:head# 2024-05-15T23:39:57.897 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a5f79dc:::benchmark_data_smithi090_166897_object60390:head# 2024-05-15T23:39:57.897 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a5fa929:::benchmark_data_smithi090_166897_object40317:head# 2024-05-15T23:39:57.898 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a636a40:::benchmark_data_smithi090_166897_object7783:head# 2024-05-15T23:39:57.898 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a63f85d:::benchmark_data_smithi090_166897_object37735:head# 2024-05-15T23:39:57.898 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a647672:::benchmark_data_smithi090_166897_object19926:head# 2024-05-15T23:39:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a6540f4:::benchmark_data_smithi090_166897_object57244:head# 2024-05-15T23:39:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a6c4591:::benchmark_data_smithi090_166897_object70036:head# 2024-05-15T23:39:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a6c94bf:::benchmark_data_smithi090_166897_object2113:head# 2024-05-15T23:39:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a6f8d60:::benchmark_data_smithi090_166897_object65357:head# 2024-05-15T23:39:57.901 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a704959:::benchmark_data_smithi090_166897_object4217:head# 2024-05-15T23:39:57.902 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a70f3db:::benchmark_data_smithi090_166897_object33708:head# 2024-05-15T23:39:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a7198e0:::benchmark_data_smithi090_166897_object54743:head# 2024-05-15T23:39:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a75570e:::benchmark_data_smithi090_166897_object10005:head# 2024-05-15T23:39:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a761999:::benchmark_data_smithi090_166897_object8307:head# 2024-05-15T23:39:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a76b917:::benchmark_data_smithi090_166897_object1502:head# 2024-05-15T23:39:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a76dcf7:::benchmark_data_smithi090_166897_object64047:head# 2024-05-15T23:39:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a777ff1:::benchmark_data_smithi090_166897_object30413:head# 2024-05-15T23:39:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a77d1ff:::benchmark_data_smithi090_166897_object494:head# 2024-05-15T23:39:57.905 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a78e2fe:::benchmark_data_smithi090_166897_object38997:head# 2024-05-15T23:39:57.905 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a796b86:::benchmark_data_smithi090_166897_object13022:head# 2024-05-15T23:39:57.905 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a799727:::benchmark_data_smithi090_166897_object25979:head# 2024-05-15T23:39:57.906 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a7ae2f1:::benchmark_data_smithi090_166897_object42972:head# 2024-05-15T23:39:57.906 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a7d9ee5:::benchmark_data_smithi090_166897_object66584:head# 2024-05-15T23:39:57.907 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a7e2ea6:::benchmark_data_smithi090_166897_object54635:head# 2024-05-15T23:39:57.907 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a7e88bf:::benchmark_data_smithi090_166897_object20502:head# 2024-05-15T23:39:57.908 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a80a2fb:::benchmark_data_smithi090_166897_object36272:head# 2024-05-15T23:39:57.908 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a8105ef:::benchmark_data_smithi090_166897_object10375:head# 2024-05-15T23:39:57.908 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a8203f8:::benchmark_data_smithi090_166897_object59467:head# 2024-05-15T23:39:57.909 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a820f9b:::benchmark_data_smithi090_166897_object20923:head# 2024-05-15T23:39:57.909 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a825d0c:::benchmark_data_smithi090_166897_object22864:head# 2024-05-15T23:39:57.910 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a86c712:::benchmark_data_smithi090_166897_object33034:head# 2024-05-15T23:39:57.910 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a8795c3:::benchmark_data_smithi090_166897_object58366:head# 2024-05-15T23:39:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a8871a7:::benchmark_data_smithi090_166897_object42989:head# 2024-05-15T23:39:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a8a72ef:::benchmark_data_smithi090_166897_object3005:head# 2024-05-15T23:39:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a8c6523:::benchmark_data_smithi090_166897_object30527:head# 2024-05-15T23:39:57.912 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a8ccf49:::benchmark_data_smithi090_166897_object32745:head# 2024-05-15T23:39:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a9006ae:::benchmark_data_smithi090_166897_object62507:head# 2024-05-15T23:39:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a91dc08:::benchmark_data_smithi090_166897_object21555:head# 2024-05-15T23:39:57.914 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a93efed:::benchmark_data_smithi090_166897_object29410:head# 2024-05-15T23:39:57.914 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a93f906:::benchmark_data_smithi090_166897_object21222:head# 2024-05-15T23:39:57.914 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a95cc32:::benchmark_data_smithi090_166897_object37136:head# 2024-05-15T23:39:57.915 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a960612:::benchmark_data_smithi090_166897_object66533:head# 2024-05-15T23:39:57.915 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a964849:::benchmark_data_smithi090_166897_object45430:head# 2024-05-15T23:39:57.916 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a969690:::benchmark_data_smithi090_166897_object27022:head# 2024-05-15T23:39:57.916 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a9700aa:::benchmark_data_smithi090_166897_object23008:head# 2024-05-15T23:39:57.917 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a971d92:::benchmark_data_smithi090_166897_object16301:head# 2024-05-15T23:39:57.917 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a97347d:::benchmark_data_smithi090_166897_object66848:head# 2024-05-15T23:39:57.917 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a98db6b:::benchmark_data_smithi090_166897_object12465:head# 2024-05-15T23:39:57.918 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a9909e8:::benchmark_data_smithi090_166897_object21697:head# 2024-05-15T23:39:57.918 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a99efb5:::benchmark_data_smithi090_166897_object31607:head# 2024-05-15T23:39:57.919 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a99fcef:::benchmark_data_smithi090_166897_object16909:head# 2024-05-15T23:39:57.919 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a9a96d3:::benchmark_data_smithi090_166897_object27468:head# 2024-05-15T23:39:57.920 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a9ae885:::benchmark_data_smithi090_166897_object59403:head# 2024-05-15T23:39:57.920 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a9d5512:::benchmark_data_smithi090_166897_object30718:head# 2024-05-15T23:39:57.921 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a9d65d3:::benchmark_data_smithi090_166897_object2832:head# 2024-05-15T23:39:57.921 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a9d8de1:::benchmark_data_smithi090_166897_object8642:head# 2024-05-15T23:39:57.922 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a9d9d5a:::benchmark_data_smithi090_166897_object55430:head# 2024-05-15T23:39:57.922 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6a9e124a:::benchmark_data_smithi090_166897_object65963:head# 2024-05-15T23:39:57.922 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aa021de:::benchmark_data_smithi090_166897_object18614:head# 2024-05-15T23:39:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aa0a236:::benchmark_data_smithi090_166897_object42000:head# 2024-05-15T23:39:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aa0d1f5:::benchmark_data_smithi090_166897_object1851:head# 2024-05-15T23:39:57.924 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aa0e81d:::benchmark_data_smithi090_166897_object25039:head# 2024-05-15T23:39:57.924 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aa1146f:::benchmark_data_smithi090_166897_object56992:head# 2024-05-15T23:39:57.925 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aa15dd4:::benchmark_data_smithi090_166897_object16297:head# 2024-05-15T23:39:57.925 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aa4aa26:::benchmark_data_smithi090_166897_object35949:head# 2024-05-15T23:39:57.925 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aa4e72e:::benchmark_data_smithi090_166897_object48908:head# 2024-05-15T23:39:57.926 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aa7711a:::benchmark_data_smithi090_166897_object5782:head# 2024-05-15T23:39:57.926 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aa86256:::benchmark_data_smithi090_166897_object17039:head# 2024-05-15T23:39:57.927 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aa896e2:::benchmark_data_smithi090_166897_object7687:head# 2024-05-15T23:39:57.927 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aa99581:::benchmark_data_smithi090_166897_object11744:head# 2024-05-15T23:39:57.930 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aaa1edc:::benchmark_data_smithi090_166897_object23490:head# 2024-05-15T23:39:57.930 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aaa45cd:::benchmark_data_smithi090_166897_object15283:head# 2024-05-15T23:39:57.930 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aabcfab:::benchmark_data_smithi090_166897_object39954:head# 2024-05-15T23:39:57.930 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aadbff8:::benchmark_data_smithi090_166897_object33642:head# 2024-05-15T23:39:57.930 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aae3888:::benchmark_data_smithi090_166897_object65493:head# 2024-05-15T23:39:57.931 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aaee7cd:::benchmark_data_smithi090_166897_object28770:head# 2024-05-15T23:39:57.931 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aaf66be:::benchmark_data_smithi090_166897_object55637:head# 2024-05-15T23:39:57.932 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aaf91e5:::benchmark_data_smithi090_166897_object3906:head# 2024-05-15T23:39:57.932 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aaf98f4:::benchmark_data_smithi090_166897_object45450:head# 2024-05-15T23:39:57.932 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ab16383:::benchmark_data_smithi090_166897_object60864:head# 2024-05-15T23:39:57.933 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ab1decc:::benchmark_data_smithi090_166897_object46090:head# 2024-05-15T23:39:57.933 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ab235cd:::benchmark_data_smithi090_166897_object3188:head# 2024-05-15T23:39:57.934 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ab2818f:::benchmark_data_smithi090_166897_object12238:head# 2024-05-15T23:39:57.934 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ab34a85:::benchmark_data_smithi090_166897_object4955:head# 2024-05-15T23:39:57.934 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ab34b1b:::benchmark_data_smithi090_166897_object32886:head# 2024-05-15T23:39:57.935 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ab59758:::benchmark_data_smithi090_166897_object12066:head# 2024-05-15T23:39:57.935 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ab5b153:::benchmark_data_smithi090_166897_object63063:head# 2024-05-15T23:39:57.936 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ab78870:::benchmark_data_smithi090_166897_object26218:head# 2024-05-15T23:39:57.936 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ab84fad:::benchmark_data_smithi090_166897_object22402:head# 2024-05-15T23:39:57.937 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ab908c1:::benchmark_data_smithi090_166897_object68430:head# 2024-05-15T23:39:57.937 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ab97fa8:::benchmark_data_smithi090_166897_object4589:head# 2024-05-15T23:39:57.937 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aba5590:::benchmark_data_smithi090_166897_object65555:head# 2024-05-15T23:39:57.938 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6abb883d:::benchmark_data_smithi090_166897_object8964:head# 2024-05-15T23:39:57.938 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6abbae16:::benchmark_data_smithi090_166897_object65767:head# 2024-05-15T23:39:57.938 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6abe2bfb:::benchmark_data_smithi090_166897_object44685:head# 2024-05-15T23:39:57.939 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6abe4923:::benchmark_data_smithi090_166897_object69148:head# 2024-05-15T23:39:57.939 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6abe50b4:::benchmark_data_smithi090_166897_object18662:head# 2024-05-15T23:39:57.940 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6abe8760:::benchmark_data_smithi090_166897_object19139:head# 2024-05-15T23:39:57.940 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ac137f5:::benchmark_data_smithi090_166897_object20010:head# 2024-05-15T23:39:57.943 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ac13e60:::benchmark_data_smithi090_166897_object43727:head# 2024-05-15T23:39:57.943 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ac269af:::benchmark_data_smithi090_166897_object46623:head# 2024-05-15T23:39:57.943 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ac38135:::benchmark_data_smithi090_166897_object5220:head# 2024-05-15T23:39:57.943 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ac3963d:::benchmark_data_smithi090_166897_object47888:head# 2024-05-15T23:39:57.943 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ac5732c:::benchmark_data_smithi090_166897_object9245:head# 2024-05-15T23:39:57.943 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ac60a59:::benchmark_data_smithi090_166897_object52694:head# 2024-05-15T23:39:57.944 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ac7d158:::benchmark_data_smithi090_166897_object19114:head# 2024-05-15T23:39:57.944 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ac86637:::benchmark_data_smithi090_166897_object66283:head# 2024-05-15T23:39:57.944 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ac9e34e:::benchmark_data_smithi090_166897_object34859:head# 2024-05-15T23:39:57.945 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6acb0213:::benchmark_data_smithi090_166897_object68888:head# 2024-05-15T23:39:57.945 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6acca9a2:::benchmark_data_smithi090_166897_object3075:head# 2024-05-15T23:39:57.946 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6acd17dd:::benchmark_data_smithi090_166897_object65757:head# 2024-05-15T23:39:57.946 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ace6293:::benchmark_data_smithi090_166897_object11221:head# 2024-05-15T23:39:57.946 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6acec768:::benchmark_data_smithi090_166897_object5070:head# 2024-05-15T23:39:57.947 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad0240a:::benchmark_data_smithi090_166897_object42792:head# 2024-05-15T23:39:57.947 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad0e65b:::benchmark_data_smithi090_166897_object19355:head# 2024-05-15T23:39:57.948 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad12514:::benchmark_data_smithi090_166897_object25110:head# 2024-05-15T23:39:57.948 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad16514:::benchmark_data_smithi090_166897_object11208:head# 2024-05-15T23:39:57.948 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad1d414:::benchmark_data_smithi090_166897_object2387:head# 2024-05-15T23:39:57.949 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad289aa:::benchmark_data_smithi090_166897_object64944:head# 2024-05-15T23:39:57.949 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad34341:::benchmark_data_smithi090_166897_object51067:head# 2024-05-15T23:39:57.950 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad349c3:::benchmark_data_smithi090_166897_object25378:head# 2024-05-15T23:39:57.950 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad4c446:::benchmark_data_smithi090_166897_object51417:head# 2024-05-15T23:39:57.950 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad5c6bb:::benchmark_data_smithi090_166897_object47050:head# 2024-05-15T23:39:57.951 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad66f46:::benchmark_data_smithi090_166897_object37067:head# 2024-05-15T23:39:57.951 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad6844d:::benchmark_data_smithi090_166897_object70651:head# 2024-05-15T23:39:57.952 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad72e05:::benchmark_data_smithi090_166897_object4751:head# 2024-05-15T23:39:57.952 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad95120:::benchmark_data_smithi090_166897_object5493:head# 2024-05-15T23:39:57.953 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad97d5d:::benchmark_data_smithi090_166897_object7721:head# 2024-05-15T23:39:57.953 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ad9804c:::benchmark_data_smithi090_166897_object62292:head# 2024-05-15T23:39:57.957 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ada11f7:::benchmark_data_smithi090_166897_object30580:head# 2024-05-15T23:39:57.957 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6adbc5ac:::benchmark_data_smithi090_166897_object2022:head# 2024-05-15T23:39:57.957 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6adca41d:::benchmark_data_smithi090_166897_object40548:head# 2024-05-15T23:39:57.957 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6adcefe1:::benchmark_data_smithi090_166897_object18571:head# 2024-05-15T23:39:57.958 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6adcfa4e:::benchmark_data_smithi090_166897_object28867:head# 2024-05-15T23:39:57.958 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6adcfd0f:::benchmark_data_smithi090_166897_object47048:head# 2024-05-15T23:39:57.958 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ae243e7:::benchmark_data_smithi090_166897_object62121:head# 2024-05-15T23:39:57.960 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ae34399:::benchmark_data_smithi090_166897_object64596:head# 2024-05-15T23:39:57.960 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ae3bf07:::benchmark_data_smithi090_166897_object33222:head# 2024-05-15T23:39:57.960 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ae4aecd:::benchmark_data_smithi090_166897_object20364:head# 2024-05-15T23:39:57.960 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ae53107:::benchmark_data_smithi090_166897_object63430:head# 2024-05-15T23:39:57.960 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ae55bcc:::benchmark_data_smithi090_166897_object65843:head# 2024-05-15T23:39:57.960 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ae647eb:::benchmark_data_smithi090_166897_object26473:head# 2024-05-15T23:39:57.963 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ae6a681:::benchmark_data_smithi090_166897_object515:head# 2024-05-15T23:39:57.963 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ae7224a:::benchmark_data_smithi090_166897_object33215:head# 2024-05-15T23:39:57.964 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ae7482c:::benchmark_data_smithi090_166897_object49097:head# 2024-05-15T23:39:57.964 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ae76b7e:::benchmark_data_smithi090_166897_object9079:head# 2024-05-15T23:39:57.965 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ae7bcc4:::benchmark_data_smithi090_166897_object4142:head# 2024-05-15T23:39:57.965 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ae7f9bc:::benchmark_data_smithi090_166897_object35943:head# 2024-05-15T23:39:57.966 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ae816d5:::benchmark_data_smithi090_166897_object69806:head# 2024-05-15T23:39:57.966 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ae9ccee:::benchmark_data_smithi090_166897_object12034:head# 2024-05-15T23:39:57.967 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aeae1a7:::benchmark_data_smithi090_166897_object17739:head# 2024-05-15T23:39:57.968 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aec1701:::benchmark_data_smithi090_166897_object25918:head# 2024-05-15T23:39:57.974 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aec3fde:::benchmark_data_smithi090_166897_object6086:head# 2024-05-15T23:39:57.974 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aed7374:::benchmark_data_smithi090_166897_object46563:head# 2024-05-15T23:39:57.974 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aee7a60:::benchmark_data_smithi090_166897_object51127:head# 2024-05-15T23:39:57.974 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6aef39d1:::benchmark_data_smithi090_166897_object61935:head# 2024-05-15T23:39:57.975 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6af0937a:::benchmark_data_smithi090_166897_object28577:head# 2024-05-15T23:39:57.975 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6af1d00d:::benchmark_data_smithi090_166897_object6771:head# 2024-05-15T23:39:57.975 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6af2a0cc:::benchmark_data_smithi090_166897_object38457:head# 2024-05-15T23:39:57.975 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6af2a706:::benchmark_data_smithi090_166897_object6402:head# 2024-05-15T23:39:57.975 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6af3d439:::benchmark_data_smithi090_166897_object6931:head# 2024-05-15T23:39:57.976 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6af73edb:::benchmark_data_smithi090_166897_object2191:head# 2024-05-15T23:39:57.979 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6af774e0:::benchmark_data_smithi090_166897_object41615:head# 2024-05-15T23:39:57.979 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6af8a22c:::benchmark_data_smithi090_166897_object10740:head# 2024-05-15T23:39:57.980 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6af98e8b:::benchmark_data_smithi090_166897_object31239:head# 2024-05-15T23:39:57.980 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6afd226c:::benchmark_data_smithi090_166897_object8485:head# 2024-05-15T23:39:57.981 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6afe828a:::benchmark_data_smithi090_166897_object26521:head# 2024-05-15T23:39:57.982 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6afef6a9:::benchmark_data_smithi090_166897_object22910:head# 2024-05-15T23:39:57.982 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b01fefe:::benchmark_data_smithi090_166897_object60316:head# 2024-05-15T23:39:57.983 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b029736:::benchmark_data_smithi090_166897_object21307:head# 2024-05-15T23:39:57.983 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b03aa1d:::benchmark_data_smithi090_166897_object16097:head# 2024-05-15T23:39:57.983 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b046a4a:::benchmark_data_smithi090_166897_object18235:head# 2024-05-15T23:39:57.984 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b0614fd:::benchmark_data_smithi090_166897_object12616:head# 2024-05-15T23:39:57.984 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b07037f:::benchmark_data_smithi090_166897_object59118:head# 2024-05-15T23:39:57.985 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b07639f:::benchmark_data_smithi090_166897_object6127:head# 2024-05-15T23:39:57.985 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b07cc74:::benchmark_data_smithi090_166897_object14858:head# 2024-05-15T23:39:57.986 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b08b62a:::benchmark_data_smithi090_166897_object4429:head# 2024-05-15T23:39:57.986 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b0a0113:::benchmark_data_smithi090_166897_object44559:head# 2024-05-15T23:39:57.987 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b0bf99f:::benchmark_data_smithi090_166897_object34788:head# 2024-05-15T23:39:57.988 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b0c1da2:::benchmark_data_smithi090_166897_object56893:head# 2024-05-15T23:39:57.989 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b0c20b4:::benchmark_data_smithi090_166897_object25176:head# 2024-05-15T23:39:57.989 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b0cd8a7:::benchmark_data_smithi090_166897_object56843:head# 2024-05-15T23:39:57.989 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b0e3685:::benchmark_data_smithi090_166897_object3046:head# 2024-05-15T23:39:57.990 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b0ed250:::benchmark_data_smithi090_166897_object27187:head# 2024-05-15T23:39:57.990 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b0f5654:::benchmark_data_smithi090_166897_object16828:head# 2024-05-15T23:39:57.991 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b103105:::benchmark_data_smithi090_166897_object42102:head# 2024-05-15T23:39:57.991 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b105e90:::benchmark_data_smithi090_166897_object65445:head# 2024-05-15T23:39:57.992 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b111d3d:::benchmark_data_smithi090_166897_object63513:head# 2024-05-15T23:39:57.992 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b131d83:::benchmark_data_smithi090_166897_object59925:head# 2024-05-15T23:39:57.992 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b13d5b0:::benchmark_data_smithi090_166897_object11232:head# 2024-05-15T23:39:57.993 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b15ef41:::benchmark_data_smithi090_166897_object15968:head# 2024-05-15T23:39:57.993 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b16633c:::benchmark_data_smithi090_166897_object23692:head# 2024-05-15T23:39:57.994 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b16fc60:::benchmark_data_smithi090_166897_object59401:head# 2024-05-15T23:39:57.995 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b19397a:::benchmark_data_smithi090_166897_object45642:head# 2024-05-15T23:39:57.995 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b19552b:::benchmark_data_smithi090_166897_object31919:head# 2024-05-15T23:39:57.996 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b199700:::benchmark_data_smithi090_166897_object63634:head# 2024-05-15T23:39:57.996 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b1b6ece:::benchmark_data_smithi090_166897_object55928:head# 2024-05-15T23:39:57.997 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b1bb102:::benchmark_data_smithi090_166897_object21639:head# 2024-05-15T23:39:57.997 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b1c81cb:::benchmark_data_smithi090_166897_object65631:head# 2024-05-15T23:39:57.997 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b1e7e33:::benchmark_data_smithi090_166897_object24677:head# 2024-05-15T23:39:58.000 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b20d703:::benchmark_data_smithi090_166897_object56120:head# 2024-05-15T23:39:58.000 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b219a19:::benchmark_data_smithi090_166897_object54882:head# 2024-05-15T23:39:58.001 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b220ce9:::benchmark_data_smithi090_166897_object39260:head# 2024-05-15T23:39:58.001 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b22cc32:::benchmark_data_smithi090_166897_object3527:head# 2024-05-15T23:39:58.002 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b22eae0:::benchmark_data_smithi090_166897_object8310:head# 2024-05-15T23:39:58.002 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b2438e1:::benchmark_data_smithi090_166897_object67815:head# 2024-05-15T23:39:58.003 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b25090d:::benchmark_data_smithi090_166897_object61145:head# 2024-05-15T23:39:58.003 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b265f59:::benchmark_data_smithi090_166897_object14433:head# 2024-05-15T23:39:58.004 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b278e2f:::benchmark_data_smithi090_166897_object7887:head# 2024-05-15T23:39:58.005 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b2907bb:::benchmark_data_smithi090_166897_object54504:head# 2024-05-15T23:39:58.006 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b294e9d:::benchmark_data_smithi090_166897_object47551:head# 2024-05-15T23:39:58.007 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b29f212:::benchmark_data_smithi090_166897_object12577:head# 2024-05-15T23:39:58.008 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b2b745b:::benchmark_data_smithi090_166897_object1332:head# 2024-05-15T23:39:58.009 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b2db13d:::benchmark_data_smithi090_166897_object16046:head# 2024-05-15T23:39:58.009 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b2e1c31:::benchmark_data_smithi090_166897_object44567:head# 2024-05-15T23:39:58.009 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b2eeaf9:::benchmark_data_smithi090_166897_object35005:head# 2024-05-15T23:39:58.010 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b3057a8:::benchmark_data_smithi090_166897_object64428:head# 2024-05-15T23:39:58.010 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b32095a:::benchmark_data_smithi090_166897_object3277:head# 2024-05-15T23:39:58.011 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b340cb9:::benchmark_data_smithi090_166897_object62650:head# 2024-05-15T23:39:58.011 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b343c28:::benchmark_data_smithi090_166897_object8823:head# 2024-05-15T23:39:58.011 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b34cce1:::benchmark_data_smithi090_166897_object19300:head# 2024-05-15T23:39:58.012 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b35c63d:::benchmark_data_smithi090_166897_object7182:head# 2024-05-15T23:39:58.012 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b384fd6:::benchmark_data_smithi090_166897_object70839:head# 2024-05-15T23:39:58.013 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b39994d:::benchmark_data_smithi090_166897_object12297:head# 2024-05-15T23:39:58.013 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b39db51:::benchmark_data_smithi090_166897_object62299:head# 2024-05-15T23:39:58.014 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b39fff2:::benchmark_data_smithi090_166897_object4620:head# 2024-05-15T23:39:58.014 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b3a997d:::benchmark_data_smithi090_166897_object17557:head# 2024-05-15T23:39:58.015 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b3ac0be:::benchmark_data_smithi090_166897_object28994:head# 2024-05-15T23:39:58.015 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b3b35dc:::benchmark_data_smithi090_166897_object19171:head# 2024-05-15T23:39:58.015 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b3bb27a:::benchmark_data_smithi090_166897_object5735:head# 2024-05-15T23:39:58.016 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b3bbcdd:::benchmark_data_smithi090_166897_object18265:head# 2024-05-15T23:39:58.016 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b3c240f:::benchmark_data_smithi090_166897_object42742:head# 2024-05-15T23:39:58.017 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b3ca721:::benchmark_data_smithi090_166897_object49791:head# 2024-05-15T23:39:58.017 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b3e277f:::benchmark_data_smithi090_166897_object66915:head# 2024-05-15T23:39:58.018 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b3f79b0:::benchmark_data_smithi090_166897_object59683:head# 2024-05-15T23:39:58.018 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b3f8fa7:::benchmark_data_smithi090_166897_object59243:head# 2024-05-15T23:39:58.018 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b3fc145:::benchmark_data_smithi090_166897_object2452:head# 2024-05-15T23:39:58.019 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b442037:::benchmark_data_smithi090_166897_object34557:head# 2024-05-15T23:39:58.019 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b442c97:::benchmark_data_smithi090_166897_object52830:head# 2024-05-15T23:39:58.019 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b446cb2:::benchmark_data_smithi090_166897_object44079:head# 2024-05-15T23:39:58.020 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b451450:::benchmark_data_smithi090_166897_object37959:head# 2024-05-15T23:39:58.020 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b45867c:::benchmark_data_smithi090_166897_object41947:head# 2024-05-15T23:39:58.021 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b45d825:::benchmark_data_smithi090_166897_object43222:head# 2024-05-15T23:39:58.021 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b45e310:::benchmark_data_smithi090_166897_object69394:head# 2024-05-15T23:39:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b468f25:::benchmark_data_smithi090_166897_object55199:head# 2024-05-15T23:39:58.022 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b469610:::benchmark_data_smithi090_166897_object61234:head# 2024-05-15T23:39:58.023 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b47f6c6:::benchmark_data_smithi090_166897_object2483:head# 2024-05-15T23:39:58.024 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b480ff4:::benchmark_data_smithi090_166897_object13582:head# 2024-05-15T23:39:58.024 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b48f3f7:::benchmark_data_smithi090_166897_object18537:head# 2024-05-15T23:39:58.024 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b49126e:::benchmark_data_smithi090_166897_object66981:head# 2024-05-15T23:39:58.025 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b492593:::benchmark_data_smithi090_166897_object31872:head# 2024-05-15T23:39:58.025 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b49c3ae:::benchmark_data_smithi090_166897_object18095:head# 2024-05-15T23:39:58.026 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b4b8205:::benchmark_data_smithi090_166897_object59609:head# 2024-05-15T23:39:58.026 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b4b9983:::benchmark_data_smithi090_166897_object55911:head# 2024-05-15T23:39:58.027 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b4e07f3:::benchmark_data_smithi090_166897_object56153:head# 2024-05-15T23:39:58.027 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b4ed069:::benchmark_data_smithi090_166897_object41648:head# 2024-05-15T23:39:58.028 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b4f5c82:::benchmark_data_smithi090_166897_object7018:head# 2024-05-15T23:39:58.028 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b509d1a:::benchmark_data_smithi090_166897_object137:head# 2024-05-15T23:39:58.029 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b52fa82:::benchmark_data_smithi090_166897_object37026:head# 2024-05-15T23:39:58.029 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b56359f:::benchmark_data_smithi090_166897_object14366:head# 2024-05-15T23:39:58.030 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b58998b:::benchmark_data_smithi090_166897_object27405:head# 2024-05-15T23:39:58.030 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b58fed9:::benchmark_data_smithi090_166897_object67403:head# 2024-05-15T23:39:58.030 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b59962f:::benchmark_data_smithi090_166897_object69273:head# 2024-05-15T23:39:58.031 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b5a2e3c:::benchmark_data_smithi090_166897_object20402:head# 2024-05-15T23:39:58.031 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b5a32a2:::benchmark_data_smithi090_166897_object33536:head# 2024-05-15T23:39:58.032 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b5a3766:::benchmark_data_smithi090_166897_object1932:head# 2024-05-15T23:39:58.032 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b5a68ff:::benchmark_data_smithi090_166897_object21870:head# 2024-05-15T23:39:58.032 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b5af82c:::benchmark_data_smithi090_166897_object42497:head# 2024-05-15T23:39:58.033 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b5d8703:::benchmark_data_smithi090_166897_object32282:head# 2024-05-15T23:39:58.033 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b5eb64d:::benchmark_data_smithi090_166897_object28035:head# 2024-05-15T23:39:58.034 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b5edf94:::benchmark_data_smithi090_166897_object64899:head# 2024-05-15T23:39:58.034 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b5ef480:::benchmark_data_smithi090_166897_object43340:head# 2024-05-15T23:39:58.034 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b5f9dbb:::benchmark_data_smithi090_166897_object64397:head# 2024-05-15T23:39:58.035 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b5fa81e:::benchmark_data_smithi090_166897_object49968:head# 2024-05-15T23:39:58.035 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b6164cf:::benchmark_data_smithi090_166897_object6570:head# 2024-05-15T23:39:58.036 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b622d99:::benchmark_data_smithi090_166897_object51190:head# 2024-05-15T23:39:58.037 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b63b04f:::benchmark_data_smithi090_166897_object8256:head# 2024-05-15T23:39:58.037 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b66cbb9:::benchmark_data_smithi090_166897_object50383:head# 2024-05-15T23:39:58.038 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b673d52:::benchmark_data_smithi090_166897_object62957:head# 2024-05-15T23:39:58.038 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b68665b:::benchmark_data_smithi090_166897_object39479:head# 2024-05-15T23:39:58.038 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b6a17d0:::benchmark_data_smithi090_166897_object70152:head# 2024-05-15T23:39:58.039 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b6aa6c1:::benchmark_data_smithi090_166897_object39811:head# 2024-05-15T23:39:58.039 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b6c30fb:::benchmark_data_smithi090_166897_object28069:head# 2024-05-15T23:39:58.040 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b6c8412:::benchmark_data_smithi090_166897_object19324:head# 2024-05-15T23:39:58.040 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b6cecf8:::benchmark_data_smithi090_166897_object17694:head# 2024-05-15T23:39:58.041 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b6cfe4a:::benchmark_data_smithi090_166897_object66911:head# 2024-05-15T23:39:58.041 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b6f77cc:::benchmark_data_smithi090_166897_object31404:head# 2024-05-15T23:39:58.041 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b6f9966:::benchmark_data_smithi090_166897_object9689:head# 2024-05-15T23:39:58.042 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b6fe241:::benchmark_data_smithi090_166897_object45361:head# 2024-05-15T23:39:58.042 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b707a80:::benchmark_data_smithi090_166897_object50502:head# 2024-05-15T23:39:58.043 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b713a84:::benchmark_data_smithi090_166897_object10578:head# 2024-05-15T23:39:58.043 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b731961:::benchmark_data_smithi090_166897_object53718:head# 2024-05-15T23:39:58.044 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b733385:::benchmark_data_smithi090_166897_object21924:head# 2024-05-15T23:39:58.044 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b7485b0:::benchmark_data_smithi090_166897_object58602:head# 2024-05-15T23:39:58.044 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b75f6e9:::benchmark_data_smithi090_166897_object43107:head# 2024-05-15T23:39:58.045 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b761944:::benchmark_data_smithi090_166897_object54487:head# 2024-05-15T23:39:58.045 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b768b9c:::benchmark_data_smithi090_166897_object51696:head# 2024-05-15T23:39:58.045 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b779bf3:::benchmark_data_smithi090_166897_object21321:head# 2024-05-15T23:39:58.046 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b7922a5:::benchmark_data_smithi090_166897_object21988:head# 2024-05-15T23:39:58.046 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b796a8b:::benchmark_data_smithi090_166897_object25326:head# 2024-05-15T23:39:58.047 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b7a245d:::benchmark_data_smithi090_166897_object55590:head# 2024-05-15T23:39:58.047 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b7b048e:::benchmark_data_smithi090_166897_object1646:head# 2024-05-15T23:39:58.048 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b7b67a9:::benchmark_data_smithi090_166897_object48519:head# 2024-05-15T23:39:58.048 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b7bb7ee:::benchmark_data_smithi090_166897_object47592:head# 2024-05-15T23:39:58.048 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b7bf040:::benchmark_data_smithi090_166897_object65826:head# 2024-05-15T23:39:58.049 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b7c7293:::benchmark_data_smithi090_166897_object31427:head# 2024-05-15T23:39:58.049 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b7f4c36:::benchmark_data_smithi090_166897_object44423:head# 2024-05-15T23:39:58.050 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b7fae26:::benchmark_data_smithi090_166897_object43876:head# 2024-05-15T23:39:58.050 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b7fbc45:::benchmark_data_smithi090_166897_object49386:head# 2024-05-15T23:39:58.051 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b816eae:::benchmark_data_smithi090_166897_object61110:head# 2024-05-15T23:39:58.051 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b829466:::benchmark_data_smithi090_166897_object49592:head# 2024-05-15T23:39:58.052 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b82b413:::benchmark_data_smithi090_166897_object34752:head# 2024-05-15T23:39:58.052 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b82c6a8:::benchmark_data_smithi090_166897_object4513:head# 2024-05-15T23:39:58.052 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b846375:::benchmark_data_smithi090_166897_object32555:head# 2024-05-15T23:39:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b84d347:::benchmark_data_smithi090_166897_object67773:head# 2024-05-15T23:39:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b84df9d:::benchmark_data_smithi090_166897_object70233:head# 2024-05-15T23:39:58.054 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b855ab3:::benchmark_data_smithi090_166897_object36686:head# 2024-05-15T23:39:58.054 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b869741:::benchmark_data_smithi090_166897_object47950:head# 2024-05-15T23:39:58.057 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b86bdcb:::benchmark_data_smithi090_166897_object27097:head# 2024-05-15T23:39:58.057 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b86f000:::benchmark_data_smithi090_166897_object27469:head# 2024-05-15T23:39:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b871907:::benchmark_data_smithi090_166897_object42575:head# 2024-05-15T23:39:58.060 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b8951de:::benchmark_data_smithi090_166897_object67026:head# 2024-05-15T23:39:58.061 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b897e7b:::benchmark_data_smithi090_166897_object41426:head# 2024-05-15T23:39:58.062 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b899603:::benchmark_data_smithi090_166897_object8501:head# 2024-05-15T23:39:58.062 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b89a2f1:::benchmark_data_smithi090_166897_object12280:head# 2024-05-15T23:39:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b8b19d8:::benchmark_data_smithi090_166897_object42422:head# 2024-05-15T23:39:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b8ce42c:::benchmark_data_smithi090_166897_object65611:head# 2024-05-15T23:39:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b8dca5b:::benchmark_data_smithi090_166897_object3367:head# 2024-05-15T23:39:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b8e29f3:::benchmark_data_smithi090_166897_object36149:head# 2024-05-15T23:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b8e453d:::benchmark_data_smithi090_166897_object25034:head# 2024-05-15T23:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b8ed158:::benchmark_data_smithi090_166897_object71087:head# 2024-05-15T23:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b8fbb03:::benchmark_data_smithi090_166897_object71053:head# 2024-05-15T23:39:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b9042f9:::benchmark_data_smithi090_166897_object21454:head# 2024-05-15T23:39:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b90e969:::benchmark_data_smithi090_166897_object67339:head# 2024-05-15T23:39:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b919182:::benchmark_data_smithi090_166897_object23130:head# 2024-05-15T23:39:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b91c901:::benchmark_data_smithi090_166897_object69428:head# 2024-05-15T23:39:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b924f96:::benchmark_data_smithi090_166897_object69510:head# 2024-05-15T23:39:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b93bc56:::benchmark_data_smithi090_166897_object70950:head# 2024-05-15T23:39:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b94360a:::benchmark_data_smithi090_166897_object66513:head# 2024-05-15T23:39:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b948631:::benchmark_data_smithi090_166897_object21329:head# 2024-05-15T23:39:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b94ad19:::benchmark_data_smithi090_166897_object40293:head# 2024-05-15T23:39:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b95bee3:::benchmark_data_smithi090_166897_object9857:head# 2024-05-15T23:39:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b962b26:::benchmark_data_smithi090_166897_object22802:head# 2024-05-15T23:39:58.070 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b970473:::benchmark_data_smithi090_166897_object56228:head# 2024-05-15T23:39:58.070 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b980070:::benchmark_data_smithi090_166897_object26740:head# 2024-05-15T23:39:58.070 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b982e93:::benchmark_data_smithi090_166897_object28195:head# 2024-05-15T23:39:58.071 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b98baf4:::benchmark_data_smithi090_166897_object58676:head# 2024-05-15T23:39:58.071 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b9ae555:::benchmark_data_smithi090_166897_object22681:head# 2024-05-15T23:39:58.073 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b9afc33:::benchmark_data_smithi090_166897_object48463:head# 2024-05-15T23:39:58.073 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b9b7adb:::benchmark_data_smithi090_166897_object51664:head# 2024-05-15T23:39:58.073 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b9b8c0a:::benchmark_data_smithi090_166897_object52869:head# 2024-05-15T23:39:58.074 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b9bd3da:::benchmark_data_smithi090_166897_object58773:head# 2024-05-15T23:39:58.074 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b9cd0c7:::benchmark_data_smithi090_166897_object18209:head# 2024-05-15T23:39:58.074 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b9cfc73:::benchmark_data_smithi090_166897_object45474:head# 2024-05-15T23:39:58.074 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6b9ed43a:::benchmark_data_smithi090_166897_object57853:head# 2024-05-15T23:39:58.075 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ba0046b:::benchmark_data_smithi090_166897_object4666:head# 2024-05-15T23:39:58.077 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ba1410f:::benchmark_data_smithi090_166897_object71054:head# 2024-05-15T23:39:58.078 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ba2a53e:::benchmark_data_smithi090_166897_object3761:head# 2024-05-15T23:39:58.078 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ba35f44:::benchmark_data_smithi090_166897_object745:head# 2024-05-15T23:39:58.079 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ba3611d:::benchmark_data_smithi090_166897_object13311:head# 2024-05-15T23:39:58.079 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ba3e405:::benchmark_data_smithi090_166897_object48046:head# 2024-05-15T23:39:58.080 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ba4353d:::benchmark_data_smithi090_166897_object61974:head# 2024-05-15T23:39:58.083 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ba4cc11:::benchmark_data_smithi090_166897_object6388:head# 2024-05-15T23:39:58.084 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ba513b0:::benchmark_data_smithi090_166897_object59959:head# 2024-05-15T23:39:58.084 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ba65c4e:::benchmark_data_smithi090_166897_object44175:head# 2024-05-15T23:39:58.085 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ba66a99:::benchmark_data_smithi090_166897_object19554:head# 2024-05-15T23:39:58.085 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ba7b70c:::benchmark_data_smithi090_166897_object40840:head# 2024-05-15T23:39:58.085 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ba8d22e:::benchmark_data_smithi090_166897_object70413:head# 2024-05-15T23:39:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ba99e68:::benchmark_data_smithi090_166897_object58979:head# 2024-05-15T23:39:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ba99fff:::benchmark_data_smithi090_166897_object62989:head# 2024-05-15T23:39:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6baa1598:::benchmark_data_smithi090_166897_object29733:head# 2024-05-15T23:39:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6baa90f1:::benchmark_data_smithi090_166897_object68649:head# 2024-05-15T23:39:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6baacc6e:::benchmark_data_smithi090_166897_object66942:head# 2024-05-15T23:39:58.095 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bab8309:::benchmark_data_smithi090_166897_object53254:head# 2024-05-15T23:39:58.095 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bae30b0:::benchmark_data_smithi090_166897_object57269:head# 2024-05-15T23:39:58.096 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6baf1f3d:::benchmark_data_smithi090_166897_object31179:head# 2024-05-15T23:39:58.096 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bb0fda1:::benchmark_data_smithi090_166897_object41640:head# 2024-05-15T23:39:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bb1a15d:::benchmark_data_smithi090_166897_object32144:head# 2024-05-15T23:39:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bb386c7:::benchmark_data_smithi090_166897_object42749:head# 2024-05-15T23:39:58.098 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bb413ee:::benchmark_data_smithi090_166897_object57470:head# 2024-05-15T23:39:58.098 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bb524d4:::benchmark_data_smithi090_166897_object46834:head# 2024-05-15T23:39:58.099 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bb5d1d8:::benchmark_data_smithi090_166897_object42028:head# 2024-05-15T23:39:58.099 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bb647fc:::benchmark_data_smithi090_166897_object3052:head# 2024-05-15T23:39:58.100 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bb7a7da:::benchmark_data_smithi090_166897_object62268:head# 2024-05-15T23:39:58.100 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bb7df8c:::benchmark_data_smithi090_166897_object4851:head# 2024-05-15T23:39:58.100 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bb94b77:::benchmark_data_smithi090_166897_object21872:head# 2024-05-15T23:39:58.101 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bb94ccf:::benchmark_data_smithi090_166897_object35721:head# 2024-05-15T23:39:58.101 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bba465a:::benchmark_data_smithi090_166897_object32794:head# 2024-05-15T23:39:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bbac679:::benchmark_data_smithi090_166897_object42270:head# 2024-05-15T23:39:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bbb0483:::benchmark_data_smithi090_166897_object55886:head# 2024-05-15T23:39:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bbc0107:::benchmark_data_smithi090_166897_object70590:head# 2024-05-15T23:39:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bbd5016:::benchmark_data_smithi090_166897_object40852:head# 2024-05-15T23:39:58.103 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bbe2f2b:::benchmark_data_smithi090_166897_object5518:head# 2024-05-15T23:39:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bbec0b3:::benchmark_data_smithi090_166897_object25023:head# 2024-05-15T23:39:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bc1192a:::benchmark_data_smithi090_166897_object40853:head# 2024-05-15T23:39:58.105 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bc20fe0:::benchmark_data_smithi090_166897_object48313:head# 2024-05-15T23:39:58.105 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bc2a19f:::benchmark_data_smithi090_166897_object405:head# 2024-05-15T23:39:58.105 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bc4c3cc:::benchmark_data_smithi090_166897_object16187:head# 2024-05-15T23:39:58.106 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bc62ef9:::benchmark_data_smithi090_166897_object64919:head# 2024-05-15T23:39:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bc83354:::benchmark_data_smithi090_166897_object3176:head# 2024-05-15T23:39:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bc8d67e:::benchmark_data_smithi090_166897_object61525:head# 2024-05-15T23:39:58.108 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bc9c103:::benchmark_data_smithi090_166897_object51197:head# 2024-05-15T23:39:58.108 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bc9d341:::benchmark_data_smithi090_166897_object13630:head# 2024-05-15T23:39:58.108 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bca9e8b:::benchmark_data_smithi090_166897_object9143:head# 2024-05-15T23:39:58.109 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bcbedb1:::benchmark_data_smithi090_166897_object7868:head# 2024-05-15T23:39:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bccd4c2:::benchmark_data_smithi090_166897_object32564:head# 2024-05-15T23:39:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bcdb99e:::benchmark_data_smithi090_166897_object19326:head# 2024-05-15T23:39:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bce29e6:::benchmark_data_smithi090_166897_object11615:head# 2024-05-15T23:39:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bce5fc4:::benchmark_data_smithi090_166897_object6737:head# 2024-05-15T23:39:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bce79f3:::benchmark_data_smithi090_166897_object16527:head# 2024-05-15T23:39:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bcfc83a:::benchmark_data_smithi090_166897_object47487:head# 2024-05-15T23:39:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bcff19c:::benchmark_data_smithi090_166897_object34716:head# 2024-05-15T23:39:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bd01539:::benchmark_data_smithi090_166897_object21637:head# 2024-05-15T23:39:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bd10921:::benchmark_data_smithi090_166897_object48206:head# 2024-05-15T23:39:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bd2f8fe:::benchmark_data_smithi090_166897_object7268:head# 2024-05-15T23:39:58.114 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bd4d792:::benchmark_data_smithi090_166897_object54182:head# 2024-05-15T23:39:58.114 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bd66b22:::benchmark_data_smithi090_166897_object65173:head# 2024-05-15T23:39:58.115 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bd76758:::benchmark_data_smithi090_166897_object15588:head# 2024-05-15T23:39:58.115 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bd99503:::benchmark_data_smithi090_166897_object26680:head# 2024-05-15T23:39:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bda2c58:::benchmark_data_smithi090_166897_object19209:head# 2024-05-15T23:39:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bda3fd6:::benchmark_data_smithi090_166897_object38995:head# 2024-05-15T23:39:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bdafc07:::benchmark_data_smithi090_166897_object28644:head# 2024-05-15T23:39:58.118 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bdd5b19:::benchmark_data_smithi090_166897_object15914:head# 2024-05-15T23:39:58.118 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bdde5c3:::benchmark_data_smithi090_166897_object16425:head# 2024-05-15T23:39:58.118 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6be02cb9:::benchmark_data_smithi090_166897_object1857:head# 2024-05-15T23:39:58.118 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6be065b4:::benchmark_data_smithi090_166897_object62696:head# 2024-05-15T23:39:58.119 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6be20e1f:::benchmark_data_smithi090_166897_object8995:head# 2024-05-15T23:39:58.121 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6be3a2be:::benchmark_data_smithi090_166897_object19284:head# 2024-05-15T23:39:58.121 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6be48854:::benchmark_data_smithi090_166897_object8819:head# 2024-05-15T23:39:58.121 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6be6a529:::benchmark_data_smithi090_166897_object66737:head# 2024-05-15T23:39:58.122 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6be6f2b2:::benchmark_data_smithi090_166897_object45123:head# 2024-05-15T23:39:58.122 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6be7272c:::benchmark_data_smithi090_166897_object53875:head# 2024-05-15T23:39:58.123 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6be7e4fd:::benchmark_data_smithi090_166897_object63766:head# 2024-05-15T23:39:58.123 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6be89202:::benchmark_data_smithi090_166897_object32472:head# 2024-05-15T23:39:58.124 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6be9fd6b:::benchmark_data_smithi090_166897_object36561:head# 2024-05-15T23:39:58.124 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6beab518:::benchmark_data_smithi090_166897_object68879:head# 2024-05-15T23:39:58.125 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bebe3f9:::benchmark_data_smithi090_166897_object15539:head# 2024-05-15T23:39:58.125 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bedb1fd:::benchmark_data_smithi090_166897_object24760:head# 2024-05-15T23:39:58.126 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bee48b0:::benchmark_data_smithi090_166897_object5628:head# 2024-05-15T23:39:58.126 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bee7f01:::benchmark_data_smithi090_166897_object12366:head# 2024-05-15T23:39:58.126 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bee8a78:::benchmark_data_smithi090_166897_object8830:head# 2024-05-15T23:39:58.127 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bee9d3e:::benchmark_data_smithi090_166897_object36808:head# 2024-05-15T23:39:58.127 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6beee1fe:::benchmark_data_smithi090_166897_object57650:head# 2024-05-15T23:39:58.128 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6beef4b6:::benchmark_data_smithi090_166897_object13352:head# 2024-05-15T23:39:58.128 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bef7f42:::benchmark_data_smithi090_166897_object62659:head# 2024-05-15T23:39:58.128 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bef9c61:::benchmark_data_smithi090_166897_object54817:head# 2024-05-15T23:39:58.130 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6befa9ae:::benchmark_data_smithi090_166897_object62938:head# 2024-05-15T23:39:58.130 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bf09108:::benchmark_data_smithi090_166897_object45727:head# 2024-05-15T23:39:58.131 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bf18151:::benchmark_data_smithi090_166897_object68424:head# 2024-05-15T23:39:58.131 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bf20d43:::benchmark_data_smithi090_166897_object4951:head# 2024-05-15T23:39:58.132 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bf26164:::benchmark_data_smithi090_166897_object27403:head# 2024-05-15T23:39:58.132 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bf3ec85:::benchmark_data_smithi090_166897_object11389:head# 2024-05-15T23:39:58.133 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bf44ec4:::benchmark_data_smithi090_166897_object30847:head# 2024-05-15T23:39:58.133 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bf50829:::benchmark_data_smithi090_166897_object18525:head# 2024-05-15T23:39:58.134 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bf63c47:::benchmark_data_smithi090_166897_object44325:head# 2024-05-15T23:39:58.134 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bf65aa9:::benchmark_data_smithi090_166897_object68978:head# 2024-05-15T23:39:58.134 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bf6746f:::benchmark_data_smithi090_166897_object3518:head# 2024-05-15T23:39:58.135 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bf844ac:::benchmark_data_smithi090_166897_object19083:head# 2024-05-15T23:39:58.135 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bf8a867:::benchmark_data_smithi090_166897_object57179:head# 2024-05-15T23:39:58.136 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bf8b6df:::benchmark_data_smithi090_166897_object61184:head# 2024-05-15T23:39:58.136 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bfa23b1:::benchmark_data_smithi090_166897_object67502:head# 2024-05-15T23:39:58.137 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bfbec64:::benchmark_data_smithi090_166897_object9351:head# 2024-05-15T23:39:58.137 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bfd22a6:::benchmark_data_smithi090_166897_object64435:head# 2024-05-15T23:39:58.139 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6bfedb04:::benchmark_data_smithi090_166897_object42918:head# 2024-05-15T23:39:58.139 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c002d30:::benchmark_data_smithi090_166897_object36315:head# 2024-05-15T23:39:58.139 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c0180b3:::benchmark_data_smithi090_166897_object5098:head# 2024-05-15T23:39:58.140 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c01a986:::benchmark_data_smithi090_166897_object62917:head# 2024-05-15T23:39:58.140 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c0381c1:::benchmark_data_smithi090_166897_object50001:head# 2024-05-15T23:39:58.141 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c04d63e:::benchmark_data_smithi090_166897_object42173:head# 2024-05-15T23:39:58.141 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c04f15b:::benchmark_data_smithi090_166897_object27721:head# 2024-05-15T23:39:58.142 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c0523b3:::benchmark_data_smithi090_166897_object63891:head# 2024-05-15T23:39:58.142 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c0574e9:::benchmark_data_smithi090_166897_object9723:head# 2024-05-15T23:39:58.143 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c05cdf3:::benchmark_data_smithi090_166897_object33838:head# 2024-05-15T23:39:58.143 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c05fa1f:::benchmark_data_smithi090_166897_object49886:head# 2024-05-15T23:39:58.144 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c070ef2:::benchmark_data_smithi090_166897_object24898:head# 2024-05-15T23:39:58.145 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c07d5aa:::benchmark_data_smithi090_166897_object4557:head# 2024-05-15T23:39:58.145 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c091db9:::benchmark_data_smithi090_166897_object27915:head# 2024-05-15T23:39:58.145 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c09bc89:::benchmark_data_smithi090_166897_object52685:head# 2024-05-15T23:39:58.146 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c0a5d93:::benchmark_data_smithi090_166897_object8464:head# 2024-05-15T23:39:58.146 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c0d08f8:::benchmark_data_smithi090_166897_object9738:head# 2024-05-15T23:39:58.147 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c0d56af:::benchmark_data_smithi090_166897_object43799:head# 2024-05-15T23:39:58.147 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c0f4549:::benchmark_data_smithi090_166897_object2914:head# 2024-05-15T23:39:58.148 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c1115c8:::benchmark_data_smithi090_166897_object30106:head# 2024-05-15T23:39:58.148 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c121749:::benchmark_data_smithi090_166897_object14104:head# 2024-05-15T23:39:58.149 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c1333be:::benchmark_data_smithi090_166897_object39440:head# 2024-05-15T23:39:58.149 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c147bfd:::benchmark_data_smithi090_166897_object33386:head# 2024-05-15T23:39:58.149 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c14fb61:::benchmark_data_smithi090_166897_object17828:head# 2024-05-15T23:39:58.150 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c159ee6:::benchmark_data_smithi090_166897_object68795:head# 2024-05-15T23:39:58.150 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c163f79:::benchmark_data_smithi090_166897_object8636:head# 2024-05-15T23:39:58.151 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c16b12b:::benchmark_data_smithi090_166897_object29106:head# 2024-05-15T23:39:58.151 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c1796b6:::benchmark_data_smithi090_166897_object18497:head# 2024-05-15T23:39:58.151 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c1aecf8:::benchmark_data_smithi090_166897_object68129:head# 2024-05-15T23:39:58.152 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c1b9823:::benchmark_data_smithi090_166897_object5443:head# 2024-05-15T23:39:58.153 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c1d7be5:::benchmark_data_smithi090_166897_object54913:head# 2024-05-15T23:39:58.153 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c1e1a91:::benchmark_data_smithi090_166897_object31329:head# 2024-05-15T23:39:58.154 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c1e2010:::benchmark_data_smithi090_166897_object63783:head# 2024-05-15T23:39:58.154 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c1ea7b8:::benchmark_data_smithi090_166897_object57252:head# 2024-05-15T23:39:58.155 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c1ff129:::benchmark_data_smithi090_166897_object20775:head# 2024-05-15T23:39:58.155 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c200b7e:::benchmark_data_smithi090_166897_object12986:head# 2024-05-15T23:39:58.156 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c20f156:::benchmark_data_smithi090_166897_object27855:head# 2024-05-15T23:39:58.156 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c210f4f:::benchmark_data_smithi090_166897_object6421:head# 2024-05-15T23:39:58.156 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c2254cd:::benchmark_data_smithi090_166897_object31834:head# 2024-05-15T23:39:58.157 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c227b35:::benchmark_data_smithi090_166897_object56996:head# 2024-05-15T23:39:58.157 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c228bd6:::benchmark_data_smithi090_166897_object30911:head# 2024-05-15T23:39:58.158 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c237da5:::benchmark_data_smithi090_166897_object15494:head# 2024-05-15T23:39:58.158 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c23bdb3:::benchmark_data_smithi090_166897_object35907:head# 2024-05-15T23:39:58.159 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c250d27:::benchmark_data_smithi090_166897_object47093:head# 2024-05-15T23:39:58.159 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c2606bb:::benchmark_data_smithi090_166897_object52945:head# 2024-05-15T23:39:58.160 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c26394d:::benchmark_data_smithi090_166897_object7754:head# 2024-05-15T23:39:58.160 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c26d958:::benchmark_data_smithi090_166897_object64140:head# 2024-05-15T23:39:58.160 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c27f61e:::benchmark_data_smithi090_166897_object61907:head# 2024-05-15T23:39:58.161 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c288fc3:::benchmark_data_smithi090_166897_object62115:head# 2024-05-15T23:39:58.161 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c2a2891:::benchmark_data_smithi090_166897_object51796:head# 2024-05-15T23:39:58.162 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c2ab45e:::benchmark_data_smithi090_166897_object20128:head# 2024-05-15T23:39:58.162 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c2bc407:::benchmark_data_smithi090_166897_object60448:head# 2024-05-15T23:39:58.163 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c2bf354:::benchmark_data_smithi090_166897_object1747:head# 2024-05-15T23:39:58.163 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c2ca06f:::benchmark_data_smithi090_166897_object40591:head# 2024-05-15T23:39:58.164 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c2cdd9a:::benchmark_data_smithi090_166897_object17863:head# 2024-05-15T23:39:58.164 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c2d09ea:::benchmark_data_smithi090_166897_object41352:head# 2024-05-15T23:39:58.165 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c2e17cc:::benchmark_data_smithi090_166897_object62405:head# 2024-05-15T23:39:58.166 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c2e4146:::benchmark_data_smithi090_166897_object40732:head# 2024-05-15T23:39:58.166 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c2e6373:::benchmark_data_smithi090_166897_object4564:head# 2024-05-15T23:39:58.167 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c2f021f:::benchmark_data_smithi090_166897_object9207:head# 2024-05-15T23:39:58.167 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c303b83:::benchmark_data_smithi090_166897_object32107:head# 2024-05-15T23:39:58.168 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c319280:::benchmark_data_smithi090_166897_object70046:head# 2024-05-15T23:39:58.168 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c31cd32:::benchmark_data_smithi090_166897_object41142:head# 2024-05-15T23:39:58.169 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c33dfc3:::benchmark_data_smithi090_166897_object39646:head# 2024-05-15T23:39:58.169 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c350fea:::benchmark_data_smithi090_166897_object40487:head# 2024-05-15T23:39:58.169 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c379365:::benchmark_data_smithi090_166897_object67901:head# 2024-05-15T23:39:58.170 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c38ecd1:::benchmark_data_smithi090_166897_object58130:head# 2024-05-15T23:39:58.170 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c3c3cc2:::benchmark_data_smithi090_166897_object68031:head# 2024-05-15T23:39:58.171 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c3e2c18:::benchmark_data_smithi090_166897_object36745:head# 2024-05-15T23:39:58.171 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c3e45e9:::benchmark_data_smithi090_166897_object20268:head# 2024-05-15T23:39:58.171 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c3f68dc:::benchmark_data_smithi090_166897_object7259:head# 2024-05-15T23:39:58.172 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c435f3d:::benchmark_data_smithi090_166897_object35967:head# 2024-05-15T23:39:58.173 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c44c68c:::benchmark_data_smithi090_166897_object56319:head# 2024-05-15T23:39:58.173 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c454b40:::benchmark_data_smithi090_166897_object61838:head# 2024-05-15T23:39:58.174 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c476952:::benchmark_data_smithi090_166897_object38898:head# 2024-05-15T23:39:58.175 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c47ea43:::benchmark_data_smithi090_166897_object43374:head# 2024-05-15T23:39:58.175 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c49dd58:::benchmark_data_smithi090_166897_object18379:head# 2024-05-15T23:39:58.176 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c4ae6e0:::benchmark_data_smithi090_166897_object56389:head# 2024-05-15T23:39:58.176 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c4c6261:::benchmark_data_smithi090_166897_object51858:head# 2024-05-15T23:39:58.177 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c4e5a08:::benchmark_data_smithi090_166897_object20924:head# 2024-05-15T23:39:58.177 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c4e7dc7:::benchmark_data_smithi090_166897_object56317:head# 2024-05-15T23:39:58.177 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c4f7784:::benchmark_data_smithi090_166897_object22760:head# 2024-05-15T23:39:58.178 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c50950e:::benchmark_data_smithi090_166897_object35898:head# 2024-05-15T23:39:58.178 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c51b09b:::benchmark_data_smithi090_166897_object42345:head# 2024-05-15T23:39:58.179 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c51d171:::benchmark_data_smithi090_166897_object66092:head# 2024-05-15T23:39:58.179 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c52d4fd:::benchmark_data_smithi090_166897_object22516:head# 2024-05-15T23:39:58.179 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c549c36:::benchmark_data_smithi090_166897_object51017:head# 2024-05-15T23:39:58.180 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c54a3ce:::benchmark_data_smithi090_166897_object57348:head# 2024-05-15T23:39:58.181 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c550e0f:::benchmark_data_smithi090_166897_object61416:head# 2024-05-15T23:39:58.181 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c553475:::benchmark_data_smithi090_166897_object54611:head# 2024-05-15T23:39:58.181 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c560091:::benchmark_data_smithi090_166897_object39063:head# 2024-05-15T23:39:58.182 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c576f32:::benchmark_data_smithi090_166897_object56926:head# 2024-05-15T23:39:58.182 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c5853b8:::benchmark_data_smithi090_166897_object48030:head# 2024-05-15T23:39:58.182 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c588d68:::benchmark_data_smithi090_166897_object31805:head# 2024-05-15T23:39:58.182 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c5a60ef:::benchmark_data_smithi090_166897_object5310:head# 2024-05-15T23:39:58.183 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c5af647:::benchmark_data_smithi090_166897_object49600:head# 2024-05-15T23:39:58.183 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c5ba499:::benchmark_data_smithi090_166897_object8458:head# 2024-05-15T23:39:58.184 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c5ce36d:::benchmark_data_smithi090_166897_object66479:head# 2024-05-15T23:39:58.185 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c5e39b7:::benchmark_data_smithi090_166897_object29435:head# 2024-05-15T23:39:58.185 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c5f84ce:::benchmark_data_smithi090_166897_object42646:head# 2024-05-15T23:39:58.186 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c6301ca:::benchmark_data_smithi090_166897_object2208:head# 2024-05-15T23:39:58.186 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c636577:::benchmark_data_smithi090_166897_object409:head# 2024-05-15T23:39:58.187 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c63a318:::benchmark_data_smithi090_166897_object52657:head# 2024-05-15T23:39:58.187 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c64ca1b:::benchmark_data_smithi090_166897_object44624:head# 2024-05-15T23:39:58.187 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c668344:::benchmark_data_smithi090_166897_object28936:head# 2024-05-15T23:39:58.188 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c66c18e:::benchmark_data_smithi090_166897_object6404:head# 2024-05-15T23:39:58.188 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c66e853:::benchmark_data_smithi090_166897_object60825:head# 2024-05-15T23:39:58.188 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c67941e:::benchmark_data_smithi090_166897_object18051:head# 2024-05-15T23:39:58.189 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c69858e:::benchmark_data_smithi090_166897_object4397:head# 2024-05-15T23:39:58.189 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c6ac09f:::benchmark_data_smithi090_166897_object26008:head# 2024-05-15T23:39:58.190 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c6caa38:::benchmark_data_smithi090_166897_object56916:head# 2024-05-15T23:39:58.190 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c6d93c4:::benchmark_data_smithi090_166897_object13122:head# 2024-05-15T23:39:58.190 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c707297:::benchmark_data_smithi090_166897_object46839:head# 2024-05-15T23:39:58.191 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c7085db:::benchmark_data_smithi090_166897_object64389:head# 2024-05-15T23:39:58.191 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c71c14a:::benchmark_data_smithi090_166897_object62289:head# 2024-05-15T23:39:58.192 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c72aecc:::benchmark_data_smithi090_166897_object54771:head# 2024-05-15T23:39:58.192 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c730878:::benchmark_data_smithi090_166897_object242:head# 2024-05-15T23:39:58.192 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c73dc9c:::benchmark_data_smithi090_166897_object51912:head# 2024-05-15T23:39:58.193 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c757c2b:::benchmark_data_smithi090_166897_object66411:head# 2024-05-15T23:39:58.194 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c783efc:::benchmark_data_smithi090_166897_object17822:head# 2024-05-15T23:39:58.195 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c78a596:::benchmark_data_smithi090_166897_object33728:head# 2024-05-15T23:39:58.195 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c79712a:::benchmark_data_smithi090_166897_object60475:head# 2024-05-15T23:39:58.195 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c79ee60:::benchmark_data_smithi090_166897_object45118:head# 2024-05-15T23:39:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c7a6316:::benchmark_data_smithi090_166897_object66746:head# 2024-05-15T23:39:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c7bbaf6:::benchmark_data_smithi090_166897_object68839:head# 2024-05-15T23:39:58.197 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c7d43e5:::benchmark_data_smithi090_166897_object522:head# 2024-05-15T23:39:58.197 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c7f39e1:::benchmark_data_smithi090_166897_object48455:head# 2024-05-15T23:39:58.197 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c8013e7:::benchmark_data_smithi090_166897_object40483:head# 2024-05-15T23:39:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c80c02a:::benchmark_data_smithi090_166897_object30010:head# 2024-05-15T23:39:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c810ca3:::benchmark_data_smithi090_166897_object14637:head# 2024-05-15T23:39:58.199 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c833b00:::benchmark_data_smithi090_166897_object57433:head# 2024-05-15T23:39:58.199 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c85a416:::benchmark_data_smithi090_166897_object61795:head# 2024-05-15T23:39:58.200 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c85bf2a:::benchmark_data_smithi090_166897_object21855:head# 2024-05-15T23:39:58.200 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c85ca7c:::benchmark_data_smithi090_166897_object38312:head# 2024-05-15T23:39:58.200 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c861328:::benchmark_data_smithi090_166897_object70588:head# 2024-05-15T23:39:58.201 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c871645:::benchmark_data_smithi090_166897_object668:head# 2024-05-15T23:39:58.201 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c87b990:::benchmark_data_smithi090_166897_object48819:head# 2024-05-15T23:39:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c8d2dde:::benchmark_data_smithi090_166897_object29093:head# 2024-05-15T23:39:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c8d97ef:::benchmark_data_smithi090_166897_object278:head# 2024-05-15T23:39:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c8dace5:::benchmark_data_smithi090_166897_object61103:head# 2024-05-15T23:39:58.203 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c8dbdc6:::benchmark_data_smithi090_166897_object3653:head# 2024-05-15T23:39:58.203 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c90644d:::benchmark_data_smithi090_166897_object4539:head# 2024-05-15T23:39:58.204 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c908019:::benchmark_data_smithi090_166897_object43070:head# 2024-05-15T23:39:58.204 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c90a283:::benchmark_data_smithi090_166897_object23644:head# 2024-05-15T23:39:58.205 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c914058:::benchmark_data_smithi090_166897_object66164:head# 2024-05-15T23:39:58.205 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c95b267:::benchmark_data_smithi090_166897_object46385:head# 2024-05-15T23:39:58.205 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c96155d:::benchmark_data_smithi090_166897_object65997:head# 2024-05-15T23:39:58.206 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c96f42e:::benchmark_data_smithi090_166897_object13427:head# 2024-05-15T23:39:58.206 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c96fadb:::benchmark_data_smithi090_166897_object39892:head# 2024-05-15T23:39:58.207 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c9704d3:::benchmark_data_smithi090_166897_object14398:head# 2024-05-15T23:39:58.207 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c97204c:::benchmark_data_smithi090_166897_object26051:head# 2024-05-15T23:39:58.208 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c9764a5:::benchmark_data_smithi090_166897_object67940:head# 2024-05-15T23:39:58.208 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c97dbfc:::benchmark_data_smithi090_166897_object64437:head# 2024-05-15T23:39:58.208 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c99b6e9:::benchmark_data_smithi090_166897_object52563:head# 2024-05-15T23:39:58.209 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c99f239:::benchmark_data_smithi090_166897_object51098:head# 2024-05-15T23:39:58.209 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c9a8156:::benchmark_data_smithi090_166897_object70100:head# 2024-05-15T23:39:58.210 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c9b2ed2:::benchmark_data_smithi090_166897_object40930:head# 2024-05-15T23:39:58.210 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c9cf02a:::benchmark_data_smithi090_166897_object35833:head# 2024-05-15T23:39:58.210 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c9d1d40:::benchmark_data_smithi090_166897_object36738:head# 2024-05-15T23:39:58.211 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c9dafb1:::benchmark_data_smithi090_166897_object41781:head# 2024-05-15T23:39:58.211 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6c9ec1bc:::benchmark_data_smithi090_166897_object25277:head# 2024-05-15T23:39:58.211 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ca0c7ff:::benchmark_data_smithi090_166897_object70232:head# 2024-05-15T23:39:58.212 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ca33fef:::benchmark_data_smithi090_166897_object64386:head# 2024-05-15T23:39:58.212 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ca3c2b9:::benchmark_data_smithi090_166897_object3086:head# 2024-05-15T23:39:58.213 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ca3cba5:::benchmark_data_smithi090_166897_object52245:head# 2024-05-15T23:39:58.213 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ca5a24e:::benchmark_data_smithi090_166897_object62393:head# 2024-05-15T23:39:58.214 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ca69000:::benchmark_data_smithi090_166897_object4454:head# 2024-05-15T23:39:58.214 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ca6e748:::benchmark_data_smithi090_166897_object19490:head# 2024-05-15T23:39:58.216 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ca7360b:::benchmark_data_smithi090_166897_object50002:head# 2024-05-15T23:39:58.216 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ca77c62:::benchmark_data_smithi090_166897_object43836:head# 2024-05-15T23:39:58.216 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ca7c319:::benchmark_data_smithi090_166897_object48653:head# 2024-05-15T23:39:58.216 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ca86729:::benchmark_data_smithi090_166897_object7124:head# 2024-05-15T23:39:58.217 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ca87ce1:::benchmark_data_smithi090_166897_object19613:head# 2024-05-15T23:39:58.217 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ca89519:::benchmark_data_smithi090_166897_object32519:head# 2024-05-15T23:39:58.217 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ca9fd86:::benchmark_data_smithi090_166897_object46340:head# 2024-05-15T23:39:58.218 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cab1b96:::benchmark_data_smithi090_166897_object12065:head# 2024-05-15T23:39:58.219 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cab216a:::benchmark_data_smithi090_166897_object33379:head# 2024-05-15T23:39:58.219 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cabd1c2:::benchmark_data_smithi090_166897_object12023:head# 2024-05-15T23:39:58.219 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cacb2ac:::benchmark_data_smithi090_166897_object66626:head# 2024-05-15T23:39:58.220 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cacbff5:::benchmark_data_smithi090_166897_object31472:head# 2024-05-15T23:39:58.220 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cafab02:::benchmark_data_smithi090_166897_object67204:head# 2024-05-15T23:39:58.220 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6caff58c:::benchmark_data_smithi090_166897_object13517:head# 2024-05-15T23:39:58.220 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cb1bf21:::benchmark_data_smithi090_166897_object20651:head# 2024-05-15T23:39:58.221 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cb307e9:::benchmark_data_smithi090_166897_object62332:head# 2024-05-15T23:39:58.221 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cb37dd8:::benchmark_data_smithi090_166897_object44956:head# 2024-05-15T23:39:58.221 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cb38d9c:::benchmark_data_smithi090_166897_object25399:head# 2024-05-15T23:39:58.222 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cb5a78d:::benchmark_data_smithi090_166897_object47530:head# 2024-05-15T23:39:58.222 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cb62378:::benchmark_data_smithi090_166897_object50563:head# 2024-05-15T23:39:58.223 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cb7788c:::benchmark_data_smithi090_166897_object47122:head# 2024-05-15T23:39:58.223 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cb8422d:::benchmark_data_smithi090_166897_object68988:head# 2024-05-15T23:39:58.223 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cb8dd16:::benchmark_data_smithi090_166897_object69280:head# 2024-05-15T23:39:58.224 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cbb852d:::benchmark_data_smithi090_166897_object54511:head# 2024-05-15T23:39:58.224 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cbb987d:::benchmark_data_smithi090_166897_object12784:head# 2024-05-15T23:39:58.225 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cbba1fe:::benchmark_data_smithi090_166897_object29028:head# 2024-05-15T23:39:58.225 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cbc4a60:::benchmark_data_smithi090_166897_object34026:head# 2024-05-15T23:39:58.225 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cbd4db2:::benchmark_data_smithi090_166897_object59474:head# 2024-05-15T23:39:58.226 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cbe8299:::benchmark_data_smithi090_166897_object21036:head# 2024-05-15T23:39:58.226 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cbf7406:::benchmark_data_smithi090_166897_object29417:head# 2024-05-15T23:39:58.227 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cc43968:::benchmark_data_smithi090_166897_object1948:head# 2024-05-15T23:39:58.227 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cc44d02:::benchmark_data_smithi090_166897_object1266:head# 2024-05-15T23:39:58.227 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cc4de8a:::benchmark_data_smithi090_166897_object30087:head# 2024-05-15T23:39:58.228 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cc53d5e:::benchmark_data_smithi090_166897_object47464:head# 2024-05-15T23:39:58.229 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cc56a67:::benchmark_data_smithi090_166897_object32611:head# 2024-05-15T23:39:58.229 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cc8f5b7:::benchmark_data_smithi090_166897_object48173:head# 2024-05-15T23:39:58.229 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cc9fb4e:::benchmark_data_smithi090_166897_object44106:head# 2024-05-15T23:39:58.230 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ccb5068:::benchmark_data_smithi090_166897_object26567:head# 2024-05-15T23:39:58.230 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ccb8e7a:::benchmark_data_smithi090_166897_object58652:head# 2024-05-15T23:39:58.231 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ccd5de1:::benchmark_data_smithi090_166897_object25861:head# 2024-05-15T23:39:58.231 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ccdc2c2:::benchmark_data_smithi090_166897_object66028:head# 2024-05-15T23:39:58.231 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cce8642:::benchmark_data_smithi090_166897_object69632:head# 2024-05-15T23:39:58.232 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ccecfbb:::benchmark_data_smithi090_166897_object47638:head# 2024-05-15T23:39:58.232 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ccfe464:::benchmark_data_smithi090_166897_object37403:head# 2024-05-15T23:39:58.233 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cd058f9:::benchmark_data_smithi090_166897_object56369:head# 2024-05-15T23:39:58.233 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cd1857c:::benchmark_data_smithi090_166897_object29641:head# 2024-05-15T23:39:58.233 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cd19d35:::benchmark_data_smithi090_166897_object26206:head# 2024-05-15T23:39:58.234 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cd2560d:::benchmark_data_smithi090_166897_object25397:head# 2024-05-15T23:39:58.234 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cd531c0:::benchmark_data_smithi090_166897_object45597:head# 2024-05-15T23:39:58.234 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cd6013b:::benchmark_data_smithi090_166897_object39326:head# 2024-05-15T23:39:58.235 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cd628d6:::benchmark_data_smithi090_166897_object2351:head# 2024-05-15T23:39:58.235 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cd79379:::benchmark_data_smithi090_166897_object3287:head# 2024-05-15T23:39:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cd7d01e:::benchmark_data_smithi090_166897_object31615:head# 2024-05-15T23:39:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cd8b644:::benchmark_data_smithi090_166897_object58089:head# 2024-05-15T23:39:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cd8cfac:::benchmark_data_smithi090_166897_object68866:head# 2024-05-15T23:39:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cdb0c71:::benchmark_data_smithi090_166897_object68166:head# 2024-05-15T23:39:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cdbe018:::benchmark_data_smithi090_166897_object42006:head# 2024-05-15T23:39:58.238 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cdc88a5:::benchmark_data_smithi090_166897_object52297:head# 2024-05-15T23:39:58.238 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cddc760:::benchmark_data_smithi090_166897_object45775:head# 2024-05-15T23:39:58.239 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cde92fc:::benchmark_data_smithi090_166897_object39903:head# 2024-05-15T23:39:58.239 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cde9df1:::benchmark_data_smithi090_166897_object6017:head# 2024-05-15T23:39:58.240 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cdf0fee:::benchmark_data_smithi090_166897_object58351:head# 2024-05-15T23:39:58.240 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cdfff8f:::benchmark_data_smithi090_166897_object30412:head# 2024-05-15T23:39:58.241 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ce1f561:::benchmark_data_smithi090_166897_object50432:head# 2024-05-15T23:39:58.241 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ce2f338:::benchmark_data_smithi090_166897_object39005:head# 2024-05-15T23:39:58.241 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ce2ff22:::benchmark_data_smithi090_166897_object53732:head# 2024-05-15T23:39:58.242 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ce35fc5:::benchmark_data_smithi090_166897_object64504:head# 2024-05-15T23:39:58.242 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ce38b91:::benchmark_data_smithi090_166897_object51762:head# 2024-05-15T23:39:58.243 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ce3cb48:::benchmark_data_smithi090_166897_object29129:head# 2024-05-15T23:39:58.243 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ce3eaf7:::benchmark_data_smithi090_166897_object3335:head# 2024-05-15T23:39:58.243 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ce4bf54:::benchmark_data_smithi090_166897_object47031:head# 2024-05-15T23:39:58.244 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ce56068:::benchmark_data_smithi090_166897_object37928:head# 2024-05-15T23:39:58.244 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ce6cb0d:::benchmark_data_smithi090_166897_object19913:head# 2024-05-15T23:39:58.245 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ce70b39:::benchmark_data_smithi090_166897_object21261:head# 2024-05-15T23:39:58.245 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ce7b0af:::benchmark_data_smithi090_166897_object44663:head# 2024-05-15T23:39:58.246 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ce96f57:::benchmark_data_smithi090_166897_object25241:head# 2024-05-15T23:39:58.246 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cea3b2c:::benchmark_data_smithi090_166897_object53003:head# 2024-05-15T23:39:58.247 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cea4a92:::benchmark_data_smithi090_166897_object61764:head# 2024-05-15T23:39:58.247 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ceac253:::benchmark_data_smithi090_166897_object39202:head# 2024-05-15T23:39:58.248 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cec32d8:::benchmark_data_smithi090_166897_object26006:head# 2024-05-15T23:39:58.248 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cedfcf4:::benchmark_data_smithi090_166897_object36479:head# 2024-05-15T23:39:58.248 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cf01528:::benchmark_data_smithi090_166897_object28494:head# 2024-05-15T23:39:58.249 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cf03133:::benchmark_data_smithi090_166897_object44219:head# 2024-05-15T23:39:58.249 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cf25b44:::benchmark_data_smithi090_166897_object52040:head# 2024-05-15T23:39:58.250 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cf417b6:::benchmark_data_smithi090_166897_object60898:head# 2024-05-15T23:39:58.250 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cf4213b:::benchmark_data_smithi090_166897_object13003:head# 2024-05-15T23:39:58.250 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cf55a0e:::benchmark_data_smithi090_166897_object43500:head# 2024-05-15T23:39:58.251 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cf5d0d2:::benchmark_data_smithi090_166897_object25457:head# 2024-05-15T23:39:58.251 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cf602a7:::benchmark_data_smithi090_166897_object65175:head# 2024-05-15T23:39:58.252 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cf6179e:::benchmark_data_smithi090_166897_object24089:head# 2024-05-15T23:39:58.252 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cf71829:::benchmark_data_smithi090_166897_object49061:head# 2024-05-15T23:39:58.252 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cf78346:::benchmark_data_smithi090_166897_object48917:head# 2024-05-15T23:39:58.253 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cf91aa6:::benchmark_data_smithi090_166897_object43335:head# 2024-05-15T23:39:58.253 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cfc4e3a:::benchmark_data_smithi090_166897_object57188:head# 2024-05-15T23:39:58.254 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cfca211:::benchmark_data_smithi090_166897_object70814:head# 2024-05-15T23:39:58.254 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cfd3672:::benchmark_data_smithi090_166897_object6307:head# 2024-05-15T23:39:58.257 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cfd3f8d:::benchmark_data_smithi090_166897_object4259:head# 2024-05-15T23:39:58.257 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cfdb9bd:::benchmark_data_smithi090_166897_object2014:head# 2024-05-15T23:39:58.257 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cfec99d:::benchmark_data_smithi090_166897_object47904:head# 2024-05-15T23:39:58.257 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cfef2f7:::benchmark_data_smithi090_166897_object21268:head# 2024-05-15T23:39:58.258 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6cffc49b:::benchmark_data_smithi090_166897_object29525:head# 2024-05-15T23:39:58.258 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d00e37e:::benchmark_data_smithi090_166897_object44657:head# 2024-05-15T23:39:58.258 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d00fb74:::benchmark_data_smithi090_166897_object4993:head# 2024-05-15T23:39:58.258 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d022611:::benchmark_data_smithi090_166897_object35908:head# 2024-05-15T23:39:58.260 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d0276d2:::benchmark_data_smithi090_166897_object35572:head# 2024-05-15T23:39:58.261 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d027ed8:::benchmark_data_smithi090_166897_object54171:head# 2024-05-15T23:39:58.261 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d03a29c:::benchmark_data_smithi090_166897_object19519:head# 2024-05-15T23:39:58.262 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d0566b4:::benchmark_data_smithi090_166897_object54219:head# 2024-05-15T23:39:58.265 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d06cd37:::benchmark_data_smithi090_166897_object51699:head# 2024-05-15T23:39:58.265 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d06fde9:::benchmark_data_smithi090_166897_object14241:head# 2024-05-15T23:39:58.265 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d074d0a:::benchmark_data_smithi090_166897_object9749:head# 2024-05-15T23:39:58.265 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d095bf0:::benchmark_data_smithi090_166897_object10808:head# 2024-05-15T23:39:58.266 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d0a596a:::benchmark_data_smithi090_166897_object23093:head# 2024-05-15T23:39:58.267 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d0ab949:::benchmark_data_smithi090_166897_object4810:head# 2024-05-15T23:39:58.267 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d0de3a8:::benchmark_data_smithi090_166897_object863:head# 2024-05-15T23:39:58.268 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d0e6355:::benchmark_data_smithi090_166897_object21424:head# 2024-05-15T23:39:58.268 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d0ed8b4:::benchmark_data_smithi090_166897_object43665:head# 2024-05-15T23:39:58.268 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d0ee517:::benchmark_data_smithi090_166897_object25193:head# 2024-05-15T23:39:58.269 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d0ef50e:::benchmark_data_smithi090_166897_object67275:head# 2024-05-15T23:39:58.269 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d10338f:::benchmark_data_smithi090_166897_object31066:head# 2024-05-15T23:39:58.269 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d10dafa:::benchmark_data_smithi090_166897_object18895:head# 2024-05-15T23:39:58.270 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d12d5aa:::benchmark_data_smithi090_166897_object37712:head# 2024-05-15T23:39:58.271 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d131ba6:::benchmark_data_smithi090_166897_object48599:head# 2024-05-15T23:39:58.271 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d145318:::benchmark_data_smithi090_166897_object27286:head# 2024-05-15T23:39:58.272 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d14f0ba:::benchmark_data_smithi090_166897_object43265:head# 2024-05-15T23:39:58.272 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d15096f:::benchmark_data_smithi090_166897_object11311:head# 2024-05-15T23:39:58.273 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d154462:::benchmark_data_smithi090_166897_object50515:head# 2024-05-15T23:39:58.273 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d157556:::benchmark_data_smithi090_166897_object21949:head# 2024-05-15T23:39:58.273 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d169746:::benchmark_data_smithi090_166897_object35590:head# 2024-05-15T23:39:58.274 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d170792:::benchmark_data_smithi090_166897_object56694:head# 2024-05-15T23:39:58.274 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d17823f:::benchmark_data_smithi090_166897_object55441:head# 2024-05-15T23:39:58.275 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d17a1da:::benchmark_data_smithi090_166897_object40205:head# 2024-05-15T23:39:58.275 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d181312:::benchmark_data_smithi090_166897_object23921:head# 2024-05-15T23:39:58.275 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d1c63e5:::benchmark_data_smithi090_166897_object19955:head# 2024-05-15T23:39:58.276 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d1ca02d:::benchmark_data_smithi090_166897_object62087:head# 2024-05-15T23:39:58.277 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d1ebc6f:::benchmark_data_smithi090_166897_object13573:head# 2024-05-15T23:39:58.277 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d1f57fa:::benchmark_data_smithi090_166897_object35290:head# 2024-05-15T23:39:58.277 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d200c44:::benchmark_data_smithi090_166897_object25566:head# 2024-05-15T23:39:58.278 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d202113:::benchmark_data_smithi090_166897_object66334:head# 2024-05-15T23:39:58.278 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d208487:::benchmark_data_smithi090_166897_object20020:head# 2024-05-15T23:39:58.279 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d20db1c:::benchmark_data_smithi090_166897_object52062:head# 2024-05-15T23:39:58.280 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d21370e:::benchmark_data_smithi090_166897_object45308:head# 2024-05-15T23:39:58.280 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d21cbf6:::benchmark_data_smithi090_166897_object58974:head# 2024-05-15T23:39:58.281 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d21f7b8:::benchmark_data_smithi090_166897_object49932:head# 2024-05-15T23:39:58.281 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d2239e1:::benchmark_data_smithi090_166897_object8841:head# 2024-05-15T23:39:58.281 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d232f07:::benchmark_data_smithi090_166897_object53684:head# 2024-05-15T23:39:58.282 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d236d4d:::benchmark_data_smithi090_166897_object40819:head# 2024-05-15T23:39:58.282 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d24e5ba:::benchmark_data_smithi090_166897_object5453:head# 2024-05-15T23:39:58.283 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d252da2:::benchmark_data_smithi090_166897_object60845:head# 2024-05-15T23:39:58.283 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d26f80d:::benchmark_data_smithi090_166897_object54030:head# 2024-05-15T23:39:58.283 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d278ea9:::benchmark_data_smithi090_166897_object23899:head# 2024-05-15T23:39:58.284 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d286cf0:::benchmark_data_smithi090_166897_object14378:head# 2024-05-15T23:39:58.284 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d2887d5:::benchmark_data_smithi090_166897_object36394:head# 2024-05-15T23:39:58.285 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d29869f:::benchmark_data_smithi090_166897_object40069:head# 2024-05-15T23:39:58.285 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d2a8ebd:::benchmark_data_smithi090_166897_object65415:head# 2024-05-15T23:39:58.285 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d2c5068:::benchmark_data_smithi090_166897_object9823:head# 2024-05-15T23:39:58.288 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d2cd99e:::benchmark_data_smithi090_166897_object42423:head# 2024-05-15T23:39:58.288 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d2e160c:::benchmark_data_smithi090_166897_object35708:head# 2024-05-15T23:39:58.290 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d2f79a8:::benchmark_data_smithi090_166897_object58743:head# 2024-05-15T23:39:58.290 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d3301d1:::benchmark_data_smithi090_166897_object58040:head# 2024-05-15T23:39:58.290 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d34ea08:::benchmark_data_smithi090_166897_object50659:head# 2024-05-15T23:39:58.290 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d353e1f:::benchmark_data_smithi090_166897_object64688:head# 2024-05-15T23:39:58.295 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d371d85:::benchmark_data_smithi090_166897_object27156:head# 2024-05-15T23:39:58.295 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d379128:::benchmark_data_smithi090_166897_object54200:head# 2024-05-15T23:39:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d38fc2c:::benchmark_data_smithi090_166897_object11936:head# 2024-05-15T23:39:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d3ca66d:::benchmark_data_smithi090_166897_object47774:head# 2024-05-15T23:39:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d3d1e99:::benchmark_data_smithi090_166897_object38622:head# 2024-05-15T23:39:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d3ef90b:::benchmark_data_smithi090_166897_object57451:head# 2024-05-15T23:39:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d3fabd6:::benchmark_data_smithi090_166897_object39095:head# 2024-05-15T23:39:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d433d81:::benchmark_data_smithi090_166897_object48361:head# 2024-05-15T23:39:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d4436fa:::benchmark_data_smithi090_166897_object38424:head# 2024-05-15T23:39:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d45cd1e:::benchmark_data_smithi090_166897_object26107:head# 2024-05-15T23:39:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d46ac55:::benchmark_data_smithi090_166897_object48837:head# 2024-05-15T23:39:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d48314d:::benchmark_data_smithi090_166897_object65234:head# 2024-05-15T23:39:58.296 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d49eb09:::benchmark_data_smithi090_166897_object66685:head# 2024-05-15T23:39:58.297 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d4a483b:::benchmark_data_smithi090_166897_object64564:head# 2024-05-15T23:39:58.297 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d4ab133:::benchmark_data_smithi090_166897_object4079:head# 2024-05-15T23:39:58.297 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d4c9f1d:::benchmark_data_smithi090_166897_object63874:head# 2024-05-15T23:39:58.298 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d4e1b88:::benchmark_data_smithi090_166897_object2745:head# 2024-05-15T23:39:58.298 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d4e27f9:::benchmark_data_smithi090_166897_object678:head# 2024-05-15T23:39:58.299 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d4e3575:::benchmark_data_smithi090_166897_object25895:head# 2024-05-15T23:39:58.299 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d4f187c:::benchmark_data_smithi090_166897_object8415:head# 2024-05-15T23:39:58.300 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d4f3490:::benchmark_data_smithi090_166897_object27283:head# 2024-05-15T23:39:58.301 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d4f8f69:::benchmark_data_smithi090_166897_object64841:head# 2024-05-15T23:39:58.301 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d51383b:::benchmark_data_smithi090_166897_object51274:head# 2024-05-15T23:39:58.301 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d513fda:::benchmark_data_smithi090_166897_object20094:head# 2024-05-15T23:39:58.302 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d544fff:::benchmark_data_smithi090_166897_object41374:head# 2024-05-15T23:39:58.304 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d54c4b5:::benchmark_data_smithi090_166897_object55089:head# 2024-05-15T23:39:58.304 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d54f27c:::benchmark_data_smithi090_166897_object47419:head# 2024-05-15T23:39:58.304 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d55b264:::benchmark_data_smithi090_166897_object4540:head# 2024-05-15T23:39:58.304 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d574b9e:::benchmark_data_smithi090_166897_object30612:head# 2024-05-15T23:39:58.304 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d582911:::benchmark_data_smithi090_166897_object64071:head# 2024-05-15T23:39:58.305 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d593dc8:::benchmark_data_smithi090_166897_object54261:head# 2024-05-15T23:39:58.305 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d59f884:::benchmark_data_smithi090_166897_object2170:head# 2024-05-15T23:39:58.305 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d5b60ea:::benchmark_data_smithi090_166897_object16643:head# 2024-05-15T23:39:58.306 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d5b9621:::benchmark_data_smithi090_166897_object18781:head# 2024-05-15T23:39:58.307 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d5b9665:::benchmark_data_smithi090_166897_object17008:head# 2024-05-15T23:39:58.312 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d5c0edf:::benchmark_data_smithi090_166897_object15432:head# 2024-05-15T23:39:58.312 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d5d1545:::benchmark_data_smithi090_166897_object53295:head# 2024-05-15T23:39:58.312 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d5d521a:::benchmark_data_smithi090_166897_object53683:head# 2024-05-15T23:39:58.312 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d5d8d4e:::benchmark_data_smithi090_166897_object52179:head# 2024-05-15T23:39:58.312 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d5ec304:::benchmark_data_smithi090_166897_object49984:head# 2024-05-15T23:39:58.312 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d5fbce0:::benchmark_data_smithi090_166897_object60395:head# 2024-05-15T23:39:58.312 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d6077e4:::benchmark_data_smithi090_166897_object1663:head# 2024-05-15T23:39:58.313 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d615d6a:::benchmark_data_smithi090_166897_object12634:head# 2024-05-15T23:39:58.313 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d63d546:::benchmark_data_smithi090_166897_object47578:head# 2024-05-15T23:39:58.313 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d6476aa:::benchmark_data_smithi090_166897_object41185:head# 2024-05-15T23:39:58.313 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d675ebf:::benchmark_data_smithi090_166897_object11495:head# 2024-05-15T23:39:58.314 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d67c4cd:::benchmark_data_smithi090_166897_object68960:head# 2024-05-15T23:39:58.315 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d684839:::benchmark_data_smithi090_166897_object27456:head# 2024-05-15T23:39:58.315 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d688e33:::benchmark_data_smithi090_166897_object35178:head# 2024-05-15T23:39:58.315 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d69f1f4:::benchmark_data_smithi090_166897_object28183:head# 2024-05-15T23:39:58.315 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d6c3097:::benchmark_data_smithi090_166897_object301:head# 2024-05-15T23:39:58.316 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d6c9282:::benchmark_data_smithi090_166897_object12312:head# 2024-05-15T23:39:58.318 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d6c942e:::benchmark_data_smithi090_166897_object10745:head# 2024-05-15T23:39:58.319 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d6d4567:::benchmark_data_smithi090_166897_object41266:head# 2024-05-15T23:39:58.319 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d6e20ba:::benchmark_data_smithi090_166897_object29366:head# 2024-05-15T23:39:58.319 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d6f8ae7:::benchmark_data_smithi090_166897_object4377:head# 2024-05-15T23:39:58.320 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d6fdb75:::benchmark_data_smithi090_166897_object67184:head# 2024-05-15T23:39:58.320 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d6fed72:::benchmark_data_smithi090_166897_object23416:head# 2024-05-15T23:39:58.321 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d7534d1:::benchmark_data_smithi090_166897_object23936:head# 2024-05-15T23:39:58.321 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d755ad9:::benchmark_data_smithi090_166897_object6078:head# 2024-05-15T23:39:58.321 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d764aaf:::benchmark_data_smithi090_166897_object2891:head# 2024-05-15T23:39:58.323 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d7822b7:::benchmark_data_smithi090_166897_object6922:head# 2024-05-15T23:39:58.323 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d78d3af:::benchmark_data_smithi090_166897_object21959:head# 2024-05-15T23:39:58.323 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d79cea9:::benchmark_data_smithi090_166897_object2213:head# 2024-05-15T23:39:58.323 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d7a811d:::benchmark_data_smithi090_166897_object3107:head# 2024-05-15T23:39:58.323 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d7d09df:::benchmark_data_smithi090_166897_object67151:head# 2024-05-15T23:39:58.324 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d7d87ea:::benchmark_data_smithi090_166897_object20009:head# 2024-05-15T23:39:58.324 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d7eec0f:::benchmark_data_smithi090_166897_object3315:head# 2024-05-15T23:39:58.325 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d7f55cf:::benchmark_data_smithi090_166897_object64940:head# 2024-05-15T23:39:58.325 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d82cb4b:::benchmark_data_smithi090_166897_object28167:head# 2024-05-15T23:39:58.326 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d835c50:::benchmark_data_smithi090_166897_object31910:head# 2024-05-15T23:39:58.326 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d846773:::benchmark_data_smithi090_166897_object25490:head# 2024-05-15T23:39:58.326 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d8470ef:::benchmark_data_smithi090_166897_object7536:head# 2024-05-15T23:39:58.327 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d847e88:::benchmark_data_smithi090_166897_object41924:head# 2024-05-15T23:39:58.327 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d874617:::benchmark_data_smithi090_166897_object62646:head# 2024-05-15T23:39:58.327 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d8752a0:::benchmark_data_smithi090_166897_object32237:head# 2024-05-15T23:39:58.330 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d875e97:::benchmark_data_smithi090_166897_object14629:head# 2024-05-15T23:39:58.330 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d87caf7:::benchmark_data_smithi090_166897_object1107:head# 2024-05-15T23:39:58.330 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d88c8a8:::benchmark_data_smithi090_166897_object17172:head# 2024-05-15T23:39:58.331 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d89a033:::benchmark_data_smithi090_166897_object24517:head# 2024-05-15T23:39:58.331 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d8a763d:::benchmark_data_smithi090_166897_object48352:head# 2024-05-15T23:39:58.332 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d8aef5d:::benchmark_data_smithi090_166897_object37510:head# 2024-05-15T23:39:58.332 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d8bb8dc:::benchmark_data_smithi090_166897_object17539:head# 2024-05-15T23:39:58.333 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d8bfb02:::benchmark_data_smithi090_166897_object52313:head# 2024-05-15T23:39:58.333 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d8d344f:::benchmark_data_smithi090_166897_object61307:head# 2024-05-15T23:39:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d8d5db5:::benchmark_data_smithi090_166897_object71016:head# 2024-05-15T23:39:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d8e2adc:::benchmark_data_smithi090_166897_object46940:head# 2024-05-15T23:39:58.335 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d8edb1c:::benchmark_data_smithi090_166897_object11205:head# 2024-05-15T23:39:58.335 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d8ef781:::benchmark_data_smithi090_166897_object14826:head# 2024-05-15T23:39:58.336 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d90412a:::benchmark_data_smithi090_166897_object50124:head# 2024-05-15T23:39:58.336 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d90454c:::benchmark_data_smithi090_166897_object40956:head# 2024-05-15T23:39:58.337 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d90e076:::benchmark_data_smithi090_166897_object1148:head# 2024-05-15T23:39:58.337 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d926eb7:::benchmark_data_smithi090_166897_object50433:head# 2024-05-15T23:39:58.338 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d94f84f:::benchmark_data_smithi090_166897_object16610:head# 2024-05-15T23:39:58.339 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d95d17a:::benchmark_data_smithi090_166897_object36448:head# 2024-05-15T23:39:58.339 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d976861:::benchmark_data_smithi090_166897_object21448:head# 2024-05-15T23:39:58.339 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d97f85e:::benchmark_data_smithi090_166897_object62247:head# 2024-05-15T23:39:58.340 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d98e548:::benchmark_data_smithi090_166897_object39532:head# 2024-05-15T23:39:58.340 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d99fe0f:::benchmark_data_smithi090_166897_object7809:head# 2024-05-15T23:39:58.341 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d9aa290:::benchmark_data_smithi090_166897_object65296:head# 2024-05-15T23:39:58.341 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d9ad261:::benchmark_data_smithi090_166897_object21308:head# 2024-05-15T23:39:58.341 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d9b0e57:::benchmark_data_smithi090_166897_object40182:head# 2024-05-15T23:39:58.342 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d9b2dc3:::benchmark_data_smithi090_166897_object14210:head# 2024-05-15T23:39:58.344 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d9d04c5:::benchmark_data_smithi090_166897_object23792:head# 2024-05-15T23:39:58.344 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6d9fefdf:::benchmark_data_smithi090_166897_object17734:head# 2024-05-15T23:39:58.344 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6da05911:::benchmark_data_smithi090_166897_object32421:head# 2024-05-15T23:39:58.344 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6da2ed01:::benchmark_data_smithi090_166897_object64061:head# 2024-05-15T23:39:58.344 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6da41869:::benchmark_data_smithi090_166897_object51945:head# 2024-05-15T23:39:58.345 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6da5ac9f:::benchmark_data_smithi090_166897_object55712:head# 2024-05-15T23:39:58.345 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6da5c2ae:::benchmark_data_smithi090_166897_object13753:head# 2024-05-15T23:39:58.346 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6da70d15:::benchmark_data_smithi090_166897_object11533:head# 2024-05-15T23:39:58.346 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6da7f7ab:::benchmark_data_smithi090_166897_object10142:head# 2024-05-15T23:39:58.347 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6da8b468:::benchmark_data_smithi090_166897_object35915:head# 2024-05-15T23:39:58.347 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6da9bd04:::benchmark_data_smithi090_166897_object34810:head# 2024-05-15T23:39:58.347 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6da9dcc3:::benchmark_data_smithi090_166897_object11280:head# 2024-05-15T23:39:58.348 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6daae3df:::benchmark_data_smithi090_166897_object8581:head# 2024-05-15T23:39:58.348 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dacb6fc:::benchmark_data_smithi090_166897_object17343:head# 2024-05-15T23:39:58.348 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dae77c2:::benchmark_data_smithi090_166897_object12220:head# 2024-05-15T23:39:58.349 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dae861b:::benchmark_data_smithi090_166897_object58066:head# 2024-05-15T23:39:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6daea24e:::benchmark_data_smithi090_166897_object70026:head# 2024-05-15T23:39:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6daeb677:::benchmark_data_smithi090_166897_object58012:head# 2024-05-15T23:39:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6daee3e0:::benchmark_data_smithi090_166897_object56341:head# 2024-05-15T23:39:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db04e50:::benchmark_data_smithi090_166897_object3354:head# 2024-05-15T23:39:58.353 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db0a74f:::benchmark_data_smithi090_166897_object63175:head# 2024-05-15T23:39:58.353 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db26e4c:::benchmark_data_smithi090_166897_object34746:head# 2024-05-15T23:39:58.353 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db2a8e0:::benchmark_data_smithi090_166897_object50218:head# 2024-05-15T23:39:58.354 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db2e6e1:::benchmark_data_smithi090_166897_object54136:head# 2024-05-15T23:39:58.355 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db38273:::benchmark_data_smithi090_166897_object22795:head# 2024-05-15T23:39:58.355 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db3f99d:::benchmark_data_smithi090_166897_object56790:head# 2024-05-15T23:39:58.355 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db4c7c2:::benchmark_data_smithi090_166897_object26814:head# 2024-05-15T23:39:58.356 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db5d774:::benchmark_data_smithi090_166897_object21131:head# 2024-05-15T23:39:58.356 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db5e084:::benchmark_data_smithi090_166897_object47079:head# 2024-05-15T23:39:58.357 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db680f9:::benchmark_data_smithi090_166897_object31285:head# 2024-05-15T23:39:58.357 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db72e61:::benchmark_data_smithi090_166897_object42534:head# 2024-05-15T23:39:58.358 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db867c3:::benchmark_data_smithi090_166897_object893:head# 2024-05-15T23:39:58.358 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db8f7c4:::benchmark_data_smithi090_166897_object9451:head# 2024-05-15T23:39:58.358 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db9a2ab:::benchmark_data_smithi090_166897_object71052:head# 2024-05-15T23:39:58.359 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6db9f509:::benchmark_data_smithi090_166897_object57946:head# 2024-05-15T23:39:58.359 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dba62fe:::benchmark_data_smithi090_166897_object51966:head# 2024-05-15T23:39:58.359 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dba8c4f:::benchmark_data_smithi090_166897_object40275:head# 2024-05-15T23:39:58.359 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dbb8c84:::benchmark_data_smithi090_166897_object68751:head# 2024-05-15T23:39:58.360 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dbcbe69:::benchmark_data_smithi090_166897_object30316:head# 2024-05-15T23:39:58.360 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dbdc34c:::benchmark_data_smithi090_166897_object62457:head# 2024-05-15T23:39:58.361 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dbdd7b2:::benchmark_data_smithi090_166897_object57662:head# 2024-05-15T23:39:58.361 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dbeea9c:::benchmark_data_smithi090_166897_object46890:head# 2024-05-15T23:39:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dbf48ba:::benchmark_data_smithi090_166897_object50074:head# 2024-05-15T23:39:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dc0baf3:::benchmark_data_smithi090_166897_object54831:head# 2024-05-15T23:39:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dc1d35c:::benchmark_data_smithi090_166897_object38162:head# 2024-05-15T23:39:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dc1fa1a:::benchmark_data_smithi090_166897_object47632:head# 2024-05-15T23:39:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dc20491:::benchmark_data_smithi090_166897_object54277:head# 2024-05-15T23:39:58.364 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dc355e6:::benchmark_data_smithi090_166897_object2602:head# 2024-05-15T23:39:58.364 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dc395eb:::benchmark_data_smithi090_166897_object51049:head# 2024-05-15T23:39:58.365 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dc55755:::benchmark_data_smithi090_166897_object45285:head# 2024-05-15T23:39:58.366 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dc5a2b0:::benchmark_data_smithi090_166897_object44866:head# 2024-05-15T23:39:58.366 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dc5a939:::benchmark_data_smithi090_166897_object68010:head# 2024-05-15T23:39:58.366 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dc64581:::benchmark_data_smithi090_166897_object56435:head# 2024-05-15T23:39:58.367 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dc84a2d:::benchmark_data_smithi090_166897_object61628:head# 2024-05-15T23:39:58.368 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dca99c3:::benchmark_data_smithi090_166897_object23712:head# 2024-05-15T23:39:58.368 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dcab96f:::benchmark_data_smithi090_166897_object68265:head# 2024-05-15T23:39:58.368 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dcaf88c:::benchmark_data_smithi090_166897_object40568:head# 2024-05-15T23:39:58.369 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dcaf9b3:::benchmark_data_smithi090_166897_object16608:head# 2024-05-15T23:39:58.369 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dccbdfe:::benchmark_data_smithi090_166897_object17133:head# 2024-05-15T23:39:58.369 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dcdb128:::benchmark_data_smithi090_166897_object54194:head# 2024-05-15T23:39:58.370 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dcec699:::benchmark_data_smithi090_166897_object12030:head# 2024-05-15T23:39:58.370 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dcf1550:::benchmark_data_smithi090_166897_object26992:head# 2024-05-15T23:39:58.371 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dcf7b39:::benchmark_data_smithi090_166897_object25910:head# 2024-05-15T23:39:58.371 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dd0f3e0:::benchmark_data_smithi090_166897_object17529:head# 2024-05-15T23:39:58.372 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dd24d19:::benchmark_data_smithi090_166897_object32476:head# 2024-05-15T23:39:58.372 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dd3757c:::benchmark_data_smithi090_166897_object61383:head# 2024-05-15T23:39:58.372 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dd4b006:::benchmark_data_smithi090_166897_object23751:head# 2024-05-15T23:39:58.373 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dd60170:::benchmark_data_smithi090_166897_object50569:head# 2024-05-15T23:39:58.373 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dd6aff8:::benchmark_data_smithi090_166897_object34818:head# 2024-05-15T23:39:58.373 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dd75818:::benchmark_data_smithi090_166897_object38014:head# 2024-05-15T23:39:58.374 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dd81fbc:::benchmark_data_smithi090_166897_object55894:head# 2024-05-15T23:39:58.374 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dd8687c:::benchmark_data_smithi090_166897_object62176:head# 2024-05-15T23:39:58.375 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dd90bbf:::benchmark_data_smithi090_166897_object38821:head# 2024-05-15T23:39:58.376 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ddae45a:::benchmark_data_smithi090_166897_object42176:head# 2024-05-15T23:39:58.376 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ddb9095:::benchmark_data_smithi090_166897_object53247:head# 2024-05-15T23:39:58.376 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ddbff34:::benchmark_data_smithi090_166897_object38284:head# 2024-05-15T23:39:58.377 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ddc6c98:::benchmark_data_smithi090_166897_object61572:head# 2024-05-15T23:39:58.377 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ddd9df5:::benchmark_data_smithi090_166897_object24602:head# 2024-05-15T23:39:58.377 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6de0e4ab:::benchmark_data_smithi090_166897_object2031:head# 2024-05-15T23:39:58.378 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6de139c0:::benchmark_data_smithi090_166897_object4436:head# 2024-05-15T23:39:58.380 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6de2f075:::benchmark_data_smithi090_166897_object18439:head# 2024-05-15T23:39:58.381 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6de368e2:::benchmark_data_smithi090_166897_object12823:head# 2024-05-15T23:39:58.381 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6de394cb:::benchmark_data_smithi090_166897_object68054:head# 2024-05-15T23:39:58.381 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6de3f654:::benchmark_data_smithi090_166897_object48394:head# 2024-05-15T23:39:58.382 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6de48a2d:::benchmark_data_smithi090_166897_object13224:head# 2024-05-15T23:39:58.386 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6de4d13e:::benchmark_data_smithi090_166897_object12843:head# 2024-05-15T23:39:58.386 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6de6030d:::benchmark_data_smithi090_166897_object30434:head# 2024-05-15T23:39:58.386 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6de73af9:::benchmark_data_smithi090_166897_object30441:head# 2024-05-15T23:39:58.387 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dec12bc:::benchmark_data_smithi090_166897_object7596:head# 2024-05-15T23:39:58.388 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6deca4b4:::benchmark_data_smithi090_166897_object34201:head# 2024-05-15T23:39:58.388 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6def08bb:::benchmark_data_smithi090_166897_object62100:head# 2024-05-15T23:39:58.388 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6def6847:::benchmark_data_smithi090_166897_object18159:head# 2024-05-15T23:39:58.389 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6def9b57:::benchmark_data_smithi090_166897_object17555:head# 2024-05-15T23:39:58.392 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df011e1:::benchmark_data_smithi090_166897_object48305:head# 2024-05-15T23:39:58.396 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df0784e:::benchmark_data_smithi090_166897_object56946:head# 2024-05-15T23:39:58.396 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df13f3b:::benchmark_data_smithi090_166897_object61088:head# 2024-05-15T23:39:58.396 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df142cf:::benchmark_data_smithi090_166897_object15983:head# 2024-05-15T23:39:58.396 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df1afaa:::benchmark_data_smithi090_166897_object26288:head# 2024-05-15T23:39:58.396 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df1e4ce:::benchmark_data_smithi090_166897_object31883:head# 2024-05-15T23:39:58.397 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df236ad:::benchmark_data_smithi090_166897_object34479:head# 2024-05-15T23:39:58.397 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df357ee:::benchmark_data_smithi090_166897_object63864:head# 2024-05-15T23:39:58.397 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df3faca:::benchmark_data_smithi090_166897_object65182:head# 2024-05-15T23:39:58.398 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df41038:::benchmark_data_smithi090_166897_object20327:head# 2024-05-15T23:39:58.400 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df430db:::benchmark_data_smithi090_166897_object10927:head# 2024-05-15T23:39:58.400 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df4316b:::benchmark_data_smithi090_166897_object10871:head# 2024-05-15T23:39:58.400 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df48d90:::benchmark_data_smithi090_166897_object10698:head# 2024-05-15T23:39:58.400 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df516f9:::benchmark_data_smithi090_166897_object42221:head# 2024-05-15T23:39:58.400 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df57f98:::benchmark_data_smithi090_166897_object13348:head# 2024-05-15T23:39:58.400 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df641d5:::benchmark_data_smithi090_166897_object62045:head# 2024-05-15T23:39:58.401 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df72361:::benchmark_data_smithi090_166897_object33055:head# 2024-05-15T23:39:58.401 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df86e2a:::benchmark_data_smithi090_166897_object25704:head# 2024-05-15T23:39:58.402 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df8ee1d:::benchmark_data_smithi090_166897_object63677:head# 2024-05-15T23:39:58.402 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df9464d:::benchmark_data_smithi090_166897_object34567:head# 2024-05-15T23:39:58.403 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6df9d80f:::benchmark_data_smithi090_166897_object29942:head# 2024-05-15T23:39:58.403 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dfa2b18:::benchmark_data_smithi090_166897_object33335:head# 2024-05-15T23:39:58.403 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dfb3ea5:::benchmark_data_smithi090_166897_object2061:head# 2024-05-15T23:39:58.404 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dfb4c0a:::benchmark_data_smithi090_166897_object4673:head# 2024-05-15T23:39:58.405 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dfb55ed:::benchmark_data_smithi090_166897_object30769:head# 2024-05-15T23:39:58.406 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dfb56c0:::benchmark_data_smithi090_166897_object38354:head# 2024-05-15T23:39:58.406 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dfceabf:::benchmark_data_smithi090_166897_object21414:head# 2024-05-15T23:39:58.406 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6dfe6b65:::benchmark_data_smithi090_166897_object2056:head# 2024-05-15T23:39:58.406 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e03a927:::benchmark_data_smithi090_166897_object63524:head# 2024-05-15T23:39:58.406 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e048561:::benchmark_data_smithi090_166897_object18563:head# 2024-05-15T23:39:58.407 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e064658:::benchmark_data_smithi090_166897_object56735:head# 2024-05-15T23:39:58.407 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e086181:::benchmark_data_smithi090_166897_object48034:head# 2024-05-15T23:39:58.408 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e088615:::benchmark_data_smithi090_166897_object14565:head# 2024-05-15T23:39:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e08b987:::benchmark_data_smithi090_166897_object61990:head# 2024-05-15T23:39:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e094a4d:::benchmark_data_smithi090_166897_object32737:head# 2024-05-15T23:39:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e0b0e29:::benchmark_data_smithi090_166897_object25353:head# 2024-05-15T23:39:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e0b103d:::benchmark_data_smithi090_166897_object66243:head# 2024-05-15T23:39:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e0c51b3:::benchmark_data_smithi090_166897_object5305:head# 2024-05-15T23:39:58.412 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e0d808e:::benchmark_data_smithi090_166897_object18315:head# 2024-05-15T23:39:58.412 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e11c27d:::benchmark_data_smithi090_166897_object34358:head# 2024-05-15T23:39:58.413 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e121a8a:::benchmark_data_smithi090_166897_object67076:head# 2024-05-15T23:39:58.413 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e12d6a4:::benchmark_data_smithi090_166897_object39187:head# 2024-05-15T23:39:58.415 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e136a1f:::benchmark_data_smithi090_166897_object42075:head# 2024-05-15T23:39:58.415 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e13a3f1:::benchmark_data_smithi090_166897_object43300:head# 2024-05-15T23:39:58.415 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e13d6a6:::benchmark_data_smithi090_166897_object67869:head# 2024-05-15T23:39:58.416 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e1640d0:::benchmark_data_smithi090_166897_object43259:head# 2024-05-15T23:39:58.416 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e1748b9:::benchmark_data_smithi090_166897_object69254:head# 2024-05-15T23:39:58.417 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e180ec6:::benchmark_data_smithi090_166897_object68780:head# 2024-05-15T23:39:58.417 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e18a1f0:::benchmark_data_smithi090_166897_object55447:head# 2024-05-15T23:39:58.417 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e1a5525:::benchmark_data_smithi090_166897_object58873:head# 2024-05-15T23:39:58.418 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e1c68e2:::benchmark_data_smithi090_166897_object42130:head# 2024-05-15T23:39:58.419 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e1d278c:::benchmark_data_smithi090_166897_object35376:head# 2024-05-15T23:39:58.419 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e1db69b:::benchmark_data_smithi090_166897_object23065:head# 2024-05-15T23:39:58.419 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e1dcb6e:::benchmark_data_smithi090_166897_object28011:head# 2024-05-15T23:39:58.420 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e1de823:::benchmark_data_smithi090_166897_object61507:head# 2024-05-15T23:39:58.420 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e1e418a:::benchmark_data_smithi090_166897_object67830:head# 2024-05-15T23:39:58.421 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e1eba9f:::benchmark_data_smithi090_166897_object61592:head# 2024-05-15T23:39:58.421 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e1f60e3:::benchmark_data_smithi090_166897_object68904:head# 2024-05-15T23:39:58.421 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e20a4fa:::benchmark_data_smithi090_166897_object40286:head# 2024-05-15T23:39:58.422 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e20fdca:::benchmark_data_smithi090_166897_object37545:head# 2024-05-15T23:39:58.422 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e21365a:::benchmark_data_smithi090_166897_object4058:head# 2024-05-15T23:39:58.423 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e216c56:::benchmark_data_smithi090_166897_object59382:head# 2024-05-15T23:39:58.423 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e21ecfe:::benchmark_data_smithi090_166897_object59728:head# 2024-05-15T23:39:58.424 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e22bdfa:::benchmark_data_smithi090_166897_object60125:head# 2024-05-15T23:39:58.424 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e22db38:::benchmark_data_smithi090_166897_object24999:head# 2024-05-15T23:39:58.425 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e232c9a:::benchmark_data_smithi090_166897_object56967:head# 2024-05-15T23:39:58.425 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e243977:::benchmark_data_smithi090_166897_object37208:head# 2024-05-15T23:39:58.427 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e254d10:::benchmark_data_smithi090_166897_object35554:head# 2024-05-15T23:39:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e25aded:::benchmark_data_smithi090_166897_object36441:head# 2024-05-15T23:39:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e267f9d:::benchmark_data_smithi090_166897_object32552:head# 2024-05-15T23:39:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e273062:::benchmark_data_smithi090_166897_object25383:head# 2024-05-15T23:39:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e27930d:::benchmark_data_smithi090_166897_object59950:head# 2024-05-15T23:39:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e28ec29:::benchmark_data_smithi090_166897_object32823:head# 2024-05-15T23:39:58.430 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e2d0018:::benchmark_data_smithi090_166897_object60876:head# 2024-05-15T23:39:58.430 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e2e8405:::benchmark_data_smithi090_166897_object15631:head# 2024-05-15T23:39:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e2eec09:::benchmark_data_smithi090_166897_object41113:head# 2024-05-15T23:39:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e2fbf2e:::benchmark_data_smithi090_166897_object66642:head# 2024-05-15T23:39:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e3035ad:::benchmark_data_smithi090_166897_object55518:head# 2024-05-15T23:39:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e30c66e:::benchmark_data_smithi090_166897_object30669:head# 2024-05-15T23:39:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e30d22d:::benchmark_data_smithi090_166897_object38329:head# 2024-05-15T23:39:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e320e5c:::benchmark_data_smithi090_166897_object38005:head# 2024-05-15T23:39:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e328fe5:::benchmark_data_smithi090_166897_object23252:head# 2024-05-15T23:39:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e332f60:::benchmark_data_smithi090_166897_object46818:head# 2024-05-15T23:39:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e339681:::benchmark_data_smithi090_166897_object52669:head# 2024-05-15T23:39:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e33abc6:::benchmark_data_smithi090_166897_object68118:head# 2024-05-15T23:39:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e35203b:::benchmark_data_smithi090_166897_object16864:head# 2024-05-15T23:39:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e35a6d4:::benchmark_data_smithi090_166897_object8097:head# 2024-05-15T23:39:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e35f49b:::benchmark_data_smithi090_166897_object24734:head# 2024-05-15T23:39:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e37124c:::benchmark_data_smithi090_166897_object50439:head# 2024-05-15T23:39:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e37e764:::benchmark_data_smithi090_166897_object50830:head# 2024-05-15T23:39:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e383678:::benchmark_data_smithi090_166897_object15366:head# 2024-05-15T23:39:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e384d07:::benchmark_data_smithi090_166897_object8460:head# 2024-05-15T23:39:58.443 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e3a5b74:::benchmark_data_smithi090_166897_object52159:head# 2024-05-15T23:39:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e3c5b1b:::benchmark_data_smithi090_166897_object7532:head# 2024-05-15T23:39:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e3e986d:::benchmark_data_smithi090_166897_object6000:head# 2024-05-15T23:39:58.445 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e3eb625:::benchmark_data_smithi090_166897_object56296:head# 2024-05-15T23:39:58.445 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e3f7340:::benchmark_data_smithi090_166897_object64766:head# 2024-05-15T23:39:58.445 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e3f8ab6:::benchmark_data_smithi090_166897_object67452:head# 2024-05-15T23:39:58.446 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e419fd3:::benchmark_data_smithi090_166897_object16104:head# 2024-05-15T23:39:58.446 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e41b206:::benchmark_data_smithi090_166897_object24325:head# 2024-05-15T23:39:58.447 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e42fb97:::benchmark_data_smithi090_166897_object44244:head# 2024-05-15T23:39:58.447 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e432a92:::benchmark_data_smithi090_166897_object43308:head# 2024-05-15T23:39:58.448 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e43aa6c:::benchmark_data_smithi090_166897_object41649:head# 2024-05-15T23:39:58.448 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e444802:::benchmark_data_smithi090_166897_object67054:head# 2024-05-15T23:39:58.448 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e45eb78:::benchmark_data_smithi090_166897_object737:head# 2024-05-15T23:39:58.450 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e4640fa:::benchmark_data_smithi090_166897_object24457:head# 2024-05-15T23:39:58.450 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e466ed3:::benchmark_data_smithi090_166897_object56139:head# 2024-05-15T23:39:58.450 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e46744e:::benchmark_data_smithi090_166897_object15354:head# 2024-05-15T23:39:58.451 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e4c4764:::benchmark_data_smithi090_166897_object41667:head# 2024-05-15T23:39:58.451 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e4dd709:::benchmark_data_smithi090_166897_object49240:head# 2024-05-15T23:39:58.452 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e4f4c48:::benchmark_data_smithi090_166897_object17121:head# 2024-05-15T23:39:58.452 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e4fd00a:::benchmark_data_smithi090_166897_object14525:head# 2024-05-15T23:39:58.452 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e5298a3:::benchmark_data_smithi090_166897_object9027:head# 2024-05-15T23:39:58.453 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e54c851:::benchmark_data_smithi090_166897_object3556:head# 2024-05-15T23:39:58.454 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e54e068:::benchmark_data_smithi090_166897_object28431:head# 2024-05-15T23:39:58.454 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e555c89:::benchmark_data_smithi090_166897_object15114:head# 2024-05-15T23:39:58.456 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e55f55e:::benchmark_data_smithi090_166897_object54839:head# 2024-05-15T23:39:58.456 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e5607c6:::benchmark_data_smithi090_166897_object12938:head# 2024-05-15T23:39:58.456 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e5618d9:::benchmark_data_smithi090_166897_object34086:head# 2024-05-15T23:39:58.457 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e5805fa:::benchmark_data_smithi090_166897_object1837:head# 2024-05-15T23:39:58.457 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e58b76a:::benchmark_data_smithi090_166897_object64762:head# 2024-05-15T23:39:58.458 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e59e89e:::benchmark_data_smithi090_166897_object10612:head# 2024-05-15T23:39:58.459 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e5a248a:::benchmark_data_smithi090_166897_object54177:head# 2024-05-15T23:39:58.459 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e5a69cb:::benchmark_data_smithi090_166897_object61298:head# 2024-05-15T23:39:58.460 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e5b8c59:::benchmark_data_smithi090_166897_object46161:head# 2024-05-15T23:39:58.460 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e5dfd50:::benchmark_data_smithi090_166897_object13099:head# 2024-05-15T23:39:58.461 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e5e81ed:::benchmark_data_smithi090_166897_object67073:head# 2024-05-15T23:39:58.461 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e5f54bd:::benchmark_data_smithi090_166897_object35436:head# 2024-05-15T23:39:58.461 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e5fdfa2:::benchmark_data_smithi090_166897_object42392:head# 2024-05-15T23:39:58.462 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e600361:::benchmark_data_smithi090_166897_object14388:head# 2024-05-15T23:39:58.463 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e61fe16:::benchmark_data_smithi090_166897_object58206:head# 2024-05-15T23:39:58.464 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e62723f:::benchmark_data_smithi090_166897_object60308:head# 2024-05-15T23:39:58.464 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e63a3e5:::benchmark_data_smithi090_166897_object60640:head# 2024-05-15T23:39:58.464 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e63bee9:::benchmark_data_smithi090_166897_object19427:head# 2024-05-15T23:39:58.465 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e63d8f0:::benchmark_data_smithi090_166897_object19004:head# 2024-05-15T23:39:58.465 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e64b377:::benchmark_data_smithi090_166897_object29020:head# 2024-05-15T23:39:58.466 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e65e39e:::benchmark_data_smithi090_166897_object16144:head# 2024-05-15T23:39:58.466 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e6731b5:::benchmark_data_smithi090_166897_object22765:head# 2024-05-15T23:39:58.467 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e6a01d7:::benchmark_data_smithi090_166897_object34260:head# 2024-05-15T23:39:58.467 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e6ac399:::benchmark_data_smithi090_166897_object37914:head# 2024-05-15T23:39:58.468 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e6c8fd1:::benchmark_data_smithi090_166897_object18362:head# 2024-05-15T23:39:58.468 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e6e1e0b:::benchmark_data_smithi090_166897_object16627:head# 2024-05-15T23:39:58.469 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e6ed39f:::benchmark_data_smithi090_166897_object5311:head# 2024-05-15T23:39:58.469 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e70434f:::benchmark_data_smithi090_166897_object1418:head# 2024-05-15T23:39:58.471 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e709f4b:::benchmark_data_smithi090_166897_object36011:head# 2024-05-15T23:39:58.471 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e70ffd6:::benchmark_data_smithi090_166897_object17625:head# 2024-05-15T23:39:58.472 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e7184b8:::benchmark_data_smithi090_166897_object17578:head# 2024-05-15T23:39:58.472 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e7277f7:::benchmark_data_smithi090_166897_object60284:head# 2024-05-15T23:39:58.472 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e727c8d:::benchmark_data_smithi090_166897_object23478:head# 2024-05-15T23:39:58.473 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e72f016:::benchmark_data_smithi090_166897_object10231:head# 2024-05-15T23:39:58.474 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e73dd49:::benchmark_data_smithi090_166897_object51083:head# 2024-05-15T23:39:58.474 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e73df2c:::benchmark_data_smithi090_166897_object37528:head# 2024-05-15T23:39:58.475 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e74bd7b:::benchmark_data_smithi090_166897_object59304:head# 2024-05-15T23:39:58.475 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e75b745:::benchmark_data_smithi090_166897_object18151:head# 2024-05-15T23:39:58.476 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e75bed5:::benchmark_data_smithi090_166897_object11153:head# 2024-05-15T23:39:58.476 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e75bf43:::benchmark_data_smithi090_166897_object29856:head# 2024-05-15T23:39:58.476 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e774750:::benchmark_data_smithi090_166897_object60846:head# 2024-05-15T23:39:58.478 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e7760ac:::benchmark_data_smithi090_166897_object40030:head# 2024-05-15T23:39:58.479 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e788b51:::benchmark_data_smithi090_166897_object69620:head# 2024-05-15T23:39:58.479 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e7899db:::benchmark_data_smithi090_166897_object12140:head# 2024-05-15T23:39:58.480 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e797e14:::benchmark_data_smithi090_166897_object3754:head# 2024-05-15T23:39:58.480 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e79ebfc:::benchmark_data_smithi090_166897_object45656:head# 2024-05-15T23:39:58.481 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e7a9062:::benchmark_data_smithi090_166897_object16665:head# 2024-05-15T23:39:58.481 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e7ad3f7:::benchmark_data_smithi090_166897_object29671:head# 2024-05-15T23:39:58.482 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e7c28a6:::benchmark_data_smithi090_166897_object54315:head# 2024-05-15T23:39:58.483 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e7d4595:::benchmark_data_smithi090_166897_object202:head# 2024-05-15T23:39:58.483 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e7df748:::benchmark_data_smithi090_166897_object25226:head# 2024-05-15T23:39:58.483 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e7eb984:::benchmark_data_smithi090_166897_object9637:head# 2024-05-15T23:39:58.484 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e7f3be2:::benchmark_data_smithi090_166897_object1121:head# 2024-05-15T23:39:58.484 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e806816:::benchmark_data_smithi090_166897_object61651:head# 2024-05-15T23:39:58.485 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e82c515:::benchmark_data_smithi090_166897_object9246:head# 2024-05-15T23:39:58.485 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e83205a:::benchmark_data_smithi090_166897_object56808:head# 2024-05-15T23:39:58.485 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e83a0e8:::benchmark_data_smithi090_166897_object53076:head# 2024-05-15T23:39:58.486 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e85feda:::benchmark_data_smithi090_166897_object2698:head# 2024-05-15T23:39:58.486 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e86f458:::benchmark_data_smithi090_166897_object57593:head# 2024-05-15T23:39:58.487 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e87a1bd:::benchmark_data_smithi090_166897_object45577:head# 2024-05-15T23:39:58.487 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e87c46d:::benchmark_data_smithi090_166897_object62036:head# 2024-05-15T23:39:58.487 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e88d2b6:::benchmark_data_smithi090_166897_object68138:head# 2024-05-15T23:39:58.488 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e892803:::benchmark_data_smithi090_166897_object66155:head# 2024-05-15T23:39:58.488 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e8a989d:::benchmark_data_smithi090_166897_object14683:head# 2024-05-15T23:39:58.489 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e8b552e:::benchmark_data_smithi090_166897_object56209:head# 2024-05-15T23:39:58.491 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e8e7f23:::benchmark_data_smithi090_166897_object48479:head# 2024-05-15T23:39:58.491 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e8ef157:::benchmark_data_smithi090_166897_object64764:head# 2024-05-15T23:39:58.491 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e90eadf:::benchmark_data_smithi090_166897_object34092:head# 2024-05-15T23:39:58.491 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e916b0c:::benchmark_data_smithi090_166897_object46689:head# 2024-05-15T23:39:58.491 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e9260ed:::benchmark_data_smithi090_166897_object38703:head# 2024-05-15T23:39:58.491 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e947e91:::benchmark_data_smithi090_166897_object46356:head# 2024-05-15T23:39:58.492 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e95336f:::benchmark_data_smithi090_166897_object67198:head# 2024-05-15T23:39:58.493 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e965edc:::benchmark_data_smithi090_166897_object1763:head# 2024-05-15T23:39:58.493 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e96acf0:::benchmark_data_smithi090_166897_object14741:head# 2024-05-15T23:39:58.493 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e972498:::benchmark_data_smithi090_166897_object51849:head# 2024-05-15T23:39:58.493 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e9757a7:::benchmark_data_smithi090_166897_object18924:head# 2024-05-15T23:39:58.494 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e998181:::benchmark_data_smithi090_166897_object45591:head# 2024-05-15T23:39:58.494 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e9a8a46:::benchmark_data_smithi090_166897_object54846:head# 2024-05-15T23:39:58.495 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e9ea2a4:::benchmark_data_smithi090_166897_object31194:head# 2024-05-15T23:39:58.495 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6e9f3611:::benchmark_data_smithi090_166897_object5820:head# 2024-05-15T23:39:58.496 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ea0364b:::benchmark_data_smithi090_166897_object1045:head# 2024-05-15T23:39:58.496 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ea41d50:::benchmark_data_smithi090_166897_object17818:head# 2024-05-15T23:39:58.497 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ea4df27:::benchmark_data_smithi090_166897_object68328:head# 2024-05-15T23:39:58.497 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ea5df4a:::benchmark_data_smithi090_166897_object23511:head# 2024-05-15T23:39:58.497 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ea67f92:::benchmark_data_smithi090_166897_object14805:head# 2024-05-15T23:39:58.497 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ea71437:::benchmark_data_smithi090_166897_object6589:head# 2024-05-15T23:39:58.498 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ea74f66:::benchmark_data_smithi090_166897_object31736:head# 2024-05-15T23:39:58.499 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ea77687:::benchmark_data_smithi090_166897_object68571:head# 2024-05-15T23:39:58.499 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ea8538f:::benchmark_data_smithi090_166897_object17117:head# 2024-05-15T23:39:58.500 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eaacb99:::benchmark_data_smithi090_166897_object40759:head# 2024-05-15T23:39:58.500 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eabcd7f:::benchmark_data_smithi090_166897_object15927:head# 2024-05-15T23:39:58.500 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eabd862:::benchmark_data_smithi090_166897_object10513:head# 2024-05-15T23:39:58.501 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eaca7ba:::benchmark_data_smithi090_166897_object18628:head# 2024-05-15T23:39:58.501 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ead7e6f:::benchmark_data_smithi090_166897_object50169:head# 2024-05-15T23:39:58.501 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eaf2f69:::benchmark_data_smithi090_166897_object11491:head# 2024-05-15T23:39:58.502 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eaf500b:::benchmark_data_smithi090_166897_object8238:head# 2024-05-15T23:39:58.502 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eb04cf2:::benchmark_data_smithi090_166897_object44718:head# 2024-05-15T23:39:58.503 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eb131f4:::benchmark_data_smithi090_166897_object14008:head# 2024-05-15T23:39:58.503 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eb40d8a:::benchmark_data_smithi090_166897_object60545:head# 2024-05-15T23:39:58.506 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eb4f4ad:::benchmark_data_smithi090_166897_object101:head# 2024-05-15T23:39:58.506 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eb618f3:::benchmark_data_smithi090_166897_object13513:head# 2024-05-15T23:39:58.507 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eb6e531:::benchmark_data_smithi090_166897_object20768:head# 2024-05-15T23:39:58.507 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eb89ea5:::benchmark_data_smithi090_166897_object34832:head# 2024-05-15T23:39:58.508 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eba9c25:::benchmark_data_smithi090_166897_object27702:head# 2024-05-15T23:39:58.508 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ebbf290:::benchmark_data_smithi090_166897_object2336:head# 2024-05-15T23:39:58.509 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ebea73b:::benchmark_data_smithi090_166897_object62761:head# 2024-05-15T23:39:58.509 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ebeda89:::benchmark_data_smithi090_166897_object15958:head# 2024-05-15T23:39:58.510 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ebffb9b:::benchmark_data_smithi090_166897_object61799:head# 2024-05-15T23:39:58.510 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec0095f:::benchmark_data_smithi090_166897_object10003:head# 2024-05-15T23:39:58.511 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec04b52:::benchmark_data_smithi090_166897_object54080:head# 2024-05-15T23:39:58.511 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec07b07:::benchmark_data_smithi090_166897_object67542:head# 2024-05-15T23:39:58.511 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec1e1a1:::benchmark_data_smithi090_166897_object58829:head# 2024-05-15T23:39:58.511 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec28d16:::benchmark_data_smithi090_166897_object54406:head# 2024-05-15T23:39:58.512 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec441ae:::benchmark_data_smithi090_166897_object51468:head# 2024-05-15T23:39:58.512 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec5e9d5:::benchmark_data_smithi090_166897_object9554:head# 2024-05-15T23:39:58.513 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec65734:::benchmark_data_smithi090_166897_object15875:head# 2024-05-15T23:39:58.513 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec65998:::benchmark_data_smithi090_166897_object61782:head# 2024-05-15T23:39:58.514 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec7002e:::benchmark_data_smithi090_166897_object45508:head# 2024-05-15T23:39:58.514 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec719bd:::benchmark_data_smithi090_166897_object925:head# 2024-05-15T23:39:58.514 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec90cf2:::benchmark_data_smithi090_166897_object57102:head# 2024-05-15T23:39:58.515 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec96617:::benchmark_data_smithi090_166897_object66096:head# 2024-05-15T23:39:58.515 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec9bef4:::benchmark_data_smithi090_166897_object1801:head# 2024-05-15T23:39:58.516 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec9dd23:::benchmark_data_smithi090_166897_object4414:head# 2024-05-15T23:39:58.516 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ec9e212:::benchmark_data_smithi090_166897_object64949:head# 2024-05-15T23:39:58.516 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ecac741:::benchmark_data_smithi090_166897_object42951:head# 2024-05-15T23:39:58.518 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ecb86b5:::benchmark_data_smithi090_166897_object27005:head# 2024-05-15T23:39:58.519 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eccc008:::benchmark_data_smithi090_166897_object58659:head# 2024-05-15T23:39:58.519 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eccf057:::benchmark_data_smithi090_166897_object43672:head# 2024-05-15T23:39:58.519 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ecd389a:::benchmark_data_smithi090_166897_object34298:head# 2024-05-15T23:39:58.520 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ece7308:::benchmark_data_smithi090_166897_object36886:head# 2024-05-15T23:39:58.520 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ed1230a:::benchmark_data_smithi090_166897_object40000:head# 2024-05-15T23:39:58.521 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ed158eb:::benchmark_data_smithi090_166897_object41397:head# 2024-05-15T23:39:58.521 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ed198f5:::benchmark_data_smithi090_166897_object38975:head# 2024-05-15T23:39:58.522 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ed1b766:::benchmark_data_smithi090_166897_object18822:head# 2024-05-15T23:39:58.522 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ed1e1ac:::benchmark_data_smithi090_166897_object61038:head# 2024-05-15T23:39:58.522 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ed30383:::benchmark_data_smithi090_166897_object64323:head# 2024-05-15T23:39:58.523 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ed3f192:::benchmark_data_smithi090_166897_object50468:head# 2024-05-15T23:39:58.523 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ed652f9:::benchmark_data_smithi090_166897_object22562:head# 2024-05-15T23:39:58.524 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ed9b4ca:::benchmark_data_smithi090_166897_object10532:head# 2024-05-15T23:39:58.525 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eda48ad:::benchmark_data_smithi090_166897_object60224:head# 2024-05-15T23:39:58.526 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6edb55ea:::benchmark_data_smithi090_166897_object1430:head# 2024-05-15T23:39:58.526 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6edb7717:::benchmark_data_smithi090_166897_object57604:head# 2024-05-15T23:39:58.526 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6edd245c:::benchmark_data_smithi090_166897_object42021:head# 2024-05-15T23:39:58.527 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eddb6d7:::benchmark_data_smithi090_166897_object31071:head# 2024-05-15T23:39:58.528 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6edead0b:::benchmark_data_smithi090_166897_object34318:head# 2024-05-15T23:39:58.528 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ee0c196:::benchmark_data_smithi090_166897_object11439:head# 2024-05-15T23:39:58.531 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ee23ebe:::benchmark_data_smithi090_166897_object9206:head# 2024-05-15T23:39:58.532 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ee2efdd:::benchmark_data_smithi090_166897_object49369:head# 2024-05-15T23:39:58.532 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ee33fbc:::benchmark_data_smithi090_166897_object27933:head# 2024-05-15T23:39:58.532 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ee48f84:::benchmark_data_smithi090_166897_object16465:head# 2024-05-15T23:39:58.533 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ee4c071:::benchmark_data_smithi090_166897_object61077:head# 2024-05-15T23:39:58.534 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ee654a0:::benchmark_data_smithi090_166897_object11837:head# 2024-05-15T23:39:58.535 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ee658d5:::benchmark_data_smithi090_166897_object53660:head# 2024-05-15T23:39:58.535 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ee76cfd:::benchmark_data_smithi090_166897_object42963:head# 2024-05-15T23:39:58.535 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ee7d07e:::benchmark_data_smithi090_166897_object63933:head# 2024-05-15T23:39:58.536 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ee7d29d:::benchmark_data_smithi090_166897_object2310:head# 2024-05-15T23:39:58.536 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eea65db:::benchmark_data_smithi090_166897_object69298:head# 2024-05-15T23:39:58.537 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eeb90e5:::benchmark_data_smithi090_166897_object17911:head# 2024-05-15T23:39:58.537 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eec1d4c:::benchmark_data_smithi090_166897_object5849:head# 2024-05-15T23:39:58.538 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eece698:::benchmark_data_smithi090_166897_object58364:head# 2024-05-15T23:39:58.538 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eef4ad5:::benchmark_data_smithi090_166897_object57651:head# 2024-05-15T23:39:58.539 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ef03240:::benchmark_data_smithi090_166897_object35312:head# 2024-05-15T23:39:58.539 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ef03bc1:::benchmark_data_smithi090_166897_object33957:head# 2024-05-15T23:39:58.539 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ef0a93e:::benchmark_data_smithi090_166897_object11471:head# 2024-05-15T23:39:58.540 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ef13d7b:::benchmark_data_smithi090_166897_object59563:head# 2024-05-15T23:39:58.540 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ef37f80:::benchmark_data_smithi090_166897_object41829:head# 2024-05-15T23:39:58.541 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ef3b22c:::benchmark_data_smithi090_166897_object30656:head# 2024-05-15T23:39:58.541 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ef3bf21:::benchmark_data_smithi090_166897_object60615:head# 2024-05-15T23:39:58.541 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ef54376:::benchmark_data_smithi090_166897_object473:head# 2024-05-15T23:39:58.541 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ef57ad2:::benchmark_data_smithi090_166897_object28527:head# 2024-05-15T23:39:58.542 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ef61920:::benchmark_data_smithi090_166897_object17879:head# 2024-05-15T23:39:58.542 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ef6e4ec:::benchmark_data_smithi090_166897_object31087:head# 2024-05-15T23:39:58.543 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ef8d6c3:::benchmark_data_smithi090_166897_object27584:head# 2024-05-15T23:39:58.543 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ef94693:::benchmark_data_smithi090_166897_object29502:head# 2024-05-15T23:39:58.543 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6efdc544:::benchmark_data_smithi090_166897_object53337:head# 2024-05-15T23:39:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eff220f:::benchmark_data_smithi090_166897_object65565:head# 2024-05-15T23:39:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6eff49d6:::benchmark_data_smithi090_166897_object41091:head# 2024-05-15T23:39:58.545 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f001027:::benchmark_data_smithi090_166897_object7634:head# 2024-05-15T23:39:58.545 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f01a764:::benchmark_data_smithi090_166897_object30195:head# 2024-05-15T23:39:58.546 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f0344d1:::benchmark_data_smithi090_166897_object57401:head# 2024-05-15T23:39:58.546 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f039d95:::benchmark_data_smithi090_166897_object18411:head# 2024-05-15T23:39:58.546 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f048071:::benchmark_data_smithi090_166897_object59664:head# 2024-05-15T23:39:58.547 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f0483e7:::benchmark_data_smithi090_166897_object25268:head# 2024-05-15T23:39:58.548 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f055ab7:::benchmark_data_smithi090_166897_object22118:head# 2024-05-15T23:39:58.548 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f056177:::benchmark_data_smithi090_166897_object22695:head# 2024-05-15T23:39:58.549 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f0587c7:::benchmark_data_smithi090_166897_object19290:head# 2024-05-15T23:39:58.549 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f061088:::benchmark_data_smithi090_166897_object70892:head# 2024-05-15T23:39:58.550 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f07b707:::benchmark_data_smithi090_166897_object28047:head# 2024-05-15T23:39:58.550 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f07e5f6:::benchmark_data_smithi090_166897_object13247:head# 2024-05-15T23:39:58.551 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f08182e:::benchmark_data_smithi090_166897_object10555:head# 2024-05-15T23:39:58.551 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f0a9d21:::benchmark_data_smithi090_166897_object19200:head# 2024-05-15T23:39:58.552 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f0b1e82:::benchmark_data_smithi090_166897_object68933:head# 2024-05-15T23:39:58.552 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f0b2245:::benchmark_data_smithi090_166897_object39149:head# 2024-05-15T23:39:58.552 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f0ba39f:::benchmark_data_smithi090_166897_object65184:head# 2024-05-15T23:39:58.553 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f0c421c:::benchmark_data_smithi090_166897_object31204:head# 2024-05-15T23:39:58.553 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f0c9c89:::benchmark_data_smithi090_166897_object22179:head# 2024-05-15T23:39:58.554 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f0d962e:::benchmark_data_smithi090_166897_object53147:head# 2024-05-15T23:39:58.555 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f0eb2ca:::benchmark_data_smithi090_166897_object58619:head# 2024-05-15T23:39:58.555 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f0ebc1e:::benchmark_data_smithi090_166897_object36914:head# 2024-05-15T23:39:58.555 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f0f3fc8:::benchmark_data_smithi090_166897_object26580:head# 2024-05-15T23:39:58.556 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f0f7af2:::benchmark_data_smithi090_166897_object58264:head# 2024-05-15T23:39:58.556 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f10edf9:::benchmark_data_smithi090_166897_object33878:head# 2024-05-15T23:39:58.557 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f111177:::benchmark_data_smithi090_166897_object41769:head# 2024-05-15T23:39:58.557 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f11370f:::benchmark_data_smithi090_166897_object51267:head# 2024-05-15T23:39:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f11e755:::benchmark_data_smithi090_166897_object57520:head# 2024-05-15T23:39:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f126f1b:::benchmark_data_smithi090_166897_object32584:head# 2024-05-15T23:39:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f128a67:::benchmark_data_smithi090_166897_object68577:head# 2024-05-15T23:39:58.559 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f1292c3:::benchmark_data_smithi090_166897_object29206:head# 2024-05-15T23:39:58.561 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f12b9f6:::benchmark_data_smithi090_166897_object15266:head# 2024-05-15T23:39:58.561 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f13679c:::benchmark_data_smithi090_166897_object29777:head# 2024-05-15T23:39:58.561 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f1493d8:::benchmark_data_smithi090_166897_object16292:head# 2024-05-15T23:39:58.562 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f15721d:::benchmark_data_smithi090_166897_object5469:head# 2024-05-15T23:39:58.562 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f16e374:::benchmark_data_smithi090_166897_object7983:head# 2024-05-15T23:39:58.562 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f175544:::benchmark_data_smithi090_166897_object15124:head# 2024-05-15T23:39:58.564 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f18d8f8:::benchmark_data_smithi090_166897_object65953:head# 2024-05-15T23:39:58.564 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f1bb615:::benchmark_data_smithi090_166897_object62858:head# 2024-05-15T23:39:58.564 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f1be369:::benchmark_data_smithi090_166897_object19920:head# 2024-05-15T23:39:58.564 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f1cec62:::benchmark_data_smithi090_166897_object12415:head# 2024-05-15T23:39:58.565 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f1d270f:::benchmark_data_smithi090_166897_object45238:head# 2024-05-15T23:39:58.565 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f1e7331:::benchmark_data_smithi090_166897_object71017:head# 2024-05-15T23:39:58.565 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f1f1f2c:::benchmark_data_smithi090_166897_object13347:head# 2024-05-15T23:39:58.566 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f209187:::benchmark_data_smithi090_166897_object63086:head# 2024-05-15T23:39:58.566 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f20fadc:::benchmark_data_smithi090_166897_object27652:head# 2024-05-15T23:39:58.567 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f213e76:::benchmark_data_smithi090_166897_object15388:head# 2024-05-15T23:39:58.567 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f223250:::benchmark_data_smithi090_166897_object29289:head# 2024-05-15T23:39:58.568 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f223d30:::benchmark_data_smithi090_166897_object15803:head# 2024-05-15T23:39:58.569 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f234f55:::benchmark_data_smithi090_166897_object52088:head# 2024-05-15T23:39:58.569 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f237f91:::benchmark_data_smithi090_166897_object66346:head# 2024-05-15T23:39:58.570 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 56 16 87922 87906 98.0799 77.4375 0.011476 0.0101937 2024-05-15T23:39:58.571 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f26c6d4:::benchmark_data_smithi090_166897_object11877:head# 2024-05-15T23:39:58.572 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f27b6fb:::benchmark_data_smithi090_166897_object34197:head# 2024-05-15T23:39:58.572 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f289772:::benchmark_data_smithi090_166897_object55324:head# 2024-05-15T23:39:58.573 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f28d7d5:::benchmark_data_smithi090_166897_object18657:head# 2024-05-15T23:39:58.573 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f29f9a8:::benchmark_data_smithi090_166897_object56668:head# 2024-05-15T23:39:58.574 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f2b271f:::benchmark_data_smithi090_166897_object40868:head# 2024-05-15T23:39:58.574 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f2b6cfb:::benchmark_data_smithi090_166897_object59668:head# 2024-05-15T23:39:58.575 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f2b9cd8:::benchmark_data_smithi090_166897_object61712:head# 2024-05-15T23:39:58.577 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f2bca82:::benchmark_data_smithi090_166897_object57055:head# 2024-05-15T23:39:58.578 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f2e31c0:::benchmark_data_smithi090_166897_object23108:head# 2024-05-15T23:39:58.578 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f2eee61:::benchmark_data_smithi090_166897_object25104:head# 2024-05-15T23:39:58.578 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f2f1e77:::benchmark_data_smithi090_166897_object25592:head# 2024-05-15T23:39:58.579 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f2f48e6:::benchmark_data_smithi090_166897_object8910:head# 2024-05-15T23:39:58.579 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f30e34f:::benchmark_data_smithi090_166897_object48092:head# 2024-05-15T23:39:58.579 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f3197f4:::benchmark_data_smithi090_166897_object55353:head# 2024-05-15T23:39:58.580 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f33a1e2:::benchmark_data_smithi090_166897_object54500:head# 2024-05-15T23:39:58.580 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f345741:::benchmark_data_smithi090_166897_object47386:head# 2024-05-15T23:39:58.581 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f348b89:::benchmark_data_smithi090_166897_object69084:head# 2024-05-15T23:39:58.581 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f36b664:::benchmark_data_smithi090_166897_object62816:head# 2024-05-15T23:39:58.582 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f36d68e:::benchmark_data_smithi090_166897_object45025:head# 2024-05-15T23:39:58.582 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f3d6418:::benchmark_data_smithi090_166897_object30935:head# 2024-05-15T23:39:58.582 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f3ea0c3:::benchmark_data_smithi090_166897_object65360:head# 2024-05-15T23:39:58.583 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f43e095:::benchmark_data_smithi090_166897_object39855:head# 2024-05-15T23:39:58.583 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f43fa18:::benchmark_data_smithi090_166897_object27124:head# 2024-05-15T23:39:58.584 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f4404f4:::benchmark_data_smithi090_166897_object61228:head# 2024-05-15T23:39:58.584 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f450f0f:::benchmark_data_smithi090_166897_object58123:head# 2024-05-15T23:39:58.584 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f458a66:::benchmark_data_smithi090_166897_object805:head# 2024-05-15T23:39:58.585 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f4593e8:::benchmark_data_smithi090_166897_object66970:head# 2024-05-15T23:39:58.585 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f478924:::benchmark_data_smithi090_166897_object10441:head# 2024-05-15T23:39:58.586 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f47b74c:::benchmark_data_smithi090_166897_object15819:head# 2024-05-15T23:39:58.586 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f482c99:::benchmark_data_smithi090_166897_object15947:head# 2024-05-15T23:39:58.586 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f489482:::benchmark_data_smithi090_166897_object53317:head# 2024-05-15T23:39:58.587 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f4a09c7:::benchmark_data_smithi090_166897_object58108:head# 2024-05-15T23:39:58.588 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f4addc2:::benchmark_data_smithi090_166897_object29702:head# 2024-05-15T23:39:58.588 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f4ade12:::benchmark_data_smithi090_166897_object7100:head# 2024-05-15T23:39:58.588 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f4b8341:::benchmark_data_smithi090_166897_object38115:head# 2024-05-15T23:39:58.589 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f4b8632:::benchmark_data_smithi090_166897_object57072:head# 2024-05-15T23:39:58.589 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f4eec9a:::benchmark_data_smithi090_166897_object60017:head# 2024-05-15T23:39:58.590 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f4f8a76:::benchmark_data_smithi090_166897_object32477:head# 2024-05-15T23:39:58.590 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f4fa976:::benchmark_data_smithi090_166897_object55136:head# 2024-05-15T23:39:58.591 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f50888c:::benchmark_data_smithi090_166897_object24031:head# 2024-05-15T23:39:58.591 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f51cd99:::benchmark_data_smithi090_166897_object36031:head# 2024-05-15T23:39:58.591 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f51ee1e:::benchmark_data_smithi090_166897_object31358:head# 2024-05-15T23:39:58.592 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f521299:::benchmark_data_smithi090_166897_object13895:head# 2024-05-15T23:39:58.592 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f52985a:::benchmark_data_smithi090_166897_object26260:head# 2024-05-15T23:39:58.592 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f53da3e:::benchmark_data_smithi090_166897_object43628:head# 2024-05-15T23:39:58.593 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f540230:::benchmark_data_smithi090_166897_object59164:head# 2024-05-15T23:39:58.593 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f54f412:::benchmark_data_smithi090_166897_object25903:head# 2024-05-15T23:39:58.594 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f550cf7:::benchmark_data_smithi090_166897_object47337:head# 2024-05-15T23:39:58.594 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f56a44d:::benchmark_data_smithi090_166897_object39709:head# 2024-05-15T23:39:58.595 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f5923f9:::benchmark_data_smithi090_166897_object30770:head# 2024-05-15T23:39:58.595 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f598eef:::benchmark_data_smithi090_166897_object2120:head# 2024-05-15T23:39:58.596 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f599be6:::benchmark_data_smithi090_166897_object56918:head# 2024-05-15T23:39:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f59bb5e:::benchmark_data_smithi090_166897_object24201:head# 2024-05-15T23:39:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f59e9ae:::benchmark_data_smithi090_166897_object22575:head# 2024-05-15T23:39:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f5a2b27:::benchmark_data_smithi090_166897_object5282:head# 2024-05-15T23:39:58.598 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f5b5ed9:::benchmark_data_smithi090_166897_object35829:head# 2024-05-15T23:39:58.598 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f5b6aaa:::benchmark_data_smithi090_166897_object32489:head# 2024-05-15T23:39:58.598 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f5d42a0:::benchmark_data_smithi090_166897_object5780:head# 2024-05-15T23:39:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f5fd926:::benchmark_data_smithi090_166897_object12813:head# 2024-05-15T23:39:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f5fed82:::benchmark_data_smithi090_166897_object66850:head# 2024-05-15T23:39:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f6060de:::benchmark_data_smithi090_166897_object35561:head# 2024-05-15T23:39:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f668d03:::benchmark_data_smithi090_166897_object44324:head# 2024-05-15T23:39:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f669bac:::benchmark_data_smithi090_166897_object63999:head# 2024-05-15T23:39:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f66a8d3:::benchmark_data_smithi090_166897_object57385:head# 2024-05-15T23:39:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f68e114:::benchmark_data_smithi090_166897_object9845:head# 2024-05-15T23:39:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f6a5a33:::benchmark_data_smithi090_166897_object25590:head# 2024-05-15T23:39:58.602 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f6b3e00:::benchmark_data_smithi090_166897_object35301:head# 2024-05-15T23:39:58.602 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f6c7ed0:::benchmark_data_smithi090_166897_object61994:head# 2024-05-15T23:39:58.603 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f6ce133:::benchmark_data_smithi090_166897_object32951:head# 2024-05-15T23:39:58.603 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f6cfc95:::benchmark_data_smithi090_166897_object19031:head# 2024-05-15T23:39:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f6edb02:::benchmark_data_smithi090_166897_object28346:head# 2024-05-15T23:39:58.605 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f6f75fb:::benchmark_data_smithi090_166897_object49274:head# 2024-05-15T23:39:58.605 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f6fe6e2:::benchmark_data_smithi090_166897_object63466:head# 2024-05-15T23:39:58.605 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f702033:::benchmark_data_smithi090_166897_object39371:head# 2024-05-15T23:39:58.605 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f71b8a4:::benchmark_data_smithi090_166897_object22250:head# 2024-05-15T23:39:58.606 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f71f297:::benchmark_data_smithi090_166897_object64453:head# 2024-05-15T23:39:58.607 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f71fe85:::benchmark_data_smithi090_166897_object30784:head# 2024-05-15T23:39:58.607 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f726b6d:::benchmark_data_smithi090_166897_object21487:head# 2024-05-15T23:39:58.607 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f76279a:::benchmark_data_smithi090_166897_object56836:head# 2024-05-15T23:39:58.607 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f771546:::benchmark_data_smithi090_166897_object42840:head# 2024-05-15T23:39:58.608 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f77caf8:::benchmark_data_smithi090_166897_object28235:head# 2024-05-15T23:39:58.608 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f7886bc:::benchmark_data_smithi090_166897_object52927:head# 2024-05-15T23:39:58.609 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f78dafb:::benchmark_data_smithi090_166897_object19460:head# 2024-05-15T23:39:58.609 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f7b8a65:::benchmark_data_smithi090_166897_object62549:head# 2024-05-15T23:39:58.610 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f7bfc4f:::benchmark_data_smithi090_166897_object29592:head# 2024-05-15T23:39:58.610 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f7c048c:::benchmark_data_smithi090_166897_object4950:head# 2024-05-15T23:39:58.610 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f7d0502:::benchmark_data_smithi090_166897_object28724:head# 2024-05-15T23:39:58.611 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f7d4815:::benchmark_data_smithi090_166897_object53049:head# 2024-05-15T23:39:58.611 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f7e3af2:::benchmark_data_smithi090_166897_object1434:head# 2024-05-15T23:39:58.611 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f7eee46:::benchmark_data_smithi090_166897_object9251:head# 2024-05-15T23:39:58.612 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f7f397d:::benchmark_data_smithi090_166897_object44018:head# 2024-05-15T23:39:58.612 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f7f519b:::benchmark_data_smithi090_166897_object12926:head# 2024-05-15T23:39:58.613 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f7ff5e7:::benchmark_data_smithi090_166897_object26922:head# 2024-05-15T23:39:58.613 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f815e23:::benchmark_data_smithi090_166897_object50701:head# 2024-05-15T23:39:58.614 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f817dfb:::benchmark_data_smithi090_166897_object64705:head# 2024-05-15T23:39:58.614 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f81a8f2:::benchmark_data_smithi090_166897_object48618:head# 2024-05-15T23:39:58.614 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f82c2fc:::benchmark_data_smithi090_166897_object28509:head# 2024-05-15T23:39:58.615 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f834b6a:::benchmark_data_smithi090_166897_object7122:head# 2024-05-15T23:39:58.615 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f878734:::benchmark_data_smithi090_166897_object19263:head# 2024-05-15T23:39:58.616 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f895b24:::benchmark_data_smithi090_166897_object69212:head# 2024-05-15T23:39:58.616 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f8a4482:::benchmark_data_smithi090_166897_object67454:head# 2024-05-15T23:39:58.617 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f8d4432:::benchmark_data_smithi090_166897_object41153:head# 2024-05-15T23:39:58.617 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f91d454:::benchmark_data_smithi090_166897_object27300:head# 2024-05-15T23:39:58.617 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f92ee47:::benchmark_data_smithi090_166897_object70292:head# 2024-05-15T23:39:58.618 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f935ca0:::benchmark_data_smithi090_166897_object58302:head# 2024-05-15T23:39:58.618 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f972d6f:::benchmark_data_smithi090_166897_object61902:head# 2024-05-15T23:39:58.619 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f986910:::benchmark_data_smithi090_166897_object10688:head# 2024-05-15T23:39:58.619 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f99924d:::benchmark_data_smithi090_166897_object16075:head# 2024-05-15T23:39:58.620 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f99ce09:::benchmark_data_smithi090_166897_object51894:head# 2024-05-15T23:39:58.620 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f9c0922:::benchmark_data_smithi090_166897_object6536:head# 2024-05-15T23:39:58.621 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f9c4049:::benchmark_data_smithi090_166897_object15753:head# 2024-05-15T23:39:58.621 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f9c4a08:::benchmark_data_smithi090_166897_object49539:head# 2024-05-15T23:39:58.621 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f9ce2bf:::benchmark_data_smithi090_166897_object1190:head# 2024-05-15T23:39:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f9ce43f:::benchmark_data_smithi090_166897_object23009:head# 2024-05-15T23:39:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f9cee34:::benchmark_data_smithi090_166897_object69274:head# 2024-05-15T23:39:58.623 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6f9d0832:::benchmark_data_smithi090_166897_object4704:head# 2024-05-15T23:39:58.623 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fa03ca0:::benchmark_data_smithi090_166897_object60259:head# 2024-05-15T23:39:58.623 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fa17408:::benchmark_data_smithi090_166897_object13168:head# 2024-05-15T23:39:58.624 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fa3ae2e:::benchmark_data_smithi090_166897_object42061:head# 2024-05-15T23:39:58.624 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fa5e9ce:::benchmark_data_smithi090_166897_object58648:head# 2024-05-15T23:39:58.625 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fa7248f:::benchmark_data_smithi090_166897_object51157:head# 2024-05-15T23:39:58.625 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fa86be8:::benchmark_data_smithi090_166897_object68220:head# 2024-05-15T23:39:58.625 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fabfcff:::benchmark_data_smithi090_166897_object21791:head# 2024-05-15T23:39:58.626 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6faf2c73:::benchmark_data_smithi090_166897_object55924:head# 2024-05-15T23:39:58.626 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6faf819c:::benchmark_data_smithi090_166897_object8059:head# 2024-05-15T23:39:58.627 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fb0cec4:::benchmark_data_smithi090_166897_object49171:head# 2024-05-15T23:39:58.627 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fb11174:::benchmark_data_smithi090_166897_object70685:head# 2024-05-15T23:39:58.627 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fb5586c:::benchmark_data_smithi090_166897_object69744:head# 2024-05-15T23:39:58.628 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fb68d31:::benchmark_data_smithi090_166897_object10327:head# 2024-05-15T23:39:58.628 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fb72762:::benchmark_data_smithi090_166897_object43835:head# 2024-05-15T23:39:58.629 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fb8c1ba:::benchmark_data_smithi090_166897_object52640:head# 2024-05-15T23:39:58.629 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fb9beed:::benchmark_data_smithi090_166897_object57898:head# 2024-05-15T23:39:58.630 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fbab61a:::benchmark_data_smithi090_166897_object42667:head# 2024-05-15T23:39:58.630 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fbacca1:::benchmark_data_smithi090_166897_object49894:head# 2024-05-15T23:39:58.630 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fbb0763:::benchmark_data_smithi090_166897_object6628:head# 2024-05-15T23:39:58.631 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fbb1fc2:::benchmark_data_smithi090_166897_object30139:head# 2024-05-15T23:39:58.631 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fbb373e:::benchmark_data_smithi090_166897_object33674:head# 2024-05-15T23:39:58.632 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fbb4e6d:::benchmark_data_smithi090_166897_object18284:head# 2024-05-15T23:39:58.632 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fbc8e24:::benchmark_data_smithi090_166897_object36899:head# 2024-05-15T23:39:58.633 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fbcf2f5:::benchmark_data_smithi090_166897_object13327:head# 2024-05-15T23:39:58.633 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fbd1623:::benchmark_data_smithi090_166897_object19254:head# 2024-05-15T23:39:58.633 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fbf6b14:::benchmark_data_smithi090_166897_object36916:head# 2024-05-15T23:39:58.634 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fbf9902:::benchmark_data_smithi090_166897_object19864:head# 2024-05-15T23:39:58.634 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fc01ce8:::benchmark_data_smithi090_166897_object8:head# 2024-05-15T23:39:58.634 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fc0ca26:::benchmark_data_smithi090_166897_object41715:head# 2024-05-15T23:39:58.635 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fc16afe:::benchmark_data_smithi090_166897_object70344:head# 2024-05-15T23:39:58.635 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fc175b5:::benchmark_data_smithi090_166897_object32829:head# 2024-05-15T23:39:58.635 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fc1adf5:::benchmark_data_smithi090_166897_object66792:head# 2024-05-15T23:39:58.636 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fc68521:::benchmark_data_smithi090_166897_object12332:head# 2024-05-15T23:39:58.636 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fc7bc5a:::benchmark_data_smithi090_166897_object8266:head# 2024-05-15T23:39:58.637 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fc7bdb2:::benchmark_data_smithi090_166897_object9757:head# 2024-05-15T23:39:58.637 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fc81bd8:::benchmark_data_smithi090_166897_object20365:head# 2024-05-15T23:39:58.638 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fc878d2:::benchmark_data_smithi090_166897_object22875:head# 2024-05-15T23:39:58.638 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fced7b0:::benchmark_data_smithi090_166897_object18106:head# 2024-05-15T23:39:58.638 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fcf4c11:::benchmark_data_smithi090_166897_object4498:head# 2024-05-15T23:39:58.639 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fcfe22b:::benchmark_data_smithi090_166897_object2146:head# 2024-05-15T23:39:58.639 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd049e1:::benchmark_data_smithi090_166897_object48480:head# 2024-05-15T23:39:58.640 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd15648:::benchmark_data_smithi090_166897_object35626:head# 2024-05-15T23:39:58.640 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd1da4f:::benchmark_data_smithi090_166897_object37650:head# 2024-05-15T23:39:58.641 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd213d5:::benchmark_data_smithi090_166897_object32547:head# 2024-05-15T23:39:58.641 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd262fd:::benchmark_data_smithi090_166897_object33888:head# 2024-05-15T23:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd45123:::benchmark_data_smithi090_166897_object27714:head# 2024-05-15T23:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd45fb5:::benchmark_data_smithi090_166897_object59742:head# 2024-05-15T23:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd4a0c1:::benchmark_data_smithi090_166897_object65152:head# 2024-05-15T23:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd50885:::benchmark_data_smithi090_166897_object63455:head# 2024-05-15T23:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd5c6c2:::benchmark_data_smithi090_166897_object18087:head# 2024-05-15T23:39:58.644 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd654e5:::benchmark_data_smithi090_166897_object41786:head# 2024-05-15T23:39:58.644 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd670f3:::benchmark_data_smithi090_166897_object20864:head# 2024-05-15T23:39:58.645 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd6a11e:::benchmark_data_smithi090_166897_object992:head# 2024-05-15T23:39:58.645 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd6ac88:::benchmark_data_smithi090_166897_object7436:head# 2024-05-15T23:39:58.646 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd765f2:::benchmark_data_smithi090_166897_object656:head# 2024-05-15T23:39:58.646 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fd7ea26:::benchmark_data_smithi090_166897_object70370:head# 2024-05-15T23:39:58.647 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fdaf28e:::benchmark_data_smithi090_166897_object44822:head# 2024-05-15T23:39:58.647 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fdc119e:::benchmark_data_smithi090_166897_object68855:head# 2024-05-15T23:39:58.647 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fdc80f4:::benchmark_data_smithi090_166897_object31078:head# 2024-05-15T23:39:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fdcd4a7:::benchmark_data_smithi090_166897_object45783:head# 2024-05-15T23:39:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fde21eb:::benchmark_data_smithi090_166897_object40432:head# 2024-05-15T23:39:58.649 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fde3d17:::benchmark_data_smithi090_166897_object66161:head# 2024-05-15T23:39:58.649 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fde5eb5:::benchmark_data_smithi090_166897_object5785:head# 2024-05-15T23:39:58.650 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fde5fc4:::benchmark_data_smithi090_166897_object64664:head# 2024-05-15T23:39:58.650 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fde6078:::benchmark_data_smithi090_166897_object62796:head# 2024-05-15T23:39:58.651 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fdebdcd:::benchmark_data_smithi090_166897_object48312:head# 2024-05-15T23:39:58.651 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fe03e9f:::benchmark_data_smithi090_166897_object26879:head# 2024-05-15T23:39:58.652 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fe08450:::benchmark_data_smithi090_166897_object45279:head# 2024-05-15T23:39:58.652 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fe14c10:::benchmark_data_smithi090_166897_object56087:head# 2024-05-15T23:39:58.653 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fe2115a:::benchmark_data_smithi090_166897_object47189:head# 2024-05-15T23:39:58.653 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fe41468:::benchmark_data_smithi090_166897_object37882:head# 2024-05-15T23:39:58.653 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fe4ac5a:::benchmark_data_smithi090_166897_object26855:head# 2024-05-15T23:39:58.654 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fe5ca73:::benchmark_data_smithi090_166897_object64263:head# 2024-05-15T23:39:58.654 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fe5dc22:::benchmark_data_smithi090_166897_object42017:head# 2024-05-15T23:39:58.655 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fe63f3a:::benchmark_data_smithi090_166897_object30011:head# 2024-05-15T23:39:58.655 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fe653d5:::benchmark_data_smithi090_166897_object59624:head# 2024-05-15T23:39:58.656 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fe9e8f0:::benchmark_data_smithi090_166897_object25584:head# 2024-05-15T23:39:58.656 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fee8fcf:::benchmark_data_smithi090_166897_object33491:head# 2024-05-15T23:39:58.657 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fef636c:::benchmark_data_smithi090_166897_object11458:head# 2024-05-15T23:39:58.660 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6fefa9ad:::benchmark_data_smithi090_166897_object45440:head# 2024-05-15T23:39:58.660 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ff09ed3:::benchmark_data_smithi090_166897_object9479:head# 2024-05-15T23:39:58.660 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ff1dbf4:::benchmark_data_smithi090_166897_object26057:head# 2024-05-15T23:39:58.660 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ff31086:::benchmark_data_smithi090_166897_object56990:head# 2024-05-15T23:39:58.660 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ff3e88e:::benchmark_data_smithi090_166897_object40814:head# 2024-05-15T23:39:58.660 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ff4984f:::benchmark_data_smithi090_166897_object55490:head# 2024-05-15T23:39:58.661 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ff8611c:::benchmark_data_smithi090_166897_object25231:head# 2024-05-15T23:39:58.661 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ff8795b:::benchmark_data_smithi090_166897_object41643:head# 2024-05-15T23:39:58.661 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ff932fc:::benchmark_data_smithi090_166897_object52353:head# 2024-05-15T23:39:58.661 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ff9597e:::benchmark_data_smithi090_166897_object39025:head# 2024-05-15T23:39:58.662 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ff9e7ea:::benchmark_data_smithi090_166897_object37779:head# 2024-05-15T23:39:58.662 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ffb2576:::benchmark_data_smithi090_166897_object12791:head# 2024-05-15T23:39:58.663 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ffb4c5e:::benchmark_data_smithi090_166897_object32602:head# 2024-05-15T23:39:58.663 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ffe197b:::benchmark_data_smithi090_166897_object63487:head# 2024-05-15T23:39:58.664 INFO:teuthology.orchestra.run.smithi088.stderr:Read #9:6ffee9e1:::benchmark_data_smithi090_166897_object21722:head# 2024-05-15T23:39:58.666 INFO:teuthology.orchestra.run.smithi088.stderr:Export successful 2024-05-15T23:39:58.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:58 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:39:58.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:58 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:39:58.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:58 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:39:58.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:58 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:39:58.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:58 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:39:58.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:58 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:39:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:39:59.527 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:39:59.570 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 57 16 88967 88951 97.5051 65.3125 0.00743912 0.0102532 2024-05-15T23:39:59.771 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:59 smithi088 ceph-mon[44699]: osdmap e762: 12 total, 11 up, 9 in 2024-05-15T23:39:59.771 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:59 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:39:59.771 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:59 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:39:59.771 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:59 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:39:59.771 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:39:59 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:39:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:59 smithi090 ceph-mon[44434]: osdmap e762: 12 total, 11 up, 9 in 2024-05-15T23:39:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:59 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:39:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:59 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:39:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:59 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:39:59.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:39:59 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:39:59.804 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -n osd.5 --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 9.6 2024-05-15T23:39:59.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:59 smithi084 ceph-mon[39784]: osdmap e762: 12 total, 11 up, 9 in 2024-05-15T23:39:59.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:59 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:39:59.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:59 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:39:59.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:59 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:39:59.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:39:59 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:40:00.089 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:40:00.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:00.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:40:00.116 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/osd.5/config 2024-05-15T23:40:00.200 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:40:00.232 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:00.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T23:40:00.434 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:40:00.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:00.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:40:00.541 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:40:00.571 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 58 16 89980 89964 96.9156 63.3125 0.0188803 0.0103153 2024-05-15T23:40:00.586 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:00.685 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:40:00.798 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:40:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:00.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:40:00.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:40:00.948 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:00.979 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:40:00.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:00.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:40:01.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: pgmap v2022: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 73 MiB/s wr, 1.17k op/s; 15015/171474 objects degraded (8.756%); 11 MiB/s, 179 objects/s recovering 2024-05-15T23:40:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:40:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:40:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: Health detail: HEALTH_WARN Degraded data redundancy: 15015/171474 objects degraded (8.756%), 4 pgs degraded 2024-05-15T23:40:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: [WRN] PG_DEGRADED: Degraded data redundancy: 15015/171474 objects degraded (8.756%), 4 pgs degraded 2024-05-15T23:40:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: pg 9.5 is active+recovering+degraded, acting [11,7] 2024-05-15T23:40:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: pg 9.6 is active+recovering+degraded, acting [3,9] 2024-05-15T23:40:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: pg 9.a is active+recovering+degraded+remapped, acting [1,7] 2024-05-15T23:40:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: pg 9.f is active+recovering+degraded, acting [3,9] 2024-05-15T23:40:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:40:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:40:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:40:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:40:01.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:40:01.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:40:01.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:00 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:40:01.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:40:01.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: pgmap v2022: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 73 MiB/s wr, 1.17k op/s; 15015/171474 objects degraded (8.756%); 11 MiB/s, 179 objects/s recovering 2024-05-15T23:40:01.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:40:01.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:40:01.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: Health detail: HEALTH_WARN Degraded data redundancy: 15015/171474 objects degraded (8.756%), 4 pgs degraded 2024-05-15T23:40:01.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: [WRN] PG_DEGRADED: Degraded data redundancy: 15015/171474 objects degraded (8.756%), 4 pgs degraded 2024-05-15T23:40:01.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: pg 9.5 is active+recovering+degraded, acting [11,7] 2024-05-15T23:40:01.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: pg 9.6 is active+recovering+degraded, acting [3,9] 2024-05-15T23:40:01.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: pg 9.a is active+recovering+degraded+remapped, acting [1,7] 2024-05-15T23:40:01.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: pg 9.f is active+recovering+degraded, acting [3,9] 2024-05-15T23:40:01.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:40:01.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:40:01.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:40:01.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:40:01.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:40:01.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:40:01.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:00 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:40:01.133 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:01.165 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:40:01.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:01.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:40:01.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:40:01.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: pgmap v2022: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.2 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 73 MiB/s wr, 1.17k op/s; 15015/171474 objects degraded (8.756%); 11 MiB/s, 179 objects/s recovering 2024-05-15T23:40:01.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:40:01.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:40:01.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: Health detail: HEALTH_WARN Degraded data redundancy: 15015/171474 objects degraded (8.756%), 4 pgs degraded 2024-05-15T23:40:01.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: [WRN] PG_DEGRADED: Degraded data redundancy: 15015/171474 objects degraded (8.756%), 4 pgs degraded 2024-05-15T23:40:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: pg 9.5 is active+recovering+degraded, acting [11,7] 2024-05-15T23:40:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: pg 9.6 is active+recovering+degraded, acting [3,9] 2024-05-15T23:40:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: pg 9.a is active+recovering+degraded+remapped, acting [1,7] 2024-05-15T23:40:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: pg 9.f is active+recovering+degraded, acting [3,9] 2024-05-15T23:40:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:40:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:40:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:40:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:40:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:40:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:40:01.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:00 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:40:01.326 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:01.361 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:40:01.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:01.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:40:01.465 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:40:01.499 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:01.523 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:40:01.527 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:40:01.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:01.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:40:01.556 INFO:teuthology.orchestra.run.smithi088.stderr:Creating an OSD daemon form without an OSD FSID value 2024-05-15T23:40:01.573 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 59 16 91046 91030 96.4022 66.625 0.0104133 0.0103714 2024-05-15T23:40:01.630 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:40:01.664 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:01.708 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:40:01.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:01.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:40:01.814 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:40:01.869 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:01.894 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:40:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:01.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:40:01.970 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:40:01.998 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:40:02.038 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:02.095 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:40:02.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:02.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:40:02.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:02 smithi090 ceph-mon[44434]: osdmap e763: 12 total, 11 up, 9 in 2024-05-15T23:40:02.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:02 smithi090 ceph-mon[44434]: pgmap v2024: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 59 MiB/s wr, 947 op/s; 15015/174354 objects degraded (8.612%); 7.1 MiB/s, 113 objects/s recovering 2024-05-15T23:40:02.168 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2020939462' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:40:02.176 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:40:02.201 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:40:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:02.257 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:40:02.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:02.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:40:02.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:40:02.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:02 smithi084 ceph-mon[39784]: osdmap e763: 12 total, 11 up, 9 in 2024-05-15T23:40:02.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:02 smithi084 ceph-mon[39784]: pgmap v2024: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 59 MiB/s wr, 947 op/s; 15015/174354 objects degraded (8.612%); 7.1 MiB/s, 113 objects/s recovering 2024-05-15T23:40:02.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2020939462' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:40:02.418 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:02.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:02 smithi088 ceph-mon[44699]: osdmap e763: 12 total, 11 up, 9 in 2024-05-15T23:40:02.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:02 smithi088 ceph-mon[44699]: pgmap v2024: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 59 MiB/s wr, 947 op/s; 15015/174354 objects degraded (8.612%); 7.1 MiB/s, 113 objects/s recovering 2024-05-15T23:40:02.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2020939462' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:40:02.571 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:40:02.571283 min lat: 0.00103205 max lat: 19.1505 avg lat: 0.0104191 2024-05-15T23:40:02.571 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:40:02.571 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 60 16 92168 92152 95.9642 70.125 0.0084225 0.0104191 2024-05-15T23:40:03.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:03 smithi090 ceph-mon[44434]: 9.9 scrub starts 2024-05-15T23:40:03.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:03 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:40:03.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2020939462' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:40:03.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:03 smithi090 ceph-mon[44434]: osdmap e764: 12 total, 11 up, 9 in 2024-05-15T23:40:03.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:03 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 15015/174354 objects degraded (8.612%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:03.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:03 smithi088 ceph-mon[44699]: 9.9 scrub starts 2024-05-15T23:40:03.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:03 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:40:03.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2020939462' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:40:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:03 smithi088 ceph-mon[44699]: osdmap e764: 12 total, 11 up, 9 in 2024-05-15T23:40:03.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:03 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 15015/174354 objects degraded (8.612%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:03.571 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 61 16 93341 93325 95.5929 73.3125 0.00546535 0.0104584 2024-05-15T23:40:03.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:03 smithi084 ceph-mon[39784]: 9.9 scrub starts 2024-05-15T23:40:03.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:03 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:40:03.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2020939462' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:40:03.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:03 smithi084 ceph-mon[39784]: osdmap e764: 12 total, 11 up, 9 in 2024-05-15T23:40:03.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:03 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 15015/174354 objects degraded (8.612%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:04 smithi090 ceph-mon[44434]: pgmap v2026: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s; 14096/181854 objects degraded (7.751%); 12 MiB/s, 190 objects/s recovering 2024-05-15T23:40:04.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:04 smithi088 ceph-mon[44699]: pgmap v2026: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s; 14096/181854 objects degraded (7.751%); 12 MiB/s, 190 objects/s recovering 2024-05-15T23:40:04.571 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 62 16 94467 94451 95.1862 70.375 0.00840684 0.0105021 2024-05-15T23:40:04.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:04 smithi084 ceph-mon[39784]: pgmap v2026: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s; 14096/181854 objects degraded (7.751%); 12 MiB/s, 190 objects/s recovering 2024-05-15T23:40:05.083 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:40:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-05-15T23:40:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-05-15T23:40:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:40:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124079 7.1 7:857eceea:::benchmark_data_smithi090_143373_object124078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e579)", 2024-05-15T23:40:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.261908+0000", 2024-05-15T23:40:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 594.81232547100001, 2024-05-15T23:40:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26989557400000003, 2024-05-15T23:40:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124079 2024-05-15T23:40:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261908+0000", 2024-05-15T23:40:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261908+0000", 2024-05-15T23:40:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261911+0000", 2024-05-15T23:40:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4880000000000001e-06 2024-05-15T23:40:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261938+0000", 2024-05-15T23:40:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7135e-05 2024-05-15T23:40:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261952+0000", 2024-05-15T23:40:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4389999999999999e-05 2024-05-15T23:40:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261963+0000", 2024-05-15T23:40:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1094000000000001e-05 2024-05-15T23:40:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263369+0000", 2024-05-15T23:40:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014062160000000001 2024-05-15T23:40:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:40:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263378+0000", 2024-05-15T23:40:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4579999999999993e-06 2024-05-15T23:40:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.497009+0000", 2024-05-15T23:40:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23363141000000001 2024-05-15T23:40:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.499810+0000", 2024-05-15T23:40:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028003059999999998 2024-05-15T23:40:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.499987+0000", 2024-05-15T23:40:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176987 2024-05-15T23:40:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.510156+0000", 2024-05-15T23:40:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010169076000000001 2024-05-15T23:40:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.528308+0000", 2024-05-15T23:40:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018152166000000001 2024-05-15T23:40:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.528368+0000", 2024-05-15T23:40:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9679999999999998e-05 2024-05-15T23:40:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.531804+0000", 2024-05-15T23:40:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034361679999999999 2024-05-15T23:40:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124097 7.1 7:8475dbd8:::benchmark_data_smithi090_143373_object124096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:40:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.261987+0000", 2024-05-15T23:40:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 594.81224630899999, 2024-05-15T23:40:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28773406099999999, 2024-05-15T23:40:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124097 2024-05-15T23:40:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261987+0000", 2024-05-15T23:40:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261987+0000", 2024-05-15T23:40:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.261988+0000", 2024-05-15T23:40:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3799999999999997e-07 2024-05-15T23:40:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262034+0000", 2024-05-15T23:40:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6520000000000002e-05 2024-05-15T23:40:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262041+0000", 2024-05-15T23:40:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5590000000000002e-06 2024-05-15T23:40:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262045+0000", 2024-05-15T23:40:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0110000000000002e-06 2024-05-15T23:40:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263404+0000", 2024-05-15T23:40:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013591110000000001 2024-05-15T23:40:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:40:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263410+0000", 2024-05-15T23:40:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780000000000003e-06 2024-05-15T23:40:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.506311+0000", 2024-05-15T23:40:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.242901425 2024-05-15T23:40:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.506407+0000", 2024-05-15T23:40:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5315999999999993e-05 2024-05-15T23:40:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.506505+0000", 2024-05-15T23:40:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8548999999999994e-05 2024-05-15T23:40:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.521629+0000", 2024-05-15T23:40:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015123319 2024-05-15T23:40:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.546928+0000", 2024-05-15T23:40:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025299687000000001 2024-05-15T23:40:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.546964+0000", 2024-05-15T23:40:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5552000000000001e-05 2024-05-15T23:40:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.549721+0000", 2024-05-15T23:40:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027575960000000002 2024-05-15T23:40:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124098 7.1 7:85fd872c:::benchmark_data_smithi090_143373_object124097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:40:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.262051+0000", 2024-05-15T23:40:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 594.81218292799997, 2024-05-15T23:40:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41649122300000002, 2024-05-15T23:40:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124098 2024-05-15T23:40:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262051+0000", 2024-05-15T23:40:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262051+0000", 2024-05-15T23:40:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262051+0000", 2024-05-15T23:40:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5699999999999998e-07 2024-05-15T23:40:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262088+0000", 2024-05-15T23:40:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6467000000000002e-05 2024-05-15T23:40:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262099+0000", 2024-05-15T23:40:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1422e-05 2024-05-15T23:40:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262111+0000", 2024-05-15T23:40:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1593000000000001e-05 2024-05-15T23:40:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263432+0000", 2024-05-15T23:40:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013212580000000001 2024-05-15T23:40:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:40:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263436+0000", 2024-05-15T23:40:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9310000000000001e-06 2024-05-15T23:40:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.510182+0000", 2024-05-15T23:40:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.24674654900000001 2024-05-15T23:40:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.510289+0000", 2024-05-15T23:40:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106941 2024-05-15T23:40:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.510417+0000", 2024-05-15T23:40:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012745399999999999 2024-05-15T23:40:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.521638+0000", 2024-05-15T23:40:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011221085 2024-05-15T23:40:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.675885+0000", 2024-05-15T23:40:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15424710599999999 2024-05-15T23:40:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.675949+0000", 2024-05-15T23:40:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4082000000000001e-05 2024-05-15T23:40:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.678542+0000", 2024-05-15T23:40:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025928779999999998 2024-05-15T23:40:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124106 7.1 7:8dcf5eef:::benchmark_data_smithi090_143373_object124105:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:40:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.262116+0000", 2024-05-15T23:40:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 594.81211754900005, 2024-05-15T23:40:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41946845300000002, 2024-05-15T23:40:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124106 2024-05-15T23:40:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262116+0000", 2024-05-15T23:40:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262116+0000", 2024-05-15T23:40:05.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262116+0000", 2024-05-15T23:40:05.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1399999999999997e-07 2024-05-15T23:40:05.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262149+0000", 2024-05-15T23:40:05.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2295999999999997e-05 2024-05-15T23:40:05.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262154+0000", 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9130000000000002e-06 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262158+0000", 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0810000000000004e-06 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263455+0000", 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012971969999999999 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:40:05.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263459+0000", 2024-05-15T23:40:05.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7859999999999998e-06 2024-05-15T23:40:05.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.513785+0000", 2024-05-15T23:40:05.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25032668899999999 2024-05-15T23:40:05.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.513908+0000", 2024-05-15T23:40:05.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012284899999999999 2024-05-15T23:40:05.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.514012+0000", 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103525 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.528239+0000", 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014226946000000001 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.678901+0000", 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15066216900000001 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.678961+0000", 2024-05-15T23:40:05.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0356000000000003e-05 2024-05-15T23:40:05.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.681585+0000", 2024-05-15T23:40:05.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026232320000000001 2024-05-15T23:40:05.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.110 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.110 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124122 7.1 7:88203a70:::benchmark_data_smithi090_143373_object124121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:40:05.110 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.262162+0000", 2024-05-15T23:40:05.111 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 594.81207181100001, 2024-05-15T23:40:05.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.42187396700000002, 2024-05-15T23:40:05.111 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.111 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.111 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.111 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124122 2024-05-15T23:40:05.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.111 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262162+0000", 2024-05-15T23:40:05.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262162+0000", 2024-05-15T23:40:05.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262162+0000", 2024-05-15T23:40:05.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2900000000000004e-07 2024-05-15T23:40:05.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262193+0000", 2024-05-15T23:40:05.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0460999999999999e-05 2024-05-15T23:40:05.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262208+0000", 2024-05-15T23:40:05.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5325e-05 2024-05-15T23:40:05.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262212+0000", 2024-05-15T23:40:05.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4809999999999998e-06 2024-05-15T23:40:05.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263485+0000", 2024-05-15T23:40:05.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001273176 2024-05-15T23:40:05.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:40:05.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263491+0000", 2024-05-15T23:40:05.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1480000000000003e-06 2024-05-15T23:40:05.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.516405+0000", 2024-05-15T23:40:05.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.252914099 2024-05-15T23:40:05.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.516538+0000", 2024-05-15T23:40:05.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013272499999999999 2024-05-15T23:40:05.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.516687+0000", 2024-05-15T23:40:05.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149693 2024-05-15T23:40:05.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.528245+0000", 2024-05-15T23:40:05.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011557596 2024-05-15T23:40:05.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.681654+0000", 2024-05-15T23:40:05.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15340879099999999 2024-05-15T23:40:05.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.681714+0000", 2024-05-15T23:40:05.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9806000000000003e-05 2024-05-15T23:40:05.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.684036+0000", 2024-05-15T23:40:05.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023221370000000002 2024-05-15T23:40:05.117 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.117 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.117 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.117 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124161 7.1 7:80dd3600:::benchmark_data_smithi090_143373_object124160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:40:05.117 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.262216+0000", 2024-05-15T23:40:05.117 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 594.812017481, 2024-05-15T23:40:05.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.42874030499999999, 2024-05-15T23:40:05.117 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.117 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.117 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.118 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.118 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124161 2024-05-15T23:40:05.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.118 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262216+0000", 2024-05-15T23:40:05.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262216+0000", 2024-05-15T23:40:05.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262217+0000", 2024-05-15T23:40:05.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8700000000000005e-07 2024-05-15T23:40:05.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262251+0000", 2024-05-15T23:40:05.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4162000000000003e-05 2024-05-15T23:40:05.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262256+0000", 2024-05-15T23:40:05.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5850000000000001e-06 2024-05-15T23:40:05.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262259+0000", 2024-05-15T23:40:05.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7790000000000002e-06 2024-05-15T23:40:05.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263546+0000", 2024-05-15T23:40:05.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012862450000000001 2024-05-15T23:40:05.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:40:05.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263552+0000", 2024-05-15T23:40:05.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9000000000000003e-06 2024-05-15T23:40:05.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.518749+0000", 2024-05-15T23:40:05.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25519711699999997 2024-05-15T23:40:05.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.518838+0000", 2024-05-15T23:40:05.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8896999999999999e-05 2024-05-15T23:40:05.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.518935+0000", 2024-05-15T23:40:05.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7421000000000001e-05 2024-05-15T23:40:05.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.535377+0000", 2024-05-15T23:40:05.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016442432 2024-05-15T23:40:05.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.687575+0000", 2024-05-15T23:40:05.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15219728799999999 2024-05-15T23:40:05.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.687629+0000", 2024-05-15T23:40:05.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4429999999999999e-05 2024-05-15T23:40:05.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.690956+0000", 2024-05-15T23:40:05.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033272620000000001 2024-05-15T23:40:05.123 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.123 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.123 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.123 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124174 7.1 7:856e5dbc:::benchmark_data_smithi090_143373_object124173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:40:05.123 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.262264+0000", 2024-05-15T23:40:05.123 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 594.81196995599998, 2024-05-15T23:40:05.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.43172603300000001, 2024-05-15T23:40:05.124 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.124 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.124 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.124 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.124 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.124 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.124 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124174 2024-05-15T23:40:05.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.124 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262264+0000", 2024-05-15T23:40:05.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262264+0000", 2024-05-15T23:40:05.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262264+0000", 2024-05-15T23:40:05.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0499999999999999e-07 2024-05-15T23:40:05.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262296+0000", 2024-05-15T23:40:05.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2254e-05 2024-05-15T23:40:05.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262301+0000", 2024-05-15T23:40:05.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.724e-06 2024-05-15T23:40:05.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.262305+0000", 2024-05-15T23:40:05.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7469999999999999e-06 2024-05-15T23:40:05.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263583+0000", 2024-05-15T23:40:05.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012785819999999999 2024-05-15T23:40:05.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:40:05.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.263587+0000", 2024-05-15T23:40:05.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.732e-06 2024-05-15T23:40:05.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.521673+0000", 2024-05-15T23:40:05.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25808625800000001 2024-05-15T23:40:05.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.521777+0000", 2024-05-15T23:40:05.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010374600000000001 2024-05-15T23:40:05.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.521947+0000", 2024-05-15T23:40:05.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017013099999999999 2024-05-15T23:40:05.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.535382+0000", 2024-05-15T23:40:05.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013435163 2024-05-15T23:40:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.691026+0000", 2024-05-15T23:40:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15564313299999999 2024-05-15T23:40:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.691082+0000", 2024-05-15T23:40:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6496000000000003e-05 2024-05-15T23:40:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.693990+0000", 2024-05-15T23:40:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029076620000000001 2024-05-15T23:40:05.130 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.130 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.130 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.130 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124212 7.1 7:81100074:::benchmark_data_smithi090_143373_object124211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:40:05.130 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.324932+0000", 2024-05-15T23:40:05.130 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 594.74930208199999, 2024-05-15T23:40:05.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37580514700000001, 2024-05-15T23:40:05.130 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.130 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.130 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.130 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.130 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.131 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.131 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124212 2024-05-15T23:40:05.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.131 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324932+0000", 2024-05-15T23:40:05.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324932+0000", 2024-05-15T23:40:05.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324933+0000", 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.497e-06 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324974+0000", 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1214999999999998e-05 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324984+0000", 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7359999999999996e-06 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.324989+0000", 2024-05-15T23:40:05.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.259e-06 2024-05-15T23:40:05.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.325040+0000", 2024-05-15T23:40:05.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1172999999999999e-05 2024-05-15T23:40:05.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:40:05.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.325051+0000", 2024-05-15T23:40:05.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0317e-05 2024-05-15T23:40:05.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.524923+0000", 2024-05-15T23:40:05.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19987254500000001 2024-05-15T23:40:05.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.525080+0000", 2024-05-15T23:40:05.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015695999999999999 2024-05-15T23:40:05.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.525237+0000", 2024-05-15T23:40:05.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015635200000000001 2024-05-15T23:40:05.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.540733+0000", 2024-05-15T23:40:05.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01549625 2024-05-15T23:40:05.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.697259+0000", 2024-05-15T23:40:05.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15652601099999999 2024-05-15T23:40:05.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.697319+0000", 2024-05-15T23:40:05.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0210999999999997e-05 2024-05-15T23:40:05.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.700737+0000", 2024-05-15T23:40:05.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003417621 2024-05-15T23:40:05.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.136 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.136 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124224 7.1 7:81c65e53:::benchmark_data_smithi090_143373_object124223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:40:05.136 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.344409+0000", 2024-05-15T23:40:05.136 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 594.72982439899999, 2024-05-15T23:40:05.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.371311897, 2024-05-15T23:40:05.136 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.136 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.137 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.137 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.137 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.137 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.137 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124224 2024-05-15T23:40:05.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.137 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344409+0000", 2024-05-15T23:40:05.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344409+0000", 2024-05-15T23:40:05.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344412+0000", 2024-05-15T23:40:05.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3180000000000001e-06 2024-05-15T23:40:05.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344455+0000", 2024-05-15T23:40:05.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3247000000000001e-05 2024-05-15T23:40:05.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344466+0000", 2024-05-15T23:40:05.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0922e-05 2024-05-15T23:40:05.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344472+0000", 2024-05-15T23:40:05.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2790000000000004e-06 2024-05-15T23:40:05.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344531+0000", 2024-05-15T23:40:05.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9055000000000003e-05 2024-05-15T23:40:05.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:40:05.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.344539+0000", 2024-05-15T23:40:05.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6040000000000004e-06 2024-05-15T23:40:05.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.531872+0000", 2024-05-15T23:40:05.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.187333584 2024-05-15T23:40:05.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.532039+0000", 2024-05-15T23:40:05.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016709099999999999 2024-05-15T23:40:05.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.532223+0000", 2024-05-15T23:40:05.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000183246 2024-05-15T23:40:05.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.545260+0000", 2024-05-15T23:40:05.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013037648000000001 2024-05-15T23:40:05.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.714171+0000", 2024-05-15T23:40:05.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.168910483 2024-05-15T23:40:05.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.714217+0000", 2024-05-15T23:40:05.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6461999999999997e-05 2024-05-15T23:40:05.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.715721+0000", 2024-05-15T23:40:05.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015039579999999999 2024-05-15T23:40:05.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.142 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.143 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124234 7.1 7:8b71c4c6:::benchmark_data_smithi090_143373_object124233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:40:05.143 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.366886+0000", 2024-05-15T23:40:05.143 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 594.707347486, 2024-05-15T23:40:05.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35046440299999998, 2024-05-15T23:40:05.143 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.143 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.143 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.143 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124234 2024-05-15T23:40:05.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.143 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366886+0000", 2024-05-15T23:40:05.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366886+0000", 2024-05-15T23:40:05.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366888+0000", 2024-05-15T23:40:05.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8920000000000001e-06 2024-05-15T23:40:05.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366934+0000", 2024-05-15T23:40:05.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5543000000000001e-05 2024-05-15T23:40:05.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366945+0000", 2024-05-15T23:40:05.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1345999999999999e-05 2024-05-15T23:40:05.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.366951+0000", 2024-05-15T23:40:05.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3400000000000003e-06 2024-05-15T23:40:05.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.367009+0000", 2024-05-15T23:40:05.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7907e-05 2024-05-15T23:40:05.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:40:05.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.367022+0000", 2024-05-15T23:40:05.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2492e-05 2024-05-15T23:40:05.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.535417+0000", 2024-05-15T23:40:05.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16839585300000001 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.535529+0000", 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011196600000000001 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.535644+0000", 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011408099999999999 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.545264+0000", 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096206910000000007 2024-05-15T23:40:05.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.715777+0000", 2024-05-15T23:40:05.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17051261100000001 2024-05-15T23:40:05.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.715818+0000", 2024-05-15T23:40:05.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1119999999999999e-05 2024-05-15T23:40:05.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.717351+0000", 2024-05-15T23:40:05.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001532561 2024-05-15T23:40:05.148 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.149 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.149 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.149 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124269 7.1 7:84b5a5c1:::benchmark_data_smithi090_143373_object124268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:40:05.149 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.428950+0000", 2024-05-15T23:40:05.149 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 594.64528366599995, 2024-05-15T23:40:05.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31337896399999998, 2024-05-15T23:40:05.149 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.149 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.149 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.149 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.149 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.149 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.149 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124269 2024-05-15T23:40:05.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.150 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.428950+0000", 2024-05-15T23:40:05.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.428950+0000", 2024-05-15T23:40:05.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.428952+0000", 2024-05-15T23:40:05.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9989999999999998e-06 2024-05-15T23:40:05.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.428998+0000", 2024-05-15T23:40:05.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5986999999999999e-05 2024-05-15T23:40:05.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.429009+0000", 2024-05-15T23:40:05.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1008e-05 2024-05-15T23:40:05.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.429017+0000", 2024-05-15T23:40:05.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1110000000000008e-06 2024-05-15T23:40:05.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.429073+0000", 2024-05-15T23:40:05.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5998999999999997e-05 2024-05-15T23:40:05.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-05-15T23:40:05.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.429081+0000", 2024-05-15T23:40:05.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.943e-06 2024-05-15T23:40:05.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.538267+0000", 2024-05-15T23:40:05.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.109185886 2024-05-15T23:40:05.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.538366+0000", 2024-05-15T23:40:05.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8715000000000006e-05 2024-05-15T23:40:05.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.538461+0000", 2024-05-15T23:40:05.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5101999999999998e-05 2024-05-15T23:40:05.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.545270+0000", 2024-05-15T23:40:05.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0068090900000000003 2024-05-15T23:40:05.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.739358+0000", 2024-05-15T23:40:05.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.194088025 2024-05-15T23:40:05.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.739415+0000", 2024-05-15T23:40:05.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6719999999999999e-05 2024-05-15T23:40:05.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.742329+0000", 2024-05-15T23:40:05.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029143789999999999 2024-05-15T23:40:05.155 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.155 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.155 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.155 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124293 7.1 7:8a970b3a:::benchmark_data_smithi090_143373_object124292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:40:05.155 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.483940+0000", 2024-05-15T23:40:05.155 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 594.59029331800002, 2024-05-15T23:40:05.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27423620599999998, 2024-05-15T23:40:05.156 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.156 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.156 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.156 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.156 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.156 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.156 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124293 2024-05-15T23:40:05.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.156 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.483940+0000", 2024-05-15T23:40:05.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.483940+0000", 2024-05-15T23:40:05.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.483943+0000", 2024-05-15T23:40:05.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.322e-06 2024-05-15T23:40:05.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.483993+0000", 2024-05-15T23:40:05.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0241000000000002e-05 2024-05-15T23:40:05.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.484005+0000", 2024-05-15T23:40:05.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2024-05-15T23:40:05.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.484012+0000", 2024-05-15T23:40:05.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2849999999999997e-06 2024-05-15T23:40:05.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.540767+0000", 2024-05-15T23:40:05.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.056755011000000001 2024-05-15T23:40:05.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.540856+0000", 2024-05-15T23:40:05.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9073000000000006e-05 2024-05-15T23:40:05.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.540949+0000", 2024-05-15T23:40:05.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2831000000000006e-05 2024-05-15T23:40:05.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.545273+0000", 2024-05-15T23:40:05.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043239929999999999 2024-05-15T23:40:05.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.755350+0000", 2024-05-15T23:40:05.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.210076601 2024-05-15T23:40:05.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.755411+0000", 2024-05-15T23:40:05.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1271000000000003e-05 2024-05-15T23:40:05.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.758176+0000", 2024-05-15T23:40:05.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027657020000000001 2024-05-15T23:40:05.160 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.161 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.161 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.161 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:124302 7.1 7:83d236b9:::benchmark_data_smithi090_143373_object124301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e579)", 2024-05-15T23:40:05.161 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:10.505719+0000", 2024-05-15T23:40:05.161 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 594.56851497599996, 2024-05-15T23:40:05.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25507802400000001, 2024-05-15T23:40:05.161 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.161 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.161 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.161 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.161 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.161 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.161 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 124302 2024-05-15T23:40:05.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.162 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505719+0000", 2024-05-15T23:40:05.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505719+0000", 2024-05-15T23:40:05.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505721+0000", 2024-05-15T23:40:05.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3760000000000002e-06 2024-05-15T23:40:05.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505778+0000", 2024-05-15T23:40:05.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6858999999999999e-05 2024-05-15T23:40:05.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505794+0000", 2024-05-15T23:40:05.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5926e-05 2024-05-15T23:40:05.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.505804+0000", 2024-05-15T23:40:05.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0219e-05 2024-05-15T23:40:05.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.543199+0000", 2024-05-15T23:40:05.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.03739543 2024-05-15T23:40:05.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.543285+0000", 2024-05-15T23:40:05.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5523000000000001e-05 2024-05-15T23:40:05.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.543379+0000", 2024-05-15T23:40:05.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4110000000000005e-05 2024-05-15T23:40:05.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.545425+0000", 2024-05-15T23:40:05.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020457280000000001 2024-05-15T23:40:05.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.758248+0000", 2024-05-15T23:40:05.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.212822821 2024-05-15T23:40:05.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.758294+0000", 2024-05-15T23:40:05.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6774999999999999e-05 2024-05-15T23:40:05.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:10.760797+0000", 2024-05-15T23:40:05.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025022569999999999 2024-05-15T23:40:05.166 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.166 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.166 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.167 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:165512 7.1 7:81ed44b8:::benchmark_data_smithi090_143373_object19341:head [delete] snapc 0=[] ondisk+write+known_if_redirected e591)", 2024-05-15T23:40:05.167 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:30:57.374026+0000", 2024-05-15T23:40:05.167 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 547.70020744399994, 2024-05-15T23:40:05.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.249226325, 2024-05-15T23:40:05.167 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.167 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.167 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.167 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.167 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.167 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.167 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 165512 2024-05-15T23:40:05.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.167 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374026+0000", 2024-05-15T23:40:05.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374026+0000", 2024-05-15T23:40:05.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374029+0000", 2024-05-15T23:40:05.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6060000000000001e-06 2024-05-15T23:40:05.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374037+0000", 2024-05-15T23:40:05.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3950000000000001e-06 2024-05-15T23:40:05.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374050+0000", 2024-05-15T23:40:05.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2826999999999999e-05 2024-05-15T23:40:05.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.374060+0000", 2024-05-15T23:40:05.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9680000000000001e-06 2024-05-15T23:40:05.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.383163+0000", 2024-05-15T23:40:05.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091029890000000006 2024-05-15T23:40:05.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:40:05.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.383237+0000", 2024-05-15T23:40:05.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4456000000000007e-05 2024-05-15T23:40:05.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.567432+0000", 2024-05-15T23:40:05.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18419463999999999 2024-05-15T23:40:05.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.567524+0000", 2024-05-15T23:40:05.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2298000000000003e-05 2024-05-15T23:40:05.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.567612+0000", 2024-05-15T23:40:05.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7293999999999995e-05 2024-05-15T23:40:05.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.600864+0000", 2024-05-15T23:40:05.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033252149000000002 2024-05-15T23:40:05.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.620769+0000", 2024-05-15T23:40:05.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019905062000000001 2024-05-15T23:40:05.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.620810+0000", 2024-05-15T23:40:05.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0685000000000001e-05 2024-05-15T23:40:05.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:30:57.623252+0000", 2024-05-15T23:40:05.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002442956 2024-05-15T23:40:05.173 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.173 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.173 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.173 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:168938 7.1 7:87745b29:::benchmark_data_smithi090_143373_object22767:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:40:05.173 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:01.604430+0000", 2024-05-15T23:40:05.173 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 543.46980353200001, 2024-05-15T23:40:05.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.246406922, 2024-05-15T23:40:05.173 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.173 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.173 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.174 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168938 2024-05-15T23:40:05.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.174 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604430+0000", 2024-05-15T23:40:05.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604430+0000", 2024-05-15T23:40:05.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604433+0000", 2024-05-15T23:40:05.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.661e-06 2024-05-15T23:40:05.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604442+0000", 2024-05-15T23:40:05.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1749999999999994e-06 2024-05-15T23:40:05.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604456+0000", 2024-05-15T23:40:05.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4354e-05 2024-05-15T23:40:05.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604468+0000", 2024-05-15T23:40:05.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1389000000000001e-05 2024-05-15T23:40:05.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.631305+0000", 2024-05-15T23:40:05.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02683758 2024-05-15T23:40:05.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:40:05.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.631363+0000", 2024-05-15T23:40:05.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8009e-05 2024-05-15T23:40:05.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.820785+0000", 2024-05-15T23:40:05.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.189421428 2024-05-15T23:40:05.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.820880+0000", 2024-05-15T23:40:05.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4852000000000006e-05 2024-05-15T23:40:05.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.821015+0000", 2024-05-15T23:40:05.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135709 2024-05-15T23:40:05.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.837594+0000", 2024-05-15T23:40:05.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016578446 2024-05-15T23:40:05.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.847339+0000", 2024-05-15T23:40:05.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0097453809999999991 2024-05-15T23:40:05.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.847400+0000", 2024-05-15T23:40:05.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1194999999999999e-05 2024-05-15T23:40:05.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.850837+0000", 2024-05-15T23:40:05.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034367429999999999 2024-05-15T23:40:05.179 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.179 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.179 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.180 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:168941 7.1 7:8a84f08b:::benchmark_data_smithi090_143373_object22770:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:40:05.180 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:01.604543+0000", 2024-05-15T23:40:05.180 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 543.46969066400004, 2024-05-15T23:40:05.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.33646746300000002, 2024-05-15T23:40:05.180 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.180 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.180 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.180 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168941 2024-05-15T23:40:05.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.180 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604543+0000", 2024-05-15T23:40:05.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604543+0000", 2024-05-15T23:40:05.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604544+0000", 2024-05-15T23:40:05.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9500000000000001e-07 2024-05-15T23:40:05.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604548+0000", 2024-05-15T23:40:05.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0380000000000001e-06 2024-05-15T23:40:05.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604554+0000", 2024-05-15T23:40:05.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1940000000000003e-06 2024-05-15T23:40:05.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.604559+0000", 2024-05-15T23:40:05.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8300000000000003e-06 2024-05-15T23:40:05.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634299+0000", 2024-05-15T23:40:05.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740566 2024-05-15T23:40:05.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:40:05.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634357+0000", 2024-05-15T23:40:05.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7884999999999999e-05 2024-05-15T23:40:05.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.916090+0000", 2024-05-15T23:40:05.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28173253799999998 2024-05-15T23:40:05.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.916206+0000", 2024-05-15T23:40:05.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116403 2024-05-15T23:40:05.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.916353+0000", 2024-05-15T23:40:05.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014637100000000001 2024-05-15T23:40:05.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.934169+0000", 2024-05-15T23:40:05.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017816117999999999 2024-05-15T23:40:05.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.939265+0000", 2024-05-15T23:40:05.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050958519999999997 2024-05-15T23:40:05.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.939322+0000", 2024-05-15T23:40:05.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7025999999999999e-05 2024-05-15T23:40:05.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.941010+0000", 2024-05-15T23:40:05.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001688747 2024-05-15T23:40:05.186 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.186 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.186 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.186 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPushReply(7.1 592/578 [PushReplyOp(7:8a84f08b:::benchmark_data_smithi090_143373_object22770:head)])", 2024-05-15T23:40:05.186 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:01.634777+0000", 2024-05-15T23:40:05.186 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 543.43945647999999, 2024-05-15T23:40:05.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28126164100000001, 2024-05-15T23:40:05.186 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.186 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.186 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-05-15T23:40:05.187 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634777+0000", 2024-05-15T23:40:05.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634777+0000", 2024-05-15T23:40:05.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634779+0000", 2024-05-15T23:40:05.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5099999999999999e-06 2024-05-15T23:40:05.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634789+0000", 2024-05-15T23:40:05.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0056e-05 2024-05-15T23:40:05.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634801+0000", 2024-05-15T23:40:05.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2532e-05 2024-05-15T23:40:05.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.634809+0000", 2024-05-15T23:40:05.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8099999999999998e-06 2024-05-15T23:40:05.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.909849+0000", 2024-05-15T23:40:05.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27503971700000002 2024-05-15T23:40:05.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:01.916039+0000", 2024-05-15T23:40:05.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061900159999999996 2024-05-15T23:40:05.189 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.189 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.189 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.190 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:169812 7.1 7:817b8891:::benchmark_data_smithi090_143373_object23641:head [delete] snapc 0=[] ondisk+write+known_if_redirected e592)", 2024-05-15T23:40:05.190 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:02.735245+0000", 2024-05-15T23:40:05.190 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 542.33898817900001, 2024-05-15T23:40:05.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31279489399999999, 2024-05-15T23:40:05.190 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.190 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.191 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.191 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 169812 2024-05-15T23:40:05.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.191 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.735245+0000", 2024-05-15T23:40:05.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.735245+0000", 2024-05-15T23:40:05.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.735248+0000", 2024-05-15T23:40:05.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3549999999999999e-06 2024-05-15T23:40:05.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.735255+0000", 2024-05-15T23:40:05.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8029999999999999e-06 2024-05-15T23:40:05.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909550+0000", 2024-05-15T23:40:05.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.174295218 2024-05-15T23:40:05.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909565+0000", 2024-05-15T23:40:05.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-05-15T23:40:05.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909644+0000", 2024-05-15T23:40:05.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9839999999999995e-05 2024-05-15T23:40:05.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:40:05.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:02.909718+0000", 2024-05-15T23:40:05.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3475999999999999e-05 2024-05-15T23:40:05.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.012591+0000", 2024-05-15T23:40:05.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102873279 2024-05-15T23:40:05.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.012700+0000", 2024-05-15T23:40:05.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108721 2024-05-15T23:40:05.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.012857+0000", 2024-05-15T23:40:05.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157305 2024-05-15T23:40:05.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.030528+0000", 2024-05-15T23:40:05.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017670788 2024-05-15T23:40:05.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.045291+0000", 2024-05-15T23:40:05.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014763108 2024-05-15T23:40:05.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.045348+0000", 2024-05-15T23:40:05.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6375000000000001e-05 2024-05-15T23:40:05.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:03.048040+0000", 2024-05-15T23:40:05.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002692776 2024-05-15T23:40:05.196 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.196 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.196 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.197 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:208781 7.1 7:87e9d900:::benchmark_data_smithi090_143373_object62610:head [delete] snapc 0=[] ondisk+write+known_if_redirected e607)", 2024-05-15T23:40:05.197 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:31:55.190408+0000", 2024-05-15T23:40:05.197 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 489.88382510299999, 2024-05-15T23:40:05.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.24431366500000001, 2024-05-15T23:40:05.197 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.197 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.197 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.197 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.197 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.197 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.197 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 208781 2024-05-15T23:40:05.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.197 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.190408+0000", 2024-05-15T23:40:05.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.190408+0000", 2024-05-15T23:40:05.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.190409+0000", 2024-05-15T23:40:05.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8899999999999998e-07 2024-05-15T23:40:05.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.190412+0000", 2024-05-15T23:40:05.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9689999999999999e-06 2024-05-15T23:40:05.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.190418+0000", 2024-05-15T23:40:05.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8969999999999996e-06 2024-05-15T23:40:05.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.190424+0000", 2024-05-15T23:40:05.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5890000000000002e-06 2024-05-15T23:40:05.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.215086+0000", 2024-05-15T23:40:05.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024662133999999999 2024-05-15T23:40:05.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:40:05.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.215146+0000", 2024-05-15T23:40:05.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9737000000000001e-05 2024-05-15T23:40:05.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.251337+0000", 2024-05-15T23:40:05.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036191359999999999 2024-05-15T23:40:05.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.251394+0000", 2024-05-15T23:40:05.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7102999999999999e-05 2024-05-15T23:40:05.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.251470+0000", 2024-05-15T23:40:05.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5964999999999995e-05 2024-05-15T23:40:05.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.262512+0000", 2024-05-15T23:40:05.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011041751000000001 2024-05-15T23:40:05.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.432850+0000", 2024-05-15T23:40:05.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.170338194 2024-05-15T23:40:05.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.432915+0000", 2024-05-15T23:40:05.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4789999999999995e-05 2024-05-15T23:40:05.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:31:55.434722+0000", 2024-05-15T23:40:05.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018072870000000001 2024-05-15T23:40:05.203 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.203 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.203 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.203 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.16764.0:248392 7.1 7:88e83ff0:::benchmark_data_smithi090_143373_object102221:head [delete] snapc 0=[] ondisk+write+known_if_redirected e616)", 2024-05-15T23:40:05.203 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:32:38.458728+0000", 2024-05-15T23:40:05.203 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 446.615505224, 2024-05-15T23:40:05.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23309224200000001, 2024-05-15T23:40:05.203 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:05.203 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:05.203 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-15T23:40:05.203 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.16764", 2024-05-15T23:40:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2062773504", 2024-05-15T23:40:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 248392 2024-05-15T23:40:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.458728+0000", 2024-05-15T23:40:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.458728+0000", 2024-05-15T23:40:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:05.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.458731+0000", 2024-05-15T23:40:05.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.379e-06 2024-05-15T23:40:05.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:05.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.458737+0000", 2024-05-15T23:40:05.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5690000000000001e-06 2024-05-15T23:40:05.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:05.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.458749+0000", 2024-05-15T23:40:05.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1450999999999999e-05 2024-05-15T23:40:05.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:05.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.458757+0000", 2024-05-15T23:40:05.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0350000000000001e-06 2024-05-15T23:40:05.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.612089+0000", 2024-05-15T23:40:05.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15333242999999999 2024-05-15T23:40:05.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2024-05-15T23:40:05.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.612703+0000", 2024-05-15T23:40:05.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061384099999999995 2024-05-15T23:40:05.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:05.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.641419+0000", 2024-05-15T23:40:05.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028715898 2024-05-15T23:40:05.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:05.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.641479+0000", 2024-05-15T23:40:05.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9755e-05 2024-05-15T23:40:05.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-05-15T23:40:05.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.641580+0000", 2024-05-15T23:40:05.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100875 2024-05-15T23:40:05.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:05.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.654633+0000", 2024-05-15T23:40:05.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013053021999999999 2024-05-15T23:40:05.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:05.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.690959+0000", 2024-05-15T23:40:05.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036326439000000002 2024-05-15T23:40:05.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-05-15T23:40:05.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.690997+0000", 2024-05-15T23:40:05.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7543000000000003e-05 2024-05-15T23:40:05.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:05.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:05.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-05-15T23:40:05.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:32:38.691821+0000", 2024-05-15T23:40:05.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082400500000000005 2024-05-15T23:40:05.209 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.209 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.209 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.209 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:05.209 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:05.209 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:40:05.406 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:05 smithi090 ceph-mon[44434]: pgmap v2027: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.7 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s; 14096/185278 objects degraded (7.608%); 8.3 MiB/s, 133 objects/s recovering 2024-05-15T23:40:05.574 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 63 16 95602 95586 94.7971 70.9375 0.0302557 0.0105468 2024-05-15T23:40:05.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:05 smithi088 ceph-mon[44699]: pgmap v2027: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.7 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s; 14096/185278 objects degraded (7.608%); 8.3 MiB/s, 133 objects/s recovering 2024-05-15T23:40:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:05 smithi084 ceph-mon[39784]: pgmap v2027: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.7 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 66 MiB/s wr, 1.05k op/s; 14096/185278 objects degraded (7.608%); 8.3 MiB/s, 133 objects/s recovering 2024-05-15T23:40:06.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T23:40:06.471 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:40:06.574 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 64 15 96601 96586 94.2925 62.5 0.0177003 0.0106034 2024-05-15T23:40:07.574 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 65 16 97701 97685 93.8986 68.6875 0.0150969 0.010647 2024-05-15T23:40:07.746 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:40:07.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:07.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:40:07.751 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:07 smithi084 ceph-mon[39784]: pgmap v2028: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 78 MiB/s wr, 1.25k op/s; 13377/191464 objects degraded (6.987%); 13 MiB/s, 204 objects/s recovering 2024-05-15T23:40:07.751 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:07 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 14096/185278 objects degraded (7.608%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:07.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:07 smithi090 ceph-mon[44434]: pgmap v2028: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 78 MiB/s wr, 1.25k op/s; 13377/191464 objects degraded (6.987%); 13 MiB/s, 204 objects/s recovering 2024-05-15T23:40:07.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:07 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 14096/185278 objects degraded (7.608%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:07.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:07 smithi088 ceph-mon[44699]: pgmap v2028: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 78 MiB/s wr, 1.25k op/s; 13377/191464 objects degraded (6.987%); 13 MiB/s, 204 objects/s recovering 2024-05-15T23:40:07.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:07 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 14096/185278 objects degraded (7.608%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:07.851 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:40:07.911 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:08.173 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:40:08.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:08.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:40:08.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:40:08.332 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:08.369 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:40:08.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:08.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:40:08.477 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:40:08.513 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:08.575 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 66 16 98835 98819 93.5488 70.875 0.00756814 0.0106874 2024-05-15T23:40:08.870 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:40:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:08.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:40:08.977 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:40:09.014 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:09.037 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:40:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:09.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:40:09.142 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:40:09.163 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:40:09.163 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-05-15T23:40:09.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.163 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.27292.0:99474 9.0 9:0452b552:::benchmark_data_smithi090_166897_object99473:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-05-15T23:40:09.163 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:40:09.141322+0000", 2024-05-15T23:40:09.164 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.020337551999999998, 2024-05-15T23:40:09.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020362910000000001, 2024-05-15T23:40:09.164 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:09.164 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:09.164 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:40:09.164 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:09.164 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.27292", 2024-05-15T23:40:09.164 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2136483304", 2024-05-15T23:40:09.164 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 99474 2024-05-15T23:40:09.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.141322+0000", 2024-05-15T23:40:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.141322+0000", 2024-05-15T23:40:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.141323+0000", 2024-05-15T23:40:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2900000000000003e-07 2024-05-15T23:40:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.141351+0000", 2024-05-15T23:40:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8076e-05 2024-05-15T23:40:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.141359+0000", 2024-05-15T23:40:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6760000000000007e-06 2024-05-15T23:40:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.141364+0000", 2024-05-15T23:40:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2909999999999998e-06 2024-05-15T23:40:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.149666+0000", 2024-05-15T23:40:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0083018780000000004 2024-05-15T23:40:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.149785+0000", 2024-05-15T23:40:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011952500000000001 2024-05-15T23:40:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:40:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.149897+0000", 2024-05-15T23:40:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111121 2024-05-15T23:40:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-05-15T23:40:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.156930+0000", 2024-05-15T23:40:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0070330180000000003 2024-05-15T23:40:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.27292.0:99466 9.8 e765/737 9:116e4a0c:::benchmark_data_smithi090_166897_object99465:head v 765'6290, mlcod=765'6286)", 2024-05-15T23:40:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:40:09.144956+0000", 2024-05-15T23:40:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.016703580999999999, 2024-05-15T23:40:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016783078999999999, 2024-05-15T23:40:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:40:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.144956+0000", 2024-05-15T23:40:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.144956+0000", 2024-05-15T23:40:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.144958+0000", 2024-05-15T23:40:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.468e-06 2024-05-15T23:40:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.144982+0000", 2024-05-15T23:40:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4278999999999999e-05 2024-05-15T23:40:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.144995+0000", 2024-05-15T23:40:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3059e-05 2024-05-15T23:40:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.145002+0000", 2024-05-15T23:40:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2069999999999998e-06 2024-05-15T23:40:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.146341+0000", 2024-05-15T23:40:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001338508 2024-05-15T23:40:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.146360+0000", 2024-05-15T23:40:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9528999999999999e-05 2024-05-15T23:40:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.27292.0:99475 9.0 9:008ef29b:::benchmark_data_smithi090_166897_object99474:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-05-15T23:40:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:40:09.145182+0000", 2024-05-15T23:40:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.016478182000000001, 2024-05-15T23:40:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016596124, 2024-05-15T23:40:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:40:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.27292", 2024-05-15T23:40:09.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2136483304", 2024-05-15T23:40:09.174 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 99475 2024-05-15T23:40:09.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.174 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:09.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:09.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.145182+0000", 2024-05-15T23:40:09.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.145182+0000", 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.145183+0000", 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.655e-06 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.145210+0000", 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6281000000000001e-05 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.145224+0000", 2024-05-15T23:40:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4082e-05 2024-05-15T23:40:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.145230+0000", 2024-05-15T23:40:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.759e-06 2024-05-15T23:40:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.156964+0000", 2024-05-15T23:40:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011733774000000001 2024-05-15T23:40:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.157090+0000", 2024-05-15T23:40:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012596300000000001 2024-05-15T23:40:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:40:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.157228+0000", 2024-05-15T23:40:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013801300000000001 2024-05-15T23:40:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.27292.0:99468 9.8 e765/737 9:152b41aa:::benchmark_data_smithi090_166897_object99467:head v 765'6291, mlcod=765'6287)", 2024-05-15T23:40:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:40:09.148794+0000", 2024-05-15T23:40:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.012865888000000001, 2024-05-15T23:40:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013027057999999999, 2024-05-15T23:40:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-05-15T23:40:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.148794+0000", 2024-05-15T23:40:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:09.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.148794+0000", 2024-05-15T23:40:09.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:09.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.148796+0000", 2024-05-15T23:40:09.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4590000000000001e-06 2024-05-15T23:40:09.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:09.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.148821+0000", 2024-05-15T23:40:09.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4763999999999999e-05 2024-05-15T23:40:09.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:09.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.148831+0000", 2024-05-15T23:40:09.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0227000000000001e-05 2024-05-15T23:40:09.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:09.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.148839+0000", 2024-05-15T23:40:09.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6070000000000002e-06 2024-05-15T23:40:09.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:09.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.153104+0000", 2024-05-15T23:40:09.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0042650450000000003 2024-05-15T23:40:09.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.153123+0000", 2024-05-15T23:40:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8644e-05 2024-05-15T23:40:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.27292.0:99480 9.5 9:aa11ddfd:::benchmark_data_smithi090_166897_object99479:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-05-15T23:40:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:40:09.153625+0000", 2024-05-15T23:40:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0080350560000000005, 2024-05-15T23:40:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0082354520000000007, 2024-05-15T23:40:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:40:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.27292", 2024-05-15T23:40:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2136483304", 2024-05-15T23:40:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 99480 2024-05-15T23:40:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.153625+0000", 2024-05-15T23:40:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.153625+0000", 2024-05-15T23:40:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.153627+0000", 2024-05-15T23:40:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2589999999999999e-06 2024-05-15T23:40:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.153651+0000", 2024-05-15T23:40:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3890999999999999e-05 2024-05-15T23:40:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.153662+0000", 2024-05-15T23:40:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0743e-05 2024-05-15T23:40:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.153670+0000", 2024-05-15T23:40:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3869999999999995e-06 2024-05-15T23:40:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.155603+0000", 2024-05-15T23:40:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019329659999999999 2024-05-15T23:40:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.155753+0000", 2024-05-15T23:40:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150109 2024-05-15T23:40:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 7", 2024-05-15T23:40:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.155906+0000", 2024-05-15T23:40:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000153015 2024-05-15T23:40:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-05-15T23:40:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.159090+0000", 2024-05-15T23:40:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031840689999999999 2024-05-15T23:40:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.27292.0:99486 9.0 9:0b9f3843:::benchmark_data_smithi090_166897_object99485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-05-15T23:40:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:40:09.158908+0000", 2024-05-15T23:40:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.002751884, 2024-05-15T23:40:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029979540000000002, 2024-05-15T23:40:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-05-15T23:40:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.27292", 2024-05-15T23:40:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2136483304", 2024-05-15T23:40:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 99486 2024-05-15T23:40:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.158908+0000", 2024-05-15T23:40:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.158908+0000", 2024-05-15T23:40:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.158910+0000", 2024-05-15T23:40:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.249e-06 2024-05-15T23:40:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.158936+0000", 2024-05-15T23:40:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5687999999999999e-05 2024-05-15T23:40:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.158945+0000", 2024-05-15T23:40:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5950000000000005e-06 2024-05-15T23:40:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.158954+0000", 2024-05-15T23:40:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3869999999999995e-06 2024-05-15T23:40:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-05-15T23:40:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.160038+0000", 2024-05-15T23:40:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001083676 2024-05-15T23:40:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-05-15T23:40:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.160152+0000", 2024-05-15T23:40:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011423500000000001 2024-05-15T23:40:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-05-15T23:40:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.160244+0000", 2024-05-15T23:40:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2272e-05 2024-05-15T23:40:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.27292.0:99488 9.0 9.e385ed0 (undecoded) ondisk+write+known_if_redirected e765)", 2024-05-15T23:40:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:40:09.159451+0000", 2024-05-15T23:40:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0022092750000000001, 2024-05-15T23:40:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002494441, 2024-05-15T23:40:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:40:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.27292", 2024-05-15T23:40:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2136483304", 2024-05-15T23:40:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 99488 2024-05-15T23:40:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.159451+0000", 2024-05-15T23:40:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.159451+0000", 2024-05-15T23:40:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.159452+0000", 2024-05-15T23:40:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.195e-06 2024-05-15T23:40:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.159470+0000", 2024-05-15T23:40:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8346e-05 2024-05-15T23:40:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.159479+0000", 2024-05-15T23:40:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6009999999999995e-06 2024-05-15T23:40:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.159484+0000", 2024-05-15T23:40:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8600000000000001e-06 2024-05-15T23:40:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.27292.0:99489 9.0 9.10d145e0 (undecoded) ondisk+write+known_if_redirected e765)", 2024-05-15T23:40:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:40:09.160174+0000", 2024-05-15T23:40:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0014861480000000001, 2024-05-15T23:40:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001803263, 2024-05-15T23:40:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:40:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.27292", 2024-05-15T23:40:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2136483304", 2024-05-15T23:40:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 99489 2024-05-15T23:40:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.160174+0000", 2024-05-15T23:40:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.160174+0000", 2024-05-15T23:40:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.160175+0000", 2024-05-15T23:40:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.159e-06 2024-05-15T23:40:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.160192+0000", 2024-05-15T23:40:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7116e-05 2024-05-15T23:40:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.160213+0000", 2024-05-15T23:40:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0590000000000001e-05 2024-05-15T23:40:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.160218+0000", 2024-05-15T23:40:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0989999999999998e-06 2024-05-15T23:40:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.27292.0:99490 9.0 9.c67610c0 (undecoded) ondisk+write+known_if_redirected e765)", 2024-05-15T23:40:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:40:09.161383+0000", 2024-05-15T23:40:09.200 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.000276545, 2024-05-15T23:40:09.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062524399999999995, 2024-05-15T23:40:09.200 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:09.200 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:09.200 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:40:09.200 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-05-15T23:40:09.200 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.27292", 2024-05-15T23:40:09.200 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "v1:172.21.15.90:0/2136483304", 2024-05-15T23:40:09.200 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 99490 2024-05-15T23:40:09.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.200 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:09.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:09.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.161383+0000", 2024-05-15T23:40:09.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:09.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.161383+0000", 2024-05-15T23:40:09.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:09.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.161385+0000", 2024-05-15T23:40:09.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6670000000000001e-06 2024-05-15T23:40:09.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:09.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.161406+0000", 2024-05-15T23:40:09.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0619999999999999e-05 2024-05-15T23:40:09.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:09.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.161417+0000", 2024-05-15T23:40:09.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1834000000000001e-05 2024-05-15T23:40:09.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:09.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.161424+0000", 2024-05-15T23:40:09.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7390000000000002e-06 2024-05-15T23:40:09.203 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.203 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:09.203 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.203 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.27292.0:99491 9.9 e765/737)", 2024-05-15T23:40:09.203 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-05-15T23:40:09.161943+0000", 2024-05-15T23:40:09.203 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4294967295.9997168, 2024-05-15T23:40:09.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6682999999999996e-05, 2024-05-15T23:40:09.203 INFO:teuthology.orchestra.run.smithi090.stdout: "continuous": false, 2024-05-15T23:40:09.203 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-05-15T23:40:09.204 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2024-05-15T23:40:09.204 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-05-15T23:40:09.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-05-15T23:40:09.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.161943+0000", 2024-05-15T23:40:09.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-05-15T23:40:09.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.161943+0000", 2024-05-15T23:40:09.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-05-15T23:40:09.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-05-15T23:40:09.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.161944+0000", 2024-05-15T23:40:09.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5090000000000001e-06 2024-05-15T23:40:09.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-05-15T23:40:09.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.161968+0000", 2024-05-15T23:40:09.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3731999999999999e-05 2024-05-15T23:40:09.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-05-15T23:40:09.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.161980+0000", 2024-05-15T23:40:09.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1627000000000001e-05 2024-05-15T23:40:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-05-15T23:40:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-05-15T23:40:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-05-15T23:40:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-05-15T23:40:09.161988+0000", 2024-05-15T23:40:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6809999999999998e-06 2024-05-15T23:40:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-05-15T23:40:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-05-15T23:40:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-05-15T23:40:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 10 2024-05-15T23:40:09.207 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:40:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:09.208 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:09.208 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:40:09.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:40:09.309 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:40:09.339 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:09.371 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:40:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:09.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:40:09.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:40:09.528 INFO:teuthology.orchestra.run.smithi088.stdout: marking collection for removal 2024-05-15T23:40:09.528 INFO:teuthology.orchestra.run.smithi088.stdout:setting '_remove' omap key 2024-05-15T23:40:09.540 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:09.550 INFO:teuthology.orchestra.run.smithi088.stdout:finish_remove_pgs 9.6_head removing 9.6 2024-05-15T23:40:09.577 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 67 16 99933 99917 93.1767 68.625 0.00861641 0.0107307 2024-05-15T23:40:09.617 INFO:teuthology.orchestra.run.smithi088.stdout:Remove successful 2024-05-15T23:40:09.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:09 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:40:09.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:09 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:40:09.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:09 smithi090 ceph-mon[44434]: osdmap e765: 12 total, 11 up, 9 in 2024-05-15T23:40:09.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:09 smithi090 ceph-mon[44434]: pgmap v2030: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 75 MiB/s wr, 1.20k op/s; 13110/193548 objects degraded (6.774%); 15 MiB/s, 238 objects/s recovering 2024-05-15T23:40:09.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:09 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:40:09.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:09 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:40:09.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:09 smithi088 ceph-mon[44699]: osdmap e765: 12 total, 11 up, 9 in 2024-05-15T23:40:09.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:09 smithi088 ceph-mon[44699]: pgmap v2030: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 75 MiB/s wr, 1.20k op/s; 13110/193548 objects degraded (6.774%); 15 MiB/s, 238 objects/s recovering 2024-05-15T23:40:09.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:09 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:40:09.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:09 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:40:09.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:09 smithi084 ceph-mon[39784]: osdmap e765: 12 total, 11 up, 9 in 2024-05-15T23:40:09.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:09 smithi084 ceph-mon[39784]: pgmap v2030: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 5.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 75 MiB/s wr, 1.20k op/s; 13110/193548 objects degraded (6.774%); 15 MiB/s, 238 objects/s recovering 2024-05-15T23:40:10.282 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:40:10.316 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:40:10.331 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:40:10.362 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:40:10.396 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:40:10.431 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:40:10.472 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:40:10.504 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:40:10.536 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:40:10.575 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 68 16 101063 101047 92.845 70.625 0.00962387 0.0107686 2024-05-15T23:40:10.576 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:40:10.615 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:40:10.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:40:10.656 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:40:10.656 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:40:10.656 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:40:10.656 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:40:10.656 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:40:10.656 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-05-15T23:40:10.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:40:10.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:40:10.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:40:10.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:40:10.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:40:10.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:40:10.677 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:10 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4096320812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:40:10.677 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:10.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:10 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4096320812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:40:10.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:10.822 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:40:10.936 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell -n osd.5 --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.9.6.5 2024-05-15T23:40:11.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:10 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4096320812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:40:11.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.181 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/osd.5/config 2024-05-15T23:40:11.575 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 69 16 102172 102156 92.504 69.3125 0.0131372 0.0108086 2024-05-15T23:40:11.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:11 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:11 smithi088 ceph-mon[44699]: pgmap v2031: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.0 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 54 MiB/s wr, 861 op/s; 13110/196346 objects degraded (6.677%); 8.4 MiB/s, 133 objects/s recovering 2024-05-15T23:40:11.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:11 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:11 smithi084 ceph-mon[39784]: pgmap v2031: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.0 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 54 MiB/s wr, 861 op/s; 13110/196346 objects degraded (6.677%); 8.4 MiB/s, 133 objects/s recovering 2024-05-15T23:40:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:12.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:12.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:12.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:12.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:12.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:12.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:11 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:12.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:11 smithi090 ceph-mon[44434]: pgmap v2031: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.0 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 54 MiB/s wr, 861 op/s; 13110/196346 objects degraded (6.677%); 8.4 MiB/s, 133 objects/s recovering 2024-05-15T23:40:12.567 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:12 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 13110/196346 objects degraded (6.677%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:12.575 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 70 16 103328 103312 92.2146 72.25 0.0128822 0.0108423 2024-05-15T23:40:12.632 INFO:teuthology.orchestra.run.smithi088.stderr:Creating an OSD daemon form without an OSD FSID value 2024-05-15T23:40:12.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:12 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 13110/196346 objects degraded (6.677%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:13.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:12 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 13110/196346 objects degraded (6.677%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:13.536 INFO:teuthology.orchestra.run.smithi088.stdout:get_pg_num_history pg_num_history pg_num_history(e749 pg_nums {1={14=1},8={654=16},9={737=16}} deleted_pools 735,8) 2024-05-15T23:40:13.536 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pgid 9.6 2024-05-15T23:40:13.546 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:600043ad:::benchmark_data_smithi090_166897_object41622:head# 2024-05-15T23:40:13.546 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.547 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6002fc21:::benchmark_data_smithi090_166897_object14756:head# 2024-05-15T23:40:13.547 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.547 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60038712:::benchmark_data_smithi090_166897_object2532:head# 2024-05-15T23:40:13.547 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.547 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6003a46f:::benchmark_data_smithi090_166897_object23915:head# 2024-05-15T23:40:13.547 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.548 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60043b15:::benchmark_data_smithi090_166897_object20640:head# 2024-05-15T23:40:13.548 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.548 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:600614fe:::benchmark_data_smithi090_166897_object65594:head# 2024-05-15T23:40:13.548 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.549 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:600692de:::benchmark_data_smithi090_166897_object25263:head# 2024-05-15T23:40:13.549 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.549 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6006f4c8:::benchmark_data_smithi090_166897_object62680:head# 2024-05-15T23:40:13.549 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.549 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60080f34:::benchmark_data_smithi090_166897_object10663:head# 2024-05-15T23:40:13.549 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.550 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6008d99a:::benchmark_data_smithi090_166897_object35902:head# 2024-05-15T23:40:13.551 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.551 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6008f364:::benchmark_data_smithi090_166897_object64021:head# 2024-05-15T23:40:13.551 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.551 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60094baf:::benchmark_data_smithi090_166897_object21769:head# 2024-05-15T23:40:13.551 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.551 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6009dfe8:::benchmark_data_smithi090_166897_object17006:head# 2024-05-15T23:40:13.551 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.551 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6009e268:::benchmark_data_smithi090_166897_object60612:head# 2024-05-15T23:40:13.551 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.551 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:600adbfc:::benchmark_data_smithi090_166897_object45954:head# 2024-05-15T23:40:13.551 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.556 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:600b1509:::benchmark_data_smithi090_166897_object19152:head# 2024-05-15T23:40:13.556 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.556 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:600b8fe2:::benchmark_data_smithi090_166897_object53053:head# 2024-05-15T23:40:13.556 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.557 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:600c210f:::benchmark_data_smithi090_166897_object11646:head# 2024-05-15T23:40:13.557 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.557 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:600c5f3f:::benchmark_data_smithi090_166897_object68521:head# 2024-05-15T23:40:13.557 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.557 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:600da86d:::benchmark_data_smithi090_166897_object41617:head# 2024-05-15T23:40:13.557 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.557 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:600dd719:::benchmark_data_smithi090_166897_object63737:head# 2024-05-15T23:40:13.557 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.557 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:600e66ba:::benchmark_data_smithi090_166897_object67065:head# 2024-05-15T23:40:13.557 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.557 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:600fdd65:::benchmark_data_smithi090_166897_object12014:head# 2024-05-15T23:40:13.557 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.557 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60108bb6:::benchmark_data_smithi090_166897_object12150:head# 2024-05-15T23:40:13.557 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.557 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6012dd58:::benchmark_data_smithi090_166897_object48212:head# 2024-05-15T23:40:13.558 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.558 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6015b4db:::benchmark_data_smithi090_166897_object2945:head# 2024-05-15T23:40:13.558 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.558 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6016a135:::benchmark_data_smithi090_166897_object12886:head# 2024-05-15T23:40:13.558 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.558 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60176914:::benchmark_data_smithi090_166897_object22474:head# 2024-05-15T23:40:13.558 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.558 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6019ea65:::benchmark_data_smithi090_166897_object65965:head# 2024-05-15T23:40:13.558 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.558 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:601c1b3a:::benchmark_data_smithi090_166897_object29401:head# 2024-05-15T23:40:13.558 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.558 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:601c848b:::benchmark_data_smithi090_166897_object11387:head# 2024-05-15T23:40:13.558 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.558 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:601d496b:::benchmark_data_smithi090_166897_object27716:head# 2024-05-15T23:40:13.559 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.559 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:601d898b:::benchmark_data_smithi090_166897_object22708:head# 2024-05-15T23:40:13.559 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.560 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:601ff967:::benchmark_data_smithi090_166897_object24763:head# 2024-05-15T23:40:13.560 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.561 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60204340:::benchmark_data_smithi090_166897_object67063:head# 2024-05-15T23:40:13.561 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.561 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60225d83:::benchmark_data_smithi090_166897_object39597:head# 2024-05-15T23:40:13.561 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.561 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6023c1e9:::benchmark_data_smithi090_166897_object70715:head# 2024-05-15T23:40:13.561 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.561 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6023c32f:::benchmark_data_smithi090_166897_object65783:head# 2024-05-15T23:40:13.562 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.563 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:602701cc:::benchmark_data_smithi090_166897_object26430:head# 2024-05-15T23:40:13.563 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.563 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:602ab461:::benchmark_data_smithi090_166897_object19908:head# 2024-05-15T23:40:13.563 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.563 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:602df5f4:::benchmark_data_smithi090_166897_object2500:head# 2024-05-15T23:40:13.563 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.564 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:602e9aea:::benchmark_data_smithi090_166897_object70129:head# 2024-05-15T23:40:13.564 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.564 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:602f0a43:::benchmark_data_smithi090_166897_object33762:head# 2024-05-15T23:40:13.564 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.565 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60309446:::benchmark_data_smithi090_166897_object12603:head# 2024-05-15T23:40:13.565 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.565 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6032eb9e:::benchmark_data_smithi090_166897_object2791:head# 2024-05-15T23:40:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60333bfe:::benchmark_data_smithi090_166897_object51378:head# 2024-05-15T23:40:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6034f97c:::benchmark_data_smithi090_166897_object62263:head# 2024-05-15T23:40:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60354e21:::benchmark_data_smithi090_166897_object44130:head# 2024-05-15T23:40:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6035ebcd:::benchmark_data_smithi090_166897_object1943:head# 2024-05-15T23:40:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60366d3d:::benchmark_data_smithi090_166897_object31798:head# 2024-05-15T23:40:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6037d39a:::benchmark_data_smithi090_166897_object259:head# 2024-05-15T23:40:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.568 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:603abae4:::benchmark_data_smithi090_166897_object62302:head# 2024-05-15T23:40:13.568 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.568 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:603b207b:::benchmark_data_smithi090_166897_object24784:head# 2024-05-15T23:40:13.568 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.568 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:603bb70b:::benchmark_data_smithi090_166897_object49264:head# 2024-05-15T23:40:13.568 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.568 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:603e2699:::benchmark_data_smithi090_166897_object14818:head# 2024-05-15T23:40:13.568 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.568 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:603f4336:::benchmark_data_smithi090_166897_object17971:head# 2024-05-15T23:40:13.568 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.568 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60402e8f:::benchmark_data_smithi090_166897_object56787:head# 2024-05-15T23:40:13.569 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.569 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60430fb8:::benchmark_data_smithi090_166897_object34672:head# 2024-05-15T23:40:13.569 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.570 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6043ecb8:::benchmark_data_smithi090_166897_object49075:head# 2024-05-15T23:40:13.570 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.570 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6044d078:::benchmark_data_smithi090_166897_object17932:head# 2024-05-15T23:40:13.570 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.570 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6045181e:::benchmark_data_smithi090_166897_object50981:head# 2024-05-15T23:40:13.570 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.570 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:604533e4:::benchmark_data_smithi090_166897_object52933:head# 2024-05-15T23:40:13.570 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.570 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:604576ca:::benchmark_data_smithi090_166897_object27631:head# 2024-05-15T23:40:13.570 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.571 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6046590d:::benchmark_data_smithi090_166897_object62222:head# 2024-05-15T23:40:13.571 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.574 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60465987:::benchmark_data_smithi090_166897_object57365:head# 2024-05-15T23:40:13.574 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.575 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:604704c3:::benchmark_data_smithi090_166897_object34658:head# 2024-05-15T23:40:13.575 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.575 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:604857f8:::benchmark_data_smithi090_166897_object69641:head# 2024-05-15T23:40:13.575 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.575 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6048d91f:::benchmark_data_smithi090_166897_object40349:head# 2024-05-15T23:40:13.575 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.575 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6049e87c:::benchmark_data_smithi090_166897_object24095:head# 2024-05-15T23:40:13.575 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.575 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:604d56fc:::benchmark_data_smithi090_166897_object22301:head# 2024-05-15T23:40:13.575 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.575 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:604ee3d9:::benchmark_data_smithi090_166897_object70120:head# 2024-05-15T23:40:13.575 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.576 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 71 16 104491 104475 91.9396 72.6875 0.00994912 0.0108747 2024-05-15T23:40:13.579 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:604eec0e:::benchmark_data_smithi090_166897_object25322:head# 2024-05-15T23:40:13.580 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.580 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:604ef649:::benchmark_data_smithi090_166897_object32775:head# 2024-05-15T23:40:13.580 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.580 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:604fc945:::benchmark_data_smithi090_166897_object34756:head# 2024-05-15T23:40:13.580 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.580 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:604fd567:::benchmark_data_smithi090_166897_object33388:head# 2024-05-15T23:40:13.580 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.580 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:605060c5:::benchmark_data_smithi090_166897_object12459:head# 2024-05-15T23:40:13.580 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.580 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6050a5c2:::benchmark_data_smithi090_166897_object42219:head# 2024-05-15T23:40:13.580 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.580 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6050f4a5:::benchmark_data_smithi090_166897_object51565:head# 2024-05-15T23:40:13.580 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.581 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60511293:::benchmark_data_smithi090_166897_object55513:head# 2024-05-15T23:40:13.581 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.581 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:605186e8:::benchmark_data_smithi090_166897_object46617:head# 2024-05-15T23:40:13.581 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.581 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60530427:::benchmark_data_smithi090_166897_object22290:head# 2024-05-15T23:40:13.581 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.581 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60533bea:::benchmark_data_smithi090_166897_object1795:head# 2024-05-15T23:40:13.581 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.581 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6053454a:::benchmark_data_smithi090_166897_object40760:head# 2024-05-15T23:40:13.581 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.581 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:605378ce:::benchmark_data_smithi090_166897_object20252:head# 2024-05-15T23:40:13.581 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.581 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60555685:::benchmark_data_smithi090_166897_object17558:head# 2024-05-15T23:40:13.581 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.581 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6057bd36:::benchmark_data_smithi090_166897_object55190:head# 2024-05-15T23:40:13.582 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.582 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60581384:::benchmark_data_smithi090_166897_object52320:head# 2024-05-15T23:40:13.582 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.582 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60594be5:::benchmark_data_smithi090_166897_object50795:head# 2024-05-15T23:40:13.582 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.582 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:605a0465:::benchmark_data_smithi090_166897_object13697:head# 2024-05-15T23:40:13.582 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.582 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:605a1df2:::benchmark_data_smithi090_166897_object24376:head# 2024-05-15T23:40:13.582 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.582 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:605d5db4:::benchmark_data_smithi090_166897_object47058:head# 2024-05-15T23:40:13.582 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.582 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:605dbbcc:::benchmark_data_smithi090_166897_object47069:head# 2024-05-15T23:40:13.582 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.583 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:605ebc40:::benchmark_data_smithi090_166897_object3010:head# 2024-05-15T23:40:13.583 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.586 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:605edd0a:::benchmark_data_smithi090_166897_object29253:head# 2024-05-15T23:40:13.586 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.586 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60613041:::benchmark_data_smithi090_166897_object34679:head# 2024-05-15T23:40:13.586 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.586 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60628c79:::benchmark_data_smithi090_166897_object42877:head# 2024-05-15T23:40:13.586 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.586 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60628ed3:::benchmark_data_smithi090_166897_object10045:head# 2024-05-15T23:40:13.586 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.586 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6063458a:::benchmark_data_smithi090_166897_object65703:head# 2024-05-15T23:40:13.587 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.587 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60652f06:::benchmark_data_smithi090_166897_object58556:head# 2024-05-15T23:40:13.587 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.587 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:606581cc:::benchmark_data_smithi090_166897_object11251:head# 2024-05-15T23:40:13.587 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.587 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6066f396:::benchmark_data_smithi090_166897_object54555:head# 2024-05-15T23:40:13.587 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.587 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:606700cf:::benchmark_data_smithi090_166897_object54066:head# 2024-05-15T23:40:13.587 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.587 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60672d81:::benchmark_data_smithi090_166897_object28133:head# 2024-05-15T23:40:13.587 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.587 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60680bfe:::benchmark_data_smithi090_166897_object19890:head# 2024-05-15T23:40:13.587 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.587 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:606865e3:::benchmark_data_smithi090_166897_object16107:head# 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:606ae3d9:::benchmark_data_smithi090_166897_object18200:head# 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:606b4616:::benchmark_data_smithi090_166897_object54015:head# 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:606c39c3:::benchmark_data_smithi090_166897_object38428:head# 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:606cce6b:::benchmark_data_smithi090_166897_object5528:head# 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:606d0342:::benchmark_data_smithi090_166897_object58228:head# 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:606d26d9:::benchmark_data_smithi090_166897_object1585:head# 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:606f4c31:::benchmark_data_smithi090_166897_object54953:head# 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.588 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:606f6cf4:::benchmark_data_smithi090_166897_object56779:head# 2024-05-15T23:40:13.589 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.589 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:606f8dc9:::benchmark_data_smithi090_166897_object199:head# 2024-05-15T23:40:13.589 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.589 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60700638:::benchmark_data_smithi090_166897_object57047:head# 2024-05-15T23:40:13.589 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.589 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:607108d4:::benchmark_data_smithi090_166897_object55820:head# 2024-05-15T23:40:13.589 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.589 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6075b927:::benchmark_data_smithi090_166897_object32403:head# 2024-05-15T23:40:13.589 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.595 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:607659fa:::benchmark_data_smithi090_166897_object3133:head# 2024-05-15T23:40:13.596 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.596 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6076f32e:::benchmark_data_smithi090_166897_object35823:head# 2024-05-15T23:40:13.596 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.596 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:607776bf:::benchmark_data_smithi090_166897_object1374:head# 2024-05-15T23:40:13.596 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.596 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6078d3cb:::benchmark_data_smithi090_166897_object18261:head# 2024-05-15T23:40:13.596 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.596 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6079a984:::benchmark_data_smithi090_166897_object31356:head# 2024-05-15T23:40:13.596 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.596 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6079bf95:::benchmark_data_smithi090_166897_object66331:head# 2024-05-15T23:40:13.596 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.596 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:607a5284:::benchmark_data_smithi090_166897_object11218:head# 2024-05-15T23:40:13.596 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.596 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:607c5316:::benchmark_data_smithi090_166897_object69532:head# 2024-05-15T23:40:13.597 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.597 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:607ca04d:::benchmark_data_smithi090_166897_object6821:head# 2024-05-15T23:40:13.597 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.597 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:607d3518:::benchmark_data_smithi090_166897_object8681:head# 2024-05-15T23:40:13.597 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.597 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:607f510b:::benchmark_data_smithi090_166897_object34420:head# 2024-05-15T23:40:13.597 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.597 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:608120b2:::benchmark_data_smithi090_166897_object45652:head# 2024-05-15T23:40:13.597 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.597 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60813bcb:::benchmark_data_smithi090_166897_object8413:head# 2024-05-15T23:40:13.597 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.597 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6081db8e:::benchmark_data_smithi090_166897_object44163:head# 2024-05-15T23:40:13.597 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.597 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6083f21f:::benchmark_data_smithi090_166897_object64074:head# 2024-05-15T23:40:13.597 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.598 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6085311b:::benchmark_data_smithi090_166897_object57937:head# 2024-05-15T23:40:13.598 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.598 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6085dd7b:::benchmark_data_smithi090_166897_object54938:head# 2024-05-15T23:40:13.598 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.598 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60862f57:::benchmark_data_smithi090_166897_object63431:head# 2024-05-15T23:40:13.598 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.598 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:608714b0:::benchmark_data_smithi090_166897_object27622:head# 2024-05-15T23:40:13.598 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.598 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6088845b:::benchmark_data_smithi090_166897_object4115:head# 2024-05-15T23:40:13.598 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.598 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6088f749:::benchmark_data_smithi090_166897_object70487:head# 2024-05-15T23:40:13.598 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.598 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:608dbbec:::benchmark_data_smithi090_166897_object57293:head# 2024-05-15T23:40:13.598 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:608e9473:::benchmark_data_smithi090_166897_object6442:head# 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:608feaff:::benchmark_data_smithi090_166897_object66616:head# 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6090f2ca:::benchmark_data_smithi090_166897_object36605:head# 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:609148d4:::benchmark_data_smithi090_166897_object60759:head# 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6091c6d6:::benchmark_data_smithi090_166897_object35634:head# 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6092405f:::benchmark_data_smithi090_166897_object62665:head# 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60939c83:::benchmark_data_smithi090_166897_object23503:head# 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6094e863:::benchmark_data_smithi090_166897_object35702:head# 2024-05-15T23:40:13.599 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6098b814:::benchmark_data_smithi090_166897_object53954:head# 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:609c3767:::benchmark_data_smithi090_166897_object56853:head# 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:609c838f:::benchmark_data_smithi090_166897_object32798:head# 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:609dd238:::benchmark_data_smithi090_166897_object63637:head# 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:609edc23:::benchmark_data_smithi090_166897_object36882:head# 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:609f82b8:::benchmark_data_smithi090_166897_object60892:head# 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60a03328:::benchmark_data_smithi090_166897_object69914:head# 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60a0cb2b:::benchmark_data_smithi090_166897_object2578:head# 2024-05-15T23:40:13.600 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.601 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60a0d24d:::benchmark_data_smithi090_166897_object71010:head# 2024-05-15T23:40:13.601 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.616 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60a1f9d1:::benchmark_data_smithi090_166897_object19624:head# 2024-05-15T23:40:13.616 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.616 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60a24ee9:::benchmark_data_smithi090_166897_object52501:head# 2024-05-15T23:40:13.616 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.617 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60a25b4e:::benchmark_data_smithi090_166897_object34800:head# 2024-05-15T23:40:13.617 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.617 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60a2e915:::benchmark_data_smithi090_166897_object58624:head# 2024-05-15T23:40:13.617 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.617 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60a33113:::benchmark_data_smithi090_166897_object11413:head# 2024-05-15T23:40:13.617 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.617 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60a475f7:::benchmark_data_smithi090_166897_object29334:head# 2024-05-15T23:40:13.617 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.617 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60a492cb:::benchmark_data_smithi090_166897_object378:head# 2024-05-15T23:40:13.617 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.617 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60a4bcc0:::benchmark_data_smithi090_166897_object14626:head# 2024-05-15T23:40:13.617 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.617 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60a69b41:::benchmark_data_smithi090_166897_object8994:head# 2024-05-15T23:40:13.617 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.618 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60a7ade0:::benchmark_data_smithi090_166897_object58208:head# 2024-05-15T23:40:13.618 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.618 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60a7d8e7:::benchmark_data_smithi090_166897_object67081:head# 2024-05-15T23:40:13.618 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.618 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60a96002:::benchmark_data_smithi090_166897_object2974:head# 2024-05-15T23:40:13.618 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.618 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60aa21c7:::benchmark_data_smithi090_166897_object23892:head# 2024-05-15T23:40:13.618 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.618 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60aaf5ae:::benchmark_data_smithi090_166897_object38870:head# 2024-05-15T23:40:13.618 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.618 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60abcb98:::benchmark_data_smithi090_166897_object41995:head# 2024-05-15T23:40:13.618 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.618 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60acd683:::benchmark_data_smithi090_166897_object27510:head# 2024-05-15T23:40:13.618 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.619 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ad114a:::benchmark_data_smithi090_166897_object70184:head# 2024-05-15T23:40:13.619 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.619 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ad943e:::benchmark_data_smithi090_166897_object11994:head# 2024-05-15T23:40:13.619 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.619 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ae2489:::benchmark_data_smithi090_166897_object69022:head# 2024-05-15T23:40:13.619 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.619 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ae3a70:::benchmark_data_smithi090_166897_object43833:head# 2024-05-15T23:40:13.619 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.619 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60af6756:::benchmark_data_smithi090_166897_object68107:head# 2024-05-15T23:40:13.619 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.619 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60afb20b:::benchmark_data_smithi090_166897_object35919:head# 2024-05-15T23:40:13.619 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.619 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60b0c3fe:::benchmark_data_smithi090_166897_object68899:head# 2024-05-15T23:40:13.619 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.619 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60b22d4b:::benchmark_data_smithi090_166897_object51316:head# 2024-05-15T23:40:13.620 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.620 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60b2a636:::benchmark_data_smithi090_166897_object13844:head# 2024-05-15T23:40:13.620 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.620 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60b2d7ae:::benchmark_data_smithi090_166897_object30800:head# 2024-05-15T23:40:13.620 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.620 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60b44999:::benchmark_data_smithi090_166897_object29556:head# 2024-05-15T23:40:13.620 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.620 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60b56593:::benchmark_data_smithi090_166897_object31381:head# 2024-05-15T23:40:13.620 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.620 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60b5956f:::benchmark_data_smithi090_166897_object19212:head# 2024-05-15T23:40:13.620 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.620 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60b86990:::benchmark_data_smithi090_166897_object14200:head# 2024-05-15T23:40:13.620 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.620 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60b953da:::benchmark_data_smithi090_166897_object7012:head# 2024-05-15T23:40:13.621 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.621 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60b984e6:::benchmark_data_smithi090_166897_object17290:head# 2024-05-15T23:40:13.621 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.621 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60b99f29:::benchmark_data_smithi090_166897_object49757:head# 2024-05-15T23:40:13.621 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.621 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60b9b28c:::benchmark_data_smithi090_166897_object41145:head# 2024-05-15T23:40:13.621 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.621 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ba6f11:::benchmark_data_smithi090_166897_object53593:head# 2024-05-15T23:40:13.621 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.621 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60bc0fa3:::benchmark_data_smithi090_166897_object41702:head# 2024-05-15T23:40:13.621 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.621 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60bc50f6:::benchmark_data_smithi090_166897_object61678:head# 2024-05-15T23:40:13.621 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.621 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60bc57f6:::benchmark_data_smithi090_166897_object20335:head# 2024-05-15T23:40:13.621 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.622 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60bc8c96:::benchmark_data_smithi090_166897_object18078:head# 2024-05-15T23:40:13.622 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.622 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60bca1bf:::benchmark_data_smithi090_166897_object48225:head# 2024-05-15T23:40:13.622 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.622 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60bdf74a:::benchmark_data_smithi090_166897_object16659:head# 2024-05-15T23:40:13.622 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.622 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60bf5313:::benchmark_data_smithi090_166897_object27481:head# 2024-05-15T23:40:13.622 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.622 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60c2d72a:::benchmark_data_smithi090_166897_object44789:head# 2024-05-15T23:40:13.622 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.622 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60c3f2d5:::benchmark_data_smithi090_166897_object34506:head# 2024-05-15T23:40:13.622 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.622 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60c50864:::benchmark_data_smithi090_166897_object5018:head# 2024-05-15T23:40:13.622 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60c69f4c:::benchmark_data_smithi090_166897_object13636:head# 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60c8298c:::benchmark_data_smithi090_166897_object55750:head# 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60c97135:::benchmark_data_smithi090_166897_object36387:head# 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60c9b450:::benchmark_data_smithi090_166897_object50571:head# 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60c9f432:::benchmark_data_smithi090_166897_object60760:head# 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ca0682:::benchmark_data_smithi090_166897_object47831:head# 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60cb3d4d:::benchmark_data_smithi090_166897_object29913:head# 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60cc8161:::benchmark_data_smithi090_166897_object22392:head# 2024-05-15T23:40:13.623 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ce7b6f:::benchmark_data_smithi090_166897_object54009:head# 2024-05-15T23:40:13.624 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ce8514:::benchmark_data_smithi090_166897_object14247:head# 2024-05-15T23:40:13.624 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ceaeed:::benchmark_data_smithi090_166897_object69851:head# 2024-05-15T23:40:13.624 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60cef96b:::benchmark_data_smithi090_166897_object13437:head# 2024-05-15T23:40:13.624 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60cfbce5:::benchmark_data_smithi090_166897_object45583:head# 2024-05-15T23:40:13.624 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60d123bb:::benchmark_data_smithi090_166897_object20123:head# 2024-05-15T23:40:13.624 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60d15757:::benchmark_data_smithi090_166897_object56780:head# 2024-05-15T23:40:13.624 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60d3ae15:::benchmark_data_smithi090_166897_object13093:head# 2024-05-15T23:40:13.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.625 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60d3f759:::benchmark_data_smithi090_166897_object3310:head# 2024-05-15T23:40:13.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.625 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60d48877:::benchmark_data_smithi090_166897_object55347:head# 2024-05-15T23:40:13.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.625 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60d59c84:::benchmark_data_smithi090_166897_object48366:head# 2024-05-15T23:40:13.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.625 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60d65099:::benchmark_data_smithi090_166897_object41235:head# 2024-05-15T23:40:13.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.625 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60d6e162:::benchmark_data_smithi090_166897_object34544:head# 2024-05-15T23:40:13.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.625 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60d8fef7:::benchmark_data_smithi090_166897_object17287:head# 2024-05-15T23:40:13.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.625 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60d91924:::benchmark_data_smithi090_166897_object61029:head# 2024-05-15T23:40:13.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.626 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60da6110:::benchmark_data_smithi090_166897_object3178:head# 2024-05-15T23:40:13.626 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.626 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60dba8dc:::benchmark_data_smithi090_166897_object45195:head# 2024-05-15T23:40:13.626 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.626 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60dc310d:::benchmark_data_smithi090_166897_object34673:head# 2024-05-15T23:40:13.626 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.626 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60dc34ee:::benchmark_data_smithi090_166897_object65391:head# 2024-05-15T23:40:13.626 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.626 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60dd01bc:::benchmark_data_smithi090_166897_object23976:head# 2024-05-15T23:40:13.626 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.626 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60dee7ef:::benchmark_data_smithi090_166897_object67819:head# 2024-05-15T23:40:13.626 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.626 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60df1a4c:::benchmark_data_smithi090_166897_object60038:head# 2024-05-15T23:40:13.626 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60e17703:::benchmark_data_smithi090_166897_object910:head# 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60e1f8ac:::benchmark_data_smithi090_166897_object18180:head# 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60e3383c:::benchmark_data_smithi090_166897_object34256:head# 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60e3eff5:::benchmark_data_smithi090_166897_object6252:head# 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60e44ce6:::benchmark_data_smithi090_166897_object28614:head# 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60e481fa:::benchmark_data_smithi090_166897_object29873:head# 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60e59df4:::benchmark_data_smithi090_166897_object21826:head# 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60e59fc8:::benchmark_data_smithi090_166897_object58991:head# 2024-05-15T23:40:13.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.628 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60e5d6f6:::benchmark_data_smithi090_166897_object52978:head# 2024-05-15T23:40:13.628 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.628 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60e65557:::benchmark_data_smithi090_166897_object30589:head# 2024-05-15T23:40:13.628 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.628 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60e977f1:::benchmark_data_smithi090_166897_object22214:head# 2024-05-15T23:40:13.628 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.628 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ea09f3:::benchmark_data_smithi090_166897_object19681:head# 2024-05-15T23:40:13.628 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.628 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ea7772:::benchmark_data_smithi090_166897_object33977:head# 2024-05-15T23:40:13.628 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.628 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ebb1b7:::benchmark_data_smithi090_166897_object16602:head# 2024-05-15T23:40:13.628 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.628 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ec25fb:::benchmark_data_smithi090_166897_object58259:head# 2024-05-15T23:40:13.628 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.641 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ed4832:::benchmark_data_smithi090_166897_object45073:head# 2024-05-15T23:40:13.641 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.641 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60edd713:::benchmark_data_smithi090_166897_object69947:head# 2024-05-15T23:40:13.641 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.642 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ef17e5:::benchmark_data_smithi090_166897_object56638:head# 2024-05-15T23:40:13.642 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.642 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ef32ab:::benchmark_data_smithi090_166897_object66797:head# 2024-05-15T23:40:13.642 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.642 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60f0c63e:::benchmark_data_smithi090_166897_object38688:head# 2024-05-15T23:40:13.642 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.642 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60f16281:::benchmark_data_smithi090_166897_object9076:head# 2024-05-15T23:40:13.642 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.642 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60f1dd2f:::benchmark_data_smithi090_166897_object20054:head# 2024-05-15T23:40:13.642 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.642 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60f22908:::benchmark_data_smithi090_166897_object26193:head# 2024-05-15T23:40:13.642 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.642 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60f29f9a:::benchmark_data_smithi090_166897_object36279:head# 2024-05-15T23:40:13.642 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.643 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60f3851b:::benchmark_data_smithi090_166897_object45190:head# 2024-05-15T23:40:13.643 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.643 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60f5ab35:::benchmark_data_smithi090_166897_object47957:head# 2024-05-15T23:40:13.643 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.643 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60f5e8b7:::benchmark_data_smithi090_166897_object17303:head# 2024-05-15T23:40:13.643 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.643 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60f6d8b8:::benchmark_data_smithi090_166897_object3409:head# 2024-05-15T23:40:13.643 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.643 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60f73f41:::benchmark_data_smithi090_166897_object30222:head# 2024-05-15T23:40:13.643 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.643 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60f83198:::benchmark_data_smithi090_166897_object57744:head# 2024-05-15T23:40:13.643 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.643 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60f891fc:::benchmark_data_smithi090_166897_object23152:head# 2024-05-15T23:40:13.643 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.643 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60fb958e:::benchmark_data_smithi090_166897_object26942:head# 2024-05-15T23:40:13.644 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.644 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60fbc10e:::benchmark_data_smithi090_166897_object52092:head# 2024-05-15T23:40:13.644 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.644 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60fc4e8b:::benchmark_data_smithi090_166897_object9146:head# 2024-05-15T23:40:13.644 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.644 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60fdcbe4:::benchmark_data_smithi090_166897_object48339:head# 2024-05-15T23:40:13.644 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.644 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60fe44fb:::benchmark_data_smithi090_166897_object27551:head# 2024-05-15T23:40:13.644 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.644 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:60ffbde9:::benchmark_data_smithi090_166897_object29144:head# 2024-05-15T23:40:13.644 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.644 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61008425:::benchmark_data_smithi090_166897_object56542:head# 2024-05-15T23:40:13.644 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.644 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6104c8e5:::benchmark_data_smithi090_166897_object21115:head# 2024-05-15T23:40:13.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.645 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61054929:::benchmark_data_smithi090_166897_object54691:head# 2024-05-15T23:40:13.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.645 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6107f9af:::benchmark_data_smithi090_166897_object49513:head# 2024-05-15T23:40:13.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.645 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:610b6627:::benchmark_data_smithi090_166897_object30748:head# 2024-05-15T23:40:13.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.645 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:610c35ea:::benchmark_data_smithi090_166897_object12270:head# 2024-05-15T23:40:13.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.645 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:610c80d0:::benchmark_data_smithi090_166897_object9719:head# 2024-05-15T23:40:13.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.645 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:610ebc92:::benchmark_data_smithi090_166897_object3034:head# 2024-05-15T23:40:13.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.645 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:610f1c36:::benchmark_data_smithi090_166897_object22625:head# 2024-05-15T23:40:13.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.646 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:610f8873:::benchmark_data_smithi090_166897_object56207:head# 2024-05-15T23:40:13.646 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.646 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:610fe05e:::benchmark_data_smithi090_166897_object49847:head# 2024-05-15T23:40:13.646 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.646 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6111741f:::benchmark_data_smithi090_166897_object54809:head# 2024-05-15T23:40:13.646 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.646 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6111edae:::benchmark_data_smithi090_166897_object30668:head# 2024-05-15T23:40:13.646 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.646 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6111f85f:::benchmark_data_smithi090_166897_object26854:head# 2024-05-15T23:40:13.646 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.646 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:611237bb:::benchmark_data_smithi090_166897_object69751:head# 2024-05-15T23:40:13.646 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.646 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61142088:::benchmark_data_smithi090_166897_object23039:head# 2024-05-15T23:40:13.650 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.650 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61143cb3:::benchmark_data_smithi090_166897_object58723:head# 2024-05-15T23:40:13.650 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.650 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6114a873:::benchmark_data_smithi090_166897_object44492:head# 2024-05-15T23:40:13.650 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.650 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:611620dd:::benchmark_data_smithi090_166897_object30383:head# 2024-05-15T23:40:13.650 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.650 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6116acce:::benchmark_data_smithi090_166897_object4400:head# 2024-05-15T23:40:13.650 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.651 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6117d233:::benchmark_data_smithi090_166897_object19575:head# 2024-05-15T23:40:13.651 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.651 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:611818f9:::benchmark_data_smithi090_166897_object13991:head# 2024-05-15T23:40:13.651 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.651 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:611844b0:::benchmark_data_smithi090_166897_object63224:head# 2024-05-15T23:40:13.651 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.651 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6119a688:::benchmark_data_smithi090_166897_object11773:head# 2024-05-15T23:40:13.651 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.651 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:611be89a:::benchmark_data_smithi090_166897_object38280:head# 2024-05-15T23:40:13.651 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.651 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:611c8631:::benchmark_data_smithi090_166897_object48070:head# 2024-05-15T23:40:13.651 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.651 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:611d83e3:::benchmark_data_smithi090_166897_object55156:head# 2024-05-15T23:40:13.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.652 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:611db2ed:::benchmark_data_smithi090_166897_object13646:head# 2024-05-15T23:40:13.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.652 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:611e595a:::benchmark_data_smithi090_166897_object50387:head# 2024-05-15T23:40:13.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.652 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:611f1b12:::benchmark_data_smithi090_166897_object63691:head# 2024-05-15T23:40:13.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.652 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:612029f8:::benchmark_data_smithi090_166897_object8772:head# 2024-05-15T23:40:13.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.652 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6120b255:::benchmark_data_smithi090_166897_object35719:head# 2024-05-15T23:40:13.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.652 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61213fb6:::benchmark_data_smithi090_166897_object15564:head# 2024-05-15T23:40:13.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.652 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61221aaf:::benchmark_data_smithi090_166897_object27768:head# 2024-05-15T23:40:13.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61232602:::benchmark_data_smithi090_166897_object33958:head# 2024-05-15T23:40:13.653 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61235fb7:::benchmark_data_smithi090_166897_object23775:head# 2024-05-15T23:40:13.653 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6123791d:::benchmark_data_smithi090_166897_object62811:head# 2024-05-15T23:40:13.653 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61243113:::benchmark_data_smithi090_166897_object66757:head# 2024-05-15T23:40:13.653 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6125de85:::benchmark_data_smithi090_166897_object61713:head# 2024-05-15T23:40:13.653 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61289494:::benchmark_data_smithi090_166897_object855:head# 2024-05-15T23:40:13.653 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:612969e5:::benchmark_data_smithi090_166897_object38685:head# 2024-05-15T23:40:13.653 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:612a0960:::benchmark_data_smithi090_166897_object10491:head# 2024-05-15T23:40:13.654 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.654 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:612a7dc9:::benchmark_data_smithi090_166897_object12414:head# 2024-05-15T23:40:13.654 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.654 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:612ab5f7:::benchmark_data_smithi090_166897_object55361:head# 2024-05-15T23:40:13.654 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.654 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:612b955d:::benchmark_data_smithi090_166897_object22653:head# 2024-05-15T23:40:13.654 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.654 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:612be218:::benchmark_data_smithi090_166897_object53835:head# 2024-05-15T23:40:13.654 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.654 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:612c55b3:::benchmark_data_smithi090_166897_object32045:head# 2024-05-15T23:40:13.654 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.654 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:612d854b:::benchmark_data_smithi090_166897_object65231:head# 2024-05-15T23:40:13.654 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.654 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:612d8c53:::benchmark_data_smithi090_166897_object53230:head# 2024-05-15T23:40:13.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.655 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:612e9488:::benchmark_data_smithi090_166897_object20254:head# 2024-05-15T23:40:13.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.655 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:612ebfb6:::benchmark_data_smithi090_166897_object48516:head# 2024-05-15T23:40:13.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.655 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:613035bf:::benchmark_data_smithi090_166897_object70671:head# 2024-05-15T23:40:13.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.655 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6130f03a:::benchmark_data_smithi090_166897_object17910:head# 2024-05-15T23:40:13.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.655 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61316cec:::benchmark_data_smithi090_166897_object52932:head# 2024-05-15T23:40:13.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.655 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61330365:::benchmark_data_smithi090_166897_object17843:head# 2024-05-15T23:40:13.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.655 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6135274c:::benchmark_data_smithi090_166897_object10448:head# 2024-05-15T23:40:13.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:613572a5:::benchmark_data_smithi090_166897_object27350:head# 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6135994a:::benchmark_data_smithi090_166897_object37116:head# 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:613635f5:::benchmark_data_smithi090_166897_object8983:head# 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6136800c:::benchmark_data_smithi090_166897_object59570:head# 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:613684fe:::benchmark_data_smithi090_166897_object7600:head# 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6136ec69:::benchmark_data_smithi090_166897_object40150:head# 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:613afcb6:::benchmark_data_smithi090_166897_object25006:head# 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:613b451b:::benchmark_data_smithi090_166897_object52016:head# 2024-05-15T23:40:13.656 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:613b5088:::benchmark_data_smithi090_166897_object40359:head# 2024-05-15T23:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:613d179b:::benchmark_data_smithi090_166897_object30209:head# 2024-05-15T23:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:613da6bd:::benchmark_data_smithi090_166897_object69884:head# 2024-05-15T23:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:613f2838:::benchmark_data_smithi090_166897_object50378:head# 2024-05-15T23:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:613f53bd:::benchmark_data_smithi090_166897_object6190:head# 2024-05-15T23:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:613f64a7:::benchmark_data_smithi090_166897_object16405:head# 2024-05-15T23:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:613f912c:::benchmark_data_smithi090_166897_object32423:head# 2024-05-15T23:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.657 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6140324b:::benchmark_data_smithi090_166897_object62877:head# 2024-05-15T23:40:13.658 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.658 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6142990a:::benchmark_data_smithi090_166897_object54356:head# 2024-05-15T23:40:13.658 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.658 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61429a05:::benchmark_data_smithi090_166897_object17690:head# 2024-05-15T23:40:13.658 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.658 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6145aa9d:::benchmark_data_smithi090_166897_object13577:head# 2024-05-15T23:40:13.658 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.658 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61476283:::benchmark_data_smithi090_166897_object46452:head# 2024-05-15T23:40:13.658 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.658 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6148a516:::benchmark_data_smithi090_166897_object70303:head# 2024-05-15T23:40:13.658 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.658 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:614b2105:::benchmark_data_smithi090_166897_object29232:head# 2024-05-15T23:40:13.658 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.672 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:614b4c15:::benchmark_data_smithi090_166897_object7299:head# 2024-05-15T23:40:13.672 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.672 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:614c7342:::benchmark_data_smithi090_166897_object12182:head# 2024-05-15T23:40:13.672 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.672 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:614c89d3:::benchmark_data_smithi090_166897_object24480:head# 2024-05-15T23:40:13.672 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.672 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:614ce1fc:::benchmark_data_smithi090_166897_object32566:head# 2024-05-15T23:40:13.672 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.672 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:614dcb2e:::benchmark_data_smithi090_166897_object46837:head# 2024-05-15T23:40:13.672 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.672 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:614de7cb:::benchmark_data_smithi090_166897_object61611:head# 2024-05-15T23:40:13.672 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.672 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:615018ff:::benchmark_data_smithi090_166897_object20673:head# 2024-05-15T23:40:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61511120:::benchmark_data_smithi090_166897_object36951:head# 2024-05-15T23:40:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61517d54:::benchmark_data_smithi090_166897_object11733:head# 2024-05-15T23:40:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6151ab82:::benchmark_data_smithi090_166897_object3815:head# 2024-05-15T23:40:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61526a91:::benchmark_data_smithi090_166897_object64622:head# 2024-05-15T23:40:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:615353e5:::benchmark_data_smithi090_166897_object4980:head# 2024-05-15T23:40:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6153d1b8:::benchmark_data_smithi090_166897_object6121:head# 2024-05-15T23:40:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61547ccc:::benchmark_data_smithi090_166897_object34757:head# 2024-05-15T23:40:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.674 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6154f289:::benchmark_data_smithi090_166897_object29014:head# 2024-05-15T23:40:13.674 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.674 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6155a349:::benchmark_data_smithi090_166897_object3391:head# 2024-05-15T23:40:13.674 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.674 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61566729:::benchmark_data_smithi090_166897_object48550:head# 2024-05-15T23:40:13.674 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.674 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6156f677:::benchmark_data_smithi090_166897_object6261:head# 2024-05-15T23:40:13.674 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.674 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61574837:::benchmark_data_smithi090_166897_object2671:head# 2024-05-15T23:40:13.674 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.674 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:615770a7:::benchmark_data_smithi090_166897_object31741:head# 2024-05-15T23:40:13.674 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.674 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:615a86a1:::benchmark_data_smithi090_166897_object41763:head# 2024-05-15T23:40:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:615b7dd5:::benchmark_data_smithi090_166897_object2965:head# 2024-05-15T23:40:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:615cc162:::benchmark_data_smithi090_166897_object43046:head# 2024-05-15T23:40:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:615dae6f:::benchmark_data_smithi090_166897_object33760:head# 2024-05-15T23:40:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:615dbeff:::benchmark_data_smithi090_166897_object44696:head# 2024-05-15T23:40:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:615dddd6:::benchmark_data_smithi090_166897_object24680:head# 2024-05-15T23:40:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:615ebc64:::benchmark_data_smithi090_166897_object35704:head# 2024-05-15T23:40:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:615ec775:::benchmark_data_smithi090_166897_object819:head# 2024-05-15T23:40:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:615ec8fe:::benchmark_data_smithi090_166897_object65001:head# 2024-05-15T23:40:13.676 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:615ef31c:::benchmark_data_smithi090_166897_object59790:head# 2024-05-15T23:40:13.676 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:616020b0:::benchmark_data_smithi090_166897_object45442:head# 2024-05-15T23:40:13.676 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61604775:::benchmark_data_smithi090_166897_object12079:head# 2024-05-15T23:40:13.676 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61614357:::benchmark_data_smithi090_166897_object59602:head# 2024-05-15T23:40:13.676 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6165ffcf:::benchmark_data_smithi090_166897_object21803:head# 2024-05-15T23:40:13.676 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61670c83:::benchmark_data_smithi090_166897_object19473:head# 2024-05-15T23:40:13.676 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6167245a:::benchmark_data_smithi090_166897_object756:head# 2024-05-15T23:40:13.677 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.677 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:616a2f4c:::benchmark_data_smithi090_166897_object52272:head# 2024-05-15T23:40:13.677 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.677 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:616b9c6f:::benchmark_data_smithi090_166897_object13974:head# 2024-05-15T23:40:13.677 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.677 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:616c7bd3:::benchmark_data_smithi090_166897_object36578:head# 2024-05-15T23:40:13.677 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.677 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:616d9881:::benchmark_data_smithi090_166897_object63649:head# 2024-05-15T23:40:13.677 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.677 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:616f3938:::benchmark_data_smithi090_166897_object14096:head# 2024-05-15T23:40:13.677 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.677 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:616f8b2a:::benchmark_data_smithi090_166897_object19898:head# 2024-05-15T23:40:13.677 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.677 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61706a8e:::benchmark_data_smithi090_166897_object62274:head# 2024-05-15T23:40:13.677 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.678 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61710fe3:::benchmark_data_smithi090_166897_object68382:head# 2024-05-15T23:40:13.678 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.678 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61715ba9:::benchmark_data_smithi090_166897_object21548:head# 2024-05-15T23:40:13.678 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.678 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6171e8de:::benchmark_data_smithi090_166897_object3571:head# 2024-05-15T23:40:13.678 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.678 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6171f22a:::benchmark_data_smithi090_166897_object38159:head# 2024-05-15T23:40:13.678 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.678 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61722985:::benchmark_data_smithi090_166897_object60175:head# 2024-05-15T23:40:13.678 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.678 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6172763d:::benchmark_data_smithi090_166897_object48525:head# 2024-05-15T23:40:13.678 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.678 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6174aaaa:::benchmark_data_smithi090_166897_object20113:head# 2024-05-15T23:40:13.679 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.679 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:617530c1:::benchmark_data_smithi090_166897_object3904:head# 2024-05-15T23:40:13.679 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.679 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61754f65:::benchmark_data_smithi090_166897_object41577:head# 2024-05-15T23:40:13.679 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.679 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61755d21:::benchmark_data_smithi090_166897_object24569:head# 2024-05-15T23:40:13.679 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.679 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61761e7b:::benchmark_data_smithi090_166897_object14903:head# 2024-05-15T23:40:13.679 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.679 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6176f5dc:::benchmark_data_smithi090_166897_object20355:head# 2024-05-15T23:40:13.679 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.679 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6177a175:::benchmark_data_smithi090_166897_object33847:head# 2024-05-15T23:40:13.679 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.679 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6177ae49:::benchmark_data_smithi090_166897_object38653:head# 2024-05-15T23:40:13.679 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.680 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6178fb8f:::benchmark_data_smithi090_166897_object25009:head# 2024-05-15T23:40:13.680 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.680 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:617950bb:::benchmark_data_smithi090_166897_object32908:head# 2024-05-15T23:40:13.680 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.680 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:617b1a56:::benchmark_data_smithi090_166897_object13076:head# 2024-05-15T23:40:13.680 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.680 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:617b8206:::benchmark_data_smithi090_166897_object24098:head# 2024-05-15T23:40:13.680 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.680 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:617c7a82:::benchmark_data_smithi090_166897_object68533:head# 2024-05-15T23:40:13.680 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.680 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:617cab4c:::benchmark_data_smithi090_166897_object45596:head# 2024-05-15T23:40:13.680 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.680 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:617db779:::benchmark_data_smithi090_166897_object5195:head# 2024-05-15T23:40:13.680 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.681 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:617f8d6c:::benchmark_data_smithi090_166897_object8791:head# 2024-05-15T23:40:13.681 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.681 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:617fb4e6:::benchmark_data_smithi090_166897_object59851:head# 2024-05-15T23:40:13.681 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.681 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6180ed6a:::benchmark_data_smithi090_166897_object11992:head# 2024-05-15T23:40:13.681 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.681 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61817a1a:::benchmark_data_smithi090_166897_object26225:head# 2024-05-15T23:40:13.681 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.681 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61823d14:::benchmark_data_smithi090_166897_object28934:head# 2024-05-15T23:40:13.681 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.681 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:618265bd:::benchmark_data_smithi090_166897_object45349:head# 2024-05-15T23:40:13.681 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.681 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6182ae38:::benchmark_data_smithi090_166897_object66634:head# 2024-05-15T23:40:13.681 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.681 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61831c5e:::benchmark_data_smithi090_166897_object41178:head# 2024-05-15T23:40:13.682 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.682 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6184f3be:::benchmark_data_smithi090_166897_object70490:head# 2024-05-15T23:40:13.682 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.682 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6185bbae:::benchmark_data_smithi090_166897_object42117:head# 2024-05-15T23:40:13.682 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.682 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6187081b:::benchmark_data_smithi090_166897_object18600:head# 2024-05-15T23:40:13.682 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.682 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61881017:::benchmark_data_smithi090_166897_object56682:head# 2024-05-15T23:40:13.682 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.682 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6188a3c3:::benchmark_data_smithi090_166897_object8176:head# 2024-05-15T23:40:13.682 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.682 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:618aee23:::benchmark_data_smithi090_166897_object66480:head# 2024-05-15T23:40:13.682 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.682 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:618aff3b:::benchmark_data_smithi090_166897_object13860:head# 2024-05-15T23:40:13.683 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.683 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:618b6202:::benchmark_data_smithi090_166897_object64901:head# 2024-05-15T23:40:13.683 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.683 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:618c17e3:::benchmark_data_smithi090_166897_object54949:head# 2024-05-15T23:40:13.683 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.683 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:618da72a:::benchmark_data_smithi090_166897_object2770:head# 2024-05-15T23:40:13.683 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.683 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:618e61db:::benchmark_data_smithi090_166897_object68332:head# 2024-05-15T23:40:13.683 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.683 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:618ef06a:::benchmark_data_smithi090_166897_object43389:head# 2024-05-15T23:40:13.683 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.683 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:618f0c01:::benchmark_data_smithi090_166897_object62564:head# 2024-05-15T23:40:13.683 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.683 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:618f474b:::benchmark_data_smithi090_166897_object45074:head# 2024-05-15T23:40:13.683 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.684 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6190a935:::benchmark_data_smithi090_166897_object10639:head# 2024-05-15T23:40:13.684 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.684 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61923ad3:::benchmark_data_smithi090_166897_object61410:head# 2024-05-15T23:40:13.684 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.684 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:619247b1:::benchmark_data_smithi090_166897_object39044:head# 2024-05-15T23:40:13.684 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.684 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6192740b:::benchmark_data_smithi090_166897_object10807:head# 2024-05-15T23:40:13.684 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.684 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6192837b:::benchmark_data_smithi090_166897_object13941:head# 2024-05-15T23:40:13.684 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.684 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6192e1e8:::benchmark_data_smithi090_166897_object43927:head# 2024-05-15T23:40:13.684 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.684 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6193778f:::benchmark_data_smithi090_166897_object69812:head# 2024-05-15T23:40:13.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.685 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6194ae84:::benchmark_data_smithi090_166897_object60352:head# 2024-05-15T23:40:13.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.685 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61954ae5:::benchmark_data_smithi090_166897_object34793:head# 2024-05-15T23:40:13.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.685 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61965227:::benchmark_data_smithi090_166897_object17235:head# 2024-05-15T23:40:13.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.685 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:619f8e90:::benchmark_data_smithi090_166897_object40391:head# 2024-05-15T23:40:13.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.685 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61a00af1:::benchmark_data_smithi090_166897_object48098:head# 2024-05-15T23:40:13.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.685 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61a0fc10:::benchmark_data_smithi090_166897_object37874:head# 2024-05-15T23:40:13.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.685 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61a16ea9:::benchmark_data_smithi090_166897_object10865:head# 2024-05-15T23:40:13.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.686 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61a19a4e:::benchmark_data_smithi090_166897_object52897:head# 2024-05-15T23:40:13.686 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.686 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61a1dd72:::benchmark_data_smithi090_166897_object24360:head# 2024-05-15T23:40:13.686 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.686 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61a24027:::benchmark_data_smithi090_166897_object23771:head# 2024-05-15T23:40:13.686 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.686 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61a2643d:::benchmark_data_smithi090_166897_object268:head# 2024-05-15T23:40:13.686 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.686 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61a3d7eb:::benchmark_data_smithi090_166897_object38659:head# 2024-05-15T23:40:13.686 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.702 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61a63121:::benchmark_data_smithi090_166897_object44626:head# 2024-05-15T23:40:13.702 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.702 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61a828a5:::benchmark_data_smithi090_166897_object18908:head# 2024-05-15T23:40:13.702 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.702 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61a86b0f:::benchmark_data_smithi090_166897_object59584:head# 2024-05-15T23:40:13.702 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.702 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61a9c5a4:::benchmark_data_smithi090_166897_object16994:head# 2024-05-15T23:40:13.702 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.702 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61a9f41d:::benchmark_data_smithi090_166897_object60227:head# 2024-05-15T23:40:13.703 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.703 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61aa0c92:::benchmark_data_smithi090_166897_object33666:head# 2024-05-15T23:40:13.703 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.703 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61aad786:::benchmark_data_smithi090_166897_object57257:head# 2024-05-15T23:40:13.703 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.703 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61ab7768:::benchmark_data_smithi090_166897_object27969:head# 2024-05-15T23:40:13.703 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.703 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61abeae5:::benchmark_data_smithi090_166897_object58693:head# 2024-05-15T23:40:13.703 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.703 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61add859:::benchmark_data_smithi090_166897_object53381:head# 2024-05-15T23:40:13.703 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.703 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61af2663:::benchmark_data_smithi090_166897_object31588:head# 2024-05-15T23:40:13.703 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.703 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61b18d3d:::benchmark_data_smithi090_166897_object60974:head# 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61b4cdea:::benchmark_data_smithi090_166897_object21203:head# 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61b6beb5:::benchmark_data_smithi090_166897_object49862:head# 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61b7b6d4:::benchmark_data_smithi090_166897_object415:head# 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61b80e70:::benchmark_data_smithi090_166897_object13833:head# 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61baa641:::benchmark_data_smithi090_166897_object30238:head# 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61bac318:::benchmark_data_smithi090_166897_object21744:head# 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61bb7b14:::benchmark_data_smithi090_166897_object34954:head# 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.704 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61bd25d3:::benchmark_data_smithi090_166897_object53711:head# 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61bed7c4:::benchmark_data_smithi090_166897_object15798:head# 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61c01c63:::benchmark_data_smithi090_166897_object7275:head# 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61c0e50b:::benchmark_data_smithi090_166897_object26437:head# 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61c18c66:::benchmark_data_smithi090_166897_object46914:head# 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61c303b7:::benchmark_data_smithi090_166897_object29412:head# 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61c32ce3:::benchmark_data_smithi090_166897_object25715:head# 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61c3448f:::benchmark_data_smithi090_166897_object37635:head# 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.705 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61c47c0f:::benchmark_data_smithi090_166897_object42948:head# 2024-05-15T23:40:13.706 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.706 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61c48a8c:::benchmark_data_smithi090_166897_object36726:head# 2024-05-15T23:40:13.706 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.706 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61c57977:::benchmark_data_smithi090_166897_object37204:head# 2024-05-15T23:40:13.706 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.706 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61c707fb:::benchmark_data_smithi090_166897_object43282:head# 2024-05-15T23:40:13.706 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.706 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61c867d2:::benchmark_data_smithi090_166897_object33448:head# 2024-05-15T23:40:13.706 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.706 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61c9e66a:::benchmark_data_smithi090_166897_object29866:head# 2024-05-15T23:40:13.706 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.706 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61cabf5d:::benchmark_data_smithi090_166897_object13758:head# 2024-05-15T23:40:13.706 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.706 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61cb0bcb:::benchmark_data_smithi090_166897_object59828:head# 2024-05-15T23:40:13.707 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.707 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61cc486a:::benchmark_data_smithi090_166897_object53967:head# 2024-05-15T23:40:13.707 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.707 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61ce9993:::benchmark_data_smithi090_166897_object66127:head# 2024-05-15T23:40:13.707 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.707 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61cfe9d8:::benchmark_data_smithi090_166897_object42153:head# 2024-05-15T23:40:13.707 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.707 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61d08847:::benchmark_data_smithi090_166897_object27214:head# 2024-05-15T23:40:13.707 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.707 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61d30894:::benchmark_data_smithi090_166897_object33712:head# 2024-05-15T23:40:13.707 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.707 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61d3212b:::benchmark_data_smithi090_166897_object39069:head# 2024-05-15T23:40:13.707 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.707 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61d36601:::benchmark_data_smithi090_166897_object11065:head# 2024-05-15T23:40:13.707 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.708 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61d38416:::benchmark_data_smithi090_166897_object13392:head# 2024-05-15T23:40:13.708 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.708 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61d47a50:::benchmark_data_smithi090_166897_object17444:head# 2024-05-15T23:40:13.708 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.708 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61d5722c:::benchmark_data_smithi090_166897_object4992:head# 2024-05-15T23:40:13.708 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.708 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61d84dd8:::benchmark_data_smithi090_166897_object50880:head# 2024-05-15T23:40:13.708 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.708 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61dc74a5:::benchmark_data_smithi090_166897_object57243:head# 2024-05-15T23:40:13.708 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.708 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61ddaa9c:::benchmark_data_smithi090_166897_object37464:head# 2024-05-15T23:40:13.708 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.708 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61ddb97c:::benchmark_data_smithi090_166897_object48238:head# 2024-05-15T23:40:13.708 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61ddba66:::benchmark_data_smithi090_166897_object19067:head# 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61de1607:::benchmark_data_smithi090_166897_object16395:head# 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61df396f:::benchmark_data_smithi090_166897_object167:head# 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61dfe9cf:::benchmark_data_smithi090_166897_object7401:head# 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61e11b44:::benchmark_data_smithi090_166897_object4176:head# 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61e11f4a:::benchmark_data_smithi090_166897_object42258:head# 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61e121ce:::benchmark_data_smithi090_166897_object13540:head# 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61e1720e:::benchmark_data_smithi090_166897_object64669:head# 2024-05-15T23:40:13.709 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.710 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61e1be76:::benchmark_data_smithi090_166897_object51799:head# 2024-05-15T23:40:13.710 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.710 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61e24a5c:::benchmark_data_smithi090_166897_object59083:head# 2024-05-15T23:40:13.710 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.710 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61e49f22:::benchmark_data_smithi090_166897_object16507:head# 2024-05-15T23:40:13.710 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.710 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61e545cf:::benchmark_data_smithi090_166897_object34178:head# 2024-05-15T23:40:13.710 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.710 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61e58d52:::benchmark_data_smithi090_166897_object70840:head# 2024-05-15T23:40:13.710 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.710 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61e82af9:::benchmark_data_smithi090_166897_object26062:head# 2024-05-15T23:40:13.710 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.710 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61e86434:::benchmark_data_smithi090_166897_object34129:head# 2024-05-15T23:40:13.710 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61e879aa:::benchmark_data_smithi090_166897_object13960:head# 2024-05-15T23:40:13.711 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61e992a5:::benchmark_data_smithi090_166897_object64527:head# 2024-05-15T23:40:13.711 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61e9eb9e:::benchmark_data_smithi090_166897_object37733:head# 2024-05-15T23:40:13.711 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61e9f89b:::benchmark_data_smithi090_166897_object49751:head# 2024-05-15T23:40:13.711 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61ea8f3c:::benchmark_data_smithi090_166897_object69304:head# 2024-05-15T23:40:13.711 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61ebaf71:::benchmark_data_smithi090_166897_object32771:head# 2024-05-15T23:40:13.711 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61ebe33b:::benchmark_data_smithi090_166897_object18419:head# 2024-05-15T23:40:13.711 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61ec4c67:::benchmark_data_smithi090_166897_object24123:head# 2024-05-15T23:40:13.712 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.712 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61ed8c58:::benchmark_data_smithi090_166897_object58537:head# 2024-05-15T23:40:13.712 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.712 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61edb473:::benchmark_data_smithi090_166897_object30843:head# 2024-05-15T23:40:13.712 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.712 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61edd8e5:::benchmark_data_smithi090_166897_object63547:head# 2024-05-15T23:40:13.712 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.712 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61eef977:::benchmark_data_smithi090_166897_object34374:head# 2024-05-15T23:40:13.712 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.712 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61efc979:::benchmark_data_smithi090_166897_object920:head# 2024-05-15T23:40:13.712 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.712 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61efff27:::benchmark_data_smithi090_166897_object31675:head# 2024-05-15T23:40:13.712 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.712 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61f0c438:::benchmark_data_smithi090_166897_object32281:head# 2024-05-15T23:40:13.712 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.713 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61f32a42:::benchmark_data_smithi090_166897_object27361:head# 2024-05-15T23:40:13.713 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.713 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61f37262:::benchmark_data_smithi090_166897_object35563:head# 2024-05-15T23:40:13.713 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.713 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61f3af8e:::benchmark_data_smithi090_166897_object17719:head# 2024-05-15T23:40:13.713 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.713 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61f41f9e:::benchmark_data_smithi090_166897_object17202:head# 2024-05-15T23:40:13.713 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.713 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61f58df1:::benchmark_data_smithi090_166897_object395:head# 2024-05-15T23:40:13.713 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.713 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61f683ca:::benchmark_data_smithi090_166897_object29461:head# 2024-05-15T23:40:13.713 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.713 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61f6cf42:::benchmark_data_smithi090_166897_object28184:head# 2024-05-15T23:40:13.713 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.714 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61f72399:::benchmark_data_smithi090_166897_object50862:head# 2024-05-15T23:40:13.714 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.714 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61f90b91:::benchmark_data_smithi090_166897_object26467:head# 2024-05-15T23:40:13.714 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.714 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61fa20c6:::benchmark_data_smithi090_166897_object14023:head# 2024-05-15T23:40:13.714 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.714 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61fa2299:::benchmark_data_smithi090_166897_object41975:head# 2024-05-15T23:40:13.714 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.714 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61fa3b6d:::benchmark_data_smithi090_166897_object21087:head# 2024-05-15T23:40:13.714 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.714 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61fa6b3f:::benchmark_data_smithi090_166897_object38695:head# 2024-05-15T23:40:13.714 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.714 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61fc21eb:::benchmark_data_smithi090_166897_object63517:head# 2024-05-15T23:40:13.714 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.714 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61fcb914:::benchmark_data_smithi090_166897_object2674:head# 2024-05-15T23:40:13.715 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.715 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61fd7645:::benchmark_data_smithi090_166897_object31696:head# 2024-05-15T23:40:13.715 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.715 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61fd952a:::benchmark_data_smithi090_166897_object48812:head# 2024-05-15T23:40:13.715 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.715 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:61fe8307:::benchmark_data_smithi090_166897_object25137:head# 2024-05-15T23:40:13.715 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.715 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62010446:::benchmark_data_smithi090_166897_object31730:head# 2024-05-15T23:40:13.715 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.715 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6201574a:::benchmark_data_smithi090_166897_object48213:head# 2024-05-15T23:40:13.715 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.715 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62023f6d:::benchmark_data_smithi090_166897_object26098:head# 2024-05-15T23:40:13.715 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.715 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6202efae:::benchmark_data_smithi090_166897_object19176:head# 2024-05-15T23:40:13.716 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.716 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62034ec4:::benchmark_data_smithi090_166897_object44852:head# 2024-05-15T23:40:13.716 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.716 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62039225:::benchmark_data_smithi090_166897_object10960:head# 2024-05-15T23:40:13.716 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.716 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62040acc:::benchmark_data_smithi090_166897_object32026:head# 2024-05-15T23:40:13.716 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.716 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62042eb0:::benchmark_data_smithi090_166897_object58731:head# 2024-05-15T23:40:13.716 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.716 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62047770:::benchmark_data_smithi090_166897_object59555:head# 2024-05-15T23:40:13.716 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.716 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62049ef5:::benchmark_data_smithi090_166897_object36673:head# 2024-05-15T23:40:13.716 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.716 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62062493:::benchmark_data_smithi090_166897_object53857:head# 2024-05-15T23:40:13.717 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.717 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6206e3e1:::benchmark_data_smithi090_166897_object32138:head# 2024-05-15T23:40:13.717 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.717 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62070ad9:::benchmark_data_smithi090_166897_object65614:head# 2024-05-15T23:40:13.717 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.717 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62070b76:::benchmark_data_smithi090_166897_object30197:head# 2024-05-15T23:40:13.717 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.717 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:620736e2:::benchmark_data_smithi090_166897_object28125:head# 2024-05-15T23:40:13.717 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.724 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:40:13.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T23:40:13.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:13.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:40:13.734 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6207ae42:::benchmark_data_smithi090_166897_object57473:head# 2024-05-15T23:40:13.734 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.734 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:620904ec:::benchmark_data_smithi090_166897_object43194:head# 2024-05-15T23:40:13.734 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.735 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62091c64:::benchmark_data_smithi090_166897_object49979:head# 2024-05-15T23:40:13.735 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.735 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6209726d:::benchmark_data_smithi090_166897_object984:head# 2024-05-15T23:40:13.735 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.735 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:620ae72f:::benchmark_data_smithi090_166897_object30234:head# 2024-05-15T23:40:13.735 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.735 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:620af87d:::benchmark_data_smithi090_166897_object38146:head# 2024-05-15T23:40:13.735 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.735 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:620c47f0:::benchmark_data_smithi090_166897_object89:head# 2024-05-15T23:40:13.735 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.735 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:620cf1ca:::benchmark_data_smithi090_166897_object14558:head# 2024-05-15T23:40:13.735 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.735 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:620d84a0:::benchmark_data_smithi090_166897_object56547:head# 2024-05-15T23:40:13.735 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:620e54e5:::benchmark_data_smithi090_166897_object31858:head# 2024-05-15T23:40:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:620f3c0b:::benchmark_data_smithi090_166897_object66822:head# 2024-05-15T23:40:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:621384a3:::benchmark_data_smithi090_166897_object19547:head# 2024-05-15T23:40:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6214c427:::benchmark_data_smithi090_166897_object10849:head# 2024-05-15T23:40:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:621500f6:::benchmark_data_smithi090_166897_object62805:head# 2024-05-15T23:40:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:621549b9:::benchmark_data_smithi090_166897_object49802:head# 2024-05-15T23:40:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62175e02:::benchmark_data_smithi090_166897_object48127:head# 2024-05-15T23:40:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62198a86:::benchmark_data_smithi090_166897_object14347:head# 2024-05-15T23:40:13.737 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.737 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:621a5959:::benchmark_data_smithi090_166897_object39764:head# 2024-05-15T23:40:13.737 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.737 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:621b927e:::benchmark_data_smithi090_166897_object43673:head# 2024-05-15T23:40:13.737 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.737 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:621bdb82:::benchmark_data_smithi090_166897_object19743:head# 2024-05-15T23:40:13.737 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.737 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:621c5b63:::benchmark_data_smithi090_166897_object17274:head# 2024-05-15T23:40:13.737 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.737 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:621c7bfd:::benchmark_data_smithi090_166897_object14186:head# 2024-05-15T23:40:13.737 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.737 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:621e2d5f:::benchmark_data_smithi090_166897_object40963:head# 2024-05-15T23:40:13.737 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.737 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:621e402f:::benchmark_data_smithi090_166897_object70870:head# 2024-05-15T23:40:13.737 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.738 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6220e335:::benchmark_data_smithi090_166897_object7507:head# 2024-05-15T23:40:13.738 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.738 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6221bce6:::benchmark_data_smithi090_166897_object60584:head# 2024-05-15T23:40:13.738 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.738 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62230ea5:::benchmark_data_smithi090_166897_object36410:head# 2024-05-15T23:40:13.738 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.738 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6224cde0:::benchmark_data_smithi090_166897_object56750:head# 2024-05-15T23:40:13.738 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.738 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6224cf11:::benchmark_data_smithi090_166897_object49370:head# 2024-05-15T23:40:13.738 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.738 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62253601:::benchmark_data_smithi090_166897_object39064:head# 2024-05-15T23:40:13.738 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.738 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:622676c4:::benchmark_data_smithi090_166897_object60947:head# 2024-05-15T23:40:13.738 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.739 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62269e37:::benchmark_data_smithi090_166897_object65299:head# 2024-05-15T23:40:13.739 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.739 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62281a43:::benchmark_data_smithi090_166897_object60001:head# 2024-05-15T23:40:13.739 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.739 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6229277a:::benchmark_data_smithi090_166897_object20948:head# 2024-05-15T23:40:13.739 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.739 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6229da49:::benchmark_data_smithi090_166897_object33180:head# 2024-05-15T23:40:13.739 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.739 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:622a66ce:::benchmark_data_smithi090_166897_object5473:head# 2024-05-15T23:40:13.739 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.739 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:622bb25c:::benchmark_data_smithi090_166897_object6943:head# 2024-05-15T23:40:13.739 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.739 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:622beaac:::benchmark_data_smithi090_166897_object19036:head# 2024-05-15T23:40:13.739 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.739 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:622d0cbb:::benchmark_data_smithi090_166897_object23954:head# 2024-05-15T23:40:13.740 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.740 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:622d7ee9:::benchmark_data_smithi090_166897_object40307:head# 2024-05-15T23:40:13.740 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.740 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:622db15f:::benchmark_data_smithi090_166897_object27524:head# 2024-05-15T23:40:13.740 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.740 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:622e046f:::benchmark_data_smithi090_166897_object17698:head# 2024-05-15T23:40:13.740 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.740 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:623089e9:::benchmark_data_smithi090_166897_object43071:head# 2024-05-15T23:40:13.740 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.740 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62333a55:::benchmark_data_smithi090_166897_object51177:head# 2024-05-15T23:40:13.740 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.740 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62338a8b:::benchmark_data_smithi090_166897_object11272:head# 2024-05-15T23:40:13.740 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.740 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62354027:::benchmark_data_smithi090_166897_object49194:head# 2024-05-15T23:40:13.741 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.741 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6236d269:::benchmark_data_smithi090_166897_object55029:head# 2024-05-15T23:40:13.741 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.741 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:623913bc:::benchmark_data_smithi090_166897_object29860:head# 2024-05-15T23:40:13.741 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.741 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6239830e:::benchmark_data_smithi090_166897_object7880:head# 2024-05-15T23:40:13.741 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.741 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:623a5017:::benchmark_data_smithi090_166897_object28373:head# 2024-05-15T23:40:13.741 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.741 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:623a734e:::benchmark_data_smithi090_166897_object31743:head# 2024-05-15T23:40:13.741 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.741 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:623bc17d:::benchmark_data_smithi090_166897_object20392:head# 2024-05-15T23:40:13.741 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.741 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:623d0116:::benchmark_data_smithi090_166897_object67120:head# 2024-05-15T23:40:13.742 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.742 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:623e2e28:::benchmark_data_smithi090_166897_object54239:head# 2024-05-15T23:40:13.742 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.742 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62410d72:::benchmark_data_smithi090_166897_object28832:head# 2024-05-15T23:40:13.742 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.742 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62418dc9:::benchmark_data_smithi090_166897_object24169:head# 2024-05-15T23:40:13.742 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.742 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:624334d2:::benchmark_data_smithi090_166897_object3742:head# 2024-05-15T23:40:13.742 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.742 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:624578ed:::benchmark_data_smithi090_166897_object7934:head# 2024-05-15T23:40:13.742 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.742 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6245ce6d:::benchmark_data_smithi090_166897_object21911:head# 2024-05-15T23:40:13.742 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.742 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6245e34e:::benchmark_data_smithi090_166897_object25568:head# 2024-05-15T23:40:13.743 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.743 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62464831:::benchmark_data_smithi090_166897_object10968:head# 2024-05-15T23:40:13.743 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.743 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62467bb7:::benchmark_data_smithi090_166897_object42493:head# 2024-05-15T23:40:13.743 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.743 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62468302:::benchmark_data_smithi090_166897_object29492:head# 2024-05-15T23:40:13.743 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.743 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62485de1:::benchmark_data_smithi090_166897_object8125:head# 2024-05-15T23:40:13.743 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.743 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62486113:::benchmark_data_smithi090_166897_object70312:head# 2024-05-15T23:40:13.743 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.743 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62495447:::benchmark_data_smithi090_166897_object25267:head# 2024-05-15T23:40:13.743 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.743 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:624a2abb:::benchmark_data_smithi090_166897_object69389:head# 2024-05-15T23:40:13.743 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:624b027c:::benchmark_data_smithi090_166897_object4537:head# 2024-05-15T23:40:13.744 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:624b5a6e:::benchmark_data_smithi090_166897_object44829:head# 2024-05-15T23:40:13.744 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:624c3dec:::benchmark_data_smithi090_166897_object8939:head# 2024-05-15T23:40:13.744 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:624ca9d3:::benchmark_data_smithi090_166897_object20246:head# 2024-05-15T23:40:13.744 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:624cedf7:::benchmark_data_smithi090_166897_object44859:head# 2024-05-15T23:40:13.744 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:624ea0bf:::benchmark_data_smithi090_166897_object52536:head# 2024-05-15T23:40:13.744 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62502605:::benchmark_data_smithi090_166897_object10947:head# 2024-05-15T23:40:13.744 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62502abc:::benchmark_data_smithi090_166897_object25738:head# 2024-05-15T23:40:13.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.745 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62506e7c:::benchmark_data_smithi090_166897_object27911:head# 2024-05-15T23:40:13.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.745 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6250813a:::benchmark_data_smithi090_166897_object61847:head# 2024-05-15T23:40:13.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.745 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6251064b:::benchmark_data_smithi090_166897_object36424:head# 2024-05-15T23:40:13.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.745 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6253047c:::benchmark_data_smithi090_166897_object20998:head# 2024-05-15T23:40:13.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.745 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62532702:::benchmark_data_smithi090_166897_object40212:head# 2024-05-15T23:40:13.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.745 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62534deb:::benchmark_data_smithi090_166897_object6840:head# 2024-05-15T23:40:13.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.745 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62539109:::benchmark_data_smithi090_166897_object24830:head# 2024-05-15T23:40:13.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.746 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62544023:::benchmark_data_smithi090_166897_object39004:head# 2024-05-15T23:40:13.746 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.746 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6255182b:::benchmark_data_smithi090_166897_object6278:head# 2024-05-15T23:40:13.746 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.746 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6255f47d:::benchmark_data_smithi090_166897_object65959:head# 2024-05-15T23:40:13.746 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.746 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6255ff2a:::benchmark_data_smithi090_166897_object15975:head# 2024-05-15T23:40:13.746 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.746 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62564dc7:::benchmark_data_smithi090_166897_object25064:head# 2024-05-15T23:40:13.746 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.746 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62568c29:::benchmark_data_smithi090_166897_object2457:head# 2024-05-15T23:40:13.746 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.746 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6256c392:::benchmark_data_smithi090_166897_object63836:head# 2024-05-15T23:40:13.746 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.746 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:625a3d5c:::benchmark_data_smithi090_166897_object6833:head# 2024-05-15T23:40:13.747 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.747 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:625a76b4:::benchmark_data_smithi090_166897_object25022:head# 2024-05-15T23:40:13.747 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.747 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:625da19b:::benchmark_data_smithi090_166897_object42257:head# 2024-05-15T23:40:13.747 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.747 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:625e2cac:::benchmark_data_smithi090_166897_object23040:head# 2024-05-15T23:40:13.747 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.747 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:625e6e6b:::benchmark_data_smithi090_166897_object49651:head# 2024-05-15T23:40:13.747 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.747 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:625eed8d:::benchmark_data_smithi090_166897_object20586:head# 2024-05-15T23:40:13.747 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.747 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6261bc80:::benchmark_data_smithi090_166897_object43985:head# 2024-05-15T23:40:13.747 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.747 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62634a57:::benchmark_data_smithi090_166897_object55690:head# 2024-05-15T23:40:13.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.748 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6263f844:::benchmark_data_smithi090_166897_object37662:head# 2024-05-15T23:40:13.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.748 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62646a08:::benchmark_data_smithi090_166897_object52703:head# 2024-05-15T23:40:13.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.748 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6265ad03:::benchmark_data_smithi090_166897_object9216:head# 2024-05-15T23:40:13.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.748 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6266d1a6:::benchmark_data_smithi090_166897_object37234:head# 2024-05-15T23:40:13.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.748 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62670587:::benchmark_data_smithi090_166897_object32475:head# 2024-05-15T23:40:13.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.748 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62674ff5:::benchmark_data_smithi090_166897_object51477:head# 2024-05-15T23:40:13.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.748 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62696f7f:::benchmark_data_smithi090_166897_object27971:head# 2024-05-15T23:40:13.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:626a891a:::benchmark_data_smithi090_166897_object55563:head# 2024-05-15T23:40:13.749 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:626bbb26:::benchmark_data_smithi090_166897_object48675:head# 2024-05-15T23:40:13.749 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:626be7ef:::benchmark_data_smithi090_166897_object44792:head# 2024-05-15T23:40:13.749 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:626c2459:::benchmark_data_smithi090_166897_object30181:head# 2024-05-15T23:40:13.749 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:626c2602:::benchmark_data_smithi090_166897_object63247:head# 2024-05-15T23:40:13.749 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:626ceaad:::benchmark_data_smithi090_166897_object21793:head# 2024-05-15T23:40:13.749 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:626d6a36:::benchmark_data_smithi090_166897_object65679:head# 2024-05-15T23:40:13.749 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62721046:::benchmark_data_smithi090_166897_object35052:head# 2024-05-15T23:40:13.750 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.750 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:627376a3:::benchmark_data_smithi090_166897_object62992:head# 2024-05-15T23:40:13.750 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.750 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6274816c:::benchmark_data_smithi090_166897_object29780:head# 2024-05-15T23:40:13.750 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.750 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62754cb4:::benchmark_data_smithi090_166897_object7684:head# 2024-05-15T23:40:13.750 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.750 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6278efed:::benchmark_data_smithi090_166897_object67203:head# 2024-05-15T23:40:13.750 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.750 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62795330:::benchmark_data_smithi090_166897_object9297:head# 2024-05-15T23:40:13.750 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.750 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6279a815:::benchmark_data_smithi090_166897_object10784:head# 2024-05-15T23:40:13.751 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.751 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:627b40db:::benchmark_data_smithi090_166897_object12045:head# 2024-05-15T23:40:13.751 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.751 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:627bc5ea:::benchmark_data_smithi090_166897_object42184:head# 2024-05-15T23:40:13.751 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.751 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:628045ba:::benchmark_data_smithi090_166897_object25096:head# 2024-05-15T23:40:13.751 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.751 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6281bee2:::benchmark_data_smithi090_166897_object49817:head# 2024-05-15T23:40:13.751 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.773 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62830d1f:::benchmark_data_smithi090_166897_object22441:head# 2024-05-15T23:40:13.773 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.774 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6283e95e:::benchmark_data_smithi090_166897_object70415:head# 2024-05-15T23:40:13.774 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.774 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6284a699:::benchmark_data_smithi090_166897_object67279:head# 2024-05-15T23:40:13.774 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.774 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6284c95e:::benchmark_data_smithi090_166897_object36173:head# 2024-05-15T23:40:13.774 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.774 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6285c6a0:::benchmark_data_smithi090_166897_object31794:head# 2024-05-15T23:40:13.774 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.774 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62861624:::benchmark_data_smithi090_166897_object69295:head# 2024-05-15T23:40:13.774 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.774 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6288246b:::benchmark_data_smithi090_166897_object42278:head# 2024-05-15T23:40:13.774 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.774 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6288bfc3:::benchmark_data_smithi090_166897_object51900:head# 2024-05-15T23:40:13.775 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.775 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:628acd4e:::benchmark_data_smithi090_166897_object44315:head# 2024-05-15T23:40:13.775 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.775 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:628c651a:::benchmark_data_smithi090_166897_object36875:head# 2024-05-15T23:40:13.775 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.775 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:628c7e1f:::benchmark_data_smithi090_166897_object34833:head# 2024-05-15T23:40:13.775 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.775 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:628dabce:::benchmark_data_smithi090_166897_object23189:head# 2024-05-15T23:40:13.775 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.775 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:628e9866:::benchmark_data_smithi090_166897_object29666:head# 2024-05-15T23:40:13.775 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.775 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:628e9f1f:::benchmark_data_smithi090_166897_object68488:head# 2024-05-15T23:40:13.775 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.775 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6291dd19:::benchmark_data_smithi090_166897_object46467:head# 2024-05-15T23:40:13.775 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.776 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6291edd7:::benchmark_data_smithi090_166897_object41761:head# 2024-05-15T23:40:13.776 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.776 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:629204c1:::benchmark_data_smithi090_166897_object15355:head# 2024-05-15T23:40:13.776 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.776 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:629684d0:::benchmark_data_smithi090_166897_object25530:head# 2024-05-15T23:40:13.776 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.776 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6296bc49:::benchmark_data_smithi090_166897_object65838:head# 2024-05-15T23:40:13.776 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.776 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6297cc3f:::benchmark_data_smithi090_166897_object17376:head# 2024-05-15T23:40:13.776 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.776 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6297d5d6:::benchmark_data_smithi090_166897_object53766:head# 2024-05-15T23:40:13.776 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.776 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:629a033b:::benchmark_data_smithi090_166897_object3125:head# 2024-05-15T23:40:13.776 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.776 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:629a2e56:::benchmark_data_smithi090_166897_object50495:head# 2024-05-15T23:40:13.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.777 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:629abc2e:::benchmark_data_smithi090_166897_object48501:head# 2024-05-15T23:40:13.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.777 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:629c0c41:::benchmark_data_smithi090_166897_object25760:head# 2024-05-15T23:40:13.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.777 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:629d569a:::benchmark_data_smithi090_166897_object17060:head# 2024-05-15T23:40:13.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.777 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:629d5a60:::benchmark_data_smithi090_166897_object23483:head# 2024-05-15T23:40:13.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.777 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:629e2c42:::benchmark_data_smithi090_166897_object1728:head# 2024-05-15T23:40:13.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.777 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:629e90d1:::benchmark_data_smithi090_166897_object63502:head# 2024-05-15T23:40:13.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.777 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:629ffce2:::benchmark_data_smithi090_166897_object29545:head# 2024-05-15T23:40:13.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.778 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62a20dc9:::benchmark_data_smithi090_166897_object49396:head# 2024-05-15T23:40:13.778 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.778 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62a27132:::benchmark_data_smithi090_166897_object19306:head# 2024-05-15T23:40:13.778 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.778 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62a46e82:::benchmark_data_smithi090_166897_object44056:head# 2024-05-15T23:40:13.778 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.778 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62a4a217:::benchmark_data_smithi090_166897_object29476:head# 2024-05-15T23:40:13.778 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.778 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62a4c6f0:::benchmark_data_smithi090_166897_object23425:head# 2024-05-15T23:40:13.778 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.778 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62a4c918:::benchmark_data_smithi090_166897_object42229:head# 2024-05-15T23:40:13.778 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.778 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62a57022:::benchmark_data_smithi090_166897_object25076:head# 2024-05-15T23:40:13.778 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.779 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62a6c611:::benchmark_data_smithi090_166897_object14257:head# 2024-05-15T23:40:13.779 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.779 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62a78406:::benchmark_data_smithi090_166897_object46539:head# 2024-05-15T23:40:13.779 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.779 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62a837db:::benchmark_data_smithi090_166897_object11569:head# 2024-05-15T23:40:13.779 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.779 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62a95b21:::benchmark_data_smithi090_166897_object44293:head# 2024-05-15T23:40:13.779 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.779 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62a9fe2a:::benchmark_data_smithi090_166897_object68434:head# 2024-05-15T23:40:13.779 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.779 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62aa439c:::benchmark_data_smithi090_166897_object35154:head# 2024-05-15T23:40:13.779 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.779 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62aa6896:::benchmark_data_smithi090_166897_object6422:head# 2024-05-15T23:40:13.779 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.780 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62aaa8f1:::benchmark_data_smithi090_166897_object34085:head# 2024-05-15T23:40:13.780 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.780 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62aaaaa1:::benchmark_data_smithi090_166897_object69646:head# 2024-05-15T23:40:13.780 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.780 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62ac7980:::benchmark_data_smithi090_166897_object45611:head# 2024-05-15T23:40:13.780 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.780 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62b0382f:::benchmark_data_smithi090_166897_object9737:head# 2024-05-15T23:40:13.780 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.780 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62b1813b:::benchmark_data_smithi090_166897_object56011:head# 2024-05-15T23:40:13.780 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.780 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62b20273:::benchmark_data_smithi090_166897_object31380:head# 2024-05-15T23:40:13.780 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.780 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62b20611:::benchmark_data_smithi090_166897_object7399:head# 2024-05-15T23:40:13.780 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.780 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62b34e26:::benchmark_data_smithi090_166897_object5682:head# 2024-05-15T23:40:13.781 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.781 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62b58adc:::benchmark_data_smithi090_166897_object57873:head# 2024-05-15T23:40:13.781 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.781 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62b5b7f9:::benchmark_data_smithi090_166897_object29567:head# 2024-05-15T23:40:13.781 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.781 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62b7501c:::benchmark_data_smithi090_166897_object42784:head# 2024-05-15T23:40:13.781 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.781 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62b83f9d:::benchmark_data_smithi090_166897_object55250:head# 2024-05-15T23:40:13.781 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.781 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62b8c44c:::benchmark_data_smithi090_166897_object27319:head# 2024-05-15T23:40:13.781 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.781 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62b8e8fe:::benchmark_data_smithi090_166897_object25710:head# 2024-05-15T23:40:13.781 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.781 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62b91426:::benchmark_data_smithi090_166897_object4133:head# 2024-05-15T23:40:13.781 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62bc159f:::benchmark_data_smithi090_166897_object35678:head# 2024-05-15T23:40:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62bc5ecf:::benchmark_data_smithi090_166897_object29008:head# 2024-05-15T23:40:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62bc9385:::benchmark_data_smithi090_166897_object33938:head# 2024-05-15T23:40:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62bcd764:::benchmark_data_smithi090_166897_object37348:head# 2024-05-15T23:40:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62bd8dfd:::benchmark_data_smithi090_166897_object30463:head# 2024-05-15T23:40:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62bf6fa2:::benchmark_data_smithi090_166897_object3027:head# 2024-05-15T23:40:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62bf9b65:::benchmark_data_smithi090_166897_object67740:head# 2024-05-15T23:40:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.782 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62c01598:::benchmark_data_smithi090_166897_object34003:head# 2024-05-15T23:40:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62c10883:::benchmark_data_smithi090_166897_object50766:head# 2024-05-15T23:40:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62c11f0e:::benchmark_data_smithi090_166897_object3892:head# 2024-05-15T23:40:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62c1dc97:::benchmark_data_smithi090_166897_object36093:head# 2024-05-15T23:40:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62c2a984:::benchmark_data_smithi090_166897_object15486:head# 2024-05-15T23:40:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62c2bb3c:::benchmark_data_smithi090_166897_object735:head# 2024-05-15T23:40:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62c36181:::benchmark_data_smithi090_166897_object14877:head# 2024-05-15T23:40:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62c53f83:::benchmark_data_smithi090_166897_object64487:head# 2024-05-15T23:40:13.783 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.784 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62c6fe23:::benchmark_data_smithi090_166897_object66538:head# 2024-05-15T23:40:13.784 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.784 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62c90a3f:::benchmark_data_smithi090_166897_object61430:head# 2024-05-15T23:40:13.784 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.784 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62c97796:::benchmark_data_smithi090_166897_object46870:head# 2024-05-15T23:40:13.784 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.784 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62c9e046:::benchmark_data_smithi090_166897_object11036:head# 2024-05-15T23:40:13.784 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.784 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62ca35dc:::benchmark_data_smithi090_166897_object14437:head# 2024-05-15T23:40:13.784 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.784 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62cb3ccc:::benchmark_data_smithi090_166897_object61244:head# 2024-05-15T23:40:13.784 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.784 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62cb7a3f:::benchmark_data_smithi090_166897_object43145:head# 2024-05-15T23:40:13.784 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.784 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62cd6d65:::benchmark_data_smithi090_166897_object67017:head# 2024-05-15T23:40:13.785 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.785 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62cfd4d6:::benchmark_data_smithi090_166897_object33546:head# 2024-05-15T23:40:13.785 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.785 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62cfe989:::benchmark_data_smithi090_166897_object48949:head# 2024-05-15T23:40:13.785 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.785 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d00718:::benchmark_data_smithi090_166897_object15498:head# 2024-05-15T23:40:13.785 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.785 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d05579:::benchmark_data_smithi090_166897_object34189:head# 2024-05-15T23:40:13.785 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.785 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d0d2b7:::benchmark_data_smithi090_166897_object66772:head# 2024-05-15T23:40:13.785 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.785 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d0e1f5:::benchmark_data_smithi090_166897_object59686:head# 2024-05-15T23:40:13.785 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.785 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d1bde8:::benchmark_data_smithi090_166897_object1982:head# 2024-05-15T23:40:13.785 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.786 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d29575:::benchmark_data_smithi090_166897_object34609:head# 2024-05-15T23:40:13.786 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.786 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d4389f:::benchmark_data_smithi090_166897_object42356:head# 2024-05-15T23:40:13.786 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.786 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d46c13:::benchmark_data_smithi090_166897_object28204:head# 2024-05-15T23:40:13.786 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.786 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d47f57:::benchmark_data_smithi090_166897_object57346:head# 2024-05-15T23:40:13.786 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.786 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d601bc:::benchmark_data_smithi090_166897_object65825:head# 2024-05-15T23:40:13.786 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.786 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d6317e:::benchmark_data_smithi090_166897_object19640:head# 2024-05-15T23:40:13.786 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.786 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d65371:::benchmark_data_smithi090_166897_object31600:head# 2024-05-15T23:40:13.786 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.786 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d71963:::benchmark_data_smithi090_166897_object12274:head# 2024-05-15T23:40:13.787 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.787 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d78f84:::benchmark_data_smithi090_166897_object37432:head# 2024-05-15T23:40:13.787 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.787 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d7a777:::benchmark_data_smithi090_166897_object20922:head# 2024-05-15T23:40:13.787 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.787 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d91c3a:::benchmark_data_smithi090_166897_object25884:head# 2024-05-15T23:40:13.787 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.787 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62d99a77:::benchmark_data_smithi090_166897_object36917:head# 2024-05-15T23:40:13.787 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.787 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62da5cfe:::benchmark_data_smithi090_166897_object11117:head# 2024-05-15T23:40:13.787 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.787 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62dbc8b1:::benchmark_data_smithi090_166897_object56545:head# 2024-05-15T23:40:13.787 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.787 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62dd06da:::benchmark_data_smithi090_166897_object39772:head# 2024-05-15T23:40:13.788 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.788 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62decf26:::benchmark_data_smithi090_166897_object3991:head# 2024-05-15T23:40:13.788 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.788 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62df8ccc:::benchmark_data_smithi090_166897_object13174:head# 2024-05-15T23:40:13.788 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.788 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62e1956b:::benchmark_data_smithi090_166897_object48748:head# 2024-05-15T23:40:13.788 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.788 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62e23af8:::benchmark_data_smithi090_166897_object8303:head# 2024-05-15T23:40:13.788 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.788 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62e2eab7:::benchmark_data_smithi090_166897_object7790:head# 2024-05-15T23:40:13.788 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.788 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62e348fa:::benchmark_data_smithi090_166897_object12388:head# 2024-05-15T23:40:13.788 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.788 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62e3a248:::benchmark_data_smithi090_166897_object23743:head# 2024-05-15T23:40:13.788 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62e3ed46:::benchmark_data_smithi090_166897_object69064:head# 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62e42292:::benchmark_data_smithi090_166897_object37479:head# 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62e50d6b:::benchmark_data_smithi090_166897_object67537:head# 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62e55a19:::benchmark_data_smithi090_166897_object52423:head# 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62e60251:::benchmark_data_smithi090_166897_object9247:head# 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62e64f63:::benchmark_data_smithi090_166897_object46327:head# 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62ea3e98:::benchmark_data_smithi090_166897_object22907:head# 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62ec3fdb:::benchmark_data_smithi090_166897_object28140:head# 2024-05-15T23:40:13.789 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.790 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62ec68dd:::benchmark_data_smithi090_166897_object18386:head# 2024-05-15T23:40:13.790 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.790 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62ec78c3:::benchmark_data_smithi090_166897_object55419:head# 2024-05-15T23:40:13.790 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.790 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62ece5c4:::benchmark_data_smithi090_166897_object44967:head# 2024-05-15T23:40:13.790 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.790 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62ed6e16:::benchmark_data_smithi090_166897_object7159:head# 2024-05-15T23:40:13.790 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.790 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62eda42f:::benchmark_data_smithi090_166897_object59526:head# 2024-05-15T23:40:13.790 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.790 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62f1a6c3:::benchmark_data_smithi090_166897_object18495:head# 2024-05-15T23:40:13.790 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.790 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62f3f782:::benchmark_data_smithi090_166897_object65770:head# 2024-05-15T23:40:13.790 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.791 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62f53682:::benchmark_data_smithi090_166897_object23336:head# 2024-05-15T23:40:13.791 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.791 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62f5f83b:::benchmark_data_smithi090_166897_object69832:head# 2024-05-15T23:40:13.791 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.791 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62f6c6e3:::benchmark_data_smithi090_166897_object56366:head# 2024-05-15T23:40:13.791 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.791 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62f7bd0f:::benchmark_data_smithi090_166897_object6456:head# 2024-05-15T23:40:13.791 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.791 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62f7cf6e:::benchmark_data_smithi090_166897_object21662:head# 2024-05-15T23:40:13.791 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.791 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62f8438a:::benchmark_data_smithi090_166897_object47901:head# 2024-05-15T23:40:13.791 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.791 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62fa3249:::benchmark_data_smithi090_166897_object3580:head# 2024-05-15T23:40:13.791 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.791 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62fa6bad:::benchmark_data_smithi090_166897_object45862:head# 2024-05-15T23:40:13.792 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.792 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62fbbab2:::benchmark_data_smithi090_166897_object13966:head# 2024-05-15T23:40:13.792 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.792 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62fc338d:::benchmark_data_smithi090_166897_object1201:head# 2024-05-15T23:40:13.792 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.792 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62fd5662:::benchmark_data_smithi090_166897_object40867:head# 2024-05-15T23:40:13.792 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.792 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62ff4473:::benchmark_data_smithi090_166897_object560:head# 2024-05-15T23:40:13.792 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.792 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:62ff9347:::benchmark_data_smithi090_166897_object9577:head# 2024-05-15T23:40:13.792 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.792 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:630099ee:::benchmark_data_smithi090_166897_object53319:head# 2024-05-15T23:40:13.792 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.792 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6301a66c:::benchmark_data_smithi090_166897_object41112:head# 2024-05-15T23:40:13.792 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.793 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63061539:::benchmark_data_smithi090_166897_object53211:head# 2024-05-15T23:40:13.793 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.793 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63066a2f:::benchmark_data_smithi090_166897_object28398:head# 2024-05-15T23:40:13.793 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.793 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63079dc7:::benchmark_data_smithi090_166897_object22552:head# 2024-05-15T23:40:13.793 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.793 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63079e1d:::benchmark_data_smithi090_166897_object6374:head# 2024-05-15T23:40:13.793 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.793 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6307b2c1:::benchmark_data_smithi090_166897_object60793:head# 2024-05-15T23:40:13.793 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.793 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6307dc83:::benchmark_data_smithi090_166897_object46803:head# 2024-05-15T23:40:13.793 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.793 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:630a9d03:::benchmark_data_smithi090_166897_object20362:head# 2024-05-15T23:40:13.793 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.793 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:630ad801:::benchmark_data_smithi090_166897_object51545:head# 2024-05-15T23:40:13.794 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.794 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:630b62fa:::benchmark_data_smithi090_166897_object68911:head# 2024-05-15T23:40:13.794 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.794 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:630bbfcc:::benchmark_data_smithi090_166897_object18952:head# 2024-05-15T23:40:13.794 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.794 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:630c358b:::benchmark_data_smithi090_166897_object24920:head# 2024-05-15T23:40:13.794 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.794 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:630cfac4:::benchmark_data_smithi090_166897_object27135:head# 2024-05-15T23:40:13.794 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.794 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:630ec903:::benchmark_data_smithi090_166897_object46196:head# 2024-05-15T23:40:13.794 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.794 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:630ee629:::benchmark_data_smithi090_166897_object29007:head# 2024-05-15T23:40:13.794 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.794 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:630f2fb4:::benchmark_data_smithi090_166897_object52331:head# 2024-05-15T23:40:13.794 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.795 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63103015:::benchmark_data_smithi090_166897_object57229:head# 2024-05-15T23:40:13.795 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.822 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631103e9:::benchmark_data_smithi090_166897_object70386:head# 2024-05-15T23:40:13.822 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.822 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6311cad7:::benchmark_data_smithi090_166897_object49215:head# 2024-05-15T23:40:13.822 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.822 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63122c2b:::benchmark_data_smithi090_166897_object41749:head# 2024-05-15T23:40:13.822 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.822 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63132eb5:::benchmark_data_smithi090_166897_object1920:head# 2024-05-15T23:40:13.822 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.823 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63150063:::benchmark_data_smithi090_166897_object32801:head# 2024-05-15T23:40:13.823 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.823 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6315e7cc:::benchmark_data_smithi090_166897_object7382:head# 2024-05-15T23:40:13.823 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.823 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6316f73f:::benchmark_data_smithi090_166897_object55833:head# 2024-05-15T23:40:13.823 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.823 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6317494b:::benchmark_data_smithi090_166897_object16842:head# 2024-05-15T23:40:13.823 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.823 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63175360:::benchmark_data_smithi090_166897_object41437:head# 2024-05-15T23:40:13.823 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.823 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63178ed6:::benchmark_data_smithi090_166897_object7613:head# 2024-05-15T23:40:13.823 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.823 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6317d64d:::benchmark_data_smithi090_166897_object50639:head# 2024-05-15T23:40:13.823 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.824 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631803ab:::benchmark_data_smithi090_166897_object23449:head# 2024-05-15T23:40:13.824 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.824 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6318f42b:::benchmark_data_smithi090_166897_object28116:head# 2024-05-15T23:40:13.824 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.824 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6319e121:::benchmark_data_smithi090_166897_object41291:head# 2024-05-15T23:40:13.824 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.824 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631ae23d:::benchmark_data_smithi090_166897_object7813:head# 2024-05-15T23:40:13.824 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.824 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631afd4a:::benchmark_data_smithi090_166897_object39351:head# 2024-05-15T23:40:13.824 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.824 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631bb9d4:::benchmark_data_smithi090_166897_object28876:head# 2024-05-15T23:40:13.824 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.824 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631bcac3:::benchmark_data_smithi090_166897_object48875:head# 2024-05-15T23:40:13.824 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.824 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631c3b77:::benchmark_data_smithi090_166897_object39884:head# 2024-05-15T23:40:13.825 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.825 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631ddc55:::benchmark_data_smithi090_166897_object68419:head# 2024-05-15T23:40:13.825 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.825 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631dec7b:::benchmark_data_smithi090_166897_object21794:head# 2024-05-15T23:40:13.825 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.825 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631defc0:::benchmark_data_smithi090_166897_object59239:head# 2024-05-15T23:40:13.825 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.825 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631e5437:::benchmark_data_smithi090_166897_object8813:head# 2024-05-15T23:40:13.825 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.825 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631ebd85:::benchmark_data_smithi090_166897_object45651:head# 2024-05-15T23:40:13.825 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.825 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631f2883:::benchmark_data_smithi090_166897_object45067:head# 2024-05-15T23:40:13.825 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.825 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631fa7b1:::benchmark_data_smithi090_166897_object42035:head# 2024-05-15T23:40:13.825 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.826 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631fed18:::benchmark_data_smithi090_166897_object70650:head# 2024-05-15T23:40:13.826 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.826 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:631ff41b:::benchmark_data_smithi090_166897_object41405:head# 2024-05-15T23:40:13.826 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.826 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63208fe3:::benchmark_data_smithi090_166897_object2626:head# 2024-05-15T23:40:13.826 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.826 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6320e17a:::benchmark_data_smithi090_166897_object877:head# 2024-05-15T23:40:13.826 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.826 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63212502:::benchmark_data_smithi090_166897_object6575:head# 2024-05-15T23:40:13.826 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.826 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6321ca7b:::benchmark_data_smithi090_166897_object9558:head# 2024-05-15T23:40:13.826 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.826 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6321e583:::benchmark_data_smithi090_166897_object45576:head# 2024-05-15T23:40:13.826 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.827 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6323c680:::benchmark_data_smithi090_166897_object51798:head# 2024-05-15T23:40:13.827 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.827 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63245c4f:::benchmark_data_smithi090_166897_object58446:head# 2024-05-15T23:40:13.827 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.827 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6325f676:::benchmark_data_smithi090_166897_object29524:head# 2024-05-15T23:40:13.827 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.827 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:632659ba:::benchmark_data_smithi090_166897_object34387:head# 2024-05-15T23:40:13.827 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.827 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:632689d4:::benchmark_data_smithi090_166897_object57211:head# 2024-05-15T23:40:13.827 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.827 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63282431:::benchmark_data_smithi090_166897_object7533:head# 2024-05-15T23:40:13.827 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.827 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63290241:::benchmark_data_smithi090_166897_object29837:head# 2024-05-15T23:40:13.827 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.827 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:632b4c44:::benchmark_data_smithi090_166897_object34837:head# 2024-05-15T23:40:13.828 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.828 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:632b7827:::benchmark_data_smithi090_166897_object18132:head# 2024-05-15T23:40:13.828 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.828 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:632bbf8d:::benchmark_data_smithi090_166897_object64731:head# 2024-05-15T23:40:13.828 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.828 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:632c062b:::benchmark_data_smithi090_166897_object39054:head# 2024-05-15T23:40:13.828 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.828 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:632de2bf:::benchmark_data_smithi090_166897_object38200:head# 2024-05-15T23:40:13.828 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.828 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:632e672a:::benchmark_data_smithi090_166897_object45993:head# 2024-05-15T23:40:13.828 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.828 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:632ee713:::benchmark_data_smithi090_166897_object12829:head# 2024-05-15T23:40:13.828 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.828 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:632f68f0:::benchmark_data_smithi090_166897_object53484:head# 2024-05-15T23:40:13.829 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.829 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:632fed66:::benchmark_data_smithi090_166897_object15403:head# 2024-05-15T23:40:13.829 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.829 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6330859f:::benchmark_data_smithi090_166897_object52995:head# 2024-05-15T23:40:13.829 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.829 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6332a1c9:::benchmark_data_smithi090_166897_object56816:head# 2024-05-15T23:40:13.829 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.829 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63332286:::benchmark_data_smithi090_166897_object3635:head# 2024-05-15T23:40:13.829 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.829 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63338271:::benchmark_data_smithi090_166897_object61281:head# 2024-05-15T23:40:13.829 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.829 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63356f34:::benchmark_data_smithi090_166897_object35104:head# 2024-05-15T23:40:13.829 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.829 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6335ca85:::benchmark_data_smithi090_166897_object40917:head# 2024-05-15T23:40:13.829 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.830 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:633630ee:::benchmark_data_smithi090_166897_object38386:head# 2024-05-15T23:40:13.830 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.830 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63367421:::benchmark_data_smithi090_166897_object49091:head# 2024-05-15T23:40:13.830 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.830 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6337d7bc:::benchmark_data_smithi090_166897_object29814:head# 2024-05-15T23:40:13.830 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.830 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6339f4d4:::benchmark_data_smithi090_166897_object18299:head# 2024-05-15T23:40:13.830 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.830 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:633a4cd7:::benchmark_data_smithi090_166897_object37995:head# 2024-05-15T23:40:13.830 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.830 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:633b0cf4:::benchmark_data_smithi090_166897_object49439:head# 2024-05-15T23:40:13.830 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.830 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:633b30ea:::benchmark_data_smithi090_166897_object46440:head# 2024-05-15T23:40:13.830 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.830 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:633d5154:::benchmark_data_smithi090_166897_object37699:head# 2024-05-15T23:40:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:633d9048:::benchmark_data_smithi090_166897_object50359:head# 2024-05-15T23:40:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:633da023:::benchmark_data_smithi090_166897_object60511:head# 2024-05-15T23:40:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:633de340:::benchmark_data_smithi090_166897_object59465:head# 2024-05-15T23:40:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:633f5937:::benchmark_data_smithi090_166897_object31189:head# 2024-05-15T23:40:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:633fb536:::benchmark_data_smithi090_166897_object51441:head# 2024-05-15T23:40:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:633fbe1c:::benchmark_data_smithi090_166897_object9185:head# 2024-05-15T23:40:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:634007ff:::benchmark_data_smithi090_166897_object5700:head# 2024-05-15T23:40:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.832 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6340b54d:::benchmark_data_smithi090_166897_object44601:head# 2024-05-15T23:40:13.832 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.832 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6341f49a:::benchmark_data_smithi090_166897_object41408:head# 2024-05-15T23:40:13.832 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.832 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6342238f:::benchmark_data_smithi090_166897_object21851:head# 2024-05-15T23:40:13.832 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.832 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:634843be:::benchmark_data_smithi090_166897_object55245:head# 2024-05-15T23:40:13.832 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.832 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:634844a8:::benchmark_data_smithi090_166897_object14382:head# 2024-05-15T23:40:13.832 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.832 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63488b4c:::benchmark_data_smithi090_166897_object69930:head# 2024-05-15T23:40:13.832 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.832 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6349b5cf:::benchmark_data_smithi090_166897_object725:head# 2024-05-15T23:40:13.832 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.832 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:634aa6d1:::benchmark_data_smithi090_166897_object25127:head# 2024-05-15T23:40:13.833 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.833 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:634b1ec1:::benchmark_data_smithi090_166897_object55540:head# 2024-05-15T23:40:13.833 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.833 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:634e1c2f:::benchmark_data_smithi090_166897_object35946:head# 2024-05-15T23:40:13.833 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.833 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63520c8e:::benchmark_data_smithi090_166897_object7678:head# 2024-05-15T23:40:13.833 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.833 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63560216:::benchmark_data_smithi090_166897_object19496:head# 2024-05-15T23:40:13.833 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.833 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6356e8d0:::benchmark_data_smithi090_166897_object12612:head# 2024-05-15T23:40:13.833 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.833 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63575273:::benchmark_data_smithi090_166897_object25455:head# 2024-05-15T23:40:13.833 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.833 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63575825:::benchmark_data_smithi090_166897_object35013:head# 2024-05-15T23:40:13.833 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.834 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6357bad5:::benchmark_data_smithi090_166897_object22421:head# 2024-05-15T23:40:13.834 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.834 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6359a4bd:::benchmark_data_smithi090_166897_object54111:head# 2024-05-15T23:40:13.834 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.834 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:635a7354:::benchmark_data_smithi090_166897_object24989:head# 2024-05-15T23:40:13.834 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.834 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:635b7e20:::benchmark_data_smithi090_166897_object60014:head# 2024-05-15T23:40:13.834 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.834 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:635bb402:::benchmark_data_smithi090_166897_object62621:head# 2024-05-15T23:40:13.834 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.834 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:635e2239:::benchmark_data_smithi090_166897_object43857:head# 2024-05-15T23:40:13.834 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.834 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:635eb67e:::benchmark_data_smithi090_166897_object31544:head# 2024-05-15T23:40:13.834 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.834 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:635f1c32:::benchmark_data_smithi090_166897_object31233:head# 2024-05-15T23:40:13.835 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.835 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:635f4ab5:::benchmark_data_smithi090_166897_object55509:head# 2024-05-15T23:40:13.835 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.835 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:636020b2:::benchmark_data_smithi090_166897_object12018:head# 2024-05-15T23:40:13.835 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.835 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6360db69:::benchmark_data_smithi090_166897_object69161:head# 2024-05-15T23:40:13.835 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.835 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63626b06:::benchmark_data_smithi090_166897_object28385:head# 2024-05-15T23:40:13.835 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.835 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6362addb:::benchmark_data_smithi090_166897_object44886:head# 2024-05-15T23:40:13.835 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.835 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:636439d7:::benchmark_data_smithi090_166897_object13738:head# 2024-05-15T23:40:13.835 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.835 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6364f0a9:::benchmark_data_smithi090_166897_object2353:head# 2024-05-15T23:40:13.836 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.836 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63664822:::benchmark_data_smithi090_166897_object18451:head# 2024-05-15T23:40:13.836 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.836 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6366f6ab:::benchmark_data_smithi090_166897_object49320:head# 2024-05-15T23:40:13.836 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.836 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6367854f:::benchmark_data_smithi090_166897_object41192:head# 2024-05-15T23:40:13.836 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.836 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63699a3e:::benchmark_data_smithi090_166897_object6289:head# 2024-05-15T23:40:13.836 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.836 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:636b1585:::benchmark_data_smithi090_166897_object8858:head# 2024-05-15T23:40:13.836 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.836 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:636bf47e:::benchmark_data_smithi090_166897_object24114:head# 2024-05-15T23:40:13.836 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.836 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:636c084c:::benchmark_data_smithi090_166897_object65463:head# 2024-05-15T23:40:13.836 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.837 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:636c4cc6:::benchmark_data_smithi090_166897_object22031:head# 2024-05-15T23:40:13.837 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.837 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63704495:::benchmark_data_smithi090_166897_object58784:head# 2024-05-15T23:40:13.837 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.837 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:637170f1:::benchmark_data_smithi090_166897_object16789:head# 2024-05-15T23:40:13.837 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.837 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6371e83e:::benchmark_data_smithi090_166897_object29644:head# 2024-05-15T23:40:13.837 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.837 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6372177b:::benchmark_data_smithi090_166897_object63193:head# 2024-05-15T23:40:13.837 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.837 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6372cef8:::benchmark_data_smithi090_166897_object51576:head# 2024-05-15T23:40:13.837 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.837 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6372d8e8:::benchmark_data_smithi090_166897_object27521:head# 2024-05-15T23:40:13.837 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.837 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:637368e6:::benchmark_data_smithi090_166897_object55711:head# 2024-05-15T23:40:13.838 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.838 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63738a7c:::benchmark_data_smithi090_166897_object5121:head# 2024-05-15T23:40:13.838 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.838 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6376dd5e:::benchmark_data_smithi090_166897_object25391:head# 2024-05-15T23:40:13.838 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.838 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6378c762:::benchmark_data_smithi090_166897_object43035:head# 2024-05-15T23:40:13.838 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.838 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6379dc50:::benchmark_data_smithi090_166897_object39075:head# 2024-05-15T23:40:13.838 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.838 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:637c0741:::benchmark_data_smithi090_166897_object51428:head# 2024-05-15T23:40:13.838 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.838 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:637c498e:::benchmark_data_smithi090_166897_object1889:head# 2024-05-15T23:40:13.838 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.838 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:637df01d:::benchmark_data_smithi090_166897_object2841:head# 2024-05-15T23:40:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6381700a:::benchmark_data_smithi090_166897_object3800:head# 2024-05-15T23:40:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63835146:::benchmark_data_smithi090_166897_object15:head# 2024-05-15T23:40:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6385428f:::benchmark_data_smithi090_166897_object47695:head# 2024-05-15T23:40:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63867c68:::benchmark_data_smithi090_166897_object51656:head# 2024-05-15T23:40:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6386a3d9:::benchmark_data_smithi090_166897_object45445:head# 2024-05-15T23:40:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63875d49:::benchmark_data_smithi090_166897_object11360:head# 2024-05-15T23:40:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:638941f5:::benchmark_data_smithi090_166897_object5523:head# 2024-05-15T23:40:13.839 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:638ab0aa:::benchmark_data_smithi090_166897_object43037:head# 2024-05-15T23:40:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:638b0ccd:::benchmark_data_smithi090_166897_object9909:head# 2024-05-15T23:40:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:638b1611:::benchmark_data_smithi090_166897_object59044:head# 2024-05-15T23:40:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:638bbe5d:::benchmark_data_smithi090_166897_object676:head# 2024-05-15T23:40:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:638d4d48:::benchmark_data_smithi090_166897_object33193:head# 2024-05-15T23:40:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:638d54d9:::benchmark_data_smithi090_166897_object40216:head# 2024-05-15T23:40:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63902368:::benchmark_data_smithi090_166897_object33501:head# 2024-05-15T23:40:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6391832b:::benchmark_data_smithi090_166897_object49961:head# 2024-05-15T23:40:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63932339:::benchmark_data_smithi090_166897_object27606:head# 2024-05-15T23:40:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63935979:::benchmark_data_smithi090_166897_object36472:head# 2024-05-15T23:40:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6393a2f2:::benchmark_data_smithi090_166897_object16098:head# 2024-05-15T23:40:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63949937:::benchmark_data_smithi090_166897_object27018:head# 2024-05-15T23:40:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6395978e:::benchmark_data_smithi090_166897_object10493:head# 2024-05-15T23:40:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63970c7c:::benchmark_data_smithi090_166897_object10852:head# 2024-05-15T23:40:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6397a030:::benchmark_data_smithi090_166897_object39433:head# 2024-05-15T23:40:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.842 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63996223:::benchmark_data_smithi090_166897_object35994:head# 2024-05-15T23:40:13.842 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.842 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63996e24:::benchmark_data_smithi090_166897_object47796:head# 2024-05-15T23:40:13.842 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.842 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6399b152:::benchmark_data_smithi090_166897_object59303:head# 2024-05-15T23:40:13.842 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.842 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6399b1d5:::benchmark_data_smithi090_166897_object27543:head# 2024-05-15T23:40:13.842 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.842 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:639aaca9:::benchmark_data_smithi090_166897_object16295:head# 2024-05-15T23:40:13.842 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.842 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:639ad1aa:::benchmark_data_smithi090_166897_object25462:head# 2024-05-15T23:40:13.842 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.842 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:639c1efa:::benchmark_data_smithi090_166897_object13624:head# 2024-05-15T23:40:13.842 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.843 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:639cdbf2:::benchmark_data_smithi090_166897_object59510:head# 2024-05-15T23:40:13.843 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.843 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63a1e757:::benchmark_data_smithi090_166897_object33531:head# 2024-05-15T23:40:13.843 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.843 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63a2b776:::benchmark_data_smithi090_166897_object67136:head# 2024-05-15T23:40:13.843 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.843 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63a3733e:::benchmark_data_smithi090_166897_object10785:head# 2024-05-15T23:40:13.843 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.843 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63a45322:::benchmark_data_smithi090_166897_object55757:head# 2024-05-15T23:40:13.843 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.843 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63a71878:::benchmark_data_smithi090_166897_object59270:head# 2024-05-15T23:40:13.843 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.843 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63ab1c50:::benchmark_data_smithi090_166897_object37826:head# 2024-05-15T23:40:13.843 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.843 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63ab7a50:::benchmark_data_smithi090_166897_object39337:head# 2024-05-15T23:40:13.844 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.844 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63ac39a4:::benchmark_data_smithi090_166897_object26954:head# 2024-05-15T23:40:13.844 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.844 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63acebb7:::benchmark_data_smithi090_166897_object22321:head# 2024-05-15T23:40:13.844 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.844 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63ada610:::benchmark_data_smithi090_166897_object34423:head# 2024-05-15T23:40:13.844 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.844 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63ae00bd:::benchmark_data_smithi090_166897_object58686:head# 2024-05-15T23:40:13.844 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.844 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63af20c5:::benchmark_data_smithi090_166897_object17076:head# 2024-05-15T23:40:13.844 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.844 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63afef5a:::benchmark_data_smithi090_166897_object32467:head# 2024-05-15T23:40:13.844 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.844 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63b0125d:::benchmark_data_smithi090_166897_object14502:head# 2024-05-15T23:40:13.844 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63b193b8:::benchmark_data_smithi090_166897_object54473:head# 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63b22f1f:::benchmark_data_smithi090_166897_object18063:head# 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63b2941a:::benchmark_data_smithi090_166897_object31502:head# 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63b2d861:::benchmark_data_smithi090_166897_object26514:head# 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63b37ed2:::benchmark_data_smithi090_166897_object56336:head# 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63b455b7:::benchmark_data_smithi090_166897_object53445:head# 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63b47313:::benchmark_data_smithi090_166897_object4411:head# 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63b593ba:::benchmark_data_smithi090_166897_object27304:head# 2024-05-15T23:40:13.845 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.846 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63b67a4a:::benchmark_data_smithi090_166897_object55155:head# 2024-05-15T23:40:13.846 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.846 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63b7f740:::benchmark_data_smithi090_166897_object51410:head# 2024-05-15T23:40:13.846 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.846 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63b97d08:::benchmark_data_smithi090_166897_object37620:head# 2024-05-15T23:40:13.846 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.846 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63b9d69b:::benchmark_data_smithi090_166897_object34295:head# 2024-05-15T23:40:13.846 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.846 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63bb2b69:::benchmark_data_smithi090_166897_object4092:head# 2024-05-15T23:40:13.846 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.846 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63bb4af5:::benchmark_data_smithi090_166897_object40559:head# 2024-05-15T23:40:13.846 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.846 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63bb738a:::benchmark_data_smithi090_166897_object6215:head# 2024-05-15T23:40:13.846 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.847 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63bc21d1:::benchmark_data_smithi090_166897_object59196:head# 2024-05-15T23:40:13.847 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.847 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63bcd852:::benchmark_data_smithi090_166897_object55551:head# 2024-05-15T23:40:13.847 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.847 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63bef727:::benchmark_data_smithi090_166897_object5693:head# 2024-05-15T23:40:13.847 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.847 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63bf3ec6:::benchmark_data_smithi090_166897_object7398:head# 2024-05-15T23:40:13.847 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.847 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63c0608d:::benchmark_data_smithi090_166897_object5550:head# 2024-05-15T23:40:13.847 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.847 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63c0db89:::benchmark_data_smithi090_166897_object55603:head# 2024-05-15T23:40:13.847 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.847 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63c18e2b:::benchmark_data_smithi090_166897_object18806:head# 2024-05-15T23:40:13.847 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.847 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63c1c4e6:::benchmark_data_smithi090_166897_object52638:head# 2024-05-15T23:40:13.848 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.848 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63c283b2:::benchmark_data_smithi090_166897_object40630:head# 2024-05-15T23:40:13.848 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.848 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63c4b039:::benchmark_data_smithi090_166897_object21191:head# 2024-05-15T23:40:13.848 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.848 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63c51acd:::benchmark_data_smithi090_166897_object37997:head# 2024-05-15T23:40:13.848 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.848 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63c5b99e:::benchmark_data_smithi090_166897_object29770:head# 2024-05-15T23:40:13.848 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.848 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63ca3527:::benchmark_data_smithi090_166897_object44978:head# 2024-05-15T23:40:13.848 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.848 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:40:13.884 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63ca6618:::benchmark_data_smithi090_166897_object52337:head# 2024-05-15T23:40:13.884 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.884 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63caf2e3:::benchmark_data_smithi090_166897_object30502:head# 2024-05-15T23:40:13.884 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.884 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63cb00cb:::benchmark_data_smithi090_166897_object48076:head# 2024-05-15T23:40:13.884 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.884 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63cb78f1:::benchmark_data_smithi090_166897_object36151:head# 2024-05-15T23:40:13.884 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.884 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63cd89de:::benchmark_data_smithi090_166897_object67781:head# 2024-05-15T23:40:13.884 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.884 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63cdb2a2:::benchmark_data_smithi090_166897_object16950:head# 2024-05-15T23:40:13.885 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.885 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63ce48c9:::benchmark_data_smithi090_166897_object33220:head# 2024-05-15T23:40:13.885 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.885 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63ce9312:::benchmark_data_smithi090_166897_object6360:head# 2024-05-15T23:40:13.885 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.885 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63ceefaf:::benchmark_data_smithi090_166897_object50150:head# 2024-05-15T23:40:13.885 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.885 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63d04e8a:::benchmark_data_smithi090_166897_object55974:head# 2024-05-15T23:40:13.885 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.885 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63d07723:::benchmark_data_smithi090_166897_object45957:head# 2024-05-15T23:40:13.885 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.885 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63d0a371:::benchmark_data_smithi090_166897_object67717:head# 2024-05-15T23:40:13.885 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.885 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63d14582:::benchmark_data_smithi090_166897_object29791:head# 2024-05-15T23:40:13.886 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.886 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63d18984:::benchmark_data_smithi090_166897_object45547:head# 2024-05-15T23:40:13.886 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.886 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63d33192:::benchmark_data_smithi090_166897_object3038:head# 2024-05-15T23:40:13.886 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.886 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63d3701f:::benchmark_data_smithi090_166897_object22098:head# 2024-05-15T23:40:13.886 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.886 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63d3aad4:::benchmark_data_smithi090_166897_object58024:head# 2024-05-15T23:40:13.886 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.886 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63d4dc12:::benchmark_data_smithi090_166897_object57046:head# 2024-05-15T23:40:13.886 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.886 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63d52d16:::benchmark_data_smithi090_166897_object11107:head# 2024-05-15T23:40:13.886 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.886 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63d6395b:::benchmark_data_smithi090_166897_object40079:head# 2024-05-15T23:40:13.887 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.887 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63d879ee:::benchmark_data_smithi090_166897_object53210:head# 2024-05-15T23:40:13.887 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.887 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63d88a69:::benchmark_data_smithi090_166897_object10505:head# 2024-05-15T23:40:13.887 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.887 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63d9ca33:::benchmark_data_smithi090_166897_object2996:head# 2024-05-15T23:40:13.887 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.887 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63d9dc3e:::benchmark_data_smithi090_166897_object26875:head# 2024-05-15T23:40:13.887 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.887 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63dc5baa:::benchmark_data_smithi090_166897_object58614:head# 2024-05-15T23:40:13.887 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.887 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63de1ad9:::benchmark_data_smithi090_166897_object49522:head# 2024-05-15T23:40:13.887 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.887 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63df0b36:::benchmark_data_smithi090_166897_object59055:head# 2024-05-15T23:40:13.888 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.888 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63dfdd8d:::benchmark_data_smithi090_166897_object1096:head# 2024-05-15T23:40:13.888 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.888 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63e008cb:::benchmark_data_smithi090_166897_object53093:head# 2024-05-15T23:40:13.888 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.888 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63e05f28:::benchmark_data_smithi090_166897_object53654:head# 2024-05-15T23:40:13.888 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.888 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63e19b70:::benchmark_data_smithi090_166897_object1926:head# 2024-05-15T23:40:13.888 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.888 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63e25d7a:::benchmark_data_smithi090_166897_object68878:head# 2024-05-15T23:40:13.888 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.888 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63e294f8:::benchmark_data_smithi090_166897_object15601:head# 2024-05-15T23:40:13.888 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.888 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63e3af5c:::benchmark_data_smithi090_166897_object4838:head# 2024-05-15T23:40:13.889 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.889 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63e4a2f1:::benchmark_data_smithi090_166897_object66854:head# 2024-05-15T23:40:13.889 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.889 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63e6995e:::benchmark_data_smithi090_166897_object69347:head# 2024-05-15T23:40:13.889 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.889 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63e6ffd0:::benchmark_data_smithi090_166897_object64464:head# 2024-05-15T23:40:13.889 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.889 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63ea23f0:::benchmark_data_smithi090_166897_object54633:head# 2024-05-15T23:40:13.889 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.889 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63ebc0c0:::benchmark_data_smithi090_166897_object53963:head# 2024-05-15T23:40:13.889 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.889 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63ef5816:::benchmark_data_smithi090_166897_object61083:head# 2024-05-15T23:40:13.889 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.889 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63efd570:::benchmark_data_smithi090_166897_object17916:head# 2024-05-15T23:40:13.890 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.890 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63f02f78:::benchmark_data_smithi090_166897_object22570:head# 2024-05-15T23:40:13.890 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.890 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63f0a532:::benchmark_data_smithi090_166897_object35128:head# 2024-05-15T23:40:13.890 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.890 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63f0f552:::benchmark_data_smithi090_166897_object13961:head# 2024-05-15T23:40:13.890 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.890 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63f1392f:::benchmark_data_smithi090_166897_object40096:head# 2024-05-15T23:40:13.890 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.890 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63f1465e:::benchmark_data_smithi090_166897_object18539:head# 2024-05-15T23:40:13.890 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.890 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63f1c1e2:::benchmark_data_smithi090_166897_object69439:head# 2024-05-15T23:40:13.890 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.890 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63f2b62f:::benchmark_data_smithi090_166897_object19201:head# 2024-05-15T23:40:13.891 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.891 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63f2b74c:::benchmark_data_smithi090_166897_object42875:head# 2024-05-15T23:40:13.891 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.891 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63f437a2:::benchmark_data_smithi090_166897_object5386:head# 2024-05-15T23:40:13.891 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.891 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63f440cb:::benchmark_data_smithi090_166897_object30563:head# 2024-05-15T23:40:13.891 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.891 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63f5580e:::benchmark_data_smithi090_166897_object19538:head# 2024-05-15T23:40:13.891 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.891 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63f6baef:::benchmark_data_smithi090_166897_object61780:head# 2024-05-15T23:40:13.891 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.891 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63f6d10e:::benchmark_data_smithi090_166897_object66961:head# 2024-05-15T23:40:13.891 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.891 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63f8eb13:::benchmark_data_smithi090_166897_object64262:head# 2024-05-15T23:40:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63fd8a90:::benchmark_data_smithi090_166897_object59958:head# 2024-05-15T23:40:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63fdf5e9:::benchmark_data_smithi090_166897_object43715:head# 2024-05-15T23:40:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:63fe75ee:::benchmark_data_smithi090_166897_object50581:head# 2024-05-15T23:40:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64026ed7:::benchmark_data_smithi090_166897_object13963:head# 2024-05-15T23:40:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:640388ea:::benchmark_data_smithi090_166897_object45788:head# 2024-05-15T23:40:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6404d5fc:::benchmark_data_smithi090_166897_object32327:head# 2024-05-15T23:40:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64062d08:::benchmark_data_smithi090_166897_object44846:head# 2024-05-15T23:40:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.893 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6406bda5:::benchmark_data_smithi090_166897_object19834:head# 2024-05-15T23:40:13.893 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.893 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6407153a:::benchmark_data_smithi090_166897_object67903:head# 2024-05-15T23:40:13.893 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.893 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6407962d:::benchmark_data_smithi090_166897_object66316:head# 2024-05-15T23:40:13.893 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.893 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6407ca25:::benchmark_data_smithi090_166897_object59032:head# 2024-05-15T23:40:13.893 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.893 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64085541:::benchmark_data_smithi090_166897_object38174:head# 2024-05-15T23:40:13.893 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.893 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:640deb9e:::benchmark_data_smithi090_166897_object41197:head# 2024-05-15T23:40:13.893 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.893 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:640fac1d:::benchmark_data_smithi090_166897_object65189:head# 2024-05-15T23:40:13.893 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.893 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6410564e:::benchmark_data_smithi090_166897_object19901:head# 2024-05-15T23:40:13.894 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.894 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6412e03e:::benchmark_data_smithi090_166897_object17318:head# 2024-05-15T23:40:13.894 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.894 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64130c63:::benchmark_data_smithi090_166897_object46537:head# 2024-05-15T23:40:13.894 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.894 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64133228:::benchmark_data_smithi090_166897_object15917:head# 2024-05-15T23:40:13.894 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.894 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64145af9:::benchmark_data_smithi090_166897_object40192:head# 2024-05-15T23:40:13.894 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.894 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6414add8:::benchmark_data_smithi090_166897_object31673:head# 2024-05-15T23:40:13.894 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.894 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64160372:::benchmark_data_smithi090_166897_object62026:head# 2024-05-15T23:40:13.894 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.894 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:641682ce:::benchmark_data_smithi090_166897_object43368:head# 2024-05-15T23:40:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6416b347:::benchmark_data_smithi090_166897_object35254:head# 2024-05-15T23:40:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64174cac:::benchmark_data_smithi090_166897_object53474:head# 2024-05-15T23:40:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6417c941:::benchmark_data_smithi090_166897_object37594:head# 2024-05-15T23:40:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:641b2f82:::benchmark_data_smithi090_166897_object21982:head# 2024-05-15T23:40:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:641ba343:::benchmark_data_smithi090_166897_object32986:head# 2024-05-15T23:40:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:641bb5fb:::benchmark_data_smithi090_166897_object47798:head# 2024-05-15T23:40:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:641c34f1:::benchmark_data_smithi090_166897_object41220:head# 2024-05-15T23:40:13.895 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.896 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:641c56e2:::benchmark_data_smithi090_166897_object39494:head# 2024-05-15T23:40:13.896 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.896 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:641ca232:::benchmark_data_smithi090_166897_object2940:head# 2024-05-15T23:40:13.896 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.896 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:641d1b6b:::benchmark_data_smithi090_166897_object28683:head# 2024-05-15T23:40:13.896 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.896 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:641ee0a3:::benchmark_data_smithi090_166897_object19662:head# 2024-05-15T23:40:13.896 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.896 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:641f40bb:::benchmark_data_smithi090_166897_object27015:head# 2024-05-15T23:40:13.896 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.896 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:641f6abd:::benchmark_data_smithi090_166897_object55644:head# 2024-05-15T23:40:13.896 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.896 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:642157ea:::benchmark_data_smithi090_166897_object7350:head# 2024-05-15T23:40:13.896 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.896 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64215fd1:::benchmark_data_smithi090_166897_object10084:head# 2024-05-15T23:40:13.897 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.897 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6421c6c9:::benchmark_data_smithi090_166897_object10847:head# 2024-05-15T23:40:13.897 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.897 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6421f663:::benchmark_data_smithi090_166897_object33438:head# 2024-05-15T23:40:13.897 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.897 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64225ac0:::benchmark_data_smithi090_166897_object4122:head# 2024-05-15T23:40:13.897 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.897 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64234076:::benchmark_data_smithi090_166897_object4255:head# 2024-05-15T23:40:13.897 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.897 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64234640:::benchmark_data_smithi090_166897_object63832:head# 2024-05-15T23:40:13.897 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.897 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64255162:::benchmark_data_smithi090_166897_object15309:head# 2024-05-15T23:40:13.897 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.897 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6426de6a:::benchmark_data_smithi090_166897_object47523:head# 2024-05-15T23:40:13.898 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.898 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6426fa48:::benchmark_data_smithi090_166897_object22724:head# 2024-05-15T23:40:13.898 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.898 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6427fbea:::benchmark_data_smithi090_166897_object1537:head# 2024-05-15T23:40:13.898 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.898 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6429ca9a:::benchmark_data_smithi090_166897_object35114:head# 2024-05-15T23:40:13.898 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.898 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6429f8d4:::benchmark_data_smithi090_166897_object31813:head# 2024-05-15T23:40:13.898 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.898 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:642ca5c4:::benchmark_data_smithi090_166897_object17155:head# 2024-05-15T23:40:13.898 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.898 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:642cbcdf:::benchmark_data_smithi090_166897_object49294:head# 2024-05-15T23:40:13.898 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.898 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:642d200b:::benchmark_data_smithi090_166897_object25548:head# 2024-05-15T23:40:13.899 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.899 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6430f793:::benchmark_data_smithi090_166897_object32238:head# 2024-05-15T23:40:13.899 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.899 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6431159a:::benchmark_data_smithi090_166897_object7250:head# 2024-05-15T23:40:13.899 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.899 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:643174e3:::benchmark_data_smithi090_166897_object42662:head# 2024-05-15T23:40:13.899 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.899 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6433518f:::benchmark_data_smithi090_166897_object36325:head# 2024-05-15T23:40:13.899 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.899 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6433babc:::benchmark_data_smithi090_166897_object54901:head# 2024-05-15T23:40:13.899 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.899 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64354abb:::benchmark_data_smithi090_166897_object66664:head# 2024-05-15T23:40:13.899 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.899 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64354d1b:::benchmark_data_smithi090_166897_object29188:head# 2024-05-15T23:40:13.899 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.900 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6435f2b0:::benchmark_data_smithi090_166897_object3816:head# 2024-05-15T23:40:13.900 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.900 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6437deef:::benchmark_data_smithi090_166897_object6298:head# 2024-05-15T23:40:13.900 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.900 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6437e620:::benchmark_data_smithi090_166897_object45658:head# 2024-05-15T23:40:13.900 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.900 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:643830ae:::benchmark_data_smithi090_166897_object42949:head# 2024-05-15T23:40:13.900 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.900 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6438e0b9:::benchmark_data_smithi090_166897_object6435:head# 2024-05-15T23:40:13.900 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.900 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:643a2361:::benchmark_data_smithi090_166897_object7241:head# 2024-05-15T23:40:13.900 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.900 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:643af633:::benchmark_data_smithi090_166897_object69477:head# 2024-05-15T23:40:13.901 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.901 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:643d5d09:::benchmark_data_smithi090_166897_object14013:head# 2024-05-15T23:40:13.901 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.901 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:643ecbc4:::benchmark_data_smithi090_166897_object14515:head# 2024-05-15T23:40:13.901 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.901 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:643eed68:::benchmark_data_smithi090_166897_object58443:head# 2024-05-15T23:40:13.901 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.901 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:643f6102:::benchmark_data_smithi090_166897_object59841:head# 2024-05-15T23:40:13.901 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.901 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6440208e:::benchmark_data_smithi090_166897_object69047:head# 2024-05-15T23:40:13.901 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.901 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64423419:::benchmark_data_smithi090_166897_object23373:head# 2024-05-15T23:40:13.901 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.901 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64427d11:::benchmark_data_smithi090_166897_object43912:head# 2024-05-15T23:40:13.901 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.902 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:644299c5:::benchmark_data_smithi090_166897_object124:head# 2024-05-15T23:40:13.902 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.902 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:644621d4:::benchmark_data_smithi090_166897_object39117:head# 2024-05-15T23:40:13.902 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.902 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64464ef9:::benchmark_data_smithi090_166897_object64814:head# 2024-05-15T23:40:13.902 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.902 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6446eb80:::benchmark_data_smithi090_166897_object62264:head# 2024-05-15T23:40:13.902 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.902 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:644793bb:::benchmark_data_smithi090_166897_object46282:head# 2024-05-15T23:40:13.902 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.902 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6448cb75:::benchmark_data_smithi090_166897_object48763:head# 2024-05-15T23:40:13.902 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.902 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6449ef0d:::benchmark_data_smithi090_166897_object18579:head# 2024-05-15T23:40:13.902 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.903 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:644a7160:::benchmark_data_smithi090_166897_object55016:head# 2024-05-15T23:40:13.903 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.903 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:644a72d0:::benchmark_data_smithi090_166897_object48566:head# 2024-05-15T23:40:13.903 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.903 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:644ad655:::benchmark_data_smithi090_166897_object56425:head# 2024-05-15T23:40:13.903 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.903 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:644c466b:::benchmark_data_smithi090_166897_object17402:head# 2024-05-15T23:40:13.903 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.903 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:644c80ef:::benchmark_data_smithi090_166897_object30867:head# 2024-05-15T23:40:13.903 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.903 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:644c94e0:::benchmark_data_smithi090_166897_object48736:head# 2024-05-15T23:40:13.903 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.903 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:644d5063:::benchmark_data_smithi090_166897_object34119:head# 2024-05-15T23:40:13.903 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:644e4d94:::benchmark_data_smithi090_166897_object34819:head# 2024-05-15T23:40:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:644e8b27:::benchmark_data_smithi090_166897_object16512:head# 2024-05-15T23:40:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:644eab3c:::benchmark_data_smithi090_166897_object51630:head# 2024-05-15T23:40:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:645032cf:::benchmark_data_smithi090_166897_object11206:head# 2024-05-15T23:40:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6453c646:::benchmark_data_smithi090_166897_object54818:head# 2024-05-15T23:40:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64541911:::benchmark_data_smithi090_166897_object42023:head# 2024-05-15T23:40:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:645456fa:::benchmark_data_smithi090_166897_object28641:head# 2024-05-15T23:40:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.904 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6455ce3d:::benchmark_data_smithi090_166897_object66598:head# 2024-05-15T23:40:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64571387:::benchmark_data_smithi090_166897_object39339:head# 2024-05-15T23:40:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64572981:::benchmark_data_smithi090_166897_object37162:head# 2024-05-15T23:40:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6457b8d5:::benchmark_data_smithi090_166897_object30988:head# 2024-05-15T23:40:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:645842f7:::benchmark_data_smithi090_166897_object59426:head# 2024-05-15T23:40:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6458a7fa:::benchmark_data_smithi090_166897_object25527:head# 2024-05-15T23:40:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:645914fe:::benchmark_data_smithi090_166897_object20051:head# 2024-05-15T23:40:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64595464:::benchmark_data_smithi090_166897_object17143:head# 2024-05-15T23:40:13.905 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:645a4c1e:::benchmark_data_smithi090_166897_object48395:head# 2024-05-15T23:40:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:645a6a0d:::benchmark_data_smithi090_166897_object17858:head# 2024-05-15T23:40:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:645c763e:::benchmark_data_smithi090_166897_object58350:head# 2024-05-15T23:40:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:645d42ea:::benchmark_data_smithi090_166897_object30856:head# 2024-05-15T23:40:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:645e1e83:::benchmark_data_smithi090_166897_object64185:head# 2024-05-15T23:40:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:645e5c7e:::benchmark_data_smithi090_166897_object18958:head# 2024-05-15T23:40:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:645f57ba:::benchmark_data_smithi090_166897_object9715:head# 2024-05-15T23:40:13.906 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64613c1a:::benchmark_data_smithi090_166897_object39526:head# 2024-05-15T23:40:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64619379:::benchmark_data_smithi090_166897_object40040:head# 2024-05-15T23:40:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6461b376:::benchmark_data_smithi090_166897_object47521:head# 2024-05-15T23:40:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6461fd61:::benchmark_data_smithi090_166897_object49467:head# 2024-05-15T23:40:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6463b5f0:::benchmark_data_smithi090_166897_object23190:head# 2024-05-15T23:40:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64643fee:::benchmark_data_smithi090_166897_object45336:head# 2024-05-15T23:40:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64646365:::benchmark_data_smithi090_166897_object5377:head# 2024-05-15T23:40:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.907 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64656240:::benchmark_data_smithi090_166897_object63777:head# 2024-05-15T23:40:13.908 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.908 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6467677b:::benchmark_data_smithi090_166897_object21921:head# 2024-05-15T23:40:13.908 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.908 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64686629:::benchmark_data_smithi090_166897_object42720:head# 2024-05-15T23:40:13.908 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.908 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:646aa77e:::benchmark_data_smithi090_166897_object32605:head# 2024-05-15T23:40:13.908 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.908 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:646b3c85:::benchmark_data_smithi090_166897_object47164:head# 2024-05-15T23:40:13.908 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.908 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:646b8b7c:::benchmark_data_smithi090_166897_object12677:head# 2024-05-15T23:40:13.908 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.908 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:646be35d:::benchmark_data_smithi090_166897_object60489:head# 2024-05-15T23:40:13.908 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.908 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:646c46ce:::benchmark_data_smithi090_166897_object23021:head# 2024-05-15T23:40:13.908 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.909 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:646c8b81:::benchmark_data_smithi090_166897_object60425:head# 2024-05-15T23:40:13.909 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.909 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:646c9231:::benchmark_data_smithi090_166897_object25829:head# 2024-05-15T23:40:13.909 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.909 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:646dce77:::benchmark_data_smithi090_166897_object42812:head# 2024-05-15T23:40:13.909 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.909 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:646e6e1d:::benchmark_data_smithi090_166897_object5798:head# 2024-05-15T23:40:13.909 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.909 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:646e80e3:::benchmark_data_smithi090_166897_object14290:head# 2024-05-15T23:40:13.909 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.909 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:646ee326:::benchmark_data_smithi090_166897_object2511:head# 2024-05-15T23:40:13.909 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.909 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:646f4c50:::benchmark_data_smithi090_166897_object38079:head# 2024-05-15T23:40:13.909 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.910 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6470a4c9:::benchmark_data_smithi090_166897_object45710:head# 2024-05-15T23:40:13.910 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.910 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6470a7b2:::benchmark_data_smithi090_166897_object6811:head# 2024-05-15T23:40:13.910 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.910 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6471ae12:::benchmark_data_smithi090_166897_object55686:head# 2024-05-15T23:40:13.910 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.910 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6471c867:::benchmark_data_smithi090_166897_object61163:head# 2024-05-15T23:40:13.910 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.910 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64727b95:::benchmark_data_smithi090_166897_object66597:head# 2024-05-15T23:40:13.910 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.910 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64737f36:::benchmark_data_smithi090_166897_object51070:head# 2024-05-15T23:40:13.910 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.910 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:647472a5:::benchmark_data_smithi090_166897_object34865:head# 2024-05-15T23:40:13.910 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.911 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6474af77:::benchmark_data_smithi090_166897_object2660:head# 2024-05-15T23:40:13.911 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.911 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64763d74:::benchmark_data_smithi090_166897_object32596:head# 2024-05-15T23:40:13.911 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.911 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:647674dd:::benchmark_data_smithi090_166897_object7699:head# 2024-05-15T23:40:13.911 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.911 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6476a7ad:::benchmark_data_smithi090_166897_object32020:head# 2024-05-15T23:40:13.911 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.911 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6476dd52:::benchmark_data_smithi090_166897_object5082:head# 2024-05-15T23:40:13.911 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.911 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6477021f:::benchmark_data_smithi090_166897_object8316:head# 2024-05-15T23:40:13.911 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.911 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64777d3a:::benchmark_data_smithi090_166897_object51241:head# 2024-05-15T23:40:13.911 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.912 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64786a87:::benchmark_data_smithi090_166897_object13329:head# 2024-05-15T23:40:13.912 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.912 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64799338:::benchmark_data_smithi090_166897_object1369:head# 2024-05-15T23:40:13.912 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.912 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64799c1f:::benchmark_data_smithi090_166897_object38302:head# 2024-05-15T23:40:13.912 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.912 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6479d625:::benchmark_data_smithi090_166897_object45389:head# 2024-05-15T23:40:13.912 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.912 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:647b154b:::benchmark_data_smithi090_166897_object43648:head# 2024-05-15T23:40:13.912 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.912 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:647e0e23:::benchmark_data_smithi090_166897_object5916:head# 2024-05-15T23:40:13.912 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.912 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:647eb90c:::benchmark_data_smithi090_166897_object19594:head# 2024-05-15T23:40:13.912 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.912 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64805e5c:::benchmark_data_smithi090_166897_object3186:head# 2024-05-15T23:40:13.913 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.913 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6482134e:::benchmark_data_smithi090_166897_object57508:head# 2024-05-15T23:40:13.913 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.913 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64826cd3:::benchmark_data_smithi090_166897_object35243:head# 2024-05-15T23:40:13.913 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.913 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:648436a3:::benchmark_data_smithi090_166897_object10845:head# 2024-05-15T23:40:13.913 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.913 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6484a38e:::benchmark_data_smithi090_166897_object33070:head# 2024-05-15T23:40:13.913 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.913 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6486ace8:::benchmark_data_smithi090_166897_object7625:head# 2024-05-15T23:40:13.913 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.913 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6487d8ea:::benchmark_data_smithi090_166897_object27706:head# 2024-05-15T23:40:13.913 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.913 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64881d81:::benchmark_data_smithi090_166897_object21153:head# 2024-05-15T23:40:13.913 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.914 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64892f6e:::benchmark_data_smithi090_166897_object31052:head# 2024-05-15T23:40:13.914 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.914 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:648958ed:::benchmark_data_smithi090_166897_object39136:head# 2024-05-15T23:40:13.914 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.914 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6489b453:::benchmark_data_smithi090_166897_object50217:head# 2024-05-15T23:40:13.914 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.914 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:648aabfc:::benchmark_data_smithi090_166897_object28121:head# 2024-05-15T23:40:13.914 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.914 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:648abb28:::benchmark_data_smithi090_166897_object17599:head# 2024-05-15T23:40:13.914 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.914 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:648b0b2e:::benchmark_data_smithi090_166897_object17926:head# 2024-05-15T23:40:13.914 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.914 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:648b83f7:::benchmark_data_smithi090_166897_object3967:head# 2024-05-15T23:40:13.914 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.915 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:648c20e4:::benchmark_data_smithi090_166897_object57780:head# 2024-05-15T23:40:13.915 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.915 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:648cacee:::benchmark_data_smithi090_166897_object67543:head# 2024-05-15T23:40:13.915 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.915 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:648d7dff:::benchmark_data_smithi090_166897_object40037:head# 2024-05-15T23:40:13.915 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.915 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:648d8d8c:::benchmark_data_smithi090_166897_object31744:head# 2024-05-15T23:40:13.915 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.915 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:648df259:::benchmark_data_smithi090_166897_object12320:head# 2024-05-15T23:40:13.915 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.915 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:648df361:::benchmark_data_smithi090_166897_object505:head# 2024-05-15T23:40:13.915 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.915 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:648ea54a:::benchmark_data_smithi090_166897_object42717:head# 2024-05-15T23:40:13.915 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.915 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:648f2e2a:::benchmark_data_smithi090_166897_object7686:head# 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64900c68:::benchmark_data_smithi090_166897_object9198:head# 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6490493d:::benchmark_data_smithi090_166897_object66370:head# 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64914d51:::benchmark_data_smithi090_166897_object43722:head# 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6492ca2a:::benchmark_data_smithi090_166897_object19274:head# 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6493ed51:::benchmark_data_smithi090_166897_object5101:head# 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64947738:::benchmark_data_smithi090_166897_object28091:head# 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6494e67c:::benchmark_data_smithi090_166897_object27162:head# 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.916 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6496235a:::benchmark_data_smithi090_166897_object63614:head# 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64962806:::benchmark_data_smithi090_166897_object61484:head# 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64975c88:::benchmark_data_smithi090_166897_object36589:head# 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6499567a:::benchmark_data_smithi090_166897_object33910:head# 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:649a6a3b:::benchmark_data_smithi090_166897_object24598:head# 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:649c6fe0:::benchmark_data_smithi090_166897_object18767:head# 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:649dd75b:::benchmark_data_smithi090_166897_object70535:head# 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:649e355e:::benchmark_data_smithi090_166897_object28810:head# 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.917 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:649f0977:::benchmark_data_smithi090_166897_object18576:head# 2024-05-15T23:40:13.918 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.918 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64a021ec:::benchmark_data_smithi090_166897_object69083:head# 2024-05-15T23:40:13.918 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.918 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64a1d2da:::benchmark_data_smithi090_166897_object68326:head# 2024-05-15T23:40:13.918 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.918 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64a1f9b2:::benchmark_data_smithi090_166897_object54281:head# 2024-05-15T23:40:13.918 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.919 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:13.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:13 smithi084 ceph-mon[39784]: pgmap v2032: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.2 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 69 MiB/s wr, 1.11k op/s; 12103/204000 objects degraded (5.933%); 12 MiB/s, 199 objects/s recovering 2024-05-15T23:40:13.920 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:13 smithi090 ceph-mon[44434]: pgmap v2032: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.2 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 69 MiB/s wr, 1.11k op/s; 12103/204000 objects degraded (5.933%); 12 MiB/s, 199 objects/s recovering 2024-05-15T23:40:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:40:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:13.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:40:13.977 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64a2d629:::benchmark_data_smithi090_166897_object59972:head# 2024-05-15T23:40:13.977 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.977 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64a2d7c5:::benchmark_data_smithi090_166897_object19301:head# 2024-05-15T23:40:13.977 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.977 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64a48d7b:::benchmark_data_smithi090_166897_object14564:head# 2024-05-15T23:40:13.977 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.977 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64a6e2cb:::benchmark_data_smithi090_166897_object1407:head# 2024-05-15T23:40:13.977 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.978 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64a846d6:::benchmark_data_smithi090_166897_object31859:head# 2024-05-15T23:40:13.978 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.978 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64a95108:::benchmark_data_smithi090_166897_object33857:head# 2024-05-15T23:40:13.978 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.978 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64a99f4b:::benchmark_data_smithi090_166897_object65655:head# 2024-05-15T23:40:13.978 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.978 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64aa7f02:::benchmark_data_smithi090_166897_object61684:head# 2024-05-15T23:40:13.978 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.978 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64aaf55a:::benchmark_data_smithi090_166897_object939:head# 2024-05-15T23:40:13.978 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.978 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64aaf984:::benchmark_data_smithi090_166897_object35868:head# 2024-05-15T23:40:13.978 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.978 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64aba15a:::benchmark_data_smithi090_166897_object3420:head# 2024-05-15T23:40:13.978 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.979 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64b05084:::benchmark_data_smithi090_166897_object59904:head# 2024-05-15T23:40:13.979 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.979 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64b18359:::benchmark_data_smithi090_166897_object44895:head# 2024-05-15T23:40:13.979 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.979 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64b190d8:::benchmark_data_smithi090_166897_object53514:head# 2024-05-15T23:40:13.979 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.979 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64b2cc15:::benchmark_data_smithi090_166897_object43834:head# 2024-05-15T23:40:13.979 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.979 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64b2cd0b:::benchmark_data_smithi090_166897_object43285:head# 2024-05-15T23:40:13.979 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.979 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64b3aab4:::benchmark_data_smithi090_166897_object18922:head# 2024-05-15T23:40:13.979 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.979 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64b4b292:::benchmark_data_smithi090_166897_object40026:head# 2024-05-15T23:40:13.979 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.980 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64b527b7:::benchmark_data_smithi090_166897_object42442:head# 2024-05-15T23:40:13.980 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.980 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64b528e5:::benchmark_data_smithi090_166897_object66936:head# 2024-05-15T23:40:13.980 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.980 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64b570f6:::benchmark_data_smithi090_166897_object48167:head# 2024-05-15T23:40:13.980 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.980 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64be5ea6:::benchmark_data_smithi090_166897_object14513:head# 2024-05-15T23:40:13.980 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.980 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64beb9f4:::benchmark_data_smithi090_166897_object43971:head# 2024-05-15T23:40:13.980 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.980 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64bf06b5:::benchmark_data_smithi090_166897_object38267:head# 2024-05-15T23:40:13.980 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.980 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64bf7c1c:::benchmark_data_smithi090_166897_object36174:head# 2024-05-15T23:40:13.980 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.980 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64c076e9:::benchmark_data_smithi090_166897_object51405:head# 2024-05-15T23:40:13.981 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.981 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64c13297:::benchmark_data_smithi090_166897_object13135:head# 2024-05-15T23:40:13.981 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.981 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64c19fbf:::benchmark_data_smithi090_166897_object27237:head# 2024-05-15T23:40:13.981 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.981 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64c1f662:::benchmark_data_smithi090_166897_object39739:head# 2024-05-15T23:40:13.981 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.981 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64c30650:::benchmark_data_smithi090_166897_object23767:head# 2024-05-15T23:40:13.981 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.981 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64c32f6b:::benchmark_data_smithi090_166897_object61657:head# 2024-05-15T23:40:13.981 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.981 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64c35caf:::benchmark_data_smithi090_166897_object25170:head# 2024-05-15T23:40:13.981 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.981 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64c430be:::benchmark_data_smithi090_166897_object21960:head# 2024-05-15T23:40:13.982 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.982 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64c4ebb9:::benchmark_data_smithi090_166897_object45886:head# 2024-05-15T23:40:13.982 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.982 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64c81457:::benchmark_data_smithi090_166897_object64846:head# 2024-05-15T23:40:13.982 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.982 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64c83de1:::benchmark_data_smithi090_166897_object37011:head# 2024-05-15T23:40:13.982 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.982 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64c89926:::benchmark_data_smithi090_166897_object48677:head# 2024-05-15T23:40:13.982 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.982 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64c89ed5:::benchmark_data_smithi090_166897_object70150:head# 2024-05-15T23:40:13.982 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.982 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64ca5263:::benchmark_data_smithi090_166897_object21834:head# 2024-05-15T23:40:13.982 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.982 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64cb70ae:::benchmark_data_smithi090_166897_object7854:head# 2024-05-15T23:40:13.983 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.983 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64cc26de:::benchmark_data_smithi090_166897_object21763:head# 2024-05-15T23:40:13.983 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.983 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64cc3478:::benchmark_data_smithi090_166897_object49863:head# 2024-05-15T23:40:13.983 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.983 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64cd6463:::benchmark_data_smithi090_166897_object23092:head# 2024-05-15T23:40:13.983 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.983 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64cd76e1:::benchmark_data_smithi090_166897_object1911:head# 2024-05-15T23:40:13.983 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.983 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64cfdcd0:::benchmark_data_smithi090_166897_object6334:head# 2024-05-15T23:40:13.983 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.983 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64d0c059:::benchmark_data_smithi090_166897_object57278:head# 2024-05-15T23:40:13.983 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.983 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64d0d2bc:::benchmark_data_smithi090_166897_object55810:head# 2024-05-15T23:40:13.983 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.984 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64d1ae08:::benchmark_data_smithi090_166897_object19946:head# 2024-05-15T23:40:13.984 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.984 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64d1aeef:::benchmark_data_smithi090_166897_object64475:head# 2024-05-15T23:40:13.984 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.984 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64d1f95b:::benchmark_data_smithi090_166897_object70369:head# 2024-05-15T23:40:13.984 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.984 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64d2f06d:::benchmark_data_smithi090_166897_object13894:head# 2024-05-15T23:40:13.984 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.984 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64d3606b:::benchmark_data_smithi090_166897_object48209:head# 2024-05-15T23:40:13.984 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.984 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64d54e4e:::benchmark_data_smithi090_166897_object12310:head# 2024-05-15T23:40:13.984 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.984 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64d7e8b8:::benchmark_data_smithi090_166897_object52084:head# 2024-05-15T23:40:13.984 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64d8f50a:::benchmark_data_smithi090_166897_object28263:head# 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64db0b7a:::benchmark_data_smithi090_166897_object44543:head# 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64db2156:::benchmark_data_smithi090_166897_object6301:head# 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64dbcf31:::benchmark_data_smithi090_166897_object59042:head# 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64dbf494:::benchmark_data_smithi090_166897_object66162:head# 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64dc702d:::benchmark_data_smithi090_166897_object21330:head# 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64dd5e82:::benchmark_data_smithi090_166897_object40406:head# 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64defb8f:::benchmark_data_smithi090_166897_object24492:head# 2024-05-15T23:40:13.985 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.986 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64e2bb4e:::benchmark_data_smithi090_166897_object24773:head# 2024-05-15T23:40:13.986 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.986 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64e56c49:::benchmark_data_smithi090_166897_object15274:head# 2024-05-15T23:40:13.986 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.986 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64e6ab34:::benchmark_data_smithi090_166897_object14764:head# 2024-05-15T23:40:13.986 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.986 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64e8bc33:::benchmark_data_smithi090_166897_object55582:head# 2024-05-15T23:40:13.986 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.986 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64e9698d:::benchmark_data_smithi090_166897_object15373:head# 2024-05-15T23:40:13.986 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.986 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64e9a58d:::benchmark_data_smithi090_166897_object60580:head# 2024-05-15T23:40:13.986 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.986 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64ecab1f:::benchmark_data_smithi090_166897_object42268:head# 2024-05-15T23:40:13.986 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.987 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64ed551e:::benchmark_data_smithi090_166897_object33150:head# 2024-05-15T23:40:13.987 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.987 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64ed744c:::benchmark_data_smithi090_166897_object1351:head# 2024-05-15T23:40:13.987 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.987 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64f08731:::benchmark_data_smithi090_166897_object23934:head# 2024-05-15T23:40:13.987 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.987 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64f1e9e5:::benchmark_data_smithi090_166897_object43225:head# 2024-05-15T23:40:13.987 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.987 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64f20cd3:::benchmark_data_smithi090_166897_object15621:head# 2024-05-15T23:40:13.987 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.987 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64f3f41e:::benchmark_data_smithi090_166897_object21212:head# 2024-05-15T23:40:13.987 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.987 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64f876fa:::benchmark_data_smithi090_166897_object33246:head# 2024-05-15T23:40:13.987 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.987 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64f90963:::benchmark_data_smithi090_166897_object11459:head# 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64fa1e42:::benchmark_data_smithi090_166897_object2068:head# 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64fa526f:::benchmark_data_smithi090_166897_object36281:head# 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64fa5b99:::benchmark_data_smithi090_166897_object20127:head# 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64fb354d:::benchmark_data_smithi090_166897_object25777:head# 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64fb9864:::benchmark_data_smithi090_166897_object30488:head# 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64fc501e:::benchmark_data_smithi090_166897_object59705:head# 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64fcb38c:::benchmark_data_smithi090_166897_object19174:head# 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.988 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64fd655d:::benchmark_data_smithi090_166897_object90:head# 2024-05-15T23:40:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64fd892f:::benchmark_data_smithi090_166897_object64163:head# 2024-05-15T23:40:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64fd9dc9:::benchmark_data_smithi090_166897_object47304:head# 2024-05-15T23:40:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64fe487c:::benchmark_data_smithi090_166897_object39860:head# 2024-05-15T23:40:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64fe8372:::benchmark_data_smithi090_166897_object66977:head# 2024-05-15T23:40:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64ff1c88:::benchmark_data_smithi090_166897_object16167:head# 2024-05-15T23:40:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:64ffd203:::benchmark_data_smithi090_166897_object53142:head# 2024-05-15T23:40:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65000c6d:::benchmark_data_smithi090_166897_object6700:head# 2024-05-15T23:40:13.989 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.990 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65004f35:::benchmark_data_smithi090_166897_object65758:head# 2024-05-15T23:40:13.990 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.990 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65020ef2:::benchmark_data_smithi090_166897_object35139:head# 2024-05-15T23:40:13.990 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.990 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65034232:::benchmark_data_smithi090_166897_object5308:head# 2024-05-15T23:40:13.990 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.990 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65038880:::benchmark_data_smithi090_166897_object245:head# 2024-05-15T23:40:13.990 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.990 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65042b9f:::benchmark_data_smithi090_166897_object51573:head# 2024-05-15T23:40:13.990 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.990 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6504f09b:::benchmark_data_smithi090_166897_object56663:head# 2024-05-15T23:40:13.990 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.990 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6505ebb2:::benchmark_data_smithi090_166897_object62258:head# 2024-05-15T23:40:13.990 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.991 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65060497:::benchmark_data_smithi090_166897_object21429:head# 2024-05-15T23:40:13.991 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.991 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65062db3:::benchmark_data_smithi090_166897_object46023:head# 2024-05-15T23:40:13.991 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.991 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65078ef7:::benchmark_data_smithi090_166897_object55268:head# 2024-05-15T23:40:13.991 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.991 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6508877a:::benchmark_data_smithi090_166897_object59337:head# 2024-05-15T23:40:13.991 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.991 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65088ea8:::benchmark_data_smithi090_166897_object8499:head# 2024-05-15T23:40:13.991 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.991 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:650a7d0a:::benchmark_data_smithi090_166897_object66053:head# 2024-05-15T23:40:13.991 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.991 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:650af4df:::benchmark_data_smithi090_166897_object68721:head# 2024-05-15T23:40:13.991 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.992 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:650c0d2b:::benchmark_data_smithi090_166897_object37258:head# 2024-05-15T23:40:13.992 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.992 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:650cebd5:::benchmark_data_smithi090_166897_object52036:head# 2024-05-15T23:40:13.992 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.992 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:650d6612:::benchmark_data_smithi090_166897_object20879:head# 2024-05-15T23:40:13.992 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.992 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:650db7e5:::benchmark_data_smithi090_166897_object48729:head# 2024-05-15T23:40:13.992 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.992 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:650e6c93:::benchmark_data_smithi090_166897_object28673:head# 2024-05-15T23:40:13.992 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.992 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:650f3417:::benchmark_data_smithi090_166897_object37509:head# 2024-05-15T23:40:13.992 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.992 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:650f540b:::benchmark_data_smithi090_166897_object55465:head# 2024-05-15T23:40:13.992 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.992 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:650fbf41:::benchmark_data_smithi090_166897_object52243:head# 2024-05-15T23:40:13.993 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.993 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:650fca30:::benchmark_data_smithi090_166897_object13933:head# 2024-05-15T23:40:13.993 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.993 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65149199:::benchmark_data_smithi090_166897_object28113:head# 2024-05-15T23:40:13.993 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.993 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6515bed9:::benchmark_data_smithi090_166897_object56813:head# 2024-05-15T23:40:13.993 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.993 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6517851f:::benchmark_data_smithi090_166897_object10398:head# 2024-05-15T23:40:13.993 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.993 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65181b53:::benchmark_data_smithi090_166897_object50498:head# 2024-05-15T23:40:13.993 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.993 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6518a3f7:::benchmark_data_smithi090_166897_object24437:head# 2024-05-15T23:40:13.993 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.993 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65191384:::benchmark_data_smithi090_166897_object46106:head# 2024-05-15T23:40:13.994 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.994 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65196553:::benchmark_data_smithi090_166897_object39370:head# 2024-05-15T23:40:13.994 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.994 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65197843:::benchmark_data_smithi090_166897_object10519:head# 2024-05-15T23:40:13.994 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.994 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:651998d5:::benchmark_data_smithi090_166897_object69070:head# 2024-05-15T23:40:13.994 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.994 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:651a39b7:::benchmark_data_smithi090_166897_object63568:head# 2024-05-15T23:40:13.994 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.994 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:651b0acd:::benchmark_data_smithi090_166897_object30880:head# 2024-05-15T23:40:13.994 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.994 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:651bbc88:::benchmark_data_smithi090_166897_object39676:head# 2024-05-15T23:40:13.994 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.994 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:651d86ab:::benchmark_data_smithi090_166897_object5558:head# 2024-05-15T23:40:13.995 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.995 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:652091d7:::benchmark_data_smithi090_166897_object53682:head# 2024-05-15T23:40:13.995 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.995 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6520bb5c:::benchmark_data_smithi090_166897_object40249:head# 2024-05-15T23:40:13.995 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.995 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65211b11:::benchmark_data_smithi090_166897_object62381:head# 2024-05-15T23:40:13.995 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.995 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65218f3b:::benchmark_data_smithi090_166897_object16241:head# 2024-05-15T23:40:13.995 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.995 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65226aaa:::benchmark_data_smithi090_166897_object9432:head# 2024-05-15T23:40:13.995 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.995 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6523c289:::benchmark_data_smithi090_166897_object59759:head# 2024-05-15T23:40:13.995 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.995 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6523ef8b:::benchmark_data_smithi090_166897_object56412:head# 2024-05-15T23:40:13.995 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.996 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6524862f:::benchmark_data_smithi090_166897_object25156:head# 2024-05-15T23:40:13.996 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.996 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6525efc7:::benchmark_data_smithi090_166897_object23458:head# 2024-05-15T23:40:13.996 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.996 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65263705:::benchmark_data_smithi090_166897_object29956:head# 2024-05-15T23:40:13.996 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.996 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65263b6a:::benchmark_data_smithi090_166897_object26020:head# 2024-05-15T23:40:13.996 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.996 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65264a6d:::benchmark_data_smithi090_166897_object53109:head# 2024-05-15T23:40:13.996 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.996 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:652686a3:::benchmark_data_smithi090_166897_object60517:head# 2024-05-15T23:40:13.996 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.996 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6526bcfe:::benchmark_data_smithi090_166897_object33548:head# 2024-05-15T23:40:13.996 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.997 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6526debd:::benchmark_data_smithi090_166897_object49146:head# 2024-05-15T23:40:13.997 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.997 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65280f82:::benchmark_data_smithi090_166897_object55235:head# 2024-05-15T23:40:13.997 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.997 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6528111e:::benchmark_data_smithi090_166897_object54948:head# 2024-05-15T23:40:13.997 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.997 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65285fa2:::benchmark_data_smithi090_166897_object11038:head# 2024-05-15T23:40:13.997 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.997 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6529b973:::benchmark_data_smithi090_166897_object20374:head# 2024-05-15T23:40:13.997 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.997 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6529da40:::benchmark_data_smithi090_166897_object38552:head# 2024-05-15T23:40:13.997 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.997 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:652b5b09:::benchmark_data_smithi090_166897_object48607:head# 2024-05-15T23:40:13.997 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.997 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:652be0e1:::benchmark_data_smithi090_166897_object70932:head# 2024-05-15T23:40:13.998 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.998 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:652bec84:::benchmark_data_smithi090_166897_object18675:head# 2024-05-15T23:40:13.998 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.998 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:652c14a1:::benchmark_data_smithi090_166897_object52925:head# 2024-05-15T23:40:13.998 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.998 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:652c6a67:::benchmark_data_smithi090_166897_object4390:head# 2024-05-15T23:40:13.998 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.998 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:652d872c:::benchmark_data_smithi090_166897_object45604:head# 2024-05-15T23:40:13.998 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.998 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:652d8bbe:::benchmark_data_smithi090_166897_object17532:head# 2024-05-15T23:40:13.998 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.998 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:652efd39:::benchmark_data_smithi090_166897_object28788:head# 2024-05-15T23:40:13.998 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.998 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:652f40b2:::benchmark_data_smithi090_166897_object2943:head# 2024-05-15T23:40:13.999 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.999 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:652f5fa6:::benchmark_data_smithi090_166897_object49126:head# 2024-05-15T23:40:13.999 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.999 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65300c5b:::benchmark_data_smithi090_166897_object138:head# 2024-05-15T23:40:13.999 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.999 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65311af1:::benchmark_data_smithi090_166897_object8911:head# 2024-05-15T23:40:13.999 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.999 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6531bae5:::benchmark_data_smithi090_166897_object5293:head# 2024-05-15T23:40:13.999 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.999 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6531c298:::benchmark_data_smithi090_166897_object14313:head# 2024-05-15T23:40:13.999 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.999 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65321cb5:::benchmark_data_smithi090_166897_object65625:head# 2024-05-15T23:40:13.999 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:13.999 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65335589:::benchmark_data_smithi090_166897_object39893:head# 2024-05-15T23:40:13.999 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.000 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65352c1e:::benchmark_data_smithi090_166897_object15308:head# 2024-05-15T23:40:14.000 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.000 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65355270:::benchmark_data_smithi090_166897_object65378:head# 2024-05-15T23:40:14.000 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.000 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:653575d9:::benchmark_data_smithi090_166897_object65464:head# 2024-05-15T23:40:14.000 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.000 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65379736:::benchmark_data_smithi090_166897_object63602:head# 2024-05-15T23:40:14.000 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.000 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6538359f:::benchmark_data_smithi090_166897_object25343:head# 2024-05-15T23:40:14.000 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.000 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6539319f:::benchmark_data_smithi090_166897_object11190:head# 2024-05-15T23:40:14.000 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.000 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6539a1fb:::benchmark_data_smithi090_166897_object39882:head# 2024-05-15T23:40:14.000 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.000 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:653c2d8e:::benchmark_data_smithi090_166897_object17182:head# 2024-05-15T23:40:14.001 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.001 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:653c77c3:::benchmark_data_smithi090_166897_object1447:head# 2024-05-15T23:40:14.001 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.001 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:653deca8:::benchmark_data_smithi090_166897_object67856:head# 2024-05-15T23:40:14.001 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.001 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:653e31aa:::benchmark_data_smithi090_166897_object40846:head# 2024-05-15T23:40:14.001 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.001 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:653ecc04:::benchmark_data_smithi090_166897_object62918:head# 2024-05-15T23:40:14.001 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.001 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:653f5cff:::benchmark_data_smithi090_166897_object24551:head# 2024-05-15T23:40:14.001 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.001 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:653f948e:::benchmark_data_smithi090_166897_object42652:head# 2024-05-15T23:40:14.001 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.001 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65400a47:::benchmark_data_smithi090_166897_object56099:head# 2024-05-15T23:40:14.002 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.002 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6541d09e:::benchmark_data_smithi090_166897_object62143:head# 2024-05-15T23:40:14.002 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.002 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6543b1df:::benchmark_data_smithi090_166897_object63929:head# 2024-05-15T23:40:14.002 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.002 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6543b4a5:::benchmark_data_smithi090_166897_object52390:head# 2024-05-15T23:40:14.002 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.002 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65447a47:::benchmark_data_smithi090_166897_object60679:head# 2024-05-15T23:40:14.002 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.002 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65468d5b:::benchmark_data_smithi090_166897_object64339:head# 2024-05-15T23:40:14.002 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.002 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6546a724:::benchmark_data_smithi090_166897_object56599:head# 2024-05-15T23:40:14.002 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.002 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65472bca:::benchmark_data_smithi090_166897_object67366:head# 2024-05-15T23:40:14.002 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.003 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6548a48c:::benchmark_data_smithi090_166897_object54625:head# 2024-05-15T23:40:14.003 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.003 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6549adcf:::benchmark_data_smithi090_166897_object65606:head# 2024-05-15T23:40:14.003 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.003 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:654bd46e:::benchmark_data_smithi090_166897_object33155:head# 2024-05-15T23:40:14.003 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.003 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:654c2639:::benchmark_data_smithi090_166897_object55718:head# 2024-05-15T23:40:14.003 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.003 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:654cdef1:::benchmark_data_smithi090_166897_object48123:head# 2024-05-15T23:40:14.003 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.003 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:654d4c8b:::benchmark_data_smithi090_166897_object19694:head# 2024-05-15T23:40:14.003 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.004 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:654da810:::benchmark_data_smithi090_166897_object8660:head# 2024-05-15T23:40:14.004 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.004 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:654eabc5:::benchmark_data_smithi090_166897_object69042:head# 2024-05-15T23:40:14.004 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.004 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:654ec508:::benchmark_data_smithi090_166897_object42633:head# 2024-05-15T23:40:14.004 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.004 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:655056a9:::benchmark_data_smithi090_166897_object53284:head# 2024-05-15T23:40:14.004 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.004 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6551276e:::benchmark_data_smithi090_166897_object56164:head# 2024-05-15T23:40:14.004 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.004 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6553b779:::benchmark_data_smithi090_166897_object10063:head# 2024-05-15T23:40:14.004 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.004 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6553ea8d:::benchmark_data_smithi090_166897_object31688:head# 2024-05-15T23:40:14.004 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.004 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6554a7c9:::benchmark_data_smithi090_166897_object8292:head# 2024-05-15T23:40:14.005 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.005 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6554c436:::benchmark_data_smithi090_166897_object40560:head# 2024-05-15T23:40:14.005 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.005 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65552908:::benchmark_data_smithi090_166897_object25100:head# 2024-05-15T23:40:14.005 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.005 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65556c79:::benchmark_data_smithi090_166897_object19822:head# 2024-05-15T23:40:14.005 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.005 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6556a667:::benchmark_data_smithi090_166897_object35303:head# 2024-05-15T23:40:14.005 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.005 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:655ab98d:::benchmark_data_smithi090_166897_object31470:head# 2024-05-15T23:40:14.005 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.005 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:655d4120:::benchmark_data_smithi090_166897_object6271:head# 2024-05-15T23:40:14.005 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.005 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:655d7152:::benchmark_data_smithi090_166897_object45697:head# 2024-05-15T23:40:14.005 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.006 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:655ec561:::benchmark_data_smithi090_166897_object21891:head# 2024-05-15T23:40:14.006 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.006 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:655f75c6:::benchmark_data_smithi090_166897_object69256:head# 2024-05-15T23:40:14.006 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.006 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:655fd207:::benchmark_data_smithi090_166897_object23717:head# 2024-05-15T23:40:14.006 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.006 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6560216d:::benchmark_data_smithi090_166897_object39157:head# 2024-05-15T23:40:14.006 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.006 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:656031b7:::benchmark_data_smithi090_166897_object24786:head# 2024-05-15T23:40:14.006 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.006 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65629f51:::benchmark_data_smithi090_166897_object14074:head# 2024-05-15T23:40:14.006 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.006 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6563f5f0:::benchmark_data_smithi090_166897_object9283:head# 2024-05-15T23:40:14.006 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65643f0a:::benchmark_data_smithi090_166897_object56108:head# 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65646a8e:::benchmark_data_smithi090_166897_object4357:head# 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65647816:::benchmark_data_smithi090_166897_object61599:head# 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65659d3c:::benchmark_data_smithi090_166897_object46210:head# 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6565fab0:::benchmark_data_smithi090_166897_object58702:head# 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6566a7b0:::benchmark_data_smithi090_166897_object51047:head# 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65671fa1:::benchmark_data_smithi090_166897_object53885:head# 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65682947:::benchmark_data_smithi090_166897_object1941:head# 2024-05-15T23:40:14.007 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.008 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65699fd6:::benchmark_data_smithi090_166897_object32636:head# 2024-05-15T23:40:14.008 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.008 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6569a412:::benchmark_data_smithi090_166897_object38339:head# 2024-05-15T23:40:14.008 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.008 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:656ac10d:::benchmark_data_smithi090_166897_object17587:head# 2024-05-15T23:40:14.008 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.008 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:656b38e0:::benchmark_data_smithi090_166897_object45119:head# 2024-05-15T23:40:14.008 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.008 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:656b8c08:::benchmark_data_smithi090_166897_object27549:head# 2024-05-15T23:40:14.008 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.008 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:656baff9:::benchmark_data_smithi090_166897_object31144:head# 2024-05-15T23:40:14.008 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.008 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:656dde87:::benchmark_data_smithi090_166897_object3622:head# 2024-05-15T23:40:14.008 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.008 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:656ea656:::benchmark_data_smithi090_166897_object36859:head# 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:656f45a2:::benchmark_data_smithi090_166897_object39563:head# 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:656f9b5e:::benchmark_data_smithi090_166897_object8833:head# 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6570c3fe:::benchmark_data_smithi090_166897_object1027:head# 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6570cfcc:::benchmark_data_smithi090_166897_object51202:head# 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6571f35f:::benchmark_data_smithi090_166897_object9953:head# 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65726fa8:::benchmark_data_smithi090_166897_object52831:head# 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:657276b7:::benchmark_data_smithi090_166897_object23959:head# 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.009 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65741715:::benchmark_data_smithi090_166897_object17019:head# 2024-05-15T23:40:14.010 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.010 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65748004:::benchmark_data_smithi090_166897_object54392:head# 2024-05-15T23:40:14.010 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.010 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:657668e5:::benchmark_data_smithi090_166897_object52579:head# 2024-05-15T23:40:14.010 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.010 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65767dbe:::benchmark_data_smithi090_166897_object67030:head# 2024-05-15T23:40:14.010 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.010 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6577d812:::benchmark_data_smithi090_166897_object6451:head# 2024-05-15T23:40:14.010 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.010 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6578639b:::benchmark_data_smithi090_166897_object14749:head# 2024-05-15T23:40:14.010 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.010 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6578b2da:::benchmark_data_smithi090_166897_object18286:head# 2024-05-15T23:40:14.010 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.010 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6579b03c:::benchmark_data_smithi090_166897_object24679:head# 2024-05-15T23:40:14.011 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.011 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6579c8fd:::benchmark_data_smithi090_166897_object20399:head# 2024-05-15T23:40:14.011 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.011 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:657a3183:::benchmark_data_smithi090_166897_object65053:head# 2024-05-15T23:40:14.011 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.011 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:657bc5d8:::benchmark_data_smithi090_166897_object67348:head# 2024-05-15T23:40:14.011 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.011 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:657be90e:::benchmark_data_smithi090_166897_object8665:head# 2024-05-15T23:40:14.011 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.011 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:657c0394:::benchmark_data_smithi090_166897_object2454:head# 2024-05-15T23:40:14.011 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.011 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:657d110e:::benchmark_data_smithi090_166897_object17415:head# 2024-05-15T23:40:14.011 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.011 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:657d6b84:::benchmark_data_smithi090_166897_object61964:head# 2024-05-15T23:40:14.011 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:657e42b2:::benchmark_data_smithi090_166897_object3329:head# 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:657f2577:::benchmark_data_smithi090_166897_object24858:head# 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:657f4445:::benchmark_data_smithi090_166897_object5024:head# 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:657f5725:::benchmark_data_smithi090_166897_object64069:head# 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6582945b:::benchmark_data_smithi090_166897_object41994:head# 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65836349:::benchmark_data_smithi090_166897_object17260:head# 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65848003:::benchmark_data_smithi090_166897_object8709:head# 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6584b8e8:::benchmark_data_smithi090_166897_object52861:head# 2024-05-15T23:40:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.013 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:658513d4:::benchmark_data_smithi090_166897_object65458:head# 2024-05-15T23:40:14.013 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.013 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:658808a0:::benchmark_data_smithi090_166897_object27937:head# 2024-05-15T23:40:14.013 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.013 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:658932b0:::benchmark_data_smithi090_166897_object38836:head# 2024-05-15T23:40:14.013 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.013 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:658973e5:::benchmark_data_smithi090_166897_object33990:head# 2024-05-15T23:40:14.013 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.013 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:658b48ba:::benchmark_data_smithi090_166897_object12925:head# 2024-05-15T23:40:14.013 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.013 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:658ca16f:::benchmark_data_smithi090_166897_object32704:head# 2024-05-15T23:40:14.013 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.013 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:658edbcb:::benchmark_data_smithi090_166897_object24649:head# 2024-05-15T23:40:14.013 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.014 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:658fdea7:::benchmark_data_smithi090_166897_object44309:head# 2024-05-15T23:40:14.014 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.014 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:658ff20b:::benchmark_data_smithi090_166897_object55740:head# 2024-05-15T23:40:14.014 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.014 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65926c48:::benchmark_data_smithi090_166897_object41350:head# 2024-05-15T23:40:14.014 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.014 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65929e96:::benchmark_data_smithi090_166897_object47327:head# 2024-05-15T23:40:14.014 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.014 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6594462a:::benchmark_data_smithi090_166897_object18990:head# 2024-05-15T23:40:14.014 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.014 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65950612:::benchmark_data_smithi090_166897_object35224:head# 2024-05-15T23:40:14.014 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.014 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65950aa8:::benchmark_data_smithi090_166897_object70435:head# 2024-05-15T23:40:14.014 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.014 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65966bee:::benchmark_data_smithi090_166897_object65718:head# 2024-05-15T23:40:14.015 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.015 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:659706bb:::benchmark_data_smithi090_166897_object31593:head# 2024-05-15T23:40:14.015 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.015 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6598a10a:::benchmark_data_smithi090_166897_object9054:head# 2024-05-15T23:40:14.015 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.015 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6598e18d:::benchmark_data_smithi090_166897_object65036:head# 2024-05-15T23:40:14.015 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.015 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6599aea6:::benchmark_data_smithi090_166897_object33350:head# 2024-05-15T23:40:14.015 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.015 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:659c0b4e:::benchmark_data_smithi090_166897_object70677:head# 2024-05-15T23:40:14.015 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.015 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:659c4faa:::benchmark_data_smithi090_166897_object63115:head# 2024-05-15T23:40:14.015 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.015 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:659da724:::benchmark_data_smithi090_166897_object55720:head# 2024-05-15T23:40:14.015 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:659f5a3b:::benchmark_data_smithi090_166897_object7739:head# 2024-05-15T23:40:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65a003eb:::benchmark_data_smithi090_166897_object47008:head# 2024-05-15T23:40:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65a03cb3:::benchmark_data_smithi090_166897_object36264:head# 2024-05-15T23:40:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65a1a72b:::benchmark_data_smithi090_166897_object52065:head# 2024-05-15T23:40:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65a273b7:::benchmark_data_smithi090_166897_object11924:head# 2024-05-15T23:40:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65a3879b:::benchmark_data_smithi090_166897_object47378:head# 2024-05-15T23:40:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.016 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65a4dd3e:::benchmark_data_smithi090_166897_object49573:head# 2024-05-15T23:40:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65a53e53:::benchmark_data_smithi090_166897_object13770:head# 2024-05-15T23:40:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65a55ca0:::benchmark_data_smithi090_166897_object11473:head# 2024-05-15T23:40:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65a58152:::benchmark_data_smithi090_166897_object33087:head# 2024-05-15T23:40:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65a67667:::benchmark_data_smithi090_166897_object55818:head# 2024-05-15T23:40:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65a69da8:::benchmark_data_smithi090_166897_object59567:head# 2024-05-15T23:40:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65a890e9:::benchmark_data_smithi090_166897_object23512:head# 2024-05-15T23:40:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65a9f7fa:::benchmark_data_smithi090_166897_object44270:head# 2024-05-15T23:40:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.018 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65aa9b36:::benchmark_data_smithi090_166897_object25971:head# 2024-05-15T23:40:14.018 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.018 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65aac95f:::benchmark_data_smithi090_166897_object23437:head# 2024-05-15T23:40:14.018 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.018 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65ab2090:::benchmark_data_smithi090_166897_object62185:head# 2024-05-15T23:40:14.018 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.018 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65ac1171:::benchmark_data_smithi090_166897_object7418:head# 2024-05-15T23:40:14.018 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.018 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65ac7468:::benchmark_data_smithi090_166897_object17460:head# 2024-05-15T23:40:14.018 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.018 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65aca308:::benchmark_data_smithi090_166897_object44416:head# 2024-05-15T23:40:14.018 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.018 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65aed303:::benchmark_data_smithi090_166897_object57454:head# 2024-05-15T23:40:14.018 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65b07717:::benchmark_data_smithi090_166897_object61085:head# 2024-05-15T23:40:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65b0fe89:::benchmark_data_smithi090_166897_object51953:head# 2024-05-15T23:40:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65b111f8:::benchmark_data_smithi090_166897_object11628:head# 2024-05-15T23:40:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65b24ad8:::benchmark_data_smithi090_166897_object65077:head# 2024-05-15T23:40:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65b2bcae:::benchmark_data_smithi090_166897_object4896:head# 2024-05-15T23:40:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65b2f7c5:::benchmark_data_smithi090_166897_object5705:head# 2024-05-15T23:40:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65b504a5:::benchmark_data_smithi090_166897_object30216:head# 2024-05-15T23:40:14.019 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.020 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65b61619:::benchmark_data_smithi090_166897_object69700:head# 2024-05-15T23:40:14.020 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.020 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65b6d8e6:::benchmark_data_smithi090_166897_object62502:head# 2024-05-15T23:40:14.020 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.020 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65b6e7d1:::benchmark_data_smithi090_166897_object27060:head# 2024-05-15T23:40:14.020 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.020 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65b7112e:::benchmark_data_smithi090_166897_object4466:head# 2024-05-15T23:40:14.020 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.020 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65b86a82:::benchmark_data_smithi090_166897_object51063:head# 2024-05-15T23:40:14.020 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.020 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65baf405:::benchmark_data_smithi090_166897_object69718:head# 2024-05-15T23:40:14.020 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.020 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65bbfb42:::benchmark_data_smithi090_166897_object21390:head# 2024-05-15T23:40:14.020 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.020 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65bc99c6:::benchmark_data_smithi090_166897_object4524:head# 2024-05-15T23:40:14.021 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.021 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65c06096:::benchmark_data_smithi090_166897_object61473:head# 2024-05-15T23:40:14.021 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.021 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65c07756:::benchmark_data_smithi090_166897_object49834:head# 2024-05-15T23:40:14.021 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.021 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65c0939f:::benchmark_data_smithi090_166897_object58465:head# 2024-05-15T23:40:14.021 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.021 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65c3185a:::benchmark_data_smithi090_166897_object48695:head# 2024-05-15T23:40:14.021 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.021 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65c3d108:::benchmark_data_smithi090_166897_object14660:head# 2024-05-15T23:40:14.021 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.021 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65c43110:::benchmark_data_smithi090_166897_object41026:head# 2024-05-15T23:40:14.021 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.021 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65c5d8b8:::benchmark_data_smithi090_166897_object64088:head# 2024-05-15T23:40:14.021 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.022 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65c696dd:::benchmark_data_smithi090_166897_object56084:head# 2024-05-15T23:40:14.022 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.022 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65c6cfd8:::benchmark_data_smithi090_166897_object39636:head# 2024-05-15T23:40:14.022 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.022 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65c95fbd:::benchmark_data_smithi090_166897_object29790:head# 2024-05-15T23:40:14.022 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.022 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65c9e521:::benchmark_data_smithi090_166897_object19677:head# 2024-05-15T23:40:14.022 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.022 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65cab72b:::benchmark_data_smithi090_166897_object47674:head# 2024-05-15T23:40:14.022 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.022 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65caeabd:::benchmark_data_smithi090_166897_object6524:head# 2024-05-15T23:40:14.022 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.022 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65cc1f58:::benchmark_data_smithi090_166897_object66675:head# 2024-05-15T23:40:14.022 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.023 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65ce17d6:::benchmark_data_smithi090_166897_object1244:head# 2024-05-15T23:40:14.023 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.023 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65d07907:::benchmark_data_smithi090_166897_object50040:head# 2024-05-15T23:40:14.023 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.023 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65d29644:::benchmark_data_smithi090_166897_object1339:head# 2024-05-15T23:40:14.023 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.023 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65d31aa8:::benchmark_data_smithi090_166897_object54952:head# 2024-05-15T23:40:14.023 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.023 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65d56c41:::benchmark_data_smithi090_166897_object54190:head# 2024-05-15T23:40:14.023 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.023 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65d5ae44:::benchmark_data_smithi090_166897_object23468:head# 2024-05-15T23:40:14.023 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.023 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65d66a9c:::benchmark_data_smithi090_166897_object43554:head# 2024-05-15T23:40:14.023 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.023 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65d6ffcf:::benchmark_data_smithi090_166897_object5345:head# 2024-05-15T23:40:14.024 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.024 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65d7db53:::benchmark_data_smithi090_166897_object46071:head# 2024-05-15T23:40:14.024 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.024 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65d84a4a:::benchmark_data_smithi090_166897_object54781:head# 2024-05-15T23:40:14.024 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.024 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65d84d53:::benchmark_data_smithi090_166897_object41754:head# 2024-05-15T23:40:14.024 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.024 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65d89600:::benchmark_data_smithi090_166897_object35885:head# 2024-05-15T23:40:14.024 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.024 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65d9f74c:::benchmark_data_smithi090_166897_object33756:head# 2024-05-15T23:40:14.024 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.024 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65db3f86:::benchmark_data_smithi090_166897_object62127:head# 2024-05-15T23:40:14.024 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.024 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65dd0f75:::benchmark_data_smithi090_166897_object64836:head# 2024-05-15T23:40:14.024 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65dd1a15:::benchmark_data_smithi090_166897_object63835:head# 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65df8cf3:::benchmark_data_smithi090_166897_object9316:head# 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65e0a109:::benchmark_data_smithi090_166897_object64808:head# 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65e11828:::benchmark_data_smithi090_166897_object29574:head# 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65e145d2:::benchmark_data_smithi090_166897_object29915:head# 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65e1836b:::benchmark_data_smithi090_166897_object6884:head# 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65e1a77a:::benchmark_data_smithi090_166897_object34345:head# 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65e39300:::benchmark_data_smithi090_166897_object6141:head# 2024-05-15T23:40:14.025 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.026 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65e461be:::benchmark_data_smithi090_166897_object32601:head# 2024-05-15T23:40:14.026 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.026 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65e5a7c9:::benchmark_data_smithi090_166897_object37961:head# 2024-05-15T23:40:14.026 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.026 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65e5aea2:::benchmark_data_smithi090_166897_object54961:head# 2024-05-15T23:40:14.026 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.026 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65e69586:::benchmark_data_smithi090_166897_object2920:head# 2024-05-15T23:40:14.026 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.026 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65e6b815:::benchmark_data_smithi090_166897_object45256:head# 2024-05-15T23:40:14.026 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.026 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65e6ecb2:::benchmark_data_smithi090_166897_object69567:head# 2024-05-15T23:40:14.026 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.026 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65e739cd:::benchmark_data_smithi090_166897_object60684:head# 2024-05-15T23:40:14.026 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.026 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65e79708:::benchmark_data_smithi090_166897_object34812:head# 2024-05-15T23:40:14.027 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.027 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65e9e669:::benchmark_data_smithi090_166897_object54849:head# 2024-05-15T23:40:14.027 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.027 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65eb230c:::benchmark_data_smithi090_166897_object20207:head# 2024-05-15T23:40:14.027 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.027 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65edbf8d:::benchmark_data_smithi090_166897_object48588:head# 2024-05-15T23:40:14.027 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.027 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65eec8b0:::benchmark_data_smithi090_166897_object22720:head# 2024-05-15T23:40:14.027 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.027 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65efa81e:::benchmark_data_smithi090_166897_object46598:head# 2024-05-15T23:40:14.027 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.027 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65f0a1cf:::benchmark_data_smithi090_166897_object68703:head# 2024-05-15T23:40:14.027 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.027 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65f0c3e8:::benchmark_data_smithi090_166897_object53521:head# 2024-05-15T23:40:14.027 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.028 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65f0d78e:::benchmark_data_smithi090_166897_object3217:head# 2024-05-15T23:40:14.028 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.028 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65f2a63b:::benchmark_data_smithi090_166897_object65256:head# 2024-05-15T23:40:14.028 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.028 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65f5fcb1:::benchmark_data_smithi090_166897_object61101:head# 2024-05-15T23:40:14.028 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.028 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65f71865:::benchmark_data_smithi090_166897_object18295:head# 2024-05-15T23:40:14.028 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.028 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65f71d40:::benchmark_data_smithi090_166897_object3940:head# 2024-05-15T23:40:14.028 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.028 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65f7564c:::benchmark_data_smithi090_166897_object48151:head# 2024-05-15T23:40:14.028 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.028 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65f7a8d8:::benchmark_data_smithi090_166897_object6743:head# 2024-05-15T23:40:14.028 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.028 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65f7b534:::benchmark_data_smithi090_166897_object10465:head# 2024-05-15T23:40:14.029 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.029 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65f8dd1e:::benchmark_data_smithi090_166897_object17472:head# 2024-05-15T23:40:14.029 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.029 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65f8ddbe:::benchmark_data_smithi090_166897_object65206:head# 2024-05-15T23:40:14.029 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.029 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65fa1d54:::benchmark_data_smithi090_166897_object45717:head# 2024-05-15T23:40:14.029 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.029 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65fb617a:::benchmark_data_smithi090_166897_object61046:head# 2024-05-15T23:40:14.029 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.029 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65fc2a1a:::benchmark_data_smithi090_166897_object67152:head# 2024-05-15T23:40:14.029 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.029 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65fcaad2:::benchmark_data_smithi090_166897_object29796:head# 2024-05-15T23:40:14.029 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.029 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65fcb79c:::benchmark_data_smithi090_166897_object56707:head# 2024-05-15T23:40:14.029 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.030 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:65feff69:::benchmark_data_smithi090_166897_object43174:head# 2024-05-15T23:40:14.030 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.030 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6600d3c7:::benchmark_data_smithi090_166897_object55856:head# 2024-05-15T23:40:14.030 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.030 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6602490a:::benchmark_data_smithi090_166897_object5703:head# 2024-05-15T23:40:14.030 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.030 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6603e6ad:::benchmark_data_smithi090_166897_object33479:head# 2024-05-15T23:40:14.030 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.030 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66047b53:::benchmark_data_smithi090_166897_object1507:head# 2024-05-15T23:40:14.030 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:40:14.081 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6604fcf8:::benchmark_data_smithi090_166897_object63318:head# 2024-05-15T23:40:14.081 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.081 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6606164e:::benchmark_data_smithi090_166897_object45107:head# 2024-05-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66063c48:::benchmark_data_smithi090_166897_object39937:head# 2024-05-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66092d3b:::benchmark_data_smithi090_166897_object23848:head# 2024-05-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:660a685d:::benchmark_data_smithi090_166897_object64388:head# 2024-05-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:660aadd5:::benchmark_data_smithi090_166897_object4941:head# 2024-05-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:660ab92f:::benchmark_data_smithi090_166897_object15655:head# 2024-05-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:660c1f13:::benchmark_data_smithi090_166897_object11845:head# 2024-05-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:660c2de1:::benchmark_data_smithi090_166897_object49260:head# 2024-05-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:660d3a0b:::benchmark_data_smithi090_166897_object70968:head# 2024-05-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:660d701a:::benchmark_data_smithi090_166897_object12957:head# 2024-05-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:660f84b4:::benchmark_data_smithi090_166897_object41014:head# 2024-05-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:660fa740:::benchmark_data_smithi090_166897_object45470:head# 2024-05-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:660fe249:::benchmark_data_smithi090_166897_object66054:head# 2024-05-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:661096a0:::benchmark_data_smithi090_166897_object10520:head# 2024-05-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.084 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6610cc7b:::benchmark_data_smithi090_166897_object24961:head# 2024-05-15T23:40:14.084 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.084 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6610faa2:::benchmark_data_smithi090_166897_object66932:head# 2024-05-15T23:40:14.084 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.084 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6611527a:::benchmark_data_smithi090_166897_object68093:head# 2024-05-15T23:40:14.084 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.084 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6611cbd6:::benchmark_data_smithi090_166897_object40158:head# 2024-05-15T23:40:14.084 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.084 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66158ad9:::benchmark_data_smithi090_166897_object59060:head# 2024-05-15T23:40:14.084 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.084 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6616f736:::benchmark_data_smithi090_166897_object22095:head# 2024-05-15T23:40:14.084 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.084 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66177984:::benchmark_data_smithi090_166897_object62776:head# 2024-05-15T23:40:14.084 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.084 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6617aa40:::benchmark_data_smithi090_166897_object52687:head# 2024-05-15T23:40:14.085 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.085 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6617dd7c:::benchmark_data_smithi090_166897_object51073:head# 2024-05-15T23:40:14.085 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.085 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66188100:::benchmark_data_smithi090_166897_object1296:head# 2024-05-15T23:40:14.085 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.085 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66198a3d:::benchmark_data_smithi090_166897_object31505:head# 2024-05-15T23:40:14.085 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.085 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6619cf7c:::benchmark_data_smithi090_166897_object10380:head# 2024-05-15T23:40:14.085 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.085 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:661dbcdb:::benchmark_data_smithi090_166897_object3121:head# 2024-05-15T23:40:14.085 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.085 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:661fd420:::benchmark_data_smithi090_166897_object20224:head# 2024-05-15T23:40:14.085 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.085 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66208294:::benchmark_data_smithi090_166897_object50806:head# 2024-05-15T23:40:14.086 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.086 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6620bb11:::benchmark_data_smithi090_166897_object20801:head# 2024-05-15T23:40:14.086 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.086 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6621308a:::benchmark_data_smithi090_166897_object4788:head# 2024-05-15T23:40:14.086 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.086 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66216be7:::benchmark_data_smithi090_166897_object59813:head# 2024-05-15T23:40:14.086 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.086 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:662184f4:::benchmark_data_smithi090_166897_object45988:head# 2024-05-15T23:40:14.086 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.086 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66221165:::benchmark_data_smithi090_166897_object46962:head# 2024-05-15T23:40:14.086 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.086 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66226208:::benchmark_data_smithi090_166897_object7082:head# 2024-05-15T23:40:14.086 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.086 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66229250:::benchmark_data_smithi090_166897_object38230:head# 2024-05-15T23:40:14.087 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.087 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66233c19:::benchmark_data_smithi090_166897_object15391:head# 2024-05-15T23:40:14.087 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.087 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6623c35c:::benchmark_data_smithi090_166897_object29788:head# 2024-05-15T23:40:14.087 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.087 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66261b83:::benchmark_data_smithi090_166897_object6882:head# 2024-05-15T23:40:14.087 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.087 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6629d6ef:::benchmark_data_smithi090_166897_object28819:head# 2024-05-15T23:40:14.087 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.087 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:662a338e:::benchmark_data_smithi090_166897_object63171:head# 2024-05-15T23:40:14.087 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.087 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:662a545d:::benchmark_data_smithi090_166897_object36708:head# 2024-05-15T23:40:14.087 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.087 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:662d4620:::benchmark_data_smithi090_166897_object16731:head# 2024-05-15T23:40:14.088 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.088 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:662d7ca3:::benchmark_data_smithi090_166897_object18273:head# 2024-05-15T23:40:14.088 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.088 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:662dda76:::benchmark_data_smithi090_166897_object2395:head# 2024-05-15T23:40:14.088 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.088 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:662e8fcc:::benchmark_data_smithi090_166897_object1879:head# 2024-05-15T23:40:14.088 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.088 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:662fa48d:::benchmark_data_smithi090_166897_object26607:head# 2024-05-15T23:40:14.088 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.088 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:662fcacf:::benchmark_data_smithi090_166897_object10034:head# 2024-05-15T23:40:14.088 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.088 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6630052a:::benchmark_data_smithi090_166897_object790:head# 2024-05-15T23:40:14.088 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.088 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6630c1a0:::benchmark_data_smithi090_166897_object13566:head# 2024-05-15T23:40:14.088 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66322cd7:::benchmark_data_smithi090_166897_object22493:head# 2024-05-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:663263e3:::benchmark_data_smithi090_166897_object38590:head# 2024-05-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:663333d9:::benchmark_data_smithi090_166897_object55874:head# 2024-05-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66337ad6:::benchmark_data_smithi090_166897_object51720:head# 2024-05-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:663498a2:::benchmark_data_smithi090_166897_object55389:head# 2024-05-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6634a18f:::benchmark_data_smithi090_166897_object57335:head# 2024-05-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6634bcf0:::benchmark_data_smithi090_166897_object63392:head# 2024-05-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66355d9a:::benchmark_data_smithi090_166897_object67129:head# 2024-05-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:663625ce:::benchmark_data_smithi090_166897_object67295:head# 2024-05-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6637b7db:::benchmark_data_smithi090_166897_object32025:head# 2024-05-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6637ff16:::benchmark_data_smithi090_166897_object15332:head# 2024-05-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:663a809f:::benchmark_data_smithi090_166897_object43033:head# 2024-05-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:663c12b5:::benchmark_data_smithi090_166897_object2009:head# 2024-05-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:663ccfe7:::benchmark_data_smithi090_166897_object51147:head# 2024-05-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:663cdfc2:::benchmark_data_smithi090_166897_object45428:head# 2024-05-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:663d7c35:::benchmark_data_smithi090_166897_object58971:head# 2024-05-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:664103ed:::benchmark_data_smithi090_166897_object11635:head# 2024-05-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6641bcc3:::benchmark_data_smithi090_166897_object31725:head# 2024-05-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66421ab2:::benchmark_data_smithi090_166897_object40230:head# 2024-05-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66444976:::benchmark_data_smithi090_166897_object20438:head# 2024-05-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:664475b6:::benchmark_data_smithi090_166897_object8330:head# 2024-05-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.092 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:664791f1:::benchmark_data_smithi090_166897_object9019:head# 2024-05-15T23:40:14.092 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.092 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6647a0d7:::benchmark_data_smithi090_166897_object6067:head# 2024-05-15T23:40:14.092 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.092 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6647f87e:::benchmark_data_smithi090_166897_object26796:head# 2024-05-15T23:40:14.092 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.092 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66481e3a:::benchmark_data_smithi090_166897_object59698:head# 2024-05-15T23:40:14.092 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.092 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6648b697:::benchmark_data_smithi090_166897_object3955:head# 2024-05-15T23:40:14.092 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.092 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66490d20:::benchmark_data_smithi090_166897_object38236:head# 2024-05-15T23:40:14.092 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.092 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:664bc451:::benchmark_data_smithi090_166897_object67121:head# 2024-05-15T23:40:14.092 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.093 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:664f95e4:::benchmark_data_smithi090_166897_object16716:head# 2024-05-15T23:40:14.093 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.093 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:664fb970:::benchmark_data_smithi090_166897_object8812:head# 2024-05-15T23:40:14.093 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.093 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66504e02:::benchmark_data_smithi090_166897_object37163:head# 2024-05-15T23:40:14.093 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.093 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66505a22:::benchmark_data_smithi090_166897_object56958:head# 2024-05-15T23:40:14.093 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.093 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6650d417:::benchmark_data_smithi090_166897_object59466:head# 2024-05-15T23:40:14.093 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.093 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66521475:::benchmark_data_smithi090_166897_object34199:head# 2024-05-15T23:40:14.093 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.093 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66530391:::benchmark_data_smithi090_166897_object36813:head# 2024-05-15T23:40:14.093 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.093 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66546e2b:::benchmark_data_smithi090_166897_object42449:head# 2024-05-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66550de1:::benchmark_data_smithi090_166897_object26344:head# 2024-05-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66577095:::benchmark_data_smithi090_166897_object1088:head# 2024-05-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6657a889:::benchmark_data_smithi090_166897_object14661:head# 2024-05-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6657b6d6:::benchmark_data_smithi090_166897_object13815:head# 2024-05-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6659bc83:::benchmark_data_smithi090_166897_object63334:head# 2024-05-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:665a0cd5:::benchmark_data_smithi090_166897_object20391:head# 2024-05-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:665a25a0:::benchmark_data_smithi090_166897_object56003:head# 2024-05-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:665ab40e:::benchmark_data_smithi090_166897_object61579:head# 2024-05-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:665e8f4d:::benchmark_data_smithi090_166897_object34454:head# 2024-05-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:665f82f5:::benchmark_data_smithi090_166897_object52504:head# 2024-05-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:665ff639:::benchmark_data_smithi090_166897_object63633:head# 2024-05-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6660d5e0:::benchmark_data_smithi090_166897_object28911:head# 2024-05-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66618123:::benchmark_data_smithi090_166897_object62291:head# 2024-05-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6662ec9d:::benchmark_data_smithi090_166897_object46744:head# 2024-05-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6662feca:::benchmark_data_smithi090_166897_object59708:head# 2024-05-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:666468b6:::benchmark_data_smithi090_166897_object23733:head# 2024-05-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6664bd45:::benchmark_data_smithi090_166897_object14407:head# 2024-05-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6664d98c:::benchmark_data_smithi090_166897_object43929:head# 2024-05-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66654a45:::benchmark_data_smithi090_166897_object36121:head# 2024-05-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66663005:::benchmark_data_smithi090_166897_object30270:head# 2024-05-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:666839ce:::benchmark_data_smithi090_166897_object12845:head# 2024-05-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66685c9c:::benchmark_data_smithi090_166897_object28135:head# 2024-05-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6668825e:::benchmark_data_smithi090_166897_object28141:head# 2024-05-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66695922:::benchmark_data_smithi090_166897_object26207:head# 2024-05-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:666aa11e:::benchmark_data_smithi090_166897_object40270:head# 2024-05-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:666b2b09:::benchmark_data_smithi090_166897_object64370:head# 2024-05-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:666d03d9:::benchmark_data_smithi090_166897_object14698:head# 2024-05-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:666d0955:::benchmark_data_smithi090_166897_object44707:head# 2024-05-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:666d9d14:::benchmark_data_smithi090_166897_object53036:head# 2024-05-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:666dd72c:::benchmark_data_smithi090_166897_object63964:head# 2024-05-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:666e09e4:::benchmark_data_smithi090_166897_object43999:head# 2024-05-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:666e6447:::benchmark_data_smithi090_166897_object4693:head# 2024-05-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6670ea9f:::benchmark_data_smithi090_166897_object40166:head# 2024-05-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6670edfb:::benchmark_data_smithi090_166897_object31031:head# 2024-05-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66751f95:::benchmark_data_smithi090_166897_object69803:head# 2024-05-15T23:40:14.099 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.099 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66753550:::benchmark_data_smithi090_166897_object37072:head# 2024-05-15T23:40:14.099 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.099 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:667620bd:::benchmark_data_smithi090_166897_object46944:head# 2024-05-15T23:40:14.099 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.099 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6678d499:::benchmark_data_smithi090_166897_object21644:head# 2024-05-15T23:40:14.099 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.099 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:667943f5:::benchmark_data_smithi090_166897_object10904:head# 2024-05-15T23:40:14.099 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.099 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:667a686a:::benchmark_data_smithi090_166897_object10941:head# 2024-05-15T23:40:14.099 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.099 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:667aef36:::benchmark_data_smithi090_166897_object30955:head# 2024-05-15T23:40:14.099 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.099 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:667b831e:::benchmark_data_smithi090_166897_object52728:head# 2024-05-15T23:40:14.099 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:667be894:::benchmark_data_smithi090_166897_object62930:head# 2024-05-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:667bf74a:::benchmark_data_smithi090_166897_object38046:head# 2024-05-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:667c9908:::benchmark_data_smithi090_166897_object57731:head# 2024-05-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:667d04e7:::benchmark_data_smithi090_166897_object62003:head# 2024-05-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:667d17f1:::benchmark_data_smithi090_166897_object32730:head# 2024-05-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:667d8011:::benchmark_data_smithi090_166897_object45269:head# 2024-05-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:667e961c:::benchmark_data_smithi090_166897_object37708:head# 2024-05-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:667eb162:::benchmark_data_smithi090_166897_object61270:head# 2024-05-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66805cff:::benchmark_data_smithi090_166897_object9476:head# 2024-05-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6681b2ab:::benchmark_data_smithi090_166897_object10670:head# 2024-05-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66822820:::benchmark_data_smithi090_166897_object47316:head# 2024-05-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6682d22c:::benchmark_data_smithi090_166897_object54287:head# 2024-05-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66839875:::benchmark_data_smithi090_166897_object8628:head# 2024-05-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6685c072:::benchmark_data_smithi090_166897_object65004:head# 2024-05-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6686788c:::benchmark_data_smithi090_166897_object28252:head# 2024-05-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66876e52:::benchmark_data_smithi090_166897_object9912:head# 2024-05-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6688557d:::benchmark_data_smithi090_166897_object44241:head# 2024-05-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6688bb3b:::benchmark_data_smithi090_166897_object26316:head# 2024-05-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:668936b7:::benchmark_data_smithi090_166897_object7365:head# 2024-05-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6689b297:::benchmark_data_smithi090_166897_object41808:head# 2024-05-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:668a836c:::benchmark_data_smithi090_166897_object52118:head# 2024-05-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:668bdfb8:::benchmark_data_smithi090_166897_object69082:head# 2024-05-15T23:40:14.103 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.103 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:668c2784:::benchmark_data_smithi090_166897_object10089:head# 2024-05-15T23:40:14.103 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.103 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:668d0b7d:::benchmark_data_smithi090_166897_object40075:head# 2024-05-15T23:40:14.103 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.103 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:668dfbc3:::benchmark_data_smithi090_166897_object50930:head# 2024-05-15T23:40:14.103 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:668fa5a2:::benchmark_data_smithi090_166897_object65455:head# 2024-05-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66904f3e:::benchmark_data_smithi090_166897_object39788:head# 2024-05-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:669090d8:::benchmark_data_smithi090_166897_object23027:head# 2024-05-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:669250ec:::benchmark_data_smithi090_166897_object40080:head# 2024-05-15T23:40:14.105 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.105 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:669528e7:::benchmark_data_smithi090_166897_object30640:head# 2024-05-15T23:40:14.105 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.105 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:669734a2:::benchmark_data_smithi090_166897_object50861:head# 2024-05-15T23:40:14.105 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.105 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6697abc9:::benchmark_data_smithi090_166897_object56880:head# 2024-05-15T23:40:14.105 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.106 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6698f1a0:::benchmark_data_smithi090_166897_object14754:head# 2024-05-15T23:40:14.106 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.106 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66990910:::benchmark_data_smithi090_166897_object68616:head# 2024-05-15T23:40:14.106 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.106 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:669a12fb:::benchmark_data_smithi090_166897_object8573:head# 2024-05-15T23:40:14.106 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.106 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:669cb481:::benchmark_data_smithi090_166897_object41992:head# 2024-05-15T23:40:14.106 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.106 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:669e8b76:::benchmark_data_smithi090_166897_object1011:head# 2024-05-15T23:40:14.106 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.106 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:669f79fc:::benchmark_data_smithi090_166897_object64305:head# 2024-05-15T23:40:14.106 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.106 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:669ffe5a:::benchmark_data_smithi090_166897_object46107:head# 2024-05-15T23:40:14.106 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.107 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66a19837:::benchmark_data_smithi090_166897_object50062:head# 2024-05-15T23:40:14.107 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.107 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66a1e911:::benchmark_data_smithi090_166897_object26304:head# 2024-05-15T23:40:14.107 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.107 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66a25cd8:::benchmark_data_smithi090_166897_object68187:head# 2024-05-15T23:40:14.107 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.107 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66a2f754:::benchmark_data_smithi090_166897_object39481:head# 2024-05-15T23:40:14.107 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.107 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66a3644a:::benchmark_data_smithi090_166897_object35014:head# 2024-05-15T23:40:14.107 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.107 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66a37d49:::benchmark_data_smithi090_166897_object53159:head# 2024-05-15T23:40:14.107 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.107 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66a44d08:::benchmark_data_smithi090_166897_object37720:head# 2024-05-15T23:40:14.107 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.107 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66a4c4a8:::benchmark_data_smithi090_166897_object28915:head# 2024-05-15T23:40:14.108 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.108 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66a5a049:::benchmark_data_smithi090_166897_object18713:head# 2024-05-15T23:40:14.108 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.108 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66a692b1:::benchmark_data_smithi090_166897_object68177:head# 2024-05-15T23:40:14.108 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.108 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66a77b4c:::benchmark_data_smithi090_166897_object69506:head# 2024-05-15T23:40:14.108 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.108 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66a7a845:::benchmark_data_smithi090_166897_object34272:head# 2024-05-15T23:40:14.108 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.108 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66a7c1f8:::benchmark_data_smithi090_166897_object53250:head# 2024-05-15T23:40:14.108 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.108 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66a8565c:::benchmark_data_smithi090_166897_object26043:head# 2024-05-15T23:40:14.108 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.108 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66a9b615:::benchmark_data_smithi090_166897_object14540:head# 2024-05-15T23:40:14.108 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.109 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66aa2d98:::benchmark_data_smithi090_166897_object47400:head# 2024-05-15T23:40:14.109 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.109 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66ab147a:::benchmark_data_smithi090_166897_object2734:head# 2024-05-15T23:40:14.109 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.109 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66af3d86:::benchmark_data_smithi090_166897_object13130:head# 2024-05-15T23:40:14.109 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.109 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66b0704c:::benchmark_data_smithi090_166897_object4811:head# 2024-05-15T23:40:14.109 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.109 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66b2e407:::benchmark_data_smithi090_166897_object63993:head# 2024-05-15T23:40:14.109 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.109 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66b4807e:::benchmark_data_smithi090_166897_object49500:head# 2024-05-15T23:40:14.109 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.109 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66b50953:::benchmark_data_smithi090_166897_object55558:head# 2024-05-15T23:40:14.109 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.109 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66b5b709:::benchmark_data_smithi090_166897_object25874:head# 2024-05-15T23:40:14.110 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.110 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66b63728:::benchmark_data_smithi090_166897_object51995:head# 2024-05-15T23:40:14.110 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.110 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66b9abf3:::benchmark_data_smithi090_166897_object51599:head# 2024-05-15T23:40:14.110 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.110 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66baab14:::benchmark_data_smithi090_166897_object32669:head# 2024-05-15T23:40:14.110 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.110 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66bb3897:::benchmark_data_smithi090_166897_object26457:head# 2024-05-15T23:40:14.110 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.110 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66bb9cf3:::benchmark_data_smithi090_166897_object48301:head# 2024-05-15T23:40:14.110 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.110 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66bc0cb1:::benchmark_data_smithi090_166897_object15110:head# 2024-05-15T23:40:14.110 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.110 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66bcddfc:::benchmark_data_smithi090_166897_object11090:head# 2024-05-15T23:40:14.111 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.111 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66bd2669:::benchmark_data_smithi090_166897_object25481:head# 2024-05-15T23:40:14.111 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.111 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66bd5eaf:::benchmark_data_smithi090_166897_object63336:head# 2024-05-15T23:40:14.111 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.111 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66bd7619:::benchmark_data_smithi090_166897_object52019:head# 2024-05-15T23:40:14.111 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.111 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66bd8ce2:::benchmark_data_smithi090_166897_object54865:head# 2024-05-15T23:40:14.111 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.111 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66bdab22:::benchmark_data_smithi090_166897_object66532:head# 2024-05-15T23:40:14.111 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.111 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66bdf51c:::benchmark_data_smithi090_166897_object27114:head# 2024-05-15T23:40:14.111 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.111 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66bf95fa:::benchmark_data_smithi090_166897_object31410:head# 2024-05-15T23:40:14.111 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.112 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66bff6bc:::benchmark_data_smithi090_166897_object49924:head# 2024-05-15T23:40:14.112 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.112 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66c11f72:::benchmark_data_smithi090_166897_object59331:head# 2024-05-15T23:40:14.112 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.112 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66c26e94:::benchmark_data_smithi090_166897_object20513:head# 2024-05-15T23:40:14.112 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.112 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66c39d0d:::benchmark_data_smithi090_166897_object41965:head# 2024-05-15T23:40:14.112 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.112 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66c48502:::benchmark_data_smithi090_166897_object67775:head# 2024-05-15T23:40:14.112 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.112 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66c54df5:::benchmark_data_smithi090_166897_object18726:head# 2024-05-15T23:40:14.112 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.112 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66c90c89:::benchmark_data_smithi090_166897_object12564:head# 2024-05-15T23:40:14.112 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.113 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66c963f8:::benchmark_data_smithi090_166897_object54663:head# 2024-05-15T23:40:14.113 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.113 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66c9efff:::benchmark_data_smithi090_166897_object65325:head# 2024-05-15T23:40:14.113 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.113 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66ca8ac7:::benchmark_data_smithi090_166897_object5256:head# 2024-05-15T23:40:14.113 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.113 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66ca9f44:::benchmark_data_smithi090_166897_object71074:head# 2024-05-15T23:40:14.113 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.113 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66cc44c5:::benchmark_data_smithi090_166897_object460:head# 2024-05-15T23:40:14.113 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.113 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66cc98dc:::benchmark_data_smithi090_166897_object53750:head# 2024-05-15T23:40:14.113 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.113 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66cea109:::benchmark_data_smithi090_166897_object27144:head# 2024-05-15T23:40:14.113 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.113 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66d18df7:::benchmark_data_smithi090_166897_object22222:head# 2024-05-15T23:40:14.114 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.114 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66d1c1a8:::benchmark_data_smithi090_166897_object49131:head# 2024-05-15T23:40:14.114 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.114 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66d1e3c7:::benchmark_data_smithi090_166897_object10666:head# 2024-05-15T23:40:14.114 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.114 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66d2f865:::benchmark_data_smithi090_166897_object3839:head# 2024-05-15T23:40:14.114 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.114 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66d5a954:::benchmark_data_smithi090_166897_object39580:head# 2024-05-15T23:40:14.114 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.114 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66d6324e:::benchmark_data_smithi090_166897_object8987:head# 2024-05-15T23:40:14.114 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.114 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66d6a3b9:::benchmark_data_smithi090_166897_object26966:head# 2024-05-15T23:40:14.114 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.114 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66d717f4:::benchmark_data_smithi090_166897_object12646:head# 2024-05-15T23:40:14.114 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.115 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66da41c0:::benchmark_data_smithi090_166897_object65677:head# 2024-05-15T23:40:14.115 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.115 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66db6185:::benchmark_data_smithi090_166897_object70136:head# 2024-05-15T23:40:14.115 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.115 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66dd3687:::benchmark_data_smithi090_166897_object6513:head# 2024-05-15T23:40:14.115 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.115 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66de4daa:::benchmark_data_smithi090_166897_object65795:head# 2024-05-15T23:40:14.115 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.115 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66df4ebe:::benchmark_data_smithi090_166897_object58927:head# 2024-05-15T23:40:14.115 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.115 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66e040fc:::benchmark_data_smithi090_166897_object59484:head# 2024-05-15T23:40:14.115 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.115 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66e08af2:::benchmark_data_smithi090_166897_object63769:head# 2024-05-15T23:40:14.115 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.116 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66e3a6f1:::benchmark_data_smithi090_166897_object2182:head# 2024-05-15T23:40:14.116 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.116 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66e3c317:::benchmark_data_smithi090_166897_object8555:head# 2024-05-15T23:40:14.116 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.116 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66e49a97:::benchmark_data_smithi090_166897_object65287:head# 2024-05-15T23:40:14.116 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.116 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66e5bc2c:::benchmark_data_smithi090_166897_object38460:head# 2024-05-15T23:40:14.116 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.116 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66e69444:::benchmark_data_smithi090_166897_object61604:head# 2024-05-15T23:40:14.116 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.116 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66e73849:::benchmark_data_smithi090_166897_object51300:head# 2024-05-15T23:40:14.116 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.116 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66e7a56e:::benchmark_data_smithi090_166897_object47486:head# 2024-05-15T23:40:14.116 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.116 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66e85bb0:::benchmark_data_smithi090_166897_object59838:head# 2024-05-15T23:40:14.117 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.117 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66e92a09:::benchmark_data_smithi090_166897_object40558:head# 2024-05-15T23:40:14.117 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.117 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66e9b400:::benchmark_data_smithi090_166897_object45700:head# 2024-05-15T23:40:14.117 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.117 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66eb76f2:::benchmark_data_smithi090_166897_object2015:head# 2024-05-15T23:40:14.117 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.117 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66ebb73d:::benchmark_data_smithi090_166897_object38189:head# 2024-05-15T23:40:14.117 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.117 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66ec1d2e:::benchmark_data_smithi090_166897_object69025:head# 2024-05-15T23:40:14.117 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.117 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66edba8c:::benchmark_data_smithi090_166897_object2565:head# 2024-05-15T23:40:14.117 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.117 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66ee552f:::benchmark_data_smithi090_166897_object23753:head# 2024-05-15T23:40:14.117 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.118 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66ee6822:::benchmark_data_smithi090_166897_object38654:head# 2024-05-15T23:40:14.118 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.118 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66ef984d:::benchmark_data_smithi090_166897_object16119:head# 2024-05-15T23:40:14.118 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.118 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66ef99bf:::benchmark_data_smithi090_166897_object59310:head# 2024-05-15T23:40:14.118 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.118 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66f14040:::benchmark_data_smithi090_166897_object52696:head# 2024-05-15T23:40:14.118 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.118 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66f1ad6e:::benchmark_data_smithi090_166897_object9926:head# 2024-05-15T23:40:14.118 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.118 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66f28dd0:::benchmark_data_smithi090_166897_object52050:head# 2024-05-15T23:40:14.118 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.118 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66f2b6f5:::benchmark_data_smithi090_166897_object24839:head# 2024-05-15T23:40:14.118 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.118 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66f42da0:::benchmark_data_smithi090_166897_object22288:head# 2024-05-15T23:40:14.119 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.119 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66f53ecd:::benchmark_data_smithi090_166897_object12378:head# 2024-05-15T23:40:14.119 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.119 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66f561d4:::benchmark_data_smithi090_166897_object5670:head# 2024-05-15T23:40:14.119 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.119 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66f5c81c:::benchmark_data_smithi090_166897_object62180:head# 2024-05-15T23:40:14.119 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.119 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66f74305:::benchmark_data_smithi090_166897_object69153:head# 2024-05-15T23:40:14.119 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.119 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66f7b573:::benchmark_data_smithi090_166897_object50045:head# 2024-05-15T23:40:14.119 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.119 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66f7b583:::benchmark_data_smithi090_166897_object15012:head# 2024-05-15T23:40:14.119 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.119 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66f8b600:::benchmark_data_smithi090_166897_object61578:head# 2024-05-15T23:40:14.119 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.120 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66f96c25:::benchmark_data_smithi090_166897_object62600:head# 2024-05-15T23:40:14.120 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.120 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66fb7a5f:::benchmark_data_smithi090_166897_object5244:head# 2024-05-15T23:40:14.120 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.120 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66fd7b62:::benchmark_data_smithi090_166897_object26451:head# 2024-05-15T23:40:14.120 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.120 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66fe3786:::benchmark_data_smithi090_166897_object58811:head# 2024-05-15T23:40:14.120 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.120 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66fecf35:::benchmark_data_smithi090_166897_object36346:head# 2024-05-15T23:40:14.120 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.120 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:66ff7c68:::benchmark_data_smithi090_166897_object22256:head# 2024-05-15T23:40:14.120 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.120 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67028daa:::benchmark_data_smithi090_166897_object23460:head# 2024-05-15T23:40:14.120 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.120 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67038067:::benchmark_data_smithi090_166897_object8758:head# 2024-05-15T23:40:14.121 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.121 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67047fcb:::benchmark_data_smithi090_166897_object58449:head# 2024-05-15T23:40:14.121 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.121 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67069da7:::benchmark_data_smithi090_166897_object17565:head# 2024-05-15T23:40:14.121 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.121 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67089433:::benchmark_data_smithi090_166897_object65930:head# 2024-05-15T23:40:14.121 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.121 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:670a27f3:::benchmark_data_smithi090_166897_object8063:head# 2024-05-15T23:40:14.121 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.121 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:670b35f5:::benchmark_data_smithi090_166897_object32456:head# 2024-05-15T23:40:14.121 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.121 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:670b5c00:::benchmark_data_smithi090_166897_object1286:head# 2024-05-15T23:40:14.121 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.121 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:670bd3b0:::benchmark_data_smithi090_166897_object63304:head# 2024-05-15T23:40:14.121 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.122 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:670c155d:::benchmark_data_smithi090_166897_object64986:head# 2024-05-15T23:40:14.122 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.122 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:670df0a7:::benchmark_data_smithi090_166897_object13589:head# 2024-05-15T23:40:14.122 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.122 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:670e0fa4:::benchmark_data_smithi090_166897_object40562:head# 2024-05-15T23:40:14.122 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.122 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6714ad91:::benchmark_data_smithi090_166897_object19090:head# 2024-05-15T23:40:14.122 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.122 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6714e444:::benchmark_data_smithi090_166897_object30543:head# 2024-05-15T23:40:14.122 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.122 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67164d6a:::benchmark_data_smithi090_166897_object65054:head# 2024-05-15T23:40:14.122 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.122 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6716a9b5:::benchmark_data_smithi090_166897_object16000:head# 2024-05-15T23:40:14.122 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6718c50d:::benchmark_data_smithi090_166897_object20566:head# 2024-05-15T23:40:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6718c95c:::benchmark_data_smithi090_166897_object6649:head# 2024-05-15T23:40:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67193c5a:::benchmark_data_smithi090_166897_object32747:head# 2024-05-15T23:40:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:671f8ec0:::benchmark_data_smithi090_166897_object43334:head# 2024-05-15T23:40:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67200eaa:::benchmark_data_smithi090_166897_object26227:head# 2024-05-15T23:40:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6722ee10:::benchmark_data_smithi090_166897_object30811:head# 2024-05-15T23:40:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6723553d:::benchmark_data_smithi090_166897_object28117:head# 2024-05-15T23:40:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67239283:::benchmark_data_smithi090_166897_object49436:head# 2024-05-15T23:40:14.124 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.124 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6723f18d:::benchmark_data_smithi090_166897_object25675:head# 2024-05-15T23:40:14.124 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.124 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67240e40:::benchmark_data_smithi090_166897_object48894:head# 2024-05-15T23:40:14.124 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.124 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6724568b:::benchmark_data_smithi090_166897_object40265:head# 2024-05-15T23:40:14.124 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.124 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6724ee7e:::benchmark_data_smithi090_166897_object52432:head# 2024-05-15T23:40:14.124 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.124 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6726aba9:::benchmark_data_smithi090_166897_object10806:head# 2024-05-15T23:40:14.124 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.124 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6726ff69:::benchmark_data_smithi090_166897_object64130:head# 2024-05-15T23:40:14.124 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.124 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67271827:::benchmark_data_smithi090_166897_object37434:head# 2024-05-15T23:40:14.124 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.125 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67275a99:::benchmark_data_smithi090_166897_object19250:head# 2024-05-15T23:40:14.125 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.125 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67279ef4:::benchmark_data_smithi090_166897_object3999:head# 2024-05-15T23:40:14.125 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.125 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:672adc2e:::benchmark_data_smithi090_166897_object64811:head# 2024-05-15T23:40:14.125 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.125 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:672c7c23:::benchmark_data_smithi090_166897_object31424:head# 2024-05-15T23:40:14.125 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.125 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:672cd52f:::benchmark_data_smithi090_166897_object24713:head# 2024-05-15T23:40:14.125 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.125 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:672e4c71:::benchmark_data_smithi090_166897_object65205:head# 2024-05-15T23:40:14.125 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.125 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:672f52f1:::benchmark_data_smithi090_166897_object46823:head# 2024-05-15T23:40:14.125 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.125 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:673010ca:::benchmark_data_smithi090_166897_object54881:head# 2024-05-15T23:40:14.126 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.126 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6730f951:::benchmark_data_smithi090_166897_object24848:head# 2024-05-15T23:40:14.126 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.126 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67323849:::benchmark_data_smithi090_166897_object29337:head# 2024-05-15T23:40:14.126 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.126 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6732f380:::benchmark_data_smithi090_166897_object34762:head# 2024-05-15T23:40:14.126 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.126 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67333714:::benchmark_data_smithi090_166897_object9067:head# 2024-05-15T23:40:14.126 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.126 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6733a2e2:::benchmark_data_smithi090_166897_object32665:head# 2024-05-15T23:40:14.126 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.126 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67349f9f:::benchmark_data_smithi090_166897_object49547:head# 2024-05-15T23:40:14.126 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.126 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:673ab56f:::benchmark_data_smithi090_166897_object6386:head# 2024-05-15T23:40:14.127 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.127 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:673b82dd:::benchmark_data_smithi090_166897_object58726:head# 2024-05-15T23:40:14.127 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.127 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:673c4683:::benchmark_data_smithi090_166897_object36186:head# 2024-05-15T23:40:14.127 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.127 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:673cb670:::benchmark_data_smithi090_166897_object41442:head# 2024-05-15T23:40:14.127 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.127 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:673cc0fc:::benchmark_data_smithi090_166897_object10206:head# 2024-05-15T23:40:14.127 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.127 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:673d0586:::benchmark_data_smithi090_166897_object2317:head# 2024-05-15T23:40:14.127 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.127 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:673d4e12:::benchmark_data_smithi090_166897_object5123:head# 2024-05-15T23:40:14.127 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.127 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:673e05e1:::benchmark_data_smithi090_166897_object49360:head# 2024-05-15T23:40:14.127 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.128 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67414849:::benchmark_data_smithi090_166897_object57109:head# 2024-05-15T23:40:14.128 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.128 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67418df8:::benchmark_data_smithi090_166897_object45384:head# 2024-05-15T23:40:14.128 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.128 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6741dae9:::benchmark_data_smithi090_166897_object26867:head# 2024-05-15T23:40:14.128 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.128 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67427619:::benchmark_data_smithi090_166897_object64376:head# 2024-05-15T23:40:14.128 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.128 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67429067:::benchmark_data_smithi090_166897_object11571:head# 2024-05-15T23:40:14.128 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.128 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6742a2fb:::benchmark_data_smithi090_166897_object45938:head# 2024-05-15T23:40:14.128 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.128 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6742c05f:::benchmark_data_smithi090_166897_object46468:head# 2024-05-15T23:40:14.128 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.128 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6743f6d2:::benchmark_data_smithi090_166897_object6822:head# 2024-05-15T23:40:14.129 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.129 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6744f312:::benchmark_data_smithi090_166897_object30112:head# 2024-05-15T23:40:14.129 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.129 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67486156:::benchmark_data_smithi090_166897_object28709:head# 2024-05-15T23:40:14.129 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.129 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67493243:::benchmark_data_smithi090_166897_object55728:head# 2024-05-15T23:40:14.129 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.129 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:674a24ee:::benchmark_data_smithi090_166897_object55128:head# 2024-05-15T23:40:14.129 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.129 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:674b870f:::benchmark_data_smithi090_166897_object17871:head# 2024-05-15T23:40:14.129 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.129 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:674cc837:::benchmark_data_smithi090_166897_object54604:head# 2024-05-15T23:40:14.129 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.129 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:674d2979:::benchmark_data_smithi090_166897_object2880:head# 2024-05-15T23:40:14.129 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.130 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6754b75e:::benchmark_data_smithi090_166897_object24491:head# 2024-05-15T23:40:14.130 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.130 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6754d1bb:::benchmark_data_smithi090_166897_object7484:head# 2024-05-15T23:40:14.130 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.130 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67552e34:::benchmark_data_smithi090_166897_object47091:head# 2024-05-15T23:40:14.130 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.130 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67556062:::benchmark_data_smithi090_166897_object4734:head# 2024-05-15T23:40:14.130 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.130 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6756e76c:::benchmark_data_smithi090_166897_object51928:head# 2024-05-15T23:40:14.130 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.130 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67594d08:::benchmark_data_smithi090_166897_object45254:head# 2024-05-15T23:40:14.130 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.130 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6759acbb:::benchmark_data_smithi090_166897_object497:head# 2024-05-15T23:40:14.130 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.130 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6759e000:::benchmark_data_smithi090_166897_object53476:head# 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:675a0a23:::benchmark_data_smithi090_166897_object11842:head# 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:675a4257:::benchmark_data_smithi090_166897_object39490:head# 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:675ab3eb:::benchmark_data_smithi090_166897_object32972:head# 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:675b02e6:::benchmark_data_smithi090_166897_object1833:head# 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:675b48ad:::benchmark_data_smithi090_166897_object52545:head# 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:675b8bcb:::benchmark_data_smithi090_166897_object33956:head# 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:675c3a6c:::benchmark_data_smithi090_166897_object61534:head# 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.131 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:675c8cbe:::benchmark_data_smithi090_166897_object18266:head# 2024-05-15T23:40:14.132 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.132 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:675cfbd4:::benchmark_data_smithi090_166897_object41488:head# 2024-05-15T23:40:14.132 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.132 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6764a7fc:::benchmark_data_smithi090_166897_object52789:head# 2024-05-15T23:40:14.132 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.132 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:676641be:::benchmark_data_smithi090_166897_object17037:head# 2024-05-15T23:40:14.132 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.132 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67687e4e:::benchmark_data_smithi090_166897_object22308:head# 2024-05-15T23:40:14.132 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.132 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67688bdd:::benchmark_data_smithi090_166897_object12637:head# 2024-05-15T23:40:14.132 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.132 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:676acd54:::benchmark_data_smithi090_166897_object30786:head# 2024-05-15T23:40:14.132 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.132 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:676b0b46:::benchmark_data_smithi090_166897_object54530:head# 2024-05-15T23:40:14.132 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.133 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:676cb7f6:::benchmark_data_smithi090_166897_object47827:head# 2024-05-15T23:40:14.133 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.133 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:676d269c:::benchmark_data_smithi090_166897_object37406:head# 2024-05-15T23:40:14.133 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.133 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:676e09ad:::benchmark_data_smithi090_166897_object50017:head# 2024-05-15T23:40:14.133 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.133 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:676f0add:::benchmark_data_smithi090_166897_object4779:head# 2024-05-15T23:40:14.133 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.133 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:676f30a7:::benchmark_data_smithi090_166897_object69358:head# 2024-05-15T23:40:14.133 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.134 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:13 smithi088 ceph-mon[44699]: pgmap v2032: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.2 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 69 MiB/s wr, 1.11k op/s; 12103/204000 objects degraded (5.933%); 12 MiB/s, 199 objects/s recovering 2024-05-15T23:40:14.136 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:14.156 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:40:14.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:14.157 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:40:14.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:40:14.209 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:676fa6d5:::benchmark_data_smithi090_166897_object8788:head# 2024-05-15T23:40:14.209 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.209 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67703d2e:::benchmark_data_smithi090_166897_object2767:head# 2024-05-15T23:40:14.209 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.209 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:677121dc:::benchmark_data_smithi090_166897_object27572:head# 2024-05-15T23:40:14.209 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.209 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67717b18:::benchmark_data_smithi090_166897_object1987:head# 2024-05-15T23:40:14.209 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.209 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67721c04:::benchmark_data_smithi090_166897_object8373:head# 2024-05-15T23:40:14.209 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.209 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67728450:::benchmark_data_smithi090_166897_object10976:head# 2024-05-15T23:40:14.210 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.210 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67783802:::benchmark_data_smithi090_166897_object37630:head# 2024-05-15T23:40:14.210 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.210 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:677a7d22:::benchmark_data_smithi090_166897_object54944:head# 2024-05-15T23:40:14.210 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.210 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:677a8867:::benchmark_data_smithi090_166897_object27020:head# 2024-05-15T23:40:14.210 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.210 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:677b3dee:::benchmark_data_smithi090_166897_object48562:head# 2024-05-15T23:40:14.210 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.210 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:677b7b90:::benchmark_data_smithi090_166897_object36261:head# 2024-05-15T23:40:14.210 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.210 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:677c4e5c:::benchmark_data_smithi090_166897_object36070:head# 2024-05-15T23:40:14.210 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.211 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:677c9b1c:::benchmark_data_smithi090_166897_object42487:head# 2024-05-15T23:40:14.211 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.211 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:677d672c:::benchmark_data_smithi090_166897_object67849:head# 2024-05-15T23:40:14.211 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.211 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:677ec246:::benchmark_data_smithi090_166897_object50992:head# 2024-05-15T23:40:14.211 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.211 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:677fcb3e:::benchmark_data_smithi090_166897_object16928:head# 2024-05-15T23:40:14.211 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.211 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:677fd4c8:::benchmark_data_smithi090_166897_object66201:head# 2024-05-15T23:40:14.211 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.211 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67814091:::benchmark_data_smithi090_166897_object46229:head# 2024-05-15T23:40:14.211 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.211 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67817b6a:::benchmark_data_smithi090_166897_object44731:head# 2024-05-15T23:40:14.211 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.212 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6782d985:::benchmark_data_smithi090_166897_object55624:head# 2024-05-15T23:40:14.212 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.212 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6785795c:::benchmark_data_smithi090_166897_object51005:head# 2024-05-15T23:40:14.212 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.212 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67861fc2:::benchmark_data_smithi090_166897_object65864:head# 2024-05-15T23:40:14.212 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.212 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6786e268:::benchmark_data_smithi090_166897_object2831:head# 2024-05-15T23:40:14.212 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.212 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6786e41a:::benchmark_data_smithi090_166897_object34161:head# 2024-05-15T23:40:14.212 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.212 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:678933cf:::benchmark_data_smithi090_166897_object60202:head# 2024-05-15T23:40:14.212 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.212 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:678ac0c2:::benchmark_data_smithi090_166897_object50547:head# 2024-05-15T23:40:14.212 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.213 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:678acde6:::benchmark_data_smithi090_166897_object69945:head# 2024-05-15T23:40:14.213 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.213 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:678ba2fd:::benchmark_data_smithi090_166897_object66339:head# 2024-05-15T23:40:14.213 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.213 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:678c8e37:::benchmark_data_smithi090_166897_object2448:head# 2024-05-15T23:40:14.213 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.213 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:679118dc:::benchmark_data_smithi090_166897_object45321:head# 2024-05-15T23:40:14.213 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.213 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67918a73:::benchmark_data_smithi090_166897_object45037:head# 2024-05-15T23:40:14.213 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.213 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6792be2c:::benchmark_data_smithi090_166897_object62577:head# 2024-05-15T23:40:14.213 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.213 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6792f9e4:::benchmark_data_smithi090_166897_object157:head# 2024-05-15T23:40:14.213 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.213 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:679314f7:::benchmark_data_smithi090_166897_object18733:head# 2024-05-15T23:40:14.214 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.214 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67931cbf:::benchmark_data_smithi090_166897_object27501:head# 2024-05-15T23:40:14.214 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.214 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6793ee7c:::benchmark_data_smithi090_166897_object31734:head# 2024-05-15T23:40:14.214 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.214 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6797c7d0:::benchmark_data_smithi090_166897_object6317:head# 2024-05-15T23:40:14.214 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.214 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6797fea1:::benchmark_data_smithi090_166897_object30968:head# 2024-05-15T23:40:14.214 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.214 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67982819:::benchmark_data_smithi090_166897_object37117:head# 2024-05-15T23:40:14.214 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.214 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6799833b:::benchmark_data_smithi090_166897_object30193:head# 2024-05-15T23:40:14.214 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.214 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:679ca3ae:::benchmark_data_smithi090_166897_object40695:head# 2024-05-15T23:40:14.215 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.215 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:679d0f68:::benchmark_data_smithi090_166897_object41861:head# 2024-05-15T23:40:14.215 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.215 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:679db75a:::benchmark_data_smithi090_166897_object38741:head# 2024-05-15T23:40:14.215 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.215 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:679e0e51:::benchmark_data_smithi090_166897_object35801:head# 2024-05-15T23:40:14.215 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.215 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:679e2b2f:::benchmark_data_smithi090_166897_object65683:head# 2024-05-15T23:40:14.215 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.215 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67a09885:::benchmark_data_smithi090_166897_object54502:head# 2024-05-15T23:40:14.215 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.215 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67a1c969:::benchmark_data_smithi090_166897_object52281:head# 2024-05-15T23:40:14.215 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.216 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67a1eddb:::benchmark_data_smithi090_166897_object12521:head# 2024-05-15T23:40:14.216 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.216 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67a2a315:::benchmark_data_smithi090_166897_object32130:head# 2024-05-15T23:40:14.216 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.216 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67a2d4f1:::benchmark_data_smithi090_166897_object46472:head# 2024-05-15T23:40:14.216 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.216 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67a3f342:::benchmark_data_smithi090_166897_object28746:head# 2024-05-15T23:40:14.216 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.216 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67a4d0d5:::benchmark_data_smithi090_166897_object35693:head# 2024-05-15T23:40:14.216 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.216 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67a7e943:::benchmark_data_smithi090_166897_object55487:head# 2024-05-15T23:40:14.216 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.216 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67a8d6d8:::benchmark_data_smithi090_166897_object19522:head# 2024-05-15T23:40:14.216 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.217 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67a9d0f5:::benchmark_data_smithi090_166897_object39208:head# 2024-05-15T23:40:14.217 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.217 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67acdf21:::benchmark_data_smithi090_166897_object7047:head# 2024-05-15T23:40:14.217 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.217 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67ad2111:::benchmark_data_smithi090_166897_object44536:head# 2024-05-15T23:40:14.217 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.217 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67ad80ee:::benchmark_data_smithi090_166897_object5905:head# 2024-05-15T23:40:14.217 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.217 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67adfaab:::benchmark_data_smithi090_166897_object38582:head# 2024-05-15T23:40:14.217 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.217 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67aef080:::benchmark_data_smithi090_166897_object30479:head# 2024-05-15T23:40:14.217 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.217 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67af28f2:::benchmark_data_smithi090_166897_object3947:head# 2024-05-15T23:40:14.217 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.218 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67b09586:::benchmark_data_smithi090_166897_object993:head# 2024-05-15T23:40:14.218 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.218 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67b11d54:::benchmark_data_smithi090_166897_object31610:head# 2024-05-15T23:40:14.218 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.218 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67b1d10e:::benchmark_data_smithi090_166897_object44644:head# 2024-05-15T23:40:14.218 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.218 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67b1d4e7:::benchmark_data_smithi090_166897_object7857:head# 2024-05-15T23:40:14.218 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.218 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67b23967:::benchmark_data_smithi090_166897_object13965:head# 2024-05-15T23:40:14.218 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.218 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67b276e5:::benchmark_data_smithi090_166897_object59932:head# 2024-05-15T23:40:14.218 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.218 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67b2adf4:::benchmark_data_smithi090_166897_object49085:head# 2024-05-15T23:40:14.218 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.219 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67b418d7:::benchmark_data_smithi090_166897_object10167:head# 2024-05-15T23:40:14.219 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.219 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67b4f63c:::benchmark_data_smithi090_166897_object25889:head# 2024-05-15T23:40:14.219 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.219 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67b5a0f2:::benchmark_data_smithi090_166897_object44419:head# 2024-05-15T23:40:14.219 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.219 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67b6816e:::benchmark_data_smithi090_166897_object40691:head# 2024-05-15T23:40:14.219 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.219 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67ba1a02:::benchmark_data_smithi090_166897_object17801:head# 2024-05-15T23:40:14.219 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.219 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67bace5c:::benchmark_data_smithi090_166897_object27624:head# 2024-05-15T23:40:14.219 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.219 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67bb129b:::benchmark_data_smithi090_166897_object25365:head# 2024-05-15T23:40:14.219 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.220 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67bc10f3:::benchmark_data_smithi090_166897_object70086:head# 2024-05-15T23:40:14.220 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.220 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67bc18e5:::benchmark_data_smithi090_166897_object61647:head# 2024-05-15T23:40:14.220 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.220 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67bd59c9:::benchmark_data_smithi090_166897_object11565:head# 2024-05-15T23:40:14.220 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.220 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67be4e5d:::benchmark_data_smithi090_166897_object52368:head# 2024-05-15T23:40:14.220 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.220 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67c06908:::benchmark_data_smithi090_166897_object37813:head# 2024-05-15T23:40:14.220 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.220 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67c1800a:::benchmark_data_smithi090_166897_object13923:head# 2024-05-15T23:40:14.220 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.220 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67c1942f:::benchmark_data_smithi090_166897_object44375:head# 2024-05-15T23:40:14.220 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.220 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67c226b1:::benchmark_data_smithi090_166897_object65437:head# 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67c4aa33:::benchmark_data_smithi090_166897_object14773:head# 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67c678ea:::benchmark_data_smithi090_166897_object20643:head# 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67c858da:::benchmark_data_smithi090_166897_object36543:head# 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67c8af54:::benchmark_data_smithi090_166897_object26452:head# 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67c95388:::benchmark_data_smithi090_166897_object37463:head# 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67cdcd9c:::benchmark_data_smithi090_166897_object21488:head# 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67ceadc7:::benchmark_data_smithi090_166897_object68335:head# 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67cefb3d:::benchmark_data_smithi090_166897_object11605:head# 2024-05-15T23:40:14.222 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.222 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67cf1aae:::benchmark_data_smithi090_166897_object67982:head# 2024-05-15T23:40:14.222 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.222 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67cf928c:::benchmark_data_smithi090_166897_object1104:head# 2024-05-15T23:40:14.222 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.222 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67cf95e8:::benchmark_data_smithi090_166897_object14376:head# 2024-05-15T23:40:14.222 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.222 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67d100d2:::benchmark_data_smithi090_166897_object52817:head# 2024-05-15T23:40:14.222 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.222 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67d140c1:::benchmark_data_smithi090_166897_object25357:head# 2024-05-15T23:40:14.222 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.222 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67d1665d:::benchmark_data_smithi090_166897_object20373:head# 2024-05-15T23:40:14.222 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67d1ca70:::benchmark_data_smithi090_166897_object36622:head# 2024-05-15T23:40:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67d2ccd6:::benchmark_data_smithi090_166897_object32175:head# 2024-05-15T23:40:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67d339e7:::benchmark_data_smithi090_166897_object39527:head# 2024-05-15T23:40:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67d34a93:::benchmark_data_smithi090_166897_object60925:head# 2024-05-15T23:40:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67d386a5:::benchmark_data_smithi090_166897_object15467:head# 2024-05-15T23:40:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67d49a53:::benchmark_data_smithi090_166897_object60844:head# 2024-05-15T23:40:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67d6cadd:::benchmark_data_smithi090_166897_object36900:head# 2024-05-15T23:40:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67d78696:::benchmark_data_smithi090_166897_object69968:head# 2024-05-15T23:40:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67d7fc44:::benchmark_data_smithi090_166897_object55530:head# 2024-05-15T23:40:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67d8b244:::benchmark_data_smithi090_166897_object10333:head# 2024-05-15T23:40:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67d9a9d1:::benchmark_data_smithi090_166897_object39246:head# 2024-05-15T23:40:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67d9f468:::benchmark_data_smithi090_166897_object18485:head# 2024-05-15T23:40:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67da1d79:::benchmark_data_smithi090_166897_object11105:head# 2024-05-15T23:40:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67db77ed:::benchmark_data_smithi090_166897_object6739:head# 2024-05-15T23:40:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.225 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67db989f:::benchmark_data_smithi090_166897_object7690:head# 2024-05-15T23:40:14.225 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.225 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67dcf617:::benchmark_data_smithi090_166897_object57989:head# 2024-05-15T23:40:14.225 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.225 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67de2648:::benchmark_data_smithi090_166897_object45138:head# 2024-05-15T23:40:14.225 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.225 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67debaa5:::benchmark_data_smithi090_166897_object27411:head# 2024-05-15T23:40:14.225 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.225 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67e2f161:::benchmark_data_smithi090_166897_object70524:head# 2024-05-15T23:40:14.225 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.225 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67e53e39:::benchmark_data_smithi090_166897_object46906:head# 2024-05-15T23:40:14.225 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.225 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67e5deb7:::benchmark_data_smithi090_166897_object29616:head# 2024-05-15T23:40:14.225 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.226 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67e85809:::benchmark_data_smithi090_166897_object54943:head# 2024-05-15T23:40:14.226 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.226 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67e872fd:::benchmark_data_smithi090_166897_object23234:head# 2024-05-15T23:40:14.226 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.226 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67e8ca8a:::benchmark_data_smithi090_166897_object9236:head# 2024-05-15T23:40:14.226 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.226 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67e9477a:::benchmark_data_smithi090_166897_object46610:head# 2024-05-15T23:40:14.226 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.226 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67ea04a8:::benchmark_data_smithi090_166897_object38957:head# 2024-05-15T23:40:14.226 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.226 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67ea44a4:::benchmark_data_smithi090_166897_object62984:head# 2024-05-15T23:40:14.226 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.226 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67ea6387:::benchmark_data_smithi090_166897_object61364:head# 2024-05-15T23:40:14.226 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.227 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67ea6439:::benchmark_data_smithi090_166897_object6129:head# 2024-05-15T23:40:14.227 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.227 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67eba059:::benchmark_data_smithi090_166897_object52618:head# 2024-05-15T23:40:14.227 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.227 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67ec1ef0:::benchmark_data_smithi090_166897_object62916:head# 2024-05-15T23:40:14.227 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.227 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67ed75dd:::benchmark_data_smithi090_166897_object48782:head# 2024-05-15T23:40:14.227 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.227 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67ee5922:::benchmark_data_smithi090_166897_object54428:head# 2024-05-15T23:40:14.227 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.227 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67f18eb7:::benchmark_data_smithi090_166897_object53286:head# 2024-05-15T23:40:14.227 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.227 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67f353a0:::benchmark_data_smithi090_166897_object57090:head# 2024-05-15T23:40:14.227 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67f8f0b4:::benchmark_data_smithi090_166897_object46092:head# 2024-05-15T23:40:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67f9a0de:::benchmark_data_smithi090_166897_object32094:head# 2024-05-15T23:40:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67fa5a90:::benchmark_data_smithi090_166897_object7380:head# 2024-05-15T23:40:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67fbe16f:::benchmark_data_smithi090_166897_object10299:head# 2024-05-15T23:40:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67fc55e8:::benchmark_data_smithi090_166897_object26009:head# 2024-05-15T23:40:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67fc670c:::benchmark_data_smithi090_166897_object28407:head# 2024-05-15T23:40:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67fd9821:::benchmark_data_smithi090_166897_object48595:head# 2024-05-15T23:40:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67ff30d4:::benchmark_data_smithi090_166897_object37685:head# 2024-05-15T23:40:14.229 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.229 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:67ff96d4:::benchmark_data_smithi090_166897_object52610:head# 2024-05-15T23:40:14.229 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.229 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6802732d:::benchmark_data_smithi090_166897_object53999:head# 2024-05-15T23:40:14.229 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.229 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68038ca5:::benchmark_data_smithi090_166897_object24563:head# 2024-05-15T23:40:14.229 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.229 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6804c3ed:::benchmark_data_smithi090_166897_object16434:head# 2024-05-15T23:40:14.229 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.229 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6804e2f5:::benchmark_data_smithi090_166897_object32647:head# 2024-05-15T23:40:14.229 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.229 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6804fb7a:::benchmark_data_smithi090_166897_object61012:head# 2024-05-15T23:40:14.229 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.229 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68051902:::benchmark_data_smithi090_166897_object67829:head# 2024-05-15T23:40:14.230 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.230 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68067297:::benchmark_data_smithi090_166897_object62452:head# 2024-05-15T23:40:14.230 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.230 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6806bc89:::benchmark_data_smithi090_166897_object11455:head# 2024-05-15T23:40:14.230 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.230 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68088049:::benchmark_data_smithi090_166897_object65156:head# 2024-05-15T23:40:14.230 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.230 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:680b27ff:::benchmark_data_smithi090_166897_object40731:head# 2024-05-15T23:40:14.230 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.230 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:680b4e0b:::benchmark_data_smithi090_166897_object4590:head# 2024-05-15T23:40:14.230 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.230 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:680c3067:::benchmark_data_smithi090_166897_object38639:head# 2024-05-15T23:40:14.230 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.230 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:680c3673:::benchmark_data_smithi090_166897_object11104:head# 2024-05-15T23:40:14.231 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.231 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:680c52cd:::benchmark_data_smithi090_166897_object50165:head# 2024-05-15T23:40:14.231 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.231 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:680c56a7:::benchmark_data_smithi090_166897_object50616:head# 2024-05-15T23:40:14.231 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.231 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:680c59e1:::benchmark_data_smithi090_166897_object31652:head# 2024-05-15T23:40:14.231 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.231 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:680c5c62:::benchmark_data_smithi090_166897_object41290:head# 2024-05-15T23:40:14.231 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.231 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:680cce03:::benchmark_data_smithi090_166897_object47810:head# 2024-05-15T23:40:14.231 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.231 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:680cdda3:::benchmark_data_smithi090_166897_object62415:head# 2024-05-15T23:40:14.231 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.231 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:680d94d4:::benchmark_data_smithi090_166897_object22733:head# 2024-05-15T23:40:14.231 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.232 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:680f2925:::benchmark_data_smithi090_166897_object14983:head# 2024-05-15T23:40:14.232 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.232 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:680f363c:::benchmark_data_smithi090_166897_object65917:head# 2024-05-15T23:40:14.232 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.232 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6813e6ae:::benchmark_data_smithi090_166897_object22211:head# 2024-05-15T23:40:14.232 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.232 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6813f3c4:::benchmark_data_smithi090_166897_object20758:head# 2024-05-15T23:40:14.232 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.232 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68159f9a:::benchmark_data_smithi090_166897_object5855:head# 2024-05-15T23:40:14.232 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.232 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68163fbf:::benchmark_data_smithi090_166897_object64505:head# 2024-05-15T23:40:14.232 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.232 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:681694ac:::benchmark_data_smithi090_166897_object61920:head# 2024-05-15T23:40:14.232 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.232 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:681721c5:::benchmark_data_smithi090_166897_object24018:head# 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:681dc021:::benchmark_data_smithi090_166897_object39218:head# 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68201329:::benchmark_data_smithi090_166897_object43557:head# 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68201676:::benchmark_data_smithi090_166897_object33520:head# 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68207426:::benchmark_data_smithi090_166897_object9123:head# 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:682077ff:::benchmark_data_smithi090_166897_object23656:head# 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68226bc5:::benchmark_data_smithi090_166897_object64538:head# 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6822cc5c:::benchmark_data_smithi090_166897_object60621:head# 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.233 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68242722:::benchmark_data_smithi090_166897_object66808:head# 2024-05-15T23:40:14.234 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.234 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6824a7d6:::benchmark_data_smithi090_166897_object37247:head# 2024-05-15T23:40:14.234 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.234 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6824bee4:::benchmark_data_smithi090_166897_object21462:head# 2024-05-15T23:40:14.234 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.234 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:682568aa:::benchmark_data_smithi090_166897_object14969:head# 2024-05-15T23:40:14.234 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.234 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:682798a0:::benchmark_data_smithi090_166897_object43959:head# 2024-05-15T23:40:14.234 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.234 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6827d2f0:::benchmark_data_smithi090_166897_object11996:head# 2024-05-15T23:40:14.234 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.234 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6829f170:::benchmark_data_smithi090_166897_object35368:head# 2024-05-15T23:40:14.234 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.234 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:682a3e34:::benchmark_data_smithi090_166897_object30887:head# 2024-05-15T23:40:14.235 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.235 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:682b33f1:::benchmark_data_smithi090_166897_object45595:head# 2024-05-15T23:40:14.235 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.235 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:682b9c2f:::benchmark_data_smithi090_166897_object63122:head# 2024-05-15T23:40:14.235 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.235 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:682d8800:::benchmark_data_smithi090_166897_object70332:head# 2024-05-15T23:40:14.235 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.235 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:682dce95:::benchmark_data_smithi090_166897_object12562:head# 2024-05-15T23:40:14.235 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.235 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:682de348:::benchmark_data_smithi090_166897_object3854:head# 2024-05-15T23:40:14.235 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.235 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:682e25d6:::benchmark_data_smithi090_166897_object48411:head# 2024-05-15T23:40:14.235 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.235 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:682e4523:::benchmark_data_smithi090_166897_object56774:head# 2024-05-15T23:40:14.235 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.236 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:682f8079:::benchmark_data_smithi090_166897_object53808:head# 2024-05-15T23:40:14.236 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.236 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68311fa2:::benchmark_data_smithi090_166897_object57550:head# 2024-05-15T23:40:14.236 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.236 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68315c00:::benchmark_data_smithi090_166897_object3080:head# 2024-05-15T23:40:14.236 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.236 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68319327:::benchmark_data_smithi090_166897_object55706:head# 2024-05-15T23:40:14.236 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.236 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6831e87d:::benchmark_data_smithi090_166897_object43409:head# 2024-05-15T23:40:14.236 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.236 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6832417f:::benchmark_data_smithi090_166897_object24529:head# 2024-05-15T23:40:14.236 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.236 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68325c44:::benchmark_data_smithi090_166897_object3016:head# 2024-05-15T23:40:14.236 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.236 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:683404a2:::benchmark_data_smithi090_166897_object39226:head# 2024-05-15T23:40:14.237 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.237 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6834fed8:::benchmark_data_smithi090_166897_object61682:head# 2024-05-15T23:40:14.237 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.237 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68350b70:::benchmark_data_smithi090_166897_object19978:head# 2024-05-15T23:40:14.237 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.237 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68364fb5:::benchmark_data_smithi090_166897_object67322:head# 2024-05-15T23:40:14.237 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.237 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68373664:::benchmark_data_smithi090_166897_object28888:head# 2024-05-15T23:40:14.237 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.237 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:683793ab:::benchmark_data_smithi090_166897_object58924:head# 2024-05-15T23:40:14.237 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.237 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6837d60f:::benchmark_data_smithi090_166897_object45127:head# 2024-05-15T23:40:14.237 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.237 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68383152:::benchmark_data_smithi090_166897_object4661:head# 2024-05-15T23:40:14.238 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.238 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6839c03e:::benchmark_data_smithi090_166897_object10991:head# 2024-05-15T23:40:14.238 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.238 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:683af3e8:::benchmark_data_smithi090_166897_object7235:head# 2024-05-15T23:40:14.238 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.238 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:683bad7d:::benchmark_data_smithi090_166897_object28940:head# 2024-05-15T23:40:14.238 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.238 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:683bb6de:::benchmark_data_smithi090_166897_object61849:head# 2024-05-15T23:40:14.238 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.238 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:683ce5c3:::benchmark_data_smithi090_166897_object56533:head# 2024-05-15T23:40:14.238 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.238 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:683fd482:::benchmark_data_smithi090_166897_object13999:head# 2024-05-15T23:40:14.238 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.239 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6840f820:::benchmark_data_smithi090_166897_object4793:head# 2024-05-15T23:40:14.239 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.239 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:684283ed:::benchmark_data_smithi090_166897_object38421:head# 2024-05-15T23:40:14.239 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.239 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68470af2:::benchmark_data_smithi090_166897_object16041:head# 2024-05-15T23:40:14.239 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.239 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68479e9d:::benchmark_data_smithi090_166897_object60048:head# 2024-05-15T23:40:14.239 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.239 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68481436:::benchmark_data_smithi090_166897_object1446:head# 2024-05-15T23:40:14.239 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.239 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6849d50a:::benchmark_data_smithi090_166897_object36055:head# 2024-05-15T23:40:14.239 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.239 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:684a7585:::benchmark_data_smithi090_166897_object56326:head# 2024-05-15T23:40:14.239 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.239 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:684b77fd:::benchmark_data_smithi090_166897_object2811:head# 2024-05-15T23:40:14.240 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.240 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:684b9b40:::benchmark_data_smithi090_166897_object54157:head# 2024-05-15T23:40:14.240 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.240 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:684bb348:::benchmark_data_smithi090_166897_object65229:head# 2024-05-15T23:40:14.240 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.240 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:684c6a5f:::benchmark_data_smithi090_166897_object48032:head# 2024-05-15T23:40:14.240 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.240 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:684d1ffe:::benchmark_data_smithi090_166897_object57327:head# 2024-05-15T23:40:14.240 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.240 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:684d60bf:::benchmark_data_smithi090_166897_object44985:head# 2024-05-15T23:40:14.240 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.240 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:684dce68:::benchmark_data_smithi090_166897_object11479:head# 2024-05-15T23:40:14.240 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.240 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:685050cd:::benchmark_data_smithi090_166897_object46089:head# 2024-05-15T23:40:14.240 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.241 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68505fb0:::benchmark_data_smithi090_166897_object8388:head# 2024-05-15T23:40:14.241 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.241 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:685089ad:::benchmark_data_smithi090_166897_object27798:head# 2024-05-15T23:40:14.241 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.241 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68509ea1:::benchmark_data_smithi090_166897_object12300:head# 2024-05-15T23:40:14.241 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.241 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6850c1bb:::benchmark_data_smithi090_166897_object1622:head# 2024-05-15T23:40:14.241 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.241 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68531ef6:::benchmark_data_smithi090_166897_object12724:head# 2024-05-15T23:40:14.241 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.241 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6857cb49:::benchmark_data_smithi090_166897_object67946:head# 2024-05-15T23:40:14.241 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.241 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6857dfed:::benchmark_data_smithi090_166897_object10970:head# 2024-05-15T23:40:14.241 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.242 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6858954d:::benchmark_data_smithi090_166897_object27865:head# 2024-05-15T23:40:14.242 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.242 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68592e6c:::benchmark_data_smithi090_166897_object52274:head# 2024-05-15T23:40:14.242 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.242 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:685d2d28:::benchmark_data_smithi090_166897_object19807:head# 2024-05-15T23:40:14.242 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.242 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:685d8a8c:::benchmark_data_smithi090_166897_object71019:head# 2024-05-15T23:40:14.242 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.242 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:685e704f:::benchmark_data_smithi090_166897_object48014:head# 2024-05-15T23:40:14.242 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.242 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:685f8978:::benchmark_data_smithi090_166897_object35788:head# 2024-05-15T23:40:14.242 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.242 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:685fe610:::benchmark_data_smithi090_166897_object37973:head# 2024-05-15T23:40:14.243 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.243 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68627a7c:::benchmark_data_smithi090_166897_object63904:head# 2024-05-15T23:40:14.243 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.243 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6863c694:::benchmark_data_smithi090_166897_object43034:head# 2024-05-15T23:40:14.243 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.243 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68662fae:::benchmark_data_smithi090_166897_object41794:head# 2024-05-15T23:40:14.243 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.243 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68665bbb:::benchmark_data_smithi090_166897_object48973:head# 2024-05-15T23:40:14.243 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.243 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6866a872:::benchmark_data_smithi090_166897_object1955:head# 2024-05-15T23:40:14.243 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.243 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6866d88e:::benchmark_data_smithi090_166897_object13413:head# 2024-05-15T23:40:14.243 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.243 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6867c99b:::benchmark_data_smithi090_166897_object5822:head# 2024-05-15T23:40:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68686551:::benchmark_data_smithi090_166897_object61371:head# 2024-05-15T23:40:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6868b954:::benchmark_data_smithi090_166897_object50640:head# 2024-05-15T23:40:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68695a78:::benchmark_data_smithi090_166897_object64601:head# 2024-05-15T23:40:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6869bff7:::benchmark_data_smithi090_166897_object54060:head# 2024-05-15T23:40:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:686b4cf1:::benchmark_data_smithi090_166897_object19887:head# 2024-05-15T23:40:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:686b8892:::benchmark_data_smithi090_166897_object26953:head# 2024-05-15T23:40:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:686eb02a:::benchmark_data_smithi090_166897_object32726:head# 2024-05-15T23:40:14.244 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:686f498b:::benchmark_data_smithi090_166897_object10482:head# 2024-05-15T23:40:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68705b81:::benchmark_data_smithi090_166897_object10999:head# 2024-05-15T23:40:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:687130c8:::benchmark_data_smithi090_166897_object58782:head# 2024-05-15T23:40:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68713e97:::benchmark_data_smithi090_166897_object982:head# 2024-05-15T23:40:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68719665:::benchmark_data_smithi090_166897_object23000:head# 2024-05-15T23:40:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68720322:::benchmark_data_smithi090_166897_object55942:head# 2024-05-15T23:40:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6872a592:::benchmark_data_smithi090_166897_object34237:head# 2024-05-15T23:40:14.245 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6872fbc6:::benchmark_data_smithi090_166897_object13555:head# 2024-05-15T23:40:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68746145:::benchmark_data_smithi090_166897_object59034:head# 2024-05-15T23:40:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68758edb:::benchmark_data_smithi090_166897_object68572:head# 2024-05-15T23:40:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:687621a9:::benchmark_data_smithi090_166897_object51516:head# 2024-05-15T23:40:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6876b7f8:::benchmark_data_smithi090_166897_object9114:head# 2024-05-15T23:40:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68772327:::benchmark_data_smithi090_166897_object55631:head# 2024-05-15T23:40:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:687736fc:::benchmark_data_smithi090_166897_object64111:head# 2024-05-15T23:40:14.246 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6878b2e0:::benchmark_data_smithi090_166897_object43841:head# 2024-05-15T23:40:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6878dbd8:::benchmark_data_smithi090_166897_object26353:head# 2024-05-15T23:40:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:687941b9:::benchmark_data_smithi090_166897_object3238:head# 2024-05-15T23:40:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:687b1443:::benchmark_data_smithi090_166897_object63511:head# 2024-05-15T23:40:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:687d1830:::benchmark_data_smithi090_166897_object63561:head# 2024-05-15T23:40:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68807cc2:::benchmark_data_smithi090_166897_object8377:head# 2024-05-15T23:40:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68817ea3:::benchmark_data_smithi090_166897_object57080:head# 2024-05-15T23:40:14.247 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.248 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6881aaf8:::benchmark_data_smithi090_166897_object49254:head# 2024-05-15T23:40:14.248 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.248 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6883933c:::benchmark_data_smithi090_166897_object39600:head# 2024-05-15T23:40:14.248 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.248 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6883a52c:::benchmark_data_smithi090_166897_object27356:head# 2024-05-15T23:40:14.248 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.248 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6883e039:::benchmark_data_smithi090_166897_object51654:head# 2024-05-15T23:40:14.248 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.248 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68851a28:::benchmark_data_smithi090_166897_object1988:head# 2024-05-15T23:40:14.248 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.248 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:688569a1:::benchmark_data_smithi090_166897_object18465:head# 2024-05-15T23:40:14.248 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.248 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6886ab68:::benchmark_data_smithi090_166897_object55950:head# 2024-05-15T23:40:14.248 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.248 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68880f52:::benchmark_data_smithi090_166897_object5103:head# 2024-05-15T23:40:14.249 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.249 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68882d0f:::benchmark_data_smithi090_166897_object13976:head# 2024-05-15T23:40:14.249 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.249 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:688861fd:::benchmark_data_smithi090_166897_object3850:head# 2024-05-15T23:40:14.249 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.249 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68888688:::benchmark_data_smithi090_166897_object70264:head# 2024-05-15T23:40:14.249 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.249 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6888ade1:::benchmark_data_smithi090_166897_object18418:head# 2024-05-15T23:40:14.249 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.249 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6889e832:::benchmark_data_smithi090_166897_object47722:head# 2024-05-15T23:40:14.249 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.249 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:688ab657:::benchmark_data_smithi090_166897_object23649:head# 2024-05-15T23:40:14.249 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.249 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:688b3341:::benchmark_data_smithi090_166897_object2833:head# 2024-05-15T23:40:14.250 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.250 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:688b7159:::benchmark_data_smithi090_166897_object10859:head# 2024-05-15T23:40:14.250 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.250 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:688c85df:::benchmark_data_smithi090_166897_object68303:head# 2024-05-15T23:40:14.250 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.250 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:688cb299:::benchmark_data_smithi090_166897_object52690:head# 2024-05-15T23:40:14.250 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.250 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:688df29c:::benchmark_data_smithi090_166897_object30775:head# 2024-05-15T23:40:14.250 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.250 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:688e6be0:::benchmark_data_smithi090_166897_object44925:head# 2024-05-15T23:40:14.250 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.250 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:688f7bdc:::benchmark_data_smithi090_166897_object22079:head# 2024-05-15T23:40:14.250 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.251 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68902311:::benchmark_data_smithi090_166897_object43830:head# 2024-05-15T23:40:14.251 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.251 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6890cce6:::benchmark_data_smithi090_166897_object26265:head# 2024-05-15T23:40:14.251 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.251 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68910bee:::benchmark_data_smithi090_166897_object38332:head# 2024-05-15T23:40:14.251 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.251 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68911630:::benchmark_data_smithi090_166897_object30160:head# 2024-05-15T23:40:14.251 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.251 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6891fd56:::benchmark_data_smithi090_166897_object49214:head# 2024-05-15T23:40:14.252 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.252 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6892900c:::benchmark_data_smithi090_166897_object34174:head# 2024-05-15T23:40:14.252 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.252 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:689463d4:::benchmark_data_smithi090_166897_object43105:head# 2024-05-15T23:40:14.252 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.252 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6894afea:::benchmark_data_smithi090_166897_object22854:head# 2024-05-15T23:40:14.252 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.252 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68962a05:::benchmark_data_smithi090_166897_object62279:head# 2024-05-15T23:40:14.252 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.252 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6898cec7:::benchmark_data_smithi090_166897_object21113:head# 2024-05-15T23:40:14.252 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.252 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6899174f:::benchmark_data_smithi090_166897_object42250:head# 2024-05-15T23:40:14.252 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.252 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:689924a2:::benchmark_data_smithi090_166897_object34938:head# 2024-05-15T23:40:14.253 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.253 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6899ae3b:::benchmark_data_smithi090_166897_object26715:head# 2024-05-15T23:40:14.253 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.253 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6899b67f:::benchmark_data_smithi090_166897_object18896:head# 2024-05-15T23:40:14.253 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.253 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6899eb88:::benchmark_data_smithi090_166897_object12073:head# 2024-05-15T23:40:14.253 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.253 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:689a10c3:::benchmark_data_smithi090_166897_object28938:head# 2024-05-15T23:40:14.253 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.253 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:689a44a0:::benchmark_data_smithi090_166897_object23700:head# 2024-05-15T23:40:14.253 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.253 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:689acb09:::benchmark_data_smithi090_166897_object67408:head# 2024-05-15T23:40:14.253 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.253 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:689b30bc:::benchmark_data_smithi090_166897_object45357:head# 2024-05-15T23:40:14.253 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.254 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:689bf64a:::benchmark_data_smithi090_166897_object33473:head# 2024-05-15T23:40:14.254 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.254 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:689d6472:::benchmark_data_smithi090_166897_object55062:head# 2024-05-15T23:40:14.254 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.254 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:689da9cc:::benchmark_data_smithi090_166897_object28081:head# 2024-05-15T23:40:14.254 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.254 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:689de3d0:::benchmark_data_smithi090_166897_object27379:head# 2024-05-15T23:40:14.254 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.254 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:689fadbf:::benchmark_data_smithi090_166897_object5610:head# 2024-05-15T23:40:14.254 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.254 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68a1f1d8:::benchmark_data_smithi090_166897_object58132:head# 2024-05-15T23:40:14.254 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.254 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68a3d22e:::benchmark_data_smithi090_166897_object50693:head# 2024-05-15T23:40:14.254 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.255 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68a50338:::benchmark_data_smithi090_166897_object743:head# 2024-05-15T23:40:14.255 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.255 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68a5eb8c:::benchmark_data_smithi090_166897_object36452:head# 2024-05-15T23:40:14.255 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.255 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68a701a9:::benchmark_data_smithi090_166897_object54002:head# 2024-05-15T23:40:14.255 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.255 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68a9e2a2:::benchmark_data_smithi090_166897_object7049:head# 2024-05-15T23:40:14.255 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.255 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68abc8d7:::benchmark_data_smithi090_166897_object48756:head# 2024-05-15T23:40:14.255 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.255 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68abf870:::benchmark_data_smithi090_166897_object46463:head# 2024-05-15T23:40:14.255 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.255 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68ac27cd:::benchmark_data_smithi090_166897_object66187:head# 2024-05-15T23:40:14.255 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.256 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68ac6389:::benchmark_data_smithi090_166897_object38120:head# 2024-05-15T23:40:14.256 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.256 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68ace43b:::benchmark_data_smithi090_166897_object63253:head# 2024-05-15T23:40:14.256 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.256 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68b051d8:::benchmark_data_smithi090_166897_object12566:head# 2024-05-15T23:40:14.256 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.256 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68b0aadc:::benchmark_data_smithi090_166897_object12898:head# 2024-05-15T23:40:14.256 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.257 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68b20de2:::benchmark_data_smithi090_166897_object59350:head# 2024-05-15T23:40:14.257 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.257 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68b22cc6:::benchmark_data_smithi090_166897_object66341:head# 2024-05-15T23:40:14.257 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.257 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68b2840e:::benchmark_data_smithi090_166897_object18069:head# 2024-05-15T23:40:14.257 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.257 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68b2cba2:::benchmark_data_smithi090_166897_object40127:head# 2024-05-15T23:40:14.257 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.257 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68b5b7a4:::benchmark_data_smithi090_166897_object7922:head# 2024-05-15T23:40:14.257 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.257 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68b608db:::benchmark_data_smithi090_166897_object69686:head# 2024-05-15T23:40:14.258 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.258 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68b63f1d:::benchmark_data_smithi090_166897_object48647:head# 2024-05-15T23:40:14.258 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.258 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68b8930d:::benchmark_data_smithi090_166897_object29931:head# 2024-05-15T23:40:14.258 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.258 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68b8ca1b:::benchmark_data_smithi090_166897_object71036:head# 2024-05-15T23:40:14.258 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.258 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68b9723e:::benchmark_data_smithi090_166897_object54595:head# 2024-05-15T23:40:14.258 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.258 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68bad3f8:::benchmark_data_smithi090_166897_object18222:head# 2024-05-15T23:40:14.258 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.258 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68baff1a:::benchmark_data_smithi090_166897_object41548:head# 2024-05-15T23:40:14.258 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.258 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68bb11dd:::benchmark_data_smithi090_166897_object44756:head# 2024-05-15T23:40:14.259 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.259 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68bb8f10:::benchmark_data_smithi090_166897_object15505:head# 2024-05-15T23:40:14.259 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.259 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68bc447f:::benchmark_data_smithi090_166897_object20821:head# 2024-05-15T23:40:14.259 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.259 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68bd8095:::benchmark_data_smithi090_166897_object29180:head# 2024-05-15T23:40:14.259 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.259 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68bedd14:::benchmark_data_smithi090_166897_object6755:head# 2024-05-15T23:40:14.259 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.259 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68bef27a:::benchmark_data_smithi090_166897_object29623:head# 2024-05-15T23:40:14.259 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.260 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68c00861:::benchmark_data_smithi090_166897_object10568:head# 2024-05-15T23:40:14.260 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.260 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68c0da69:::benchmark_data_smithi090_166897_object10905:head# 2024-05-15T23:40:14.260 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.260 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68c27514:::benchmark_data_smithi090_166897_object50993:head# 2024-05-15T23:40:14.260 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.260 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68c371af:::benchmark_data_smithi090_166897_object70359:head# 2024-05-15T23:40:14.260 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.260 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68c4a5ef:::benchmark_data_smithi090_166897_object8311:head# 2024-05-15T23:40:14.260 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.260 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68c4f26f:::benchmark_data_smithi090_166897_object54670:head# 2024-05-15T23:40:14.260 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.260 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68c54edd:::benchmark_data_smithi090_166897_object1317:head# 2024-05-15T23:40:14.260 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.261 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68c6a752:::benchmark_data_smithi090_166897_object22943:head# 2024-05-15T23:40:14.261 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.261 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68c74895:::benchmark_data_smithi090_166897_object65281:head# 2024-05-15T23:40:14.261 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.261 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68c775e3:::benchmark_data_smithi090_166897_object3093:head# 2024-05-15T23:40:14.261 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.261 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68c82170:::benchmark_data_smithi090_166897_object64326:head# 2024-05-15T23:40:14.261 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.261 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68c901a5:::benchmark_data_smithi090_166897_object34784:head# 2024-05-15T23:40:14.261 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.262 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68c94848:::benchmark_data_smithi090_166897_object2869:head# 2024-05-15T23:40:14.262 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.262 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68c9a752:::benchmark_data_smithi090_166897_object63009:head# 2024-05-15T23:40:14.262 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.262 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68ca6305:::benchmark_data_smithi090_166897_object18424:head# 2024-05-15T23:40:14.262 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.262 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68ca8f1d:::benchmark_data_smithi090_166897_object67736:head# 2024-05-15T23:40:14.262 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.262 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68cb08a0:::benchmark_data_smithi090_166897_object20876:head# 2024-05-15T23:40:14.262 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.262 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68cb16d0:::benchmark_data_smithi090_166897_object61762:head# 2024-05-15T23:40:14.262 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.262 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68cb2b16:::benchmark_data_smithi090_166897_object2825:head# 2024-05-15T23:40:14.262 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.263 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68cb3f51:::benchmark_data_smithi090_166897_object17616:head# 2024-05-15T23:40:14.263 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.263 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68cb506a:::benchmark_data_smithi090_166897_object31227:head# 2024-05-15T23:40:14.263 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.263 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68cb569f:::benchmark_data_smithi090_166897_object58953:head# 2024-05-15T23:40:14.263 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.263 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68cba4bf:::benchmark_data_smithi090_166897_object39631:head# 2024-05-15T23:40:14.263 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.263 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68cc2cac:::benchmark_data_smithi090_166897_object8010:head# 2024-05-15T23:40:14.263 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.263 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68cd1134:::benchmark_data_smithi090_166897_object6627:head# 2024-05-15T23:40:14.263 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.263 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68cda7d8:::benchmark_data_smithi090_166897_object29766:head# 2024-05-15T23:40:14.265 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.265 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68cde388:::benchmark_data_smithi090_166897_object60261:head# 2024-05-15T23:40:14.265 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.265 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68cfa734:::benchmark_data_smithi090_166897_object9681:head# 2024-05-15T23:40:14.265 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.265 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68d10211:::benchmark_data_smithi090_166897_object68261:head# 2024-05-15T23:40:14.265 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.265 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68d3a072:::benchmark_data_smithi090_166897_object3958:head# 2024-05-15T23:40:14.266 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.266 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68d699e1:::benchmark_data_smithi090_166897_object36178:head# 2024-05-15T23:40:14.266 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.266 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68d6f9a9:::benchmark_data_smithi090_166897_object12998:head# 2024-05-15T23:40:14.266 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.266 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68d72855:::benchmark_data_smithi090_166897_object15869:head# 2024-05-15T23:40:14.266 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.266 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68d7c878:::benchmark_data_smithi090_166897_object20422:head# 2024-05-15T23:40:14.266 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.266 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68d863dd:::benchmark_data_smithi090_166897_object18027:head# 2024-05-15T23:40:14.266 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.266 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68d91573:::benchmark_data_smithi090_166897_object34576:head# 2024-05-15T23:40:14.266 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.267 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68d9a3c0:::benchmark_data_smithi090_166897_object17055:head# 2024-05-15T23:40:14.267 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.267 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68d9f56d:::benchmark_data_smithi090_166897_object59811:head# 2024-05-15T23:40:14.267 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.267 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68da0ef4:::benchmark_data_smithi090_166897_object63925:head# 2024-05-15T23:40:14.267 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.267 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68dbb1c5:::benchmark_data_smithi090_166897_object9979:head# 2024-05-15T23:40:14.267 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.267 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68dd98a9:::benchmark_data_smithi090_166897_object36658:head# 2024-05-15T23:40:14.267 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.267 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68ddd5a1:::benchmark_data_smithi090_166897_object64275:head# 2024-05-15T23:40:14.267 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.267 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68e04fb3:::benchmark_data_smithi090_166897_object11908:head# 2024-05-15T23:40:14.267 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.267 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68e0c151:::benchmark_data_smithi090_166897_object31656:head# 2024-05-15T23:40:14.268 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.268 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68e1547b:::benchmark_data_smithi090_166897_object65459:head# 2024-05-15T23:40:14.268 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.268 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68e26ff3:::benchmark_data_smithi090_166897_object48415:head# 2024-05-15T23:40:14.268 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.268 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68e2a49d:::benchmark_data_smithi090_166897_object51732:head# 2024-05-15T23:40:14.268 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.268 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68e3c5a4:::benchmark_data_smithi090_166897_object35756:head# 2024-05-15T23:40:14.268 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.268 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68e534ce:::benchmark_data_smithi090_166897_object48450:head# 2024-05-15T23:40:14.268 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.268 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68e55f5f:::benchmark_data_smithi090_166897_object61716:head# 2024-05-15T23:40:14.268 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.268 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68e58728:::benchmark_data_smithi090_166897_object25290:head# 2024-05-15T23:40:14.269 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.269 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68e7b86d:::benchmark_data_smithi090_166897_object17051:head# 2024-05-15T23:40:14.269 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.269 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68e83c64:::benchmark_data_smithi090_166897_object58142:head# 2024-05-15T23:40:14.269 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.269 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68e88296:::benchmark_data_smithi090_166897_object4280:head# 2024-05-15T23:40:14.269 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.269 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68e8e6bb:::benchmark_data_smithi090_166897_object3292:head# 2024-05-15T23:40:14.269 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.269 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68eb089a:::benchmark_data_smithi090_166897_object50531:head# 2024-05-15T23:40:14.269 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.269 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68ebc35b:::benchmark_data_smithi090_166897_object17890:head# 2024-05-15T23:40:14.269 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.269 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68ebf869:::benchmark_data_smithi090_166897_object37905:head# 2024-05-15T23:40:14.270 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.270 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68eca068:::benchmark_data_smithi090_166897_object66564:head# 2024-05-15T23:40:14.270 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.270 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68ef6614:::benchmark_data_smithi090_166897_object30516:head# 2024-05-15T23:40:14.270 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.270 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68f0c91e:::benchmark_data_smithi090_166897_object57934:head# 2024-05-15T23:40:14.270 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.270 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68f1e881:::benchmark_data_smithi090_166897_object65034:head# 2024-05-15T23:40:14.270 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.270 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68f1fdbe:::benchmark_data_smithi090_166897_object33585:head# 2024-05-15T23:40:14.270 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.270 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68f22f50:::benchmark_data_smithi090_166897_object67895:head# 2024-05-15T23:40:14.270 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.270 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68f33fe9:::benchmark_data_smithi090_166897_object64579:head# 2024-05-15T23:40:14.271 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.271 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68f34391:::benchmark_data_smithi090_166897_object43700:head# 2024-05-15T23:40:14.271 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.271 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68f443a9:::benchmark_data_smithi090_166897_object21294:head# 2024-05-15T23:40:14.271 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.271 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68f57424:::benchmark_data_smithi090_166897_object37865:head# 2024-05-15T23:40:14.271 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.271 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68f589ec:::benchmark_data_smithi090_166897_object39716:head# 2024-05-15T23:40:14.271 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.271 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68f7824c:::benchmark_data_smithi090_166897_object54842:head# 2024-05-15T23:40:14.271 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.271 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68f83df3:::benchmark_data_smithi090_166897_object61149:head# 2024-05-15T23:40:14.271 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.271 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68f8c822:::benchmark_data_smithi090_166897_object14156:head# 2024-05-15T23:40:14.272 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.272 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68f9b85f:::benchmark_data_smithi090_166897_object35650:head# 2024-05-15T23:40:14.272 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.272 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68fc2a2f:::benchmark_data_smithi090_166897_object1146:head# 2024-05-15T23:40:14.272 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.272 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68fd4077:::benchmark_data_smithi090_166897_object17952:head# 2024-05-15T23:40:14.272 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.272 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68fdb50c:::benchmark_data_smithi090_166897_object296:head# 2024-05-15T23:40:14.272 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.272 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68fe2bea:::benchmark_data_smithi090_166897_object22353:head# 2024-05-15T23:40:14.272 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.272 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68fe5f85:::benchmark_data_smithi090_166897_object61152:head# 2024-05-15T23:40:14.272 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.272 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68fef56a:::benchmark_data_smithi090_166897_object32480:head# 2024-05-15T23:40:14.273 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.273 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:68ff9db1:::benchmark_data_smithi090_166897_object3170:head# 2024-05-15T23:40:14.273 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.273 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69006905:::benchmark_data_smithi090_166897_object31261:head# 2024-05-15T23:40:14.273 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.273 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:690089da:::benchmark_data_smithi090_166897_object48300:head# 2024-05-15T23:40:14.273 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.273 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6903172b:::benchmark_data_smithi090_166897_object63918:head# 2024-05-15T23:40:14.273 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.273 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6903193d:::benchmark_data_smithi090_166897_object9075:head# 2024-05-15T23:40:14.273 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.273 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6903659c:::benchmark_data_smithi090_166897_object7340:head# 2024-05-15T23:40:14.273 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.273 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6903fddf:::benchmark_data_smithi090_166897_object2341:head# 2024-05-15T23:40:14.273 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.274 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69041481:::benchmark_data_smithi090_166897_object41582:head# 2024-05-15T23:40:14.274 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.274 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6904c877:::benchmark_data_smithi090_166897_object30965:head# 2024-05-15T23:40:14.274 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.274 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6905897b:::benchmark_data_smithi090_166897_object9364:head# 2024-05-15T23:40:14.274 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.274 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6905cff3:::benchmark_data_smithi090_166897_object30512:head# 2024-05-15T23:40:14.274 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.274 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6905e151:::benchmark_data_smithi090_166897_object22842:head# 2024-05-15T23:40:14.274 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.274 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:690681a1:::benchmark_data_smithi090_166897_object52157:head# 2024-05-15T23:40:14.274 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.274 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:690733d3:::benchmark_data_smithi090_166897_object68539:head# 2024-05-15T23:40:14.274 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.275 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69091091:::benchmark_data_smithi090_166897_object63147:head# 2024-05-15T23:40:14.275 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.275 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6909d99f:::benchmark_data_smithi090_166897_object67449:head# 2024-05-15T23:40:14.275 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.275 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:690a55b2:::benchmark_data_smithi090_166897_object29643:head# 2024-05-15T23:40:14.275 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.275 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:690a9c4c:::benchmark_data_smithi090_166897_object37392:head# 2024-05-15T23:40:14.275 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.275 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69131d8c:::benchmark_data_smithi090_166897_object13850:head# 2024-05-15T23:40:14.275 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.275 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69136da6:::benchmark_data_smithi090_166897_object813:head# 2024-05-15T23:40:14.275 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.275 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:691438e6:::benchmark_data_smithi090_166897_object14996:head# 2024-05-15T23:40:14.275 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6914459a:::benchmark_data_smithi090_166897_object50958:head# 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6914c4bd:::benchmark_data_smithi090_166897_object70667:head# 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69154cdc:::benchmark_data_smithi090_166897_object46048:head# 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:691579db:::benchmark_data_smithi090_166897_object44328:head# 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6917b9ac:::benchmark_data_smithi090_166897_object45778:head# 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:691821ad:::benchmark_data_smithi090_166897_object59864:head# 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6918d350:::benchmark_data_smithi090_166897_object69088:head# 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69193d15:::benchmark_data_smithi090_166897_object31387:head# 2024-05-15T23:40:14.276 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.277 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6919897b:::benchmark_data_smithi090_166897_object45830:head# 2024-05-15T23:40:14.277 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.277 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6919fadf:::benchmark_data_smithi090_166897_object12239:head# 2024-05-15T23:40:14.277 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.277 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:691b3a78:::benchmark_data_smithi090_166897_object24861:head# 2024-05-15T23:40:14.277 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.277 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:691cb19a:::benchmark_data_smithi090_166897_object56965:head# 2024-05-15T23:40:14.277 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.277 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:691e6728:::benchmark_data_smithi090_166897_object34603:head# 2024-05-15T23:40:14.277 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.277 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:691f7af5:::benchmark_data_smithi090_166897_object34948:head# 2024-05-15T23:40:14.277 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.277 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6921378b:::benchmark_data_smithi090_166897_object37748:head# 2024-05-15T23:40:14.277 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.278 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:692155ed:::benchmark_data_smithi090_166897_object47004:head# 2024-05-15T23:40:14.278 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.278 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69218ab2:::benchmark_data_smithi090_166897_object16854:head# 2024-05-15T23:40:14.278 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.278 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6922387d:::benchmark_data_smithi090_166897_object70002:head# 2024-05-15T23:40:14.278 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.278 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:692327ae:::benchmark_data_smithi090_166897_object44530:head# 2024-05-15T23:40:14.278 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.278 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69232c28:::benchmark_data_smithi090_166897_object56931:head# 2024-05-15T23:40:14.278 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.278 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:692365ed:::benchmark_data_smithi090_166897_object33885:head# 2024-05-15T23:40:14.278 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.278 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:692431d4:::benchmark_data_smithi090_166897_object9016:head# 2024-05-15T23:40:14.278 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.279 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6924dfcb:::benchmark_data_smithi090_166897_object28862:head# 2024-05-15T23:40:14.279 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.279 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6925ee67:::benchmark_data_smithi090_166897_object377:head# 2024-05-15T23:40:14.279 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.279 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:692bd369:::benchmark_data_smithi090_166897_object19086:head# 2024-05-15T23:40:14.279 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.279 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:692c1345:::benchmark_data_smithi090_166897_object35326:head# 2024-05-15T23:40:14.279 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.279 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:692cd662:::benchmark_data_smithi090_166897_object16764:head# 2024-05-15T23:40:14.279 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.279 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:692f2dbf:::benchmark_data_smithi090_166897_object70459:head# 2024-05-15T23:40:14.279 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.279 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:692f7806:::benchmark_data_smithi090_166897_object56863:head# 2024-05-15T23:40:14.279 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.280 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:693353a8:::benchmark_data_smithi090_166897_object42695:head# 2024-05-15T23:40:14.280 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.280 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:693368be:::benchmark_data_smithi090_166897_object1314:head# 2024-05-15T23:40:14.280 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.280 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6935759f:::benchmark_data_smithi090_166897_object70774:head# 2024-05-15T23:40:14.280 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.280 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6936ab06:::benchmark_data_smithi090_166897_object37298:head# 2024-05-15T23:40:14.280 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.280 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6936d260:::benchmark_data_smithi090_166897_object65264:head# 2024-05-15T23:40:14.280 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.280 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:693aa704:::benchmark_data_smithi090_166897_object33935:head# 2024-05-15T23:40:14.280 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.280 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:693b0269:::benchmark_data_smithi090_166897_object48950:head# 2024-05-15T23:40:14.280 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.281 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:693ba396:::benchmark_data_smithi090_166897_object53622:head# 2024-05-15T23:40:14.281 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.281 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:693e3d57:::benchmark_data_smithi090_166897_object14786:head# 2024-05-15T23:40:14.281 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.281 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:693f4536:::benchmark_data_smithi090_166897_object20186:head# 2024-05-15T23:40:14.281 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.281 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6942d647:::benchmark_data_smithi090_166897_object9282:head# 2024-05-15T23:40:14.281 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.281 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69430b2f:::benchmark_data_smithi090_166897_object49543:head# 2024-05-15T23:40:14.281 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.281 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:694386ba:::benchmark_data_smithi090_166897_object3051:head# 2024-05-15T23:40:14.281 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.281 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6943a353:::benchmark_data_smithi090_166897_object8887:head# 2024-05-15T23:40:14.281 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.282 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6943d770:::benchmark_data_smithi090_166897_object61555:head# 2024-05-15T23:40:14.282 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.282 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:694431a3:::benchmark_data_smithi090_166897_object43608:head# 2024-05-15T23:40:14.282 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.282 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69485dec:::benchmark_data_smithi090_166897_object59022:head# 2024-05-15T23:40:14.282 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.282 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:694936e9:::benchmark_data_smithi090_166897_object19447:head# 2024-05-15T23:40:14.282 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.282 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69493bcb:::benchmark_data_smithi090_166897_object57318:head# 2024-05-15T23:40:14.282 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.282 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69495619:::benchmark_data_smithi090_166897_object43844:head# 2024-05-15T23:40:14.282 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.282 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6949b11e:::benchmark_data_smithi090_166897_object65727:head# 2024-05-15T23:40:14.282 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:694abdb3:::benchmark_data_smithi090_166897_object43741:head# 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:694b69a3:::benchmark_data_smithi090_166897_object48436:head# 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:694ba53f:::benchmark_data_smithi090_166897_object46503:head# 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:694ead04:::benchmark_data_smithi090_166897_object48596:head# 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6954597d:::benchmark_data_smithi090_166897_object14242:head# 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6954e43c:::benchmark_data_smithi090_166897_object68004:head# 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:695559fb:::benchmark_data_smithi090_166897_object24777:head# 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:695625d3:::benchmark_data_smithi090_166897_object2478:head# 2024-05-15T23:40:14.283 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.284 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69580ec2:::benchmark_data_smithi090_166897_object56601:head# 2024-05-15T23:40:14.284 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.284 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69582245:::benchmark_data_smithi090_166897_object24728:head# 2024-05-15T23:40:14.284 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.284 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69589950:::benchmark_data_smithi090_166897_object22331:head# 2024-05-15T23:40:14.284 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.284 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:695a94a8:::benchmark_data_smithi090_166897_object48565:head# 2024-05-15T23:40:14.284 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.284 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:695abfb3:::benchmark_data_smithi090_166897_object52977:head# 2024-05-15T23:40:14.284 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.284 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:695b6754:::benchmark_data_smithi090_166897_object31935:head# 2024-05-15T23:40:14.284 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.284 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:695be8a6:::benchmark_data_smithi090_166897_object12422:head# 2024-05-15T23:40:14.284 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.285 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:695cd563:::benchmark_data_smithi090_166897_object2496:head# 2024-05-15T23:40:14.285 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.285 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69617df1:::benchmark_data_smithi090_166897_object38972:head# 2024-05-15T23:40:14.285 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.285 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:696184c0:::benchmark_data_smithi090_166897_object27385:head# 2024-05-15T23:40:14.285 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.285 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6961f791:::benchmark_data_smithi090_166897_object60043:head# 2024-05-15T23:40:14.285 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.285 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:696353f8:::benchmark_data_smithi090_166897_object44413:head# 2024-05-15T23:40:14.285 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.285 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6963551d:::benchmark_data_smithi090_166897_object11030:head# 2024-05-15T23:40:14.285 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.285 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6964570b:::benchmark_data_smithi090_166897_object63360:head# 2024-05-15T23:40:14.285 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.286 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:696592e6:::benchmark_data_smithi090_166897_object22281:head# 2024-05-15T23:40:14.286 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.286 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69664953:::benchmark_data_smithi090_166897_object55343:head# 2024-05-15T23:40:14.286 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.286 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6966bd6b:::benchmark_data_smithi090_166897_object40151:head# 2024-05-15T23:40:14.286 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.286 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:696707e2:::benchmark_data_smithi090_166897_object59586:head# 2024-05-15T23:40:14.286 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.286 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69679216:::benchmark_data_smithi090_166897_object36131:head# 2024-05-15T23:40:14.286 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.286 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69689225:::benchmark_data_smithi090_166897_object29962:head# 2024-05-15T23:40:14.286 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.286 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6969e791:::benchmark_data_smithi090_166897_object16526:head# 2024-05-15T23:40:14.286 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.287 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:696c651d:::benchmark_data_smithi090_166897_object51022:head# 2024-05-15T23:40:14.287 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.287 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:696ce774:::benchmark_data_smithi090_166897_object66110:head# 2024-05-15T23:40:14.287 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:40:14.355 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:14.385 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:696ce7b8:::benchmark_data_smithi090_166897_object18511:head# 2024-05-15T23:40:14.385 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.385 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:696f04d3:::benchmark_data_smithi090_166897_object43391:head# 2024-05-15T23:40:14.385 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.386 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:696fd9e3:::benchmark_data_smithi090_166897_object66254:head# 2024-05-15T23:40:14.386 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.386 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:696ff249:::benchmark_data_smithi090_166897_object32081:head# 2024-05-15T23:40:14.386 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.386 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:697123f8:::benchmark_data_smithi090_166897_object64094:head# 2024-05-15T23:40:14.386 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.386 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69716677:::benchmark_data_smithi090_166897_object40801:head# 2024-05-15T23:40:14.386 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.386 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69721512:::benchmark_data_smithi090_166897_object70257:head# 2024-05-15T23:40:14.386 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.386 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6977135c:::benchmark_data_smithi090_166897_object63671:head# 2024-05-15T23:40:14.386 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.386 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6977d708:::benchmark_data_smithi090_166897_object65272:head# 2024-05-15T23:40:14.387 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.387 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:697abf16:::benchmark_data_smithi090_166897_object102:head# 2024-05-15T23:40:14.387 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.387 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:697ae5fc:::benchmark_data_smithi090_166897_object37372:head# 2024-05-15T23:40:14.387 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.387 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:697b5065:::benchmark_data_smithi090_166897_object8016:head# 2024-05-15T23:40:14.387 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.387 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:697bbd53:::benchmark_data_smithi090_166897_object67388:head# 2024-05-15T23:40:14.387 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.387 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:697be02b:::benchmark_data_smithi090_166897_object67987:head# 2024-05-15T23:40:14.387 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.387 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:697e2e22:::benchmark_data_smithi090_166897_object54118:head# 2024-05-15T23:40:14.387 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.387 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:697e6e91:::benchmark_data_smithi090_166897_object7566:head# 2024-05-15T23:40:14.388 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.388 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:697f0dcd:::benchmark_data_smithi090_166897_object17950:head# 2024-05-15T23:40:14.388 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.388 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:697f1f18:::benchmark_data_smithi090_166897_object37924:head# 2024-05-15T23:40:14.388 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.388 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6980eed0:::benchmark_data_smithi090_166897_object55283:head# 2024-05-15T23:40:14.388 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.388 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6980f268:::benchmark_data_smithi090_166897_object48900:head# 2024-05-15T23:40:14.388 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.388 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6981affa:::benchmark_data_smithi090_166897_object68156:head# 2024-05-15T23:40:14.388 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.388 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6984b5b0:::benchmark_data_smithi090_166897_object64954:head# 2024-05-15T23:40:14.388 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.388 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69861e4e:::benchmark_data_smithi090_166897_object68668:head# 2024-05-15T23:40:14.388 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.389 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:698631ec:::benchmark_data_smithi090_166897_object43663:head# 2024-05-15T23:40:14.389 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.389 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:698959b8:::benchmark_data_smithi090_166897_object2597:head# 2024-05-15T23:40:14.389 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.389 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:698a0dcd:::benchmark_data_smithi090_166897_object44670:head# 2024-05-15T23:40:14.389 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.389 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:698a35ee:::benchmark_data_smithi090_166897_object43770:head# 2024-05-15T23:40:14.389 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.389 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:698db010:::benchmark_data_smithi090_166897_object28771:head# 2024-05-15T23:40:14.389 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.389 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:698f0a6b:::benchmark_data_smithi090_166897_object51723:head# 2024-05-15T23:40:14.389 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.389 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69906023:::benchmark_data_smithi090_166897_object36380:head# 2024-05-15T23:40:14.389 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.389 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6990ea90:::benchmark_data_smithi090_166897_object61207:head# 2024-05-15T23:40:14.390 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.390 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6992af4b:::benchmark_data_smithi090_166897_object26849:head# 2024-05-15T23:40:14.390 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.390 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6992f9c8:::benchmark_data_smithi090_166897_object15385:head# 2024-05-15T23:40:14.390 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.390 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6993c00a:::benchmark_data_smithi090_166897_object16466:head# 2024-05-15T23:40:14.390 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.390 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:699461fc:::benchmark_data_smithi090_166897_object12147:head# 2024-05-15T23:40:14.390 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.390 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6994b0c9:::benchmark_data_smithi090_166897_object16190:head# 2024-05-15T23:40:14.390 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.390 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:699542e0:::benchmark_data_smithi090_166897_object41505:head# 2024-05-15T23:40:14.390 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.391 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6996a18f:::benchmark_data_smithi090_166897_object2910:head# 2024-05-15T23:40:14.391 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.391 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6996f14f:::benchmark_data_smithi090_166897_object41229:head# 2024-05-15T23:40:14.391 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.391 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6998534e:::benchmark_data_smithi090_166897_object56322:head# 2024-05-15T23:40:14.391 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.391 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69987545:::benchmark_data_smithi090_166897_object10139:head# 2024-05-15T23:40:14.391 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.391 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6998935b:::benchmark_data_smithi090_166897_object28243:head# 2024-05-15T23:40:14.391 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.391 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6999d48a:::benchmark_data_smithi090_166897_object51121:head# 2024-05-15T23:40:14.391 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.391 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:699a0b53:::benchmark_data_smithi090_166897_object5507:head# 2024-05-15T23:40:14.391 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.391 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:699b98fe:::benchmark_data_smithi090_166897_object64411:head# 2024-05-15T23:40:14.392 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.392 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:699be1c5:::benchmark_data_smithi090_166897_object8575:head# 2024-05-15T23:40:14.392 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.392 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:699bf465:::benchmark_data_smithi090_166897_object16047:head# 2024-05-15T23:40:14.392 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.392 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:699c8774:::benchmark_data_smithi090_166897_object57234:head# 2024-05-15T23:40:14.392 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.392 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:699d158f:::benchmark_data_smithi090_166897_object33566:head# 2024-05-15T23:40:14.392 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.392 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:699e1bf2:::benchmark_data_smithi090_166897_object27369:head# 2024-05-15T23:40:14.392 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.392 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:699ece7e:::benchmark_data_smithi090_166897_object29921:head# 2024-05-15T23:40:14.392 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.392 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:699f4ea7:::benchmark_data_smithi090_166897_object39620:head# 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69a16ea6:::benchmark_data_smithi090_166897_object59292:head# 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69a26369:::benchmark_data_smithi090_166897_object22338:head# 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69a38088:::benchmark_data_smithi090_166897_object66121:head# 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69a38bc3:::benchmark_data_smithi090_166897_object52636:head# 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69a3bdde:::benchmark_data_smithi090_166897_object39464:head# 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69a4229d:::benchmark_data_smithi090_166897_object3381:head# 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69a47f1a:::benchmark_data_smithi090_166897_object42088:head# 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.393 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69a5ea15:::benchmark_data_smithi090_166897_object20663:head# 2024-05-15T23:40:14.394 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.394 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69a6164c:::benchmark_data_smithi090_166897_object4223:head# 2024-05-15T23:40:14.394 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.394 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69a6dd22:::benchmark_data_smithi090_166897_object25409:head# 2024-05-15T23:40:14.394 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.394 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69a7968b:::benchmark_data_smithi090_166897_object62225:head# 2024-05-15T23:40:14.394 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.394 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69a8c229:::benchmark_data_smithi090_166897_object25138:head# 2024-05-15T23:40:14.394 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.394 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69a914fb:::benchmark_data_smithi090_166897_object697:head# 2024-05-15T23:40:14.394 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.394 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69a96b59:::benchmark_data_smithi090_166897_object45211:head# 2024-05-15T23:40:14.394 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.394 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69a97041:::benchmark_data_smithi090_166897_object16729:head# 2024-05-15T23:40:14.395 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.395 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69aacbdb:::benchmark_data_smithi090_166897_object36438:head# 2024-05-15T23:40:14.395 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.395 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69ac3d52:::benchmark_data_smithi090_166897_object53000:head# 2024-05-15T23:40:14.395 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.395 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69aca965:::benchmark_data_smithi090_166897_object20932:head# 2024-05-15T23:40:14.395 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.395 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69acba68:::benchmark_data_smithi090_166897_object56162:head# 2024-05-15T23:40:14.395 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.395 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69ad3474:::benchmark_data_smithi090_166897_object25442:head# 2024-05-15T23:40:14.395 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.395 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69ad4549:::benchmark_data_smithi090_166897_object21428:head# 2024-05-15T23:40:14.395 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.395 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69adcd88:::benchmark_data_smithi090_166897_object31757:head# 2024-05-15T23:40:14.395 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.396 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69aee348:::benchmark_data_smithi090_166897_object36881:head# 2024-05-15T23:40:14.396 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.396 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69af5cb8:::benchmark_data_smithi090_166897_object63914:head# 2024-05-15T23:40:14.396 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.396 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69af7645:::benchmark_data_smithi090_166897_object9790:head# 2024-05-15T23:40:14.396 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.396 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69afcd5d:::benchmark_data_smithi090_166897_object49335:head# 2024-05-15T23:40:14.396 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.396 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69afdf62:::benchmark_data_smithi090_166897_object31558:head# 2024-05-15T23:40:14.396 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.396 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b00517:::benchmark_data_smithi090_166897_object67088:head# 2024-05-15T23:40:14.396 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.396 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b16d20:::benchmark_data_smithi090_166897_object9331:head# 2024-05-15T23:40:14.396 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.396 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b1c866:::benchmark_data_smithi090_166897_object13564:head# 2024-05-15T23:40:14.397 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.397 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b3ad0a:::benchmark_data_smithi090_166897_object15069:head# 2024-05-15T23:40:14.397 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.397 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b41531:::benchmark_data_smithi090_166897_object17523:head# 2024-05-15T23:40:14.397 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.397 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b4bedf:::benchmark_data_smithi090_166897_object22702:head# 2024-05-15T23:40:14.397 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.397 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b50d6c:::benchmark_data_smithi090_166897_object54370:head# 2024-05-15T23:40:14.397 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.397 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b571f8:::benchmark_data_smithi090_166897_object34908:head# 2024-05-15T23:40:14.397 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.397 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b5f285:::benchmark_data_smithi090_166897_object43273:head# 2024-05-15T23:40:14.397 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.397 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b66e5e:::benchmark_data_smithi090_166897_object57649:head# 2024-05-15T23:40:14.398 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.398 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b68e9d:::benchmark_data_smithi090_166897_object59279:head# 2024-05-15T23:40:14.398 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.398 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b7715b:::benchmark_data_smithi090_166897_object14148:head# 2024-05-15T23:40:14.398 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.398 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b7ee42:::benchmark_data_smithi090_166897_object36016:head# 2024-05-15T23:40:14.398 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.398 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b7f6b3:::benchmark_data_smithi090_166897_object6156:head# 2024-05-15T23:40:14.398 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.398 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b8525b:::benchmark_data_smithi090_166897_object69367:head# 2024-05-15T23:40:14.398 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.398 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b8c3ae:::benchmark_data_smithi090_166897_object47408:head# 2024-05-15T23:40:14.398 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.398 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69b98b74:::benchmark_data_smithi090_166897_object29519:head# 2024-05-15T23:40:14.399 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.399 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69bc2ed7:::benchmark_data_smithi090_166897_object3551:head# 2024-05-15T23:40:14.399 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.399 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69bcf21a:::benchmark_data_smithi090_166897_object42135:head# 2024-05-15T23:40:14.399 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.399 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69be0ba9:::benchmark_data_smithi090_166897_object51438:head# 2024-05-15T23:40:14.399 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.399 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69becfbf:::benchmark_data_smithi090_166897_object8691:head# 2024-05-15T23:40:14.399 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.399 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69bf475d:::benchmark_data_smithi090_166897_object70047:head# 2024-05-15T23:40:14.399 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.399 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69bf935d:::benchmark_data_smithi090_166897_object15280:head# 2024-05-15T23:40:14.399 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.399 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69bfee4e:::benchmark_data_smithi090_166897_object63580:head# 2024-05-15T23:40:14.399 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.400 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69c058a7:::benchmark_data_smithi090_166897_object34839:head# 2024-05-15T23:40:14.400 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.400 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69c19795:::benchmark_data_smithi090_166897_object3431:head# 2024-05-15T23:40:14.400 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.400 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69c1983a:::benchmark_data_smithi090_166897_object40883:head# 2024-05-15T23:40:14.400 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.400 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69c21108:::benchmark_data_smithi090_166897_object31861:head# 2024-05-15T23:40:14.400 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.400 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69c276dc:::benchmark_data_smithi090_166897_object41911:head# 2024-05-15T23:40:14.400 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.400 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69c5d468:::benchmark_data_smithi090_166897_object23877:head# 2024-05-15T23:40:14.400 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.400 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69c86206:::benchmark_data_smithi090_166897_object61206:head# 2024-05-15T23:40:14.400 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.400 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69c8c42e:::benchmark_data_smithi090_166897_object69318:head# 2024-05-15T23:40:14.401 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.401 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69c8d507:::benchmark_data_smithi090_166897_object42281:head# 2024-05-15T23:40:14.401 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.401 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69c945cb:::benchmark_data_smithi090_166897_object40375:head# 2024-05-15T23:40:14.401 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.401 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69c9b0a1:::benchmark_data_smithi090_166897_object15053:head# 2024-05-15T23:40:14.401 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.401 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69caf5a1:::benchmark_data_smithi090_166897_object12809:head# 2024-05-15T23:40:14.401 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.401 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69ce9406:::benchmark_data_smithi090_166897_object65665:head# 2024-05-15T23:40:14.401 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.401 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69cef1c1:::benchmark_data_smithi090_166897_object4623:head# 2024-05-15T23:40:14.401 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.401 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d04748:::benchmark_data_smithi090_166897_object33490:head# 2024-05-15T23:40:14.401 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.402 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d1f74c:::benchmark_data_smithi090_166897_object54729:head# 2024-05-15T23:40:14.402 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.402 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d1f93f:::benchmark_data_smithi090_166897_object14701:head# 2024-05-15T23:40:14.402 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.402 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d209fc:::benchmark_data_smithi090_166897_object32529:head# 2024-05-15T23:40:14.402 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.402 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d27867:::benchmark_data_smithi090_166897_object57306:head# 2024-05-15T23:40:14.402 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.402 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d37634:::benchmark_data_smithi090_166897_object53214:head# 2024-05-15T23:40:14.402 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.402 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d3bf82:::benchmark_data_smithi090_166897_object19884:head# 2024-05-15T23:40:14.402 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.402 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d4b796:::benchmark_data_smithi090_166897_object40528:head# 2024-05-15T23:40:14.402 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.403 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d57afc:::benchmark_data_smithi090_166897_object57969:head# 2024-05-15T23:40:14.403 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.403 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d57e08:::benchmark_data_smithi090_166897_object69017:head# 2024-05-15T23:40:14.403 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.403 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d58bc7:::benchmark_data_smithi090_166897_object7883:head# 2024-05-15T23:40:14.403 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.403 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d6b83e:::benchmark_data_smithi090_166897_object61601:head# 2024-05-15T23:40:14.403 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.403 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d7bdb9:::benchmark_data_smithi090_166897_object616:head# 2024-05-15T23:40:14.403 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.403 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d7f067:::benchmark_data_smithi090_166897_object1856:head# 2024-05-15T23:40:14.403 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.403 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d8717f:::benchmark_data_smithi090_166897_object37564:head# 2024-05-15T23:40:14.403 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.404 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d88e95:::benchmark_data_smithi090_166897_object22667:head# 2024-05-15T23:40:14.404 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.404 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d99ec7:::benchmark_data_smithi090_166897_object6382:head# 2024-05-15T23:40:14.404 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.404 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69d9a70e:::benchmark_data_smithi090_166897_object52392:head# 2024-05-15T23:40:14.404 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.404 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69da6f00:::benchmark_data_smithi090_166897_object37860:head# 2024-05-15T23:40:14.404 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.404 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69daa6e4:::benchmark_data_smithi090_166897_object5065:head# 2024-05-15T23:40:14.404 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.404 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69dc89f3:::benchmark_data_smithi090_166897_object22759:head# 2024-05-15T23:40:14.404 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.404 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69de8053:::benchmark_data_smithi090_166897_object20816:head# 2024-05-15T23:40:14.405 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.405 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69df610b:::benchmark_data_smithi090_166897_object7816:head# 2024-05-15T23:40:14.405 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.405 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69e09667:::benchmark_data_smithi090_166897_object17554:head# 2024-05-15T23:40:14.405 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.405 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69e1ef36:::benchmark_data_smithi090_166897_object8632:head# 2024-05-15T23:40:14.405 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.405 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69e21ce5:::benchmark_data_smithi090_166897_object40661:head# 2024-05-15T23:40:14.405 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.405 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69e7307e:::benchmark_data_smithi090_166897_object38857:head# 2024-05-15T23:40:14.405 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.405 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69e9aeaf:::benchmark_data_smithi090_166897_object22238:head# 2024-05-15T23:40:14.405 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.405 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69ea6a98:::benchmark_data_smithi090_166897_object28827:head# 2024-05-15T23:40:14.405 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.406 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69eb7437:::benchmark_data_smithi090_166897_object70198:head# 2024-05-15T23:40:14.406 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.406 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69ed7960:::benchmark_data_smithi090_166897_object33012:head# 2024-05-15T23:40:14.406 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.406 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69ee523d:::benchmark_data_smithi090_166897_object12782:head# 2024-05-15T23:40:14.406 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.406 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69ee8eb7:::benchmark_data_smithi090_166897_object65659:head# 2024-05-15T23:40:14.406 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.406 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69ee8f62:::benchmark_data_smithi090_166897_object8982:head# 2024-05-15T23:40:14.406 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.406 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69f02e73:::benchmark_data_smithi090_166897_object23573:head# 2024-05-15T23:40:14.406 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.406 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69f1e041:::benchmark_data_smithi090_166897_object49892:head# 2024-05-15T23:40:14.406 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.407 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69f2efc3:::benchmark_data_smithi090_166897_object4172:head# 2024-05-15T23:40:14.407 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.407 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69f3b8ca:::benchmark_data_smithi090_166897_object26159:head# 2024-05-15T23:40:14.407 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.407 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69f3bb1c:::benchmark_data_smithi090_166897_object58709:head# 2024-05-15T23:40:14.407 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.407 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69f40e06:::benchmark_data_smithi090_166897_object59638:head# 2024-05-15T23:40:14.407 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.407 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69f78d9d:::benchmark_data_smithi090_166897_object13323:head# 2024-05-15T23:40:14.407 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.407 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69f96fcb:::benchmark_data_smithi090_166897_object70531:head# 2024-05-15T23:40:14.407 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.407 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69fb4080:::benchmark_data_smithi090_166897_object24499:head# 2024-05-15T23:40:14.407 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.407 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69fc5662:::benchmark_data_smithi090_166897_object53269:head# 2024-05-15T23:40:14.408 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.408 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:69fd7405:::benchmark_data_smithi090_166897_object29298:head# 2024-05-15T23:40:14.408 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.408 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a006b7d:::benchmark_data_smithi090_166897_object58716:head# 2024-05-15T23:40:14.408 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.408 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a00c109:::benchmark_data_smithi090_166897_object32512:head# 2024-05-15T23:40:14.408 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.408 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a00fa46:::benchmark_data_smithi090_166897_object33786:head# 2024-05-15T23:40:14.408 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.408 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a01550c:::benchmark_data_smithi090_166897_object20773:head# 2024-05-15T23:40:14.408 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.408 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a01ac91:::benchmark_data_smithi090_166897_object2799:head# 2024-05-15T23:40:14.408 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.408 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a02ef21:::benchmark_data_smithi090_166897_object16497:head# 2024-05-15T23:40:14.408 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.409 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a02fba6:::benchmark_data_smithi090_166897_object50831:head# 2024-05-15T23:40:14.409 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.409 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a0318e7:::benchmark_data_smithi090_166897_object35643:head# 2024-05-15T23:40:14.409 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.409 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a0325c1:::benchmark_data_smithi090_166897_object1340:head# 2024-05-15T23:40:14.409 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.409 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a039969:::benchmark_data_smithi090_166897_object30296:head# 2024-05-15T23:40:14.409 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.409 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a040d41:::benchmark_data_smithi090_166897_object10611:head# 2024-05-15T23:40:14.409 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.409 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a053510:::benchmark_data_smithi090_166897_object58203:head# 2024-05-15T23:40:14.409 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.409 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a059746:::benchmark_data_smithi090_166897_object51758:head# 2024-05-15T23:40:14.409 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.410 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a07feef:::benchmark_data_smithi090_166897_object52006:head# 2024-05-15T23:40:14.410 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.410 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a09cf2b:::benchmark_data_smithi090_166897_object42981:head# 2024-05-15T23:40:14.410 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.410 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a0a2c93:::benchmark_data_smithi090_166897_object63218:head# 2024-05-15T23:40:14.410 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.410 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a0a4f23:::benchmark_data_smithi090_166897_object63858:head# 2024-05-15T23:40:14.410 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.410 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a0f65aa:::benchmark_data_smithi090_166897_object2742:head# 2024-05-15T23:40:14.410 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.410 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a0fb374:::benchmark_data_smithi090_166897_object4387:head# 2024-05-15T23:40:14.410 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.410 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a11595c:::benchmark_data_smithi090_166897_object20111:head# 2024-05-15T23:40:14.410 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.411 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a121ccb:::benchmark_data_smithi090_166897_object64612:head# 2024-05-15T23:40:14.411 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.411 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a125d09:::benchmark_data_smithi090_166897_object36209:head# 2024-05-15T23:40:14.411 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.411 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a13e492:::benchmark_data_smithi090_166897_object59284:head# 2024-05-15T23:40:14.411 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.411 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a15c6dd:::benchmark_data_smithi090_166897_object24484:head# 2024-05-15T23:40:14.411 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.411 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a17d015:::benchmark_data_smithi090_166897_object30366:head# 2024-05-15T23:40:14.411 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.411 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a191799:::benchmark_data_smithi090_166897_object17151:head# 2024-05-15T23:40:14.411 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.411 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a1a7693:::benchmark_data_smithi090_166897_object48686:head# 2024-05-15T23:40:14.411 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.411 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a1bf47c:::benchmark_data_smithi090_166897_object28481:head# 2024-05-15T23:40:14.412 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.412 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a1c2307:::benchmark_data_smithi090_166897_object68603:head# 2024-05-15T23:40:14.412 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.412 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a1d2ad2:::benchmark_data_smithi090_166897_object4308:head# 2024-05-15T23:40:14.412 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.412 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a1dbe6a:::benchmark_data_smithi090_166897_object69068:head# 2024-05-15T23:40:14.412 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.412 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a1dd0ae:::benchmark_data_smithi090_166897_object35988:head# 2024-05-15T23:40:14.412 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.412 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a1f991c:::benchmark_data_smithi090_166897_object60362:head# 2024-05-15T23:40:14.412 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.412 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a210623:::benchmark_data_smithi090_166897_object5789:head# 2024-05-15T23:40:14.412 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.412 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a232f16:::benchmark_data_smithi090_166897_object71051:head# 2024-05-15T23:40:14.413 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.413 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a24dd4f:::benchmark_data_smithi090_166897_object22901:head# 2024-05-15T23:40:14.413 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.413 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a24f05e:::benchmark_data_smithi090_166897_object38618:head# 2024-05-15T23:40:14.413 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.413 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a26d589:::benchmark_data_smithi090_166897_object60575:head# 2024-05-15T23:40:14.413 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.413 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a2745da:::benchmark_data_smithi090_166897_object32567:head# 2024-05-15T23:40:14.413 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.413 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a274651:::benchmark_data_smithi090_166897_object12697:head# 2024-05-15T23:40:14.413 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.413 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a278921:::benchmark_data_smithi090_166897_object67768:head# 2024-05-15T23:40:14.413 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.413 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a286179:::benchmark_data_smithi090_166897_object65122:head# 2024-05-15T23:40:14.413 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.414 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a28b77d:::benchmark_data_smithi090_166897_object28327:head# 2024-05-15T23:40:14.414 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.414 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a28f736:::benchmark_data_smithi090_166897_object26314:head# 2024-05-15T23:40:14.414 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.414 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a29fb3d:::benchmark_data_smithi090_166897_object54791:head# 2024-05-15T23:40:14.414 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.414 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a2beef3:::benchmark_data_smithi090_166897_object6798:head# 2024-05-15T23:40:14.414 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.414 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a2c7991:::benchmark_data_smithi090_166897_object56688:head# 2024-05-15T23:40:14.414 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.414 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a2da965:::benchmark_data_smithi090_166897_object50750:head# 2024-05-15T23:40:14.414 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.414 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a2db7b6:::benchmark_data_smithi090_166897_object47772:head# 2024-05-15T23:40:14.414 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.415 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a2dca35:::benchmark_data_smithi090_166897_object28111:head# 2024-05-15T23:40:14.415 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.415 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a2e7784:::benchmark_data_smithi090_166897_object25258:head# 2024-05-15T23:40:14.415 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.415 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a2ed805:::benchmark_data_smithi090_166897_object17896:head# 2024-05-15T23:40:14.415 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.415 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a2f4951:::benchmark_data_smithi090_166897_object59105:head# 2024-05-15T23:40:14.415 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.415 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a30774d:::benchmark_data_smithi090_166897_object3750:head# 2024-05-15T23:40:14.415 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.415 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a30cc13:::benchmark_data_smithi090_166897_object58744:head# 2024-05-15T23:40:14.415 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.415 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a316e83:::benchmark_data_smithi090_166897_object58134:head# 2024-05-15T23:40:14.415 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.415 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a328ab7:::benchmark_data_smithi090_166897_object22085:head# 2024-05-15T23:40:14.416 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.416 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a32c27a:::benchmark_data_smithi090_166897_object23054:head# 2024-05-15T23:40:14.416 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.416 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a339b3c:::benchmark_data_smithi090_166897_object57285:head# 2024-05-15T23:40:14.416 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.416 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a34b3c9:::benchmark_data_smithi090_166897_object13381:head# 2024-05-15T23:40:14.416 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.416 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3528cb:::benchmark_data_smithi090_166897_object20935:head# 2024-05-15T23:40:14.416 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.416 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3567c7:::benchmark_data_smithi090_166897_object26247:head# 2024-05-15T23:40:14.416 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.416 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a371f93:::benchmark_data_smithi090_166897_object5248:head# 2024-05-15T23:40:14.416 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.416 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a372567:::benchmark_data_smithi090_166897_object5404:head# 2024-05-15T23:40:14.417 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.417 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a373943:::benchmark_data_smithi090_166897_object31637:head# 2024-05-15T23:40:14.417 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.417 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a381416:::benchmark_data_smithi090_166897_object34972:head# 2024-05-15T23:40:14.417 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.417 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a382add:::benchmark_data_smithi090_166897_object30835:head# 2024-05-15T23:40:14.417 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.417 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3887ad:::benchmark_data_smithi090_166897_object16889:head# 2024-05-15T23:40:14.417 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.417 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3aa601:::benchmark_data_smithi090_166897_object52032:head# 2024-05-15T23:40:14.417 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.417 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3b18f3:::benchmark_data_smithi090_166897_object64042:head# 2024-05-15T23:40:14.417 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.417 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3bca67:::benchmark_data_smithi090_166897_object12287:head# 2024-05-15T23:40:14.418 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.418 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3bcea7:::benchmark_data_smithi090_166897_object32070:head# 2024-05-15T23:40:14.418 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.418 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3cd630:::benchmark_data_smithi090_166897_object29108:head# 2024-05-15T23:40:14.418 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.418 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3d3d53:::benchmark_data_smithi090_166897_object695:head# 2024-05-15T23:40:14.418 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.418 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3d532d:::benchmark_data_smithi090_166897_object32103:head# 2024-05-15T23:40:14.418 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.418 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3d711f:::benchmark_data_smithi090_166897_object9180:head# 2024-05-15T23:40:14.418 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.418 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3d791f:::benchmark_data_smithi090_166897_object33927:head# 2024-05-15T23:40:14.418 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.418 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3e5690:::benchmark_data_smithi090_166897_object58111:head# 2024-05-15T23:40:14.419 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.419 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3ed491:::benchmark_data_smithi090_166897_object32299:head# 2024-05-15T23:40:14.419 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.419 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3f3c57:::benchmark_data_smithi090_166897_object26842:head# 2024-05-15T23:40:14.419 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.419 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a3f9e97:::benchmark_data_smithi090_166897_object57571:head# 2024-05-15T23:40:14.419 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.419 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a41713d:::benchmark_data_smithi090_166897_object68067:head# 2024-05-15T23:40:14.419 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.419 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a421cd0:::benchmark_data_smithi090_166897_object20797:head# 2024-05-15T23:40:14.419 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.419 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a431597:::benchmark_data_smithi090_166897_object35357:head# 2024-05-15T23:40:14.419 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.420 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a438146:::benchmark_data_smithi090_166897_object19269:head# 2024-05-15T23:40:14.420 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.420 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a448adc:::benchmark_data_smithi090_166897_object20799:head# 2024-05-15T23:40:14.420 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.420 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a455417:::benchmark_data_smithi090_166897_object5958:head# 2024-05-15T23:40:14.420 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.420 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a468874:::benchmark_data_smithi090_166897_object17757:head# 2024-05-15T23:40:14.420 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.420 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a482e9e:::benchmark_data_smithi090_166897_object38584:head# 2024-05-15T23:40:14.420 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.420 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a485a3f:::benchmark_data_smithi090_166897_object16496:head# 2024-05-15T23:40:14.420 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.420 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a48b7cc:::benchmark_data_smithi090_166897_object43757:head# 2024-05-15T23:40:14.420 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a4a42de:::benchmark_data_smithi090_166897_object63804:head# 2024-05-15T23:40:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a4cf7f2:::benchmark_data_smithi090_166897_object42025:head# 2024-05-15T23:40:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a4eb07f:::benchmark_data_smithi090_166897_object21792:head# 2024-05-15T23:40:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a4ed6f4:::benchmark_data_smithi090_166897_object41050:head# 2024-05-15T23:40:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a4ef4ee:::benchmark_data_smithi090_166897_object63624:head# 2024-05-15T23:40:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a4f35c2:::benchmark_data_smithi090_166897_object31396:head# 2024-05-15T23:40:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a50f16d:::benchmark_data_smithi090_166897_object41971:head# 2024-05-15T23:40:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.421 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a513e53:::benchmark_data_smithi090_166897_object16499:head# 2024-05-15T23:40:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a513fb9:::benchmark_data_smithi090_166897_object56888:head# 2024-05-15T23:40:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a556751:::benchmark_data_smithi090_166897_object40841:head# 2024-05-15T23:40:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a55e29e:::benchmark_data_smithi090_166897_object14587:head# 2024-05-15T23:40:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a560525:::benchmark_data_smithi090_166897_object54543:head# 2024-05-15T23:40:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a5679fc:::benchmark_data_smithi090_166897_object22844:head# 2024-05-15T23:40:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a56876d:::benchmark_data_smithi090_166897_object60359:head# 2024-05-15T23:40:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a57cec5:::benchmark_data_smithi090_166897_object32485:head# 2024-05-15T23:40:14.423 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.423 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a5a07d0:::benchmark_data_smithi090_166897_object20476:head# 2024-05-15T23:40:14.423 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.423 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a5a1200:::benchmark_data_smithi090_166897_object23995:head# 2024-05-15T23:40:14.423 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.423 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a5ad6f3:::benchmark_data_smithi090_166897_object20819:head# 2024-05-15T23:40:14.423 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.423 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a5c7faa:::benchmark_data_smithi090_166897_object62734:head# 2024-05-15T23:40:14.423 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.423 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a5c98f4:::benchmark_data_smithi090_166897_object27087:head# 2024-05-15T23:40:14.423 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.423 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a5d4c2b:::benchmark_data_smithi090_166897_object10164:head# 2024-05-15T23:40:14.423 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.423 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a5f79dc:::benchmark_data_smithi090_166897_object60390:head# 2024-05-15T23:40:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a5fa929:::benchmark_data_smithi090_166897_object40317:head# 2024-05-15T23:40:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a636a40:::benchmark_data_smithi090_166897_object7783:head# 2024-05-15T23:40:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a63f85d:::benchmark_data_smithi090_166897_object37735:head# 2024-05-15T23:40:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a647672:::benchmark_data_smithi090_166897_object19926:head# 2024-05-15T23:40:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a6540f4:::benchmark_data_smithi090_166897_object57244:head# 2024-05-15T23:40:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a6c4591:::benchmark_data_smithi090_166897_object70036:head# 2024-05-15T23:40:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a6c94bf:::benchmark_data_smithi090_166897_object2113:head# 2024-05-15T23:40:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.425 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a6f8d60:::benchmark_data_smithi090_166897_object65357:head# 2024-05-15T23:40:14.425 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.425 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a704959:::benchmark_data_smithi090_166897_object4217:head# 2024-05-15T23:40:14.425 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.425 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a70f3db:::benchmark_data_smithi090_166897_object33708:head# 2024-05-15T23:40:14.425 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.425 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a7198e0:::benchmark_data_smithi090_166897_object54743:head# 2024-05-15T23:40:14.425 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.425 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a75570e:::benchmark_data_smithi090_166897_object10005:head# 2024-05-15T23:40:14.425 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.425 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a761999:::benchmark_data_smithi090_166897_object8307:head# 2024-05-15T23:40:14.425 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.425 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a76b917:::benchmark_data_smithi090_166897_object1502:head# 2024-05-15T23:40:14.426 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.426 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a76dcf7:::benchmark_data_smithi090_166897_object64047:head# 2024-05-15T23:40:14.426 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.426 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a777ff1:::benchmark_data_smithi090_166897_object30413:head# 2024-05-15T23:40:14.426 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.426 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a77d1ff:::benchmark_data_smithi090_166897_object494:head# 2024-05-15T23:40:14.426 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.426 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a78e2fe:::benchmark_data_smithi090_166897_object38997:head# 2024-05-15T23:40:14.426 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.426 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a796b86:::benchmark_data_smithi090_166897_object13022:head# 2024-05-15T23:40:14.426 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.426 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a799727:::benchmark_data_smithi090_166897_object25979:head# 2024-05-15T23:40:14.426 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.426 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a7ae2f1:::benchmark_data_smithi090_166897_object42972:head# 2024-05-15T23:40:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a7d9ee5:::benchmark_data_smithi090_166897_object66584:head# 2024-05-15T23:40:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a7e2ea6:::benchmark_data_smithi090_166897_object54635:head# 2024-05-15T23:40:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a7e88bf:::benchmark_data_smithi090_166897_object20502:head# 2024-05-15T23:40:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a80a2fb:::benchmark_data_smithi090_166897_object36272:head# 2024-05-15T23:40:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a8105ef:::benchmark_data_smithi090_166897_object10375:head# 2024-05-15T23:40:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a8203f8:::benchmark_data_smithi090_166897_object59467:head# 2024-05-15T23:40:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a820f9b:::benchmark_data_smithi090_166897_object20923:head# 2024-05-15T23:40:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.428 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a825d0c:::benchmark_data_smithi090_166897_object22864:head# 2024-05-15T23:40:14.428 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.428 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a86c712:::benchmark_data_smithi090_166897_object33034:head# 2024-05-15T23:40:14.428 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.428 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a8795c3:::benchmark_data_smithi090_166897_object58366:head# 2024-05-15T23:40:14.428 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.428 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a8871a7:::benchmark_data_smithi090_166897_object42989:head# 2024-05-15T23:40:14.428 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.428 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a8a72ef:::benchmark_data_smithi090_166897_object3005:head# 2024-05-15T23:40:14.428 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.428 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a8c6523:::benchmark_data_smithi090_166897_object30527:head# 2024-05-15T23:40:14.428 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.428 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a8ccf49:::benchmark_data_smithi090_166897_object32745:head# 2024-05-15T23:40:14.428 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.428 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a9006ae:::benchmark_data_smithi090_166897_object62507:head# 2024-05-15T23:40:14.429 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.429 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a91dc08:::benchmark_data_smithi090_166897_object21555:head# 2024-05-15T23:40:14.429 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.429 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a93efed:::benchmark_data_smithi090_166897_object29410:head# 2024-05-15T23:40:14.429 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.429 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a93f906:::benchmark_data_smithi090_166897_object21222:head# 2024-05-15T23:40:14.429 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.429 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a95cc32:::benchmark_data_smithi090_166897_object37136:head# 2024-05-15T23:40:14.429 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.429 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a960612:::benchmark_data_smithi090_166897_object66533:head# 2024-05-15T23:40:14.429 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.429 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a964849:::benchmark_data_smithi090_166897_object45430:head# 2024-05-15T23:40:14.429 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.429 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a969690:::benchmark_data_smithi090_166897_object27022:head# 2024-05-15T23:40:14.430 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.430 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a9700aa:::benchmark_data_smithi090_166897_object23008:head# 2024-05-15T23:40:14.430 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.430 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a971d92:::benchmark_data_smithi090_166897_object16301:head# 2024-05-15T23:40:14.430 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.430 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a97347d:::benchmark_data_smithi090_166897_object66848:head# 2024-05-15T23:40:14.430 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.430 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a98db6b:::benchmark_data_smithi090_166897_object12465:head# 2024-05-15T23:40:14.430 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.430 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a9909e8:::benchmark_data_smithi090_166897_object21697:head# 2024-05-15T23:40:14.430 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.430 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a99efb5:::benchmark_data_smithi090_166897_object31607:head# 2024-05-15T23:40:14.430 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.430 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a99fcef:::benchmark_data_smithi090_166897_object16909:head# 2024-05-15T23:40:14.431 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.431 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a9a96d3:::benchmark_data_smithi090_166897_object27468:head# 2024-05-15T23:40:14.431 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.431 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a9ae885:::benchmark_data_smithi090_166897_object59403:head# 2024-05-15T23:40:14.431 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.431 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a9d5512:::benchmark_data_smithi090_166897_object30718:head# 2024-05-15T23:40:14.431 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.431 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a9d65d3:::benchmark_data_smithi090_166897_object2832:head# 2024-05-15T23:40:14.431 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.431 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a9d8de1:::benchmark_data_smithi090_166897_object8642:head# 2024-05-15T23:40:14.431 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.431 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a9d9d5a:::benchmark_data_smithi090_166897_object55430:head# 2024-05-15T23:40:14.431 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.431 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6a9e124a:::benchmark_data_smithi090_166897_object65963:head# 2024-05-15T23:40:14.432 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.432 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aa021de:::benchmark_data_smithi090_166897_object18614:head# 2024-05-15T23:40:14.432 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.432 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aa0a236:::benchmark_data_smithi090_166897_object42000:head# 2024-05-15T23:40:14.432 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.432 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aa0d1f5:::benchmark_data_smithi090_166897_object1851:head# 2024-05-15T23:40:14.432 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.432 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aa0e81d:::benchmark_data_smithi090_166897_object25039:head# 2024-05-15T23:40:14.432 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.432 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aa1146f:::benchmark_data_smithi090_166897_object56992:head# 2024-05-15T23:40:14.432 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.432 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aa15dd4:::benchmark_data_smithi090_166897_object16297:head# 2024-05-15T23:40:14.432 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.432 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aa4aa26:::benchmark_data_smithi090_166897_object35949:head# 2024-05-15T23:40:14.432 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.433 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aa4e72e:::benchmark_data_smithi090_166897_object48908:head# 2024-05-15T23:40:14.433 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.433 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aa7711a:::benchmark_data_smithi090_166897_object5782:head# 2024-05-15T23:40:14.433 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.433 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aa86256:::benchmark_data_smithi090_166897_object17039:head# 2024-05-15T23:40:14.433 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.433 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aa896e2:::benchmark_data_smithi090_166897_object7687:head# 2024-05-15T23:40:14.433 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.433 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aa99581:::benchmark_data_smithi090_166897_object11744:head# 2024-05-15T23:40:14.433 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.433 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aaa1edc:::benchmark_data_smithi090_166897_object23490:head# 2024-05-15T23:40:14.433 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.433 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aaa45cd:::benchmark_data_smithi090_166897_object15283:head# 2024-05-15T23:40:14.433 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.434 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aabcfab:::benchmark_data_smithi090_166897_object39954:head# 2024-05-15T23:40:14.434 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.434 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aadbff8:::benchmark_data_smithi090_166897_object33642:head# 2024-05-15T23:40:14.434 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.434 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aae3888:::benchmark_data_smithi090_166897_object65493:head# 2024-05-15T23:40:14.434 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.434 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aaee7cd:::benchmark_data_smithi090_166897_object28770:head# 2024-05-15T23:40:14.434 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.434 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aaf66be:::benchmark_data_smithi090_166897_object55637:head# 2024-05-15T23:40:14.434 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.434 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aaf91e5:::benchmark_data_smithi090_166897_object3906:head# 2024-05-15T23:40:14.434 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.434 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aaf98f4:::benchmark_data_smithi090_166897_object45450:head# 2024-05-15T23:40:14.435 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.435 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ab16383:::benchmark_data_smithi090_166897_object60864:head# 2024-05-15T23:40:14.435 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.435 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ab1decc:::benchmark_data_smithi090_166897_object46090:head# 2024-05-15T23:40:14.435 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.435 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ab235cd:::benchmark_data_smithi090_166897_object3188:head# 2024-05-15T23:40:14.435 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.435 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ab2818f:::benchmark_data_smithi090_166897_object12238:head# 2024-05-15T23:40:14.435 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.435 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ab34a85:::benchmark_data_smithi090_166897_object4955:head# 2024-05-15T23:40:14.435 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.435 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ab34b1b:::benchmark_data_smithi090_166897_object32886:head# 2024-05-15T23:40:14.435 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.435 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ab59758:::benchmark_data_smithi090_166897_object12066:head# 2024-05-15T23:40:14.435 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.436 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ab5b153:::benchmark_data_smithi090_166897_object63063:head# 2024-05-15T23:40:14.436 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.436 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ab78870:::benchmark_data_smithi090_166897_object26218:head# 2024-05-15T23:40:14.436 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.436 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ab84fad:::benchmark_data_smithi090_166897_object22402:head# 2024-05-15T23:40:14.436 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.436 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ab908c1:::benchmark_data_smithi090_166897_object68430:head# 2024-05-15T23:40:14.436 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.436 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ab97fa8:::benchmark_data_smithi090_166897_object4589:head# 2024-05-15T23:40:14.436 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.436 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aba5590:::benchmark_data_smithi090_166897_object65555:head# 2024-05-15T23:40:14.436 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.436 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6abb883d:::benchmark_data_smithi090_166897_object8964:head# 2024-05-15T23:40:14.436 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.436 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6abbae16:::benchmark_data_smithi090_166897_object65767:head# 2024-05-15T23:40:14.437 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.437 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6abe2bfb:::benchmark_data_smithi090_166897_object44685:head# 2024-05-15T23:40:14.437 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.437 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6abe4923:::benchmark_data_smithi090_166897_object69148:head# 2024-05-15T23:40:14.437 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.437 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6abe50b4:::benchmark_data_smithi090_166897_object18662:head# 2024-05-15T23:40:14.437 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.437 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6abe8760:::benchmark_data_smithi090_166897_object19139:head# 2024-05-15T23:40:14.437 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.437 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ac137f5:::benchmark_data_smithi090_166897_object20010:head# 2024-05-15T23:40:14.437 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.437 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ac13e60:::benchmark_data_smithi090_166897_object43727:head# 2024-05-15T23:40:14.437 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.437 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ac269af:::benchmark_data_smithi090_166897_object46623:head# 2024-05-15T23:40:14.437 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.438 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ac38135:::benchmark_data_smithi090_166897_object5220:head# 2024-05-15T23:40:14.438 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.438 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ac3963d:::benchmark_data_smithi090_166897_object47888:head# 2024-05-15T23:40:14.438 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.438 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ac5732c:::benchmark_data_smithi090_166897_object9245:head# 2024-05-15T23:40:14.438 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.438 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ac60a59:::benchmark_data_smithi090_166897_object52694:head# 2024-05-15T23:40:14.438 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.438 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ac7d158:::benchmark_data_smithi090_166897_object19114:head# 2024-05-15T23:40:14.438 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.438 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ac86637:::benchmark_data_smithi090_166897_object66283:head# 2024-05-15T23:40:14.438 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.439 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ac9e34e:::benchmark_data_smithi090_166897_object34859:head# 2024-05-15T23:40:14.439 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.439 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6acb0213:::benchmark_data_smithi090_166897_object68888:head# 2024-05-15T23:40:14.439 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.439 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6acca9a2:::benchmark_data_smithi090_166897_object3075:head# 2024-05-15T23:40:14.439 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.439 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6acd17dd:::benchmark_data_smithi090_166897_object65757:head# 2024-05-15T23:40:14.439 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.439 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ace6293:::benchmark_data_smithi090_166897_object11221:head# 2024-05-15T23:40:14.439 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.439 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6acec768:::benchmark_data_smithi090_166897_object5070:head# 2024-05-15T23:40:14.439 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.439 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad0240a:::benchmark_data_smithi090_166897_object42792:head# 2024-05-15T23:40:14.439 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.440 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad0e65b:::benchmark_data_smithi090_166897_object19355:head# 2024-05-15T23:40:14.440 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.440 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad12514:::benchmark_data_smithi090_166897_object25110:head# 2024-05-15T23:40:14.440 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.440 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad16514:::benchmark_data_smithi090_166897_object11208:head# 2024-05-15T23:40:14.440 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.440 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad1d414:::benchmark_data_smithi090_166897_object2387:head# 2024-05-15T23:40:14.440 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.440 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad289aa:::benchmark_data_smithi090_166897_object64944:head# 2024-05-15T23:40:14.440 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.440 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad34341:::benchmark_data_smithi090_166897_object51067:head# 2024-05-15T23:40:14.440 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.440 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad349c3:::benchmark_data_smithi090_166897_object25378:head# 2024-05-15T23:40:14.440 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.441 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad4c446:::benchmark_data_smithi090_166897_object51417:head# 2024-05-15T23:40:14.441 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.441 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad5c6bb:::benchmark_data_smithi090_166897_object47050:head# 2024-05-15T23:40:14.441 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.441 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad66f46:::benchmark_data_smithi090_166897_object37067:head# 2024-05-15T23:40:14.441 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.441 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad6844d:::benchmark_data_smithi090_166897_object70651:head# 2024-05-15T23:40:14.441 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.441 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad72e05:::benchmark_data_smithi090_166897_object4751:head# 2024-05-15T23:40:14.441 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.441 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad95120:::benchmark_data_smithi090_166897_object5493:head# 2024-05-15T23:40:14.441 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.441 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad97d5d:::benchmark_data_smithi090_166897_object7721:head# 2024-05-15T23:40:14.441 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.442 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ad9804c:::benchmark_data_smithi090_166897_object62292:head# 2024-05-15T23:40:14.442 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.442 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ada11f7:::benchmark_data_smithi090_166897_object30580:head# 2024-05-15T23:40:14.442 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.442 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6adbc5ac:::benchmark_data_smithi090_166897_object2022:head# 2024-05-15T23:40:14.442 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.442 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6adca41d:::benchmark_data_smithi090_166897_object40548:head# 2024-05-15T23:40:14.442 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.442 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6adcefe1:::benchmark_data_smithi090_166897_object18571:head# 2024-05-15T23:40:14.442 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.442 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6adcfa4e:::benchmark_data_smithi090_166897_object28867:head# 2024-05-15T23:40:14.442 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.442 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6adcfd0f:::benchmark_data_smithi090_166897_object47048:head# 2024-05-15T23:40:14.442 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ae243e7:::benchmark_data_smithi090_166897_object62121:head# 2024-05-15T23:40:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ae34399:::benchmark_data_smithi090_166897_object64596:head# 2024-05-15T23:40:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ae3bf07:::benchmark_data_smithi090_166897_object33222:head# 2024-05-15T23:40:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ae4aecd:::benchmark_data_smithi090_166897_object20364:head# 2024-05-15T23:40:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ae53107:::benchmark_data_smithi090_166897_object63430:head# 2024-05-15T23:40:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ae55bcc:::benchmark_data_smithi090_166897_object65843:head# 2024-05-15T23:40:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ae647eb:::benchmark_data_smithi090_166897_object26473:head# 2024-05-15T23:40:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ae6a681:::benchmark_data_smithi090_166897_object515:head# 2024-05-15T23:40:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ae7224a:::benchmark_data_smithi090_166897_object33215:head# 2024-05-15T23:40:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ae7482c:::benchmark_data_smithi090_166897_object49097:head# 2024-05-15T23:40:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ae76b7e:::benchmark_data_smithi090_166897_object9079:head# 2024-05-15T23:40:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ae7bcc4:::benchmark_data_smithi090_166897_object4142:head# 2024-05-15T23:40:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ae7f9bc:::benchmark_data_smithi090_166897_object35943:head# 2024-05-15T23:40:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ae816d5:::benchmark_data_smithi090_166897_object69806:head# 2024-05-15T23:40:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ae9ccee:::benchmark_data_smithi090_166897_object12034:head# 2024-05-15T23:40:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.445 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aeae1a7:::benchmark_data_smithi090_166897_object17739:head# 2024-05-15T23:40:14.445 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.445 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aec1701:::benchmark_data_smithi090_166897_object25918:head# 2024-05-15T23:40:14.445 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.445 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aec3fde:::benchmark_data_smithi090_166897_object6086:head# 2024-05-15T23:40:14.445 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.445 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aed7374:::benchmark_data_smithi090_166897_object46563:head# 2024-05-15T23:40:14.445 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.445 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aee7a60:::benchmark_data_smithi090_166897_object51127:head# 2024-05-15T23:40:14.445 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.445 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6aef39d1:::benchmark_data_smithi090_166897_object61935:head# 2024-05-15T23:40:14.445 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.445 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6af0937a:::benchmark_data_smithi090_166897_object28577:head# 2024-05-15T23:40:14.445 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.446 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6af1d00d:::benchmark_data_smithi090_166897_object6771:head# 2024-05-15T23:40:14.446 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.446 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6af2a0cc:::benchmark_data_smithi090_166897_object38457:head# 2024-05-15T23:40:14.446 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.446 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6af2a706:::benchmark_data_smithi090_166897_object6402:head# 2024-05-15T23:40:14.446 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.446 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6af3d439:::benchmark_data_smithi090_166897_object6931:head# 2024-05-15T23:40:14.446 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.446 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6af73edb:::benchmark_data_smithi090_166897_object2191:head# 2024-05-15T23:40:14.446 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.446 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6af774e0:::benchmark_data_smithi090_166897_object41615:head# 2024-05-15T23:40:14.446 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.446 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6af8a22c:::benchmark_data_smithi090_166897_object10740:head# 2024-05-15T23:40:14.446 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6af98e8b:::benchmark_data_smithi090_166897_object31239:head# 2024-05-15T23:40:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6afd226c:::benchmark_data_smithi090_166897_object8485:head# 2024-05-15T23:40:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6afe828a:::benchmark_data_smithi090_166897_object26521:head# 2024-05-15T23:40:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6afef6a9:::benchmark_data_smithi090_166897_object22910:head# 2024-05-15T23:40:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b01fefe:::benchmark_data_smithi090_166897_object60316:head# 2024-05-15T23:40:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b029736:::benchmark_data_smithi090_166897_object21307:head# 2024-05-15T23:40:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b03aa1d:::benchmark_data_smithi090_166897_object16097:head# 2024-05-15T23:40:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b046a4a:::benchmark_data_smithi090_166897_object18235:head# 2024-05-15T23:40:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b0614fd:::benchmark_data_smithi090_166897_object12616:head# 2024-05-15T23:40:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b07037f:::benchmark_data_smithi090_166897_object59118:head# 2024-05-15T23:40:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b07639f:::benchmark_data_smithi090_166897_object6127:head# 2024-05-15T23:40:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b07cc74:::benchmark_data_smithi090_166897_object14858:head# 2024-05-15T23:40:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b08b62a:::benchmark_data_smithi090_166897_object4429:head# 2024-05-15T23:40:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b0a0113:::benchmark_data_smithi090_166897_object44559:head# 2024-05-15T23:40:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b0bf99f:::benchmark_data_smithi090_166897_object34788:head# 2024-05-15T23:40:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b0c1da2:::benchmark_data_smithi090_166897_object56893:head# 2024-05-15T23:40:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b0c20b4:::benchmark_data_smithi090_166897_object25176:head# 2024-05-15T23:40:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b0cd8a7:::benchmark_data_smithi090_166897_object56843:head# 2024-05-15T23:40:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b0e3685:::benchmark_data_smithi090_166897_object3046:head# 2024-05-15T23:40:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b0ed250:::benchmark_data_smithi090_166897_object27187:head# 2024-05-15T23:40:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b0f5654:::benchmark_data_smithi090_166897_object16828:head# 2024-05-15T23:40:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b103105:::benchmark_data_smithi090_166897_object42102:head# 2024-05-15T23:40:14.449 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.450 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b105e90:::benchmark_data_smithi090_166897_object65445:head# 2024-05-15T23:40:14.450 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.450 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b111d3d:::benchmark_data_smithi090_166897_object63513:head# 2024-05-15T23:40:14.450 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.450 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b131d83:::benchmark_data_smithi090_166897_object59925:head# 2024-05-15T23:40:14.450 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.450 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b13d5b0:::benchmark_data_smithi090_166897_object11232:head# 2024-05-15T23:40:14.450 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.450 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b15ef41:::benchmark_data_smithi090_166897_object15968:head# 2024-05-15T23:40:14.450 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.450 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b16633c:::benchmark_data_smithi090_166897_object23692:head# 2024-05-15T23:40:14.450 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.450 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b16fc60:::benchmark_data_smithi090_166897_object59401:head# 2024-05-15T23:40:14.450 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.451 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b19397a:::benchmark_data_smithi090_166897_object45642:head# 2024-05-15T23:40:14.451 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.451 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b19552b:::benchmark_data_smithi090_166897_object31919:head# 2024-05-15T23:40:14.451 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.451 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b199700:::benchmark_data_smithi090_166897_object63634:head# 2024-05-15T23:40:14.451 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.451 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b1b6ece:::benchmark_data_smithi090_166897_object55928:head# 2024-05-15T23:40:14.451 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.451 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b1bb102:::benchmark_data_smithi090_166897_object21639:head# 2024-05-15T23:40:14.451 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.451 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b1c81cb:::benchmark_data_smithi090_166897_object65631:head# 2024-05-15T23:40:14.451 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.451 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b1e7e33:::benchmark_data_smithi090_166897_object24677:head# 2024-05-15T23:40:14.451 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.451 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b20d703:::benchmark_data_smithi090_166897_object56120:head# 2024-05-15T23:40:14.452 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.452 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b219a19:::benchmark_data_smithi090_166897_object54882:head# 2024-05-15T23:40:14.452 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.452 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b220ce9:::benchmark_data_smithi090_166897_object39260:head# 2024-05-15T23:40:14.452 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.452 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b22cc32:::benchmark_data_smithi090_166897_object3527:head# 2024-05-15T23:40:14.452 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.452 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b22eae0:::benchmark_data_smithi090_166897_object8310:head# 2024-05-15T23:40:14.452 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.452 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b2438e1:::benchmark_data_smithi090_166897_object67815:head# 2024-05-15T23:40:14.452 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.452 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b25090d:::benchmark_data_smithi090_166897_object61145:head# 2024-05-15T23:40:14.452 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.452 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b265f59:::benchmark_data_smithi090_166897_object14433:head# 2024-05-15T23:40:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b278e2f:::benchmark_data_smithi090_166897_object7887:head# 2024-05-15T23:40:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b2907bb:::benchmark_data_smithi090_166897_object54504:head# 2024-05-15T23:40:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b294e9d:::benchmark_data_smithi090_166897_object47551:head# 2024-05-15T23:40:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b29f212:::benchmark_data_smithi090_166897_object12577:head# 2024-05-15T23:40:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b2b745b:::benchmark_data_smithi090_166897_object1332:head# 2024-05-15T23:40:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b2db13d:::benchmark_data_smithi090_166897_object16046:head# 2024-05-15T23:40:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b2e1c31:::benchmark_data_smithi090_166897_object44567:head# 2024-05-15T23:40:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.454 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b2eeaf9:::benchmark_data_smithi090_166897_object35005:head# 2024-05-15T23:40:14.454 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.454 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b3057a8:::benchmark_data_smithi090_166897_object64428:head# 2024-05-15T23:40:14.454 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.454 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b32095a:::benchmark_data_smithi090_166897_object3277:head# 2024-05-15T23:40:14.454 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.454 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b340cb9:::benchmark_data_smithi090_166897_object62650:head# 2024-05-15T23:40:14.454 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.454 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b343c28:::benchmark_data_smithi090_166897_object8823:head# 2024-05-15T23:40:14.454 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.454 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b34cce1:::benchmark_data_smithi090_166897_object19300:head# 2024-05-15T23:40:14.454 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.454 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b35c63d:::benchmark_data_smithi090_166897_object7182:head# 2024-05-15T23:40:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b384fd6:::benchmark_data_smithi090_166897_object70839:head# 2024-05-15T23:40:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b39994d:::benchmark_data_smithi090_166897_object12297:head# 2024-05-15T23:40:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b39db51:::benchmark_data_smithi090_166897_object62299:head# 2024-05-15T23:40:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b39fff2:::benchmark_data_smithi090_166897_object4620:head# 2024-05-15T23:40:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b3a997d:::benchmark_data_smithi090_166897_object17557:head# 2024-05-15T23:40:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b3ac0be:::benchmark_data_smithi090_166897_object28994:head# 2024-05-15T23:40:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b3b35dc:::benchmark_data_smithi090_166897_object19171:head# 2024-05-15T23:40:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b3bb27a:::benchmark_data_smithi090_166897_object5735:head# 2024-05-15T23:40:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b3bbcdd:::benchmark_data_smithi090_166897_object18265:head# 2024-05-15T23:40:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b3c240f:::benchmark_data_smithi090_166897_object42742:head# 2024-05-15T23:40:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b3ca721:::benchmark_data_smithi090_166897_object49791:head# 2024-05-15T23:40:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b3e277f:::benchmark_data_smithi090_166897_object66915:head# 2024-05-15T23:40:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b3f79b0:::benchmark_data_smithi090_166897_object59683:head# 2024-05-15T23:40:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b3f8fa7:::benchmark_data_smithi090_166897_object59243:head# 2024-05-15T23:40:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b3fc145:::benchmark_data_smithi090_166897_object2452:head# 2024-05-15T23:40:14.457 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.457 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b442037:::benchmark_data_smithi090_166897_object34557:head# 2024-05-15T23:40:14.457 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.457 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b442c97:::benchmark_data_smithi090_166897_object52830:head# 2024-05-15T23:40:14.457 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.457 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b446cb2:::benchmark_data_smithi090_166897_object44079:head# 2024-05-15T23:40:14.457 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.457 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b451450:::benchmark_data_smithi090_166897_object37959:head# 2024-05-15T23:40:14.457 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.457 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b45867c:::benchmark_data_smithi090_166897_object41947:head# 2024-05-15T23:40:14.457 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.457 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b45d825:::benchmark_data_smithi090_166897_object43222:head# 2024-05-15T23:40:14.457 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.457 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b45e310:::benchmark_data_smithi090_166897_object69394:head# 2024-05-15T23:40:14.458 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.458 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b468f25:::benchmark_data_smithi090_166897_object55199:head# 2024-05-15T23:40:14.458 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.458 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b469610:::benchmark_data_smithi090_166897_object61234:head# 2024-05-15T23:40:14.458 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.458 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b47f6c6:::benchmark_data_smithi090_166897_object2483:head# 2024-05-15T23:40:14.458 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.458 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b480ff4:::benchmark_data_smithi090_166897_object13582:head# 2024-05-15T23:40:14.458 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.458 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b48f3f7:::benchmark_data_smithi090_166897_object18537:head# 2024-05-15T23:40:14.458 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.458 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b49126e:::benchmark_data_smithi090_166897_object66981:head# 2024-05-15T23:40:14.458 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.458 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b492593:::benchmark_data_smithi090_166897_object31872:head# 2024-05-15T23:40:14.459 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.459 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b49c3ae:::benchmark_data_smithi090_166897_object18095:head# 2024-05-15T23:40:14.459 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.459 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b4b8205:::benchmark_data_smithi090_166897_object59609:head# 2024-05-15T23:40:14.459 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.459 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b4b9983:::benchmark_data_smithi090_166897_object55911:head# 2024-05-15T23:40:14.459 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.459 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b4e07f3:::benchmark_data_smithi090_166897_object56153:head# 2024-05-15T23:40:14.459 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.459 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b4ed069:::benchmark_data_smithi090_166897_object41648:head# 2024-05-15T23:40:14.460 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.460 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b4f5c82:::benchmark_data_smithi090_166897_object7018:head# 2024-05-15T23:40:14.460 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.460 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b509d1a:::benchmark_data_smithi090_166897_object137:head# 2024-05-15T23:40:14.460 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.460 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b52fa82:::benchmark_data_smithi090_166897_object37026:head# 2024-05-15T23:40:14.460 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.460 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b56359f:::benchmark_data_smithi090_166897_object14366:head# 2024-05-15T23:40:14.460 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.460 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b58998b:::benchmark_data_smithi090_166897_object27405:head# 2024-05-15T23:40:14.460 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.460 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b58fed9:::benchmark_data_smithi090_166897_object67403:head# 2024-05-15T23:40:14.460 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.460 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b59962f:::benchmark_data_smithi090_166897_object69273:head# 2024-05-15T23:40:14.460 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.461 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b5a2e3c:::benchmark_data_smithi090_166897_object20402:head# 2024-05-15T23:40:14.461 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.461 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b5a32a2:::benchmark_data_smithi090_166897_object33536:head# 2024-05-15T23:40:14.461 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.461 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b5a3766:::benchmark_data_smithi090_166897_object1932:head# 2024-05-15T23:40:14.461 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.461 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b5a68ff:::benchmark_data_smithi090_166897_object21870:head# 2024-05-15T23:40:14.461 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.461 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b5af82c:::benchmark_data_smithi090_166897_object42497:head# 2024-05-15T23:40:14.461 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.461 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b5d8703:::benchmark_data_smithi090_166897_object32282:head# 2024-05-15T23:40:14.461 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.461 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b5eb64d:::benchmark_data_smithi090_166897_object28035:head# 2024-05-15T23:40:14.461 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.461 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b5edf94:::benchmark_data_smithi090_166897_object64899:head# 2024-05-15T23:40:14.462 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.462 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b5ef480:::benchmark_data_smithi090_166897_object43340:head# 2024-05-15T23:40:14.462 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.462 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b5f9dbb:::benchmark_data_smithi090_166897_object64397:head# 2024-05-15T23:40:14.462 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.462 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b5fa81e:::benchmark_data_smithi090_166897_object49968:head# 2024-05-15T23:40:14.462 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.462 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b6164cf:::benchmark_data_smithi090_166897_object6570:head# 2024-05-15T23:40:14.462 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.462 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b622d99:::benchmark_data_smithi090_166897_object51190:head# 2024-05-15T23:40:14.462 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.462 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b63b04f:::benchmark_data_smithi090_166897_object8256:head# 2024-05-15T23:40:14.462 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.462 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b66cbb9:::benchmark_data_smithi090_166897_object50383:head# 2024-05-15T23:40:14.463 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.463 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b673d52:::benchmark_data_smithi090_166897_object62957:head# 2024-05-15T23:40:14.463 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.463 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b68665b:::benchmark_data_smithi090_166897_object39479:head# 2024-05-15T23:40:14.463 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.463 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b6a17d0:::benchmark_data_smithi090_166897_object70152:head# 2024-05-15T23:40:14.463 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.463 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b6aa6c1:::benchmark_data_smithi090_166897_object39811:head# 2024-05-15T23:40:14.463 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.463 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b6c30fb:::benchmark_data_smithi090_166897_object28069:head# 2024-05-15T23:40:14.463 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.463 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b6c8412:::benchmark_data_smithi090_166897_object19324:head# 2024-05-15T23:40:14.463 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.463 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b6cecf8:::benchmark_data_smithi090_166897_object17694:head# 2024-05-15T23:40:14.463 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.464 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b6cfe4a:::benchmark_data_smithi090_166897_object66911:head# 2024-05-15T23:40:14.464 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.464 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b6f77cc:::benchmark_data_smithi090_166897_object31404:head# 2024-05-15T23:40:14.464 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.464 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b6f9966:::benchmark_data_smithi090_166897_object9689:head# 2024-05-15T23:40:14.464 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.464 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b6fe241:::benchmark_data_smithi090_166897_object45361:head# 2024-05-15T23:40:14.464 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.464 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b707a80:::benchmark_data_smithi090_166897_object50502:head# 2024-05-15T23:40:14.464 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.464 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b713a84:::benchmark_data_smithi090_166897_object10578:head# 2024-05-15T23:40:14.464 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.464 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b731961:::benchmark_data_smithi090_166897_object53718:head# 2024-05-15T23:40:14.464 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.465 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b733385:::benchmark_data_smithi090_166897_object21924:head# 2024-05-15T23:40:14.465 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.465 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b7485b0:::benchmark_data_smithi090_166897_object58602:head# 2024-05-15T23:40:14.465 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.465 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b75f6e9:::benchmark_data_smithi090_166897_object43107:head# 2024-05-15T23:40:14.465 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.465 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b761944:::benchmark_data_smithi090_166897_object54487:head# 2024-05-15T23:40:14.465 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.465 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b768b9c:::benchmark_data_smithi090_166897_object51696:head# 2024-05-15T23:40:14.465 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.465 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b779bf3:::benchmark_data_smithi090_166897_object21321:head# 2024-05-15T23:40:14.465 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.465 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b7922a5:::benchmark_data_smithi090_166897_object21988:head# 2024-05-15T23:40:14.465 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.465 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b796a8b:::benchmark_data_smithi090_166897_object25326:head# 2024-05-15T23:40:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b7a245d:::benchmark_data_smithi090_166897_object55590:head# 2024-05-15T23:40:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b7b048e:::benchmark_data_smithi090_166897_object1646:head# 2024-05-15T23:40:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b7b67a9:::benchmark_data_smithi090_166897_object48519:head# 2024-05-15T23:40:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b7bb7ee:::benchmark_data_smithi090_166897_object47592:head# 2024-05-15T23:40:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b7bf040:::benchmark_data_smithi090_166897_object65826:head# 2024-05-15T23:40:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b7c7293:::benchmark_data_smithi090_166897_object31427:head# 2024-05-15T23:40:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b7f4c36:::benchmark_data_smithi090_166897_object44423:head# 2024-05-15T23:40:14.467 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.467 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b7fae26:::benchmark_data_smithi090_166897_object43876:head# 2024-05-15T23:40:14.467 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.467 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b7fbc45:::benchmark_data_smithi090_166897_object49386:head# 2024-05-15T23:40:14.467 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.467 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b816eae:::benchmark_data_smithi090_166897_object61110:head# 2024-05-15T23:40:14.467 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.467 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b829466:::benchmark_data_smithi090_166897_object49592:head# 2024-05-15T23:40:14.467 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.467 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b82b413:::benchmark_data_smithi090_166897_object34752:head# 2024-05-15T23:40:14.467 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.467 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b82c6a8:::benchmark_data_smithi090_166897_object4513:head# 2024-05-15T23:40:14.467 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.468 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b846375:::benchmark_data_smithi090_166897_object32555:head# 2024-05-15T23:40:14.468 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.468 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b84d347:::benchmark_data_smithi090_166897_object67773:head# 2024-05-15T23:40:14.468 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.468 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b84df9d:::benchmark_data_smithi090_166897_object70233:head# 2024-05-15T23:40:14.468 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.468 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b855ab3:::benchmark_data_smithi090_166897_object36686:head# 2024-05-15T23:40:14.468 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.468 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b869741:::benchmark_data_smithi090_166897_object47950:head# 2024-05-15T23:40:14.468 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.468 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b86bdcb:::benchmark_data_smithi090_166897_object27097:head# 2024-05-15T23:40:14.468 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.468 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b86f000:::benchmark_data_smithi090_166897_object27469:head# 2024-05-15T23:40:14.468 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.468 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b871907:::benchmark_data_smithi090_166897_object42575:head# 2024-05-15T23:40:14.469 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.469 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b8951de:::benchmark_data_smithi090_166897_object67026:head# 2024-05-15T23:40:14.469 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.469 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b897e7b:::benchmark_data_smithi090_166897_object41426:head# 2024-05-15T23:40:14.469 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.469 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b899603:::benchmark_data_smithi090_166897_object8501:head# 2024-05-15T23:40:14.469 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.469 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b89a2f1:::benchmark_data_smithi090_166897_object12280:head# 2024-05-15T23:40:14.469 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.469 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b8b19d8:::benchmark_data_smithi090_166897_object42422:head# 2024-05-15T23:40:14.469 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.469 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b8ce42c:::benchmark_data_smithi090_166897_object65611:head# 2024-05-15T23:40:14.469 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.470 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b8dca5b:::benchmark_data_smithi090_166897_object3367:head# 2024-05-15T23:40:14.470 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.470 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b8e29f3:::benchmark_data_smithi090_166897_object36149:head# 2024-05-15T23:40:14.470 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.470 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b8e453d:::benchmark_data_smithi090_166897_object25034:head# 2024-05-15T23:40:14.470 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.470 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b8ed158:::benchmark_data_smithi090_166897_object71087:head# 2024-05-15T23:40:14.470 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.470 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b8fbb03:::benchmark_data_smithi090_166897_object71053:head# 2024-05-15T23:40:14.470 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.470 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b9042f9:::benchmark_data_smithi090_166897_object21454:head# 2024-05-15T23:40:14.470 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.470 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b90e969:::benchmark_data_smithi090_166897_object67339:head# 2024-05-15T23:40:14.470 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.471 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b919182:::benchmark_data_smithi090_166897_object23130:head# 2024-05-15T23:40:14.471 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.471 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b91c901:::benchmark_data_smithi090_166897_object69428:head# 2024-05-15T23:40:14.471 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.471 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b924f96:::benchmark_data_smithi090_166897_object69510:head# 2024-05-15T23:40:14.471 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.471 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b93bc56:::benchmark_data_smithi090_166897_object70950:head# 2024-05-15T23:40:14.471 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.471 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b94360a:::benchmark_data_smithi090_166897_object66513:head# 2024-05-15T23:40:14.471 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.471 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b948631:::benchmark_data_smithi090_166897_object21329:head# 2024-05-15T23:40:14.471 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.471 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b94ad19:::benchmark_data_smithi090_166897_object40293:head# 2024-05-15T23:40:14.471 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.472 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b95bee3:::benchmark_data_smithi090_166897_object9857:head# 2024-05-15T23:40:14.472 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.472 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b962b26:::benchmark_data_smithi090_166897_object22802:head# 2024-05-15T23:40:14.472 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.472 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b970473:::benchmark_data_smithi090_166897_object56228:head# 2024-05-15T23:40:14.472 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.472 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b980070:::benchmark_data_smithi090_166897_object26740:head# 2024-05-15T23:40:14.472 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.472 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b982e93:::benchmark_data_smithi090_166897_object28195:head# 2024-05-15T23:40:14.472 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.472 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b98baf4:::benchmark_data_smithi090_166897_object58676:head# 2024-05-15T23:40:14.472 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.472 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b9ae555:::benchmark_data_smithi090_166897_object22681:head# 2024-05-15T23:40:14.472 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.473 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b9afc33:::benchmark_data_smithi090_166897_object48463:head# 2024-05-15T23:40:14.473 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.473 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b9b7adb:::benchmark_data_smithi090_166897_object51664:head# 2024-05-15T23:40:14.473 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.473 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b9b8c0a:::benchmark_data_smithi090_166897_object52869:head# 2024-05-15T23:40:14.473 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.473 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b9bd3da:::benchmark_data_smithi090_166897_object58773:head# 2024-05-15T23:40:14.473 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.473 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b9cd0c7:::benchmark_data_smithi090_166897_object18209:head# 2024-05-15T23:40:14.473 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.473 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b9cfc73:::benchmark_data_smithi090_166897_object45474:head# 2024-05-15T23:40:14.473 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.473 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6b9ed43a:::benchmark_data_smithi090_166897_object57853:head# 2024-05-15T23:40:14.473 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.474 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ba0046b:::benchmark_data_smithi090_166897_object4666:head# 2024-05-15T23:40:14.474 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.474 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ba1410f:::benchmark_data_smithi090_166897_object71054:head# 2024-05-15T23:40:14.474 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.474 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ba2a53e:::benchmark_data_smithi090_166897_object3761:head# 2024-05-15T23:40:14.474 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.474 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ba35f44:::benchmark_data_smithi090_166897_object745:head# 2024-05-15T23:40:14.474 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.474 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ba3611d:::benchmark_data_smithi090_166897_object13311:head# 2024-05-15T23:40:14.474 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.474 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ba3e405:::benchmark_data_smithi090_166897_object48046:head# 2024-05-15T23:40:14.474 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.474 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ba4353d:::benchmark_data_smithi090_166897_object61974:head# 2024-05-15T23:40:14.474 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.475 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ba4cc11:::benchmark_data_smithi090_166897_object6388:head# 2024-05-15T23:40:14.475 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.475 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ba513b0:::benchmark_data_smithi090_166897_object59959:head# 2024-05-15T23:40:14.475 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.475 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ba65c4e:::benchmark_data_smithi090_166897_object44175:head# 2024-05-15T23:40:14.475 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.475 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ba66a99:::benchmark_data_smithi090_166897_object19554:head# 2024-05-15T23:40:14.475 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.475 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ba7b70c:::benchmark_data_smithi090_166897_object40840:head# 2024-05-15T23:40:14.475 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.475 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ba8d22e:::benchmark_data_smithi090_166897_object70413:head# 2024-05-15T23:40:14.475 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.475 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ba99e68:::benchmark_data_smithi090_166897_object58979:head# 2024-05-15T23:40:14.475 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.475 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ba99fff:::benchmark_data_smithi090_166897_object62989:head# 2024-05-15T23:40:14.476 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.476 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6baa1598:::benchmark_data_smithi090_166897_object29733:head# 2024-05-15T23:40:14.476 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.476 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6baa90f1:::benchmark_data_smithi090_166897_object68649:head# 2024-05-15T23:40:14.476 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.476 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6baacc6e:::benchmark_data_smithi090_166897_object66942:head# 2024-05-15T23:40:14.476 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.476 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bab8309:::benchmark_data_smithi090_166897_object53254:head# 2024-05-15T23:40:14.476 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.476 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bae30b0:::benchmark_data_smithi090_166897_object57269:head# 2024-05-15T23:40:14.476 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.476 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6baf1f3d:::benchmark_data_smithi090_166897_object31179:head# 2024-05-15T23:40:14.476 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.476 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bb0fda1:::benchmark_data_smithi090_166897_object41640:head# 2024-05-15T23:40:14.476 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.477 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bb1a15d:::benchmark_data_smithi090_166897_object32144:head# 2024-05-15T23:40:14.477 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.477 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bb386c7:::benchmark_data_smithi090_166897_object42749:head# 2024-05-15T23:40:14.477 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.477 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bb413ee:::benchmark_data_smithi090_166897_object57470:head# 2024-05-15T23:40:14.477 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.477 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bb524d4:::benchmark_data_smithi090_166897_object46834:head# 2024-05-15T23:40:14.477 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.477 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bb5d1d8:::benchmark_data_smithi090_166897_object42028:head# 2024-05-15T23:40:14.477 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.477 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bb647fc:::benchmark_data_smithi090_166897_object3052:head# 2024-05-15T23:40:14.477 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.477 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bb7a7da:::benchmark_data_smithi090_166897_object62268:head# 2024-05-15T23:40:14.477 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.478 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bb7df8c:::benchmark_data_smithi090_166897_object4851:head# 2024-05-15T23:40:14.478 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.478 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bb94b77:::benchmark_data_smithi090_166897_object21872:head# 2024-05-15T23:40:14.478 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.478 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bb94ccf:::benchmark_data_smithi090_166897_object35721:head# 2024-05-15T23:40:14.478 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.478 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bba465a:::benchmark_data_smithi090_166897_object32794:head# 2024-05-15T23:40:14.478 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.478 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bbac679:::benchmark_data_smithi090_166897_object42270:head# 2024-05-15T23:40:14.478 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.478 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bbb0483:::benchmark_data_smithi090_166897_object55886:head# 2024-05-15T23:40:14.478 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.478 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bbc0107:::benchmark_data_smithi090_166897_object70590:head# 2024-05-15T23:40:14.478 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.478 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bbd5016:::benchmark_data_smithi090_166897_object40852:head# 2024-05-15T23:40:14.479 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.479 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bbe2f2b:::benchmark_data_smithi090_166897_object5518:head# 2024-05-15T23:40:14.479 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.479 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bbec0b3:::benchmark_data_smithi090_166897_object25023:head# 2024-05-15T23:40:14.479 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.479 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bc1192a:::benchmark_data_smithi090_166897_object40853:head# 2024-05-15T23:40:14.479 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.479 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bc20fe0:::benchmark_data_smithi090_166897_object48313:head# 2024-05-15T23:40:14.479 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.479 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bc2a19f:::benchmark_data_smithi090_166897_object405:head# 2024-05-15T23:40:14.479 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.479 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bc4c3cc:::benchmark_data_smithi090_166897_object16187:head# 2024-05-15T23:40:14.479 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.479 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bc62ef9:::benchmark_data_smithi090_166897_object64919:head# 2024-05-15T23:40:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bc83354:::benchmark_data_smithi090_166897_object3176:head# 2024-05-15T23:40:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bc8d67e:::benchmark_data_smithi090_166897_object61525:head# 2024-05-15T23:40:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bc9c103:::benchmark_data_smithi090_166897_object51197:head# 2024-05-15T23:40:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bc9d341:::benchmark_data_smithi090_166897_object13630:head# 2024-05-15T23:40:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bca9e8b:::benchmark_data_smithi090_166897_object9143:head# 2024-05-15T23:40:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bcbedb1:::benchmark_data_smithi090_166897_object7868:head# 2024-05-15T23:40:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bccd4c2:::benchmark_data_smithi090_166897_object32564:head# 2024-05-15T23:40:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bcdb99e:::benchmark_data_smithi090_166897_object19326:head# 2024-05-15T23:40:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bce29e6:::benchmark_data_smithi090_166897_object11615:head# 2024-05-15T23:40:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bce5fc4:::benchmark_data_smithi090_166897_object6737:head# 2024-05-15T23:40:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bce79f3:::benchmark_data_smithi090_166897_object16527:head# 2024-05-15T23:40:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bcfc83a:::benchmark_data_smithi090_166897_object47487:head# 2024-05-15T23:40:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bcff19c:::benchmark_data_smithi090_166897_object34716:head# 2024-05-15T23:40:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bd01539:::benchmark_data_smithi090_166897_object21637:head# 2024-05-15T23:40:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bd10921:::benchmark_data_smithi090_166897_object48206:head# 2024-05-15T23:40:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bd2f8fe:::benchmark_data_smithi090_166897_object7268:head# 2024-05-15T23:40:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bd4d792:::benchmark_data_smithi090_166897_object54182:head# 2024-05-15T23:40:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bd66b22:::benchmark_data_smithi090_166897_object65173:head# 2024-05-15T23:40:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bd76758:::benchmark_data_smithi090_166897_object15588:head# 2024-05-15T23:40:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bd99503:::benchmark_data_smithi090_166897_object26680:head# 2024-05-15T23:40:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bda2c58:::benchmark_data_smithi090_166897_object19209:head# 2024-05-15T23:40:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.483 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bda3fd6:::benchmark_data_smithi090_166897_object38995:head# 2024-05-15T23:40:14.483 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.483 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bdafc07:::benchmark_data_smithi090_166897_object28644:head# 2024-05-15T23:40:14.483 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.483 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bdd5b19:::benchmark_data_smithi090_166897_object15914:head# 2024-05-15T23:40:14.483 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.483 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bdde5c3:::benchmark_data_smithi090_166897_object16425:head# 2024-05-15T23:40:14.483 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.483 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6be02cb9:::benchmark_data_smithi090_166897_object1857:head# 2024-05-15T23:40:14.483 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.483 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6be065b4:::benchmark_data_smithi090_166897_object62696:head# 2024-05-15T23:40:14.483 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.483 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6be20e1f:::benchmark_data_smithi090_166897_object8995:head# 2024-05-15T23:40:14.483 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.484 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6be3a2be:::benchmark_data_smithi090_166897_object19284:head# 2024-05-15T23:40:14.484 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.484 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6be48854:::benchmark_data_smithi090_166897_object8819:head# 2024-05-15T23:40:14.484 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.484 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6be6a529:::benchmark_data_smithi090_166897_object66737:head# 2024-05-15T23:40:14.484 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.484 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6be6f2b2:::benchmark_data_smithi090_166897_object45123:head# 2024-05-15T23:40:14.484 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.484 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6be7272c:::benchmark_data_smithi090_166897_object53875:head# 2024-05-15T23:40:14.484 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.484 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6be7e4fd:::benchmark_data_smithi090_166897_object63766:head# 2024-05-15T23:40:14.484 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.484 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6be89202:::benchmark_data_smithi090_166897_object32472:head# 2024-05-15T23:40:14.484 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.485 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6be9fd6b:::benchmark_data_smithi090_166897_object36561:head# 2024-05-15T23:40:14.485 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.485 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6beab518:::benchmark_data_smithi090_166897_object68879:head# 2024-05-15T23:40:14.485 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.485 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bebe3f9:::benchmark_data_smithi090_166897_object15539:head# 2024-05-15T23:40:14.485 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.485 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bedb1fd:::benchmark_data_smithi090_166897_object24760:head# 2024-05-15T23:40:14.485 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.485 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bee48b0:::benchmark_data_smithi090_166897_object5628:head# 2024-05-15T23:40:14.485 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.485 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bee7f01:::benchmark_data_smithi090_166897_object12366:head# 2024-05-15T23:40:14.485 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.485 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bee8a78:::benchmark_data_smithi090_166897_object8830:head# 2024-05-15T23:40:14.485 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:14.487 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:40:14.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:40:14.577 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 72 16 105421 105405 91.47 58.125 0.00401735 0.0109306 2024-05-15T23:40:14.590 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:40:14.600 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bee9d3e:::benchmark_data_smithi090_166897_object36808:head# 2024-05-15T23:40:14.600 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.600 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6beee1fe:::benchmark_data_smithi090_166897_object57650:head# 2024-05-15T23:40:14.600 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.600 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6beef4b6:::benchmark_data_smithi090_166897_object13352:head# 2024-05-15T23:40:14.600 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.600 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bef7f42:::benchmark_data_smithi090_166897_object62659:head# 2024-05-15T23:40:14.600 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.600 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bef9c61:::benchmark_data_smithi090_166897_object54817:head# 2024-05-15T23:40:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6befa9ae:::benchmark_data_smithi090_166897_object62938:head# 2024-05-15T23:40:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bf09108:::benchmark_data_smithi090_166897_object45727:head# 2024-05-15T23:40:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bf18151:::benchmark_data_smithi090_166897_object68424:head# 2024-05-15T23:40:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bf20d43:::benchmark_data_smithi090_166897_object4951:head# 2024-05-15T23:40:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bf26164:::benchmark_data_smithi090_166897_object27403:head# 2024-05-15T23:40:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bf3ec85:::benchmark_data_smithi090_166897_object11389:head# 2024-05-15T23:40:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bf44ec4:::benchmark_data_smithi090_166897_object30847:head# 2024-05-15T23:40:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.602 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bf50829:::benchmark_data_smithi090_166897_object18525:head# 2024-05-15T23:40:14.602 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.602 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bf63c47:::benchmark_data_smithi090_166897_object44325:head# 2024-05-15T23:40:14.602 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.602 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bf65aa9:::benchmark_data_smithi090_166897_object68978:head# 2024-05-15T23:40:14.602 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.602 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bf6746f:::benchmark_data_smithi090_166897_object3518:head# 2024-05-15T23:40:14.602 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.602 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bf844ac:::benchmark_data_smithi090_166897_object19083:head# 2024-05-15T23:40:14.602 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.602 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bf8a867:::benchmark_data_smithi090_166897_object57179:head# 2024-05-15T23:40:14.602 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.602 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bf8b6df:::benchmark_data_smithi090_166897_object61184:head# 2024-05-15T23:40:14.602 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.603 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bfa23b1:::benchmark_data_smithi090_166897_object67502:head# 2024-05-15T23:40:14.603 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.603 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bfbec64:::benchmark_data_smithi090_166897_object9351:head# 2024-05-15T23:40:14.603 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.603 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bfd22a6:::benchmark_data_smithi090_166897_object64435:head# 2024-05-15T23:40:14.603 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.603 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6bfedb04:::benchmark_data_smithi090_166897_object42918:head# 2024-05-15T23:40:14.603 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.603 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c002d30:::benchmark_data_smithi090_166897_object36315:head# 2024-05-15T23:40:14.603 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.603 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c0180b3:::benchmark_data_smithi090_166897_object5098:head# 2024-05-15T23:40:14.603 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.603 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c01a986:::benchmark_data_smithi090_166897_object62917:head# 2024-05-15T23:40:14.603 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.603 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c0381c1:::benchmark_data_smithi090_166897_object50001:head# 2024-05-15T23:40:14.604 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.604 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c04d63e:::benchmark_data_smithi090_166897_object42173:head# 2024-05-15T23:40:14.604 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.604 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c04f15b:::benchmark_data_smithi090_166897_object27721:head# 2024-05-15T23:40:14.604 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.604 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c0523b3:::benchmark_data_smithi090_166897_object63891:head# 2024-05-15T23:40:14.604 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.604 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c0574e9:::benchmark_data_smithi090_166897_object9723:head# 2024-05-15T23:40:14.604 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.604 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c05cdf3:::benchmark_data_smithi090_166897_object33838:head# 2024-05-15T23:40:14.604 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.604 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c05fa1f:::benchmark_data_smithi090_166897_object49886:head# 2024-05-15T23:40:14.604 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.604 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c070ef2:::benchmark_data_smithi090_166897_object24898:head# 2024-05-15T23:40:14.604 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.605 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c07d5aa:::benchmark_data_smithi090_166897_object4557:head# 2024-05-15T23:40:14.605 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.605 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c091db9:::benchmark_data_smithi090_166897_object27915:head# 2024-05-15T23:40:14.605 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.605 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c09bc89:::benchmark_data_smithi090_166897_object52685:head# 2024-05-15T23:40:14.605 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.605 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c0a5d93:::benchmark_data_smithi090_166897_object8464:head# 2024-05-15T23:40:14.605 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.605 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c0d08f8:::benchmark_data_smithi090_166897_object9738:head# 2024-05-15T23:40:14.605 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.605 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c0d56af:::benchmark_data_smithi090_166897_object43799:head# 2024-05-15T23:40:14.605 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.605 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c0f4549:::benchmark_data_smithi090_166897_object2914:head# 2024-05-15T23:40:14.605 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.605 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c1115c8:::benchmark_data_smithi090_166897_object30106:head# 2024-05-15T23:40:14.606 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.606 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c121749:::benchmark_data_smithi090_166897_object14104:head# 2024-05-15T23:40:14.606 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.606 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c1333be:::benchmark_data_smithi090_166897_object39440:head# 2024-05-15T23:40:14.606 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.606 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c147bfd:::benchmark_data_smithi090_166897_object33386:head# 2024-05-15T23:40:14.606 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.606 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c14fb61:::benchmark_data_smithi090_166897_object17828:head# 2024-05-15T23:40:14.606 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.606 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c159ee6:::benchmark_data_smithi090_166897_object68795:head# 2024-05-15T23:40:14.606 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.606 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c163f79:::benchmark_data_smithi090_166897_object8636:head# 2024-05-15T23:40:14.606 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.607 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c16b12b:::benchmark_data_smithi090_166897_object29106:head# 2024-05-15T23:40:14.607 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.607 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c1796b6:::benchmark_data_smithi090_166897_object18497:head# 2024-05-15T23:40:14.607 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.607 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c1aecf8:::benchmark_data_smithi090_166897_object68129:head# 2024-05-15T23:40:14.607 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.607 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c1b9823:::benchmark_data_smithi090_166897_object5443:head# 2024-05-15T23:40:14.607 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.607 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c1d7be5:::benchmark_data_smithi090_166897_object54913:head# 2024-05-15T23:40:14.607 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.607 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c1e1a91:::benchmark_data_smithi090_166897_object31329:head# 2024-05-15T23:40:14.607 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.607 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c1e2010:::benchmark_data_smithi090_166897_object63783:head# 2024-05-15T23:40:14.607 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.607 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c1ea7b8:::benchmark_data_smithi090_166897_object57252:head# 2024-05-15T23:40:14.608 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.608 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c1ff129:::benchmark_data_smithi090_166897_object20775:head# 2024-05-15T23:40:14.608 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.608 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c200b7e:::benchmark_data_smithi090_166897_object12986:head# 2024-05-15T23:40:14.608 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.608 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c20f156:::benchmark_data_smithi090_166897_object27855:head# 2024-05-15T23:40:14.608 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.608 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c210f4f:::benchmark_data_smithi090_166897_object6421:head# 2024-05-15T23:40:14.608 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.608 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c2254cd:::benchmark_data_smithi090_166897_object31834:head# 2024-05-15T23:40:14.608 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.608 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c227b35:::benchmark_data_smithi090_166897_object56996:head# 2024-05-15T23:40:14.608 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.608 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c228bd6:::benchmark_data_smithi090_166897_object30911:head# 2024-05-15T23:40:14.609 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.609 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c237da5:::benchmark_data_smithi090_166897_object15494:head# 2024-05-15T23:40:14.609 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.609 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c23bdb3:::benchmark_data_smithi090_166897_object35907:head# 2024-05-15T23:40:14.609 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.609 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c250d27:::benchmark_data_smithi090_166897_object47093:head# 2024-05-15T23:40:14.609 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.609 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c2606bb:::benchmark_data_smithi090_166897_object52945:head# 2024-05-15T23:40:14.609 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.609 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c26394d:::benchmark_data_smithi090_166897_object7754:head# 2024-05-15T23:40:14.609 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.609 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c26d958:::benchmark_data_smithi090_166897_object64140:head# 2024-05-15T23:40:14.609 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.609 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c27f61e:::benchmark_data_smithi090_166897_object61907:head# 2024-05-15T23:40:14.609 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.610 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c288fc3:::benchmark_data_smithi090_166897_object62115:head# 2024-05-15T23:40:14.610 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.610 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c2a2891:::benchmark_data_smithi090_166897_object51796:head# 2024-05-15T23:40:14.610 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.610 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c2ab45e:::benchmark_data_smithi090_166897_object20128:head# 2024-05-15T23:40:14.610 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.610 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c2bc407:::benchmark_data_smithi090_166897_object60448:head# 2024-05-15T23:40:14.610 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.610 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c2bf354:::benchmark_data_smithi090_166897_object1747:head# 2024-05-15T23:40:14.610 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.610 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c2ca06f:::benchmark_data_smithi090_166897_object40591:head# 2024-05-15T23:40:14.610 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.610 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c2cdd9a:::benchmark_data_smithi090_166897_object17863:head# 2024-05-15T23:40:14.610 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.611 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c2d09ea:::benchmark_data_smithi090_166897_object41352:head# 2024-05-15T23:40:14.611 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.611 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c2e17cc:::benchmark_data_smithi090_166897_object62405:head# 2024-05-15T23:40:14.611 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.611 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c2e4146:::benchmark_data_smithi090_166897_object40732:head# 2024-05-15T23:40:14.611 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.611 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c2e6373:::benchmark_data_smithi090_166897_object4564:head# 2024-05-15T23:40:14.611 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.611 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c2f021f:::benchmark_data_smithi090_166897_object9207:head# 2024-05-15T23:40:14.611 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.611 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c303b83:::benchmark_data_smithi090_166897_object32107:head# 2024-05-15T23:40:14.611 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.611 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c319280:::benchmark_data_smithi090_166897_object70046:head# 2024-05-15T23:40:14.611 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.611 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c31cd32:::benchmark_data_smithi090_166897_object41142:head# 2024-05-15T23:40:14.612 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.612 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c33dfc3:::benchmark_data_smithi090_166897_object39646:head# 2024-05-15T23:40:14.612 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.612 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c350fea:::benchmark_data_smithi090_166897_object40487:head# 2024-05-15T23:40:14.612 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.612 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c379365:::benchmark_data_smithi090_166897_object67901:head# 2024-05-15T23:40:14.612 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.612 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c38ecd1:::benchmark_data_smithi090_166897_object58130:head# 2024-05-15T23:40:14.612 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.612 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c3c3cc2:::benchmark_data_smithi090_166897_object68031:head# 2024-05-15T23:40:14.612 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.612 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c3e2c18:::benchmark_data_smithi090_166897_object36745:head# 2024-05-15T23:40:14.612 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.612 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c3e45e9:::benchmark_data_smithi090_166897_object20268:head# 2024-05-15T23:40:14.612 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.613 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c3f68dc:::benchmark_data_smithi090_166897_object7259:head# 2024-05-15T23:40:14.613 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.613 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c435f3d:::benchmark_data_smithi090_166897_object35967:head# 2024-05-15T23:40:14.613 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.613 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c44c68c:::benchmark_data_smithi090_166897_object56319:head# 2024-05-15T23:40:14.613 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.613 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c454b40:::benchmark_data_smithi090_166897_object61838:head# 2024-05-15T23:40:14.613 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.613 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c476952:::benchmark_data_smithi090_166897_object38898:head# 2024-05-15T23:40:14.613 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.613 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c47ea43:::benchmark_data_smithi090_166897_object43374:head# 2024-05-15T23:40:14.613 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.613 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c49dd58:::benchmark_data_smithi090_166897_object18379:head# 2024-05-15T23:40:14.613 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.614 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c4ae6e0:::benchmark_data_smithi090_166897_object56389:head# 2024-05-15T23:40:14.614 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.614 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c4c6261:::benchmark_data_smithi090_166897_object51858:head# 2024-05-15T23:40:14.614 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.614 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c4e5a08:::benchmark_data_smithi090_166897_object20924:head# 2024-05-15T23:40:14.614 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.614 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c4e7dc7:::benchmark_data_smithi090_166897_object56317:head# 2024-05-15T23:40:14.614 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.614 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c4f7784:::benchmark_data_smithi090_166897_object22760:head# 2024-05-15T23:40:14.614 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.614 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c50950e:::benchmark_data_smithi090_166897_object35898:head# 2024-05-15T23:40:14.614 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.614 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c51b09b:::benchmark_data_smithi090_166897_object42345:head# 2024-05-15T23:40:14.614 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c51d171:::benchmark_data_smithi090_166897_object66092:head# 2024-05-15T23:40:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c52d4fd:::benchmark_data_smithi090_166897_object22516:head# 2024-05-15T23:40:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c549c36:::benchmark_data_smithi090_166897_object51017:head# 2024-05-15T23:40:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c54a3ce:::benchmark_data_smithi090_166897_object57348:head# 2024-05-15T23:40:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c550e0f:::benchmark_data_smithi090_166897_object61416:head# 2024-05-15T23:40:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c553475:::benchmark_data_smithi090_166897_object54611:head# 2024-05-15T23:40:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c560091:::benchmark_data_smithi090_166897_object39063:head# 2024-05-15T23:40:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.615 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c576f32:::benchmark_data_smithi090_166897_object56926:head# 2024-05-15T23:40:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c5853b8:::benchmark_data_smithi090_166897_object48030:head# 2024-05-15T23:40:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c588d68:::benchmark_data_smithi090_166897_object31805:head# 2024-05-15T23:40:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c5a60ef:::benchmark_data_smithi090_166897_object5310:head# 2024-05-15T23:40:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c5af647:::benchmark_data_smithi090_166897_object49600:head# 2024-05-15T23:40:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c5ba499:::benchmark_data_smithi090_166897_object8458:head# 2024-05-15T23:40:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c5ce36d:::benchmark_data_smithi090_166897_object66479:head# 2024-05-15T23:40:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c5e39b7:::benchmark_data_smithi090_166897_object29435:head# 2024-05-15T23:40:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c5f84ce:::benchmark_data_smithi090_166897_object42646:head# 2024-05-15T23:40:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c6301ca:::benchmark_data_smithi090_166897_object2208:head# 2024-05-15T23:40:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c636577:::benchmark_data_smithi090_166897_object409:head# 2024-05-15T23:40:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c63a318:::benchmark_data_smithi090_166897_object52657:head# 2024-05-15T23:40:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c64ca1b:::benchmark_data_smithi090_166897_object44624:head# 2024-05-15T23:40:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c668344:::benchmark_data_smithi090_166897_object28936:head# 2024-05-15T23:40:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c66c18e:::benchmark_data_smithi090_166897_object6404:head# 2024-05-15T23:40:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c66e853:::benchmark_data_smithi090_166897_object60825:head# 2024-05-15T23:40:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c67941e:::benchmark_data_smithi090_166897_object18051:head# 2024-05-15T23:40:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c69858e:::benchmark_data_smithi090_166897_object4397:head# 2024-05-15T23:40:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c6ac09f:::benchmark_data_smithi090_166897_object26008:head# 2024-05-15T23:40:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c6caa38:::benchmark_data_smithi090_166897_object56916:head# 2024-05-15T23:40:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c6d93c4:::benchmark_data_smithi090_166897_object13122:head# 2024-05-15T23:40:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c707297:::benchmark_data_smithi090_166897_object46839:head# 2024-05-15T23:40:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c7085db:::benchmark_data_smithi090_166897_object64389:head# 2024-05-15T23:40:14.619 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.619 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c71c14a:::benchmark_data_smithi090_166897_object62289:head# 2024-05-15T23:40:14.619 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.619 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c72aecc:::benchmark_data_smithi090_166897_object54771:head# 2024-05-15T23:40:14.619 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.619 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c730878:::benchmark_data_smithi090_166897_object242:head# 2024-05-15T23:40:14.619 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.619 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c73dc9c:::benchmark_data_smithi090_166897_object51912:head# 2024-05-15T23:40:14.619 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.619 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c757c2b:::benchmark_data_smithi090_166897_object66411:head# 2024-05-15T23:40:14.619 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.619 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c783efc:::benchmark_data_smithi090_166897_object17822:head# 2024-05-15T23:40:14.619 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.619 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c78a596:::benchmark_data_smithi090_166897_object33728:head# 2024-05-15T23:40:14.619 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.620 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c79712a:::benchmark_data_smithi090_166897_object60475:head# 2024-05-15T23:40:14.620 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.620 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c79ee60:::benchmark_data_smithi090_166897_object45118:head# 2024-05-15T23:40:14.620 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.620 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c7a6316:::benchmark_data_smithi090_166897_object66746:head# 2024-05-15T23:40:14.620 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.620 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c7bbaf6:::benchmark_data_smithi090_166897_object68839:head# 2024-05-15T23:40:14.620 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.620 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c7d43e5:::benchmark_data_smithi090_166897_object522:head# 2024-05-15T23:40:14.620 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.620 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c7f39e1:::benchmark_data_smithi090_166897_object48455:head# 2024-05-15T23:40:14.620 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.620 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c8013e7:::benchmark_data_smithi090_166897_object40483:head# 2024-05-15T23:40:14.620 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.620 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c80c02a:::benchmark_data_smithi090_166897_object30010:head# 2024-05-15T23:40:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c810ca3:::benchmark_data_smithi090_166897_object14637:head# 2024-05-15T23:40:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c833b00:::benchmark_data_smithi090_166897_object57433:head# 2024-05-15T23:40:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c85a416:::benchmark_data_smithi090_166897_object61795:head# 2024-05-15T23:40:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c85bf2a:::benchmark_data_smithi090_166897_object21855:head# 2024-05-15T23:40:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c85ca7c:::benchmark_data_smithi090_166897_object38312:head# 2024-05-15T23:40:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c861328:::benchmark_data_smithi090_166897_object70588:head# 2024-05-15T23:40:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.621 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c871645:::benchmark_data_smithi090_166897_object668:head# 2024-05-15T23:40:14.622 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.622 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c87b990:::benchmark_data_smithi090_166897_object48819:head# 2024-05-15T23:40:14.622 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.622 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c8d2dde:::benchmark_data_smithi090_166897_object29093:head# 2024-05-15T23:40:14.622 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.622 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c8d97ef:::benchmark_data_smithi090_166897_object278:head# 2024-05-15T23:40:14.622 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.622 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c8dace5:::benchmark_data_smithi090_166897_object61103:head# 2024-05-15T23:40:14.622 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.622 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c8dbdc6:::benchmark_data_smithi090_166897_object3653:head# 2024-05-15T23:40:14.622 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.622 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c90644d:::benchmark_data_smithi090_166897_object4539:head# 2024-05-15T23:40:14.622 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c908019:::benchmark_data_smithi090_166897_object43070:head# 2024-05-15T23:40:14.623 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c90a283:::benchmark_data_smithi090_166897_object23644:head# 2024-05-15T23:40:14.623 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c914058:::benchmark_data_smithi090_166897_object66164:head# 2024-05-15T23:40:14.623 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c95b267:::benchmark_data_smithi090_166897_object46385:head# 2024-05-15T23:40:14.623 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c96155d:::benchmark_data_smithi090_166897_object65997:head# 2024-05-15T23:40:14.623 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c96f42e:::benchmark_data_smithi090_166897_object13427:head# 2024-05-15T23:40:14.623 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c96fadb:::benchmark_data_smithi090_166897_object39892:head# 2024-05-15T23:40:14.623 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.623 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c9704d3:::benchmark_data_smithi090_166897_object14398:head# 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c97204c:::benchmark_data_smithi090_166897_object26051:head# 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c9764a5:::benchmark_data_smithi090_166897_object67940:head# 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c97dbfc:::benchmark_data_smithi090_166897_object64437:head# 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c99b6e9:::benchmark_data_smithi090_166897_object52563:head# 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c99f239:::benchmark_data_smithi090_166897_object51098:head# 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c9a8156:::benchmark_data_smithi090_166897_object70100:head# 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c9b2ed2:::benchmark_data_smithi090_166897_object40930:head# 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.624 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c9cf02a:::benchmark_data_smithi090_166897_object35833:head# 2024-05-15T23:40:14.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.625 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c9d1d40:::benchmark_data_smithi090_166897_object36738:head# 2024-05-15T23:40:14.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.625 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c9dafb1:::benchmark_data_smithi090_166897_object41781:head# 2024-05-15T23:40:14.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.625 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6c9ec1bc:::benchmark_data_smithi090_166897_object25277:head# 2024-05-15T23:40:14.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.625 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ca0c7ff:::benchmark_data_smithi090_166897_object70232:head# 2024-05-15T23:40:14.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.625 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ca33fef:::benchmark_data_smithi090_166897_object64386:head# 2024-05-15T23:40:14.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.625 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ca3c2b9:::benchmark_data_smithi090_166897_object3086:head# 2024-05-15T23:40:14.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.625 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ca3cba5:::benchmark_data_smithi090_166897_object52245:head# 2024-05-15T23:40:14.625 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.626 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ca5a24e:::benchmark_data_smithi090_166897_object62393:head# 2024-05-15T23:40:14.626 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.626 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ca69000:::benchmark_data_smithi090_166897_object4454:head# 2024-05-15T23:40:14.626 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.626 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ca6e748:::benchmark_data_smithi090_166897_object19490:head# 2024-05-15T23:40:14.626 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.626 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ca7360b:::benchmark_data_smithi090_166897_object50002:head# 2024-05-15T23:40:14.626 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.626 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ca77c62:::benchmark_data_smithi090_166897_object43836:head# 2024-05-15T23:40:14.626 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.626 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ca7c319:::benchmark_data_smithi090_166897_object48653:head# 2024-05-15T23:40:14.626 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.626 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ca86729:::benchmark_data_smithi090_166897_object7124:head# 2024-05-15T23:40:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ca87ce1:::benchmark_data_smithi090_166897_object19613:head# 2024-05-15T23:40:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ca89519:::benchmark_data_smithi090_166897_object32519:head# 2024-05-15T23:40:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ca9fd86:::benchmark_data_smithi090_166897_object46340:head# 2024-05-15T23:40:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cab1b96:::benchmark_data_smithi090_166897_object12065:head# 2024-05-15T23:40:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cab216a:::benchmark_data_smithi090_166897_object33379:head# 2024-05-15T23:40:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cabd1c2:::benchmark_data_smithi090_166897_object12023:head# 2024-05-15T23:40:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cacb2ac:::benchmark_data_smithi090_166897_object66626:head# 2024-05-15T23:40:14.627 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.628 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cacbff5:::benchmark_data_smithi090_166897_object31472:head# 2024-05-15T23:40:14.628 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.628 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cafab02:::benchmark_data_smithi090_166897_object67204:head# 2024-05-15T23:40:14.628 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.628 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6caff58c:::benchmark_data_smithi090_166897_object13517:head# 2024-05-15T23:40:14.628 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.628 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cb1bf21:::benchmark_data_smithi090_166897_object20651:head# 2024-05-15T23:40:14.628 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.628 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cb307e9:::benchmark_data_smithi090_166897_object62332:head# 2024-05-15T23:40:14.628 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.628 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cb37dd8:::benchmark_data_smithi090_166897_object44956:head# 2024-05-15T23:40:14.628 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.628 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cb38d9c:::benchmark_data_smithi090_166897_object25399:head# 2024-05-15T23:40:14.628 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.628 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cb5a78d:::benchmark_data_smithi090_166897_object47530:head# 2024-05-15T23:40:14.629 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.629 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cb62378:::benchmark_data_smithi090_166897_object50563:head# 2024-05-15T23:40:14.629 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.629 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cb7788c:::benchmark_data_smithi090_166897_object47122:head# 2024-05-15T23:40:14.629 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.629 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cb8422d:::benchmark_data_smithi090_166897_object68988:head# 2024-05-15T23:40:14.629 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.629 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cb8dd16:::benchmark_data_smithi090_166897_object69280:head# 2024-05-15T23:40:14.629 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.629 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cbb852d:::benchmark_data_smithi090_166897_object54511:head# 2024-05-15T23:40:14.629 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.629 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cbb987d:::benchmark_data_smithi090_166897_object12784:head# 2024-05-15T23:40:14.629 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.629 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cbba1fe:::benchmark_data_smithi090_166897_object29028:head# 2024-05-15T23:40:14.629 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.630 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cbc4a60:::benchmark_data_smithi090_166897_object34026:head# 2024-05-15T23:40:14.630 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.630 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cbd4db2:::benchmark_data_smithi090_166897_object59474:head# 2024-05-15T23:40:14.630 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.630 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cbe8299:::benchmark_data_smithi090_166897_object21036:head# 2024-05-15T23:40:14.630 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.630 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cbf7406:::benchmark_data_smithi090_166897_object29417:head# 2024-05-15T23:40:14.630 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.630 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cc43968:::benchmark_data_smithi090_166897_object1948:head# 2024-05-15T23:40:14.630 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.630 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cc44d02:::benchmark_data_smithi090_166897_object1266:head# 2024-05-15T23:40:14.630 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.630 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cc4de8a:::benchmark_data_smithi090_166897_object30087:head# 2024-05-15T23:40:14.630 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cc53d5e:::benchmark_data_smithi090_166897_object47464:head# 2024-05-15T23:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cc56a67:::benchmark_data_smithi090_166897_object32611:head# 2024-05-15T23:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cc8f5b7:::benchmark_data_smithi090_166897_object48173:head# 2024-05-15T23:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cc9fb4e:::benchmark_data_smithi090_166897_object44106:head# 2024-05-15T23:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ccb5068:::benchmark_data_smithi090_166897_object26567:head# 2024-05-15T23:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ccb8e7a:::benchmark_data_smithi090_166897_object58652:head# 2024-05-15T23:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ccd5de1:::benchmark_data_smithi090_166897_object25861:head# 2024-05-15T23:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ccdc2c2:::benchmark_data_smithi090_166897_object66028:head# 2024-05-15T23:40:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cce8642:::benchmark_data_smithi090_166897_object69632:head# 2024-05-15T23:40:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ccecfbb:::benchmark_data_smithi090_166897_object47638:head# 2024-05-15T23:40:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ccfe464:::benchmark_data_smithi090_166897_object37403:head# 2024-05-15T23:40:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cd058f9:::benchmark_data_smithi090_166897_object56369:head# 2024-05-15T23:40:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cd1857c:::benchmark_data_smithi090_166897_object29641:head# 2024-05-15T23:40:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cd19d35:::benchmark_data_smithi090_166897_object26206:head# 2024-05-15T23:40:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cd2560d:::benchmark_data_smithi090_166897_object25397:head# 2024-05-15T23:40:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.633 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cd531c0:::benchmark_data_smithi090_166897_object45597:head# 2024-05-15T23:40:14.633 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.633 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cd6013b:::benchmark_data_smithi090_166897_object39326:head# 2024-05-15T23:40:14.633 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.633 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cd628d6:::benchmark_data_smithi090_166897_object2351:head# 2024-05-15T23:40:14.633 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.633 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cd79379:::benchmark_data_smithi090_166897_object3287:head# 2024-05-15T23:40:14.633 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.633 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cd7d01e:::benchmark_data_smithi090_166897_object31615:head# 2024-05-15T23:40:14.633 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.633 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cd8b644:::benchmark_data_smithi090_166897_object58089:head# 2024-05-15T23:40:14.633 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.633 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cd8cfac:::benchmark_data_smithi090_166897_object68866:head# 2024-05-15T23:40:14.633 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.633 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cdb0c71:::benchmark_data_smithi090_166897_object68166:head# 2024-05-15T23:40:14.634 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.634 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cdbe018:::benchmark_data_smithi090_166897_object42006:head# 2024-05-15T23:40:14.634 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.634 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cdc88a5:::benchmark_data_smithi090_166897_object52297:head# 2024-05-15T23:40:14.634 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.634 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cddc760:::benchmark_data_smithi090_166897_object45775:head# 2024-05-15T23:40:14.634 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.634 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cde92fc:::benchmark_data_smithi090_166897_object39903:head# 2024-05-15T23:40:14.634 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.634 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cde9df1:::benchmark_data_smithi090_166897_object6017:head# 2024-05-15T23:40:14.634 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.634 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cdf0fee:::benchmark_data_smithi090_166897_object58351:head# 2024-05-15T23:40:14.634 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.634 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cdfff8f:::benchmark_data_smithi090_166897_object30412:head# 2024-05-15T23:40:14.635 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.635 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ce1f561:::benchmark_data_smithi090_166897_object50432:head# 2024-05-15T23:40:14.635 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.635 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ce2f338:::benchmark_data_smithi090_166897_object39005:head# 2024-05-15T23:40:14.635 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.635 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ce2ff22:::benchmark_data_smithi090_166897_object53732:head# 2024-05-15T23:40:14.635 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.635 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ce35fc5:::benchmark_data_smithi090_166897_object64504:head# 2024-05-15T23:40:14.635 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.635 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ce38b91:::benchmark_data_smithi090_166897_object51762:head# 2024-05-15T23:40:14.635 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.635 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ce3cb48:::benchmark_data_smithi090_166897_object29129:head# 2024-05-15T23:40:14.635 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.635 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ce3eaf7:::benchmark_data_smithi090_166897_object3335:head# 2024-05-15T23:40:14.635 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.636 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ce4bf54:::benchmark_data_smithi090_166897_object47031:head# 2024-05-15T23:40:14.636 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.636 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ce56068:::benchmark_data_smithi090_166897_object37928:head# 2024-05-15T23:40:14.636 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.636 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ce6cb0d:::benchmark_data_smithi090_166897_object19913:head# 2024-05-15T23:40:14.636 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.636 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ce70b39:::benchmark_data_smithi090_166897_object21261:head# 2024-05-15T23:40:14.636 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.636 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ce7b0af:::benchmark_data_smithi090_166897_object44663:head# 2024-05-15T23:40:14.636 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.636 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ce96f57:::benchmark_data_smithi090_166897_object25241:head# 2024-05-15T23:40:14.636 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.636 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cea3b2c:::benchmark_data_smithi090_166897_object53003:head# 2024-05-15T23:40:14.636 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.636 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cea4a92:::benchmark_data_smithi090_166897_object61764:head# 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ceac253:::benchmark_data_smithi090_166897_object39202:head# 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cec32d8:::benchmark_data_smithi090_166897_object26006:head# 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cedfcf4:::benchmark_data_smithi090_166897_object36479:head# 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cf01528:::benchmark_data_smithi090_166897_object28494:head# 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cf03133:::benchmark_data_smithi090_166897_object44219:head# 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cf25b44:::benchmark_data_smithi090_166897_object52040:head# 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cf417b6:::benchmark_data_smithi090_166897_object60898:head# 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.637 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cf4213b:::benchmark_data_smithi090_166897_object13003:head# 2024-05-15T23:40:14.638 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.638 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cf55a0e:::benchmark_data_smithi090_166897_object43500:head# 2024-05-15T23:40:14.638 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.638 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cf5d0d2:::benchmark_data_smithi090_166897_object25457:head# 2024-05-15T23:40:14.638 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.638 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cf602a7:::benchmark_data_smithi090_166897_object65175:head# 2024-05-15T23:40:14.638 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.638 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cf6179e:::benchmark_data_smithi090_166897_object24089:head# 2024-05-15T23:40:14.638 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.638 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cf71829:::benchmark_data_smithi090_166897_object49061:head# 2024-05-15T23:40:14.638 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.638 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cf78346:::benchmark_data_smithi090_166897_object48917:head# 2024-05-15T23:40:14.638 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.638 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cf91aa6:::benchmark_data_smithi090_166897_object43335:head# 2024-05-15T23:40:14.639 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.639 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cfc4e3a:::benchmark_data_smithi090_166897_object57188:head# 2024-05-15T23:40:14.639 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.639 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cfca211:::benchmark_data_smithi090_166897_object70814:head# 2024-05-15T23:40:14.639 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.639 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cfd3672:::benchmark_data_smithi090_166897_object6307:head# 2024-05-15T23:40:14.639 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.639 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cfd3f8d:::benchmark_data_smithi090_166897_object4259:head# 2024-05-15T23:40:14.639 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.639 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cfdb9bd:::benchmark_data_smithi090_166897_object2014:head# 2024-05-15T23:40:14.639 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.639 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cfec99d:::benchmark_data_smithi090_166897_object47904:head# 2024-05-15T23:40:14.639 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.639 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cfef2f7:::benchmark_data_smithi090_166897_object21268:head# 2024-05-15T23:40:14.640 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.640 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6cffc49b:::benchmark_data_smithi090_166897_object29525:head# 2024-05-15T23:40:14.640 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.640 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d00e37e:::benchmark_data_smithi090_166897_object44657:head# 2024-05-15T23:40:14.640 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.640 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d00fb74:::benchmark_data_smithi090_166897_object4993:head# 2024-05-15T23:40:14.640 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.640 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d022611:::benchmark_data_smithi090_166897_object35908:head# 2024-05-15T23:40:14.640 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.640 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d0276d2:::benchmark_data_smithi090_166897_object35572:head# 2024-05-15T23:40:14.640 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.640 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d027ed8:::benchmark_data_smithi090_166897_object54171:head# 2024-05-15T23:40:14.640 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.640 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d03a29c:::benchmark_data_smithi090_166897_object19519:head# 2024-05-15T23:40:14.640 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.641 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d0566b4:::benchmark_data_smithi090_166897_object54219:head# 2024-05-15T23:40:14.641 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.641 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d06cd37:::benchmark_data_smithi090_166897_object51699:head# 2024-05-15T23:40:14.641 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.641 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d06fde9:::benchmark_data_smithi090_166897_object14241:head# 2024-05-15T23:40:14.641 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.641 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d074d0a:::benchmark_data_smithi090_166897_object9749:head# 2024-05-15T23:40:14.641 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.641 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d095bf0:::benchmark_data_smithi090_166897_object10808:head# 2024-05-15T23:40:14.641 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.641 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d0a596a:::benchmark_data_smithi090_166897_object23093:head# 2024-05-15T23:40:14.641 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.641 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d0ab949:::benchmark_data_smithi090_166897_object4810:head# 2024-05-15T23:40:14.641 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.641 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d0de3a8:::benchmark_data_smithi090_166897_object863:head# 2024-05-15T23:40:14.642 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.642 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d0e6355:::benchmark_data_smithi090_166897_object21424:head# 2024-05-15T23:40:14.642 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.642 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d0ed8b4:::benchmark_data_smithi090_166897_object43665:head# 2024-05-15T23:40:14.642 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.642 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d0ee517:::benchmark_data_smithi090_166897_object25193:head# 2024-05-15T23:40:14.642 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.642 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d0ef50e:::benchmark_data_smithi090_166897_object67275:head# 2024-05-15T23:40:14.642 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.642 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d10338f:::benchmark_data_smithi090_166897_object31066:head# 2024-05-15T23:40:14.642 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.642 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d10dafa:::benchmark_data_smithi090_166897_object18895:head# 2024-05-15T23:40:14.642 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.642 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d12d5aa:::benchmark_data_smithi090_166897_object37712:head# 2024-05-15T23:40:14.642 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.643 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d131ba6:::benchmark_data_smithi090_166897_object48599:head# 2024-05-15T23:40:14.643 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.643 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d145318:::benchmark_data_smithi090_166897_object27286:head# 2024-05-15T23:40:14.643 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.643 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d14f0ba:::benchmark_data_smithi090_166897_object43265:head# 2024-05-15T23:40:14.643 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.643 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d15096f:::benchmark_data_smithi090_166897_object11311:head# 2024-05-15T23:40:14.643 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.643 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d154462:::benchmark_data_smithi090_166897_object50515:head# 2024-05-15T23:40:14.643 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.643 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d157556:::benchmark_data_smithi090_166897_object21949:head# 2024-05-15T23:40:14.643 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.643 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d169746:::benchmark_data_smithi090_166897_object35590:head# 2024-05-15T23:40:14.643 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d170792:::benchmark_data_smithi090_166897_object56694:head# 2024-05-15T23:40:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d17823f:::benchmark_data_smithi090_166897_object55441:head# 2024-05-15T23:40:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d17a1da:::benchmark_data_smithi090_166897_object40205:head# 2024-05-15T23:40:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d181312:::benchmark_data_smithi090_166897_object23921:head# 2024-05-15T23:40:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d1c63e5:::benchmark_data_smithi090_166897_object19955:head# 2024-05-15T23:40:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d1ca02d:::benchmark_data_smithi090_166897_object62087:head# 2024-05-15T23:40:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d1ebc6f:::benchmark_data_smithi090_166897_object13573:head# 2024-05-15T23:40:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.644 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d1f57fa:::benchmark_data_smithi090_166897_object35290:head# 2024-05-15T23:40:14.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.645 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d200c44:::benchmark_data_smithi090_166897_object25566:head# 2024-05-15T23:40:14.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.645 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d202113:::benchmark_data_smithi090_166897_object66334:head# 2024-05-15T23:40:14.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.645 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d208487:::benchmark_data_smithi090_166897_object20020:head# 2024-05-15T23:40:14.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.645 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d20db1c:::benchmark_data_smithi090_166897_object52062:head# 2024-05-15T23:40:14.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.645 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d21370e:::benchmark_data_smithi090_166897_object45308:head# 2024-05-15T23:40:14.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.645 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d21cbf6:::benchmark_data_smithi090_166897_object58974:head# 2024-05-15T23:40:14.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.645 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d21f7b8:::benchmark_data_smithi090_166897_object49932:head# 2024-05-15T23:40:14.645 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.646 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d2239e1:::benchmark_data_smithi090_166897_object8841:head# 2024-05-15T23:40:14.646 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.646 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d232f07:::benchmark_data_smithi090_166897_object53684:head# 2024-05-15T23:40:14.646 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.646 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d236d4d:::benchmark_data_smithi090_166897_object40819:head# 2024-05-15T23:40:14.646 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.646 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d24e5ba:::benchmark_data_smithi090_166897_object5453:head# 2024-05-15T23:40:14.646 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.646 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d252da2:::benchmark_data_smithi090_166897_object60845:head# 2024-05-15T23:40:14.646 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.646 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d26f80d:::benchmark_data_smithi090_166897_object54030:head# 2024-05-15T23:40:14.646 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.646 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d278ea9:::benchmark_data_smithi090_166897_object23899:head# 2024-05-15T23:40:14.646 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.646 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d286cf0:::benchmark_data_smithi090_166897_object14378:head# 2024-05-15T23:40:14.647 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.647 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d2887d5:::benchmark_data_smithi090_166897_object36394:head# 2024-05-15T23:40:14.647 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.647 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d29869f:::benchmark_data_smithi090_166897_object40069:head# 2024-05-15T23:40:14.647 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.647 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d2a8ebd:::benchmark_data_smithi090_166897_object65415:head# 2024-05-15T23:40:14.647 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.647 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d2c5068:::benchmark_data_smithi090_166897_object9823:head# 2024-05-15T23:40:14.647 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.647 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d2cd99e:::benchmark_data_smithi090_166897_object42423:head# 2024-05-15T23:40:14.647 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.647 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d2e160c:::benchmark_data_smithi090_166897_object35708:head# 2024-05-15T23:40:14.647 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.647 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d2f79a8:::benchmark_data_smithi090_166897_object58743:head# 2024-05-15T23:40:14.647 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.648 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d3301d1:::benchmark_data_smithi090_166897_object58040:head# 2024-05-15T23:40:14.648 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.648 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d34ea08:::benchmark_data_smithi090_166897_object50659:head# 2024-05-15T23:40:14.648 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.648 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d353e1f:::benchmark_data_smithi090_166897_object64688:head# 2024-05-15T23:40:14.648 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.648 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d371d85:::benchmark_data_smithi090_166897_object27156:head# 2024-05-15T23:40:14.648 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.648 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d379128:::benchmark_data_smithi090_166897_object54200:head# 2024-05-15T23:40:14.648 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.648 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d38fc2c:::benchmark_data_smithi090_166897_object11936:head# 2024-05-15T23:40:14.648 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.648 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d3ca66d:::benchmark_data_smithi090_166897_object47774:head# 2024-05-15T23:40:14.648 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d3d1e99:::benchmark_data_smithi090_166897_object38622:head# 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d3ef90b:::benchmark_data_smithi090_166897_object57451:head# 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d3fabd6:::benchmark_data_smithi090_166897_object39095:head# 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d433d81:::benchmark_data_smithi090_166897_object48361:head# 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d4436fa:::benchmark_data_smithi090_166897_object38424:head# 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d45cd1e:::benchmark_data_smithi090_166897_object26107:head# 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d46ac55:::benchmark_data_smithi090_166897_object48837:head# 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d48314d:::benchmark_data_smithi090_166897_object65234:head# 2024-05-15T23:40:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.650 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d49eb09:::benchmark_data_smithi090_166897_object66685:head# 2024-05-15T23:40:14.650 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.650 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d4a483b:::benchmark_data_smithi090_166897_object64564:head# 2024-05-15T23:40:14.650 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.650 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d4ab133:::benchmark_data_smithi090_166897_object4079:head# 2024-05-15T23:40:14.650 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.650 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d4c9f1d:::benchmark_data_smithi090_166897_object63874:head# 2024-05-15T23:40:14.650 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.650 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d4e1b88:::benchmark_data_smithi090_166897_object2745:head# 2024-05-15T23:40:14.650 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.650 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d4e27f9:::benchmark_data_smithi090_166897_object678:head# 2024-05-15T23:40:14.650 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.650 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d4e3575:::benchmark_data_smithi090_166897_object25895:head# 2024-05-15T23:40:14.650 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.651 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d4f187c:::benchmark_data_smithi090_166897_object8415:head# 2024-05-15T23:40:14.651 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.651 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d4f3490:::benchmark_data_smithi090_166897_object27283:head# 2024-05-15T23:40:14.651 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.651 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d4f8f69:::benchmark_data_smithi090_166897_object64841:head# 2024-05-15T23:40:14.651 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.651 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d51383b:::benchmark_data_smithi090_166897_object51274:head# 2024-05-15T23:40:14.651 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.651 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d513fda:::benchmark_data_smithi090_166897_object20094:head# 2024-05-15T23:40:14.651 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.651 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d544fff:::benchmark_data_smithi090_166897_object41374:head# 2024-05-15T23:40:14.651 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.651 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d54c4b5:::benchmark_data_smithi090_166897_object55089:head# 2024-05-15T23:40:14.651 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.651 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d54f27c:::benchmark_data_smithi090_166897_object47419:head# 2024-05-15T23:40:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d55b264:::benchmark_data_smithi090_166897_object4540:head# 2024-05-15T23:40:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d574b9e:::benchmark_data_smithi090_166897_object30612:head# 2024-05-15T23:40:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d582911:::benchmark_data_smithi090_166897_object64071:head# 2024-05-15T23:40:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d593dc8:::benchmark_data_smithi090_166897_object54261:head# 2024-05-15T23:40:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d59f884:::benchmark_data_smithi090_166897_object2170:head# 2024-05-15T23:40:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d5b60ea:::benchmark_data_smithi090_166897_object16643:head# 2024-05-15T23:40:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d5b9621:::benchmark_data_smithi090_166897_object18781:head# 2024-05-15T23:40:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d5b9665:::benchmark_data_smithi090_166897_object17008:head# 2024-05-15T23:40:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d5c0edf:::benchmark_data_smithi090_166897_object15432:head# 2024-05-15T23:40:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d5d1545:::benchmark_data_smithi090_166897_object53295:head# 2024-05-15T23:40:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d5d521a:::benchmark_data_smithi090_166897_object53683:head# 2024-05-15T23:40:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d5d8d4e:::benchmark_data_smithi090_166897_object52179:head# 2024-05-15T23:40:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d5ec304:::benchmark_data_smithi090_166897_object49984:head# 2024-05-15T23:40:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d5fbce0:::benchmark_data_smithi090_166897_object60395:head# 2024-05-15T23:40:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d6077e4:::benchmark_data_smithi090_166897_object1663:head# 2024-05-15T23:40:14.654 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.654 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d615d6a:::benchmark_data_smithi090_166897_object12634:head# 2024-05-15T23:40:14.654 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.654 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d63d546:::benchmark_data_smithi090_166897_object47578:head# 2024-05-15T23:40:14.654 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.654 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d6476aa:::benchmark_data_smithi090_166897_object41185:head# 2024-05-15T23:40:14.654 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.654 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d675ebf:::benchmark_data_smithi090_166897_object11495:head# 2024-05-15T23:40:14.654 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.654 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d67c4cd:::benchmark_data_smithi090_166897_object68960:head# 2024-05-15T23:40:14.654 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.654 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d684839:::benchmark_data_smithi090_166897_object27456:head# 2024-05-15T23:40:14.654 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.654 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d688e33:::benchmark_data_smithi090_166897_object35178:head# 2024-05-15T23:40:14.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.655 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d69f1f4:::benchmark_data_smithi090_166897_object28183:head# 2024-05-15T23:40:14.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.655 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d6c3097:::benchmark_data_smithi090_166897_object301:head# 2024-05-15T23:40:14.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.655 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d6c9282:::benchmark_data_smithi090_166897_object12312:head# 2024-05-15T23:40:14.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.655 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d6c942e:::benchmark_data_smithi090_166897_object10745:head# 2024-05-15T23:40:14.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.655 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d6d4567:::benchmark_data_smithi090_166897_object41266:head# 2024-05-15T23:40:14.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.655 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d6e20ba:::benchmark_data_smithi090_166897_object29366:head# 2024-05-15T23:40:14.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.655 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d6f8ae7:::benchmark_data_smithi090_166897_object4377:head# 2024-05-15T23:40:14.655 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d6fdb75:::benchmark_data_smithi090_166897_object67184:head# 2024-05-15T23:40:14.656 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d6fed72:::benchmark_data_smithi090_166897_object23416:head# 2024-05-15T23:40:14.656 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d7534d1:::benchmark_data_smithi090_166897_object23936:head# 2024-05-15T23:40:14.656 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d755ad9:::benchmark_data_smithi090_166897_object6078:head# 2024-05-15T23:40:14.656 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d764aaf:::benchmark_data_smithi090_166897_object2891:head# 2024-05-15T23:40:14.656 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d7822b7:::benchmark_data_smithi090_166897_object6922:head# 2024-05-15T23:40:14.656 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d78d3af:::benchmark_data_smithi090_166897_object21959:head# 2024-05-15T23:40:14.656 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.656 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d79cea9:::benchmark_data_smithi090_166897_object2213:head# 2024-05-15T23:40:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d7a811d:::benchmark_data_smithi090_166897_object3107:head# 2024-05-15T23:40:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d7d09df:::benchmark_data_smithi090_166897_object67151:head# 2024-05-15T23:40:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d7d87ea:::benchmark_data_smithi090_166897_object20009:head# 2024-05-15T23:40:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d7eec0f:::benchmark_data_smithi090_166897_object3315:head# 2024-05-15T23:40:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d7f55cf:::benchmark_data_smithi090_166897_object64940:head# 2024-05-15T23:40:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d82cb4b:::benchmark_data_smithi090_166897_object28167:head# 2024-05-15T23:40:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d835c50:::benchmark_data_smithi090_166897_object31910:head# 2024-05-15T23:40:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.658 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d846773:::benchmark_data_smithi090_166897_object25490:head# 2024-05-15T23:40:14.658 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.658 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d8470ef:::benchmark_data_smithi090_166897_object7536:head# 2024-05-15T23:40:14.658 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.658 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d847e88:::benchmark_data_smithi090_166897_object41924:head# 2024-05-15T23:40:14.658 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.658 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d874617:::benchmark_data_smithi090_166897_object62646:head# 2024-05-15T23:40:14.658 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.658 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d8752a0:::benchmark_data_smithi090_166897_object32237:head# 2024-05-15T23:40:14.658 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.658 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d875e97:::benchmark_data_smithi090_166897_object14629:head# 2024-05-15T23:40:14.658 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.658 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d87caf7:::benchmark_data_smithi090_166897_object1107:head# 2024-05-15T23:40:14.658 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d88c8a8:::benchmark_data_smithi090_166897_object17172:head# 2024-05-15T23:40:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d89a033:::benchmark_data_smithi090_166897_object24517:head# 2024-05-15T23:40:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d8a763d:::benchmark_data_smithi090_166897_object48352:head# 2024-05-15T23:40:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d8aef5d:::benchmark_data_smithi090_166897_object37510:head# 2024-05-15T23:40:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d8bb8dc:::benchmark_data_smithi090_166897_object17539:head# 2024-05-15T23:40:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d8bfb02:::benchmark_data_smithi090_166897_object52313:head# 2024-05-15T23:40:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d8d344f:::benchmark_data_smithi090_166897_object61307:head# 2024-05-15T23:40:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.659 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d8d5db5:::benchmark_data_smithi090_166897_object71016:head# 2024-05-15T23:40:14.660 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.660 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d8e2adc:::benchmark_data_smithi090_166897_object46940:head# 2024-05-15T23:40:14.660 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.660 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d8edb1c:::benchmark_data_smithi090_166897_object11205:head# 2024-05-15T23:40:14.660 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.660 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d8ef781:::benchmark_data_smithi090_166897_object14826:head# 2024-05-15T23:40:14.660 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.660 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d90412a:::benchmark_data_smithi090_166897_object50124:head# 2024-05-15T23:40:14.660 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.660 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d90454c:::benchmark_data_smithi090_166897_object40956:head# 2024-05-15T23:40:14.660 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.660 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d90e076:::benchmark_data_smithi090_166897_object1148:head# 2024-05-15T23:40:14.660 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.660 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d926eb7:::benchmark_data_smithi090_166897_object50433:head# 2024-05-15T23:40:14.660 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d94f84f:::benchmark_data_smithi090_166897_object16610:head# 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d95d17a:::benchmark_data_smithi090_166897_object36448:head# 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d976861:::benchmark_data_smithi090_166897_object21448:head# 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d97f85e:::benchmark_data_smithi090_166897_object62247:head# 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d98e548:::benchmark_data_smithi090_166897_object39532:head# 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d99fe0f:::benchmark_data_smithi090_166897_object7809:head# 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d9aa290:::benchmark_data_smithi090_166897_object65296:head# 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d9ad261:::benchmark_data_smithi090_166897_object21308:head# 2024-05-15T23:40:14.661 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.662 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d9b0e57:::benchmark_data_smithi090_166897_object40182:head# 2024-05-15T23:40:14.662 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.662 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d9b2dc3:::benchmark_data_smithi090_166897_object14210:head# 2024-05-15T23:40:14.662 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.662 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d9d04c5:::benchmark_data_smithi090_166897_object23792:head# 2024-05-15T23:40:14.662 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.662 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6d9fefdf:::benchmark_data_smithi090_166897_object17734:head# 2024-05-15T23:40:14.662 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.662 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6da05911:::benchmark_data_smithi090_166897_object32421:head# 2024-05-15T23:40:14.662 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.662 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6da2ed01:::benchmark_data_smithi090_166897_object64061:head# 2024-05-15T23:40:14.662 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.662 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6da41869:::benchmark_data_smithi090_166897_object51945:head# 2024-05-15T23:40:14.662 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.663 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6da5ac9f:::benchmark_data_smithi090_166897_object55712:head# 2024-05-15T23:40:14.663 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.663 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6da5c2ae:::benchmark_data_smithi090_166897_object13753:head# 2024-05-15T23:40:14.663 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.663 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6da70d15:::benchmark_data_smithi090_166897_object11533:head# 2024-05-15T23:40:14.663 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.663 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6da7f7ab:::benchmark_data_smithi090_166897_object10142:head# 2024-05-15T23:40:14.663 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.663 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6da8b468:::benchmark_data_smithi090_166897_object35915:head# 2024-05-15T23:40:14.663 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.663 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6da9bd04:::benchmark_data_smithi090_166897_object34810:head# 2024-05-15T23:40:14.663 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.663 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6da9dcc3:::benchmark_data_smithi090_166897_object11280:head# 2024-05-15T23:40:14.663 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.663 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6daae3df:::benchmark_data_smithi090_166897_object8581:head# 2024-05-15T23:40:14.664 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.664 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dacb6fc:::benchmark_data_smithi090_166897_object17343:head# 2024-05-15T23:40:14.664 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.664 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dae77c2:::benchmark_data_smithi090_166897_object12220:head# 2024-05-15T23:40:14.664 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.664 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dae861b:::benchmark_data_smithi090_166897_object58066:head# 2024-05-15T23:40:14.664 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.664 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6daea24e:::benchmark_data_smithi090_166897_object70026:head# 2024-05-15T23:40:14.664 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.664 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6daeb677:::benchmark_data_smithi090_166897_object58012:head# 2024-05-15T23:40:14.664 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.664 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6daee3e0:::benchmark_data_smithi090_166897_object56341:head# 2024-05-15T23:40:14.664 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.664 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db04e50:::benchmark_data_smithi090_166897_object3354:head# 2024-05-15T23:40:14.664 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.665 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db0a74f:::benchmark_data_smithi090_166897_object63175:head# 2024-05-15T23:40:14.665 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.665 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db26e4c:::benchmark_data_smithi090_166897_object34746:head# 2024-05-15T23:40:14.665 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.665 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db2a8e0:::benchmark_data_smithi090_166897_object50218:head# 2024-05-15T23:40:14.665 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.665 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db2e6e1:::benchmark_data_smithi090_166897_object54136:head# 2024-05-15T23:40:14.665 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.665 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db38273:::benchmark_data_smithi090_166897_object22795:head# 2024-05-15T23:40:14.665 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.665 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db3f99d:::benchmark_data_smithi090_166897_object56790:head# 2024-05-15T23:40:14.665 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.665 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db4c7c2:::benchmark_data_smithi090_166897_object26814:head# 2024-05-15T23:40:14.666 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.666 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db5d774:::benchmark_data_smithi090_166897_object21131:head# 2024-05-15T23:40:14.666 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.666 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db5e084:::benchmark_data_smithi090_166897_object47079:head# 2024-05-15T23:40:14.666 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.666 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db680f9:::benchmark_data_smithi090_166897_object31285:head# 2024-05-15T23:40:14.666 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.666 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db72e61:::benchmark_data_smithi090_166897_object42534:head# 2024-05-15T23:40:14.666 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.666 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db867c3:::benchmark_data_smithi090_166897_object893:head# 2024-05-15T23:40:14.666 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.666 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db8f7c4:::benchmark_data_smithi090_166897_object9451:head# 2024-05-15T23:40:14.666 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.666 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db9a2ab:::benchmark_data_smithi090_166897_object71052:head# 2024-05-15T23:40:14.666 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6db9f509:::benchmark_data_smithi090_166897_object57946:head# 2024-05-15T23:40:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dba62fe:::benchmark_data_smithi090_166897_object51966:head# 2024-05-15T23:40:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dba8c4f:::benchmark_data_smithi090_166897_object40275:head# 2024-05-15T23:40:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dbb8c84:::benchmark_data_smithi090_166897_object68751:head# 2024-05-15T23:40:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dbcbe69:::benchmark_data_smithi090_166897_object30316:head# 2024-05-15T23:40:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dbdc34c:::benchmark_data_smithi090_166897_object62457:head# 2024-05-15T23:40:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dbdd7b2:::benchmark_data_smithi090_166897_object57662:head# 2024-05-15T23:40:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.667 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dbeea9c:::benchmark_data_smithi090_166897_object46890:head# 2024-05-15T23:40:14.668 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.668 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dbf48ba:::benchmark_data_smithi090_166897_object50074:head# 2024-05-15T23:40:14.668 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.668 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dc0baf3:::benchmark_data_smithi090_166897_object54831:head# 2024-05-15T23:40:14.668 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.668 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dc1d35c:::benchmark_data_smithi090_166897_object38162:head# 2024-05-15T23:40:14.668 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.668 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dc1fa1a:::benchmark_data_smithi090_166897_object47632:head# 2024-05-15T23:40:14.668 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.668 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dc20491:::benchmark_data_smithi090_166897_object54277:head# 2024-05-15T23:40:14.668 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.668 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dc355e6:::benchmark_data_smithi090_166897_object2602:head# 2024-05-15T23:40:14.668 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.668 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dc395eb:::benchmark_data_smithi090_166897_object51049:head# 2024-05-15T23:40:14.668 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.669 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dc55755:::benchmark_data_smithi090_166897_object45285:head# 2024-05-15T23:40:14.669 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.669 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dc5a2b0:::benchmark_data_smithi090_166897_object44866:head# 2024-05-15T23:40:14.669 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.669 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dc5a939:::benchmark_data_smithi090_166897_object68010:head# 2024-05-15T23:40:14.669 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.669 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dc64581:::benchmark_data_smithi090_166897_object56435:head# 2024-05-15T23:40:14.669 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.669 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dc84a2d:::benchmark_data_smithi090_166897_object61628:head# 2024-05-15T23:40:14.669 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.669 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dca99c3:::benchmark_data_smithi090_166897_object23712:head# 2024-05-15T23:40:14.669 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.669 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dcab96f:::benchmark_data_smithi090_166897_object68265:head# 2024-05-15T23:40:14.669 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.669 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dcaf88c:::benchmark_data_smithi090_166897_object40568:head# 2024-05-15T23:40:14.670 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.670 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dcaf9b3:::benchmark_data_smithi090_166897_object16608:head# 2024-05-15T23:40:14.670 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.670 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dccbdfe:::benchmark_data_smithi090_166897_object17133:head# 2024-05-15T23:40:14.670 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.670 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dcdb128:::benchmark_data_smithi090_166897_object54194:head# 2024-05-15T23:40:14.670 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.670 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dcec699:::benchmark_data_smithi090_166897_object12030:head# 2024-05-15T23:40:14.670 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.670 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dcf1550:::benchmark_data_smithi090_166897_object26992:head# 2024-05-15T23:40:14.670 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.670 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dcf7b39:::benchmark_data_smithi090_166897_object25910:head# 2024-05-15T23:40:14.670 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.670 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dd0f3e0:::benchmark_data_smithi090_166897_object17529:head# 2024-05-15T23:40:14.671 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.671 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dd24d19:::benchmark_data_smithi090_166897_object32476:head# 2024-05-15T23:40:14.671 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.671 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dd3757c:::benchmark_data_smithi090_166897_object61383:head# 2024-05-15T23:40:14.671 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.671 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dd4b006:::benchmark_data_smithi090_166897_object23751:head# 2024-05-15T23:40:14.671 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.671 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dd60170:::benchmark_data_smithi090_166897_object50569:head# 2024-05-15T23:40:14.671 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.671 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dd6aff8:::benchmark_data_smithi090_166897_object34818:head# 2024-05-15T23:40:14.671 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.672 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dd75818:::benchmark_data_smithi090_166897_object38014:head# 2024-05-15T23:40:14.672 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.672 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dd81fbc:::benchmark_data_smithi090_166897_object55894:head# 2024-05-15T23:40:14.672 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.672 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dd8687c:::benchmark_data_smithi090_166897_object62176:head# 2024-05-15T23:40:14.672 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.672 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dd90bbf:::benchmark_data_smithi090_166897_object38821:head# 2024-05-15T23:40:14.672 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.672 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ddae45a:::benchmark_data_smithi090_166897_object42176:head# 2024-05-15T23:40:14.672 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.672 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ddb9095:::benchmark_data_smithi090_166897_object53247:head# 2024-05-15T23:40:14.672 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.672 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ddbff34:::benchmark_data_smithi090_166897_object38284:head# 2024-05-15T23:40:14.672 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.672 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ddc6c98:::benchmark_data_smithi090_166897_object61572:head# 2024-05-15T23:40:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ddd9df5:::benchmark_data_smithi090_166897_object24602:head# 2024-05-15T23:40:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6de0e4ab:::benchmark_data_smithi090_166897_object2031:head# 2024-05-15T23:40:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6de139c0:::benchmark_data_smithi090_166897_object4436:head# 2024-05-15T23:40:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6de2f075:::benchmark_data_smithi090_166897_object18439:head# 2024-05-15T23:40:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6de368e2:::benchmark_data_smithi090_166897_object12823:head# 2024-05-15T23:40:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6de394cb:::benchmark_data_smithi090_166897_object68054:head# 2024-05-15T23:40:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6de3f654:::benchmark_data_smithi090_166897_object48394:head# 2024-05-15T23:40:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.674 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6de48a2d:::benchmark_data_smithi090_166897_object13224:head# 2024-05-15T23:40:14.674 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.674 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6de4d13e:::benchmark_data_smithi090_166897_object12843:head# 2024-05-15T23:40:14.674 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.674 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6de6030d:::benchmark_data_smithi090_166897_object30434:head# 2024-05-15T23:40:14.674 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.674 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6de73af9:::benchmark_data_smithi090_166897_object30441:head# 2024-05-15T23:40:14.674 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.674 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dec12bc:::benchmark_data_smithi090_166897_object7596:head# 2024-05-15T23:40:14.674 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.674 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6deca4b4:::benchmark_data_smithi090_166897_object34201:head# 2024-05-15T23:40:14.674 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.674 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6def08bb:::benchmark_data_smithi090_166897_object62100:head# 2024-05-15T23:40:14.674 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.675 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6def6847:::benchmark_data_smithi090_166897_object18159:head# 2024-05-15T23:40:14.675 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.675 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6def9b57:::benchmark_data_smithi090_166897_object17555:head# 2024-05-15T23:40:14.675 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.675 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df011e1:::benchmark_data_smithi090_166897_object48305:head# 2024-05-15T23:40:14.675 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.675 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df0784e:::benchmark_data_smithi090_166897_object56946:head# 2024-05-15T23:40:14.675 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.675 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df13f3b:::benchmark_data_smithi090_166897_object61088:head# 2024-05-15T23:40:14.675 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.675 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df142cf:::benchmark_data_smithi090_166897_object15983:head# 2024-05-15T23:40:14.675 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.675 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df1afaa:::benchmark_data_smithi090_166897_object26288:head# 2024-05-15T23:40:14.675 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df1e4ce:::benchmark_data_smithi090_166897_object31883:head# 2024-05-15T23:40:14.676 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df236ad:::benchmark_data_smithi090_166897_object34479:head# 2024-05-15T23:40:14.676 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df357ee:::benchmark_data_smithi090_166897_object63864:head# 2024-05-15T23:40:14.676 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df3faca:::benchmark_data_smithi090_166897_object65182:head# 2024-05-15T23:40:14.676 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df41038:::benchmark_data_smithi090_166897_object20327:head# 2024-05-15T23:40:14.676 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df430db:::benchmark_data_smithi090_166897_object10927:head# 2024-05-15T23:40:14.676 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df4316b:::benchmark_data_smithi090_166897_object10871:head# 2024-05-15T23:40:14.676 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.676 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df48d90:::benchmark_data_smithi090_166897_object10698:head# 2024-05-15T23:40:14.677 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.677 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df516f9:::benchmark_data_smithi090_166897_object42221:head# 2024-05-15T23:40:14.677 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.677 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df57f98:::benchmark_data_smithi090_166897_object13348:head# 2024-05-15T23:40:14.677 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.677 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df641d5:::benchmark_data_smithi090_166897_object62045:head# 2024-05-15T23:40:14.677 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.677 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df72361:::benchmark_data_smithi090_166897_object33055:head# 2024-05-15T23:40:14.677 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.677 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df86e2a:::benchmark_data_smithi090_166897_object25704:head# 2024-05-15T23:40:14.677 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.677 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df8ee1d:::benchmark_data_smithi090_166897_object63677:head# 2024-05-15T23:40:14.677 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.678 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df9464d:::benchmark_data_smithi090_166897_object34567:head# 2024-05-15T23:40:14.678 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.678 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6df9d80f:::benchmark_data_smithi090_166897_object29942:head# 2024-05-15T23:40:14.678 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.678 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dfa2b18:::benchmark_data_smithi090_166897_object33335:head# 2024-05-15T23:40:14.678 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.678 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dfb3ea5:::benchmark_data_smithi090_166897_object2061:head# 2024-05-15T23:40:14.678 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.678 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dfb4c0a:::benchmark_data_smithi090_166897_object4673:head# 2024-05-15T23:40:14.678 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.678 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dfb55ed:::benchmark_data_smithi090_166897_object30769:head# 2024-05-15T23:40:14.678 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.678 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dfb56c0:::benchmark_data_smithi090_166897_object38354:head# 2024-05-15T23:40:14.678 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.679 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dfceabf:::benchmark_data_smithi090_166897_object21414:head# 2024-05-15T23:40:14.679 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.679 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6dfe6b65:::benchmark_data_smithi090_166897_object2056:head# 2024-05-15T23:40:14.679 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.679 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e03a927:::benchmark_data_smithi090_166897_object63524:head# 2024-05-15T23:40:14.679 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.679 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e048561:::benchmark_data_smithi090_166897_object18563:head# 2024-05-15T23:40:14.679 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.679 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e064658:::benchmark_data_smithi090_166897_object56735:head# 2024-05-15T23:40:14.679 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.679 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e086181:::benchmark_data_smithi090_166897_object48034:head# 2024-05-15T23:40:14.679 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.679 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e088615:::benchmark_data_smithi090_166897_object14565:head# 2024-05-15T23:40:14.679 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.680 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e08b987:::benchmark_data_smithi090_166897_object61990:head# 2024-05-15T23:40:14.680 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.680 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e094a4d:::benchmark_data_smithi090_166897_object32737:head# 2024-05-15T23:40:14.680 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.680 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e0b0e29:::benchmark_data_smithi090_166897_object25353:head# 2024-05-15T23:40:14.680 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.680 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e0b103d:::benchmark_data_smithi090_166897_object66243:head# 2024-05-15T23:40:14.680 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.680 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e0c51b3:::benchmark_data_smithi090_166897_object5305:head# 2024-05-15T23:40:14.680 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.680 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e0d808e:::benchmark_data_smithi090_166897_object18315:head# 2024-05-15T23:40:14.680 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.680 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e11c27d:::benchmark_data_smithi090_166897_object34358:head# 2024-05-15T23:40:14.680 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.681 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e121a8a:::benchmark_data_smithi090_166897_object67076:head# 2024-05-15T23:40:14.681 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.681 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e12d6a4:::benchmark_data_smithi090_166897_object39187:head# 2024-05-15T23:40:14.681 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.681 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e136a1f:::benchmark_data_smithi090_166897_object42075:head# 2024-05-15T23:40:14.681 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.681 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e13a3f1:::benchmark_data_smithi090_166897_object43300:head# 2024-05-15T23:40:14.681 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.681 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e13d6a6:::benchmark_data_smithi090_166897_object67869:head# 2024-05-15T23:40:14.681 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.681 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e1640d0:::benchmark_data_smithi090_166897_object43259:head# 2024-05-15T23:40:14.681 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.681 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e1748b9:::benchmark_data_smithi090_166897_object69254:head# 2024-05-15T23:40:14.681 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.682 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e180ec6:::benchmark_data_smithi090_166897_object68780:head# 2024-05-15T23:40:14.682 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.682 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e18a1f0:::benchmark_data_smithi090_166897_object55447:head# 2024-05-15T23:40:14.682 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.682 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e1a5525:::benchmark_data_smithi090_166897_object58873:head# 2024-05-15T23:40:14.682 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.682 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e1c68e2:::benchmark_data_smithi090_166897_object42130:head# 2024-05-15T23:40:14.682 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.682 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e1d278c:::benchmark_data_smithi090_166897_object35376:head# 2024-05-15T23:40:14.682 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.682 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e1db69b:::benchmark_data_smithi090_166897_object23065:head# 2024-05-15T23:40:14.682 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.682 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e1dcb6e:::benchmark_data_smithi090_166897_object28011:head# 2024-05-15T23:40:14.682 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.683 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e1de823:::benchmark_data_smithi090_166897_object61507:head# 2024-05-15T23:40:14.683 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.683 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e1e418a:::benchmark_data_smithi090_166897_object67830:head# 2024-05-15T23:40:14.683 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.683 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e1eba9f:::benchmark_data_smithi090_166897_object61592:head# 2024-05-15T23:40:14.683 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.683 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e1f60e3:::benchmark_data_smithi090_166897_object68904:head# 2024-05-15T23:40:14.683 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.683 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e20a4fa:::benchmark_data_smithi090_166897_object40286:head# 2024-05-15T23:40:14.683 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.683 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e20fdca:::benchmark_data_smithi090_166897_object37545:head# 2024-05-15T23:40:14.683 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.683 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e21365a:::benchmark_data_smithi090_166897_object4058:head# 2024-05-15T23:40:14.683 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.683 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e216c56:::benchmark_data_smithi090_166897_object59382:head# 2024-05-15T23:40:14.684 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.684 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e21ecfe:::benchmark_data_smithi090_166897_object59728:head# 2024-05-15T23:40:14.684 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.684 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e22bdfa:::benchmark_data_smithi090_166897_object60125:head# 2024-05-15T23:40:14.684 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.684 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e22db38:::benchmark_data_smithi090_166897_object24999:head# 2024-05-15T23:40:14.684 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.684 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e232c9a:::benchmark_data_smithi090_166897_object56967:head# 2024-05-15T23:40:14.684 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.684 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e243977:::benchmark_data_smithi090_166897_object37208:head# 2024-05-15T23:40:14.684 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.684 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e254d10:::benchmark_data_smithi090_166897_object35554:head# 2024-05-15T23:40:14.684 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.684 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e25aded:::benchmark_data_smithi090_166897_object36441:head# 2024-05-15T23:40:14.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.685 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e267f9d:::benchmark_data_smithi090_166897_object32552:head# 2024-05-15T23:40:14.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.685 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e273062:::benchmark_data_smithi090_166897_object25383:head# 2024-05-15T23:40:14.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.685 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e27930d:::benchmark_data_smithi090_166897_object59950:head# 2024-05-15T23:40:14.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.685 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e28ec29:::benchmark_data_smithi090_166897_object32823:head# 2024-05-15T23:40:14.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.685 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e2d0018:::benchmark_data_smithi090_166897_object60876:head# 2024-05-15T23:40:14.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.685 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e2e8405:::benchmark_data_smithi090_166897_object15631:head# 2024-05-15T23:40:14.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.685 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e2eec09:::benchmark_data_smithi090_166897_object41113:head# 2024-05-15T23:40:14.685 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.686 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e2fbf2e:::benchmark_data_smithi090_166897_object66642:head# 2024-05-15T23:40:14.686 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.686 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e3035ad:::benchmark_data_smithi090_166897_object55518:head# 2024-05-15T23:40:14.686 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.686 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e30c66e:::benchmark_data_smithi090_166897_object30669:head# 2024-05-15T23:40:14.686 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.686 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e30d22d:::benchmark_data_smithi090_166897_object38329:head# 2024-05-15T23:40:14.686 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.686 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e320e5c:::benchmark_data_smithi090_166897_object38005:head# 2024-05-15T23:40:14.686 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.686 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e328fe5:::benchmark_data_smithi090_166897_object23252:head# 2024-05-15T23:40:14.686 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.686 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e332f60:::benchmark_data_smithi090_166897_object46818:head# 2024-05-15T23:40:14.687 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.687 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e339681:::benchmark_data_smithi090_166897_object52669:head# 2024-05-15T23:40:14.687 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.687 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e33abc6:::benchmark_data_smithi090_166897_object68118:head# 2024-05-15T23:40:14.687 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.687 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e35203b:::benchmark_data_smithi090_166897_object16864:head# 2024-05-15T23:40:14.687 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.687 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e35a6d4:::benchmark_data_smithi090_166897_object8097:head# 2024-05-15T23:40:14.687 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.687 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e35f49b:::benchmark_data_smithi090_166897_object24734:head# 2024-05-15T23:40:14.687 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.687 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e37124c:::benchmark_data_smithi090_166897_object50439:head# 2024-05-15T23:40:14.687 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.687 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e37e764:::benchmark_data_smithi090_166897_object50830:head# 2024-05-15T23:40:14.687 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.688 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e383678:::benchmark_data_smithi090_166897_object15366:head# 2024-05-15T23:40:14.688 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.688 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e384d07:::benchmark_data_smithi090_166897_object8460:head# 2024-05-15T23:40:14.688 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.688 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e3a5b74:::benchmark_data_smithi090_166897_object52159:head# 2024-05-15T23:40:14.688 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.688 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e3c5b1b:::benchmark_data_smithi090_166897_object7532:head# 2024-05-15T23:40:14.688 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.688 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e3e986d:::benchmark_data_smithi090_166897_object6000:head# 2024-05-15T23:40:14.688 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.688 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e3eb625:::benchmark_data_smithi090_166897_object56296:head# 2024-05-15T23:40:14.688 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.688 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e3f7340:::benchmark_data_smithi090_166897_object64766:head# 2024-05-15T23:40:14.688 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.689 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e3f8ab6:::benchmark_data_smithi090_166897_object67452:head# 2024-05-15T23:40:14.689 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.689 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e419fd3:::benchmark_data_smithi090_166897_object16104:head# 2024-05-15T23:40:14.689 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.689 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e41b206:::benchmark_data_smithi090_166897_object24325:head# 2024-05-15T23:40:14.689 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.689 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e42fb97:::benchmark_data_smithi090_166897_object44244:head# 2024-05-15T23:40:14.689 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.689 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e432a92:::benchmark_data_smithi090_166897_object43308:head# 2024-05-15T23:40:14.689 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.689 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e43aa6c:::benchmark_data_smithi090_166897_object41649:head# 2024-05-15T23:40:14.689 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.689 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e444802:::benchmark_data_smithi090_166897_object67054:head# 2024-05-15T23:40:14.689 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.690 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e45eb78:::benchmark_data_smithi090_166897_object737:head# 2024-05-15T23:40:14.690 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.690 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e4640fa:::benchmark_data_smithi090_166897_object24457:head# 2024-05-15T23:40:14.690 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.690 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e466ed3:::benchmark_data_smithi090_166897_object56139:head# 2024-05-15T23:40:14.690 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.690 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e46744e:::benchmark_data_smithi090_166897_object15354:head# 2024-05-15T23:40:14.690 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.690 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e4c4764:::benchmark_data_smithi090_166897_object41667:head# 2024-05-15T23:40:14.690 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.690 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e4dd709:::benchmark_data_smithi090_166897_object49240:head# 2024-05-15T23:40:14.690 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.690 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e4f4c48:::benchmark_data_smithi090_166897_object17121:head# 2024-05-15T23:40:14.690 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e4fd00a:::benchmark_data_smithi090_166897_object14525:head# 2024-05-15T23:40:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e5298a3:::benchmark_data_smithi090_166897_object9027:head# 2024-05-15T23:40:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e54c851:::benchmark_data_smithi090_166897_object3556:head# 2024-05-15T23:40:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e54e068:::benchmark_data_smithi090_166897_object28431:head# 2024-05-15T23:40:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e555c89:::benchmark_data_smithi090_166897_object15114:head# 2024-05-15T23:40:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e55f55e:::benchmark_data_smithi090_166897_object54839:head# 2024-05-15T23:40:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e5607c6:::benchmark_data_smithi090_166897_object12938:head# 2024-05-15T23:40:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e5618d9:::benchmark_data_smithi090_166897_object34086:head# 2024-05-15T23:40:14.692 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.692 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e5805fa:::benchmark_data_smithi090_166897_object1837:head# 2024-05-15T23:40:14.692 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.692 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e58b76a:::benchmark_data_smithi090_166897_object64762:head# 2024-05-15T23:40:14.692 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.692 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e59e89e:::benchmark_data_smithi090_166897_object10612:head# 2024-05-15T23:40:14.692 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.692 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e5a248a:::benchmark_data_smithi090_166897_object54177:head# 2024-05-15T23:40:14.692 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.692 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e5a69cb:::benchmark_data_smithi090_166897_object61298:head# 2024-05-15T23:40:14.692 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.692 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e5b8c59:::benchmark_data_smithi090_166897_object46161:head# 2024-05-15T23:40:14.692 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.692 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e5dfd50:::benchmark_data_smithi090_166897_object13099:head# 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e5e81ed:::benchmark_data_smithi090_166897_object67073:head# 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e5f54bd:::benchmark_data_smithi090_166897_object35436:head# 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e5fdfa2:::benchmark_data_smithi090_166897_object42392:head# 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e600361:::benchmark_data_smithi090_166897_object14388:head# 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e61fe16:::benchmark_data_smithi090_166897_object58206:head# 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e62723f:::benchmark_data_smithi090_166897_object60308:head# 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e63a3e5:::benchmark_data_smithi090_166897_object60640:head# 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.693 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e63bee9:::benchmark_data_smithi090_166897_object19427:head# 2024-05-15T23:40:14.694 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.694 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e63d8f0:::benchmark_data_smithi090_166897_object19004:head# 2024-05-15T23:40:14.694 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.694 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e64b377:::benchmark_data_smithi090_166897_object29020:head# 2024-05-15T23:40:14.694 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.694 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e65e39e:::benchmark_data_smithi090_166897_object16144:head# 2024-05-15T23:40:14.694 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.694 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e6731b5:::benchmark_data_smithi090_166897_object22765:head# 2024-05-15T23:40:14.694 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.694 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e6a01d7:::benchmark_data_smithi090_166897_object34260:head# 2024-05-15T23:40:14.694 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.694 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e6ac399:::benchmark_data_smithi090_166897_object37914:head# 2024-05-15T23:40:14.694 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.694 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e6c8fd1:::benchmark_data_smithi090_166897_object18362:head# 2024-05-15T23:40:14.695 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.695 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e6e1e0b:::benchmark_data_smithi090_166897_object16627:head# 2024-05-15T23:40:14.695 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.695 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e6ed39f:::benchmark_data_smithi090_166897_object5311:head# 2024-05-15T23:40:14.695 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.695 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e70434f:::benchmark_data_smithi090_166897_object1418:head# 2024-05-15T23:40:14.695 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.695 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e709f4b:::benchmark_data_smithi090_166897_object36011:head# 2024-05-15T23:40:14.695 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.695 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e70ffd6:::benchmark_data_smithi090_166897_object17625:head# 2024-05-15T23:40:14.695 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.695 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e7184b8:::benchmark_data_smithi090_166897_object17578:head# 2024-05-15T23:40:14.695 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.696 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e7277f7:::benchmark_data_smithi090_166897_object60284:head# 2024-05-15T23:40:14.696 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.696 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e727c8d:::benchmark_data_smithi090_166897_object23478:head# 2024-05-15T23:40:14.696 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.696 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e72f016:::benchmark_data_smithi090_166897_object10231:head# 2024-05-15T23:40:14.696 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.696 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e73dd49:::benchmark_data_smithi090_166897_object51083:head# 2024-05-15T23:40:14.696 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.696 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e73df2c:::benchmark_data_smithi090_166897_object37528:head# 2024-05-15T23:40:14.696 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.696 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e74bd7b:::benchmark_data_smithi090_166897_object59304:head# 2024-05-15T23:40:14.696 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.696 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e75b745:::benchmark_data_smithi090_166897_object18151:head# 2024-05-15T23:40:14.696 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.696 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e75bed5:::benchmark_data_smithi090_166897_object11153:head# 2024-05-15T23:40:14.697 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.697 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e75bf43:::benchmark_data_smithi090_166897_object29856:head# 2024-05-15T23:40:14.697 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.697 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e774750:::benchmark_data_smithi090_166897_object60846:head# 2024-05-15T23:40:14.697 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.697 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e7760ac:::benchmark_data_smithi090_166897_object40030:head# 2024-05-15T23:40:14.697 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.697 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e788b51:::benchmark_data_smithi090_166897_object69620:head# 2024-05-15T23:40:14.697 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.697 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e7899db:::benchmark_data_smithi090_166897_object12140:head# 2024-05-15T23:40:14.697 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.697 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e797e14:::benchmark_data_smithi090_166897_object3754:head# 2024-05-15T23:40:14.697 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.697 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e79ebfc:::benchmark_data_smithi090_166897_object45656:head# 2024-05-15T23:40:14.697 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.698 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e7a9062:::benchmark_data_smithi090_166897_object16665:head# 2024-05-15T23:40:14.698 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.698 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e7ad3f7:::benchmark_data_smithi090_166897_object29671:head# 2024-05-15T23:40:14.698 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.698 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e7c28a6:::benchmark_data_smithi090_166897_object54315:head# 2024-05-15T23:40:14.698 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.698 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e7d4595:::benchmark_data_smithi090_166897_object202:head# 2024-05-15T23:40:14.698 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.698 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e7df748:::benchmark_data_smithi090_166897_object25226:head# 2024-05-15T23:40:14.698 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.698 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e7eb984:::benchmark_data_smithi090_166897_object9637:head# 2024-05-15T23:40:14.698 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.698 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e7f3be2:::benchmark_data_smithi090_166897_object1121:head# 2024-05-15T23:40:14.698 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.699 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e806816:::benchmark_data_smithi090_166897_object61651:head# 2024-05-15T23:40:14.699 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.699 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e82c515:::benchmark_data_smithi090_166897_object9246:head# 2024-05-15T23:40:14.699 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.699 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e83205a:::benchmark_data_smithi090_166897_object56808:head# 2024-05-15T23:40:14.699 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.699 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e83a0e8:::benchmark_data_smithi090_166897_object53076:head# 2024-05-15T23:40:14.699 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.699 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e85feda:::benchmark_data_smithi090_166897_object2698:head# 2024-05-15T23:40:14.699 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.699 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e86f458:::benchmark_data_smithi090_166897_object57593:head# 2024-05-15T23:40:14.699 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.699 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e87a1bd:::benchmark_data_smithi090_166897_object45577:head# 2024-05-15T23:40:14.699 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.700 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e87c46d:::benchmark_data_smithi090_166897_object62036:head# 2024-05-15T23:40:14.700 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.700 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e88d2b6:::benchmark_data_smithi090_166897_object68138:head# 2024-05-15T23:40:14.700 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.700 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e892803:::benchmark_data_smithi090_166897_object66155:head# 2024-05-15T23:40:14.700 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.700 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e8a989d:::benchmark_data_smithi090_166897_object14683:head# 2024-05-15T23:40:14.700 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.700 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e8b552e:::benchmark_data_smithi090_166897_object56209:head# 2024-05-15T23:40:14.700 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.700 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e8e7f23:::benchmark_data_smithi090_166897_object48479:head# 2024-05-15T23:40:14.700 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.700 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e8ef157:::benchmark_data_smithi090_166897_object64764:head# 2024-05-15T23:40:14.700 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.700 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e90eadf:::benchmark_data_smithi090_166897_object34092:head# 2024-05-15T23:40:14.701 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.701 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e916b0c:::benchmark_data_smithi090_166897_object46689:head# 2024-05-15T23:40:14.701 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.701 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e9260ed:::benchmark_data_smithi090_166897_object38703:head# 2024-05-15T23:40:14.701 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.701 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e947e91:::benchmark_data_smithi090_166897_object46356:head# 2024-05-15T23:40:14.701 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.701 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e95336f:::benchmark_data_smithi090_166897_object67198:head# 2024-05-15T23:40:14.701 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.701 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e965edc:::benchmark_data_smithi090_166897_object1763:head# 2024-05-15T23:40:14.701 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.701 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e96acf0:::benchmark_data_smithi090_166897_object14741:head# 2024-05-15T23:40:14.701 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.701 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e972498:::benchmark_data_smithi090_166897_object51849:head# 2024-05-15T23:40:14.701 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.702 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e9757a7:::benchmark_data_smithi090_166897_object18924:head# 2024-05-15T23:40:14.702 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.702 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e998181:::benchmark_data_smithi090_166897_object45591:head# 2024-05-15T23:40:14.702 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.702 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e9a8a46:::benchmark_data_smithi090_166897_object54846:head# 2024-05-15T23:40:14.702 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.702 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e9ea2a4:::benchmark_data_smithi090_166897_object31194:head# 2024-05-15T23:40:14.702 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.702 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6e9f3611:::benchmark_data_smithi090_166897_object5820:head# 2024-05-15T23:40:14.702 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.702 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ea0364b:::benchmark_data_smithi090_166897_object1045:head# 2024-05-15T23:40:14.702 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.702 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ea41d50:::benchmark_data_smithi090_166897_object17818:head# 2024-05-15T23:40:14.702 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.703 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ea4df27:::benchmark_data_smithi090_166897_object68328:head# 2024-05-15T23:40:14.703 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.703 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ea5df4a:::benchmark_data_smithi090_166897_object23511:head# 2024-05-15T23:40:14.703 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.703 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ea67f92:::benchmark_data_smithi090_166897_object14805:head# 2024-05-15T23:40:14.703 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.703 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ea71437:::benchmark_data_smithi090_166897_object6589:head# 2024-05-15T23:40:14.703 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.703 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ea74f66:::benchmark_data_smithi090_166897_object31736:head# 2024-05-15T23:40:14.703 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.703 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ea77687:::benchmark_data_smithi090_166897_object68571:head# 2024-05-15T23:40:14.703 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.703 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ea8538f:::benchmark_data_smithi090_166897_object17117:head# 2024-05-15T23:40:14.703 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.703 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eaacb99:::benchmark_data_smithi090_166897_object40759:head# 2024-05-15T23:40:14.704 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.704 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eabcd7f:::benchmark_data_smithi090_166897_object15927:head# 2024-05-15T23:40:14.704 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.704 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eabd862:::benchmark_data_smithi090_166897_object10513:head# 2024-05-15T23:40:14.704 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.704 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eaca7ba:::benchmark_data_smithi090_166897_object18628:head# 2024-05-15T23:40:14.704 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.704 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ead7e6f:::benchmark_data_smithi090_166897_object50169:head# 2024-05-15T23:40:14.704 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.704 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eaf2f69:::benchmark_data_smithi090_166897_object11491:head# 2024-05-15T23:40:14.704 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.704 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eaf500b:::benchmark_data_smithi090_166897_object8238:head# 2024-05-15T23:40:14.704 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.704 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eb04cf2:::benchmark_data_smithi090_166897_object44718:head# 2024-05-15T23:40:14.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.705 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eb131f4:::benchmark_data_smithi090_166897_object14008:head# 2024-05-15T23:40:14.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.705 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eb40d8a:::benchmark_data_smithi090_166897_object60545:head# 2024-05-15T23:40:14.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.705 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eb4f4ad:::benchmark_data_smithi090_166897_object101:head# 2024-05-15T23:40:14.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.705 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eb618f3:::benchmark_data_smithi090_166897_object13513:head# 2024-05-15T23:40:14.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.705 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eb6e531:::benchmark_data_smithi090_166897_object20768:head# 2024-05-15T23:40:14.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.705 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eb89ea5:::benchmark_data_smithi090_166897_object34832:head# 2024-05-15T23:40:14.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.705 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eba9c25:::benchmark_data_smithi090_166897_object27702:head# 2024-05-15T23:40:14.705 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.706 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ebbf290:::benchmark_data_smithi090_166897_object2336:head# 2024-05-15T23:40:14.706 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.706 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ebea73b:::benchmark_data_smithi090_166897_object62761:head# 2024-05-15T23:40:14.706 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.706 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ebeda89:::benchmark_data_smithi090_166897_object15958:head# 2024-05-15T23:40:14.706 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.706 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ebffb9b:::benchmark_data_smithi090_166897_object61799:head# 2024-05-15T23:40:14.706 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.706 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec0095f:::benchmark_data_smithi090_166897_object10003:head# 2024-05-15T23:40:14.706 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.706 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec04b52:::benchmark_data_smithi090_166897_object54080:head# 2024-05-15T23:40:14.706 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.706 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec07b07:::benchmark_data_smithi090_166897_object67542:head# 2024-05-15T23:40:14.706 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.707 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec1e1a1:::benchmark_data_smithi090_166897_object58829:head# 2024-05-15T23:40:14.707 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.707 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec28d16:::benchmark_data_smithi090_166897_object54406:head# 2024-05-15T23:40:14.707 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.707 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec441ae:::benchmark_data_smithi090_166897_object51468:head# 2024-05-15T23:40:14.707 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.707 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec5e9d5:::benchmark_data_smithi090_166897_object9554:head# 2024-05-15T23:40:14.707 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.707 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec65734:::benchmark_data_smithi090_166897_object15875:head# 2024-05-15T23:40:14.707 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.707 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec65998:::benchmark_data_smithi090_166897_object61782:head# 2024-05-15T23:40:14.707 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.707 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec7002e:::benchmark_data_smithi090_166897_object45508:head# 2024-05-15T23:40:14.707 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.707 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec719bd:::benchmark_data_smithi090_166897_object925:head# 2024-05-15T23:40:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec90cf2:::benchmark_data_smithi090_166897_object57102:head# 2024-05-15T23:40:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec96617:::benchmark_data_smithi090_166897_object66096:head# 2024-05-15T23:40:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec9bef4:::benchmark_data_smithi090_166897_object1801:head# 2024-05-15T23:40:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec9dd23:::benchmark_data_smithi090_166897_object4414:head# 2024-05-15T23:40:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ec9e212:::benchmark_data_smithi090_166897_object64949:head# 2024-05-15T23:40:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ecac741:::benchmark_data_smithi090_166897_object42951:head# 2024-05-15T23:40:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ecb86b5:::benchmark_data_smithi090_166897_object27005:head# 2024-05-15T23:40:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eccc008:::benchmark_data_smithi090_166897_object58659:head# 2024-05-15T23:40:14.709 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eccf057:::benchmark_data_smithi090_166897_object43672:head# 2024-05-15T23:40:14.709 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ecd389a:::benchmark_data_smithi090_166897_object34298:head# 2024-05-15T23:40:14.709 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ece7308:::benchmark_data_smithi090_166897_object36886:head# 2024-05-15T23:40:14.709 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ed1230a:::benchmark_data_smithi090_166897_object40000:head# 2024-05-15T23:40:14.709 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ed158eb:::benchmark_data_smithi090_166897_object41397:head# 2024-05-15T23:40:14.709 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ed198f5:::benchmark_data_smithi090_166897_object38975:head# 2024-05-15T23:40:14.709 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.709 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ed1b766:::benchmark_data_smithi090_166897_object18822:head# 2024-05-15T23:40:14.710 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.710 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ed1e1ac:::benchmark_data_smithi090_166897_object61038:head# 2024-05-15T23:40:14.710 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.710 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ed30383:::benchmark_data_smithi090_166897_object64323:head# 2024-05-15T23:40:14.710 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.710 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ed3f192:::benchmark_data_smithi090_166897_object50468:head# 2024-05-15T23:40:14.710 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.710 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ed652f9:::benchmark_data_smithi090_166897_object22562:head# 2024-05-15T23:40:14.710 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.710 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ed9b4ca:::benchmark_data_smithi090_166897_object10532:head# 2024-05-15T23:40:14.710 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.710 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eda48ad:::benchmark_data_smithi090_166897_object60224:head# 2024-05-15T23:40:14.710 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.710 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6edb55ea:::benchmark_data_smithi090_166897_object1430:head# 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6edb7717:::benchmark_data_smithi090_166897_object57604:head# 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6edd245c:::benchmark_data_smithi090_166897_object42021:head# 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eddb6d7:::benchmark_data_smithi090_166897_object31071:head# 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6edead0b:::benchmark_data_smithi090_166897_object34318:head# 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ee0c196:::benchmark_data_smithi090_166897_object11439:head# 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ee23ebe:::benchmark_data_smithi090_166897_object9206:head# 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ee2efdd:::benchmark_data_smithi090_166897_object49369:head# 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.711 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ee33fbc:::benchmark_data_smithi090_166897_object27933:head# 2024-05-15T23:40:14.712 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.712 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ee48f84:::benchmark_data_smithi090_166897_object16465:head# 2024-05-15T23:40:14.712 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.712 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ee4c071:::benchmark_data_smithi090_166897_object61077:head# 2024-05-15T23:40:14.712 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.712 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ee654a0:::benchmark_data_smithi090_166897_object11837:head# 2024-05-15T23:40:14.712 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.712 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ee658d5:::benchmark_data_smithi090_166897_object53660:head# 2024-05-15T23:40:14.712 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.720 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ee76cfd:::benchmark_data_smithi090_166897_object42963:head# 2024-05-15T23:40:14.720 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.720 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ee7d07e:::benchmark_data_smithi090_166897_object63933:head# 2024-05-15T23:40:14.720 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ee7d29d:::benchmark_data_smithi090_166897_object2310:head# 2024-05-15T23:40:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eea65db:::benchmark_data_smithi090_166897_object69298:head# 2024-05-15T23:40:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eeb90e5:::benchmark_data_smithi090_166897_object17911:head# 2024-05-15T23:40:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eec1d4c:::benchmark_data_smithi090_166897_object5849:head# 2024-05-15T23:40:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eece698:::benchmark_data_smithi090_166897_object58364:head# 2024-05-15T23:40:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eef4ad5:::benchmark_data_smithi090_166897_object57651:head# 2024-05-15T23:40:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ef03240:::benchmark_data_smithi090_166897_object35312:head# 2024-05-15T23:40:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ef03bc1:::benchmark_data_smithi090_166897_object33957:head# 2024-05-15T23:40:14.722 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.722 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ef0a93e:::benchmark_data_smithi090_166897_object11471:head# 2024-05-15T23:40:14.722 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.722 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ef13d7b:::benchmark_data_smithi090_166897_object59563:head# 2024-05-15T23:40:14.722 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.722 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ef37f80:::benchmark_data_smithi090_166897_object41829:head# 2024-05-15T23:40:14.722 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.722 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ef3b22c:::benchmark_data_smithi090_166897_object30656:head# 2024-05-15T23:40:14.722 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.722 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ef3bf21:::benchmark_data_smithi090_166897_object60615:head# 2024-05-15T23:40:14.722 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.722 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ef54376:::benchmark_data_smithi090_166897_object473:head# 2024-05-15T23:40:14.722 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.722 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ef57ad2:::benchmark_data_smithi090_166897_object28527:head# 2024-05-15T23:40:14.723 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.723 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ef61920:::benchmark_data_smithi090_166897_object17879:head# 2024-05-15T23:40:14.723 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.723 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ef6e4ec:::benchmark_data_smithi090_166897_object31087:head# 2024-05-15T23:40:14.723 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.723 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ef8d6c3:::benchmark_data_smithi090_166897_object27584:head# 2024-05-15T23:40:14.723 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.723 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ef94693:::benchmark_data_smithi090_166897_object29502:head# 2024-05-15T23:40:14.723 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.723 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6efdc544:::benchmark_data_smithi090_166897_object53337:head# 2024-05-15T23:40:14.723 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.723 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eff220f:::benchmark_data_smithi090_166897_object65565:head# 2024-05-15T23:40:14.723 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.723 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6eff49d6:::benchmark_data_smithi090_166897_object41091:head# 2024-05-15T23:40:14.724 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.724 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f001027:::benchmark_data_smithi090_166897_object7634:head# 2024-05-15T23:40:14.724 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.724 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f01a764:::benchmark_data_smithi090_166897_object30195:head# 2024-05-15T23:40:14.724 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.724 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f0344d1:::benchmark_data_smithi090_166897_object57401:head# 2024-05-15T23:40:14.724 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.724 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f039d95:::benchmark_data_smithi090_166897_object18411:head# 2024-05-15T23:40:14.724 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.724 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f048071:::benchmark_data_smithi090_166897_object59664:head# 2024-05-15T23:40:14.724 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.724 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f0483e7:::benchmark_data_smithi090_166897_object25268:head# 2024-05-15T23:40:14.724 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.724 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f055ab7:::benchmark_data_smithi090_166897_object22118:head# 2024-05-15T23:40:14.725 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.725 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f056177:::benchmark_data_smithi090_166897_object22695:head# 2024-05-15T23:40:14.725 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.725 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f0587c7:::benchmark_data_smithi090_166897_object19290:head# 2024-05-15T23:40:14.725 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.725 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f061088:::benchmark_data_smithi090_166897_object70892:head# 2024-05-15T23:40:14.725 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.725 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f07b707:::benchmark_data_smithi090_166897_object28047:head# 2024-05-15T23:40:14.725 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.725 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f07e5f6:::benchmark_data_smithi090_166897_object13247:head# 2024-05-15T23:40:14.725 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.725 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f08182e:::benchmark_data_smithi090_166897_object10555:head# 2024-05-15T23:40:14.725 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.725 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f0a9d21:::benchmark_data_smithi090_166897_object19200:head# 2024-05-15T23:40:14.725 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.726 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f0b1e82:::benchmark_data_smithi090_166897_object68933:head# 2024-05-15T23:40:14.726 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.726 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f0b2245:::benchmark_data_smithi090_166897_object39149:head# 2024-05-15T23:40:14.726 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.726 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f0ba39f:::benchmark_data_smithi090_166897_object65184:head# 2024-05-15T23:40:14.726 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.726 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f0c421c:::benchmark_data_smithi090_166897_object31204:head# 2024-05-15T23:40:14.726 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.726 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f0c9c89:::benchmark_data_smithi090_166897_object22179:head# 2024-05-15T23:40:14.726 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.726 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f0d962e:::benchmark_data_smithi090_166897_object53147:head# 2024-05-15T23:40:14.726 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.726 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f0eb2ca:::benchmark_data_smithi090_166897_object58619:head# 2024-05-15T23:40:14.727 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.727 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f0ebc1e:::benchmark_data_smithi090_166897_object36914:head# 2024-05-15T23:40:14.727 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.727 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f0f3fc8:::benchmark_data_smithi090_166897_object26580:head# 2024-05-15T23:40:14.727 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.727 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f0f7af2:::benchmark_data_smithi090_166897_object58264:head# 2024-05-15T23:40:14.727 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.727 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f10edf9:::benchmark_data_smithi090_166897_object33878:head# 2024-05-15T23:40:14.727 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.727 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f111177:::benchmark_data_smithi090_166897_object41769:head# 2024-05-15T23:40:14.727 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.727 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f11370f:::benchmark_data_smithi090_166897_object51267:head# 2024-05-15T23:40:14.727 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.727 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f11e755:::benchmark_data_smithi090_166897_object57520:head# 2024-05-15T23:40:14.727 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.728 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f126f1b:::benchmark_data_smithi090_166897_object32584:head# 2024-05-15T23:40:14.728 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.728 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f128a67:::benchmark_data_smithi090_166897_object68577:head# 2024-05-15T23:40:14.728 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.728 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f1292c3:::benchmark_data_smithi090_166897_object29206:head# 2024-05-15T23:40:14.728 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.728 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f12b9f6:::benchmark_data_smithi090_166897_object15266:head# 2024-05-15T23:40:14.728 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.728 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f13679c:::benchmark_data_smithi090_166897_object29777:head# 2024-05-15T23:40:14.728 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.728 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f1493d8:::benchmark_data_smithi090_166897_object16292:head# 2024-05-15T23:40:14.728 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.728 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f15721d:::benchmark_data_smithi090_166897_object5469:head# 2024-05-15T23:40:14.728 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f16e374:::benchmark_data_smithi090_166897_object7983:head# 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f175544:::benchmark_data_smithi090_166897_object15124:head# 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f18d8f8:::benchmark_data_smithi090_166897_object65953:head# 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f1bb615:::benchmark_data_smithi090_166897_object62858:head# 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f1be369:::benchmark_data_smithi090_166897_object19920:head# 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f1cec62:::benchmark_data_smithi090_166897_object12415:head# 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f1d270f:::benchmark_data_smithi090_166897_object45238:head# 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f1e7331:::benchmark_data_smithi090_166897_object71017:head# 2024-05-15T23:40:14.729 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.730 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f1f1f2c:::benchmark_data_smithi090_166897_object13347:head# 2024-05-15T23:40:14.730 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.730 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f209187:::benchmark_data_smithi090_166897_object63086:head# 2024-05-15T23:40:14.730 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.730 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f20fadc:::benchmark_data_smithi090_166897_object27652:head# 2024-05-15T23:40:14.730 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.730 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f213e76:::benchmark_data_smithi090_166897_object15388:head# 2024-05-15T23:40:14.730 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.730 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f223250:::benchmark_data_smithi090_166897_object29289:head# 2024-05-15T23:40:14.730 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.730 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f223d30:::benchmark_data_smithi090_166897_object15803:head# 2024-05-15T23:40:14.730 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.730 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f234f55:::benchmark_data_smithi090_166897_object52088:head# 2024-05-15T23:40:14.730 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.731 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f237f91:::benchmark_data_smithi090_166897_object66346:head# 2024-05-15T23:40:14.731 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.731 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f26c6d4:::benchmark_data_smithi090_166897_object11877:head# 2024-05-15T23:40:14.731 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.731 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f27b6fb:::benchmark_data_smithi090_166897_object34197:head# 2024-05-15T23:40:14.731 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.731 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f289772:::benchmark_data_smithi090_166897_object55324:head# 2024-05-15T23:40:14.731 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.731 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f28d7d5:::benchmark_data_smithi090_166897_object18657:head# 2024-05-15T23:40:14.731 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.731 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f29f9a8:::benchmark_data_smithi090_166897_object56668:head# 2024-05-15T23:40:14.731 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.731 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f2b271f:::benchmark_data_smithi090_166897_object40868:head# 2024-05-15T23:40:14.731 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.731 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f2b6cfb:::benchmark_data_smithi090_166897_object59668:head# 2024-05-15T23:40:14.732 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.732 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f2b9cd8:::benchmark_data_smithi090_166897_object61712:head# 2024-05-15T23:40:14.732 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.732 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f2bca82:::benchmark_data_smithi090_166897_object57055:head# 2024-05-15T23:40:14.732 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.732 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f2e31c0:::benchmark_data_smithi090_166897_object23108:head# 2024-05-15T23:40:14.732 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.732 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:14.743 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f2eee61:::benchmark_data_smithi090_166897_object25104:head# 2024-05-15T23:40:14.743 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.743 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f2f1e77:::benchmark_data_smithi090_166897_object25592:head# 2024-05-15T23:40:14.743 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.743 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f2f48e6:::benchmark_data_smithi090_166897_object8910:head# 2024-05-15T23:40:14.743 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.743 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f30e34f:::benchmark_data_smithi090_166897_object48092:head# 2024-05-15T23:40:14.743 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.743 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f3197f4:::benchmark_data_smithi090_166897_object55353:head# 2024-05-15T23:40:14.743 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.743 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f33a1e2:::benchmark_data_smithi090_166897_object54500:head# 2024-05-15T23:40:14.743 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f345741:::benchmark_data_smithi090_166897_object47386:head# 2024-05-15T23:40:14.744 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f348b89:::benchmark_data_smithi090_166897_object69084:head# 2024-05-15T23:40:14.744 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f36b664:::benchmark_data_smithi090_166897_object62816:head# 2024-05-15T23:40:14.744 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f36d68e:::benchmark_data_smithi090_166897_object45025:head# 2024-05-15T23:40:14.744 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f3d6418:::benchmark_data_smithi090_166897_object30935:head# 2024-05-15T23:40:14.744 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f3ea0c3:::benchmark_data_smithi090_166897_object65360:head# 2024-05-15T23:40:14.744 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f43e095:::benchmark_data_smithi090_166897_object39855:head# 2024-05-15T23:40:14.744 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.744 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f43fa18:::benchmark_data_smithi090_166897_object27124:head# 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f4404f4:::benchmark_data_smithi090_166897_object61228:head# 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f450f0f:::benchmark_data_smithi090_166897_object58123:head# 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f458a66:::benchmark_data_smithi090_166897_object805:head# 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f4593e8:::benchmark_data_smithi090_166897_object66970:head# 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f478924:::benchmark_data_smithi090_166897_object10441:head# 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f47b74c:::benchmark_data_smithi090_166897_object15819:head# 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f482c99:::benchmark_data_smithi090_166897_object15947:head# 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.745 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f489482:::benchmark_data_smithi090_166897_object53317:head# 2024-05-15T23:40:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f4a09c7:::benchmark_data_smithi090_166897_object58108:head# 2024-05-15T23:40:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f4addc2:::benchmark_data_smithi090_166897_object29702:head# 2024-05-15T23:40:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f4ade12:::benchmark_data_smithi090_166897_object7100:head# 2024-05-15T23:40:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f4b8341:::benchmark_data_smithi090_166897_object38115:head# 2024-05-15T23:40:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f4b8632:::benchmark_data_smithi090_166897_object57072:head# 2024-05-15T23:40:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f4eec9a:::benchmark_data_smithi090_166897_object60017:head# 2024-05-15T23:40:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f4f8a76:::benchmark_data_smithi090_166897_object32477:head# 2024-05-15T23:40:14.747 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.747 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f4fa976:::benchmark_data_smithi090_166897_object55136:head# 2024-05-15T23:40:14.747 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.747 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f50888c:::benchmark_data_smithi090_166897_object24031:head# 2024-05-15T23:40:14.747 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.747 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f51cd99:::benchmark_data_smithi090_166897_object36031:head# 2024-05-15T23:40:14.747 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.747 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f51ee1e:::benchmark_data_smithi090_166897_object31358:head# 2024-05-15T23:40:14.747 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.747 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f521299:::benchmark_data_smithi090_166897_object13895:head# 2024-05-15T23:40:14.747 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.747 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f52985a:::benchmark_data_smithi090_166897_object26260:head# 2024-05-15T23:40:14.747 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.747 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f53da3e:::benchmark_data_smithi090_166897_object43628:head# 2024-05-15T23:40:14.747 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f540230:::benchmark_data_smithi090_166897_object59164:head# 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f54f412:::benchmark_data_smithi090_166897_object25903:head# 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f550cf7:::benchmark_data_smithi090_166897_object47337:head# 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f56a44d:::benchmark_data_smithi090_166897_object39709:head# 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f5923f9:::benchmark_data_smithi090_166897_object30770:head# 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f598eef:::benchmark_data_smithi090_166897_object2120:head# 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f599be6:::benchmark_data_smithi090_166897_object56918:head# 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f59bb5e:::benchmark_data_smithi090_166897_object24201:head# 2024-05-15T23:40:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f59e9ae:::benchmark_data_smithi090_166897_object22575:head# 2024-05-15T23:40:14.749 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f5a2b27:::benchmark_data_smithi090_166897_object5282:head# 2024-05-15T23:40:14.749 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f5b5ed9:::benchmark_data_smithi090_166897_object35829:head# 2024-05-15T23:40:14.749 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f5b6aaa:::benchmark_data_smithi090_166897_object32489:head# 2024-05-15T23:40:14.749 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f5d42a0:::benchmark_data_smithi090_166897_object5780:head# 2024-05-15T23:40:14.749 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f5fd926:::benchmark_data_smithi090_166897_object12813:head# 2024-05-15T23:40:14.749 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f5fed82:::benchmark_data_smithi090_166897_object66850:head# 2024-05-15T23:40:14.749 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.749 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f6060de:::benchmark_data_smithi090_166897_object35561:head# 2024-05-15T23:40:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f668d03:::benchmark_data_smithi090_166897_object44324:head# 2024-05-15T23:40:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f669bac:::benchmark_data_smithi090_166897_object63999:head# 2024-05-15T23:40:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f66a8d3:::benchmark_data_smithi090_166897_object57385:head# 2024-05-15T23:40:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f68e114:::benchmark_data_smithi090_166897_object9845:head# 2024-05-15T23:40:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f6a5a33:::benchmark_data_smithi090_166897_object25590:head# 2024-05-15T23:40:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f6b3e00:::benchmark_data_smithi090_166897_object35301:head# 2024-05-15T23:40:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f6c7ed0:::benchmark_data_smithi090_166897_object61994:head# 2024-05-15T23:40:14.751 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.751 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f6ce133:::benchmark_data_smithi090_166897_object32951:head# 2024-05-15T23:40:14.751 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.751 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f6cfc95:::benchmark_data_smithi090_166897_object19031:head# 2024-05-15T23:40:14.751 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.751 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f6edb02:::benchmark_data_smithi090_166897_object28346:head# 2024-05-15T23:40:14.751 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.751 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f6f75fb:::benchmark_data_smithi090_166897_object49274:head# 2024-05-15T23:40:14.751 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.751 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f6fe6e2:::benchmark_data_smithi090_166897_object63466:head# 2024-05-15T23:40:14.751 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.751 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f702033:::benchmark_data_smithi090_166897_object39371:head# 2024-05-15T23:40:14.751 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.751 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f71b8a4:::benchmark_data_smithi090_166897_object22250:head# 2024-05-15T23:40:14.752 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.752 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f71f297:::benchmark_data_smithi090_166897_object64453:head# 2024-05-15T23:40:14.752 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.752 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f71fe85:::benchmark_data_smithi090_166897_object30784:head# 2024-05-15T23:40:14.752 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.752 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f726b6d:::benchmark_data_smithi090_166897_object21487:head# 2024-05-15T23:40:14.752 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.752 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f76279a:::benchmark_data_smithi090_166897_object56836:head# 2024-05-15T23:40:14.752 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.752 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f771546:::benchmark_data_smithi090_166897_object42840:head# 2024-05-15T23:40:14.752 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.752 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f77caf8:::benchmark_data_smithi090_166897_object28235:head# 2024-05-15T23:40:14.752 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.752 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f7886bc:::benchmark_data_smithi090_166897_object52927:head# 2024-05-15T23:40:14.752 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f78dafb:::benchmark_data_smithi090_166897_object19460:head# 2024-05-15T23:40:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f7b8a65:::benchmark_data_smithi090_166897_object62549:head# 2024-05-15T23:40:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f7bfc4f:::benchmark_data_smithi090_166897_object29592:head# 2024-05-15T23:40:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f7c048c:::benchmark_data_smithi090_166897_object4950:head# 2024-05-15T23:40:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f7d0502:::benchmark_data_smithi090_166897_object28724:head# 2024-05-15T23:40:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f7d4815:::benchmark_data_smithi090_166897_object53049:head# 2024-05-15T23:40:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f7e3af2:::benchmark_data_smithi090_166897_object1434:head# 2024-05-15T23:40:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f7eee46:::benchmark_data_smithi090_166897_object9251:head# 2024-05-15T23:40:14.754 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.754 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f7f397d:::benchmark_data_smithi090_166897_object44018:head# 2024-05-15T23:40:14.754 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.754 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f7f519b:::benchmark_data_smithi090_166897_object12926:head# 2024-05-15T23:40:14.754 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.754 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f7ff5e7:::benchmark_data_smithi090_166897_object26922:head# 2024-05-15T23:40:14.754 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.754 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f815e23:::benchmark_data_smithi090_166897_object50701:head# 2024-05-15T23:40:14.754 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.754 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f817dfb:::benchmark_data_smithi090_166897_object64705:head# 2024-05-15T23:40:14.754 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.754 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f81a8f2:::benchmark_data_smithi090_166897_object48618:head# 2024-05-15T23:40:14.754 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.754 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f82c2fc:::benchmark_data_smithi090_166897_object28509:head# 2024-05-15T23:40:14.755 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.755 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f834b6a:::benchmark_data_smithi090_166897_object7122:head# 2024-05-15T23:40:14.755 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.755 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f878734:::benchmark_data_smithi090_166897_object19263:head# 2024-05-15T23:40:14.755 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.755 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f895b24:::benchmark_data_smithi090_166897_object69212:head# 2024-05-15T23:40:14.755 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.755 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f8a4482:::benchmark_data_smithi090_166897_object67454:head# 2024-05-15T23:40:14.755 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.755 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f8d4432:::benchmark_data_smithi090_166897_object41153:head# 2024-05-15T23:40:14.755 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.755 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f91d454:::benchmark_data_smithi090_166897_object27300:head# 2024-05-15T23:40:14.755 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.755 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f92ee47:::benchmark_data_smithi090_166897_object70292:head# 2024-05-15T23:40:14.756 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.756 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f935ca0:::benchmark_data_smithi090_166897_object58302:head# 2024-05-15T23:40:14.756 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.756 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f972d6f:::benchmark_data_smithi090_166897_object61902:head# 2024-05-15T23:40:14.756 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.756 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f986910:::benchmark_data_smithi090_166897_object10688:head# 2024-05-15T23:40:14.756 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.756 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f99924d:::benchmark_data_smithi090_166897_object16075:head# 2024-05-15T23:40:14.756 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.756 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f99ce09:::benchmark_data_smithi090_166897_object51894:head# 2024-05-15T23:40:14.756 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.756 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f9c0922:::benchmark_data_smithi090_166897_object6536:head# 2024-05-15T23:40:14.756 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.756 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f9c4049:::benchmark_data_smithi090_166897_object15753:head# 2024-05-15T23:40:14.757 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.757 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f9c4a08:::benchmark_data_smithi090_166897_object49539:head# 2024-05-15T23:40:14.757 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.757 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f9ce2bf:::benchmark_data_smithi090_166897_object1190:head# 2024-05-15T23:40:14.757 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.757 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:40:14.758 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f9ce43f:::benchmark_data_smithi090_166897_object23009:head# 2024-05-15T23:40:14.759 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.759 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f9cee34:::benchmark_data_smithi090_166897_object69274:head# 2024-05-15T23:40:14.759 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.759 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6f9d0832:::benchmark_data_smithi090_166897_object4704:head# 2024-05-15T23:40:14.759 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.759 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fa03ca0:::benchmark_data_smithi090_166897_object60259:head# 2024-05-15T23:40:14.759 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.759 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fa17408:::benchmark_data_smithi090_166897_object13168:head# 2024-05-15T23:40:14.759 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.759 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fa3ae2e:::benchmark_data_smithi090_166897_object42061:head# 2024-05-15T23:40:14.759 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.759 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fa5e9ce:::benchmark_data_smithi090_166897_object58648:head# 2024-05-15T23:40:14.759 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.760 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fa7248f:::benchmark_data_smithi090_166897_object51157:head# 2024-05-15T23:40:14.760 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.760 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fa86be8:::benchmark_data_smithi090_166897_object68220:head# 2024-05-15T23:40:14.760 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:14.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:40:14.771 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fabfcff:::benchmark_data_smithi090_166897_object21791:head# 2024-05-15T23:40:14.771 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.771 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6faf2c73:::benchmark_data_smithi090_166897_object55924:head# 2024-05-15T23:40:14.771 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.772 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6faf819c:::benchmark_data_smithi090_166897_object8059:head# 2024-05-15T23:40:14.772 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.772 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fb0cec4:::benchmark_data_smithi090_166897_object49171:head# 2024-05-15T23:40:14.772 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.772 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fb11174:::benchmark_data_smithi090_166897_object70685:head# 2024-05-15T23:40:14.772 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.772 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fb5586c:::benchmark_data_smithi090_166897_object69744:head# 2024-05-15T23:40:14.772 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.772 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fb68d31:::benchmark_data_smithi090_166897_object10327:head# 2024-05-15T23:40:14.772 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.772 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fb72762:::benchmark_data_smithi090_166897_object43835:head# 2024-05-15T23:40:14.772 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.772 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fb8c1ba:::benchmark_data_smithi090_166897_object52640:head# 2024-05-15T23:40:14.772 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.773 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fb9beed:::benchmark_data_smithi090_166897_object57898:head# 2024-05-15T23:40:14.773 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.773 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fbab61a:::benchmark_data_smithi090_166897_object42667:head# 2024-05-15T23:40:14.773 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.773 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fbacca1:::benchmark_data_smithi090_166897_object49894:head# 2024-05-15T23:40:14.773 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.773 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fbb0763:::benchmark_data_smithi090_166897_object6628:head# 2024-05-15T23:40:14.773 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.773 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fbb1fc2:::benchmark_data_smithi090_166897_object30139:head# 2024-05-15T23:40:14.773 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.773 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fbb373e:::benchmark_data_smithi090_166897_object33674:head# 2024-05-15T23:40:14.773 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.774 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fbb4e6d:::benchmark_data_smithi090_166897_object18284:head# 2024-05-15T23:40:14.774 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.774 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fbc8e24:::benchmark_data_smithi090_166897_object36899:head# 2024-05-15T23:40:14.774 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.774 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fbcf2f5:::benchmark_data_smithi090_166897_object13327:head# 2024-05-15T23:40:14.774 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.774 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fbd1623:::benchmark_data_smithi090_166897_object19254:head# 2024-05-15T23:40:14.774 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.774 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fbf6b14:::benchmark_data_smithi090_166897_object36916:head# 2024-05-15T23:40:14.774 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.774 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fbf9902:::benchmark_data_smithi090_166897_object19864:head# 2024-05-15T23:40:14.774 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.775 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fc01ce8:::benchmark_data_smithi090_166897_object8:head# 2024-05-15T23:40:14.775 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.775 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fc0ca26:::benchmark_data_smithi090_166897_object41715:head# 2024-05-15T23:40:14.775 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.775 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fc16afe:::benchmark_data_smithi090_166897_object70344:head# 2024-05-15T23:40:14.775 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.775 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fc175b5:::benchmark_data_smithi090_166897_object32829:head# 2024-05-15T23:40:14.775 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.775 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fc1adf5:::benchmark_data_smithi090_166897_object66792:head# 2024-05-15T23:40:14.775 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.775 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fc68521:::benchmark_data_smithi090_166897_object12332:head# 2024-05-15T23:40:14.775 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.775 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fc7bc5a:::benchmark_data_smithi090_166897_object8266:head# 2024-05-15T23:40:14.775 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.775 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fc7bdb2:::benchmark_data_smithi090_166897_object9757:head# 2024-05-15T23:40:14.776 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.776 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fc81bd8:::benchmark_data_smithi090_166897_object20365:head# 2024-05-15T23:40:14.776 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.776 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fc878d2:::benchmark_data_smithi090_166897_object22875:head# 2024-05-15T23:40:14.776 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.776 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fced7b0:::benchmark_data_smithi090_166897_object18106:head# 2024-05-15T23:40:14.776 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.776 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fcf4c11:::benchmark_data_smithi090_166897_object4498:head# 2024-05-15T23:40:14.776 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.776 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fcfe22b:::benchmark_data_smithi090_166897_object2146:head# 2024-05-15T23:40:14.776 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.776 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd049e1:::benchmark_data_smithi090_166897_object48480:head# 2024-05-15T23:40:14.776 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.776 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd15648:::benchmark_data_smithi090_166897_object35626:head# 2024-05-15T23:40:14.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.777 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd1da4f:::benchmark_data_smithi090_166897_object37650:head# 2024-05-15T23:40:14.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.777 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd213d5:::benchmark_data_smithi090_166897_object32547:head# 2024-05-15T23:40:14.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.777 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd262fd:::benchmark_data_smithi090_166897_object33888:head# 2024-05-15T23:40:14.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.777 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd45123:::benchmark_data_smithi090_166897_object27714:head# 2024-05-15T23:40:14.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.777 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd45fb5:::benchmark_data_smithi090_166897_object59742:head# 2024-05-15T23:40:14.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.777 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd4a0c1:::benchmark_data_smithi090_166897_object65152:head# 2024-05-15T23:40:14.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.777 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd50885:::benchmark_data_smithi090_166897_object63455:head# 2024-05-15T23:40:14.777 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.778 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd5c6c2:::benchmark_data_smithi090_166897_object18087:head# 2024-05-15T23:40:14.778 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.778 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd654e5:::benchmark_data_smithi090_166897_object41786:head# 2024-05-15T23:40:14.778 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.778 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd670f3:::benchmark_data_smithi090_166897_object20864:head# 2024-05-15T23:40:14.778 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.778 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd6a11e:::benchmark_data_smithi090_166897_object992:head# 2024-05-15T23:40:14.778 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.778 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd6ac88:::benchmark_data_smithi090_166897_object7436:head# 2024-05-15T23:40:14.778 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.778 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd765f2:::benchmark_data_smithi090_166897_object656:head# 2024-05-15T23:40:14.778 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.778 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fd7ea26:::benchmark_data_smithi090_166897_object70370:head# 2024-05-15T23:40:14.778 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.779 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fdaf28e:::benchmark_data_smithi090_166897_object44822:head# 2024-05-15T23:40:14.779 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.779 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fdc119e:::benchmark_data_smithi090_166897_object68855:head# 2024-05-15T23:40:14.779 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.779 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fdc80f4:::benchmark_data_smithi090_166897_object31078:head# 2024-05-15T23:40:14.779 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.779 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fdcd4a7:::benchmark_data_smithi090_166897_object45783:head# 2024-05-15T23:40:14.779 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.779 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fde21eb:::benchmark_data_smithi090_166897_object40432:head# 2024-05-15T23:40:14.779 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.779 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fde3d17:::benchmark_data_smithi090_166897_object66161:head# 2024-05-15T23:40:14.779 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.779 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fde5eb5:::benchmark_data_smithi090_166897_object5785:head# 2024-05-15T23:40:14.779 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.779 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fde5fc4:::benchmark_data_smithi090_166897_object64664:head# 2024-05-15T23:40:14.780 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.780 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fde6078:::benchmark_data_smithi090_166897_object62796:head# 2024-05-15T23:40:14.780 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.780 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fdebdcd:::benchmark_data_smithi090_166897_object48312:head# 2024-05-15T23:40:14.780 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.780 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fe03e9f:::benchmark_data_smithi090_166897_object26879:head# 2024-05-15T23:40:14.780 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.780 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fe08450:::benchmark_data_smithi090_166897_object45279:head# 2024-05-15T23:40:14.780 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.780 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fe14c10:::benchmark_data_smithi090_166897_object56087:head# 2024-05-15T23:40:14.780 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.780 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fe2115a:::benchmark_data_smithi090_166897_object47189:head# 2024-05-15T23:40:14.780 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.780 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fe41468:::benchmark_data_smithi090_166897_object37882:head# 2024-05-15T23:40:14.780 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fe4ac5a:::benchmark_data_smithi090_166897_object26855:head# 2024-05-15T23:40:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fe5ca73:::benchmark_data_smithi090_166897_object64263:head# 2024-05-15T23:40:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fe5dc22:::benchmark_data_smithi090_166897_object42017:head# 2024-05-15T23:40:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fe63f3a:::benchmark_data_smithi090_166897_object30011:head# 2024-05-15T23:40:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fe653d5:::benchmark_data_smithi090_166897_object59624:head# 2024-05-15T23:40:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fe9e8f0:::benchmark_data_smithi090_166897_object25584:head# 2024-05-15T23:40:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fee8fcf:::benchmark_data_smithi090_166897_object33491:head# 2024-05-15T23:40:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fef636c:::benchmark_data_smithi090_166897_object11458:head# 2024-05-15T23:40:14.782 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.782 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6fefa9ad:::benchmark_data_smithi090_166897_object45440:head# 2024-05-15T23:40:14.782 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.782 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ff09ed3:::benchmark_data_smithi090_166897_object9479:head# 2024-05-15T23:40:14.782 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.782 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ff1dbf4:::benchmark_data_smithi090_166897_object26057:head# 2024-05-15T23:40:14.782 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.782 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ff31086:::benchmark_data_smithi090_166897_object56990:head# 2024-05-15T23:40:14.782 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.782 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ff3e88e:::benchmark_data_smithi090_166897_object40814:head# 2024-05-15T23:40:14.782 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.782 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ff4984f:::benchmark_data_smithi090_166897_object55490:head# 2024-05-15T23:40:14.782 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ff8611c:::benchmark_data_smithi090_166897_object25231:head# 2024-05-15T23:40:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ff8795b:::benchmark_data_smithi090_166897_object41643:head# 2024-05-15T23:40:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ff932fc:::benchmark_data_smithi090_166897_object52353:head# 2024-05-15T23:40:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ff9597e:::benchmark_data_smithi090_166897_object39025:head# 2024-05-15T23:40:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ff9e7ea:::benchmark_data_smithi090_166897_object37779:head# 2024-05-15T23:40:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ffb2576:::benchmark_data_smithi090_166897_object12791:head# 2024-05-15T23:40:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ffb4c5e:::benchmark_data_smithi090_166897_object32602:head# 2024-05-15T23:40:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ffe197b:::benchmark_data_smithi090_166897_object63487:head# 2024-05-15T23:40:14.784 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.784 INFO:teuthology.orchestra.run.smithi088.stdout:Write #9:6ffee9e1:::benchmark_data_smithi090_166897_object21722:head# 2024-05-15T23:40:14.784 INFO:teuthology.orchestra.run.smithi088.stdout:snapset 0=[]:{} 2024-05-15T23:40:14.784 INFO:teuthology.orchestra.run.smithi088.stdout:write_pg epoch 749 info 9.6( v 749'4391 (0'0,749'4391] local-lis/les=737/738 n=4391 ec=737/737 lis/c=737/737 les/c/f=738/738/0 sis=737) 2024-05-15T23:40:14.784 INFO:teuthology.orchestra.run.smithi088.stdout:Import successful 2024-05-15T23:40:14.862 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:40:14.903 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:15.248 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:40:15.248 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:40:15.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:15 smithi090 ceph-mon[44434]: pgmap v2033: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.3 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 70 MiB/s wr, 1.11k op/s; 12103/207544 objects degraded (5.832%); 12 MiB/s, 199 objects/s recovering 2024-05-15T23:40:15.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:15 smithi088 ceph-mon[44699]: pgmap v2033: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.3 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 70 MiB/s wr, 1.11k op/s; 12103/207544 objects degraded (5.832%); 12 MiB/s, 199 objects/s recovering 2024-05-15T23:40:15.575 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 73 16 106559 106543 91.1913 71.125 0.00644013 0.0109635 2024-05-15T23:40:15.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:15 smithi084 ceph-mon[39784]: pgmap v2033: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.3 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 70 MiB/s wr, 1.11k op/s; 12103/207544 objects degraded (5.832%); 12 MiB/s, 199 objects/s recovering 2024-05-15T23:40:15.994 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/exp.9.6.5 2024-05-15T23:40:16.066 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:40:16.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:16.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:40:16.079 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-05-15T23:40:16.080 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-15T23:40:16.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:40:16.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:40:16.240 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:16.575 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 74 16 107589 107573 90.8289 64.375 0.00904641 0.0110079 2024-05-15T23:40:16.883 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:40:17.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:40:17.576 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 75 16 108489 108473 90.3679 56.25 0.00914506 0.011064 2024-05-15T23:40:17.614 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:40:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:17.618 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:40:17.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:17 smithi084 ceph-mon[39784]: pgmap v2034: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.5 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 69 MiB/s wr, 1.11k op/s; 11337/213620 objects degraded (5.307%); 13 MiB/s, 203 objects/s recovering 2024-05-15T23:40:17.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:17 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 12103/207544 objects degraded (5.832%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:17.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:40:17.760 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:17.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:17 smithi090 ceph-mon[44434]: pgmap v2034: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.5 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 69 MiB/s wr, 1.11k op/s; 11337/213620 objects degraded (5.307%); 13 MiB/s, 203 objects/s recovering 2024-05-15T23:40:17.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:17 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 12103/207544 objects degraded (5.832%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:17.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:17 smithi088 ceph-mon[44699]: pgmap v2034: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.5 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 69 MiB/s wr, 1.11k op/s; 11337/213620 objects degraded (5.307%); 13 MiB/s, 203 objects/s recovering 2024-05-15T23:40:17.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:17 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 12103/207544 objects degraded (5.832%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:18.444 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:40:18.576 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 76 16 109530 109514 90.0349 65.0625 0.0325679 0.0111045 2024-05-15T23:40:18.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:18 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2208061463' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:40:18.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:18 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2208061463' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:40:18.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:18 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2208061463' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:40:19.578 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 77 16 110435 110419 89.6002 56.5625 0.0207618 0.0111589 2024-05-15T23:40:19.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:19 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:40:19.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2208061463' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:40:19.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:19 smithi090 ceph-mon[44434]: osdmap e766: 12 total, 11 up, 9 in 2024-05-15T23:40:19.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:19 smithi090 ceph-mon[44434]: pgmap v2036: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.6 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 69 MiB/s wr, 1.10k op/s; 11057/215588 objects degraded (5.129%); 13 MiB/s, 205 objects/s recovering 2024-05-15T23:40:19.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:19 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:40:19.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2208061463' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:40:19.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:19 smithi088 ceph-mon[44699]: osdmap e766: 12 total, 11 up, 9 in 2024-05-15T23:40:19.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:19 smithi088 ceph-mon[44699]: pgmap v2036: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.6 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 69 MiB/s wr, 1.10k op/s; 11057/215588 objects degraded (5.129%); 13 MiB/s, 205 objects/s recovering 2024-05-15T23:40:19.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:19 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:40:19.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2208061463' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:40:19.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:19 smithi084 ceph-mon[39784]: osdmap e766: 12 total, 11 up, 9 in 2024-05-15T23:40:19.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:19 smithi084 ceph-mon[39784]: pgmap v2036: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.6 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 69 MiB/s wr, 1.10k op/s; 11057/215588 objects degraded (5.129%); 13 MiB/s, 205 objects/s recovering 2024-05-15T23:40:19.959 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:40:19.959 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:40:20.138 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:40:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:20.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:40:20.244 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:40:20.285 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:20.428 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:40:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:20.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:40:20.535 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:40:20.573 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:20.576 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 78 16 111553 111537 89.3473 69.875 0.00715814 0.0111906 2024-05-15T23:40:20.942 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T23:40:21.354 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:40:21.576 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 79 16 112564 112548 89.0162 63.1875 0.0280877 0.0112321 2024-05-15T23:40:21.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:21 smithi090 ceph-mon[44434]: pgmap v2037: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 68 MiB/s wr, 1.08k op/s; 11057/217988 objects degraded (5.072%); 13 MiB/s, 205 objects/s recovering 2024-05-15T23:40:21.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:21 smithi088 ceph-mon[44699]: pgmap v2037: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 68 MiB/s wr, 1.08k op/s; 11057/217988 objects degraded (5.072%); 13 MiB/s, 205 objects/s recovering 2024-05-15T23:40:21.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:21 smithi084 ceph-mon[39784]: pgmap v2037: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 68 MiB/s wr, 1.08k op/s; 11057/217988 objects degraded (5.072%); 13 MiB/s, 205 objects/s recovering 2024-05-15T23:40:22.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:40:22.580 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:2024-05-15 23:40:22.577221 min lat: 0.00103205 max lat: 19.1505 avg lat: 0.0112697 2024-05-15T23:40:22.580 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-15T23:40:22.580 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 80 16 113603 113587 88.7142 64.9375 0.0167774 0.0112697 2024-05-15T23:40:22.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:22 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 11057/217988 objects degraded (5.072%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:22.757 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:22 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 11057/217988 objects degraded (5.072%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:22.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:22 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 11057/217988 objects degraded (5.072%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:23.586 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 81 16 114719 114703 88.4796 69.75 0.0050246 0.0113 2024-05-15T23:40:23.709 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:40:23.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:23 smithi090 ceph-mon[44434]: pgmap v2038: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 65 MiB/s wr, 1.04k op/s; 9908/224876 objects degraded (4.406%); 14 MiB/s, 219 objects/s recovering 2024-05-15T23:40:23.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:23 smithi088 ceph-mon[44699]: pgmap v2038: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 65 MiB/s wr, 1.04k op/s; 9908/224876 objects degraded (4.406%); 14 MiB/s, 219 objects/s recovering 2024-05-15T23:40:23.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:23 smithi084 ceph-mon[39784]: pgmap v2038: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 6.9 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 65 MiB/s wr, 1.04k op/s; 9908/224876 objects degraded (4.406%); 14 MiB/s, 219 objects/s recovering 2024-05-15T23:40:23.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:23.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:40:24.020 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:40:24.021 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:40:24.098 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:40:24.129 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:24.154 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:40:24.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:24.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:40:24.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:40:24.287 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:24.309 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:40:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:24.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:40:24.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:40:24.466 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:24.577 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 82 16 115781 115765 88.21 66.375 0.0202423 0.0113343 2024-05-15T23:40:24.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T23:40:24.928 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:40:24.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:24.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:40:24.995 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:40:25.036 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:40:25.042 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:40:25.108 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:25.577 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 83 16 116807 116791 87.9198 64.125 0.0118489 0.0113719 2024-05-15T23:40:25.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:25 smithi090 ceph-mon[44434]: pgmap v2039: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 64 MiB/s wr, 1.03k op/s; 9908/228124 objects degraded (4.343%); 14 MiB/s, 219 objects/s recovering 2024-05-15T23:40:25.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:25 smithi088 ceph-mon[44699]: pgmap v2039: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 64 MiB/s wr, 1.03k op/s; 9908/228124 objects degraded (4.343%); 14 MiB/s, 219 objects/s recovering 2024-05-15T23:40:25.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:25 smithi084 ceph-mon[39784]: pgmap v2039: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 64 MiB/s wr, 1.03k op/s; 9908/228124 objects degraded (4.343%); 14 MiB/s, 219 objects/s recovering 2024-05-15T23:40:26.101 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:40:26.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:26.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:40:26.208 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:40:26.241 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:26.492 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:40:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:26.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:40:26.578 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 84 16 117874 117858 87.667 66.6875 0.0195608 0.0114048 2024-05-15T23:40:26.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:40:26.653 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:26.697 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:40:26.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:26.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:40:26.803 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:40:26.888 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:26.921 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:40:26.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:26.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:40:27.030 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:40:27.084 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:27.578 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 85 16 118928 118912 87.4106 65.875 0.0322779 0.0114385 2024-05-15T23:40:27.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:27 smithi090 ceph-mon[44434]: pgmap v2040: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 64 MiB/s wr, 1.03k op/s; 9065/234166 objects degraded (3.871%); 14 MiB/s, 227 objects/s recovering 2024-05-15T23:40:27.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:27 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 9908/228124 objects degraded (4.343%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:27 smithi088 ceph-mon[44699]: pgmap v2040: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 64 MiB/s wr, 1.03k op/s; 9065/234166 objects degraded (3.871%); 14 MiB/s, 227 objects/s recovering 2024-05-15T23:40:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:27 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 9908/228124 objects degraded (4.343%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:27.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:27 smithi084 ceph-mon[39784]: pgmap v2040: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 64 MiB/s wr, 1.03k op/s; 9065/234166 objects degraded (3.871%); 14 MiB/s, 227 objects/s recovering 2024-05-15T23:40:27.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:27 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 9908/228124 objects degraded (4.343%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:28.049 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:40:28.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:28.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:40:28.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:40:28.203 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:28.243 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:40:28.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:28.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:40:28.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:40:28.389 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:28.428 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:40:28.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:28.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:40:28.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:40:28.578 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 86 16 119921 119905 87.1159 62.0625 0.00989442 0.0114772 2024-05-15T23:40:28.586 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:28.624 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:40:28.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:28.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:40:28.732 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:40:28.763 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:28.785 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:40:28.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:28.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:40:28.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:40:28.942 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:28.975 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:40:28.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:28.983 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:40:29.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:40:29.132 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:29.166 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:40:29.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:29.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:40:29.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:40:29.345 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:29.382 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:40:29.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:29.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:40:29.484 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:40:29.513 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:29.534 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:40:29.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:29.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:40:29.578 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 87 16 120992 120976 86.8839 66.9375 0.010259 0.0115077 2024-05-15T23:40:29.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:40:29.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:29 smithi084 ceph-mon[39784]: pgmap v2041: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 61 MiB/s wr, 972 op/s; 8705/236134 objects degraded (3.686%); 13 MiB/s, 215 objects/s recovering 2024-05-15T23:40:29.782 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:29.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:29 smithi090 ceph-mon[44434]: pgmap v2041: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 61 MiB/s wr, 972 op/s; 8705/236134 objects degraded (3.686%); 13 MiB/s, 215 objects/s recovering 2024-05-15T23:40:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:29 smithi088 ceph-mon[44699]: pgmap v2041: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 61 MiB/s wr, 972 op/s; 8705/236134 objects degraded (3.686%); 13 MiB/s, 215 objects/s recovering 2024-05-15T23:40:30.251 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:40:30.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:30.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:40:30.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:40:30.433 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:30.578 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 88 16 122002 121986 86.6139 63.125 0.0151246 0.0115432 2024-05-15T23:40:30.617 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:40:30.636 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:40:31.229 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:40:31.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:31.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:40:31.335 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:40:31.380 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:31.420 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:40:31.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:31.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:40:31.528 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:40:31.570 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:31.578 INFO:tasks.radosbench.radosbench.2.smithi090.stdout: 89 16 123115 123099 86.4223 69.5625 0.0139692 0.0115695 2024-05-15T23:40:31.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:31 smithi090 ceph-mon[44434]: from='client.17553 v1:172.21.15.84:0/2013429846' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:40:31.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:31 smithi090 ceph-mon[44434]: pgmap v2042: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 60 MiB/s wr, 964 op/s; 8705/238746 objects degraded (3.646%); 12 MiB/s, 195 objects/s recovering 2024-05-15T23:40:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:31 smithi088 ceph-mon[44699]: from='client.17553 v1:172.21.15.84:0/2013429846' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:40:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:31 smithi088 ceph-mon[44699]: pgmap v2042: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 60 MiB/s wr, 964 op/s; 8705/238746 objects degraded (3.646%); 12 MiB/s, 195 objects/s recovering 2024-05-15T23:40:31.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:31 smithi084 ceph-mon[39784]: from='client.17553 v1:172.21.15.84:0/2013429846' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:40:31.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:31 smithi084 ceph-mon[39784]: pgmap v2042: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 60 MiB/s wr, 964 op/s; 8705/238746 objects degraded (3.646%); 12 MiB/s, 195 objects/s recovering 2024-05-15T23:40:32.140 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":2041,"stamp":"2024-05-15T23:40:29.348944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7738098208,"num_objects":118067,"num_object_clones":0,"num_object_copies":236134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118067,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":118188,"num_write_kb":7558310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8801,"num_bytes_recovered":577975904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118159,"ondisk_log_size":118159,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":8,"kb":843706368,"kb_used":20907520,"kb_used_data":14991860,"kb_used_omap":39,"kb_used_meta":5915544,"kb_avail":822798848,"statfs":{"total":863955320832,"available":842546020352,"internally_reserved":0,"allocated":15351664640,"data_stored":15348337965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40938,"internal_metadata":6057517078},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":695009280,"num_objects":10605,"num_object_clones":0,"num_object_copies":21210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10605,"num_write_kb":678720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2352,"num_bytes_recovered":154140672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.907966"},"pg_stats":[{"pgid":"9.f","version":"766'7260","reported_seq":10910,"reported_epoch":766,"state":"active+recovering+degraded","last_fresh":"2024-05-15T23:40:27.170262+0000","last_change":"2024-05-15T23:39:43.669795+0000","last_active":"2024-05-15T23:40:27.170262+0000","last_peered":"2024-05-15T23:40:27.170262+0000","last_clean":"2024-05-15T23:39:25.389963+0000","last_became_active":"2024-05-15T23:39:43.643192+0000","last_became_peered":"2024-05-15T23:39:43.643192+0000","last_unstale":"2024-05-15T23:40:27.170262+0000","last_undegraded":"2024-05-15T23:39:43.429912+0000","last_fullsized":"2024-05-15T23:40:27.170262+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":7260,"log_dups_size":0,"ondisk_log_size":7260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475791360,"num_objects":7260,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2559,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7260,"num_write_kb":464640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1790,"num_bytes_recovered":117309440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2559},{"shards":"3,9","objects":4701}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"766'7240","reported_seq":7307,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T23:40:24.909272+0000","last_change":"2024-05-15T23:38:50.403048+0000","last_active":"2024-05-15T23:40:24.909272+0000","last_peered":"2024-05-15T23:40:24.909272+0000","last_clean":"2024-05-15T23:40:24.909272+0000","last_became_active":"2024-05-15T23:38:50.402740+0000","last_became_peered":"2024-05-15T23:38:50.402740+0000","last_unstale":"2024-05-15T23:40:24.909272+0000","last_undegraded":"2024-05-15T23:40:24.909272+0000","last_fullsized":"2024-05-15T23:40:24.909272+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":7240,"log_dups_size":0,"ondisk_log_size":7240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474480640,"num_objects":7240,"num_object_clones":0,"num_object_copies":14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":463360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"766'7470","reported_seq":7537,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T23:40:24.935155+0000","last_change":"2024-05-15T23:38:50.401477+0000","last_active":"2024-05-15T23:40:24.935155+0000","last_peered":"2024-05-15T23:40:24.935155+0000","last_clean":"2024-05-15T23:40:24.935155+0000","last_became_active":"2024-05-15T23:38:50.401123+0000","last_became_peered":"2024-05-15T23:38:50.401123+0000","last_unstale":"2024-05-15T23:40:24.935155+0000","last_undegraded":"2024-05-15T23:40:24.935155+0000","last_fullsized":"2024-05-15T23:40:24.935155+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":7470,"log_dups_size":0,"ondisk_log_size":7470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489553920,"num_objects":7470,"num_object_clones":0,"num_object_copies":14940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7470,"num_write_kb":478080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"766'7429","reported_seq":7496,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T23:40:27.161539+0000","last_change":"2024-05-15T23:38:50.402679+0000","last_active":"2024-05-15T23:40:27.161539+0000","last_peered":"2024-05-15T23:40:27.161539+0000","last_clean":"2024-05-15T23:40:27.161539+0000","last_became_active":"2024-05-15T23:38:50.402428+0000","last_became_peered":"2024-05-15T23:38:50.402428+0000","last_unstale":"2024-05-15T23:40:27.161539+0000","last_undegraded":"2024-05-15T23:40:27.161539+0000","last_fullsized":"2024-05-15T23:40:27.161539+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":7429,"log_dups_size":0,"ondisk_log_size":7429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486866944,"num_objects":7429,"num_object_clones":0,"num_object_copies":14858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7429,"num_write_kb":475456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"766'7515","reported_seq":7582,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T23:40:29.213557+0000","last_change":"2024-05-15T23:38:50.402430+0000","last_active":"2024-05-15T23:40:29.213557+0000","last_peered":"2024-05-15T23:40:29.213557+0000","last_clean":"2024-05-15T23:40:29.213557+0000","last_became_active":"2024-05-15T23:38:50.401815+0000","last_became_peered":"2024-05-15T23:38:50.401815+0000","last_unstale":"2024-05-15T23:40:29.213557+0000","last_undegraded":"2024-05-15T23:40:29.213557+0000","last_fullsized":"2024-05-15T23:40:29.213557+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":7515,"log_dups_size":0,"ondisk_log_size":7515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:40:30.032578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":492503040,"num_objects":7515,"num_object_clones":0,"num_object_copies":15030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7515,"num_write_kb":480960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"766'7339","reported_seq":12638,"reported_epoch":766,"state":"active+recovering+degraded+remapped","last_fresh":"2024-05-15T23:40:27.093570+0000","last_change":"2024-05-15T23:39:44.562552+0000","last_active":"2024-05-15T23:40:27.093570+0000","last_peered":"2024-05-15T23:40:27.093570+0000","last_clean":"2024-05-15T23:39:25.406948+0000","last_became_active":"2024-05-15T23:39:44.542316+0000","last_became_peered":"2024-05-15T23:39:44.542316+0000","last_unstale":"2024-05-15T23:40:27.093570+0000","last_undegraded":"2024-05-15T23:39:44.442609+0000","last_fullsized":"2024-05-15T23:40:27.093570+0000","mapping_epoch":759,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":7339,"log_dups_size":0,"ondisk_log_size":7339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480968704,"num_objects":7339,"num_object_clones":0,"num_object_copies":14678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1781,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7339,"num_write_kb":469696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2614,"num_bytes_recovered":171311104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[1,7],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1781},{"shards":"1,7","objects":5558}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"766'7414","reported_seq":7495,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T23:40:26.428490+0000","last_change":"2024-05-15T23:40:02.210276+0000","last_active":"2024-05-15T23:40:26.428490+0000","last_peered":"2024-05-15T23:40:26.428490+0000","last_clean":"2024-05-15T23:40:26.428490+0000","last_became_active":"2024-05-15T23:38:50.404189+0000","last_became_peered":"2024-05-15T23:38:50.404189+0000","last_unstale":"2024-05-15T23:40:26.428490+0000","last_undegraded":"2024-05-15T23:40:26.428490+0000","last_fullsized":"2024-05-15T23:40:26.428490+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":1049,"log_size":7414,"log_dups_size":0,"ondisk_log_size":7414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:40:28.059433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485883904,"num_objects":7414,"num_object_clones":0,"num_object_copies":14828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7414,"num_write_kb":474496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"766'7279","reported_seq":7406,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T23:40:26.434282+0000","last_change":"2024-05-15T23:39:55.792219+0000","last_active":"2024-05-15T23:40:26.434282+0000","last_peered":"2024-05-15T23:40:26.434282+0000","last_clean":"2024-05-15T23:40:26.434282+0000","last_became_active":"2024-05-15T23:38:50.401416+0000","last_became_peered":"2024-05-15T23:38:50.401416+0000","last_unstale":"2024-05-15T23:40:26.434282+0000","last_undegraded":"2024-05-15T23:40:26.434282+0000","last_fullsized":"2024-05-15T23:40:26.434282+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"761'5246","last_scrub_stamp":"2024-05-15T23:39:55.792018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:39:55.792018+0000","objects_scrubbed":5229,"log_size":7279,"log_dups_size":0,"ondisk_log_size":7279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:21.653921+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477036544,"num_objects":7279,"num_object_clones":0,"num_object_copies":14558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7279,"num_write_kb":465856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"766'7324","reported_seq":11284,"reported_epoch":766,"state":"active+recovering+degraded","last_fresh":"2024-05-15T23:40:27.167592+0000","last_change":"2024-05-15T23:39:43.668135+0000","last_active":"2024-05-15T23:40:27.167592+0000","last_peered":"2024-05-15T23:40:27.167592+0000","last_clean":"2024-05-15T23:39:25.398249+0000","last_became_active":"2024-05-15T23:39:43.637604+0000","last_became_peered":"2024-05-15T23:39:43.637604+0000","last_unstale":"2024-05-15T23:40:27.167592+0000","last_undegraded":"2024-05-15T23:39:43.429785+0000","last_fullsized":"2024-05-15T23:40:27.167592+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":7324,"log_dups_size":0,"ondisk_log_size":7324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479985664,"num_objects":7324,"num_object_clones":0,"num_object_copies":14648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7324,"num_write_kb":468736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1944,"num_bytes_recovered":127401984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2447},{"shards":"3,9","objects":4877}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"766'7319","reported_seq":12285,"reported_epoch":766,"state":"active+recovering+degraded","last_fresh":"2024-05-15T23:40:28.039072+0000","last_change":"2024-05-15T23:39:43.532605+0000","last_active":"2024-05-15T23:40:28.039072+0000","last_peered":"2024-05-15T23:40:28.039072+0000","last_clean":"2024-05-15T23:39:25.408696+0000","last_became_active":"2024-05-15T23:39:43.514269+0000","last_became_peered":"2024-05-15T23:39:43.514269+0000","last_unstale":"2024-05-15T23:40:28.039072+0000","last_undegraded":"2024-05-15T23:39:43.421623+0000","last_fullsized":"2024-05-15T23:40:28.039072+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":7319,"log_dups_size":0,"ondisk_log_size":7319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479657984,"num_objects":7319,"num_object_clones":0,"num_object_copies":14638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7319,"num_write_kb":468416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2447,"num_bytes_recovered":160366592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":["11"],"object_location_counts":[{"shards":"7,11","objects":5401},{"shards":"11","objects":1918}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"766'7426","reported_seq":7493,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T23:40:27.045913+0000","last_change":"2024-05-15T23:38:50.398678+0000","last_active":"2024-05-15T23:40:27.045913+0000","last_peered":"2024-05-15T23:40:27.045913+0000","last_clean":"2024-05-15T23:40:27.045913+0000","last_became_active":"2024-05-15T23:38:50.398410+0000","last_became_peered":"2024-05-15T23:38:50.398410+0000","last_unstale":"2024-05-15T23:40:27.045913+0000","last_undegraded":"2024-05-15T23:40:27.045913+0000","last_fullsized":"2024-05-15T23:40:27.045913+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":7426,"log_dups_size":0,"ondisk_log_size":7426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486670336,"num_objects":7426,"num_object_clones":0,"num_object_copies":14852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7426,"num_write_kb":475264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"766'7245","reported_seq":7312,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T23:40:25.207938+0000","last_change":"2024-05-15T23:38:50.402064+0000","last_active":"2024-05-15T23:40:25.207938+0000","last_peered":"2024-05-15T23:40:25.207938+0000","last_clean":"2024-05-15T23:40:25.207938+0000","last_became_active":"2024-05-15T23:38:50.401779+0000","last_became_peered":"2024-05-15T23:38:50.401779+0000","last_unstale":"2024-05-15T23:40:25.207938+0000","last_undegraded":"2024-05-15T23:40:25.207938+0000","last_fullsized":"2024-05-15T23:40:25.207938+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":7245,"log_dups_size":0,"ondisk_log_size":7245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474808320,"num_objects":7245,"num_object_clones":0,"num_object_copies":14490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7245,"num_write_kb":463680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"766'7453","reported_seq":7520,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T23:40:27.179031+0000","last_change":"2024-05-15T23:38:50.402094+0000","last_active":"2024-05-15T23:40:27.179031+0000","last_peered":"2024-05-15T23:40:27.179031+0000","last_clean":"2024-05-15T23:40:27.179031+0000","last_became_active":"2024-05-15T23:38:50.401872+0000","last_became_peered":"2024-05-15T23:38:50.401872+0000","last_unstale":"2024-05-15T23:40:27.179031+0000","last_undegraded":"2024-05-15T23:40:27.179031+0000","last_fullsized":"2024-05-15T23:40:27.179031+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":7453,"log_dups_size":0,"ondisk_log_size":7453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488439808,"num_objects":7453,"num_object_clones":0,"num_object_copies":14906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7453,"num_write_kb":476992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"766'7453","reported_seq":7520,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T23:40:27.152142+0000","last_change":"2024-05-15T23:38:50.401343+0000","last_active":"2024-05-15T23:40:27.152142+0000","last_peered":"2024-05-15T23:40:27.152142+0000","last_clean":"2024-05-15T23:40:27.152142+0000","last_became_active":"2024-05-15T23:38:50.401074+0000","last_became_peered":"2024-05-15T23:38:50.401074+0000","last_unstale":"2024-05-15T23:40:27.152142+0000","last_undegraded":"2024-05-15T23:40:27.152142+0000","last_fullsized":"2024-05-15T23:40:27.152142+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":7453,"log_dups_size":0,"ondisk_log_size":7453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488439808,"num_objects":7453,"num_object_clones":0,"num_object_copies":14906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7453,"num_write_kb":476992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"766'7549","reported_seq":7616,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T23:40:27.993089+0000","last_change":"2024-05-15T23:38:50.402309+0000","last_active":"2024-05-15T23:40:27.993089+0000","last_peered":"2024-05-15T23:40:27.993089+0000","last_clean":"2024-05-15T23:40:27.993089+0000","last_became_active":"2024-05-15T23:38:50.401998+0000","last_became_peered":"2024-05-15T23:38:50.401998+0000","last_unstale":"2024-05-15T23:40:27.993089+0000","last_undegraded":"2024-05-15T23:40:27.993089+0000","last_fullsized":"2024-05-15T23:40:27.993089+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":7549,"log_dups_size":0,"ondisk_log_size":7549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494731264,"num_objects":7549,"num_object_clones":0,"num_object_copies":15098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7549,"num_write_kb":483136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.8","version":"766'7350","reported_seq":7417,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T23:40:25.217419+0000","last_change":"2024-05-15T23:38:50.404457+0000","last_active":"2024-05-15T23:40:25.217419+0000","last_peered":"2024-05-15T23:40:25.217419+0000","last_clean":"2024-05-15T23:40:25.217419+0000","last_became_active":"2024-05-15T23:38:50.404155+0000","last_became_peered":"2024-05-15T23:38:50.404155+0000","last_unstale":"2024-05-15T23:40:25.217419+0000","last_undegraded":"2024-05-15T23:40:25.217419+0000","last_fullsized":"2024-05-15T23:40:25.217419+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":7350,"log_dups_size":0,"ondisk_log_size":7350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481689600,"num_objects":7350,"num_object_clones":0,"num_object_copies":14700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7350,"num_write_kb":470400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1659,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-05-15T23:40:18.446090+0000","last_change":"2024-05-15T23:38:23.156562+0000","last_active":"2024-05-15T23:40:18.446090+0000","last_peered":"2024-05-15T23:40:18.446090+0000","last_clean":"2024-05-15T23:40:18.446090+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:40:18.446090+0000","last_undegraded":"2024-05-15T23:40:18.446090+0000","last_fullsized":"2024-05-15T23:40:18.446090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:38:23.156474+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:38:23.156474+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7737507840,"num_objects":118065,"num_object_clones":0,"num_object_copies":236130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118065,"num_write_kb":7556160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8795,"num_bytes_recovered":576389120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15860170752,"data_stored":15860170752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118065,"ondisk_log_size":118065,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351417,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2799636,"kb_used_data":1911884,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":90945516,"statfs":{"total":95995035648,"available":93128208384,"internally_reserved":0,"allocated":1957769216,"data_stored":1957396805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":909048260},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580349,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2497484,"kb_used_data":1874884,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91247668,"statfs":{"total":95995035648,"available":93437612032,"internally_reserved":0,"allocated":1919881216,"data_stored":1919512377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":637532608},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776581,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2799372,"kb_used_data":1983812,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":90945780,"statfs":{"total":95995035648,"available":93128478720,"internally_reserved":0,"allocated":2031423488,"data_stored":2031054649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":835123648},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":520,"seq":2233382994085,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594188,"kb_used_data":480900,"kb_used_omap":24,"kb_used_meta":113255,"kb_avail":93150964,"statfs":{"total":95995035648,"available":95386587136,"internally_reserved":0,"allocated":492441600,"data_stored":492072761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25357,"internal_metadata":115973363},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267150,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3000660,"kb_used_data":2319756,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":90744492,"statfs":{"total":95995035648,"available":92922359808,"internally_reserved":0,"allocated":2375430144,"data_stored":2375057753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":697235914},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568496084,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575342,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3552332,"kb_used_data":2664580,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":90192820,"statfs":{"total":95995035648,"available":92357447680,"internally_reserved":0,"allocated":2728529920,"data_stored":2728161081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":909048262},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016752,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":209360,"kb_used_data":708,"kb_used_omap":4,"kb_used_meta":208635,"kb_avail":93535792,"statfs":{"total":95995035648,"available":95780651008,"internally_reserved":0,"allocated":724992,"data_stored":356153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":213642947},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150181,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3806668,"kb_used_data":2832836,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":89938484,"statfs":{"total":95995035648,"available":92097007616,"internally_reserved":0,"allocated":2900824064,"data_stored":2900455225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":997194183},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986652,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1647820,"kb_used_data":922500,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92097332,"statfs":{"total":95995035648,"available":94307667968,"internally_reserved":0,"allocated":944640000,"data_stored":944271161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2446393344,"data_stored":2446393344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2902589440,"data_stored":2902589440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":951975936,"data_stored":951975936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1040711680,"data_stored":1040711680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2104885248,"data_stored":2104885248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":492503040,"data_stored":492503040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2034237440,"data_stored":2034237440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1926496256,"data_stored":1926496256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1960378368,"data_stored":1960378368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:40:32.141 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:40:32.246 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:40:32.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:32.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:40:32.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:40:32.421 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:32.459 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:40:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:32.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:40:32.574 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:40:32.578 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total time run: 90.0102 2024-05-15T23:40:32.579 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Total writes made: 124185 2024-05-15T23:40:32.579 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Write size: 65536 2024-05-15T23:40:32.579 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Object size: 65536 2024-05-15T23:40:32.579 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Bandwidth (MB/sec): 86.2298 2024-05-15T23:40:32.579 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Bandwidth: 75.8584 2024-05-15T23:40:32.579 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max bandwidth (MB/sec): 355.688 2024-05-15T23:40:32.579 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min bandwidth (MB/sec): 0 2024-05-15T23:40:32.579 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average IOPS: 1379 2024-05-15T23:40:32.579 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev IOPS: 1213.75 2024-05-15T23:40:32.579 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max IOPS: 5691 2024-05-15T23:40:32.580 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min IOPS: 0 2024-05-15T23:40:32.580 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Average Latency(s): 0.0115962 2024-05-15T23:40:32.580 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Stddev Latency(s): 0.209642 2024-05-15T23:40:32.580 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Max latency(s): 19.1505 2024-05-15T23:40:32.580 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Min latency(s): 0.00103205 2024-05-15T23:40:32.586 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Cleaning up (deleting benchmark objects) 2024-05-15T23:40:32.604 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:40:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:32.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:40:32.733 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:40:32.762 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:32 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 8705/238746 objects degraded (3.646%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:32.764 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:32.788 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:40:32.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:32.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:40:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:32 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 8705/238746 objects degraded (3.646%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:32.814 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:32 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 8705/238746 objects degraded (3.646%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:32.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:40:32.936 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:33.126 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:40:33.153 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:40:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:33.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:40:33.260 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:40:33.341 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:33.455 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:40:33.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:33.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:40:33.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:40:33.622 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:33.669 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:40:33.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:33.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:40:33.714 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:33 smithi084 ceph-mon[39784]: pgmap v2043: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 73 MiB/s wr, 1.17k op/s; 7414/245866 objects degraded (3.015%); 19 MiB/s, 303 objects/s recovering 2024-05-15T23:40:33.776 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:40:33.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:33 smithi090 ceph-mon[44434]: pgmap v2043: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 73 MiB/s wr, 1.17k op/s; 7414/245866 objects degraded (3.015%); 19 MiB/s, 303 objects/s recovering 2024-05-15T23:40:33.802 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:33 smithi088 ceph-mon[44699]: pgmap v2043: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 73 MiB/s wr, 1.17k op/s; 7414/245866 objects degraded (3.015%); 19 MiB/s, 303 objects/s recovering 2024-05-15T23:40:33.804 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:33.826 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:40:33.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:33.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:40:33.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:40:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:34.005 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:40:34.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:34.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:40:34.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:40:34.160 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:34.202 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:40:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:34.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:40:34.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:40:34.357 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:34.385 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:40:34.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:34.390 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:40:34.491 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:40:34.565 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:35 smithi090 ceph-mon[44434]: pgmap v2044: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 85 B/s rd, 60 MiB/s wr, 1.03k op/s; 7414/245990 objects degraded (3.014%); 13 MiB/s, 207 objects/s recovering 2024-05-15T23:40:35.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:35 smithi088 ceph-mon[44699]: pgmap v2044: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 85 B/s rd, 60 MiB/s wr, 1.03k op/s; 7414/245990 objects degraded (3.014%); 13 MiB/s, 207 objects/s recovering 2024-05-15T23:40:35.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:35 smithi084 ceph-mon[39784]: pgmap v2044: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 85 B/s rd, 60 MiB/s wr, 1.03k op/s; 7414/245990 objects degraded (3.014%); 13 MiB/s, 207 objects/s recovering 2024-05-15T23:40:36.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:40:36.657 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:40:36.688 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:40:36.714 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:40:36.733 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:40:36.739 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:40:36.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:36.746 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:36 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:40:36.746 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:36 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:40:36.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:40:36.781 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:40:36.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:36 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:40:36.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:36 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:40:36.814 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:40:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:36 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:40:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:36 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:40:36.848 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:40:36.851 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:40:36.878 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:36.886 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:40:36.900 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:40:36.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:36.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:40:36.923 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:40:36.962 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:40:36.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:40:36.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:40:36.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:40:36.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:40:36.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:40:36.963 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-05-15T23:40:36.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:40:36.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:40:36.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:40:36.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:40:36.964 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:40:36.964 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:40:37.006 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:40:37.051 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:37.808 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:37 smithi090 ceph-mon[44434]: osdmap e767: 12 total, 11 up, 9 in 2024-05-15T23:40:37.808 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:37 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/216874684' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:40:37.808 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:37.808 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:37.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:37.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:37.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:37.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:37 smithi090 ceph-mon[44434]: pgmap v2046: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 102 B/s rd, 44 MiB/s wr, 1.08k op/s; 6398/241034 objects degraded (2.654%); 17 MiB/s, 266 objects/s recovering 2024-05-15T23:40:37.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:37 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 7414/245990 objects degraded (3.014%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:37 smithi088 ceph-mon[44699]: osdmap e767: 12 total, 11 up, 9 in 2024-05-15T23:40:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:37 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/216874684' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:40:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:37 smithi088 ceph-mon[44699]: pgmap v2046: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 102 B/s rd, 44 MiB/s wr, 1.08k op/s; 6398/241034 objects degraded (2.654%); 17 MiB/s, 266 objects/s recovering 2024-05-15T23:40:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:37 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 7414/245990 objects degraded (3.014%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:37.832 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:37 smithi084 ceph-mon[39784]: osdmap e767: 12 total, 11 up, 9 in 2024-05-15T23:40:37.832 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:37 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/216874684' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:40:37.832 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:37.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:37.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:37.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:37.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:37.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:37 smithi084 ceph-mon[39784]: pgmap v2046: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 102 B/s rd, 44 MiB/s wr, 1.08k op/s; 6398/241034 objects degraded (2.654%); 17 MiB/s, 266 objects/s recovering 2024-05-15T23:40:37.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:37 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 7414/245990 objects degraded (3.014%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:38.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:39.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:39 smithi090 ceph-mon[44434]: pgmap v2047: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 102 B/s rd, 38 MiB/s wr, 1.08k op/s; 5969/238916 objects degraded (2.498%); 17 MiB/s, 273 objects/s recovering 2024-05-15T23:40:39.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:39 smithi088 ceph-mon[44699]: pgmap v2047: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 102 B/s rd, 38 MiB/s wr, 1.08k op/s; 5969/238916 objects degraded (2.498%); 17 MiB/s, 273 objects/s recovering 2024-05-15T23:40:39.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:39 smithi084 ceph-mon[39784]: pgmap v2047: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 102 B/s rd, 38 MiB/s wr, 1.08k op/s; 5969/238916 objects degraded (2.498%); 17 MiB/s, 273 objects/s recovering 2024-05-15T23:40:40.426 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:40:40.432 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:40:40.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:40.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:40:40.542 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:40:40.571 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:40.595 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:40:40.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:40.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:40:40.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:40:40.747 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:40.775 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:40:40.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:40.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:40:40.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:40:40.953 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:41.606 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:41 smithi084 ceph-mon[39784]: pgmap v2048: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.2 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 102 B/s rd, 30 MiB/s wr, 1.09k op/s; 5969/236130 objects degraded (2.528%); 17 MiB/s, 273 objects/s recovering 2024-05-15T23:40:41.694 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:40:41.695 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:40:41.695 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-05-15T23:40:41.695 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:40:41.810 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:41 smithi090 ceph-mon[44434]: pgmap v2048: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.2 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 102 B/s rd, 30 MiB/s wr, 1.09k op/s; 5969/236130 objects degraded (2.528%); 17 MiB/s, 273 objects/s recovering 2024-05-15T23:40:41.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:41 smithi088 ceph-mon[44699]: pgmap v2048: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.2 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 102 B/s rd, 30 MiB/s wr, 1.09k op/s; 5969/236130 objects degraded (2.528%); 17 MiB/s, 273 objects/s recovering 2024-05-15T23:40:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:40:42.200 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:40:42.553 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T23:40:42.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:42 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 5969/236130 objects degraded (2.528%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:42.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:42 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 5969/236130 objects degraded (2.528%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:42.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:42 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 5969/236130 objects degraded (2.528%), 4 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:43.458 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:40:43.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:43 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/808235804' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:40:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:43 smithi084 ceph-mon[39784]: from='client.37493 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:40:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:43 smithi084 ceph-mon[39784]: pgmap v2049: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 102 B/s rd, 7.4 MiB/s wr, 1.11k op/s; 4326/228422 objects degraded (1.894%); 19 MiB/s, 308 objects/s recovering 2024-05-15T23:40:43.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:43 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/808235804' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:40:43.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:43 smithi090 ceph-mon[44434]: from='client.37493 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:40:43.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:43 smithi090 ceph-mon[44434]: pgmap v2049: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 102 B/s rd, 7.4 MiB/s wr, 1.11k op/s; 4326/228422 objects degraded (1.894%); 19 MiB/s, 308 objects/s recovering 2024-05-15T23:40:43.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:43 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/808235804' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:40:43.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:43 smithi088 ceph-mon[44699]: from='client.37493 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:40:43.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:43 smithi088 ceph-mon[44699]: pgmap v2049: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 7.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 102 B/s rd, 7.4 MiB/s wr, 1.11k op/s; 4326/228422 objects degraded (1.894%); 19 MiB/s, 308 objects/s recovering 2024-05-15T23:40:44.201 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:40:44.394 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:40:44.427 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:40:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:44.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:40:44.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:40:44.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:44 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:40:44.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:44 smithi084 ceph-mon[39784]: from='client.37493 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:40:44.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:44 smithi084 ceph-mon[39784]: osdmap e768: 12 total, 11 up, 9 in 2024-05-15T23:40:44.573 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:44.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:44 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:40:44.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:44 smithi090 ceph-mon[44434]: from='client.37493 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:40:44.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:44 smithi090 ceph-mon[44434]: osdmap e768: 12 total, 11 up, 9 in 2024-05-15T23:40:44.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:44 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:40:44.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:44 smithi088 ceph-mon[44699]: from='client.37493 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:40:44.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:44 smithi088 ceph-mon[44699]: osdmap e768: 12 total, 11 up, 9 in 2024-05-15T23:40:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:45 smithi090 ceph-mon[44434]: osdmap e769: 12 total, 11 up, 9 in 2024-05-15T23:40:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:45 smithi090 ceph-mon[44434]: 9.4 scrub starts 2024-05-15T23:40:45.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:45 smithi090 ceph-mon[44434]: pgmap v2052: 17 pgs: 1 peering, 3 active+recovering+degraded, 13 active+clean; 6.9 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 4091/224788 objects degraded (1.820%); 18 MiB/s, 288 objects/s recovering 2024-05-15T23:40:45.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:45 smithi088 ceph-mon[44699]: osdmap e769: 12 total, 11 up, 9 in 2024-05-15T23:40:45.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:45 smithi088 ceph-mon[44699]: 9.4 scrub starts 2024-05-15T23:40:45.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:45 smithi088 ceph-mon[44699]: pgmap v2052: 17 pgs: 1 peering, 3 active+recovering+degraded, 13 active+clean; 6.9 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 4091/224788 objects degraded (1.820%); 18 MiB/s, 288 objects/s recovering 2024-05-15T23:40:45.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:45 smithi084 ceph-mon[39784]: osdmap e769: 12 total, 11 up, 9 in 2024-05-15T23:40:45.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:45 smithi084 ceph-mon[39784]: 9.4 scrub starts 2024-05-15T23:40:45.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:45 smithi084 ceph-mon[39784]: pgmap v2052: 17 pgs: 1 peering, 3 active+recovering+degraded, 13 active+clean; 6.9 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 4091/224788 objects degraded (1.820%); 18 MiB/s, 288 objects/s recovering 2024-05-15T23:40:46.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:46 smithi090 ceph-mon[44434]: 9.b scrub starts 2024-05-15T23:40:46.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:46 smithi090 ceph-mon[44434]: 9.4 scrub ok 2024-05-15T23:40:46.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:46 smithi090 ceph-mon[44434]: osdmap e770: 12 total, 11 up, 9 in 2024-05-15T23:40:46.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:46 smithi090 ceph-mon[44434]: 9.b scrub ok 2024-05-15T23:40:46.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:46 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:40:46.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:46 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:40:46.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:46 smithi088 ceph-mon[44699]: 9.b scrub starts 2024-05-15T23:40:46.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:46 smithi088 ceph-mon[44699]: 9.4 scrub ok 2024-05-15T23:40:46.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:46 smithi088 ceph-mon[44699]: osdmap e770: 12 total, 11 up, 9 in 2024-05-15T23:40:46.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:46 smithi088 ceph-mon[44699]: 9.b scrub ok 2024-05-15T23:40:46.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:46 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:40:46.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:46 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:40:46.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:46 smithi084 ceph-mon[39784]: 9.b scrub starts 2024-05-15T23:40:46.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:46 smithi084 ceph-mon[39784]: 9.4 scrub ok 2024-05-15T23:40:46.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:46 smithi084 ceph-mon[39784]: osdmap e770: 12 total, 11 up, 9 in 2024-05-15T23:40:46.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:46 smithi084 ceph-mon[39784]: 9.b scrub ok 2024-05-15T23:40:46.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:46 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:40:46.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:46 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:40:47.223 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:40:47.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:47 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:40:47.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:47 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:40:47.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:47 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:40:47.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:47 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:40:47.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:47 smithi084 ceph-mon[39784]: osdmap e771: 12 total, 11 up, 9 in 2024-05-15T23:40:47.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:47 smithi084 ceph-mon[39784]: pgmap v2055: 17 pgs: 1 peering, 3 active+recovering+degraded, 13 active+clean; 6.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 3386/220242 objects degraded (1.537%); 15 MiB/s, 234 objects/s recovering 2024-05-15T23:40:47.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:47 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 4091/224788 objects degraded (1.820%), 3 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:47.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:47 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:40:47.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:47 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:40:47.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:47 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:40:47.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:47 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:40:47.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:47 smithi090 ceph-mon[44434]: osdmap e771: 12 total, 11 up, 9 in 2024-05-15T23:40:47.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:47 smithi090 ceph-mon[44434]: pgmap v2055: 17 pgs: 1 peering, 3 active+recovering+degraded, 13 active+clean; 6.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 3386/220242 objects degraded (1.537%); 15 MiB/s, 234 objects/s recovering 2024-05-15T23:40:47.810 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:47 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 4091/224788 objects degraded (1.820%), 3 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:47.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:47 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:40:47.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:47 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:40:47.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:47 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:40:47.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:47 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:40:47.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:47 smithi088 ceph-mon[44699]: osdmap e771: 12 total, 11 up, 9 in 2024-05-15T23:40:47.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:47 smithi088 ceph-mon[44699]: pgmap v2055: 17 pgs: 1 peering, 3 active+recovering+degraded, 13 active+clean; 6.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 3386/220242 objects degraded (1.537%); 15 MiB/s, 234 objects/s recovering 2024-05-15T23:40:47.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:47 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 4091/224788 objects degraded (1.820%), 3 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:48.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:48 smithi090 ceph-mon[44434]: 9.9 scrub starts 2024-05-15T23:40:48.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:48 smithi088 ceph-mon[44699]: 9.9 scrub starts 2024-05-15T23:40:48.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:48 smithi084 ceph-mon[39784]: 9.9 scrub starts 2024-05-15T23:40:49.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:40:49.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:49 smithi090 ceph-mon[44434]: 9.9 scrub ok 2024-05-15T23:40:49.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:49 smithi090 ceph-mon[44434]: 9.0 scrub starts 2024-05-15T23:40:49.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:49 smithi090 ceph-mon[44434]: pgmap v2056: 17 pgs: 1 peering, 2 active+recovering+degraded, 14 active+clean; 6.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 829 op/s; 2762/218614 objects degraded (1.263%); 17 MiB/s, 264 objects/s recovering 2024-05-15T23:40:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:49 smithi088 ceph-mon[44699]: 9.9 scrub ok 2024-05-15T23:40:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:49 smithi088 ceph-mon[44699]: 9.0 scrub starts 2024-05-15T23:40:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:49 smithi088 ceph-mon[44699]: pgmap v2056: 17 pgs: 1 peering, 2 active+recovering+degraded, 14 active+clean; 6.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 829 op/s; 2762/218614 objects degraded (1.263%); 17 MiB/s, 264 objects/s recovering 2024-05-15T23:40:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:49 smithi084 ceph-mon[39784]: 9.9 scrub ok 2024-05-15T23:40:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:49 smithi084 ceph-mon[39784]: 9.0 scrub starts 2024-05-15T23:40:49.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:49 smithi084 ceph-mon[39784]: pgmap v2056: 17 pgs: 1 peering, 2 active+recovering+degraded, 14 active+clean; 6.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 829 op/s; 2762/218614 objects degraded (1.263%); 17 MiB/s, 264 objects/s recovering 2024-05-15T23:40:50.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:50 smithi084 ceph-mon[39784]: 9.5 scrub starts 2024-05-15T23:40:50.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:50 smithi084 ceph-mon[39784]: 9.0 scrub ok 2024-05-15T23:40:50.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:50 smithi090 ceph-mon[44434]: 9.5 scrub starts 2024-05-15T23:40:50.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:50 smithi090 ceph-mon[44434]: 9.0 scrub ok 2024-05-15T23:40:50.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:50 smithi088 ceph-mon[44699]: 9.5 scrub starts 2024-05-15T23:40:50.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:50 smithi088 ceph-mon[44699]: 9.0 scrub ok 2024-05-15T23:40:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:51 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:40:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:51 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:40:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:51 smithi090 ceph-mon[44434]: 9.5 scrub ok 2024-05-15T23:40:51.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:51 smithi090 ceph-mon[44434]: pgmap v2057: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 6.6 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 899 op/s; 2762/215386 objects degraded (1.282%); 14 MiB/s, 227 objects/s recovering 2024-05-15T23:40:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:51 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:40:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:51 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:40:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:51 smithi088 ceph-mon[44699]: 9.5 scrub ok 2024-05-15T23:40:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:51 smithi088 ceph-mon[44699]: pgmap v2057: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 6.6 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 899 op/s; 2762/215386 objects degraded (1.282%); 14 MiB/s, 227 objects/s recovering 2024-05-15T23:40:51.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:51 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:40:51.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:51 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:40:51.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:51 smithi084 ceph-mon[39784]: 9.5 scrub ok 2024-05-15T23:40:51.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:51 smithi084 ceph-mon[39784]: pgmap v2057: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 6.6 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 899 op/s; 2762/215386 objects degraded (1.282%); 14 MiB/s, 227 objects/s recovering 2024-05-15T23:40:52.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:52 smithi084 ceph-mon[39784]: 9.e scrub starts 2024-05-15T23:40:52.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:52 smithi084 ceph-mon[39784]: from='client.17568 v1:172.21.15.84:0/946345999' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:40:52.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:52 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 2762/215386 objects degraded (1.282%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:52.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:52 smithi090 ceph-mon[44434]: 9.e scrub starts 2024-05-15T23:40:52.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:52 smithi090 ceph-mon[44434]: from='client.17568 v1:172.21.15.84:0/946345999' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:40:52.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:52 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 2762/215386 objects degraded (1.282%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:52.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:52 smithi088 ceph-mon[44699]: 9.e scrub starts 2024-05-15T23:40:52.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:52 smithi088 ceph-mon[44699]: from='client.17568 v1:172.21.15.84:0/946345999' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:40:52.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:52 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 2762/215386 objects degraded (1.282%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:52.879 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:40:52.879 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:40:53.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:53 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:40:53.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:53 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:40:53.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:53 smithi090 ceph-mon[44434]: osdmap e772: 12 total, 11 up, 9 in 2024-05-15T23:40:53.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:53 smithi090 ceph-mon[44434]: 9.e scrub ok 2024-05-15T23:40:53.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:53 smithi090 ceph-mon[44434]: 9.d scrub starts 2024-05-15T23:40:53.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:53 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:40:53.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:53 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:40:53.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:53 smithi090 ceph-mon[44434]: pgmap v2059: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 6.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 2179/208782 objects degraded (1.044%); 15 MiB/s, 242 objects/s recovering 2024-05-15T23:40:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:53 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:40:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:53 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:40:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:53 smithi088 ceph-mon[44699]: osdmap e772: 12 total, 11 up, 9 in 2024-05-15T23:40:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:53 smithi088 ceph-mon[44699]: 9.e scrub ok 2024-05-15T23:40:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:53 smithi088 ceph-mon[44699]: 9.d scrub starts 2024-05-15T23:40:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:53 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:40:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:53 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:40:53.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:53 smithi088 ceph-mon[44699]: pgmap v2059: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 6.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 2179/208782 objects degraded (1.044%); 15 MiB/s, 242 objects/s recovering 2024-05-15T23:40:53.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:53 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:40:53.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:53 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:40:53.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:53 smithi084 ceph-mon[39784]: osdmap e772: 12 total, 11 up, 9 in 2024-05-15T23:40:53.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:53 smithi084 ceph-mon[39784]: 9.e scrub ok 2024-05-15T23:40:53.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:53 smithi084 ceph-mon[39784]: 9.d scrub starts 2024-05-15T23:40:53.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:53 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:40:53.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:53 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:40:53.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:53 smithi084 ceph-mon[39784]: pgmap v2059: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 6.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 2179/208782 objects degraded (1.044%); 15 MiB/s, 242 objects/s recovering 2024-05-15T23:40:54.640 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":2057,"stamp":"2024-05-15T23:40:51.361044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7058162232,"num_objects":107693,"num_object_clones":0,"num_object_copies":215386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107693,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":140804,"num_write_kb":7949991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14746,"num_bytes_recovered":967456352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140775,"ondisk_log_size":140775,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":8,"kb":843706368,"kb_used":20130160,"kb_used_data":14122764,"kb_used_omap":39,"kb_used_meta":6007192,"kb_avail":823576208,"statfs":{"total":863955320832,"available":843342036992,"internally_reserved":0,"allocated":14461710336,"data_stored":14458112922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40938,"internal_metadata":6151364630},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,3,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-406454272,"num_objects":-6202,"num_object_clones":0,"num_object_copies":-12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1564,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6202,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1566,"num_bytes_recovered":102498304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.892825"},"pg_stats":[{"pgid":"9.f","version":"771'8498","reported_seq":14366,"reported_epoch":771,"state":"active+recovering+degraded","last_fresh":"2024-05-15T23:40:47.242534+0000","last_change":"2024-05-15T23:39:43.669795+0000","last_active":"2024-05-15T23:40:47.242534+0000","last_peered":"2024-05-15T23:40:47.242534+0000","last_clean":"2024-05-15T23:39:25.389963+0000","last_became_active":"2024-05-15T23:39:43.643192+0000","last_became_peered":"2024-05-15T23:39:43.643192+0000","last_unstale":"2024-05-15T23:40:47.242534+0000","last_undegraded":"2024-05-15T23:39:43.429912+0000","last_fullsized":"2024-05-15T23:40:47.242534+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":8498,"log_dups_size":0,"ondisk_log_size":8498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440139776,"num_objects":6716,"num_object_clones":0,"num_object_copies":13432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1456,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8498,"num_write_kb":486848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2893,"num_bytes_recovered":189595648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1456},{"shards":"3,9","objects":5260}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"771'8927","reported_seq":9004,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-05-15T23:40:49.921570+0000","last_change":"2024-05-15T23:38:50.403048+0000","last_active":"2024-05-15T23:40:49.921570+0000","last_peered":"2024-05-15T23:40:49.921570+0000","last_clean":"2024-05-15T23:40:49.921570+0000","last_became_active":"2024-05-15T23:38:50.402740+0000","last_became_peered":"2024-05-15T23:38:50.402740+0000","last_unstale":"2024-05-15T23:40:49.921570+0000","last_undegraded":"2024-05-15T23:40:49.921570+0000","last_fullsized":"2024-05-15T23:40:49.921570+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":8927,"log_dups_size":0,"ondisk_log_size":8927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431685632,"num_objects":6587,"num_object_clones":0,"num_object_copies":13174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8927,"num_write_kb":496448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"771'9136","reported_seq":9214,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-05-15T23:40:49.928343+0000","last_change":"2024-05-15T23:38:50.401477+0000","last_active":"2024-05-15T23:40:49.928343+0000","last_peered":"2024-05-15T23:40:49.928343+0000","last_clean":"2024-05-15T23:40:49.928343+0000","last_became_active":"2024-05-15T23:38:50.401123+0000","last_became_peered":"2024-05-15T23:38:50.401123+0000","last_unstale":"2024-05-15T23:40:49.928343+0000","last_undegraded":"2024-05-15T23:40:49.928343+0000","last_fullsized":"2024-05-15T23:40:49.928343+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":9136,"log_dups_size":0,"ondisk_log_size":9136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445448216,"num_objects":6798,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6798,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9136,"num_write_kb":509825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"771'8775","reported_seq":8852,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-05-15T23:40:47.213746+0000","last_change":"2024-05-15T23:38:50.402679+0000","last_active":"2024-05-15T23:40:47.213746+0000","last_peered":"2024-05-15T23:40:47.213746+0000","last_clean":"2024-05-15T23:40:47.213746+0000","last_became_active":"2024-05-15T23:38:50.402428+0000","last_became_peered":"2024-05-15T23:38:50.402428+0000","last_unstale":"2024-05-15T23:40:47.213746+0000","last_undegraded":"2024-05-15T23:40:47.213746+0000","last_fullsized":"2024-05-15T23:40:47.213746+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":8775,"log_dups_size":0,"ondisk_log_size":8775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446365696,"num_objects":6811,"num_object_clones":0,"num_object_copies":13622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8775,"num_write_kb":498752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"771'8745","reported_seq":8899,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-05-15T23:40:49.224497+0000","last_change":"2024-05-15T23:40:45.650576+0000","last_active":"2024-05-15T23:40:49.224497+0000","last_peered":"2024-05-15T23:40:49.224497+0000","last_clean":"2024-05-15T23:40:49.224497+0000","last_became_active":"2024-05-15T23:38:50.401815+0000","last_became_peered":"2024-05-15T23:38:50.401815+0000","last_unstale":"2024-05-15T23:40:49.224497+0000","last_undegraded":"2024-05-15T23:40:49.224497+0000","last_fullsized":"2024-05-15T23:40:49.224497+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'8522","last_scrub_stamp":"2024-05-15T23:40:45.650391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:45.650391+0000","objects_scrubbed":6888,"log_size":8745,"log_dups_size":0,"ondisk_log_size":8745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:58.125696+0000","scrub_duration":745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436273152,"num_objects":6657,"num_object_clones":0,"num_object_copies":13314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8745,"num_write_kb":492864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"771'8812","reported_seq":17694,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-05-15T23:40:49.877901+0000","last_change":"2024-05-15T23:40:45.515010+0000","last_active":"2024-05-15T23:40:49.877901+0000","last_peered":"2024-05-15T23:40:49.877901+0000","last_clean":"2024-05-15T23:40:49.877901+0000","last_became_active":"2024-05-15T23:40:45.476827+0000","last_became_peered":"2024-05-15T23:40:45.476827+0000","last_unstale":"2024-05-15T23:40:49.877901+0000","last_undegraded":"2024-05-15T23:40:49.877901+0000","last_fullsized":"2024-05-15T23:40:49.877901+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":8812,"log_dups_size":0,"ondisk_log_size":8812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429260800,"num_objects":6550,"num_object_clones":0,"num_object_copies":13100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8812,"num_write_kb":491584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4397,"num_bytes_recovered":288030720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"770'8749","reported_seq":8838,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-05-15T23:40:46.450838+0000","last_change":"2024-05-15T23:40:02.210276+0000","last_active":"2024-05-15T23:40:46.450838+0000","last_peered":"2024-05-15T23:40:46.450838+0000","last_clean":"2024-05-15T23:40:46.450838+0000","last_became_active":"2024-05-15T23:38:50.404189+0000","last_became_peered":"2024-05-15T23:38:50.404189+0000","last_unstale":"2024-05-15T23:40:46.450838+0000","last_undegraded":"2024-05-15T23:40:46.450838+0000","last_fullsized":"2024-05-15T23:40:46.450838+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":1049,"log_size":8749,"log_dups_size":0,"ondisk_log_size":8749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452395008,"num_objects":6903,"num_object_clones":0,"num_object_copies":13806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8749,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"770'8596","reported_seq":8731,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-05-15T23:40:46.446332+0000","last_change":"2024-05-15T23:39:55.792219+0000","last_active":"2024-05-15T23:40:46.446332+0000","last_peered":"2024-05-15T23:40:46.446332+0000","last_clean":"2024-05-15T23:40:46.446332+0000","last_became_active":"2024-05-15T23:38:50.401416+0000","last_became_peered":"2024-05-15T23:38:50.401416+0000","last_unstale":"2024-05-15T23:40:46.446332+0000","last_undegraded":"2024-05-15T23:40:46.446332+0000","last_fullsized":"2024-05-15T23:40:46.446332+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"761'5246","last_scrub_stamp":"2024-05-15T23:39:55.792018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:39:55.792018+0000","objects_scrubbed":5229,"log_size":8596,"log_dups_size":0,"ondisk_log_size":8596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:21.653921+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446038016,"num_objects":6806,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8596,"num_write_kb":492864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"771'8605","reported_seq":14857,"reported_epoch":771,"state":"active+recovering+degraded","last_fresh":"2024-05-15T23:40:47.246528+0000","last_change":"2024-05-15T23:39:43.668135+0000","last_active":"2024-05-15T23:40:47.246528+0000","last_peered":"2024-05-15T23:40:47.246528+0000","last_clean":"2024-05-15T23:39:25.398249+0000","last_became_active":"2024-05-15T23:39:43.637604+0000","last_became_peered":"2024-05-15T23:39:43.637604+0000","last_unstale":"2024-05-15T23:40:47.246528+0000","last_undegraded":"2024-05-15T23:39:43.429785+0000","last_fullsized":"2024-05-15T23:40:47.246528+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":8605,"log_dups_size":0,"ondisk_log_size":8605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439681024,"num_objects":6709,"num_object_clones":0,"num_object_copies":13418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8605,"num_write_kb":490048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3085,"num_bytes_recovered":202178560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1306},{"shards":"3,9","objects":5403}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"771'8637","reported_seq":17452,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-05-15T23:40:48.049478+0000","last_change":"2024-05-15T23:40:47.046820+0000","last_active":"2024-05-15T23:40:48.049478+0000","last_peered":"2024-05-15T23:40:48.049478+0000","last_clean":"2024-05-15T23:40:48.049478+0000","last_became_active":"2024-05-15T23:39:43.514269+0000","last_became_peered":"2024-05-15T23:39:43.514269+0000","last_unstale":"2024-05-15T23:40:48.049478+0000","last_undegraded":"2024-05-15T23:40:48.049478+0000","last_fullsized":"2024-05-15T23:40:48.049478+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":8637,"log_dups_size":0,"ondisk_log_size":8637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437190656,"num_objects":6671,"num_object_clones":0,"num_object_copies":13342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8637,"num_write_kb":489856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":286064640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"771'8762","reported_seq":8915,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-05-15T23:40:47.136324+0000","last_change":"2024-05-15T23:40:45.307444+0000","last_active":"2024-05-15T23:40:47.136324+0000","last_peered":"2024-05-15T23:40:47.136324+0000","last_clean":"2024-05-15T23:40:47.136324+0000","last_became_active":"2024-05-15T23:38:50.398410+0000","last_became_peered":"2024-05-15T23:38:50.398410+0000","last_unstale":"2024-05-15T23:40:47.136324+0000","last_undegraded":"2024-05-15T23:40:47.136324+0000","last_fullsized":"2024-05-15T23:40:47.136324+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'8645","last_scrub_stamp":"2024-05-15T23:40:45.307336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:45.307336+0000","objects_scrubbed":6921,"log_size":8762,"log_dups_size":0,"ondisk_log_size":8762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:03.552738+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445775872,"num_objects":6802,"num_object_clones":0,"num_object_copies":13604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8762,"num_write_kb":498048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"771'8921","reported_seq":8998,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-05-15T23:40:50.206675+0000","last_change":"2024-05-15T23:38:50.402064+0000","last_active":"2024-05-15T23:40:50.206675+0000","last_peered":"2024-05-15T23:40:50.206675+0000","last_clean":"2024-05-15T23:40:50.206675+0000","last_became_active":"2024-05-15T23:38:50.401779+0000","last_became_peered":"2024-05-15T23:38:50.401779+0000","last_unstale":"2024-05-15T23:40:50.206675+0000","last_undegraded":"2024-05-15T23:40:50.206675+0000","last_fullsized":"2024-05-15T23:40:50.206675+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":8921,"log_dups_size":0,"ondisk_log_size":8921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429457408,"num_objects":6553,"num_object_clones":0,"num_object_copies":13106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8921,"num_write_kb":495168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"771'8796","reported_seq":8873,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-05-15T23:40:47.198945+0000","last_change":"2024-05-15T23:38:50.402094+0000","last_active":"2024-05-15T23:40:47.198945+0000","last_peered":"2024-05-15T23:40:47.198945+0000","last_clean":"2024-05-15T23:40:47.198945+0000","last_became_active":"2024-05-15T23:38:50.401872+0000","last_became_peered":"2024-05-15T23:38:50.401872+0000","last_unstale":"2024-05-15T23:40:47.198945+0000","last_undegraded":"2024-05-15T23:40:47.198945+0000","last_fullsized":"2024-05-15T23:40:47.198945+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":8796,"log_dups_size":0,"ondisk_log_size":8796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447610880,"num_objects":6830,"num_object_clones":0,"num_object_copies":13660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8796,"num_write_kb":500032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"771'8775","reported_seq":8852,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-05-15T23:40:47.228556+0000","last_change":"2024-05-15T23:38:50.401343+0000","last_active":"2024-05-15T23:40:47.228556+0000","last_peered":"2024-05-15T23:40:47.228556+0000","last_clean":"2024-05-15T23:40:47.228556+0000","last_became_active":"2024-05-15T23:38:50.401074+0000","last_became_peered":"2024-05-15T23:38:50.401074+0000","last_unstale":"2024-05-15T23:40:47.228556+0000","last_undegraded":"2024-05-15T23:40:47.228556+0000","last_fullsized":"2024-05-15T23:40:47.228556+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":8775,"log_dups_size":0,"ondisk_log_size":8775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450035712,"num_objects":6867,"num_object_clones":0,"num_object_copies":13734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8775,"num_write_kb":500544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"771'8901","reported_seq":8978,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-05-15T23:40:48.036648+0000","last_change":"2024-05-15T23:38:50.402309+0000","last_active":"2024-05-15T23:40:48.036648+0000","last_peered":"2024-05-15T23:40:48.036648+0000","last_clean":"2024-05-15T23:40:48.036648+0000","last_became_active":"2024-05-15T23:38:50.401998+0000","last_became_peered":"2024-05-15T23:38:50.401998+0000","last_unstale":"2024-05-15T23:40:48.036648+0000","last_undegraded":"2024-05-15T23:40:48.036648+0000","last_fullsized":"2024-05-15T23:40:48.036648+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":8901,"log_dups_size":0,"ondisk_log_size":8901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445317120,"num_objects":6795,"num_object_clones":0,"num_object_copies":13590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8901,"num_write_kb":502272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.8","version":"771'9046","reported_seq":9123,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-05-15T23:40:50.213596+0000","last_change":"2024-05-15T23:38:50.404457+0000","last_active":"2024-05-15T23:40:50.213596+0000","last_peered":"2024-05-15T23:40:50.213596+0000","last_clean":"2024-05-15T23:40:50.213596+0000","last_became_active":"2024-05-15T23:38:50.404155+0000","last_became_peered":"2024-05-15T23:38:50.404155+0000","last_unstale":"2024-05-15T23:40:50.213596+0000","last_undegraded":"2024-05-15T23:40:50.213596+0000","last_fullsized":"2024-05-15T23:40:50.213596+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":9046,"log_dups_size":0,"ondisk_log_size":9046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434896896,"num_objects":6636,"num_object_clones":0,"num_object_copies":13272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9046,"num_write_kb":501824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1669,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-05-15T23:40:46.480889+0000","last_change":"2024-05-15T23:38:23.156562+0000","last_active":"2024-05-15T23:40:46.480889+0000","last_peered":"2024-05-15T23:40:46.480889+0000","last_clean":"2024-05-15T23:40:46.480889+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:40:46.480889+0000","last_undegraded":"2024-05-15T23:40:46.480889+0000","last_fullsized":"2024-05-15T23:40:46.480889+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:38:23.156474+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:38:23.156474+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7057571864,"num_objects":107691,"num_object_clones":0,"num_object_copies":215382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107691,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140681,"num_write_kb":7947841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14740,"num_bytes_recovered":965869568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14990188544,"data_stored":14990180400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140681,"ondisk_log_size":140681,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351421,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2621840,"kb_used_data":1734088,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":91123312,"statfs":{"total":95995035648,"available":93310271488,"internally_reserved":0,"allocated":1775706112,"data_stored":1775303871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":909048260},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580353,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2385788,"kb_used_data":1759028,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":91359364,"statfs":{"total":95995035648,"available":93551988736,"internally_reserved":0,"allocated":1801244672,"data_stored":1800851878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":641792448},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776586,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2775432,"kb_used_data":1954304,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":90969720,"statfs":{"total":95995035648,"available":93152993280,"internally_reserved":0,"allocated":2001207296,"data_stored":2000808627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":840825280},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":520,"seq":2233382994089,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":549128,"kb_used_data":427520,"kb_used_omap":24,"kb_used_meta":121575,"kb_avail":93196024,"statfs":{"total":95995035648,"available":95432728576,"internally_reserved":0,"allocated":437780480,"data_stored":437381811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25357,"internal_metadata":124493043},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267155,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3036564,"kb_used_data":2337612,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":90708588,"statfs":{"total":95995035648,"available":92885594112,"internally_reserved":0,"allocated":2393714688,"data_stored":2393308395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":715717066},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568496088,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575346,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3382292,"kb_used_data":2448708,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":90362860,"statfs":{"total":95995035648,"available":92531568640,"internally_reserved":0,"allocated":2507476992,"data_stored":2507074251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":955972038},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016756,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":209424,"kb_used_data":768,"kb_used_omap":4,"kb_used_meta":208635,"kb_avail":93535728,"statfs":{"total":95995035648,"available":95780585472,"internally_reserved":0,"allocated":786432,"data_stored":387763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":213642947},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150185,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3595092,"kb_used_data":2607296,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":90150060,"statfs":{"total":95995035648,"available":92313661440,"internally_reserved":0,"allocated":2669871104,"data_stored":2669472435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1011415495},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986657,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1574600,"kb_used_data":853440,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92170552,"statfs":{"total":95995035648,"available":94382645248,"internally_reserved":0,"allocated":873922560,"data_stored":873523891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2241728512,"data_stored":2241724440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2666659840,"data_stored":2666659840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":872873984,"data_stored":872873984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1040711680,"data_stored":1040711680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2167738368,"data_stored":2167734296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":436273152,"data_stored":436273152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1997799424,"data_stored":1997799424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1793916928,"data_stored":1793916928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1772486656,"data_stored":1772486656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:40:54.643 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:40:54.677 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:40:54.677 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:40:54.677 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:40:54.678 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:40:54.678 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:40:54.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:54 smithi088 ceph-mon[44699]: osdmap e773: 12 total, 11 up, 9 in 2024-05-15T23:40:54.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:54 smithi088 ceph-mon[44699]: 9.d scrub ok 2024-05-15T23:40:54.870 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:40:54.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:54 smithi084 ceph-mon[39784]: osdmap e773: 12 total, 11 up, 9 in 2024-05-15T23:40:54.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:54 smithi084 ceph-mon[39784]: 9.d scrub ok 2024-05-15T23:40:54.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:54.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:40:54.979 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:40:55.009 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:55.033 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:40:55.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:55.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:40:55.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:54 smithi090 ceph-mon[44434]: osdmap e773: 12 total, 11 up, 9 in 2024-05-15T23:40:55.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:54 smithi090 ceph-mon[44434]: 9.d scrub ok 2024-05-15T23:40:55.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:40:55.177 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:55.223 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:40:55.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:55.229 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:40:55.330 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:40:55.366 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:55.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T23:40:55.411 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:40:55.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:55.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:40:55.422 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:40:55.518 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:40:55.572 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:55.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:55 smithi084 ceph-mon[39784]: 9.a scrub starts 2024-05-15T23:40:55.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:55 smithi084 ceph-mon[39784]: pgmap v2061: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+degraded, 14 active+clean; 6.3 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 960 op/s; 2179/204854 objects degraded (1.064%); 9.4 MiB/s, 150 objects/s recovering 2024-05-15T23:40:55.683 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:40:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:55.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:40:55.791 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:40:55.817 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:55 smithi088 ceph-mon[44699]: 9.a scrub starts 2024-05-15T23:40:55.818 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:55 smithi088 ceph-mon[44699]: pgmap v2061: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+degraded, 14 active+clean; 6.3 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 960 op/s; 2179/204854 objects degraded (1.064%); 9.4 MiB/s, 150 objects/s recovering 2024-05-15T23:40:55.820 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:55.843 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:40:55.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:55.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:40:55.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:40:55.990 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:56.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:55 smithi090 ceph-mon[44434]: 9.a scrub starts 2024-05-15T23:40:56.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:55 smithi090 ceph-mon[44434]: pgmap v2061: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+degraded, 14 active+clean; 6.3 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 960 op/s; 2179/204854 objects degraded (1.064%); 9.4 MiB/s, 150 objects/s recovering 2024-05-15T23:40:56.630 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:40:56.638 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:40:56.648 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:40:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:56.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:40:56.671 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:40:56.694 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:40:56.723 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:40:56.753 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:40:56.754 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:40:56.781 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:40:56.794 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:56.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:56 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:40:56.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:56 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:40:56.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:56 smithi088 ceph-mon[44699]: osdmap e774: 12 total, 11 up, 9 in 2024-05-15T23:40:56.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:56 smithi088 ceph-mon[44699]: 9.a scrub ok 2024-05-15T23:40:56.824 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:40:56.825 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:40:56.828 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:56 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:40:56.828 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:56 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:40:56.829 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:56 smithi090 ceph-mon[44434]: osdmap e774: 12 total, 11 up, 9 in 2024-05-15T23:40:56.829 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:56 smithi090 ceph-mon[44434]: 9.a scrub ok 2024-05-15T23:40:56.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:56.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:40:56.867 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:40:56.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:56 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:40:56.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:56 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:40:56.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:56 smithi084 ceph-mon[39784]: osdmap e774: 12 total, 11 up, 9 in 2024-05-15T23:40:56.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:56 smithi084 ceph-mon[39784]: 9.a scrub ok 2024-05-15T23:40:56.895 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-05-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:40:56.930 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:40:56.954 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:40:56.959 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:56.977 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:40:56.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:56.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:40:57.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:40:57.133 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:57.169 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:40:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:57.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:40:57.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:40:57.323 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:57.359 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:40:57.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:57.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:40:57.473 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:40:57.525 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:57.553 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:40:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:57.557 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:40:57.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:40:57.712 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:57.733 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:40:57.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:57.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:40:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1766718745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:40:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:57 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:57 smithi090 ceph-mon[44434]: pgmap v2063: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+degraded, 14 active+clean; 6.1 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.33k op/s; 1517/199414 objects degraded (0.761%); 13 MiB/s, 207 objects/s recovering 2024-05-15T23:40:57.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:57 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 2179/204854 objects degraded (1.064%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:57.840 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:40:57.866 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1766718745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:40:57.866 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.866 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.866 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.866 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.866 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.867 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.867 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.867 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:57 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.867 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:57 smithi088 ceph-mon[44699]: pgmap v2063: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+degraded, 14 active+clean; 6.1 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.33k op/s; 1517/199414 objects degraded (0.761%); 13 MiB/s, 207 objects/s recovering 2024-05-15T23:40:57.867 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:57 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 2179/204854 objects degraded (1.064%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:57.868 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1766718745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:40:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:57 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:57 smithi084 ceph-mon[39784]: pgmap v2063: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+degraded, 14 active+clean; 6.1 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.33k op/s; 1517/199414 objects degraded (0.761%); 13 MiB/s, 207 objects/s recovering 2024-05-15T23:40:57.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:57 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 2179/204854 objects degraded (1.064%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:40:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:40:57.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:57.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:40:57.996 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:40:58.025 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:58.047 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:40:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:58.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:40:58.153 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:40:58.181 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:58.204 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:40:58.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:58.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:40:58.308 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:40:58.350 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:58.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:58.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:58.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:58 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:58.659 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:40:58.660 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:40:58.876 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:40:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:59.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:58 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:59.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:59.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:59.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:58 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:40:59.108 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:40:59.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:59.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:40:59.218 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:40:59.274 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:59.312 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:40:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:59.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:40:59.416 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:40:59.446 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:59.462 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:40:59.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:59.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:40:59.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:40:59.602 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:59.614 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:40:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:40:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:59.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:40:59.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:40:59.772 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:40:59.814 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:40:59.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:40:59.818 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:40:59.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:59 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/646196930' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:40:59.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:40:59 smithi088 ceph-mon[44699]: pgmap v2064: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+degraded, 14 active+clean; 6.0 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 927 op/s; 1517/197436 objects degraded (0.768%); 6.1 MiB/s, 96 objects/s recovering 2024-05-15T23:40:59.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:59 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/646196930' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:40:59.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:40:59 smithi084 ceph-mon[39784]: pgmap v2064: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+degraded, 14 active+clean; 6.0 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 927 op/s; 1517/197436 objects degraded (0.768%); 6.1 MiB/s, 96 objects/s recovering 2024-05-15T23:40:59.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:40:59.979 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:59 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/646196930' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:40:59.979 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:40:59 smithi090 ceph-mon[44434]: pgmap v2064: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+degraded, 14 active+clean; 6.0 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 927 op/s; 1517/197436 objects degraded (0.768%); 6.1 MiB/s, 96 objects/s recovering 2024-05-15T23:40:59.980 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:00.008 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:00.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:00.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:00.113 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:41:00.140 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:41:00.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:00.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:41:00.266 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:41:00.295 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:00.314 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:41:00.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:00.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:41:00.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:41:00.476 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:41:00.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:00.522 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:41:00.623 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:41:00.661 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:41:00.665 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:00.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:00 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:41:00.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:00 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/646196930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:41:00.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:00 smithi090 ceph-mon[44434]: osdmap e775: 12 total, 11 up, 9 in 2024-05-15T23:41:00.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:00 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:41:00.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:00 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/646196930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:41:00.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:00 smithi084 ceph-mon[39784]: osdmap e775: 12 total, 11 up, 9 in 2024-05-15T23:41:01.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:00 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:41:01.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:00 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/646196930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:41:01.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:00 smithi088 ceph-mon[44699]: osdmap e775: 12 total, 11 up, 9 in 2024-05-15T23:41:01.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:01 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:41:01.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:01 smithi084 ceph-mon[39784]: pgmap v2066: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 5.9 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 951 op/s; 1517/193906 objects degraded (0.782%); 5.3 MiB/s, 84 objects/s recovering 2024-05-15T23:41:02.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:01 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:41:02.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:01 smithi090 ceph-mon[44434]: pgmap v2066: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 5.9 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 951 op/s; 1517/193906 objects degraded (0.782%); 5.3 MiB/s, 84 objects/s recovering 2024-05-15T23:41:02.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:01 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:41:02.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:01 smithi088 ceph-mon[44699]: pgmap v2066: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 5.9 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 951 op/s; 1517/193906 objects degraded (0.782%); 5.3 MiB/s, 84 objects/s recovering 2024-05-15T23:41:02.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:02 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:41:02.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:02 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:41:02.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:02 smithi084 ceph-mon[39784]: osdmap e776: 12 total, 11 up, 9 in 2024-05-15T23:41:02.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:02 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 1517/193906 objects degraded (0.782%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:41:03.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:02 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:41:03.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:02 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:41:03.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:02 smithi090 ceph-mon[44434]: osdmap e776: 12 total, 11 up, 9 in 2024-05-15T23:41:03.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:02 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 1517/193906 objects degraded (0.782%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:41:03.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:02 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:41:03.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:02 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:41:03.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:02 smithi088 ceph-mon[44699]: osdmap e776: 12 total, 11 up, 9 in 2024-05-15T23:41:03.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:02 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 1517/193906 objects degraded (0.782%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:41:03.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:03 smithi084 ceph-mon[39784]: pgmap v2068: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 5.7 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 860/187516 objects degraded (0.459%); 11 MiB/s, 169 objects/s recovering 2024-05-15T23:41:04.031 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:41:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:04.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:41:04.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:03 smithi090 ceph-mon[44434]: pgmap v2068: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 5.7 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 860/187516 objects degraded (0.459%); 11 MiB/s, 169 objects/s recovering 2024-05-15T23:41:04.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:03 smithi088 ceph-mon[44699]: pgmap v2068: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 5.7 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 860/187516 objects degraded (0.459%); 11 MiB/s, 169 objects/s recovering 2024-05-15T23:41:04.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:41:04.199 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:04.557 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:41:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:04.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:41:04.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:41:04.731 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:04.764 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:41:04.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:04.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:41:04.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:04 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:41:04.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:04 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:41:04.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:04 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:41:04.866 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:41:04.894 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:04 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:41:04.895 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:04 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:41:04.895 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:04 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:41:04.897 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:04.918 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:41:04.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:04.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:41:05.023 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:41:05.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:04 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:41:05.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:04 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:41:05.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:04 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:41:05.073 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:05.176 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:41:05.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:05.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:41:05.290 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:05.332 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:05.422 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:05.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:05.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:05.528 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:41:05.556 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:41:05.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:05.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:41:05.682 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:05.718 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:05.748 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:05.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:05.754 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:05.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:05 smithi088 ceph-mon[44699]: pgmap v2069: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 5.6 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 984 op/s; 860/183664 objects degraded (0.468%); 5.1 MiB/s, 82 objects/s recovering 2024-05-15T23:41:05.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:41:05.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:05 smithi084 ceph-mon[39784]: pgmap v2069: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 5.6 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 984 op/s; 860/183664 objects degraded (0.468%); 5.1 MiB/s, 82 objects/s recovering 2024-05-15T23:41:05.893 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:05.921 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:41:05.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:05.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:41:06.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:41:06.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:05 smithi090 ceph-mon[44434]: pgmap v2069: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 5.6 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 984 op/s; 860/183664 objects degraded (0.468%); 5.1 MiB/s, 82 objects/s recovering 2024-05-15T23:41:06.090 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:06.110 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:06.132 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:41:06.136 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:41:06.136 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:41:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:06.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:41:06.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:06.299 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:06.330 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:06.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:06.338 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:06.395 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:41:06.400 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:41:06.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:06.498 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:06.528 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:06.532 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:06.633 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:06.681 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:06.838 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:06.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:06.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:06.946 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:41:06.976 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:06.999 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:41:07.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:07.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:41:07.105 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:41:07.143 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:07.173 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:41:07.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:07.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:41:07.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:41:07.285 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:07.334 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:07.365 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:07.366 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:07.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:07 smithi090 ceph-mon[44434]: from='client.17586 v1:172.21.15.84:0/3623210066' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:41:07.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:07 smithi090 ceph-mon[44434]: pgmap v2070: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.18k op/s; 184/178516 objects degraded (0.103%); 10 MiB/s, 166 objects/s recovering 2024-05-15T23:41:07.432 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:07 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 860/183664 objects degraded (0.468%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:41:07.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:41:07.522 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:07.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:07 smithi088 ceph-mon[44699]: from='client.17586 v1:172.21.15.84:0/3623210066' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:41:07.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:07 smithi088 ceph-mon[44699]: pgmap v2070: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.18k op/s; 184/178516 objects degraded (0.103%); 10 MiB/s, 166 objects/s recovering 2024-05-15T23:41:07.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:07 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 860/183664 objects degraded (0.468%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:41:07.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:07 smithi084 ceph-mon[39784]: from='client.17586 v1:172.21.15.84:0/3623210066' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:41:07.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:07 smithi084 ceph-mon[39784]: pgmap v2070: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.18k op/s; 184/178516 objects degraded (0.103%); 10 MiB/s, 166 objects/s recovering 2024-05-15T23:41:07.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:07 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 860/183664 objects degraded (0.468%), 2 pgs degraded (PG_DEGRADED) 2024-05-15T23:41:07.979 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:41:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:07.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:41:08.084 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:41:08.112 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:08.133 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:41:08.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:08.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:41:08.237 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:41:08.265 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:08.287 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:41:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:08.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:41:08.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:41:08.439 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:08.467 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:41:08.470 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T23:41:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:08.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:41:08.574 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:41:08.602 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:08.621 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:41:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:08.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:41:08.725 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:41:08.753 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:08.773 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:41:08.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:08.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:41:08.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:08.924 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:09.140 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:41:09.154 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:41:09.180 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:41:09.205 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:41:09.227 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:41:09.261 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:41:09.288 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:41:09.315 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:41:09.341 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:41:09.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:09 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3484997582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:41:09.374 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:41:09.408 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:41:09.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:41:09.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:41:09.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:41:09.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:41:09.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:41:09.408 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-05-15T23:41:09.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:41:09.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:41:09.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:41:09.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:41:09.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:41:09.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:41:09.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:09 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3484997582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:41:09.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:09 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3484997582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:41:09.621 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:09.661 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":2069,"stamp":"2024-05-15T23:41:05.368032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6018695736,"num_objects":91832,"num_object_clones":0,"num_object_copies":183664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91832,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":156665,"num_write_kb":7949991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16648,"num_bytes_recovered":1092105824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156536,"ondisk_log_size":156536,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":8,"kb":843706368,"kb_used":18217408,"kb_used_data":12178084,"kb_used_omap":39,"kb_used_meta":6039128,"kb_avail":825488960,"statfs":{"total":863955320832,"available":845300695040,"internally_reserved":0,"allocated":12470358016,"data_stored":12467036830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40938,"internal_metadata":6184067094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-516096000,"num_objects":-7875,"num_object_clones":0,"num_object_copies":-15750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7875,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":43057152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001627"},"pg_stats":[{"pgid":"9.f","version":"776'9484","reported_seq":17218,"reported_epoch":776,"state":"active+recovering+degraded","last_fresh":"2024-05-15T23:41:02.273066+0000","last_change":"2024-05-15T23:39:43.669795+0000","last_active":"2024-05-15T23:41:02.273066+0000","last_peered":"2024-05-15T23:41:02.273066+0000","last_clean":"2024-05-15T23:39:25.389963+0000","last_became_active":"2024-05-15T23:39:43.643192+0000","last_became_peered":"2024-05-15T23:39:43.643192+0000","last_unstale":"2024-05-15T23:41:02.273066+0000","last_undegraded":"2024-05-15T23:39:43.429912+0000","last_fullsized":"2024-05-15T23:41:02.273066+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":9484,"log_dups_size":0,"ondisk_log_size":9484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375521280,"num_objects":5730,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9484,"num_write_kb":486848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3823,"num_bytes_recovered":250544128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":526},{"shards":"3,9","objects":5204}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"776'9902","reported_seq":10063,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-05-15T23:41:04.956528+0000","last_change":"2024-05-15T23:40:52.788179+0000","last_active":"2024-05-15T23:41:04.956528+0000","last_peered":"2024-05-15T23:41:04.956528+0000","last_clean":"2024-05-15T23:41:04.956528+0000","last_became_active":"2024-05-15T23:38:50.402740+0000","last_became_peered":"2024-05-15T23:38:50.402740+0000","last_unstale":"2024-05-15T23:41:04.956528+0000","last_undegraded":"2024-05-15T23:41:04.956528+0000","last_fullsized":"2024-05-15T23:41:04.956528+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'9081","last_scrub_stamp":"2024-05-15T23:40:52.788078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:52.788078+0000","objects_scrubbed":6463,"log_size":9902,"log_dups_size":0,"ondisk_log_size":9902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:58.533630+0000","scrub_duration":960,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367788032,"num_objects":5612,"num_object_clones":0,"num_object_copies":11224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9902,"num_write_kb":496448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"776'10199","reported_seq":10362,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-05-15T23:41:04.956600+0000","last_change":"2024-05-15T23:40:53.768684+0000","last_active":"2024-05-15T23:41:04.956600+0000","last_peered":"2024-05-15T23:41:04.956600+0000","last_clean":"2024-05-15T23:41:04.956600+0000","last_became_active":"2024-05-15T23:38:50.401123+0000","last_became_peered":"2024-05-15T23:38:50.401123+0000","last_unstale":"2024-05-15T23:41:04.956600+0000","last_undegraded":"2024-05-15T23:41:04.956600+0000","last_fullsized":"2024-05-15T23:41:04.956600+0000","mapping_epoch":737,"log_start":"743'100","ondisk_log_start":"743'100","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"773'9423","last_scrub_stamp":"2024-05-15T23:40:53.768529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:53.768529+0000","objects_scrubbed":6535,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:14.948562+0000","scrub_duration":969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375783448,"num_objects":5735,"num_object_clones":0,"num_object_copies":11470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5735,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10199,"num_write_kb":509825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"776'9770","reported_seq":9857,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-05-15T23:41:02.204855+0000","last_change":"2024-05-15T23:38:50.402679+0000","last_active":"2024-05-15T23:41:02.204855+0000","last_peered":"2024-05-15T23:41:02.204855+0000","last_clean":"2024-05-15T23:41:02.204855+0000","last_became_active":"2024-05-15T23:38:50.402428+0000","last_became_peered":"2024-05-15T23:38:50.402428+0000","last_unstale":"2024-05-15T23:41:02.204855+0000","last_undegraded":"2024-05-15T23:41:02.204855+0000","last_fullsized":"2024-05-15T23:41:02.204855+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":9770,"log_dups_size":0,"ondisk_log_size":9770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381157376,"num_objects":5816,"num_object_clones":0,"num_object_copies":11632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9770,"num_write_kb":498752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"776'9709","reported_seq":9873,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-05-15T23:41:04.225362+0000","last_change":"2024-05-15T23:40:45.650576+0000","last_active":"2024-05-15T23:41:04.225362+0000","last_peered":"2024-05-15T23:41:04.225362+0000","last_clean":"2024-05-15T23:41:04.225362+0000","last_became_active":"2024-05-15T23:38:50.401815+0000","last_became_peered":"2024-05-15T23:38:50.401815+0000","last_unstale":"2024-05-15T23:41:04.225362+0000","last_undegraded":"2024-05-15T23:41:04.225362+0000","last_fullsized":"2024-05-15T23:41:04.225362+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'8522","last_scrub_stamp":"2024-05-15T23:40:45.650391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:45.650391+0000","objects_scrubbed":6888,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:58.125696+0000","scrub_duration":745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373096448,"num_objects":5693,"num_object_clones":0,"num_object_copies":11386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9709,"num_write_kb":492864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"776'9805","reported_seq":18767,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-05-15T23:41:04.921580+0000","last_change":"2024-05-15T23:40:55.754806+0000","last_active":"2024-05-15T23:41:04.921580+0000","last_peered":"2024-05-15T23:41:04.921580+0000","last_clean":"2024-05-15T23:41:04.921580+0000","last_became_active":"2024-05-15T23:40:45.476827+0000","last_became_peered":"2024-05-15T23:40:45.476827+0000","last_unstale":"2024-05-15T23:41:04.921580+0000","last_undegraded":"2024-05-15T23:41:04.921580+0000","last_fullsized":"2024-05-15T23:41:04.921580+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"774'9184","last_scrub_stamp":"2024-05-15T23:40:55.754657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:55.754657+0000","objects_scrubbed":6210,"log_size":9805,"log_dups_size":0,"ondisk_log_size":9805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:14.046965+0000","scrub_duration":922,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364183552,"num_objects":5557,"num_object_clones":0,"num_object_copies":11114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9805,"num_write_kb":491584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4397,"num_bytes_recovered":288030720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"775'9742","reported_seq":9918,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-05-15T23:41:01.464643+0000","last_change":"2024-05-15T23:40:48.476758+0000","last_active":"2024-05-15T23:41:01.464643+0000","last_peered":"2024-05-15T23:41:01.464643+0000","last_clean":"2024-05-15T23:41:01.464643+0000","last_became_active":"2024-05-15T23:38:50.404189+0000","last_became_peered":"2024-05-15T23:38:50.404189+0000","last_unstale":"2024-05-15T23:41:01.464643+0000","last_undegraded":"2024-05-15T23:41:01.464643+0000","last_fullsized":"2024-05-15T23:41:01.464643+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"771'8884","last_scrub_stamp":"2024-05-15T23:40:48.476557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:48.476557+0000","objects_scrubbed":6814,"log_size":9742,"log_dups_size":0,"ondisk_log_size":9742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:02.814899+0000","scrub_duration":1372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387317760,"num_objects":5910,"num_object_clones":0,"num_object_copies":11820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9742,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"775'9613","reported_seq":9759,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-05-15T23:41:01.479268+0000","last_change":"2024-05-15T23:39:55.792219+0000","last_active":"2024-05-15T23:41:01.479268+0000","last_peered":"2024-05-15T23:41:01.479268+0000","last_clean":"2024-05-15T23:41:01.479268+0000","last_became_active":"2024-05-15T23:38:50.401416+0000","last_became_peered":"2024-05-15T23:38:50.401416+0000","last_unstale":"2024-05-15T23:41:01.479268+0000","last_undegraded":"2024-05-15T23:41:01.479268+0000","last_fullsized":"2024-05-15T23:41:01.479268+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"761'5246","last_scrub_stamp":"2024-05-15T23:39:55.792018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:39:55.792018+0000","objects_scrubbed":5229,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:21.653921+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379387904,"num_objects":5789,"num_object_clones":0,"num_object_copies":11578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":492864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"776'9580","reported_seq":17786,"reported_epoch":776,"state":"active+recovering+degraded","last_fresh":"2024-05-15T23:41:02.271653+0000","last_change":"2024-05-15T23:39:43.668135+0000","last_active":"2024-05-15T23:41:02.271653+0000","last_peered":"2024-05-15T23:41:02.271653+0000","last_clean":"2024-05-15T23:39:25.398249+0000","last_became_active":"2024-05-15T23:39:43.637604+0000","last_became_peered":"2024-05-15T23:39:43.637604+0000","last_unstale":"2024-05-15T23:41:02.271653+0000","last_undegraded":"2024-05-15T23:39:43.429785+0000","last_fullsized":"2024-05-15T23:41:02.271653+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":9580,"log_dups_size":0,"ondisk_log_size":9580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375783424,"num_objects":5734,"num_object_clones":0,"num_object_copies":11468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9580,"num_write_kb":490048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4057,"num_bytes_recovered":265879552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":334},{"shards":"3,9","objects":5400}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"776'9589","reported_seq":18488,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-05-15T23:41:03.056902+0000","last_change":"2024-05-15T23:40:51.174588+0000","last_active":"2024-05-15T23:41:03.056902+0000","last_peered":"2024-05-15T23:41:03.056902+0000","last_clean":"2024-05-15T23:41:03.056902+0000","last_became_active":"2024-05-15T23:39:43.514269+0000","last_became_peered":"2024-05-15T23:39:43.514269+0000","last_unstale":"2024-05-15T23:41:03.056902+0000","last_undegraded":"2024-05-15T23:41:03.056902+0000","last_fullsized":"2024-05-15T23:41:03.056902+0000","mapping_epoch":758,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":771,"parent":"0.0","parent_split_bits":0,"last_scrub":"771'8828","last_scrub_stamp":"2024-05-15T23:40:51.174438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:51.174438+0000","objects_scrubbed":6515,"log_size":9589,"log_dups_size":0,"ondisk_log_size":9589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:59.748880+0000","scrub_duration":1137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374800384,"num_objects":5719,"num_object_clones":0,"num_object_copies":11438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9589,"num_write_kb":489856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":286064640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"776'9716","reported_seq":9881,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-05-15T23:41:02.084890+0000","last_change":"2024-05-15T23:40:45.307444+0000","last_active":"2024-05-15T23:41:02.084890+0000","last_peered":"2024-05-15T23:41:02.084890+0000","last_clean":"2024-05-15T23:41:02.084890+0000","last_became_active":"2024-05-15T23:38:50.398410+0000","last_became_peered":"2024-05-15T23:38:50.398410+0000","last_unstale":"2024-05-15T23:41:02.084890+0000","last_undegraded":"2024-05-15T23:41:02.084890+0000","last_fullsized":"2024-05-15T23:41:02.084890+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'8645","last_scrub_stamp":"2024-05-15T23:40:45.307336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:45.307336+0000","objects_scrubbed":6921,"log_size":9716,"log_dups_size":0,"ondisk_log_size":9716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:03.552738+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383254528,"num_objects":5848,"num_object_clones":0,"num_object_copies":11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9716,"num_write_kb":498048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"776'9907","reported_seq":9995,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-05-15T23:41:05.275959+0000","last_change":"2024-05-15T23:38:50.402064+0000","last_active":"2024-05-15T23:41:05.275959+0000","last_peered":"2024-05-15T23:41:05.275959+0000","last_clean":"2024-05-15T23:41:05.275959+0000","last_became_active":"2024-05-15T23:38:50.401779+0000","last_became_peered":"2024-05-15T23:38:50.401779+0000","last_unstale":"2024-05-15T23:41:05.275959+0000","last_undegraded":"2024-05-15T23:41:05.275959+0000","last_fullsized":"2024-05-15T23:41:05.275959+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":9907,"log_dups_size":0,"ondisk_log_size":9907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364838912,"num_objects":5567,"num_object_clones":0,"num_object_copies":11134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9907,"num_write_kb":495168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"776'9829","reported_seq":9917,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-05-15T23:41:02.246825+0000","last_change":"2024-05-15T23:38:50.402094+0000","last_active":"2024-05-15T23:41:02.246825+0000","last_peered":"2024-05-15T23:41:02.246825+0000","last_clean":"2024-05-15T23:41:02.246825+0000","last_became_active":"2024-05-15T23:38:50.401872+0000","last_became_peered":"2024-05-15T23:38:50.401872+0000","last_unstale":"2024-05-15T23:41:02.246825+0000","last_undegraded":"2024-05-15T23:41:02.246825+0000","last_fullsized":"2024-05-15T23:41:02.246825+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":9829,"log_dups_size":0,"ondisk_log_size":9829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379912192,"num_objects":5797,"num_object_clones":0,"num_object_copies":11594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9829,"num_write_kb":500032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"776'9763","reported_seq":9851,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-05-15T23:41:02.220194+0000","last_change":"2024-05-15T23:38:50.401343+0000","last_active":"2024-05-15T23:41:02.220194+0000","last_peered":"2024-05-15T23:41:02.220194+0000","last_clean":"2024-05-15T23:41:02.220194+0000","last_became_active":"2024-05-15T23:38:50.401074+0000","last_became_peered":"2024-05-15T23:38:50.401074+0000","last_unstale":"2024-05-15T23:41:02.220194+0000","last_undegraded":"2024-05-15T23:41:02.220194+0000","last_fullsized":"2024-05-15T23:41:02.220194+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":9763,"log_dups_size":0,"ondisk_log_size":9763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385286144,"num_objects":5879,"num_object_clones":0,"num_object_copies":11758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9763,"num_write_kb":500544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"776'9891","reported_seq":10054,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-05-15T23:41:03.051385+0000","last_change":"2024-05-15T23:40:50.387123+0000","last_active":"2024-05-15T23:41:03.051385+0000","last_peered":"2024-05-15T23:41:03.051385+0000","last_clean":"2024-05-15T23:41:03.051385+0000","last_became_active":"2024-05-15T23:38:50.401998+0000","last_became_peered":"2024-05-15T23:38:50.401998+0000","last_unstale":"2024-05-15T23:41:03.051385+0000","last_undegraded":"2024-05-15T23:41:03.051385+0000","last_fullsized":"2024-05-15T23:41:03.051385+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"771'9050","last_scrub_stamp":"2024-05-15T23:40:50.386878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:50.386878+0000","objects_scrubbed":6683,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:57.422858+0000","scrub_duration":1397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380436480,"num_objects":5805,"num_object_clones":0,"num_object_copies":11610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9891,"num_write_kb":502272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.8","version":"776'10043","reported_seq":10131,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-05-15T23:41:05.282511+0000","last_change":"2024-05-15T23:38:50.404457+0000","last_active":"2024-05-15T23:41:05.282511+0000","last_peered":"2024-05-15T23:41:05.282511+0000","last_clean":"2024-05-15T23:41:05.282511+0000","last_became_active":"2024-05-15T23:38:50.404155+0000","last_became_peered":"2024-05-15T23:38:50.404155+0000","last_unstale":"2024-05-15T23:41:05.282511+0000","last_undegraded":"2024-05-15T23:41:05.282511+0000","last_fullsized":"2024-05-15T23:41:05.282511+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369557504,"num_objects":5639,"num_object_clones":0,"num_object_copies":11278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10043,"num_write_kb":501824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1687,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-05-15T23:41:01.614691+0000","last_change":"2024-05-15T23:40:50.840277+0000","last_active":"2024-05-15T23:41:01.614691+0000","last_peered":"2024-05-15T23:41:01.614691+0000","last_clean":"2024-05-15T23:41:01.614691+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:41:01.614691+0000","last_undegraded":"2024-05-15T23:41:01.614691+0000","last_fullsized":"2024-05-15T23:41:01.614691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:40:50.840191+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:40:50.840191+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:15.227987+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6018105368,"num_objects":91830,"num_object_clones":0,"num_object_copies":183660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91830,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156542,"num_write_kb":7947841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16642,"num_bytes_recovered":1090519040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13040099328,"data_stored":13040091184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156442,"ondisk_log_size":156442,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351424,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2400984,"kb_used_data":1495180,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":91344168,"statfs":{"total":95995035648,"available":93536428032,"internally_reserved":0,"allocated":1531064320,"data_stored":1530692096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":927529412},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580356,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2129156,"kb_used_data":1506552,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91615996,"statfs":{"total":95995035648,"available":93814779904,"internally_reserved":0,"allocated":1542709248,"data_stored":1542346571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":637532608},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776589,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2593612,"kb_used_data":1764164,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":91151540,"statfs":{"total":95995035648,"available":93339176960,"internally_reserved":0,"allocated":1806503936,"data_stored":1806135284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":849344960},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":520,"seq":2233382994092,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":475340,"kb_used_data":366212,"kb_used_omap":24,"kb_used_meta":109095,"kb_avail":93269812,"statfs":{"total":95995035648,"available":95508287488,"internally_reserved":0,"allocated":375001088,"data_stored":374632436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25357,"internal_metadata":111713523},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267158,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2693912,"kb_used_data":1990800,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":91051240,"statfs":{"total":95995035648,"available":93236469760,"internally_reserved":0,"allocated":2038579200,"data_stored":2038202924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":719976906},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568496091,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575349,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3034052,"kb_used_data":2096252,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":90711100,"statfs":{"total":95995035648,"available":92888166400,"internally_reserved":0,"allocated":2146562048,"data_stored":2146195299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":960231878},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016759,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":209360,"kb_used_data":708,"kb_used_omap":4,"kb_used_meta":208635,"kb_avail":93535792,"statfs":{"total":95995035648,"available":95780651008,"internally_reserved":0,"allocated":724992,"data_stored":356340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":213642947},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150188,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3220628,"kb_used_data":2228740,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":90524524,"statfs":{"total":95995035648,"available":92697112576,"internally_reserved":0,"allocated":2282229760,"data_stored":2281861108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1015675335},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986660,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1460364,"kb_used_data":729476,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92284788,"statfs":{"total":95995035648,"available":94499622912,"internally_reserved":0,"allocated":746983424,"data_stored":746614772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1913458688,"data_stored":1913454616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2275475456,"data_stored":2275475456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":744423424,"data_stored":744423424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1040711680,"data_stored":1040711680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1839927296,"data_stored":1839923224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":373096448,"data_stored":373096448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1804206080,"data_stored":1804206080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1534132224,"data_stored":1534132224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1514668032,"data_stored":1514668032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:41:09.662 INFO:tasks.cephadm.ceph_manager.ceph:PG 9.f is not active+clean 2024-05-15T23:41:09.662 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '9.f', 'version': "776'9484", 'reported_seq': 17218, 'reported_epoch': 776, 'state': 'active+recovering+degraded', 'last_fresh': '2024-05-15T23:41:02.273066+0000', 'last_change': '2024-05-15T23:39:43.669795+0000', 'last_active': '2024-05-15T23:41:02.273066+0000', 'last_peered': '2024-05-15T23:41:02.273066+0000', 'last_clean': '2024-05-15T23:39:25.389963+0000', 'last_became_active': '2024-05-15T23:39:43.643192+0000', 'last_became_peered': '2024-05-15T23:39:43.643192+0000', 'last_unstale': '2024-05-15T23:41:02.273066+0000', 'last_undegraded': '2024-05-15T23:39:43.429912+0000', 'last_fullsized': '2024-05-15T23:41:02.273066+0000', 'mapping_epoch': 758, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 737, 'last_epoch_clean': 738, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-15T23:38:49.378874+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-15T23:38:49.378874+0000', 'last_clean_scrub_stamp': '2024-05-15T23:38:49.378874+0000', 'objects_scrubbed': 0, 'log_size': 9484, 'log_dups_size': 0, 'ondisk_log_size': 9484, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 375521280, 'num_objects': 5730, 'num_object_clones': 0, 'num_object_copies': 11460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9484, 'num_write_kb': 486848, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3823, 'num_bytes_recovered': 250544128, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 9], 'acting': [3, 9], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 526}, {'shards': '3,9', 'objects': 5204}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-15T23:41:09.662 INFO:tasks.cephadm.ceph_manager.ceph:PG 9.6 is not active+clean 2024-05-15T23:41:09.662 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '9.6', 'version': "776'9580", 'reported_seq': 17786, 'reported_epoch': 776, 'state': 'active+recovering+degraded', 'last_fresh': '2024-05-15T23:41:02.271653+0000', 'last_change': '2024-05-15T23:39:43.668135+0000', 'last_active': '2024-05-15T23:41:02.271653+0000', 'last_peered': '2024-05-15T23:41:02.271653+0000', 'last_clean': '2024-05-15T23:39:25.398249+0000', 'last_became_active': '2024-05-15T23:39:43.637604+0000', 'last_became_peered': '2024-05-15T23:39:43.637604+0000', 'last_unstale': '2024-05-15T23:41:02.271653+0000', 'last_undegraded': '2024-05-15T23:39:43.429785+0000', 'last_fullsized': '2024-05-15T23:41:02.271653+0000', 'mapping_epoch': 758, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 737, 'last_epoch_clean': 738, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-15T23:38:49.378874+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-15T23:38:49.378874+0000', 'last_clean_scrub_stamp': '2024-05-15T23:38:49.378874+0000', 'objects_scrubbed': 0, 'log_size': 9580, 'log_dups_size': 0, 'ondisk_log_size': 9580, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 375783424, 'num_objects': 5734, 'num_object_clones': 0, 'num_object_copies': 11468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 334, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5734, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9580, 'num_write_kb': 490048, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4057, 'num_bytes_recovered': 265879552, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 9], 'acting': [3, 9], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 334}, {'shards': '3,9', 'objects': 5400}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-15T23:41:09.663 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph status --format=json 2024-05-15T23:41:09.895 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:09.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:09.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:10.010 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:41:10.038 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:10.057 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:41:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:10.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:41:10.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:41:10.201 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:10.243 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:41:10.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:10.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:41:10.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:10 smithi088 ceph-mon[44699]: pgmap v2071: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 184/176474 objects degraded (0.104%); 8.5 MiB/s, 136 objects/s recovering 2024-05-15T23:41:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:10 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.352 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:41:10.380 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:10.400 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:41:10.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:10.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:41:10.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:41:10.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.542 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.543 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:10 smithi090 ceph-mon[44434]: pgmap v2071: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 184/176474 objects degraded (0.104%); 8.5 MiB/s, 136 objects/s recovering 2024-05-15T23:41:10.543 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.543 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:10 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.553 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:10.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:10 smithi084 ceph-mon[39784]: pgmap v2071: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 184/176474 objects degraded (0.104%); 8.5 MiB/s, 136 objects/s recovering 2024-05-15T23:41:10.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:10 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:10.583 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:41:10.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:10.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:41:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:41:10.606 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:41:10.688 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:41:10.716 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:10.739 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:41:10.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:10.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:41:10.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:10.882 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:11.039 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:11.049 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:11.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:11.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:11.193 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:11.434 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:11 smithi084 ceph-mon[39784]: pgmap v2072: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.3 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s; 184/173364 objects degraded (0.106%); 8.3 MiB/s, 133 objects/s recovering 2024-05-15T23:41:11.602 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:11.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:11 smithi090 ceph-mon[44434]: pgmap v2072: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.3 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s; 184/173364 objects degraded (0.106%); 8.3 MiB/s, 133 objects/s recovering 2024-05-15T23:41:11.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:11 smithi088 ceph-mon[44699]: pgmap v2072: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.3 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s; 184/173364 objects degraded (0.106%); 8.3 MiB/s, 133 objects/s recovering 2024-05-15T23:41:12.607 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:41:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:12 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:41:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:12 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:41:12.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:12 smithi090 ceph-mon[44434]: Health check update: Degraded data redundancy: 184/173364 objects degraded (0.106%), 1 pg degraded (PG_DEGRADED) 2024-05-15T23:41:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:12 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:41:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:12 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:41:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:12 smithi088 ceph-mon[44699]: Health check update: Degraded data redundancy: 184/173364 objects degraded (0.106%), 1 pg degraded (PG_DEGRADED) 2024-05-15T23:41:12.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:12 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:41:12.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:12 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:41:12.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:12 smithi084 ceph-mon[39784]: Health check update: Degraded data redundancy: 184/173364 objects degraded (0.106%), 1 pg degraded (PG_DEGRADED) 2024-05-15T23:41:13.717 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:41:13.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:13 smithi090 ceph-mon[44434]: osdmap e777: 12 total, 11 up, 9 in 2024-05-15T23:41:13.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1512473146' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:41:13.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:13 smithi090 ceph-mon[44434]: from='client.37526 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:41:13.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:13 smithi090 ceph-mon[44434]: pgmap v2074: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 5.4 MiB/s, 85 objects/s recovering 2024-05-15T23:41:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:13 smithi088 ceph-mon[44699]: osdmap e777: 12 total, 11 up, 9 in 2024-05-15T23:41:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1512473146' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:41:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:13 smithi088 ceph-mon[44699]: from='client.37526 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:41:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:13 smithi088 ceph-mon[44699]: pgmap v2074: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 5.4 MiB/s, 85 objects/s recovering 2024-05-15T23:41:13.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:13 smithi084 ceph-mon[39784]: osdmap e777: 12 total, 11 up, 9 in 2024-05-15T23:41:13.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1512473146' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:41:13.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:13 smithi084 ceph-mon[39784]: from='client.37526 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:41:13.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:13 smithi084 ceph-mon[39784]: pgmap v2074: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 5.4 MiB/s, 85 objects/s recovering 2024-05-15T23:41:14.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:14 smithi090 ceph-mon[44434]: 9.c scrub starts 2024-05-15T23:41:14.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:14 smithi090 ceph-mon[44434]: 9.8 scrub starts 2024-05-15T23:41:14.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:14 smithi090 ceph-mon[44434]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 184/173364 objects degraded (0.106%), 1 pg degraded) 2024-05-15T23:41:14.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:14 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:41:14.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:14 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:41:14.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:14 smithi090 ceph-mon[44434]: from='client.37526 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:41:14.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:14 smithi090 ceph-mon[44434]: osdmap e778: 12 total, 11 up, 9 in 2024-05-15T23:41:14.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:14 smithi088 ceph-mon[44699]: 9.c scrub starts 2024-05-15T23:41:14.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:14 smithi088 ceph-mon[44699]: 9.8 scrub starts 2024-05-15T23:41:14.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:14 smithi088 ceph-mon[44699]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 184/173364 objects degraded (0.106%), 1 pg degraded) 2024-05-15T23:41:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:14 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:41:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:14 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:41:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:14 smithi088 ceph-mon[44699]: from='client.37526 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:41:14.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:14 smithi088 ceph-mon[44699]: osdmap e778: 12 total, 11 up, 9 in 2024-05-15T23:41:14.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:14 smithi084 ceph-mon[39784]: 9.c scrub starts 2024-05-15T23:41:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:14 smithi084 ceph-mon[39784]: 9.8 scrub starts 2024-05-15T23:41:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:14 smithi084 ceph-mon[39784]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 184/173364 objects degraded (0.106%), 1 pg degraded) 2024-05-15T23:41:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:14 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:41:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:14 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:41:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:14 smithi084 ceph-mon[39784]: from='client.37526 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:41:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:14 smithi084 ceph-mon[39784]: osdmap e778: 12 total, 11 up, 9 in 2024-05-15T23:41:14.957 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:14.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:14.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:15.062 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:41:15.091 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:15.113 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:41:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:15.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:41:15.217 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:41:15.244 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:41:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:15.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:41:15.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:41:15.421 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:15.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:15 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:41:15.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:15 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:41:15.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:15 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:41:15.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:15 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:41:15.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:15 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:41:15.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:15 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:41:15.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:15 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:41:15.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:15 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:41:15.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:15 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:41:15.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:15 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:41:15.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:15 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:41:15.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:15 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:41:15.894 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:41:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:15.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:41:16.003 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:16.044 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:16.500 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:16.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:16.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:16.607 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:41:16.635 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:16.655 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:41:16.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:16.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:41:16.759 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:41:16.805 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:17.562 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:41:17.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:17.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:41:17.667 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:41:17.695 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:17 smithi088 ceph-mon[44699]: pgmap v2077: 17 pgs: 17 active+clean; 4.8 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.25k op/s; 1.4 MiB/s, 22 objects/s recovering 2024-05-15T23:41:17.697 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:17.719 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:41:17.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:17.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:41:17.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:17 smithi090 ceph-mon[44434]: pgmap v2077: 17 pgs: 17 active+clean; 4.8 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.25k op/s; 1.4 MiB/s, 22 objects/s recovering 2024-05-15T23:41:17.823 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:17.858 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:17 smithi084 ceph-mon[39784]: pgmap v2077: 17 pgs: 17 active+clean; 4.8 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.25k op/s; 1.4 MiB/s, 22 objects/s recovering 2024-05-15T23:41:17.861 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:18.008 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:18.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:18.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:41:18.158 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:18.583 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:41:18.583 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:41:18.584 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-05-15T23:41:18.584 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:41:18.920 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:19.144 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:41:19.144 INFO:teuthology.orchestra.run.smithi084.stdout:{"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2558,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":778,"num_osds":12,"num_up_osds":11,"osd_up_since":1715816365,"num_in_osds":9,"osd_in_since":1715816382,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17}],"num_pgs":17,"num_pools":2,"num_objects":78217,"data_bytes":5126423096,"bytes_used":16927850496,"bytes_avail":847027470336,"bytes_total":863955320832,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":1506866,"recovering_keys_per_sec":0,"num_objects_recovered":184,"num_bytes_recovered":12058624,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1252},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.84:8443/"}},"servicemap":{"epoch":52,"modified":"2024-05-15T23:41:03.367735+0000","services":{}},"progress_events":{}} 2024-05-15T23:41:19.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2650304342' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T23:41:19.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T23:41:19.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2650304342' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T23:41:19.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2650304342' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-05-15T23:41:19.649 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:41:19.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:19.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:41:19.759 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:41:19.796 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:19.926 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:41:20.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:20 smithi084 ceph-mon[39784]: pgmap v2078: 17 pgs: 17 active+clean; 4.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s; 1.4 MiB/s, 22 objects/s recovering 2024-05-15T23:41:20.556 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:20 smithi090 ceph-mon[44434]: pgmap v2078: 17 pgs: 17 active+clean; 4.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s; 1.4 MiB/s, 22 objects/s recovering 2024-05-15T23:41:20.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:20 smithi088 ceph-mon[44699]: pgmap v2078: 17 pgs: 17 active+clean; 4.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s; 1.4 MiB/s, 22 objects/s recovering 2024-05-15T23:41:21.261 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:41:21.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:21.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:41:21.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:41:21.413 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:21.469 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:41:21.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:21.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:41:21.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:21.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:21 smithi084 ceph-mon[39784]: pgmap v2079: 17 pgs: 17 active+clean; 4.6 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 953 op/s 2024-05-15T23:41:21.623 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:21.654 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:21.662 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:21 smithi090 ceph-mon[44434]: pgmap v2079: 17 pgs: 17 active+clean; 4.6 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 953 op/s 2024-05-15T23:41:21.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:21.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:21.739 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-05-15T23:41:21.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:41:21.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:21.812 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:21.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:21 smithi088 ceph-mon[44699]: pgmap v2079: 17 pgs: 17 active+clean; 4.6 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 953 op/s 2024-05-15T23:41:21.851 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:21.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:21.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:21.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:22.023 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:22.064 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:22.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:22.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:22.174 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:41:22.210 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:22.749 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:23.069 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:41:23.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:23.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:41:23.173 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:41:23.205 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:23.228 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:41:23.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:23.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:41:23.331 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:41:23.360 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:23.381 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:41:23.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:23.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:41:23.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:41:23.525 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:41:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:41:23.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:23 smithi088 ceph-mon[44699]: pgmap v2080: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.13k op/s 2024-05-15T23:41:23.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:41:23.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:41:23.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:23 smithi084 ceph-mon[39784]: pgmap v2080: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.13k op/s 2024-05-15T23:41:23.665 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:41:23.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:23.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:41:23.773 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:41:23.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:41:23.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:41:23.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:23 smithi090 ceph-mon[44434]: pgmap v2080: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.13k op/s 2024-05-15T23:41:23.803 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:23.825 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:41:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:23.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:41:23.930 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:24.002 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:24.809 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:24 smithi090 ceph-mon[44434]: osdmap e779: 12 total, 11 up, 9 in 2024-05-15T23:41:24.810 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:24 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1879044216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:41:24.810 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:24 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:24.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:24 smithi088 ceph-mon[44699]: osdmap e779: 12 total, 11 up, 9 in 2024-05-15T23:41:24.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:24 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1879044216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:41:24.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:24 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:24.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:24 smithi084 ceph-mon[39784]: osdmap e779: 12 total, 11 up, 9 in 2024-05-15T23:41:24.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:24 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1879044216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:41:24.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:24 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.178 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:25.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:25.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:25.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:41:25.335 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:25.361 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:41:25.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:25.365 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:41:25.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:41:25.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:41:25.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:41:25.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:41:25.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:41:25.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:41:25.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:25 smithi084 ceph-mon[39784]: pgmap v2082: 17 pgs: 17 active+clean; 4.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:41:25.501 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:25.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:41:25.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:41:25.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:41:25.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:41:25.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:41:25.807 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:25 smithi090 ceph-mon[44434]: pgmap v2082: 17 pgs: 17 active+clean; 4.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:41:25.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:41:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:41:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:41:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:41:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:41:25.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:25 smithi088 ceph-mon[44699]: pgmap v2082: 17 pgs: 17 active+clean; 4.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:41:25.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:41:25.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:41:25.944 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:41:25.944 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:41:25.944 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:41:25.944 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:41:25.944 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:41:25.944 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:41:25.944 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:41:25.944 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:41:25.944 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:41:25.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:41:25.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:41:25.945 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:41:25.945 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:41:25.945 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:41:25.945 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-05-15T23:41:25.945 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:41:25.945 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:41:25.945 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:41:25.945 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:41:25.945 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:41:25.945 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:41:26.596 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:41:26.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:26.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:41:26.621 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:41:26.639 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:41:26.710 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:41:26.745 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:26.874 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:41:26.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:26.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:41:26.983 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:41:27.010 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:27.031 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:41:27.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:27.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:41:27.136 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:41:27.181 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:27.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:27 smithi084 ceph-mon[39784]: pgmap v2083: 17 pgs: 17 active+clean; 4.1 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:41:27.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:27 smithi090 ceph-mon[44434]: pgmap v2083: 17 pgs: 17 active+clean; 4.1 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:41:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:27 smithi088 ceph-mon[44699]: pgmap v2083: 17 pgs: 17 active+clean; 4.1 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:41:27.938 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:41:27.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:27.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:41:27.956 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:28.043 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:41:28.071 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:28.091 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:41:28.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:28.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:41:28.196 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:41:28.224 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:28.245 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:41:28.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:28.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:41:28.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:41:28.388 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:28.419 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:41:28.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:28.426 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:41:28.454 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:41:28.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:28.572 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:28.603 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:28.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:28.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:28.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:41:28.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:41:28.754 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2318231266' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:41:28.755 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:28 smithi090 ceph-mon[44434]: from='client.27409 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:41:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:28.786 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:41:28.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:28.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:41:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2318231266' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:41:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:28 smithi088 ceph-mon[44699]: from='client.27409 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:41:28.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2318231266' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:41:28.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:28 smithi084 ceph-mon[39784]: from='client.27409 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:41:28.893 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:41:28.920 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:41:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:28.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:41:29.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:41:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:29.143 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:41:29.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:29.152 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:41:29.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:29.302 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:29.336 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:29.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:29.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:29.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:29.511 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:29.550 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:29.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:29.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:29.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:29 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:41:29.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:29 smithi084 ceph-mon[39784]: from='client.27409 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:41:29.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:29 smithi084 ceph-mon[39784]: osdmap e780: 12 total, 11 up, 9 in 2024-05-15T23:41:29.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:29 smithi084 ceph-mon[39784]: pgmap v2085: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:41:29.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:29.703 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:29 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:41:29.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:29 smithi090 ceph-mon[44434]: from='client.27409 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:41:29.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:29 smithi090 ceph-mon[44434]: osdmap e780: 12 total, 11 up, 9 in 2024-05-15T23:41:29.704 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:29 smithi090 ceph-mon[44434]: pgmap v2085: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:41:29.705 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:29.745 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:29.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:29.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:29 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:41:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:29 smithi088 ceph-mon[44699]: from='client.27409 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:41:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:29 smithi088 ceph-mon[44699]: osdmap e780: 12 total, 11 up, 9 in 2024-05-15T23:41:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:29 smithi088 ceph-mon[44699]: pgmap v2085: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:41:29.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:29.887 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:29.919 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:29.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:30.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:30.066 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:30.142 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:30.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:30.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:30.249 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:41:30.277 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:30.299 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:41:30.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:30.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:41:30.403 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:41:30.431 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:30.452 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:41:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:30.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:41:30.556 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:41:30.584 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:30.608 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:41:30.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:30.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:41:30.713 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:41:30.751 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:30.785 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:41:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:30.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:41:30.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:30.929 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:30.981 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:41:30.982 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:41:30.991 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:30.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:30.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:31.096 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:41:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:31.167 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:41:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:31.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:41:31.272 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:31.312 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:31.333 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:41:31.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:31 smithi084 ceph-mon[39784]: pgmap v2086: 17 pgs: 17 active+clean; 3.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 951 op/s 2024-05-15T23:41:31.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:31 smithi090 ceph-mon[44434]: pgmap v2086: 17 pgs: 17 active+clean; 3.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 951 op/s 2024-05-15T23:41:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:31 smithi088 ceph-mon[44699]: pgmap v2086: 17 pgs: 17 active+clean; 3.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 951 op/s 2024-05-15T23:41:32.619 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:32.773 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:32.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:32.880 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:41:32.923 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:41:32.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:32.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:41:33.069 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:33.119 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:33.301 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:41:33.315 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:41:34.008 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:34.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:34.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:34.115 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:41:34.158 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:33 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:41:34.158 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:33 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:41:34.158 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:33 smithi084 ceph-mon[39784]: osdmap e781: 12 total, 11 up, 9 in 2024-05-15T23:41:34.158 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:33 smithi084 ceph-mon[39784]: from='client.37547 v1:172.21.15.84:0/571795065' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:41:34.158 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:33 smithi084 ceph-mon[39784]: pgmap v2088: 17 pgs: 17 active+clean; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s 2024-05-15T23:41:34.159 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:33 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:41:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:33 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:41:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:33 smithi090 ceph-mon[44434]: osdmap e781: 12 total, 11 up, 9 in 2024-05-15T23:41:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:33 smithi090 ceph-mon[44434]: from='client.37547 v1:172.21.15.84:0/571795065' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:41:34.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:33 smithi090 ceph-mon[44434]: pgmap v2088: 17 pgs: 17 active+clean; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s 2024-05-15T23:41:34.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:33 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:41:34.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:33 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:41:34.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:33 smithi088 ceph-mon[44699]: osdmap e781: 12 total, 11 up, 9 in 2024-05-15T23:41:34.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:33 smithi088 ceph-mon[44699]: from='client.37547 v1:172.21.15.84:0/571795065' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:41:34.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:33 smithi088 ceph-mon[44699]: pgmap v2088: 17 pgs: 17 active+clean; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s 2024-05-15T23:41:34.349 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:41:34.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:34.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:41:34.376 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":2087,"stamp":"2024-05-15T23:41:32.869726+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4026991160,"num_objects":61441,"num_object_clones":0,"num_object_copies":122882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61441,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":187056,"num_write_kb":7949991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17508,"num_bytes_recovered":1148466784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161127,"ondisk_log_size":161127,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":8,"kb":843706368,"kb_used":14423612,"kb_used_data":8251040,"kb_used_omap":39,"kb_used_meta":6172376,"kb_avail":829282756,"statfs":{"total":863955320832,"available":849185542144,"internally_reserved":0,"allocated":8449064960,"data_stored":8445521849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40938,"internal_metadata":6320513046},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-682622976,"num_objects":-10416,"num_object_clones":0,"num_object_copies":-20832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10416,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.442013"},"pg_stats":[{"pgid":"9.f","version":"780'11535","reported_seq":20336,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-05-15T23:41:32.308975+0000","last_change":"2024-05-15T23:41:09.439865+0000","last_active":"2024-05-15T23:41:32.308975+0000","last_peered":"2024-05-15T23:41:32.308975+0000","last_clean":"2024-05-15T23:41:32.308975+0000","last_became_active":"2024-05-15T23:39:43.643192+0000","last_became_peered":"2024-05-15T23:39:43.643192+0000","last_unstale":"2024-05-15T23:41:32.308975+0000","last_undegraded":"2024-05-15T23:41:32.308975+0000","last_fullsized":"2024-05-15T23:41:32.308975+0000","mapping_epoch":758,"log_start":"744'1500","ondisk_log_start":"744'1500","created":737,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:33.172667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241106944,"num_objects":3679,"num_object_clones":0,"num_object_copies":7358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11535,"num_write_kb":486848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4349,"num_bytes_recovered":285016064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"780'11595","reported_seq":11762,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-05-15T23:41:29.955990+0000","last_change":"2024-05-15T23:40:52.788179+0000","last_active":"2024-05-15T23:41:29.955990+0000","last_peered":"2024-05-15T23:41:29.955990+0000","last_clean":"2024-05-15T23:41:29.955990+0000","last_became_active":"2024-05-15T23:38:50.402740+0000","last_became_peered":"2024-05-15T23:38:50.402740+0000","last_unstale":"2024-05-15T23:41:29.955990+0000","last_undegraded":"2024-05-15T23:41:29.955990+0000","last_fullsized":"2024-05-15T23:41:29.955990+0000","mapping_epoch":737,"log_start":"744'1500","ondisk_log_start":"744'1500","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'9081","last_scrub_stamp":"2024-05-15T23:40:52.788078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:52.788078+0000","objects_scrubbed":6463,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:58.533630+0000","scrub_duration":960,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256835584,"num_objects":3919,"num_object_clones":0,"num_object_copies":7838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":496448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"780'11972","reported_seq":12143,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-05-15T23:41:29.943458+0000","last_change":"2024-05-15T23:40:53.768684+0000","last_active":"2024-05-15T23:41:29.943458+0000","last_peered":"2024-05-15T23:41:29.943458+0000","last_clean":"2024-05-15T23:41:29.943458+0000","last_became_active":"2024-05-15T23:38:50.401123+0000","last_became_peered":"2024-05-15T23:38:50.401123+0000","last_unstale":"2024-05-15T23:41:29.943458+0000","last_undegraded":"2024-05-15T23:41:29.943458+0000","last_fullsized":"2024-05-15T23:41:29.943458+0000","mapping_epoch":737,"log_start":"745'1900","ondisk_log_start":"745'1900","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"773'9423","last_scrub_stamp":"2024-05-15T23:40:53.768529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:53.768529+0000","objects_scrubbed":6535,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:14.948562+0000","scrub_duration":969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259588120,"num_objects":3962,"num_object_clones":0,"num_object_copies":7924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3962,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11972,"num_write_kb":509825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"780'11900","reported_seq":12018,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-05-15T23:41:32.295500+0000","last_change":"2024-05-15T23:41:13.494498+0000","last_active":"2024-05-15T23:41:32.295500+0000","last_peered":"2024-05-15T23:41:32.295500+0000","last_clean":"2024-05-15T23:41:32.295500+0000","last_became_active":"2024-05-15T23:38:50.402428+0000","last_became_peered":"2024-05-15T23:38:50.402428+0000","last_unstale":"2024-05-15T23:41:32.295500+0000","last_undegraded":"2024-05-15T23:41:32.295500+0000","last_fullsized":"2024-05-15T23:41:32.295500+0000","mapping_epoch":737,"log_start":"745'1800","ondisk_log_start":"745'1800","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":1899,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:37.126386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241565696,"num_objects":3686,"num_object_clones":0,"num_object_copies":7372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11900,"num_write_kb":498752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"780'11443","reported_seq":11616,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-05-15T23:41:29.232943+0000","last_change":"2024-05-15T23:40:45.650576+0000","last_active":"2024-05-15T23:41:29.232943+0000","last_peered":"2024-05-15T23:41:29.232943+0000","last_clean":"2024-05-15T23:41:29.232943+0000","last_became_active":"2024-05-15T23:38:50.401815+0000","last_became_peered":"2024-05-15T23:38:50.401815+0000","last_unstale":"2024-05-15T23:41:29.232943+0000","last_undegraded":"2024-05-15T23:41:29.232943+0000","last_fullsized":"2024-05-15T23:41:29.232943+0000","mapping_epoch":737,"log_start":"744'1400","ondisk_log_start":"744'1400","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'8522","last_scrub_stamp":"2024-05-15T23:40:45.650391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:45.650391+0000","objects_scrubbed":6888,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:58.125696+0000","scrub_duration":745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259457024,"num_objects":3959,"num_object_clones":0,"num_object_copies":7918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11443,"num_write_kb":492864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"780'11472","reported_seq":20442,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-05-15T23:41:29.957619+0000","last_change":"2024-05-15T23:40:55.754806+0000","last_active":"2024-05-15T23:41:29.957619+0000","last_peered":"2024-05-15T23:41:29.957619+0000","last_clean":"2024-05-15T23:41:29.957619+0000","last_became_active":"2024-05-15T23:40:45.476827+0000","last_became_peered":"2024-05-15T23:40:45.476827+0000","last_unstale":"2024-05-15T23:41:29.957619+0000","last_undegraded":"2024-05-15T23:41:29.957619+0000","last_fullsized":"2024-05-15T23:41:29.957619+0000","mapping_epoch":769,"log_start":"744'1400","ondisk_log_start":"744'1400","created":737,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"774'9184","last_scrub_stamp":"2024-05-15T23:40:55.754657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:55.754657+0000","objects_scrubbed":6210,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:14.046965+0000","scrub_duration":922,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254935040,"num_objects":3890,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11472,"num_write_kb":491584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4397,"num_bytes_recovered":288030720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"780'11832","reported_seq":12018,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-05-15T23:41:31.503350+0000","last_change":"2024-05-15T23:40:48.476758+0000","last_active":"2024-05-15T23:41:31.503350+0000","last_peered":"2024-05-15T23:41:31.503350+0000","last_clean":"2024-05-15T23:41:31.503350+0000","last_became_active":"2024-05-15T23:38:50.404189+0000","last_became_peered":"2024-05-15T23:38:50.404189+0000","last_unstale":"2024-05-15T23:41:31.503350+0000","last_undegraded":"2024-05-15T23:41:31.503350+0000","last_fullsized":"2024-05-15T23:41:31.503350+0000","mapping_epoch":737,"log_start":"745'1800","ondisk_log_start":"745'1800","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"771'8884","last_scrub_stamp":"2024-05-15T23:40:48.476557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:48.476557+0000","objects_scrubbed":6814,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:02.814899+0000","scrub_duration":1372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250347520,"num_objects":3820,"num_object_clones":0,"num_object_copies":7640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11832,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"780'11691","reported_seq":11847,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-05-15T23:41:31.492400+0000","last_change":"2024-05-15T23:39:55.792219+0000","last_active":"2024-05-15T23:41:31.492400+0000","last_peered":"2024-05-15T23:41:31.492400+0000","last_clean":"2024-05-15T23:41:31.492400+0000","last_became_active":"2024-05-15T23:38:50.401416+0000","last_became_peered":"2024-05-15T23:38:50.401416+0000","last_unstale":"2024-05-15T23:41:31.492400+0000","last_undegraded":"2024-05-15T23:41:31.492400+0000","last_fullsized":"2024-05-15T23:41:31.492400+0000","mapping_epoch":737,"log_start":"744'1600","ondisk_log_start":"744'1600","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"761'5246","last_scrub_stamp":"2024-05-15T23:39:55.792018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:39:55.792018+0000","objects_scrubbed":5229,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:33.723146+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243204096,"num_objects":3711,"num_object_clones":0,"num_object_copies":7422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11691,"num_write_kb":492864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"780'11644","reported_seq":20528,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-05-15T23:41:32.318980+0000","last_change":"2024-05-15T23:41:06.834872+0000","last_active":"2024-05-15T23:41:32.318980+0000","last_peered":"2024-05-15T23:41:32.318980+0000","last_clean":"2024-05-15T23:41:32.318980+0000","last_became_active":"2024-05-15T23:39:43.637604+0000","last_became_peered":"2024-05-15T23:39:43.637604+0000","last_unstale":"2024-05-15T23:41:32.318980+0000","last_undegraded":"2024-05-15T23:41:32.318980+0000","last_fullsized":"2024-05-15T23:41:32.318980+0000","mapping_epoch":758,"log_start":"744'1600","ondisk_log_start":"744'1600","created":737,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:33.172634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240517120,"num_objects":3670,"num_object_clones":0,"num_object_copies":7340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11644,"num_write_kb":490048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4391,"num_bytes_recovered":287768576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"779'11269","reported_seq":20174,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-05-15T23:41:28.116842+0000","last_change":"2024-05-15T23:40:51.174588+0000","last_active":"2024-05-15T23:41:28.116842+0000","last_peered":"2024-05-15T23:41:28.116842+0000","last_clean":"2024-05-15T23:41:28.116842+0000","last_became_active":"2024-05-15T23:39:43.514269+0000","last_became_peered":"2024-05-15T23:39:43.514269+0000","last_unstale":"2024-05-15T23:41:28.116842+0000","last_undegraded":"2024-05-15T23:41:28.116842+0000","last_fullsized":"2024-05-15T23:41:28.116842+0000","mapping_epoch":758,"log_start":"744'1200","ondisk_log_start":"744'1200","created":737,"last_epoch_clean":771,"parent":"0.0","parent_split_bits":0,"last_scrub":"771'8828","last_scrub_stamp":"2024-05-15T23:40:51.174438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:51.174438+0000","objects_scrubbed":6515,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:59.748880+0000","scrub_duration":1137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264699904,"num_objects":4039,"num_object_clones":0,"num_object_copies":8078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11269,"num_write_kb":489856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":286064640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"780'11791","reported_seq":11964,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-05-15T23:41:32.164827+0000","last_change":"2024-05-15T23:40:45.307444+0000","last_active":"2024-05-15T23:41:32.164827+0000","last_peered":"2024-05-15T23:41:32.164827+0000","last_clean":"2024-05-15T23:41:32.164827+0000","last_became_active":"2024-05-15T23:38:50.398410+0000","last_became_peered":"2024-05-15T23:38:50.398410+0000","last_unstale":"2024-05-15T23:41:32.164827+0000","last_undegraded":"2024-05-15T23:41:32.164827+0000","last_fullsized":"2024-05-15T23:41:32.164827+0000","mapping_epoch":737,"log_start":"745'1700","ondisk_log_start":"745'1700","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'8645","last_scrub_stamp":"2024-05-15T23:40:45.307336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:45.307336+0000","objects_scrubbed":6921,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:03.552738+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247267328,"num_objects":3773,"num_object_clones":0,"num_object_copies":7546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11791,"num_write_kb":498048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"780'11640","reported_seq":11735,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-05-15T23:41:30.305493+0000","last_change":"2024-05-15T23:38:50.402064+0000","last_active":"2024-05-15T23:41:30.305493+0000","last_peered":"2024-05-15T23:41:30.305493+0000","last_clean":"2024-05-15T23:41:30.305493+0000","last_became_active":"2024-05-15T23:38:50.401779+0000","last_became_peered":"2024-05-15T23:38:50.401779+0000","last_unstale":"2024-05-15T23:41:30.305493+0000","last_undegraded":"2024-05-15T23:41:30.305493+0000","last_fullsized":"2024-05-15T23:41:30.305493+0000","mapping_epoch":737,"log_start":"744'1600","ondisk_log_start":"744'1600","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:32.124863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251265024,"num_objects":3834,"num_object_clones":0,"num_object_copies":7668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11640,"num_write_kb":495168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"780'11887","reported_seq":11983,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-05-15T23:41:32.281685+0000","last_change":"2024-05-15T23:38:50.402094+0000","last_active":"2024-05-15T23:41:32.281685+0000","last_peered":"2024-05-15T23:41:32.281685+0000","last_clean":"2024-05-15T23:41:32.281685+0000","last_became_active":"2024-05-15T23:38:50.401872+0000","last_became_peered":"2024-05-15T23:38:50.401872+0000","last_unstale":"2024-05-15T23:41:32.281685+0000","last_undegraded":"2024-05-15T23:41:32.281685+0000","last_fullsized":"2024-05-15T23:41:32.281685+0000","mapping_epoch":737,"log_start":"745'1800","ondisk_log_start":"745'1800","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:33.172593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245039104,"num_objects":3739,"num_object_clones":0,"num_object_copies":7478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11887,"num_write_kb":500032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"780'11858","reported_seq":11954,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-05-15T23:41:32.325846+0000","last_change":"2024-05-15T23:38:50.401343+0000","last_active":"2024-05-15T23:41:32.325846+0000","last_peered":"2024-05-15T23:41:32.325846+0000","last_clean":"2024-05-15T23:41:32.325846+0000","last_became_active":"2024-05-15T23:38:50.401074+0000","last_became_peered":"2024-05-15T23:38:50.401074+0000","last_unstale":"2024-05-15T23:41:32.325846+0000","last_undegraded":"2024-05-15T23:41:32.325846+0000","last_fullsized":"2024-05-15T23:41:32.325846+0000","mapping_epoch":737,"log_start":"745'1800","ondisk_log_start":"745'1800","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:33.172714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247988224,"num_objects":3784,"num_object_clones":0,"num_object_copies":7568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11858,"num_write_kb":500544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"779'11644","reported_seq":11813,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-05-15T23:41:28.123056+0000","last_change":"2024-05-15T23:40:50.387123+0000","last_active":"2024-05-15T23:41:28.123056+0000","last_peered":"2024-05-15T23:41:28.123056+0000","last_clean":"2024-05-15T23:41:28.123056+0000","last_became_active":"2024-05-15T23:38:50.401998+0000","last_became_peered":"2024-05-15T23:38:50.401998+0000","last_unstale":"2024-05-15T23:41:28.123056+0000","last_undegraded":"2024-05-15T23:41:28.123056+0000","last_fullsized":"2024-05-15T23:41:28.123056+0000","mapping_epoch":737,"log_start":"744'1600","ondisk_log_start":"744'1600","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"771'9050","last_scrub_stamp":"2024-05-15T23:40:50.386878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:50.386878+0000","objects_scrubbed":6683,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:57.422858+0000","scrub_duration":1397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265551872,"num_objects":4052,"num_object_clones":0,"num_object_copies":8104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11644,"num_write_kb":502272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.8","version":"780'11760","reported_seq":11878,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-05-15T23:41:30.307380+0000","last_change":"2024-05-15T23:41:13.496285+0000","last_active":"2024-05-15T23:41:30.307380+0000","last_peered":"2024-05-15T23:41:30.307380+0000","last_clean":"2024-05-15T23:41:30.307380+0000","last_became_active":"2024-05-15T23:38:50.404155+0000","last_became_peered":"2024-05-15T23:38:50.404155+0000","last_unstale":"2024-05-15T23:41:30.307380+0000","last_undegraded":"2024-05-15T23:41:30.307380+0000","last_fullsized":"2024-05-15T23:41:30.307380+0000","mapping_epoch":737,"log_start":"745'1700","ondisk_log_start":"745'1700","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":1899,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:32.124831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257032192,"num_objects":3922,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11760,"num_write_kb":501824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1695,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-05-15T23:41:28.444585+0000","last_change":"2024-05-15T23:40:50.840277+0000","last_active":"2024-05-15T23:41:28.444585+0000","last_peered":"2024-05-15T23:41:28.444585+0000","last_clean":"2024-05-15T23:41:28.444585+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:41:28.444585+0000","last_undegraded":"2024-05-15T23:41:28.444585+0000","last_fullsized":"2024-05-15T23:41:28.444585+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:40:50.840191+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:40:50.840191+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:15.227987+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4026400792,"num_objects":61439,"num_object_clones":0,"num_object_copies":122878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186933,"num_write_kb":7947841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17502,"num_bytes_recovered":1146880000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9104465920,"data_stored":9104457776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161033,"ondisk_log_size":161033,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351429,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1950520,"kb_used_data":1053040,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":91794632,"statfs":{"total":95995035648,"available":93997703168,"internally_reserved":0,"allocated":1078312960,"data_stored":1077922761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":919009732},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580362,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1641984,"kb_used_data":984628,"kb_used_omap":1,"kb_used_meta":657278,"kb_avail":92103168,"statfs":{"total":95995035648,"available":94313644032,"internally_reserved":0,"allocated":1008259072,"data_stored":1007866420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":673053120},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776594,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2109948,"kb_used_data":1244404,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":91635204,"statfs":{"total":95995035648,"available":93834448896,"internally_reserved":0,"allocated":1274269696,"data_stored":1273877044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":886307264},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":520,"seq":2233382994097,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":386624,"kb_used_data":255284,"kb_used_omap":24,"kb_used_meta":131303,"kb_avail":93358528,"statfs":{"total":95995035648,"available":95599132672,"internally_reserved":0,"allocated":261410816,"data_stored":261018164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25357,"internal_metadata":134454515},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267163,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2137100,"kb_used_data":1397888,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":91608052,"statfs":{"total":95995035648,"available":93806645248,"internally_reserved":0,"allocated":1431437312,"data_stored":1431037036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":756939210},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568496097,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575355,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2317184,"kb_used_data":1375288,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":91427968,"statfs":{"total":95995035648,"available":93622239232,"internally_reserved":0,"allocated":1408294912,"data_stored":1407898188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":964491718},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016765,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483452,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":209408,"kb_used_data":756,"kb_used_omap":4,"kb_used_meta":208635,"kb_avail":93535744,"statfs":{"total":95995035648,"available":95780601856,"internally_reserved":0,"allocated":774144,"data_stored":381492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":213642947},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150194,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2417788,"kb_used_data":1430068,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":91327364,"statfs":{"total":95995035648,"available":93519220736,"internally_reserved":0,"allocated":1464389632,"data_stored":1463996980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1011415495},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986665,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1253056,"kb_used_data":509684,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92492096,"statfs":{"total":95995035648,"available":94711906304,"internally_reserved":0,"allocated":521916416,"data_stored":521523764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1230704640,"data_stored":1230700568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1457324032,"data_stored":1457324032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":514457600,"data_stored":514457600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1040711680,"data_stored":1040711680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1277628416,"data_stored":1277624344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":259457024,"data_stored":259457024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1266417664,"data_stored":1266417664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":994836480,"data_stored":994836480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1062928384,"data_stored":1062928384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:41:34.403 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T23:41:34.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:34.526 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:34.558 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:34.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:34.570 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:34.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:41:34.702 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:34.735 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:41:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:34.745 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:41:34.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:34.910 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:34.967 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:34.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:34.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:35.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:41:35.128 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:35.278 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:41:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:35.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:41:35.356 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:35.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:35.433 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:35.468 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:35.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:35.477 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:35.578 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:41:35.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:35 smithi084 ceph-mon[39784]: pgmap v2089: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:41:35.610 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:35.671 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:41:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:35.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:41:35.774 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:41:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:35 smithi090 ceph-mon[44434]: pgmap v2089: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:41:35.802 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:35 smithi088 ceph-mon[44699]: pgmap v2089: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:41:35.804 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:35.825 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:41:35.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:35.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:41:35.928 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:41:35.957 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:35.981 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:41:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:35.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:41:35.991 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:36.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:36.133 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:36.143 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:41:36.170 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:36.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:41:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:36.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:36.202 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:41:36.242 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:41:36.281 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:36.289 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:41:36.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:41:36.340 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:36.346 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:41:36.370 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:36.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:36.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:36.387 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:41:36.412 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:41:36.454 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:41:36.481 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:41:36.487 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:41:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:41:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:41:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:41:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:41:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:41:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-05-15T23:41:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:41:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:41:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:41:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:41:36.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:41:36.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:41:36.511 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:36.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3280542897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:41:36.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:36 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:36.529 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:41:36.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:36.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:41:36.633 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:41:36.662 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:36.684 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:41:36.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:36.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:41:36.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:41:36.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3280542897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:41:36.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:36 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3280542897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:41:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:36 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:36.851 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:36.886 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:41:36.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:36.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:41:36.995 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:41:37.033 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:37.378 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:41:37.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:37 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:37 smithi090 ceph-mon[44434]: pgmap v2090: 17 pgs: 17 active+clean; 3.4 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.18k op/s 2024-05-15T23:41:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:37 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:37 smithi088 ceph-mon[44699]: pgmap v2090: 17 pgs: 17 active+clean; 3.4 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.18k op/s 2024-05-15T23:41:37.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:37 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:37.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:37 smithi084 ceph-mon[39784]: pgmap v2090: 17 pgs: 17 active+clean; 3.4 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.18k op/s 2024-05-15T23:41:38.094 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:41:38.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:38.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:41:38.199 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:41:38.238 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:38.339 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:41:38.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:38.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:41:38.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:38.499 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:38.530 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:38.534 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:38.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:38.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:41:38.674 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:38.704 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:41:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:38.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:41:38.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:38.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:41:38.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:38.873 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:41:38.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:38.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:41:38.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:39.018 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:39.053 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:39.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:39.054 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:39.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:41:39.193 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:39.231 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:41:39.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:39.235 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:41:39.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:41:39.373 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:39.731 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:39 smithi084 ceph-mon[39784]: pgmap v2091: 17 pgs: 17 active+clean; 3.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.13k op/s 2024-05-15T23:41:39.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:39 smithi090 ceph-mon[44434]: pgmap v2091: 17 pgs: 17 active+clean; 3.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.13k op/s 2024-05-15T23:41:39.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:39 smithi088 ceph-mon[44699]: pgmap v2091: 17 pgs: 17 active+clean; 3.3 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.13k op/s 2024-05-15T23:41:40.016 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:41:40.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:40.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:41:40.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:40.169 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:40.269 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:40.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:40.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:40.374 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:41:40.411 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:41.177 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:41:41.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:41.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:41:41.285 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:41.337 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:41.379 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:41.386 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:41.487 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:41:41.515 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:41.535 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:41:41.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:41.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:41:41.574 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:41:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-05-15T23:41:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-05-15T23:41:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [] 2024-05-15T23:41:41.574 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:41:41.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:41.688 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:41 smithi090 ceph-mon[44434]: pgmap v2092: 17 pgs: 17 active+clean; 3.2 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:41:41.690 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:41.732 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:41.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:41.744 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:41.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:41 smithi088 ceph-mon[44699]: pgmap v2092: 17 pgs: 17 active+clean; 3.2 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:41:41.845 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:41:41.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:41 smithi084 ceph-mon[39784]: pgmap v2092: 17 pgs: 17 active+clean; 3.2 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:41:41.872 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:41:41.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:41.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:41:41.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:42.040 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:42.068 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:42.077 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:42.179 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:41:42.206 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:42.228 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:41:42.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:42.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:41:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:41:42.290 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:41:42.333 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:42.383 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:42.410 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:42.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:42.484 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:41:42.516 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:41:42.542 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:42 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/339377604' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:41:42.543 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:42 smithi088 ceph-mon[44699]: from='client.37559 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:41:42.544 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:42.564 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:41:42.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:42.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:41:42.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:42 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/339377604' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:41:42.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:42 smithi084 ceph-mon[39784]: from='client.37559 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:41:42.668 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:41:42.699 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:42.743 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:41:42.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:42.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:41:42.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:42 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/339377604' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:41:42.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:42 smithi090 ceph-mon[44434]: from='client.37559 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:41:42.847 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:42.888 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:43.339 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:43.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:43.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:43.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:43.477 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:43.521 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:43.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:43.524 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:43.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:43 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:41:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:43 smithi084 ceph-mon[39784]: from='client.37559 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:41:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:43 smithi084 ceph-mon[39784]: osdmap e782: 12 total, 11 up, 9 in 2024-05-15T23:41:43.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:43 smithi084 ceph-mon[39784]: pgmap v2094: 17 pgs: 17 active+clean; 3.0 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:41:43.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:41:43.667 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:43.704 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:41:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:43.708 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:43 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:41:43.708 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:43 smithi090 ceph-mon[44434]: from='client.37559 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:41:43.708 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:43 smithi090 ceph-mon[44434]: osdmap e782: 12 total, 11 up, 9 in 2024-05-15T23:41:43.708 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:43 smithi090 ceph-mon[44434]: pgmap v2094: 17 pgs: 17 active+clean; 3.0 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:41:43.709 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:41:43.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:41:43.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:43 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:41:43.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:43 smithi088 ceph-mon[44699]: from='client.37559 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:41:43.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:43 smithi088 ceph-mon[44699]: osdmap e782: 12 total, 11 up, 9 in 2024-05-15T23:41:43.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:43 smithi088 ceph-mon[44699]: pgmap v2094: 17 pgs: 17 active+clean; 3.0 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:41:43.846 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:43.890 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:41:43.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:43.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:41:43.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T23:41:43.945 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:44.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:41:44.040 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:44.066 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:41:44.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:44.070 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:41:44.171 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:41:44.202 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:44.234 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:41:44.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:44.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:41:44.290 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:41:44.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:41:44.339 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:41:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:44.658 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:41:44.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:44.664 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:41:44.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:44.835 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:44.868 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:44.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:44.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:44.949 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:44 smithi090 ceph-mon[44434]: 9.7 scrub starts 2024-05-15T23:41:44.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:41:45.022 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:45.064 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:41:45.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:45.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:41:45.172 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:41:45.201 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:44 smithi084 ceph-mon[39784]: 9.7 scrub starts 2024-05-15T23:41:45.209 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:45.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:44 smithi088 ceph-mon[44699]: 9.7 scrub starts 2024-05-15T23:41:45.360 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:41:45.462 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:41:45.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:45.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:41:45.538 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:45.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:41:45.614 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:45.958 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:45 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:41:45.958 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:45 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:41:45.958 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:45 smithi090 ceph-mon[44434]: 9.7 scrub ok 2024-05-15T23:41:45.958 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:45 smithi090 ceph-mon[44434]: pgmap v2095: 17 pgs: 17 active+clean; 2.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:41:46.086 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:41:46.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:46.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:41:46.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:45 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:41:46.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:45 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:41:46.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:45 smithi084 ceph-mon[39784]: 9.7 scrub ok 2024-05-15T23:41:46.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:45 smithi084 ceph-mon[39784]: pgmap v2095: 17 pgs: 17 active+clean; 2.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:41:46.127 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:46.191 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:41:46.231 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:46.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:45 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:41:46.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:45 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:41:46.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:45 smithi088 ceph-mon[44699]: 9.7 scrub ok 2024-05-15T23:41:46.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:45 smithi088 ceph-mon[44699]: pgmap v2095: 17 pgs: 17 active+clean; 2.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:41:46.384 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:41:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:46.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:41:46.490 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:41:46.495 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:41:46.495 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:46.534 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:47.019 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:41:47.019 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:41:47.019 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-05-15T23:41:47.019 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:41:47.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:46 smithi090 ceph-mon[44434]: osdmap e783: 12 total, 11 up, 9 in 2024-05-15T23:41:47.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:46 smithi088 ceph-mon[44699]: osdmap e783: 12 total, 11 up, 9 in 2024-05-15T23:41:47.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:46 smithi084 ceph-mon[39784]: osdmap e783: 12 total, 11 up, 9 in 2024-05-15T23:41:47.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T23:41:48.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:47 smithi090 ceph-mon[44434]: from='client.17619 v1:172.21.15.84:0/2035612472' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:41:48.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:47 smithi090 ceph-mon[44434]: pgmap v2097: 17 pgs: 17 active+clean; 2.7 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:41:48.191 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:48.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:48.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:48.215 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:41:48.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:48.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:47 smithi088 ceph-mon[44699]: from='client.17619 v1:172.21.15.84:0/2035612472' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:41:48.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:47 smithi088 ceph-mon[44699]: pgmap v2097: 17 pgs: 17 active+clean; 2.7 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:41:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:48.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:47 smithi084 ceph-mon[39784]: from='client.17619 v1:172.21.15.84:0/2035612472' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:41:48.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:47 smithi084 ceph-mon[39784]: pgmap v2097: 17 pgs: 17 active+clean; 2.7 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.23k op/s 2024-05-15T23:41:48.372 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:48.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:48.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:48.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:48.525 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:48.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:48.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:48.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:41:48.703 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:48.771 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:41:48.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:48.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:41:48.883 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:41:48.922 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:49.047 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:41:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:49.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:41:49.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:41:49.195 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:49.452 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:41:49.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:49.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:41:49.558 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:41:49.592 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:49.688 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:49 smithi090 ceph-mon[44434]: pgmap v2098: 17 pgs: 17 active+clean; 2.7 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s 2024-05-15T23:41:49.799 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:41:49.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:49 smithi084 ceph-mon[39784]: pgmap v2098: 17 pgs: 17 active+clean; 2.7 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s 2024-05-15T23:41:49.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:49.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:41:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:49 smithi088 ceph-mon[44699]: pgmap v2098: 17 pgs: 17 active+clean; 2.7 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s 2024-05-15T23:41:49.903 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:41:49.942 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:50.413 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-05-15T23:41:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-05-15T23:41:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-05-15T23:41:50.413 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-05-15T23:41:50.414 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-05-15T23:41:50.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:50 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:41:50.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:50 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:41:50.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:50 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:41:50.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:50 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:41:50.857 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:50 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:41:50.857 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:50 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:41:50.892 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":2096,"stamp":"2024-05-15T23:41:45.955980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3115582008,"num_objects":47534,"num_object_clones":0,"num_object_copies":95068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47534,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":200963,"num_write_kb":7949991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17508,"num_bytes_recovered":1148466784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160734,"ondisk_log_size":160734,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":8,"kb":843706368,"kb_used":12585164,"kb_used_data":6388432,"kb_used_omap":39,"kb_used_meta":6195992,"kb_avail":831121204,"statfs":{"total":863955320832,"available":851068112896,"internally_reserved":0,"allocated":6541754368,"data_stored":6538475007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40938,"internal_metadata":6344695830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,11,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-544407552,"num_objects":-8307,"num_object_clones":0,"num_object_copies":-16614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8307,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.577904"},"pg_stats":[{"pgid":"9.f","version":"781'12222","reported_seq":21026,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-05-15T23:41:42.333688+0000","last_change":"2024-05-15T23:41:09.439865+0000","last_active":"2024-05-15T23:41:42.333688+0000","last_peered":"2024-05-15T23:41:42.333688+0000","last_clean":"2024-05-15T23:41:42.333688+0000","last_became_active":"2024-05-15T23:39:43.643192+0000","last_became_peered":"2024-05-15T23:39:43.643192+0000","last_unstale":"2024-05-15T23:41:42.333688+0000","last_undegraded":"2024-05-15T23:41:42.333688+0000","last_fullsized":"2024-05-15T23:41:42.333688+0000","mapping_epoch":758,"log_start":"745'2200","ondisk_log_start":"745'2200","created":737,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:47.185456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196083712,"num_objects":2992,"num_object_clones":0,"num_object_copies":5984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12222,"num_write_kb":486848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4349,"num_bytes_recovered":285016064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"782'12608","reported_seq":12781,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-05-15T23:41:44.981091+0000","last_change":"2024-05-15T23:40:52.788179+0000","last_active":"2024-05-15T23:41:44.981091+0000","last_peered":"2024-05-15T23:41:44.981091+0000","last_clean":"2024-05-15T23:41:44.981091+0000","last_became_active":"2024-05-15T23:38:50.402740+0000","last_became_peered":"2024-05-15T23:38:50.402740+0000","last_unstale":"2024-05-15T23:41:44.981091+0000","last_undegraded":"2024-05-15T23:41:44.981091+0000","last_fullsized":"2024-05-15T23:41:44.981091+0000","mapping_epoch":737,"log_start":"745'2600","ondisk_log_start":"745'2600","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'9081","last_scrub_stamp":"2024-05-15T23:40:52.788078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:52.788078+0000","objects_scrubbed":6463,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:58.533630+0000","scrub_duration":960,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190447616,"num_objects":2906,"num_object_clones":0,"num_object_copies":5812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12608,"num_write_kb":496448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"782'13085","reported_seq":13260,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-05-15T23:41:44.925087+0000","last_change":"2024-05-15T23:40:53.768684+0000","last_active":"2024-05-15T23:41:44.925087+0000","last_peered":"2024-05-15T23:41:44.925087+0000","last_clean":"2024-05-15T23:41:44.925087+0000","last_became_active":"2024-05-15T23:38:50.401123+0000","last_became_peered":"2024-05-15T23:38:50.401123+0000","last_unstale":"2024-05-15T23:41:44.925087+0000","last_undegraded":"2024-05-15T23:41:44.925087+0000","last_fullsized":"2024-05-15T23:41:44.925087+0000","mapping_epoch":737,"log_start":"745'3000","ondisk_log_start":"745'3000","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"773'9423","last_scrub_stamp":"2024-05-15T23:40:53.768529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:53.768529+0000","objects_scrubbed":6535,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:14.948562+0000","scrub_duration":969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186646552,"num_objects":2849,"num_object_clones":0,"num_object_copies":5698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13085,"num_write_kb":509825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"781'12570","reported_seq":12690,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-05-15T23:41:42.294936+0000","last_change":"2024-05-15T23:41:13.494498+0000","last_active":"2024-05-15T23:41:42.294936+0000","last_peered":"2024-05-15T23:41:42.294936+0000","last_clean":"2024-05-15T23:41:42.294936+0000","last_became_active":"2024-05-15T23:38:50.402428+0000","last_became_peered":"2024-05-15T23:38:50.402428+0000","last_unstale":"2024-05-15T23:41:42.294936+0000","last_undegraded":"2024-05-15T23:41:42.294936+0000","last_fullsized":"2024-05-15T23:41:42.294936+0000","mapping_epoch":737,"log_start":"745'2500","ondisk_log_start":"745'2500","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":1899,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:47.183509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197656576,"num_objects":3016,"num_object_clones":0,"num_object_copies":6032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12570,"num_write_kb":498752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"782'12541","reported_seq":12717,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-05-15T23:41:44.272786+0000","last_change":"2024-05-15T23:40:45.650576+0000","last_active":"2024-05-15T23:41:44.272786+0000","last_peered":"2024-05-15T23:41:44.272786+0000","last_clean":"2024-05-15T23:41:44.272786+0000","last_became_active":"2024-05-15T23:38:50.401815+0000","last_became_peered":"2024-05-15T23:38:50.401815+0000","last_unstale":"2024-05-15T23:41:44.272786+0000","last_undegraded":"2024-05-15T23:41:44.272786+0000","last_fullsized":"2024-05-15T23:41:44.272786+0000","mapping_epoch":737,"log_start":"745'2500","ondisk_log_start":"745'2500","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'8522","last_scrub_stamp":"2024-05-15T23:40:45.650391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:45.650391+0000","objects_scrubbed":6888,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:58.125696+0000","scrub_duration":745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187498496,"num_objects":2861,"num_object_clones":0,"num_object_copies":5722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12541,"num_write_kb":492864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"782'12508","reported_seq":21482,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-05-15T23:41:44.936081+0000","last_change":"2024-05-15T23:40:55.754806+0000","last_active":"2024-05-15T23:41:44.936081+0000","last_peered":"2024-05-15T23:41:44.936081+0000","last_clean":"2024-05-15T23:41:44.936081+0000","last_became_active":"2024-05-15T23:40:45.476827+0000","last_became_peered":"2024-05-15T23:40:45.476827+0000","last_unstale":"2024-05-15T23:41:44.936081+0000","last_undegraded":"2024-05-15T23:41:44.936081+0000","last_fullsized":"2024-05-15T23:41:44.936081+0000","mapping_epoch":769,"log_start":"745'2500","ondisk_log_start":"745'2500","created":737,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"774'9184","last_scrub_stamp":"2024-05-15T23:40:55.754657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:55.754657+0000","objects_scrubbed":6210,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:14.046965+0000","scrub_duration":922,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187039744,"num_objects":2854,"num_object_clones":0,"num_object_copies":5708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12508,"num_write_kb":491584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4397,"num_bytes_recovered":288030720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"781'12516","reported_seq":12704,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-05-15T23:41:41.455775+0000","last_change":"2024-05-15T23:40:48.476758+0000","last_active":"2024-05-15T23:41:41.455775+0000","last_peered":"2024-05-15T23:41:41.455775+0000","last_clean":"2024-05-15T23:41:41.455775+0000","last_became_active":"2024-05-15T23:38:50.404189+0000","last_became_peered":"2024-05-15T23:38:50.404189+0000","last_unstale":"2024-05-15T23:41:41.455775+0000","last_undegraded":"2024-05-15T23:41:41.455775+0000","last_fullsized":"2024-05-15T23:41:41.455775+0000","mapping_epoch":737,"log_start":"745'2500","ondisk_log_start":"745'2500","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"771'8884","last_scrub_stamp":"2024-05-15T23:40:48.476557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:48.476557+0000","objects_scrubbed":6814,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:02.814899+0000","scrub_duration":1372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":3136,"num_object_clones":0,"num_object_copies":6272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12516,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"781'12342","reported_seq":12500,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-05-15T23:41:41.488131+0000","last_change":"2024-05-15T23:39:55.792219+0000","last_active":"2024-05-15T23:41:41.488131+0000","last_peered":"2024-05-15T23:41:41.488131+0000","last_clean":"2024-05-15T23:41:41.488131+0000","last_became_active":"2024-05-15T23:38:50.401416+0000","last_became_peered":"2024-05-15T23:38:50.401416+0000","last_unstale":"2024-05-15T23:41:41.488131+0000","last_undegraded":"2024-05-15T23:41:41.488131+0000","last_fullsized":"2024-05-15T23:41:41.488131+0000","mapping_epoch":737,"log_start":"745'2300","ondisk_log_start":"745'2300","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"761'5246","last_scrub_stamp":"2024-05-15T23:39:55.792018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:39:55.792018+0000","objects_scrubbed":5229,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:42.701969+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200540160,"num_objects":3060,"num_object_clones":0,"num_object_copies":6120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12342,"num_write_kb":492864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"781'12325","reported_seq":21213,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-05-15T23:41:42.284589+0000","last_change":"2024-05-15T23:41:06.834872+0000","last_active":"2024-05-15T23:41:42.284589+0000","last_peered":"2024-05-15T23:41:42.284589+0000","last_clean":"2024-05-15T23:41:42.284589+0000","last_became_active":"2024-05-15T23:39:43.637604+0000","last_became_peered":"2024-05-15T23:39:43.637604+0000","last_unstale":"2024-05-15T23:41:42.284589+0000","last_undegraded":"2024-05-15T23:41:42.284589+0000","last_fullsized":"2024-05-15T23:41:42.284589+0000","mapping_epoch":758,"log_start":"745'2300","ondisk_log_start":"745'2300","created":737,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:47.183576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195887104,"num_objects":2989,"num_object_clones":0,"num_object_copies":5978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12325,"num_write_kb":490048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4391,"num_bytes_recovered":287768576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"782'12331","reported_seq":21242,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-05-15T23:41:43.137276+0000","last_change":"2024-05-15T23:40:51.174588+0000","last_active":"2024-05-15T23:41:43.137276+0000","last_peered":"2024-05-15T23:41:43.137276+0000","last_clean":"2024-05-15T23:41:43.137276+0000","last_became_active":"2024-05-15T23:39:43.514269+0000","last_became_peered":"2024-05-15T23:39:43.514269+0000","last_unstale":"2024-05-15T23:41:43.137276+0000","last_undegraded":"2024-05-15T23:41:43.137276+0000","last_fullsized":"2024-05-15T23:41:43.137276+0000","mapping_epoch":758,"log_start":"745'2300","ondisk_log_start":"745'2300","created":737,"last_epoch_clean":771,"parent":"0.0","parent_split_bits":0,"last_scrub":"771'8828","last_scrub_stamp":"2024-05-15T23:40:51.174438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:51.174438+0000","objects_scrubbed":6515,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:59.748880+0000","scrub_duration":1137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195100672,"num_objects":2977,"num_object_clones":0,"num_object_copies":5954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12331,"num_write_kb":489856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":286064640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"781'12521","reported_seq":12696,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-05-15T23:41:42.169208+0000","last_change":"2024-05-15T23:40:45.307444+0000","last_active":"2024-05-15T23:41:42.169208+0000","last_peered":"2024-05-15T23:41:42.169208+0000","last_clean":"2024-05-15T23:41:42.169208+0000","last_became_active":"2024-05-15T23:38:50.398410+0000","last_became_peered":"2024-05-15T23:38:50.398410+0000","last_unstale":"2024-05-15T23:41:42.169208+0000","last_undegraded":"2024-05-15T23:41:42.169208+0000","last_fullsized":"2024-05-15T23:41:42.169208+0000","mapping_epoch":737,"log_start":"745'2500","ondisk_log_start":"745'2500","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'8645","last_scrub_stamp":"2024-05-15T23:40:45.307336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:45.307336+0000","objects_scrubbed":6921,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:03.552738+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199426048,"num_objects":3043,"num_object_clones":0,"num_object_copies":6086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12521,"num_write_kb":498048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"782'12660","reported_seq":12760,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-05-15T23:41:45.320426+0000","last_change":"2024-05-15T23:38:50.402064+0000","last_active":"2024-05-15T23:41:45.320426+0000","last_peered":"2024-05-15T23:41:45.320426+0000","last_clean":"2024-05-15T23:41:45.320426+0000","last_became_active":"2024-05-15T23:38:50.401779+0000","last_became_peered":"2024-05-15T23:38:50.401779+0000","last_unstale":"2024-05-15T23:41:45.320426+0000","last_undegraded":"2024-05-15T23:41:45.320426+0000","last_fullsized":"2024-05-15T23:41:45.320426+0000","mapping_epoch":737,"log_start":"745'2600","ondisk_log_start":"745'2600","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184418304,"num_objects":2814,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12660,"num_write_kb":495168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"781'12581","reported_seq":12679,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-05-15T23:41:42.335070+0000","last_change":"2024-05-15T23:38:50.402094+0000","last_active":"2024-05-15T23:41:42.335070+0000","last_peered":"2024-05-15T23:41:42.335070+0000","last_clean":"2024-05-15T23:41:42.335070+0000","last_became_active":"2024-05-15T23:38:50.401872+0000","last_became_peered":"2024-05-15T23:38:50.401872+0000","last_unstale":"2024-05-15T23:41:42.335070+0000","last_undegraded":"2024-05-15T23:41:42.335070+0000","last_fullsized":"2024-05-15T23:41:42.335070+0000","mapping_epoch":737,"log_start":"745'2500","ondisk_log_start":"745'2500","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:47.183544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199557120,"num_objects":3045,"num_object_clones":0,"num_object_copies":6090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12581,"num_write_kb":500032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"781'12539","reported_seq":12637,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-05-15T23:41:42.328216+0000","last_change":"2024-05-15T23:38:50.401343+0000","last_active":"2024-05-15T23:41:42.328216+0000","last_peered":"2024-05-15T23:41:42.328216+0000","last_clean":"2024-05-15T23:41:42.328216+0000","last_became_active":"2024-05-15T23:38:50.401074+0000","last_became_peered":"2024-05-15T23:38:50.401074+0000","last_unstale":"2024-05-15T23:41:42.328216+0000","last_undegraded":"2024-05-15T23:41:42.328216+0000","last_fullsized":"2024-05-15T23:41:42.328216+0000","mapping_epoch":737,"log_start":"745'2500","ondisk_log_start":"745'2500","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:47.185476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203358208,"num_objects":3103,"num_object_clones":0,"num_object_copies":6206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12539,"num_write_kb":500544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"782'12685","reported_seq":12860,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-05-15T23:41:43.119170+0000","last_change":"2024-05-15T23:40:50.387123+0000","last_active":"2024-05-15T23:41:43.119170+0000","last_peered":"2024-05-15T23:41:43.119170+0000","last_clean":"2024-05-15T23:41:43.119170+0000","last_became_active":"2024-05-15T23:38:50.401998+0000","last_became_peered":"2024-05-15T23:38:50.401998+0000","last_unstale":"2024-05-15T23:41:43.119170+0000","last_undegraded":"2024-05-15T23:41:43.119170+0000","last_fullsized":"2024-05-15T23:41:43.119170+0000","mapping_epoch":737,"log_start":"745'2600","ondisk_log_start":"745'2600","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"771'9050","last_scrub_stamp":"2024-05-15T23:40:50.386878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:40:50.386878+0000","objects_scrubbed":6683,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:41:57.422858+0000","scrub_duration":1397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197328896,"num_objects":3011,"num_object_clones":0,"num_object_copies":6022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12685,"num_write_kb":502272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.8","version":"782'12806","reported_seq":12928,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-05-15T23:41:45.287114+0000","last_change":"2024-05-15T23:41:13.496285+0000","last_active":"2024-05-15T23:41:45.287114+0000","last_peered":"2024-05-15T23:41:45.287114+0000","last_clean":"2024-05-15T23:41:45.287114+0000","last_became_active":"2024-05-15T23:38:50.404155+0000","last_became_peered":"2024-05-15T23:38:50.404155+0000","last_unstale":"2024-05-15T23:41:45.287114+0000","last_undegraded":"2024-05-15T23:41:45.287114+0000","last_fullsized":"2024-05-15T23:41:45.287114+0000","mapping_epoch":737,"log_start":"745'2800","ondisk_log_start":"745'2800","created":737,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T23:38:49.378874+0000","last_clean_scrub_stamp":"2024-05-15T23:38:49.378874+0000","objects_scrubbed":1899,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188481536,"num_objects":2876,"num_object_clones":0,"num_object_copies":5752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":501824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"57'94","reported_seq":1699,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-05-15T23:41:42.463975+0000","last_change":"2024-05-15T23:40:50.840277+0000","last_active":"2024-05-15T23:41:42.463975+0000","last_peered":"2024-05-15T23:41:42.463975+0000","last_clean":"2024-05-15T23:41:42.463975+0000","last_became_active":"2024-05-15T23:03:59.533393+0000","last_became_peered":"2024-05-15T23:03:59.533393+0000","last_unstale":"2024-05-15T23:41:42.463975+0000","last_undegraded":"2024-05-15T23:41:42.463975+0000","last_fullsized":"2024-05-15T23:41:42.463975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:40:50.840191+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:40:50.840191+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:42:15.227987+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3114991640,"num_objects":47532,"num_object_clones":0,"num_object_copies":95064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200840,"num_write_kb":7947841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17502,"num_bytes_recovered":1146880000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7264018432,"data_stored":7264010288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160640,"ondisk_log_size":160640,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351432,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1680288,"kb_used_data":782684,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":92064864,"statfs":{"total":95995035648,"available":94274420736,"internally_reserved":0,"allocated":801468416,"data_stored":801120142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":919009732},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580364,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1454344,"kb_used_data":805184,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92290808,"statfs":{"total":95995035648,"available":94505787392,"internally_reserved":0,"allocated":824508416,"data_stored":824109789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":664533440},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776597,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1779676,"kb_used_data":914132,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":91965476,"statfs":{"total":95995035648,"available":94172647424,"internally_reserved":0,"allocated":936071168,"data_stored":935726466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":886307264},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":520,"seq":2233382994100,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321372,"kb_used_data":185876,"kb_used_omap":24,"kb_used_meta":135463,"kb_avail":93423780,"statfs":{"total":95995035648,"available":95665950720,"internally_reserved":0,"allocated":190337024,"data_stored":189992322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25357,"internal_metadata":138714355},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267167,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1776680,"kb_used_data":1033312,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":91968472,"statfs":{"total":95995035648,"available":94175715328,"internally_reserved":0,"allocated":1058111488,"data_stored":1057759162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":761199050},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568496099,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575357,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2081612,"kb_used_data":1125828,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":91663540,"statfs":{"total":95995035648,"available":93863464960,"internally_reserved":0,"allocated":1152847872,"data_stored":1152445173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":978713030},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":1705102016767,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483455,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":209432,"kb_used_data":660,"kb_used_omap":4,"kb_used_meta":208635,"kb_avail":93535720,"statfs":{"total":95995035648,"available":95780577280,"internally_reserved":0,"allocated":675840,"data_stored":331138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":213642947},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150196,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2172168,"kb_used_data":1166336,"kb_used_omap":1,"kb_used_meta":1005758,"kb_avail":91572984,"statfs":{"total":95995035648,"available":93770735616,"internally_reserved":0,"allocated":1194328064,"data_stored":1193929437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1029896647},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1109592,"kb_used_data":374420,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":92635560,"statfs":{"total":95995035648,"available":94858813440,"internally_reserved":0,"allocated":383406080,"data_stored":383061378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":752679365},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1001328640,"data_stored":1001324568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1189871616,"data_stored":1189871616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":375848960,"data_stored":375848960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1040711680,"data_stored":1040711680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":935858176,"data_stored":935854104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":187498496,"data_stored":187498496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":931201024,"data_stored":931201024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":812908544,"data_stored":812908544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":788791296,"data_stored":788791296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:41:50.893 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-15T23:41:50.893 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-05-15T23:41:50.999 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:41:51.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:51.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:41:51.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:51.157 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:51.204 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:51.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:51.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T23:41:51.317 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:41:51.349 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:51.396 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:41:51.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:51.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:41:51.460 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:41:51.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:51 smithi090 ceph-mon[44434]: osdmap e784: 12 total, 11 up, 9 in 2024-05-15T23:41:51.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:51 smithi090 ceph-mon[44434]: pgmap v2100: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 920 op/s 2024-05-15T23:41:51.479 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2635883677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:41:51.492 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:41:51.518 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:41:51.521 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:41:51.548 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:41:51.549 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:51.579 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:41:51.603 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:41:51.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:51 smithi084 ceph-mon[39784]: osdmap e784: 12 total, 11 up, 9 in 2024-05-15T23:41:51.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:51 smithi084 ceph-mon[39784]: pgmap v2100: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 920 op/s 2024-05-15T23:41:51.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2635883677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:41:51.630 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:41:51.660 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:41:51.686 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:41:51.689 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:41:51.718 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:41:51.748 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:41:51.748 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:41:51.748 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:41:51.748 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:41:51.748 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:41:51.748 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:41:51.748 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-05-15T23:41:51.748 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:41:51.749 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:41:51.749 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:41:51.749 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:41:51.749 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:41:51.749 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:41:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:51 smithi088 ceph-mon[44699]: osdmap e784: 12 total, 11 up, 9 in 2024-05-15T23:41:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:51 smithi088 ceph-mon[44699]: pgmap v2100: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 920 op/s 2024-05-15T23:41:51.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2635883677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:41:52.559 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:41:52.559 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:41:52.607 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:41:52.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:52.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:41:52.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:41:52.746 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:52.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:52 smithi084 ceph-mon[39784]: 9.c scrub starts 2024-05-15T23:41:52.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:52 smithi084 ceph-mon[39784]: 9.c scrub ok 2024-05-15T23:41:52.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.748 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.748 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:52 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.754 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:52 smithi090 ceph-mon[44434]: 9.c scrub starts 2024-05-15T23:41:52.754 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.755 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.755 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.755 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:52 smithi090 ceph-mon[44434]: 9.c scrub ok 2024-05-15T23:41:52.755 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.755 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.755 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.755 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.755 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:52 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:52 smithi088 ceph-mon[44699]: 9.c scrub starts 2024-05-15T23:41:52.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:52 smithi088 ceph-mon[44699]: 9.c scrub ok 2024-05-15T23:41:52.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.822 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:52 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:52.830 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:41:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:52.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:41:52.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:41:52.993 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:53.118 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:41:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:53.131 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:41:53.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:41:53.276 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:53.307 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:41:53.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:53.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:41:53.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:53.497 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:53.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:53.511 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:41:53.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:53.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:53.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:53 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:53.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:53 smithi084 ceph-mon[39784]: 9.8 scrub starts 2024-05-15T23:41:53.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:53 smithi084 ceph-mon[39784]: 9.8 scrub ok 2024-05-15T23:41:53.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:53 smithi084 ceph-mon[39784]: pgmap v2101: 17 pgs: 17 active+clean; 2.4 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s 2024-05-15T23:41:53.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:53 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/774949974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:41:53.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:53.648 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:53.702 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:53.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:53.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:53.810 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:53.810 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:53.810 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:53 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:53.810 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:53 smithi090 ceph-mon[44434]: 9.8 scrub starts 2024-05-15T23:41:53.810 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:53 smithi090 ceph-mon[44434]: 9.8 scrub ok 2024-05-15T23:41:53.810 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:53 smithi090 ceph-mon[44434]: pgmap v2101: 17 pgs: 17 active+clean; 2.4 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s 2024-05-15T23:41:53.810 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:53 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/774949974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:41:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:53 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:41:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:53 smithi088 ceph-mon[44699]: 9.8 scrub starts 2024-05-15T23:41:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:53 smithi088 ceph-mon[44699]: 9.8 scrub ok 2024-05-15T23:41:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:53 smithi088 ceph-mon[44699]: pgmap v2101: 17 pgs: 17 active+clean; 2.4 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s 2024-05-15T23:41:53.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:53 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/774949974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:41:53.847 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:53.877 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:53.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:53.880 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:53.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:54.001 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:41:54.001 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:41:54.038 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:54.294 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:54.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:54.297 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:54.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:54.448 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:54.557 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:54 smithi090 ceph-mon[44434]: 9.3 scrub starts 2024-05-15T23:41:54.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:54 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:41:54.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:54 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:41:54.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:54 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:41:54.558 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:54 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:41:54.559 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/774949974' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:41:54.559 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:54 smithi090 ceph-mon[44434]: osdmap e785: 12 total, 11 up, 9 in 2024-05-15T23:41:54.559 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:54 smithi090 ceph-mon[44434]: 9.3 scrub ok 2024-05-15T23:41:54.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:41:54.598 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:54 smithi084 ceph-mon[39784]: 9.3 scrub starts 2024-05-15T23:41:54.598 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:54 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:41:54.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:54 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:41:54.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:54 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:41:54.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:54 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:41:54.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/774949974' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:41:54.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:54 smithi084 ceph-mon[39784]: osdmap e785: 12 total, 11 up, 9 in 2024-05-15T23:41:54.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:54 smithi084 ceph-mon[39784]: 9.3 scrub ok 2024-05-15T23:41:54.784 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-05-15T23:41:54.817 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:54.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:54 smithi088 ceph-mon[44699]: 9.3 scrub starts 2024-05-15T23:41:54.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:54 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:41:54.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:54 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:41:54.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:54 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:41:54.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:54 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:41:54.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/774949974' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:41:54.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:54 smithi088 ceph-mon[44699]: osdmap e785: 12 total, 11 up, 9 in 2024-05-15T23:41:54.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:54 smithi088 ceph-mon[44699]: 9.3 scrub ok 2024-05-15T23:41:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:54.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:54.922 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:41:54.949 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:54.968 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:54.968 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:41:54.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:54.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:41:55.073 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:41:55.099 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:55.121 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:41:55.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:55.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:41:55.225 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:41:55.254 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:55.274 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:41:55.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:55.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:41:55.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:41:55.423 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:55.457 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:41:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:55.461 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:41:55.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:55 smithi084 ceph-mon[39784]: pgmap v2103: 17 pgs: 17 active+clean; 2.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 996 op/s 2024-05-15T23:41:55.562 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:41:55.610 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:55.806 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:55 smithi090 ceph-mon[44434]: pgmap v2103: 17 pgs: 17 active+clean; 2.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 996 op/s 2024-05-15T23:41:55.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:55 smithi088 ceph-mon[44699]: pgmap v2103: 17 pgs: 17 active+clean; 2.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 996 op/s 2024-05-15T23:41:56.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:41:57.062 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:57.070 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:41:57.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:57.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:41:57.178 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:41:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:57.228 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:41:57.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:57.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:41:57.335 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:41:57.379 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:57.412 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:41:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:57.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:41:57.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:41:57.559 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:41:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:57.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:41:57.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:41:57.746 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:57.772 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:41:57.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:57.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:41:57.821 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:57 smithi084 ceph-mon[39784]: 9.2 scrub starts 2024-05-15T23:41:57.821 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:57 smithi084 ceph-mon[39784]: 9.2 scrub ok 2024-05-15T23:41:57.821 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:57 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:41:57.821 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:57 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:41:57.821 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:57 smithi084 ceph-mon[39784]: osdmap e786: 12 total, 11 up, 9 in 2024-05-15T23:41:57.879 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:41:57.907 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:57.932 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:41:57.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:41:57.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:41:58.036 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:41:58.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:57 smithi090 ceph-mon[44434]: 9.2 scrub starts 2024-05-15T23:41:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:57 smithi090 ceph-mon[44434]: 9.2 scrub ok 2024-05-15T23:41:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:57 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:41:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:57 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:41:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:57 smithi090 ceph-mon[44434]: osdmap e786: 12 total, 11 up, 9 in 2024-05-15T23:41:58.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:57 smithi088 ceph-mon[44699]: 9.2 scrub starts 2024-05-15T23:41:58.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:57 smithi088 ceph-mon[44699]: 9.2 scrub ok 2024-05-15T23:41:58.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:57 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:41:58.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:57 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:41:58.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:57 smithi088 ceph-mon[44699]: osdmap e786: 12 total, 11 up, 9 in 2024-05-15T23:41:58.071 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:41:59.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:58 smithi084 ceph-mon[39784]: 9.6 scrub starts 2024-05-15T23:41:59.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:58 smithi084 ceph-mon[39784]: pgmap v2105: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.1 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:41:59.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:41:58 smithi084 ceph-mon[39784]: 9.6 scrub ok 2024-05-15T23:41:59.290 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:41:59.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:58 smithi090 ceph-mon[44434]: 9.6 scrub starts 2024-05-15T23:41:59.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:58 smithi090 ceph-mon[44434]: pgmap v2105: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.1 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:41:59.304 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:41:58 smithi090 ceph-mon[44434]: 9.6 scrub ok 2024-05-15T23:41:59.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:58 smithi088 ceph-mon[44699]: 9.6 scrub starts 2024-05-15T23:41:59.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:58 smithi088 ceph-mon[44699]: pgmap v2105: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.1 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s 2024-05-15T23:41:59.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:41:58 smithi088 ceph-mon[44699]: 9.6 scrub ok 2024-05-15T23:42:01.012 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:00 smithi084 ceph-mon[39784]: pgmap v2106: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:42:01.030 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:00 smithi084 ceph-mon[39784]: 9.e scrub starts 2024-05-15T23:42:01.030 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:00 smithi084 ceph-mon[39784]: 9.e scrub ok 2024-05-15T23:42:01.030 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:00 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:42:01.030 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:00 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:42:01.034 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:42:01.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:01.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:42:01.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:42:01.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:00 smithi090 ceph-mon[44434]: pgmap v2106: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:42:01.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:00 smithi090 ceph-mon[44434]: 9.e scrub starts 2024-05-15T23:42:01.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:00 smithi090 ceph-mon[44434]: 9.e scrub ok 2024-05-15T23:42:01.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:00 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:42:01.195 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:00 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:42:01.198 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:01.235 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:42:01.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:01.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:42:01.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:00 smithi088 ceph-mon[44699]: pgmap v2106: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:42:01.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:00 smithi088 ceph-mon[44699]: 9.e scrub starts 2024-05-15T23:42:01.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:00 smithi088 ceph-mon[44699]: 9.e scrub ok 2024-05-15T23:42:01.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:00 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:42:01.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:00 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:42:01.341 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:42:01.377 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:02.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:01 smithi084 ceph-mon[39784]: osdmap e787: 12 total, 11 up, 9 in 2024-05-15T23:42:02.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:01 smithi084 ceph-mon[39784]: 9.0 scrub starts 2024-05-15T23:42:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:01 smithi090 ceph-mon[44434]: osdmap e787: 12 total, 11 up, 9 in 2024-05-15T23:42:02.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:01 smithi090 ceph-mon[44434]: 9.0 scrub starts 2024-05-15T23:42:02.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:01 smithi088 ceph-mon[44699]: osdmap e787: 12 total, 11 up, 9 in 2024-05-15T23:42:02.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:01 smithi088 ceph-mon[44699]: 9.0 scrub starts 2024-05-15T23:42:03.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:02 smithi084 ceph-mon[39784]: pgmap v2108: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.9 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 946 op/s 2024-05-15T23:42:03.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:02 smithi084 ceph-mon[39784]: 9.0 scrub ok 2024-05-15T23:42:03.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:02 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:42:03.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:02 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:42:03.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:02 smithi084 ceph-mon[39784]: 9.b scrub starts 2024-05-15T23:42:03.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:02 smithi084 ceph-mon[39784]: 9.b scrub ok 2024-05-15T23:42:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:02 smithi090 ceph-mon[44434]: pgmap v2108: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.9 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 946 op/s 2024-05-15T23:42:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:02 smithi090 ceph-mon[44434]: 9.0 scrub ok 2024-05-15T23:42:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:02 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:42:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:02 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:42:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:02 smithi090 ceph-mon[44434]: 9.b scrub starts 2024-05-15T23:42:03.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:02 smithi090 ceph-mon[44434]: 9.b scrub ok 2024-05-15T23:42:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:02 smithi088 ceph-mon[44699]: pgmap v2108: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.9 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 946 op/s 2024-05-15T23:42:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:02 smithi088 ceph-mon[44699]: 9.0 scrub ok 2024-05-15T23:42:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:02 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:42:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:02 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:42:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:02 smithi088 ceph-mon[44699]: 9.b scrub starts 2024-05-15T23:42:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:02 smithi088 ceph-mon[44699]: 9.b scrub ok 2024-05-15T23:42:03.446 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:42:03.446 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:42:03.633 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:42:03.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:03.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:42:03.737 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:42:03.765 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:03.784 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:42:03.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:03.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:42:03.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:42:03.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:42:03.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:42:03.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:42:03.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:42:03.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:42:03.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:42:03.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:42:03.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:42:03.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:42:03.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:42:03.791 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:42:03.791 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:42:03.791 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:42:03.791 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:42:03.791 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:42:03.791 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-05-15T23:42:03.791 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:42:03.791 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:42:03.791 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:42:03.791 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:42:03.791 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:42:03.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:03.918 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:42:03.930 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:03.957 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:03.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:03.965 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:04.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:42:04.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:03 smithi088 ceph-mon[44699]: 9.f scrub starts 2024-05-15T23:42:04.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:03 smithi088 ceph-mon[44699]: 9.f scrub ok 2024-05-15T23:42:04.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:03 smithi088 ceph-mon[44699]: osdmap e788: 12 total, 11 up, 9 in 2024-05-15T23:42:04.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2151988121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:42:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:03 smithi088 ceph-mon[44699]: pgmap v2110: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.7 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:42:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:03 smithi088 ceph-mon[44699]: 9.5 scrub starts 2024-05-15T23:42:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2217839637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:42:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:03 smithi088 ceph-mon[44699]: from='client.37589 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:42:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:03 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:03 smithi084 ceph-mon[39784]: 9.f scrub starts 2024-05-15T23:42:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:03 smithi084 ceph-mon[39784]: 9.f scrub ok 2024-05-15T23:42:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:03 smithi084 ceph-mon[39784]: osdmap e788: 12 total, 11 up, 9 in 2024-05-15T23:42:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2151988121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:42:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:03 smithi084 ceph-mon[39784]: pgmap v2110: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.7 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:42:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:03 smithi084 ceph-mon[39784]: 9.5 scrub starts 2024-05-15T23:42:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2217839637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:42:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:03 smithi084 ceph-mon[39784]: from='client.37589 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:42:04.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:03 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.122 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:04.157 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:42:04.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:04.163 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:42:04.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:03 smithi090 ceph-mon[44434]: 9.f scrub starts 2024-05-15T23:42:04.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:03 smithi090 ceph-mon[44434]: 9.f scrub ok 2024-05-15T23:42:04.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:03 smithi090 ceph-mon[44434]: osdmap e788: 12 total, 11 up, 9 in 2024-05-15T23:42:04.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2151988121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:42:04.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:03 smithi090 ceph-mon[44434]: pgmap v2110: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.7 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.24k op/s 2024-05-15T23:42:04.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:03 smithi090 ceph-mon[44434]: 9.5 scrub starts 2024-05-15T23:42:04.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2217839637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:42:04.164 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:03 smithi090 ceph-mon[44434]: from='client.37589 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:42:04.165 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:03 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:42:04.316 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:04.347 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:42:04.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:04.355 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:42:04.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:42:04.506 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:04.539 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:42:04.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:04.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:42:04.647 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:42:04.678 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:04.699 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:42:04.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:04.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:42:04.803 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:42:04.820 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-05-15T23:42:04.833 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:04.852 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:42:04.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:04.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:42:04.956 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:42:04.983 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.983 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:04 smithi088 ceph-mon[44699]: 9.1 scrub starts 2024-05-15T23:42:04.983 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.983 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.983 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.983 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:04 smithi088 ceph-mon[44699]: 9.5 scrub ok 2024-05-15T23:42:04.984 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.984 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.984 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:04 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:04.984 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:04 smithi088 ceph-mon[44699]: 9.1 scrub ok 2024-05-15T23:42:04.984 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:04 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:42:04.984 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:04 smithi088 ceph-mon[44699]: from='client.37589 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:42:04.984 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:04 smithi088 ceph-mon[44699]: osdmap e789: 12 total, 11 up, 9 in 2024-05-15T23:42:04.985 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:05.006 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:42:05.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:05.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:42:05.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:42:05.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:05.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:04 smithi084 ceph-mon[39784]: 9.1 scrub starts 2024-05-15T23:42:05.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:05.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:05.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:05.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:04 smithi084 ceph-mon[39784]: 9.5 scrub ok 2024-05-15T23:42:05.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:05.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:05.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:04 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:05.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:04 smithi084 ceph-mon[39784]: 9.1 scrub ok 2024-05-15T23:42:05.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:04 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:42:05.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:04 smithi084 ceph-mon[39784]: from='client.37589 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:42:05.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:04 smithi084 ceph-mon[39784]: osdmap e789: 12 total, 11 up, 9 in 2024-05-15T23:42:05.142 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:05.271 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:42:05.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:05.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:42:05.301 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:04 smithi090 ceph-mon[44434]: 9.1 scrub starts 2024-05-15T23:42:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:04 smithi090 ceph-mon[44434]: 9.5 scrub ok 2024-05-15T23:42:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:04 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:04 smithi090 ceph-mon[44434]: 9.1 scrub ok 2024-05-15T23:42:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:04 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:42:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:04 smithi090 ceph-mon[44434]: from='client.37589 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:42:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:04 smithi090 ceph-mon[44434]: osdmap e789: 12 total, 11 up, 9 in 2024-05-15T23:42:05.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:05.446 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:05.479 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:05.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:05.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:05.589 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:42:05.628 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:05.730 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:42:05.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:05.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:42:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:42:05.752 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:42:05.810 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:05.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:05.883 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:05.912 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:05.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:05.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:06.016 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:42:06.043 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:06.065 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:42:06.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:06.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:42:06.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:05 smithi084 ceph-mon[39784]: pgmap v2112: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.6 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.18k op/s 2024-05-15T23:42:06.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:42:06.208 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:06.306 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:05 smithi090 ceph-mon[44434]: pgmap v2112: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.6 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.18k op/s 2024-05-15T23:42:06.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:05 smithi088 ceph-mon[44699]: pgmap v2112: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.6 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.18k op/s 2024-05-15T23:42:06.371 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:42:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:06.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:42:06.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:42:06.518 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:06.546 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:42:06.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:06.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:42:06.655 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:42:06.683 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:06.704 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:42:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:06.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:42:06.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:42:06.861 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:06.907 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:42:06.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:06.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:42:07.018 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:42:07.045 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:07.068 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:42:07.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:07.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:42:07.172 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:42:07.217 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:07.240 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:42:07.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:07.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:42:07.299 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set nodeep-scrub 2024-05-15T23:42:07.344 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:42:07.371 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:07.391 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:42:07.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:07.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:42:07.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:42:07.523 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:07.543 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:42:07.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:07.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:42:07.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:42:07.696 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:07.729 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:42:07.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:07.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:42:07.752 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:42:07.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:42:07.862 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:42:07.862 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:42:07.862 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:42:07.862 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:42:07.862 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:42:07.884 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:07.914 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:42:07.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:07.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:42:08.022 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:42:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:08.070 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:42:08.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:08.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:42:08.122 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:08.174 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:42:08.201 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:08.223 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:42:08.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:08.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:42:08.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:42:08.405 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:08.436 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:42:08.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:08.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:42:08.543 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:42:08.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:08 smithi084 ceph-mon[39784]: pgmap v2113: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.5 GiB data, 9.1 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 949 op/s 2024-05-15T23:42:08.603 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:08.620 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-05-15T23:42:08.629 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:42:08.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:08.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:42:08.734 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:08.771 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:08 smithi090 ceph-mon[44434]: pgmap v2113: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.5 GiB data, 9.1 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 949 op/s 2024-05-15T23:42:08.771 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:08.817 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:08.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:08 smithi088 ceph-mon[44699]: pgmap v2113: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.5 GiB data, 9.1 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 949 op/s 2024-05-15T23:42:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:08.823 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:08.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:42:08.981 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:10.622 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:10.642 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:42:10.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:10.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:42:10.747 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:42:10.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:10 smithi088 ceph-mon[44699]: pgmap v2114: 17 pgs: 17 active+clean; 1.4 GiB data, 8.9 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:42:10.776 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:10 smithi090 ceph-mon[44434]: pgmap v2114: 17 pgs: 17 active+clean; 1.4 GiB data, 8.9 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:42:10.803 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:42:10.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:10.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:42:10.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:10 smithi084 ceph-mon[39784]: pgmap v2114: 17 pgs: 17 active+clean; 1.4 GiB data, 8.9 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:42:10.910 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:42:10.964 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:11.456 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:42:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:11.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:42:11.468 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:11.561 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:42:11.589 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:11.611 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:42:11.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:11.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:42:11.715 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:42:11.761 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:11.808 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:42:11.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:11.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:42:11.913 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:42:11.969 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:12 smithi090 ceph-mon[44434]: pgmap v2115: 17 pgs: 17 active+clean; 1.3 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 953 op/s 2024-05-15T23:42:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:12 smithi088 ceph-mon[44699]: pgmap v2115: 17 pgs: 17 active+clean; 1.3 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 953 op/s 2024-05-15T23:42:12.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:12 smithi084 ceph-mon[39784]: pgmap v2115: 17 pgs: 17 active+clean; 1.3 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 953 op/s 2024-05-15T23:42:13.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:13 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:42:13.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:13 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:42:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:13 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:42:13.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:13 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:42:13.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:13 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:42:13.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:13 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:42:14.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:14 smithi084 ceph-mon[39784]: pgmap v2116: 17 pgs: 17 active+clean; 1.1 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s 2024-05-15T23:42:14.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:14 smithi084 ceph-mon[39784]: osdmap e790: 12 total, 11 up, 9 in 2024-05-15T23:42:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:14 smithi090 ceph-mon[44434]: pgmap v2116: 17 pgs: 17 active+clean; 1.1 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s 2024-05-15T23:42:14.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:14 smithi090 ceph-mon[44434]: osdmap e790: 12 total, 11 up, 9 in 2024-05-15T23:42:14.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:14 smithi088 ceph-mon[44699]: pgmap v2116: 17 pgs: 17 active+clean; 1.1 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s 2024-05-15T23:42:14.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:14 smithi088 ceph-mon[44699]: osdmap e790: 12 total, 11 up, 9 in 2024-05-15T23:42:15.093 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:42:15.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:15.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:42:15.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:15.254 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:15.283 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:15.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:15.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:15.395 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:42:15.422 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:15.444 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:42:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:15.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:42:15.484 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-05-15T23:42:15.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:42:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:15 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1766228795' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:42:15.554 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:15 smithi090 ceph-mon[44434]: from='client.37595 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:42:15.555 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:15 smithi090 ceph-mon[44434]: pgmap v2118: 17 pgs: 17 active+clean; 946 MiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-05-15T23:42:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:15.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:15 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1766228795' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:42:15.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:15 smithi084 ceph-mon[39784]: from='client.37595 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:42:15.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:15 smithi084 ceph-mon[39784]: pgmap v2118: 17 pgs: 17 active+clean; 946 MiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-05-15T23:42:15.629 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:42:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:15.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:42:15.734 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:42:15.758 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:15 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1766228795' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:42:15.759 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:15 smithi088 ceph-mon[44699]: from='client.37595 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:42:15.759 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:15 smithi088 ceph-mon[44699]: pgmap v2118: 17 pgs: 17 active+clean; 946 MiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-05-15T23:42:15.761 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:15.784 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:42:15.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:15.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:42:15.866 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:42:15.867 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:42:15.885 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:42:15.891 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:42:15.924 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:42:15.932 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:15.961 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:42:16.000 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:42:16.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:16.006 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:42:16.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:42:16.032 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:42:16.075 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:42:16.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:42:16.108 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:42:16.142 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:42:16.144 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:16.176 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:42:16.176 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:42:16.176 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:42:16.176 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:42:16.176 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:42:16.177 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:42:16.177 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:42:16.177 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:42:16.177 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-05-15T23:42:16.177 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:42:16.177 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:42:16.177 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:42:16.177 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:42:16.177 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:42:16.177 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:42:16.185 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:42:16.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:16.193 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:42:16.294 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:42:16.322 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:16.341 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:42:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:16.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:42:16.445 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:42:16.474 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:16.494 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:42:16.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:16.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:42:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:16 smithi088 ceph-mon[44699]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:42:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:16 smithi088 ceph-mon[44699]: from='client.37595 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:42:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:16 smithi088 ceph-mon[44699]: osdmap e791: 12 total, 11 up, 9 in 2024-05-15T23:42:16.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:16 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3327418690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:42:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:16.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:16 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:16.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:42:16.641 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:16.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:16 smithi084 ceph-mon[39784]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:42:16.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:16 smithi084 ceph-mon[39784]: from='client.37595 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:42:16.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:16 smithi084 ceph-mon[39784]: osdmap e791: 12 total, 11 up, 9 in 2024-05-15T23:42:16.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:16 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3327418690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:42:16.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:16.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:16.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:16.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:16 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:16.750 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:42:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:16.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:42:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:16 smithi090 ceph-mon[44434]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:42:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:16 smithi090 ceph-mon[44434]: from='client.37595 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-05-15T23:42:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:16 smithi090 ceph-mon[44434]: osdmap e791: 12 total, 11 up, 9 in 2024-05-15T23:42:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:16 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3327418690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:42:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:16 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:16.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:42:16.888 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:17.484 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:17 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:17 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:42:17.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:17 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:42:17.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:17 smithi084 ceph-mon[39784]: osdmap e792: 12 total, 11 up, 9 in 2024-05-15T23:42:17.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:17 smithi084 ceph-mon[39784]: pgmap v2121: 17 pgs: 17 active+clean; 880 MiB data, 7.8 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-05-15T23:42:17.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:17 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:17 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:42:17.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:17 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:42:17.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:17 smithi090 ceph-mon[44434]: osdmap e792: 12 total, 11 up, 9 in 2024-05-15T23:42:17.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:17 smithi090 ceph-mon[44434]: pgmap v2121: 17 pgs: 17 active+clean; 880 MiB data, 7.8 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-05-15T23:42:17.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:17 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:17 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:42:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:17 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:42:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:17 smithi088 ceph-mon[44699]: osdmap e792: 12 total, 11 up, 9 in 2024-05-15T23:42:17.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:17 smithi088 ceph-mon[44699]: pgmap v2121: 17 pgs: 17 active+clean; 880 MiB data, 7.8 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-05-15T23:42:17.955 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:42:17.956 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:42:18.256 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:42:18.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:18.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:42:18.291 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-05-15T23:42:18.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:18.405 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:18.440 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:18.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:18.449 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:18.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:42:18.591 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:18.624 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:42:18.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:18.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:42:18.707 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:18.727 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:42:18.757 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:18.777 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:42:18.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:18.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:42:18.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:42:18.925 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:18.994 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:42:19.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:19.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:42:19.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:19.160 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:19.198 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:42:19.200 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:19.207 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:19.308 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:42:19.359 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:19.467 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:19 smithi090 ceph-mon[44434]: pgmap v2122: 17 pgs: 17 active+clean; 706 MiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 986 op/s 2024-05-15T23:42:19.512 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:42:19.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:19.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:42:19.516 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:19 smithi084 ceph-mon[39784]: pgmap v2122: 17 pgs: 17 active+clean; 706 MiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 986 op/s 2024-05-15T23:42:19.617 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:42:19.644 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:19.667 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:42:19.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:19.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:42:19.772 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:42:19.799 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:19 smithi088 ceph-mon[44699]: pgmap v2122: 17 pgs: 17 active+clean; 706 MiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 986 op/s 2024-05-15T23:42:19.801 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:19.824 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:42:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:19.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:42:19.928 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:42:19.957 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-15T23:42:19.974 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:20.009 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:42:20.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:20.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:42:20.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:42:20.163 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:20.205 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:42:20.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:20.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:42:20.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:42:20.362 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:20.381 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:42:20.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:20.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:42:20.489 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:42:20.519 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:20.542 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:42:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:20.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:42:20.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:20.666 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:20.686 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:20.718 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:20.719 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:42:20.719 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:42:20.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:20.725 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:20.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:42:20.869 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:20.896 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:42:20.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:20.908 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:42:21.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:42:21.042 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:21.088 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:42:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:21.095 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:42:21.195 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:42:21.236 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:22.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:22 smithi090 ceph-mon[44434]: pgmap v2123: 17 pgs: 17 active+clean; 598 MiB data, 7.3 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 963 op/s 2024-05-15T23:42:22.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:22 smithi088 ceph-mon[44699]: pgmap v2123: 17 pgs: 17 active+clean; 598 MiB data, 7.3 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 963 op/s 2024-05-15T23:42:22.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:22 smithi084 ceph-mon[39784]: pgmap v2123: 17 pgs: 17 active+clean; 598 MiB data, 7.3 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 963 op/s 2024-05-15T23:42:24.343 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:24.381 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-05-15T23:42:24.601 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:42:24.603 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:24 smithi084 ceph-mon[39784]: pgmap v2124: 17 pgs: 17 active+clean; 378 MiB data, 6.8 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.13k op/s 2024-05-15T23:42:24.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:24.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:42:24.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:42:24.754 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:24.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:24 smithi090 ceph-mon[44434]: pgmap v2124: 17 pgs: 17 active+clean; 378 MiB data, 6.8 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.13k op/s 2024-05-15T23:42:24.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:24 smithi088 ceph-mon[44699]: pgmap v2124: 17 pgs: 17 active+clean; 378 MiB data, 6.8 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.13k op/s 2024-05-15T23:42:24.849 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:42:24.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:24.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:42:24.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:42:24.998 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:25.032 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:42:25.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:25.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:42:25.139 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:42:25.169 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:25.189 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:42:25.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:25.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:42:25.293 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4 2024-05-15T23:42:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:25.342 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4" found: no such container 2024-05-15T23:42:25.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:25.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.4' 2024-05-15T23:42:25.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:42:25.489 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:26.251 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:42:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:26.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:42:26.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:42:26.413 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:26.491 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:42:26.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:26 smithi084 ceph-mon[39784]: pgmap v2125: 17 pgs: 17 active+clean; 252 MiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:42:26.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:26 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:42:26.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:26 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/956719221' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:42:26.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:26 smithi084 ceph-mon[39784]: from='client.37607 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:42:26.804 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:26 smithi090 ceph-mon[44434]: pgmap v2125: 17 pgs: 17 active+clean; 252 MiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:42:26.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:26 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:42:26.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:26 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/956719221' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:42:26.805 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:26 smithi090 ceph-mon[44434]: from='client.37607 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:42:26.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:26 smithi088 ceph-mon[44699]: pgmap v2125: 17 pgs: 17 active+clean; 252 MiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-05-15T23:42:26.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:26 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:42:26.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:26 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/956719221' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:42:26.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:26 smithi088 ceph-mon[44699]: from='client.37607 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:42:26.878 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:42:26.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:26.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:42:26.987 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:42:27.018 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:27.048 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Removed 124185 objects 2024-05-15T23:42:27.048 INFO:tasks.radosbench.radosbench.2.smithi090.stdout:Clean up completed and total clean up time :114.462 2024-05-15T23:42:27.229 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:42:27.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:27.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:42:27.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:27.369 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:27.389 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:27.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:27.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:42:27.520 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:27.573 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:42:27.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:27.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:42:27.656 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:27.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:27.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:27 smithi090 ceph-mon[44434]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:42:27.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:27 smithi090 ceph-mon[44434]: from='client.37607 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:42:27.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:27 smithi090 ceph-mon[44434]: osdmap e793: 12 total, 11 up, 9 in 2024-05-15T23:42:27.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:42:27.702 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:42:27.703 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:27 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:42:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:27.723 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:27.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:27.726 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:27 smithi088 ceph-mon[44699]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:42:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:27 smithi088 ceph-mon[44699]: from='client.37607 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:42:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:27 smithi088 ceph-mon[44699]: osdmap e793: 12 total, 11 up, 9 in 2024-05-15T23:42:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:42:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:42:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:27 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:42:27.827 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:42:27.855 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:27.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:27 smithi084 ceph-mon[39784]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:42:27.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:27 smithi084 ceph-mon[39784]: from='client.37607 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:42:27.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:27 smithi084 ceph-mon[39784]: osdmap e793: 12 total, 11 up, 9 in 2024-05-15T23:42:27.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:42:27.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:42:27.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:27 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' 2024-05-15T23:42:27.875 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:42:27.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:27.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:42:27.979 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7 2024-05-15T23:42:28.008 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:28.027 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7" found: no such container 2024-05-15T23:42:28.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:28.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.7' 2024-05-15T23:42:28.131 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:42:28.161 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:28.284 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:42:28.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:28.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:42:28.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:42:28.416 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:28.434 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:42:28.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:28.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:42:28.539 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:42:28.572 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:28.594 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:42:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:28.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:42:28.634 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:42:28.653 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:42:28.673 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:42:28.692 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:42:28.698 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:42:28.714 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:42:28.724 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:28 smithi084 ceph-mon[39784]: pgmap v2127: 17 pgs: 17 active+clean; 188 MiB data, 6.4 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:42:28.726 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:28.735 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:42:28.756 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:42:28.775 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:42:28.796 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:42:28.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:28 smithi090 ceph-mon[44434]: pgmap v2127: 17 pgs: 17 active+clean; 188 MiB data, 6.4 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:42:28.816 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:42:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:28 smithi088 ceph-mon[44699]: pgmap v2127: 17 pgs: 17 active+clean; 188 MiB data, 6.4 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-05-15T23:42:28.836 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:42:28.836 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:42:28.836 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:42:28.836 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:42:28.836 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:42:28.836 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:42:28.837 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-05-15T23:42:28.837 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:42:28.837 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:42:28.837 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:42:28.837 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:42:28.837 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:42:28.837 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:42:29.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:42:29.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:29 smithi084 ceph-mon[39784]: 9.a scrub starts 2024-05-15T23:42:29.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:29 smithi084 ceph-mon[39784]: 9.a scrub ok 2024-05-15T23:42:29.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:29 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3785577994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:42:29.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:29 smithi084 ceph-mon[39784]: 9.4 scrub starts 2024-05-15T23:42:29.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:29 smithi084 ceph-mon[39784]: 9.4 scrub ok 2024-05-15T23:42:29.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:29 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:29 smithi090 ceph-mon[44434]: 9.a scrub starts 2024-05-15T23:42:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:29 smithi090 ceph-mon[44434]: 9.a scrub ok 2024-05-15T23:42:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:29 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3785577994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:42:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:29 smithi090 ceph-mon[44434]: 9.4 scrub starts 2024-05-15T23:42:29.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:29 smithi090 ceph-mon[44434]: 9.4 scrub ok 2024-05-15T23:42:29.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:29 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:29 smithi088 ceph-mon[44699]: 9.a scrub starts 2024-05-15T23:42:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:29 smithi088 ceph-mon[44699]: 9.a scrub ok 2024-05-15T23:42:29.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:29 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3785577994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:42:29.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:29 smithi088 ceph-mon[44699]: 9.4 scrub starts 2024-05-15T23:42:29.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:29 smithi088 ceph-mon[44699]: 9.4 scrub ok 2024-05-15T23:42:29.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:29 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:42:29.827 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:42:29.898 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-05-15T23:42:29.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-05-15T23:42:30.375 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:42:30.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:30.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:42:30.405 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:30.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:42:30.506 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:30.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:30.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:30.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:30.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:30 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:30.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:30 smithi084 ceph-mon[39784]: pgmap v2128: 17 pgs: 17 active+clean; 44 MiB data, 6.1 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:42:30.654 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:42:30.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:30.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:42:30.757 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:42:30.789 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:30.789 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:30.789 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:30.789 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:30 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:30.789 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:30 smithi088 ceph-mon[44699]: pgmap v2128: 17 pgs: 17 active+clean; 44 MiB data, 6.1 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:42:30.791 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:30.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:30 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:30 smithi090 ceph-mon[44434]: pgmap v2128: 17 pgs: 17 active+clean; 44 MiB data, 6.1 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s 2024-05-15T23:42:30.812 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:42:30.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:30.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:42:30.894 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-05-15T23:42:30.894 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-05-15T23:42:30.895 DEBUG:teuthology.orchestra.run.smithi088:osd.5> sudo journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.5.service 2024-05-15T23:42:30.897 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl start ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.5 2024-05-15T23:42:30.916 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:42:30.927 INFO:journalctl@ceph.osd.5.smithi088.stdout:-- Logs begin at Wed 2024-05-15 22:43:48 UTC. -- 2024-05-15T23:42:30.945 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:31.019 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:42:31.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:31.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:42:31.070 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:42:30 smithi088 systemd[1]: Starting Ceph osd.5 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:42:31.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:31.155 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:31.179 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:31.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:31.183 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:31.284 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:42:31.313 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:31.511 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:42:31.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:31.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:42:31.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:31 smithi084 ceph-mon[39784]: 9.d scrub starts 2024-05-15T23:42:31.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:31 smithi084 ceph-mon[39784]: 9.d scrub ok 2024-05-15T23:42:31.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:31 smithi084 ceph-mon[39784]: 9.9 scrub starts 2024-05-15T23:42:31.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:31 smithi084 ceph-mon[39784]: 9.9 scrub ok 2024-05-15T23:42:31.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:31 smithi088 ceph-mon[44699]: 9.d scrub starts 2024-05-15T23:42:31.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:31 smithi088 ceph-mon[44699]: 9.d scrub ok 2024-05-15T23:42:31.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:31 smithi088 ceph-mon[44699]: 9.9 scrub starts 2024-05-15T23:42:31.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:31 smithi088 ceph-mon[44699]: 9.9 scrub ok 2024-05-15T23:42:31.614 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:31.645 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:31.666 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:31.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:31.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:31.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9 2024-05-15T23:42:31.796 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:31 smithi090 ceph-mon[44434]: 9.d scrub starts 2024-05-15T23:42:31.796 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:31 smithi090 ceph-mon[44434]: 9.d scrub ok 2024-05-15T23:42:31.796 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:31 smithi090 ceph-mon[44434]: 9.9 scrub starts 2024-05-15T23:42:31.796 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:31 smithi090 ceph-mon[44434]: 9.9 scrub ok 2024-05-15T23:42:31.798 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:31.819 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9" found: no such container 2024-05-15T23:42:31.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:31.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.9' 2024-05-15T23:42:31.827 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-15T23:42:31.923 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:42:31.948 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:32.028 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-05-15T23:42:32.028 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-05-15T23:42:32.028 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-05-15T23:42:32.029 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-05-15T23:42:32.029 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-05-15T23:42:32.519 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:42:32 smithi088 bash[174409]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-05-15T23:42:32.519 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:42:32 smithi088 bash[174409]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-15T23:42:32.519 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:42:32 smithi088 bash[174409]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-05-15T23:42:32.520 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:42:32 smithi088 bash[174409]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-05-15T23:42:32.520 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:42:32 smithi088 bash[174409]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-05-15T23:42:32.520 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:42:32 smithi088 bash[174409]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-05-15T23:42:32.520 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:42:32 smithi088 bash[174409]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-15T23:42:32.520 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:42:32 smithi088 bash[174409]: --> ceph-volume raw activate successful for osd ID: 5 2024-05-15T23:42:32.604 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:42:32.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:32 smithi084 ceph-mon[39784]: pgmap v2129: 17 pgs: 17 active+clean; 4.5 MiB data, 6.0 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 949 op/s 2024-05-15T23:42:32.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:32 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:42:32.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:32 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:42:32.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:32.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:42:32.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-05-15T23:42:32.694 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:32.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:32.743 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:32.765 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:32.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:32.769 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:32 smithi090 ceph-mon[44434]: pgmap v2129: 17 pgs: 17 active+clean; 4.5 MiB data, 6.0 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 949 op/s 2024-05-15T23:42:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:32 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:42:32.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:32 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:42:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:32 smithi088 ceph-mon[44699]: pgmap v2129: 17 pgs: 17 active+clean; 4.5 MiB data, 6.0 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 949 op/s 2024-05-15T23:42:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:32 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:42:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:32 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:42:32.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1 2024-05-15T23:42:32.898 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:33.147 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1" found: no such container 2024-05-15T23:42:33.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:33.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.1' 2024-05-15T23:42:33.251 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:42:33.283 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:33.306 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:42:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:33.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:42:33.327 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T23:42:33.354 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:42:33 smithi088 bash[174661]: 55c7cd40ef1e7dd1c715f75f1ae173c1b8cb20c1065e5829ce7d7d51c0b29c15 2024-05-15T23:42:33.354 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:42:33 smithi088 systemd[1]: Started Ceph osd.5 for 3999fb7e-130e-11ef-bc99-c7b262605968. 2024-05-15T23:42:33.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:33.439 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:33.460 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:33.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:33.463 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:33.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:42:33.593 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:33.594 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:42:33.613 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:42:33.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:33.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:42:33.717 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:42:33.745 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:33.765 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:42:33.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:33.768 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:42:33.772 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:33.868 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:42:33.895 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:33.915 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:42:33.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:33.919 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:42:34.020 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2 2024-05-15T23:42:34.050 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:34.070 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:42:33 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:42:33.756+0000 7f1554d2b7c0 -1 Falling back to public interface 2024-05-15T23:42:34.537 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:42:34.752 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:34 smithi088 ceph-mon[44699]: pgmap v2130: 17 pgs: 17 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 604 op/s 2024-05-15T23:42:34.752 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:34 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1226329795' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:42:34.801 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:34 smithi090 ceph-mon[44434]: pgmap v2130: 17 pgs: 17 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 604 op/s 2024-05-15T23:42:34.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:34 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1226329795' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:42:34.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:34 smithi084 ceph-mon[39784]: pgmap v2130: 17 pgs: 17 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 604 op/s 2024-05-15T23:42:34.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:34 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1226329795' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:42:35.133 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2" found: no such container 2024-05-15T23:42:35.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:35.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.2' 2024-05-15T23:42:35.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:35.279 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:35.301 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:35.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:35.305 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:35.348 INFO:teuthology.orchestra.run.smithi088.stderr:no valid command found; 10 closest matches: 2024-05-15T23:42:35.348 INFO:teuthology.orchestra.run.smithi088.stderr:0 2024-05-15T23:42:35.348 INFO:teuthology.orchestra.run.smithi088.stderr:1 2024-05-15T23:42:35.348 INFO:teuthology.orchestra.run.smithi088.stderr:2 2024-05-15T23:42:35.348 INFO:teuthology.orchestra.run.smithi088.stderr:abort 2024-05-15T23:42:35.349 INFO:teuthology.orchestra.run.smithi088.stderr:assert 2024-05-15T23:42:35.349 INFO:teuthology.orchestra.run.smithi088.stderr:config diff 2024-05-15T23:42:35.349 INFO:teuthology.orchestra.run.smithi088.stderr:config diff get 2024-05-15T23:42:35.349 INFO:teuthology.orchestra.run.smithi088.stderr:config get 2024-05-15T23:42:35.349 INFO:teuthology.orchestra.run.smithi088.stderr:config help [] 2024-05-15T23:42:35.349 INFO:teuthology.orchestra.run.smithi088.stderr:config set ... 2024-05-15T23:42:35.349 INFO:teuthology.orchestra.run.smithi088.stderr:admin_socket: invalid command 2024-05-15T23:42:35.406 DEBUG:teuthology.orchestra.run.smithi088:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6 2024-05-15T23:42:35.437 INFO:teuthology.orchestra.run.smithi088.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:35.661 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:35 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:42:35.661 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:35 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:42:35.661 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:35 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1226329795' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:42:35.661 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:35 smithi088 ceph-mon[44699]: osdmap e794: 12 total, 11 up, 9 in 2024-05-15T23:42:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6" found: no such container 2024-05-15T23:42:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:35.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi088 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.6' 2024-05-15T23:42:35.770 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:42:35.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:35 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:42:35.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:35 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:42:35.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:35 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1226329795' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:42:35.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:35 smithi084 ceph-mon[39784]: osdmap e794: 12 total, 11 up, 9 in 2024-05-15T23:42:35.799 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:35 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-05-15T23:42:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:35 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:42:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:35 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1226329795' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:42:35.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:35 smithi090 ceph-mon[44434]: osdmap e794: 12 total, 11 up, 9 in 2024-05-15T23:42:35.898 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:42:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:35.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:42:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-15T23:42:35.964 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-05-15T23:42:36.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:42:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:42:36.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:36.058 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:42:36.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:42:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:36.209 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:42:36.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:36.212 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:42:36.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:42:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:36.362 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:42:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:36.365 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:42:36.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3 2024-05-15T23:42:36.495 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:36.544 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_7' removed 2024-05-15T23:42:36.568 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3" found: no such container 2024-05-15T23:42:36.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:36.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.3' 2024-05-15T23:42:36.572 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:36.672 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10 2024-05-15T23:42:36.698 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:36 smithi090 ceph-mon[44434]: pgmap v2132: 17 pgs: 17 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 451 op/s 2024-05-15T23:42:36.698 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2855792424' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:42:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:36.721 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10" found: no such container 2024-05-15T23:42:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:36.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.10' 2024-05-15T23:42:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:36 smithi088 ceph-mon[44699]: pgmap v2132: 17 pgs: 17 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 451 op/s 2024-05-15T23:42:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2855792424' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:42:36.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11 2024-05-15T23:42:36.853 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:36.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:36 smithi084 ceph-mon[39784]: pgmap v2132: 17 pgs: 17 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 451 op/s 2024-05-15T23:42:36.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2855792424' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-15T23:42:36.873 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11" found: no such container 2024-05-15T23:42:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:36.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.11' 2024-05-15T23:42:36.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8 2024-05-15T23:42:37.004 INFO:teuthology.orchestra.run.smithi090.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:37.023 INFO:teuthology.orchestra.run.smithi090.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8" found: no such container 2024-05-15T23:42:37.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:37.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi090 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.8' 2024-05-15T23:42:37.128 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0 2024-05-15T23:42:37.175 INFO:teuthology.orchestra.run.smithi084.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-05-15T23:42:37.294 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd set noscrub 2024-05-15T23:42:37.320 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:42:36 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:42:36.838+0000 7f1554d2b7c0 -1 osd.5 749 log_to_monitors true 2024-05-15T23:42:37.423 INFO:teuthology.orchestra.run.smithi084.stderr:Error: no container with name or ID "ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0" found: no such container 2024-05-15T23:42:37.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-05-15T23:42:37.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 125: 'sudo docker kill -s 1 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd.0' 2024-05-15T23:42:37.436 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-05-15T23:42:37.449 INFO:tasks.thrashosds:joining thrashosds 2024-05-15T23:42:37.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:37 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2855792424' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:42:37.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:37 smithi084 ceph-mon[39784]: osdmap e795: 12 total, 11 up, 9 in 2024-05-15T23:42:37.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:37 smithi084 ceph-mon[39784]: from='osd.5 v1:172.21.15.88:6804/2238397473' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T23:42:37.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:37 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2855792424' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:42:37.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:37 smithi090 ceph-mon[44434]: osdmap e795: 12 total, 11 up, 9 in 2024-05-15T23:42:37.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:37 smithi090 ceph-mon[44434]: from='osd.5 v1:172.21.15.88:6804/2238397473' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T23:42:37.820 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:42:37 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:42:37.564+0000 7f1546182700 -1 osd.5 796 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T23:42:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:37 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2855792424' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-05-15T23:42:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:37 smithi088 ceph-mon[44699]: osdmap e795: 12 total, 11 up, 9 in 2024-05-15T23:42:37.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:37 smithi088 ceph-mon[44699]: from='osd.5 v1:172.21.15.88:6804/2238397473' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T23:42:38.461 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-05-15T23:42:38.462 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-05-15T23:42:38.462 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-05-15T23:42:38.462 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-05-15T23:42:38.482 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-05-15T23:42:38.502 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-05-15T23:42:38.523 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-05-15T23:42:38.543 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-05-15T23:42:38.564 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-05-15T23:42:38.584 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-05-15T23:42:38.605 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-05-15T23:42:38.605 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-15T23:42:38.606 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-15T23:42:38.606 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-15T23:42:38.606 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-15T23:42:38.606 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-15T23:42:38.606 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-05-15T23:42:38.606 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-15T23:42:38.606 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-15T23:42:38.606 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-05-15T23:42:38.606 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-05-15T23:42:38.606 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-05-15T23:42:38.606 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-05-15T23:42:38.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:38 smithi084 ceph-mon[39784]: pgmap v2134: 1 pgs: 1 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:38.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:38 smithi084 ceph-mon[39784]: from='osd.5 v1:172.21.15.88:6804/2238397473' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-15T23:42:38.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:38 smithi084 ceph-mon[39784]: osdmap e796: 12 total, 11 up, 9 in 2024-05-15T23:42:38.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:38 smithi084 ceph-mon[39784]: from='osd.5 v1:172.21.15.88:6804/2238397473' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:42:38.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:38 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:42:38.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:38 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:42:38.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:38 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:42:38.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:38 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1786744935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:42:38.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:38 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:38 smithi090 ceph-mon[44434]: pgmap v2134: 1 pgs: 1 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:38 smithi090 ceph-mon[44434]: from='osd.5 v1:172.21.15.88:6804/2238397473' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-15T23:42:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:38 smithi090 ceph-mon[44434]: osdmap e796: 12 total, 11 up, 9 in 2024-05-15T23:42:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:38 smithi090 ceph-mon[44434]: from='osd.5 v1:172.21.15.88:6804/2238397473' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:42:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:38 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:42:38.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:38 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:42:38.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:38 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:42:38.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:38 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1786744935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:42:38.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:38 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:38.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:38 smithi088 ceph-mon[44699]: pgmap v2134: 1 pgs: 1 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:38.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:38 smithi088 ceph-mon[44699]: from='osd.5 v1:172.21.15.88:6804/2238397473' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-15T23:42:38.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:38 smithi088 ceph-mon[44699]: osdmap e796: 12 total, 11 up, 9 in 2024-05-15T23:42:38.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:38 smithi088 ceph-mon[44699]: from='osd.5 v1:172.21.15.88:6804/2238397473' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-05-15T23:42:38.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:38 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:42:38.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:38 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:42:38.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:38 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:42:38.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:38 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1786744935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-05-15T23:42:38.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:38 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.659 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-05-15T23:42:39.660 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-05-15T23:42:39.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:39 smithi088 ceph-mon[44699]: osd.5 v1:172.21.15.88:6804/2238397473 boot 2024-05-15T23:42:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:39 smithi088 ceph-mon[44699]: osdmap e797: 12 total, 12 up, 9 in 2024-05-15T23:42:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:39 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:42:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:39 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:39 smithi084 ceph-mon[39784]: osd.5 v1:172.21.15.88:6804/2238397473 boot 2024-05-15T23:42:39.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:39 smithi084 ceph-mon[39784]: osdmap e797: 12 total, 12 up, 9 in 2024-05-15T23:42:39.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:39 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:42:39.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:39.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:39 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:40.021 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:39 smithi090 ceph-mon[44434]: osd.5 v1:172.21.15.88:6804/2238397473 boot 2024-05-15T23:42:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:39 smithi090 ceph-mon[44434]: osdmap e797: 12 total, 12 up, 9 in 2024-05-15T23:42:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:39 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T23:42:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:39 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:40.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:40 smithi088 ceph-mon[44699]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:40 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:42:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:40 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:42:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:40 smithi088 ceph-mon[44699]: pgmap v2137: 1 pgs: 1 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:40.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:40 smithi088 ceph-mon[44699]: osdmap e798: 12 total, 12 up, 9 in 2024-05-15T23:42:40.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:40 smithi084 ceph-mon[39784]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:40.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:40 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:42:40.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:40 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:42:40.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:40 smithi084 ceph-mon[39784]: pgmap v2137: 1 pgs: 1 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:40.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:40 smithi084 ceph-mon[39784]: osdmap e798: 12 total, 12 up, 9 in 2024-05-15T23:42:40.965 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T23:42:41.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:40 smithi090 ceph-mon[44434]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-05-15T23:42:41.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:40 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:42:41.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:40 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:42:41.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:40 smithi090 ceph-mon[44434]: pgmap v2137: 1 pgs: 1 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:41.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:40 smithi090 ceph-mon[44434]: osdmap e798: 12 total, 12 up, 9 in 2024-05-15T23:42:41.217 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:42:41.217 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:42:41.251 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:42:42.789 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:42 smithi088 ceph-mon[44699]: pgmap v2139: 1 pgs: 1 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:42.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:42 smithi084 ceph-mon[39784]: pgmap v2139: 1 pgs: 1 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:42.938 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-05-15T23:42:43.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:42 smithi090 ceph-mon[44434]: pgmap v2139: 1 pgs: 1 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:43.089 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:42:43.089 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:42:43.089 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2024-05-15T23:42:43.089 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:42:43.570 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-05-15T23:42:43.773 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:43 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/382459228' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:42:43.805 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:42:43.806 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-15T23:42:43.806 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [1, 3, 4, 7, 11, 9, 10, 2, 8] 2024-05-15T23:42:43.807 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd out 7 2024-05-15T23:42:43.840 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:43 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/382459228' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:42:44.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:43 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/382459228' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-05-15T23:42:44.322 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:44.805 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:44 smithi084 ceph-mon[39784]: pgmap v2140: 1 pgs: 1 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:44.805 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:44 smithi084 ceph-mon[39784]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:42:44.805 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/382459228' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:42:44.805 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:44 smithi084 ceph-mon[39784]: osdmap e799: 12 total, 12 up, 9 in 2024-05-15T23:42:44.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:44 smithi088 ceph-mon[44699]: pgmap v2140: 1 pgs: 1 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:44 smithi088 ceph-mon[44699]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:42:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/382459228' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:42:44.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:44 smithi088 ceph-mon[44699]: osdmap e799: 12 total, 12 up, 9 in 2024-05-15T23:42:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:44 smithi090 ceph-mon[44434]: pgmap v2140: 1 pgs: 1 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:44 smithi090 ceph-mon[44434]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-05-15T23:42:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/382459228' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-05-15T23:42:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:44 smithi090 ceph-mon[44434]: osdmap e799: 12 total, 12 up, 9 in 2024-05-15T23:42:45.493 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:42:45.493 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:42:46.221 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-05-15T23:42:46.263 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset noscrub 2024-05-15T23:42:46.483 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:42:46.530 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:46.571 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.7. 2024-05-15T23:42:46.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:46 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:42:46.410+0000 7fa0938b7700 -1 mon.a@0(leader).osd e799 definitely_dead 0 2024-05-15T23:42:46.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:46 smithi084 ceph-mon[39784]: pgmap v2142: 1 pgs: 1 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:46.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:46 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/702553591' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-05-15T23:42:46.839 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:46 smithi084 ceph-mon[39784]: Client client.admin marked osd.7 out, while it was still marked up 2024-05-15T23:42:46.944 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:46 smithi088 ceph-mon[44699]: pgmap v2142: 1 pgs: 1 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:46.945 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:46 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/702553591' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-05-15T23:42:46.945 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:46 smithi088 ceph-mon[44699]: Client client.admin marked osd.7 out, while it was still marked up 2024-05-15T23:42:47.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:46 smithi090 ceph-mon[44434]: pgmap v2142: 1 pgs: 1 active+clean; 577 KiB data, 6.0 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:47.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:46 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/702553591' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-05-15T23:42:47.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:46 smithi090 ceph-mon[44434]: Client client.admin marked osd.7 out, while it was still marked up 2024-05-15T23:42:47.781 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:47 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/702553591' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-05-15T23:42:47.781 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:47 smithi084 ceph-mon[39784]: osdmap e800: 12 total, 12 up, 8 in 2024-05-15T23:42:47.781 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:47 smithi084 ceph-mon[39784]: osdmap e801: 12 total, 12 up, 8 in 2024-05-15T23:42:47.874 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:47 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/702553591' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-05-15T23:42:47.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:47 smithi088 ceph-mon[44699]: osdmap e800: 12 total, 12 up, 8 in 2024-05-15T23:42:47.875 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:47 smithi088 ceph-mon[44699]: osdmap e801: 12 total, 12 up, 8 in 2024-05-15T23:42:48.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:47 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/702553591' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-05-15T23:42:48.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:47 smithi090 ceph-mon[44434]: osdmap e800: 12 total, 12 up, 8 in 2024-05-15T23:42:48.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:47 smithi090 ceph-mon[44434]: osdmap e801: 12 total, 12 up, 8 in 2024-05-15T23:42:48.166 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:42:48.166 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:42:48.735 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-05-15T23:42:48.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:48 smithi084 ceph-mon[39784]: pgmap v2144: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-05-15T23:42:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:42:49.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:48 smithi090 ceph-mon[44434]: pgmap v2144: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-05-15T23:42:49.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:48 smithi088 ceph-mon[44699]: pgmap v2144: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-05-15T23:42:49.597 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-05-15T23:42:49.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:49 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3028572653' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:42:49.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:49 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3028572653' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:42:50.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:49 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3028572653' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-05-15T23:42:50.340 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd unset nodeep-scrub 2024-05-15T23:42:50.612 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:50.690 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:42:50.690 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:42:50.710 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:50 smithi088 ceph-mon[44699]: pgmap v2146: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-05-15T23:42:50.710 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:50 smithi088 ceph-mon[44699]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-05-15T23:42:50.710 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:50 smithi088 ceph-mon[44699]: Cluster is now healthy 2024-05-15T23:42:50.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3028572653' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:42:50.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:50 smithi088 ceph-mon[44699]: osdmap e802: 12 total, 12 up, 8 in 2024-05-15T23:42:50.846 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:50 smithi084 ceph-mon[39784]: pgmap v2146: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-05-15T23:42:50.846 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:50 smithi084 ceph-mon[39784]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-05-15T23:42:50.846 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:50 smithi084 ceph-mon[39784]: Cluster is now healthy 2024-05-15T23:42:50.846 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3028572653' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:42:50.846 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:50 smithi084 ceph-mon[39784]: osdmap e802: 12 total, 12 up, 8 in 2024-05-15T23:42:51.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:50 smithi090 ceph-mon[44434]: pgmap v2146: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-05-15T23:42:51.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:50 smithi090 ceph-mon[44434]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-05-15T23:42:51.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:50 smithi090 ceph-mon[44434]: Cluster is now healthy 2024-05-15T23:42:51.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3028572653' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-05-15T23:42:51.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:50 smithi090 ceph-mon[44434]: osdmap e802: 12 total, 12 up, 8 in 2024-05-15T23:42:51.789 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:51 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:42:51.789 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:51 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:42:51.981 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:51 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:42:51.982 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:51 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:42:52.013 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-05-15T23:42:52.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:51 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:42:52.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:51 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:42:52.283 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:42:52.805 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:52 smithi088 ceph-mon[44699]: pgmap v2148: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-05-15T23:42:52.805 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:52 smithi088 ceph-mon[44699]: osdmap e803: 12 total, 12 up, 8 in 2024-05-15T23:42:52.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:52 smithi084 ceph-mon[39784]: pgmap v2148: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-05-15T23:42:52.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:52 smithi084 ceph-mon[39784]: osdmap e803: 12 total, 12 up, 8 in 2024-05-15T23:42:53.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:52 smithi090 ceph-mon[44434]: pgmap v2148: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-05-15T23:42:53.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:52 smithi090 ceph-mon[44434]: osdmap e803: 12 total, 12 up, 8 in 2024-05-15T23:42:53.344 INFO:tasks.thrashosds.thrasher:inning osd 2024-05-15T23:42:53.344 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-05-15T23:42:53.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd in 0 2024-05-15T23:42:53.610 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-05-15T23:42:53.624 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:53.640 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:53 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/910974204' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:42:53.786 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:53 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/910974204' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:42:54.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:53 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/910974204' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-05-15T23:42:54.072 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:42:54.073 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:42:54.744 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:54 smithi088 ceph-mon[44699]: pgmap v2150: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-05-15T23:42:54.744 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/910974204' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:42:54.744 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:54 smithi088 ceph-mon[44699]: osdmap e804: 12 total, 12 up, 8 in 2024-05-15T23:42:54.771 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-05-15T23:42:54.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:54 smithi084 ceph-mon[39784]: pgmap v2150: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-05-15T23:42:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/910974204' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:42:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:54 smithi084 ceph-mon[39784]: osdmap e804: 12 total, 12 up, 8 in 2024-05-15T23:42:55.034 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:42:55.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:54 smithi090 ceph-mon[44434]: pgmap v2150: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-05-15T23:42:55.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/910974204' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-05-15T23:42:55.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:54 smithi090 ceph-mon[44434]: osdmap e804: 12 total, 12 up, 8 in 2024-05-15T23:42:56.614 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.0. 2024-05-15T23:42:56.860 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:42:56.860 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:42:56.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:56 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:42:56.455+0000 7fa0938b7700 -1 mon.a@0(leader).osd e804 definitely_dead 0 2024-05-15T23:42:56.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:56 smithi084 ceph-mon[39784]: pgmap v2152: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-05-15T23:42:56.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:56 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/966836223' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-05-15T23:42:56.886 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:56 smithi088 ceph-mon[44699]: pgmap v2152: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-05-15T23:42:56.886 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:56 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/966836223' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-05-15T23:42:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:56 smithi090 ceph-mon[44434]: pgmap v2152: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-05-15T23:42:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:56 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/966836223' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-05-15T23:42:57.286 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-05-15T23:42:57.286 INFO:tasks.thrashosds.thrasher:inning osd 2024-05-15T23:42:57.286 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-05-15T23:42:57.286 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd in 5 2024-05-15T23:42:57.520 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-05-15T23:42:57.554 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:42:57.786 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:42:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/966836223' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-05-15T23:42:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:57 smithi088 ceph-mon[44699]: osdmap e805: 12 total, 12 up, 9 in 2024-05-15T23:42:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/966836223' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-05-15T23:42:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:57 smithi090 ceph-mon[44434]: osdmap e805: 12 total, 12 up, 9 in 2024-05-15T23:42:58.109 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/966836223' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-05-15T23:42:58.109 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:57 smithi084 ceph-mon[39784]: osdmap e805: 12 total, 12 up, 9 in 2024-05-15T23:42:58.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:58 smithi084 ceph-mon[39784]: pgmap v2154: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:58 smithi090 ceph-mon[44434]: pgmap v2154: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:59.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:58 smithi088 ceph-mon[44699]: pgmap v2154: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:42:59.355 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:42:59.356 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:42:59.633 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.5. 2024-05-15T23:42:59.667 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:59 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:42:59.667 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:59 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:42:59.667 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:42:59 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/463612551' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-05-15T23:42:59.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:59 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:42:59.581+0000 7fa0938b7700 -1 mon.a@0(leader).osd e805 definitely_dead 0 2024-05-15T23:42:59.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:59 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:42:59.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:59 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:42:59.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:42:59 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/463612551' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-05-15T23:42:59.982 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-05-15T23:43:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:59 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:43:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:59 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:43:00.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:42:59 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/463612551' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-05-15T23:43:00.246 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:43:00.392 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-05-15T23:43:00.392 INFO:tasks.thrashosds.thrasher:inning osd 2024-05-15T23:43:00.393 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-05-15T23:43:00.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd in 6 2024-05-15T23:43:00.657 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:00.732 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:00 smithi088 ceph-mon[44699]: pgmap v2155: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:43:00.733 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:00 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/463612551' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-05-15T23:43:00.733 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:00 smithi088 ceph-mon[44699]: osdmap e806: 12 total, 12 up, 10 in 2024-05-15T23:43:00.935 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:00 smithi084 ceph-mon[39784]: pgmap v2155: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:43:00.935 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:00 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/463612551' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-05-15T23:43:00.935 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:00 smithi084 ceph-mon[39784]: osdmap e806: 12 total, 12 up, 10 in 2024-05-15T23:43:01.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:00 smithi090 ceph-mon[44434]: pgmap v2155: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:43:01.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:00 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/463612551' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-05-15T23:43:01.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:00 smithi090 ceph-mon[44434]: osdmap e806: 12 total, 12 up, 10 in 2024-05-15T23:43:01.989 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:01.989 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:02.719 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:02 smithi088 ceph-mon[44699]: pgmap v2157: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:43:02.745 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-05-15T23:43:02.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:02 smithi084 ceph-mon[39784]: pgmap v2157: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:43:03.008 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:43:03.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:02 smithi090 ceph-mon[44434]: pgmap v2157: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:43:03.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:02 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:43:02.882+0000 7fa0938b7700 -1 mon.a@0(leader).osd e806 definitely_dead 0 2024-05-15T23:43:03.643 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.6. 2024-05-15T23:43:04.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:03 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3199835881' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-05-15T23:43:04.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:03 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3199835881' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-05-15T23:43:04.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:03 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3199835881' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-05-15T23:43:04.434 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-05-15T23:43:04.435 INFO:tasks.thrashosds.thrasher:inning osd 2024-05-15T23:43:04.435 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-05-15T23:43:04.435 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd in 7 2024-05-15T23:43:04.655 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:04.655 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:04.706 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:04.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:04 smithi084 ceph-mon[39784]: pgmap v2158: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:43:04.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3199835881' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-05-15T23:43:04.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:04 smithi084 ceph-mon[39784]: osdmap e807: 12 total, 12 up, 11 in 2024-05-15T23:43:04.934 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:04 smithi088 ceph-mon[44699]: pgmap v2158: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:43:04.934 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3199835881' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-05-15T23:43:04.935 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:04 smithi088 ceph-mon[44699]: osdmap e807: 12 total, 12 up, 11 in 2024-05-15T23:43:05.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:04 smithi090 ceph-mon[44434]: pgmap v2158: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 799 GiB / 805 GiB avail 2024-05-15T23:43:05.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3199835881' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-05-15T23:43:05.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:04 smithi090 ceph-mon[44434]: osdmap e807: 12 total, 12 up, 11 in 2024-05-15T23:43:05.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-05-15T23:43:05.520 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:43:05.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:05 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:43:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:05 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:43:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:05 smithi084 ceph-mon[39784]: osdmap e808: 12 total, 12 up, 11 in 2024-05-15T23:43:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:05 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:43:05.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:05 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:43:05.979 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:05 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:43:05.979 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:05 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:43:05.979 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:05 smithi088 ceph-mon[44699]: osdmap e808: 12 total, 12 up, 11 in 2024-05-15T23:43:05.979 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:05 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:43:05.979 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:05 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:43:06.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:05 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:43:06.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:05 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:43:06.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:05 smithi090 ceph-mon[44434]: osdmap e808: 12 total, 12 up, 11 in 2024-05-15T23:43:06.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:05 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:43:06.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:05 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:43:06.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:06 smithi084 ceph-mon[39784]: pgmap v2161: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 888 GiB / 894 GiB avail 2024-05-15T23:43:06.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:06 smithi084 ceph-mon[39784]: osdmap e809: 12 total, 12 up, 11 in 2024-05-15T23:43:06.916 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:06 smithi088 ceph-mon[44699]: pgmap v2161: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 888 GiB / 894 GiB avail 2024-05-15T23:43:06.916 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:06 smithi088 ceph-mon[44699]: osdmap e809: 12 total, 12 up, 11 in 2024-05-15T23:43:07.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:06 smithi090 ceph-mon[44434]: pgmap v2161: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 888 GiB / 894 GiB avail 2024-05-15T23:43:07.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:06 smithi090 ceph-mon[44434]: osdmap e809: 12 total, 12 up, 11 in 2024-05-15T23:43:07.165 INFO:teuthology.orchestra.run.smithi088.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:07.165 INFO:teuthology.orchestra.run.smithi088.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:07.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:07 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:43:07.023+0000 7fa0938b7700 -1 mon.a@0(leader).osd e809 definitely_dead 0 2024-05-15T23:43:07.665 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.7. 2024-05-15T23:43:07.745 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:07 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3735133834' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-05-15T23:43:07.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-05-15T23:43:07.970 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:07 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3735133834' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-05-15T23:43:08.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:07 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3735133834' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-05-15T23:43:08.054 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:43:08.351 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-05-15T23:43:08.351 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-05-15T23:43:08.621 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:08.926 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:08 smithi084 ceph-mon[39784]: pgmap v2163: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 888 GiB / 894 GiB avail 2024-05-15T23:43:08.927 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3735133834' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-05-15T23:43:08.927 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:08 smithi084 ceph-mon[39784]: osdmap e810: 12 total, 12 up, 12 in 2024-05-15T23:43:09.027 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:08 smithi090 ceph-mon[44434]: pgmap v2163: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 888 GiB / 894 GiB avail 2024-05-15T23:43:09.027 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3735133834' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-05-15T23:43:09.027 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:08 smithi090 ceph-mon[44434]: osdmap e810: 12 total, 12 up, 12 in 2024-05-15T23:43:09.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:08 smithi088 ceph-mon[44699]: pgmap v2163: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 888 GiB / 894 GiB avail 2024-05-15T23:43:09.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3735133834' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-05-15T23:43:09.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:08 smithi088 ceph-mon[44699]: osdmap e810: 12 total, 12 up, 12 in 2024-05-15T23:43:10.160 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:10.160 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:10.322 INFO:teuthology.orchestra.run.smithi084.stdout:mon.a: {} 2024-05-15T23:43:10.332 INFO:teuthology.orchestra.run.smithi084.stdout:mon.c: {} 2024-05-15T23:43:10.343 INFO:teuthology.orchestra.run.smithi084.stdout:mon.b: {} 2024-05-15T23:43:10.344 INFO:teuthology.orchestra.run.smithi084.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-15T23:43:10.344 INFO:teuthology.orchestra.run.smithi084.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-15T23:43:10.344 INFO:teuthology.orchestra.run.smithi084.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-15T23:43:10.776 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:10 smithi090 ceph-mon[44434]: pgmap v2165: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 977 GiB / 983 GiB avail 2024-05-15T23:43:10.776 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:10 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2896117300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T23:43:10.803 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-05-15T23:43:11.025 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:10 smithi084 ceph-mon[39784]: pgmap v2165: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 977 GiB / 983 GiB avail 2024-05-15T23:43:11.025 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:10 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2896117300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T23:43:11.051 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 1 1 2024-05-15T23:43:11.067 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:43:11.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:10 smithi088 ceph-mon[44699]: pgmap v2165: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 977 GiB / 983 GiB avail 2024-05-15T23:43:11.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:10 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2896117300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T23:43:11.313 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:12.855 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:12.855 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:12.878 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:12 smithi090 ceph-mon[44434]: pgmap v2166: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:12.878 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:12 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:43:12.878 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:12 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:43:12.941 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:12 smithi084 ceph-mon[39784]: pgmap v2166: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:12.941 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:12 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:43:12.941 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:12 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:43:13.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:12 smithi088 ceph-mon[44699]: pgmap v2166: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:13.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:12 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:43:13.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:12 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:43:13.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-05-15T23:43:13.683 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2024-05-15T23:43:13.876 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:43:14.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:13 smithi090 ceph-mon[44434]: osdmap e811: 12 total, 12 up, 12 in 2024-05-15T23:43:14.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:13 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/708518174' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:43:14.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:13 smithi088 ceph-mon[44699]: osdmap e811: 12 total, 12 up, 12 in 2024-05-15T23:43:14.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:13 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/708518174' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:43:14.081 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:13 smithi084 ceph-mon[39784]: osdmap e811: 12 total, 12 up, 12 in 2024-05-15T23:43:14.081 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:13 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/708518174' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:43:14.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 1 1 2024-05-15T23:43:14.737 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:14 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/708518174' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:43:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:14 smithi084 ceph-mon[39784]: osdmap e812: 12 total, 12 up, 12 in 2024-05-15T23:43:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:14 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:43:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:14 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:43:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:14 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:43:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:14 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:43:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:14 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:43:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:14 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:43:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:14 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:43:15.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:14 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/708518174' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:43:15.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:14 smithi090 ceph-mon[44434]: osdmap e812: 12 total, 12 up, 12 in 2024-05-15T23:43:15.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:14 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:43:15.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:14 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:43:15.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:14 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:43:15.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:14 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:43:15.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:14 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:43:15.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:14 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:43:15.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:14 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:43:15.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:14 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/708518174' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:43:15.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:14 smithi088 ceph-mon[44699]: osdmap e812: 12 total, 12 up, 12 in 2024-05-15T23:43:15.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:14 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T23:43:15.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:14 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T23:43:15.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:14 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T23:43:15.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:14 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:43:15.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:14 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:43:15.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:14 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:43:15.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:14 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:43:15.671 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:15.671 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:16.013 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:15 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:43:16.013 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:15 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:43:16.044 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:15 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:43:16.044 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:15 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:43:16.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:15 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:43:16.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:15 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:43:16.305 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-05-15T23:43:16.572 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:43:17.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:16 smithi090 ceph-mon[44434]: pgmap v2170: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:17.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:16 smithi090 ceph-mon[44434]: osdmap e813: 12 total, 12 up, 12 in 2024-05-15T23:43:17.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:16 smithi088 ceph-mon[44699]: pgmap v2170: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:17.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:16 smithi088 ceph-mon[44699]: osdmap e813: 12 total, 12 up, 12 in 2024-05-15T23:43:17.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:16 smithi084 ceph-mon[39784]: pgmap v2170: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:17.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:16 smithi084 ceph-mon[39784]: osdmap e813: 12 total, 12 up, 12 in 2024-05-15T23:43:17.702 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-05-15T23:43:17.938 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:17 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/925703980' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:43:18.070 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:17 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/925703980' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:43:18.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:17 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/925703980' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-05-15T23:43:18.484 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 3 1 2024-05-15T23:43:18.533 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:18.533 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:18.755 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:18.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:18 smithi084 ceph-mon[39784]: pgmap v2172: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:18.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:18 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/925703980' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:43:18.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:18 smithi084 ceph-mon[39784]: osdmap e814: 12 total, 12 up, 12 in 2024-05-15T23:43:18.890 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:18 smithi090 ceph-mon[44434]: pgmap v2172: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:18.890 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:18 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/925703980' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:43:18.890 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:18 smithi090 ceph-mon[44434]: osdmap e814: 12 total, 12 up, 12 in 2024-05-15T23:43:19.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:18 smithi088 ceph-mon[44699]: pgmap v2172: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:19.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:18 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/925703980' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-05-15T23:43:19.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:18 smithi088 ceph-mon[44699]: osdmap e814: 12 total, 12 up, 12 in 2024-05-15T23:43:19.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-05-15T23:43:19.477 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:43:20.704 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-05-15T23:43:21.047 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:20 smithi090 ceph-mon[44434]: pgmap v2174: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:21.047 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:20 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2167027635' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:43:21.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:20 smithi084 ceph-mon[39784]: pgmap v2174: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:21.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:20 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2167027635' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:43:21.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:20 smithi088 ceph-mon[44699]: pgmap v2174: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:21.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:20 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2167027635' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:43:21.325 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:21.325 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:21.416 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 3 1 2024-05-15T23:43:21.680 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:21.985 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:21 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2167027635' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:43:21.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:21 smithi084 ceph-mon[39784]: osdmap e815: 12 total, 12 up, 12 in 2024-05-15T23:43:21.998 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:21 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2167027635' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:43:21.998 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:21 smithi090 ceph-mon[44434]: osdmap e815: 12 total, 12 up, 12 in 2024-05-15T23:43:22.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-05-15T23:43:22.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:21 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2167027635' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:43:22.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:21 smithi088 ceph-mon[44699]: osdmap e815: 12 total, 12 up, 12 in 2024-05-15T23:43:22.288 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:43:22.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:22 smithi090 ceph-mon[44434]: pgmap v2176: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:22.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:22 smithi084 ceph-mon[39784]: pgmap v2176: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:23.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:22 smithi088 ceph-mon[44699]: pgmap v2176: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:23.723 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-05-15T23:43:23.979 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:23.979 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:23.992 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:23 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:43:23.992 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:23 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:43:23.992 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:23 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3675182311' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:43:24.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:23 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:43:24.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:23 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:43:24.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:23 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3675182311' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:43:24.088 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:23 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:43:24.088 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:23 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:43:24.088 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:23 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3675182311' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-05-15T23:43:24.511 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 4 1 2024-05-15T23:43:24.648 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-05-15T23:43:24.780 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:24.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:24 smithi084 ceph-mon[39784]: pgmap v2177: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:24.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:24 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:43:24.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:24 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:43:24.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:24 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3675182311' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:43:24.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:24 smithi084 ceph-mon[39784]: osdmap e816: 12 total, 12 up, 12 in 2024-05-15T23:43:24.913 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:43:25.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:24 smithi090 ceph-mon[44434]: pgmap v2177: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:25.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:24 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:43:25.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:24 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:43:25.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:24 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3675182311' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:43:25.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:24 smithi090 ceph-mon[44434]: osdmap e816: 12 total, 12 up, 12 in 2024-05-15T23:43:25.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:24 smithi088 ceph-mon[44699]: pgmap v2177: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:25.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:24 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:43:25.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:24 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:43:25.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:24 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3675182311' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-05-15T23:43:25.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:24 smithi088 ceph-mon[44699]: osdmap e816: 12 total, 12 up, 12 in 2024-05-15T23:43:26.726 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:26.726 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:27.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:26 smithi090 ceph-mon[44434]: pgmap v2179: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:27.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:26 smithi088 ceph-mon[44699]: pgmap v2179: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:27.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:26 smithi084 ceph-mon[39784]: pgmap v2179: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:27.381 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-05-15T23:43:27.644 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:43:27.735 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2024-05-15T23:43:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:43:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:43:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:27 smithi090 ceph-mon[44434]: osdmap e817: 12 total, 12 up, 12 in 2024-05-15T23:43:28.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:27 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3568233821' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:43:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:43:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:43:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:27 smithi088 ceph-mon[44699]: osdmap e817: 12 total, 12 up, 12 in 2024-05-15T23:43:28.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:27 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3568233821' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:43:28.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:43:28.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:43:28.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:27 smithi084 ceph-mon[39784]: osdmap e817: 12 total, 12 up, 12 in 2024-05-15T23:43:28.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:27 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3568233821' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:43:28.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 4 1 2024-05-15T23:43:28.806 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:28.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:28 smithi084 ceph-mon[39784]: pgmap v2181: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:28.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3568233821' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:43:28.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:28 smithi084 ceph-mon[39784]: osdmap e818: 12 total, 12 up, 12 in 2024-05-15T23:43:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:28 smithi090 ceph-mon[44434]: pgmap v2181: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3568233821' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:43:29.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:28 smithi090 ceph-mon[44434]: osdmap e818: 12 total, 12 up, 12 in 2024-05-15T23:43:29.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:28 smithi088 ceph-mon[44699]: pgmap v2181: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:29.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3568233821' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:43:29.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:28 smithi088 ceph-mon[44699]: osdmap e818: 12 total, 12 up, 12 in 2024-05-15T23:43:29.396 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:29.397 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:30.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-05-15T23:43:30.336 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:43:31.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:30 smithi090 ceph-mon[44434]: pgmap v2183: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:31.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:30 smithi088 ceph-mon[44699]: pgmap v2183: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:31.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:30 smithi084 ceph-mon[39784]: pgmap v2183: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:31.756 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-05-15T23:43:31.862 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2917772512' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:43:32.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2917772512' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:43:32.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2917772512' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-05-15T23:43:32.154 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:32.154 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:32.537 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 6 1 2024-05-15T23:43:32.807 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-05-15T23:43:32.812 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:32.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:32 smithi084 ceph-mon[39784]: pgmap v2184: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:32.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:32 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2917772512' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:43:32.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:32 smithi084 ceph-mon[39784]: osdmap e819: 12 total, 12 up, 12 in 2024-05-15T23:43:33.051 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:32 smithi090 ceph-mon[44434]: pgmap v2184: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:33.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:32 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2917772512' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:43:33.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:32 smithi090 ceph-mon[44434]: osdmap e819: 12 total, 12 up, 12 in 2024-05-15T23:43:33.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:32 smithi088 ceph-mon[44699]: pgmap v2184: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:33.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:32 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2917772512' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-05-15T23:43:33.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:32 smithi088 ceph-mon[44699]: osdmap e819: 12 total, 12 up, 12 in 2024-05-15T23:43:33.603 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:34.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:33 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:43:34.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:33 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:43:34.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:33 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:43:34.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:33 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:43:34.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:33 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:43:34.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:33 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:43:35.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:34 smithi090 ceph-mon[44434]: pgmap v2186: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:35.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:34 smithi090 ceph-mon[44434]: osdmap e820: 12 total, 12 up, 12 in 2024-05-15T23:43:35.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:34 smithi088 ceph-mon[44699]: pgmap v2186: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:35.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:34 smithi088 ceph-mon[44699]: osdmap e820: 12 total, 12 up, 12 in 2024-05-15T23:43:35.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:34 smithi084 ceph-mon[39784]: pgmap v2186: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:35.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:34 smithi084 ceph-mon[39784]: osdmap e820: 12 total, 12 up, 12 in 2024-05-15T23:43:36.367 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:36.368 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:36.764 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 1 (10000) 2024-05-15T23:43:37.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:36 smithi090 ceph-mon[44434]: pgmap v2188: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:37.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1574814828' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T23:43:37.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:36 smithi088 ceph-mon[44699]: pgmap v2188: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:37.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1574814828' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T23:43:37.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:36 smithi084 ceph-mon[39784]: pgmap v2188: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:37.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1574814828' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T23:43:37.424 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-05-15T23:43:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:37 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1574814828' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T23:43:38.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:37 smithi090 ceph-mon[44434]: osdmap e821: 12 total, 12 up, 12 in 2024-05-15T23:43:38.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:37 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1574814828' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T23:43:38.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:37 smithi088 ceph-mon[44699]: osdmap e821: 12 total, 12 up, 12 in 2024-05-15T23:43:38.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:37 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1574814828' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-05-15T23:43:38.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:37 smithi084 ceph-mon[39784]: osdmap e821: 12 total, 12 up, 12 in 2024-05-15T23:43:38.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 6 1 2024-05-15T23:43:38.148 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:38.901 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:39.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:38 smithi090 ceph-mon[44434]: pgmap v2190: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:39.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:38 smithi088 ceph-mon[44699]: pgmap v2190: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:39.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:38 smithi084 ceph-mon[39784]: pgmap v2190: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:39.990 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:39.990 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:41.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:40 smithi090 ceph-mon[44434]: pgmap v2191: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:41.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:40 smithi088 ceph-mon[44699]: pgmap v2191: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:41.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:40 smithi084 ceph-mon[39784]: pgmap v2191: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:42.022 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-05-15T23:43:42.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=2470741623,config=3313370793,health=3897446751,kv=2625160729}) 2024-05-15T23:43:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=120296119,logm=1106861782}) 2024-05-15T23:43:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=602665001}) 2024-05-15T23:43:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=330153065}) 2024-05-15T23:43:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659978040}) 2024-05-15T23:43:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=668731790}) 2024-05-15T23:43:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4182865225}) 2024-05-15T23:43:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273736607}) 2024-05-15T23:43:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2100042598}) 2024-05-15T23:43:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3046872440}) 2024-05-15T23:43:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762393597}) 2024-05-15T23:43:42.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309149509}) 2024-05-15T23:43:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2654832779}) 2024-05-15T23:43:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018512722}) 2024-05-15T23:43:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2868360082}) 2024-05-15T23:43:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3441603604}) 2024-05-15T23:43:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2947495895}) 2024-05-15T23:43:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2587105076}) 2024-05-15T23:43:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2910327882}) 2024-05-15T23:43:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091220664}) 2024-05-15T23:43:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4043836242}) 2024-05-15T23:43:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2242232760}) 2024-05-15T23:43:42.054 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2206299964}) 2024-05-15T23:43:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1215987013}) 2024-05-15T23:43:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1608648784}) 2024-05-15T23:43:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3678045612}) 2024-05-15T23:43:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1442400151}) 2024-05-15T23:43:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2987361492}) 2024-05-15T23:43:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1099382960}) 2024-05-15T23:43:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4106740071}) 2024-05-15T23:43:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1264386369}) 2024-05-15T23:43:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38585600}) 2024-05-15T23:43:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3031938119}) 2024-05-15T23:43:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3323930823}) 2024-05-15T23:43:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3239726191}) 2024-05-15T23:43:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2611181370}) 2024-05-15T23:43:42.055 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=392504795}) 2024-05-15T23:43:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4102706958}) 2024-05-15T23:43:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3476307000}) 2024-05-15T23:43:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3150209853}) 2024-05-15T23:43:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1694303886}) 2024-05-15T23:43:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=199532007}) 2024-05-15T23:43:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2029311889}) 2024-05-15T23:43:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=288316751}) 2024-05-15T23:43:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273081540}) 2024-05-15T23:43:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2795291686}) 2024-05-15T23:43:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=438623467}) 2024-05-15T23:43:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4031544661}) 2024-05-15T23:43:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3095737913}) 2024-05-15T23:43:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1113535356}) 2024-05-15T23:43:42.056 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=916599160}) 2024-05-15T23:43:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3175175771}) 2024-05-15T23:43:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3300182930}) 2024-05-15T23:43:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2553703114}) 2024-05-15T23:43:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2240647498}) 2024-05-15T23:43:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2647345923}) 2024-05-15T23:43:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104830544}) 2024-05-15T23:43:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=495293748}) 2024-05-15T23:43:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695965383}) 2024-05-15T23:43:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=168289022}) 2024-05-15T23:43:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2962493196}) 2024-05-15T23:43:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3977028849}) 2024-05-15T23:43:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3567642828}) 2024-05-15T23:43:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=958815913}) 2024-05-15T23:43:42.057 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4046410510}) 2024-05-15T23:43:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=178645249}) 2024-05-15T23:43:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=321276066}) 2024-05-15T23:43:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=551604301}) 2024-05-15T23:43:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3520042480}) 2024-05-15T23:43:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=928068998}) 2024-05-15T23:43:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1494722052}) 2024-05-15T23:43:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1506620191}) 2024-05-15T23:43:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2440362341}) 2024-05-15T23:43:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1145308370}) 2024-05-15T23:43:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2828908074}) 2024-05-15T23:43:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=369819366}) 2024-05-15T23:43:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1432488611}) 2024-05-15T23:43:42.058 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3483266564}) 2024-05-15T23:43:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2264640253}) 2024-05-15T23:43:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=681387245}) 2024-05-15T23:43:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4219127632}) 2024-05-15T23:43:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1913235066}) 2024-05-15T23:43:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1516026517}) 2024-05-15T23:43:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110711265}) 2024-05-15T23:43:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2607139558}) 2024-05-15T23:43:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2010485395}) 2024-05-15T23:43:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3943813774}) 2024-05-15T23:43:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=172146871}) 2024-05-15T23:43:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3315762389}) 2024-05-15T23:43:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3554737603}) 2024-05-15T23:43:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2636806008}) 2024-05-15T23:43:42.059 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2252575705}) 2024-05-15T23:43:42.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1001608255}) 2024-05-15T23:43:42.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4072231116}) 2024-05-15T23:43:42.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2323555137}) 2024-05-15T23:43:42.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1116494677}) 2024-05-15T23:43:42.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3740995739}) 2024-05-15T23:43:42.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {logm=50,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=9} crc {logm=3263833459,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2657882846,mon_config_key=2125840432}) 2024-05-15T23:43:42.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2968416119}) 2024-05-15T23:43:42.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3412579213}) 2024-05-15T23:43:42.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=75,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=7} crc {mon_config_key=1639139787,monmap=1478146745,osd_metadata=2683863825,osd_pg_creating=4210770337,osdmap=1700942396}) 2024-05-15T23:43:42.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=14554621}) 2024-05-15T23:43:42.060 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:41 smithi090 ceph-mon[44434]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=14} crc {osdmap=3392815532}) 2024-05-15T23:43:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=2470741623,config=3313370793,health=3897446751,kv=2625160729}) 2024-05-15T23:43:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=120296119,logm=1106861782}) 2024-05-15T23:43:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=602665001}) 2024-05-15T23:43:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=330153065}) 2024-05-15T23:43:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659978040}) 2024-05-15T23:43:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=668731790}) 2024-05-15T23:43:42.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4182865225}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273736607}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2100042598}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3046872440}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762393597}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309149509}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2654832779}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018512722}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2868360082}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3441603604}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2947495895}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2587105076}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2910327882}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091220664}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4043836242}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2242232760}) 2024-05-15T23:43:42.072 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2206299964}) 2024-05-15T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1215987013}) 2024-05-15T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1608648784}) 2024-05-15T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3678045612}) 2024-05-15T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1442400151}) 2024-05-15T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2987361492}) 2024-05-15T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1099382960}) 2024-05-15T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4106740071}) 2024-05-15T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1264386369}) 2024-05-15T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38585600}) 2024-05-15T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3031938119}) 2024-05-15T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3323930823}) 2024-05-15T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3239726191}) 2024-05-15T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2611181370}) 2024-05-15T23:43:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=392504795}) 2024-05-15T23:43:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4102706958}) 2024-05-15T23:43:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3476307000}) 2024-05-15T23:43:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3150209853}) 2024-05-15T23:43:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1694303886}) 2024-05-15T23:43:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=199532007}) 2024-05-15T23:43:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2029311889}) 2024-05-15T23:43:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=288316751}) 2024-05-15T23:43:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273081540}) 2024-05-15T23:43:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2795291686}) 2024-05-15T23:43:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=438623467}) 2024-05-15T23:43:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4031544661}) 2024-05-15T23:43:42.074 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3095737913}) 2024-05-15T23:43:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1113535356}) 2024-05-15T23:43:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=916599160}) 2024-05-15T23:43:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3175175771}) 2024-05-15T23:43:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3300182930}) 2024-05-15T23:43:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2553703114}) 2024-05-15T23:43:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2240647498}) 2024-05-15T23:43:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2647345923}) 2024-05-15T23:43:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104830544}) 2024-05-15T23:43:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=495293748}) 2024-05-15T23:43:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695965383}) 2024-05-15T23:43:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=168289022}) 2024-05-15T23:43:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2962493196}) 2024-05-15T23:43:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3977028849}) 2024-05-15T23:43:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3567642828}) 2024-05-15T23:43:42.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=958815913}) 2024-05-15T23:43:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4046410510}) 2024-05-15T23:43:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=178645249}) 2024-05-15T23:43:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=321276066}) 2024-05-15T23:43:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=551604301}) 2024-05-15T23:43:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3520042480}) 2024-05-15T23:43:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=928068998}) 2024-05-15T23:43:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1494722052}) 2024-05-15T23:43:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1506620191}) 2024-05-15T23:43:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2440362341}) 2024-05-15T23:43:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1145308370}) 2024-05-15T23:43:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2828908074}) 2024-05-15T23:43:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=369819366}) 2024-05-15T23:43:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1432488611}) 2024-05-15T23:43:42.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3483266564}) 2024-05-15T23:43:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2264640253}) 2024-05-15T23:43:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=681387245}) 2024-05-15T23:43:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4219127632}) 2024-05-15T23:43:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1913235066}) 2024-05-15T23:43:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1516026517}) 2024-05-15T23:43:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110711265}) 2024-05-15T23:43:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2607139558}) 2024-05-15T23:43:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2010485395}) 2024-05-15T23:43:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3943813774}) 2024-05-15T23:43:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=172146871}) 2024-05-15T23:43:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3315762389}) 2024-05-15T23:43:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3554737603}) 2024-05-15T23:43:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2636806008}) 2024-05-15T23:43:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2252575705}) 2024-05-15T23:43:42.077 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1001608255}) 2024-05-15T23:43:42.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4072231116}) 2024-05-15T23:43:42.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2323555137}) 2024-05-15T23:43:42.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1116494677}) 2024-05-15T23:43:42.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3740995739}) 2024-05-15T23:43:42.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {logm=50,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=9} crc {logm=3263833459,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2657882846,mon_config_key=2125840432}) 2024-05-15T23:43:42.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2968416119}) 2024-05-15T23:43:42.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3412579213}) 2024-05-15T23:43:42.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=75,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=7} crc {mon_config_key=1639139787,monmap=1478146745,osd_metadata=2683863825,osd_pg_creating=4210770337,osdmap=1700942396}) 2024-05-15T23:43:42.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=14554621}) 2024-05-15T23:43:42.078 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:41 smithi088 ceph-mon[44699]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=14} crc {osdmap=3392815532}) 2024-05-15T23:43:42.088 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=2470741623,config=3313370793,health=3897446751,kv=2625160729}) 2024-05-15T23:43:42.088 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=120296119,logm=1106861782}) 2024-05-15T23:43:42.088 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=602665001}) 2024-05-15T23:43:42.088 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=330153065}) 2024-05-15T23:43:42.088 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659978040}) 2024-05-15T23:43:42.088 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=668731790}) 2024-05-15T23:43:42.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4182865225}) 2024-05-15T23:43:42.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273736607}) 2024-05-15T23:43:42.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2100042598}) 2024-05-15T23:43:42.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3046872440}) 2024-05-15T23:43:42.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762393597}) 2024-05-15T23:43:42.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309149509}) 2024-05-15T23:43:42.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2654832779}) 2024-05-15T23:43:42.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018512722}) 2024-05-15T23:43:42.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2868360082}) 2024-05-15T23:43:42.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3441603604}) 2024-05-15T23:43:42.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2947495895}) 2024-05-15T23:43:42.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2587105076}) 2024-05-15T23:43:42.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2910327882}) 2024-05-15T23:43:42.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091220664}) 2024-05-15T23:43:42.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4043836242}) 2024-05-15T23:43:42.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2242232760}) 2024-05-15T23:43:42.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2206299964}) 2024-05-15T23:43:42.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1215987013}) 2024-05-15T23:43:42.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1608648784}) 2024-05-15T23:43:42.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3678045612}) 2024-05-15T23:43:42.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1442400151}) 2024-05-15T23:43:42.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2987361492}) 2024-05-15T23:43:42.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1099382960}) 2024-05-15T23:43:42.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4106740071}) 2024-05-15T23:43:42.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1264386369}) 2024-05-15T23:43:42.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38585600}) 2024-05-15T23:43:42.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3031938119}) 2024-05-15T23:43:42.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3323930823}) 2024-05-15T23:43:42.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3239726191}) 2024-05-15T23:43:42.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2611181370}) 2024-05-15T23:43:42.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=392504795}) 2024-05-15T23:43:42.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4102706958}) 2024-05-15T23:43:42.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3476307000}) 2024-05-15T23:43:42.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3150209853}) 2024-05-15T23:43:42.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1694303886}) 2024-05-15T23:43:42.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=199532007}) 2024-05-15T23:43:42.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2029311889}) 2024-05-15T23:43:42.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=288316751}) 2024-05-15T23:43:42.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273081540}) 2024-05-15T23:43:42.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2795291686}) 2024-05-15T23:43:42.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=438623467}) 2024-05-15T23:43:42.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4031544661}) 2024-05-15T23:43:42.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3095737913}) 2024-05-15T23:43:42.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1113535356}) 2024-05-15T23:43:42.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=916599160}) 2024-05-15T23:43:42.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3175175771}) 2024-05-15T23:43:42.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3300182930}) 2024-05-15T23:43:42.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2553703114}) 2024-05-15T23:43:42.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2240647498}) 2024-05-15T23:43:42.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2647345923}) 2024-05-15T23:43:42.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104830544}) 2024-05-15T23:43:42.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=495293748}) 2024-05-15T23:43:42.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695965383}) 2024-05-15T23:43:42.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=168289022}) 2024-05-15T23:43:42.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2962493196}) 2024-05-15T23:43:42.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3977028849}) 2024-05-15T23:43:42.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3567642828}) 2024-05-15T23:43:42.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=958815913}) 2024-05-15T23:43:42.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4046410510}) 2024-05-15T23:43:42.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=178645249}) 2024-05-15T23:43:42.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=321276066}) 2024-05-15T23:43:42.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=551604301}) 2024-05-15T23:43:42.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3520042480}) 2024-05-15T23:43:42.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=928068998}) 2024-05-15T23:43:42.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1494722052}) 2024-05-15T23:43:42.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1506620191}) 2024-05-15T23:43:42.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2440362341}) 2024-05-15T23:43:42.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1145308370}) 2024-05-15T23:43:42.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2828908074}) 2024-05-15T23:43:42.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=369819366}) 2024-05-15T23:43:42.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1432488611}) 2024-05-15T23:43:42.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3483266564}) 2024-05-15T23:43:42.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2264640253}) 2024-05-15T23:43:42.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=681387245}) 2024-05-15T23:43:42.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4219127632}) 2024-05-15T23:43:42.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1913235066}) 2024-05-15T23:43:42.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1516026517}) 2024-05-15T23:43:42.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110711265}) 2024-05-15T23:43:42.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2607139558}) 2024-05-15T23:43:42.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2010485395}) 2024-05-15T23:43:42.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3943813774}) 2024-05-15T23:43:42.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=172146871}) 2024-05-15T23:43:42.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3315762389}) 2024-05-15T23:43:42.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3554737603}) 2024-05-15T23:43:42.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2636806008}) 2024-05-15T23:43:42.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2252575705}) 2024-05-15T23:43:42.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1001608255}) 2024-05-15T23:43:42.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4072231116}) 2024-05-15T23:43:42.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2323555137}) 2024-05-15T23:43:42.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1116494677}) 2024-05-15T23:43:42.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3740995739}) 2024-05-15T23:43:42.096 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {logm=50,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=9} crc {logm=3263833459,mdsmap=2840952586,mgr=523987623,mgr_command_descs=3040664224,mgr_metadata=1894328593,mgrstat=2657882846,mon_config_key=2125840432}) 2024-05-15T23:43:42.096 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2968416119}) 2024-05-15T23:43:42.096 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3412579213}) 2024-05-15T23:43:42.096 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=75,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=7} crc {mon_config_key=1639139787,monmap=1478146745,osd_metadata=2683863825,osd_pg_creating=4210770337,osdmap=1700942396}) 2024-05-15T23:43:42.096 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=14554621}) 2024-05-15T23:43:42.096 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:41 smithi084 ceph-mon[39784]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=14} crc {osdmap=3392815532}) 2024-05-15T23:43:42.329 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:42.792 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-05-15T23:43:42.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:42 smithi084 ceph-mon[39784]: pgmap v2192: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:42.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:42 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/232566797' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T23:43:43.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:42 smithi090 ceph-mon[44434]: pgmap v2192: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:43.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:42 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/232566797' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T23:43:43.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:42 smithi088 ceph-mon[44699]: pgmap v2192: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:43.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:42 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/232566797' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-05-15T23:43:44.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:43 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/232566797' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-05-15T23:43:44.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:43 smithi090 ceph-mon[44434]: osdmap e822: 12 total, 12 up, 12 in 2024-05-15T23:43:44.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:43 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/232566797' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-05-15T23:43:44.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:43 smithi084 ceph-mon[39784]: osdmap e822: 12 total, 12 up, 12 in 2024-05-15T23:43:44.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:43 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/232566797' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-05-15T23:43:44.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:43 smithi088 ceph-mon[44699]: osdmap e822: 12 total, 12 up, 12 in 2024-05-15T23:43:44.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 7 1 2024-05-15T23:43:44.648 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:44.648 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:44.649 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:44.946 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:44 smithi084 ceph-mon[39784]: pgmap v2194: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:44 smithi090 ceph-mon[44434]: pgmap v2194: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:45.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:44 smithi088 ceph-mon[44699]: pgmap v2194: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:45.693 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-05-15T23:43:46.775 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:47.028 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:46 smithi084 ceph-mon[39784]: pgmap v2195: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:47.028 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:46 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:43:47.028 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:46 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:43:47.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:46 smithi090 ceph-mon[44434]: pgmap v2195: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:47.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:46 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:43:47.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:46 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:43:47.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:46 smithi088 ceph-mon[44699]: pgmap v2195: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:47.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:46 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:43:47.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:46 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:43:48.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:47 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:43:48.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:47 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:43:48.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:47 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:43:48.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:47 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:43:48.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:47 smithi088 ceph-mon[44699]: osdmap e823: 12 total, 12 up, 12 in 2024-05-15T23:43:48.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:47 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:43:48.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:47 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:43:48.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:47 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:43:48.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:47 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:43:48.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:47 smithi084 ceph-mon[39784]: osdmap e823: 12 total, 12 up, 12 in 2024-05-15T23:43:48.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:47 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:43:48.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:47 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:43:48.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:47 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:43:48.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:47 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:43:48.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:47 smithi090 ceph-mon[44434]: osdmap e823: 12 total, 12 up, 12 in 2024-05-15T23:43:48.701 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:48.701 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:48.820 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-05-15T23:43:49.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:48 smithi088 ceph-mon[44699]: pgmap v2197: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:49.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:48 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3983253468' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:43:49.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:48 smithi084 ceph-mon[39784]: pgmap v2197: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:49.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:48 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3983253468' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:43:49.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:48 smithi090 ceph-mon[44434]: pgmap v2197: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:49.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:48 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3983253468' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:43:49.681 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 7 1 2024-05-15T23:43:50.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:49 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3983253468' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:43:50.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:49 smithi084 ceph-mon[39784]: osdmap e824: 12 total, 12 up, 12 in 2024-05-15T23:43:50.164 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-05-15T23:43:50.284 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:50.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:49 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3983253468' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:43:50.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:49 smithi090 ceph-mon[44434]: osdmap e824: 12 total, 12 up, 12 in 2024-05-15T23:43:50.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:49 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3983253468' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:43:50.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:49 smithi088 ceph-mon[44699]: osdmap e824: 12 total, 12 up, 12 in 2024-05-15T23:43:51.088 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:51.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:50 smithi084 ceph-mon[39784]: pgmap v2199: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:51.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:50 smithi090 ceph-mon[44434]: pgmap v2199: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:51.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:50 smithi088 ceph-mon[44699]: pgmap v2199: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:52.954 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:52 smithi084 ceph-mon[39784]: pgmap v2200: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:53.196 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:53.196 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:53.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:52 smithi090 ceph-mon[44434]: pgmap v2200: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:53.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:52 smithi088 ceph-mon[44699]: pgmap v2200: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:54.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-05-15T23:43:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:54.832 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-05-15T23:43:54.966 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:54 smithi084 ceph-mon[39784]: pgmap v2201: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:54.966 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:54 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3945090317' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:43:55.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:54 smithi090 ceph-mon[44434]: pgmap v2201: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:55.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:54 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3945090317' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:43:55.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:54 smithi088 ceph-mon[44699]: pgmap v2201: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:55.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:54 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3945090317' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-05-15T23:43:55.595 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 9 1 2024-05-15T23:43:56.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:55 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3945090317' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:43:56.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:55 smithi084 ceph-mon[39784]: osdmap e825: 12 total, 12 up, 12 in 2024-05-15T23:43:56.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:55 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3945090317' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:43:56.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:55 smithi090 ceph-mon[44434]: osdmap e825: 12 total, 12 up, 12 in 2024-05-15T23:43:56.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:55 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3945090317' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-05-15T23:43:56.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:55 smithi088 ceph-mon[44699]: osdmap e825: 12 total, 12 up, 12 in 2024-05-15T23:43:56.330 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:43:57.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:56 smithi084 ceph-mon[39784]: pgmap v2203: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:57.177 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-05-15T23:43:57.177 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:43:57.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:56 smithi090 ceph-mon[44434]: pgmap v2203: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:57.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:56 smithi088 ceph-mon[44699]: pgmap v2203: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:58.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-05-15T23:43:59.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:58 smithi084 ceph-mon[39784]: pgmap v2204: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:59.084 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:43:59.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:58 smithi090 ceph-mon[44434]: pgmap v2204: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:59.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:58 smithi088 ceph-mon[44699]: pgmap v2204: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:43:59.851 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.9 to 1 (10000) 2024-05-15T23:44:00.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:43:59 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2266447877' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:44:00.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:43:59 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2266447877' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:44:00.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:43:59 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2266447877' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:44:00.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 9 1 2024-05-15T23:44:00.851 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:44:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:44:01.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:00 smithi084 ceph-mon[39784]: pgmap v2205: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:01.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:00 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2266447877' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:44:01.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:00 smithi084 ceph-mon[39784]: osdmap e826: 12 total, 12 up, 12 in 2024-05-15T23:44:01.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:00 smithi090 ceph-mon[44434]: pgmap v2205: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:01.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:00 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2266447877' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:44:01.134 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:00 smithi090 ceph-mon[44434]: osdmap e826: 12 total, 12 up, 12 in 2024-05-15T23:44:01.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:00 smithi088 ceph-mon[44699]: pgmap v2205: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:01.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:00 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2266447877' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:44:01.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:00 smithi088 ceph-mon[44699]: osdmap e826: 12 total, 12 up, 12 in 2024-05-15T23:44:01.476 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-05-15T23:44:01.739 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:44:02.992 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:02 smithi090 ceph-mon[44434]: pgmap v2207: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:02.992 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1425036254' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:44:03.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:02 smithi084 ceph-mon[39784]: pgmap v2207: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:03.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1425036254' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:44:03.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:02 smithi088 ceph-mon[44699]: pgmap v2207: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:03.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1425036254' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-05-15T23:44:03.455 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-05-15T23:44:03.570 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:44:03.570 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:44:04.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-05-15T23:44:04.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 10 1 2024-05-15T23:44:04.430 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.c/config 2024-05-15T23:44:04.520 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:04.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:04 smithi084 ceph-mon[39784]: pgmap v2208: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:04.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:04 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1425036254' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:44:04.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:04 smithi084 ceph-mon[39784]: osdmap e827: 12 total, 12 up, 12 in 2024-05-15T23:44:04.719 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:04 smithi090 ceph-mon[44434]: pgmap v2208: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:04.719 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:04 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1425036254' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:44:04.719 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:04 smithi090 ceph-mon[44434]: osdmap e827: 12 total, 12 up, 12 in 2024-05-15T23:44:04.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:04 smithi088 ceph-mon[44699]: pgmap v2208: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:04.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:04 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1425036254' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-05-15T23:44:04.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:04 smithi088 ceph-mon[44699]: osdmap e827: 12 total, 12 up, 12 in 2024-05-15T23:44:06.202 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-05-15T23:44:06.202 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-15T23:44:06.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:06 smithi090 ceph-mon[44434]: pgmap v2210: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:06.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:06 smithi088 ceph-mon[44699]: pgmap v2210: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:06.862 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-05-15T23:44:06.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:06 smithi084 ceph-mon[39784]: pgmap v2210: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:07.140 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:44:07.472 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.10 to 1 (10000) 2024-05-15T23:44:07.611 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:07 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1904830356' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:44:07.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:07 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1904830356' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:44:07.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:07 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1904830356' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:44:08.515 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 10 1 2024-05-15T23:44:08.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:08 smithi088 ceph-mon[44699]: pgmap v2211: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:08 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1904830356' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:44:08.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:08 smithi088 ceph-mon[44699]: osdmap e828: 12 total, 12 up, 12 in 2024-05-15T23:44:08.783 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:08.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:08 smithi090 ceph-mon[44434]: pgmap v2211: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:08.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:08 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1904830356' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:44:08.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:08 smithi090 ceph-mon[44434]: osdmap e828: 12 total, 12 up, 12 in 2024-05-15T23:44:08.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:08 smithi084 ceph-mon[39784]: pgmap v2211: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:08.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:08 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1904830356' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:44:08.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:08 smithi084 ceph-mon[39784]: osdmap e828: 12 total, 12 up, 12 in 2024-05-15T23:44:08.895 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:44:08.895 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:44:08.895 INFO:teuthology.orchestra.run.smithi088.stdout: "num_ops": 0 2024-05-15T23:44:08.895 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:44:09.612 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-05-15T23:44:09.875 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:44:10.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:10 smithi084 ceph-mon[39784]: pgmap v2213: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:10.751 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:10 smithi088 ceph-mon[44699]: pgmap v2213: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:10.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:10 smithi090 ceph-mon[44434]: pgmap v2213: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:11.486 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-05-15T23:44:11.616 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:44:11.617 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [], 2024-05-15T23:44:11.617 INFO:teuthology.orchestra.run.smithi088.stdout: "complaint_time": 30, 2024-05-15T23:44:11.617 INFO:teuthology.orchestra.run.smithi088.stdout: "num_blocked_ops": 0 2024-05-15T23:44:11.617 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:44:11.636 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:11 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:44:11.636 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:11 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:44:11.636 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:11 smithi088 ceph-mon[44699]: osdmap e829: 12 total, 12 up, 12 in 2024-05-15T23:44:11.636 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:11 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2421516611' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:44:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:11 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:44:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:11 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:44:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:11 smithi090 ceph-mon[44434]: osdmap e829: 12 total, 12 up, 12 in 2024-05-15T23:44:11.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:11 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2421516611' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:44:11.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:11 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:44:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:11 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:44:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:11 smithi084 ceph-mon[39784]: osdmap e829: 12 total, 12 up, 12 in 2024-05-15T23:44:11.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:11 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2421516611' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-05-15T23:44:12.246 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 11 1 2024-05-15T23:44:12.402 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-05-15T23:44:12.517 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:12.671 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.b/config 2024-05-15T23:44:12.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:12 smithi084 ceph-mon[39784]: pgmap v2215: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:12.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:12 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2421516611' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:44:12.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:12 smithi084 ceph-mon[39784]: osdmap e830: 12 total, 12 up, 12 in 2024-05-15T23:44:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:12 smithi090 ceph-mon[44434]: pgmap v2215: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:12 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2421516611' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:44:12.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:12 smithi090 ceph-mon[44434]: osdmap e830: 12 total, 12 up, 12 in 2024-05-15T23:44:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:12 smithi088 ceph-mon[44699]: pgmap v2215: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:12.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:12 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2421516611' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-05-15T23:44:12.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:12 smithi088 ceph-mon[44699]: osdmap e830: 12 total, 12 up, 12 in 2024-05-15T23:44:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-15T23:44:14.422 INFO:teuthology.orchestra.run.smithi088.stdout: "size": 20, 2024-05-15T23:44:14.422 INFO:teuthology.orchestra.run.smithi088.stdout: "duration": 600, 2024-05-15T23:44:14.422 INFO:teuthology.orchestra.run.smithi088.stdout: "ops": [] 2024-05-15T23:44:14.422 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-15T23:44:14.763 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:14 smithi088 ceph-mon[44699]: pgmap v2217: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:14.764 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:14 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:44:14.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:14 smithi090 ceph-mon[44434]: pgmap v2217: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:14.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:14 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:44:14.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:14 smithi084 ceph-mon[39784]: pgmap v2217: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:14.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:14 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:44:15.502 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.11 to 1 (10000) 2024-05-15T23:44:15.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:15 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/511555930' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:44:15.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:15 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/511555930' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:44:15.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:15 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/511555930' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:44:16.326 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 11 1 2024-05-15T23:44:16.600 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:16.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:16 smithi084 ceph-mon[39784]: pgmap v2218: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:16 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/511555930' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:44:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:16 smithi084 ceph-mon[39784]: osdmap e831: 12 total, 12 up, 12 in 2024-05-15T23:44:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:16 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:44:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:16 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:44:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:44:16.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:44:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:16 smithi090 ceph-mon[44434]: pgmap v2218: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:16 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/511555930' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:44:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:16 smithi090 ceph-mon[44434]: osdmap e831: 12 total, 12 up, 12 in 2024-05-15T23:44:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:16 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:44:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:16 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:44:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:44:16.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:44:16.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:16 smithi088 ceph-mon[44699]: pgmap v2218: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:16.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:16 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/511555930' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:44:16.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:16 smithi088 ceph-mon[44699]: osdmap e831: 12 total, 12 up, 12 in 2024-05-15T23:44:16.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:16 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:44:16.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:16 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:44:16.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:44:16.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:44:17.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:17 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:44:17.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:17 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:44:17.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:17 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:44:18.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:18 smithi090 ceph-mon[44434]: pgmap v2220: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:18.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:18 smithi088 ceph-mon[44699]: pgmap v2220: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:18.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:18 smithi084 ceph-mon[39784]: pgmap v2220: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:19.519 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-05-15T23:44:19.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:19 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3708495942' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:44:19.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:19 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:44:19.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:19 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:44:19.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:19 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3708495942' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:44:19.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:19 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:44:19.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:19 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:44:19.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:19 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3708495942' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-05-15T23:44:19.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:19 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:44:19.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:19 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:44:20.291 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 2 1 2024-05-15T23:44:20.561 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:20.798 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:20 smithi084 ceph-mon[39784]: pgmap v2221: 1 pgs: 1 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:20.798 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:20 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3708495942' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:44:20.798 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:20 smithi084 ceph-mon[39784]: osdmap e832: 12 total, 12 up, 12 in 2024-05-15T23:44:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:20 smithi090 ceph-mon[44434]: pgmap v2221: 1 pgs: 1 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:20 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3708495942' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:44:20.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:20 smithi090 ceph-mon[44434]: osdmap e832: 12 total, 12 up, 12 in 2024-05-15T23:44:20.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:20 smithi088 ceph-mon[44699]: pgmap v2221: 1 pgs: 1 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:20.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:20 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3708495942' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-05-15T23:44:20.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:20 smithi088 ceph-mon[44699]: osdmap e832: 12 total, 12 up, 12 in 2024-05-15T23:44:22.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:22 smithi090 ceph-mon[44434]: pgmap v2223: 1 pgs: 1 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:22.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:22 smithi088 ceph-mon[44699]: pgmap v2223: 1 pgs: 1 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:22.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:22 smithi084 ceph-mon[39784]: pgmap v2223: 1 pgs: 1 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:23.529 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2024-05-15T23:44:23.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:23 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/674580156' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T23:44:23.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:23 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/674580156' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T23:44:23.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:23 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/674580156' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T23:44:24.347 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 2 1 2024-05-15T23:44:24.618 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:24.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:24 smithi084 ceph-mon[39784]: pgmap v2224: 1 pgs: 1 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:24.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:24 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/674580156' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T23:44:24.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:24 smithi084 ceph-mon[39784]: osdmap e833: 12 total, 12 up, 12 in 2024-05-15T23:44:24.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:24 smithi090 ceph-mon[44434]: pgmap v2224: 1 pgs: 1 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:24.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:24 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/674580156' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T23:44:24.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:24 smithi090 ceph-mon[44434]: osdmap e833: 12 total, 12 up, 12 in 2024-05-15T23:44:24.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:24 smithi088 ceph-mon[44699]: pgmap v2224: 1 pgs: 1 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:24.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:24 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/674580156' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-05-15T23:44:24.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:24 smithi088 ceph-mon[44699]: osdmap e833: 12 total, 12 up, 12 in 2024-05-15T23:44:26.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:26 smithi090 ceph-mon[44434]: pgmap v2226: 1 pgs: 1 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:26.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:26 smithi088 ceph-mon[44699]: pgmap v2226: 1 pgs: 1 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:26.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:26 smithi084 ceph-mon[39784]: pgmap v2226: 1 pgs: 1 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:27.542 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-05-15T23:44:27.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:44:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:44:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:27 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2884025355' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T23:44:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:27 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:44:27.803 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:27 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:44:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:44:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:44:27.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:27 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2884025355' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T23:44:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:27 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:44:27.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:27 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:44:27.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:44:27.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:44:27.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:27 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2884025355' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-05-15T23:44:27.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:27 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:44:27.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:27 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:44:28.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 0 1 2024-05-15T23:44:28.614 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:28.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:28 smithi084 ceph-mon[39784]: pgmap v2227: 1 pgs: 1 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:28.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:28 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2884025355' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-05-15T23:44:28.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:28 smithi084 ceph-mon[39784]: osdmap e834: 12 total, 12 up, 12 in 2024-05-15T23:44:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:28 smithi090 ceph-mon[44434]: pgmap v2227: 1 pgs: 1 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:28 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2884025355' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-05-15T23:44:28.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:28 smithi090 ceph-mon[44434]: osdmap e834: 12 total, 12 up, 12 in 2024-05-15T23:44:28.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:28 smithi088 ceph-mon[44699]: pgmap v2227: 1 pgs: 1 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:28 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2884025355' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-05-15T23:44:28.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:28 smithi088 ceph-mon[44699]: osdmap e834: 12 total, 12 up, 12 in 2024-05-15T23:44:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:30 smithi090 ceph-mon[44434]: pgmap v2229: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:30 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:44:30.802 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:30 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:44:30.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:30 smithi088 ceph-mon[44699]: pgmap v2229: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:30.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:30 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:44:30.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:30 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:44:30.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:30 smithi084 ceph-mon[39784]: pgmap v2229: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:30.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:30 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:44:30.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:30 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:44:31.554 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2024-05-15T23:44:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:31 smithi088 ceph-mon[44699]: osdmap e835: 12 total, 12 up, 12 in 2024-05-15T23:44:31.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1901647925' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:44:31.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:31 smithi088 ceph-mon[44699]: from='client.17844 v1:172.21.15.84:0/1901647925' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:44:31.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:31 smithi084 ceph-mon[39784]: osdmap e835: 12 total, 12 up, 12 in 2024-05-15T23:44:31.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1901647925' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:44:31.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:31 smithi084 ceph-mon[39784]: from='client.17844 v1:172.21.15.84:0/1901647925' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:44:32.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:31 smithi090 ceph-mon[44434]: osdmap e835: 12 total, 12 up, 12 in 2024-05-15T23:44:32.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1901647925' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:44:32.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:31 smithi090 ceph-mon[44434]: from='client.17844 v1:172.21.15.84:0/1901647925' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:44:32.300 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 0 1 2024-05-15T23:44:32.571 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:32 smithi088 ceph-mon[44699]: pgmap v2231: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:32 smithi088 ceph-mon[44699]: from='client.17844 v1:172.21.15.84:0/1901647925' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T23:44:32.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:32 smithi088 ceph-mon[44699]: osdmap e836: 12 total, 12 up, 12 in 2024-05-15T23:44:32.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:32 smithi084 ceph-mon[39784]: pgmap v2231: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:32.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:32 smithi084 ceph-mon[39784]: from='client.17844 v1:172.21.15.84:0/1901647925' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T23:44:32.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:32 smithi084 ceph-mon[39784]: osdmap e836: 12 total, 12 up, 12 in 2024-05-15T23:44:33.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:32 smithi090 ceph-mon[44434]: pgmap v2231: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:33.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:32 smithi090 ceph-mon[44434]: from='client.17844 v1:172.21.15.84:0/1901647925' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-05-15T23:44:33.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:32 smithi090 ceph-mon[44434]: osdmap e836: 12 total, 12 up, 12 in 2024-05-15T23:44:34.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:34 smithi088 ceph-mon[44699]: pgmap v2233: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:34.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:34 smithi084 ceph-mon[39784]: pgmap v2233: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:35.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:34 smithi090 ceph-mon[44434]: pgmap v2233: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:35.568 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-05-15T23:44:35.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:35 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3638359972' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:44:35.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:35 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3638359972' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:44:36.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:35 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3638359972' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-05-15T23:44:36.321 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 8 1 2024-05-15T23:44:36.593 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:36 smithi088 ceph-mon[44699]: pgmap v2234: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:36.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:36 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3638359972' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-05-15T23:44:36.821 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:36 smithi088 ceph-mon[44699]: osdmap e837: 12 total, 12 up, 12 in 2024-05-15T23:44:36.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:36 smithi084 ceph-mon[39784]: pgmap v2234: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:36.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:36 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3638359972' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-05-15T23:44:36.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:36 smithi084 ceph-mon[39784]: osdmap e837: 12 total, 12 up, 12 in 2024-05-15T23:44:37.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:36 smithi090 ceph-mon[44434]: pgmap v2234: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:37.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:36 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3638359972' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-05-15T23:44:37.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:36 smithi090 ceph-mon[44434]: osdmap e837: 12 total, 12 up, 12 in 2024-05-15T23:44:38.576 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.8 to 1 (10000) 2024-05-15T23:44:38.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:38 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:44:38.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:38 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:44:38.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:38 smithi084 ceph-mon[39784]: pgmap v2236: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:38.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:38 smithi084 ceph-mon[39784]: osdmap e838: 12 total, 12 up, 12 in 2024-05-15T23:44:38.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:38 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/769684060' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:44:39.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:38 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:44:39.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:38 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:44:39.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:38 smithi090 ceph-mon[44434]: pgmap v2236: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:39.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:38 smithi090 ceph-mon[44434]: osdmap e838: 12 total, 12 up, 12 in 2024-05-15T23:44:39.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:38 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/769684060' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:44:39.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:38 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:44:39.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:38 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:44:39.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:38 smithi088 ceph-mon[44699]: pgmap v2236: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:39.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:38 smithi088 ceph-mon[44699]: osdmap e838: 12 total, 12 up, 12 in 2024-05-15T23:44:39.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:38 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/769684060' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:44:39.789 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:39 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/769684060' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:44:39.789 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:39 smithi084 ceph-mon[39784]: osdmap e839: 12 total, 12 up, 12 in 2024-05-15T23:44:39.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 8 1 2024-05-15T23:44:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:39 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/769684060' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:44:40.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:39 smithi090 ceph-mon[44434]: osdmap e839: 12 total, 12 up, 12 in 2024-05-15T23:44:40.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:39 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/769684060' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:44:40.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:39 smithi088 ceph-mon[44699]: osdmap e839: 12 total, 12 up, 12 in 2024-05-15T23:44:40.091 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:40.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:40 smithi084 ceph-mon[39784]: pgmap v2239: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:41.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:40 smithi090 ceph-mon[44434]: pgmap v2239: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:41.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:40 smithi088 ceph-mon[44699]: pgmap v2239: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:42.600 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-05-15T23:44:42.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:42 smithi084 ceph-mon[39784]: pgmap v2240: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:42.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:42 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:44:42.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:42 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:44:42.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:42 smithi084 ceph-mon[39784]: osdmap e840: 12 total, 12 up, 12 in 2024-05-15T23:44:42.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:42 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/302130245' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:44:43.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:42 smithi090 ceph-mon[44434]: pgmap v2240: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:43.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:42 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:44:43.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:42 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:44:43.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:42 smithi090 ceph-mon[44434]: osdmap e840: 12 total, 12 up, 12 in 2024-05-15T23:44:43.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:42 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/302130245' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:44:43.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:42 smithi088 ceph-mon[44699]: pgmap v2240: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:43.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:42 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:44:43.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:42 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:44:43.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:42 smithi088 ceph-mon[44699]: osdmap e840: 12 total, 12 up, 12 in 2024-05-15T23:44:43.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:42 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/302130245' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-05-15T23:44:43.444 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd reweight 5 1 2024-05-15T23:44:43.716 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:43.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:43 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/302130245' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:44:43.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:43 smithi084 ceph-mon[39784]: osdmap e841: 12 total, 12 up, 12 in 2024-05-15T23:44:44.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:43 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/302130245' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:44:44.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:43 smithi090 ceph-mon[44434]: osdmap e841: 12 total, 12 up, 12 in 2024-05-15T23:44:44.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:43 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/302130245' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-05-15T23:44:44.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:43 smithi088 ceph-mon[44699]: osdmap e841: 12 total, 12 up, 12 in 2024-05-15T23:44:44.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:44 smithi084 ceph-mon[39784]: pgmap v2243: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:44.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:44 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:44:44.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:44 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:44:44.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:44 smithi084 ceph-mon[39784]: osdmap e842: 12 total, 12 up, 12 in 2024-05-15T23:44:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:44 smithi090 ceph-mon[44434]: pgmap v2243: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:44 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:44:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:44 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:44:45.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:44 smithi090 ceph-mon[44434]: osdmap e842: 12 total, 12 up, 12 in 2024-05-15T23:44:45.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:44 smithi088 ceph-mon[44699]: pgmap v2243: 1 pgs: 1 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:45.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:44 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:44:45.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:44 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:44:45.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:44 smithi088 ceph-mon[44699]: osdmap e842: 12 total, 12 up, 12 in 2024-05-15T23:44:45.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:45 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:44:45.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:45 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:44:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:45 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:44:46.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:45 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:44:46.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:45 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:44:46.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:45 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:44:46.613 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-05-15T23:44:46.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:46 smithi084 ceph-mon[39784]: pgmap v2245: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:46.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:46 smithi084 ceph-mon[39784]: osdmap e843: 12 total, 12 up, 12 in 2024-05-15T23:44:46.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:46 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/916741469' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:44:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:46 smithi090 ceph-mon[44434]: pgmap v2245: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:46 smithi090 ceph-mon[44434]: osdmap e843: 12 total, 12 up, 12 in 2024-05-15T23:44:47.053 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:46 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/916741469' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:44:47.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:46 smithi088 ceph-mon[44699]: pgmap v2245: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:47.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:46 smithi088 ceph-mon[44699]: osdmap e843: 12 total, 12 up, 12 in 2024-05-15T23:44:47.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:46 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/916741469' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:44:47.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd primary-affinity 5 1 2024-05-15T23:44:47.807 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:47.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:47 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/916741469' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:44:47.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:47 smithi084 ceph-mon[39784]: osdmap e844: 12 total, 12 up, 12 in 2024-05-15T23:44:48.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:47 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/916741469' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:44:48.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:47 smithi090 ceph-mon[44434]: osdmap e844: 12 total, 12 up, 12 in 2024-05-15T23:44:48.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:47 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/916741469' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:44:48.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:47 smithi088 ceph-mon[44699]: osdmap e844: 12 total, 12 up, 12 in 2024-05-15T23:44:49.008 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:48 smithi084 ceph-mon[39784]: pgmap v2248: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:49.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:48 smithi090 ceph-mon[44434]: pgmap v2248: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:49.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:48 smithi088 ceph-mon[44699]: pgmap v2248: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:50.633 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-05-15T23:44:50.983 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:50 smithi084 ceph-mon[39784]: pgmap v2249: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:50.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/488180559' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:44:51.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:50 smithi090 ceph-mon[44434]: pgmap v2249: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:51.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/488180559' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:44:51.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:50 smithi088 ceph-mon[44699]: pgmap v2249: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:51.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/488180559' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-05-15T23:44:51.322 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-05-15T23:44:51.322 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-05-15T23:44:51.322 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-05-15T23:44:51.322 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-05-15T23:44:51.323 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-15T23:44:51.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump --format=json 2024-05-15T23:44:51.592 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:51.899 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:51 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/488180559' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:44:51.899 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:51 smithi084 ceph-mon[39784]: osdmap e845: 12 total, 12 up, 12 in 2024-05-15T23:44:52.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:51 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/488180559' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:44:52.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:51 smithi090 ceph-mon[44434]: osdmap e845: 12 total, 12 up, 12 in 2024-05-15T23:44:52.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:51 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/488180559' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-05-15T23:44:52.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:51 smithi088 ceph-mon[44699]: osdmap e845: 12 total, 12 up, 12 in 2024-05-15T23:44:52.829 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:52 smithi084 ceph-mon[39784]: pgmap v2251: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:53.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:52 smithi090 ceph-mon[44434]: pgmap v2251: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:53.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:52 smithi088 ceph-mon[44699]: pgmap v2251: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:53.611 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:44:53.612 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":845,"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","created":"2024-05-15T22:56:41.212493+0000","modified":"2024-05-15T23:44:50.623638+0000","last_up_change":"2024-05-15T23:42:38.545626+0000","last_in_change":"2024-05-15T23:43:07.654639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":33,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T22:59:38.603575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80159cc8-1b6c-4aa6-a114-966786d7be1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":355,"up_from":397,"up_thru":283,"down_at":356,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6801","nonce":3077815271}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6802","nonce":3077815271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6804","nonce":3077815271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6803","nonce":3077815271}]},"public_addr":"172.21.15.84:6801/3077815271","cluster_addr":"172.21.15.84:6802/3077815271","heartbeat_back_addr":"172.21.15.84:6804/3077815271","heartbeat_front_addr":"172.21.15.84:6803/3077815271","state":["exists","up"]},{"osd":1,"uuid":"9c2c76c9-ce0f-4c2d-9d96-c87a10187607","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":768,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6805","nonce":2140830229}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6806","nonce":2140830229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6808","nonce":2140830229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6807","nonce":2140830229}]},"public_addr":"172.21.15.84:6805/2140830229","cluster_addr":"172.21.15.84:6806/2140830229","heartbeat_back_addr":"172.21.15.84:6808/2140830229","heartbeat_front_addr":"172.21.15.84:6807/2140830229","state":["exists","up"]},{"osd":2,"uuid":"e8514daa-b5f9-4f2f-a432-53f4a4eb86bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":261,"up_from":291,"up_thru":685,"down_at":262,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6809","nonce":941507493}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6810","nonce":941507493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6812","nonce":941507493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6811","nonce":941507493}]},"public_addr":"172.21.15.84:6809/941507493","cluster_addr":"172.21.15.84:6810/941507493","heartbeat_back_addr":"172.21.15.84:6812/941507493","heartbeat_front_addr":"172.21.15.84:6811/941507493","state":["exists","up"]},{"osd":3,"uuid":"5d9965f9-c0b2-48ce-869c-582990985623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":758,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6813","nonce":2204039828}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6814","nonce":2204039828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6816","nonce":2204039828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6815","nonce":2204039828}]},"public_addr":"172.21.15.84:6813/2204039828","cluster_addr":"172.21.15.84:6814/2204039828","heartbeat_back_addr":"172.21.15.84:6816/2204039828","heartbeat_front_addr":"172.21.15.84:6815/2204039828","state":["exists","up"]},{"osd":4,"uuid":"4bc50523-8480-44d2-8d76-31479e839c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":549,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6800","nonce":2267955292}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":2267955292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":2267955292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":2267955292}]},"public_addr":"172.21.15.88:6800/2267955292","cluster_addr":"172.21.15.88:6801/2267955292","heartbeat_back_addr":"172.21.15.88:6803/2267955292","heartbeat_front_addr":"172.21.15.88:6802/2267955292","state":["exists","up"]},{"osd":5,"uuid":"d4db8d7f-afc7-43dc-81fe-f70c290d8b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":37,"last_clean_end":749,"up_from":797,"up_thru":737,"down_at":750,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":2238397473}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":2238397473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":2238397473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":2238397473}]},"public_addr":"172.21.15.88:6804/2238397473","cluster_addr":"172.21.15.88:6805/2238397473","heartbeat_back_addr":"172.21.15.88:6807/2238397473","heartbeat_front_addr":"172.21.15.88:6806/2238397473","state":["exists","up"]},{"osd":6,"uuid":"a2773454-2318-4a5b-8f31-0952b1b07f2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":808,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":380856869}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":380856869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":380856869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":380856869}]},"public_addr":"172.21.15.88:6808/380856869","cluster_addr":"172.21.15.88:6809/380856869","heartbeat_back_addr":"172.21.15.88:6811/380856869","heartbeat_front_addr":"172.21.15.88:6810/380856869","state":["exists","up"]},{"osd":7,"uuid":"028ca7b6-3e00-4466-a33f-b726cd60cb34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":769,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":1044709063}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":1044709063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":1044709063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":1044709063}]},"public_addr":"172.21.15.88:6812/1044709063","cluster_addr":"172.21.15.88:6813/1044709063","heartbeat_back_addr":"172.21.15.88:6815/1044709063","heartbeat_front_addr":"172.21.15.88:6814/1044709063","state":["exists","up"]},{"osd":8,"uuid":"02cab286-d487-4a66-982c-3cf28f55e851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":61,"last_clean_end":484,"up_from":520,"up_thru":737,"down_at":485,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6800","nonce":703598286}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6801","nonce":703598286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6803","nonce":703598286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6802","nonce":703598286}]},"public_addr":"172.21.15.90:6800/703598286","cluster_addr":"172.21.15.90:6801/703598286","heartbeat_back_addr":"172.21.15.90:6803/703598286","heartbeat_front_addr":"172.21.15.90:6802/703598286","state":["exists","up"]},{"osd":9,"uuid":"47d38bc7-39ba-4b35-af24-e282354dd613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":737,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6804","nonce":2794571133}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6805","nonce":2794571133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6807","nonce":2794571133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6806","nonce":2794571133}]},"public_addr":"172.21.15.90:6804/2794571133","cluster_addr":"172.21.15.90:6805/2794571133","heartbeat_back_addr":"172.21.15.90:6807/2794571133","heartbeat_front_addr":"172.21.15.90:6806/2794571133","state":["exists","up"]},{"osd":10,"uuid":"d7b2aeae-d012-41eb-9f91-e4c05c649473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":737,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6808","nonce":3735380744}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6809","nonce":3735380744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6811","nonce":3735380744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6810","nonce":3735380744}]},"public_addr":"172.21.15.90:6808/3735380744","cluster_addr":"172.21.15.90:6809/3735380744","heartbeat_back_addr":"172.21.15.90:6811/3735380744","heartbeat_front_addr":"172.21.15.90:6810/3735380744","state":["exists","up"]},{"osd":11,"uuid":"2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":800,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6812","nonce":2265615270}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6813","nonce":2265615270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6815","nonce":2265615270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6814","nonce":2265615270}]},"public_addr":"172.21.15.90:6812/2265615270","cluster_addr":"172.21.15.90:6813/2265615270","heartbeat_back_addr":"172.21.15.90:6815/2265615270","heartbeat_front_addr":"172.21.15.90:6814/2265615270","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:41.473703+0000","dead_epoch":355},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:18.807208+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:43:33.581529+0000","dead_epoch":261},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:43.461906+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:37.019668+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:43:45.988914+0000","dead_epoch":749},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:44.649984+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:27.067773+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:09.992313+0000","dead_epoch":484},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:26.683111+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:43:46.503965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:30.278854+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[6,3]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/2581408231":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/1136984393":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/2443201864":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1252671888":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1451001371":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/4084814866":"2024-05-16T22:57:38.505263+0000","172.21.15.84:0/4076409285":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/2988977694":"2024-05-16T22:57:38.505263+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T23:44:53.971 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:53 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/4260817908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:44:54.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:53 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/4260817908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:44:54.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:53 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/4260817908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:44:54.440 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-15T23:44:54.440 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd dump --format=json 2024-05-15T23:44:54.705 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:54.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:54 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:44:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:54 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:44:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:54 smithi084 ceph-mon[39784]: pgmap v2252: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:54.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:54 smithi084 ceph-mon[39784]: osdmap e846: 12 total, 12 up, 12 in 2024-05-15T23:44:55.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:54 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:44:55.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:54 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:44:55.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:54 smithi090 ceph-mon[44434]: pgmap v2252: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:55.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:54 smithi090 ceph-mon[44434]: osdmap e846: 12 total, 12 up, 12 in 2024-05-15T23:44:55.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:54 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:44:55.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:54 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:44:55.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:54 smithi088 ceph-mon[44699]: pgmap v2252: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:55.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:54 smithi088 ceph-mon[44699]: osdmap e846: 12 total, 12 up, 12 in 2024-05-15T23:44:56.714 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:44:56.714 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":846,"fsid":"3999fb7e-130e-11ef-bc99-c7b262605968","created":"2024-05-15T22:56:41.212493+0000","modified":"2024-05-15T23:44:53.635082+0000","last_up_change":"2024-05-15T23:42:38.545626+0000","last_in_change":"2024-05-15T23:43:07.654639+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":33,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T22:59:38.603575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80159cc8-1b6c-4aa6-a114-966786d7be1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":355,"up_from":397,"up_thru":283,"down_at":356,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6801","nonce":3077815271}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6802","nonce":3077815271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6804","nonce":3077815271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6803","nonce":3077815271}]},"public_addr":"172.21.15.84:6801/3077815271","cluster_addr":"172.21.15.84:6802/3077815271","heartbeat_back_addr":"172.21.15.84:6804/3077815271","heartbeat_front_addr":"172.21.15.84:6803/3077815271","state":["exists","up"]},{"osd":1,"uuid":"9c2c76c9-ce0f-4c2d-9d96-c87a10187607","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":768,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6805","nonce":2140830229}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6806","nonce":2140830229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6808","nonce":2140830229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6807","nonce":2140830229}]},"public_addr":"172.21.15.84:6805/2140830229","cluster_addr":"172.21.15.84:6806/2140830229","heartbeat_back_addr":"172.21.15.84:6808/2140830229","heartbeat_front_addr":"172.21.15.84:6807/2140830229","state":["exists","up"]},{"osd":2,"uuid":"e8514daa-b5f9-4f2f-a432-53f4a4eb86bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":261,"up_from":291,"up_thru":685,"down_at":262,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6809","nonce":941507493}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6810","nonce":941507493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6812","nonce":941507493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6811","nonce":941507493}]},"public_addr":"172.21.15.84:6809/941507493","cluster_addr":"172.21.15.84:6810/941507493","heartbeat_back_addr":"172.21.15.84:6812/941507493","heartbeat_front_addr":"172.21.15.84:6811/941507493","state":["exists","up"]},{"osd":3,"uuid":"5d9965f9-c0b2-48ce-869c-582990985623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":758,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6813","nonce":2204039828}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6814","nonce":2204039828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6816","nonce":2204039828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.84:6815","nonce":2204039828}]},"public_addr":"172.21.15.84:6813/2204039828","cluster_addr":"172.21.15.84:6814/2204039828","heartbeat_back_addr":"172.21.15.84:6816/2204039828","heartbeat_front_addr":"172.21.15.84:6815/2204039828","state":["exists","up"]},{"osd":4,"uuid":"4bc50523-8480-44d2-8d76-31479e839c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":549,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6800","nonce":2267955292}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":2267955292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":2267955292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":2267955292}]},"public_addr":"172.21.15.88:6800/2267955292","cluster_addr":"172.21.15.88:6801/2267955292","heartbeat_back_addr":"172.21.15.88:6803/2267955292","heartbeat_front_addr":"172.21.15.88:6802/2267955292","state":["exists","up"]},{"osd":5,"uuid":"d4db8d7f-afc7-43dc-81fe-f70c290d8b20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":37,"last_clean_end":749,"up_from":797,"up_thru":737,"down_at":750,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":2238397473}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":2238397473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":2238397473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":2238397473}]},"public_addr":"172.21.15.88:6804/2238397473","cluster_addr":"172.21.15.88:6805/2238397473","heartbeat_back_addr":"172.21.15.88:6807/2238397473","heartbeat_front_addr":"172.21.15.88:6806/2238397473","state":["exists","up"]},{"osd":6,"uuid":"a2773454-2318-4a5b-8f31-0952b1b07f2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":808,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":380856869}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":380856869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":380856869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":380856869}]},"public_addr":"172.21.15.88:6808/380856869","cluster_addr":"172.21.15.88:6809/380856869","heartbeat_back_addr":"172.21.15.88:6811/380856869","heartbeat_front_addr":"172.21.15.88:6810/380856869","state":["exists","up"]},{"osd":7,"uuid":"028ca7b6-3e00-4466-a33f-b726cd60cb34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":769,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":1044709063}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":1044709063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":1044709063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":1044709063}]},"public_addr":"172.21.15.88:6812/1044709063","cluster_addr":"172.21.15.88:6813/1044709063","heartbeat_back_addr":"172.21.15.88:6815/1044709063","heartbeat_front_addr":"172.21.15.88:6814/1044709063","state":["exists","up"]},{"osd":8,"uuid":"02cab286-d487-4a66-982c-3cf28f55e851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":61,"last_clean_end":484,"up_from":520,"up_thru":737,"down_at":485,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6800","nonce":703598286}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6801","nonce":703598286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6803","nonce":703598286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6802","nonce":703598286}]},"public_addr":"172.21.15.90:6800/703598286","cluster_addr":"172.21.15.90:6801/703598286","heartbeat_back_addr":"172.21.15.90:6803/703598286","heartbeat_front_addr":"172.21.15.90:6802/703598286","state":["exists","up"]},{"osd":9,"uuid":"47d38bc7-39ba-4b35-af24-e282354dd613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":737,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6804","nonce":2794571133}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6805","nonce":2794571133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6807","nonce":2794571133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6806","nonce":2794571133}]},"public_addr":"172.21.15.90:6804/2794571133","cluster_addr":"172.21.15.90:6805/2794571133","heartbeat_back_addr":"172.21.15.90:6807/2794571133","heartbeat_front_addr":"172.21.15.90:6806/2794571133","state":["exists","up"]},{"osd":10,"uuid":"d7b2aeae-d012-41eb-9f91-e4c05c649473","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":737,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6808","nonce":3735380744}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6809","nonce":3735380744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6811","nonce":3735380744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6810","nonce":3735380744}]},"public_addr":"172.21.15.90:6808/3735380744","cluster_addr":"172.21.15.90:6809/3735380744","heartbeat_back_addr":"172.21.15.90:6811/3735380744","heartbeat_front_addr":"172.21.15.90:6810/3735380744","state":["exists","up"]},{"osd":11,"uuid":"2ad7c53d-ca1c-43a8-9340-07fd7a5cd6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":800,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6812","nonce":2265615270}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6813","nonce":2265615270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6815","nonce":2265615270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.90:6814","nonce":2265615270}]},"public_addr":"172.21.15.90:6812/2265615270","cluster_addr":"172.21.15.90:6813/2265615270","heartbeat_back_addr":"172.21.15.90:6815/2265615270","heartbeat_front_addr":"172.21.15.90:6814/2265615270","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:41.473703+0000","dead_epoch":355},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:18.807208+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:52.759860+0000","dead_epoch":261},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:43.461906+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:37.019668+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:43:45.988914+0000","dead_epoch":749},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:44.649984+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:27.067773+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:09.992313+0000","dead_epoch":484},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:26.683111+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:43:46.503965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T23:44:30.278854+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[6,3]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/2581408231":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/1136984393":"2024-05-16T22:57:38.505263+0000","172.21.15.84:6800/2443201864":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1252671888":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/1451001371":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/4084814866":"2024-05-16T22:57:38.505263+0000","172.21.15.84:0/4076409285":"2024-05-16T22:57:09.403385+0000","172.21.15.84:0/2988977694":"2024-05-16T22:57:38.505263+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T23:44:57.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:56 smithi090 ceph-mon[44434]: pgmap v2254: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:57.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:56 smithi088 ceph-mon[44699]: pgmap v2254: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:57.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:56 smithi084 ceph-mon[39784]: pgmap v2254: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:57.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-15T23:44:57.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-15T23:44:57.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-15T23:44:57.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-15T23:44:57.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-15T23:44:57.420 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-15T23:44:57.420 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-15T23:44:57.421 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-15T23:44:57.421 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-15T23:44:57.421 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-15T23:44:57.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-15T23:44:57.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-15T23:44:57.725 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:57 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/177530722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:44:58.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:57 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/177530722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:44:58.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:57 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/177530722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T23:44:58.160 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:58.170 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:58.170 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:58.178 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:58.185 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:58.188 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:58.244 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:58.244 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:44:59.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:44:58 smithi090 ceph-mon[44434]: pgmap v2255: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:59.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:44:58 smithi088 ceph-mon[44699]: pgmap v2255: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:44:59.080 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:44:58 smithi084 ceph-mon[39784]: pgmap v2255: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:00.106 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:00.111 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:00.968 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:00 smithi084 ceph-mon[39784]: pgmap v2256: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:01.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:00 smithi090 ceph-mon[44434]: pgmap v2256: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:01.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:00 smithi088 ceph-mon[44699]: pgmap v2256: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:03.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:02 smithi090 ceph-mon[44434]: pgmap v2257: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:03.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:02 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:45:03.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:02 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:45:03.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:02 smithi090 ceph-mon[44434]: osdmap e847: 12 total, 12 up, 12 in 2024-05-15T23:45:03.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:02 smithi088 ceph-mon[44699]: pgmap v2257: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:03.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:02 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:45:03.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:02 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:45:03.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:02 smithi088 ceph-mon[44699]: osdmap e847: 12 total, 12 up, 12 in 2024-05-15T23:45:03.109 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:02 smithi084 ceph-mon[39784]: pgmap v2257: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:03.109 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:02 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:45:03.109 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:02 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:45:03.110 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:02 smithi084 ceph-mon[39784]: osdmap e847: 12 total, 12 up, 12 in 2024-05-15T23:45:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:03 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:45:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:03 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:45:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:03 smithi084 ceph-mon[39784]: 1.0 scrub starts 2024-05-15T23:45:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:03 smithi084 ceph-mon[39784]: 1.0 scrub ok 2024-05-15T23:45:03.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:03 smithi084 ceph-mon[39784]: osdmap e848: 12 total, 12 up, 12 in 2024-05-15T23:45:04.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:03 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:45:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:03 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:45:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:03 smithi088 ceph-mon[44699]: 1.0 scrub starts 2024-05-15T23:45:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:03 smithi088 ceph-mon[44699]: 1.0 scrub ok 2024-05-15T23:45:04.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:03 smithi088 ceph-mon[44699]: osdmap e848: 12 total, 12 up, 12 in 2024-05-15T23:45:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:03 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:45:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:03 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:45:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:03 smithi090 ceph-mon[44434]: 1.0 scrub starts 2024-05-15T23:45:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:03 smithi090 ceph-mon[44434]: 1.0 scrub ok 2024-05-15T23:45:04.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:03 smithi090 ceph-mon[44434]: osdmap e848: 12 total, 12 up, 12 in 2024-05-15T23:45:05.048 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:04 smithi084 ceph-mon[39784]: pgmap v2260: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:05.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:04 smithi088 ceph-mon[44699]: pgmap v2260: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:05.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:04 smithi090 ceph-mon[44434]: pgmap v2260: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:06.830 INFO:teuthology.orchestra.run.smithi084.stdout:55834575398 2024-05-15T23:45:06.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-15T23:45:06.913 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:06.914 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:07.052 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:06 smithi084 ceph-mon[39784]: pgmap v2261: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:07.071 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:06 smithi088 ceph-mon[44699]: pgmap v2261: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:07.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:06 smithi090 ceph-mon[44434]: pgmap v2261: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:08.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:08 smithi084 ceph-mon[39784]: pgmap v2262: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:09.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:08 smithi088 ceph-mon[44699]: pgmap v2262: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:09.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:08 smithi090 ceph-mon[44434]: pgmap v2262: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:10.948 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:10 smithi084 ceph-mon[39784]: pgmap v2263: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:11.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:10 smithi088 ceph-mon[44699]: pgmap v2263: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:11.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:10 smithi090 ceph-mon[44434]: pgmap v2263: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:13.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:12 smithi088 ceph-mon[44699]: pgmap v2264: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:13.081 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:12 smithi084 ceph-mon[39784]: pgmap v2264: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:13.082 INFO:teuthology.orchestra.run.smithi084.stdout:317827580405 2024-05-15T23:45:13.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-15T23:45:13.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:12 smithi090 ceph-mon[44434]: pgmap v2264: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:15.070 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:14 smithi088 ceph-mon[44699]: pgmap v2265: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:15.076 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:14 smithi084 ceph-mon[39784]: pgmap v2265: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:15.077 INFO:teuthology.orchestra.run.smithi084.stdout:2233382994142 2024-05-15T23:45:15.078 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-15T23:45:15.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:14 smithi090 ceph-mon[44434]: pgmap v2265: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:16.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:16 smithi084 ceph-mon[39784]: pgmap v2266: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:16.869 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:16 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:45:17.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:16 smithi090 ceph-mon[44434]: pgmap v2266: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:17.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:16 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:45:17.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:16 smithi088 ceph-mon[44699]: pgmap v2266: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:17.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:16 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:45:18.595 INFO:teuthology.orchestra.run.smithi084.stdout:197568496143 2024-05-15T23:45:18.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-15T23:45:19.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:19 smithi090 ceph-mon[44434]: pgmap v2267: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:19.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:19 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:45:19.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:19 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:45:19.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:19 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:45:19.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:19 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:45:19.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:19 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:45:19.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:19 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:45:19.303 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:19 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:45:19.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:19 smithi088 ceph-mon[44699]: pgmap v2267: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:19.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:19 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:45:19.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:19 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:45:19.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:19 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:45:19.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:19 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:45:19.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:19 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:45:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:19 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:45:19.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:19 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:45:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:19 smithi084 ceph-mon[39784]: pgmap v2267: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:19 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:45:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:19 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:45:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:19 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:45:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:19 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:45:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:45:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:19 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:45:19.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:19 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:45:20.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:20 smithi090 ceph-mon[44434]: osdmap e849: 12 total, 12 up, 12 in 2024-05-15T23:45:20.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:20 smithi088 ceph-mon[44699]: osdmap e849: 12 total, 12 up, 12 in 2024-05-15T23:45:20.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:20 smithi084 ceph-mon[39784]: osdmap e849: 12 total, 12 up, 12 in 2024-05-15T23:45:20.532 INFO:teuthology.orchestra.run.smithi084.stdout:1705102016811 2024-05-15T23:45:20.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-15T23:45:20.815 INFO:teuthology.orchestra.run.smithi084.stdout:347892351476 2024-05-15T23:45:20.815 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-15T23:45:21.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:21 smithi084 ceph-mon[39784]: pgmap v2269: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:21.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:21 smithi090 ceph-mon[44434]: pgmap v2269: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:21.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:21 smithi088 ceph-mon[44699]: pgmap v2269: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:21.912 INFO:teuthology.orchestra.run.smithi084.stdout:133143986711 2024-05-15T23:45:21.913 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-15T23:45:22.701 INFO:teuthology.orchestra.run.smithi084.stdout:3423088934947 2024-05-15T23:45:22.702 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-15T23:45:23.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:23 smithi084 ceph-mon[39784]: pgmap v2270: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:23.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:23 smithi090 ceph-mon[44434]: pgmap v2270: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:23.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:23 smithi088 ceph-mon[44699]: pgmap v2270: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:23.582 INFO:teuthology.orchestra.run.smithi084.stdout:111669150240 2024-05-15T23:45:23.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-15T23:45:23.876 INFO:teuthology.orchestra.run.smithi084.stdout:292057776640 2024-05-15T23:45:23.876 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-15T23:45:24.077 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:24.088 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:24.088 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:24.641 INFO:teuthology.orchestra.run.smithi084.stdout:1249835483497 2024-05-15T23:45:24.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-15T23:45:25.208 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:25 smithi084 ceph-mon[39784]: pgmap v2271: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:25.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:25 smithi090 ceph-mon[44434]: pgmap v2271: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:25.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:25 smithi088 ceph-mon[44699]: pgmap v2271: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:25.842 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:25.845 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:25.849 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:27.254 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:27 smithi084 ceph-mon[39784]: pgmap v2272: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:27.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:27 smithi090 ceph-mon[44434]: pgmap v2272: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:27.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:27 smithi088 ceph-mon[44699]: pgmap v2272: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:28.025 INFO:teuthology.orchestra.run.smithi084.stdout:227633267213 2024-05-15T23:45:28.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-15T23:45:28.250 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:29.302 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:29 smithi090 ceph-mon[44434]: pgmap v2273: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:29.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:29 smithi088 ceph-mon[44699]: pgmap v2273: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:29.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:29 smithi084 ceph-mon[39784]: pgmap v2273: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:30.316 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:30 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:45:30.316 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:30 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:45:30.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:30 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:45:30.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:30 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:45:30.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:30 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:45:30.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:30 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:45:30.673 INFO:teuthology.orchestra.run.smithi084.stdout:2233382994146 2024-05-15T23:45:31.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:31 smithi088 ceph-mon[44699]: pgmap v2274: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:31 smithi088 ceph-mon[44699]: osdmap e850: 12 total, 12 up, 12 in 2024-05-15T23:45:31.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:31 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3414840160' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-15T23:45:31.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:31 smithi084 ceph-mon[39784]: pgmap v2274: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:31.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:31 smithi084 ceph-mon[39784]: osdmap e850: 12 total, 12 up, 12 in 2024-05-15T23:45:31.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:31 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3414840160' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-15T23:45:31.467 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:31.467 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:31.467 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:31 smithi090 ceph-mon[44434]: pgmap v2274: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:31 smithi090 ceph-mon[44434]: osdmap e850: 12 total, 12 up, 12 in 2024-05-15T23:45:31.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:31 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3414840160' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-15T23:45:32.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:32 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:45:32.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:32 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:45:32.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:32 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:45:32.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:32 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:45:32.552 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2233382994142 got 2233382994146 for osd.8 2024-05-15T23:45:32.552 DEBUG:teuthology.parallel:result is None 2024-05-15T23:45:32.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:32 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:45:32.553 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:32 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:45:33.320 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:33 smithi088 ceph-mon[44699]: pgmap v2276: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:33.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:33 smithi088 ceph-mon[44699]: osdmap e851: 12 total, 12 up, 12 in 2024-05-15T23:45:33.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:33 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:45:33.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:33 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:45:33.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:33 smithi084 ceph-mon[39784]: pgmap v2276: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:33.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:33 smithi084 ceph-mon[39784]: osdmap e851: 12 total, 12 up, 12 in 2024-05-15T23:45:33.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:33 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:45:33.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:33 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:45:33.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:33 smithi090 ceph-mon[44434]: pgmap v2276: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:33.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:33 smithi090 ceph-mon[44434]: osdmap e851: 12 total, 12 up, 12 in 2024-05-15T23:45:33.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:33 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:45:33.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:33 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:45:34.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:34 smithi084 ceph-mon[39784]: osdmap e852: 12 total, 12 up, 12 in 2024-05-15T23:45:34.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:34 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/515268097' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-15T23:45:34.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:34 smithi090 ceph-mon[44434]: osdmap e852: 12 total, 12 up, 12 in 2024-05-15T23:45:34.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:34 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/515268097' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-15T23:45:34.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:34 smithi088 ceph-mon[44699]: osdmap e852: 12 total, 12 up, 12 in 2024-05-15T23:45:34.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:34 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/515268097' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-15T23:45:34.967 INFO:teuthology.orchestra.run.smithi084.stdout:55834575403 2024-05-15T23:45:35.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:35 smithi084 ceph-mon[39784]: pgmap v2279: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:35.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:35 smithi090 ceph-mon[44434]: pgmap v2279: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:35.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:35 smithi088 ceph-mon[44699]: pgmap v2279: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:36.300 INFO:teuthology.orchestra.run.smithi084.stdout:197568496146 2024-05-15T23:45:37.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:37 smithi084 ceph-mon[39784]: pgmap v2280: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:37.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:37 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3662775127' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-15T23:45:37.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:37 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/3749650694' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-15T23:45:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:37 smithi090 ceph-mon[44434]: pgmap v2280: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:37 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3662775127' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-15T23:45:37.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:37 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/3749650694' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-15T23:45:37.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:37 smithi088 ceph-mon[44699]: pgmap v2280: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:37 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3662775127' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-15T23:45:37.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:37 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/3749650694' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-15T23:45:38.546 INFO:teuthology.orchestra.run.smithi084.stdout:1249835483502 2024-05-15T23:45:39.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:39 smithi084 ceph-mon[39784]: pgmap v2281: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:39.142 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1249835483497 got 1249835483502 for osd.2 2024-05-15T23:45:39.143 DEBUG:teuthology.parallel:result is None 2024-05-15T23:45:39.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:39 smithi090 ceph-mon[44434]: pgmap v2281: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:39.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:39 smithi088 ceph-mon[44699]: pgmap v2281: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:39.586 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575398 got 55834575403 for osd.1 2024-05-15T23:45:39.586 DEBUG:teuthology.parallel:result is None 2024-05-15T23:45:40.120 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568496143 got 197568496146 for osd.6 2024-05-15T23:45:40.120 DEBUG:teuthology.parallel:result is None 2024-05-15T23:45:40.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:40 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:45:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:40 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:45:40.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:40 smithi084 ceph-mon[39784]: osdmap e853: 12 total, 12 up, 12 in 2024-05-15T23:45:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:40 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:45:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:40 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:45:40.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:40 smithi090 ceph-mon[44434]: osdmap e853: 12 total, 12 up, 12 in 2024-05-15T23:45:40.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:40 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:45:40.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:40 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:45:40.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:40 smithi088 ceph-mon[44699]: osdmap e853: 12 total, 12 up, 12 in 2024-05-15T23:45:41.041 INFO:teuthology.orchestra.run.smithi084.stdout:1705102016815 2024-05-15T23:45:41.212 INFO:teuthology.orchestra.run.smithi084.stdout:347892351480 2024-05-15T23:45:41.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:41 smithi084 ceph-mon[39784]: pgmap v2283: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:41.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:41 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/668432871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-15T23:45:41.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:41 smithi090 ceph-mon[44434]: pgmap v2283: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:41.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:41 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/668432871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-15T23:45:41.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:41 smithi088 ceph-mon[44699]: pgmap v2283: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:41.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:41 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/668432871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-15T23:45:41.778 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1705102016811 got 1705102016815 for osd.0 2024-05-15T23:45:41.779 DEBUG:teuthology.parallel:result is None 2024-05-15T23:45:42.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:42 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2511845756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-15T23:45:42.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:42 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2511845756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-15T23:45:42.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:42 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2511845756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-15T23:45:43.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:43 smithi084 ceph-mon[39784]: pgmap v2284: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:43.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:43 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:45:43.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:43 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:45:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:43 smithi090 ceph-mon[44434]: pgmap v2284: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:43 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:45:43.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:43 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:45:43.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:43 smithi088 ceph-mon[44699]: pgmap v2284: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:43.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:43 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:45:43.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:43 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:45:43.575 INFO:teuthology.orchestra.run.smithi084.stdout:133143986716 2024-05-15T23:45:43.803 INFO:teuthology.orchestra.run.smithi084.stdout:3423088934951 2024-05-15T23:45:44.010 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:44.010 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:44.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:44 smithi084 ceph-mon[39784]: osdmap e854: 12 total, 12 up, 12 in 2024-05-15T23:45:44.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1157017825' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-15T23:45:44.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:44 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/178927237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-15T23:45:44.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:44 smithi090 ceph-mon[44434]: osdmap e854: 12 total, 12 up, 12 in 2024-05-15T23:45:44.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1157017825' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-15T23:45:44.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:44 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/178927237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-15T23:45:44.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:44 smithi088 ceph-mon[44699]: osdmap e854: 12 total, 12 up, 12 in 2024-05-15T23:45:44.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1157017825' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-15T23:45:44.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:44 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/178927237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-15T23:45:45.200 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986711 got 133143986716 for osd.4 2024-05-15T23:45:45.200 DEBUG:teuthology.parallel:result is None 2024-05-15T23:45:45.294 INFO:teuthology.orchestra.run.smithi084.stdout:111669150245 2024-05-15T23:45:45.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:45 smithi084 ceph-mon[39784]: pgmap v2286: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:45.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:45 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/896172462' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-15T23:45:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:45 smithi090 ceph-mon[44434]: pgmap v2286: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:45.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:45 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/896172462' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-15T23:45:45.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:45 smithi088 ceph-mon[44699]: pgmap v2286: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:45.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:45 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/896172462' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-15T23:45:47.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:47 smithi084 ceph-mon[39784]: pgmap v2287: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:47.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:47 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:45:47.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:47 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:45:47.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:47 smithi084 ceph-mon[39784]: osdmap e855: 12 total, 12 up, 12 in 2024-05-15T23:45:47.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:47 smithi084 ceph-mon[39784]: purged_snaps scrub starts 2024-05-15T23:45:47.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:47 smithi084 ceph-mon[39784]: purged_snaps scrub ok 2024-05-15T23:45:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:47 smithi090 ceph-mon[44434]: pgmap v2287: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:47 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:45:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:47 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:45:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:47 smithi090 ceph-mon[44434]: osdmap e855: 12 total, 12 up, 12 in 2024-05-15T23:45:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:47 smithi090 ceph-mon[44434]: purged_snaps scrub starts 2024-05-15T23:45:47.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:47 smithi090 ceph-mon[44434]: purged_snaps scrub ok 2024-05-15T23:45:47.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:47 smithi088 ceph-mon[44699]: pgmap v2287: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:47 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:45:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:47 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:45:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:47 smithi088 ceph-mon[44699]: osdmap e855: 12 total, 12 up, 12 in 2024-05-15T23:45:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:47 smithi088 ceph-mon[44699]: purged_snaps scrub starts 2024-05-15T23:45:47.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:47 smithi088 ceph-mon[44699]: purged_snaps scrub ok 2024-05-15T23:45:47.668 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669150240 got 111669150245 for osd.3 2024-05-15T23:45:47.669 DEBUG:teuthology.parallel:result is None 2024-05-15T23:45:48.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:48 smithi084 ceph-mon[39784]: osdmap e856: 12 total, 12 up, 12 in 2024-05-15T23:45:48.411 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3423088934947 got 3423088934951 for osd.5 2024-05-15T23:45:48.411 DEBUG:teuthology.parallel:result is None 2024-05-15T23:45:48.549 INFO:teuthology.orchestra.run.smithi084.stdout:317827580414 2024-05-15T23:45:48.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:48 smithi090 ceph-mon[44434]: osdmap e856: 12 total, 12 up, 12 in 2024-05-15T23:45:48.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:48 smithi088 ceph-mon[44699]: osdmap e856: 12 total, 12 up, 12 in 2024-05-15T23:45:49.241 INFO:teuthology.orchestra.run.smithi084.stdout:292057776646 2024-05-15T23:45:49.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:49 smithi090 ceph-mon[44434]: pgmap v2290: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:49.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:49 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1600834906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-15T23:45:49.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:49 smithi084 ceph-mon[39784]: pgmap v2290: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:49.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:49 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1600834906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-15T23:45:49.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:49 smithi088 ceph-mon[44699]: pgmap v2290: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:49.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:49 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1600834906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-15T23:45:49.584 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351476 got 347892351480 for osd.11 2024-05-15T23:45:49.584 DEBUG:teuthology.parallel:result is None 2024-05-15T23:45:50.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:50 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/1141216127' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-15T23:45:50.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:50 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/1141216127' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-15T23:45:50.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:50 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/1141216127' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-15T23:45:50.690 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827580405 got 317827580414 for osd.10 2024-05-15T23:45:50.690 DEBUG:teuthology.parallel:result is None 2024-05-15T23:45:51.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:51 smithi084 ceph-mon[39784]: pgmap v2291: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:51.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:51 smithi090 ceph-mon[44434]: pgmap v2291: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:51.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:51 smithi088 ceph-mon[44699]: pgmap v2291: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:51.602 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776640 got 292057776646 for osd.9 2024-05-15T23:45:51.602 DEBUG:teuthology.parallel:result is None 2024-05-15T23:45:52.224 INFO:teuthology.orchestra.run.smithi084.stdout:227633267218 2024-05-15T23:45:52.948 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633267213 got 227633267218 for osd.7 2024-05-15T23:45:52.948 DEBUG:teuthology.parallel:result is None 2024-05-15T23:45:52.948 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-05-15T23:45:52.948 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:45:53.216 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:53.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:53 smithi084 ceph-mon[39784]: pgmap v2292: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:53.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:53 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/2558904483' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-15T23:45:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:53 smithi090 ceph-mon[44434]: pgmap v2292: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:53.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:53 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/2558904483' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-15T23:45:53.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:53 smithi088 ceph-mon[44699]: pgmap v2292: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:53.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:53 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/2558904483' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-15T23:45:55.366 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:45:55.369 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:45:55.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:55 smithi090 ceph-mon[44434]: pgmap v2293: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:55.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:55 smithi088 ceph-mon[44699]: pgmap v2293: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:55.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:55 smithi084 ceph-mon[39784]: pgmap v2293: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:56.018 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":2293,"stamp":"2024-05-15T23:45:53.439267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":6587720,"kb_used_data":10592,"kb_used_omap":211,"kb_used_meta":6576748,"kb_avail":1118354104,"statfs":{"total":1151940427776,"available":1145194602496,"internally_reserved":0,"allocated":10846208,"data_stored":5610528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":216769,"internal_metadata":6734590271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001644"},"pg_stats":[{"pgid":"1.0","version":"57'94","reported_seq":131,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-05-15T23:45:47.139455+0000","last_change":"2024-05-15T23:45:02.420461+0000","last_active":"2024-05-15T23:45:47.139455+0000","last_peered":"2024-05-15T23:45:47.139455+0000","last_clean":"2024-05-15T23:45:47.139455+0000","last_became_active":"2024-05-15T23:43:05.657502+0000","last_became_peered":"2024-05-15T23:43:05.657502+0000","last_unstale":"2024-05-15T23:45:47.139455+0000","last_undegraded":"2024-05-15T23:45:47.139455+0000","last_fullsized":"2024-05-15T23:45:47.139455+0000","mapping_epoch":807,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":809,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:45:02.420348+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:45:02.420348+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:46:19.218927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351483,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":912168,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":92832984,"statfs":{"total":95995035648,"available":95060975616,"internally_reserved":0,"allocated":815104,"data_stored":379371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":933231044},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580415,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":645612,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93099540,"statfs":{"total":95995035648,"available":95333928960,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":660273600},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776647,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":888556,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92856596,"statfs":{"total":95995035648,"available":95085154304,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":909048256},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":520,"seq":2233382994150,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":150184,"kb_used_data":796,"kb_used_omap":24,"kb_used_meta":149351,"kb_avail":93594968,"statfs":{"total":95995035648,"available":95841247232,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25357,"internal_metadata":152935667},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267218,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":758056,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92987096,"statfs":{"total":95995035648,"available":95218786304,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":775420362},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181044,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93564108,"statfs":{"total":95995035648,"available":95809646592,"internally_reserved":0,"allocated":1347584,"data_stored":908319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":184023494},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575408,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":952424,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92792728,"statfs":{"total":95995035648,"available":95019753472,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":974453190},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":397,"seq":1705102016818,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":38440,"kb_used_data":796,"kb_used_omap":36,"kb_used_meta":37595,"kb_avail":93706712,"statfs":{"total":95995035648,"available":95955673088,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37643,"internal_metadata":38497525},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483505,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":209448,"kb_used_data":796,"kb_used_omap":4,"kb_used_meta":208635,"kb_avail":93535704,"statfs":{"total":95995035648,"available":95780560896,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":213642947},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150247,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1025140,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":1023806,"kb_avail":92720012,"statfs":{"total":95995035648,"available":94945292288,"internally_reserved":0,"allocated":1347584,"data_stored":908319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1048377799},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":744172,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93000980,"statfs":{"total":95995035648,"available":95233003520,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":797,"seq":3423088934953,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":82476,"kb_used_data":796,"kb_used_omap":133,"kb_used_meta":81530,"kb_avail":93662676,"statfs":{"total":95995035648,"available":95910580224,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":136594,"internal_metadata":83487342},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:45:56.019 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph pg dump --format=json 2024-05-15T23:45:56.286 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:56.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:56 smithi084 ceph-mon[39784]: from='client.18036 v1:172.21.15.84:0/1676299367' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:45:56.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:56 smithi090 ceph-mon[44434]: from='client.18036 v1:172.21.15.84:0/1676299367' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:45:56.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:56 smithi088 ceph-mon[44699]: from='client.18036 v1:172.21.15.84:0/1676299367' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:45:57.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:57 smithi090 ceph-mon[44434]: pgmap v2294: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:57.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:57 smithi088 ceph-mon[44699]: pgmap v2294: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:57.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:57 smithi084 ceph-mon[39784]: pgmap v2294: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:58.335 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-05-15T23:45:58.338 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-05-15T23:45:59.017 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":2295,"stamp":"2024-05-15T23:45:57.440248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":6587720,"kb_used_data":10592,"kb_used_omap":211,"kb_used_meta":6576748,"kb_avail":1118354104,"statfs":{"total":1151940427776,"available":1145194602496,"internally_reserved":0,"allocated":10846208,"data_stored":5610528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":216769,"internal_metadata":6734590271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.301632"},"pg_stats":[{"pgid":"1.0","version":"57'94","reported_seq":131,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-05-15T23:45:47.139455+0000","last_change":"2024-05-15T23:45:02.420461+0000","last_active":"2024-05-15T23:45:47.139455+0000","last_peered":"2024-05-15T23:45:47.139455+0000","last_clean":"2024-05-15T23:45:47.139455+0000","last_became_active":"2024-05-15T23:43:05.657502+0000","last_became_peered":"2024-05-15T23:43:05.657502+0000","last_unstale":"2024-05-15T23:45:47.139455+0000","last_undegraded":"2024-05-15T23:45:47.139455+0000","last_fullsized":"2024-05-15T23:45:47.139455+0000","mapping_epoch":807,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":809,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'94","last_scrub_stamp":"2024-05-15T23:45:02.420348+0000","last_deep_scrub":"57'94","last_deep_scrub_stamp":"2024-05-15T23:02:23.944154+0000","last_clean_scrub_stamp":"2024-05-15T23:45:02.420348+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-15T23:46:19.218927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351483,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":912168,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":92832984,"statfs":{"total":95995035648,"available":95060975616,"internally_reserved":0,"allocated":815104,"data_stored":379371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":933231044},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827580416,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":645612,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93099540,"statfs":{"total":95995035648,"available":95333928960,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":660273600},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776648,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":888556,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92856596,"statfs":{"total":95995035648,"available":95085154304,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":909048256},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":520,"seq":2233382994151,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":150184,"kb_used_data":796,"kb_used_omap":24,"kb_used_meta":149351,"kb_avail":93594968,"statfs":{"total":95995035648,"available":95841247232,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25357,"internal_metadata":152935667},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":758056,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92987096,"statfs":{"total":95995035648,"available":95218786304,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":775420362},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181044,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93564108,"statfs":{"total":95995035648,"available":95809646592,"internally_reserved":0,"allocated":1347584,"data_stored":908319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":184023494},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575409,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":952424,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92792728,"statfs":{"total":95995035648,"available":95019753472,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":974453190},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":397,"seq":1705102016819,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":38440,"kb_used_data":796,"kb_used_omap":36,"kb_used_meta":37595,"kb_avail":93706712,"statfs":{"total":95995035648,"available":95955673088,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37643,"internal_metadata":38497525},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":291,"seq":1249835483506,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":209448,"kb_used_data":796,"kb_used_omap":4,"kb_used_meta":208635,"kb_avail":93535704,"statfs":{"total":95995035648,"available":95780560896,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4413,"internal_metadata":213642947},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150247,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1025140,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":1023806,"kb_avail":92720012,"statfs":{"total":95995035648,"available":94945292288,"internally_reserved":0,"allocated":1347584,"data_stored":908319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1048377799},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986719,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":744172,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93000980,"statfs":{"total":95995035648,"available":95233003520,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":797,"seq":3423088934954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":82476,"kb_used_data":796,"kb_used_omap":133,"kb_used_meta":81530,"kb_avail":93662676,"statfs":{"total":95995035648,"available":95910580224,"internally_reserved":0,"allocated":815104,"data_stored":379391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":136594,"internal_metadata":83487342},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T23:45:59.018 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-05-15T23:45:59.018 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6cd0f8013e2dea00c3a29a0d8b10656b132d7c80 shell --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-05-15T23:45:59.287 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/mon.a/config 2024-05-15T23:45:59.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:59 smithi084 ceph-mon[39784]: pgmap v2295: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:59.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:45:59 smithi084 ceph-mon[39784]: from='client.18042 v1:172.21.15.84:0/2822051175' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:45:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:59 smithi090 ceph-mon[44434]: pgmap v2295: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:59.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:45:59 smithi090 ceph-mon[44434]: from='client.18042 v1:172.21.15.84:0/2822051175' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:45:59.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:59 smithi088 ceph-mon[44699]: pgmap v2295: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:45:59.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:45:59 smithi088 ceph-mon[44699]: from='client.18042 v1:172.21.15.84:0/2822051175' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T23:46:01.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:46:01 smithi090 ceph-mon[44434]: pgmap v2296: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:46:01.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:46:01 smithi088 ceph-mon[44699]: pgmap v2296: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:46:01.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:46:01 smithi084 ceph-mon[39784]: pgmap v2296: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-15T23:46:01.987 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-15T23:46:01.997 INFO:tasks.cephadm:Teardown begin 2024-05-15T23:46:01.998 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T23:46:02.051 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T23:46:02.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T23:46:02.247 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-15T23:46:02.247 DEBUG:teuthology.orchestra.run.smithi084:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-15T23:46:02.278 DEBUG:teuthology.orchestra.run.smithi088:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-15T23:46:02.305 DEBUG:teuthology.orchestra.run.smithi090:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-15T23:46:02.332 INFO:tasks.cephadm:Stopping all daemons... 2024-05-15T23:46:02.332 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-15T23:46:02.332 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mon.a 2024-05-15T23:46:02.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:46:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/154700260' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-05-15T23:46:02.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:46:02 smithi084 ceph-mon[39784]: from='client.? v1:172.21.15.84:0/154700260' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-05-15T23:46:02.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:46:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:46:02.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:46:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:46:02.358 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:46:02 smithi084 ceph-mon[39784]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:46:02.358 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:46:02 smithi084 ceph-mon[39784]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:46:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:46:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/154700260' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-05-15T23:46:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:46:02 smithi090 ceph-mon[44434]: from='client.? v1:172.21.15.84:0/154700260' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-05-15T23:46:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:46:02 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:46:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:46:02 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:46:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:46:02 smithi090 ceph-mon[44434]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:46:02.552 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:46:02 smithi090 ceph-mon[44434]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:46:02.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:46:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/154700260' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-05-15T23:46:02.570 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:46:02 smithi088 ceph-mon[44699]: from='client.? v1:172.21.15.84:0/154700260' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-05-15T23:46:02.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:46:02 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T23:46:02.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:46:02 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T23:46:02.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:46:02 smithi088 ceph-mon[44699]: from='mgr.14152 v1:172.21.15.84:0/3766436449' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T23:46:02.571 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:46:02 smithi088 ceph-mon[44699]: from='mgr.14152 ' entity='mgr.y' 2024-05-15T23:46:02.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:46:02 smithi084 systemd[1]: Stopping Ceph mon.a for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:46:02.994 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:46:02 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:46:02.623+0000 7fa0990c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:46:02.995 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:46:02 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a[39761]: 2024-05-15T23:46:02.623+0000 7fa0990c2700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-15T23:46:03.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:May 15 23:46:03 smithi084 bash[329604]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-a 2024-05-15T23:46:03.520 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mon.a.service' 2024-05-15T23:46:03.602 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:46:03.602 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-15T23:46:03.602 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-15T23:46:03.602 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mon.b 2024-05-15T23:46:03.928 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:46:03 smithi088 systemd[1]: Stopping Ceph mon.b for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:46:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:46:03 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-b[44676]: 2024-05-15T23:46:03.925+0000 7f41cc525700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:46:04.321 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:46:03 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-b[44676]: 2024-05-15T23:46:03.925+0000 7f41cc525700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-15T23:46:04.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:May 15 23:46:04 smithi088 bash[178790]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-b 2024-05-15T23:46:04.983 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mon.b.service' 2024-05-15T23:46:05.024 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:46:05.024 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-15T23:46:05.024 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-15T23:46:05.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mon.c 2024-05-15T23:46:05.333 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:46:05 smithi090 systemd[1]: Stopping Ceph mon.c for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:46:05.692 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:46:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-c[44404]: 2024-05-15T23:46:05.330+0000 7f1d7034f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:46:05.693 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:46:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-c[44404]: 2024-05-15T23:46:05.330+0000 7f1d7034f700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-15T23:46:06.052 INFO:journalctl@ceph.mon.c.smithi090.stdout:May 15 23:46:05 smithi090 bash[178721]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mon-c 2024-05-15T23:46:06.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mon.c.service' 2024-05-15T23:46:06.350 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:46:06.350 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-15T23:46:06.350 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-15T23:46:06.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mgr.y 2024-05-15T23:46:06.869 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 23:46:06 smithi084 systemd[1]: Stopping Ceph mgr.y for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:46:06.869 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 23:46:06 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y[40035]: 2024-05-15T23:46:06.595+0000 7fcee690f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:46:07.368 INFO:journalctl@ceph.mgr.y.smithi084.stdout:May 15 23:46:07 smithi084 bash[329750]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-y 2024-05-15T23:46:07.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mgr.y.service' 2024-05-15T23:46:07.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:46:07.625 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-15T23:46:07.625 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-15T23:46:07.626 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mgr.x 2024-05-15T23:46:08.070 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 23:46:07 smithi088 systemd[1]: Stopping Ceph mgr.x for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:46:08.503 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 23:46:08 smithi088 bash[178949]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-mgr-x 2024-05-15T23:46:08.691 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 23:46:08 smithi088 systemd[1]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-05-15T23:46:08.691 INFO:journalctl@ceph.mgr.x.smithi088.stdout:May 15 23:46:08 smithi088 systemd[1]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-05-15T23:46:08.693 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@mgr.x.service' 2024-05-15T23:46:08.729 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:46:08.729 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-15T23:46:08.729 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-15T23:46:08.729 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.0 2024-05-15T23:46:09.118 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:46:08 smithi084 systemd[1]: Stopping Ceph osd.0 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:46:09.119 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:46:09 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-0[188475]: 2024-05-15T23:46:09.005+0000 7f66d1775700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:46:09.119 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:46:09 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-0[188475]: 2024-05-15T23:46:09.005+0000 7f66d1775700 -1 osd.0 856 *** Got signal Terminated *** 2024-05-15T23:46:09.119 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:46:09 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-0[188475]: 2024-05-15T23:46:09.005+0000 7f66d1775700 -1 osd.0 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:46:15.119 INFO:journalctl@ceph.osd.0.smithi084.stdout:May 15 23:46:14 smithi084 bash[329895]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-0 2024-05-15T23:46:17.243 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.0.service' 2024-05-15T23:46:17.286 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:46:17.286 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-15T23:46:17.286 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-15T23:46:17.286 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.1 2024-05-15T23:46:17.619 INFO:journalctl@ceph.osd.1.smithi084.stdout:May 15 23:46:17 smithi084 systemd[1]: Stopping Ceph osd.1 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:46:18.118 INFO:journalctl@ceph.osd.1.smithi084.stdout:May 15 23:46:17 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-1[65448]: 2024-05-15T23:46:17.629+0000 7fb382f68700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:46:18.119 INFO:journalctl@ceph.osd.1.smithi084.stdout:May 15 23:46:17 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-1[65448]: 2024-05-15T23:46:17.629+0000 7fb382f68700 -1 osd.1 856 *** Got signal Terminated *** 2024-05-15T23:46:18.119 INFO:journalctl@ceph.osd.1.smithi084.stdout:May 15 23:46:17 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-1[65448]: 2024-05-15T23:46:17.629+0000 7fb382f68700 -1 osd.1 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:46:23.868 INFO:journalctl@ceph.osd.1.smithi084.stdout:May 15 23:46:23 smithi084 bash[330168]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-1 2024-05-15T23:46:25.654 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.1.service' 2024-05-15T23:46:25.694 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:46:25.695 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-15T23:46:25.695 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-15T23:46:25.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.2 2024-05-15T23:46:26.003 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:46:25 smithi084 systemd[1]: Stopping Ceph osd.2 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:46:26.368 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:46:26 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-2[152788]: 2024-05-15T23:46:26.001+0000 7f50e9889700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:46:26.369 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:46:26 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-2[152788]: 2024-05-15T23:46:26.001+0000 7f50e9889700 -1 osd.2 856 *** Got signal Terminated *** 2024-05-15T23:46:26.369 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:46:26 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-2[152788]: 2024-05-15T23:46:26.001+0000 7f50e9889700 -1 osd.2 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:46:32.369 INFO:journalctl@ceph.osd.2.smithi084.stdout:May 15 23:46:32 smithi084 bash[330432]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-2 2024-05-15T23:46:33.971 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.2.service' 2024-05-15T23:46:34.011 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:46:34.012 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-15T23:46:34.012 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-15T23:46:34.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.3 2024-05-15T23:46:34.321 INFO:journalctl@ceph.osd.3.smithi084.stdout:May 15 23:46:34 smithi084 systemd[1]: Stopping Ceph osd.3 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:46:34.618 INFO:journalctl@ceph.osd.3.smithi084.stdout:May 15 23:46:34 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-3[80671]: 2024-05-15T23:46:34.319+0000 7f8f13ab3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:46:34.619 INFO:journalctl@ceph.osd.3.smithi084.stdout:May 15 23:46:34 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-3[80671]: 2024-05-15T23:46:34.319+0000 7f8f13ab3700 -1 osd.3 856 *** Got signal Terminated *** 2024-05-15T23:46:34.619 INFO:journalctl@ceph.osd.3.smithi084.stdout:May 15 23:46:34 smithi084 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-3[80671]: 2024-05-15T23:46:34.319+0000 7f8f13ab3700 -1 osd.3 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:46:36.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:35 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:35.720+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:36.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:35 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:35.727+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:37.054 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:36 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:36.715+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:37.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:36 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:36.705+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:37.570 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:37 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:37.213+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:37.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:37 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:37.688+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:37.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:37 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:37.486+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:38.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:37 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:37.700+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:38.570 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:38 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:38.248+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:38.775 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:38 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:38.505+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:38.775 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:38 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:38.712+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:39.052 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:38 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:38.774+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:39.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:38 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:38.688+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:39.570 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:39 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:39.231+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:39.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:39 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:39.538+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:39.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:39 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:39.725+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:40.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:39 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:39.708+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:40.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:39 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:39.820+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:40.566 INFO:journalctl@ceph.osd.3.smithi084.stdout:May 15 23:46:40 smithi084 bash[330695]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-3 2024-05-15T23:46:40.570 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:40 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:40.262+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:40.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:40 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:40.545+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:40.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:40 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:40.682+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:41.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:40 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:40.755+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:41.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:40 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:40.832+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:41.570 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:41 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:41.272+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:41.796 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:41 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:41.523+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:41.796 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:41 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:41.669+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:42.052 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:41 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:41.795+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:42.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:41 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:41.798+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:42.105 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.3.service' 2024-05-15T23:46:42.144 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:46:42.144 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-15T23:46:42.144 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-15T23:46:42.144 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.4 2024-05-15T23:46:42.455 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:42 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:42.279+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:42.455 INFO:journalctl@ceph.osd.4.smithi088.stdout:May 15 23:46:42 smithi088 systemd[1]: Stopping Ceph osd.4 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:46:42.763 INFO:journalctl@ceph.osd.4.smithi088.stdout:May 15 23:46:42 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-4[50000]: 2024-05-15T23:46:42.453+0000 7f424c13d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:46:42.763 INFO:journalctl@ceph.osd.4.smithi088.stdout:May 15 23:46:42 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-4[50000]: 2024-05-15T23:46:42.453+0000 7f424c13d700 -1 osd.4 856 *** Got signal Terminated *** 2024-05-15T23:46:42.763 INFO:journalctl@ceph.osd.4.smithi088.stdout:May 15 23:46:42 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-4[50000]: 2024-05-15T23:46:42.453+0000 7f424c13d700 -1 osd.4 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:46:42.800 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:42 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:42.488+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:42.800 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:42 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:42.645+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:43.052 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:42 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:42.798+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:43.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:42 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:42.761+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:43.571 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:43 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:43.288+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:43.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:43 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:43.506+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:43.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:43 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:43.685+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:44.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:43 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:43.804+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:44.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:43 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:43.804+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:44.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:43 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:43.839+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:44.570 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:44 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:44.251+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:44.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:44 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:44.666+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:44.803 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:44 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:44.515+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:44.803 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:44 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:44.515+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:45.258 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:44 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:44.846+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:45.258 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:44 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:44.846+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:45.258 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:45 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:45.049+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:46:45.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:44 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:44.818+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:45.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:44 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:44.818+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:45.570 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:45 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:45.256+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:46.052 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:45 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:45.790+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:46.052 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:45 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:45.790+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:46.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:45 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:45.564+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:46.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:45 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:45.564+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:46.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:45 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:45.696+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:46.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:45 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:45.696+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:46:46.069 INFO:journalctl@ceph.osd.4.smithi088.stdout:May 15 23:46:45 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-4[50000]: 2024-05-15T23:46:45.829+0000 7f4244729700 -1 osd.4 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.198838+0000 front 2024-05-15T23:46:20.198813+0000 (oldest deadline 2024-05-15T23:46:45.498827+0000) 2024-05-15T23:46:46.069 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:45 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:45.817+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:46.069 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:45 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:45.817+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:46.320 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:46 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:46.214+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:46.321 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:46 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:46.067+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:46:47.052 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:46 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:46.784+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:47.052 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:46 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:46.784+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:47.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:46 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:46.563+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:47.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:46 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:46.563+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:47.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:46 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:46.699+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:47.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:46 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:46.699+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:46:47.070 INFO:journalctl@ceph.osd.4.smithi088.stdout:May 15 23:46:46 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-4[50000]: 2024-05-15T23:46:46.806+0000 7f4244729700 -1 osd.4 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.198838+0000 front 2024-05-15T23:46:20.198813+0000 (oldest deadline 2024-05-15T23:46:45.498827+0000) 2024-05-15T23:46:47.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:46 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:46.778+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:47.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:46 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:46.778+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:47.071 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:47 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:47.026+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:46:47.570 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:47 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:47.226+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:47.802 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:47 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:47.757+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:47.802 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:47 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:47.757+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:47.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:47 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:47.523+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:47.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:47 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:47.523+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:47.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:47 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:47.666+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:47.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:47 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:47.666+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:46:48.041 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:47 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:47.774+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:48.041 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:47 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:47.774+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:48.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:47 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:47.824+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:46:48.307 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:48 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:48.226+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:48.307 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:48 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:48.040+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:46:48.570 INFO:journalctl@ceph.osd.4.smithi088.stdout:May 15 23:46:48 smithi088 bash[179112]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-4 2024-05-15T23:46:48.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:48 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:48.528+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:48.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:48 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:48.528+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:48.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:48 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:48.664+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:48.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:48 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:48.664+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:46:49.071 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:49 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:49.023+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:46:49.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:48 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:48.777+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:49.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:48 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:48.777+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:49.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:48 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:48.803+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:49.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:48 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:48.803+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:49.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:48 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:48.868+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:46:49.571 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:49 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:49.189+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:49.571 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:49 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:49.189+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:22.482811+0000 front 2024-05-15T23:46:22.482798+0000 (oldest deadline 2024-05-15T23:46:48.382896+0000) 2024-05-15T23:46:49.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:49 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:49.493+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:49.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:49 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:49.493+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:49.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:49 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:49.660+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:49.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:49 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:49.660+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:46:49.981 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:49 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:49.762+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:49.981 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:49 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:49.762+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:50.300 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:49 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:49.979+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:46:50.300 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:50 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:50.167+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:50.300 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:50 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:50.167+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:22.482811+0000 front 2024-05-15T23:46:22.482798+0000 (oldest deadline 2024-05-15T23:46:48.382896+0000) 2024-05-15T23:46:50.301 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:49 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:49.811+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:50.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:49 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:49.811+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:50.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:49 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:49.872+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:46:50.307 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.4.service' 2024-05-15T23:46:50.347 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:46:50.347 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-15T23:46:50.347 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-15T23:46:50.348 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.5 2024-05-15T23:46:50.570 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:50 smithi088 systemd[1]: Stopping Ceph osd.5 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:46:50.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:50 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:50.511+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:50.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:50 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:50.511+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:50.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:50 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:50.625+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:50.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:50 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:50.625+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:46:50.977 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:50 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:50.811+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:50.978 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:50 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:50.811+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:50.978 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:50 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:50.672+0000 7f154b391700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:46:50.978 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:50 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:50.672+0000 7f154b391700 -1 osd.5 856 *** Got signal Terminated *** 2024-05-15T23:46:50.978 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:50 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:50.672+0000 7f154b391700 -1 osd.5 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:46:51.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:50 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:50.812+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:51.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:50 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:50.812+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:51.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:50 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:50.913+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:46:51.320 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:50 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:50.977+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:46:51.321 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:51 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:51.133+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:51.321 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:51 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:51.133+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:22.482811+0000 front 2024-05-15T23:46:22.482798+0000 (oldest deadline 2024-05-15T23:46:48.382896+0000) 2024-05-15T23:46:51.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:51 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:51.498+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:51.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:51 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:51.498+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:51.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:51 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:51.592+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:51.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:51 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:51.592+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:46:52.070 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:51 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:51.950+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:46:52.071 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:51 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:51.950+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:27.696479+0000 front 2024-05-15T23:46:27.696458+0000 (oldest deadline 2024-05-15T23:46:51.796608+0000) 2024-05-15T23:46:52.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:51 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:51.777+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:52.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:51 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:51.777+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:52.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:51 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:51.827+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:52.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:51 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:51.827+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:52.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:51 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:51.827+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:46:52.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:51 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:51.869+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:46:52.570 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:52 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:52.171+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:52.571 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:52 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:52.171+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:22.482811+0000 front 2024-05-15T23:46:22.482798+0000 (oldest deadline 2024-05-15T23:46:48.382896+0000) 2024-05-15T23:46:52.571 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:52 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:52.171+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.383420+0000 front 2024-05-15T23:46:28.383570+0000 (oldest deadline 2024-05-15T23:46:51.883562+0000) 2024-05-15T23:46:52.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:52 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:52.466+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:52.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:52 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:52.466+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:52.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:52 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:52.609+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:52.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:52 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:52.609+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:46:53.221 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:52 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:52.828+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:53.221 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:52 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:52.828+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:53.222 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:52 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:52.919+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:46:53.222 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:52 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:52.919+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:27.696479+0000 front 2024-05-15T23:46:27.696458+0000 (oldest deadline 2024-05-15T23:46:51.796608+0000) 2024-05-15T23:46:53.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:52 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:52.824+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:53.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:52 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:52.824+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:53.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:52 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:52.824+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:46:53.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:52 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:52.829+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:46:53.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:52 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:52.829+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:46:53.570 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:53 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:53.220+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:53.571 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:53 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:53.220+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:22.482811+0000 front 2024-05-15T23:46:22.482798+0000 (oldest deadline 2024-05-15T23:46:48.382896+0000) 2024-05-15T23:46:53.571 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:53 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:53.220+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.383420+0000 front 2024-05-15T23:46:28.383570+0000 (oldest deadline 2024-05-15T23:46:51.883562+0000) 2024-05-15T23:46:53.796 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:53 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:53.571+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:53.797 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:53 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:53.571+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:46:53.797 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:53 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:53.516+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:53.797 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:53 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:53.516+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:53.797 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:53 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:53.516+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:46:54.052 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:53 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:53.799+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:46:54.052 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:53 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:53.799+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:46:54.052 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:53 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:53.795+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:54.052 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:53 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:53.795+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:54.052 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:53 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:53.795+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:46:54.070 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:53 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:53.942+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:46:54.070 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:53 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:53.942+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:27.696479+0000 front 2024-05-15T23:46:27.696458+0000 (oldest deadline 2024-05-15T23:46:51.796608+0000) 2024-05-15T23:46:54.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:53 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:53.816+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:54.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:53 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:53.816+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:54.570 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:54 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:54.176+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:54.571 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:54 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:54.176+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:22.482811+0000 front 2024-05-15T23:46:22.482798+0000 (oldest deadline 2024-05-15T23:46:48.382896+0000) 2024-05-15T23:46:54.571 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:54 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:54.176+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.383420+0000 front 2024-05-15T23:46:28.383570+0000 (oldest deadline 2024-05-15T23:46:51.883562+0000) 2024-05-15T23:46:55.052 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:54 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:54.781+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:46:55.052 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:54 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:54.781+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:46:55.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:54 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:54.558+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:55.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:54 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:54.558+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:55.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:54 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:54.558+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:46:55.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:54 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:54.763+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:55.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:54 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:54.763+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:55.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:54 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:54.763+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:46:55.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:54 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:54.567+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:55.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:54 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:54.567+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:46:55.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:54 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:54.567+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:46:55.070 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:54 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:54.935+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:46:55.071 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:54 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:54.935+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:27.696479+0000 front 2024-05-15T23:46:27.696458+0000 (oldest deadline 2024-05-15T23:46:51.796608+0000) 2024-05-15T23:46:55.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:54 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:54.792+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:55.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:54 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:54.792+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:55.571 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:55 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:55.169+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.881849+0000 front 2024-05-15T23:46:11.881771+0000 (oldest deadline 2024-05-15T23:46:37.181788+0000) 2024-05-15T23:46:55.571 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:55 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:55.169+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:22.482811+0000 front 2024-05-15T23:46:22.482798+0000 (oldest deadline 2024-05-15T23:46:48.382896+0000) 2024-05-15T23:46:55.571 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:55 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5[174676]: 2024-05-15T23:46:55.169+0000 7f1547985700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.383420+0000 front 2024-05-15T23:46:28.383570+0000 (oldest deadline 2024-05-15T23:46:51.883562+0000) 2024-05-15T23:46:55.819 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:55 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:55.592+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:55.820 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:55 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:55.592+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:55.820 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:55 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:55.592+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:46:55.820 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:55 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:55.799+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:55.820 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:55 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:55.799+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:55.820 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:55 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:55.799+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:46:55.820 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:55 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:55.564+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:55.820 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:55 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:55.564+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:46:55.821 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:55 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:55.564+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:46:56.070 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:55 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:55.886+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:46:56.071 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:55 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:55.886+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:27.696479+0000 front 2024-05-15T23:46:27.696458+0000 (oldest deadline 2024-05-15T23:46:51.796608+0000) 2024-05-15T23:46:56.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:55 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:55.752+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:56.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:55 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:55.752+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:56.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:55 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:55.752+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:46:56.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:55 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:55.818+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:46:56.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:55 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:55.818+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:46:56.722 INFO:journalctl@ceph.osd.5.smithi088.stdout:May 15 23:46:56 smithi088 bash[179381]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-5 2024-05-15T23:46:56.802 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:56 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:56.761+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:56.802 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:56 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:56.761+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:56.802 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:56 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:56.761+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:46:56.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:56 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:56.562+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:56.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:56 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:56.562+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:56.803 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:56 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:56.562+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:46:56.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:56 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:56.546+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:56.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:56 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:56.546+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:46:56.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:56 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:56.546+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:46:57.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:56 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:56.719+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:57.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:56 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:56.719+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:57.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:56 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:56.719+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:46:57.071 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:56 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:56.850+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:46:57.071 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:56 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:56.850+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:27.696479+0000 front 2024-05-15T23:46:27.696458+0000 (oldest deadline 2024-05-15T23:46:51.796608+0000) 2024-05-15T23:46:57.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:56 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:56.811+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:46:57.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:56 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:56.811+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:46:57.846 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:57 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:57.603+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:57.846 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:57 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:57.603+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:57.846 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:57 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:57.603+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:46:57.847 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:57 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:57.772+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:57.847 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:57 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:57.772+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:57.847 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:57 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:57.772+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:46:57.847 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:57 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:57.560+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:57.847 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:57 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:57.560+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:46:57.847 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:57 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:57.560+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:46:57.882 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:57 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:57.710+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:57.882 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:57 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:57.710+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:57.882 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:57 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:57.710+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:46:58.175 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:57 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:57.880+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:46:58.176 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:57 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:57.880+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:27.696479+0000 front 2024-05-15T23:46:27.696458+0000 (oldest deadline 2024-05-15T23:46:51.796608+0000) 2024-05-15T23:46:58.184 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.5.service' 2024-05-15T23:46:58.226 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:46:58.226 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-15T23:46:58.226 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-15T23:46:58.226 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.6 2024-05-15T23:46:58.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:57 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:57.845+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:46:58.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:57 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:57.845+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:46:58.508 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:58 smithi088 systemd[1]: Stopping Ceph osd.6 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:46:58.508 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:58 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:58.506+0000 7ff5c9987700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:46:58.508 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:58 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:58.506+0000 7ff5c9987700 -1 osd.6 856 *** Got signal Terminated *** 2024-05-15T23:46:58.508 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:58 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:58.506+0000 7ff5c9987700 -1 osd.6 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:46:58.820 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:58 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:58.746+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:46:58.820 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:58 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:58.746+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:46:58.820 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:58 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:58.746+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:46:58.887 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:58 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:58.765+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:58.888 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:58 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:58.765+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:58.888 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:58 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:58.765+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:46:58.888 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:58 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:58.652+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:58.888 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:58 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:58.652+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:58.888 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:58 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:58.652+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:46:58.888 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:58 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:58.594+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:58.888 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:58 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:58.594+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:46:58.888 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:58 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:58.594+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:46:59.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:58 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:58.886+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:46:59.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:58 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:58.886+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:46:59.320 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:58 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:58.872+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:46:59.321 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:58 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:58.872+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:27.696479+0000 front 2024-05-15T23:46:27.696458+0000 (oldest deadline 2024-05-15T23:46:51.796608+0000) 2024-05-15T23:46:59.936 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:59 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:59.634+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:46:59.936 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:59 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:59.634+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:46:59.936 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:46:59 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:46:59.634+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:46:59.937 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:59 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:59.758+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:46:59.937 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:59 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:59.758+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:46:59.937 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:46:59 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:46:59.758+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:46:59.937 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:59 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:59.638+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:46:59.937 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:59 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:59.638+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:46:59.937 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:46:59 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:46:59.638+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:00.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:59 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:59.793+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:47:00.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:59 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:59.793+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:47:00.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:46:59 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:46:59.793+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:47:00.071 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:59 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:59.879+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:47:00.071 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:46:59 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:46:59.879+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:27.696479+0000 front 2024-05-15T23:46:27.696458+0000 (oldest deadline 2024-05-15T23:46:51.796608+0000) 2024-05-15T23:47:00.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:59 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:59.935+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:00.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:46:59 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:46:59.935+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:00.964 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:00 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:00.797+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:00.964 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:00 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:00.797+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:00.965 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:00 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:00.797+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:00.965 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:00 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:00.679+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:00.965 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:00 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:00.679+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:00.965 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:00 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:00.679+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:00.965 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:00 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:00.610+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:00.965 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:00 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:00.610+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:00.965 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:00 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:00.610+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:01.070 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:47:00 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:47:00.894+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:47:01.070 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:47:00 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:47:00.894+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:27.696479+0000 front 2024-05-15T23:46:27.696458+0000 (oldest deadline 2024-05-15T23:46:51.796608+0000) 2024-05-15T23:47:01.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:00 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:00.802+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:47:01.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:00 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:00.802+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:47:01.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:00 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:00.802+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:47:01.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:00 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:00.963+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:01.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:00 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:00.963+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:01.927 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:01 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:01.792+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:47:01.927 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:01 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:01.792+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:47:01.927 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:01 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:01.792+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:47:01.966 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:01 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:01.633+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:01.966 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:01 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:01.633+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:01.966 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:01 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:01.633+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:01.967 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:01 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:01.819+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:01.967 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:01 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:01.819+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:01.967 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:01 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:01.819+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:01.967 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:01 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:01.579+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:01.967 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:01 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:01.579+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:01.967 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:01 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:01.579+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:01.967 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:01 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:01.579+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:02.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:01 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:01.965+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:02.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:01 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:01.965+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:02.320 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:47:01 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:47:01.925+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:47:02.321 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:47:01 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:47:01.925+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:27.696479+0000 front 2024-05-15T23:46:27.696458+0000 (oldest deadline 2024-05-15T23:46:51.796608+0000) 2024-05-15T23:47:02.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:02 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:02.536+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:02.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:02 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:02.536+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:02.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:02 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:02.536+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:02.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:02 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:02.536+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:02.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:02 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:02.673+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:02.802 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:02 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:02.673+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:02.803 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:02 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:02.673+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:02.803 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:02 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:02.673+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:03.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:02 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:02.774+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:47:03.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:02 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:02.774+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:47:03.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:02 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:02.774+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:47:03.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:02 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:02.774+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.325332+0000 front 2024-05-15T23:46:35.325273+0000 (oldest deadline 2024-05-15T23:47:02.325964+0000) 2024-05-15T23:47:03.071 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:47:02 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:47:02.928+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.194486+0000 front 2024-05-15T23:46:20.194543+0000 (oldest deadline 2024-05-15T23:46:44.894457+0000) 2024-05-15T23:47:03.071 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:47:02 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:47:02.928+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:27.696479+0000 front 2024-05-15T23:46:27.696458+0000 (oldest deadline 2024-05-15T23:46:51.796608+0000) 2024-05-15T23:47:03.071 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:47:02 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6[65031]: 2024-05-15T23:47:02.928+0000 7ff5c5f7b700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:37.999293+0000 front 2024-05-15T23:46:37.999307+0000 (oldest deadline 2024-05-15T23:47:01.999941+0000) 2024-05-15T23:47:03.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:02 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:02.925+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:03.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:02 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:02.925+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:03.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:02 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:02.843+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:03.303 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:02 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:02.843+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:03.303 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:02 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:02.843+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:03.871 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:03 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:03.708+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:03.871 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:03 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:03.708+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:03.871 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:03 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:03.708+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:03.871 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:03 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:03.708+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:03.871 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:03 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:03.554+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:03.871 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:03 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:03.554+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:03.872 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:03 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:03.554+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:03.872 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:03 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:03.554+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:04.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:03 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:03.789+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:47:04.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:03 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:03.789+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:47:04.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:03 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:03.789+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:47:04.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:03 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:03.789+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.325332+0000 front 2024-05-15T23:46:35.325273+0000 (oldest deadline 2024-05-15T23:47:02.325964+0000) 2024-05-15T23:47:04.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:03 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:03.870+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:04.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:03 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:03.870+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:04.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:03 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:03.870+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:04.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:03 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:03.877+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:04.303 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:03 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:03.877+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:04.303 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:03 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:03.877+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:04.570 INFO:journalctl@ceph.osd.6.smithi088.stdout:May 15 23:47:04 smithi088 bash[179651]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-6 2024-05-15T23:47:04.827 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:04 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:04.574+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:04.827 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:04 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:04.574+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:04.828 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:04 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:04.574+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:04.828 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:04 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:04.574+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:04.828 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:04 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:04.675+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:04.828 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:04 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:04.675+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:04.828 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:04 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:04.675+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:04.828 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:04 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:04.675+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:05.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:04 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:04.793+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:47:05.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:04 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:04.793+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:47:05.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:04 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:04.793+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:47:05.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:04 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:04.793+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.325332+0000 front 2024-05-15T23:46:35.325273+0000 (oldest deadline 2024-05-15T23:47:02.325964+0000) 2024-05-15T23:47:05.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:04 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:04.826+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:05.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:04 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:04.826+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:05.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:04 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:04.826+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:05.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:04 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:04.916+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:05.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:04 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:04.916+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:05.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:04 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:04.916+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:05.929 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:05.847+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:05.929 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:05.847+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:05.930 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:05.847+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:05.930 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:05.847+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.264811+0000 front 2024-05-15T23:46:35.264741+0000 (oldest deadline 2024-05-15T23:47:05.265250+0000) 2024-05-15T23:47:05.930 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:05.709+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:05.930 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:05.709+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:05.930 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:05.709+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:05.930 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:05.709+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:05.930 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:05.615+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:05.931 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:05.615+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:05.931 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:05.615+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:05.931 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:05.615+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:06.069 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:05 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:05.761+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:47:06.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:05 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:05.761+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:47:06.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:05 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:05.761+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:47:06.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:05 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:05.761+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.325332+0000 front 2024-05-15T23:46:35.325273+0000 (oldest deadline 2024-05-15T23:47:02.325964+0000) 2024-05-15T23:47:06.076 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.6.service' 2024-05-15T23:47:06.121 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:47:06.121 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-15T23:47:06.121 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-15T23:47:06.121 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.7 2024-05-15T23:47:06.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:05.928+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:06.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:05.928+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:06.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:05 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:05.928+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:06.418 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:06 smithi088 systemd[1]: Stopping Ceph osd.7 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:47:06.756 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:06 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:06.417+0000 7f7db31aa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:47:06.757 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:06 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:06.417+0000 7f7db31aa700 -1 osd.7 856 *** Got signal Terminated *** 2024-05-15T23:47:06.757 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:06 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:06.417+0000 7f7db31aa700 -1 osd.7 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:47:06.757 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:06 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:06.756+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:47:06.947 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:06 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:06.687+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:06.947 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:06 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:06.687+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:06.947 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:06 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:06.687+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:06.947 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:06 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:06.687+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:06.947 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:06 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:06.824+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:06.947 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:06 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:06.824+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:06.947 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:06 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:06.824+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:06.948 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:06 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:06.824+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.264811+0000 front 2024-05-15T23:46:35.264741+0000 (oldest deadline 2024-05-15T23:47:05.265250+0000) 2024-05-15T23:47:06.948 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:06 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:06.617+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:06.948 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:06 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:06.617+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:06.948 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:06 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:06.617+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:06.948 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:06 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:06.617+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:07.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:06 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:06.756+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:47:07.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:06 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:06.756+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:47:07.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:06 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:06.756+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.325332+0000 front 2024-05-15T23:46:35.325273+0000 (oldest deadline 2024-05-15T23:47:02.325964+0000) 2024-05-15T23:47:07.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:06 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:06.946+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:07.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:06 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:06.946+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:07.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:06 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:06.946+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:07.934 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:07 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:07.729+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:07.934 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:07 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:07.729+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:07.934 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:07 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:07.729+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:07.934 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:07 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:07.729+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:07.934 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:07 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:07.775+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:07.934 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:07 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:07.775+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:07.935 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:07 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:07.775+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:07.935 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:07 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:07.775+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.264811+0000 front 2024-05-15T23:46:35.264741+0000 (oldest deadline 2024-05-15T23:47:05.265250+0000) 2024-05-15T23:47:07.935 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:07 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:07.660+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:07.935 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:07 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:07.660+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:07.935 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:07 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:07.660+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:07.935 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:07 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:07.660+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:08.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:07 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:07.771+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:47:08.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:07 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:07.771+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:47:08.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:07 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:07.771+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:47:08.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:07 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:07.771+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.325332+0000 front 2024-05-15T23:46:35.325273+0000 (oldest deadline 2024-05-15T23:47:02.325964+0000) 2024-05-15T23:47:08.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:07 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:07.932+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:08.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:07 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:07.932+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:08.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:07 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:07.932+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:08.935 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:08.712+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:08.935 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:08.712+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:08.935 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:08.712+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:08.935 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:08.712+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:08.936 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:08.810+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:08.936 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:08.810+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:08.936 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:08.810+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:08.936 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:08.810+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.264811+0000 front 2024-05-15T23:46:35.264741+0000 (oldest deadline 2024-05-15T23:47:05.265250+0000) 2024-05-15T23:47:08.936 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:08.810+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:45.265831+0000 front 2024-05-15T23:46:45.265754+0000 (oldest deadline 2024-05-15T23:47:08.165664+0000) 2024-05-15T23:47:08.936 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:08.681+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:08.936 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:08.681+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:08.936 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:08.681+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:08.936 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:08.681+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:09.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:08 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:08.772+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:47:09.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:08 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:08.772+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:47:09.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:08 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:08.772+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:47:09.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:08 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:08.772+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.325332+0000 front 2024-05-15T23:46:35.325273+0000 (oldest deadline 2024-05-15T23:47:02.325964+0000) 2024-05-15T23:47:09.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:08 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:08.772+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:45.127754+0000 front 2024-05-15T23:46:45.127845+0000 (oldest deadline 2024-05-15T23:47:08.627884+0000) 2024-05-15T23:47:09.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:08.933+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:09.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:08.933+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:09.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:08.933+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:09.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:08 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:08.933+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:09.981 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:09.684+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:09.981 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:09.684+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:09.982 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:09.684+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:09.982 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:09.684+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:09.982 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:09.774+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:09.982 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:09.774+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:09.982 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:09.774+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:09.982 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:09.774+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.264811+0000 front 2024-05-15T23:46:35.264741+0000 (oldest deadline 2024-05-15T23:47:05.265250+0000) 2024-05-15T23:47:09.982 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:09.774+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:45.265831+0000 front 2024-05-15T23:46:45.265754+0000 (oldest deadline 2024-05-15T23:47:08.165664+0000) 2024-05-15T23:47:09.983 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:09.726+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:09.983 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:09.726+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:09.983 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:09.726+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:09.983 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:09.726+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:09.983 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:09.726+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:10.070 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:09 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:09.815+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:47:10.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:09 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:09.815+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:47:10.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:09 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:09.815+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:47:10.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:09 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:09.815+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.325332+0000 front 2024-05-15T23:46:35.325273+0000 (oldest deadline 2024-05-15T23:47:02.325964+0000) 2024-05-15T23:47:10.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:09 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:09.815+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:45.127754+0000 front 2024-05-15T23:46:45.127845+0000 (oldest deadline 2024-05-15T23:47:08.627884+0000) 2024-05-15T23:47:10.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:09.980+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:10.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:09.980+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:10.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:09.980+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:10.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:09 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:09.980+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:11.014 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:10 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:10.807+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:11.014 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:10 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:10.807+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:11.014 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:10 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:10.807+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:11.014 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:10 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:10.807+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.264811+0000 front 2024-05-15T23:46:35.264741+0000 (oldest deadline 2024-05-15T23:47:05.265250+0000) 2024-05-15T23:47:11.014 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:10 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:10.807+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:45.265831+0000 front 2024-05-15T23:46:45.265754+0000 (oldest deadline 2024-05-15T23:47:08.165664+0000) 2024-05-15T23:47:11.014 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:10 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:10.770+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:11.015 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:10 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:10.770+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:11.015 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:10 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:10.770+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:11.015 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:10 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:10.770+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:11.015 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:10 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:10.770+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:11.015 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:10 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:10.645+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:11.015 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:10 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:10.645+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:11.015 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:10 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:10.645+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:11.015 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:10 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:10.645+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:11.015 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:10 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:10.645+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:11.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:10 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:10.805+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:10.821517+0000 front 2024-05-15T23:46:10.821593+0000 (oldest deadline 2024-05-15T23:46:35.521370+0000) 2024-05-15T23:47:11.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:10 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:10.805+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.422623+0000 front 2024-05-15T23:46:21.422560+0000 (oldest deadline 2024-05-15T23:46:43.623009+0000) 2024-05-15T23:47:11.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:10 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:10.805+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.624912+0000 front 2024-05-15T23:46:30.624881+0000 (oldest deadline 2024-05-15T23:46:55.324836+0000) 2024-05-15T23:47:11.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:10 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:10.805+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.325332+0000 front 2024-05-15T23:46:35.325273+0000 (oldest deadline 2024-05-15T23:47:02.325964+0000) 2024-05-15T23:47:11.071 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:10 smithi088 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7[72302]: 2024-05-15T23:47:10.806+0000 7f7daf79e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:45.127754+0000 front 2024-05-15T23:46:45.127845+0000 (oldest deadline 2024-05-15T23:47:08.627884+0000) 2024-05-15T23:47:11.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:11.013+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:11.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:11.013+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:11.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:11.013+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:11.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:11.013+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:12.009 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:11.814+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:12.009 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:11.814+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:12.009 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:11.814+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:12.009 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:11.814+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:12.009 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:11.814+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:12.009 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:11.762+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:12.009 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:11.762+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:12.009 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:11.762+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:12.009 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:11.762+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.264811+0000 front 2024-05-15T23:46:35.264741+0000 (oldest deadline 2024-05-15T23:47:05.265250+0000) 2024-05-15T23:47:12.010 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:11.762+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:45.265831+0000 front 2024-05-15T23:46:45.265754+0000 (oldest deadline 2024-05-15T23:47:08.165664+0000) 2024-05-15T23:47:12.010 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:11.686+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:12.010 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:11.686+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:12.010 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:11.686+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:12.010 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:11.686+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:12.010 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:11 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:11.686+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:12.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:12.008+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:12.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:12.008+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:12.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:12.008+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:12.303 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:12.008+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:12.570 INFO:journalctl@ceph.osd.7.smithi088.stdout:May 15 23:47:12 smithi088 bash[179919]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-7 2024-05-15T23:47:13.018 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:12.859+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:13.018 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:12.859+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:13.018 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:12.859+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:13.019 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:12.859+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:13.019 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:12.859+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:13.019 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:12.794+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:13.020 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:12.794+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:13.020 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:12.794+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:13.020 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:12.794+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.264811+0000 front 2024-05-15T23:46:35.264741+0000 (oldest deadline 2024-05-15T23:47:05.265250+0000) 2024-05-15T23:47:13.020 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:12.794+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:45.265831+0000 front 2024-05-15T23:46:45.265754+0000 (oldest deadline 2024-05-15T23:47:08.165664+0000) 2024-05-15T23:47:13.020 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:12.656+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:13.021 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:12.656+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:13.021 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:12.656+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:13.021 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:12.656+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:13.021 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:12 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:12.656+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:13.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:13.016+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:13.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:13.016+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:13.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:13.016+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:13.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:13.016+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:13.999 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.7.service' 2024-05-15T23:47:14.038 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:47:14.038 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-15T23:47:14.039 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-15T23:47:14.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.8 2024-05-15T23:47:14.039 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:13.811+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:14.039 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:13.811+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:14.040 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:13.811+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:14.040 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:13.811+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:14.040 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:13.811+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:14.040 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:13.808+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:14.040 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:13.808+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:14.040 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:13.808+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:14.040 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:13.808+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.264811+0000 front 2024-05-15T23:46:35.264741+0000 (oldest deadline 2024-05-15T23:47:05.265250+0000) 2024-05-15T23:47:14.040 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:13.808+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:45.265831+0000 front 2024-05-15T23:46:45.265754+0000 (oldest deadline 2024-05-15T23:47:08.165664+0000) 2024-05-15T23:47:14.041 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:13.692+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:14.041 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:13.693+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:14.041 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:13.693+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:14.041 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:13.693+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:14.041 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:13 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:13.693+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:14.302 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:14 smithi090 systemd[1]: Stopping Ceph osd.8 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:47:14.303 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:14.036+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:14.303 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:14.036+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:14.303 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:14.036+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:14.303 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:14.036+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:14.709 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:14.315+0000 7f40494e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:47:14.709 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:14.315+0000 7f40494e9700 -1 osd.8 856 *** Got signal Terminated *** 2024-05-15T23:47:14.709 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:14.315+0000 7f40494e9700 -1 osd.8 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:47:15.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:14.835+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:15.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:14.835+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:15.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:14.835+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:15.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:14.835+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:15.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:14.835+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:15.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:14.765+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:15.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:14.765+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:15.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:14.765+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:15.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:14.765+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.264811+0000 front 2024-05-15T23:46:35.264741+0000 (oldest deadline 2024-05-15T23:47:05.265250+0000) 2024-05-15T23:47:15.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:14.765+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:45.265831+0000 front 2024-05-15T23:46:45.265754+0000 (oldest deadline 2024-05-15T23:47:08.165664+0000) 2024-05-15T23:47:15.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:14.708+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:15.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:14.708+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:15.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:14.708+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:15.054 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:14.708+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:15.054 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:14 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:14.708+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:15.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:15.054+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:15.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:15.054+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:15.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:15.054+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:15.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:15.054+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:16.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:15.804+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:16.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:15.804+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:16.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:15.804+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:16.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:15.804+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:16.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:15.804+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:16.052 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:15.761+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:16.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:15.761+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:16.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:15.761+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:16.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:15.761+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.264811+0000 front 2024-05-15T23:46:35.264741+0000 (oldest deadline 2024-05-15T23:47:05.265250+0000) 2024-05-15T23:47:16.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:15.761+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:45.265831+0000 front 2024-05-15T23:46:45.265754+0000 (oldest deadline 2024-05-15T23:47:08.165664+0000) 2024-05-15T23:47:16.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:15.692+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:16.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:15.692+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:16.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:15.692+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:16.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:15.692+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:16.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:15 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:15.692+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:16.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:16.068+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:16.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:16.068+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:16.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:16.068+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:16.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:16.068+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:17.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:16.769+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:17.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:16.769+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:17.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:16.769+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:17.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:16.769+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:17.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:16.769+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:17.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:16.781+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:17.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:16.781+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:17.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:16.781+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:17.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:16.781+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.264811+0000 front 2024-05-15T23:46:35.264741+0000 (oldest deadline 2024-05-15T23:47:05.265250+0000) 2024-05-15T23:47:17.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:16.781+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:45.265831+0000 front 2024-05-15T23:46:45.265754+0000 (oldest deadline 2024-05-15T23:47:08.165664+0000) 2024-05-15T23:47:17.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:16.649+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:17.054 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:16.649+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:17.054 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:16.649+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:17.054 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:16.649+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:17.054 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:16 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:16.649+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:17.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:17.097+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:17.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:17.097+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:17.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:17.097+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:17.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:17.097+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:18.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:17.773+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:18.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:17.773+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:18.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:17.773+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:18.054 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:17.773+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:18.054 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:17.773+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:18.054 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:17.766+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:18.055 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:17.766+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:18.055 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:17.766+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:18.055 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:17.766+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.264811+0000 front 2024-05-15T23:46:35.264741+0000 (oldest deadline 2024-05-15T23:47:05.265250+0000) 2024-05-15T23:47:18.055 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:17.766+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:45.265831+0000 front 2024-05-15T23:46:45.265754+0000 (oldest deadline 2024-05-15T23:47:08.165664+0000) 2024-05-15T23:47:18.055 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:17.687+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:18.055 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:17.687+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:18.055 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:17.687+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:18.055 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:17.687+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:18.055 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:17 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:17.687+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:18.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:18.145+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:18.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:18.145+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:18.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:18.145+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:18.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:18.145+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:19.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:18.789+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:19.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:18.789+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:19.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:18.789+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:19.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:18.789+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:19.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:18.789+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:19.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:18.726+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:12.160398+0000 front 2024-05-15T23:46:12.160245+0000 (oldest deadline 2024-05-15T23:46:38.059956+0000) 2024-05-15T23:47:19.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:18.726+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.960954+0000 front 2024-05-15T23:46:20.960797+0000 (oldest deadline 2024-05-15T23:46:44.460893+0000) 2024-05-15T23:47:19.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:18.726+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.663505+0000 front 2024-05-15T23:46:30.663593+0000 (oldest deadline 2024-05-15T23:46:51.763620+0000) 2024-05-15T23:47:19.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:18.726+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:35.264811+0000 front 2024-05-15T23:46:35.264741+0000 (oldest deadline 2024-05-15T23:47:05.265250+0000) 2024-05-15T23:47:19.053 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8[136364]: 2024-05-15T23:47:18.726+0000 7f4045add700 -1 osd.8 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:45.265831+0000 front 2024-05-15T23:46:45.265754+0000 (oldest deadline 2024-05-15T23:47:08.165664+0000) 2024-05-15T23:47:19.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:18.711+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:19.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:18.711+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:19.054 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:18.711+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:19.054 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:18.711+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:19.054 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:18 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:18.711+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:19.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:19 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:19.191+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:19.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:19 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:19.191+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:19.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:19 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:19.191+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:19.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:19 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:19.191+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:19.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:19 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:19.191+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6806 osd.5 since back 2024-05-15T23:46:55.014515+0000 front 2024-05-15T23:46:55.014530+0000 (oldest deadline 2024-05-15T23:47:18.514468+0000) 2024-05-15T23:47:20.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:19 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:19.825+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:20.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:19 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:19.825+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:20.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:19 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:19.825+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:20.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:19 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:19.825+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:20.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:19 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:19.825+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:20.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:19 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:19.675+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:20.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:19 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:19.675+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:20.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:19 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:19.675+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:20.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:19 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:19.675+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:20.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:19 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:19.675+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:20.552 INFO:journalctl@ceph.osd.8.smithi090.stdout:May 15 23:47:20 smithi090 bash[178891]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-8 2024-05-15T23:47:20.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:20 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:20.186+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:20.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:20 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:20.186+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:20.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:20 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:20.186+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:20.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:20 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:20.186+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:20.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:20 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:20.186+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6806 osd.5 since back 2024-05-15T23:46:55.014515+0000 front 2024-05-15T23:46:55.014530+0000 (oldest deadline 2024-05-15T23:47:18.514468+0000) 2024-05-15T23:47:21.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:20 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:20.848+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:21.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:20 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:20.848+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:21.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:20 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:20.848+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:21.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:20 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:20.848+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:21.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:20 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:20.848+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:21.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:20 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:20.668+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:21.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:20 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:20.668+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:21.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:20 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:20.668+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:21.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:20 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:20.668+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:21.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:20 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:20.668+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:21.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:21 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:21.180+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:21.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:21 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:21.180+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:21.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:21 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:21.180+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:21.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:21 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:21.180+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:21.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:21 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:21.180+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6806 osd.5 since back 2024-05-15T23:46:55.014515+0000 front 2024-05-15T23:46:55.014530+0000 (oldest deadline 2024-05-15T23:47:18.514468+0000) 2024-05-15T23:47:22.036 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:21 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:21.800+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:22.036 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:21 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:21.800+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:22.036 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:21 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:21.800+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:22.036 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:21 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:21.800+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:22.036 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:21 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:21.800+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:22.037 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:21 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:21.639+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:22.037 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:21 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:21.639+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:22.037 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:21 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:21.639+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:22.037 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:21 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:21.639+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:22.037 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:21 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:21.639+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:22.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:22.142+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:22.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:22.142+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:22.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:22.142+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:22.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:22.142+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:22.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:22.142+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6806 osd.5 since back 2024-05-15T23:46:55.014515+0000 front 2024-05-15T23:46:55.014530+0000 (oldest deadline 2024-05-15T23:47:18.514468+0000) 2024-05-15T23:47:22.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.8.service' 2024-05-15T23:47:22.557 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:47:22.557 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-15T23:47:22.558 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-15T23:47:22.558 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.9 2024-05-15T23:47:22.802 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:22 smithi090 systemd[1]: Stopping Ceph osd.9 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:47:22.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:22.612+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:22.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:22.612+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:22.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:22.612+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:22.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:22.612+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:22.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:22.612+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:23.106 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:22.863+0000 7fd16c029700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:47:23.106 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:22.863+0000 7fd16c029700 -1 osd.9 856 *** Got signal Terminated *** 2024-05-15T23:47:23.106 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:22.863+0000 7fd16c029700 -1 osd.9 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:47:23.106 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:23 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:23.105+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:23.106 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:22.842+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:23.107 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:22.842+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:23.107 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:22.842+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:23.107 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:22.842+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:23.107 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:22 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:22.842+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:23.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:23 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:23.105+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:23.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:23 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:23.105+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:23.553 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:23 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:23.105+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:23.553 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:23 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:23.105+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6806 osd.5 since back 2024-05-15T23:46:55.014515+0000 front 2024-05-15T23:46:55.014530+0000 (oldest deadline 2024-05-15T23:47:18.514468+0000) 2024-05-15T23:47:23.854 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:23 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:23.573+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:23.854 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:23 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:23.573+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:23.855 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:23 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:23.573+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:23.855 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:23 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:23.573+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:23.855 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:23 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:23.573+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:24.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:24.091+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:24.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:24.091+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:24.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:24.091+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:24.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:24.091+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:24.302 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:24.091+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6806 osd.5 since back 2024-05-15T23:46:55.014515+0000 front 2024-05-15T23:46:55.014530+0000 (oldest deadline 2024-05-15T23:47:18.514468+0000) 2024-05-15T23:47:24.303 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:23 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:23.853+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:24.303 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:23 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:23.853+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:24.303 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:23 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:23.853+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:24.303 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:23 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:23.853+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:24.303 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:23 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:23.853+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:24.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:24.549+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:24.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:24.549+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:24.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:24.549+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:24.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:24.549+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:24.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:24.549+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:25.122 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:24.807+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:25.122 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:24.807+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:25.122 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:24.807+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:25.122 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:24.807+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:25.122 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:24.807+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:25.123 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:24 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:24.807+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:01.214104+0000 front 2024-05-15T23:47:01.214223+0000 (oldest deadline 2024-05-15T23:47:24.114010+0000) 2024-05-15T23:47:25.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:25.121+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:25.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:25.121+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:25.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:25.121+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:25.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:25.121+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:25.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:25.121+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6806 osd.5 since back 2024-05-15T23:46:55.014515+0000 front 2024-05-15T23:46:55.014530+0000 (oldest deadline 2024-05-15T23:47:18.514468+0000) 2024-05-15T23:47:25.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:25.121+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.215549+0000 front 2024-05-15T23:47:03.215598+0000 (oldest deadline 2024-05-15T23:47:24.915467+0000) 2024-05-15T23:47:26.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:25.568+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:26.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:25.568+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:26.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:25.568+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:26.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:25.568+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:26.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:25.568+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:26.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:25.767+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:26.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:25.767+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:26.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:25.767+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:26.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:25.767+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:26.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:25.767+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:26.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:25 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:25.767+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:01.214104+0000 front 2024-05-15T23:47:01.214223+0000 (oldest deadline 2024-05-15T23:47:24.114010+0000) 2024-05-15T23:47:26.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:26.129+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:26.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:26.129+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:26.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:26.129+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:26.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:26.129+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:26.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:26.129+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6806 osd.5 since back 2024-05-15T23:46:55.014515+0000 front 2024-05-15T23:46:55.014530+0000 (oldest deadline 2024-05-15T23:47:18.514468+0000) 2024-05-15T23:47:26.552 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:26.129+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.215549+0000 front 2024-05-15T23:47:03.215598+0000 (oldest deadline 2024-05-15T23:47:24.915467+0000) 2024-05-15T23:47:27.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:26.618+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:27.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:26.618+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:27.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:26.618+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:27.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:26.618+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:27.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:26.618+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:27.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:26.737+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:27.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:26.737+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:27.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:26.737+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:27.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:26.737+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:27.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:26.737+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:27.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:26 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:26.737+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:01.214104+0000 front 2024-05-15T23:47:01.214223+0000 (oldest deadline 2024-05-15T23:47:24.114010+0000) 2024-05-15T23:47:27.553 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:27.171+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:21.809334+0000 front 2024-05-15T23:46:21.809383+0000 (oldest deadline 2024-05-15T23:46:47.709294+0000) 2024-05-15T23:47:27.553 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:27.171+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.510712+0000 front 2024-05-15T23:46:30.510756+0000 (oldest deadline 2024-05-15T23:46:52.810829+0000) 2024-05-15T23:47:27.553 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:27.171+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:39.212589+0000 front 2024-05-15T23:46:39.212653+0000 (oldest deadline 2024-05-15T23:47:03.312533+0000) 2024-05-15T23:47:27.553 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:27.171+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:43.313059+0000 front 2024-05-15T23:46:43.313165+0000 (oldest deadline 2024-05-15T23:47:08.013168+0000) 2024-05-15T23:47:27.553 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:27.171+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6806 osd.5 since back 2024-05-15T23:46:55.014515+0000 front 2024-05-15T23:46:55.014530+0000 (oldest deadline 2024-05-15T23:47:18.514468+0000) 2024-05-15T23:47:27.553 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9[57439]: 2024-05-15T23:47:27.171+0000 7fd16861d700 -1 osd.9 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.215549+0000 front 2024-05-15T23:47:03.215598+0000 (oldest deadline 2024-05-15T23:47:24.915467+0000) 2024-05-15T23:47:28.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:27.589+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:28.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:27.589+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:28.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:27.589+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:28.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:27.589+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:28.052 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:27.589+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:28.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:27.766+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:28.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:27.766+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:28.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:27.766+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:28.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:27.766+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:28.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:27.766+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:28.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:27 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:27.766+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:01.214104+0000 front 2024-05-15T23:47:01.214223+0000 (oldest deadline 2024-05-15T23:47:24.114010+0000) 2024-05-15T23:47:29.052 INFO:journalctl@ceph.osd.9.smithi090.stdout:May 15 23:47:28 smithi090 bash[179162]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-9 2024-05-15T23:47:29.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:28 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:28.762+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:29.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:28 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:28.762+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:29.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:28 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:28.762+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:29.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:28 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:28.762+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:29.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:28 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:28.762+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:29.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:28 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:28.762+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:01.214104+0000 front 2024-05-15T23:47:01.214223+0000 (oldest deadline 2024-05-15T23:47:24.114010+0000) 2024-05-15T23:47:29.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:28 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:28.635+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:29.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:28 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:28.635+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:29.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:28 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:28.635+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:29.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:28 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:28.635+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:29.053 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:28 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:28.635+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:29.054 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:28 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:28.635+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:29.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:29 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:29.613+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:29.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:29 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:29.613+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:29.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:29 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:29.613+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:29.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:29 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:29.613+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:29.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:29 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:29.613+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:29.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:29 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:29.613+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:30.218 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:29 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:29.804+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:30.218 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:29 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:29.804+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:30.219 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:29 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:29.804+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:30.219 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:29 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:29.804+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:30.219 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:29 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:29.804+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:30.219 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:29 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:29.804+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:01.214104+0000 front 2024-05-15T23:47:01.214223+0000 (oldest deadline 2024-05-15T23:47:24.114010+0000) 2024-05-15T23:47:30.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.9.service' 2024-05-15T23:47:30.580 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:47:30.580 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-15T23:47:30.581 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-15T23:47:30.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.10 2024-05-15T23:47:30.796 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:30 smithi090 systemd[1]: Stopping Ceph osd.10 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:47:30.796 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:30 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:30.575+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:30.796 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:30 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:30.575+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:30.797 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:30 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:30.575+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:30.797 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:30 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:30.575+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:30.797 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:30 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:30.575+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:30.797 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:30 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:30.575+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:31.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:30 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:30.795+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:31.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:30 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:30.795+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:31.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:30 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:30.795+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:31.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:30 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:30.795+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:31.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:30 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:30.795+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:31.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:30 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:30.795+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:01.214104+0000 front 2024-05-15T23:47:01.214223+0000 (oldest deadline 2024-05-15T23:47:24.114010+0000) 2024-05-15T23:47:31.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:30 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:30.903+0000 7efe82b1c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:47:31.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:30 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:30.903+0000 7efe82b1c700 -1 osd.10 856 *** Got signal Terminated *** 2024-05-15T23:47:31.053 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:30 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:30.903+0000 7efe82b1c700 -1 osd.10 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:47:31.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:31 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:31.530+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:31.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:31 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:31.530+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:31.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:31 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:31.530+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:31.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:31 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:31.530+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:31.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:31 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:31.530+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:31.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:31 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:31.530+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:32.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:31 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:31.834+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:32.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:31 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:31.834+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:32.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:31 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:31.834+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:32.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:31 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:31.834+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:32.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:31 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:31.834+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:32.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:31 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:31.834+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:01.214104+0000 front 2024-05-15T23:47:01.214223+0000 (oldest deadline 2024-05-15T23:47:24.114010+0000) 2024-05-15T23:47:32.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:32 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:32.529+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:32.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:32 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:32.529+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:32.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:32 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:32.529+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:32.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:32 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:32.529+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:32.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:32 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:32.529+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:32.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:32 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:32.529+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:32.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:32 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:32.529+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6814 osd.7 since back 2024-05-15T23:47:11.144302+0000 front 2024-05-15T23:47:11.144289+0000 (oldest deadline 2024-05-15T23:47:32.244107+0000) 2024-05-15T23:47:33.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:32 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:32.807+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:33.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:32 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:32.807+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:33.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:32 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:32.807+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:33.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:32 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:32.807+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:33.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:32 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:32.807+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:33.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:32 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:32.807+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:01.214104+0000 front 2024-05-15T23:47:01.214223+0000 (oldest deadline 2024-05-15T23:47:24.114010+0000) 2024-05-15T23:47:33.775 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:33 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:33.481+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:33.775 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:33 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:33.481+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:33.776 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:33 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:33.481+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:33.776 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:33 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:33.481+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:33.776 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:33 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:33.481+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:33.776 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:33 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:33.481+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:33.776 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:33 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:33.481+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6814 osd.7 since back 2024-05-15T23:47:11.144302+0000 front 2024-05-15T23:47:11.144289+0000 (oldest deadline 2024-05-15T23:47:32.244107+0000) 2024-05-15T23:47:34.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:33 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:33.775+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:34.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:33 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:33.775+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:34.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:33 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:33.775+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:34.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:33 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:33.775+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:34.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:33 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:33.775+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:34.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:33 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:33.775+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:01.214104+0000 front 2024-05-15T23:47:01.214223+0000 (oldest deadline 2024-05-15T23:47:24.114010+0000) 2024-05-15T23:47:34.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:34 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:34.439+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:34.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:34 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:34.439+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:34.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:34 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:34.439+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:34.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:34 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:34.439+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:34.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:34 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:34.439+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:34.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:34 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:34.439+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:34.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:34 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:34.439+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6814 osd.7 since back 2024-05-15T23:47:11.144302+0000 front 2024-05-15T23:47:11.144289+0000 (oldest deadline 2024-05-15T23:47:32.244107+0000) 2024-05-15T23:47:35.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:34 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:34.816+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:35.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:34 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:34.816+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:35.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:34 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:34.816+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:35.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:34 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:34.816+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:35.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:34 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:34.816+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:35.303 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:34 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:34.816+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:01.214104+0000 front 2024-05-15T23:47:01.214223+0000 (oldest deadline 2024-05-15T23:47:24.114010+0000) 2024-05-15T23:47:35.303 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:34 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:34.816+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6814 osd.7 since back 2024-05-15T23:47:11.115881+0000 front 2024-05-15T23:47:11.115934+0000 (oldest deadline 2024-05-15T23:47:34.615870+0000) 2024-05-15T23:47:35.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:35 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:35.447+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:35.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:35 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:35.447+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:35.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:35 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:35.447+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:35.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:35 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:35.447+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:35.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:35 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:35.447+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:35.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:35 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:35.447+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:35.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:35 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:35.447+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6814 osd.7 since back 2024-05-15T23:47:11.144302+0000 front 2024-05-15T23:47:11.144289+0000 (oldest deadline 2024-05-15T23:47:32.244107+0000) 2024-05-15T23:47:36.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:35 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:35.826+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:11.606382+0000 front 2024-05-15T23:46:11.606395+0000 (oldest deadline 2024-05-15T23:46:36.906262+0000) 2024-05-15T23:47:36.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:35 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:35.826+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:19.807379+0000 front 2024-05-15T23:46:19.807293+0000 (oldest deadline 2024-05-15T23:46:44.507408+0000) 2024-05-15T23:47:36.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:35 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:35.826+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:28.608409+0000 front 2024-05-15T23:46:28.608450+0000 (oldest deadline 2024-05-15T23:46:53.308355+0000) 2024-05-15T23:47:36.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:35 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:35.826+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.209550+0000 front 2024-05-15T23:46:36.209571+0000 (oldest deadline 2024-05-15T23:47:02.109514+0000) 2024-05-15T23:47:36.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:35 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:35.826+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:46.111394+0000 front 2024-05-15T23:46:46.111335+0000 (oldest deadline 2024-05-15T23:47:09.011221+0000) 2024-05-15T23:47:36.302 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:35 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:35.826+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:01.214104+0000 front 2024-05-15T23:47:01.214223+0000 (oldest deadline 2024-05-15T23:47:24.114010+0000) 2024-05-15T23:47:36.303 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:35 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10[64502]: 2024-05-15T23:47:35.826+0000 7efe7f110700 -1 osd.10 856 heartbeat_check: no reply from 172.21.15.88:6814 osd.7 since back 2024-05-15T23:47:11.115881+0000 front 2024-05-15T23:47:11.115934+0000 (oldest deadline 2024-05-15T23:47:34.615870+0000) 2024-05-15T23:47:36.725 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:36 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:36.401+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:36.725 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:36 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:36.401+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:36.726 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:36 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:36.401+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:36.726 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:36 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:36.401+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:36.726 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:36 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:36.401+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:36.726 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:36 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:36.401+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:36.726 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:36 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:36.401+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6814 osd.7 since back 2024-05-15T23:47:11.144302+0000 front 2024-05-15T23:47:11.144289+0000 (oldest deadline 2024-05-15T23:47:32.244107+0000) 2024-05-15T23:47:37.052 INFO:journalctl@ceph.osd.10.smithi090.stdout:May 15 23:47:36 smithi090 bash[179429]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-10 2024-05-15T23:47:37.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:37 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:37.425+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:37.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:37 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:37.425+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:37.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:37 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:37.425+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:37.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:37 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:37.425+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:37.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:37 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:37.425+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:37.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:37 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:37.425+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:37.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:37 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:37.425+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6814 osd.7 since back 2024-05-15T23:47:11.144302+0000 front 2024-05-15T23:47:11.144289+0000 (oldest deadline 2024-05-15T23:47:32.244107+0000) 2024-05-15T23:47:38.553 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.10.service' 2024-05-15T23:47:38.595 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:47:38.595 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-15T23:47:38.595 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-15T23:47:38.595 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.11 2024-05-15T23:47:38.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:38 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:38.435+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:38.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:38 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:38.435+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:38.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:38 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:38.435+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:38.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:38 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:38.435+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:38.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:38 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:38.435+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:38.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:38 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:38.435+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:38.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:38 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:38.435+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6814 osd.7 since back 2024-05-15T23:47:11.144302+0000 front 2024-05-15T23:47:11.144289+0000 (oldest deadline 2024-05-15T23:47:32.244107+0000) 2024-05-15T23:47:38.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:38 smithi090 systemd[1]: Stopping Ceph osd.11 for 3999fb7e-130e-11ef-bc99-c7b262605968... 2024-05-15T23:47:39.302 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:38 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:38.901+0000 7fe566bdf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T23:47:39.302 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:38 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:38.901+0000 7fe566bdf700 -1 osd.11 856 *** Got signal Terminated *** 2024-05-15T23:47:39.302 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:38 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:38.901+0000 7fe566bdf700 -1 osd.11 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T23:47:39.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:39 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:39.408+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:39.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:39 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:39.409+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:39.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:39 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:39.409+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:39.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:39 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:39.409+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:39.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:39 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:39.409+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:39.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:39 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:39.409+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:39.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:39 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:39.409+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6814 osd.7 since back 2024-05-15T23:47:11.144302+0000 front 2024-05-15T23:47:11.144289+0000 (oldest deadline 2024-05-15T23:47:32.244107+0000) 2024-05-15T23:47:40.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:40 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:40.400+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:40.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:40 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:40.400+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:40.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:40 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:40.400+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:40.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:40 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:40.400+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:40.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:40 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:40.400+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:40.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:40 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:40.400+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:40.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:40 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:40.400+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6814 osd.7 since back 2024-05-15T23:47:11.144302+0000 front 2024-05-15T23:47:11.144289+0000 (oldest deadline 2024-05-15T23:47:32.244107+0000) 2024-05-15T23:47:41.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:41 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:41.370+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:41.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:41 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:41.370+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:41.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:41 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:41.370+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:41.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:41 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:41.370+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:41.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:41 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:41.370+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:41.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:41 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:41.370+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:41.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:41 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:41.370+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6814 osd.7 since back 2024-05-15T23:47:11.144302+0000 front 2024-05-15T23:47:11.144289+0000 (oldest deadline 2024-05-15T23:47:32.244107+0000) 2024-05-15T23:47:42.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:42 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:42.339+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:42.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:42 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:42.339+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:42.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:42 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:42.339+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:42.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:42 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:42.339+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:42.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:42 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:42.339+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:42.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:42 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:42.339+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:42.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:42 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:42.339+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6814 osd.7 since back 2024-05-15T23:47:11.144302+0000 front 2024-05-15T23:47:11.144289+0000 (oldest deadline 2024-05-15T23:47:32.244107+0000) 2024-05-15T23:47:42.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:42 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:42.339+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.90:6802 osd.8 since back 2024-05-15T23:47:17.246580+0000 front 2024-05-15T23:47:17.246636+0000 (oldest deadline 2024-05-15T23:47:41.946616+0000) 2024-05-15T23:47:43.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:43 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:43.303+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-05-15T23:46:13.932901+0000 front 2024-05-15T23:46:13.932791+0000 (oldest deadline 2024-05-15T23:46:35.032972+0000) 2024-05-15T23:47:43.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:43 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:43.303+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-05-15T23:46:20.234425+0000 front 2024-05-15T23:46:20.234351+0000 (oldest deadline 2024-05-15T23:46:45.534517+0000) 2024-05-15T23:47:43.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:43 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:43.303+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-05-15T23:46:30.835873+0000 front 2024-05-15T23:46:30.835709+0000 (oldest deadline 2024-05-15T23:46:54.335594+0000) 2024-05-15T23:47:43.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:43 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:43.303+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-05-15T23:46:36.636993+0000 front 2024-05-15T23:46:36.636933+0000 (oldest deadline 2024-05-15T23:47:00.736837+0000) 2024-05-15T23:47:43.802 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:43 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:43.303+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6802 osd.4 since back 2024-05-15T23:46:47.340724+0000 front 2024-05-15T23:46:47.340599+0000 (oldest deadline 2024-05-15T23:47:10.240497+0000) 2024-05-15T23:47:43.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:43 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:43.303+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6810 osd.6 since back 2024-05-15T23:47:03.042819+0000 front 2024-05-15T23:47:03.042867+0000 (oldest deadline 2024-05-15T23:47:28.342787+0000) 2024-05-15T23:47:43.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:43 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:43.303+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.88:6814 osd.7 since back 2024-05-15T23:47:11.144302+0000 front 2024-05-15T23:47:11.144289+0000 (oldest deadline 2024-05-15T23:47:32.244107+0000) 2024-05-15T23:47:43.803 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:43 smithi090 ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11[71695]: 2024-05-15T23:47:43.303+0000 7fe55f1cb700 -1 osd.11 856 heartbeat_check: no reply from 172.21.15.90:6802 osd.8 since back 2024-05-15T23:47:17.246580+0000 front 2024-05-15T23:47:17.246636+0000 (oldest deadline 2024-05-15T23:47:41.946616+0000) 2024-05-15T23:47:44.947 INFO:journalctl@ceph.osd.11.smithi090.stdout:May 15 23:47:44 smithi090 bash[179689]: ceph-3999fb7e-130e-11ef-bc99-c7b262605968-osd-11 2024-05-15T23:47:46.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3999fb7e-130e-11ef-bc99-c7b262605968@osd.11.service' 2024-05-15T23:47:46.450 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T23:47:46.450 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-15T23:47:46.450 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 --force --keep-logs 2024-05-15T23:47:46.683 INFO:teuthology.orchestra.run.smithi084.stdout:Deleting cluster with fsid: 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T23:47:47.835 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 --force --keep-logs 2024-05-15T23:47:48.065 INFO:teuthology.orchestra.run.smithi088.stdout:Deleting cluster with fsid: 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T23:47:49.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 --force --keep-logs 2024-05-15T23:47:49.453 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T23:47:50.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T23:47:50.514 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T23:47:50.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T23:47:50.580 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-15T23:47:50.583 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754/remote/smithi084/crash 2024-05-15T23:47:50.583 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/crash -- . 2024-05-15T23:47:50.651 INFO:teuthology.orchestra.run.smithi084.stderr:tar: /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/crash: Cannot open: No such file or directory 2024-05-15T23:47:50.651 INFO:teuthology.orchestra.run.smithi084.stderr:tar: Error is not recoverable: exiting now 2024-05-15T23:47:50.653 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754/remote/smithi088/crash 2024-05-15T23:47:50.654 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/crash -- . 2024-05-15T23:47:50.715 INFO:teuthology.orchestra.run.smithi088.stderr:tar: /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/crash: Cannot open: No such file or directory 2024-05-15T23:47:50.715 INFO:teuthology.orchestra.run.smithi088.stderr:tar: Error is not recoverable: exiting now 2024-05-15T23:47:50.717 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754/remote/smithi090/crash 2024-05-15T23:47:50.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/crash -- . 2024-05-15T23:47:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/crash: Cannot open: No such file or directory 2024-05-15T23:47:50.788 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2024-05-15T23:47:50.788 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-15T23:47:50.789 DEBUG:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-05-15T23:47:50.900 INFO:teuthology.orchestra.run.smithi084.stdout:2024-05-15T23:10:00.003453+0000 mon.a (mon.0) 1128 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-05-15T23:47:50.900 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-05-15T23:47:50.900 DEBUG:teuthology.orchestra.run.smithi084:> sudo egrep '\[SEC\]' /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-05-15T23:47:50.939 DEBUG:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]' /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-05-15T23:47:51.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo egrep '\[WRN\]' /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-05-15T23:47:51.091 INFO:teuthology.orchestra.run.smithi084.stdout:2024-05-15T23:10:00.003453+0000 mon.a (mon.0) 1128 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-05-15T23:47:51.091 INFO:tasks.cephadm:Compressing logs... 2024-05-15T23:47:51.092 DEBUG:teuthology.orchestra.run.smithi084:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-15T23:47:51.134 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-15T23:47:51.136 DEBUG:teuthology.orchestra.run.smithi090:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-15T23:47:51.171 INFO:teuthology.orchestra.run.smithi088.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-15T23:47:51.171 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-15T23:47:51.174 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.6.log 2024-05-15T23:47:51.174 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-volume.log 2024-05-15T23:47:51.175 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.audit.log 2024-05-15T23:47:51.175 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.6.log: /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.cephadm.log 2024-05-15T23:47:51.176 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.7.log 2024-05-15T23:47:51.176 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-mgr.x.log 2024-05-15T23:47:51.176 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/objectstore_tool.7.log 2024-05-15T23:47:51.176 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-mon.b.log 2024-05-15T23:47:51.176 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.log 2024-05-15T23:47:51.177 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.4.log 2024-05-15T23:47:51.177 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.5.log 2024-05-15T23:47:51.183 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.4.log: /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.5.log: 93.0% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-mgr.x.log.gz 2024-05-15T23:47:51.185 INFO:teuthology.orchestra.run.smithi088.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-15T23:47:51.191 INFO:teuthology.orchestra.run.smithi084.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-15T23:47:51.192 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-15T23:47:51.204 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-volume.log 2024-05-15T23:47:51.205 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.2.log 2024-05-15T23:47:51.205 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-mgr.y.log 2024-05-15T23:47:51.205 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-volume.log: /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.audit.log 2024-05-15T23:47:51.205 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.cephadm.log 2024-05-15T23:47:51.205 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.3.log 2024-05-15T23:47:51.205 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.0.log 2024-05-15T23:47:51.205 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/objectstore_tool.8.log 2024-05-15T23:47:51.205 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/objectstore_tool.7.log 2024-05-15T23:47:51.206 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-mon.a.log 2024-05-15T23:47:51.206 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.log 2024-05-15T23:47:51.206 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.1.log 2024-05-15T23:47:51.214 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.3.log: /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.0.log: /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/objectstore_tool.8.log: /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/objectstore_tool.7.log: /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-mon.a.log: /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.log: /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.1.log: 94.4% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/objectstore_tool.8.log.gz 2024-05-15T23:47:51.217 INFO:teuthology.orchestra.run.smithi088.stderr: 82.3% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.cephadm.log.gz 2024-05-15T23:47:51.217 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- : No such file or directory/var/log/ceph/cephadm.log 2024-05-15T23:47:51.218 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-05-15T23:47:51.226 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.11.log 2024-05-15T23:47:51.226 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-volume.log 2024-05-15T23:47:51.226 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.audit.log 2024-05-15T23:47:51.226 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.cephadm.log 2024-05-15T23:47:51.226 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/objectstore_tool.6.log 2024-05-15T23:47:51.226 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.audit.log: /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.10.log 2024-05-15T23:47:51.226 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/objectstore_tool.6.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.8.log 2024-05-15T23:47:51.226 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/objectstore_tool.7.log 2024-05-15T23:47:51.226 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-mon.c.log 2024-05-15T23:47:51.227 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.9.log 2024-05-15T23:47:51.227 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.log 2024-05-15T23:47:51.227 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.9.log: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-15T23:47:51.227 INFO:teuthology.orchestra.run.smithi084.stderr: 87.6% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.log.gz 2024-05-15T23:47:51.307 INFO:teuthology.orchestra.run.smithi084.stderr: 96.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-15T23:47:51.324 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.log: 82.3% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.cephadm.log.gz 2024-05-15T23:47:51.387 INFO:teuthology.orchestra.run.smithi090.stderr: 89.6% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.audit.log.gz 2024-05-15T23:47:51.404 INFO:teuthology.orchestra.run.smithi090.stderr: 94.4% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/objectstore_tool.6.log.gz 2024-05-15T23:47:51.556 INFO:teuthology.orchestra.run.smithi088.stderr: 89.6% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.audit.log.gz 2024-05-15T23:47:51.561 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.log.gz 2024-05-15T23:47:51.639 INFO:teuthology.orchestra.run.smithi088.stderr: 91.3% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-volume.log.gz 2024-05-15T23:47:51.736 INFO:teuthology.orchestra.run.smithi088.stderr: 94.2% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/objectstore_tool.7.log.gz 2024-05-15T23:47:51.910 INFO:teuthology.orchestra.run.smithi084.stderr: 91.5% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-volume.log.gz 2024-05-15T23:47:51.935 INFO:teuthology.orchestra.run.smithi090.stderr: 91.4% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-volume.log.gz 2024-05-15T23:47:51.960 INFO:teuthology.orchestra.run.smithi084.stderr: 82.8% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.cephadm.log.gz 2024-05-15T23:47:52.025 INFO:teuthology.orchestra.run.smithi084.stderr: 89.6% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.audit.log.gz 2024-05-15T23:47:52.061 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph.log.gz 2024-05-15T23:47:52.270 INFO:teuthology.orchestra.run.smithi090.stderr: 94.4% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/objectstore_tool.7.log.gz 2024-05-15T23:47:52.463 INFO:teuthology.orchestra.run.smithi084.stderr: 94.6% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/objectstore_tool.7.log.gz 2024-05-15T23:47:52.477 INFO:teuthology.orchestra.run.smithi084.stderr: 91.3% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-mgr.y.log.gz 2024-05-15T23:47:54.024 INFO:teuthology.orchestra.run.smithi088.stderr: 90.8% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-mon.b.log.gz 2024-05-15T23:47:57.597 INFO:teuthology.orchestra.run.smithi090.stderr: 90.6% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-mon.c.log.gz 2024-05-15T23:47:59.364 INFO:teuthology.orchestra.run.smithi084.stderr: 89.8% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-mon.a.log.gz 2024-05-15T23:48:53.478 INFO:teuthology.orchestra.run.smithi088.stderr: 94.3% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.6.log.gz 2024-05-15T23:49:11.235 INFO:teuthology.orchestra.run.smithi084.stderr: 94.0% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.0.log.gz 2024-05-15T23:49:51.945 INFO:teuthology.orchestra.run.smithi088.stderr: 94.2% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.4.log.gz 2024-05-15T23:50:47.495 INFO:teuthology.orchestra.run.smithi088.stderr: 93.1% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.7.log.gz 2024-05-15T23:50:53.754 INFO:teuthology.orchestra.run.smithi084.stderr: 92.0% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.2.log.gz 2024-05-15T23:51:14.353 INFO:teuthology.orchestra.run.smithi088.stderr: 92.6% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.5.log.gz 2024-05-15T23:51:14.355 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T23:51:14.355 INFO:teuthology.orchestra.run.smithi088.stderr:real 3m23.200s 2024-05-15T23:51:14.355 INFO:teuthology.orchestra.run.smithi088.stderr:user 5m3.272s 2024-05-15T23:51:14.355 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m12.727s 2024-05-15T23:52:09.552 INFO:teuthology.orchestra.run.smithi084.stderr: 93.1% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.1.log.gz 2024-05-15T23:52:12.330 INFO:teuthology.orchestra.run.smithi084.stderr: 93.2% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.3.log.gz 2024-05-15T23:52:12.332 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-05-15T23:52:12.332 INFO:teuthology.orchestra.run.smithi084.stderr:real 4m21.183s 2024-05-15T23:52:12.332 INFO:teuthology.orchestra.run.smithi084.stderr:user 5m59.309s 2024-05-15T23:52:12.332 INFO:teuthology.orchestra.run.smithi084.stderr:sys 0m14.889s 2024-05-15T23:53:08.930 INFO:teuthology.orchestra.run.smithi090.stderr: 92.7% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.8.log.gz 2024-05-15T23:53:45.088 INFO:teuthology.orchestra.run.smithi090.stderr: 94.0% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.9.log.gz 2024-05-15T23:54:56.903 INFO:teuthology.orchestra.run.smithi090.stderr: 91.9% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.10.log.gz 2024-05-15T23:54:58.446 INFO:teuthology.orchestra.run.smithi090.stderr: 93.7% -- replaced with /var/log/ceph/3999fb7e-130e-11ef-bc99-c7b262605968/ceph-osd.11.log.gz 2024-05-15T23:54:58.448 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-05-15T23:54:58.448 INFO:teuthology.orchestra.run.smithi090.stderr:real 7m7.293s 2024-05-15T23:54:58.448 INFO:teuthology.orchestra.run.smithi090.stderr:user 6m42.363s 2024-05-15T23:54:58.448 INFO:teuthology.orchestra.run.smithi090.stderr:sys 0m16.711s 2024-05-15T23:54:58.448 INFO:tasks.cephadm:Archiving logs... 2024-05-15T23:54:58.449 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754/remote/smithi084/log 2024-05-15T23:54:58.450 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-15T23:55:59.955 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754/remote/smithi088/log 2024-05-15T23:55:59.956 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-15T23:56:51.551 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754/remote/smithi090/log 2024-05-15T23:56:51.553 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-15T23:58:00.777 INFO:tasks.cephadm:Removing cluster... 2024-05-15T23:58:00.778 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 --force 2024-05-15T23:58:01.111 INFO:teuthology.orchestra.run.smithi084.stdout:Deleting cluster with fsid: 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T23:58:01.758 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 --force 2024-05-15T23:58:02.187 INFO:teuthology.orchestra.run.smithi088.stdout:Deleting cluster with fsid: 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T23:58:02.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3999fb7e-130e-11ef-bc99-c7b262605968 --force 2024-05-15T23:58:03.204 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: 3999fb7e-130e-11ef-bc99-c7b262605968 2024-05-15T23:58:03.894 INFO:tasks.cephadm:Removing cephadm ... 2024-05-15T23:58:03.894 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-15T23:58:03.926 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-15T23:58:04.092 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-15T23:58:04.116 INFO:tasks.cephadm:Teardown complete 2024-05-15T23:58:04.117 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-15T23:58:04.126 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-15T23:58:04.126 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-15T23:58:04.129 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-15T23:58:04.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-15T23:58:04.298 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-15T23:58:04.308 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-15T23:58:04.308 DEBUG:teuthology.orchestra.run.smithi084:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T23:58:04.311 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T23:58:04.313 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T23:58:04.328 INFO:teuthology.orchestra.run.smithi084.stderr:bash: ntpq: command not found 2024-05-15T23:58:04.330 INFO:teuthology.orchestra.run.smithi090.stderr:bash: ntpq: command not found 2024-05-15T23:58:04.330 INFO:teuthology.orchestra.run.smithi088.stderr:bash: ntpq: command not found 2024-05-15T23:58:04.348 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T23:58:04.348 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-05-15T23:58:04.348 INFO:teuthology.orchestra.run.smithi090.stdout:^* hv01.front.sepia.ceph.com 3 6 377 40 +1329us[+1316us] +/- 50ms 2024-05-15T23:58:04.348 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 49 -1381us[-1394us] +/- 55ms 2024-05-15T23:58:04.349 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 -189us[ -203us] +/- 75ms 2024-05-15T23:58:04.349 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T23:58:04.363 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T23:58:04.363 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-05-15T23:58:04.363 INFO:teuthology.orchestra.run.smithi088.stdout:^* hv01.front.sepia.ceph.com 3 7 377 112 +1345us[+1350us] +/- 49ms 2024-05-15T23:58:04.363 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 111 -1392us[-1392us] +/- 54ms 2024-05-15T23:58:04.363 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 111 -189us[ -189us] +/- 74ms 2024-05-15T23:58:04.363 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T23:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T23:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-05-15T23:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout:^* hv01.front.sepia.ceph.com 3 6 377 44 +1414us[+1415us] +/- 50ms 2024-05-15T23:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 47 -1326us[-1326us] +/- 55ms 2024-05-15T23:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 -131us[ -131us] +/- 75ms 2024-05-15T23:58:04.392 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T23:58:04.393 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-15T23:58:04.403 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-15T23:58:04.403 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-15T23:58:04.412 DEBUG:teuthology.orchestra.run.smithi084:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-15T23:58:04.537 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-15T23:58:04.714 DEBUG:teuthology.orchestra.run.smithi090:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-15T23:58:04.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-05-15T23:58:04.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T23:58:04.971 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-05-15T23:58:05.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T23:58:05.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-05-15T23:58:05.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T23:58:05.041 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-15T23:58:05.051 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-15T23:58:05.074 INFO:teuthology.task.internal:Duration was 4423.187871 seconds 2024-05-15T23:58:05.074 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-15T23:58:05.082 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-15T23:58:05.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-15T23:58:05.085 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-15T23:58:05.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-15T23:58:05.170 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T23:58:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T23:58:05.186 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T23:58:05.407 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-15T23:58:05.407 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi084.front.sepia.ceph.com 2024-05-15T23:58:05.408 DEBUG:teuthology.orchestra.run.smithi084:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-15T23:58:05.447 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2024-05-15T23:58:05.447 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-15T23:58:05.480 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2024-05-15T23:58:05.481 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-15T23:58:05.519 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-15T23:58:05.520 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-15T23:58:05.522 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-15T23:58:05.525 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-15T23:58:05.695 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-15T23:58:05.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-15T23:58:05.697 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-15T23:58:05.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-15T23:58:05.955 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-15T23:58:05.981 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-15T23:58:05.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-15T23:58:06.050 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-15T23:58:06.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-15T23:58:06.174 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-15T23:58:06.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-15T23:58:06.242 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-15T23:58:06.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-15T23:58:06.271 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = core 2024-05-15T23:58:06.286 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2024-05-15T23:58:06.306 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2024-05-15T23:58:06.320 DEBUG:teuthology.orchestra.run.smithi084:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-15T23:58:06.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T23:58:06.349 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-15T23:58:06.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T23:58:06.367 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-15T23:58:06.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T23:58:06.384 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-15T23:58:06.409 INFO:teuthology.task.internal:Transferring archived files... 2024-05-15T23:58:06.410 DEBUG:teuthology.misc:Transferring archived files from smithi084:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754/remote/smithi084 2024-05-15T23:58:06.411 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-15T23:58:06.502 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754/remote/smithi088 2024-05-15T23:58:06.502 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-15T23:58:06.626 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754/remote/smithi090 2024-05-15T23:58:06.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-15T23:58:06.714 INFO:teuthology.task.internal:Removing archive directory... 2024-05-15T23:58:06.714 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-15T23:58:06.717 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-15T23:58:06.722 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-15T23:58:06.744 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-15T23:58:06.860 INFO:teuthology.task.internal:Not uploading archives. 2024-05-15T23:58:06.860 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-15T23:58:07.055 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-15T23:58:07.055 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-15T23:58:07.058 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-15T23:58:07.060 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-15T23:58:07.074 INFO:teuthology.orchestra.run.smithi084.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 15 23:58 /home/ubuntu/cephtest 2024-05-15T23:58:07.076 INFO:teuthology.orchestra.run.smithi088.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 15 23:58 /home/ubuntu/cephtest 2024-05-15T23:58:07.078 INFO:teuthology.orchestra.run.smithi090.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 15 23:58 /home/ubuntu/cephtest 2024-05-15T23:58:07.079 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-15T23:58:07.121 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-15T23:58:07.144 INFO:teuthology.nuke:Checking targets against current locks 2024-05-15T23:58:07.172 DEBUG:teuthology.nuke:shortname: smithi084 2024-05-15T23:58:07.172 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T23:58:07.198 DEBUG:teuthology.nuke:shortname: smithi088 2024-05-15T23:58:07.198 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T23:58:07.203 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 22:35:06.053221', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK9Nf4cHWUuLl3T7vPa3UepgO9fpmPaqZFc3jDJWWFvigOE6SkIM8vIpDYoXQqZHWdFJb7CUlq6DSbADMftJz+M='} 2024-05-15T23:58:07.222 DEBUG:teuthology.nuke:shortname: smithi090 2024-05-15T23:58:07.222 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T23:58:07.227 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 22:35:06.055009', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNpKE5Nb7ypAPxSuFMWe0Q/sPYGfgEw475HpNeOGwj8Vrg7jwUodI/qj+KqvcUcPn00OO479Zk1hcgfbtfztS5g='} 2024-05-15T23:58:07.250 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-15_21:09:29-rados-wip-yuri5-testing-2024-05-15-0804-distro-default-smithi/7707754', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 22:35:06.051182', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKVJiiXiSPIa49WcVsk5RBTt8Yy+lKpFPg7I+1ipH/OgvGENv9W5uCOj4qcxk+NZJSuHPWyqc1PfUMkCqacxr0k='} 2024-05-15T23:58:07.259 INFO:teuthology.orchestra.console.smithi084:Power off 2024-05-15T23:58:07.260 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-15T23:58:07.282 DEBUG:teuthology.orchestra.console.smithi084:power off output: Chassis Power Control: Down/Off 2024-05-15T23:58:07.282 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T23:58:07.299 INFO:teuthology.orchestra.console.smithi088:Power off 2024-05-15T23:58:07.299 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-15T23:58:07.314 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-05-15T23:58:07.327 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off 2024-05-15T23:58:07.327 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T23:58:07.342 INFO:teuthology.orchestra.console.smithi090:Power off 2024-05-15T23:58:07.342 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-15T23:58:07.357 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-05-15T23:58:07.366 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off 2024-05-15T23:58:07.366 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T23:58:07.390 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-05-15T23:58:11.315 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T23:58:11.358 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T23:58:11.391 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T23:58:11.426 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-05-15T23:58:11.468 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-05-15T23:58:11.501 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-05-15T23:58:15.431 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T23:58:15.469 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T23:58:15.502 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T23:58:15.542 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-05-15T23:58:15.579 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off 2024-05-15T23:58:15.613 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-05-15T23:58:15.680 INFO:teuthology.orchestra.console.smithi088:Power off completed 2024-05-15T23:58:19.544 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T23:58:19.614 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T23:58:19.656 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off 2024-05-15T23:58:19.725 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off 2024-05-15T23:58:19.758 INFO:teuthology.orchestra.console.smithi084:Power off completed 2024-05-15T23:58:19.826 INFO:teuthology.orchestra.console.smithi090:Power off completed 2024-05-15T23:58:19.931 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} duration: 4423.187871456146 failure_reason: '"2024-05-15T23:10:00.003453+0000 mon.a (mon.0) 1128 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2024-05-15T23:58:19.931 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-15T23:58:20.013 INFO:teuthology.run:FAIL